I'm trying to export data from queryset to different file formats and I'm getting an issue with json format.
This is my little piece of code :
from django.core import serializers
def export_categories_json(request):
with open("categories.json", "w") as out:
data = serializers.serialize("json", Category.objects.all().values_list('id', 'name'))
out.write(data)
Then, I set in my template a button which call this function and should download the json file.
But I have this issue :
Traceback:
File "/home/val/.pyenv/versions/Publication3.6.2/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
41. response = get_response(request)
File "/home/val/.pyenv/versions/Publication3.6.2/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
187. response = self.process_exception_by_middleware(e, request)
File "/home/val/.pyenv/versions/Publication3.6.2/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
185. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/val/Bureau/Projets/Publication/publication/src/web/views/exports.py" in export_categories_json
276. data = serializers.serialize("json", Category.objects.all().values_list('id', 'name'))
File "/home/val/.pyenv/versions/Publication3.6.2/lib/python3.6/site-packages/django/core/serializers/__init__.py" in serialize
129. s.serialize(queryset, **options)
File "/home/val/.pyenv/versions/Publication3.6.2/lib/python3.6/site-packages/django/core/serializers/base.py" in serialize
84. concrete_model = obj._meta.concrete_model
Exception Type: AttributeError at /Category/json
Exception Value: 'tuple' object has no attribute '_meta'
Something is wrong in my code ?
I can't send a queryset to json file ?
EDIT :
I wrote this and it works, but json is shown in my template. How I can export this one to a json file ?
def export_categories_json(request):
from django.http import JsonResponse
data = list(Category.objects.values())
return JsonResponse(data, safe=False)
Please do not do serialization yourself: Django has some builtin serializiation functionality, and you can subclass a serializer to change its behavior.
Your view also does not return a HTTP response, but this is a contract it should satisfy (well it should return a HTTP response, or it should raise some error).
Instead you write content to a file, but writing to files is typically not a good idea (unless you expect the filesize to be huge, in which case you can use a temporary file). By using files, you create race conditions, a hacker might also aim to "inject" a different filename and thus overwrting certain files to run arbitrary code, or changing credentials, and finally it is possible that the server has certain permissions making it impossible to write to a file (the permissions of the directory).
Django allows you to see a HTTP response a s a stream object, to which content can be written, like:
from django.http import HttpResponse
from django.core import serializers
def export_categories_json(request):
response = new HttpResponse(content_type='application/json')
response['Content-Disposition'] = 'attachment;filename=categories.json'
serializers.serialize(
'json',
Category.objects.all(),
fields=['name'],
stream=response
)
return response
Django's serialization is for models, but you are using .values_list() which returns plain Python lists.
In your specific case, you can simply use the built-in json module:
import json
def export_categories_json(request):
with open("categories.json", "w") as out:
values = list(Category.objects.all().values_list('id', 'name'))
json.dump(values, out)
django serialize except the querset object ,
Category.objects.all().values_list('id', 'name')
this will return the tuple, you can replace with below
data = list(Category.objects.all().values('id', 'name'))
with open("file.json", "w+") as file:
file.write(data)
If you want to write the json representation of something to a file, use json.dump:
import json
from django.shortcuts import redirect
def export_to_json(request):
with open('export.json', 'w') as f:
json.dump(list(Category.objects.all().values_list('id', 'name')), f)
return redirect('/')
Related
I have setup Django REST framework endpoint that allows me to upload a csv file.
The serializers.py looks like this:
from rest_framework import serializers
class UploadSerializer(serializers.Serializer):
file_uploaded = serializers.FileField()
class Meta:
fields = ['file_uploaded']
In my views.py file, I'm trying to read data from uploaded csv like this:
class UploadViewSet(viewsets.ViewSet):
serializer_class = UploadSerializer
def create(self, request):
file_uploaded = request.FILES.get('file_uploaded')
with open(file_uploaded, mode ='r')as file:
csvFile = csv.reader(file)
for lines in csvFile:
print(lines)
I'm getting the following error:
... line 37, in create
with open(file_uploaded, mode ='r') as file:
TypeError: expected str, bytes or os.PathLike object, not InMemoryUploadedFile
I have checked type() of file_uploaded and It is <class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
How can I read this file into dictionary or dataframe so I can extract the data I need from it?
When you do request.FILES.get('file_uploaded') it returns back an InMemoryUploadedFile which is a wrapper around a file object. You can access the file object using the file attribute.
file_uploaded # <InMemoryUploadedFile: xxx (xxx/xxx)>
file_object = file_uploaded.file
This file_object can then be opened.
an InMemoryFileObject can be used pretty much in the same way as an open file., so you don't need to open it.
def create(self, request):
file_upload = request.FILES.get("file_uploaded")
csvFile = csv.reader(file_upload)
for line in csvFile:
print(line)
This has some good information on file handling in django.
https://docs.djangoproject.com/en/4.1/topics/http/file-uploads/#handling-uploaded-files-with-a-model
Trying to make endpoint that can read uploaded .sav (SPSS) file and create model with data from it. For getting data from it I'm using pyreadstat library. But now when I'm trying to run it I have an error expected str, bytes or os.PathLike object, not InMemoryUploadedFile
How I can change this code so pyreadstat can correctly read the given file?
from rest_framework import generics, status
import pandas as pd
import pyreadstat
from rest_framework.response import Response
from .models import Research, Data
from .serializers import FileUploadSerializer, SaveFileSerializer
class UploadFileView(generics.CreateAPIView):
serializer_class = FileUploadSerializer
def post(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
file = serializer.validated_data['file']
df, meta = pyreadstat.read_sav(file, user_missing=True)
json_df = df.to_json(orient='table')
rsch = Research.objects.get(pk=1)
Data.objects.create(research=rsch, data={})
Change this:
pyreadstat.read_sav(file, user_missing=True)
To this:
pyreadstat.read_sav(file.getvalue(), user_missing=True)
Hope it works.
Django uploaded files are mostly file like, and they only proxy certain methods. See the FileProxyMixin source for more details on specific methods, if you are curious.
To get the uploaded contents, use the read() method:
file = serializer.validated_data['file']
contents = file.read()
df, meta = pyreadstat.read_sav(contents, user_missing=True)
In this case you are getting an InMemoryUploadFile but this could change if Django decides to write to disk due to the upload being too large.
Documentation on django uploaded file objects:
https://docs.djangoproject.com/en/4.0/ref/files/uploads/#.
I have this serializer which I am trying to test:
class AttachmentSerializer(CustomModelSerializer):
order = serializers.PrimaryKeyRelatedField()
file = FileField()
class Meta:
model = Attachment
fields = (
'id',
'order',
'name',
'file_type',
'file',
'created_at',
)
My test simply checks whether it is valid or not:
def test_serializer_create(self):
self.data = {
'order': self.order.pk,
'name': 'sample_name',
'file_type': 'image',
'created_at': datetime.now(),
'file': open(self.image_path, 'rb').read()
}
serializer = AttachmentSerializer(data=self.data)
self.assertTrue(serializer.is_valid())
And I am constantly getting this error:
{'file': ['No file was submitted. Check the encoding type on the form.']}
I tried to create a file in a number of different ways, such as with StringIO/BytesIO, File and etc. to no avail.
What might be wrong?
from django.core.files.uploadedfile import SimpleUploadedFile
content = SimpleUploadedFile("file.txt", "filecontentstring")
data = {'content': content}
try smth like that , because if you check FileField serializer's code - it expects UploadedFile that should have name and size:
def to_internal_value(self, data):
try:
# `UploadedFile` objects should have name and size attributes.
file_name = data.name
file_size = data.size
except AttributeError:
self.fail('invalid')
and StringIO or opened file objects doesn't have size attribute.
I ran into a similar problem. It turned out that Django REST Framework FileField cannot be used with a JSON API parser. DRF documentation states that "Most parsers, such as e.g. JSON don't support file uploads."
Your question does not show which parser you configured, but given how common JSON is, it may be the culprit. You can set a different parser either across the board, or for a specific API view, as described here.
Once the parser problem was fixed, I made the test work with a Django File, but perhaps other approaches could work too:
from django.core.files import File
def test_create(self):
...
data = {
'file': File(open(path_to_test_file, 'rb')),
}
...
The thing is that you pass an opened file to the APIClient / APIRequestFactory, not to the view itself. The Django request will wraps the file to an UploadedFile which is what you should use.
I have a django app which was running on 1.4.2 version and working completely fine, but recently i updated it to django 1.6.5 and facing some wierd errors like below
Actually i am getting this during user/client registration process in my site functionality
Request URL: http://example.com/client/registration/
Django Version: 1.6.5
Exception Type: TypeError
Exception Value: <Client: test one> is not JSON serializable
Exception Location: /usr/lib/python2.7/json/encoder.py in default, line 184
Python Executable: /home/user/.virtualenvs/test_proj/bin/python
Python Version: 2.7.5
traceback
Traceback:
File "/home/user/.virtualenvs/test_proj/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
199. response = middleware_method(request, response)
File "/home/user/.virtualenvs/test_proj/local/lib/python2.7/site-packages/django/contrib/sessions/middleware.py" in process_response
38. request.session.save()
File "/home/user/.virtualenvs/test_proj/local/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py" in save
57. session_data=self.encode(self._get_session(no_load=must_create)),
File "/home/user/.virtualenvs/test_proj/local/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py" in encode
87. serialized = self.serializer().dumps(session_dict)
File "/home/user/.virtualenvs/test_proj/local/lib/python2.7/site-packages/django/core/signing.py" in dumps
88. return json.dumps(obj, separators=(',', ':')).encode('latin-1')
File "/usr/lib/python2.7/json/__init__.py" in dumps
250. sort_keys=sort_keys, **kw).encode(obj)
File "/usr/lib/python2.7/json/encoder.py" in encode
207. chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py" in iterencode
270. return _iterencode(o, 0)
File "/usr/lib/python2.7/json/encoder.py" in default
184. raise TypeError(repr(o) + " is not JSON serializable")
Exception Type: TypeError at /client/registration/
Exception Value: <Client: test one> is not JSON serializable
I am confused on why the above json error has been appearing after updation and by the way i am using a customized json field in some of my models as below
proj/utils.py
from django.db import models
from django.utils import simplejson as json
from django.core.serializers.json import DjangoJSONEncoder
class JSONField(models.TextField):
'''JSONField is a generic textfield that neatly serializes/unserializes
JSON objects seamlessly'''
# Used so to_python() is called
__metaclass__ = models.SubfieldBase
def to_python(self, value):
'''Convert our string value to JSON after we load it from the DB'''
if value == '':
return None
try:
if isinstance(value, basestring):
return json.loads(value)
except ValueError:
pass
return value
def get_db_prep_save(self, value, connection=None):
'''Convert our JSON object to a string before we save'''
if not value or value == '':
return None
if isinstance(value, (dict, list)):
value = json.dumps(value, mimetype="application/json")
return super(JSONField, self).get_db_prep_save(value, connection=connection)
from south.modelsinspector import add_introspection_rules
add_introspection_rules([], ["^proj\.util\.jsonfield\.JSONField"])
settings.py
SERIALIZATION_MODULES = {
'custom_json': 'proj.util.json_serializer',
}
json_serializer.py
from django.core.serializers.json import Serializer as JSONSerializer
from django.utils.encoding import is_protected_type
# JSONFields that are normally incorrectly serialized as strings
json_fields = ['field_1', 'field_2']
class Serializer(JSONSerializer):
"""
A fix on JSONSerializer in order to prevent stringifying JSONField data.
"""
def handle_field(self, obj, field):
value = field._get_val_from_obj(obj)
# Protected types (i.e., primitives like None, numbers, dates,
# and Decimals) are passed through as is. All other values are
# converted to string first.
if is_protected_type(value) or field.name in json_fields:
self._current[field.name] = value
else:
self._current[field.name] = field.value_to_string(obj)
So how to solve the above error ? can some one give me an explanation of what happening to cause the error ?
Django 1.6 changed the serializer from pickle to json. pickle can serialize things that json can't.
You can change the value of SESSION_SERIALIZER in your settings.py to get back the behaviour from Django before version 1.6.
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer'
You might want to read about session serialization in the documentation.
Setting this line to settings.py will clear the error when upgraded to django 1.6 version
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer'
After analyzing the traceback, it seems that the JSONEncoder can not serialize the instance of your Client model. Generally, you got such error if you try to serialize a model related to other models (Many2ManyField, etc.) using json or simplejson libraries.
See this https://docs.djangoproject.com/en/dev/topics/serialization/, You can also use some 3rd-party packages such as DjangoRestFramework depending on your needs.
I am a bit confused and I need some help.
I am displaying my objects using ModelFormset, then I am dynamically removing them using Ajax and then saving all of the objects again also using Ajax call. Everything is dynamic and the page is not reloaded at any time.
The problem is that when Django tries to save the whole formset using Ajax alfter an object or two has been deleted, it looks for the deleted object(s) and raises an IndexError: list index out of range, because the object(s) isn't at the queryset anymore.
This is how I am displaying and saving the formsets (simplified version - I think this is where the error comes from):
def App(request, slug):
TopicFormSet = modelformset_factory(Topic, form=TopicForm, extra=0, fields=('name',), can_delete=True)
SummaryFormSet = modelformset_factory(Summary, form=SummaryForm, extra=0, fields=('content',), can_delete=True)
tquery = user.topic_set.all().order_by('date')
squery = user.summary_set.all().order_by('date')
# saving formsets:
if request.method == 'POST' and request.is_ajax():
# the following two lines is where the error comes from:
t_formset = TopicFormSet(request.POST) # formset instance
s_formset = SummaryFormSet(request.POST) # formset instance
s_formset.save()
t_formset.save()
return render (blah...)
This is how I am removing objects (this is a different view):
def Remove_topic(request, slug, id):
topic = Topic.objects.get(pk=id)
summary = Summary.objects.get(topic = topic) # foreign key relatonship
topic.delete()
summary.delete()
# Ajax stuff....
if request.is_ajax():
return HttpResponse('blah..')
I have tried placing queryset = tquery and queryset = squery when instantiating t_formset and s_formset, but it didn't help. What should I do ? I am using Postgres db if that's useful.
The error:
> File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/home/eimantas/Desktop/Projects/Lynx/lynx/views.py", line 122, in App
t_formset = TopicFormSet(request.POST, queryset = tquery)
File "/usr/local/lib/python2.7/dist-packages/django/forms/models.py", line 441, in __init__
super(BaseModelFormSet, self).__init__(**defaults)
File "/usr/local/lib/python2.7/dist-packages/django/forms/formsets.py", line 56, in __init__
self._construct_forms()
File "/usr/local/lib/python2.7/dist-packages/django/forms/formsets.py", line 124, in _construct_forms
self.forms.append(self._construct_form(i))
File "/usr/local/lib/python2.7/dist-packages/django/forms/models.py", line 468, in _construct_form
kwargs['instance'] = self.get_queryset()[i]
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 198, in __getitem__
return self._result_cache[k]
IndexError: list index out of range
This may be a case of a cascaded delete that is already deleting the summary object:
When an object referenced by a ForeignKey is deleted, Django by
default emulates the behavior of the SQL constraint ON DELETE CASCADE
and also deletes the object containing the ForeignKey.
https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.ForeignKey.on_delete
It has nothing to do with second view and Ajax calls. I think that You have messed up management form's fields. Like initial_form_count, total_form_count or something similar.
Another important point. Do not save formset before checking if it is valid:
t_formset = TopicFormSet(request.POST)
if t_formset.is_valid():
t_formset.save()
In G+ group I was adviced that technically it is possible to reset or "reload" the Queryset, but it would be very difficult to maintain "at all levels" and probably would give no benefit. I was adviced to use iteration and check if each object has been saved successfully when saving the formset forms (I would have to overwrite form = TopicForm's and form = SummaryForm's save() method.)
I decided not to use formsets at all, but to list and save each object individually, it will be better for me and my app's business logic.