I have this serializer which I am trying to test:
class AttachmentSerializer(CustomModelSerializer):
order = serializers.PrimaryKeyRelatedField()
file = FileField()
class Meta:
model = Attachment
fields = (
'id',
'order',
'name',
'file_type',
'file',
'created_at',
)
My test simply checks whether it is valid or not:
def test_serializer_create(self):
self.data = {
'order': self.order.pk,
'name': 'sample_name',
'file_type': 'image',
'created_at': datetime.now(),
'file': open(self.image_path, 'rb').read()
}
serializer = AttachmentSerializer(data=self.data)
self.assertTrue(serializer.is_valid())
And I am constantly getting this error:
{'file': ['No file was submitted. Check the encoding type on the form.']}
I tried to create a file in a number of different ways, such as with StringIO/BytesIO, File and etc. to no avail.
What might be wrong?
from django.core.files.uploadedfile import SimpleUploadedFile
content = SimpleUploadedFile("file.txt", "filecontentstring")
data = {'content': content}
try smth like that , because if you check FileField serializer's code - it expects UploadedFile that should have name and size:
def to_internal_value(self, data):
try:
# `UploadedFile` objects should have name and size attributes.
file_name = data.name
file_size = data.size
except AttributeError:
self.fail('invalid')
and StringIO or opened file objects doesn't have size attribute.
I ran into a similar problem. It turned out that Django REST Framework FileField cannot be used with a JSON API parser. DRF documentation states that "Most parsers, such as e.g. JSON don't support file uploads."
Your question does not show which parser you configured, but given how common JSON is, it may be the culprit. You can set a different parser either across the board, or for a specific API view, as described here.
Once the parser problem was fixed, I made the test work with a Django File, but perhaps other approaches could work too:
from django.core.files import File
def test_create(self):
...
data = {
'file': File(open(path_to_test_file, 'rb')),
}
...
The thing is that you pass an opened file to the APIClient / APIRequestFactory, not to the view itself. The Django request will wraps the file to an UploadedFile which is what you should use.
Related
I generate a file in python, and want to "upload" that file to the django database. This way it is automatically put inside the media folder, and organized neatly with all other files of my application.
Now here is what I tried: (type hinting used, since it's python 3.6)
# forms.py
class UploadForm(forms.ModelForm):
class Meta:
model = UploadedFile
fields = ('document',)
# models.py
class UploadedFile(models.Model):
document = models.FileField(upload_to=get_upload_path)
# mimetype is generated by filename on save
mimetype = models.CharField(max_length=255)
# ... additional fields like temporary
def get_upload_path(instance: UploadedFile, filename):
if instance.temporary:
return "uploaded_files/temp/" + filename
return "uploaded_files/" + filename
# views.py, file_out has been generated
with open(file_out, 'rb') as local_file:
from django.core.files import File
form = UploadForm(dict(), {'document': File(local_file)})
print(form.errors)
if form.is_valid():
file = form.save(commit=False)
# ... set additional fields
file.save()
form.save_m2m()
return file
Now this is not the only thing I've tried. First I've gone with setting the FileField directly, but that resulted in the save() to fail, while the mimetype field is set. Because the original file sits outside the media folder, and thus a suspicious file action is triggered.
Also, the form gives some feedback about the "upload", through the form.errors.
Depending on my approach, either the save() fails as mentioned above -- meaning the "uploading" does not actually copy the file in the media folder -- or the form returns the error that no file was transmitted, and tells to check the form protocol.
Now my theory is, that I would have to go and initialize my own instance of InMemoryUploadedFile, but I could not figure out how to do that myself, and no documentation was available on the internet.
It feels like I'm taking the wrong approach from the get go. How would one do this properly?
Do you have get_upload_path defined? If not, that would explain the errors you're getting.
From what I can see you're on the right track. If you don't need a dynamic path for your uploads, if you just want them in media/uploads, you can pass in a string value for upload_to (from the Django docs):
# file will be uploaded to MEDIA_ROOT/uploads
document = models.FileField(upload_to='uploads/')
First of all, thanks to Franey for pointing me at storage documentation which lead me to contentfile documentation.
The ContentFile actually solves the problem, because it basically is the self-instantiated version of InMemoryUploadedFile that I was looking for. It's a django File that is not stored on disk.
Here's the full solution:
# views.py, file_out has been generated
with open(file_out, 'rb') as local_file:
from django.core.files.base import ContentFile
# we need to provide a name. Otherwise the Storage.save
# method reveives a None-parameter and breaks.
form = UploadForm(dict(), {'document': ContentFile(local_file.read(), name=name)})
if form.is_valid():
file = form.save(commit=False)
# ... set additional fields
file.save()
form.save_m2m()
return file
I have a Django model that saves filename as "uuid4().pdf". Where uuid4 generates a random uuid for each instance created. This file name is also stored on the amazon s3 server with the same name.
I am trying to add a custom disposition for filename that i upload to amazon s3, this is because i want to see a custom name whenever i download the file not the uuid one. At the same time, i want the files to stored on s3 with the uuid filename.
So, I am using django-storages with python 2.7. I have tried adding content_disposition in settings like this:
AWS_CONTENT_DISPOSITION = 'core.utils.s3.get_file_name'
where get_file_name() returns the filename.
I have also tried adding this to the settings:
AWS_HEADERS = {
'Content-Disposition': 'attachments; filename="%s"'% get_file_name(),
}
no luck!
Do anyone of you know to implement this.
Current version of S3Boto3Storage from django-storages supports AWS_S3_OBJECT_PARAMETERS global settings variable, which allows modify ContentDisposition too. But the problem is that it is applied as is to all objects that are uploaded to s3 and, moreover, affects all models working with the storage, which may turn to be not the expected result.
The following hack worked for me.
from storages.backends.s3boto3 import S3Boto3Storage
class DownloadableS3Boto3Storage(S3Boto3Storage):
def _save_content(self, obj, content, parameters):
"""
The method is called by the storage for every file being uploaded to S3.
Below we take care of setting proper ContentDisposition header for
the file.
"""
filename = obj.key.split('/')[-1]
parameters.update({'ContentDisposition': f'attachment; filename="{filename}"'})
return super()._save_content(obj, content, parameters)
Here we override native save method of the storage object and make sure proper content disposition is set on each file.
Of course, you need to feed this storage to the field you working on:
my_file_filed = models.FileField(upload_to='mypath', storage=DownloadableS3Boto3Storage())
In case someone finds this, like I did: none of the solutions mentioned on SO worked for me in Django 3.0.
Docstring of S3Boto3Storage suggests overriding S3Boto3Storage.get_object_parameters, however this method only receives name of the uploaded file, which at this point has been changed by upload_to and can differ from the original.
What worked is the following:
class S3Boto3CustomStorage(S3Boto3Storage):
"""Override some upload parameters, such as ContentDisposition header."""
def _get_write_parameters(self, name, content):
"""Set ContentDisposition header using original file name.
While docstring recomments overriding `get_object_parameters` for this purpose,
`get_object_parameters` only gets a `name` which is not the original file name,
but the result of `upload_to`.
"""
params = super()._get_write_parameters(name, content)
original_name = getattr(content, 'name', None)
if original_name and name != original_name:
content_disposition = f'attachment; filename="{original_name}"'
params['ContentDisposition'] = content_disposition
return params
and then using this storage in the file field, e.g.:
file_field = models.FileField(
upload_to=some_func,
storage=S3Boto3CustomStorage(),
)
Whatever solution you come up with, do not change file_field.storage.object_parameters directly (e.g. in model's save() as it's been suggested in a similar question), because this will change ContentDisposition header for subsequent file uploads of any field that uses the same storage. Which is not what you probably want.
I guess you are using S3BotoStorage from django-storages, so while uploading the file to S3, override the save() method of the model, and set the header there.
I am giving an example below:
class ModelName(models.Model):
sthree = S3BotoStorage()
def file_name(self,filename):
ext = filename.split('.')[-1]
name = "%s/%s.%s" % ("downloads", uuid.uuid4(), ext)
return name
upload_file = models.FileField(upload_to=file_name,storage = sthree)
def save(self):
self.upload_file.storage.headers = {'Content-Disposition': 'attachments; filename="%s"' %self.upload_file.name}
super(ModelName, self).save()
One way can be giving ResponseContentDisposition parameter to S3Boto3Storage.url() method. In this case you don't have to create a custom storage.
Example model:
class MyModel(models.Model):
file = models.FileField(upload_to=generate_upload_path)
original_filename = models.CharField(max_length=255)
Creating URL for your file:
# obj is instance of MyModel
url = obj.file.storage.url(
obj.file.name,
parameters={
'ResponseContentDisposition': f'inline; filename={obj.original_filename}',
},
)
If you want to force browser to download the file, replace inline with attachment.
If you are using non-ascii filenames, check how Django encodes filename for Content-Disposition header in FileResponse.
I need to change default file upload behavior in django and the documentation on the django site is rather confusing.
I have a model with a field as follows:
class document (models.Model):
name = models.CharField(max_length=200)
file = models.FileField(null=True, upload_to='uploads/')
I need to create a .json file that will contain meta data when a file is uploaded. For example if I upload a file mydocument.docx I need to create mydocument.json file within the uploads/ folder and add meta information about the document.
From what I can decipher from the documentation I need to create a file upload handler as a subclass of django.core.files.uploadhandler.FileUploadHandler. It also goes on to say I can define this anywhere I want.
My questions: Where is the best place to define my subclass? Also from the documentation found here https://docs.djangoproject.com/en/1.8/ref/files/uploads/#writing-custom-upload-handlers looks like the subclass would look like the following:
class FileUploadHandler(object):
def handle_raw_input(self, input_data, META, content_length, boundary, encoding=None):
# do the acctual writing to disk
def file_complete(self, file_size):
# some logic to create json file
Does anyone have a working example of a upload handler class that works for django version 1.8?
One option could be to do the .json file generation on the (model) form used to initially upload the file. Override the save() method of the ModelForm to generate the file immediately after the model has been saved.
class DocumentForm(forms.ModelForm):
class Meta(object):
model = Document
fields = 'name', 'file'
def save(self, commit=True):
saved_document = super().save(commit)
with open(saved_document.file.path + '.json', mode='w') as fh:
fh.write(json.dumps({
"size": saved_document.file.size,
"uploaded": timezone.now().isoformat()
}))
return saved_document
I've tested this locally but YMMV if you are using custom storages for working with things like S3.
I'm switching to SVG images to represent categories on my e-commerce platform. I was using models.ImageField in the Category model to store the images before, but the forms.ImageField validation is not capable of handling a vector-based image (and therefore rejects it).
I don't require thorough validation against harmful files, since all uploads will be done via the Django Admin. It looks like I'll have to switch to a models.FileField in my model, but I do want warnings against uploading invalid images.
Nick Khlestov wrote a SVGAndImageFormField (find source within the article, I don't have enough reputation to post more links) over django-rest-framework's ImageField. How do I use this solution over Django's ImageField (and not the DRF one)?
I have never used SVGAndImageFormField so I cannot really comment on that. Personally I would have opted for a simple application of FileField, but that clearly depends on the project requirements. I will expand on that below:
As mentioned in the comment, the basic difference between an ImageField and a FileField is that the first checks if a file is an image using Pillow:
Inherits all attributes and methods from FileField, but also validates that the uploaded object is a valid image.
Reference: Django docs, Django source code
It also offers a couple of attributes possibly irrelevant to the SVG case (height, width).
Therefore, the model field could be:
svg = models.FileField(upload_to=..., validators=[validate_svg])
You can use a function like is_svg as provided in the relevant question:
How can I say a file is SVG without using a magic number?
Then a function to validate SVG:
def validate_svg(file, valid):
if not is_svg(file):
raise ValidationError("File not svg")
It turns out that SVGAndImageFormField has no dependencies on DRF's ImageField, it only adds to the validation done by django.forms.ImageField.
So to accept SVGs in the Django Admin I changed the model's ImageField to a FileField and specified an override as follows:
class MyModelForm(forms.ModelForm):
class Meta:
model = MyModel
exclude = []
field_classes = {
'image_field': SVGAndImageFormField,
}
class MyModelAdmin(admin.ModelAdmin):
form = MyModelForm
admin.site.register(MyModel, MyModelAdmin)
It now accepts all previous image formats along with SVG.
EDIT: Just found out that this works even if you don't switch from models.ImageField to models.FileField. The height and width attributes of models.ImageField will still work for raster image types, and will be set to None for SVG.
Here is a solution that works as a simple model field, that you can put instead of models.ImageField:
class Icon(models.Model):
image_file = SVGAndImageField()
You need to define following classes and functions somewhere in your code:
from django.db import models
class SVGAndImageField(models.ImageField):
def formfield(self, **kwargs):
defaults = {'form_class': SVGAndImageFieldForm}
defaults.update(kwargs)
return super().formfield(**defaults)
And here is how SVGAndImageFieldForm looks like:
from django import forms
from django.core.exceptions import ValidationError
class SVGAndImageFieldForm(forms.ImageField):
def to_python(self, data):
try:
f = super().to_python(data)
except ValidationError:
return validate_svg(data)
return f
Function validate_svg I took from other solutions:
import xml.etree.cElementTree as et
def validate_svg(f):
# Find "start" word in file and get "tag" from there
f.seek(0)
tag = None
try:
for event, el in et.iterparse(f, ('start',)):
tag = el.tag
break
except et.ParseError:
pass
# Check that this "tag" is correct
if tag != '{http://www.w3.org/2000/svg}svg':
raise ValidationError('Uploaded file is not an image or SVG file.')
# Do not forget to "reset" file
f.seek(0)
return f
Also if you want to use SVG files only model field - you can do it more simple.
Just create class, inherited from models.FileField, and in __init__ method you can add validate_svg function to kwargs['validators'].
Or just add this validator to models.FileField and be happy :)
As stated in the comments, validation for SVGAndImageFormField will fail because extensions are checked using django.core.validators.validate_image_file_extension, which is the default validator for an ImageField.
A workaround for this would be creating a custom validator adding "svg" to the accepted extensions.
Edited: Thanks #Ilya Semenov for your comment
from django.core.validators import (
get_available_image_extensions,
FileExtensionValidator,
)
def validate_image_and_svg_file_extension(value):
allowed_extensions = get_available_image_extensions() + ["svg"]
return FileExtensionValidator(allowed_extensions=allowed_extensions)(value)
Then, override the default_validators attribute in the SvgAndImageFormField:
class SVGAndImageFormField(DjangoImageField):
default_validators = [validate_image_and_svg_file_extension]
# ...
from django.forms import ModelForm, FileField
class TemplatesModelForm(ModelForm):
class Meta:
model = Templates
exclude = []
field_classes = {
'image': FileField,
}
#admin.register(Templates)
class TemplatesAdmin(admin.ModelAdmin):
form = TemplatesModelForm
its work
I am quite new to django and recently I have a requirement of a JSON output, for which I use the following django code:
data = serializers.serialize("json", Mymodel.objects.all())
It works great, except that I get a output of:
[{"pk": 8970859016715811, "model": "myapp.mymodel", "fields": {"reviews": "3.5", "title": .....}}]
However, I would like the output to be simply either:
[{"reviews": "3.5", "title": .....}]
or,
[{"id": "8970859016715811", "reviews": "3.5", "title": .....}]
I was wondering if someone could point me to the right direction as to how to achieve this.
You can add 'fields' parameter to the serialize-function, like this:
data = serializers.serialize('xml', SomeModel.objects.all(), fields=('name','size'))
See: https://docs.djangoproject.com/en/dev/topics/serialization/
EDIT 1:
You can customize the serializer to get only the fields you specify.
From Override Django Object Serializer to get rid of specified model:
from django.core.serializers.python import Serializer
class MySerialiser(Serializer):
def end_object( self, obj ):
self._current['id'] = obj._get_pk_val()
self.objects.append( self._current )
# views.py
serializer = MySerialiser()
data = serializer.serialize(some_qs)
You'll need to write a custom Json serializer. Something like this should do the trick:
class FlatJsonSerializer(Serializer):
def get_dump_object(self, obj):
data = self._current
if not self.selected_fields or 'id' in self.selected_fields:
data['id'] = obj.id
return data
def end_object(self, obj):
if not self.first:
self.stream.write(', ')
json.dump(self.get_dump_object(obj), self.stream,
cls=DjangoJSONEncoder)
self._current = None
def start_serialization(self):
self.stream.write("[")
def end_serialization(self):
self.stream.write("]")
def getvalue(self):
return super(Serializer, self).getvalue()
The you can use it like this:
s = FlatJsonSerializer()
s.serialize(MyModel.objects.all())
Or you could register the serializer with django.core.serializers.register_serializer and then use the familiar serializers.serialize shortcut.
Take a look at the django implementation as a reference if you need further customization: https://github.com/django/django/blob/master/django/core/serializers/json.py#L21-62
I just came across this as I was having the same problem. I also solved this with a custom serializer, tried the "EDIT 1" method but it didn't work too well as it stripped away all the goodies that the django JSON encoder already did (decimal, date serialization), which you can rewrite it yourself but why bother. I think a much less intrusive way is to inherit the JSON serializer directly like this.
from django.core.serializers.json import Serializer
from django.utils.encoding import smart_text
class MyModelSerializer(Serializer):
def get_dump_object(self, obj):
self._current['id'] = smart_text(obj._get_pk_val(), strings_only=True)
return self._current
Sso the main culprit that writes the fields and model thing is at the parent level python serializer and this way, you also automatically get the fields filtering that's already built into django's JSON serializer. Call it like this
serializer = MyModelSerializer()
data = serializer.serialize(<queryset>, <optional>fields=('field1', 'field2'))
import json
_all_data = Reporter.objects. all()
json_data = json.dumps([{'name': reporter.full_name} for reporter in _all_data])
return HttpResponse(json_data, content_type='application/json')
Here Reporter is your Model