Develop Reference

"Upload" file from disk in Django

I generate a file in python, and want to "upload" that file to the django database. This way it is automatically put inside the media folder, and organized neatly with all other files of my application.
Now here is what I tried: (type hinting used, since it's python 3.6)
# forms.py
class UploadForm(forms.ModelForm):
class Meta:
model = UploadedFile
fields = ('document',)
# models.py
class UploadedFile(models.Model):
document = models.FileField(upload_to=get_upload_path)
# mimetype is generated by filename on save
mimetype = models.CharField(max_length=255)
# ... additional fields like temporary
def get_upload_path(instance: UploadedFile, filename):
if instance.temporary:
return "uploaded_files/temp/" + filename
return "uploaded_files/" + filename
# views.py, file_out has been generated
with open(file_out, 'rb') as local_file:
from django.core.files import File
form = UploadForm(dict(), {'document': File(local_file)})
print(form.errors)
if form.is_valid():
file = form.save(commit=False)
# ... set additional fields
file.save()
form.save_m2m()
return file
Now this is not the only thing I've tried. First I've gone with setting the FileField directly, but that resulted in the save() to fail, while the mimetype field is set. Because the original file sits outside the media folder, and thus a suspicious file action is triggered.
Also, the form gives some feedback about the "upload", through the form.errors.
Depending on my approach, either the save() fails as mentioned above -- meaning the "uploading" does not actually copy the file in the media folder -- or the form returns the error that no file was transmitted, and tells to check the form protocol.
Now my theory is, that I would have to go and initialize my own instance of InMemoryUploadedFile, but I could not figure out how to do that myself, and no documentation was available on the internet.
It feels like I'm taking the wrong approach from the get go. How would one do this properly?

Do you have get_upload_path defined? If not, that would explain the errors you're getting.
From what I can see you're on the right track. If you don't need a dynamic path for your uploads, if you just want them in media/uploads, you can pass in a string value for upload_to (from the Django docs):
# file will be uploaded to MEDIA_ROOT/uploads
document = models.FileField(upload_to='uploads/')

First of all, thanks to Franey for pointing me at storage documentation which lead me to contentfile documentation.
The ContentFile actually solves the problem, because it basically is the self-instantiated version of InMemoryUploadedFile that I was looking for. It's a django File that is not stored on disk.
Here's the full solution:
# views.py, file_out has been generated
with open(file_out, 'rb') as local_file:
from django.core.files.base import ContentFile
# we need to provide a name. Otherwise the Storage.save
# method reveives a None-parameter and breaks.
form = UploadForm(dict(), {'document': ContentFile(local_file.read(), name=name)})
if form.is_valid():
file = form.save(commit=False)
# ... set additional fields
file.save()
form.save_m2m()
return file

Allow SVG files to be uploaded to ImageField via Django admin

I'm switching to SVG images to represent categories on my e-commerce platform. I was using models.ImageField in the Category model to store the images before, but the forms.ImageField validation is not capable of handling a vector-based image (and therefore rejects it).
I don't require thorough validation against harmful files, since all uploads will be done via the Django Admin. It looks like I'll have to switch to a models.FileField in my model, but I do want warnings against uploading invalid images.
Nick Khlestov wrote a SVGAndImageFormField (find source within the article, I don't have enough reputation to post more links) over django-rest-framework's ImageField. How do I use this solution over Django's ImageField (and not the DRF one)?

I have never used SVGAndImageFormField so I cannot really comment on that. Personally I would have opted for a simple application of FileField, but that clearly depends on the project requirements. I will expand on that below:
As mentioned in the comment, the basic difference between an ImageField and a FileField is that the first checks if a file is an image using Pillow:
Inherits all attributes and methods from FileField, but also validates that the uploaded object is a valid image.
Reference: Django docs, Django source code
It also offers a couple of attributes possibly irrelevant to the SVG case (height, width).
Therefore, the model field could be:
svg = models.FileField(upload_to=..., validators=[validate_svg])
You can use a function like is_svg as provided in the relevant question:
How can I say a file is SVG without using a magic number?
Then a function to validate SVG:
def validate_svg(file, valid):
if not is_svg(file):
raise ValidationError("File not svg")

It turns out that SVGAndImageFormField has no dependencies on DRF's ImageField, it only adds to the validation done by django.forms.ImageField.
So to accept SVGs in the Django Admin I changed the model's ImageField to a FileField and specified an override as follows:
class MyModelForm(forms.ModelForm):
class Meta:
model = MyModel
exclude = []
field_classes = {
'image_field': SVGAndImageFormField,
}
class MyModelAdmin(admin.ModelAdmin):
form = MyModelForm
admin.site.register(MyModel, MyModelAdmin)
It now accepts all previous image formats along with SVG.
EDIT: Just found out that this works even if you don't switch from models.ImageField to models.FileField. The height and width attributes of models.ImageField will still work for raster image types, and will be set to None for SVG.

Here is a solution that works as a simple model field, that you can put instead of models.ImageField:
class Icon(models.Model):
image_file = SVGAndImageField()
You need to define following classes and functions somewhere in your code:
from django.db import models
class SVGAndImageField(models.ImageField):
def formfield(self, **kwargs):
defaults = {'form_class': SVGAndImageFieldForm}
defaults.update(kwargs)
return super().formfield(**defaults)
And here is how SVGAndImageFieldForm looks like:
from django import forms
from django.core.exceptions import ValidationError
class SVGAndImageFieldForm(forms.ImageField):
def to_python(self, data):
try:
f = super().to_python(data)
except ValidationError:
return validate_svg(data)
return f
Function validate_svg I took from other solutions:
import xml.etree.cElementTree as et
def validate_svg(f):
# Find "start" word in file and get "tag" from there
f.seek(0)
tag = None
try:
for event, el in et.iterparse(f, ('start',)):
tag = el.tag
break
except et.ParseError:
pass
# Check that this "tag" is correct
if tag != '{http://www.w3.org/2000/svg}svg':
raise ValidationError('Uploaded file is not an image or SVG file.')
# Do not forget to "reset" file
f.seek(0)
return f
Also if you want to use SVG files only model field - you can do it more simple.
Just create class, inherited from models.FileField, and in __init__ method you can add validate_svg function to kwargs['validators'].
Or just add this validator to models.FileField and be happy :)

As stated in the comments, validation for SVGAndImageFormField will fail because extensions are checked using django.core.validators.validate_image_file_extension, which is the default validator for an ImageField.
A workaround for this would be creating a custom validator adding "svg" to the accepted extensions.
Edited: Thanks #Ilya Semenov for your comment
from django.core.validators import (
get_available_image_extensions,
FileExtensionValidator,
)
def validate_image_and_svg_file_extension(value):
allowed_extensions = get_available_image_extensions() + ["svg"]
return FileExtensionValidator(allowed_extensions=allowed_extensions)(value)
Then, override the default_validators attribute in the SvgAndImageFormField:
class SVGAndImageFormField(DjangoImageField):
default_validators = [validate_image_and_svg_file_extension]
# ...

from django.forms import ModelForm, FileField
class TemplatesModelForm(ModelForm):
class Meta:
model = Templates
exclude = []
field_classes = {
'image': FileField,
}
#admin.register(Templates)
class TemplatesAdmin(admin.ModelAdmin):
form = TemplatesModelForm
its work

Set Django's FileField to an existing file

I have an existing file on disk (say /folder/file.txt) and a FileField model field in Django.
When I do
instance.field = File(file('/folder/file.txt'))
instance.save()
it re-saves the file as file_1.txt (the next time it's _2, etc.).
I understand why, but I don't want this behavior - I know the file I want the field to be associated with is really there waiting for me, and I just want Django to point to it.
How?

just set instance.field.name to the path of your file
e.g.
class Document(models.Model):
file = FileField(upload_to=get_document_path)
description = CharField(max_length=100)
doc = Document()
doc.file.name = 'path/to/file' # must be relative to MEDIA_ROOT
doc.file
<FieldFile: path/to/file>

If you want to do this permanently, you need to create your own FileStorage class
import os
from django.conf import settings
from django.core.files.storage import FileSystemStorage
class MyFileStorage(FileSystemStorage):
# This method is actually defined in Storage
def get_available_name(self, name):
if self.exists(name):
os.remove(os.path.join(settings.MEDIA_ROOT, name))
return name # simply returns the name passed
Now in your model, you use your modified MyFileStorage
from mystuff.customs import MyFileStorage
mfs = MyFileStorage()
class SomeModel(model.Model):
my_file = model.FileField(storage=mfs)

try this (doc):
instance.field.name = <PATH RELATIVE TO MEDIA_ROOT>
instance.save()

It's right to write own storage class. However get_available_name is not the right method to override.
get_available_name is called when Django sees a file with same name and tries to get a new available file name. It's not the method that causes the rename. the method caused that is _save. Comments in _save is pretty good and you can easily find it opens file for writing with flag os.O_EXCL which will throw an OSError if same file name already exists. Django catches this Error then calls get_available_name to get a new name.
So I think the correct way is to override _save and call os.open() without flag os.O_EXCL. The modification is quite simple however the method is a little be long so I don't paste it here. Tell me if you need more help :)

I had exactly the same problem! then I realize that my Models were causing that. example I hade my models like this:
class Tile(models.Model):
image = models.ImageField()
Then, I wanted to have more the one tile referencing the same file in the disk! The way that I found to solve that was change my Model structure to this:
class Tile(models.Model):
image = models.ForeignKey(TileImage)
class TileImage(models.Model):
image = models.ImageField()
Which after I realize that make more sense, because if I want the same file being saved more then one in my DB I have to create another table for it!
I guess you can solve your problem like that too, just hoping that you can change the models!
EDIT
Also I guess you can use a different storage, like this for instance: SymlinkOrCopyStorage
http://code.welldev.org/django-storages/src/11bef0c2a410/storages/backends/symlinkorcopy.py

You should define your own storage, inherit it from FileSystemStorage, and override OS_OPEN_FLAGS class attribute and get_available_name() method:
Django Version: 3.1
Project/core/files/storages/backends/local.py
import os
from django.core.files.storage import FileSystemStorage
class OverwriteStorage(FileSystemStorage):
"""
FileSystemStorage subclass that allows overwrite the already existing
files.
Be careful using this class, as user-uploaded files will overwrite
already existing files.
"""
# The combination that don't makes os.open() raise OSError if the
# file already exists before it's opened.
OS_OPEN_FLAGS = os.O_WRONLY | os.O_TRUNC | os.O_CREAT | getattr(os, 'O_BINARY', 0)
def get_available_name(self, name, max_length=None):
"""
This method will be called before starting the save process.
"""
return name
In your model, use your custom OverwriteStorage
myapp/models.py
from django.db import models
from core.files.storages.backends.local import OverwriteStorage
class MyModel(models.Model):
my_file = models.FileField(storage=OverwriteStorage())

The answers work fine if you are using the app's filesystem to store your files. But, If your are using boto3 and uploading to sth like AWS S3 and maybe you want to set a file already existing in an S3 bucket to your model's FileField then, this is what you need.
We have a simple model class with a filefield:
class Image(models.Model):
img = models.FileField()
owner = models.ForeignKey(get_user_model(), on_delete=models.CASCADE, related_name='images')
date_added = models.DateTimeField(editable=False)
date_modified = models.DateTimeField(editable=True)
from botocore.exceptions import ClientError
import boto3
s3 = boto3.client(
's3',
aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
)
s3_key = S3_DIR + '/' + filename
bucket_name = os.getenv("AWS_STORAGE_BUCKET_NAME")
try:
s3.upload_file(local_file_path, bucket_name, s3_key)
# we want to store it to our db model called **Image** after s3 upload is complete so,
image_data = Image()
image_data.img.name = s3_key # this does it !!
image_data.owner = get_user_model().objects.get(id=owner_id)
image_data.save()
except ClientError as e:
print(f"failed uploading to s3 {e}")
Setting the S3 KEY into the name field of the FileField does the trick. As much i have tested everything related works as expected e.g previewing the image file in django admin. fetching the images from db appends the root s3 bucket prefix (or, the cloudfront cdn prefix) to the s3 keys of the files too. Ofcourse, its given that, i already had a working setup of the django settings.py for boto and s3.

Replacing a Django image doesn't delete original

In Django, if you have a ImageFile in a model, deleting will remove the associated file from disk as well as removing the record from the database.
Shouldn't replacing an image also remove the unneeded file from disk? Instead, I see that it keeps the original and adds the replacement.
Now deleting the object won't delete the original file only the replacement.
Are there any good strategies to doing this? I don't want to have a bunch of orphan files if my users replace their images frequently.

The best strategy I've found is to make a custom save method in the model:
class Photo(models.Model):
image = ImageField(...) # works with FileField also
def save(self, *args, **kwargs):
# delete old file when replacing by updating the file
try:
this = Photo.objects.get(id=self.id)
if this.image != self.image:
this.image.delete(save=False)
except: pass # when new photo then we do nothing, normal case
super(Photo, self).save(*args, **kwargs)
And beware, as with the updating which doesn't delete the back end file, deleting an instance model (here Photo) will not delete the back-end file, not in Django 1.3 anyway, you'll have to add more custom code to do that (or regularly do some dirty cron job).
Finally test all your update/delete cases with your ForeignKey, ManytoMany and others relations to check if the back-end files are correctly deleted. Believe only what you test.

Shouldn't replacing an image also remove the unneeded file from disk?
In the olden days, FileField was eager to clean up orphaned files. But that changed in Django 1.2:
In earlier Django versions, when a model instance containing a FileField was deleted, FileField took it upon itself to also delete the file from the backend storage. This opened the door to several potentially serious data-loss scenarios, including rolled-back transactions and fields on different models referencing the same file. In Django 1.2.5, FileField will never delete files from the backend storage.

The code in the following working example will, upon uploading an image in an ImageField, detect if a file with the same name exists, and in that case, delete that file before storing the new one.
It could easily be modified so that it deletes the old file regardless of the filename. But that's not what I wanted in my project.
Add the following class:
from django.core.files.storage import FileSystemStorage
class OverwriteStorage(FileSystemStorage):
def _save(self, name, content):
if self.exists(name):
self.delete(name)
return super(OverwriteStorage, self)._save(name, content)
def get_available_name(self, name):
return name
And use it with ImageField like so:
class MyModel(models.Model):
myfield = models.ImageField(
'description of purpose',
upload_to='folder_name',
storage=OverwriteStorage(), ### using OverwriteStorage here
max_length=500,
null=True,
blank=True,
height_field='height',
width_field='width'
)
height = models.IntegerField(blank=True, null=True)
width = models.IntegerField(blank=True, null=True)

If you don't use transactions or you don't afraid of loosing files on transaction rollback, you can use django-cleanup

There have been a number of tickets regarding this issue though it is likely this will not make it into the core. The most comprehensive is http://code.djangoproject.com/ticket/11663. The patches and ticket comments can give you some direction if you are looking for a solution.
You can also consider using a different StorageBackend such as the Overwrite File Storage System given by Django snippet 976. http://djangosnippets.org/snippets/976/. You can change your default storage to this backend or you can override it on each FileField/ImageField declaration.

Here is a code that can work with or without upload_to=... or blank=True, and when the submitted file has the same name as the old one.
(py3 syntax, tested on Django 1.7)
class Attachment(models.Model):
document = models.FileField(...) # or ImageField
def delete(self, *args, **kwargs):
self.document.delete(save=False)
super().delete(*args, **kwargs)
def save(self, *args, **kwargs):
if self.pk:
old = self.__class__._default_manager.get(pk=self.pk)
if old.document.name and (not self.document._committed or not self.document.name):
old.document.delete(save=False)
super().save(*args, **kwargs)
Remember that this kind of solution is only applicable if you are in a non transactional context (no rollback, because the file is definitively lost)

I used a simple method with popen, so when i save my Info model i delete the former file before linking to the new:
import os
try:
os.popen("rm %s" % str(info.photo.path))
except:
#deal with error
pass
info.photo = nd['photo']

I save the original file and if it has changed - delete it.
class Document(models.Model):
document = FileField()
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._document = self.document
def save(self, *args, **kwargs):
if self.document != self._document:
self._document.delete()
super().save(*args, **kwargs)

How to validate django form only when adding not editing

How could we make the django form to not validate if we are editing, not adding a new record. The code as following :
class PageForm(forms.Form):
name = forms.CharField(max_length=100,widget=forms.TextInput(attrs={'class':'textInput'}))
description = forms.CharField(max_length=300, required=False,widget=forms.TextInput(attrs={'class':'textInput'}))
body = forms.CharField(widget=forms.Textarea)
template = forms.CharField(max_length=30,widget=forms.TextInput(attrs={'class':'textInput'}))
navbar = forms.BooleanField(required=False, widget=forms.Select(choices=(('True','True'),
('False', 'False'))))
publish = forms.BooleanField(widget=forms.Select(choices=(('Published','Publish Now'),
('Private','Private'),
('Draft','Draft'))))
def save(self, page=None, commit=True):
data = self.cleaned_data
if not page:
page = models.Page(key_name=data['name'].replace(' ','-'))
page.name = data['name']
page.description = data['description']
page.body = data['body']
page.template = data['template']
page.publish = data['publish']
if commit: page.put()
return page
# prevent the same page 's name
def clean_name(self):
name = self.cleaned_data['name']
query = models.Page.all(keys_only=True)
query.filter('name = ', name)
page = query.get()
if page:
raise forms.ValidationError('Page name "%s" was already used before' % name)
return name
The purpose of this name validation is to prevent the records with the same name. BUt i found that, it also validate on edit, so we couldn't edit records, since it will said 'records with same name already exist'.
Actually for editing, the page param on save function wont be none, but prev record instead, and wil be none on saving a new one. But how we read this param, on clean_name function so we can now whether it is editing or creating?
Thanks a lot!

in your clean method, you can use self.initial to know whether it is adding or editing. If it is editing, the self.initial will not be empty. But when it is adding, self.initial will be dictionary of what the previous value.

If you are editing form, then the form has some instance, and you can check if that exists.
If it does, then you are probably editing existing object.. right?
Example:
If you are editing object with form, you create form object much like this:
form = MyForm(instance = myobject)
Then in your form class methods you can check if form has saved instance in a way that it is described here:
Test if Django ModelForm has instance

in your clean_name function exclude the current object from queryset
query.filter('name = ', name).exclude(pk=self.pk)
or change the if condition to check that page and current object are not the same.

Sorry, I couldn't comment below your guys post, don't know why.
#sunn0 : I didn't use django models, coz deploy the app in appengine, so use appengine model instead.
#Zayatzz : May you show a little code how to do it? Since whether we are adding or editing, we always bound the form to request.POST before validation, so don't know how to differentiate.
#Ashok : I made a workaround based on your suggestion. Since previously I didn't pass the pk to form, but passing the prev object as param instead, so couldn't exclude by using pk. So, I change the code and put additional key as pk (if create, let key empty, but if edit fill key with pk) and just check in if condition, if key field not empty, then it means we are editing. Not sure if it is best practice, but it works anyway.

I can suggest to override form's init method
https://stackoverflow.com/a/70845558/15080117
because there is an argument instance.