Develop Reference

How to programmatically upload local file as Django model field?

I'm having troubles trying to upload files to FileField from local path.
I have correctly configurred CDN backend in S3 bucket and use it as PrivateMediaStorage for one of my model fields:
class MyModel(models.Model):
some_file = models.FileField(storage=PrivateMediaStorage())
...
With this very simple configuration whenever I'm creating/updating model through django-admin it is saved and file attached as some_file is correctly uploaded to S3 bucket.
Yet if I try to create/update model instance programmatically, say through custom manage.py command, model instance itself is created but attachment is never uploaded to CDN. Here's simplified version of code I'm using to upload files:
class Command(BaseCommand):
help = 'Creates dummy instance for quicker configuration'
def handle(self, *args, **options):
some_file = os.path.join(os.path.dirname(__file__), '../../../temporary/some_image.png')
if not os.path.exists(some_file):
raise CommandError(f'File {some_file} does not exist')
else:
instance, created = MyModel.objects.get_or_create(defaults={'some_file': some_file}, ...)
What is missing in my implementation and what needs to be adjusted to allow file uploads from local storage?

You're passing a string (the result of os.path.join()) to your some_file field, but you need to pass it an actual File object.
The easiest way to save a file on a model directly is to use the FieldFile's save() method.
As a working solution for case provided in question a valid way of creating a record would be:
instance = MyModel.objects.create(some_file=File(file=open(some_file, 'rb'), name='some_name.png'))
Or even better to use pathlib to obtain name dynamically:
from pathlib import Path
instance = MyModel.objects.create(some_file=File(file=open(some_file, 'rb'), name=Path(some_file).name))
Note that fetching a row based on the file is unlikely to work, AFAIK each time you open a file, doing a get_or_create() with the File instance as argument will probably create a new row each time. Better put file fields into defaults:
with open(some_file, 'rb') as file:
instance, created = MyModel.objects.get_or_create(
some_other_field=...,
defaults={'some_file': File(
file=file,
name=pathlib.Path(some_file).name
)}
)

you can also do something like this.
some_file = os.path.join(os.path.dirname(__file__), '../../../temporary/some_image.png')
instance.some_file.name = some_file
instance.save()

Getting the Django-generated unique name of a file uploaded multiple times

I am using DRF backend to upload files. In my specific case I will want to get the name of the file, after it has been uploaded. The reason is that if a user uploads a file with same name, I am still able to process it independently.
views.py:
class ImageUploaderView(viewsets.ModelViewSet):
renderer_classes = [renderers.JSONRenderer]
queryset = ImageUploader.objects.all()
serializer_class = ImageUploaderSerializer
parser_classes = (MultiPartParser,)
serializer.py:
class ImageUploaderSerializer(serializers.ModelSerializer):
class Meta:
model = ImageUploader
models.py:
class ImageUploader(models.Model):
# name=models.ImageField(upload_to='media')
name=models.FileField(upload_to='media')
I tried to put signals and hooks after the model definitions but I am not being able to get this filename. Can someone shed some light pls?
UPDATE: Let me elaborate what I want to achieve essentially:
User1 hits endpoint "/api/calculate_interest_rate" which is rendered
by a frontend React component. "calculate_interest_rate" is served by
DRF, and lets the user upload a CSV file. This will be stored as
"user1.csv", the file is processed and then tabulated (rendered by
React).
At the same time and in parallel to User1, User2 hits the same endpoint "/api/calculate_interest_rate" and
by mistake he saves his file as "user1.csv", and uploads it to the systemn.
So I want to be able to detect both names of the file in order to process it. By using always the same default filename (ex. using the OverwriteStorage() technique), I will probably cause chaos when two or more users are using the same filename. Therefore I am looking into a technique that allows me to get the filename as is, and process it immediately.

How about using storage option?
class OverwriteStorage(FileSystemStorage):
def get_available_name(self, name, max_length=None):
print("filename", name)
#parts = name.split('.') you can separate name and extension.
return super().get_available_name(name)
upload_image = models.ImageField(
upload_to=[yourpath],
default=[defaultname],
storage=OverwriteStorage()
)

I suggest you to following this configuration:
1. Change your MEDIA_ROOT and MEDIA_URL inside file of settings.py
MEDIA_URL = '/media/'
MEDIA_ROOT = '/path/to/env/projectname/media'
2. Then, I suggest you to change your upload_to='media to upload_to='images/%Y/%m/%d, also rename your field of name with image.
class ImageUploader(models.Model):
image = models.FileField(upload_to='images/%Y/%m/%d')
# OR
# image = models.ImageField(upload_to='images/%Y/%m/%d')
Explanation; If you following this configuration, you could have uploaded images are following, eg: /media/images/2017/01/29/yourimage.jpg. This is one way to handle the problem of duplicated files.
3. But if you want to upload file with multiple times without duplicate files, you can using deconstructible;
import os, time, uuid
from django.db import models
from django.utils.deconstruct import deconstructible
class ImageUploader(models.Model):
#deconstructible
class PathAndRename(object):
def __init__(self, sub_path):
self.path = sub_path
def __call__(self, instance, filename):
# eg: filename = 'my uploaded file.jpg'
ext = filename.split('.')[-1] #eg: '.jpg'
uid = uuid.uuid4().hex[:10] #eg: '567ae32f97'
# eg: 'my-uploaded-file'
new_name = '-'.join(filename.replace('.%s' % ext, '').split())
# eg: 'my-uploaded-file_64c942aa64.jpg'
renamed_filename = '%(new_name)s_%(uid)s.%(ext)s' % {'new_name': new_name, 'uid': uid, 'ext': ext}
# eg: 'images/2017/01/29/my-uploaded-file_64c942aa64.jpg'
return os.path.join(self.path, renamed_filename)
image_path = time.strftime('images/%Y/%m/%d')
image = models.ImageField(upload_to=PathAndRename(self.image_path))

Add Dynamic Content Disposition for file names(amazon S3) in python

I have a Django model that saves filename as "uuid4().pdf". Where uuid4 generates a random uuid for each instance created. This file name is also stored on the amazon s3 server with the same name.
I am trying to add a custom disposition for filename that i upload to amazon s3, this is because i want to see a custom name whenever i download the file not the uuid one. At the same time, i want the files to stored on s3 with the uuid filename.
So, I am using django-storages with python 2.7. I have tried adding content_disposition in settings like this:
AWS_CONTENT_DISPOSITION = 'core.utils.s3.get_file_name'
where get_file_name() returns the filename.
I have also tried adding this to the settings:
AWS_HEADERS = {
'Content-Disposition': 'attachments; filename="%s"'% get_file_name(),
}
no luck!
Do anyone of you know to implement this.

Current version of S3Boto3Storage from django-storages supports AWS_S3_OBJECT_PARAMETERS global settings variable, which allows modify ContentDisposition too. But the problem is that it is applied as is to all objects that are uploaded to s3 and, moreover, affects all models working with the storage, which may turn to be not the expected result.
The following hack worked for me.
from storages.backends.s3boto3 import S3Boto3Storage
class DownloadableS3Boto3Storage(S3Boto3Storage):
def _save_content(self, obj, content, parameters):
"""
The method is called by the storage for every file being uploaded to S3.
Below we take care of setting proper ContentDisposition header for
the file.
"""
filename = obj.key.split('/')[-1]
parameters.update({'ContentDisposition': f'attachment; filename="{filename}"'})
return super()._save_content(obj, content, parameters)
Here we override native save method of the storage object and make sure proper content disposition is set on each file.
Of course, you need to feed this storage to the field you working on:
my_file_filed = models.FileField(upload_to='mypath', storage=DownloadableS3Boto3Storage())

In case someone finds this, like I did: none of the solutions mentioned on SO worked for me in Django 3.0.
Docstring of S3Boto3Storage suggests overriding S3Boto3Storage.get_object_parameters, however this method only receives name of the uploaded file, which at this point has been changed by upload_to and can differ from the original.
What worked is the following:
class S3Boto3CustomStorage(S3Boto3Storage):
"""Override some upload parameters, such as ContentDisposition header."""
def _get_write_parameters(self, name, content):
"""Set ContentDisposition header using original file name.
While docstring recomments overriding `get_object_parameters` for this purpose,
`get_object_parameters` only gets a `name` which is not the original file name,
but the result of `upload_to`.
"""
params = super()._get_write_parameters(name, content)
original_name = getattr(content, 'name', None)
if original_name and name != original_name:
content_disposition = f'attachment; filename="{original_name}"'
params['ContentDisposition'] = content_disposition
return params
and then using this storage in the file field, e.g.:
file_field = models.FileField(
upload_to=some_func,
storage=S3Boto3CustomStorage(),
)
Whatever solution you come up with, do not change file_field.storage.object_parameters directly (e.g. in model's save() as it's been suggested in a similar question), because this will change ContentDisposition header for subsequent file uploads of any field that uses the same storage. Which is not what you probably want.

I guess you are using S3BotoStorage from django-storages, so while uploading the file to S3, override the save() method of the model, and set the header there.
I am giving an example below:
class ModelName(models.Model):
sthree = S3BotoStorage()
def file_name(self,filename):
ext = filename.split('.')[-1]
name = "%s/%s.%s" % ("downloads", uuid.uuid4(), ext)
return name
upload_file = models.FileField(upload_to=file_name,storage = sthree)
def save(self):
self.upload_file.storage.headers = {'Content-Disposition': 'attachments; filename="%s"' %self.upload_file.name}
super(ModelName, self).save()

One way can be giving ResponseContentDisposition parameter to S3Boto3Storage.url() method. In this case you don't have to create a custom storage.
Example model:
class MyModel(models.Model):
file = models.FileField(upload_to=generate_upload_path)
original_filename = models.CharField(max_length=255)
Creating URL for your file:
# obj is instance of MyModel
url = obj.file.storage.url(
obj.file.name,
parameters={
'ResponseContentDisposition': f'inline; filename={obj.original_filename}',
},
)
If you want to force browser to download the file, replace inline with attachment.
If you are using non-ascii filenames, check how Django encodes filename for Content-Disposition header in FileResponse.

Django. How to programatically rename files uploaded using django-filer

Let's say I am selecting 10 files to be uploaded using django-filer. They initially have very random names. I'd like to have a set of rules according to which they ought to be renamed and only then passed for further processing (thumbnails etc.).
I need to actually rename everything, especially filename, not just Image model name.
I tried catching pre_save signal for Image model and altering instance.original_filename but that's not renaming a filename. Or maybe should I subclass and override something from filer package?
I'd be grateful for code example cause this is a little bit to hard for me.

I used form_valid(self, form) in views.py to process and manipulate my images. The complete code is a bit long and very specific, but I'll post a few snipplets which should show the idea of how to generate filenames:
def form_valid(self, form):
upload = self.request.FILES['profilbild_original'] #coming from a very simple form
self.request.user.student.profilbild_original = upload
self.request.user.student.save()
#no renaming was required here, but now I did some work:
inputfilepath = os.path.join(my_app.settings.MEDIA_ROOT, profilbild_path(self.request.user, str(upload)))
original = Image.open(inputfilepath)
original.thumbnail((200,200), Image.ANTIALIAS)
filename = str(upload)+'.thumbnail_200_200_aa.jpg'
filepath = profilbild_path(self.request.user, filename)
filepath = os.path.join(my_app.settings.MEDIA_ROOT, filepath)
original.save(filepath, 'JPEG', quality=90)
self.request.user.student.profilbild = profilbild_path(self.request.user, filename).replace("\\", "/")
self.request.user.student.save()
return super(ProfilbildView, self).form_valid(form)
profilbild_pathis a function according to https://docs.djangoproject.com/en/1.3/ref/models/fields/#django.db.models.FileField.upload_to :
def profilbild_path(instance, filename):
return os.path.join('profilbilder', str(instance.id), filename)
I hope this gives you some clues.

Set Django's FileField to an existing file

I have an existing file on disk (say /folder/file.txt) and a FileField model field in Django.
When I do
instance.field = File(file('/folder/file.txt'))
instance.save()
it re-saves the file as file_1.txt (the next time it's _2, etc.).
I understand why, but I don't want this behavior - I know the file I want the field to be associated with is really there waiting for me, and I just want Django to point to it.
How?

just set instance.field.name to the path of your file
e.g.
class Document(models.Model):
file = FileField(upload_to=get_document_path)
description = CharField(max_length=100)
doc = Document()
doc.file.name = 'path/to/file' # must be relative to MEDIA_ROOT
doc.file
<FieldFile: path/to/file>

If you want to do this permanently, you need to create your own FileStorage class
import os
from django.conf import settings
from django.core.files.storage import FileSystemStorage
class MyFileStorage(FileSystemStorage):
# This method is actually defined in Storage
def get_available_name(self, name):
if self.exists(name):
os.remove(os.path.join(settings.MEDIA_ROOT, name))
return name # simply returns the name passed
Now in your model, you use your modified MyFileStorage
from mystuff.customs import MyFileStorage
mfs = MyFileStorage()
class SomeModel(model.Model):
my_file = model.FileField(storage=mfs)

try this (doc):
instance.field.name = <PATH RELATIVE TO MEDIA_ROOT>
instance.save()

It's right to write own storage class. However get_available_name is not the right method to override.
get_available_name is called when Django sees a file with same name and tries to get a new available file name. It's not the method that causes the rename. the method caused that is _save. Comments in _save is pretty good and you can easily find it opens file for writing with flag os.O_EXCL which will throw an OSError if same file name already exists. Django catches this Error then calls get_available_name to get a new name.
So I think the correct way is to override _save and call os.open() without flag os.O_EXCL. The modification is quite simple however the method is a little be long so I don't paste it here. Tell me if you need more help :)

I had exactly the same problem! then I realize that my Models were causing that. example I hade my models like this:
class Tile(models.Model):
image = models.ImageField()
Then, I wanted to have more the one tile referencing the same file in the disk! The way that I found to solve that was change my Model structure to this:
class Tile(models.Model):
image = models.ForeignKey(TileImage)
class TileImage(models.Model):
image = models.ImageField()
Which after I realize that make more sense, because if I want the same file being saved more then one in my DB I have to create another table for it!
I guess you can solve your problem like that too, just hoping that you can change the models!
EDIT
Also I guess you can use a different storage, like this for instance: SymlinkOrCopyStorage
http://code.welldev.org/django-storages/src/11bef0c2a410/storages/backends/symlinkorcopy.py

You should define your own storage, inherit it from FileSystemStorage, and override OS_OPEN_FLAGS class attribute and get_available_name() method:
Django Version: 3.1
Project/core/files/storages/backends/local.py
import os
from django.core.files.storage import FileSystemStorage
class OverwriteStorage(FileSystemStorage):
"""
FileSystemStorage subclass that allows overwrite the already existing
files.
Be careful using this class, as user-uploaded files will overwrite
already existing files.
"""
# The combination that don't makes os.open() raise OSError if the
# file already exists before it's opened.
OS_OPEN_FLAGS = os.O_WRONLY | os.O_TRUNC | os.O_CREAT | getattr(os, 'O_BINARY', 0)
def get_available_name(self, name, max_length=None):
"""
This method will be called before starting the save process.
"""
return name
In your model, use your custom OverwriteStorage
myapp/models.py
from django.db import models
from core.files.storages.backends.local import OverwriteStorage
class MyModel(models.Model):
my_file = models.FileField(storage=OverwriteStorage())

The answers work fine if you are using the app's filesystem to store your files. But, If your are using boto3 and uploading to sth like AWS S3 and maybe you want to set a file already existing in an S3 bucket to your model's FileField then, this is what you need.
We have a simple model class with a filefield:
class Image(models.Model):
img = models.FileField()
owner = models.ForeignKey(get_user_model(), on_delete=models.CASCADE, related_name='images')
date_added = models.DateTimeField(editable=False)
date_modified = models.DateTimeField(editable=True)
from botocore.exceptions import ClientError
import boto3
s3 = boto3.client(
's3',
aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
)
s3_key = S3_DIR + '/' + filename
bucket_name = os.getenv("AWS_STORAGE_BUCKET_NAME")
try:
s3.upload_file(local_file_path, bucket_name, s3_key)
# we want to store it to our db model called **Image** after s3 upload is complete so,
image_data = Image()
image_data.img.name = s3_key # this does it !!
image_data.owner = get_user_model().objects.get(id=owner_id)
image_data.save()
except ClientError as e:
print(f"failed uploading to s3 {e}")
Setting the S3 KEY into the name field of the FileField does the trick. As much i have tested everything related works as expected e.g previewing the image file in django admin. fetching the images from db appends the root s3 bucket prefix (or, the cloudfront cdn prefix) to the s3 keys of the files too. Ofcourse, its given that, i already had a working setup of the django settings.py for boto and s3.