I have an application that allows for users to upload CSV files with data which is then graphed and displayed to the user. These files are saved as media within my media folder. In my graph view however I need to open the file and process it. My problem is that I can only open files that are within my project's current working directory and any time that I attempt to upload a file from somewhere outside of that directory I get this error:
File b'TEST.CSV' does not exist
I have attempted this, but without success:
file_upload_dir = os.path.join(settings.MEDIA_ROOT, 'Data_Files')
data_file = open(os.path.join(file_upload_dir, new_file), 'rb')
The variable new_file is only the name of the file saved from in a session and not a path to that file. Data_Files is a directory within the media directory that contains the uploaded files.
My media settings for Django are
SETTINGS_DIR = os.path.dirname(__file__)
PROJECT_PATH = os.path.join(SETTINGS_DIR, os.pardir)
PROJECT_PATH = os.path.abspath(PROJECT_PATH)
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(PROJECT_PATH, 'media')
Is there a way to reference the media files properly from a view?
Here is the output of file_upload_dir and the location of the new file.
>>> print(file_upload_dir)
C:\\Users\\vut46744\\Desktop\\graphite_project\\media\\Data_Files
>>> print(os.path.join(file_upload_dir, new_file))
C:\\Users\\vut46744\\Desktop\\graphite_project\\media\\Data_Files\\TEST.CSV
Normally, you should not access files using open() in a Django app. You should use the storage API. This allows your code to play well with Django settings, and potential third party apps that augment this API.
https://docs.djangoproject.com/en/1.7/topics/files/#file-storage
So here you should be doing something like
from django.core.files.storage import default_storage
f = default_storage.open(os.path.join('Data_Files', new_file), 'r')
data = f.read()
f.close()
print(data)
By the way, if you want it to be modular, it would be a good idea to have a custom storage class, allowing easy configuration and use of your app, should your requirements change. Also, that allows putting files outside of MEDIA_ROOT. This sample storage will put them in settings.UPLOADS_ROOT (and default to MEDIA_ROOT if the setting is not found).
# Put this in a storage.py files in your app
from django.conf import settings
from django.core.files.storage import FileSystemStorage, get_storage_class
from django.utils.functional import LazyObject
class UploadsStorage(FileSystemStorage):
def __init__(self, location=None, base_url=None, *args, **kwargs):
if location is None:
location = getattr(settings, 'UPLOADS_ROOT', None)
super(UploadsStorage, self).__init__(location, base_url, *args, **kwargs)
self.base_url = None # forbid any URL generation for uploads
class ConfiguredStorage(LazyObject):
def _setup(self):
storage = getattr(settings, 'UPLOADS_STORAGE', None)
klass = UploadsStorage if storage is None else get_storage_class(storage)
self._wrapped = klass()
uploads_storage = ConfiguredStorage()
We create a very simple storage here. It's just the regular one, but that reads its files from another directory. Then we set up a lazy object that will allow overriding that storage from settings.
So now your code becomes:
from myapp.storage import uploads_storage
f = uploads_storage.open(new_files, 'r')
And in your settings, you set UPLOADS_ROOT to whatever you like. Probably something outside your media directory. And if someday you decide to store uploads in a database instead, you can set UPLOADS_STORAGE to a database-backed storage, your code will happily use it.
Related
in models.py,
class Files(models.Model):
File = models.FileField(upload_to="documents/")
in my views.py, all works perfectly when i remove the 3rd line
file = Files(File=f)
file.File.upload_to('Media/')
file.save()
Question is.
I want to create new upload_to Folder from views.py only. is it even POSSIBLE ?
I know the instance function method of creating Folder used in models.py, i dont want to use it
You can set the name of the file, and thus work with:
from django.conf import settings
from os.path import join as pathjoin
file = Files(File=f)
old_path = car.photo.path
file.File.name = f'Media/{file.File.name}'
new_path = pathjoin(settings.MEDIA_ROOT, file.File.name)
rename(path, new_path)
file.save()
We thus first construct a Files object with f as File, then we check the path, create a new name, and then move the file to the new directory and update the Files object in the database.
You can do this in bulk by enumerating over the Files items and set the file as described in the answer. Instead of saving the object directly, you can collect these in a list and perform a .bulk_update(…) [Django-doc]:
from django.conf import settings
from os.path import join as pathjoin
files = list(Files.objects.all())
for file in files:
old_path = car.photo.path
file.File.name = f'Media/{file.File.name}'
new_path = pathjoin(settings.MEDIA_ROOT, file.File.name)
rename(path, new_path)
Files.objects.bulk_update(files, ['File'])
My models.imageField uploads images to a directory outside of the django directory path. In storages.py I have a function 'upload_to' that sets the directory and filename. It all works perfectly except that I cannot overwrite an existing file with the same name, django simply appends some random text to the file name.
My question is:
How can I overwrite the remote file?
What I have tried:
I have played around with the script suggested here:
https://www.itachay.com/2020/04/override-replace-uploaded-file-in-django.html
Which would work if I was using MEDIA_ROOT, but with my remote file I only ever get the django "SuspiciousFileOperation" warning.
Here is my 'working' code albeit that it won't overwrite any existing file of the same name.
models.py
from .storages import upload_to, remote_storage
class Newsletter(models.Model):
image_1 = models.ImageField(upload_to=upload_to, storage=remote_storage)
storages.py
from django.core.files.storage import FileSystemStorage
def upload_to(instance, filename):
dir_name = 'volume-'+str(instance.pk)
filename = 'image_1'+filename[-4:]
return '%s/%s' % (dir_name, filename)
remote_storage = FileSystemStorage(
location='/Users/some_dir/Documents/dev/mediatest/',
base_url='/Users/some_dir/Documents/dev/mediatest/'
)
My Tries:
TRY 1:
settings.py
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
MEDIA_TRACK = '/Users/some_dir/Documents/dev/mediatest/'
models.py
from .storages import upload_to, OverwriteStorage
class Newsletter(models.Model):
image_1 = models.ImageField(upload_to=upload_to, storage=OverwriteStorage())
storages.py
FileSystemStorage(
location='/Users/some_dir/Documents/dev/mediatest/',
base_url='/Users/some_dir/Documents/dev/mediatest/'
)
class OverwriteStorage(FileSystemStorage):
def get_available_name(self, name, max_length=None):
if self.exists(name):
os.remove(os.path.join(settings.MEDIA_TRACK, name))
return name
Try 1 Condition:
New record created and image is saved.
Record edited and a new image is selected and saved.
Try 1 RESULT:
Image is saved to the local django directory, not the remote media/volume-1 (no errors)
FileNotFoundError at /admin/newsletter/newsletter/6/change/
[Errno 2] No such file or directory: '/Users/some_dir/Documents/dev/mediatest/volume-1/image_1.png'
TRY 2:
The only change to the above script is to OverwriteStorage()
class OverwriteStorage(FileSystemStorage):
def get_available_name(self, name, max_length=None):
if self.exists(os.path.join(settings.MEDIA_TRACK, name)):
os.remove(os.path.join(settings.MEDIA_TRACK, name))
return name
Try 2 Condition:
Same record edited and image is saved.
Try 2 Result
SuspiciousFileOperation at /admin/newsletter/newsletter/6/change/
The joined path (/Users/some_dir/Documents/dev/mediatest/volume-1/image_1.png) is located outside of the base path component (/Users/some_dir/Documents/dev/backoffice/media)
I'm having troubles trying to upload files to FileField from local path.
I have correctly configurred CDN backend in S3 bucket and use it as PrivateMediaStorage for one of my model fields:
class MyModel(models.Model):
some_file = models.FileField(storage=PrivateMediaStorage())
...
With this very simple configuration whenever I'm creating/updating model through django-admin it is saved and file attached as some_file is correctly uploaded to S3 bucket.
Yet if I try to create/update model instance programmatically, say through custom manage.py command, model instance itself is created but attachment is never uploaded to CDN. Here's simplified version of code I'm using to upload files:
class Command(BaseCommand):
help = 'Creates dummy instance for quicker configuration'
def handle(self, *args, **options):
some_file = os.path.join(os.path.dirname(__file__), '../../../temporary/some_image.png')
if not os.path.exists(some_file):
raise CommandError(f'File {some_file} does not exist')
else:
instance, created = MyModel.objects.get_or_create(defaults={'some_file': some_file}, ...)
What is missing in my implementation and what needs to be adjusted to allow file uploads from local storage?
You're passing a string (the result of os.path.join()) to your some_file field, but you need to pass it an actual File object.
The easiest way to save a file on a model directly is to use the FieldFile's save() method.
As a working solution for case provided in question a valid way of creating a record would be:
instance = MyModel.objects.create(some_file=File(file=open(some_file, 'rb'), name='some_name.png'))
Or even better to use pathlib to obtain name dynamically:
from pathlib import Path
instance = MyModel.objects.create(some_file=File(file=open(some_file, 'rb'), name=Path(some_file).name))
Note that fetching a row based on the file is unlikely to work, AFAIK each time you open a file, doing a get_or_create() with the File instance as argument will probably create a new row each time. Better put file fields into defaults:
with open(some_file, 'rb') as file:
instance, created = MyModel.objects.get_or_create(
some_other_field=...,
defaults={'some_file': File(
file=file,
name=pathlib.Path(some_file).name
)}
)
you can also do something like this.
some_file = os.path.join(os.path.dirname(__file__), '../../../temporary/some_image.png')
instance.some_file.name = some_file
instance.save()
I'm developing a Django project and I have the following "problem".
I have a local directory that has subdirectories, in which I got some PDF files. For example:
Main Dir:
|-->2000
____|-->A
________|-->file1.pdf
________|-->file2.pdf
____|-->B
________|-->file3.pdf
____|-->C
________|-->...
____|-->D
________|-->...
|-->2001
___|--> ...
|-->2002
___|--> ...
All the folders contain thousands of PDF files.
I want to display this directory in a Django view and let the user browse it by clicking the subdirectories and the PDF files so he can view them in his browser and maybe even add a "Download PDF" button. I also want to format it a bit, maybe add a search function too in the future if possible.
It is my first time working with local files and Django so I'm a bit lost.
Thank you for your time!
You could use ListView like this
class FileObject():
name = ''
def __init__(self, name):
self.name = name
class DirListView(ListView):
template_name = 'main/list_dir.html'
def get_queryset(self):
files = []
for filename in os.listdir(path):
fileobject = FileObject(name=filename)
files.append(fileobject)
return files
This ListView the could be used like any other ListView. Not sure about the permission and locations about path. This must be steared by settings.py I guess
How do you control version of your static files in Django? I wrote custom templatetag which adds the modification date as GET parameter of file URL, but would like to know - whether I'm doing.
Tag code:
import os
from django import template
from django.conf import settings
register = template.Library()
#register.simple_tag
def sstatic(path):
'''
Returns absolute URL to static file with versioning.
'''
full_path = os.path.join(settings.STATIC_ROOT, path)
try:
# Get file modification time.
mtime = os.path.getmtime(full_path)
return '%s%s?%s' % (settings.STATIC_URL, path, mtime)
except OSError:
# Returns normal url if this file was not found in filesystem.
return '%s%s' % (settings.STATIC_URL, path)
Applications such as django-compressor, and django-pipeline are good for these sort of things.
Django 1.7 added ManifestStaticFilesStorage which is designed to do this as part of the collectstatic cycle by appending hashes to filenames. By default it applies to all filetypes.
If you want to use it but want to specify which filetypes are versioned, then I wrote an extension that allows you to include / exclude files by path pattern.
As of Django 2.2 this ManifestStaticFilesStorage is still the recommended way to do this.