I want to make Django accept url that consists of infinite number of parameters. Something like dropbox has going on. Each parameter for each folder the file is in. There could be infinite number of subfolders. Parameter is alphanumeric.
You can create a URL that accepts an arbitrarily long parameter than contains some form of separator and then in your view split the parameter by this separator. For a path like parameter:
url(r'^prefix/(?P<path>[a-zA-Z\/]*)/$', your_view),
def your_view(request, path):
folders = path.split('/')
Now any request to this URL like prefix/folder1/folder2/folder3/ and folders will contain ['folder1', 'folder2', 'folder3']
I don't think so that you can define infinite url parameters with django URLS. Because with django you have to declare the urls that you site will gonna use.
But if you are talking specifically of folder and files, you can do this with FileField in a model, that allows you to decide where you can save files and save the path in instance.filefield.url. then insted of search django urls, search nginx url, something like this. {site_url}/static/{any}/{folder}/{you}/{want}/{etc}/.
Inclusive, you can use models like this.
class Folder(models.Model):
name = models.CharField(...)
parent = mdoels.ForeignKey('self')
...
def get_path_to_here(self):
# here you have the full url getting all parents folders.
...
def func_to_declare_where_save_it(instance, filename):
return os.path.join(
f'{instance.folder.get_path_to_here()}',
f'{filename}')
class File(models.Model):
name = models.CharField(...)
folder = models.ForeignKey(Folder)
file = models.FileField(upload_to=func_to_declare_where_save_it)
Related
I have a Django project where users can register and add their XML/CSV feeds.
What the project does is (everyday morning):
download the feed
parse the feed and store it to the model Product
generate XML/CSV export from these products in a different format
Now the problem is that users can register anytime during the project lifetime and many feeds have to be either downloaded, parsed or exported in a specific way.
So I need to be able to react quickly when the feed is added and write custom functions/classes/scripts to download/parse or export these sources.
Let's say it would be enough to add or extend some base classes like Downloader,Parser or Exporter
I'm looking for the most common way to do this.
I tried multiple approaches. For example, I've created a package parsers and there is a __init__.py with CHOICES attribute. When a user adds new feed, I create a parser__sourcename.py file which I add into parsers package which contains Parser class which extends BaseParser class with parse method where is the custom code.
Then I add import into __init__.py file so it looks like this:
from feeds_core.parsers.parser__source1 import Parser as ParserForSource1
from feeds_core.parsers.parser__source2 import Parser as ParserForSource2
from feeds_core.parsers.parser__source3 import Parser as ParserForSource3
PARSER__SOURCE1 = 'source1'
PARSER__SOURCE2 = 'source2'
PARSER__SOURCE3 = 'source3'
CHOICES = {
PARSER__SOURCE1: ParserForSource1,
PARSER__SOURCE2: ParserForSource2,
PARSER__SOURCE3: ParserForSource3,
}
def get_parser(choice):
return CHOICES[choice]
Then I have a model Source with this field:
PARSER_CHOICES = [(x, x) for x in CHOICES.keys()]
parser = models.CharField(max_length=128, choices=PARSER_CHOICES, null=True, blank=True)
def get_parser(self):
...
elif self.parser:
return self.get_parser_class()(self.get_last_filepaths(), self.user_eshop, self)
else:
raise self.NoParserDefinedException
def parse(self): # TODO raise spec exc
self.parse_started()
self.get_parser().parse()
self.parse_ended()
def get_parser_class(self) -> BaseParser:
return get_parser(self.parser)
And when there is a new feed, I create the file, modify the __init__.py and choose the parser for this source in the Django admin interface.
But it is little bit complicated and moreover, I'm afraid I have to restart production server every time.
Do you have any ideas/experiences? Is there some best practice how to do such things?
A very basic implementation where you would add a file to a directory (settings.PARSER_FILE_DIR), it would then be an available choice for Source.parser_file, then using importlib you would load the file and extract the Profile class out of it. settings.PARSER_FILE_DIR would probably have to be in your PYTHONPATH
This is an extremely basic example and you would need a lot of error handling and would need to make sure it was very secure
import importlib
class Source(models.Model):
parser_file = models.FilePathField(path=settings.PARSER_FILE_DIR, match='.*\.py$')
def get_parser_class(self):
return importlib.import_module(self.parser_file.replace('.py', '')).Parser
I'm developing a Django project and I have the following "problem".
I have a local directory that has subdirectories, in which I got some PDF files. For example:
Main Dir:
|-->2000
____|-->A
________|-->file1.pdf
________|-->file2.pdf
____|-->B
________|-->file3.pdf
____|-->C
________|-->...
____|-->D
________|-->...
|-->2001
___|--> ...
|-->2002
___|--> ...
All the folders contain thousands of PDF files.
I want to display this directory in a Django view and let the user browse it by clicking the subdirectories and the PDF files so he can view them in his browser and maybe even add a "Download PDF" button. I also want to format it a bit, maybe add a search function too in the future if possible.
It is my first time working with local files and Django so I'm a bit lost.
Thank you for your time!
You could use ListView like this
class FileObject():
name = ''
def __init__(self, name):
self.name = name
class DirListView(ListView):
template_name = 'main/list_dir.html'
def get_queryset(self):
files = []
for filename in os.listdir(path):
fileobject = FileObject(name=filename)
files.append(fileobject)
return files
This ListView the could be used like any other ListView. Not sure about the permission and locations about path. This must be steared by settings.py I guess
I am using DRF backend to upload files. In my specific case I will want to get the name of the file, after it has been uploaded. The reason is that if a user uploads a file with same name, I am still able to process it independently.
views.py:
class ImageUploaderView(viewsets.ModelViewSet):
renderer_classes = [renderers.JSONRenderer]
queryset = ImageUploader.objects.all()
serializer_class = ImageUploaderSerializer
parser_classes = (MultiPartParser,)
serializer.py:
class ImageUploaderSerializer(serializers.ModelSerializer):
class Meta:
model = ImageUploader
models.py:
class ImageUploader(models.Model):
# name=models.ImageField(upload_to='media')
name=models.FileField(upload_to='media')
I tried to put signals and hooks after the model definitions but I am not being able to get this filename. Can someone shed some light pls?
UPDATE: Let me elaborate what I want to achieve essentially:
User1 hits endpoint "/api/calculate_interest_rate" which is rendered
by a frontend React component. "calculate_interest_rate" is served by
DRF, and lets the user upload a CSV file. This will be stored as
"user1.csv", the file is processed and then tabulated (rendered by
React).
At the same time and in parallel to User1, User2 hits the same endpoint "/api/calculate_interest_rate" and
by mistake he saves his file as "user1.csv", and uploads it to the systemn.
So I want to be able to detect both names of the file in order to process it. By using always the same default filename (ex. using the OverwriteStorage() technique), I will probably cause chaos when two or more users are using the same filename. Therefore I am looking into a technique that allows me to get the filename as is, and process it immediately.
How about using storage option?
class OverwriteStorage(FileSystemStorage):
def get_available_name(self, name, max_length=None):
print("filename", name)
#parts = name.split('.') you can separate name and extension.
return super().get_available_name(name)
upload_image = models.ImageField(
upload_to=[yourpath],
default=[defaultname],
storage=OverwriteStorage()
)
I suggest you to following this configuration:
1. Change your MEDIA_ROOT and MEDIA_URL inside file of settings.py
MEDIA_URL = '/media/'
MEDIA_ROOT = '/path/to/env/projectname/media'
2. Then, I suggest you to change your upload_to='media to upload_to='images/%Y/%m/%d, also rename your field of name with image.
class ImageUploader(models.Model):
image = models.FileField(upload_to='images/%Y/%m/%d')
# OR
# image = models.ImageField(upload_to='images/%Y/%m/%d')
Explanation; If you following this configuration, you could have uploaded images are following, eg: /media/images/2017/01/29/yourimage.jpg. This is one way to handle the problem of duplicated files.
3. But if you want to upload file with multiple times without duplicate files, you can using deconstructible;
import os, time, uuid
from django.db import models
from django.utils.deconstruct import deconstructible
class ImageUploader(models.Model):
#deconstructible
class PathAndRename(object):
def __init__(self, sub_path):
self.path = sub_path
def __call__(self, instance, filename):
# eg: filename = 'my uploaded file.jpg'
ext = filename.split('.')[-1] #eg: '.jpg'
uid = uuid.uuid4().hex[:10] #eg: '567ae32f97'
# eg: 'my-uploaded-file'
new_name = '-'.join(filename.replace('.%s' % ext, '').split())
# eg: 'my-uploaded-file_64c942aa64.jpg'
renamed_filename = '%(new_name)s_%(uid)s.%(ext)s' % {'new_name': new_name, 'uid': uid, 'ext': ext}
# eg: 'images/2017/01/29/my-uploaded-file_64c942aa64.jpg'
return os.path.join(self.path, renamed_filename)
image_path = time.strftime('images/%Y/%m/%d')
image = models.ImageField(upload_to=PathAndRename(self.image_path))
I have a Django model that saves filename as "uuid4().pdf". Where uuid4 generates a random uuid for each instance created. This file name is also stored on the amazon s3 server with the same name.
I am trying to add a custom disposition for filename that i upload to amazon s3, this is because i want to see a custom name whenever i download the file not the uuid one. At the same time, i want the files to stored on s3 with the uuid filename.
So, I am using django-storages with python 2.7. I have tried adding content_disposition in settings like this:
AWS_CONTENT_DISPOSITION = 'core.utils.s3.get_file_name'
where get_file_name() returns the filename.
I have also tried adding this to the settings:
AWS_HEADERS = {
'Content-Disposition': 'attachments; filename="%s"'% get_file_name(),
}
no luck!
Do anyone of you know to implement this.
Current version of S3Boto3Storage from django-storages supports AWS_S3_OBJECT_PARAMETERS global settings variable, which allows modify ContentDisposition too. But the problem is that it is applied as is to all objects that are uploaded to s3 and, moreover, affects all models working with the storage, which may turn to be not the expected result.
The following hack worked for me.
from storages.backends.s3boto3 import S3Boto3Storage
class DownloadableS3Boto3Storage(S3Boto3Storage):
def _save_content(self, obj, content, parameters):
"""
The method is called by the storage for every file being uploaded to S3.
Below we take care of setting proper ContentDisposition header for
the file.
"""
filename = obj.key.split('/')[-1]
parameters.update({'ContentDisposition': f'attachment; filename="{filename}"'})
return super()._save_content(obj, content, parameters)
Here we override native save method of the storage object and make sure proper content disposition is set on each file.
Of course, you need to feed this storage to the field you working on:
my_file_filed = models.FileField(upload_to='mypath', storage=DownloadableS3Boto3Storage())
In case someone finds this, like I did: none of the solutions mentioned on SO worked for me in Django 3.0.
Docstring of S3Boto3Storage suggests overriding S3Boto3Storage.get_object_parameters, however this method only receives name of the uploaded file, which at this point has been changed by upload_to and can differ from the original.
What worked is the following:
class S3Boto3CustomStorage(S3Boto3Storage):
"""Override some upload parameters, such as ContentDisposition header."""
def _get_write_parameters(self, name, content):
"""Set ContentDisposition header using original file name.
While docstring recomments overriding `get_object_parameters` for this purpose,
`get_object_parameters` only gets a `name` which is not the original file name,
but the result of `upload_to`.
"""
params = super()._get_write_parameters(name, content)
original_name = getattr(content, 'name', None)
if original_name and name != original_name:
content_disposition = f'attachment; filename="{original_name}"'
params['ContentDisposition'] = content_disposition
return params
and then using this storage in the file field, e.g.:
file_field = models.FileField(
upload_to=some_func,
storage=S3Boto3CustomStorage(),
)
Whatever solution you come up with, do not change file_field.storage.object_parameters directly (e.g. in model's save() as it's been suggested in a similar question), because this will change ContentDisposition header for subsequent file uploads of any field that uses the same storage. Which is not what you probably want.
I guess you are using S3BotoStorage from django-storages, so while uploading the file to S3, override the save() method of the model, and set the header there.
I am giving an example below:
class ModelName(models.Model):
sthree = S3BotoStorage()
def file_name(self,filename):
ext = filename.split('.')[-1]
name = "%s/%s.%s" % ("downloads", uuid.uuid4(), ext)
return name
upload_file = models.FileField(upload_to=file_name,storage = sthree)
def save(self):
self.upload_file.storage.headers = {'Content-Disposition': 'attachments; filename="%s"' %self.upload_file.name}
super(ModelName, self).save()
One way can be giving ResponseContentDisposition parameter to S3Boto3Storage.url() method. In this case you don't have to create a custom storage.
Example model:
class MyModel(models.Model):
file = models.FileField(upload_to=generate_upload_path)
original_filename = models.CharField(max_length=255)
Creating URL for your file:
# obj is instance of MyModel
url = obj.file.storage.url(
obj.file.name,
parameters={
'ResponseContentDisposition': f'inline; filename={obj.original_filename}',
},
)
If you want to force browser to download the file, replace inline with attachment.
If you are using non-ascii filenames, check how Django encodes filename for Content-Disposition header in FileResponse.
If I have a Django model with a FileField e.g.,
class Probe(models.Model):
name = models.CharField(max_length=200, unique=True)
nanoz_file = models.FileField(upload_to='nanoz_file', blank=True)
is there a way to prevent an uploaded file from being overwritten if a user uploads a new file in the Admin interface?
Also if I do keep the old files around is there a way I can relate the previous files back to the model instance?
I.e., I'd like to be able to list all files uploaded to the nanoz_file field for a given model instance.
Django never overwrites an uploaded file. If you upload 'foo.png' twice, the second will be 'foo_1.png' - I just tested this but don't take my word for it: try it too !
All you have to do (or let django-reversion do) is keep track of the previous file names.
You can use this structure:
class File(models.Model):
name = models.CharField()
file = models.FileField(upload_to='files_storage/')
belongs = models.ForeignKey('self')
creation = models.DateTimeField(auto_now_add=True)
Then in the view you can use something like:
def edit_file(request, ...):
# Get the file model instance
file_model = ... # Code to get the instance
# Create a new instance of the model with the old file path
old_file = File(name='file1-v2', file=file_model.file, belongs=file_model)
old_file.save()
# Update the file_model with the new file data
Hope this helps!
Just as jpic said, you could try django-reversion, or
track names of past files in sorts of DB, for example in seperated table row, in customized field or in gfk field.
glob files online, as long as filename is managed.
For the second way, actually for dealing w/ all user uploads, its better to name the file by the pattern you designed instead of using the raw name (you could also store the raw name for later usage) . For your case, since the name field is unique, the field is suitable as the base for generating filename of the uploaded files, if it rarely changes:
import os.path
from django.hash_compat import sha_constructor
def upload_to(self, filename):
return 'nanoz_file/%s%s' % (
sha_constructor(self.name).hexdigest(), os.path.splitext(filename)[-1])
class Probe(models.Model):
name = models.CharField(max_length=200, unique=True)
nanoz_file = models.FileField(upload_to=upload_to, blank=True)
Then in your view, you could fetch the list of names of all files of Probe instance probe by
import glob
# be careful to operate directory securely
glob.glob(os.path.join(
os.path.dirname(probe.nanoz_file.path),
'%s*' % sha_constructor(probe.name).hexdigest()))