I've written an app that uses forms to collect information that is then sent in an email. Many of these forms have a filefield used to attach files to the email. I'd like to validate two things, the size of the file (to ensure the emails are accepted by our mail server. I'd also like to check the file extension, to discourage attaching file types not useable for our users.
(This is the python class I'm trying to extend)
class FileField(Field):
widget = FileInput
default_error_messages = {
'invalid': _(u"No file was submitted. Check the encoding type on the form."),
'missing': _(u"No file was submitted."),
'empty': _(u"The submitted file is empty."),
'max_length': _(u'Ensure this filename has at most %(max)d characters (it has %(length)d).'),
}
def __init__(self, *args, **kwargs):
self.max_length = kwargs.pop('max_length', None)
super(FileField, self).__init__(*args, **kwargs)
def clean(self, data, initial=None):
super(FileField, self).clean(initial or data)
if not self.required and data in EMPTY_VALUES:
return None
elif not data and initial:
return initial
# UploadedFile objects should have name and size attributes.
try:
file_name = data.name
file_size = data.size
except AttributeError:
raise ValidationError(self.error_messages['invalid'])
if self.max_length is not None and len(file_name) > self.max_length:
error_values = {'max': self.max_length, 'length': len(file_name)}
raise ValidationError(self.error_messages['max_length'] % error_values)
if not file_name:
raise ValidationError(self.error_messages['invalid'])
if not file_size:
raise ValidationError(self.error_messages['empty'])
return data
Just overload the "clean" method:
def clean(self, data, initial=None):
try:
if data.size > somesize:
raise ValidationError('File is too big')
(junk, ext) = os.path.splitext(data.name)
if not ext in ('.jpg', '.gif', '.png'):
raise ValidationError('Invalid file type')
except AttributeError:
raise ValidationError(self.error_messages['invalid'])
return FileField.clean(self, data, initial)
in my opinion subclassing the actual field class is way to much effort. It should be easier to simply extend your form class. You could add a method which "cleans" the file field.
For example:
class MyForm(forms.Form):
attachment = forms.FileField(...)
def clean_attachment(self):
data = self.cleaned_data['attachment'] // UploadedFile object
exts = ['jpg', 'png'] // allowed extensions
// 1. check file size
if data.size > x:
raise forms.ValidationError("file to big")
// 2. check file extension
file_extension = data.name.split('.')[1] // simple method
if file_extension not in exts:
raise forms.ValidationError("Wrong file type")
return data
This is only a basic example and there are some rough edges. But you can use this example and improve it until you have a version that works for you.
Recommended readings:
Django Doc - Cleaning a specific field
Django Doc - UploadedFile class
Django Doc - File class
This is what I ended up doing:
In my app's setting file:
exts = ['doc', 'docx', 'pdf', 'jpg', 'png', 'xls', 'xlsx', '.xlsm', '.xlsb']
max_email_attach_size = 10485760 #10MB written in bytes
In a new file I called formfunctions:
from django import forms
from django.forms.util import ErrorList, ValidationError
from app.settings import exts, max_email_attach_size
class SizedFileField(forms.FileField):
def clean(self, data, initial=None):
if not data in (None, ''):
try:
if data.size > max_email_attach_size:
raise ValidationError("The file is too big")
file_extension = data.name.split('.')[1]
if file_extension not in exts:
raise ValidationError("Invalid File Type")
except AttributeError:
raise ValidationError(self.error_messages['invalid'])
return forms.FileField.clean(self, data, initial)
and in my forms file:
from formfunctions import SizedFileField
An example class from the forms file:
class ExampleClass(forms.Form):
Email_Body = forms.CharField(widget=forms.Textarea, required=False)
Todays_Date = forms.CharField()
Attachment = SizedFileField(required=False)
Related
I have a model Message that has a FileField. My API accepts files in Base64 encoding so they can be sent alongside other data.
To know a filename and an extension, there is one more field attachment_filename in the serializer that is not a model field. It is used inside Base64Field.
I want to be able to validate if there are both attachment_filename , attachment, or none of them.
The problem is that if the attachment_filename is read-only, it is not present in validate - data variable.
On the other hand, if it's required=False, allow_null=True, the serializer raises an error when creating a message:
TypeError: ChatMessage() got an unexpected keyword argument 'attachment_filename'
Code:
class Base64File(Base64FileField): # todo make accept a list of extensions (finite eg. pdf, xlsx, csv, txt )
ALLOWED_TYPES = ['pdf', 'xlsx', 'png', 'jpg', 'jpeg', 'docx', 'doc', 'zip']
def get_file_extension(self, filename, decoded_file):
extension = self.get_full_name().split('.')[-1]
return extension
def get_file_name(self, decoded_file):
attachment_filename = self.get_full_name()
return '.'.join(attachment_filename.split('.')[:-1])
def get_full_name(self):
return self.context['request'].data['attachment_filename'] # todo validate name
class ChatMessageSerializer(serializers.ModelSerializer):
attachment = Base64File(required=False)
attachment_filename = serializers.CharField(required=False, allow_null=True)
class Meta:
model = ChatMessage
fields = '__all__'
def validate(self, data):
"""
Validation of start and end date.
"""
attachment = data.get('attachment')
attachment_filename = data.get('attachment_filename')
if bool(attachment) ^ bool(attachment_filename):
raise serializers.ValidationError("Either none or both 'attachment' and 'attachment_filename' must be present")
# del data['attachment_filename'] # works but dirty
return data
How to make it work?
EDIT
I managed to make it work by adding
del data['attachment_filename']
before validate method return but that seems to be too "dirty".
You should handle this behavior in serializer.save method, for example, you can pop it from validated_data like that:
def save(self, **kwargs):
self.validated_data.pop("attachment_filename")
return super().save(**kwargs)
I want to download multiple files in Django without creating zip acrhive.
I have a valid which uses zip (create 1 zip file and download it)
But I have to implement downloading several files without zip archive creating. How can I modify my code?
class DocumentView(GenericAPIView):
def get(self, request, *args, **kwargs):
document_type = kwargs.get("document_type", None)
user_id = kwargs.get("user_id", None)
try:
user = User.objects.get(id=user_id)
except User.DoesNotExist:
raise NotFound("This is user not found.")
if document_type == 'vehicle_photo':
user_vehicle = user.vehicle.select_related().first()
documents = user_vehicle.photos.all()
else:
documents = user.document_owner.select_related().filter(document_type=document_type)
in_memory = BytesIO()
zip_filename = f"{document_type}_{user_id}.zip"
zip_archive = ZipFile(in_memory, "w")
for document in documents:
f_dir, f_name = os.path.split(document.photo.url if document_type == "vehicle_photo" else
document.file.url)
zip_path = f"{settings.ROOT_DIR}{f_dir}"
zip_archive.write(zip_path+"/"+f_name, f_name)
# Save zip file
zip_archive.close()
response = HttpResponse(content_type="application/zip")
response['Content-Disposition'] = f'attachment; filename={zip_filename}'
in_memory.seek(0)
response.write(in_memory.read())
return response
I'm trying to upload a file to a django backend using graphene-file-upload which has the mutation to link the backend to the react frontend where I'm trying to use apollo-upload client to link it with graphql.
In my django model an empty file is being successfully uploaded but it is not uploading the real file which I'm selecting but uploads an empty file.
Like it uploads nothing {} but the instance is created in the database where another story is added which is empty.
Here is some of my code.
My Database Model. models.py
class Story(models.Model):
file = models.FileField(null=True)
created_at = models.DateTimeField(auto_now_add=True)
MY schema.py
from graphene_file_upload.scalars import Upload
class StoryType(DjangoObjectType):
class Meta:
model = Story
class UploadFile(graphene.Mutation):
story = graphene.Field(StoryType)
class Arguments:
file = Upload()
def mutate(self, info, file):
for line in file:
print(line)
story = Story(file=file)
story.save()
return UploadFile(story=story)
My frontend File.js
import React from 'react';
import { Mutation } from 'react-apollo';
import {withStyles} from '#material-ui/core/styles';
import gql from 'graphql-tag';
const styles = theme => ({
layoutRoot: {}
});
const UploadFile = () => (
<Mutation
mutation={gql`
mutation($file: Upload!) {
uploadFile(file: $file) {
story {
file
}
}
}
`}
>
{mutate => (
<input
type="file"
required
onChange={({
target: {
validity,
files: [file]
}
}) => validity.valid && mutate({ variables: { file } })}
/>
)}
</Mutation>
)
export default withStyles(styles, {withTheme: true})(UploadFile);
It's working for me now I've overridden parse_body in GraphQLView, in order for it to deal with multipart/form-data correctly.
# views.py
from django.http.response import HttpResponseBadRequest
from graphene_django.views import GraphQLView
class MyGraphQLView(GraphQLView):
def parse_body(self, request):
content_type = self.get_content_type(request)
if content_type == "application/graphql":
return {"query": request.body.decode()}
elif content_type == "application/json":
# noinspection PyBroadException
try:
body = request.body.decode("utf-8")
except Exception as e:
raise HttpError(HttpResponseBadRequest(str(e)))
try:
request_json = json.loads(body)
if self.batch:
assert isinstance(request_json, list), (
"Batch requests should receive a list, but received {}."
).format(repr(request_json))
assert (
len(request_json) > 0
), "Received an empty list in the batch request."
else:
assert isinstance(
request_json, dict
), "The received data is not a valid JSON query."
return request_json
except AssertionError as e:
raise HttpError(HttpResponseBadRequest(str(e)))
except (TypeError, ValueError):
raise HttpError(HttpResponseBadRequest("POST body sent invalid JSON."))
# Added for graphql file uploads
elif content_type == 'multipart/form-data':
operations = json.loads(request.POST['operations'])
files_map = json.loads(request.POST['map'])
return place_files_in_operations(
operations, files_map, request.FILES)
elif content_type in [
"application/x-www-form-urlencoded",
#"multipart/form-data",
]:
return request.POST
return {}
def place_files_in_operations(operations, files_map, files):
# operations: dict or list
# files_map: {filename: [path, path, ...]}
# files: {filename: FileStorage}
fmap = []
for key, values in files_map.items():
for val in values:
path = val.split('.')
fmap.append((path, key))
return _place_files_in_operations(operations, fmap, files)
def _place_files_in_operations(ops, fmap, fobjs):
for path, fkey in fmap:
ops = _place_file_in_operations(ops, path, fobjs[fkey])
return ops
def _place_file_in_operations(ops, path, obj):
if len(path) == 0:
return obj
if isinstance(ops, list):
key = int(path[0])
sub = _place_file_in_operations(ops[key], path[1:], obj)
return _insert_in_list(ops, key, sub)
if isinstance(ops, dict):
key = path[0]
sub = _place_file_in_operations(ops[key], path[1:], obj)
return _insert_in_dict(ops, key, sub)
raise TypeError('Expected ops to be list or dict')
def _insert_in_dict(dct, key, val):
return {**dct, key: val}
def _insert_in_list(lst, key, val):
return [*lst[:key], val, *lst[key+1:]]
Requirements
from python_graphql_client import GraphqlClient
import asyncio
client = GraphqlClient(endpoint="http://localhost:8000/graphql")
Client-side -
def test_archive_upload(client):
file_header = "data:application/zip;base64,"
with open("files/Archive.zip", 'rb') as file:
query = """
mutation uploadFile($file: Upload) {
uploadFile(file:$file) {
ok
}
}
"""
file = file.read()
file = file_header + base64.b64encode(file).decode("UTF-8")
variables = {"file": file}
data = client.execute(query=query, variables=variables)
Run -
asyncio.get_event_loop().run_until_complete(test_archive_upload(client))
Server-side -
file = file.split(",")
file[0] = file[0]+","
file_type = guess_extension(guess_type(file[0])[0])
file = base64.b64decode(file[1])
with open("files/test"+ file_type, "wb") as w_file:
w_file.write(file)
I have set up a form and view to upload multiple *.gpx files to my website at once. These files are validated using a clean() method on the form and then once validated passed to a function for processing.
When I upload some invalid files the clean() method catches them and informs the user as expected.
When I upload some valid files the processing function crashes with an error saying the files are empty.
If I comment out the clean() method then the valid files are uploaded fine.
What can be happening to the form during the clean() method than means the files are being blanked?
here is my form:
class UploadGpxForm(forms.Form):
gpx_file = forms.FileField(widget=forms.ClearableFileInput(attrs={'multiple': True}))
here is my view:
class UploadGpxView(FormView):
form_class = UploadGpxForm
template_name = 'dashboard/upload.html' # Replace with your template.
success_url = reverse_lazy('dashboard:index') # Replace with your URL or reverse().
def post(self, request, *args, **kwargs):
form_class = self.get_form_class()
form = self.get_form(form_class)
files = request.FILES.getlist('gpx_file')
if form.is_valid():
for f in files:
SaveGPXtoPostGIS(f)
return self.form_valid(form)
else:
return self.form_invalid(form)
Here is my clean method for the UploadGpxForm:
def clean(self):
file_errors=[]
files = list(self.files.getlist('gpx_file'))
for f in list(files):
#check file has only one full stop in it.
if len(f.name.split('.')) != 2:
file_errors.append(ValidationError(
_('%(file_name)s has not been uploaded:'\
'File type is not supported')
, params = { 'file_name': f.name }
, code = 'file_type')
)
#check file doesn't breach the file size listed in settings
if f.content_type in settings.DASHBOARD_UPLOAD_FILE_TYPES:
if f._size > settings.DASHBOARD_UPLOAD_FILE_MAX_SIZE:
file_errors.append(ValidationError(
_('%(file_name)s has not been uploaded: File too big.'\
'Please keep filesize under %(setting_size)s.'\
'Current filesize %(file_size)s') ,
params = {
'file_name': f.name,
'setting_size': filesizeformat(
settings.DASHBOARD_UPLOAD_FILE_MAX_SIZE),
'file_size': filesizeformat(f._size)
},
code = 'file_size'
)
)
#check it is one of our allowed file types
else:
file_errors.append(ValidationError(
_('%(file_name)s has not been uploaded:'\
'File type is not supported')
, params = { 'file_name' : f.name }
, code = 'file_type'
)
)
#next check the file hasn't been uploaded before
#generate MD5
md5hash = md5()
for chunk in f.chunks():
md5hash.update(chunk)
file_hash = md5hash.hexdigest()
if gpxTrack.objects.filter(file_hash=file_hash).exists():
file_errors.append(ValidationError(
_('%(file_name)s has not been uploaded as a identical file'\
'has already been uploaded previously'),
params = { 'file_name' : f.name },
code = 'file_hash'))
#finally raise errors if there are any
if len(file_errors) > 0:
raise ValidationError(file_errors)
else:
return files
When you read the file content (for calculating md5 hash) you need to move the file object’s position to the beginning (0th byte) using file.seek:
md5hash = md5()
for chunk in f.chunks():
md5hash.update(chunk)
file_hash = md5hash.hexdigest()
f.seek(0) #<-- add this line
Im using django 1.10 + Apache in Linux.
I've created a small webapp to upload documents (with dropzone.js) and want to implement the ability for a user to specify who can view/modify/delete a specific file but i can't figure out a way how. I attempted using a ManyToManyField but maybe im not understading the Field itself correctly.
The "Document" model is this:
Model
class Document(models.Model):
file = models.FileField(upload_to = 'files/')
#validators=[validate_file_type])
uploaded_at = models.DateTimeField(auto_now_add = True)
extension = models.CharField(max_length = 30, blank = True)
thumbnail = models.ImageField(blank = True, null = True)
is_public = models.BooleanField(default = False)
accesible_by = models.ManyToManyField(User) #This is my attempt at doing this task.
def clean(self):
self.extension = self.file.name.split('/')[-1].split('.')[-1]
if self.extension == 'xlsx' or self.extension == 'xls':
self.thumbnail = 'xlsx.png'
elif self.extension == 'pptx' or self.extension == 'ppt':
self.thumbnail = 'pptx.png'
elif self.extension == 'docx' or self.extension == 'doc':
self.thumbnail = 'docx.png'
def delete(self, *args, **kwargs):
#delete file from /media/files
self.file.delete(save = False)
#call parent delete method.
super().delete(*args, **kwargs)
#Redirect to file list page.
def get_absolute_url(self):
return reverse('dashby-files:files')
def __str__(self):
return self.file.name.split('/')[-1]
class Meta():
ordering = ['-uploaded_at']
My View to handle the creation of documents:
View
class DocumentCreate(CreateView):
model = Document
fields = ['file', 'is_public']
def form_valid(self, form):
self.object = form.save(commit = False)
## I guess here i would Add the (self.request.user) to the accesible_by Field.
self.object.save()
data = {'status': 'success'}
response = JSONResponse(data, mimetype =
response_mimetype(self.request))
return response
Thanks in advance to anyone for any ideas or suggestions...
You have a model and a view that hopefully works for adding new documents, you still have a number of steps to go.
You'll need a place to assign users that can view/modify/delete your files. If you need to store access levels (view/delete...), your accessible_by will not suffice and you'll do well with a through table to add more information like access level.
You need to write views for various actions like view, delete... that users will request and here you ensure users have the right privileges. An implementation would be to get the request.user and the document id, look up if the user has the permission for what she's doing, return an http unauthorized exception or allow the action to proceed.
Edit: My question is about how can I assign user-permissions to each
individual file
If we're keeping this to access control from the django level, using the document model you already have, and you've taken some steps and for every document, you can assign users (accessible_by). Something like this can get you started:
from django.core.exceptions import PermissionDenied
def view_document(request, doc_pk):
doc = get_object_or_404(Document, pk=doc_pk)
if not doc.accessible_by.filter(username=request.user.username):
raise PermissionDenied
#perform rest of action
Or do you mean to use the permissions framework itself?