Develop Reference

Ignore Django Model fields in migrations

I want to create a Django model that contains a FileField to store image and video files, but I want to validate the files before saving the instance. I've thought about adding three fields:
file: A FileField field. This will only be used to have a file column in the database, but serializers won't use it (instead they will use the two next fields).
file_image: An ImageField to perform image file validation. Before the model instance is saved, the file will be assigned to the file field. I don't want this field to have a dabatase representation.
file_video: A VideoField (custom field) to perform video file validation. Before the model instance is saved, the file will be assigned to the file field. I don't want this field to have a dabatase representation.
Of course, file_image and file_video won't be set at the same time.
The problem is preventing makemigrations from including file_image and file_video in the migrations. I could edit the migration file by hand, but I wonder if there is any way to automatically ignore these fields.
class MyModel(models.Model):
file = models.ImageField()
file_image = models.ImageField() # Not an actual column
file_video = models.VideoField() # Not an actual column
def save(self, *args, **kwargs):
if self.file_image.file is not None:
self.file.file = self.file_image.file
elif self.file_video.file is not None:
self.file.file = self.file_video.file
else:
raise ValidationError()
super().save(*args, **kwargs)

Your model is a representation of what's in the database. I would advise you not to fight against the ORM in this manner. Instead, I would perform the validation within the form class that's used when creating/updating the instance. With the form you can define the fields, file_video and file_image, then whichever is used, use that field to write to file.

Ok, so I abandoned my original idea of adding extra fields to the model and I only left file field, as suggested by #schillingt. Now I'm using a custom validation in the clean() method to validate the file type.
from django.db import models
from django.db.models.fields.files import ImageFieldFile
from django import forms
from custom_fields import forms as custom_forms
from custom_fields import fields as custom_fields
class MyModel(models.Model):
file = models.ImageField()
def clean(self):
# Get the MIME type to have a hint of the file type
import magic
mime_type = magic.from_buffer(self.file.file.read(1024), mime=True)
if mime_type.startswith('image'):
image_field = ImageFieldFile(self, models.ImageField(),
self.file.name)
image_field.file = self.file.file
if image_field.width:
self.width = image_field.width
self.height = image_field.height
self.media_type = mime_type
else:
raise ValidationError({'file': forms.ImageField.
default_error_messages['invalid_image']})
elif mime_type.startswith('video'):
video_field = custom_fields.VideoFieldFile(self,
custom_fields.VideoField(),
self.file.name)
video_field.file = self.file.file
if video_field.width:
self.width = video_field.width
self.height = video_field.height
self.duration = video_field.duration
self.media_type = mime_type
else:
raise ValidationError({'src': custom_forms.VideoField.
default_error_messages['invalid_video']})
else:
raise ValidationError({'src': 'The file is neither an image nor a video.'})
def save(self, *args, **kwargs):
self.full_clean()
super().save(*args, **kwargs)
I know it's an ugly solution, but could this work? I mean, I tested it and it seems to work, but surely there will be a better and more elegant way.

File manipulation in Django models.py

I am building a Django app that saves an .stl file passed using a formulary and my goal is to open the file, extract some information with a script that is already tested, and save this information in the same register that the file.
I am doing this:
from stl import mesh # numpy-stl library
def informationGeneration(stl_route, *args, **kwargs):
# scripts that generates the information
myMesh = mesh.Mesh.from_file(stl_route) # here the error appears
return myMesh.areas.shape[0]
class Piece(models.Model):
"""Piece model."""
# ...
file = models.FileField(upload_to='pieces/files', default='NA')
# ...
information = models.IntegerField(default=0)
def save(self, *args, **kwargs):
"""Overriding the save method."""
self.information = informationGeneration(self.file)
super().save(*args, **kwargs)
def __str__(self):
# ...
The problem is that when I try to save a new instance, numpy-stl detects an error, self.file is not the .stl file, is an alement of the formulary.
Then, I use a form:
class PieceForm(forms.ModelForm):
"""Pieces model form."""
class Meta:
"""Form settings."""
model = Piece
fields = ('file')
How can I pass the file and not the route?

Piece.file is not a path, it's a models.FileField. To get the path, you have to use self.file.path.
Just beware that if there's actually no file for this field, self.file.path will raise an exception (ValueError, "the file attribute has no file associated with it"), so it's better to test before. models.FileField have a false value in a boolean context, so you want:
if self.file:
self.information = informationGeneration(self.file.path)
A couple notes:
1/ a function is an action, so it's name should be a verb (ie "extract_informations")
2/ you probably don't want to re-parse the file's content each and every time your object is saved, only when the file has changed. You can use a md5sum (stored in the model) to check this.
3/ I have not double-checked but I really dont think you should use a default for this field - if you want to make it optional, use blank=True and null=True.

Where is a django validator function's return value stored?

In my django app, this is my validator.py
from django.core.exceptions import ValidationError
from django.core.validators import URLValidator
def validate_url(value):
url_validator = URLValidator()
url_invalid = False
try:
url_validator(value)
except:
url_invalid = True
try:
value = "http://"+value
url_validator(value)
url_invalid = False
except:
url_invalid = True
if url_invalid:
raise ValidationError("Invalid Data for this field")
return value
which is used to validate this :
from django import forms
from .validators import validate_url
class SubmitUrlForm(forms.Form):
url = forms.CharField(label="Submit URL",validators=[validate_url])
When I enter URL like google.co.in, and print the value right before returning it from validate_url, it prints http://google.co.in but when I try to get the cleaned_data['url'] in my views, it still shows google.co.in. So where does the value returned by my validator go and do I need to explicitly edit the clean() functions to change the url field value??
The doc says the following:
The clean() method on a Field subclass is responsible for running to_python(), validate(), and run_validators() in the correct order and propagating their errors. If, at any time, any of the methods raise ValidationError, the validation stops and that error is raised. This method returns the clean data, which is then inserted into the cleaned_data dictionary of the form.
I am still not sure where the validator return value goes and if it is possible to change cleaned_data dict using the validator.

From the docs:
A validator is merely a callable object or function that takes a value
and simply returns nothing if the value is valid or raises a
ValidationError if not.
The return value is simply ignored.
If you want to be able to modify the value you may use clean_field on the forms as described here:
class SubmitUrlForm(forms.Form):
url = ...
def clean_url(self):
value = self.cleaned_data['url']
...
return updated_value
Validators are only about validating the data, hence that is why the return value of the validator gets ignored.
You are looking for data "cleaning" (transforming it in a common form). In Django, Forms are responsible for data cleaning.

Use URLField. It validates value and prepends http if neccessary.

Django: removed object causes IndexError

I am a bit confused and I need some help.
I am displaying my objects using ModelFormset, then I am dynamically removing them using Ajax and then saving all of the objects again also using Ajax call. Everything is dynamic and the page is not reloaded at any time.
The problem is that when Django tries to save the whole formset using Ajax alfter an object or two has been deleted, it looks for the deleted object(s) and raises an IndexError: list index out of range, because the object(s) isn't at the queryset anymore.
This is how I am displaying and saving the formsets (simplified version - I think this is where the error comes from):
def App(request, slug):
TopicFormSet = modelformset_factory(Topic, form=TopicForm, extra=0, fields=('name',), can_delete=True)
SummaryFormSet = modelformset_factory(Summary, form=SummaryForm, extra=0, fields=('content',), can_delete=True)
tquery = user.topic_set.all().order_by('date')
squery = user.summary_set.all().order_by('date')
# saving formsets:
if request.method == 'POST' and request.is_ajax():
# the following two lines is where the error comes from:
t_formset = TopicFormSet(request.POST) # formset instance
s_formset = SummaryFormSet(request.POST) # formset instance
s_formset.save()
t_formset.save()
return render (blah...)
This is how I am removing objects (this is a different view):
def Remove_topic(request, slug, id):
topic = Topic.objects.get(pk=id)
summary = Summary.objects.get(topic = topic) # foreign key relatonship
topic.delete()
summary.delete()
# Ajax stuff....
if request.is_ajax():
return HttpResponse('blah..')
I have tried placing queryset = tquery and queryset = squery when instantiating t_formset and s_formset, but it didn't help. What should I do ? I am using Postgres db if that's useful.
The error:
> File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/home/eimantas/Desktop/Projects/Lynx/lynx/views.py", line 122, in App
t_formset = TopicFormSet(request.POST, queryset = tquery)
File "/usr/local/lib/python2.7/dist-packages/django/forms/models.py", line 441, in __init__
super(BaseModelFormSet, self).__init__(**defaults)
File "/usr/local/lib/python2.7/dist-packages/django/forms/formsets.py", line 56, in __init__
self._construct_forms()
File "/usr/local/lib/python2.7/dist-packages/django/forms/formsets.py", line 124, in _construct_forms
self.forms.append(self._construct_form(i))
File "/usr/local/lib/python2.7/dist-packages/django/forms/models.py", line 468, in _construct_form
kwargs['instance'] = self.get_queryset()[i]
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 198, in __getitem__
return self._result_cache[k]
IndexError: list index out of range

This may be a case of a cascaded delete that is already deleting the summary object:
When an object referenced by a ForeignKey is deleted, Django by
default emulates the behavior of the SQL constraint ON DELETE CASCADE
and also deletes the object containing the ForeignKey.
https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.ForeignKey.on_delete

It has nothing to do with second view and Ajax calls. I think that You have messed up management form's fields. Like initial_form_count, total_form_count or something similar.
Another important point. Do not save formset before checking if it is valid:
t_formset = TopicFormSet(request.POST)
if t_formset.is_valid():
t_formset.save()

In G+ group I was adviced that technically it is possible to reset or "reload" the Queryset, but it would be very difficult to maintain "at all levels" and probably would give no benefit. I was adviced to use iteration and check if each object has been saved successfully when saving the formset forms (I would have to overwrite form = TopicForm's and form = SummaryForm's save() method.)
I decided not to use formsets at all, but to list and save each object individually, it will be better for me and my app's business logic.

Subclass a class in Django to accept an additional kwarg

I am trying to subclass an ajax uploader to accept a hash_id of a django model (so that it may create the model upon the successful upload of an image) and am having trouble passing the additional kwarg (widget2_hash_id). I would appreciate guidance on how to properly add the kwarg.
views.py:
class S3UploadBackend_Widget2EditableImage(S3UploadBackend):
def upload(self, *args, **kwargs):
self.widget2_hash_id = kwargs.pop('widget2_hash_id')
k = Key(self._bucket)
chunk = uploaded.read()
k.set_contents_from_string(chunk)
# create uploaded file
fh = tempfile.TemporaryFile()
k.get_contents_to_file(fh)
fh.seek(0)
saveable_file = SimpleUploadedFile(k.name, fh.read())
# delete aws key and close tempfile
_media_bucket.delete_key(k)
fh.close()
self.widget2 = Widget2.objects.get(hash_id = self.widget2_hash_id)
self.widget2_editable_image = Widget2EditableImage(image = saveable_file, widget2 = self.widget2)
self.widget2_editable_image.save()
if k.key:
self.key = k.key
return True
else:
# Key creation failed.
return False
def upload_complete(self, request, filename):
# Manually add S3 key to ajaxuploader JSONresponse
res = {"aws_file_key": self.key, "url": self.widget2_editable_image.image.url}
views.py:
widget2_editable_image_ajax_uploader = AjaxFileUploader(backend=S3UploadBackend_Widget2EditableImage)
urls.py:
(r'^widget2/widget2_image_upload/(?P<widget2_hash_id>[a-fA-F0-9]+)/$', 'widget2.views.widget2_editable_image_ajax_uploader'),
Traceback:
Traceback (most recent call last):
File "/home/zain/XXX/lib/Django-1.3.1/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
TypeError: __call__() got an unexpected keyword argument 'widget2_hash_id'
[20/Aug/2012 20:50:44] "POST /widget2/widget2_image_upload/d9dc4fab3d5e0eb45995/?qqfile=s3Zas.jpg HTTP/1.1" 500 870358
EDIT: I tried this inside the class and get the same error:
def __init__(self, *args, **kwargs):
try:
self.widget2 = Widget2.objects.get(hash_id = kwargs.pop('widget2_hash_id'))
except KeyError:
self.widget2_hash_id = None
super(S3UploadBackend_Widget2EditableImage, self).__init__(*args, **kwargs)
EDIT2: here is the AjaxFileUploader class:
class AjaxFileUploader(object):
def __init__(self, backend=None, **kwargs):
if backend is None:
backend = LocalUploadBackend
self.get_backend = lambda: backend(**kwargs)
def __call__(self, request, **kwargs):
return self._ajax_upload(request)
def _ajax_upload(self, request):
if request.method == "POST":
if request.is_ajax():
# the file is stored raw in the request
upload = request
is_raw = True
# AJAX Upload will pass the filename in the querystring if it
# is the "advanced" ajax upload
try:
filename = request.GET['qqfile']
except KeyError:
return HttpResponseBadRequest("AJAX request not valid")
# not an ajax upload, so it was the "basic" iframe version with
# submission via form
else:
is_raw = False
if len(request.FILES) == 1:
# FILES is a dictionary in Django but Ajax Upload gives
# the uploaded file an ID based on a random number, so it
# cannot be guessed here in the code. Rather than editing
# Ajax Upload to pass the ID in the querystring, observe
# that each upload is a separate request, so FILES should
# only have one entry. Thus, we can just grab the first
# (and only) value in the dict.
upload = request.FILES.values()[0]
else:
raise Http404("Bad Upload")
filename = upload.name
backend = self.get_backend()
# custom filename handler
# Override filename to avoid collisons
filename = unicode(hashlib.sha1(str(datetime.datetime.now())).hexdigest()[0:6]) + filename
filename = (backend.update_filename(request, filename)
or filename)
# save the file
backend.setup(filename)
success = backend.upload(upload, filename, is_raw)
# callback
extra_context = backend.upload_complete(request, filename)
# let Ajax Upload know whether we saved it or not
ret_json = {'success': success, 'filename': filename}
if extra_context is not None:
ret_json.update(extra_context)
return HttpResponse(json.dumps(ret_json, cls=DjangoJSONEncoder))

The explanation
In your urls.py, you have this:
(r'^widget2/widget2_image_upload/(?P<widget2_hash_id>[a-fA-F0-9]+)/$', 'widget2.views.widget2_editable_image_ajax_uploader'),
When this url gets matched, django will dispatch the request, as well as the args to the given view, so, if we break it down, the following arguments will be passed to widget2.views.widget2_editable_image_ajax_uploader (the view):
request
*args: []
*kwargs: {'widget2_hash_id':'somehashid',}
Now, what does dispatch mean in django vocabulary? Well, basically, it means call function.
It so happens that widget2.views.widget2_editable_image_ajax_uploader has a call method, so django will happily call this method with the arguments mentionned above.
Unfortunately, as your view does not accept the widget2_hash_id kwarg, python is going to raise an error. Which it does.
I think that the point were you got lost is that you missed the fact that it's the view that gets passed the extra arg, not the backend. It's then the view's job to pass it down to the backend.
The solutions
The easy one
You just need to modify your __call__ method so that it accepts the additional arg, and then passes it on to the _ajax_upload method.
Then, you need to pass the argument to your backend. To this end, you should change the following line:
backend.upload(upload, filename, is_raw)
to
backend.upload(upload, filename, is_raw, widget2_hash_id)
The right one
Now, I must say that I'm a bit confused as to what you are doing. You seem to be using objects as views, but then, why wouldn't you use django's awesome class-based views?
The documentation used to be pretty lacking (I honestly can't tell if that's still the case), so here's a small guide to get you started.
Take some time to study and use them. They are very easy to use and extend, and they will save you a huge amount of time in the long run.
On a closing note
Please do use django forms, because they rock.