Develop Reference

Django Rest: how to mock FileField and ImageField to_representation()

I have a model with 2 fields: image (ImageField) and media (FileField). They both use a custom GoogleCloudMediaStorage.
When writing my tests, I cannot access GCS, so I mocked the writing of the files by doing this:
from unittest.mock import patch
#patch('my_project.storage_backends.CustomGoogleCloudMediaStorage.save')
def test(self, mock_save):
mock_save.return_value = 'mocked_file.png'
# rest of the test
And it's working fine. The problem is now I have to test update and list views. When I do so, I have of course this error:
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application.
After digging it a bit, I see that the error is triggered by the serializer to_representation(). And more precisely in the FileField:
def to_representation(self, value):
if not value:
return None
use_url = getattr(self, 'use_url', api_settings.UPLOADED_FILES_USE_URL)
if use_url:
try:
url = value.url # !! THIS LINE CAUSE THE ISSUE
except AttributeError:
return None
request = self.context.get('request', None)
if request is not None:
return request.build_absolute_uri(url)
return url
So I tried to mock FileField.to_representation() doing this:
#patch('django.db.models.FileField.to_representation')
def test_update(self, mock_representation):
mock_representation.return_value = 'mocked_training_file_url'
data = {
'title': 'Updated',
}
response = self.client.patch(. . .)
But I have this error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/unittest/mock.py", line 1322, in patched
with self.decoration_helper(patched,
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/unittest/mock.py", line 1304, in decoration_helper
arg = exit_stack.enter_context(patching)
File "/usr/local/lib/python3.8/contextlib.py", line 425, in enter_context
result = _cm_type.__enter__(cm)
File "/usr/local/lib/python3.8/unittest/mock.py", line 1393, in __enter__
original, local = self.get_original()
File "/usr/local/lib/python3.8/unittest/mock.py", line 1366, in get_original
raise AttributeError(
AttributeError: <class 'django.db.models.fields.files.FileField'> does not have the attribute 'to_representation'
Any idea how I can mock FileField and ImageField to_representation() ? Am I doing this correctly

Django fails when saving a serializer with foreign key references

So I had a model with one foreign key reference that was working really well for a few weeks. I ended up having to add another foreign key reference to the model and now I'm not able to save any records using a serializer. I should mention I'm using Django 2.1.
Here's the model I'm using (some fields removed for simplicity):
class Finding(models.Model):
finding_id = models.IntegerField(null=False, blank=False, primary_key=True, db_index=True)
name = models.CharField(null=True, blank=True, max_length=255)
web_app = models.ForeignKey(WebApplication, on_delete=models.CASCADE)
q_id = models.ForeignKey(StagingQkb, on_delete=models.CASCADE)
The new FK is the q_id field. This was just a plain int before. I've actually blown away the old table and the new one is still totally empty so I know it's not a problem with the existing data (the tables with the foreign keys are still intact). When saving Findings before, I would just give the PK of the WebApplication object for 'web_app'. As far as I can tell this is still working. The 'q_id' field, when inserted the same way, complains that it needs an int/string/bytes instead of a StagingQkb object. Well, I'm not giving it a StagingQkb object so what gives!
Here's the serializer:
class FindingSerializer(serializers.ModelSerializer):
class Meta:
model = Finding
fields = '__all__'
The data I'm feeding to the serializer looks like this:
data = {'finding_id': 5514989,
'name': 'Sample-Name',
'q_id': 12345,
'web_app': 67890}
When I insert the data into the serializer I'm doing the following:
>>> fs = FindingSerializer(data=data)
>>> fs.is_valid()
True
>>> fs.save()
The error I'm getting after running the above code looks like this:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/opt/oss/python35/lib/python3.5/site-packages/rest_framework/serializers.py", line 214, in save
self.instance = self.create(validated_data)
File "/opt/oss/python35/lib/python3.5/site-packages/rest_framework/serializers.py", line 959, in create
raise TypeError(msg)
TypeError: Got a `TypeError` when calling `Finding.objects.create()`. This may be because you have a writable field on the serializer class that is not a valid argument to `Finding.objects.create()`. You may need to make the field read-only, or override the FindingSerializer.create() method to handle this correctly.
Original exception was:
Traceback (most recent call last):
File "/opt/oss/python35/lib/python3.5/site-packages/rest_framework/serializers.py", line 940, in create
instance = ModelClass._default_manager.create(**validated_data)
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/query.py", line 413, in create
obj.save(force_insert=True, using=self.db)
File "/home/dpz3w0q/AutonomousPrimeD/autonomousprimed/Finding/models.py", line 83, in save
q_id=self.q_id)
File "/home/dpz3w0q/AutonomousPrimeD/autonomousprimed/Finding/models.py", line 173, in evaluate
if ScoreMatrix.objects.filter(q_id=q_id).count() > 0:
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/query.py", line 844, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/query.py", line 862, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1263, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1287, in _add_q
split_subq=split_subq,
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1225, in build_filter
condition = self.build_lookup(lookups, col, value)
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1096, in build_lookup
lookup = lookup_class(lhs, rhs)
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/lookups.py", line 20, in __init__
self.rhs = self.get_prep_lookup()
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/lookups.py", line 70, in get_prep_lookup
return self.lhs.output_field.get_prep_value(self.rhs)
File "/opt/oss/python35/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 1807, in get_prep_value
return int(value)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'StagingQkb'
I'm completely stuck here. I haven't changed the way I'm passing data to the serializer, so I really can't tell what I've done wrong. If anyone is able to help me work through this I'd really appreciate it!

Thanks to #WillKeeling I realized that I had a reference to Finding.q_id in my save() method that I forgot to update after making the FK changes. The PK of the foreign table is qkb_id, so I changed the reference to Finding.q_id.qkb_id and I was able to save the model.

DRF - 'str' object has no attribute 'pk'

I've got a front-end that sends JSON to the back-end to switch the input of a digital audio stream, with an optional time component to schedule the switch for the future. Here are the components of making this work:
from views.py:
class SwitchStreamView(views.APIView):
"""
A custom endpoint for switching inputs of radio streams
"""
queryset = RadioStream.objects.all()
def post(self, request, format=None):
serializer = serializers.RadioSwitchSerializer(data=request.data, many=True)
serializer.is_valid()
for stream in serializer.data:
if stream.schedule_time is None:
tasks.switch_stream(stream.mnemonic, stream.current_input)
else:
tasks.schedule_switch(stream.mnemonic, stream.current_input, stream.schedule_time)
return HttpResponse('')
from serializers.py:
class RadioSwitchSerializer(serializers.ModelSerializer):
schedule_time = serializers.SerializerMethodField()
def get_schedule_time(self, obj):
return obj.get('schedule_time', None)
class Meta:
model = RadioStream
fields = ('mnemonic', 'current_input', 'schedule_time')
The issue I'm having is that however I try and send a test JSON snippet, I'm getting errors. With this setup, sending
[
{
"mnemonic": "TEST",
"current_input": "TEST"
}
]
results in the error 'str' object has no attribute 'pk', but if I change RadioSwitchSerializer(data=request.data, many=True) to many=False, and send
{
"mnemonic": "TEST",
"current_input": "TEST"
}
I get the response 'str' object has no attribute 'schedule_time' instead.
My plan was to use mnemonic to identify the stream, and current_input to identify which input to switch it to. My questions are; Why is this not working, and should I be using a non-Model serializer for this custom action instead of trying to fit the action into the existing fields of the model?
Edit: Here is the traceback
Internal Server Error: /api/switch/
Traceback (most recent call last):
File "...\lib\site-packages\django\core\handlers\exception.py", line 35, in inner
response = get_response(request)
File "...\lib\site-packages\django\core\handlers\base.py", line 128, in _get_response
response = self.process_exception_by_middleware(e, request)
File "...\lib\site-packages\django\core\handlers\base.py", line 126, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "...\lib\site-packages\django\views\decorators\csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "...\lib\site-packages\django\views\generic\base.py", line 69, in view
return self.dispatch(request, *args, **kwargs)
File "...\lib\site-packages\rest_framework\views.py", line 489, in dispatch
response = self.handle_exception(exc)
File "...\lib\site-packages\rest_framework\views.py", line 449, in handle_exception
self.raise_uncaught_exception(exc)
File "...\lib\site-packages\rest_framework\views.py", line 486, in dispatch
response = handler(request, *args, **kwargs)
File "...\radio_switching\views.py", line 45, in post
for stream in serializer.data:
File "...\lib\site-packages\rest_framework\serializers.py", line 738, in data
ret = super(ListSerializer, self).data
File "...\lib\site-packages\rest_framework\serializers.py", line 266, in data
self._data = self.get_initial()
File "...\lib\site-packages\rest_framework\serializers.py", line 573, in get_initial
return self.to_representation(self.initial_data)
File "...\lib\site-packages\rest_framework\serializers.py", line 656, in to_representation
self.child.to_representation(item) for item in iterable
File "...\lib\site-packages\rest_framework\serializers.py", line 656, in <listcomp>
self.child.to_representation(item) for item in iterable
File "...\lib\site-packages\rest_framework\serializers.py", line 500, in to_representation
ret[field.field_name] = field.to_representation(attribute)
File "...\lib\site-packages\rest_framework\relations.py", line 259, in to_representation
return value.pk
AttributeError: 'str' object has no attribute 'pk'

replace your views.py by this snippet,
class SwitchStreamView(views.APIView):
"""
A custom endpoint for switching inputs of radio streams
"""
queryset = RadioStream.objects.all()
def post(self, request, format=None):
serializer = serializers.RadioSwitchSerializer(data=request.data, many=True)
serializer.is_valid()
for stream in serializer.data:
if 'schedule_time' not in stream:
tasks.switch_stream(stream['mnemonic'], stream['current_input'])
else:
tasks.schedule_switch(stream['mnemonic'], stream['current_input'], stream['schedule_time'])
return HttpResponse('')
The reason for the error is, you are trying to access a python dict using dot operator. To access dictionary elements, you can use the familiar square brackets along with the key to obtaining it's value. Here is the official doc
EDIT
AttributeError: 'str' object has no attribute 'pk' this error because, somewhere you trying to access .pk from a str object
Reproducing the error
In [7]: my_str = 'this is just a string'
In [8]: my_str.pk
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-8-5bc39c990550> in <module>()
----> 1 my_str.pk
AttributeError: 'str' object has no attribute 'pk'

There were a few issues, but it turns out the root issue was that serlializer.data was for whatever reason returning a string and not an array of objects. I ended up replacing the ModelSerializer with a regular Serializer here:
class RadioSwitchSerializer(serializers.Serializer):
mnemonic = serializers.CharField(max_length=20)
new_input = serializers.CharField(max_length=20)
schedule_time = serializers.DateTimeField(allow_null=True)
As per Jerin's answer, I also fixed up the lines accessing the stream dictionary to use square brackets rather than the dot operator. These two things have fixed the issue.

Django: model OneToOneField with User can't add default value

I have an model and I want to add an OneToOneField to hold the creator of an object:
models.py
creator = models.OneToOneField(User, blank=True,
default=User.objects.filter(
username="antoni4040"))
I already have a database with items and just want the default value to be the admin user, which has the username "antoni4040". When I try to migrate without the default field it asks for a default value, so I can't get away with it. But here's what I get when running makemigrations:
Migrations for 'jokes_app':
0008_joke_creator.py:
- Add field creator to joke
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/core/management/commands/makemigrations.py", line 150, in handle
self.write_migration_files(changes)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/core/management/commands/makemigrations.py", line 178, in write_migration_files
migration_string = writer.as_string()
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/db/migrations/writer.py", line 167, in as_string
operation_string, operation_imports = OperationWriter(operation).serialize()
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/db/migrations/writer.py", line 124, in serialize
_write(arg_name, arg_value)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/db/migrations/writer.py", line 88, in _write
arg_string, arg_imports = MigrationWriter.serialize(_arg_value)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/db/migrations/writer.py", line 433, in serialize
return cls.serialize_deconstructed(path, args, kwargs)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/db/migrations/writer.py", line 318, in serialize_deconstructed
arg_string, arg_imports = cls.serialize(arg)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/db/migrations/writer.py", line 517, in serialize
item_string, item_imports = cls.serialize(item)
File "/home/antoni4040/Documents/Jokes_Website/django-jokes/venv/lib/python3.4/site-packages/django/db/migrations/writer.py", line 540, in serialize
"topics/migrations/#migration-serializing" % (value, get_docs_version())
ValueError: Cannot serialize: <User: antoni4040>
There are some values Django cannot serialize into migration files.
For more, see https://docs.djangoproject.com/en/1.9/topics/migrations/#migration-serializing
What am I doing wrong?

You are using the queryset User.objects.filter(username="antoni4040"). To get a model instance, you would use User.objects.get(username="antoni4040").
However, you shouldn't use a a model instance for as the default in the model field. There's no error handling if the user does not exist in the database. In fact, if models.py is loaded before you run the initial migrations, then the User table won't even exist so the query will give an error.
The logic to set the default user should go in the view (or Django admin) instead of the models file.
In the admin, you could define a custom form that sets the initial value for the creator field.
class MyModelForm(forms.ModelForm):
class Meta:
model = MyModel
def __init__(self, *args, **kwargs):
try:
user = User.objects.get(username="antoni4040")
kwargs['initial']['creator'] = user
except MyModel.DoesNotExist:
pass
super(MyModelForm, self).__init__(*args, **kwargs)
Then use that model form in your admin.
class MyModelAdmin(admin.ModelAdmin):
form = MyModelForm
...

If you really wanted to do this you can open up a shell and get the id for that User, then set the default to that.
The problem is, why would you do that? The first time you add a model it will use the default correctly. The second time you try to do that, it will fail because it's one to one, not many to one, so it will fail.

Django TemplateSyntaxError: current transaction is aborted, what does this exception mean? Does postgresql 8.4 work fine with django?

The full text of the error is:
TemplateSyntaxError at /
Caught an exception while rendering:
current transaction is aborted,
commands ignored until end of
transaction block
I've recently reinstalled all the software on my computer. The code used to work no problem before. It also still works no problem on a friend's computer and on a development server.
The only thing I can think of, which may have changed is the postgresql server version (and I'm not actually certain whether I tried running it on 8.4 on my old installation or not - it definitely worked on 8.3).
Is anyone able to confirm that Django has problems with postgresql 8.4 and/or has any hints as to why I'm having these errors?
Edit 1
In response to Dominic...
This doesn't happen just on one page or with one tag (though there are some pages which seem to be ok). The only thing the tags and variables that cause errors have in common is that they happen to access the database somewhere along the way (though not all tags that access the database cause the error). Furthermore, the same code does not create TemplateSyntaxError on other computers.
If I remove a variable or custom template tag that is making the error then this is the chain of events that happens:
I remove the variable or tag that is supposedely causing the error and refresh the page.
Django picks a different variable or tag and the same error occurs, so I keep removing them.
Once all the variables and tags that cause the error are removed, I stop getting a proper error page. I get a plain white page with the following traceback:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 279, in run
self.result = application(self.environ, self.start_response)
File "/usr/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 651, in __call__
return self.application(environ, start_response)
File "/usr/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 245, in __call__
response = middleware_method(request, response)
File "/usr/lib/python2.6/site-packages/debug_toolbar/middleware.py", line 90, in process_response
response.content = replace_insensitive(smart_unicode(response.content), u'', smart_unicode(self.debug_toolbars[request].render_toolbar() + u''))
File "/usr/lib/python2.6/site-packages/debug_toolbar/toolbar/loader.py", line 72, in render_toolbar
'BASE_URL': self.request.META.get('SCRIPT_NAME', ''),
File "/usr/lib/python2.6/site-packages/django/template/loader.py", line 108, in render_to_string
return t.render(context_instance)
File "/usr/lib/python2.6/site-packages/django/test/utils.py", line 29, in instrumented_test_render
return self.nodelist.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 779, in render
bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py", line 71, in render_node
result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/defaulttags.py", line 155, in render
nodelist.append(node.render(context))
File "/usr/lib/python2.6/site-packages/django/template/defaulttags.py", line 243, in render
return self.nodelist_true.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 779, in render
bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py", line 81, in render_node
raise wrapped
TemplateSyntaxError: Caught an exception while rendering: current transaction is aborted, commands ignored until end of transaction block
Original Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/django/template/debug.py", line 71, in render_node
result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/debug.py", line 87, in render
output = force_unicode(self.filter_expression.resolve(context))
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 546, in resolve
obj = self.var.resolve(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 687, in resolve
value = self._resolve_lookup(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 722, in _resolve_lookup
current = current()
File "/usr/lib/python2.6/site-packages/debug_toolbar/panels/template.py", line 64, in content
pformat(k(self.request))) for k in get_standard_processors()
File "/usr/lib/python2.6/site-packages/django/core/context_processors.py", line 27, in auth
'messages': user.get_and_delete_messages(),
File "/usr/lib/python2.6/site-packages/django/contrib/auth/models.py", line 263, in get_and_delete_messages
for m in self.message_set.all():
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 106, in _result_iter
self._fill_cache()
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 692, in _fill_cache
self._result_cache.append(self._iter.next())
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 238, in iterator
for row in self.query.results_iter():
File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py", line 287, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py", line 2369, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.6/site-packages/debug_toolbar/panels/sql.py", line 91, in execute
return self.cursor.execute(sql, params)
InternalError: current transaction is aborted, commands ignored until end of transaction block

That exception means that there was an error in some SQL that is getting executed. Since Django runs all the SQL inside of a database transaction, all the SQL that is being executed after the error gets ignored. So:
BEGIN;
SELECT * FROM table;
SELECT missing_column FROM table WHERE id = 1; -- generates an error because the column is missing
SELECT * FROM another_table; -- this statement and all following statements get ignored until the next COMMIT;
COMMIT;
To figure out the problem, find your log file for PostgreSQL and run tail -f /path/to/postgresql_error.log. Then refresh the page. You should see the error come up in the log file.

If you're moving from a different DB, note that there might be some considerations to bear in mind with the way that Django with Postgres uses transactions. Have a read here:
http://docs.djangoproject.com/en/dev/topics/db/transactions/?from=olddocs#handling-exceptions-within-postgresql-transactions
The quick answer is usually to turn on database level autocommit by adding:
'OPTIONS': {'autocommit': True,}
to your database settings. But it's worth having a read up on it and understanding the coo.
Rolo.