Develop Reference

Pycharm debugger, does not go deep enough : Django project. [ fairly new to python/Django ]

I'm currently debugging my Django project [kind of first project in Django/python]. The debug 'Step Over' button in Pycharm seems to be going properly. However at a particular point, it does not go deep enough.
Below is the code with the debug point (commented that particular line).
Please note, it's a properly working project, although I have trimmed down the codes (like skipping import statements, class definitions etc) here to make it as simple as possible to represent my question.
File: create_report_view.py Here is where I have set the debug point.
from project.project_core.forms import BrandModelForm
def post(self, request):
brand_form = BrandModelForm(request.POST, request.FILES)
if brand_form.is_valid(): # Debug point
file_name_errors, csv_name_errors, file_name_alphanumeric_error = self._validate_data(brand_form)
print(" And debug step goes on... ")
So, the debugger skips what happens inside the .is_valid() call, where my validation runs, and jumps to next line in the above code. How do I force it to debug the .is_valid() method as well ?
Dependent code blocks.
Below is the definition of my model, where i mention the validators in it.
File : models.py
from django.db.models import Model
from project.project_core.validators import ReportTypeValidator
class BrandModel(Model):
report_model = models.ForeignKey(ReportModel, models.CASCADE)
name = models.CharField(max_length=256, db_index=True)
review_file = models.FileField(upload_to='reviews/', validators=[ReportReviewValidator])
number_of_reviews = models.IntegerField(blank=True, null=True)
File : forms.py
from project.project_core.models import BrandModel
class BrandModelForm(forms.ModelForm):
class Meta:
model = BrandModel
exclude = ('report_model',)
labels = {
'name': 'Brand Name'
}
File :validators.py
from django.utils.deconstruct import deconstructible
ReportReviewValidator = UploadFileValidator(REVIEW_FILE_COLUMNS, CHARACTER_LIMIT_CONFIGS)
#deconstructible
class UploadFileValidator:
def __init__(self, review_file_columns: List[str], character_limit_configs: List[CharacterLimitConfig]):
self.csv_column_validator = CsvColumnsValidator(self.review_file_columns)
self.char_limit_validator = CharLimitValidator(self.char_limit_configs)
def __call__(self, field_file: FieldFile) -> None:
row_list = self.csv_column_validator(field_file)
I want the debugger to reach the above block of code, i.e the class UploadFileValidator
What should I be doing for that ?

To achieve this, you need to put your breakpoint one line above and use "step into" to get into it:
def post(self, request):
brand_form = BrandModelForm(request.POST, request.FILES) # Debug point
if brand_form.is_valid(): # Step Into this
file_name_errors, csv_name_errors, file_name_alphanumeric_error = self._validate_data(brand_form)
Note that this will get you to the is_valid() method, which is django's way of checking if your form is valid. It is not going to instantly take you to your validator. You will have to navigate through the whole process of django's form validation, and at some point you will end up in your own validator. If you want to specifically get into your validator's code, put a breakpoint inside instead.

errors in django tests

im trying to run the following test but im not sure why its not working since when tests are ran a test db is created and that means that the newly created item gets an id of 1 but im still getting an error that no object matches the query
model
from django.db import models
# Create your models here.
class Post(models.Model):
text = models.TextField()
def __str__(self):
return self.text[:50]
tests
from django.test import TestCase
from .models import Post
# Create your tests here.
class PostModelTest(TestCase):
def setup(self):
post = Post.objects.create(text="just a test")
def test_text_content(self):
post = Post.objects.get(id=1)
expected_obect_name = f'{post.text}'
self.assertEqual(expected_obect_name, 'just a test')
here's the error

First of all you should name method setUp not setup. It's case-sensetive. But it's probably will not solve your problem, since django doesn't reset auto id for each TestCase class. So if you have another test case with same code the error will happened again. To solve it you need to save post id as e.g. self.post_id and use it in test methods instead of 1:
class PostModelTest(TestCase):
def setUp(self):
post = Post.objects.create(text="just a test")
self.post_id = post.pk
def test_text_content(self):
post = Post.objects.get(id=self.post_id)
expected_obect_name = f'{post.text}'
self.assertEqual(expected_obect_name, 'just a test')

How do I run tests against a Django data migration?

Using the following example from the documentation:
def combine_names(apps, schema_editor):
Person = apps.get_model("yourappname", "Person")
for person in Person.objects.all():
person.name = "%s %s" % (person.first_name, person.last_name)
person.save()
class Migration(migrations.Migration):
dependencies = [
('yourappname', '0001_initial'),
]
operations = [
migrations.RunPython(combine_names),
]
How would I create and run a test against this migration, confirming that the data is migrated correctly?

I was doing some google to address the same question and found an article that nailed the hammer on the nail for me and seemed less hacky than existing answers. So, putting this here in case it helps anyone else coming though.
The proposed the following subclass of Django's TestCase:
from django.apps import apps
from django.test import TestCase
from django.db.migrations.executor import MigrationExecutor
from django.db import connection
class TestMigrations(TestCase):
#property
def app(self):
return apps.get_containing_app_config(type(self).__module__).name
migrate_from = None
migrate_to = None
def setUp(self):
assert self.migrate_from and self.migrate_to, \
"TestCase '{}' must define migrate_from and migrate_to properties".format(type(self).__name__)
self.migrate_from = [(self.app, self.migrate_from)]
self.migrate_to = [(self.app, self.migrate_to)]
executor = MigrationExecutor(connection)
old_apps = executor.loader.project_state(self.migrate_from).apps
# Reverse to the original migration
executor.migrate(self.migrate_from)
self.setUpBeforeMigration(old_apps)
# Run the migration to test
executor = MigrationExecutor(connection)
executor.loader.build_graph() # reload.
executor.migrate(self.migrate_to)
self.apps = executor.loader.project_state(self.migrate_to).apps
def setUpBeforeMigration(self, apps):
pass
And an example use case that they proposed was:
class TagsTestCase(TestMigrations):
migrate_from = '0009_previous_migration'
migrate_to = '0010_migration_being_tested'
def setUpBeforeMigration(self, apps):
BlogPost = apps.get_model('blog', 'Post')
self.post_id = BlogPost.objects.create(
title = "A test post with tags",
body = "",
tags = "tag1 tag2",
).id
def test_tags_migrated(self):
BlogPost = self.apps.get_model('blog', 'Post')
post = BlogPost.objects.get(id=self.post_id)
self.assertEqual(post.tags.count(), 2)
self.assertEqual(post.tags.all()[0].name, "tag1")
self.assertEqual(post.tags.all()[1].name, "tag2")

You can use django-test-migrations package. It is suited for testing: data migrations, schema migrations, and migrations' order.
Here's how it works:
from django_test_migrations.migrator import Migrator
# You can specify any database alias you need:
migrator = Migrator(database='default')
old_state = migrator.before(('main_app', '0002_someitem_is_clean'))
SomeItem = old_state.apps.get_model('main_app', 'SomeItem')
# One instance will be `clean`, the other won't be:
SomeItem.objects.create(string_field='a')
SomeItem.objects.create(string_field='a b')
assert SomeItem.objects.count() == 2
assert SomeItem.objects.filter(is_clean=True).count() == 2
new_state = migrator.after(('main_app', '0003_auto_20191119_2125'))
SomeItem = new_state.apps.get_model('main_app', 'SomeItem')
assert SomeItem.objects.count() == 2
# One instance is clean, the other is not:
assert SomeItem.objects.filter(is_clean=True).count() == 1
assert SomeItem.objects.filter(is_clean=False).count() == 1
We also have native integrations for both pytest:
#pytest.mark.django_db
def test_main_migration0002(migrator):
"""Ensures that the second migration works."""
old_state = migrator.before(('main_app', '0002_someitem_is_clean'))
SomeItem = old_state.apps.get_model('main_app', 'SomeItem')
...
And unittest:
from django_test_migrations.contrib.unittest_case import MigratorTestCase
class TestDirectMigration(MigratorTestCase):
"""This class is used to test direct migrations."""
migrate_from = ('main_app', '0002_someitem_is_clean')
migrate_to = ('main_app', '0003_auto_20191119_2125')
def prepare(self):
"""Prepare some data before the migration."""
SomeItem = self.old_state.apps.get_model('main_app', 'SomeItem')
SomeItem.objects.create(string_field='a')
SomeItem.objects.create(string_field='a b')
def test_migration_main0003(self):
"""Run the test itself."""
SomeItem = self.new_state.apps.get_model('main_app', 'SomeItem')
assert SomeItem.objects.count() == 2
assert SomeItem.objects.filter(is_clean=True).count() == 1
Full guide: https://sobolevn.me/2019/10/testing-django-migrations
Github: https://github.com/wemake-services/django-test-migrations
PyPI: https://pypi.org/project/django-test-migrations/

EDIT:
These other answers make more sense:
https://stackoverflow.com/a/56212859
https://stackoverflow.com/a/59016744, if you don't mind the extra (dev) dependency
ORIGINAL:
Running your data-migration functions (such as combine_names from the OP's example) through some basic unit-tests, before actually applying them, makes sense to me too.
At first glance this should not be much more difficult than your normal Django unit-tests: migrations are Python modules and the migrations/ folder is a package, so it is possible to import things from them. However, it took some time to get this working.
The first difficulty arises due to the fact that the default migration file names start with a number. For example, suppose the code from the OP's (i.e. Django's) data-migration example sits in 0002_my_data_migration.py, then it is tempting to use
from yourappname.migrations.0002_my_data_migration import combine_names
but that would raise a SyntaxError because the module name starts with a number (0).
There are at least two ways to make this work:
Rename the migration file so it does not start with a number. This should be perfectly fine according to the docs: "Django just cares that each migration has a different name." Then you can just use import as above.
If you want to stick to the default numbered migration file names, you can use Python's import_module (see docs and this SO question).
The second difficulty arises from the fact that your data-migration functions are designed to be passed into RunPython (docs), so they expect two input arguments by default: apps and schema_editor. To see where these come from, you can inspect the source.
Now, I'm not sure this works for every case (please, anyone, comment if you can clarify), but for our case, it was sufficient to import apps from django.apps and get the schema_editor from the active database connection (django.db.connection).
The following is a stripped-down example showing how you can implement this for the OP example, assuming the migration file is called 0002_my_data_migration.py:
from importlib import import_module
from django.test import TestCase
from django.apps import apps
from django.db import connection
from yourappname.models import Person
# Our filename starts with a number, so we use import_module
data_migration = import_module('yourappname.migrations.0002_my_data_migration')
class DataMigrationTests(TestCase):
def __init__(self, *args, **kwargs):
super(DataMigrationTests, self).__init__(*args, **kwargs)
# Some test values
self.first_name = 'John'
self.last_name = 'Doe'
def test_combine_names(self):
# Create a dummy Person
Person.objects.create(first_name=self.first_name,
last_name=self.last_name,
name=None)
# Run the data migration function
data_migration.combine_names(apps, connection.schema_editor())
# Test the result
person = Person.objects.get(id=1)
self.assertEqual('{} {}'.format(self.first_name, self.last_name), person.name)

You could add a crude if statement to a prior migration that tests if the test suite is running, and adds initial data if it is -- that way you can just write a test to check if the objects are in the final state you want them in. Just make sure your conditional is compatible with production, here's an example that would work with python manage.py test:
import sys
if 'test in sys.argv:
# do steps to update your operations
For a more "complete" solution, this older blog post has some good info and more up-to-date comments for inspiration:
https://micknelson.wordpress.com/2013/03/01/testing-django-migrations/#comments

Resolving circular imports in celery and django

I have a Django app that uses Celery to offload some tasks. Mainly, it defers the computation of some fields in a database table.
So, I have a tasks.py:
from models import MyModel
from celery import shared_task
#shared_task
def my_task(id):
qs = MyModel.objects.filter(some_field=id)
for record in qs:
my_value = #do some computations
record.my_field = my_value
record.save()
And in models.py
from django.db import models
from tasks import my_task
class MyModel(models.Model):
field1 = models.IntegerField()
#more fields
my_field = models.FloatField(null=True)
#staticmethod
def load_from_file(file):
#parse file, set fields from file
my_task.delay(id)
Now obviously, this won't work because of a circular import (models imports tasks and tasks imports models).
I've resolved this for the moment by calling my_task.delay() from views.py, but it seems to make sense to keep the model logic within the model class. Is there a better way of doing this?

The solution posted by joshua is very good, but when I first tried it, I found that my #receiver decorators had no effect. That was because the tasks module wasn't imported anywhere, which was expected as I used task auto-discovery.
There is, however, another way to decouple tasks.py from modules.py. Namely, tasks can be sent by name and they don't have to be evaluated (imported) in the process that sends them:
from django.db import models
#from tasks import my_task
import celery
class MyModel(models.Model):
field1 = models.IntegerField()
#more fields
my_field = models.FloatField(null=True)
#staticmethod
def load_from_file(file):
#parse file, set fields from file
#my_task.delay(id)
celery.current_app.send_task('myapp.tasks.my_task', (id,))
send_task() is a method on Celery app objects.
In this solution it is important to take care of correct, predictable names for your tasks.

In your models instead of importing the my_task at the beginning of the file, you can import it just before you use it. It will solve circular imports problem.
from django.db import models
class MyModel(models.Model):
field1 = models.IntegerField()
#more fields
my_field = models.FloatField(null=True)
#staticmethod
def load_from_file(file):
#parse file, set fields from file
from tasks import my_task # import here instead of top
my_task.delay(id)
Alternatively, you can also do same thing in your tasks.py. You can import your models just before you use it instead of beginning.
Alternative:
You can use send_task method to call your task
from celery import current_app
from django.db import models
class MyModel(models.Model):
field1 = models.IntegerField()
#more fields
my_field = models.FloatField(null=True)
#staticmethod
def load_from_file(file):
#parse file, set fields from file
current_app.send_task('myapp.tasks.my_task', (id,))

Just to toss one more not-great solution into this list, what I've ended up doing is relying on django's now-built-in app registry.
So in tasks.py, rather than importing from models, you use apps.get_model() to gain access to the model.
I do this with a helper method with a healthy bit of documentation just to express why this is painful:
from django.apps import apps
def _model(model_name):
"""Generically retrieve a model object.
This is a hack around Django/Celery's inherent circular import
issues with tasks.py/models.py. In order to keep clean abstractions, we use
this to avoid importing from models, introducing a circular import.
No solutions for this are good so far (unnecessary signals, inline imports,
serializing the whole object, tasks forced to be in model, this), so we
use this because at least the annoyance is constrained to tasks.
"""
return apps.get_model('my_app', model_name)
And then:
#shared_task
def some_task(post_id):
post = _model('Post').objects.get(pk=post_id)
You could certainly just use apps.get_model() directly though.

Use signals.
tasks.py
from models import MyModel, my_signal
from celery import shared_task
from django.dispatch import receiver
#shared_task
def my_task(id):
qs = MyModel.objects.filter(some_field=id)
for record in qs:
my_value = #do some computations
record.my_field = my_value
record.save()
#receiver(my_signal)
def my_receiver(sender, **kwargs):
my_task.delay(kwargs['id'])
models.py
from django.db import models
from tasks import my_task
from django.dispatch import Signal
my_signal = Signal(providing_args=['id'])
class MyModel(models.Model):
field1 = models.IntegerField()
#more fields
my_field = models.FloatField(null=True)
#staticmethod
def load_from_file(file):
#parse file, set fields from file
my_signal.send(sender=?, id=?)

Not sure if this is anyone else's problem, but I took a few hours, and I found a solution...mainly, the key from the documentation:
Using the #shared_task decorator
The tasks you write will probably live in reusable apps, and reusable apps cannot depend on the project itself, so you also cannot import your app instance directly.
Basically what I was doing was this...
####
# project/coolapp/tasks.py -- DON'T DO THIS
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
app = Celery("coolapp")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
#app.task(bind=True)
def some_task(self, some_id):
from coolapp.models import CoolPerson
####
# project/coolapp/__init__.py -- DON'T DO THIS
from __future__ import absolute_import, unicode_literals
from .tasks import app as celery_app
__all__ = ("celery_app",)
Therefore, I was getting weird errors about missing app labels (a clear indication of a circular import).
The solution...
Refactor project/coolapp/tasks.py -> project/project/tasks.py and project/coolapp/__init__.py -> project/project/__init__.py.
IMPORTANT: This does not (and should not) be added to INSTALLED_APPS. Otherwise, you'll get the circular import.
So then to start the woker:
celery -A project.project worker -l INFO
Also, a little debugging tip...
When you want to find out if your tasks are properly discovered, put this in project/project/app.py:
app.autodiscover_tasks()
assert "project.app.tasks.some_task" in app.tasks
Otherwise, you'll have to start up the worker only to realize your tasks aren't included in the app, then you'll have to wait for shutdown.

django factory boy factory with OneToOne relationship and related field

I am using Factory Boy to create test factories for my django app. The model I am having an issue with is a very basic Account model which has a OneToOne relation to the django User auth model (using django < 1.5):
# models.py
from django.contrib.auth.models import User
from django.db import models
class Account(models.Model):
user = models.OneToOneField(User)
currency = models.CharField(max_length=3, default='USD')
balance = models.CharField(max_length="5", default='0.00')
Here are my factories:
# factories.py
from django.db.models.signals import post_save
from django.contrib.auth.models import User
import factory
from models import Account
class AccountFactory(factory.django.DjangoModelFactory):
FACTORY_FOR = Account
user = factory.SubFactory('app.factories.UserFactory')
currency = 'USD'
balance = '50.00'
class UserFactory(factory.django.DjangoModelFactory):
FACTORY_FOR = User
username = 'bob'
account = factory.RelatedFactory(AccountFactory)
So I am expecting the factory boy to create a related UserFactory whenever AccountFactory is invoked:
# tests.py
from django.test import TestCase
from factories import AccountFactory
class AccountTest(TestCase):
def setUp(self):
self.factory = AccountFactory()
def test_factory_boy(self):
print self.factory.id
When running the test however, it looks like multiple User models are being create, and I am seeing an integriy error:
IntegrityError: column username is not unique
The documentation does mention watching out for loops when dealing with circular imports, but I am not sure whether that is whats going on, nor how I would remedy it. docs
If anyone familiar with Factory Boy could chime in or provide some insight as to what may be causing this integrity error it would be much appreciated!

I believe this is because you have a circular reference in your factory definitions. Try removing the line account = factory.RelatedFactory(AccountFactory) from the UserFactory definition. If you are always going to invoke the account creation through AccountFactory, then you shouldn't need this line.
Also, you may consider attaching a sequence to the name field, so that if you ever do need more than one account, it'll generate them automatically.
Change: username = "bob" to username = factory.Sequence(lambda n : "bob {}".format(n)) and your users will be named "bob 1", "bob 2", etc.

To pass result of calling UserFactory to AccountFactory you should use factory_related_name (docs)
Code above works next way:
AccountFactory for instantiating needs SubFactory(UserFactory).
UserFactory instantiates User.
UserFactory after instantiating calls RelatedFactory(AccountFactory)
Recursion,.. that is broken due to unique username constraint (you probably want to generate usernames via FuzzyText or Sequence)
So you need write UserFactory like this:
class UserFactory(factory.django.DjangoModelFactory):
account = factory.RelatedFactory(AccountFactory, factory_related_name='user')
username = factory.Sequence(lambda a: 'email%04d#somedomain.com' % a)
# rest of code
But you can still experience issues with already written tests. Imagine you have in tests places like next:
user = UserFactory()
account = Account(user=user)
Then adding RelatedFactory will break tests. If you haven't lots of tests and contributors in your project, you could rewrite them. But if not, it is not an option. Here is how it could be handled:
class UserFactory(factory.django.DjangoModelFactory):
class Params:
generate_account = factory.Trait(
account=factory.RelatedFactory(AccountFactory, factory_related_name='user')
)
Then code above won't be broken, because default call of UserFactory won't instantiate AccountFactory. To instantiate user with account:
user_with_account = UserFactory(generate_account=True)