Develop Reference

Django test runner not respecting unittest.TestCase?

For one of my functional tests, I decided to use unittest.TestCase instead of a Django test class because it was convenient when cleaning up the test to have direct access to my local development database in the test itself.
Running the test in isolation like so passes as I'd expect:
$ python manage.py test functional_tests.test_functionality
System check identified no issues (0 silenced).
...
----------------------------------------------------------------------
Ran 3 tests in 0.040s
OK
When I try to run all tests at the same time, however, that test specifically errors out, complaining that an object DoesNotExist, as though it were using the Django test database:
$ python manage.py test functional_tests
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
..................E..
======================================================================
ERROR: some_functional_test (functional_tests.test_functionality.FunctionalTest)
----------------------------------------------------------------------
Traceback (most recent call last):
... etc.
app.models.Object.DoesNotExist: Object matching query does not exist.
----------------------------------------------------------------------
Ran 21 tests in 0.226s
FAILED (errors=1)
Destroying test database for alias 'default'...
I assume the error is with my trying use Object.objects.latest('created') when no Objects exist in Django's test database.
Is there some way to prevent Django from wrapping all tests in whatever it is about the test runner that prevents my test from accessing an Object directly?

First a little explanation.
By default when you run ./manage.py test django test-runner make few steps which involves creating test database (with test_ prefix to each database name from app settings), running migration and destroying test database (more details on runner steps could find here)
A good explanation on how django treat test database is described here
In your case when you run unittest.TestCase in isolation no test database is created:
$ python manage.py test functional_tests.test_functionality
System check identified no issues (0 silenced).
...
----------------------------------------------------------------------
Ran 3 tests in 0.040s
OK
( ^no logs about creating test database)
That's because there were no django.test.TestCase called. We can see it from sources (virgin unittest.TestCase doesn't have databases property, when django.TestCase it has)
But when you call whole module (python manage.py test functional_tests) looks like you have some django.test.TestCase tests in suite so that's why new test database is creating:
$ python manage.py test functional_tests
Creating test database for alias 'default'... # <-- THIS ONE
##<skipped for readability>
Destroying test database for alias 'default'...
And as you mentioned tests failed because there were no prepared objects for them.
Solution
At this point I see few options to solve this.
Prepare test data for tests explicitly (by fixtures or manually in tests or setups) so they would be independent from current state of database
Explicitly use desired database.
Run tests with --keepdb option (ie ./manage.py test --keepdb, it will use the existing database and will not destroy it after test runs) and set same test database name in app settings as working database (in such case it will not append test_ prefix for test db)
Since you don't want to use django.TestCase, so don't use them at all? Replace them with unittest.TestCase and it will not create test database

IntegrityError: Problem installing fixture

I upload my first Django-project into DigitalOcean. After command python manage.py loaddata initial_data.json, I have received this message:
django.db.utils.IntegrityError: Problem installing fixture
'/webapps/django_shop/shop/initial_data.json': Could not load
contenttypes.ContentType(pk=3): duplicate key value violates unique
constraint "django_content_type_app_label_76bd3d3b_uniq" DETAIL: Key
(app_label, model)=(auth, permission) already exists.
How can I fix it?

I had the same problem and I solved this way
DB with data to export from
python manage.py dumpdata --exclude auth.permission --exclude contenttypes > db.json
New DB to import to
python manage.py flush
// Important! Disable all signals on models pre_save and post_save
python manage.py loaddata db.json
// Do not forget to enable all signals that you disabled

It looks like you've generated fixtures that include Django's default data set, i.e. the built-in entries that are inserted normally as part of the first migrate run for some of Django's plumbing data types.
You should review your fixture process, because content type entries will be created automatically when your (and Django's) apps' migrations are run, so they should not be present in fixtures. It's possible there are other tables that will have this same problem, so now would be a good time to make sure you're not including any other data that would result in this situation.

Django backup strategy with dumpdata and migrations

As in this question, I set up a dumpdata-based backup system for my database. The setup is akin to running a cron script that calls dumpdata and moves the backup to a remote server, with the aim of simply using loaddata to recover the database. However, I'm not sure this plays well with migrations. loaddata now has an ignorenonexistent switch to deal with deleted models/fields, but it is not able to resolve cases where columns were added with one-off defaults or apply RunPython code.
The way I see it, there are two sub-problems to address:
Tag each dumpdata output file with the current version of each app
Splice the fixtures into the migration path
I'm stumped about how to tackle the first problem without introducing a ton of overhead. Would it be enough to save an extra file per backup that contained an {app_name: migration_number} mapping?
The second problem I think is easier once the first one is solved, since the process is roughly:
Create a new database
Run migrations forward to the appropriate point for each app
Call loaddata with the given fixture file
Run the rest of the migrations
There's some code in this question (linked from the bug report) that I think could be adapted for this purpose.
Since these are fairly regular/large snapshots of the database, I don't want to keep them as data migrations cluttering up the migrations directory.

I am taking the following steps to backup, restore or transfer my postgresql database between any instance of my project:
The idea is to keep the least possible migrations as if manage.py makemigrations was run for the first time on an empty database.
Let's assume that we have a working database to our development environment. This database is a current copy of the production database that should not be open to any changes. We have added models, altered attributes etc and those actions have generated additional migrations.
Now the database is ready to be migrated to production which -as stated before- is not open to public so it is not altered in any way. In order to achieve this:
I perform the normal procedure in the development environment.
I copy the project to the production environment.
I perform the normal procedure in the production environment
We make the changes in our development environment. No changes should happen in the production database because they will be overridden.
Normal Procedure
Before anything else, I have a backup of the project directory (which includes a requirements.txt file), a backup of the database and -of course- git is a friend of mine.
I take a dumpdata backup in case I need it. However, dumpdata has some serious limitations regarding content types, permissions or other cases where a natural foreignkey should be used:
./manage.py dumpdata --exclude auth.permission --exclude contenttypes --exclude admin.LogEntry --exclude sessions --indent 2 > db.json
I take a pg_dump backup to use:
pg_dump -U $user -Fc $database --exclude-table=django_migrations > path/to/backup-dir/db.dump
Only if I want to merge existing migrations in one, I delete all migrations from every application.
In my case the migrations folder is a symlink, so I use the following script:
#!/bin/bash
for dir in $(find -L -name "migrations")
do
rm -Rf $dir/*
done
I delete and recreate the database:
For example, a bash script can include the following commands:
su -l postgres -c "PGPASSWORD=$password psql -c 'drop database $database ;'"
su -l postgres -c "createdb --owner $username $database"
su -l postgres -c "PGPASSWORD=$password psql $database -U $username -c 'CREATE EXTENSION $extension ;'"
I restore the database from the dump:
pg_restore -Fc -U $username -d $database path/to/backup-dir/db.dump
If migrations were deleted in step 3, I recreate them in the following way:
./manage.py makemigrations <app1> <app2> ... <appn>
... by using the following script:
#!/bin/bash
apps=()
for app in $(find ./ -maxdepth 1 -type d ! -path "./<project-folder> ! -path "./.*" ! -path "./")
do
apps+=(${app#??})
done
all_apps=$(printf "%s " "${apps[#]}")
./manage.py makemigrations $all_apps
I migrate using a fake migration:
./manage.py migrate --fake
In case something has gone completely wrong and everything is ***, (this can happen, indeed), I can use the backup to revert everything to its previous working state. If I would like to use the db.json file from step one, it goes like this:
When pg_dump or pg_restore fails
I perform the steps:
3 (delete migrations)
4 (delete and recreate the database)
6 (makemigrations)
and then:
Apply the migrations:
./manage.py migrate
Load the data from db.json:
./manage.py loaddata path/to/db.json
Then I try to find out why my previous effort was not successful.
When the steps are performed successfully, I copy the project to the server and perform the same ones to that box.
This way, I always keep the least number of migrations and I am able to use pg_dump and pg_restore to any box that shares the same project.

Force delete of any previous test database (autoclobber) when running Django unit tests, eg, in PyCharm

I am running Django unit tests against a multithreaded app. Often a thread hasn't terminated by the time the unit test finishes, so the test database cannot be deleted. When I next run the tests, I get the message:
Type 'yes' if you would like to try deleting the test database 'test_appname', or 'no' to cancel`
The create_test_db autoclobber option is the functionality I want, but how can I use that? I can't find any examples or clues. I'm working in the PyCharm IDE, which is pretty configurable. I just want to delete the test database silently every time.
I'm putting tests in Transaction TestCase classes, running setup_test_environment() then Client().post(reverse(etc..))..

In Pycharm django tests, you can enable the option input and enter --noinput
see screenshot below

If you're using PyCharm and want to run a single test using the green arrow but you keep getting this error, you can modify the default django tests configuration template, so that you don't have to keep setting the --noinput option on each.

If you are using flush in some other way, (eg, on the pre-existing development database like here), the --noinput option supresses the user prompt, eg:
from django.core.management import call_command
call_command('flush', '--noinput')

My answer was to create a script like this:
export PGPASSWORD=the_password
if [[ `psql -h 127.0.0.1 -d postgres -U username -p 5432 -tAc "SELECT 1 FROM pg_database WHERE datname='test_djangoprojectname'"` == "1" ]]
then
psql -h 127.0.0.1 -d postgres -U username -p 5432 -a -w -c "SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'test_djangoprojectname' AND pid <> pg_backend_pid();"
psql -h 127.0.0.1 -d postgres -U username -p 5432 -a -w -c "DROP DATABASE test_djangoprojectname;"
fi
The -d setting is database name - it can be any database your user has access to, except the one you are deleting.
The default username is postgres.
The -p setting is the port your database is on - 5432 is the default.
Save as (for example) del_test_db.sh (Windows users see below), then
chmod +x del_test_db.sh
Then in PyCharm:
Run, Edit Configurations...
Unfold Defaults, click Django tests
In the Before launch window click +, External tools, click +
Under Program, select your file del_test_db.sh, give the command a name (eg, 'del test db') and click OK.
Select your tool in the list and click OK
You may need to unfold Django tests in the left and delete existing test configurations
Then the script force deletes the test database before every run.
This works on Mac OS X and Ubuntu etc. For Windows the process is the same, except instead of export use SET, save the commands as a .bat file instead of .sh, and you don't need to chmod +x, and use the following syntax for the IF statement in the batch file:
if 'command' == '1' (
...
)
Apologies I'm unable to check this as I don't have a Windows machine.
Thanks to this answer for the code checking whether the database exists.

PyCharm can't find Django test

I'm running into this very same issue, but I can't find a solution for this.
Find the dummy Django project I created for this here.
This is my test configuration:
This is the project structure:
Trying to run the test results in:
AttributeError: 'super' object has no attribute 'run_tests'
Process finished with exit code 137
Empty test suite.
Obviously, running this from the shell gives no errors:
(django)mosquito#mosquito-X550LD ~/python_projects/django_test $ python manage.py test
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Destroying test database for alias 'default'...
Any idea what I'm doing wrong??
Thanks,
Alejandro

In the project configuration, add another environment variable of
DJANGO_SETTINGS_MODULE = django_test.settings
You can do this by clicking on the "..." icon