I want to write tests that can show whether or not the database is in sync with my models.py file. Actually I have already written them, only to find out that django creates a new database each time the tests are run based on the models.py file.
Is there any way I can make the models.py test use the existing database schema? The one that's in mysql/postgresql, and not the one that's in /myapp/models.py ?
I don't care about the data that's in the database, I only care about it's schema i.e. I want my tests to notice if a table in the database has less fields than the schema in my models.py file.
I'm using the unittest framework (actually the django extension to it) if this has any relevance.
thanks
What we did was override the default test_runner so that it wouldn't create a new database to test against. This way, it runs the test against whatever our current local database looks like. But be very careful if you use this method because any changes to data you make in your tests will be permanent. I made sure that all our tests restores any changes back to their original state, and keep our pristine version of our database on the server and backed up.
So to do this you need to copy the run_test method from django.test.simple to a location in your project -- I put mine in myproject/test/test_runner.py
Then make the following changes to that method:
// change
old_name = settings.DATABASE_NAME
from django.db import connection
connection.creation.create_test_db(verbosity, autoclobber=not interactive)
result = unittest.TextTestRunner(verbosity=verbosity).run(suite)
connection.creation.destroy_test_db(old_name, verbosity)
// to:
result = unittest.TextTestRunner(verbosity=verbosity).run(suite)
Make sure to do all the necessary imports at the top and then in your settings file set the setting:
TEST_RUNNER = 'myproject.test.test_runner.run_tests'
Now when you run ./manage.py test Django will run the tests against the current state of your database rather than creating a new version based on your current model definitions.
Another thing you can do is create a copy of your database locally, and then do a check in your new run_test() method like this:
if settings.DATABASE_NAME != 'my_test_db':
sys.exit("You cannot run tests using the %s database. Please switch DATABASE_NAME to my_test_db in settings.py" % settings.DATABASE_NAME)
That way there's no danger of running tests against your main database.
Related
I am having trouble running the django migrations on a new database because of the get_or_create() function in a form.
form.py
class EmployeeForm(...):
# Make sure that we have a directory before initializing the form
default_upload_directory = Directory.objects.get_or_create(name='Employee', parent=None)[0]
This is the important part of a custom form I wrote. It will link the Employee's uploaded files to a default directory. Everything works fine except for the migration. When I run it on a new database it will complain with django.db.utils.OperationalError: no such table: filemanager_directory. It also happens if I use a standard objects.get(..) to access the Directory object.
I am not sure how to solve this. Obviously the table does not exist, yet. On an existing database with the tables already in place everything works fine.
EDIT: The migrations I am trying to run are not related to this app. Furthermore, as a workaround, you can comment out the line, run the migrations and reactivate it... But this is not a neat solution...
This is, quite simply, not a thing you should do. You shouldn't run any kind of database access in any code that runs at import time, which this does.
There should never be a need to do this in any case. If your goal is to ensure an object exists when the form is instantiated, put it in the __init__ method.
But you should also reflect that ensuring that database objects exist is the job of migrations in the first place. Write a data migration with a RunPython method to do this.
If one is using Django, what happens with changes made directly to the database (in my case postgres) through either pgadmin or psql?
How are such changes handled by migrations? Do they take precedence over what the ORM thinks the state of affairs is, or does Django override them and impose it's own sense of change history?
Finally, how are any of these issues effected, or avoided, by git, if at all?
Thanks.
You can exclude a model completely from the django migrations, and then you are responsible to adjust the schema to the django code (or the django code to the existing schema):
class SomeModel(models.Model):
class Meta:
managed = False
db_table = "some_table_name"
name = models.Fields....
Note that you can't have it both ways, so migrations are preferred when possible. You can always define a custom SQL migration, that will save the need for external changes. However, sometimes you do need to handle the schema elsewhere instead of migrations, and then use managed=False
The migrations system does not look at your current schema at all. It builds up its picture from the graph of previous migrations and the current state of models.py. That means that if you make changes to the schema from outside this system, it will be out of sync; if you then make the equivalent change in models.py and create migrations, when you run them you will probably get an error.
For that reason, you should avoid doing this. If it's done already, you could apply the conflicting migration in fake mode, which simply marks it as done without actually running the code against the database. But it's simpler to do everything via migrations in the first place.
git has no impact on this at all, other than to reiterate that migrations are code, and should be added to your git repo.
I have a bootstrap script that performs syncdb and migrate:
import settings
from django.core.management import setup_environ, call_command
setup_environ(settings) # Setting up the env settings
call_command('syncdb', migrate=True, interactive=False) # Sync the database
Pre-Requisites:
django-south for migrations.
Process happening:
initial_data fixture contains data for a model that is created by migrations.
syncdb is executed it creates all the tables except for those apps where migrations exists.
Post syncdb it tries to load initial_data and raises error of db not found because the table for app with migrations were not created by syncdb. [ Problem ]
Then it performs migration which creates the db.
Post migration it automatically loads initial_data successfully this time.
Problem:
How can I get rid of the error, when it tries to load fixture for the table that is not yet created ?
Can I edit the above script in a way so that it loads initial_data only after performing the migration ?
You could disable loading initial data when syncdb:
call_command('syncdb', load_initial_data=False, interactive=False)
call_command('migrate', interactive=False)
From the source code of syncdb.py:
# Stealth option -- 'load_initial_data' is used by the testing setup
# process to disable initial fixture loading.
load_initial_data = options.get('load_initial_data', True)
There are a few ways you can solve this:
Exclude the apps from the initial data dump by just calling the apps you want to populate data for
Could try this library https://github.com/davedash/django-fixture-magic
You could write a custom management command to populate the models you require
You can also use data migration that comes with south http://south.aeracode.org/docs/tutorial/part3.html
Personally I would go with either 1 or 3. With the first point, store the fixtures individually in each app under a fixtures folder. However, this is pain to update if your models change. So writing a custom management command might be the most painless.
Is there a way to execute a Django fixture just once - when the appropriate table is created? I have some initial data that should be put in the app tables, but once the tables are there, I don't want every ./manage.py syncdb to refresh the data. According to Django docs it seems this can only be done for fixtures in SQL format and not JSON / YAML:
http://docs.djangoproject.com/en/1.3/howto/initial-data/
You're going to want to use the post_syncdb signal, and filter/manually load the fixture via the underlying methods when specific apps or models are created.
The django projet I'm working on has a ton of initial_data fixture data. It seems by default the only way to have data load automatically is to have a file in your app folder called fixtures, and the file needs to be named initial_data.ext (ext being xml or json or yaml or something).
This is really unflexable, I think. I'd rather have a fixtures folder, and then inside that a initial_data folder, and then inside there, one file for each model in that app. Or something to that effect. IS this possible to do in django now? Or maybe some other scheme of better fixture organization.
In my experience, hard-coded fixtures are a pain to write and a pain to maintain. Wherever a model change breaks a fixture, the Django initial load will return a very unfriendly error message and you will end-up adding a bunch a of print's in the Django core in order to find where the problem is coming from.
One of the developer I work with has developed a very good library to overcome this problem, it's called django-dynamic-fixture and we really to love it. Here is how it works:
Suppose you have this models:
class Author(models.Model):
name = models.CharField()
class Book(models.Model):
author = models.ForeingKey(Author, required=True)
title = models.CharField()
In order to create a book instance in your test, all you have to do is
from django_dynamic_fixture import get
from app import Book
class MyTest(TestCase):
def setUp(self):
self.book = get(Book)
The django-dynamic-fixture automatically creates for you any dependencies that are required for the Book model to exist. This is just a simple example but the library can handle very complex model structures.
You can reorganize your initial data fixtures however you want and then write a post_syncdb signal handler which loads them. So it will then be automatically loaded on syncdb, according to the logic defined by you.
See: https://docs.djangoproject.com/en/1.3/ref/signals/#post-syncdb
Yes, you can split fixtures into multiple files with subfolder structures. You can specify fixture files to load and create a script that loads some or all of them. I have done this before so can confirm that it works.
Example: django-admin.py loaddata application/module/model.json
See loaddata documentation for more information.
A hacky way to load an additional initial_data.json or two is to create additional empty apps within your Django project that has nothing but the fixtures folder and the initial_data.json file. If you need the fixture loaded before the other apps' fixtures, you could name it something like aa1. If you need another one you could name it aa2. Your directory structure would look like this:
aa1/
fixtures/
initial_data.json
aa2/
fixtures/
initial_data.json
myrealapp/
fixtures/
initial_data.json
...
You would need to add the apps to INSTALLED_APPS in settings.py.
You can then populate the fixture_data.json files with arbitrary app information as necessary:
(virtualenv) ./manage.py dumpdata --indent=4 auth > aa1/fixtures/initial_data.json
(virtualenv) ./manage.py dumpdata --indent=4 oauth2 > aa2/fixtures/initial_data.json
(virtualenv) ./manage.py dumpdata --indent=4 myrealapp > myrealapp/fixtures/initial_data.json
When you run python manage.py syncdb, each of the fixtures will be loaded in automatically in alphabetical order.
As I mentioned, this is quite hacky, but if you only need a couple extra initial_data.json files and need to be able to control the order they are loaded in, this works.