My Django unit tests are quite slow. I'm using in-memory sqlite and keeping fixtures to a minimum. However it still takes about 5 seconds to load a fixture.
I've read json fixtures are faster than yaml ones and am therefore using json.
My suspicion is that the fixtures are using a slow json library (json or simplejson instead of cjson).
My questions are these:
Can anyone confirm the fixture loading is the slow part or have an
idea on how to go about this?
Does anyone know what json library django uses to load fixtures?
Django fixtures are loaded using loaddata management command which uses built-in django deserializers which use json module from the standard library.
Fixtures are slow and difficult to maintain. Consider using model factories instead. Basically, there are two major players out there:
factory_boy
model_mommy
Also see:
On Fixtures and Factories
Factory Boy as an Alternative to Django Testing Fixtures
Related
I'm trying to understand best-practices around Django test data loading. I have a simple API that has some very long lists of returned JSON data, and I want to write a behaviorial test to make sure it works right.
Fixtures, or something like them, seem like they should do the trick - but it looks like Django fixtures are intended for seeding the database for tests, and not providing more general sampled/serialized data - e.g. I haven't been able to find an API that lets me load data from one specific fixture.
What's the generally-accepted way for me to manually store/version a file on disk that contains data that will be loaded and self.assertEqual'ed against in my test cases?
Any advice appreciated - thank you!
you should take a look at factory_boy or django model mommy. They are preferred way to load fixtures as compared to the default fixture loading supported by django. But if you really need to use django default fixture,
class SiteTests(TestCase):
#This is the fixture:
#- fields: {content: lots of stuff, query: test, title:
test, url: 'http://google.com'}
#model: mine.site
#pk: 1
fixtures = ['mine']
where mine is the name of the fixture file. Take a look at http://django-testing-docs.readthedocs.org/en/latest/fixtures.html on how it is done.
When I run "python manage.py syncdb" I want not empty DB but with the data I want. If there a hook in Django to run a number of foo.save() lines?
From Django docs about fixtures:
Providing initial data with fixtures
A fixture is a collection of data that Django knows how to import into a database. The most straightforward way of creating a fixture if you’ve already got some data is to use the manage.py dumpdata command. Or, you can write fixtures by hand; fixtures can be written as JSON, XML or YAML (with PyYAML installed) documents. The serialization documentation has more details about each of these supported serialization formats.
Before version 1.7 Django there was a mechanism to load fixtures automatically:
If you create a fixture named initial_data.[xml/yaml/json], that fixture will be loaded every time you run migrate. This is extremely convenient, but be careful: remember that the data will be refreshed every time you run migrate. So don’t use initial_data for data you’ll want to edit.
If you are using Django>=1.7, you must issue the loaddata manage command or create a migration:
If an application uses migrations, there is no automatic loading of fixtures. Since migrations will be required for applications in Django 2.0, this behavior is considered deprecated. If you want to load initial data for an app, consider doing it in a data migration.
The JSON serializer used to choke on large inputs (tried to load everything in memory or something like that); the XML serializer used to behave better for larger fixtures.
Assuming you are using Django 1.7, you can write a data migration to insert any data you need.
There's a whole django docs page about it.
In short, you have three options:
Provide a set of "fixtures", that describe models. This way is DB agnostic and can be used with any DB django can talk to.
Provide a set of SQL scripts to run. Benefit you can get here is to use some database specific data types/programming capabilities/etc. However, it's considered deprecated in Django 1.7 and will be removed in Django 2.0
Create a set of data migrations (comes with Django >=1.7, for earlier versions use South)
Also, loading initial data can occur every time migrate is run:
If you create a fixture named initial_data.[xml/yaml/json], that fixture will be loaded every time you run migrate. This is extremely convenient, but be careful: remember that the data will be refreshed every time you run migrate. So don’t use initial_data for data you’ll want to edit.
For a Django test I'd like to load a fixture, which is in a csv file. What is the best way to do that?
Django's built-in fixtures functionality doesn't support CSV. You'd need to process the file automatically using the csv module, probably in the test's setUp method.
For easier reuse I'd suggest my Django-PyFixture library. You'd then program your fixture in raw python, but it would semantically look like one more fixture.
A straight forward question:
Is there some easy way to write fixtures (i.e. in JSON format, but I don't care really) when using models that incorporate PickleFields?
EDIT:
In the end I think I'll just get rid of fixtures altogether. I'll use named *.py scripts that will create all the objects for me. I've always found fixtures quite cumbersome anyway.
Assuming you are using django-picklefield you can use dumpdata/loaddata just like you would with any other model. Tested it briefly and everything works fine.
I have an initial_data fixture that I want to load everytime except for production. I already have different settings file for production and non-production deployments.
Any suggestions on how to accomplish this?
Clarification: I do not want test fixtures. Basically, I just need the fixture to be loaded based on a setting change of some sort. I'll be digging into the Django code to see if I could figure out an elegant way to accomplish this.
You can actually setup different test fixtures for each test if you want:
http://docs.djangoproject.com/en/dev/topics/testing/#topics-testing-fixtures
If you only want to load the fixtures in one time, you can also write a custom TestRunner that will allow you to do that setup at the beginning:
docs.djangoproject.com/en/dev/topics/testing/#using-different-testing-frameworks
Both of those will still load the data from the production fixtures as that is done with syncdb, but you can override the data, or even delete it all. This may not be optimal if you are loading large amounts of data into your production product. If this is the case, I would recommend you adding a custom command like load_production_data that allows you to do it quickly and easily from the command line.
The easiest way is to use manage.py testserver [fixture ...]
If this is a staging (rather than dev) deployment, though, you may not want to use django's builtin server. In that case, a quick (if hacky) way of doing what you're after is to have the fixtures in an app (called, for example, "undeployed") that is only installed in your non-production settings.