I have a form which enables a user to register on our website. Now I need to export all the data to excel, so I turned towards the import-export package. I have 3 models, Customer, Reference and Contact. The latter two both have a m2m with Customer. I also created Resources for these models. When I use Resource().export() at the end of my done() method in my form view, it exports all existing objects in the database, which is not what I want.
I tried googling this and only got one result, which basically says I need to use before_export(), but I can't find anywhere in the docs how it actually works.
I tried querying my customer manually like:
customer = Customer.objects.filter(pk=customer.id)
customer_data = CustomerResource().export(customer)
which works fine but then I'm stuck with the related references and contacts: reference_data = ReferenceResource().export(customer.references) gives me an TypeError saying 'ManyRelatedManager' object is not iterable. Which makes sense because export() expects an queryset, but I'm not sure if it's possible getting it that way.
Any help very appreciated!
One way is to override get_queryset(), you could potentially try to load all related data in a single query:
class ReferenceResource(resources.ModelResource):
def __init__(self, customer_id):
super().__init__()
self.customer_id = customer_id
def get_queryset(self):
qs = Customer.objects.filter(pk=self.customer.id)
# additional filtering here
return qs
class Meta:
model = Reference
# add fields as appropriate
fields = ('id', )
To handle m2m relationships, you may be able to modify the queryset to add these additional fields.
This isn't the complete answer but it may help you make progress.
Suppose there is a production database, there is some data in it. I need to migrate in the next tricky case.
There is a model (already in db), say Model, it has foreign keys to other models.
class ModelA: ...
class ModelX: ...
class Model:
a = models.ForeignKey(ModelA, default = A)
x = models.ForeignKey(ModelX, default = X)
And we need to create one more model ModelY to which Model should refer. And when creating a Model, an object should have some default value related to some ModelY object, which is obviously not yet available, but we should create it during migration.
class ModelY: ...
class Model:
y = models.ForeignKey (ModelY, default = ??????)
So the migration sequence should be:
Create ModelY table
Create a default object in this table, put its id somewhere
Create a new field y in the Model table, with the default value taken
from the previous paragraph
And I'd like to automate all of this, of course. So to avoid necessity to apply one migration by hands, then create some object, then write down it's id and then use this id as default value for new field, and only then apply another migration with this new field.
And I'd also like to do it all in one step, so define both ModelY and a new field y in the old model, generate migration, fix it somehow, and then apply at once and make it work.
Are there any best practices for such case? In particular, where to store this newly created object's id? Some dedicated table in same db?
You won't be able to do this in a single migration file, however you'll be able to create several migrations files to achieve this. I'll have a go at helping you out though I'm not totally certain this is what you want, it should teach you a thing or two about Django migrations.
I'm going to refer to two types of migrations here, one is a schema migration, and these are the migration files you typically generate after changing your models. The other is a data migration, and these need to be created using the --empty option of the makemigrations command, e.g. python manage.py makemigrations my_app --empty, and are used to move data around, set data on null columns that need to be changed to non-null, etc.
class ModelY(models.Model):
# Fields ...
is_default = models.BooleanField(default=False, help_text="Will be specified true by the data migration")
class Model(models.Model):
# Fields ...
y = models.ForeignKey(ModelY, null=True, default=None)
You'll notice that y accepts null, we can change this later, for now you can run python manage.py makemigrations to generate the schema migration.
To generate your first data migration run the command python manage.py makemigrations <app_name> --empty. You'll see an empty migration file in your migrations folder. You should add two methods, one that is going to create your default ModelY instance and assign it to your existing Model instances, and another that will be a stub method so Django will let you reverse your migrations later if needed.
from __future__ import unicode_literals
from django.db import migrations
def migrate_model_y(apps, schema_editor):
"""Create a default ModelY instance, and apply this to all our existing models"""
ModelY = apps.get_model("my_app", "ModelY")
default_model_y = ModelY.objects.create(something="something", is_default=True)
Model = apps.get_model("my_app", "Model")
models = Model.objects.all()
for model in models:
model.y = default_model_y
model.save()
def reverse_migrate_model_y(apps, schema_editor):
"""This is necessary to reverse migrations later, if we need to"""
return
class Migration(migrations.Migration):
dependencies = [("my_app", "0100_auto_1092839172498")]
operations = [
migrations.RunPython(
migrate_model_y, reverse_code=reverse_migrate_model_y
)
]
Do not directly import your models to this migration! The models need to be returned through the apps.get_model("my_app", "my_model") method in order to get the Model as it was at this migration's point in time. If in the future you add more fields and run this migration your models fields may not match the databases columns (because the model is from the future, sort of...), and you could receive some errors about missing columns in the database and such. Also be wary of using custom methods on your models/managers in migrations because you won't have access to them from this proxy Model, usually I may duplicate some code to a migration so it always runs the same.
Now we can go back and modify the Model model to ensure y is not null and that it picks up the default ModelY instance in the future:
def get_default_model_y():
default_model_y = ModelY.objects.filter(is_default=True).first()
assert default_model_y is not None, "There is no default ModelY to populate with!!!"
return default_model_y.pk # We must return the primary key used by the relation, not the instance
class Model(models.Model):
# Fields ...
y = models.ForeignKey(ModelY, default=get_default_model_y)
Now you should run python manage.py makemigrations again to create another schema migration.
You shouldn't mix schema migrations and data migrations, because of the way migrations are wrapped in transactions it can cause database errors which will complain about trying to create/alter tables and execute INSERT queries in a transaction.
Finally you can run python manage.py migrate and it should create a default ModelY object, add it to a ForeignKey of your Model, and remove the null to make it like a default ForeignKey.
Finally I came to the following solution.
First I accept the idea to identify default object by isDefault attribute and wrote some abstract model to deal with it, keeping data integrity as much as possible (code is in bottom of the post).
What I don't like much in accepted solution, is the data migrations are mixed with schema migrations. It's easy to lost them, i.e. during squashing. Occasionally I am also deleting migrations at all, when I am sure all my production and backup databases are in consistence with the code, so I can generate single initial migration and fake it. Keeping data migration together with schema migrations breaks this workflow.
So I decide to keep all data migrations in single file outside of migrations package. So I create data.py in my app package and put all data migrations in single function migratedata, keeping in mind that this function can be called on early stages, when some models still may not exist, so we need to catch LookupError exception for apps registry access. Than I use this function for every RunPython operations in data migrations.
So the workflow looks like that (we assume Model and ModelX are already in place):
1) Create ModelY:
class ModelY(Defaultable):
y_name = models.CharField(max_length=255, default='ModelY')
2) Generate migration:
manage.py makemigration
3) Add data migration in data.py (add name of the model to defaultable list in my case):
# data.py in myapp
def migratedata(apps, schema_editor):
defaultables = ['ModelX', 'ModelY']
for m in defaultables:
try:
M = apps.get_model('myapp', m)
if not M.objects.filter(isDefault=True).exists():
M.objects.create(isDefault=True)
except LookupError as e:
print '[{} : ignoring]'.format(e)
# owner model, should be after defaults to support squashed migrations over empty database scenario
Model = apps.get_model('myapp', 'Model')
if not Model.objects.all().exists():
Model.objects.create()
4) Edit migration by adding operation RunPython:
from myapp.data import migratedata
class Migration(migrations.Migration):
...
operations = [
migrations.CreateModel(name='ModelY', ...),
migrations.RunPython(migratedata, reverse_code=migratedata),
]
5) Add ForeignKey(ModelY) to Model:
class Model(models.Model):
# SET_DEFAULT ensures that there will be no integrity issues, but make sure default object exists
y = models.ForeignKey(ModelY, default=ModelY.default, on_delete=models.SET_DEFAULT)
6) Generate migration again:
manage.py makemigration
7) Migrate:
manage.py migrate
8) Done!
The whole chain can be applied to empty database, it will create final schema and fill it with initial data.
When we sure, that our db is in sync with code we can easily remove long chain of migrations, generate single initial one, add RunPython(migratedata, ...) to it, and then migrate with --fake-initial (delete django_migrations table before).
Huh, so so tricky solution for such simple task!
Finally there is Defaultable model source code:
class Defaultable(models.Model):
class Meta:
abstract = True
isDefault = models.BooleanField(default=False)
#classmethod
def default(cls):
# type: (Type[Defaultable]) -> Defaultable
"""
Search for default object in given model.
Returning None is useful when applying sqashed migrations on empty database,
the ForeignKey with this default can still be non-nullable, as return value
is not used during migration if there is no model instance (Django is not pushing
returned default to the SQL level).
Take a note on only(), this is kind of dirty hack to avoide problems during
model evolution, as default() can be called in migrations within some
historical project state, so ideally we should use model from this historical
apps registry, but we have no access to it globally.
:return: Default object id, or None if no or many.
"""
try:
return cls.objects.only('id', 'isDefault').get(isDefault=True).id
except cls.DoesNotExist:
return None
# take care of data integrity
def save(self, *args, **kwargs):
super(Defaultable, self).save(*args, **kwargs)
if self.isDefault: # Ensure only one default, so make all others non default
self.__class__.objects.filter(~Q(id=self.id), isDefault=True).update(isDefault=False)
else: # Ensure at least one default exists
if not self.__class__.objects.filter(isDefault=True).exists():
self.__class__.objects.filter(id=self.id).update(isDefault=True)
def __init__(self, *args, **kwargs):
super(Defaultable, self).__init__(*args, **kwargs)
# noinspection PyShadowingNames,PyUnusedLocal
def pre_delete_defaultable(instance, **kwargs):
if instance.isDefault:
raise IntegrityError, "Can not delete default object {}".format(instance.__class__.__name__)
pre_delete.connect(pre_delete_defaultable, self.__class__, weak=False, dispatch_uid=self._meta.db_table)
I left my previous answer just to show search for thoughts. Finally I've founded fully automatic solution, so it's not necessary anymore to manually edit django generated migrations, but the price is monkey patching, as often.
The idea is to provide callable for default of ForeignKey, which creates default instance of referenced model, if it is not exists. But the problem is, that this callable can be called not only in final Django project stage, but also during migrations, with old project stages, so it can be called for deleted model on early stages, when the model was still existing.
The standard solution in RunPython operations is to use apps registry from the migration state, but this feature unavailable for our callable, cause this registry is provided as argument for RunPython and not available globally. But to support all scenarios of migration applying and rollback we need to detect are we in migration or not, and access appropriate apps registry.
The only solution is to monkey patch AddField and RemoveField operations to keep migration apps registry in global variable, if we are in migration.
migration_apps = None
def set_migration_apps(apps):
global migration_apps
migration_apps = apps
def get_or_create_default(model_name, app_name):
M = (migration_apps or django.apps.apps).get_model(app_name, model_name)
try:
return M.objects.get(isDefault=True).id
except M.DoesNotExist as e:
o = M.objects.create(isDefault=True)
print '{}.{} default object not found, creating default object : OK'.format(model_name, app_name)
return o
def monkey_patch_fields_operations():
def patch(klass):
old_database_forwards = klass.database_forwards
def database_forwards(self, app_label, schema_editor, from_state, to_state):
set_migration_apps(to_state.apps)
old_database_forwards(self, app_label, schema_editor, from_state, to_state)
klass.database_forwards = database_forwards
old_database_backwards = klass.database_backwards
def database_backwards(self, app_label, schema_editor, from_state, to_state):
set_migration_apps(to_state.apps)
old_database_backwards(self, app_label, schema_editor, from_state, to_state)
klass.database_backwards = database_backwards
patch(django.db.migrations.AddField)
patch(django.db.migrations.RemoveField)
The rest, including Defaultable model with data integrity check are in GitHub repository
I am using Django's dumpdata to save data and loaddata to reload it. I am also using natural keys. My model looks similar to this:
class LinkManager(models.Manager):
def get_by_natural_key(self, url):
return self.get(url=url)
class Link(models.Model):
objects = LinkManager()
title = models.CharField(max_length=200)
url = models.URLField()
def natural_key(self):
return (self.url, )
If I export and reimport the data, Django recognizes that the objects already exist and doesn't create duplicates. If I change the title, it correctly updates the objects. However, if I change the URL, it correctly treats it as a new object - although I forgot to mark url unique! How does it guess my intent?
How does django know that my url field is the natural key? There is no get_natural_fields function. Django could call natural_key on the class instead of an instance to get the fields, but that seems really brittle:
>>> [f.field_name for f in Link.natural_key(Link)]
['url']
The reason I want to know this is that I am writing my own special importer (to replace my use of loaddata), and I would like to take advantage of natural keys without hardcoding the natural key (or the "identifying" fields) for each model. Currently, I "identify" an object by it's unique fields - I do:
obj, created = Model.objects.update_or_create(**identifying, defaults=other)
but Django seems to be choosing it's "identifying" fields differently.
I think I've found it out. Django does not just call get_by_natural_key, it first calls natural_key. How does it do that, if it doesn't have an instance of the model?
It simply creates an instance, not backed by the database, from the constructor (d'oh!): Model(**data). See build_instance in django.core.serializers.base. Then it calls natural_key on the newly created object, and immediately get_by_natural_key to retrive the pk that belongs to the object, if present in the database. This way, Django does not need to know what fields the natural key depends on, it just needs to know how to get it from data. You can just call save() on the retrieved instance, if it is in the database it will have a pk and will update, if not it will create a new row.
Source of the build_instance function (Django 1.11.2):
def build_instance(Model, data, db):
"""
Build a model instance.
If the model instance doesn't have a primary key and the model supports
natural keys, try to retrieve it from the database.
"""
obj = Model(**data)
if (obj.pk is None and hasattr(Model, 'natural_key') and
hasattr(Model._default_manager, 'get_by_natural_key')):
natural_key = obj.natural_key()
try:
obj.pk = Model._default_manager.db_manager(db).get_by_natural_key(*natural_key).pk
except Model.DoesNotExist:
pass
return obj
I want to Exclude link from a Data field.
Like, I have a model named "Profile". in the Model, there is a field named "facebook". in that field, data is saved like "https://www.facebook.com/user_name". I want to exclude the "https://www.facebook.com/" and keep only the "user_name" data.
So how I can do that with South migration?
I have made a data migration with ./manage.py datamigration , and in that migration I have changed as following
class Migration(DataMigration):
def forwards(self, orm):
"Write your forwards methods here."
orm.Profile.objects.exclude(facebook='https://www.facebook.com/').update(facebook='')
def backwards(self, orm):
"Write your backwards methods here."
raise RuntimeError('Cannot reverse this migration.')
But, it seems that its deleting all the Data of that field.
So what should I do in order to keep only the "user_name" and exclude the "https:/www.facebook.com/" data?
Regards
This isn't exactly what South is for... but if this is how you need to handle it, I'm pretty sure you can do the following (but test it on a copy first):
def forwards(self, orm):
for profile in orm.Profile.objects.all():
profile.facebook = profile.facebook.split('/')[-1]
profile.save()
I recently added a new method to my model "News":
def slug(self):
return slugify(self.title)
However I can't seem to call it from anywhere. Imagine I have a News object called n. When trying to call
n.slug()
I always get an error like:
AttributeError: 'News' object has no attribute 'slug'
I'm a total beginner at Django and Python and I'm sure this is just a stupid mistake on my side. By the way I tried restarting the dev server and syncing the db after I added the method, both don't change a thing. Note that I have used model methods like this before without a problem :S
Edit:
Here is the model:
from django.template.defaultfilters import slugify
class News(models.Model):
title = models.CharField(max_length=100)
[...]
def slug(self):
return slugify(self.title)
Here is some example code on how I call the method. I first tried to call it in a template, but it didn't work. Then I changed my view so it just returns the slug, but the error remains. Same when I try it in the shell.
from fbki.models import News
def news_detail(request, slug, news_id):
news = News.objects.get(id = news_id)
return HttpResponse(news.slug())
There are, no errors. Please check what class you by
from fbki.models import News
it looks like you have two copies in project, and modify other class. You check by renaming your class News to News1. It you will same error - I'm right.
I've had some errors similar to this.
I believe that the problem is an inconsistency between your model and the actual schema in your database.
Run manage.py sql myapp and verify the contents match what is in sql> show schema (etc.)
If they don't match, you have to use your SQL client and drop the old table so that you can re-run manage.py syncdb to make them match again.
Your method should work once the schema are present.