This is an extension to this question: How to move a model between two Django apps (Django 1.7)
I need to move a bunch of models from old_app to new_app. The best answer seems to be Ozan's, but with required foreign key references, things are bit trickier. #halfnibble presents a solution in the comments to Ozan's answer, but I'm still having trouble with the precise order of steps (e.g. when do I copy the models over to new_app, when do I delete the models from old_app, which migrations will sit in old_app.migrations vs. new_app.migrations, etc.)
Any help is much appreciated!
Migrating a model between apps.
The short answer is, don't do it!!
But that answer rarely works in the real world of living projects and production databases. Therefore, I have created a sample GitHub repo to demonstrate this rather complicated process.
I am using MySQL. (No, those aren't my real credentials).
The Problem
The example I'm using is a factory project with a cars app that initially has a Car model and a Tires model.
factory
|_ cars
|_ Car
|_ Tires
The Car model has a ForeignKey relationship with Tires. (As in, you specify the tires via the car model).
However, we soon realize that Tires is going to be a large model with its own views, etc., and therefore we want it in its own app. The desired structure is therefore:
factory
|_ cars
|_ Car
|_ tires
|_ Tires
And we need to keep the ForeignKey relationship between Car and Tires because too much depends on preserving the data.
The Solution
Step 1. Setup initial app with bad design.
Browse through the code of step 1.
Step 2. Create an admin interface and add a bunch of data containing ForeignKey relationships.
View step 2.
Step 3. Decide to move the Tires model to its own app. Meticulously cut and paste code into the new tires app. Make sure you update the Car model to point to the new tires.Tires model.
Then run ./manage.py makemigrations and backup the database somewhere (just in case this fails horribly).
Finally, run ./manage.py migrate and see the error message of doom,
django.db.utils.IntegrityError: (1217, 'Cannot delete or update a parent row: a foreign key constraint fails')
View code and migrations so far in step 3.
Step 4. The tricky part. The auto-generated migration fails to see that you've merely copied a model to a different app. So, we have to do some things to remedy this.
You can follow along and view the final migrations with comments in step 4. I did test this to verify it works.
First, we are going to work on cars. You have to make a new, empty migration. This migration actually needs to run before the most recently created migration (the one that failed to execute). Therefore, I renumbered the migration I created and changed the dependencies to run my custom migration first and then the last auto-generated migration for the cars app.
You can create an empty migration with:
./manage.py makemigrations --empty cars
Step 4.a. Make custom old_app migration.
In this first custom migration, I'm only going to perform a "database_operations" migration. Django gives you the option to split "state" and "database" operations. You can see how this is done by viewing the code here.
My goal in this first step is to rename the database tables from oldapp_model to newapp_model without messing with Django's state. You have to figure out what Django would have named your database table based on the app name and model name.
Now you are ready to modify the initial tires migration.
Step 4.b. Modify new_app initial migration
The operations are fine, but we only want to modify the "state" and not the database. Why? Because we are keeping the database tables from the cars app. Also, you need to make sure that the previously made custom migration is a dependency of this migration. See the tires migration file.
So, now we have renamed cars.Tires to tires.Tires in the database, and changed the Django state to recognize the tires.Tires table.
Step 4.c. Modify old_app last auto-generated migration.
Going back to cars, we need to modify that last auto-generated migration. It should require our first custom cars migration, and the initial tires migration (that we just modified).
Here we should leave the AlterField operations because the Car model is pointing to a different model (even though it has the same data). However, we need to remove the lines of migration concerning DeleteModel because the cars.Tires model no longer exists. It has fully converted into tires.Tires. View this migration.
Step 4.d. Clean up stale model in old_app.
Last but not least, you need to make a final custom migration in the cars app. Here, we will do a "state" operation only to delete the cars.Tires model. It is state-only because the database table for cars.Tires has already been renamed. This last migration cleans up the remaining Django state.
Just now moved two models from old_app to new_app, but the FK references were in some models from app_x and app_y, instead of models from old_app.
In this case, follow the steps provided by Nostalg.io like this:
Move the models from old_app to new_app, then update the import statements across the code base.
makemigrations.
Follow Step 4.a. But use AlterModelTable for all moved models. Two for me.
Follow Step 4.b. as is.
Follow Step 4.c. But also, for each app that has a newly generated migration file, manually edit them, so you migrate the state_operations instead.
Follow Step 4.d But use DeleteModel for all moved models.
Notes:
All the edited auto-generated migration files from other apps have a dependency on the custom migration file from old_app where AlterModelTable is used to rename the table(s). (created in Step 4.a.)
In my case, I had to remove the auto-generated migration file from old_app because I didn't have any AlterField operations, only DeleteModel and RemoveField operations. Or keep it with empty operations = []
To avoid migration exceptions when creating the test DB from scratch, make sure the custom migration from old_app created at Step 4.a. has all previous migration dependencies from other apps.
old_app
0020_auto_others
0021_custom_rename_models.py
dependencies:
('old_app', '0020_auto_others'),
('app_x', '0002_auto_20170608_1452'),
('app_y', '0005_auto_20170608_1452'),
('new_app', '0001_initial'),
0022_auto_maybe_empty_operations.py
dependencies:
('old_app', '0021_custom_rename_models'),
0023_custom_clean_models.py
dependencies:
('old_app', '0022_auto_maybe_empty_operations'),
app_x
0001_initial.py
0002_auto_20170608_1452.py
0003_update_fk_state_operations.py
dependencies
('app_x', '0002_auto_20170608_1452'),
('old_app', '0021_custom_rename_models'),
app_y
0004_auto_others_that_could_use_old_refs.py
0005_auto_20170608_1452.py
0006_update_fk_state_operations.py
dependencies
('app_y', '0005_auto_20170608_1452'),
('old_app', '0021_custom_rename_models'),
BTW: There is an open ticket about this: https://code.djangoproject.com/ticket/24686
In case you need to move the model and you don't have access to the app anymore (or you don't want the access), you can create a new Operation and consider to create a new model only if the migrated model does not exist.
In this example I am passing 'MyModel' from old_app to myapp.
class MigrateOrCreateTable(migrations.CreateModel):
def __init__(self, source_table, dst_table, *args, **kwargs):
super(MigrateOrCreateTable, self).__init__(*args, **kwargs)
self.source_table = source_table
self.dst_table = dst_table
def database_forwards(self, app_label, schema_editor, from_state, to_state):
table_exists = self.source_table in schema_editor.connection.introspection.table_names()
if table_exists:
with schema_editor.connection.cursor() as cursor:
cursor.execute("RENAME TABLE {} TO {};".format(self.source_table, self.dst_table))
else:
return super(MigrateOrCreateTable, self).database_forwards(app_label, schema_editor, from_state, to_state)
class Migration(migrations.Migration):
dependencies = [
('myapp', '0002_some_migration'),
]
operations = [
MigrateOrCreateTable(
source_table='old_app_mymodel',
dst_table='myapp_mymodel',
name='MyModel',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=18))
],
),
]
I've built a management command to do just that - move a model from one Django app to another - based on nostalgic.io's suggestions at https://stackoverflow.com/a/30613732/1639699
You can find it on GitHub at alexei/django-move-model
You can do this relatively straightforwardly, but you need to follow these steps, which are summarized from a question in the Django Users' Group.
Before moving your model to the new app, which we will call new, add the db_table option to the current model's Meta class. We will call the model that you want to move M. But you can do multiple models at once if you want to.
class M(models.Model):
a = models.ForeignKey(B, on_delete=models.CASCADE)
b = models.IntegerField()
class Meta:
db_table = "new_M"
Run python manage.py makemigrations. This generates a new migration file that will rename the table in the database from current_M to new_M. We will refer to this migration file as x later on.
Now move the models to your new app. Remove the reference to db_table because Django will automatically put it in the table called new_M.
Make the new migrations. Run python manage.py makemigrations. This will generate two new migrations files in our example. The first one will be in the new app. Verify that in the dependencies property, Django has listed x from the previous migrations file. The second one will be in the current app. Now wrap the operations list in both migrations files in a call to SeparateDatabaseAndState to be like so:
operations = [
SeparateDatabaseAndState([], [
migrations.CreateModel(...), ...
]),
]
Run python manage.py migrate. You are done. The time to do this is relatively fast because unlike some answers, you're not copying records from one table to the other. You are just renaming tables, which is a fast operation by itself.
After work was done I tried to make new migration. But I facing with following error:
ValueError: Unhandled pending operations for models:
oldapp.modelname (referred to by fields: oldapp.HistoricalProductModelName.model_ref_obj)
If your Django model using HistoricalRecords field don't forget add additinal models/tables while following #Nostalg.io answer.
Add following item to database_operations at the first step (4.a):
migrations.AlterModelTable('historicalmodelname', 'newapp_historicalmodelname'),
and add additional Delete into state_operations at the last step (4.d):
migrations.DeleteModel(name='HistoricalModleName'),
Nostalg.io's way worked in forwards (auto-generating all other apps FKs referencing it). But i needed also backwards. For this, the backward AlterTable has to happen before any FKs are backwarded (in original it would happen after that). So for this, i split the AlterTable in to 2 separate AlterTableF and AlterTableR, each working only in one direction, then using forward one instead of the original in first custom migration, and reverse one in the last cars migration (both happen in cars app). Something like this:
#cars/migrations/0002...py :
class AlterModelTableF( migrations.AlterModelTable):
def database_backwards(self, app_label, schema_editor, from_state, to_state):
print( 'nothing back on', app_label, self.name, self.table)
class Migration(migrations.Migration):
dependencies = [
('cars', '0001_initial'),
]
database_operations= [
AlterModelTableF( 'tires', 'tires_tires' ),
]
operations = [
migrations.SeparateDatabaseAndState( database_operations= database_operations)
]
#cars/migrations/0004...py :
class AlterModelTableR( migrations.AlterModelTable):
def database_forwards(self, app_label, schema_editor, from_state, to_state):
print( 'nothing forw on', app_label, self.name, self.table)
def database_backwards(self, app_label, schema_editor, from_state, to_state):
super().database_forwards( app_label, schema_editor, from_state, to_state)
class Migration(migrations.Migration):
dependencies = [
('cars', '0003_auto_20150603_0630'),
]
# This needs to be a state-only operation because the database model was renamed, and no longer exists according to Django.
state_operations = [
migrations.DeleteModel(
name='Tires',
),
]
database_operations= [
AlterModelTableR( 'tires', 'tires_tires' ),
]
operations = [
# After this state operation, the Django DB state should match the actual database structure.
migrations.SeparateDatabaseAndState( state_operations=state_operations,
database_operations=database_operations)
]
This worked for me but I'm sure I'll hear why it's a terrible idea. Add this function and an operation that calls it to your old_app migration:
def migrate_model(apps, schema_editor):
old_model = apps.get_model('old_app', 'MovingModel')
new_model = apps.get_model('new_app', 'MovingModel')
for mod in old_model.objects.all():
mod.__class__ = new_model
mod.save()
class Migration(migrations.Migration):
dependencies = [
('new_app', '0006_auto_20171027_0213'),
]
operations = [
migrations.RunPython(migrate_model),
migrations.DeleteModel(
name='MovingModel',
),
]
Step 1: backup your database!
Make sure your new_app migration is run first, and/or a requirement of the old_app migration. Decline deleting the stale content type until you've completed the old_app migration.
after Django 1.9 you may want to step thru a bit more carefully:
Migration1: Create new table
Migration2: Populate table
Migration3: Alter fields on other tables
Migration4: Delete old table
Coming back to this after a couple of months (after successfully implementing Lucianovici's approach), It seems to me that it becomes much simpler if you take care to point db_table to the old table (if you only care about the code organisation and don't mind outdated names in the database).
You won't need AlterModelTable migrations, so there's no need for the custom first step.
You still need to change the models and relations without touching the database.
So what I did was just take the automatic migrations from Django and wrap them into migrations.SeparateDatabaseAndState.
Note (again) that this only could work if you took care to point db_table to the old table for each model.
I'm not sure if something is wrong with this that I don't see yet, but it seemed to have worked on my devel system (which I took care to backup, of course). All data looks intact. I'll take a closer look to check if any problems come up...
Maybe it's also possible to later rename the database tables as well in a separate step, making this whole process less complicated.
Coming this is one a little late but if you want the easiest path AND don't care too much about preserving your migration history. The simple solution is just to wipe migrations and refresh.
I had a rather complicated app and after trying the above solutions without success for hours, I realized that I could just do.
rm cars/migrations/*
./manage.py makemigrations
./manage.py migrate --fake-initial
Presto! The migration history is still in Git if I need it. And since this is essentially a no-op, rolling back wasn't a concern.
Related
I am building a Blog App and I am trying to add inbuilt initial instances in django admin so when user clone the repo , then user will see several initial blogs every time even after reset the database.
I didn't find anywhere to set the initial data. I also tried How to set initial data for Django admin model add instance form? But it was not what i am trying to do.
models.py
class BlogPost(models.Model):
title = models.CharField(max_length=1000)
body = models.CharField(max_length=1000)
I tried to use Providing data with fixtures But I have no idea , How can I store in.
Any help would be much Appreciated. Thank You.
Fixtures still need to be loaded manually. You can add that step into installation instruction, something like "to load example data, install the provided fixture via manage.py loaddata ./my_blog_fixture.json
If you want to have the data inserted into database without any user's action, then you are looking for a Data Migration
that's a kind of database migration which does not change the database structure in any way, but it executes a custom command, eg. inserting some data. An example (adjustments needed to match your app name) below. You can either generate an empty migration (recommended) or append RunCommand into an existing migration.
To generate a new empty migration run makemigrations
$ manage.py makemigrations your_app_name --empty
then edit the migration and add RunPython there (see linked docs above).
from django.db import migrations
def insert_blogpost(apps, schema_editor):
BlogPost = apps.get_model('your_app_name', 'BlogPost')
post = BlogPost(title="hello", body="post content")
post.save()
class Migration(migrations.Migration):
dependencies = [
('your_app_name', '0001_initial'),
]
operations = [
migrations.RunPython(insert_blogpost),
]
I noticed that when you run Django Migration, some data is added to table auth_permission when a new model is created. Is there any way you can take advantage of migration process to do the same with your own models? I red Django Documentation about writing Migrations, but it only covers the creation of brand new migration processes, it doesn't say anything about using the default one for your own porpoises.
You can write migrations.RunPython like this
def insert_data(apps, schema_editor):
YourModel = apps.get_model('<appname>', '<ModelName>')
# now insert data with YourModel.
class Migration(migrations.Migration):
dependencies = [
('<appname>', '<dependency>'),
]
operations = [
migrations.RunPython(insert_data),
]
notice these:
Model Loaded in migration hasn't implemented methods. for example is you customize save method, you must write that code here
if you want to make migration reversible you must write a new function for reverse operation and add argument reverse_code to your RunPython line. you can use migrations.RunPython.noop to make reverse migration do nothing.
I have quite a complex project architecture which involves several applications whose models contains cross references.
For example, I have a billing.Premium model - which belongs to the billing app - that is referenced by another model whose name is payments.PaymentJob through a one to one field:
('premium', models.OneToOneField(on_delete=django.db.models.deletion.PROTECT, to='billing.Premium', verbose_name='premium'))
(This code comes from one of payment's migrations)
But I have come to some point when I need to rename billing.Premium to billing.PremiumInstallment and this is when the funny part comes: after having refactored my code to replace the model name, I try to django-admin makemigrations, it leads to the following error:
ValueError: The field payments.PaymentJob.premium was declared with a lazy reference to 'billing.premium', but app 'billing' doesn't provide model 'premium'.
It appears like my migration has been broken since I have renamed the model of an external application. I do not know how to fix this in a fancy way, I mean generating some migration without error and that would be applied when I run django-admin migrate.
Any idea?
According to the docs for the RenameModel operation
You may have to manually add this if you change the model’s name and quite a few of its fields at once; to the autodetector, this will look like you deleted a model with the old name and added a new one with a different name, and the migration it creates will lose any data in the old table.
You should manually create a migration and add the RenameModel operation to it
class Migration(migrations.Migration):
dependencies = [
('billing', 'xxxx_previous_migration'),
]
operations = [
migrations.RenameModel('Premium', 'PremiumInstallment')
]
If one is using Django, what happens with changes made directly to the database (in my case postgres) through either pgadmin or psql?
How are such changes handled by migrations? Do they take precedence over what the ORM thinks the state of affairs is, or does Django override them and impose it's own sense of change history?
Finally, how are any of these issues effected, or avoided, by git, if at all?
Thanks.
You can exclude a model completely from the django migrations, and then you are responsible to adjust the schema to the django code (or the django code to the existing schema):
class SomeModel(models.Model):
class Meta:
managed = False
db_table = "some_table_name"
name = models.Fields....
Note that you can't have it both ways, so migrations are preferred when possible. You can always define a custom SQL migration, that will save the need for external changes. However, sometimes you do need to handle the schema elsewhere instead of migrations, and then use managed=False
The migrations system does not look at your current schema at all. It builds up its picture from the graph of previous migrations and the current state of models.py. That means that if you make changes to the schema from outside this system, it will be out of sync; if you then make the equivalent change in models.py and create migrations, when you run them you will probably get an error.
For that reason, you should avoid doing this. If it's done already, you could apply the conflicting migration in fake mode, which simply marks it as done without actually running the code against the database. But it's simpler to do everything via migrations in the first place.
git has no impact on this at all, other than to reiterate that migrations are code, and should be added to your git repo.
I'm running into quite interesting situation.
I need to extend default django's Group model with some fields.
I tried to use inheritance first, e.g. inherit from Group model and change some references, but seems I can't change all needed references, so, this way completely breaks django permission system.
Then I found this answer: How do I extend the Django Group model? where guy suggested to use field.contribute_to_class() method.
I have put this adjustment right above the model definition in < myapp >. (don't ask me why do I need roles for group, it's not my idea, I just need them :D)
if not hasattr(Group, 'roles'):
field = models.ManyToManyField(
Role, verbose_name=_('roles'), blank=True,
help_text=_('List of roles attached to this group'),
related_name='groups')
field.contribute_to_class(Group, 'roles')
class MyGroup(Group):
class Meta:
proxy = True
def _sync_permissions(self):
"""
This method will sync group permissions with all attached Roles.
"""
self.permissions.clear()
for role in self.roles.all():
self.permissions.add(role.permissions)
self.save()
This part seems to be working (it really modifies django.contrib.auth.models.Group model)
But what I need next is to generate a migration for the Group model.
If I simply run ./manage.py makemigrations <myapp> it generates a migration for Group model, but tries to put it inside django.contrib.auth application, that is definitely not what I need.
So, my question here is:
Is there a way to tell django to generate a migration for Group model, but not to create a migration file under python libs directory, but rather create it inside < myapp > or just output the migration code?
the location where django looks for migrations cn be custominzed using MIGRATION_MODULES in your settings.py, anyway this means that ALL the migrations (not only the new) must be there.
You need to copy the original migrations and manually update them when you upgrade Django
You can create a dedicated package so to not clash with your migrations
Es.
MIGRATION_MODULES = {
'django.contrib.auth' : 'myapp.auth_migrations',
'myapp': 'myapp.migrations' # this line is only to clarify. IT'S NOT NEEDED AT ALL
}
I have not seen the field.contribute_to_class() to class solution to this problem before.
For me it looks a bit dirty, like monkey patching.
You would go back and use a 1:1 relation to Group, or inherit from Group.
You say that you can't change all needed references to Group? What is missing?
I know this is not an answer to your question, but maybe a solution to your problem :-)