I've been trying to wrap my head around how to avoid creating a mess with Django migrations. I tried "don't delete migrations" and for the most part that worked, but when I wanted to delete an app off my codebase, I struggled and fell back into a mess.
I was thinking, it would be far easier to untangle these issue if one could:
1- delete all current migrations
2- create a migration from an existing database schema (rather than from Django models) <- this is the missing step
3- run migrate --fake to synch DB with migrations
4- run makemigrations to add any non-applied changes to the migration chain (this time from Django models)
5- run migrate to synch the DB with the new changes
The only step that I don't know how to do is step 2. Is this doable? I don't know of any built in module or tool that does this so I'm wondering why not.
Some research showed python manage.py inspectDB gets me partially through step 2. But is there an automated way to infer migrations from an existing database?
Related
I have been assigned the task to work on project involving react UI and django backend. There are two versions of this app:
older version which runs on older versions of react and python (v2)
newer version which runs on newer versions of react and python (v3)
I have been tasked to move some functionality from older version to newer version and test it with postgres database dump from a live environment running older version. The challenge could be differences in the database schema of newer and older versions. (But, most possibly there wont be much differences if some minor ones.)
So I proceeded to take the database dump, attached it to the database running on my laptop and specified its name in my django's settings.ini. But when I started my django app, it gave me error
You have 7 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, lab, otherlabs. Run 'python manage.py migrate' to apply them.
When I ran python manage.py migrate, it gave me an error:
Migration abc.0001_initial is applied before its dependency auth.0011_xyz on database 'default'.
So, I deleted record corresponding to abc.0001_initial from django_migrations table and reran python manage.py migrate. Now I got same error for migration of another project def but with the same dependency auth.0011_xyz:
Migration def.0001_initial is applied before its dependency auth.0011_xyz on database 'default'.
Should I proceed with deleting record corresponding to def.0001_initial too? Am afraid that this will continue till I delete all such 0001_initial-suffixed records. There are 35 projects and hence 35 such 0001_initial-suffixed records.
Q1. Is it safe to delete them all and run the migrations? Is deleting all migrations and rerunning them really bad idea if they are schema-only migrations (we use fixtures for importing data)? Or is it simply impossible to do as most of those migrations will fail as tables corresponding to most models are already there in the database (or this wont give any error?), so the only solution remained is to delete migrations record one by one from db till I get no errors. But wont this still give error as I am only deleting migrations record and not actual tables created by migrations? Is there any other approach?
Q2. Are migrations completely safe to "re"-run? I mean if one fails because corresponding table / columns are already there or for some other reasons, will it leave database in consistent state or will it mess up the database?
Q3. Also will it continue to execute rest of the migrations after one fails? OR
Q4. Can / should I execute rest of the migrations if one fails? If yes, do I need to pass some arguments to python manage.py migrate or its default behavior?
Update
I tried removing all migrations from the django_migrations table. And then running python manage.py makemigrations followed by python manage.py migrate. But migration did not complete and terminated with error "relation already exist". So applied fake migration python manage.py migrate --fake. It succeeded (seems that it faked / skipped all migrations as all corresponding tables were present, so it did not create new columns in already existing tables). But then my REST API calls started failing with error in response saying: "column xyz does not exist".
Manually copying data seems impossible since there are 156 tables 🥲
Please let me know if there are any solution to this.
You need to delete the migration row from django_migrations for migration def.0001_initial, then apply migrate auth 0011_xyz to fix the dependency and then you should can to do migrate def 0001_initial --fake to fake the migration and add the row you deleted before to django_migrations.
I have an existing database filled with a bunch of data. I want to migrate to Django so I went ahead and created the necessary models with python manage.py inspectdb. I haven't changed anything besides removed the managed = False since I want Django to manage the tables (mistake perhaps for initial migration?).
So now that the models are ready, how can I generate the first migration file so that I can start changing the fields to generate additional migrations (renaming a field here and there). I understand python manage.py migrate will handle the creation of Django-specific models but does not actually create any migration files? Some sources indicate the first migration file should be run with --fake so it's not applied. Will the next migration files remember to run the first one as fake and only apply the next ones?
You want makemigrations to create the migrations. The migrate command applies migrations, it does not create them.
You can use the --fake-initial option so that Django does not try to create the tables that already exist. Note that --fake and --fake-initial are two different commands.
When you run migrate, the django_migrations table is updated to store the currently applied migrations. The migration files themselves are not changed. The --fake command updates the django_migrations table without running the migration. That means that if you use it incorrectly your database and django_migrations table can get out of sync, which can be difficult to fix.
I have a Python/Django proyect. Due to some rolls back, and other mixed stuff we ended up in a kind of odd scenario.
The current scenario is like this:
DB has the correct tables
DB can't be rolled back or dropped
Code is up to date
Migrations folder is behind the DB by one or two migrations. (These migrations were applied from somewhere else and that "somewhere else" doesn't exist anymore)
I add and alter some models
I run makemigrations
New migrations are created, but it's a mix of new tables and some tables that already exist in the DB.
If I run migrate it will complain that some of the tables that I'm trying to create already exist.
What I need:
To be able to run the migrations and kind of "ignore" the existing tables and apply the new ones. Or any alternative way to achieve this.
Is that possible?
When you apply a migration, Django inserts a row in a table called django_migrations. That's the only way Django knows which migrations have been applied already and which have not. So the rows in that table have to match the files in your migrations directory. If you've lost the migration files after they were applied, or done anything else to get things out of sync, you'll have problems.. because the migration numbers in your database refer to different migration files than the ones in your project.
So before you do anything else, you need to bring things back into sync by deleting the django_migrations table rows for any migration files that you've lost somehow and can't get back. The table should contain rows for only those migrations that you do have and that were actually applied to the database correctly.
Now you need to deal with any changes in your database that Django Migrations doesn't know about.. and for that there are a few options:
If things worked out such that the database changes that were already applied to the database are in different migration files than the ones that weren't, then you can fix it by running your migrations one at a time using the --fake option on any changes that are in reality already in the database. The fake option just writes the row to the django_migrations table marking the migration as done. Only do this if the database does in fact already have all the changes contained in that migration file.
And those migration files that contain only changes which have not been applied to the database, run without the --fake option and Django will apply them. eg:
# database already has it
manage.py migrate myapp 0003 --fake
# need it
manage.py migrate myapp 0004
# database already has it
manage.py migrate myapp 0005 --fake
If you have migration files where some but not all of the changes have been applied, then you have a bigger problem. In that case, there are several ways to go about it (choose ONLY ONE):
Edit the migration files to put changes that have already been applied (whether Django did it or you did it manually does not matter) into lower number migrations, and put everything you need done into higher numbered files. Now you can --fake the lower number ones, and run the higher numbered ones as normal. Let's say you have 10 changes you made to your models, and 5 of those changes are actually in the database already, but Django doesn't know about them.. so when you run makemigrations, a new migration is created with all 10 changes. This will normally fail because the database server can't for example add a column that already exists. Move these already-applied changes out of your new migration file, into the previous (already applied) migration file. Django will then assume that these were applied with the previous migration and will not try to apply them again. You can then migrate as normal and the new changes will be applied.
If you don't want to touch your older migration file, a cleaner way to do this is to first run makemigrations --empty appname to create an empty migration file. Then run makemigrations which will create another migration with all the changes that Django thinks need to be done. Move the already done migrations from that file into the empty migration you created.. then --fake that one. This will put Django's understanding of what the database looks like will be in sync with reality and you can then migrate as normal, applying the changes in the last migration file.
Get rid of any new migrations you just created using makemigrations. Now, comment out or put back anything in your models that has not been applied to the database, leaving your code matching what's actually in the database. Now you can do makemigrations and migrate appname --fake and you will get things back in sync. Then uncomment your new code and run 'makemigrations' then migrate as normal and the changes will be applied. If the changes are small (for example, adding a few fields), sometimes this is easiest. If the changes are large, it isn't....
You can go ahead and (carefully) make the database changes yourself, bringing the database up to date. Now just run migrate --fake and if you didn't mess up then everything will be ok. Again, this is easy for smaller changes, not as easy for complicated ones.
You can run manage.py sqlmigrate > mychanges.sql. This generates mychanges.sql containing all the SQL Django WOULD have executed against the database. Now edit that file to remove any changes that have already been applied, leaving what needs to be done. Execute that SQL using pgadmin or psql (you're using postgresql I hope). Now the changes have all been made.. so you can run manage.py migrate --fake, this will bring Django into sync with reality and you should be all set. If your SQL skills are sufficient, this is probably the most straightforward solution.
I should add two warnings:
First, if you apply a later migration, eg 0003_foobar.py, and then things don't work out and you decide to try going back and apply 0002_bazbuz.py, then Django will TAKE STUFF OUT OF YOUR DATABASE. For example a column you might have added in 0003 will be dropped along with its data. Since you say you can't lose data, be very careful about going back.
Second, do not rush into running --fake migrations. Make sure that the entire migration you are about to fake is actually in the database already. Else it gets very confusing. If you do regret faking migrations and don't want to roll back, you can erase django's knowledge of the faked migration by deleting that row from the django_migrations table. It is ok to do this.. if you understand what you are doing. If you know that the migration really was not applied, then it's ok.
This blog post really nails it. https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html
Let me summarize the steps in his scenario 2 (you have a production database and want to change schema/models in one or more apps). In my case, I had two apps, queue and routingslip, that had model modifications that I needed to apply to a production system. Key was I already had the database, so this is where --fake-initial comes into play.
Here are the steps I followed. As always, backup everything before starting. I do work in a VM so I just took a snapshot before going forward.
1) Remove the migration history for each app.
python manage.py migrate --fake queue zero
python manage.py migrate --fake routingslip zero
2) Blow away any migration files in the entire project within which the app(s) reside.
find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete
3) Make migrations
python manage.py makemigrations
4) Apply the migrations, faking initial because the database already exists and we just want the changes:
python manage.py migrate --fake-initial
Worked great for me.
If you don't have any migration files or you lost the previous file and want to migrate new changes, then you need to follow the following steps carefully:
# To create a new migration file before changing the models.
cmd: python manage.py makemigrations app_name
# Fake migrate
cmd: python manage.py migrate app_name 0005 --fake #[0005 is the migration file number created just now. It'll seem like 0005_add_address or something like this.]
# To create a new migration file after changing the models.
cmd: python manage.py makemigrations app_name
# database already has it
cmd: python manage.py migrate app_name 0006 #[0006 is the migration file number created just now.]
I am trying to setup a Django app locally in a new machine but migrations seem to be totally broken. They need to be performed in a particular order, which worked in the first machine I set the environment in a couple months ago, but now there are inconsistencies (although I am pretty sure no new migrations were generated).
So the only solution I can think of is exporting the database from the old machine, where it is working, to the new one. Would that work?
This would not solve the broken migrations issue, but at least I can work on the code till there's a proper soltuion.
Answering this question:
So the only solution I can think of is exporting the database from the old machine, where it is working, to the new one. Would that work?
Yes, this can work if you are sure that your database is in sync with your models. It is actually the way to go, if you want to be best prepared of updating your production environment.
get a dump from the current production machine
create a new database and load the dump
check whether there are differences between the models and the migration history (this is more reliable with the new Django migrations, South was an external tool and had not all of the possibilities) (e.g. ./manage.py showmigrations (1.10), ./manage.py migrate --list (1.7-1.9 and South)
If you are confident that no migrations have to be run but the listing shows differences then do: ./manage.py migrate --fake
Note, in newer versions you can do ./manage.py migrate and it will report that everything is in order if the models and the migrations are in sync. This can be a sanity check before you deploy onto production.
I want to revert my last migration (0157) by running its Migration.backwards() method. Since I am reverting the migration in production server I want to run it automatically during code deployment. Deployment script executes these steps:
Pull code changes
Run migrations: manage.py migrate <app>
Refresh Apache to use newest code: touch django.wsgi
If I could, I would create new migration file which would tell South to backward migrate to 0156:
migrations/0158_backward__migrate_to_0156.py
This commited migration would be deployed to production and executed during manage.py migrate <app> command. In this case I wouldn't have to execute backward migration by hand, like suggested in these answers.
Lets say, I have created two data migrations, first for user's Payment, second for User model. I have implemented backwards() methods for both migrations in case I'd have to revert these data migrations. I've deployed these two migrations to production. And suddenly find out that Payment migration contains an error. I want to revert my two last data migrations as fast as possible. What is the fastest safe way to do it?
Since I am reverting the migration in production server I want to run
it automatically during code deployment.
IMHO the safest path is
run manage.py migrate <app> (i.e. apply all existing migrations, i.e. up to 0156)
undo the changes in your model
run manage.py schemamigration <app> --auto
This will create a new migration 0157 that effectively reverts the previous migration 0156. Then simply apply the new migration by running manage.py migrate <app> again. As I understand, your code deployment will just do that.
Apparently the codeline has migrations up to #157 and now the developer decided that the last one was not a good idea after all. So the plan is to go back to #156.
Two scenarios:
(a) migration #157 was not released or deployed anywhere yet.
Simply revert the last change from models.py and delete migration #157.py from the source archive. Any deployment will take the system to level 156; "157 was never there".
(b) there have been deployments of the latest software with migration #157.
In this case the previous strategy will obviously not work. So you need to create a migration #158 to undo #157. Revert the change in models.py and run
django manage.py migrate <app> 0157
django manage.py schemamigration <app> --auto
This will auto-generate a new migration #158, which will contain the inverse schema migration compared to #157.
If schemamigration is giving trouble because of django Model validation (something that can happen if you have custom validators which check stuff outside the ORM box), I suggest the following workaround:
<django project>/<app>/management/commands/checkmigrations.py
from south.management.commands import schemamigration
class Command(schemamigration.Command):
requires_model_validation = False
help = "schemamigration without model validation"
This command becomes available in manage.py:
django manage.py checkmigrations <app> --auto
There's no silver bullet here. The simplest solution I can think of would be to - in your dev env of course - manually migrate back to 0156, manually update your migration's history table (sorry I can't remember the table's name now) to fool south in thinking you're still #0158, then run schemamigration again. Not garanteed to work but might be worth trying.