How do I transfer test database to product database - python

I'm working on an Django project, and building and testing with a database on GCP. Its full of test data and kind of a mess.
Now I want to release the app with a new and fresh another database.
How do I migrate to the new database? with all those migrations/ folder?
I don't want to delete the folder cause the development might continue.
Data do not need to be preserved. It's test data only.
Django version is 2.2;
Python 3.7
Thank you.
========= update
After changing the settings.py, python manage.py makemigrations says no changes detected.
Then I did python manage.py migrate, and now it complains about relation does not exist.
=============== update2
The problem seems to be that, I had a table name Customer, and I changed it to 'Client'. Now it's complaining about "psycopg2.errors.UndefinedTable: relation "app_customer" does not exist".
How can I fix it, maybe without deleting all files in migrations/?
================ update final
After eliminating all possibilities, I have found out that the "new" database is not new at all. I migrated on that database some months ago.
Now I created a fresh new one and migrate worked like a charm.
Again, thank you all for your suggestions.

The migration folder is your friend. No need to delete it. You can flatten migrations if you feel the migration folder is getting too large with many migration files, but you don't need to.
If you plan to use a new database in GCP, just change the settings.py file, which is typically located in the project_name/project_name/ folder. Locate the DATABASES section to reflect the new database credentials.
Once your app is pointing at the new database, run python manage.py migrate. This will build the database schema with the necessary tables to start populating the new database.

You don't need to delete any migrations/ folder, you can use the same migration files in product database, if you don't need the test data just delete the database i.e. db.sqlite3 (I guess you are using default database).
Note: Migration files don't effect your database structure(if you delete and do makemigrations or use the existing one it remains the same)

Related

After delete database ,my Django does't work

I have some problem with my database, so I deleted db.sqlite3 and Migrations manually.
than I recreate the database by using manage.py makemigrations <appname> manage. py migrate <appname>
Everything looks normal,but when I get into localhost:8000, It is a blank page without anything (even if I never change the templates).
To my confused, I can get into the admin page.
Are there any mistakes in this process?what happened to my django app?
Now what you can do is :
1) Take backup of your code
2) Create another project with other name or same name, it will have it's own sqlite db
3) Place your code in that project and then run migrations, it will recreate your db schema.
And it should work.

How to force migrations to a DB if some tables already exist in Django?

I have a Python/Django proyect. Due to some rolls back, and other mixed stuff we ended up in a kind of odd scenario.
The current scenario is like this:
DB has the correct tables
DB can't be rolled back or dropped
Code is up to date
Migrations folder is behind the DB by one or two migrations. (These migrations were applied from somewhere else and that "somewhere else" doesn't exist anymore)
I add and alter some models
I run makemigrations
New migrations are created, but it's a mix of new tables and some tables that already exist in the DB.
If I run migrate it will complain that some of the tables that I'm trying to create already exist.
What I need:
To be able to run the migrations and kind of "ignore" the existing tables and apply the new ones. Or any alternative way to achieve this.
Is that possible?
When you apply a migration, Django inserts a row in a table called django_migrations. That's the only way Django knows which migrations have been applied already and which have not. So the rows in that table have to match the files in your migrations directory. If you've lost the migration files after they were applied, or done anything else to get things out of sync, you'll have problems.. because the migration numbers in your database refer to different migration files than the ones in your project.
So before you do anything else, you need to bring things back into sync by deleting the django_migrations table rows for any migration files that you've lost somehow and can't get back. The table should contain rows for only those migrations that you do have and that were actually applied to the database correctly.
Now you need to deal with any changes in your database that Django Migrations doesn't know about.. and for that there are a few options:
If things worked out such that the database changes that were already applied to the database are in different migration files than the ones that weren't, then you can fix it by running your migrations one at a time using the --fake option on any changes that are in reality already in the database. The fake option just writes the row to the django_migrations table marking the migration as done. Only do this if the database does in fact already have all the changes contained in that migration file.
And those migration files that contain only changes which have not been applied to the database, run without the --fake option and Django will apply them. eg:
# database already has it
manage.py migrate myapp 0003 --fake
# need it
manage.py migrate myapp 0004
# database already has it
manage.py migrate myapp 0005 --fake
If you have migration files where some but not all of the changes have been applied, then you have a bigger problem. In that case, there are several ways to go about it (choose ONLY ONE):
Edit the migration files to put changes that have already been applied (whether Django did it or you did it manually does not matter) into lower number migrations, and put everything you need done into higher numbered files. Now you can --fake the lower number ones, and run the higher numbered ones as normal. Let's say you have 10 changes you made to your models, and 5 of those changes are actually in the database already, but Django doesn't know about them.. so when you run makemigrations, a new migration is created with all 10 changes. This will normally fail because the database server can't for example add a column that already exists. Move these already-applied changes out of your new migration file, into the previous (already applied) migration file. Django will then assume that these were applied with the previous migration and will not try to apply them again. You can then migrate as normal and the new changes will be applied.
If you don't want to touch your older migration file, a cleaner way to do this is to first run makemigrations --empty appname to create an empty migration file. Then run makemigrations which will create another migration with all the changes that Django thinks need to be done. Move the already done migrations from that file into the empty migration you created.. then --fake that one. This will put Django's understanding of what the database looks like will be in sync with reality and you can then migrate as normal, applying the changes in the last migration file.
Get rid of any new migrations you just created using makemigrations. Now, comment out or put back anything in your models that has not been applied to the database, leaving your code matching what's actually in the database. Now you can do makemigrations and migrate appname --fake and you will get things back in sync. Then uncomment your new code and run 'makemigrations' then migrate as normal and the changes will be applied. If the changes are small (for example, adding a few fields), sometimes this is easiest. If the changes are large, it isn't....
You can go ahead and (carefully) make the database changes yourself, bringing the database up to date. Now just run migrate --fake and if you didn't mess up then everything will be ok. Again, this is easy for smaller changes, not as easy for complicated ones.
You can run manage.py sqlmigrate > mychanges.sql. This generates mychanges.sql containing all the SQL Django WOULD have executed against the database. Now edit that file to remove any changes that have already been applied, leaving what needs to be done. Execute that SQL using pgadmin or psql (you're using postgresql I hope). Now the changes have all been made.. so you can run manage.py migrate --fake, this will bring Django into sync with reality and you should be all set. If your SQL skills are sufficient, this is probably the most straightforward solution.
I should add two warnings:
First, if you apply a later migration, eg 0003_foobar.py, and then things don't work out and you decide to try going back and apply 0002_bazbuz.py, then Django will TAKE STUFF OUT OF YOUR DATABASE. For example a column you might have added in 0003 will be dropped along with its data. Since you say you can't lose data, be very careful about going back.
Second, do not rush into running --fake migrations. Make sure that the entire migration you are about to fake is actually in the database already. Else it gets very confusing. If you do regret faking migrations and don't want to roll back, you can erase django's knowledge of the faked migration by deleting that row from the django_migrations table. It is ok to do this.. if you understand what you are doing. If you know that the migration really was not applied, then it's ok.
This blog post really nails it. https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html
Let me summarize the steps in his scenario 2 (you have a production database and want to change schema/models in one or more apps). In my case, I had two apps, queue and routingslip, that had model modifications that I needed to apply to a production system. Key was I already had the database, so this is where --fake-initial comes into play.
Here are the steps I followed. As always, backup everything before starting. I do work in a VM so I just took a snapshot before going forward.
1) Remove the migration history for each app.
python manage.py migrate --fake queue zero
python manage.py migrate --fake routingslip zero
2) Blow away any migration files in the entire project within which the app(s) reside.
find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete
3) Make migrations
python manage.py makemigrations
4) Apply the migrations, faking initial because the database already exists and we just want the changes:
python manage.py migrate --fake-initial
Worked great for me.
If you don't have any migration files or you lost the previous file and want to migrate new changes, then you need to follow the following steps carefully:
# To create a new migration file before changing the models.
cmd: python manage.py makemigrations app_name
# Fake migrate
cmd: python manage.py migrate app_name 0005 --fake #[0005 is the migration file number created just now. It'll seem like 0005_add_address or something like this.]
# To create a new migration file after changing the models.
cmd: python manage.py makemigrations app_name
# database already has it
cmd: python manage.py migrate app_name 0006 #[0006 is the migration file number created just now.]

Can I delete the django migration files inside migrations directory

I personally like django for its MVC ideals. But while i am running Django migrations in version 1.7 each and every migrations i do in it is stored inside the migrations directory. If i delete those file it is throwing an error while migration.
I Tested like this. I created a new Django project and initiated a git repo . I ran some 3-4 migrations in Django which resulted in
3-4 migration files under the migrations directory. I tried deleting the very older migration files i.e (1st and 2nd migration files) and tried to run
python manage.py makemigrations
which does cause some error like "migration files not found". Later i did a git stash which restored the deleted files. Now i tried to run the same command again and it was working fine.
What my question is if a person runs some 50 changes in db during development all the migration files are stored in migrations directory. Is it possible to delete those files and do changes to db again without any interruption?
The answer is "it depends".
If you are working against a production DB, or some DB that can't periodically blow away for whatever reason, then you absolutely want to keep around the migration files that you've applied to your DB. They should be checked into source control with the rest of your code.
Now, for a situation like yours, the easiest way to discard your 50 migrations would be to just blow away the db (and it's 50 migrations) and start from scratch given your current models. It's oftentimes a good idea to do this periodically as you evolve your models during development.
Its ok to blow away your models when you blow away your DB because syncdb will build a blank db using your current models. It'll then optionally populate the db using any initial fixtures. Conceptually, there is no longer anything that you've migrated from at such a point, so you don't need to keep around your old migrations for your old db. They are no longer relevant.
It's not usually good to delete migration files that have been applied to your DB unless you are either 1) blowing away the DB altogether, or 2) reverting the migrations first.
You might also appreciate knowing that when you apply migrations to a db it's also recording those migrations in a special table in the db itself. That's why things go haywire when you just delete the migration files. They have to stay in sync with the migration table
The answer is "Do not delete migration files".
To understand why we shouldn't delete migration files, you need to understand how migration works in frameworks.
Migration files are the history of your database. One migration file is created based on the migration files created in the past. Deleting migration files means losing your history. This historical info is recorded in the django_migrations table in your database. if you delete migration files, you will get dependency errors. So Don't try to lose your history by deleting your migration files.
If you want to keep your DB, but decrease the number of migration files, one option is squashing the migrations into one (or few, if complex dependencies) migration.
From the official documentation:
You are encouraged to make migrations freely and not worry about how many you have; the migration code is optimized to deal with hundreds at a time without much slowdown. However, eventually you will want to move back from having several hundred migrations to just a few, and that’s where squashing comes in.
Before squashing, you should be aware that "model interdependencies in Django can get very complex, and squashing may result in migrations that do not run", and therefore manual work may be needed.
For detailed information about how to make the squashing, refer to the docs: https://docs.djangoproject.com/en/dev/topics/migrations/#squashing-migrations
If models match database it is safe to delete migration files.
Currently, with Django 3 I can safely remove the migrations directory, then run python manage.py makemigrations myapp and python manage.py migrate. After that I have 0001_initial.py migration file and my production database is intact. This works when models already match database.
In my opinion, it would be a bad idea. You can always roll back migrations if you make a mistake. Also, as migrations grow too large, you can also "squash" them. I learned about this from an article written by DoorDash.
You are encouraged to make migrations freely and not worry about how many you have; the migration code is optimized to deal with hundreds at a time without much slowdown. However, eventually you will want to move back from having several hundred migrations to just a few, and that’s where squashing comes in.
Squashing migrations: https://docs.djangoproject.com/en/3.2/topics/migrations/#squashing-migrations
DoorDash article: https://doordash.engineering/2017/05/15/tips-for-building-high-quality-django-apps-at-scale/
It probably isn't a good idea (apparently), but if you are going to do it...
do not remove the __init__.py files.
In *nix:
cd [your project directory]
find . -path "*/migrations/[0-9][0-9][0-9][0-9]_*.py" -delete
find . -path "*/migrations/*.pyc" -delete

syncdb is not creating tables again?

I am stuck with this issue: I had some migration problems and I tried many times and on the way, I deleted migrations and tried again and even deleted one table in db. there is no data in db, so I don't have to fear. But now if I try syncdb it is not creating the table I deleted manually.
Honestly, I get really stuck every time with this such kind of migration issues.
What should I do to create the tables again?
To be sure you are deleting south history and to use syncdb as if no south exists, you have to delete migrations in two places:
In the database: remove tables created by south
Migration files: remove folders used by south. They are like "app_name/migrations"
If you don't need any data in your database (because it is testing for example), instead of remove only south tables, y recommend to do a complete reset. The commando to do that is
python manage.py reset_db --router=default
Hope it helps!
Some extra help
Generally, when you use syncdb and south is in your INSTALLED_APPS you get a message saying that some apps won't be synced and that you have to use migrations. Try to do a syncdb and look at the apps that can't be synced. If there are some of them, remove the "migrations" folder inside that apps and make a syncdb again. The app should't appear again in the list that can't be synced.
are you using south?
If you are, there is a migration history database that exists.
Make sure to delete the row mentionnaing the migration you want to run again.
Try to use the comand reset in the app, something like that:
python manage.py reset my_app
Try renaming the migration file and running python manage.py syncdb.

How to handle refactor with south (django)?

I have installed south and make some migrations. Now there's a 'migrations' directory in the folder app. My question is: when I am refactoring models, which entries in the migration directory files I must apply the changes? I think some entries are related directly with the database schema, and others with the code itself. I couldn't fina an answer to this in the south docs.
Make the changes to your models then run python manage.py schemamigration yourapp --auto. This will create the migrations for you (you'll see a new file in your migrations directory every time you do this process).
Sometimes you really need to edit a migration manually, but you should try and avoid it. Particularly if you have already run the migration (the south app keeps a record of which migrations have been run so it knows the state of your database).
South is designed to support moving between different versions of your code without breaking your database. Each migration file in the migrations directory represents a snapshot of your code (specifically a snapshot of your models.py). You migrate from version to version by running python manage.py migrate yourapp version_no

Categories

Resources