I'm currently deploying a project to Heroku, however it returns me the following error: ValueError: Related model 'store.user' cannot be resolved.
But there is a way to avoid such error locally. You simply do:
py manage.py migrate store
And then
py manage.py migrate
In other words, if I migrate separetely, I won't face such error. However, Heroku migrates all together, then the deploy fails because of this error.
If I proceed locally as Heroku does, i.e. run
py manage.py migrate
I can effortlessly click on and delete the db.sqlite3 file and makemigrations and migrate separetely. Then the issue would be resolved. However, that's not possible when deploying to Heroku. Thus, how can I delete this file only via terminal? I have been searching, but people only say you click on the file and the delete it, which for me it is not possible.
Thanks
However, Heroku migrates all together, then the deploy fails because of this error
Heroku does nothing of the sort.
Migrations only run on Heroku when you tell them to, either by running something like
heroku run python manage.py migrate
or because you have declared a release process that should run when you deploy a new version, e.g.:
web: gunicorn app.wsgi
release: python manage.py migrate
In both cases you are in charge of what that command is. If you don't want to run python manage.py migrate every time you deploy, remove the release process from your Procfile.
if I migrate separately, I won't face such error
This is concerning.
There could be several causes for this. One common issue is missing dependencies in your migration. If you write a migration in app1 that depends on certain models and tables from app2 existing, you need to add a dependency to the relevant migration in app1, e.g. something like this example from the documentation:
class Migration(migrations.Migration):
dependencies = [
('app1', '0001_initial'),
# added dependency to enable using models from app2 in move_m1
('app2', '0004_foobar'),
]
operations = [
migrations.RunPython(move_m1),
]
If migrations are written properly, such that they accurately reflect the code in each commit, build off each other, have dependencies declared, etc. you should always be able to safely apply them to your database.
Finally, you ask about deleting db.sqlite.
Heroku's filesystem is ephemeral. It gets baked into your application slug at build time, and any changes you make to it get reset every time your dyno restarts. This happens frequently (at least once per day).
This means you can't effectively delete your database file. But it also means you can't save data and expect it to be there when you look it up later! SQLite isn't a good fit for Heroku.
A client-server database like PostgreSQL is a much better choice. Heroku offers its own Postgres service with a free tier.
If you switch, I strongly urge you to switch in development as well. Django's ORM makes it relatively easy to change, but database engines aren't drop-in replacements for each other. I've seen real-world examples where developing on SQLite and deploying to Postgres or MySQL causes things to fail in production that worked in development.
I am trying to setup a Django app locally in a new machine but migrations seem to be totally broken. They need to be performed in a particular order, which worked in the first machine I set the environment in a couple months ago, but now there are inconsistencies (although I am pretty sure no new migrations were generated).
So the only solution I can think of is exporting the database from the old machine, where it is working, to the new one. Would that work?
This would not solve the broken migrations issue, but at least I can work on the code till there's a proper soltuion.
Answering this question:
So the only solution I can think of is exporting the database from the old machine, where it is working, to the new one. Would that work?
Yes, this can work if you are sure that your database is in sync with your models. It is actually the way to go, if you want to be best prepared of updating your production environment.
get a dump from the current production machine
create a new database and load the dump
check whether there are differences between the models and the migration history (this is more reliable with the new Django migrations, South was an external tool and had not all of the possibilities) (e.g. ./manage.py showmigrations (1.10), ./manage.py migrate --list (1.7-1.9 and South)
If you are confident that no migrations have to be run but the listing shows differences then do: ./manage.py migrate --fake
Note, in newer versions you can do ./manage.py migrate and it will report that everything is in order if the models and the migrations are in sync. This can be a sanity check before you deploy onto production.
I am developing a small web application using Django and Elasticbeanstalk.
I created a EB application with two environments (staging and production), created a RDS instance and assigned it to my EB environments.
For development I use a local database, because deploying to AWS takes quite some time.
However, I am having troubles with the migrations. Because I develop and test locally every couple of minutes, I tend to have different migrations locally and on the two environments.
So once I deploy the current version of the app to a certain environment, the "manage.py migrate" fails most of the times because tables already exist or do not exist even though they should (because another environment already created the tables).
So I was wondering how to handle the migration process when using multiple environments for development, staging and production with some common and some exclusive database instances that might not reflect the same structure all the time?
Should I exclude the migration files from the code repository and the eb deployment and run makemigrations & migrate after every deployment? Should I not run migrations automatically using the .ebextensions and apply all the migrations manually through one of the instances?
What's the recommended way of using the same Django application with different database instances on different environments?
Seems that you might have deleted the table or migrations at some point of time.
When you run makemigrations, django create migratins and when you run migrate, it creates database whichever is specified in settings file.
One thing is if you keep on creating migrations and do not run it in a particular database, it will be absolutely fine. Whenever you switch to databsse and run migrations, it will handle it as every database will store the point upto which migrations have been run until now in django-migrations table and will start running next migrations only.
To solve your problem, you can delete all databases and migration files and start afresh as you are perhaps testing right now. Things will go fine untill you delete a migration or a database in any of the server.
If you have precious data, you should get into migration files and tables to analyse and manage things.
Should I be adding the Django migration files in the .gitignore file?
I've recently been getting a lot of git issues due to migration conflicts and was wondering if I should be marking migration files as ignore.
If so, how would I go about adding all of the migrations that I have in my apps, and adding them to the .gitignore file?
Quoting from the Django migrations documentation:
The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
If you follow this process, you shouldn't be getting any merge conflicts in the migration files.
When merging version control branches, you still may encounter a situation where you have multiple migrations based on the same parent migration, e.g. if to different developers introduced a migration concurrently. One way of resolving this situation is to introduce a merge_migration. Often this can be done automatically with the command
./manage.py makemigrations --merge
which will introduce a new migration that depends on all current head migrations. Of course this only works when there is no conflict between the head migrations, in which case you will have to resolve the problem manually.
Given that some people here suggested that you shouldn't commit your migrations to version control, I'd like to expand on the reasons why you actually should do so.
First, you need a record of the migrations applied to your production systems. If you deploy changes to production and want to migrate the database, you need a description of the current state. You can create a separate backup of the migrations applied to each production database, but this seems unnecessarily cumbersome.
Second, migrations often contain custom, handwritten code. It's not always possible to automatically generate them with ./manage.py makemigrations.
Third, migrations should be included in code review. They are significant changes to your production system, and there are lots of things that can go wrong with them.
So in short, if you care about your production data, please check your migrations into version control.
You can follow the below process.
You can run makemigrations locally and this creates the migration file. Commit this new migration file to repo.
In my opinion you should not run makemigrations in production at all. You can run migrate in production and you will see the migrations are applied from the migration file that you committed from local. This way you can avoid all conflicts.
IN LOCAL ENV, to create the migration files,
python manage.py makemigrations
python manage.py migrate
Now commit these newly created files, something like below.
git add app/migrations/...
git commit -m 'add migration files' app/migrations/...
IN PRODUCTION ENV, run only the below command.
python manage.py migrate
Quote from the 2022 docs, Django 4.0. (two separate commands = makemigrations and migrate)
The reason that there are separate commands to make and apply
migrations is because you’ll commit migrations to your version control
system and ship them with your app; they not only make your
development easier, they’re also useable by other developers and in
production.
https://docs.djangoproject.com/en/4.0/intro/tutorial02/
TL;DR: commit migrations, resolve migration conflicts, adjust your git workflow.
Feels like you'd need to adjust your git workflow, instead of ignoring conflicts.
Ideally, every new feature is developed in a different branch, and merged back with a pull request.
PRs cannot be merged if there's a conflict, therefore who needs to merge his feature needs to resolve the conflict, migrations included. This might need coordination between different teams.
It is important though to commit migration files! If a conflict arises, Django might even help you solve those conflicts ;)
I can't imagine why you would be getting conflicts, unless you're editing the migrations somehow? That usually ends badly - if someone misses some intermediate commits then they won't be upgrading from the correct version, and their copy of the database will be corrupted.
The process that I follow is pretty simple - whenever you change the models for an app, you also commit a migration, and then that migration doesn't change - if you need something different in the model, then you change the model and commit a new migration alongside your changes.
In greenfield projects, you can often delete the migrations and start over from scratch with a 0001_ migration when you release, but if you have production code, then you can't (though you can squash migrations down into one).
The solution usually used, is that, before anything is merged into master, the developer must pull any remote changes. If there's a conflict in migration versions, he should rename his local migration (the remote one has been run by other devs, and, potentially, in production), to N+1.
During development it might be okay to just not-commit migrations (don't add an ignore though, just don't add them). But once you've gone into production, you'll need them in order to keep the schema in sync with model changes.
You then need to edit the file, and change the dependencies to the latest remote version.
This works for Django migrations, as well as other similar apps (sqlalchemy+alembic, RoR, etc).
Gitignore the migrations, if You have separate DBs for Development, Staging and Production environment. For dev. purposes You can use local sqlite DB and play with migrations locally.
I would recommend You to create four additional branches:
Master - Clean fresh code without migrations. Nobody is connected to this branch. Used for code reviews only
Development - daily development. Push/pull accepted. Each developer is working on sqlite DB
Cloud_DEV_env - remote cloud/server DEV environment. Pull only. Keep migrations locally on machine, which is used for the code deployment and remote migrations of Dev database
Cloud_STAG_env - remote cloud/server STAG environment. Pull only. Keep migrations locally on machine, which is used for the code deployment and remote migrations of Stag database
Cloud_PROD_env - remote cloud/server DEV environment. Pull only. Keep migrations locally on machine, which is used for the code deployment and remote migrations of Prod database
Notes:
2, 3, 4 - migrations can be kept in repos but there should be strict rules of pull requests merging, so we decided to find a person, responsible for deployments, so the only guy who has all the migration files - our deploy-er. He keeps the remote DB migrations each time we have any changes in Models.
You should think of migrations as a version control system for your database schema. makemigrations is responsible for packaging up your model changes into individual migration files - analogous to commits - and migrate is responsible for applying those to your database.
The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
golden rule : Make once on dev and migrate on all
Having a bunch of migration files in git is messy. There is only one file in migration folder that you should not ignore. That file is init.py file, If you ignore it, python will no longer look for submodules inside the directory, so any attempts to import the modules will fail. So the question should be how to ignore all migration files but init.py?
The solution is:
Add '0*.py' to .gitignore files and it does the job perfectly.
Hope this helps someone.
Committing your migrations is just a recipe for disaster. Because the migrations are somewhat or a chain that can be traced back, if you have dependences from a former migration e.g a pip module which you used at some point in your project lifecycle and then stopped using. You might find bread crumbs of such dependences in your migrations thread and you have to manually remove these imports from the migrations file.
Verdict, except you are a god tier Django dev, probably avoid adding migrations to your commits.
Short answer
I propose excluding migrations in the repo. After code merge, just run ./manage.py makemigrations and you are all set.
Long answer
I don't think you should put migrations files into repo. It will spoil the migration states in other person's dev environment and other prod and stage environment. (refer to Sugar Tang's comment for examples).
In my point of view, the purpose of Django migrations is to find gaps between previous model states and new model states, and then serialise the gap. If your model changes after code merge, you can simple do makemigrations to find out the gap. Why do you want to manually and carefully merge other migrations when you can achieve the same automatically and bug free? Django documentation says,
They*(migrations)*’re designed to be mostly automatic
; please keep it that way. To merge migrations manually, you have to fully understand what others have changed and any dependence of the changes. That's a lot of overhead and error prone. So tracking models file is sufficient.
It is a good topic on the workflow. I am open to other options.
I have installed south and make some migrations. Now there's a 'migrations' directory in the folder app. My question is: when I am refactoring models, which entries in the migration directory files I must apply the changes? I think some entries are related directly with the database schema, and others with the code itself. I couldn't fina an answer to this in the south docs.
Make the changes to your models then run python manage.py schemamigration yourapp --auto. This will create the migrations for you (you'll see a new file in your migrations directory every time you do this process).
Sometimes you really need to edit a migration manually, but you should try and avoid it. Particularly if you have already run the migration (the south app keeps a record of which migrations have been run so it knows the state of your database).
South is designed to support moving between different versions of your code without breaking your database. Each migration file in the migrations directory represents a snapshot of your code (specifically a snapshot of your models.py). You migrate from version to version by running python manage.py migrate yourapp version_no