Django on GAE - How to automatically 'migrate' on deploy? - python

Django v1.11
Postgresql v9.6
Currently, I use 2 Google CloudSQL databases, one for development and one for production. Whenever I make changes to my models, I run python manage.py migrate to update the tables in the development database. This migration does not affect the production database, however.
Now, whenever I git push changes to my Django project, TravisCI automatically runs tests and deploys the code to Google App Engine. Currently, it runs on GAE flexible environment (so I can use Python 3.5)
What I want to have is for Travis or GAE to automatically run python manage.py migrate on the production database before runserver. However, I can't figure out how to run custom commands during a deploy.
I've tried looking around GAE and Travis documentation and adding scripts to .travis.yml and app.yaml, but to no avail.
As of now, anytime there is a model change, I have to migrate the production database locally in a very hacky way. Ideally, GAE will migrate at the beginning of every deploy.

Not sure if you have seen this:
Travis CI Script Deployment
A reference from a similar issue:
How can I run a script as part of a Travis CI build?
Also, consider a database migration tool embedded with your source code, Postgresql is supported (something similar to FlywayDB migration ):
Yoyo database migrations¶

Related

Delete db.sqlite via terminal (Django)

I'm currently deploying a project to Heroku, however it returns me the following error: ValueError: Related model 'store.user' cannot be resolved.
But there is a way to avoid such error locally. You simply do:
py manage.py migrate store
And then
py manage.py migrate
In other words, if I migrate separetely, I won't face such error. However, Heroku migrates all together, then the deploy fails because of this error.
If I proceed locally as Heroku does, i.e. run
py manage.py migrate
I can effortlessly click on and delete the db.sqlite3 file and makemigrations and migrate separetely. Then the issue would be resolved. However, that's not possible when deploying to Heroku. Thus, how can I delete this file only via terminal? I have been searching, but people only say you click on the file and the delete it, which for me it is not possible.
Thanks
However, Heroku migrates all together, then the deploy fails because of this error
Heroku does nothing of the sort.
Migrations only run on Heroku when you tell them to, either by running something like
heroku run python manage.py migrate
or because you have declared a release process that should run when you deploy a new version, e.g.:
web: gunicorn app.wsgi
release: python manage.py migrate
In both cases you are in charge of what that command is. If you don't want to run python manage.py migrate every time you deploy, remove the release process from your Procfile.
if I migrate separately, I won't face such error
This is concerning.
There could be several causes for this. One common issue is missing dependencies in your migration. If you write a migration in app1 that depends on certain models and tables from app2 existing, you need to add a dependency to the relevant migration in app1, e.g. something like this example from the documentation:
class Migration(migrations.Migration):
dependencies = [
('app1', '0001_initial'),
# added dependency to enable using models from app2 in move_m1
('app2', '0004_foobar'),
]
operations = [
migrations.RunPython(move_m1),
]
If migrations are written properly, such that they accurately reflect the code in each commit, build off each other, have dependencies declared, etc. you should always be able to safely apply them to your database.
Finally, you ask about deleting db.sqlite.
Heroku's filesystem is ephemeral. It gets baked into your application slug at build time, and any changes you make to it get reset every time your dyno restarts. This happens frequently (at least once per day).
This means you can't effectively delete your database file. But it also means you can't save data and expect it to be there when you look it up later! SQLite isn't a good fit for Heroku.
A client-server database like PostgreSQL is a much better choice. Heroku offers its own Postgres service with a free tier.
If you switch, I strongly urge you to switch in development as well. Django's ORM makes it relatively easy to change, but database engines aren't drop-in replacements for each other. I've seen real-world examples where developing on SQLite and deploying to Postgres or MySQL causes things to fail in production that worked in development.

How to run makemigrations for dockerized django app

I have been trying to setup a django app using this guide:
https://testdriven.io/blog/dockerizing-django-with-postgres-gunicorn-and-nginx/
And it was a great resource to get everything up and running. However i have problem now when it comes to migrations. Say i add a model, add data into my db, and later i change the model. Now i can run makemigrations using the exec method on the container, but when i shut down my migrations are not stored in my local files i save to my git project, instead they are lost as i spin down the container.
Does anyone know how to solve this, how do you makemigrations is such a setup where you run two dockerized django/postgres dev/prod environments?

Django Migration Process for Elasticbeanstalk / Multiple Databases

I am developing a small web application using Django and Elasticbeanstalk.
I created a EB application with two environments (staging and production), created a RDS instance and assigned it to my EB environments.
For development I use a local database, because deploying to AWS takes quite some time.
However, I am having troubles with the migrations. Because I develop and test locally every couple of minutes, I tend to have different migrations locally and on the two environments.
So once I deploy the current version of the app to a certain environment, the "manage.py migrate" fails most of the times because tables already exist or do not exist even though they should (because another environment already created the tables).
So I was wondering how to handle the migration process when using multiple environments for development, staging and production with some common and some exclusive database instances that might not reflect the same structure all the time?
Should I exclude the migration files from the code repository and the eb deployment and run makemigrations & migrate after every deployment? Should I not run migrations automatically using the .ebextensions and apply all the migrations manually through one of the instances?
What's the recommended way of using the same Django application with different database instances on different environments?
Seems that you might have deleted the table or migrations at some point of time.
When you run makemigrations, django create migratins and when you run migrate, it creates database whichever is specified in settings file.
One thing is if you keep on creating migrations and do not run it in a particular database, it will be absolutely fine. Whenever you switch to databsse and run migrations, it will handle it as every database will store the point upto which migrations have been run until now in django-migrations table and will start running next migrations only.
To solve your problem, you can delete all databases and migration files and start afresh as you are perhaps testing right now. Things will go fine untill you delete a migration or a database in any of the server.
If you have precious data, you should get into migration files and tables to analyse and manage things.

Heroku and Django Running Server

I'm following this tutorial: http://tutorial.djangogirls.org/en/domain/README.html
When I do python manage.py runserver it works fine. It also works when I run
heroku ps:scale web=1
then
heroku open
with python manage.py runserver it shows my blog posts and everything that I had added. But when I run the server with heroku open there are no posts as if the database is missing or something.
Why is that? Why do the two commands launch the same web page but with different posts/different databases?
Which brings me to my follow up question: how do I know when I need to run migrate or makemigrations again for the server? Would doing so fix the problem? And what exactly do those commands do/why are they necessary?
Thanks
EDIT:
Bonus question: Why do my posts display in descending order of time? The new posts are on the bottom of the page rather than the top. How can I change this?
There is a difference between your local development and your deployed project.
I assume you created your posts local. So they are saved local into your database. Local you use a filebased database defined in the settings 'django.db.backends.sqlite3' that means when you run manage.py syncdb the file is created with all tables inside. When you deploy your code to heroku the code is pushed to the server and runs from this place. This can be everywhere, so it can't connect to your local databasefile. For your project you has to setup a database on heroku as well. I recommend to read this article. When you want to transfer your data you can create a database dump local and load all data to you heroku database. Described here and here.
Another point, you don't run the server with heroku open or by fire python manage.py runserver. The heroku server is automatically started when your git push heroku master is done. It use the configs from your Procfile.
When you want to migrate your heroku database you has to run heroku run python manage.py migrate <app_name> than the migration is done remotely on the heroku server. You have to run this command everytime you changed a model and added a migrationfile with python manage.py makemigration <app_name>. When you did this you has to migrate you database local and remote. That means that you change the database structure to match your models. Remember the models are only a abstraction(orm) of your database.
I don't know your project but the order sees legit. Try to imagine this as rows. first row comes first. So the last entered row is on the bottom. You can change the order of the queryset with something like.order_by('-id'). So you get all entries in reverse order.

Django project using wrong (old) database settings

recently I started a small Django project that I developed on a local machine using a SQLite3 database and the integrated development server. I now copied the whole project to a server running Debian.
Everything worked well as long as I kept using the SQLite3 database. Now I wanted to switch to a local MySQL database, so I changed the settings.py file in my project's root folder, created the database and added a user. I then ran syncdb and it created the needed tables without any problems.
Now I wanted to use the app, but I keep getting errors, because Django can not find the tables - neither the 'standard tables' like django_sessions nor my own tables - even though they are there (I checked the database).
The one thing that seems awkward about the DatabaseError pages I get is the following line:
Exception Location: /usr/local/lib/python2.6/dist-packages/django/db/backends/sqlite3/base.py in execute, line 234
It seems like Django is still using the SQLite3 backend even though I set it to use the MySQL backend.
The other thing that nearly makes me freak out is this: I deleted the data.sqlite file in my app's root folder using rm. But when I use my app, the file is being recreated!
Can anyone tell me where I went wrong?
When running in production with mod_wsgi or mod_python, new code isn't incorporated until you reload/restart the webserver.
If you are using apache with mod_wsgi, you can also touch the wsgi.py file to restart the python code only.
If running on apache run "sudo /etc/init.d/apache2 reload" everytime you make a change.

Categories

Resources