Develop Reference

"transaction is active" error while migrating django model

I renamed some field in my model, and ran
python manage.py makemigration # successful
python manage.py migrate
On the second command I get
NotSupportedError: Renaming the 'my_model'.''my_column' while in a transaction is not supported on SQLite because it would break referential integrity. Try adding atomic = False to the Migration class
However, I don't see which transaction it means. There is no python or sqlite process that is running at the time I get that error. Is some lock left in sqlite or django file? And how do I fix that??

Go to the app folder in which you have renamed some field in the model.
when you have ran this command
python manage.py makemigration.
This in the app folder inside migration folder would have made a migration file (last file, eg: 000_initial).
Open that file inside that Migration Class would be written in the beginning of that class add this.
atomic = False
It will look something like this
class Migration(migrations.Migration):
atomic = False
That will help you run command error free:
python manage.py migrate
For more Reference check: https://docs.djangoproject.com/en/2.1/howto/writing-migrations/

Django migrations old database, forgot one model before

I have existing database/production system pre-migration support for Django. We did start to use Djangos migration 2 years back it turns out I did forget to install migrations for one model that now causes problems.
2 years back I had the following models; Location, Tool and a third Log that points to instances of Location and Tool.
The 0001_inital.py for Log model has a dependency to Tool for 0001_inital but for Locations it points to _ first _
Now, today I am trying to get Location to use migration (so I later can add things to it, needed now) for the first time...
Running makemigration Location works and generates a new clean migration directory and all, but then when doing migrate --fake I get the following django.db.migrations.exceptions.InconsistentMigrationHistory: Migration log.0001_initial is applied before its dependency locations.0001_initial on database 'default'.
Understand this is caused by the mistake earlier by forgetting to get migrations done for Locations when we started to use this in Django - any ideas how to resolve this in a good way?

Solved it - here is how it went.
1) reset the migration history for Log
>> python manage.py migrate --fake log zero
2) create first migrations for Location (this is what we forgot 2 years back)
>> python manage.py makemigrations locations
3) Edited the 0001_initial.py for Log and changed reference in it dependencies for Location from'_ first _' to '0001_initial'
4) clear (if any) migrations for locations
>> python manage.py migrate --fake locations zero
5) recreate projects all migration history
>> python manage.py showmigrations
>> python manage.py migrate --fake
6) Done - tested it by trying run migration to see if thing wirk
>> python manage.py migrate
No migrations to apply
:-)

Django backup strategy with dumpdata and migrations

As in this question, I set up a dumpdata-based backup system for my database. The setup is akin to running a cron script that calls dumpdata and moves the backup to a remote server, with the aim of simply using loaddata to recover the database. However, I'm not sure this plays well with migrations. loaddata now has an ignorenonexistent switch to deal with deleted models/fields, but it is not able to resolve cases where columns were added with one-off defaults or apply RunPython code.
The way I see it, there are two sub-problems to address:
Tag each dumpdata output file with the current version of each app
Splice the fixtures into the migration path
I'm stumped about how to tackle the first problem without introducing a ton of overhead. Would it be enough to save an extra file per backup that contained an {app_name: migration_number} mapping?
The second problem I think is easier once the first one is solved, since the process is roughly:
Create a new database
Run migrations forward to the appropriate point for each app
Call loaddata with the given fixture file
Run the rest of the migrations
There's some code in this question (linked from the bug report) that I think could be adapted for this purpose.
Since these are fairly regular/large snapshots of the database, I don't want to keep them as data migrations cluttering up the migrations directory.

I am taking the following steps to backup, restore or transfer my postgresql database between any instance of my project:
The idea is to keep the least possible migrations as if manage.py makemigrations was run for the first time on an empty database.
Let's assume that we have a working database to our development environment. This database is a current copy of the production database that should not be open to any changes. We have added models, altered attributes etc and those actions have generated additional migrations.
Now the database is ready to be migrated to production which -as stated before- is not open to public so it is not altered in any way. In order to achieve this:
I perform the normal procedure in the development environment.
I copy the project to the production environment.
I perform the normal procedure in the production environment
We make the changes in our development environment. No changes should happen in the production database because they will be overridden.
Normal Procedure
Before anything else, I have a backup of the project directory (which includes a requirements.txt file), a backup of the database and -of course- git is a friend of mine.
I take a dumpdata backup in case I need it. However, dumpdata has some serious limitations regarding content types, permissions or other cases where a natural foreignkey should be used:
./manage.py dumpdata --exclude auth.permission --exclude contenttypes --exclude admin.LogEntry --exclude sessions --indent 2 > db.json
I take a pg_dump backup to use:
pg_dump -U $user -Fc $database --exclude-table=django_migrations > path/to/backup-dir/db.dump
Only if I want to merge existing migrations in one, I delete all migrations from every application.
In my case the migrations folder is a symlink, so I use the following script:
#!/bin/bash
for dir in $(find -L -name "migrations")
do
rm -Rf $dir/*
done
I delete and recreate the database:
For example, a bash script can include the following commands:
su -l postgres -c "PGPASSWORD=$password psql -c 'drop database $database ;'"
su -l postgres -c "createdb --owner $username $database"
su -l postgres -c "PGPASSWORD=$password psql $database -U $username -c 'CREATE EXTENSION $extension ;'"
I restore the database from the dump:
pg_restore -Fc -U $username -d $database path/to/backup-dir/db.dump
If migrations were deleted in step 3, I recreate them in the following way:
./manage.py makemigrations <app1> <app2> ... <appn>
... by using the following script:
#!/bin/bash
apps=()
for app in $(find ./ -maxdepth 1 -type d ! -path "./<project-folder> ! -path "./.*" ! -path "./")
do
apps+=(${app#??})
done
all_apps=$(printf "%s " "${apps[#]}")
./manage.py makemigrations $all_apps
I migrate using a fake migration:
./manage.py migrate --fake
In case something has gone completely wrong and everything is ***, (this can happen, indeed), I can use the backup to revert everything to its previous working state. If I would like to use the db.json file from step one, it goes like this:
When pg_dump or pg_restore fails
I perform the steps:
3 (delete migrations)
4 (delete and recreate the database)
6 (makemigrations)
and then:
Apply the migrations:
./manage.py migrate
Load the data from db.json:
./manage.py loaddata path/to/db.json
Then I try to find out why my previous effort was not successful.
When the steps are performed successfully, I copy the project to the server and perform the same ones to that box.
This way, I always keep the least number of migrations and I am able to use pg_dump and pg_restore to any box that shares the same project.

Django can't destroy and create test databases properly

When I try to run my unittest, this is what I get:
python manage.py test dbank --settings=databank_web.settings.dqs.dev_hooman
Creating test database for alias 'default'...
Creating test database for alias 'global'...
Creating test database for alias 'optin_db'...
Creating test database for alias 'vpd3'...
Creating test database for alias 'user_db'...
Creating test database for alias 'vpd1'...
Creating test database for alias 'vpd2'...
.
----------------------------------------------------------------------
Ran 1 test in 0.327s
OK
Destroying test database for alias 'default'...
Warning: Table 'mysql.proc' doesn't exist
It couldn't destroy the database. It gets better, when I rerun the test:
python manage.py test dbank --settings=databank_web.settings.dqs.dev_hooman
Creating test database for alias 'default'...
Creating test database for alias 'global'...
Got an error creating the test database: (1007, "Can't create database 'test_dqs12_full2'; database exists")
Type 'yes' if you would like to try deleting the test database 'test_dqs12_full2', or 'no' to cancel: yes
Destroying old test database 'global'...
Got an error recreating the test database: Table 'mysql.proc' doesn't exist
Any idea why this is going wrong?
Running latest homebrew + mysql-5.6.21 + Django 1.5.5

I have finally found the reason.
Our application has around 7 databases. When I was about to drop them manually to get a clean state, I accidentally also dropped the mysql database. This database should not be ever deleted, as it contains vital information about the root user.
When I realised this, I tried to mitigate by reinstalling MySQL via brew.
brew uninstall mysql
brew install mysql
This worked and mysql database showed up again when I ran show databases;, however the problem persisted. Thats when I came here for help.
As it seems things are a bit more complicated than that with brew.
This is how I got it working again:
1) brew uninstall mysql
2) sudo rm -r /usr/local/var/mysql/
3) brew install mysql
4) unset TMPDIR
5) mysql_install_db --verbose --user='whoami' --basedir="$(brew --prefix mysql)" --datadir=/usr/local/var/mysql --tmpdir=/tmp
6) mysql.server restart
7) mysql_secure_installation
Now it works. Hope this helps other fellow mac users. Thanks.

Well, there are two errors indicating that the mysql.proc table is missing. You might be able to generate it running mysql_update.

Generate Database Schema using Python

I want to generate a basic DB schema for my django project to display all my Apps with Models and Model Fields with boundary conditions etc. Is there already any DB schema generator for django in python? Or otherwise how should i go about doing it.

If your talking about needing to see the SQL schema, run ./manage.py sqlall <appname>
If you want a visualisation of the schema you can get django-extensions and run ./manage.py graph_models -a -g -o my_project.png. This will produce a pretty schema graph for you, but generally omits border conditions. you may want to check the options to add more data. http://readthedocs.org/docs/django-extensions/en/latest/graph_models.html

manage.py sql <appname appname ...> (docs)

Using Your DB
As mentioned in the tutorial, you can use your database's command line client to get the schema.
Example using sqlite:
python manage.py dbshell
> .schema
You may need to install sqlite3 for this to work.
Using Django
You used to be able to use python manage.py sql ..., but it has been deprecated in 1.9 in favor of migrations. You can check out the initial migration scripts using:
python manage.py sqlmigrate myapp 0001_initial
(From Answer: Equivalent of sqlall in Django 1.9?)