Develop Reference

Is Django Make Indexes for ManyToMany field?

i ask if Django make database indexes for ManyToMany Field ... and if yes is it do this for the model i provide in through ?
i just want to make sure that database will go fast when it have a lot of data

It does for plain m2m fields - can't tell for sure for "through" tables since I don't have any in my current project and don't have access to other projects ATM, but it still should be the case since this index is necessary for the UNIQUE constraint.
FWIW you can easily check it by yourself by looking at the tables definitions in your database (in MySQL / mariadb using the show create table yourtablename).
This being said, db index are only useful when there are mainly distinct values, and can actually degrade performances (depending on your db vendor etc) if that's not the case - for example, if you have a field with like 3 or 4 possible values (ie "gender" or something similar), indexing it might not yield the expected results. Should not be an issue for m2m tables since the (table1_id, table2_id) pair is supposed to be unique, but the point is that you should not believe that just adding indexes will automagically boost your performances - tuning SQL database performances is a trade in and by itself, and actually depends a lot of how you actually use your data.

Alembic migration acting on different versions of same database

I am trying to use alembic migrations to act on different versions of the same database. An example would be that I have two databases, one live and one for testing. Each of them might be in different states of migration. For one, the test database might not exist at all.
Say live has a table table1 with columns A and B. Now I would like to add column C. I change my model to include C and I generate a migration script that has the following code
op.add_column('table1', sa.Column('C', sa.String(), nullable=True))
This works fine with the existing live database.
If I now call alembic upgrade head referring to a non-existing test database, I get an (Operational Error) duplicate column name... error. I assume this is due to my model containing the C column and that alembic/sqlalchemy creates the full table automatically if it does not exist.
Should I simply trap the error or is there a better way of doing this?

I would suggest that immediately after your test db is newly created you should stamp it with head
command.stamp(configs_for_test_db, 'head')
This will go ahead and insert the head revision number into the appropriate alembic table without actually running migrations so that the revision number will reflect the state of the db (namely that your newly created db is up to date wrt migrations). After the db is stamped, alembic upgrade should behave properly.

Automatic database refactoring to reconcile with refactored python code in OpenERP?

When I refactor a Python class (e.g. change the name/type of a field), what is the best way to also reflect these changes in the database?
Currently, I manually run SQL scripts to transfer data to new columns and prune old ones, but this is slow and error prone.
Anything like Liquibase for Java?
Thanks,
Anton

The OpenERP ORM is capable of synchronizing the database schema with the python definition of the models, but it cannot perform complex migration operations. Its purpose is to create or update the database tables and columns necessary to support the newest version of your models, i.e. allow inserting, updating and reading records.
As part of this synchronization, it supports simple schema alterations: as long as the name and type of a column does not change, the ORM will try to update the SQL constraints (NOT NULL, UNIQUE, FOREIGN KEY, etc.), and the ONDELETE rule, if any.
If the type changed, the old column will be renamed to be out of the way (but kept around so you can manually migrate the data), and a new appropriate column will be created. You'll need a manual migration operation if you want to migrate the data [*].
If the only thing that changes is the name of the column, you can give the ORM a hint about that using the oldname column property. It tells the ORM that an old column needs to be renamed instead of creating a new one. For example, this column definition would make the ORM attempt to rename partner_id to customer_id if the customer_id column is missing:
'customer_id': fields.many2one('res.partner', 'Customer', oldname='partner_id'),
You can start the synchronization logic either by launching the server with `-u <module> -d <database> or by clicking on the "Update" button of the corresponding module in the Settings menu of that database (--update is a synonym of -u, but make sure to pass -d to have it start the synchronization on the database directly).
Note that you should not use the oldname parameter when you're in development and just trying out different database schema definitions. This option is mainly useful for keeping track of field renamings between major versions of a given module. For work-in-progress development you don't care about keeping track of all the iterations so using -u is sufficient, you should periodically start again on a new empty database anyway.
[*] There is limited and mostly undocumented support for embedding your own migration scripts in your modules, where the ORM will automatically execute them depending on the detected module version. Unfortunately the best way to understand how this work is to read the source code.

Just run openerp-server with -u option:
python openerp-server.py -c <your config file> -u <your module>

Django-DB-Migrations: cannot ALTER TABLE because it has pending trigger events

I want to remove null=True from a TextField:
- footer=models.TextField(null=True, blank=True)
+ footer=models.TextField(blank=True, default='')
I created a schema migration:
manage.py schemamigration fooapp --auto
Since some footer columns contain NULL I get this error if I run the migration:
django.db.utils.IntegrityError: column "footer" contains null values
I added this to the schema migration:
for sender in orm['fooapp.EmailSender'].objects.filter(footer=None):
sender.footer=''
sender.save()
Now I get:
django.db.utils.DatabaseError: cannot ALTER TABLE "fooapp_emailsender" because it has pending trigger events
What is wrong?

Another reason for this maybe because you try to set a column to NOT NULL when it actually already has NULL values.

Every migration is inside a transaction. In PostgreSQL you must not update the table and then alter the table schema in one transaction.
You need to split the data migration and the schema migration. First create the data migration with this code:
for sender in orm['fooapp.EmailSender'].objects.filter(footer=None):
sender.footer=''
sender.save()
Then create the schema migration:
manage.py schemamigration fooapp --auto
Now you have two transactions and the migration in two steps should work.

At the operations I put SET CONSTRAINTS:
operations = [
migrations.RunSQL('SET CONSTRAINTS ALL IMMEDIATE;'),
migrations.RunPython(migration_func),
migrations.RunSQL('SET CONSTRAINTS ALL DEFERRED;'),
]

If you are adding a non-nullable field, you need to do it in two migrations:
AddField and RunPython to populate it
AlterField to change the field to be non-nullable
Explanation
On PostgreSQL and SQLite, this problem can occur if you have a sufficiently complex RunPython command combined with schema alterations in the same migration. For example, if you are adding a non-nullable field, the typical migration steps for this is:
AddField to add the field as nullable
RunRython to populate it
AlterField to change the field to be non-nullable
On SQLite and Postgres, this can cause problems because the whole thing is being done in one transaction.
The Django docs have a specific warning about this:
On databases that do support DDL transactions (SQLite and PostgreSQL), RunPython operations do not have any transactions automatically added besides the transactions created for each migration. Thus, on PostgreSQL, for example, you should avoid combining schema changes and RunPython operations in the same migration or you may hit errors like OperationalError: cannot ALTER TABLE "mytable" because it has pending trigger events.
If this is the case, the solution is to separate your migration into multiple migrations. In general, the way to split is to have a first migration containing the steps up through the run_python command and the second migration containing all the ones after it. Thus, in the case described above, the pattern would be the AddField and RunPython in one migration, and the AlterField in a second.

Have just hit this problem. You can also use db.start_transaction() and db.commit_transaction() in the schema migration to separate data changes from schema changes. Probably not so clean as to have a separate data migration but in my case I would need schema, data, and then another schema migration so I decided to do it all at once.

You are altering the column schema. That footer column can no longer contain a blank value. There are most likely blank values already stored in the DB for that column. Django is going to update those blank rows in your DB from blank to the now default value with the migrate command. Django tries to update the rows where footer column has a blank value and change the schema at the same time it seems (I'm not sure).
The problem is you can't alter the same column schema you are trying to update the values for at the same time.
One solution would be to delete the migrations file updating the schema. Then, run a script to update all those values to your default value. Then re-run the migration to update the schema. This way, the update is already done. Django migration is only altering the schema.

In my case I've got
AddField
RunPython
RemoveField
Then I just moved the last RemoveFied to the new migration file, that fixed the problem

step 1)the solution is to remove the latest migration from the migration folder and remove the latest added fields in models.
step 2)then again makemigration and migrate
step 3)At the last add the field again that has been removed in the first step
step 4)then again makemigration and migrate
Problem solved

In South, can I copy the value of an old column to a new one?

One of my Django models is a subclass and I want to change its superclass to one that is very similar to the original one. In particular, the new superclass describes the same object and has the same primary key. How can I make South create the new OneToOne field and copy the values from the old one to the new one?

In south, there are two kinds of migrations: schema migrations and data migrations.
After you've created the schemamigration, create a corresponding data migration:
./manage.py datamigration <app> <migration_name>
Do not run the migration (yet). Instead, open up the migration file you just created.
You'll find the method named forwards(). Into this you define the procedure by which values from old tables get copied to new tables.
If you're changing the structure of a given table to a more complex layout, a common method is to have two schema migrations around a data migration: the first schema migration adds fields, the data migration translates the old fields to the new fields, and the second schema migration deletes the old fields. You can do just about anything with the database with the forwards() method, so long as you keep track of which schema (previous or current) you're accessing. Generally, you only read from the orm.-related, and write to the traditional Django accessors.
The South Data Migration Tutorial covers this in some detail. It shows you how to use South's orm reference to access the database using the schema prior to the schema migration and gives access to the database without Django complaining about fields it doesn't understand.
If you're renaming a class, that can be tricky-- it involves creating the new table, migrating from one to the other, and deleting the old table. South can do it, but it might take more than one pass through shifting schemas and data migrations.
South also has the backwards() method, which allows you to return your database tables to a previous step. In some cases, this may be impossible; the new table may record information that will be lost in a downgrade. I recommend using throwing an exception in backwards() if you're not in DEBUG mode.