I'm working on a project that I inherited, and I want to add a table to my database that is very similar to one that already exists. Basically, we have a table to log users for our website, and I want to create a second table to specifically log users that our site fails to do a task for.
Since I didn't write the site myself, and am pretty new to both SQL and Django, I'm a little paranoid about running a migration (we have a lot of really sensitive data that I'm paranoid about wiping).
Instead of having a django migration create the table itself, can I create the second table in MySQL, and the corresponding model in Django, and then have this model "recognize" the SQL table? without explicitly using a migration?
SHORT ANSWER: Yes.
MEDIUM ANSWER: Yes. But you will have to figure out how Django would have created the table, and do it by hand. That's not terribly hard.
Django may also spit out some warnings on startup about migrations being needed...but those are warnings, and if the app works, then you're OK.
LONG ANSWER: Yes. But for the sake of your sanity and sleep quality, get a completely separate development environment and test your backups. (But you knew that already.)
Related
I need in my project (based on Postgresql) to export few models as SQLite dump. It must be made 'on-demand' f.e. on user request.
I can prepare appropriate database manually, but I would like to omit the duplication of information about schema. I dream about solution like 'dumpdata app-name' but instead of JSON/XML/YAML there should be SQLite.
Is there such solution?
P.S. For too overbearing - it's not broad question. Possibilities are only two: there is such snippet, helper etc. or there is not and it should be done individually. I can't find it by my own so I ask for help.
To sum up details (some people could not figure them out and could put my question 'on hold'):
there is Django project with main Postgresql database
I'm already processing request from users (through an API)
one of request is "make a dump of some models (tables) in SQLite format"
I can prepare temporary SQLite database and manually fill it with data
I'm looking for powerful and universal tool (solution) which will do such export automatically (from some of my Django models to SQLite)
I put my first Django site online eight months ago. It was both a proof of concept as well as my first experience with Django. Fast forward eight months, I have validated my idea, but since it was a proof of concept and my first Django project, the code is pretty messy. Essentially, I am going to be re-writing the majority of the site, including re-engineering the models.
This is all fine and good. I have all my new models planned out. Essentially, I am going to create a new database to develop off of and let South manage any new database schema changes I make.
It is important to note two things:
I will not be creating a new project, just a new database.
This will be the first time I am incorporating South into the project and I would prefer to start with fresh models and a fresh database.
My question is, when I create the new database, will importing the contents of the old auth_* and django_* tables into the new auth_* and django_* tables create any problems? I have had some users register using the original proof of concept and I don't want to lose their information. I've never had to do this before so I'm not sure if there will be any repercussions.
If you use sql dump, such as
mysqldump -uusername -ppassword db_name table_name > xxxx.sql
mysql -uusername -ppassword new_db_name < xxxx.sql
The database's side is fine，if your backend is some other db，you can still find the similar commands.
For a new db, i think you need to export/import auth_user, i'm not quite sure if you need other contents in django_* tables. You can do this step by step, and see whether the new project works.
I'm programming a web application using sqlalchemy. Everything was smooth during the first phase of development when the site was not in production. I could easily change the database schema by simply deleting the old sqlite database and creating a new one from scratch.
Now the site is in production and I need to preserve the data, but I still want to keep my original development speed by easily converting the database to the new schema.
So let's say that I have model.py at revision 50 and model.py a revision 75, describing the schema of the database. Between those two schema most changes are trivial, for example a new column is declared with a default value and I just want to add this default value to old records.
Eventually a few changes may not be trivial and require some pre-computation.
How do (or would) you handle fast changing web applications with, say, one or two new version of the production code per day ?
By the way, the site is written in Pylons if this makes any difference.
Alembic is a new database migrations tool, written by the author of SQLAlchemy. I've found it much easier to use than sqlalchemy-migrate. It also works seamlessly with Flask-SQLAlchemy.
Auto generate the schema migration script from your SQLAlchemy models:
alembic revision --autogenerate -m "description of changes"
Then apply the new schema changes to your database:
alembic upgrade head
More info here: http://readthedocs.org/docs/alembic/
What we do.
Use "major version"."minor version" identification of your applications. Major version is the schema version number. The major number is no some random "enough new functionality" kind of thing. It's a formal declaration of compatibility with database schema.
Release 2.3 and 2.4 both use schema version 2.
Release 3.1 uses the version 3 schema.
Make the schema version very, very visible. For SQLite, this means keep the schema version number in the database file name. For MySQL, use the database name.
Write migration scripts. 2to3.py, 3to4.py. These scripts work in two phases. (1) Query the old data into the new structure creating simple CSV or JSON files. (2) Load the new structure from the simple CSV or JSON files with no further processing. These extract files -- because they're in the proper structure, are fast to load and can easily be used as unit test fixtures. Also, you never have two databases open at the same time. This makes the scripts slightly simpler. Finally, the load files can be used to move the data to another database server.
It's very, very hard to "automate" schema migration. It's easy (and common) to have database surgery so profound that an automated script can't easily map data from old schema to new schema.
Use sqlalchemy-migrate.
It is designed to support an agile approach to database design, and make it easier to keep development and production databases in sync, as schema changes are required. It makes schema versioning easy.
Think of it as a version control for your database schema. You commit each schema change to it, and it will be able to go forwards/backwards on the schema versions. That way you can upgrade a client and it will know exactly which set of changes to apply on that client's database.
It does what S.Lott proposes in his answer, automatically for you. Makes a hard thing easy.
The best way to deal with your problem is to reflect your schema instead doing it the declarative way. I wrote an article about the reflective approach here:
http://petrushev.wordpress.com/2010/06/16/reflective-approach-on-sqlalchemy-usage/
but there are other resources about this also. In this manner, every time you make changes to your schema, all you need to do is restart the app and the reflection will fetch the new metadata for the changes in tables. This is quite fast and sqlalchemy does it only once per process. Of course, you'll have to manage the relationships changes you make yourself.
If I change a field in a Django model, how can I synchronize it with the database tables? Do I need to do it manually on the database or is there a tool that does helps with the process?
Alas, Django does not support any easy solution to this.
The only thing django will do for you, is restart your database with new tables that match your new models:
$ #DON'T DO THIS UNLESS YOU CAN AFFORD TO LOSE ALL YOUR DATA!
$ python PROJECT_DIR/manage.py syncdb
the next option is to use the various sql* options to manage.py to see what django would do to match the current models to the database, then issue your own ALTER TABLE commands to make everything work right. Of course this is error prone and difficult.
The real solution is to use a database migration tool, such as south to generate migration code.
Here is a similar question with discussion about various database migration options for django.
Can't seem to be able to add a comment to the marked answer, probably because I haven't got enough rep (be nice if SO told me so though).
Anyway, just wanted to add that in the answered post, I believe it is wrong about syncdb - syncdb does not touch tables once they have been created and have data in them. You should not lose data by calling it (otherwise, how could you add new tables for new apps?)
I believe the poster was referring to the reset command instead, which does result in data loss - it will drop the table and recreate it and hence it'll have all the latest model changes.
Django Evolution can help, but the best option really is to plan out your schema in advance, or to make simple modifications manually. Or, to be willing to toast your test data by dropping tables and re-syncing.
Django does not provide for this out of the box.
Here's some information from the Django Book on doing it by hand (see Making Changes to a Database Schema). This works for straightforward, simple changes.
Longer-term, you'll probably want to use a migration tool. There are three major options:
django-evolution
Dmigrations (written by Simon Willison, one of the creators of Django) (works only with MySQL)
South
EDIT: Looking through the question linked by TokenMacGuy, I'll add two more to the list for the sake of completeness:
Migratory
simplemigrations
Just to throw in an extra opinion - dmigrations is pretty nice and clear to use, but I'd say South is your best bet. Again, it's easy to get into, but it's more powerful and also has support for more database backends than just MySQL. It even handles MSSQL, if that's your thing
as I usually don't do the up front design of my models in Django projects I end up modifying the models a lot and thus deleting my test database every time (because "syncdb" won't ever alter the tables automatically for you). Below lies my workflow and I'd like to hear about yours. Any thoughts welcome..
Modify the model.
Delete the test database. (always a simple sqlite database for me.)
Run "syncdb".
Generate some test data via code.
goto 1.
A secondary question regarding this.. In case your workflow is like above, how do you execute the 4. step? Do you generate the test data manually or is there a proper hook point in Django apps where you can inject the test-data-generating-code at server startup?\
TIA.
Steps 2 & 3 can be done in one step:
manage.py reset appname
Step 4 is most easily managed, from my understanding, by using fixtures
This is a job for Django's fixtures. They are convenient because they are database independent and the test harness (and manage.py) have built-in support for them.
To use them:
Set up your data in your app (call
it "foo") using the admin tool
Create a fixtures directory in your
"foo" app directory
Type: python manage.py dumpdata --indent=4 foo > foo/fixtures/foo.json
Now, after your syncdb stage, you just type:
python manage.py loaddata foo.json
And your data will be re-created.
If you want them in a test case:
class FooTests(TestCase):
fixtures = ['foo.json']
Note that you will have to recreate or manually update your fixtures if your schema changes drastically.
You can read more about fixtures in the django docs for Fixture Loading
Here's what we do.
Apps are named with a Schema version number. appa_2, appb_1, etc.
Minor changes don't change the number.
Major changes increment the number. Syncdb works. And a "data migration" script can be written.
def migrate_appa_2_to_3():
for a in appa_2.SomeThing.objects.all():
appa_3.AnotherThing.create( a.this, a.that )
appa_3.NewThing.create( a.another, a.yetAnother )
for b in ...
The point is that drop and recreate isn't always appropriate. It's sometimes helpful to move data form the old model to the new model without rebuilding from scratch.
South is the coolest.
Though good ol' reset works best when data doesn't matter.
http://south.aeracode.org/
To add to Matthew's response, I often also use custom SQL to provide initial data as documented here.
Django just looks for files in <app>/sql/<modelname>.sql and runs them after creating tables during syncdb or sqlreset. I use custom SQL when I need to do something like populate my Django tables from other non-Django database tables.
Personally my development db is for a project I'm working on right now is rather large, so I use dmigrations to create db migration scripts to modify the db (rather than wiping out the db everytime like I did in the beginning).
Edit: Actually, I'm using South now :-)