I've recently inherited a production codebase for a webapp written using Django. Up until now, the database the project has been using has been the default SQLite3 database, but now that more people are using the app, moving to Postgres is necessary.
I've been able to set up an empty postgres database with the project that works fine. The problem I'm encountering is in moving the data from the old project to the new one. I can run
python manage.py dumpdata --natural-foreign > dump.json
to dump the data, which works fine, but when I switch to postgres in settings.py and run
python manage.py loaddata dump.json
I get the following error:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/core/management/base.py", line 390, in run_from_argv
self.execute(*args, **cmd_options)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/core/management/base.py", line 441, in execute
output = self.handle(*args, **options)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 60, in handle
self.loaddata(fixture_labels)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 90, in loaddata
self.load_label(fixture_label)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 147, in load_label
obj.save(using=self.using)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 173, in save
models.Model.save_base(self.object, using=using, raw=True)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/models/base.py", line 738, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/models/base.py", line 803, in _save_table
forced_update)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/models/base.py", line 853, in _do_update
return filtered._update(values) > 0
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/models/query.py", line 580, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1062, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
cursor.execute(sql, params)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/mnt/d/Code/MCJobTrack/_VENV/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: Problem installing fixture '/mnt/d/Code/MCJobTrack/jobtrack_project/dump.json': Could not load contenttypes.ContentType(pk=15): duplicate key value violates unique constraint "django_content_type_app_label_45f3b1d93ec8c61c_uniq"
DETAIL: Key (app_label, model)=(jobtrack, purchaseorder) already exists.
So far I've tried:
Running TRUNCATE django_content_type RESTART IDENTITY CASCADE; in the dbshell
Excluding contenttypes when dumping the data. (I get the error django.core.serializers.base.DeserializationError: Problem installing fixture '/mnt/d/Code/MCJobTrack/jobtrack_project/dump_no_contenttypes.json': ContentType matching query does not exist.)
Resetting the primary key sequence
Using the --natural-primary tag when dumping the data
Any help would be greatly appreciated.
You probably need to use --natural-primary as well, so that the primary keys of content types are not exported to the fixture:
python manage.py dumpdata --natural-foreign --natural-primary > dump.json
Turns out there were two things I needed to do:
Dump the data with --natural-primary, --natural-foreign, and -e
contenttypes
Have loaddata ignore post_save signals
It still takes a very long time to load the data, but it works.
Related
I am trying to unitest my models in my django application that consisits of some migrations. I have prepared my test, but when running
./manage.py test my_app
i get the following error
django.db.utils.OperationalError: (1005, "Can't create table 'test_rhombusdb.#sql-4ca_2c' (errno: 150)")
The database is created normally, So I guess I have permissions to create the db.
I really don't know which table this is reffering to. Migrations pass normal and db works fine in production. What could be the issue here?
what other info would you like?
my stack trace
File "./manage.py", line 18, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/test.py", line 30, in run_from_argv
super(Command, self).run_from_argv(argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 393, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/test.py", line 74, in execute
super(Command, self).execute(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 444, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/test.py", line 90, in handle
failures = test_runner.run_tests(test_labels)
File "/usr/local/lib/python2.7/dist-packages/django/test/runner.py", line 210, in run_tests
old_config = self.setup_databases()
File "/usr/local/lib/python2.7/dist-packages/django/test/runner.py", line 166, in setup_databases
**kwargs
File "/usr/local/lib/python2.7/dist-packages/django/test/runner.py", line 370, in setup_databases
serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/creation.py", line 368, in create_test_db
test_flush=not keepdb,
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 120, in call_command
return command.execute(*args, **defaults)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 444, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 179, in handle
created_models = self.sync_apps(connection, executor.loader.unmigrated_apps)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 317, in sync_apps
cursor.execute(statement)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 97, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 62, in execute
return self.cursor.execute(sql)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 124, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
django.db.utils.OperationalError: (1005, "Can't create table 'testdb.#sql-4ca_7c' (errno: 150)")
I can see there is an error when migration is called.
PS: I droped my database(real one but the developement one) and tried to recreate tables running migrate. I came with the same error. Running migrate again worked like a charm....Something is really really really wrong here....
PS2: Deleting all migration folders and starting without gave me the same error after running migrate with no migration folders present....Very bad error very difficult to debug
PS3: There is an issue with the foreign key on one (at least so far i've tested) of my models. I tried deactivating apps and activating them in terms, and when I got to a specific one, error occured.I isolated one model, and gave me the issue. I commented out the foreign keys and worked. Uncomment one of them error again.
class Patient(models.Model):
#many fields here
doctor = models.ForeignKey(settings.AUTH_USER_MODEL, related_name="patients")
It worked with 1.6 and sqlite3, I moved to 1.8 and mysql, and boom...error.
AUTH_USER_MODEL = 'users.MyCustomUser'
MyCustomUser is a class inheriting AbstractBaseUser and PermissionsMixin.
I had the same (pretty general) error message. For me it came down to having forgot to run makemigrations on a new app. See also:
https://docs.djangoproject.com/en/1.8/topics/migrations/#dependencies
I have a problem with Django quite incomprehensible.
I used models.py to model my database. This database runs on the production server. I want to change it, so I create a new database on the development server. The problem happens when I do syncdb, Django tries to create twice the same table, as you can see below:
python manage.py syncdb
Operations to perform:
Synchronize unmigrated apps: inventaryApp, newPrpvApp, grappelli, debug_toolbar, registration, import_export
Apply all migrations: sitetree, sessions, admin, sites, auth, contenttypes
Synchronizing apps without migrations:
Creating tables...
Creating table u_institutions
Creating table u_users
Creating table o_organes
Creating table o_degats
Creating table o_validation
Creating table o_m_determination
Creating table o_stades_dvlpmt
Creating table o_types
Creating table precision_date
Creating table t_categories
Creating table t_etat_synonymie
Creating table g_langues
Creating table g_pays
Creating table g_regions
Creating table g_communes
Creating table pays_langues
Creating table newprpvapp_userprofile
Creating table t_noms_verna
Creating table langues_noms_verna
Creating table maladies
Creating table t_familles
Creating table t_genres
Creating table t_especes
Creating table images
Creating table esp_noms_verna
Creating table t_infra_types
Creating table t_infra_sp
Creating table t_synonymes
Creating table o_contexte
Creating table o_interception
Creating table o_hotes
Creating table hotes_organes
Creating table o_organismes_associes
Creating table u_institutions
As you can see, the u_institutions table is created the first, but also at the end. And this is where the execution of the command stops to show:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
output = self.handle(*args, **options)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/base.py", line 533, in handle
return self.handle_noargs(**options)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 27, in handle_noargs
call_command("migrate", **options)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 115, in call_command
return klass.execute(*args, **defaults)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
output = self.handle(*args, **options)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 128, in handle
created_models = self.sync_apps(connection, executor.loader.unmigrated_apps)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 239, in sync_apps
cursor.execute(statement)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 81, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 63, in execute
return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "u_institutions" already exists
I have verified that the table had not been set twice in model.py, and it was not. I use Django 1.7, and psycopg 2.6. Do you have an idea please? Thank you all!
Edit
I had already executed makemigrations, and everything was good. Now it indicates that no change is detected, which is probably normal.
When I try migrate, I get :
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
output = self.handle(*args, **options)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 128, in handle
created_models = self.sync_apps(connection, executor.loader.unmigrated_apps)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 239, in sync_apps
cursor.execute(statement)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 81, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 63, in execute
return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "u_institutions" already exists
And migrate --fake return :
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
output = self.handle(*args, **options)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 128, in handle
created_models = self.sync_apps(connection, executor.loader.unmigrated_apps)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 239, in sync_apps
cursor.execute(statement)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 81, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/hugo/DEV/.virtualenvs/env_eprpv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 63, in execute
return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "u_institutions" already exists
It seems that you have already created your database schema. Please, use migration (as long as you use Django 1.7). syncdb is deprecated.
python manage.py makemigrations
python manage.py migrate
If you will face any errors here (like relation "..." already exists) run this command to mark this tables as applied:
python manage.py migrate --fake
UPDATE #2 - Dec 13 (STILL NEED HELP)
The Github repo for the project: https://github.com/onlyandrewn/gale
From what I understand looking at my Terminal, it seems to keep referring to previous migrations after I try to ./manage.py makemigrations blog and ./manage.py migrate blog
The problem may stem from an old error for non nullable fields when I mistakenly put a string into a field called order = models.PositiveSmallIntegerField() which should have been an integer. Now, I would like to add a new field called rank, and I go through the process of ./manage.py makemigrations blog and ./manage.py migrate blog but I keep getting prompted with this in the Terminal:
You have unapplied migrations; your app may not work properly until they are applied.
Run 'python manage.py migrate' to apply them.
When I do run python manage.py, I get this long list of stuff in my Terminal:
Operations to perform:
Apply all migrations: blog
Running migrations:
**Applying blog.0007_auto_20141212_1622...Traceback (most recent call last):**
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Library/Python/2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Library/Python/2.7/site-packages/django/core/management/base.py", line 338, in execute
output = self.handle(*args, **options)
File "/Library/Python/2.7/site-packages/django/core/management/commands/migrate.py", line 160, in handle
executor.migrate(targets, plan, fake=options.get("fake", False))
File "/Library/Python/2.7/site-packages/django/db/migrations/executor.py", line 63, in migrate
self.apply_migration(migration, fake=fake)
File "/Library/Python/2.7/site-packages/django/db/migrations/executor.py", line 97, in apply_migration
migration.apply(project_state, schema_editor)
File "/Library/Python/2.7/site-packages/django/db/migrations/migration.py", line 107, in apply
operation.database_forwards(self.app_label, schema_editor, project_state, new_state)
File "/Library/Python/2.7/site-packages/django/db/migrations/operations/fields.py", line 139, in database_forwards
schema_editor.alter_field(from_model, from_field, to_field)
File "/Library/Python/2.7/site-packages/django/db/backends/schema.py", line 473, in alter_field
self._alter_field(model, old_field, new_field, old_type, new_type, old_db_params, new_db_params, strict)
File "/Library/Python/2.7/site-packages/django/db/backends/sqlite3/schema.py", line 190, in _alter_field
self._remake_table(model, alter_fields=[(old_field, new_field)])
File "/Library/Python/2.7/site-packages/django/db/backends/sqlite3/schema.py", line 135, in _remake_table
self.quote_name(model._meta.db_table),
File "/Library/Python/2.7/site-packages/django/db/backends/schema.py", line 99, in execute
cursor.execute(sql, params)
File "/Library/Python/2.7/site-packages/django/db/backends/utils.py", line 81, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/Library/Python/2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/Library/Python/2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Library/Python/2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/Library/Python/2.7/site-packages/django/db/backends/sqlite3/base.py", line 485, in execute
return Database.Cursor.execute(self, query, params)
**django.db.utils.IntegrityError: column order is not unique**
Two things that stand out to me in this long slew of text:
Applying blog.0007_auto_20141212_1622...Traceback (most recent call last): (The current migration is closer to 25, #7 is a very old migration)
django.db.utils.IntegrityError: column order is not unique
order = models.PositiveSmallIntegerField()
means IntegerField and NOT Text- or CharField.
and you gave "http://" which IS string.
change this to default number like 0 or 0000 whatever..
and run migrations again
"I'm wondering how to get rid of it in the Terminal." - you better wonder how to fix the problem then it goes away from your terminal
after run makemigrations django put 'http://' in your field definition, creating a wrong migration
check your migrations directory, edit the last one and manually remove 'http://' from the field definition
EDIT:
the migration 0007 create a unique consttraint on the field order but your db already contains duplicates so that migration cannot succeed.
you have a totally inconsistent situation, as your system is under development (ie not yet released) it's better that you remove all the migrations, remove the db a run makemigrations again.
I have a cron job that downloads data from MySQL every night at 3. I can test this connection and download and it works. Occasionally the download partially fails. (partial download) If I try and re-run the py script, it barks. Duplicate entry error for key 2.
I want to be able to run a script and erase only the entries for the previous night so I can rerun the script that populates the db. There are three other tables tied to this one. What is django going to do if is create a SQL script that deletes yesterdays records? Will it automatically delete necessary additions to the other tables, or should I do this in the script also?
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.6/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.6/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.6/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.6/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/usr/local/django/grease/greaseboard/management/commands/import_patients.py", line 27, in handle
mrn = row.MRN,
File "/usr/local/lib/python2.6/site-packages/django/db/models/manager.py", line 134, in get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/usr/local/lib/python2.6/site-packages/django/db/models/query.py", line 449, in get_or_create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python2.6/site-packages/django/db/models/base.py", line 463, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/usr/local/lib/python2.6/site-packages/django/db/models/base.py", line 551, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "/usr/local/lib/python2.6/site-packages/django/db/models/manager.py", line 203, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/usr/local/lib/python2.6/site-packages/django/db/models/query.py", line 1576, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 910, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/util.py", line 40, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 114, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python2.6/site-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
django.db.utils.IntegrityError: (1062, "Duplicate entry '000xxxxxxxx' for key 2")
No idea how large the work is, but for similar problem I used transactions alongside savepoints - https://docs.djangoproject.com/en/dev/topics/db/transactions/#savepoints
so given something like:
transaction.auto_commit(False)
try:
for raw_input in streaming_filelikeobject.readline():
product = do_work(raw_input)
MyTable(**product).save()
except SomeFileIOError:
transaction.rollback()
else:
transaction.commit()
Another idea would be to have a batch_id column and assign that on the start of each batch.
For very large data sets, you could use something like Memcache/Redis for managing inventory.
transaction.auto_commit(False)
try:
for raw_input in streaming_filelikeobject.readline():
product = do_work(raw_input)
if redis_conn.sadd("my_input_set", product['some_unique_id']):
MyTable(**product).save()
except SomeFileIOError:
transaction.rollback()
else:
transaction.commit()
.sadd() is a Redis command that returns true if an element doesn't exist already in a redis set.
Please note I am typing this stuff of the top of my head so method django transaction method signatures may not be authoritative.
When I issue the command:
python manage.py syncdb --database=mydb
It shows the output as follows...
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table django_content_type
Traceback (most recent call last):
File "manage.py", line 14, in <module>
execute_manager(settings)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 459, in execute_manager
utility.execute()
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
return self.handle_noargs(**options)
File "/usr/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 110, in handle_noargs
emit_post_sync_signal(created_models, verbosity, interactive, db)
File "/usr/lib/python2.7/site-packages/django/core/management/sql.py", line 189, in emit_post_sync_signal
interactive=interactive, db=db)
File "/usr/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 172, in send
response = receiver(signal=self, sender=sender, **named)
File "/usr/lib/python2.7/site-packages/django/contrib/auth/management/__init__.py", line 35, in create_permissions
ctype = ContentType.objects.get_for_model(klass)
File "/usr/lib/python2.7/site-packages/django/contrib/contenttypes/models.py", line 42, in get_for_model
defaults = {'name': smart_unicode(opts.verbose_name_raw)},
File "/usr/lib/python2.7/site-packages/django/db/models/manager.py", line 134, in get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 442, in get_or_create
return self.get(**lookup), False
File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 361, in get
num = len(clone)
File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 85, in __len__
self._result_cache = list(self.iterator())
File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 291, in iterator
for row in compiler.results_iter():
File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 763, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 818, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.7/site-packages/django/db/backends/util.py", line 40, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 337, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.DatabaseError: no such table: django_content_type
I have a custom db router that is basically set to django's example, except i have a custom attribute on MY models so that it knows which database they default to. syncdb works when in my settings.py INSTALLED_APPS I comment out: django.contrib.auth and django.contrib.contenttypes. Had this problem for a while, but have been putting it off until now, when I need to start on authentication. If you want my db router I'll post also
I've explained a similar problem here:
django.db.utils.IntegrityError: (1062, "Duplicate entry '22-add_' for key 'content_type_id'")
You don't need to comment out contrib.auth and contrib.contenttypes. Just make sure all django models - users, sessions, permissions are only used in 1 database, which could be considered as a master.
This will not directly solve your problem, but can be a starting point when dealing with multiple db's and db routers. What you need to know is that each model has it's content type in the database.
The problem occur when django objects - user/session/permission are not restricted to a single database - then they are created into each database. And since a content-type makes a model unique, having content-types for a single type in multiple databases could lead to the problem explained in the other SO question above.
In Django 1.4 you have to launch this command in the top-level folder of the project (path_to_project/you_project NOT path_to_project/you_project/you_project)