Develop Reference

How to VACUUM a SQLite database from within an Alembic migration?

I am using SqlAlchemy Alembic to perform DB migrations on a SQLite database. One of my migrations removes many redundant records and I would like to VACUUM the database after the deletion.
Here's how I'm trying to do this in my migration's upgrade() method:
def upgrade():
# deleting records here using op.execute()...
op.execute("VACUUM")
When the migration runs it fails with this error message:
E sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) cannot VACUUM from within a transaction
E [SQL: VACUUM]
E (Background on this error at: https://sqlalche.me/e/14/e3q8)```
The link only provides a rather general description of what an OperationalError is.
How can I overcome this error and VACUUM my database from within my migration?
Is there a way to exclude this specific command or this specific migration from running in a transaction?
PS - In general I would like my migrations to run in transactions so I would prefer not to change Alembic's default behavior (as set in env.py).

I was able to successfully execute the VACUUM command in my migration by wrapping it like so:
with op.get_context().autocommit_block():
op.execute("VACUUM")
This did not require any changes to env.py.

OperationalError: cursor "_django_curs_<id>" does not exist

We have an online store web-app which is powered by django, postgresql and heroku.
For a specific campaign (you can think a campaign like a product to purchase), we have sold 10k+ copies successfully. Yet some of our users are encountered this error according to our Sentry reports. Common specification of these users is; none of them have address information before the purchase. Generally, users fill out address form right after registering. If they don't, they need to fill the form while purchasing the product and submit them together.
This is how the trace looks like:
OperationalError: cursor "_django_curs_140398688327424_146" does not exist
(66 additional frame(s) were not displayed)
...
File "store/apps/store_main/templatetags/store_form_filters.py", line 31, in render_form
return render_to_string('widgets/store_form_renderer.html', ctx)
File "store/apps/store_main/templatetags/store_form_filters.py", line 20, in render_widget
return render_to_string('widgets/store_widget_renderer.html', ctx)
File "store/apps/store_main/widgets.py", line 40, in render
attrs=attrs) + "<span class='js-select-support select-arrow'></span><div class='js-select-support select-arrow-space'><b></b></div>"
OperationalError: cursor "_django_curs_140398688327424_146" does not exist
So another weird common thing, there are exception messages between sql queries before the failure. You can see it in the image below:
I'm adding it if they are somehow related. What may also be related is, the users who get this error are the users who tries to purchase the campaign right after a bulk mailing. So, extensive traffic might be the reason yet we are also not sure.
We asked Heroku about the problem since they are hosting the postgres, yet they do not have any clue either.
I know the formal reason of this error is trying to reach the cursor after a commit. Since it is destroyed after transaction, trying to reach it cause this error yet I don't see this in our scenario. We are not touching the cursor in any way. What am I missing? What may produce this error? How to prevent it? Any ideas would be appreciated.

The reason for your error might be that you added fields to a model and forgot to makemigrations and migrate.
Have a look at this answer: When in run test cases then I will get this error: psycopg2.OperationalError: cursor "_django_curs_140351416325888_23" does not exist

If you're using django-pytest and have enabled the optimization --reuse-db and have made DB migrations between test runs you need to re-create the DB tables again.
pytest --create-db

Most likely you have forgotten to makemigrations and migrate them to the database.
If you are sure you did make the migrations and running python manage.py makemigrations and python manage.py migrate will not find the changes you made then the database and models are not in sync.
Sometimes this situation can be very frustrating if you have a big database. You will need to manually inspect your models.
To help out you can try this trick, which has been working for me.
Step 1 (delete all migrations from apps)
I am going to assume you are using the unix terminal. run sudo rm -rv */migrations/*. This removes all the migrations files and caches.
Step 2 (make the migration folders in each app)
run the command mkdir <app-folder>/migrations && touch <app-folder>/__init__.py. Replace with the name of the app that you have in the INSTALLED_APPS list in your default django settings file.
Step 3 (Make Migrations)
Here we populate the migrations folders in each app with migration files. Run python manage.py makemigrations. Run the second command. python manage.py migrate --fake. We are using the --fake flag because the data already exists in the database and so we do not want to populate name database tables that are already there or you will be greeted with the already exists error
If this did not work, then you will need to temper with some additional fields on the database. such as the django_migrations or a similarly named table. This is not recommended as there are other database tables that depend on it such as the django_contenttypes and this will throw you in a chain of related tables that you will manually inspect which is very painful.

Error while running inspectdb in Django with MySQL Connector and Python 3

I'm going to post a question and then post the resolution to the problem, I've found the solution after being stuck for some time so I thought it might be valuable for other people using the command inspectdb to generate models in Django from a MySQL database, using MySQL Connector for Python 3.3
command:
C:\myproj\myproj> manage.py inspectdb --database=mydb > models.py
Resulting Error (Traceback omitted for brevity):
django.db.utils.IntegrityError: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''decimal' at line 1

So the solution is very quick, please note that the credits go to Marcin Miklaszewski who posted the bug and the related solution here: http://bugs.mysql.com/bug.php?id=72478
However I thought that it would have been easier to find here on StackOverflow.
There is an error in line 65 of the file (Assuming you have python installed in C:\Python33):
C:\Python33\lib\site-packages\mysql\connector\django\introspection.py
replace the line 65:
"table_schema = DATABASE() AND data_type='decimal", [table_name])"
with:
"table_schema = DATABASE() AND data_type='decimal'", [table_name])"
(note the missing apostrophe after the word decimal in the first version).
Now your inspectdb command will run correctly.

Schema Migration after dropping table

I added a column to my models.py and it was giving me issues. On the road to trying to solve the problems I've done a couple things.
Dropped the table: ./manage.py sqlclear app | ./manage.py dbshell
Tried to "reset" the schema: ./manage.py schemamigration app --initial
Tried to migrate: ./manage.py migrate app
After doing all these things, I get this error after trying to migrate:
FATAL ERROR - The following SQL query failed: CREATE TABLE "projects_project" ("id" integer NOT NULL PRIMARY KEY)
The error was: table "projects_project" already exists
Question: How do I repair my database? I don't care about any of the data in the db.
Edit:
One of the related posts took me to this link Django South - table already exists . Apparently if you fake the migration all is well.
./manage.py migrate myapp --fake
I'm still unsure of all the reprocussions of this but I guess thats what docs are for.

Welll, the error points it out pretty clearly:
table "projects_project" already exists
You can either do it the quick and dirty way and drop the table. In that case log into your DMBS. If it's MySQL, you'll just open the terminal and typ:
mysql -u root -p YOURPASSWORD
Then select the database:
use your_database;
Finally, drop the table:
DROP TABLE projects_project;
You should be able to migrate now.
The elegant way would be to undo the migration. But every framework has it's own way to do that. You need to figure that out first - or give us more information.

Django Migrating to a new Database

I just joined a project using Django, and am attempting to initialize my own development server. When I attempt to do so, the migration fails for one of my apps. A model for this app has a sorl.thumbnail.ImageField, to add a logo. When the migration is attempted, I get the following error message:
FATAL ERROR - The following SQL query failed: ALTER TABLE "accounts_account" ADD CONSTRAINT "logo_id_refs_file_ptr_id_7c3d1997" FOREIGN KEY ("logo_id") REFERENCES "filer_image" ("file_ptr_id") DEFERRABLE INITIALLY DEFERRED;
The error was: relation "filer_image" does not exist
I'm not sure what the problem could be, as apparently there was no problem in creating the production database or subsequent migrations.
I believe I have correctly installed sorl-thumbnail, and all of its dependencies.
Your help would be much apperciated.

I'm pulling the answer provided by Johndt6 from a comment to this answer for future search-ability.
The solution is to add filer to the INSTALLED_APPS tuple.