I have a fairly large code base in django, with several applications, and at some point I had a problem: every time I change or add models and try to do a migration, an error appears: django.db.utils.ProgrammingError: relation "appname_modelname" already exists. The problem has been going on for some time - the migration file is always created under the same number - 0114 (I could not find this file, by the way), and all new fixes are recorded in it along with the previous ones, which is why the problem grows, like snowball.
I did not want to delve into the solution and just manually removed all the innovations from the database - everything that caused the "already exists" conflict to appear. So, in order for the migration to succeed, I had to manually delete all models or table fields that I created after this problem appeared. But now I'm starting to work in prod with this code and fill it with data, and it is no longer possible to delete all the data from the corresponding tables. I have no idea why this problem appeared and how to solve it and would really appreciate your advice.
I tried to make a fake, but then, obviously, the fixes I need simply do not get into the database.
I have a hypothesis as to what caused this: I have three docker containers, one api, and two for celery. In the api container, I wrote the command in the bash file at startup
python3 /usr/srv/h_api/src/manage.py makemigrations && python3 /usr/srv/h_api/src/manage.py migrate
instead of
python3 /usr/srv/h_api/src/manage.py makemigrations
so I don't have to do it manually every time. I don't understand why, but judging by the coincidence of time, it seems that the problem started because of this. It's been weeks since I changed that line back, but the problem remains.
How do I commit previous changes to the database and have django create the next migration file numbered 0115?
It may be a bit risky but it has worked for me in the past. I suggest creating a copy of your project in another folder and trying this safely away from the original project. Also if you are using a postgresql database then just switch to a dummy database although it shouldn't make a difference but just to be on the safer side.
Inside of your migrations folder, inside the app folder, try deleting the all the files inside of pycache EXCEPT init.cpython-39.pyc and 0001_initial.cpython-39.pyc and inside of the migrations folder delete all files EXCEPT init.py and 0001_initial.py.
IMPORTANT: Delete from the pycache INSIDE of migrations NOT the one outside of it.
The approach below has often helped me out, if its a production database its good to backup and also your migrations folder before getting started.
Start by deleting your current applications migrations folder
temporarily add a field to your models.
Makemigrations then migrate
Delete the temporarily added field from your model
Makemigrations and migrate
Related
I am using Django and I have an app inside called "stuff". I decided I didn't need it anymore so I removed all usage of it from all my files, I removed it from settings.py under installed apps, and then I just deleted its entire folder.
Is that okay to do? So far everything seems to still be running fine and it no longer shows up in my admin panel which is what I want. I was reading some other posts on how to remove a Django App and they were doing a lot of extra things like this:
https://stackoverflow.com/questions/3329773/django-how-to-completely-uninstall-a-django-app#:~:text=To%20remove%20the%20app%20from,your%20PYTHONPATH%20where%20it%20resides.
Should I be worried something bad will happen down the road? I can makemigrations and changes and do everything perfectly fine currently.
In case the "stuff" app had some models, It is a good idea to remove the tables associated to them from the database. Although they should no interfer in the correct execution of your site.
The easiest way in my opinion is to do python manage.py migrate stuff zero, before doing all the operations you describe.
At this point, if you pushed your code to a production environment, it is probably better not to mesh with the database directly and leave it as it is.
In case you have not put your website in production, this is you don't have any data in your production database, then you don't have to worry about it since the app no longer exists.
I've never done that on production so I cannot tell you that in my experience all the other steps are not needed. But I don't fell your approach is discorageous.
Every time I do a: python manage.py runserver
And I load the site, python gets data and puts this in my database.
Even when I already filled some info in the database. Enough to get a view of what I am working on.
Now it is not loading the information I want and instead putting in new information to add to the database so it can work with some data.
What is the reason my data in the database is not being processed?
And how do I stop new data being loaded into the database.
May be it is happening due to migration file first sometimes when you migrate models into database query language with same number
python manage.py makemigrations 0001
This "0001" has to be changed everytime
To solve your problem once delete the migrations file and then again migrate all models and then try
Tell if this work
Can django delete/erease/clean data of single database/class in an app managing more disjunctive classes? Or is it safer to build separate app for each separate/disjunctive database?
I have found how to delete all data from all project databases, i.e.:
python manage.py flush
I was only surpruised that user IDs and class IDs have not been reset to 1, but previously used ID numbers stayed somewhere in project memory.
I have also found that it is possible to delete/clean all databases of signle app, e.g. here or here.
python manage.py migrate <app> zero
Here even IDs were reset. Does somebody know how to reset user IDs to start again from number 1?
But nowhere, I have found how to delete/erase/clean/reset signle database/class of the app with more than one database. I have only found for ruby that there it could be possible like:
Model.delete_all
or
Model.destroy_all
But nobody confirmed there, that commads really work. Will/Can it work also in django?
Kind regards, Rene
We have an online store web-app which is powered by django, postgresql and heroku.
For a specific campaign (you can think a campaign like a product to purchase), we have sold 10k+ copies successfully. Yet some of our users are encountered this error according to our Sentry reports. Common specification of these users is; none of them have address information before the purchase. Generally, users fill out address form right after registering. If they don't, they need to fill the form while purchasing the product and submit them together.
This is how the trace looks like:
OperationalError: cursor "_django_curs_140398688327424_146" does not exist
(66 additional frame(s) were not displayed)
...
File "store/apps/store_main/templatetags/store_form_filters.py", line 31, in render_form
return render_to_string('widgets/store_form_renderer.html', ctx)
File "store/apps/store_main/templatetags/store_form_filters.py", line 20, in render_widget
return render_to_string('widgets/store_widget_renderer.html', ctx)
File "store/apps/store_main/widgets.py", line 40, in render
attrs=attrs) + "<span class='js-select-support select-arrow'></span><div class='js-select-support select-arrow-space'><b></b></div>"
OperationalError: cursor "_django_curs_140398688327424_146" does not exist
So another weird common thing, there are exception messages between sql queries before the failure. You can see it in the image below:
I'm adding it if they are somehow related. What may also be related is, the users who get this error are the users who tries to purchase the campaign right after a bulk mailing. So, extensive traffic might be the reason yet we are also not sure.
We asked Heroku about the problem since they are hosting the postgres, yet they do not have any clue either.
I know the formal reason of this error is trying to reach the cursor after a commit. Since it is destroyed after transaction, trying to reach it cause this error yet I don't see this in our scenario. We are not touching the cursor in any way. What am I missing? What may produce this error? How to prevent it? Any ideas would be appreciated.
The reason for your error might be that you added fields to a model and forgot to makemigrations and migrate.
Have a look at this answer: When in run test cases then I will get this error: psycopg2.OperationalError: cursor "_django_curs_140351416325888_23" does not exist
If you're using django-pytest and have enabled the optimization --reuse-db and have made DB migrations between test runs you need to re-create the DB tables again.
pytest --create-db
Most likely you have forgotten to makemigrations and migrate them to the database.
If you are sure you did make the migrations and running python manage.py makemigrations and python manage.py migrate will not find the changes you made then the database and models are not in sync.
Sometimes this situation can be very frustrating if you have a big database. You will need to manually inspect your models.
To help out you can try this trick, which has been working for me.
Step 1 (delete all migrations from apps)
I am going to assume you are using the unix terminal. run sudo rm -rv */migrations/*. This removes all the migrations files and caches.
Step 2 (make the migration folders in each app)
run the command mkdir <app-folder>/migrations && touch <app-folder>/__init__.py. Replace with the name of the app that you have in the INSTALLED_APPS list in your default django settings file.
Step 3 (Make Migrations)
Here we populate the migrations folders in each app with migration files. Run python manage.py makemigrations. Run the second command. python manage.py migrate --fake. We are using the --fake flag because the data already exists in the database and so we do not want to populate name database tables that are already there or you will be greeted with the already exists error
If this did not work, then you will need to temper with some additional fields on the database. such as the django_migrations or a similarly named table. This is not recommended as there are other database tables that depend on it such as the django_contenttypes and this will throw you in a chain of related tables that you will manually inspect which is very painful.
as I usually don't do the up front design of my models in Django projects I end up modifying the models a lot and thus deleting my test database every time (because "syncdb" won't ever alter the tables automatically for you). Below lies my workflow and I'd like to hear about yours. Any thoughts welcome..
Modify the model.
Delete the test database. (always a simple sqlite database for me.)
Run "syncdb".
Generate some test data via code.
goto 1.
A secondary question regarding this.. In case your workflow is like above, how do you execute the 4. step? Do you generate the test data manually or is there a proper hook point in Django apps where you can inject the test-data-generating-code at server startup?\
TIA.
Steps 2 & 3 can be done in one step:
manage.py reset appname
Step 4 is most easily managed, from my understanding, by using fixtures
This is a job for Django's fixtures. They are convenient because they are database independent and the test harness (and manage.py) have built-in support for them.
To use them:
Set up your data in your app (call
it "foo") using the admin tool
Create a fixtures directory in your
"foo" app directory
Type: python manage.py dumpdata --indent=4 foo > foo/fixtures/foo.json
Now, after your syncdb stage, you just type:
python manage.py loaddata foo.json
And your data will be re-created.
If you want them in a test case:
class FooTests(TestCase):
fixtures = ['foo.json']
Note that you will have to recreate or manually update your fixtures if your schema changes drastically.
You can read more about fixtures in the django docs for Fixture Loading
Here's what we do.
Apps are named with a Schema version number. appa_2, appb_1, etc.
Minor changes don't change the number.
Major changes increment the number. Syncdb works. And a "data migration" script can be written.
def migrate_appa_2_to_3():
for a in appa_2.SomeThing.objects.all():
appa_3.AnotherThing.create( a.this, a.that )
appa_3.NewThing.create( a.another, a.yetAnother )
for b in ...
The point is that drop and recreate isn't always appropriate. It's sometimes helpful to move data form the old model to the new model without rebuilding from scratch.
South is the coolest.
Though good ol' reset works best when data doesn't matter.
http://south.aeracode.org/
To add to Matthew's response, I often also use custom SQL to provide initial data as documented here.
Django just looks for files in <app>/sql/<modelname>.sql and runs them after creating tables during syncdb or sqlreset. I use custom SQL when I need to do something like populate my Django tables from other non-Django database tables.
Personally my development db is for a project I'm working on right now is rather large, so I use dmigrations to create db migration scripts to modify the db (rather than wiping out the db everytime like I did in the beginning).
Edit: Actually, I'm using South now :-)