I need to delete fields from an existing django model that already have a few objects associated with it. Deleting the fields from models.py gives me an error (obviously as there are table columns still associated with them). The data in body2 and body3 are not necessary for my app.
I have copied that data from those fields to the body field. How would I go about deleting these fields without dropping the table entirely?
class Post(models.Model):
#some fields
body =EditorJsField(editorjs_config=editorjs_config)
body2 =EditorJsField(editorjs_config=editorjs_config)
body3 =EditorJsField(editorjs_config=editorjs_config)
I deleted body2 and body3 and ran migrations and when creating a new object, I get errors such as this.
django.db.utils.IntegrityError: null value in column "body2" of relation "second_posts" violates not-null constraint
DETAIL: Failing row contains (20, Wave | Deceptiveness, and unpredictability of nature, 2021-07-19 13:40:32.274815+00, 2021-07-19 13:40:32.274815+00, {"time":1626702023175,"blocks":[{"type":"paragraph","data":{"tex..., null, null, Just how unpredictable is nature? Nature is all around us, yet, ..., image/upload/v1626702035/dfaormaooiaa8felspqd.jpg, wave--deceptiveness-and-unpredictability-of-nature, #66c77c, l, 1, 1, 0).
This is the code that I'm using to save the sanitized data(after I've deleted those fields of course.)
post = Posts.objects.create(
body=form.cleaned_data.get('body'),
#
)
Since nobody seemed to have an answer, and since it looked like this error was an anomaly, I went the non-python way and ran SQL queries and dropped the columns. For those of you who ran into the same problem,
Warning, you will lose all the data in the fields you would like to delete using this method
First, make Django aware of the changes
Delete the fields you want to be deleted and run migrations.
Before
class Post(models.Model):
#some fields
body =EditorJsField(editorjs_config=editorjs_config)
body2 =EditorJsField(editorjs_config=editorjs_config)
body3 =EditorJsField(editorjs_config=editorjs_config)
After
class Post(models.Model):
#some fields
body =EditorJsField(editorjs_config=editorjs_config)
Command Prompt
python manage.py makemigrations
python manage.py migrate
Drop the columns using SQL queries
First connect to your database(I used postgres). The name of the table should be something like appname_model. My app was called "Second" and the model was Post. So, the table was called second_post.
See if the columns still persist after the migrations using,
In the SQL command prompt
/d second_post
This should give you a nice diagram of the database with all the columns listed on the left side. To drop those columns, type,
ALTER TABLE second_post DROP COLUMN body2;
ALTER TABLE second_post DROP COLUMN body3;
After entering each query, the prompt should return a string ALTER TABLE if successful.
If you want to drop the data completely, you need to create a Django migration (using ./manage.py makemigrations preferably) that will remove those columns from the database.
Alternatively, if you want to play safe and persist the old data, you can first make those fields as nullable, then create migrations for them and at the end either just don't use them anymore or remove those columns from the model, but don't reflect it in migrations (you'll need to fake the removal of those columns in migrations though if you'll ever need to run another migration in this app).
Related
I am new to django.
I have changed some fields in my already created Django model. But It says this message when I try to apply migrations on it:
It is impossible to add a non-nullable field 'name' to table_name without specifying a default. This is because the database needs something to populate existing rows.
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
2) Quit and manually define a default value in models.py.
Although I have deleted the data of this table from database. I cannot set it's default value because the field has to store unique values. Do I need to delete my previous migration file related to that table?
I have applied data migrations, but still getting the same error when applying migrations again:
def add_name_and_teacher(apps, schema_editor):
Student = apps.get_model('app_name', 'Student')
Teacher = apps.get_model('app_name', 'Teacher')
for student in Student.objects.all():
student.name = 'name'
student.teacher = Teacher.objects.get(id=1)
student.save()
class Migration(migrations.Migration):
dependencies = [
('app', '0045_standup_standupupdate'),
]
operations = [
migrations.RunPython(add_name_and_teacher),
]
So, before you had a nullable field "name". This means that it's possible to have null set as that field's value.
If you add a not null constraint to that field (null=False), and you run the migrations, you will get an integrity error from the database because there are rows in that table that have null set as that field's value.
In case you just made two migrations where first you added a nullable field, but then remembered it mustn't be nullable and you added the not null constraint, you should simply revert your migrations and delete the previous migration. It's the cleanest solution.
You can revert by running python manage.py migrate <app name> <the migration that you want to keep>
Then you simply delete the new migrations and run python manage.py makemigrations again.
In case the migration with the nullable field was defined very early on and there is already data there and it's impossible to delete that migration, you will need to figure out how to populate that data. Since you say that there is also the unique constraint, you can't just provide a default because it will cause issues with that constraint.
My suggestion is to edit the migration file and add migrations.RunSQL where you write custom SQL code which will insert values to the field. Make sure you place the RunSQL operation before the operation that adds the not null constraint (it should be AlterField or AddConstraint) as they are run in order.
You could also use migrations.RunPython, but I prefer the RunSQL because future changes in the code might break your migrations which is a hassle to deal with.
Docs for RunSQL
I've got a Django model like the following..
class ExampleModel(models.Model):
name = models.CharField(...)
related_user = models.ForeignKey(UserTypeA, related_name='related_example', blank=True, null=True, on_delete=models.SET_NULL)
where I recently had to make a change to the related_user field by changing the ForeignKey from UserTypeA to UserTypeB.
Of course, this raises an error in the terminal when I attempt to python manage.py makemigration...
django.db.utils.IntegrityError: insert or update on table "models_examplemodel" violates foreign key constraint "models_examplemodel_related_user_id_ac0c6018_fk_accounts_"
DETAIL: Key (related_user_id)=(13) is not present in table "accounts_usertypea".
What's the safest way to go about making these changes? Currently I'm in development so I'm happy to delete my data/migrations/whatever, but I imagine in production this would be difficult.
The ideal behaviour I'd like to see here is the relations from ExampleModel and UserTypeA just being deleted, and so the current relationships would be set to NULL. Thoughts?
if you simply want to drop UserTypeA and use UserTypeB with None values simply do this:
remove related_user field
generate migrations
add related_user field
generate migrtions
If you want to do something more complecated (fill UserTypeB based on UserTypeA) these are the steps
add realted_user_b field with default as null
generate migration file
write a data migrations file which fills realted_user_b based on current data docs
remove realted_user field
generate migration file
rename realted_user_b to realted_user
generate migration file
In my database i have column:
currency = models.CharField(max_length=10, blank=True, null=True)
I want to change this column from CharField to IntegerField. So in models.py i change this:
currency = models.IntegerField(blank=True, null=True)
then i made migrations: python manage.py makemigrations and python manage.py migrate. After that actions it rise error:
django.db.utils.ProgrammingError: column "currency" cannot be cast automatically to type integer
HINT: Specify a USING expression to perform the conversion.
After that in pgAdmin3 console i made this changes:
ALTER TABLE my_table ALTER COLUMN currency TYPE integer USING (currency::integer);
But i still got that error, I tried to change all back, but error doesn't disappear. What i have to do to escape this error. Thank you
I think django migrations does not perform casting, I looked in the documentation but I did not find any thing about column casting.
if the existing data is not that important for you, you can delete the column and create a new one
first step remove currency from you model and apply migration
add again the currency with the new definition and apply again the migration
if you want to keep your data, you need to give your new column a different name and use the old column as a temporary column to hold the data during the transition.
Important: Postgresql is more strongly typed in recent versions, and as explained here some casting may not work in PosgreSQL unless it's explicitly done. And it required to be more specific about the type. So you have to make the right choice based on your values:
alter table my_table alter column currency type bigint using currency::bigint
or maybe:
alter table my_table alter column currency type numeric(10,0) using currency::numeric
It is a PSQL issue when changing from certain data types to others... I had a similar problem an I did something a bit hackey but it worked ... but only because I didn't have any important data in the column
1) delete the most recent migration for that app
2) delete/comment out the "column" on the object
3) migrate the app with the missing column
4) reinstate/uncomment the offending "column"
5) migrate again
this is all a long way to delete and recreate the column on the actual db without using sql ... figured I would share in case it might help someone down the road
I have a model that has one column right now, description, that populates an area on a few different pages of the site. The client wants this split up into a couple different columns, so they can populate different values in certain parts of the site. So I have to change that description column to frontpage_description and resourcepage_description. I am trying to come up with a way to do this in South so that the value of the description column is the (initial) default value for both of the "new" columns. This is what I have so far:
# file: project/app/xxxx_mymigration.py
import datetime
from south.db import db
from south.v2 import SchemaMigration
from django.db import models
class Migration(SchemaMigration):
def forwards(self, orm):
db.rename_column('myapp', 'description', 'frontpage_description')
db.add_column('myapp', 'resourcepage_description', self.gf('myfields.TextField')(default='CHANGEME'), keep_default=False)
def backwards(self, orm):
db.rename_column('myapp', 'frontpage_description', 'description')
db.delete_column('myapp', 'resourcepage_description')
models = {
# ...
The part I am wondering about is the self.gf(...)(default='CHANGEME'), I am wondering if there is a way to set the value of description or frontpage_description to be the default value for resourcepage_description?
I have looked into the orm parameter, which does allow you to access models during the migration, but all the examples I have come across involve using it to define relations during a migration, and not actually accessing individual records.
Am I just going to have to split this up into a schemamigration and a datamigration?
This is exactly a three-step: schema migration to add the columns, a data migration to set them, and another schema migration to delete the original description column. 90% of that is done for you from the ./manage command, so it's not as if this is tragically more work.
I'm creating a set of SQL full database copy scripts using MySQL's INTO OUTFILE and LOAD DATA LOCAL INFILE.
Specifically:
SELECT {columns} FROM {table} INTO OUTFILE '{table}.csv'
LOAD DATA LOCAL INFILE '{table}.csv' REPLACE INTO {table} {columns}
Because of this, I don't need just the tables, I also need the columns for the tables.
I can get all of the tables and columns, but this doesn't include m2m tables:
from django.db.models import get_models()
for model in get_models():
table = model._meta.db_table
columns = [field.column for field in model._meta.fields]
I can also get all of the tables, but this doesn't give me access to the columns:
from django.db import connection
tables = connection.introspection.table_names()
How do you get every table and every corresponding column on that table for a Django project?
More details:
I'm doing this on a reasonably large dataset (>1GB) so using the flat file method seems to be the only reasonable way to make this large of a copy in MySQL. I already have the schema copied over (using ./manage.py syncdb --migrate) and the issue I'm having is specifically with copying the data, which requires me to have the tables and columns to create proper SQL statements. Also, the reason I can't use default column ordering is because the production database I'm copying from has different column ordering than what is created with a fresh syncdb (due to many months worth of migrations and schema changes).
Have you taken a look at manage.py ?
You can get boatloads of SQL information, for example to get all the create table syntax for an app within your project you can do:
python manage.py sqlall <appname>
If you type:
python manage.py help
You can see a ton of other features.
I dug in to the source to find this solution. I feel like there's probably a better way, but this does the trick.
This first block gets all of the normal (non-m2m) tables and their columns
from django.db import connection
from django.apps import apps
table_info = []
tables = connection.introspection.table_names()
seen_models = connection.introspection.installed_models(tables)
for model in apps.get_models():
if model._meta.proxy:
continue
table = model._meta.db_table
if table not in tables:
continue
columns = [field.column for field in model._meta.fields]
table_info.append((table, columns))
This next block was the tricky part. It gets all the m2m field tables and their columns.
for model in apps.get_models():
for field in model._meta.local_many_to_many:
if not field.creates_table:
continue
table = field.m2m_db_table()
if table not in tables:
continue
columns = ['id'] # They always have an id column
columns.append(field.m2m_column_name())
columns.append(field.m2m_reverse_name())
table_info.append((table, columns))
Have you looked into "manage.py dumpdata" and "manage.py loaddata"? They dump and load in json format. I use it to dump stuff from one site and overwrite another site's database. It doesn't have an "every database" option on dumpdata, but you can call it in a loop on the results of a "manage.py dbshell" command.