I have the following raw SQL command that I am executing inside a Django migration file. I need to use raw SQL because Django does not support generated columns.
ALTER TABLE dockets_document ADD COLUMN search_vector tsvector GENERATED ALWAYS AS (
setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
setweight(to_tsvector('english', coalesce(f_arr2text(content),'')), 'B') ||
setweight(jsonb_to_tsvector('english', coalesce(tables), '["all"]'), 'C')
) STORED;
My models.py file has the following field:
search_vector = SearchVectorField(null=True)
This line triggers a migration that generates the column for me, then my custom migration applies the SQL. The custom migration fails because the column was already created (with a corresponding index) so ADD COLUMN returns the error ERROR: column "search_vector" of relation "dockets_document" already exists. I tried using ALTER COLUMN in place of ADD COLUMN but it did not work (ERROR: syntax error at or near "tsvector").
I tried removing the field from the models.py file but then Django doesn't know that the field exists and won't allow me to query against the column. And it doesn't feel right to remove it either.
How can I convert the existing null column into a GENERATED column instead?
You could do RunSQL and add state_operation as documented
The state_operations argument allows you to supply operations that are
equivalent to the SQL in terms of project state. For example, if you
are manually creating a column, you should pass in a list containing
an AddField operation here so that the autodetector still has an
up-to-date state of the model
in manually generated empty migration
Related
I am new to django.
I have changed some fields in my already created Django model. But It says this message when I try to apply migrations on it:
It is impossible to add a non-nullable field 'name' to table_name without specifying a default. This is because the database needs something to populate existing rows.
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
2) Quit and manually define a default value in models.py.
Although I have deleted the data of this table from database. I cannot set it's default value because the field has to store unique values. Do I need to delete my previous migration file related to that table?
I have applied data migrations, but still getting the same error when applying migrations again:
def add_name_and_teacher(apps, schema_editor):
Student = apps.get_model('app_name', 'Student')
Teacher = apps.get_model('app_name', 'Teacher')
for student in Student.objects.all():
student.name = 'name'
student.teacher = Teacher.objects.get(id=1)
student.save()
class Migration(migrations.Migration):
dependencies = [
('app', '0045_standup_standupupdate'),
]
operations = [
migrations.RunPython(add_name_and_teacher),
]
So, before you had a nullable field "name". This means that it's possible to have null set as that field's value.
If you add a not null constraint to that field (null=False), and you run the migrations, you will get an integrity error from the database because there are rows in that table that have null set as that field's value.
In case you just made two migrations where first you added a nullable field, but then remembered it mustn't be nullable and you added the not null constraint, you should simply revert your migrations and delete the previous migration. It's the cleanest solution.
You can revert by running python manage.py migrate <app name> <the migration that you want to keep>
Then you simply delete the new migrations and run python manage.py makemigrations again.
In case the migration with the nullable field was defined very early on and there is already data there and it's impossible to delete that migration, you will need to figure out how to populate that data. Since you say that there is also the unique constraint, you can't just provide a default because it will cause issues with that constraint.
My suggestion is to edit the migration file and add migrations.RunSQL where you write custom SQL code which will insert values to the field. Make sure you place the RunSQL operation before the operation that adds the not null constraint (it should be AlterField or AddConstraint) as they are run in order.
You could also use migrations.RunPython, but I prefer the RunSQL because future changes in the code might break your migrations which is a hassle to deal with.
Docs for RunSQL
I am working in Python, using Pandas to pull data from a TSV, convert it to a data frame, then syncing that data frame to a temp table in postgres using df.to_sql. That process works great.
However, once that table exists, I want to move all the rows from that table to the permanent table. The two tables will always be identical. The permanent table has a unique index, so if the id already exists it should update the row instead.
Here is my attempt to upsert all rows from one table to another:
WITH moved_rows AS (
DELETE FROM my_table_tmp a
RETURNING a.*
)
INSERT INTO my_table
SELECT * FROM moved_rows
ON CONFLICT ("unique_id") DO
UPDATE SET
Field_A = excluded.Field_A,
Field_B = excluded.Field_B,
Field_C = excluded.Field_C
Unfortunately, when I run this, I get this error:
psycopg2.errors.UndefinedColumn: column excluded.field_a does not exist
LINE 10: Field_A = excluded.Field_A,
^
HINT: Perhaps you meant to reference the column "excluded.Field_A".
But in fact, that column does exist. What am I missing here? I've tried removing Field_A from the set and then I get the same error with Field_B.
Answering my own question here - the issue is that Postgres ignores capitalization unless it's quoted.
This was not clear in the example I posted because I obscured the naming of the fields I was working with. I've updated them now to show the issue.
In order to fix this you need to wrap your field names with double quotes. E.g. "Field_A" = excluded."Field_A"
i am creating a database using SQLAlchemy and I Need to do Migration to my data as i am using df_sql function for converting my csv into dataframe and then to tables in sqlalchemy. As i do this i need to do Migration to add new column and values inside it and assign Primary and foreign key Features. I saw someting related to Alembic and flask but am not sure how to upgrade it as also am working on Jupyter. Any ideas of how i can update delete and assign keys to my tables would be very helpful. Done until the table creation.
metadata.tables.keys()
dict_keys(['table1', 'table2'])
I also tried directly to create a temp table and copy ist values and assinging Primary key but am getting error with my column names as it has Special characters so i cant create duplicate too. Rename property too doesnt work
Column: date
Column: time_stamp
Column: timeslices[5].profilerDataProcess[8]_C0[us]
Column: timeslices[4].profilerDataProcess[54]_C0[us]
Column: timeslices[4]profilerDataProcess[50]_C0[us]
Column: timeslices[4].profilerDataProcess[49]_C0[us]
Column: timeslices[0].profilerDataProcess[14]_C0[us]
In my database i have column:
currency = models.CharField(max_length=10, blank=True, null=True)
I want to change this column from CharField to IntegerField. So in models.py i change this:
currency = models.IntegerField(blank=True, null=True)
then i made migrations: python manage.py makemigrations and python manage.py migrate. After that actions it rise error:
django.db.utils.ProgrammingError: column "currency" cannot be cast automatically to type integer
HINT: Specify a USING expression to perform the conversion.
After that in pgAdmin3 console i made this changes:
ALTER TABLE my_table ALTER COLUMN currency TYPE integer USING (currency::integer);
But i still got that error, I tried to change all back, but error doesn't disappear. What i have to do to escape this error. Thank you
I think django migrations does not perform casting, I looked in the documentation but I did not find any thing about column casting.
if the existing data is not that important for you, you can delete the column and create a new one
first step remove currency from you model and apply migration
add again the currency with the new definition and apply again the migration
if you want to keep your data, you need to give your new column a different name and use the old column as a temporary column to hold the data during the transition.
Important: Postgresql is more strongly typed in recent versions, and as explained here some casting may not work in PosgreSQL unless it's explicitly done. And it required to be more specific about the type. So you have to make the right choice based on your values:
alter table my_table alter column currency type bigint using currency::bigint
or maybe:
alter table my_table alter column currency type numeric(10,0) using currency::numeric
It is a PSQL issue when changing from certain data types to others... I had a similar problem an I did something a bit hackey but it worked ... but only because I didn't have any important data in the column
1) delete the most recent migration for that app
2) delete/comment out the "column" on the object
3) migrate the app with the missing column
4) reinstate/uncomment the offending "column"
5) migrate again
this is all a long way to delete and recreate the column on the actual db without using sql ... figured I would share in case it might help someone down the road
I have an existing model in the database. I want to enhance it with an hstore field. I installed hstore Postgres extension, django-hstore app, changed the appropriate settings in the django project:
SOUTH_DATABASE_ADAPTERS = {'default': 'south.db.postgresql_psycopg2'}
DATABASES = {
'default': {
'ENGINE': 'django_hstore.postgresql_psycopg2',
...
I checked that the django app works with new settings -- it does. So I added the new field to one of the models:
data = hstore.DictionaryField(db_index=True)
Next step: db migration. And here I'm lost. When trying to create a migration for the new field, I get this:
The field 'Project.data' does not have a default specified, yet is NOT NULL.
Since you are adding this field, you MUST specify a default
value to use for existing rows. Would you like to:
1. Quit now, and add a default to the field in models.py
2. Specify a one-off value to use for existing columns now
What do I do here? Did I miss something? I didn't find any references to a default value (or null=True) in any django-hstore related article.
this message usually appears when South is trying to update your models on the database and finds existing rows on the table yo are trying to modify. In order to continue and create the new field on database you must specify a value for the existing rows of tha table you are migrating. What I usually do, if it is a development stage, I go for option number 2 and set the value to 0, {} empty dict, or even NULL, depending on the field type.
As already mentionned, when you hit 2, you can either go for an empty string ("") or fill the field the way it's stored:
? The field 'MyModel.data' does not have a default specified, yet is NOT NULL.
? Since you are adding this field, you MUST specify a default
? value to use for existing rows. Would you like to:
? 1. Quit now, and add a default to the field in models.py
? 2. Specify a one-off value to use for existing columns now
? Please select a choice: 2
? Please enter Python code for your one-off default value.
? The datetime module is available, so you can do e.g. datetime.date.today()
>>> "foo=>bar,foo2=>bar2"