Develop Reference

Change VARCHAR length of columns created dynamically in SQL table using sqlalchemy

I need to create columns dynamically in table1 based on the values retrieved from the form. I am using sqlalchemy to do the same and using the below code:
engine.execute('ALTER TABLE %s ADD COLUMN %s %s;' % ('table1', col_name, "VARCHAR(100)"))
In the above statement:
table1: name of the table we are inserting the column dynamically.
col_name: string containing the column name that is to be inserted.
VARCHAR(100): column type
The above code runs without any error and the new column is added. However, all the columns being created have the datatype as VARCHAR(60) in the table. I need to increase the length of column. Also, I'm not using flask-sqlalchemy.
Any ideas what might be causing the problem.

PostgreSQL - create a generated column for a column that already exists

I have the following raw SQL command that I am executing inside a Django migration file. I need to use raw SQL because Django does not support generated columns.
ALTER TABLE dockets_document ADD COLUMN search_vector tsvector GENERATED ALWAYS AS (
setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
setweight(to_tsvector('english', coalesce(f_arr2text(content),'')), 'B') ||
setweight(jsonb_to_tsvector('english', coalesce(tables), '["all"]'), 'C')
) STORED;
My models.py file has the following field:
search_vector = SearchVectorField(null=True)
This line triggers a migration that generates the column for me, then my custom migration applies the SQL. The custom migration fails because the column was already created (with a corresponding index) so ADD COLUMN returns the error ERROR: column "search_vector" of relation "dockets_document" already exists. I tried using ALTER COLUMN in place of ADD COLUMN but it did not work (ERROR: syntax error at or near "tsvector").
I tried removing the field from the models.py file but then Django doesn't know that the field exists and won't allow me to query against the column. And it doesn't feel right to remove it either.
How can I convert the existing null column into a GENERATED column instead?

You could do RunSQL and add state_operation as documented
The state_operations argument allows you to supply operations that are
equivalent to the SQL in terms of project state. For example, if you
are manually creating a column, you should pass in a list containing
an AddField operation here so that the autodetector still has an
up-to-date state of the model
in manually generated empty migration

How to map data types from python pandas to postgres table?

I have a pandas.DataFrame with columns having different data types like object, int64 , etc.
I have a postgresql table created with appropriate data types. I want to insert all the dataframe data into postgresql table. How should manage to do this?
Note : The data in pandas is coming from another source so the
data types are not specified manually by me.

The easiest way is to use sqlalchemy:
from sqlalchemy import create_engine
engine = create_engine('postgresql://abc:def#localhost:5432/database')
df.to_sql('table_name', engine, if_exists='replace')
If the table exists, you can choose what you want to do with if_exists option
if_exists {‘fail’, ‘replace’, ‘append’}, default ‘fail’
If the table does not exist, it will create a new table with the corresponding datatypes.

Maybe you have the problem I had that you want to create new columns on the existing table, and then the solution to replace or append the table does not work for me. Shortly for me it looks like this ( I guess for the converting of datatypes is no general solution and you should adapt for your need):
lg.debug('table gets extended with the columns: '+",".join(dataframe.dtypes))
#check whether we have to add a field
df_postgres={'object':'text','int64':'bigint','float64':'numeric','bool':'boolean','datetime64':'timestamp','timedelta':'interval'}
for col in dataframe.columns:
#convert the columns to postgres:
if str(dataframe.dtypes[col]) in df_postgres:
dbo.table_column_if_not_exists(self.table_name,col,df_postgres[str(dataframe.dtypes[col])],original_endpoint)
else:
lg.error('Fieldtype '+str(dataframe.dtypes[col])+' is not configured')
and the function to create the columns:
def table_column_if_not_exists(self,table,name,dtype,original_endpoint=''):
self.query(query='ALTER TABLE '+table+' ADD COLUMN IF NOT EXISTS '+name+' '+dtype)
#make a comment when we know which source create this column
if original_endpoint!='':
self.query(query='comment on column '+table+'.'+name+" IS '"+original_endpoint+"'")

cannot get sqlalchemy and pandas (to_sql) to write an dataframe index date into a MySQL DB

I read from an API the following data into a pandas dataframe:
Now, I want to write this data into a MySQL-DB-table, using pandas to_sql:
In MySQL, the column is set up correctly, but has not written the values:
Then I looked in the debugger to show me the dataframe:
I thought it would maybe a formatting issue, and added the following lines:
In the debugger, it looks now fine:
But now, in the database, it wants to write the index column as text
... and interrupts the execution with an error:
Is there a way to get this going, aka to write df index data as date into a MySQL DB using pandas to_SQL in connection with a sqlalchemy engine?
Edit:
Table schema:
DataFrame Header:

It seems you are using Date column as primary key. I would suggest not to use that as primary key instead you should use Date + Ticker as primary key.

Use pandas.DataFrame.to_sql to update a JSON column in MySQL

I have a MySQL table with a JSON column.
I am able to write into this JSON field following either answer to this question,
but I have not yet found a solution to be able to update the json field with the same method.
The solution I tried:
out_df.to_sql('my_table', my_sql_alchemy_engine, if_exists='append')
can go 2 ways:
if there are no unique keys in my table, the rows are simply added at the end of the table, with the JSON field containing only the values I wanted to make an update with
If there are unique fields, i get an IntegrityError
Is there any way to make that work, or should I find another solution to update my JSON fields ?