I am running into a weird error trying to append dataframe to MySQL table using pandas to_sql function. I have not been able to find answer to this anywhere. Here is a test example:
test_df = pd.DataFrame(['a','b','c','d'], columns = ['char'])
with engine.begin() as connection:
test_df.to_sql(name='test', con=connection, if_exists='append')
The above runs successfuly, I can see the table being created in my database.
new_df = pd.DataFrame(['e','f'], columns = ['char'])
with engine.begin() as connection:
new_df.to_sql(name='test', con=connection, if_exists='append')
However, when I try to append more data, I get following error:
OperationalError: (MySQLdb._exceptions.OperationalError) (1050, "Table 'test' already exists")
[SQL:
CREATE TABLE test (
`index` BIGINT,
`char` TEXT
)
]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
This is very confusing. Since I did not see anyone encounter this error, could it be a bad installation of my packages. Any help will be greatly appreciated.
Thanks to the comment by Rouhollah. I made the "append' to work by replacing
engine = create_engine(f"mysql://{user}:{password}#{host}:{port}")
with
engine = create_engine(f"mysql://{user}:{password}#{host}:{port}/{database}")
previously, I was accessing database using engine.execute(f"USE {database}") which seems to break the append function.
Related
In my snowflake db, I have already created tables. I used the sqlalchemy inspector library to check if it was created.
engine = create_engine("snowflake://my_connection_string")
db = {}
metadata = MetaData(bind=engine)
inspector = inspect(engine)
schemas = inspector.get_schema_names()
for schema in schemas:
print(f"schema: {schema}")
for table_name in inspector.get_table_names(schema=schema):
print(f"table name: {table_name}")
db[f"{schema}.{table_name}"] = Table(f"{schema}.{table_name}", metadata)
print("Columns: ", end=': ')
for column in inspector.get_columns(table_name, schema=schema):
print(f"{column['name']}", end=',')
print()
However, I'm having trouble fetching those tables using sqlalchemy metadata:
engine = create_engine("snowflake://my_connection_string")
meta_data = MetaData(bind=engine)
MetaData.reflect(meta_data)
when I run the above code I get the following error:
ProgrammingError: (snowflake.connector.errors.ProgrammingError) 001059 (22023): SQL compilation error:
Must specify the full search path starting from database for TEST_DB
[SQL: SHOW /* sqlalchemy:_get_schema_primary_keys */PRIMARY KEYS IN SCHEMA test_db]
(Background on this error at: https://sqlalche.me/e/14/f405)
Any suggestions would be much appreciated!
Hello and thank you for taking the time to read this. For days I'm figuring out why I get this error when I try to load Account data towards an mssql database. The connection is fine.
But I keep on getting these errors:
(pyodbc.ProgrammingError) ('Invalid parameter type. param-index=17 param-type=collections.OrderedDict', 'HY105')
Exception: (102, b"Incorrect syntax near 'Id'.DB-Lib error message 20018, severity 15:\nGeneral SQL Server error: Check messages from the SQL Server\n")
Exception: One or more values in the dataframe have more characters than possible in the database table. The maximum number of characters in each column are:
How can I circumvent these errors and load the data without errors:
I use this for instance:
engine = sal.create_engine('mssql+pyodbc:///?odbc_connect={}'.format(params))
conn = engine.connect()
for entity in ['Account']:
df = get_salesforce_data(sf=sf, sf_object=entity, method=method)
df.to_sql(entity, con = engine, if_exists ='append', index = False, chunksize = 1000)
There 94 columns in this Account table?
Thank you for thinking with me
Im trying to make a join on 2 databases in MSSQL.
here is the SQL query:
SELECT od.Indice, cs.Argued
FROM HN_Ondata.dbo.ODCalls as od
JOIN HN_ADMIN.dbo.CallStatus as cs ON od.CallStatusGroup = cs.StatusGroup
I have tried:
create two engines making the tables with autoload and query it.
create two engines opening two session and making a subquery.
create two engines create a CTE of table2.
create a metadata bind to database1 reflect table1 then call reflect(bind=database2) for table2
always end up with this error:
pymssql.ProgrammingError: (208, b"Invalid object name 'CallStatus'.DB-Lib error message 20018, severity 16:\nGeneral SQL Server error: Check messages from the SQL Server\n")
The current solution i got is using session.execute and write down raw sql, i could stick with it but im currious is there any way doing that with sqlalchemy ORM ?
EDIT 1:
Here is my code :
db1 = DatabaseManager(settings.DATABASE['hrm'], database='HN_Ondata')
db2 = DatabaseManager(settings.DATABASE['hrm'], database='HN_ADMIN')
metadata1 = MetaData(bind=db1.database)
metadata2 = MetaData(bind=db2.database)
table1 = Table('ODCalls', metadata1, autoload=True)
table2 = Table('CallStatus', metadata2, autoload=True)
with db1.session(raise_err=True) as session:
result = (
session
.query(table1.c.Indice, table2.c.Argued)
.join(table2, table1.c.CallStatusGroup == table2.c.StatusGroup)
.all()
)
who produce the following query:
SELECT [ODCalls].[Indice] AS [ODCalls_Indice], [CallStatus].[Argued] AS [CallStatus_Argued]
FROM [ODCalls]
JOIN [CallStatus] ON [ODCalls].[CallStatusGroup] = [CallStatus].[StatusGroup]
Found the solution thank's to Ryan Gadsdon and Ilja Everilä pointing me the way.
You need to precise database.schema in Table schema parameters like this:
table1 = Table('ODCalls', metadata1, autoload=True, schema='HN_Odcalls.dbo')
Specify Database in schema is needed only if the table refer to a database who your engine is not connected, if you precise database.schema in schema parameters you can then use the table with any engine connected to any database on the same server.
http://docs.sqlalchemy.org/en/latest/dialects/mssql.html#multipart-schema-names
I'm trying to connect to one our our internal databases using the following code:
engine = create_engine('postgresql+psycopg2://{user}:{passw}#{host}:{port}/{db}'.format(
user=config3.CANVAS_USERNAME,
passw=config3.CANVAS_PWD,
host=config3.CANVAS_BOUNCER,
port=config3.CANVAS_PORT,
db='cluster17dr'
))
metadata = MetaData()
metadata.reflect(bind=engine)
print(metadata.tables)
And my only result is a table called 'spatial_ref_sys', which I assume is some kind of metadata. I know that my login stuff is correct, because this works perfectly:
with ppg2.connect(
database='cluster17dr',
user=config3.CANVAS_USERNAME,
password=config3.CANVAS_PWD,
host=config3.CANVAS_BOUNCER,
port=config3.CANVAS_PORT) as conn:
cur = conn.cursor()
sql = 'SELECT * FROM canvas.switchman_shards LIMIT 10'
cur.execute(sql)
res = cur.fetchall()
print(res)
Any ideas as to what I'm missing in my connection using SQLAlchemy?
By default, if no schema name is specified, SQLAlchemy will only give you tables under the default schema. If you want to reflect tables in a schema other than the default schema (which defaults to public in PostgreSQL), you need to specify the schema keyword to .reflect():
metadata.reflect(..., schema="canvas")
I'm new to Python. I'm executing this very basic query:
connection = psycopg2.connect("dbname='test' host='localhost' user='admin' password='pass' port='9100'")
cur = connection.cursor()
cur.execute("""SELECT id FROM pages WEHERE uri = %(uri)s""", {'uri': uri})
row = cur.fetchall()
and keep getting this error:
<class 'psycopg2.ProgrammingError'>
('syntax error at or near "uri"\nLINE 1: SELECT id FROM pages WEHERE uri = \'http://example.com/index.php...\n ^\n',)
uri is a string and has the value http://example.com/index.php
Could you please help me?? This is making me crazy
It should be:
cur.execute("""SELECT id FROM pages WHERE uri = %(uri)s""", {'uri': uri})
That is, it should be where instead of wehere. Since there is no function like wehere in SQL, the syntax error is thrown.
The error itself is self-explanatory. Next time, read the error message closely.