I have a Falsk app that is using sqlalchemy-aurora-data-api the package to connect to MySQL db from the AWS RDS.
In the app I am using raw queries eg. f'SELECT * FROM customers WHERE id={id}'
And to execute this query I use
conn.execute(customer_query).one()
and this is the connection string that I use to connect to the dB, not using sessions here
conn = create_engine(f'mysql+auroradataapi://:#/{db_name}',
connect_args=dict(aurora_cluster_arn=cluster_arn, secret_arn=secret_arn)).connect()
There is a random error that comes unexpected
Some error occurred in fetching data from customers table (botocore.errorfactory.BadRequestException) An error occurred (BadRequestException) when calling the RollbackTransaction operation: Invalid transaction ID
(Background on this error at: https://sqlalche.me/e/14/dbapi)
I can't really figure out the solution to this problem.
How can I solve this problem?
Related
This question already has an answer here:
Read from the server failed (20004) (SQLExecDirectW)Read from the server failed (20004) (SQLExecDirectW)
(1 answer)
Closed last month.
I am using mssql server in ubuntu, and tries to access using pyodbc
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc:///?odbc_connects="Driver=;SERVER=;DATABASE=;UID=;PWD=;port=;TDS_Version=8.0")
using this statement, I got access to the database,
I tried to read the data from the table using sqlalchemy from flask
session.query(table1).filter(table1.col>4).all()
I got this error
sqlalchemy.exc.OperationalError: (pyodbc.OperationalError) ('08S01', '[08S01] [FreeTDS][SQL Server]Write to the server failed (20006) (SQLExecDirectW)')
The important note: this error happens randomly, which means, when executing this query, sometimes, I got the data and sometimes, I got this error
A few times, I got this error:
ERROR - sqlalchemy.pool.impl.QueuePool - Exception during reset or similar
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 502, in do_rollback
dbapi_connection.rollback()
pyodbc.OperationalError: ('08S01', '[08S01] [FreeTDS][SQL Server]Write to the server failed (20006) (SQLEndTran)')
What is the possible reason for this error? Thanks.
Can you try below! and let me know if it works!!
from your statement - when you say sometimes it works and sometimes it dosen't - does the same query work again when you re-run to reset the conection again by running the create_engine again. if yes then please try setting up connection like below and try
from sqlalchemy import create_engine
from sqlalchemy.engine import URL
connection_string = "DRIVER=;SERVER=;DATABASE=;UID=;PWD=;?autocommit=true"
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
engine = create_engine(connection_url)
I have added IMPORT URL from sqlalchemy, you can ignore it if you like (i prefer my connection this way)
now the main difference in my connection is , if you look at this line specifically
connection_string = "DRIVER=;SERVER=;DATABASE=;UID=;PWD=;?autocommit=true"
is using this in the end - autocommit=true - please give it a try.
I'm trying to connect to Teradata with SQLAlchemy and then save the results as pandas.DataFrame.
engine = create_engine(f"teradata://{UID}:{PWD}#host/?dsn={DSN}")
query = """SELECT TOP 10 * from db_5.mobile_sa"""
with engine.connect() as connect:
df = pd.read_sql(query,connect)
This results in an error:
sqlalchemy.exc.DatabaseError: (teradata.api.DatabaseError) (11470, '[HYC00] [Cloudera][ODBC] (11470) Transactions are not supported.') (Background on this error at: https://sqlalche.me/e/14/4xp6)
I've found that I should try to set autocommit to true, so I tried solutions from SQLAlchemy documentation and added isolation_level="AUTOCOMMIT" to create_engine() , but this time I got
TypeError: Invalid argument(s) 'isolation_level' sent to create_engine(), using configuration TeradataDialect/SingletonThreadPool/Engine. Please check that the keyword arguments are appropriate for this combination of components.
I also tried to add &autocommit=true to connection url, but that didn't seem to have an effect.
How do I go around it?
I'm learning tortoise and am trying to configure Postgres with register_tortoise looking like below:
register_tortoise(
app,
db_url="postgresql://postgres:password#localhost:5432/testdb",
modules={'models': ['db.models.users']},
generate_schemas=True,
add_exception_handlers=True
)
But I get this error
tortoise.exceptions.ConfigurationError: Unknown DB scheme: postgresql
ERROR: Application startup failed. Exiting.
Could someone help me figure out the error?
I have a simple test script to ensure I can connect to and read/write using GCF to Google Cloud SQL instance.
def update_db(request):
import sqlalchemy
# connect to GCP SQL db
user = 'root'
pw = 'XXX'
conn_name = 'hb-project-319121:us-east4:hb-data'
db_name = 'Homebase'
url = 'mysql+pymysql://{}:{}#/{}?unix_socket=/cloudsql/{}'.format(user,pw,db_name,conn_name)
engine = sqlalchemy.create_engine(url,pool_size=1,max_overflow=0)
# Returns
return f'Success'
This is successful but when I try to add a connection:
conn = engine.connect()
I get the following error:
pymysql.err.OperationalError: (1045, "Access denied for user 'root'#'cloudsqlproxy~' (using password: YES)")
Seems odd that engine can be created but no connection can be made to it? Worth noting that any kind of execute using the engine, e.g. engine.execute('select * from table limit 3') will also lead to the error above.
Any thoughts are appreciated. Thanks for your time
Cheers,
Dave
When the create_engine method is called, it creates the necessary objects, but does not actually attempt to connect to the database.
Per the SQLAlchemy documentation:
[create_engine] creates a Dialect object tailored towards [the database in the connection string], as well as a Pool object which will establish a DBAPI connection at [host]:[port] when a connection request is first received.
The error message you receive from MySQL is likely due to the fact that the special user account for the Cloud SQL Auth proxy has not yet been created.
You need to create a user in CloudSQL (MySQL) ('[USER_NAME]'#'cloudsqlproxy~%' or '[USER_NAME]'#'cloudsqlproxy~[IP_ADDRESS]' ).
The following code is working only using pyodbc -
conn_string = 'DRIVER='+driver+';SERVER='+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password
conn = pyodbc.connect(conn_string)
sql_query = 'SELECT * FROM table'
df = pd.read_sql(sql_query,conn)
However, when I try to use the pass through exact pyodbc string method of creating a sqlaclehmy engine, I get a permission error -
conn_string = 'DRIVER='+driver+';SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password
params = urllib.parse.quote_plus(conn_string)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
engine.connect()
Error -
SAWarning: Could not fetch transaction isolation level, tried views: ('sys.dm_exec_sessions',
'sys.dm_pdw_nodes_exec_sessions'); final error was: ('42000', '[42000] [Microsoft]
[ODBC Driver 17 for SQL Server][SQL Server]User does not have permission to perform this action.
(6004) (SQLExecDirectW)')
(I tried with and without the port included)
I'm confused on where this permission error is coming from because its working when only using pyodbc. is there some parameter im missing here?
Solution - https://github.com/catherinedevlin/ipython-sql/issues/105
I know this is an older issue, but since I just resolved a similar
issue, I'd thought I would share. This is an issue with SQL DW, not
the SQL Alchemy engine that ipython-sql uses.
I did notice that configuring SQL Magic to use autocommit doesn't seem
to work with MSSQL. Go ahead and add that to your connection string:
%sql mssql+pyodbc://user:pwd#MSSQL_DSN?autocommit=true
You should be able to connect using an Admin account. For general
users:
"Could not fetch transaction isolation level" is a SQL error. This
means the user does not have permissions to see what the current
isolation is. For Azure SQL Data Warehouse, those permissions are in
this view:
https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-exec-connections-transact-sql
Since VIEW SERVER STATE is not supported in SQL DW, you need to use
"GRANT VIEW DATABASE STATE TO user" instead:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/43a4f051-2a12-4a15-8362-3d7a67f5c88f/unable-to-give-permission-on-azure-sql-data-warehouse-catalog-views-and-dmvs?forum=AzureSQLDataWarehouse