I'm trying to connect to Teradata with SQLAlchemy and then save the results as pandas.DataFrame.
engine = create_engine(f"teradata://{UID}:{PWD}#host/?dsn={DSN}")
query = """SELECT TOP 10 * from db_5.mobile_sa"""
with engine.connect() as connect:
df = pd.read_sql(query,connect)
This results in an error:
sqlalchemy.exc.DatabaseError: (teradata.api.DatabaseError) (11470, '[HYC00] [Cloudera][ODBC] (11470) Transactions are not supported.') (Background on this error at: https://sqlalche.me/e/14/4xp6)
I've found that I should try to set autocommit to true, so I tried solutions from SQLAlchemy documentation and added isolation_level="AUTOCOMMIT" to create_engine() , but this time I got
TypeError: Invalid argument(s) 'isolation_level' sent to create_engine(), using configuration TeradataDialect/SingletonThreadPool/Engine. Please check that the keyword arguments are appropriate for this combination of components.
I also tried to add &autocommit=true to connection url, but that didn't seem to have an effect.
How do I go around it?
Related
I have a Falsk app that is using sqlalchemy-aurora-data-api the package to connect to MySQL db from the AWS RDS.
In the app I am using raw queries eg. f'SELECT * FROM customers WHERE id={id}'
And to execute this query I use
conn.execute(customer_query).one()
and this is the connection string that I use to connect to the dB, not using sessions here
conn = create_engine(f'mysql+auroradataapi://:#/{db_name}',
connect_args=dict(aurora_cluster_arn=cluster_arn, secret_arn=secret_arn)).connect()
There is a random error that comes unexpected
Some error occurred in fetching data from customers table (botocore.errorfactory.BadRequestException) An error occurred (BadRequestException) when calling the RollbackTransaction operation: Invalid transaction ID
(Background on this error at: https://sqlalche.me/e/14/dbapi)
I can't really figure out the solution to this problem.
How can I solve this problem?
First I query my cosmos data and put it into a dataframe. Then I connect to my database and try using to_sql:
params = urllib.parse.quote_plus(r'Driver=SQL Server;Server={SERVERNAME},1433;Database=cosmosTest;Trusted_Connection=yes;TrustServerCertificate=no;Connection Timeout=0;')
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine = create_engine(conn_str,echo=True)
conn = engine.connect()
df.to_sql('Document', conn, if_exists='replace', index = False)
When I try and run my script I run into this error message
('HY004', '[HY004] [Microsoft][ODBC SQL Server Driver]Invalid SQL data type (0) (SQLBindParameter)')
I'm not sure if I'm receiving this error due to something I'm doing wrong or if the data I'm querying from cosmos doesn't play well with SQL Server. I've checked a couple other posts about this issue, but they didn't really pertain to me. Any suggestions on what I can try here?
The following code is working only using pyodbc -
conn_string = 'DRIVER='+driver+';SERVER='+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password
conn = pyodbc.connect(conn_string)
sql_query = 'SELECT * FROM table'
df = pd.read_sql(sql_query,conn)
However, when I try to use the pass through exact pyodbc string method of creating a sqlaclehmy engine, I get a permission error -
conn_string = 'DRIVER='+driver+';SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password
params = urllib.parse.quote_plus(conn_string)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
engine.connect()
Error -
SAWarning: Could not fetch transaction isolation level, tried views: ('sys.dm_exec_sessions',
'sys.dm_pdw_nodes_exec_sessions'); final error was: ('42000', '[42000] [Microsoft]
[ODBC Driver 17 for SQL Server][SQL Server]User does not have permission to perform this action.
(6004) (SQLExecDirectW)')
(I tried with and without the port included)
I'm confused on where this permission error is coming from because its working when only using pyodbc. is there some parameter im missing here?
Solution - https://github.com/catherinedevlin/ipython-sql/issues/105
I know this is an older issue, but since I just resolved a similar
issue, I'd thought I would share. This is an issue with SQL DW, not
the SQL Alchemy engine that ipython-sql uses.
I did notice that configuring SQL Magic to use autocommit doesn't seem
to work with MSSQL. Go ahead and add that to your connection string:
%sql mssql+pyodbc://user:pwd#MSSQL_DSN?autocommit=true
You should be able to connect using an Admin account. For general
users:
"Could not fetch transaction isolation level" is a SQL error. This
means the user does not have permissions to see what the current
isolation is. For Azure SQL Data Warehouse, those permissions are in
this view:
https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-exec-connections-transact-sql
Since VIEW SERVER STATE is not supported in SQL DW, you need to use
"GRANT VIEW DATABASE STATE TO user" instead:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/43a4f051-2a12-4a15-8362-3d7a67f5c88f/unable-to-give-permission-on-azure-sql-data-warehouse-catalog-views-and-dmvs?forum=AzureSQLDataWarehouse
I am trying to write my python dataframe to a MSSQL server. The table design has already been created in the server.
Below is the code I am attempting to use. But I get an error.
import sqlalchemy as sa
import pyodbc
engine = sa.create_engine('mssql+pyodbc://username:password#server/database', echo = True)
# engine
df_EVENT5_14.to_sql("MODREPORT", engine)
This is the error I'm getting:
DBAPIError: (pyodbc.Error) ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)')
You have to use correct connection string as below
engine = sqlalchemy.create_engine('mssql+pyodbc://'+user+':'+password+'#'+host+':'+port+'/'+database+'?'\
+'driver=SQL+Server+Native+Client+11.0')
Please make sure that, put correct details into variables that used in connection string. You may use different driver as well according to your please change the driver, in my case I put SQL+Server+Native+Client+11.0.
I'm new to Python + Flask + Flask Appbuilder but I am a professional Java developer. I've been working on a small app that I initially used SqlLite and now I want to move into SQL Server, which will be the production database.
I can't seem to get the connection right.
I have tried using a DSN but I get an error message indicating there is a mismatch between the driver and something else (Python?). The searches on this error seem to indicate the driver is 32 bit and Python is 64. Still I can't get that working so I thought I'd try to connect directly. I'd prefer not using a DSN anyway. I've searched the web and can't find an example that works for me.
I have imported pyodbc. This is the current way I'm trying to connect:
params = urllib.quote_plus("DRIVER={SQL Server};SERVER=devsql07:1433;DATABASE=DevOpsSnippets;UID=<user>;PWD=<password>")
SQLALCHEMY_DATABASE_URI = "mssql+pyodbc:///?odbc_connect=%s" % params
This produces the following error message:
2016-02-17 07:11:38,115:ERROR:flask_appbuilder.security.sqla.manager:DB Creation and initialization failed: (pyodbc.Error) ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]Invalid connection. (14) (SQLDriverConnect); [01000] [Microsoft][ODBC SQL Server Driver][DBNETLIB]ConnectionOpen (ParseConnectParams()). (14)')
Can anyone help me get this connection correct?
I really appreciate any help.
if you are using pyodbc you should be able to connect this way
import pyodbc
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=yourServer;DATABASE=yourDatabase;UID=;PWD=')
#putting to use
SQL = "select Field1, Field2 from someTable"
cursor = cnxn.cursor()
cursor.execute(SQL)
row = cursor.fetchall()
for r in row:
print r[0] #field1
print r[1] #field2
The port should be specified through a comma. Specify the connection string as
DRIVER={SQL Server};SERVER=devsql07,1433;DATABASE.....