When I Try to connect oracle server with SQLAlchemy. I'm getting this error.
NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:oracle.oracledb
from sqlalchemy.engine import create_engine
DIALECT = 'oracle'
SQL_DRIVER = 'oracledb'
USERNAME = 'username' #enter your username
PASSWORD = 'password' #enter your password
HOST = 'host url' #enter the oracle db host url
PORT = 1533 # enter the oracle port number
SERVICE = 'service name' # enter the oracle db service name
ENGINE_PATH_WIN_AUTH = DIALECT + '+' + SQL_DRIVER + '://' + USERNAME + ':' + PASSWORD +'#' + HOST + ':' + str(PORT) + '/?service_name=' + SERVICE
engine = create_engine(ENGINE_PATH_WIN_AUTH)
#test query
import pandas as pd
test_df = pd.read_sql_query('SELECT * FROM global_name', engine)
any different method to connect?
For completeness (since the answer is already in comments): with SQLAlchemy 1.4 add this to your top level script file:
import sys
import oracledb
oracledb.version = "8.3.0"
sys.modules["cx_Oracle"] = oracledb
and then proceed as if you were using cx_Oracle. The create_engine() should begin with oracle: like:
# SQLAlchemy 1.4 with python-oracledb or cx_Oracle
engine = create_engine('oracle://...
The sys.modules etc snippet isn't needed for SQLAlchemy 2.0. With this version create_engine() should begin with oracle+oracledb: like:
# SQLAlchemy 2.0 with python-oracledb
engine = create_engine('oracle+oracledb://...
These posts are good references:
Using python-oracledb 1.0 with SQLAlchemy, Pandas, Django and Flask
Using SQLAlchemy 2.0 (development) with python-oracledb for Oracle Database
Related
I try to connect to Azure SQL using SQLAlchemy in python, authenticating with JWT generated for App registration. Despite checking almost any combination of parameters and granting all potentially meaningful permissions, I still get following error:
[28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user ''. (18456)
I did the following:
Created App registration 'test-app'.
Executed on the db:
CREATE USER [test-app] FROM EXTERNAL PROVIDER;
EXEC sp_addrolemember N'db_datareader', N'test-app';
GRANT SELECT ON test_vw to [test-app];
Added 'test-app' to Contributor and Reader roles for the SQL Server.
Whitelisted IP in the portal - I'm able to log in through SSMS and 'AAD Universal with MFA' using personal account.
Generated token with (tenant ID matches the one used by the organization):
POST https://login.microsoftonline.com/9ff8761-1be4-4729-b88df-e421e19d92f/oauth2/v2.0/token HTTP/1.1
Content-Type: application/x-www-form-urlencoded
client_id=[client_id]
client_secret=[client_secret]
scope=https://database.windows.net/.default
grant_type=client_credentials
Used generated JWT token in following python code:
from sqlalchemy import create_engine
import urllib
import struct
server = "test-server.database.windows.net"
database = "test-db"
driver = "{ODBC Driver 17 for SQL Server}"
jwt_token = "<jwt_token>"
SQL_COPT_SS_ACCESS_TOKEN = 1256
exptoken = b""
for i in bytes(jwt_token, "UTF-8"):
exptoken += bytes({i})
exptoken += bytes(1)
tokenstruct = struct.pack("=i", len(exptoken)) + exptoken
params = urllib.parse.quote_plus(
f"Driver={driver};Server={server};Database={database};Encrypt=yes"
)
conn_str = f"mssql+pyodbc:///?odbc_connect={params}"
engine_azure = create_engine(
conn_str,
connect_args={"attrs_before": {SQL_COPT_SS_ACCESS_TOKEN: tokenstruct}},
echo=True,
)
with engine_azure.connect() as con:
rs = con.execute("SELECT TOP 100 * FROM test_vw")
for row in rs:
print(row)
Any idea what I've missed?
I registered an app and copied ClientID, TenantId, generated client secret and copied ClientSecret value.I created azure SQL database and copied servername, databasename in azure portal.
I tried to connect azure SQL database by using token by replacing ClientID, TenantId, ClientSecret:
I installed SQLALchemy and adal packages using
pip install sqlalchemy
pip install adal
Connecting to Azure SQL database using below code:
from sqlalchemy import create_engine
import adal
server = '<servername>'
database = '<databasename>'
client_id = '<ClientId>'
tenant_id = '<TenantId>'
authority_host_uri = 'https://login.microsoftonline.com'
authority_uri = authority_host_uri + '/' + tenant_id
resource_uri = 'https://database.windows.net/'
client_secret = 'NbH8Q~x8PsGqD4C~8HUGzLmql_iRS7dxOMv2Bcjj'
context = adal.AuthenticationContext(authority_uri)
token = context.acquire_token_with_client_credentials(resource_uri, client_id, client_secret)
conn_str = (
'Driver={ODBC Driver 17 for SQL Server};'
f'Server={server};'
f'Database={database};'
'Encrypt=yes;'
'TrustServerCertificate=yes;'
f'AccessToken={token["accessToken"]};'
)
engine = create_engine('mssql+pyodbc:///?odbc_connect={}'.format(conn_str))
with engine.connect() as conn:
result = conn.execute('SELECT * FROM your-table')
for row in result:
print(row)
I got below error:
I created new group and added to registered app by following below procedure:
Goto-->Azure Active directory-->Enterprise Applications
Select the app and go to--> Group/Users-->Add User/Group select the user/group and click on assign.
Set that user/group as admin of SQL server by clicking Set admin:
I tried to connect SQL database again:
It worked successfully for me kindly check from your end.
I wish to use sqlachemy with teradata dialect to push some csv into a table.
So far I wrote this :
import pandas as pd
from sqlalchemy import create_engine
user = '******'
pasw = '******'
host = 'FTGPRDTD'
DATABASE = 'DB_FTG_SRS_DATALAB'
# connect
td_engine = create_engine('teradata://'+ user +':' + pasw + '#'+ DBCNAME + ':1025/')
print ('ok step one')
print(td_engine)
# execute sql
df = pd.read_csv(r'C:/Users/c92434/Desktop/Load.csv')
print('df chargé')
df.to_sql(name= 'mdc_load', con = td_engine, index=False, schema = DATABASE,
if_exists='replace')
print ('ok step two')
This is the error message I get :
DatabaseError: (teradata.api.DatabaseError) (0, '[08001] [TPT][ODBC SQL Server Wire Protocol driver]Invalid Connection Data., [TPT][ODBC SQL Server Wire Protocol driver]Invalid attribute in connection string: DBCNAME.')
(Background on this error at: http://sqlalche.me/e/4xp6)
What I can I do ?
Hopefully you've solved this by now, but I had success with this. Looking at what you provided, it looks like the host information you set is not being used in the connection string. My example includes the dtype parameter, which I use to define the data type for each column so they don't show up as CLOB.
database = "database_name"
table = "mdc_load"
user = "user"
password = "password"
host = 'FTGPRDTD:1025'
td_engine = create_engine(f'teradata://{user}:{password}#{host}/?database={database}&driver=Teradata&authentication=LDAP')
conn = td_engine.connect()
data.to_sql(name=table, con=conn, index=False, if_exists='replace', dtype=destType)
conn.close()
The "teradata" dialect (sqlalchemy-teradata module) relies on a Teradata ODBC driver being separately installed on the client platform. If you have multiple ODBC drivers installed that include the word Teradata in the name (for example, because you installed TPT with the Teradata-branded drivers for other database platforms), you may need to explicitly specify the one to be used by appending an optional parameter to your connection string, e.g.
td_engine = create_engine('teradata://'+ user +':' + pasw + '#'+ DBCNAME + ':1025/?driver=Teradata Database ODBC Driver 16.20')
Alternatively, you could use the "teradatasql" dialect (teradatasqlalchemy module) which does not require ODBC.
I am trying to connect to an Azure database using SQLAlchemy in Python.
My code is the following:
engine_azure = \
create_engine('mssql+pyodbc://{Server admin login}:{password}#{Server name}.database.windows.net:1433/{AdventureWorksLT}', echo=True)
I get the following message:
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\connectors\pyodbc.py:92: SAWarning: No driver name specified; this is expected by PyODBC when using DSN-less connections
"No driver name specified; "
Then I run the following code:
print(engine_azure.table_names())
I get the following message:
DBAPIError: (pyodbc.Error) ('01S00', '[01S00] [Microsoft][ODBC Driver Manager] Invalid connection string attribute (0) (SQLDriverConnect)')
There are 2 issues with your connection string:
As per the SQLAlchemy documentation: The delimeters must be URL escaped when using a pass-through exact pyodbc string.
And you do not specify the sql driver name either.
You can use the code below, which works fine at my side:
import pyodbc
from sqlalchemy import create_engine
import urllib
params = urllib.parse.quote_plus \ # urllib.parse.quote_plus for python 3
(r'Driver={ODBC Driver 13 for SQL Server};Server=tcp:yourDBServerName.database.windows.net,1433;Database=dbname;Uid=username#dbserverName;Pwd=xxx;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;')
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine_azure = create_engine(conn_str,echo=True)
print('connection is ok')
print(engine_azure.table_names())
Test result:
And for the connection string, you can get it by going to azure portal -> your database -> connection strings(select the ODBC in this case):
This is what i use in Python3:
params = urllib.parse.quote_plus(
'Driver=%s;' % driver +
'Server=tcp:%s,1433;' % server +
'Database=%s;' % database +
'Uid=%s;' % username +
'Pwd={%s};' % password +
'Encrypt=yes;' +
'TrustServerCertificate=no;' +
'Connection Timeout=30;')
conn_str = 'mssql+pyodbc:///?odbc_connect=' + params
engine = create_engine(conn_str)
Python3 snippet that I am using with ODBC Driver 17 for SQL Server. Cost me some time to figure it all out, especially the driver version and params.
import urllib
from sqlalchemy import create_engine
driver = "{ODBC Driver 17 for SQL Server}"
server = "<server-name>.database.windows.net"
database = "<db-name>"
user = "<db-user>"
password = "<db-password>"
conn = f"""Driver={driver};Server=tcp:{server},1433;Database={database};
Uid={user};Pwd={password};Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;"""
params = urllib.parse.quote_plus(conn)
conn_str = 'mssql+pyodbc:///?autocommit=true&odbc_connect={}'.format(params)
engine = create_engine(conn_str, echo=True)
engine.execute("SELECT 1")
Furthermore, I needed to install the following drivers/tools on macOS:
brew install msodbcsql17 mssql-tools
Step 1: Install Azure SQL DB Drivers
Install the new version of SQL DB Drivers using official documentation:
Linux, MacOS, Windows
Major update to previous answers: use the last supported version of DB driver ODBC Driver 17 for SQL Server instead of outdated versions ODBC Driver 13 for SQL Server or versions without explicitly defined a version, e.g. SQL Server.
Step 2: Install sqlalchemy package
Just print in the terminal: pip install SQLAlchemy
Step 3: Wrapping specific DB logic in AzureDbConnection class
from dataclasses import dataclass
from typing import Dict, Any, Iterable
from pandas import DataFrame
from sqlalchemy import create_engine, inspect
import urllib
#dataclass(frozen=True)
class ConnectionSettings:
"""Connection Settings."""
server: str
database: str
username: str
password: str
driver: str = '{ODBC Driver 18 for SQL Server}'
timeout: int = 30
class AzureDbConnection:
"""
Azure SQL database connection.
"""
def __init__(self, conn_settings: ConnectionSettings, echo: bool = False) -> None:
conn_params = urllib.parse.quote_plus(
'Driver=%s;' % conn_settings.driver +
'Server=tcp:%s.database.windows.net,1433;' % conn_settings.server +
'Database=%s;' % conn_settings.database +
'Uid=%s;' % conn_settings.username +
'Pwd=%s;' % conn_settings.password +
'Encrypt=yes;' +
'TrustServerCertificate=no;' +
'Connection Timeout=%s;' % conn_settings.timeout
)
conn_string = f'mssql+pyodbc:///?odbc_connect={conn_params}'
self.db = create_engine(conn_string, echo=echo)
def connect(self) -> None:
"""Estimate connection."""
self.conn = self.db.connect()
def get_tables(self) -> Iterable[str]:
"""Get list of tables."""
inspector = inspect(self.db)
return [t for t in inspector.get_table_names()]
def dispose(self) -> None:
"""Dispose opened connections."""
self.conn.close()
self.db.dispose()
Major update to previous answers: do not forget to close the connection and dispose of the DB engine explicitly as soon as it stops being needed.
Enjoy!
Set connection settings and credentials using Azure DB blade on Azure Portal:
conn_settings = ConnectionSettings(
server='<db_server_name>',
database='<db_name>',
username='<user_name>',
password='***')
Open DB connection:
db_conn = AzureDbConnection(conn_settings)
db_conn.connect()
Test connection (for example, get available tables list), do other stuff, and close it finally:
try:
for t in db_conn.get_tables():
print(t)
# Do another DB-related stuff:
# ...
finally:
db_conn.dispose()
None of the solutions posted so far worked for me.
Instead, I had to specify Driver ({SQL Server}) which worked perfectly.
params = urllib.parse.quote_plus("DRIVER={SQL Server};SERVER=sqlhost.database.windows.net;DATABASE=pythonSQL;UID=username#sqldb;PWD=password56789")
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine_azure = create_engine(conn_str,echo=True)
Source:
https://gist.github.com/timmyreilly/f4a351eda5dd45aa9d56411d27573d7c
I am using falsk SQLAlchemy with MYSQL with all default configuration(pool size and timeout)
I don't understand why do I get out of DB connections from the MYSQL DB?
My server is not that heavy loaded.
Can someone please explain how the flask sql alchemy get and release DB connections?
If I have a pool thread of 20 on my apache mod_wsgi server that means by theory that i can have 20 db connection opens all the time and that's it no?
How flask sql alchemy handle close and restore those connections.
Thanks
try this code
import sqlalchemy
dbhost = 'localhost' #host name
dbuser = 'root' #mysql username
dbpass = 'admin' #mysql password
dbname = 'mytable' #database name
engine = sqlalchemy.create_engine('mysql://'+dbuser+':'+dbpass+'#'+dbhost ) # connect to server
DB_URI = 'mysql://' + dbuser + ':' + dbpass + '#' + dbhost + '/' +dbname
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI']=DB_URI
db = SQLAlchemy(app)
am trying to connect with MSSQL remotely which is in windows from ubuntu using sqlalchemy.I creted DSN like below
dbinfo.py:
username = 'XXX'
pw = 'XXX'
host = '190.122.12.214'
drivername = 'SQL Server'
database = 'XXX'
extra_param=''
and i mported the dbinfo.py file into db_handler.py :
import transaction
from z3c.saconfig import Session as SASession
from z3c.saconfig import EngineFactory
from zope import component
from zope.sqlalchemy import mark_changed
# sqlalchemy
import sqlalchemy as sa
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from redindia.loginpage import dbinfo
info = {
'username' : dbinfo.username,
'pw' : dbinfo.pw,
'host' : dbinfo.host,
'drivername' : dbinfo.drivername,
'database' : dbinfo.database,
'extra_param' : ''
}
drivername = str(info['drivername'])
username = str(info['username'])
host = str(info['host'])
database = str(info['database'])
extra_param = str(info['extra_param'])
def getDb():
pass
def getSession(testing=False):
try:
return SASession()
except component.ComponentLookupError:
pass
# construct url to open database
_testing_ = ''
if testing:
_testing_ = '_testing'
if info['pw'] != '':
DSN = drivername+'://'+username+':' + info['pw'] +'#'+host+'/'+database+_testing_+'?charset=utf8'+extra_param
else:
DSN = drivername+'://'+username+'#'+host+'/'+database+_testing_+'?charset=utf8'+extra_param
engine_factory = EngineFactory(DSN, pool_recycle=7200)
engine = engine_factory()
## create a global session
from z3c.saconfig import GloballyScopedSession
utility = GloballyScopedSession(bind=engine) # i think, without engine, it will find above provided one...
from z3c.saconfig.interfaces import IScopedSession
component.provideUtility(utility, provides=IScopedSession)
return SASession()
session = getSession()
engine = session.get_bind()
Base = declarative_base(engine)
Base.metadata.reflect()
tables = Base.metadata.tables
and then connecting details below mentioned
def callStoreProcedure(self):
form = self.request.form
area = form.get('Area')
session = getSession()
result = session.execute("select * from BBBB")
result_set = result.fetchall()
return result_set
and i configure ODBC connectivity settings
etc/odbc.ini:
[SQL Server]
Description=my dsn
Driver=SQL Server
Database=XXX
Servername=190.122.12.214
UID=XXX
PWD=XXX
etc/odbcinst.ini:
[SQL Server]
Description = sql Driver
Driver = /usr/local/lib/libtdsodbc.so
Setup=/usr/local/lib/libtdsS.so
UsageCount = 1
I configured the settings like above.But i can't able to connect MSSQL.am getting the error like below
"ArgumentError: Could not parse rfc1738 URL from string 'SQL Server://XXX:XXX#190.122.12.214/XXX?charset=utf8'"
Plz can anyone help me to solve this issues.Thanks in advance.
The SQLAlchemy engine URL should begin with either mssql or mssql+pyodbc. See the Engine Configuration documentation.
I created dbhandler.py file. It contains the details about the database connectivity.The details are below
db_handler.py:
from sqlalchemy import create_engine
def getSession(self):
DSN="mssql://UID:PWD#IPADDRESS/DBNAME"
return DSN
our .py file
from xxxx.yyyy.db_handler import getSession
from sqlalchemy import create_engine
def callStoreProcedure(self):
form = self.request.form
DSN = getSession(self)
engine = create_engine(DSN)
cursor = engine.execute("select * from tablename")
result = cursor.fetchall()
return result
Now i have connected with the database.