Develop Reference

Connect to Oracle database from SQLAlchemy using pyodbc

I have set up a data source name(DSN) in ODBC driver and supplying that in a query.
My below code is working like a charm.
import pyodbc as db
cnxn = db.connect('DSN=Oracle Prod DW;PWD=******')
I want to create a sqlalchemy connection for the same, but I fail. I tried different approaches but it didn't work. I just want to supply a password and DSN.

Oracle dialect + ODBC Driver is not seem to be supported by SqlAlchemy
https://docs.sqlalchemy.org/en/13/core/engines.html#oracle
Only in Java Runtime you can do that apparently
https://docs.sqlalchemy.org/en/13/dialects/oracle.html#module-sqlalchemy.dialects.oracle.zxjdbc
https://www.jython.org/jython-old-sites/archive/21/docs/zxjdbc.html
That being said
If you have an oracle client installation with proper tnsnames setup
You can do something like follows
Install cx_Oracle
Setup tnsnames i.e.
DEVDB=
(DESCRIPTION =
(ADDRESS =(PROTOCOL =TCP)(HOST =10.10.10.11)(PORT =1521))
(CONNECT_DATA =
(SERVER =DEDICATED)
(SERVICE_NAME =SVCDEV)
)
)
Code
import sqlalchemy as alc
from sqlalchemy.orm import sessionmaker
import cx_Oracle
import pandas as pd
conn_str = 'oracle://DEVDB'
engine = alc.create_engine(conn_str, echo=False)
Session = sessionmaker(bind=engine)
# YOU MIGHT NEED THIS sometimes
# cx_Oracle.init_oracle_client(lib_dir=r"C:\oracle\x64\product\19.0.0\client_1\bin")
sess = Session()
result = sess.execute("select 'foo' from dual")
df = pd.DataFrame(result.fetchall(), columns=result.keys())
print(df.to_string())

Exporting pandas dataframe to MySQL using SQLAlchemy

I intend to export a pandas dataframe to MySQL using SQLAlchemy. Despite referring to all previous posts, I am unable to solve the issue:
import pandas as pd
import pymysql
from sqlalchemy import create_engine
df=pd.read_excel(r"C:\Users\mazin\1-601.xlsx")
cnx = create_engine('mysql+pymysql://[root]:[aUtO1115]#[localhost]:[3306]/[patenting in psis]', echo=False)
df.to_sql(name='inventor_dataset', con=cnx, if_exists = 'replace', index=False)
Following is the error:
OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect
to MySQL server on 'localhost]:[3306' ([Errno 11001] getaddrinfo
failed)")

After a lot of tinkering with the code and exploring different packages in Python, I was able to make the code work.
Code:
import mysql.connector
import sqlalchemy
database_username = 'root'
database_password = 'mysql'
database_ip = '127.0.0.1'
database_name = 'patenting_in_psi'
database_connection = sqlalchemy.create_engine('mysql+mysqlconnector://{0}:{1}#{2}/{3}'.
format(database_username, database_password,
database_ip, database_name), pool_recycle=1, pool_timeout=57600).connect()
df22.to_sql(con=database_connection, name='university_dataset_ca', if_exists='append',chunksize=100)
database_connection.close()

Python: SQL Alchemy Create Engine Syntax Issues

Let's say I have the following connection information for a MSSQL server:
'Driver={SQL Server};'
'Server=VCAB18RPACRGZ12\GNRSRZ11,1414;'
'Database=sampleDB;'
'uid=sampleID;'
'pwd=samplePW'
I want to write a python dataframe to the MSSQL server as a table. I have the following code:
from sqlalchemy import create_engine
connection = create_engine('mssql+pyodbc://sampleID:samplePW#myhost:VCAB18RPACRGZ12\GNRSRZ11,1414/sampleDB?driver=SQL+Server+Native+Client+10.0')
My above connection code is erroring out. I'm not sure exactly where my connection information is supposed to go in the create_engine statement.
This is my error ...
ValueError: invalid literal for int() with base 10:
'VCAB18RPACRGZ12\GNRSRZ11,1414'

Your Server Address is not correct.
If 1414 is the port#, you should use ":" instead of ",".
The SQLAlchemy uses pyodbc as the default DBAPI. pymssql is also available.
Below is the connection string sample:
# pyodbc -DSN
engine = create_engine('mssql+pyodbc://scott:tiger#mydsn')
# pymssql
engine = create_engine('mssql+pymssql://scott:tiger#hostname:port/dbname')
# pyodbc -DSN Less connection
from sqlalchemy import create_engine
#assumes driver name=[SQL+Server+Native+Client+10.0]
#engine = create_engine('mssql+pyodbc://username:password#hostname:port/databasename?driver=SQL+Server+Native+Client+10.0')
engine = create_engine(r'mssql+pyodbc://sampleID:samplePW#VCAB18RPACRGZ12\GNRSRZ11:1414/sampleDB?driver=SQL+Server+Native+Client+10.0')
print engine

How to load data into blaze from hive2

All,
I am attempting to load data into blaze from a hive2 thrift server. I would like to do some analysis similar to what is posted here. Here is my current process.
import blaze as bz
import sqlalchemy
import impala
conn = connect(host='myhost.url.com', port=10000, database='mydb', user='hive', auth_mechanism='PLAIN')
engine = sqlalchemy.create_engine('hive://', creator=conn)
data = bz.data(engine)
I am able to make the connection and generate the engine, but when I run bz.data it fails with the error
TypeError: 'HiveServer2Connection' object is not callable
Any help is appreciated.
Answer
from pyhive import import hive
import sqlalchemy
from impala.dbapi import import connect
def conn():
return connect(host='myhost.com', port=10000, database='database', user='username', auth_mechanism='PLAIN')
engine = sqlalchemy.create_engine('hive://', creator=conn)
#Workaround
import blaze as bz
data = bz.data(engine)

from pyhive import import hive
import sqlalchemy
from impala.dbapi import import connect
def conn():
return connect(host='myhost.com', port=10000, database='database', user='username', auth_mechanism='PLAIN')
engine = sqlalchemy.create_engine('hive://', creator=conn)
#Workaround
import blaze as bz
data = bz.data(engine)

I was having this same issue when using impyla to connect to Impala with SQLAlchemy. Making conn a function instead of assigning it to a variable worked.

impala connection via sqlalchemy

I'm new to hadoop and impala. I managed to connect to impala by installing impyla and executing the following code. This is connection by LDAP:
from impala.dbapi import connect
from impala.util import as_pandas
conn = connect(host="server.lrd.com",port=21050, database='tcad',auth_mechanism='PLAIN', user="alexcj", use_ssl=True,timeout=20, password="secret1pass")
I'm then able to grab a cursor and execute queries as:
cursor = conn.cursor()
cursor.execute('SELECT * FROM tab_2014_m LIMIT 10')
df = as_pandas(cursor)
I'd like to be able use sqlalchemy to connect to impala and be able to use some nice sqlalchemy functions. I found a test file in imyla source code that illustrates how to create an sqlalchemy engine with impala driver like:
engine = create_engine('impala://localhost')
I'd like to be able to do that but I'm not able to because my call to the connect function above has a lot more parameters; and I do not know how to pass those to sqlalchemy's create_engine to get a successful connection. Has anyone done this? Thanks.

As explained at https://github.com/cloudera/impyla/issues/214
import sqlalchemy
def conn():
return connect(host='some_host',
port=21050,
database='default',
timeout=20,
use_ssl=True,
ca_cert='some_pem',
user=user, password=pwd,
auth_mechanism='PLAIN')
engine = sqlalchemy.create_engine('impala://', creator=conn)

If your Impala is secured by Kerberos below script works (due to some reason I need to use hive:// instead of impala://)
import sqlalchemy
from sqlalchemy.engine import create_engine
connect_args={'auth': 'KERBEROS', 'kerberos_service_name': 'impala'}
engine = create_engine('hive://impalad-host:21050', connect_args=connect_args)
conn = engine.connect()
ResultProxy = conn.execute("SELECT * FROM db1.table1 LIMIT 5")
print(ResultProxy.fetchall())

import time
from sqlalchemy import create_engine, MetaData, Table, select, and_
ENGINE = create_engine(
'impala://{host}:{port}/{database}'.format(
host=host, # your host
port=port,
database=database,
)
)
METADATA = MetaData(ENGINE)
TABLES = {
'table': Table('table_name', METADATA, autoload=True),
}