I have been running SQL queries (client side) from DB2 databases using ibm_db & ibm_db_dbi with pandas. However our company implemented new security standards and I would need a way to secure the connection as well.
Running Python3.7 and DB2 10.5
Below is my current connection string:
import ibm_db
import ibm_db_dbi
import pandas as pd
driver = 'IBM DB2 ODBC DRIVER'
database = 'DB0001'
hostname = 'my.host.com'
port = '1234'
protocol = 'TCPIP'
uid = 'user'
pwd = 'password'
security = 'SSL'
dsn = (
f'DRIVER={driver};'
f'DATABASE={database};'
f'HOSTNAME={hostname};'
f'PORT={port};'
f'PROTOCOL={protocol};'
f'UID={uid};'
f'PWD={pwd};'
f'SECURITY={security};'
)
test_query = 'SELECT 1 FROM SYSIBM.SYSDUMMY1'
conn_engine = ibm_db.connect(dsn, '', '')
db_conn = ibm_db_dbi.Connection(conn_engine)
df = pd.read_sql(test_query, db_conn)
Is there any way to incorporate SSL for this code?
This is, unfortunately, a little complicated, and (hopefully) your DBA can help with some of this.
If you're using a Db2 10.5 Fixpack 5 (or newer) client, then you just need to add a couple of parameters in your DSN string:
Security=ssl;
SslServerCertificate=/path/to/file.arm;
Your DBA should be able to provide you with the SslServerCertificate file (or contents).
If you installed a Db2 client separately from the python ibm_db package, then depending on which Db2 client you have installed (i.e. providing the native libraries for ibm_db, you might need to install some additional libraries (the IBM GSKit libraries, which provide the SSL functionality for the Db2 client).
Try to set the following properties:
Security=SSL, SSLClientKeystoredb, SSLClientKeystoreDBPassword or SSLClientKeystash as described at the link.
With some newer versions of the Db2 driver and to connect to Db2 on Cloud and Db2 Warehouse on Cloud, addint just Security=SSL; to the list of parameters works.
Related
Need to use cx_oracle module with python 3.x version to connect Oracle 19c with TLS. There are firewalls and proxies.
How to implement it?
Do I have to use Oracle Wallets?
Is TLS 1.2 good enough or I need TLS 1.3 in 2022?
import cx_Oracle
conn = cx_Oracle.connect("uname/pwd#//localhost:1521/sd")
cur = conn.cursor()
cur.execute("SELECT 'Hello' FROM dual")
res = cur.fetchall()
If this is a cloud DB you no longer always need a wallet. With recent Oracle Client libraries you can use 1-way TLS. See Easy wallet-less connections to Oracle Autonomous Databases in Python.
I am using python's pyodbc library to connect with the Oracle Database. The authentication to my non-prod servers were set-up as Basic Authentication so my connection string worked well.
It was tough to understand and set up the right drivers initially, But then did manage to get through. Here is what the code looks like for connection and worked well for my other servers.
import textwrap
import pyodbc
connection_string = textwrap.dedent('''Driver={driver};
DBQ={hostname}:{port}/{sid};
UID={username};
PWD={password};
Connection Timeout=30;
Trusted_Connection="yes"
'''.format(driver = 'Oracle in instantclient_11_2',
hostname = <hostname>,
port = <port>,
sid = <sid>,
username = <username>,
password = <passwd>
))
connection = pyodbc.connect(connection_string)
To my surprise PROD is authenticated via LDAP only. To which I have the LDAP Server, Context, and DB Service along with the credentials.
I tried creating my connection string as
connection_string = textwrap.dedent('''Driver={driver};
DBQ={hostname}:{port}/{sid};
UID={username};
PWD={password};
Connection Timeout=30;
Authentication=LDAP;
Trusted_Connection="yes"
'''.format(driver = 'Oracle in instantclient_11_2',
hostname = <ldap_server>,
port = 389,
sid = <db_service>,
username = <username>,
password = <passwd>
))
To which I was not surprised to know it wont work. Tried going through many links but could not get through. Has anyone tried this before or please let me know what I must be missing.
I could definitely connect in SQL Developer using the LDAP Authentication. And since TNS Listener is not configured on this server I cannot use Basic Authentication to connect.
And help would be welcome on this.
Setting up Oracle Drivers - (This is a pre-requisite if exist please ignore)
Download Oracle Instant Client for Microsoft Windows (x64) 64-bit. Alternatively download available for your 32 bit system.
Choose your version of Oracle Database. My version was 11.2.0.4.0
Download below files -
i. Instant Client Package - Basic
ii. Instant Client Package - ODBC
Unzip both the packages into the same directory such as C:\oracle\instantclient_19_3.
Execute odbc_install.exe from the Instant Client directory. If Instant Client is 11g or lower, start the command prompt with the Administrator privilege.
Create C:\oracle\instantclient_19_3\network\admin folder to place tnsnames.ora, sqlnet.ora.
Set Environment variable - TNS_ADMIN to above folder location.
Add C:\oracle\instantclient_19_3 to PATH Environment variable.
Add C:\oracle\instantclient_19_3 to ORACLE_HOME (optional, if connection doesn't work try this.)
LDAP Authentication
Create two files: sqlnet.ora and ldap.ora
ldap.ora
# Place this file in the network/admin subdirectory or your
# $ORACLE_HOME location.
# LDAP Server name should be added here. Rest all the values remains unchanged.
DIRECTORY_SERVERS = (your-server.your-organization:389:636)
DEFAULT_ADMIN_CONTEXT = "ldap-ou-designation"
DIRECTORY_SERVER_TYPE = OID
sqlnet.ora
# Place this file in the network/admin subdirectory or your
# $ORACLE_HOME location.
SQLNET.AUTHENTICATION_SERVICES=(NTS)
NAMES.DIRECTORY_PATH = (LDAP)
Verify C:\oracle\instantclient_19_3 is present in the Path else try updating the path in the Python Code as below
import os
lib_dir=r"C:\oracle\instantclient_19_3"
os.environ["PATH"] = lib_dir + ";" + os.environ["PATH"]
With all this set-up in place you should have
username, password and DB Service name lets call it as db_service here
with cx_Oracle
import cx_Oracle
con = cx_Oracle.connect('{0}/{1}#{2}'.format(user, password, db_service))
version_script = "SELECT * FROM v$version"
cursor = con.cursor()
cursor.execute(version_script)
version = cursor.fetchall()
print(version[0][0])
Ouput
-----
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
with pyodbc
import pyodbc
con = pyodbc.connect('Driver={0};DBQ={1};UID={2};PWD={3}'.format(driver, db_service, user, password))
version_script = "SELECT * FROM v$version"
cursor = con.cursor()
cursor.execute(version_script)
version = cursor.fetchall()
print(version[0][0])
Ouput
-----
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
#Drivers for pyodbc
pyodbc.drivers()
Output
------
['ODBC Driver 17 for SQL Server',
'Oracle in instantclient_11_2']
References :
https://www.oracle.com/database/technologies/instant-client/winx64-64-downloads.html
https://www.oracle.com/database/technologies/releasenote-odbc-ic.html
https://eikonomega.medium.com/connecting-to-oracle-database-with-cx-oracle-and-ldap-5da7925a305c
Goal: Connect to remote MSSQL 2016 server via Python.
Main approach: Closely followed tutorial in https://github.com/mkleehammer/pyodbc/wiki/Connecting-to-SQL-Server-from-Mac-OSX .
Problem: Able to connect via tsql, but isql is not working. Errors
[S1000][unixODBC][FreeTDS][SQL Server]Unable to connect to data source
[37000][unixODBC][FreeTDS][SQL Server]Login failed for user 'DOMAIN\user-p'
[37000][unixODBC][FreeTDS][SQL Server]Cannot open database "TIT_BI_OPERATIONS" requested by the login. The login failed.
Things tried:
Different ODBC drivers 13.1, 17, FreeTDS
Inclusion/exclusion of escape character in the user name.
Host name vs host ip.
Settings:
odbc.ini
[ODS_DSN]
Description = Connection to ODS MS_SQL 2016
Driver = FreeTDS
Servername = ODS_DSN
Port = 40000
Database = TIT_BI_OPERATIONS
odbcinst.ini
[FreeTDS]
Driver=/usr/local/lib/libtdsodbc.so
Setup=/usr/local/lib/libtdsodbc.so
UsageCount=1
freetds.conf
[ODS_DSN]
host = 164.10.17.77
port = 40000
tds version = 7.4
client charset = UTF-8
Notes:
Even though, its not very promising to run python without connecting through tsql and isql first, i still tried without success. Using pyodbc, pypodbc, sqlalchemy.
Most errors in the form: Login failed for user 'DOMAIN\user-p'
For ODBC driver 13: Can't open lib '/usr/local/lib/libmsodbcsql.13.dylib'
I am able to connect via SQL PRO STUDIO, using exact same credentials.
If you have any thoughts which direction to go to climb out of this connection problem, it would be greatly appreciated. Thank you!
If you're using Windows domain auth, you'll have to use FreeTDS. Oddly enough, Windows domain auth isn't supported by the Microsoft ODBC Driver, only FreeTDS.
Since you can connect with the tsql command, that means FreeTDS is working. I'd recommend connecting directly from Python explicitly. Try a connection string like this:
import pyodbc
con = pyodbc.connect(
r"DRIVER={FreeTDS};"
r"SERVER=164.10.17.77;"
r"PORT=40000;"
r"DATABASE=TIT_BI_OPERATIONS;"
f"UID=DOMAIN\\user-p;"
f"PWD=yourpassword;"
r"TDS_Version=7.3;"
)
cursor = con.cursor();
cursor.execute("SELECT 'this' AS that")
for row in cursor.fetchall():
print(row)
Note that you do need two backslashes in the UID field to connect with Windows domain auth; that is not a typo!
I'm trying to connect to a DB2 database using pyodbc on Mac OS X and I can't seem to get this going. I have had success using FreeTDS to connect to MS SQL Server under similar conditions.
First, I installed db2exc_client_952_MAC_x86_64 from IBM. After successfully installing that, I went to add the driver via ODBC Administrator.
Then I've tried running this python:
import pyodbc
cnxn = pyodbc.connect('Driver={IBM DB2 ODBC Driver}; Hostname=myhost.com; Port=50300; Protocol=TCPIP; Database=DB2ET; CurrentSchema=SCHM1; UID=MYUID; PWD=MYPWD;');
And I receive this error:
Traceback (most recent call last):
File "", line 1, in
pyodbc.Error: ('HY000', '[] \xf8\x92\x90\x81\x9b\xf8\x93\x90\x81\x82\xf8\x96\xb0\x81\x9d\xf8\x93\x80\x81\x83\xf8\x88\x80\x81\x89\xf8\x9c\xa0\x81\x84\xf8\x9d\xa0\x81\xa9\xf8\x9c\xa0\x81\xa5\xf8\x88\x80\x81\x9d\xf8\x94\x90\x81\x93\xf8\x8c\x90\x81\x8c\xf8\x8c\xa0\x80\xb0\xf8\x93\xa0\x80\xb4\xf8\x88\x80\x80\xa0\xf8\x88\x80\x81\x81\xf8\x98\x90\x81\xa4\xf8\x98\x90\x81\xb4\xf8\x98\x90\x81\xa2\xf8\x99\x90\x81\xb3\xf8\x98\xb0\x80\xa0\xf8\x9b\xa0\x81\xaf\xf8\x99\x90\x81\xae\xf8\x9d\x80\x81\xa3\xf8\x9b\xb0\x81\xa9\xf8\x88\x80\x81\xae\xf8\x9b\xb0\x81\xa4\xf8\x9c\xb0\x81\xa5\xf8\x9b\xa0\x80\xa0\xf8\x9d\x80\x81\xaf\xf8\x99\x90\x80\xa0\xf8\x9a\x90\x81\xb8\xf8\x9d\x80\x81\xb3\xf8\x88\x80\x80\xae\xf8\x94\xb0\x80\xa0\xf8\x93\x80\x81\x91\xf8\x95\x80\x81\x93\xf8\x95\x80\x81\x81\xf8\x8f\x90\x81\x85\xf8\x8e\x80\x80\xb0\xf8\x8c\x80\x80\xb0\xf2\xa0\x80\xb3\xfc\xa0\x9c\xb0\x80\x80\xfd\xa1\x99\x82\x81\xa1\xfd\xa1\x98\xa6\x85\xb4\xfd\xa9\x88\x86\x95\xb3\xfd\x82\x91\x82\x81\xae\xfd\xaf\x99\xa2\x80\xb2\xfd\x96\x93\x92\x81\xb2\xfd\x93\x82\xa2\xb9\x93\xff\x7f (-1024) (SQLDriverConnect)')
I gathered from searching that HY000 may be a connectionString problem, but I'm not sure how to decipher the rest of the error.
Any ideas?
It appears connection strings vary by database and the connection string you are using is not compatible with DB2.
Here's some DB2 example connection strings:
http://www.connectionstrings.com/ibm-db2
Here's the best reference I could find as to what is valid in a DB2 connection string:
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=%2Fcom.ibm.swg.im.dbclient.adonet.ref.doc%2Fdoc%2FDB2ConnectionClass.html
Based on that, I'd convert your connection string from:
Driver={IBM DB2 ODBC Driver}; Hostname=myhost.com; Port=50300; Protocol=TCPIP; Database=DB2ET; CurrentSchema=SCHM1; UID=MYUID; PWD=MYPWD;
To:
Driver={IBM DB2 ODBC Driver}; Server=myhost.com:50300; Database=DB2ET; CurrentSchema=SCHM1; UID=MYUID; PWD=MYPWD;
Here's what my connection string looks like in PHP (I know you're using Python but I think they're very similar):
$this->db_connection = new PDO("odbc:DRIVER={iSeries Access ODBC Driver};SYSTEM=10.xxx.xxx.xxx;PROTOCOL=TCPIP", $temp_username, $temp_password);
Notice that I say "SYSTEM=..." and you say "Hostname=..." - I think that could be important.
Is your DSN set up completely? On my Linux machines I have two files that contain all of our configuration information for DSN's.
odbc.ini
[primary]
Description = primary
Driver = iSeries Access ODBC Driver
System = xxx.xxx.xxx.xxx
UserID = xxxxxxxxxx
Password = xxxxxxxxxx
Naming = 0
DefaultLibraries = QGPL
Database = xxxxxxxxxx
ConnectionType = 0
CommitMode = 2
ExtendedDynamic = 0
DefaultPkgLibrary = QGPL
DefaultPackage = A/DEFAULT(IBM),2,0,1,0,512
AllowDataCompression = 1
LibraryView = 0
AllowUnsupportedChar = 0
ForceTranslation = 0
Trace = 0
and odbcinst.ini
[iSeries Access ODBC Driver]
Description = iSeries Access for Linux ODBC Driver
Driver = /usr/lib/libcwbodbc.so
Setup = /usr/lib/libcwbodbcs.so
NOTE1 = If using unixODBC 2.2.11 or later and you want the 32 and 64-bit ODBC drivers to share DSN's,
NOTE2 = the following Driver64/Setup64 keywords will provide that support.
Driver64 = /usr/lib/lib64/libcwbodbc.so
Setup64 = /usr/lib/lib64/libcwbodbcs.so
Threading = 2
DontDLClose = 1
UsageCount = 1
I believe you enter similar information in the dialog you show, probably under the DSN tab.
Have you enabled ODBC tracing? To do this in Linux I have to add this to the odbcinst.ini file:
[ODBC]
Trace = Yes
TraceFile = /tmp/odbc.log
This dumped a LOT of information for me and helped to diagnose some problems I've had in the past.
According to this page (http://code.google.com/p/pyodbc/wiki/ConnectionStrings), "The most important thing to know is that pyodbc does not even look at the connection string -- it is passed directly to SQLDriverConnect unmodified." If that's true then changing your "Hostname" to "SYSTEM" should do the trick.
EDIT
Any chance this could be a problem with encoding? Unicode trying to talk to a UTF-8 server (or something similar)? Have you tried setting the CCSID? When you configure the driver do you have options such as "BinAsChar", "CCSID" or "Host CCSID"?
By default I believe the CCSID is 37 (US/Canada) and I think Unicode is 1208.
Why not define a User DSN using a driver and then use the DSN in the connect() method?
pyodbc.connect( 'DSN=MYDSN;UID=MYUSER;PWD=MYPASS' )
And by the way using any driver other than the "iSeries Access ODBC Driver" might raise those DB2 licensing errors. Maybe this is what the message is saying.
The driver is installed when you install "IBM i Access for Windows 7.1"
I have a running DBeaver connection to a DB2 database and now I would like to connect also in Python with the DB2 database securely over SSL. As authenticating I would like to use my windows account instead of giving a PW and UserID, to do so in DBeaver I changed:
javax.net.ssl.keyStoreType = Windows-ROOT
javax.net.ssl.trustStoreType = Windows-MY
Is this also possible with the Python connection?
I tried the following to set up the connection:
import ibm_db
connID = ibm_db.connect('DRIVER={IBM DB2 ODBC DRIVER};'
'DATABASE=XXXX;'
'HOSTNAME=XXXXYYY.net;'
'PORT=9797;'
'PROTOCOL=TCPIP;'
'SECURITY=ssl;'
'SSLSERVERCERTIFICATE=C:/Users/..../cert.cer;'
, '', '')
if connID is None:
print("\nERROR: Unable to connect.")
exit(-1)
print("Connected!")
I get the following error:
SQLCODE=-30082][CLI Driver] SQL30082N Security processing failed with reason "17" ("UNSUPPORTED FUNCTION"). SQLSTATE=08001
Some details:
M:\>"C:\Users\.conda\pkgs\ibm_db-3.0.1-py37hfa6e2cd_1\Lib\site-packages\clidriver\bin\db2level.exe"
DB21085I This instance or install (instance name, where applicable: "*") uses
"64" bits and DB2 code release "SQL11013" with level identifier "0204010F".
Informational tokens are "DB2 v11.1.3031.295", "s1804271300",
"DYN1804271300WIN64", and Fix Pack "3a".
Product is installed at
"C:\Users\.conda\pkgs\ibm_db-3.0.1-py37hfa6e2cd_1\Lib\site-packages\cli
driver" with DB2 Copy Name "IBM Data Server Driver For ODBC and CLI".
I'm using:
Python version 2.7.17
Ibm_db version 3.0.1