Develop Reference

SQL Login Failed [2800] with Windows Authentication Only when Using pyodbc, and Only for Certain Databases

Thank you for taking the time to look at my question!
I have a class ODBCSources which returns a SQLAlchemy/pyodbc engine for use with pretty much anything you could want it for. While logged in as a user with database editing permissions and using windows authentication, the code works for some databases on myserver.mycompany.local but not one database in particular. The problem database was created a week or so ago vs years ago for the ones that I can connect to with pyodbc. As the same user, I am able to log into the all of the databases using windows authentication + SSMS, Azure Data Studio, and/or DataGrip.
My code works for other databases on the same server, so I don't think the issue is with my connection string (But, the connection string must have a database name component, not an ODBC name component -- See my answer below). Additionally, the MS ODBC Data Source Administrator gives the feel good 'TESTS COMPLETED SUCCESSFULLY!' message when I test the connection corresponding to the problem database.
Is there a SQL-Server setting that I need to enable to allow python connections to the problem database? (No -- See my answer below)
import getpass
import pyodbc
from sqlalchemy import create_engine
class ODBCSources():
"""
A class to return sql alchemy engine for use in something like `pd.to_sql`
"""
def __init__(self, login = False):
self.sources = pyodbc.dataSources().keys()
self.login = login
self.user = ""
self.pwd = ""
def make_engine(self, odbc_source, driver = 'ODBC Driver 17 for SQL Server'):
if odbc_source not in self.sources:
raise ValueError(f'{odbc_source} is not available.\nAvailable Sources: {self.sources}')
else:
if self.login:
# This just gets the currently logged in user. Not really much more flexible
# than windows auth, but I'm not concerned with this
user = getpass.getuser()
pwd = getpass.getpass()
engine = create_engine(f'mssql+pyodbc://{user}:{pwd}#myserver.mycompany.local/{odbc_source}?driver={driver}', echo = False)
else:
user = self.user
pwd = self.pwd
engine = create_engine(f'mssql+pyodbc://{user}:{pwd}#myserver.mycompany.local/{odbc_source}?driver={driver}', echo = False)
return engine

I resolved the issue by creating a new ODBC datasource having a the same name as my database . . . which was really not necessary at all, but led me to the root cause root cause of my issue: my connection string is suitable for connection via database names, not ODBC names.
It so happened that in the past I created all of my ODBC connections with the same name as their corresponding database, so I was able to get by thinking that my connection string in create_engine had an ODBC component (it does not), because I was actually entering correct database names.

Connect with Oracle Database via LDAP Authentication

I am using python's pyodbc library to connect with the Oracle Database. The authentication to my non-prod servers were set-up as Basic Authentication so my connection string worked well.
It was tough to understand and set up the right drivers initially, But then did manage to get through. Here is what the code looks like for connection and worked well for my other servers.
import textwrap
import pyodbc
connection_string = textwrap.dedent('''Driver={driver};
DBQ={hostname}:{port}/{sid};
UID={username};
PWD={password};
Connection Timeout=30;
Trusted_Connection="yes"
'''.format(driver = 'Oracle in instantclient_11_2',
hostname = <hostname>,
port = <port>,
sid = <sid>,
username = <username>,
password = <passwd>
))
connection = pyodbc.connect(connection_string)
To my surprise PROD is authenticated via LDAP only. To which I have the LDAP Server, Context, and DB Service along with the credentials.
I tried creating my connection string as
connection_string = textwrap.dedent('''Driver={driver};
DBQ={hostname}:{port}/{sid};
UID={username};
PWD={password};
Connection Timeout=30;
Authentication=LDAP;
Trusted_Connection="yes"
'''.format(driver = 'Oracle in instantclient_11_2',
hostname = <ldap_server>,
port = 389,
sid = <db_service>,
username = <username>,
password = <passwd>
))
To which I was not surprised to know it wont work. Tried going through many links but could not get through. Has anyone tried this before or please let me know what I must be missing.
I could definitely connect in SQL Developer using the LDAP Authentication. And since TNS Listener is not configured on this server I cannot use Basic Authentication to connect.
And help would be welcome on this.

Setting up Oracle Drivers - (This is a pre-requisite if exist please ignore)
Download Oracle Instant Client for Microsoft Windows (x64) 64-bit. Alternatively download available for your 32 bit system.
Choose your version of Oracle Database. My version was 11.2.0.4.0
Download below files -
i. Instant Client Package - Basic
ii. Instant Client Package - ODBC
Unzip both the packages into the same directory such as C:\oracle\instantclient_19_3.
Execute odbc_install.exe from the Instant Client directory. If Instant Client is 11g or lower, start the command prompt with the Administrator privilege.
Create C:\oracle\instantclient_19_3\network\admin folder to place tnsnames.ora, sqlnet.ora.
Set Environment variable - TNS_ADMIN to above folder location.
Add C:\oracle\instantclient_19_3 to PATH Environment variable.
Add C:\oracle\instantclient_19_3 to ORACLE_HOME (optional, if connection doesn't work try this.)
LDAP Authentication
Create two files: sqlnet.ora and ldap.ora
ldap.ora
# Place this file in the network/admin subdirectory or your
# $ORACLE_HOME location.
# LDAP Server name should be added here. Rest all the values remains unchanged.
DIRECTORY_SERVERS = (your-server.your-organization:389:636)
DEFAULT_ADMIN_CONTEXT = "ldap-ou-designation"
DIRECTORY_SERVER_TYPE = OID
sqlnet.ora
# Place this file in the network/admin subdirectory or your
# $ORACLE_HOME location.
SQLNET.AUTHENTICATION_SERVICES=(NTS)
NAMES.DIRECTORY_PATH = (LDAP)
Verify C:\oracle\instantclient_19_3 is present in the Path else try updating the path in the Python Code as below
import os
lib_dir=r"C:\oracle\instantclient_19_3"
os.environ["PATH"] = lib_dir + ";" + os.environ["PATH"]
With all this set-up in place you should have
username, password and DB Service name lets call it as db_service here
with cx_Oracle
import cx_Oracle
con = cx_Oracle.connect('{0}/{1}#{2}'.format(user, password, db_service))
version_script = "SELECT * FROM v$version"
cursor = con.cursor()
cursor.execute(version_script)
version = cursor.fetchall()
print(version[0][0])
Ouput
-----
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
with pyodbc
import pyodbc
con = pyodbc.connect('Driver={0};DBQ={1};UID={2};PWD={3}'.format(driver, db_service, user, password))
version_script = "SELECT * FROM v$version"
cursor = con.cursor()
cursor.execute(version_script)
version = cursor.fetchall()
print(version[0][0])
Ouput
-----
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
#Drivers for pyodbc
pyodbc.drivers()
Output
------
['ODBC Driver 17 for SQL Server',
'Oracle in instantclient_11_2']
References :
https://www.oracle.com/database/technologies/instant-client/winx64-64-downloads.html
https://www.oracle.com/database/technologies/releasenote-odbc-ic.html
https://eikonomega.medium.com/connecting-to-oracle-database-with-cx-oracle-and-ldap-5da7925a305c

Connect to db2 database with Python

I'm working on an app that needs to connect to an ibm db2 database. Using DBeaver I can successfully connect to the database (I provide him the db2cc.jar and db2cc4.jar files).
It looks to me as DBeaver is using my Window's credentials to login, because I didn't need to input any login or password to connect.
Now, I've been trying to connect to the same database using python 3.7 and pypi's latest version of the ibm_db package. I didn't install anything else.
import ibm_db
# ...
connection_string = "DATABASE=" + self.params['schema'] + ";" + \
"HOSTNAME=" + self.params['host'] + ";" + \
"PORT=" + self.params['port'] + ";" + \
"PROTOCOL=TCPIP;" + \
"SECURITYMECHANISM=4;" + \
"UID=" + self.params['user'] + ";" + \
"PWD=" + self.params['password'] + ";"
try:
self.connection = ibm_db.connect(connection_string, "", "")
# ...
Using my Windows credentials in the parameters, I get the following error message:
Connection error
Bad credentials
SQLCODE=-30082
08001
From what I've seen on stack overflow connecting to a db2 database is complicated...
Does someone know how to connect ? Using the windows credentials or otherwise...
Thanks !

Db2 works fine with Python.
You need to be aware of some basics before you start such as:
what operating-system runs the target Db2-database and
what kind of client is being used (java, odbc/cli, .net etc), and
what kind of authentication/encrpytion is in place for the database
(ssl, server based authentication/+/-/encryption etc.).
is the remote database rdbms Apache DERBY or Db2.
Find out these basics before you start. You have to speak with people who run the Db2-server.
Note: in your question you mention (SecurityMechanism=4) com.ibm.db2.jcc.DB2BaseDataSource.USER_ONLY_SECURITY - this is not relevant for non-JAVA clients, it is relevant is the database manager is DERBY .
For python, the ibm_db package is not a java application.
DBeaver is a java application (hence it uses db2jcc.jar or db2jcc4.jar and a licence-file to connect to the remote database).
You can only use your Windows credentials for connecting to a Db2-database when that Db2-database run on Microsoft-Windows, and the credentials work on the hostname running the Db2-server. For any other combinations, the administrator must issue you a userid/password that is relevant for the target hostname.
The ibm_db package needs a Db2-client to be installed. The Db2-client is a separate installable. There are different kinds of Db2-client depending both on which operating-system runs your Db2-server and how much functionality you need to have in your Db2-client. If your remote Db2-server runs on Linux, Unix, Windows or Z/OS then you can use the "IBM Data Server Runtime Client" which you can either download from IBM's passport advantage website, or get from your internal IT folks. If your Db2-server runs on i-Series (AS/400) you should get its drivers from your i-Series administrator. For either Z/OS or i-Series you will additionally need a license file (which costs money) and you should get that from your administrator, unless your company uses a gateway product called Db2-connect in which case you don't need a separate license file on your workstation.

Try the following connection string, If your db2 client and server are on the same host.
Change the 'mydb' and 'DB2' (db2 instance name, you can get it with db2ilist utility) constants according your case.
ibm_db.connect('DATABASE=mydb;Instance=DB2;PROTOCOL=IPC;', '', '')

add ibm_db library and import ibm_db on top in the .py file.
def get_db_connection():
"""
This will help to get db2 connection for query execution
:return: conn
"""
dsn_driver = "{IBM DB2 ODBC DRIVER}"
dsn_database = "BLUDB"
dsn_hostname = "your_hostname"
dsn_port = "50000"
dsn_protocol = "TCPIP"
dsn_uid = "your_userid"
dsn_pwd = "your_pwd"
dsn = (
"DRIVER={0};"
"DATABASE={1};"
"HOSTNAME={2};"
"PORT={3};"
"PROTOCOL={4};"
"UID={5};"
"PWD={6};").format(dsn_driver, dsn_database, dsn_hostname, dsn_port, dsn_protocol, dsn_uid, dsn_pwd)
try:
conn = ibm_db.connect(dsn, "", "")
print("Connected!")
return conn
except Exception:
print("\nERROR: Unable to connect to the \'" + dsn_database + "\' server.")
print("error: ", ibm_db.conn_errormsg())
exit(-1)
function get_db_connection() will return the connection object. On that connection object you can perform operation like:
conn = get_db_connection()
list_results = []
select_query = 'SELECT a.STATUS, a.ID FROM "TABLE_NAME" AS a'
print(select_query)
selectStmt = ibm_db.exec_immediate(conn, select_query)
while ibm_db.fetch_row(selectStmt) != False:
list_results.append(ibm_db.result(selectStmt, 'ID'))
ibm_db.close(conn)

Python Vertica: How to use Kerberos authentication with vertica-python module?

I am using Uber's vertica-python native python adaptor (https://github.com/uber/vertica-python), and I am trying to integrate kerberos authentication with it. Is there a way I can do it?
Problem: The database user I want to use with my script has authentication method kerberos set to highest priority, seems that vertica-python adaptor has only password based authentication. I read Vertica documentation it said if the priority is set to use Kerberos then its the only authentication method that Vertica asks for.

I'm looking for a similar answer to use Vertica with Airflow.
In the other hand I can tell you how I use Vertica with Kerberos and Python, in my case I found the best solution is using ODBC.
First install a ODBC client, if you are using Linux or Mac you can follow next site: unixodbc
Then configure a DSN in your client machine, you can just edit the default file on your home folder like this (examples for Mac and Linux):
~/.odbc.ini:
[ConnectionAliasABC]
Description = Database description
# Driver for MAC
# Driver = /Library/Vertica/ODBC/lib/libverticaodbc.dylib
# Driver for Linux
Driver = /opt/vertica/lib64/libverticaodbc.so
Database = DatabaseName
ServerName = ServerHost
UID = username
Port = 5433
KerberosServiceName = kerberosServiceName
KerberosHostname = kerberosHostname
Then you can use the next code snippet:
import pyodbc
# here you should init a kerberos ticket, we use a keytab file for this
initKerberos()
odbc_dsn = 'ConnectionAliasABC' # same as the .odbc.ini file section
print(f'Connecting to Vertica using dsn: {odbc_dsn}')
connection = pyodbc.connect(f'DSN={odbc_dsn}')
connection.setdecoding(pyodbc.SQL_CHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WCHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WMETADATA, encoding='utf-16')
connection.setencoding(encoding='utf-8')
cursor = conn.cursor()
cursor.execute("SELECT dummy as test_column_name FROM DUAL")
row = cursor.fetchone()
print(f'query result: {row[0]}') # this should show an 'X'

How to connect to a remote PostgreSQL database through SSL with Python

I want to connect to a remote PostgreSQL database through Python to do some basic data analysis. This database requires SSL (verify-ca), along with three files (which I have):
Server root certificate file
Client certificate file
Client key file
I have not been able to find a tutorial which describes how to make this connection with Python.
Any help is appreciated.

Use the psycopg2 module.
You will need to use the ssl options in your connection string, or add them as key word arguments:
import psycopg2
conn = psycopg2.connect(dbname='yourdb', user='dbuser', password='abcd1234', host='server', port='5432', sslmode='require')
In this case sslmode specifies that SSL is required.
To perform server certificate verification you can set sslmode to verify-full or verify-ca. You need to supply the path to the server certificate in sslrootcert. Also set the sslcert and sslkey values to your client certificate and key respectively.
It is explained in detail in the PostgreSQL Connection Strings documentation (see also Parameter Key Words) and in SSL Support.

You may also use an ssh tunnel with paramiko and sshtunnel:
import psycopg2
import paramiko
from sshtunnel import SSHTunnelForwarder
mypkey = paramiko.RSAKey.from_private_key_file('/path/to/private/key')
tunnel = SSHTunnelForwarder(
(host_ip, 22),
ssh_username=username,
ssh_pkey=mypkey,
remote_bind_address=('localhost', psql_port))
tunnel.start()
conn = psycopg2.connect(dbname='gisdata', user=psql_username, password=psql_password, host='127.0.0.1', port=tunnel.local_bind_port)

If you need to connect to your PostgresSQL database with an SSL certificate using psycopg2, you'll need to put your certificate SSL certificate in a subdirectory of your python program, and then you can reference the certificate in your connection string. I believe you could also set an environment variable as well, but in my example my SSL certificate will be in a subdirectory.
My python script is in a directory which looks like:
/Users/myusername/Desktop/MyCoolPythonProgram/test_database_connection.py
And my SSL certificate is in a directory which looks like:
/Users/myusername/Desktop/MyCoolPythonProgram/database/ssl_certificate/database/ssl_certificate/ca-certificate.crt
My HOSTNAME is a URL from DigitalOcean, but yours might be an IP Address instead.
This is what my test_database_connection.py script looks like:
import psycopg2
import os
POSTGRES_DATABASE_HOST_ADDRESS = "your-database-name-do-user-12345678-0.b.db.ondigitalocean.com"
POSTGRES_DATABASE_NAME = "defaultdb"
POSTGRES_USERNAME = "doadmin"
POSTGRES_PASSWORD = "$uperD00P3Rp#$$W0RDg0E$here"
# HOW TO (Relative Path Python): https://stackoverflow.com/questions/918154/relative-paths-in-python
path_to_current_directory = os.path.dirname(__file__)
relative_path_to_ssl_cert = 'database/ssl_certificate/ca-certificate.crt'
SSL_ROOT_CERT = os.path.join(path_to_current_directory , relative_path_to_ssl_cert )
POSTGRES_CONNECTION_PORT = "1234" # Set this to the correct port! Mine is provided by DigitalOcean and it's NOT 1234
db_info = "host='%s' dbname='%s' user='%s' password='%s' sslmode='require' sslrootcert='%s' port='%s'" % (POSTGRES_DATABASE_HOST_ADDRESS, POSTGRES_DATABASE_NAME, POSTGRES_USERNAME, POSTGRES_PASSWORD, SSL_ROOT_CERT, POSTGRES_CONNECTION_PORT)
postgres_connection = psycopg2.connect(db_info)
with postgres_connection:
with postgres_connection.cursor() as postgres_cursor:
sql = "SELECT * FROM your_table;"
postgres_cursor.execute(sql)
results = postgres_cursor.fetchall()
for row in results:
print("row in result")
print("Connection Success!")
# Close Database Cursor/Connection
postgres_cursor.close()

Adding this for completeness and because I couldn't find it anywhere else on SO. Like #mhawke says, you can use psycopg2, but you can also use any other Python database modules (ORMs, etc) that allow you to manually specify a database postgresql URI (postgresql://[user[:password]#][netloc][:port][/dbname][?param1=value1&...]) to connect to since the sslmode="require" parameter that psycopg2.connect uses to enforce ssl connections is just part of the postgresql:// URI that you use to connect to your database (see 33.1.2. Parameter Key Words). So, if you wanted to use sqlalchemy or another ORM instead of vanilla psycopg2, you can tack your desired sslmode onto the end of your database URI and connect that way.
import sqlalchemy
DATABASE_URI = "postgresql://postgres:postgres#localhost:5432/dbname"
# sqlalchemy 1.4+ uses postgresql:// instead of postgres://
ssl_mode = "?sslmode=require"
DATABASE_URI += ssl_mode
engine = sqlalchemy.create_engine(URI)
Session = sqlalchemy.orm.sessionmaker(bind=engine)
There's a nifty figure (Table 33.1) in the postgres documentation on SSL Support that breaks down the different options you can supply. If you want to use any of the fancier options that require you to specify a path to a specific certificate, you can drop it in with a format string.