Develop Reference

Need help in fixing sql connection issue using service account though pyodbc

When I try to run the below code I am getting following error. Tried referring other posts but it does not seem to help.
The server principal "{Myuser}" is not able to access the database "{database}" under the current security context. (916) (SQLExecDirectW)'
import pyodbc
cnxn = pyodbc.connect("Driver={SQL Server};"
'Server=XXXXXXX;'
'Port=xxxx'
'Username={service_account}'
'Password={password of service_account}'
'Database={detabase};'
'Trusted_Connection=yes;')
results = cnxn.cursor()
results.execute('SELECT * FROM {database}.dbo.{tablename}')
for row in results:
print(row)

If you are going to use Microsoft's ODBC driver for SQL Server and Trusted_Connection=yes (SQL Server Windows authentication) then you don't pass the Windows credentials in the connection string. Instead, you run your Python app as the Windows user. Ways to do that include
launching the Python app from a Windows command prompt using the RUNAS command, or
Shiftright_clicking your app's icon, then choose "Run as different user".
Another option might be to use the FreeTDS ODBC driver. It supports the older NTLM authentication protocol and allows you to specify the DOMAIN, UID, and PWD (for Windows authentication) in the connection string.

This worked for me:
import pyodbc
password = "<password>"
conn = pyodbc.connect("Driver={SQL Server};Server=<host>;Port=<port>;UID=
<username>;PWD=" + password + ";Database=<dbname>;")
cursor = conn.cursor()
cursor.execute('SELECT * FROM <table>')
for row in cursor:
print(row)

Connect to db2 database with Python

I'm working on an app that needs to connect to an ibm db2 database. Using DBeaver I can successfully connect to the database (I provide him the db2cc.jar and db2cc4.jar files).
It looks to me as DBeaver is using my Window's credentials to login, because I didn't need to input any login or password to connect.
Now, I've been trying to connect to the same database using python 3.7 and pypi's latest version of the ibm_db package. I didn't install anything else.
import ibm_db
# ...
connection_string = "DATABASE=" + self.params['schema'] + ";" + \
"HOSTNAME=" + self.params['host'] + ";" + \
"PORT=" + self.params['port'] + ";" + \
"PROTOCOL=TCPIP;" + \
"SECURITYMECHANISM=4;" + \
"UID=" + self.params['user'] + ";" + \
"PWD=" + self.params['password'] + ";"
try:
self.connection = ibm_db.connect(connection_string, "", "")
# ...
Using my Windows credentials in the parameters, I get the following error message:
Connection error
Bad credentials
SQLCODE=-30082
08001
From what I've seen on stack overflow connecting to a db2 database is complicated...
Does someone know how to connect ? Using the windows credentials or otherwise...
Thanks !

Db2 works fine with Python.
You need to be aware of some basics before you start such as:
what operating-system runs the target Db2-database and
what kind of client is being used (java, odbc/cli, .net etc), and
what kind of authentication/encrpytion is in place for the database
(ssl, server based authentication/+/-/encryption etc.).
is the remote database rdbms Apache DERBY or Db2.
Find out these basics before you start. You have to speak with people who run the Db2-server.
Note: in your question you mention (SecurityMechanism=4) com.ibm.db2.jcc.DB2BaseDataSource.USER_ONLY_SECURITY - this is not relevant for non-JAVA clients, it is relevant is the database manager is DERBY .
For python, the ibm_db package is not a java application.
DBeaver is a java application (hence it uses db2jcc.jar or db2jcc4.jar and a licence-file to connect to the remote database).
You can only use your Windows credentials for connecting to a Db2-database when that Db2-database run on Microsoft-Windows, and the credentials work on the hostname running the Db2-server. For any other combinations, the administrator must issue you a userid/password that is relevant for the target hostname.
The ibm_db package needs a Db2-client to be installed. The Db2-client is a separate installable. There are different kinds of Db2-client depending both on which operating-system runs your Db2-server and how much functionality you need to have in your Db2-client. If your remote Db2-server runs on Linux, Unix, Windows or Z/OS then you can use the "IBM Data Server Runtime Client" which you can either download from IBM's passport advantage website, or get from your internal IT folks. If your Db2-server runs on i-Series (AS/400) you should get its drivers from your i-Series administrator. For either Z/OS or i-Series you will additionally need a license file (which costs money) and you should get that from your administrator, unless your company uses a gateway product called Db2-connect in which case you don't need a separate license file on your workstation.

Try the following connection string, If your db2 client and server are on the same host.
Change the 'mydb' and 'DB2' (db2 instance name, you can get it with db2ilist utility) constants according your case.
ibm_db.connect('DATABASE=mydb;Instance=DB2;PROTOCOL=IPC;', '', '')

add ibm_db library and import ibm_db on top in the .py file.
def get_db_connection():
"""
This will help to get db2 connection for query execution
:return: conn
"""
dsn_driver = "{IBM DB2 ODBC DRIVER}"
dsn_database = "BLUDB"
dsn_hostname = "your_hostname"
dsn_port = "50000"
dsn_protocol = "TCPIP"
dsn_uid = "your_userid"
dsn_pwd = "your_pwd"
dsn = (
"DRIVER={0};"
"DATABASE={1};"
"HOSTNAME={2};"
"PORT={3};"
"PROTOCOL={4};"
"UID={5};"
"PWD={6};").format(dsn_driver, dsn_database, dsn_hostname, dsn_port, dsn_protocol, dsn_uid, dsn_pwd)
try:
conn = ibm_db.connect(dsn, "", "")
print("Connected!")
return conn
except Exception:
print("\nERROR: Unable to connect to the \'" + dsn_database + "\' server.")
print("error: ", ibm_db.conn_errormsg())
exit(-1)
function get_db_connection() will return the connection object. On that connection object you can perform operation like:
conn = get_db_connection()
list_results = []
select_query = 'SELECT a.STATUS, a.ID FROM "TABLE_NAME" AS a'
print(select_query)
selectStmt = ibm_db.exec_immediate(conn, select_query)
while ibm_db.fetch_row(selectStmt) != False:
list_results.append(ibm_db.result(selectStmt, 'ID'))
ibm_db.close(conn)

Python Vertica: How to use Kerberos authentication with vertica-python module?

I am using Uber's vertica-python native python adaptor (https://github.com/uber/vertica-python), and I am trying to integrate kerberos authentication with it. Is there a way I can do it?
Problem: The database user I want to use with my script has authentication method kerberos set to highest priority, seems that vertica-python adaptor has only password based authentication. I read Vertica documentation it said if the priority is set to use Kerberos then its the only authentication method that Vertica asks for.

I'm looking for a similar answer to use Vertica with Airflow.
In the other hand I can tell you how I use Vertica with Kerberos and Python, in my case I found the best solution is using ODBC.
First install a ODBC client, if you are using Linux or Mac you can follow next site: unixodbc
Then configure a DSN in your client machine, you can just edit the default file on your home folder like this (examples for Mac and Linux):
~/.odbc.ini:
[ConnectionAliasABC]
Description = Database description
# Driver for MAC
# Driver = /Library/Vertica/ODBC/lib/libverticaodbc.dylib
# Driver for Linux
Driver = /opt/vertica/lib64/libverticaodbc.so
Database = DatabaseName
ServerName = ServerHost
UID = username
Port = 5433
KerberosServiceName = kerberosServiceName
KerberosHostname = kerberosHostname
Then you can use the next code snippet:
import pyodbc
# here you should init a kerberos ticket, we use a keytab file for this
initKerberos()
odbc_dsn = 'ConnectionAliasABC' # same as the .odbc.ini file section
print(f'Connecting to Vertica using dsn: {odbc_dsn}')
connection = pyodbc.connect(f'DSN={odbc_dsn}')
connection.setdecoding(pyodbc.SQL_CHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WCHAR, encoding='utf-8')
connection.setdecoding(pyodbc.SQL_WMETADATA, encoding='utf-16')
connection.setencoding(encoding='utf-8')
cursor = conn.cursor()
cursor.execute("SELECT dummy as test_column_name FROM DUAL")
row = cursor.fetchone()
print(f'query result: {row[0]}') # this should show an 'X'

Authenticate from Linux to Windows SQL Server with pyodbc

I am trying to connect from a linux machine to a windows SQL Server with pyodbc.
I do have a couple of constraints:
Need to log on with a windows domain account
Need to use python3
Need to do it from Linux to Windows
Need to connect to a specific instance
I set up the environment as described by microsoft and have it working (I can import pyodbc and use the configured mussel driver).
I am not familiar with Windows domain authentication and what not, so there is where my problem is.
My connection string:
DRIVER={ODBC Driver 17 for SQL Server};SERVER=myserver.mydomain.com;PORT=1433;DATABASE=MyDatabase;Domain=MyCompanyDomain;Instance=MyInstance;UID=myDomainUser;PWD=XXXXXXXX;Trusted_Connection=yes;Integrated_Security=SSPI
Supposedly one should use "Trusted_Connection" to use the Windows domain authentication instead of directly authenticating with the SQL server.
The error I get when running pyodbc.connect(connString):
pyodbc.Error: ('HY000', '[HY000] [unixODBC][Microsoft][ODBC Driver 17 for SQL Server]SSPI Provider: No Kerberos credentials available (851968) (SQLDriverConnect)')
From other sources I read this should work on Windows as this code would use the credentials of the currently logged in user.
My question is how can I connect to a Windows SQL Server instance from Linux using Windows Domain credentials.

You must obtain a Kerberos ticket for this to work. Your example doesn't specify whether your Linux system is set up to authenticate via Kerberos or whether you have previously obtained a Kerberos ticket before your code hits your connection string.
If your Linux system is set up to authenticate via Kerberos, then as a proof of concept you can obtain a Kerberos ticket using kinit from the command line. Here's what works for me in python3 running in Ubuntu on Windows via the WSL. The python code:
#!/usr/bin/env python
# minimal example using Kerberos auth
import sys
import re
import pyodbc
driver='{ODBC Driver 17 for SQL Server}'
server = sys.argv[1]
database = sys.argv[2]
# trusted_connection uses kerberos ticket and ignores UID and PASSWORD in connection string
# https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/using-integrated-authentication?view=sql-server-ver15
try:
cnxn = pyodbc.connect(driver=driver, server=server, database=database, trusted_connection='yes')
cursor = cnxn.cursor()
except pyodbc.Error as ex:
msg = ex.args[1]
if re.search('No Kerberos', msg):
print('You must login using kinit before using this script.')
exit(1)
else:
raise
# Sample select query
cursor.execute("SELECT ##version;")
row = cursor.fetchone()
while row:
print(row[0])
row = cursor.fetchone()
print('success')
This tells you if you don't have a ticket. Since it uses a ticket you don't have to specify a user or password in the script. It will ignore both.
Now we run it:
user#localhost:~# kdestroy # make sure there are no active tickets
kdestroy: No credentials cache found while destroying cache
user#localhost:~# python pyodbc_sql_server_test.py tcp:dbserver.example.com mydatabase
You must login using kinit before using this script.
user#localhost:~# kinit
Password for user#DOMAIN.LOCAL:
user#localhost:~# python pyodbc_sql_server_test.py tcp:dbserver.example.com mydatabase
Microsoft SQL Server 2016 (SP2-GDR) (KB4505220) - 13.0.5101.9 (X64)
Jun 15 2019 23:15:58
Copyright (c) Microsoft Corporation
Enterprise Edition (64-bit) on Windows Server 2016 Datacenter 10.0 <X64> (Build 14393: )
success
user#localhost:~#
You may also have success obtaining a Kerberos ticket from python code that runs before you make this connection but that is beyond the scope of this answer. A search for python Kerberos modules might point you toward a solution.
It also appears possible to set up the Linux system so that as soon as a user logs in it automatically obtains a Kerberos ticket that can be passed to other processes. That is also outside of the scope of this answer but a search for automatic Kerberos ticket upon Linux login may yield some clues.

I ended up using the pymssql library which basically is pyodbc on top of the FreeTDS driver. It worked out of the box.
Weird how I had such a hard time discovering this library..

I find two ways for same task. I have MSSQL server with AD auth.
You can use JVM.
Load and install JAVA https://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html. Also install JPype1 version 0.6.3 pip install JPype==0.6.3. Version above 0.6.3 won't work correct
import jaydebeapi
import pandas as pd
driver_name = "net.sourceforge.jtds.jdbc.Driver"
connection_url="jdbc:jtds:sqlserver://<server>:<port>/<database name>"
connection_properties = {
"domain": "<domain name>",
"user": "<username>",
"password": "<pwd>"}
jar_path = <path to jsds>"/jtds-1.3.1.jar"
CONN = jaydebeapi.connect(driver_name, connection_url, connection_properties, jar_path)
sql = "SELECT * FROM INFORMATION_SCHEMA.COLUMNS"
df = pd.read_sql(sql, CONN)
This version was too slow for me.
Also You can use pyodbc via FreeTDS. To create a FreeTDS connection Install FreeTDS on your Linux apt-get install tdsodbc freetds-bin, configure FreeTDS /etc/odbcinst.ini like this:
[FreeTDS]
Description=FreeTDS
Driver=/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup=/usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
and turn it on odbcinst -i -d -f /etc/odbcinst.ini
After that, you can use pyodbc
import pandas as pd
import pyodbc
CONN =pyodbc.connect('DRIVER={FreeTDS};'
'Server=<server>;'
'Database=<database>;'
'UID=<domain name>\\<username>;'
'PWD=<password>;'
'TDS_Version=8.0;'
'Port=1433;')
sql = "SELECT * FROM INFORMATION_SCHEMA.COLUMNS"
df = pd.read_sql(sql, CONN)
It's works much faster

I was trying to do the same thing and after reading the OPs answer I tested out pymssql and noticed that it worked with just the below:
pymssql.connect(server='myserver', user='domain\username', password='password', database='mydb')
After realizing that that was all pymssql needed I went back to pyodbc and was able to get it working with:
pyodbc.connect("DRIVER={FreeTDS};SERVER=myserver;PORT=1433;DATABASE=mydb;UID=domain\username;PWD=password;TDS_Version=8.0")
I just wanted to thank you for posting this as it helped me so greatly!!!! :)

Generating windows authentication via Linux is complex. EasySoftDB (commercial) used to be able to handle this, and FreeTDS has some convoluted support.
https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/using-integrated-authentication
My suggestion is to move away from Windows Authentication and use SQL authentication. Really there is no security difference, except that you are providing a username and password in the connection string. But this would make your life a lot easier.

I had the same issue and got the docker container for airflow using windows authentication by adding a few things to my airflow build. The apt install needs to be run as root.
USER root
RUN apt install -y krb5-config
RUN apt-get install -y krb5-user
COPY krb5.conf /etc/krb5.conf
In the krb5.conf file
[appdefaults]
default_lifetime = 52hrs
krb4_convert = false
krb4_convert_524 = false
ksu = {
forwardable = false
}
pam = {
minimum_uid = 100
forwardable = true
}
pam-afs-session = {
minimum_uid = 100
}
[libdefaults]
default_realm = DEFAULT_DOMAIN
ticket_lifetime = 52h
renew_lifetime = 90d
forwardable = true
noaddresses = true
allow_weak_crypto = true
rdns = false
[realms]
MY.COMPANY.LOCAL = {
kdc = SERVER.DEFAULT_DOMAIN
default_domain = DEFAULT_DOMAIN
}
[domain_realm]
my.company.local = DEFAULT_DOMAIN
[logging]
kdc = SYSLOG:NOTICE
admin_server = SYSLOG:NOTICE
default = SYSLOG:NOTICE
DEFAULT_DOMAIN for me is DOMAIN.COMPANY.COM. Others have .LOCAL at the end. Make sure it is all caps in the file. I had an error the first time I tried to authenticate.
Rebuild and then launch the shell for the airflow worker.
Run kinit USER
It will prompt for a password. Running klist afterwards to confirm you have a ticket. Once you get this working you should be able to authenticate to the server from python.

How can I access Oracle from Python?

How can I access Oracle from Python? I have downloaded a cx_Oracle msi installer, but Python can't import the library.
I get the following error:
import cx_Oracle
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
import cx_Oracle
ImportError: DLL load failed: The specified module could not be found.
I will be grateful for any help.

Here's what worked for me. My Python and Oracle versions are slightly different from yours, but the same approach should apply. Just make sure the cx_Oracle binary installer version matches your Oracle client and Python versions.
My versions:
Python 2.7
Oracle Instant Client 11G R2
cx_Oracle 5.0.4 (Unicode, Python 2.7, Oracle 11G)
Windows XP SP3
Steps:
Download the Oracle Instant Client package. I used instantclient-basic-win32-11.2.0.1.0.zip. Unzip it to C:\your\path\to\instantclient_11_2
Download and run the cx_Oracle binary installer. I used cx_Oracle-5.0.4-11g-unicode.win32-py2.7.msi. I installed it for all users and pointed it to the Python 2.7 location it found in the registry.
Set the ORACLE_HOME and PATH environment variables via a batch script or whatever mechanism makes sense in your app context, so that they point to the Oracle Instant Client directory. See oracle_python.bat source below. I'm sure there must be a more elegant solution for this, but I wanted to limit my system-wide changes as much as possible. Make sure you put the targeted Oracle Instant Client directory at the beginning of the PATH (or at least ahead of any other Oracle client directories). Right now, I'm only doing command-line stuff so I just run oracle_python.bat in the shell before running any programs that require cx_Oracle.
Run regedit and check to see if there's an NLS_LANG key set at \HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE. If so, rename the key (I changed it to NLS_LANG_OLD) or unset it. This key should only be used as the default NLS_LANG value for Oracle 7 client, so it's safe to remove it unless you happen to be using Oracle 7 client somewhere else. As always, be sure to backup your registry before making changes.
Now, you should be able to import cx_Oracle in your Python program. See the oracle_test.py source below. Note that I had to set the connection and SQL strings to Unicode for my version of cx_Oracle.
Source: oracle_python.bat
#echo off
set ORACLE_HOME=C:\your\path\to\instantclient_11_2
set PATH=%ORACLE_HOME%;%PATH%
Source: oracle_test.py
import cx_Oracle
conn_str = u'user/password#host:port/service'
conn = cx_Oracle.connect(conn_str)
c = conn.cursor()
c.execute(u'select your_col_1, your_col_2 from your_table')
for row in c:
print row[0], "-", row[1]
conn.close()
Possible Issues:
"ORA-12705: Cannot access NLS data files or invalid environment specified" - I ran into this before I made the NLS_LANG registry change.
"TypeError: argument 1 must be unicode, not str" - if you need to set the connection string to Unicode.
"TypeError: expecting None or a string" - if you need to set the SQL string to Unicode.
"ImportError: DLL load failed: The specified procedure could not be found." - may indicate that cx_Oracle can't find the appropriate Oracle client DLL.

You can use any of the following way based on Service Name or SID whatever you have.
With SID:
import cx_Oracle
dsn_tns = cx_Oracle.makedsn('server', 'port', 'sid')
conn = cx_Oracle.connect(user='username', password='password', dsn=dsn_tns)
c = conn.cursor()
c.execute('select count(*) from TABLE_NAME')
for row in c:
print(row)
conn.close()
OR
With Service Name:
import cx_Oracle
dsn_tns = cx_Oracle.makedsn('server', 'port', service_name='service_name')
conn = cx_Oracle.connect(user='username', password='password', dsn=dsn_tns)
c = conn.cursor()
c.execute('select count(*) from TABLE_NAME')
for row in c:
print(row)
conn.close()

Here is how my code looks like. It also shows an example of how to use query parameters using a dictionary. It works on using Python 3.6:
import cx_Oracle
CONN_INFO = {
'host': 'xxx.xx.xxx.x',
'port': 12345,
'user': 'SOME_SCHEMA',
'psw': 'SECRETE',
'service': 'service.server.com'
}
CONN_STR = '{user}/{psw}#{host}:{port}/{service}'.format(**CONN_INFO)
QUERY = '''
SELECT
*
FROM
USER
WHERE
NAME = :name
'''
class DB:
def __init__(self):
self.conn = cx_Oracle.connect(CONN_STR)
def query(self, query, params=None):
cursor = self.conn.cursor()
result = cursor.execute(query, params).fetchall()
cursor.close()
return result
db = DB()
result = db.query(QUERY, {'name': 'happy'})

import cx_Oracle
dsn_tns = cx_Oracle.makedsn('host', 'port', service_name='give service name')
conn = cx_Oracle.connect(user='username', password='password', dsn=dsn_tns)
c = conn.cursor()
c.execute('select count(*) from schema.table_name')
for row in c:
print row
conn.close()
Note :
In (dsn_tns) if needed, place an 'r' before any parameter in order to address any special character such as '\'.
In (conn) if needed, place an 'r' before any parameter in order to address any special character such as '\'. For example, if your user name contains '\', you'll need to place 'r' before the user name: user=r'User Name' or password=r'password'
use triple quotes if you want to spread your query across multiple lines.

Note if you are using pandas you can access it in following way:
import pandas as pd
import cx_Oracle
conn= cx_Oracle.connect('username/pwd#host:port/service_name')
try:
query = '''
SELECT * from dual
'''
df = pd.read_sql(con = conn, sql = query)
finally:
conn.close()
df.head()

In addition to the Oracle instant client, you may also need to install the Oracle ODAC components and put the path to them into your system path. cx_Oracle seems to need access to the oci.dll file that is installed with them.
Also check that you get the correct version (32bit or 64bit) of them that matches your: python, cx_Oracle, and instant client versions.

In addition to cx_Oracle, you need to have the Oracle client library installed and the paths set correctly in order for cx_Oracle to find it - try opening the cx_Oracle DLL in "Dependency Walker" (http://www.dependencywalker.com/) to see what the missing DLL is.

Ensure these two and it should work:-
Python, Oracle instantclient and cx_Oracle are 32 bit.
Set the environment variables.
Fixes this issue on windows like a charm.

If you are using virtualenv, it is not as trivial to get the driver using the installer. What you can do then: install it as described by Devon. Then copy over cx_Oracle.pyd and the cx_Oracle-XXX.egg-info folder from Python\Lib\site-packages
into the Lib\site-packages from your virtual env. Of course, also here, architecture and version are important.

import cx_Oracle
from sshtunnel import SSHTunnelForwarder
# remote server variables
remote_ip_address = "<PUBLIC_IP_ADDRESS_OF_DB_SERVER>"
remote_os_username = "<OS_USERNAME>"
ssh_private_key = "<PATH_TO_PRIVATE_KEY>"
# Oracle database variables
database_username = "<DATABASE_USER>"
database_password = "<DATABASE_USER_PASSWORD>"
database_server_sid = "<ORACLE_SID>"
def server_connection():
server = SSHTunnelForwarder(
remote_ip_address,
ssh_username=remote_os_username,
ssh_password=ssh_private_key,
remote_bind_address=('localhost', 1521) # default Oracle DB port
)
return server
def database_connection():
data_source_name = cx_Oracle.makedsn("localhost",
server.local_bind_port,
service_name=database_server_sid)
connection = cx_Oracle.connect(database_username,
database_password,
data_source_name,
mode=cx_Oracle.SYSDBA) # If logging in with SYSDBA privs,
# leave out if not.
return connection
def database_execute():
connection = database_connection()
cursor = connection.cursor()
for row in cursor.execute("SELECT * FROM HELLO_WORLD_TABLE"):
print(row)
if __name__ == '__main__':
server = server_connection()
server.start() # start remote server connection
connection = database_connection() # create Oracle database connection
database_execute() # execute query
connection.close() # close Oracle database connection
server.stop() # close remote server connection
If you're accessing the Oracle database via a bastion tunnel, you just need to modify this piece of code:
def server_connection():
server = SSHTunnelForwarder(
remote_ip_address, # public IP of bastion server
ssh_username=remote_os_username,
ssh_password=ssh_private_key,
remote_bind_address=('localhost', 1521),
local_bind_address=('0.0.0.0', 3333) # Suppose local bind is '3333'
)
return server