Develop Reference

Anaconda3's python cannot find kerberos credential cache

I run into an error when creating an ODBC connection to a MSSQL server using Anaconda's version of Python 3:
pyodbc.Error: ('HY000', '[HY000] [Microsoft][ODBC Driver 17 for SQL Server]SSPI Provider: No Kerberos credentials available (default cache: KEYRING:persistent:1918003883) (851968) (SQLDriverConnect)')
The server has joined a Windows Active Directory domain and Kerberos realm via SSSD. I can SSH into the server, and retrieve a TGT using kinit. I can even see the credential cache with klist. But the python process cannot seem to either find the Kerberos TGT or the Kerberos credential cache.
the setup:
python
$ /mnt/ds/anaconda3/bin/python --version
Python 3.6.5 :: Anaconda, Inc.
test.py
from pyodbc import connect
connection = connect('DSN=MyDSN')
/etc/odbc.ini
[MyDSN]
#Driver=ODBC Driver 13 for SQL Server
Driver=ODBC Driver 17 for SQL Server
Description=MyMSSQL ODBC Driver
Trace=No
Server=MyMSSQL
Trusted_Connection=Yes
/etc/odbcinst.ini
[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
Driver=/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.1.so.0.1
UsageCount=1
Red Hat Enterprise Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)
$ uname -r
3.10.0-862.2.3.el7.x86_64
msodbcsql17
$ sudo yum info msodbcsql17
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Installed Packages
Name : msodbcsql17
Arch : x86_64
Version : 17.1.0.1
Release : 1
Size : 17 M
Repo : installed
From repo : packages-microsoft-com-prod
Summary : ODBC Driver for Microsoft(R) SQL Server(R)
License : https://aka.ms/odbc170eula
Description : This package provides an ODBC driver that can connect to Microsoft(R) SQL Server(R).
unixODBC
$ sudo yum info unixODBC
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Installed Packages
Name : unixODBC
Arch : x86_64
Version : 2.3.1
Release : 11.el7
Size : 1.2 M
Repo : installed
From repo : rhui-REGION-rhel-server-releases
Summary : A complete ODBC driver manager for Linux
URL : http://www.unixODBC.org/
License : GPLv2+ and LGPLv2+
Description : Install unixODBC if you want to access databases through ODBC.
: You will also need the mysql-connector-odbc package if you want to access
: a MySQL database, and/or the postgresql-odbc package for PostgreSQL.
$ /mnt/ds/anaconda3/bin/conda list unixodbc
# packages in environment at /mnt/ds/anaconda3:
#
# Name Version Build Channel
unixodbc 2.3.6 h1bed415_0
pyodbc
$ /mnt/ds/anaconda3/bin/conda list pyodbc
# packages in environment at /mnt/ds/anaconda3:
#
# Name Version Build Channel
pyodbc 4.0.23 py36hf484d3e_0
Here are some things I've tried:
Using Python-2.7.15, as packaged by Anaconda2. That worked!
Using isql. I ran isql MyDSN and that connected.
There are two unixODBC libraries (one installed via yum; the other with conda). By default, it will use conda's, but I forced it to use the system unixODBC package with LD_PRELOAD. Same error.
I tried downgrading the database driver to msodbcsql-13.1.9.2-1 and then to msodbcsql-13.0.1.0-1. Same error.
I tried swapping out PyODBC for TurbODBC, another Python ODBC library. Same error.
I created a separate environment in conda with python-3.5. And that worked! Still not sure why.
I wrote a simple C program that interfaced with unixODBC. That program was able to connect to the MSSQL server via Kerberos just fine.
I ran the python2 positive test case and the python3 negative test case through strace to review the system calls. I thought that might reveal something. It seems that they both start looking for the client.keytab file on the file system. Then, in the postive test case, it will fallback to searching the kernel's keyring, where it will successfully find the credential cache and proceed. However, in the negative test case, it simply retries to find client.keytab, and never attempts to search the keyring.
I enabled the unixODBC trace option, one with Python3 test case, and the other with Python2 test case. Unfortunately, the traces (shown below) don't reveal anything to me.
py3-unixodbc.trace
[ODBC][8741][1527046794.480751][__handles.c][460]
Exit:[SQL_SUCCESS]
Environment = 0x55eea73ed130
[ODBC][8741][1527046794.480806][SQLSetEnvAttr.c][189]
Entry:
Environment = 0x55eea73ed130
Attribute = SQL_ATTR_ODBC_VERSION
Value = 0x3
StrLen = 4
[ODBC][8741][1527046794.480824][SQLSetEnvAttr.c][363]
Exit:[SQL_SUCCESS]
[ODBC][8741][1527046794.480843][SQLAllocHandle.c][375]
Entry:
Handle Type = 2
Input Handle = 0x55eea73ed130
[ODBC][8741][1527046794.480861][SQLAllocHandle.c][493]
Exit:[SQL_SUCCESS]
Output Handle = 0x55eea7400500
[ODBC][8741][1527046794.481176][SQLDriverConnectW.c][290]
Entry:
Connection = 0x55eea7400500
Window Hdl = (nil)
Str In = [DSN=MyDSN][length = 15]
Str Out = (nil)
Str Out Max = 0
Str Out Ptr = (nil)
Completion = 0
UNICODE Using encoding ASCII 'ISO8859-1' and UNICODE 'UCS-2LE'
[ODBC][8741][1527046794.575566][__handles.c][460]
Exit:[SQL_SUCCESS]
Environment = 0x55eea746e360
[ODBC][8741][1527046794.575614][SQLGetEnvAttr.c][157]
Entry:
Environment = 0x55eea746e360
Attribute = 65002
Value = 0x7ffd399177f0
Buffer Len = 128
StrLen = 0x7ffd3991778c
[ODBC][8741][1527046794.575632][SQLGetEnvAttr.c][264]
Exit:[SQL_SUCCESS]
[ODBC][8741][1527046794.575651][SQLFreeHandle.c][219]
Entry:
Handle Type = 1
Input Handle = 0x55eea746e360
py2-unixodbc.trace
[ODBC][8746][1527046842.073439][__handles.c][460]
Exit:[SQL_SUCCESS]
Environment = 0x185e2e0
[ODBC][8746][1527046842.073530][SQLSetEnvAttr.c][189]
Entry:
Environment = 0x185e2e0
Attribute = SQL_ATTR_ODBC_VERSION
Value = 0x3
StrLen = 4
[ODBC][8746][1527046842.073552][SQLSetEnvAttr.c][363]
Exit:[SQL_SUCCESS]
[ODBC][8746][1527046842.073572][SQLAllocHandle.c][375]
Entry:
Handle Type = 2
Input Handle = 0x185e2e0
[ODBC][8746][1527046842.073590][SQLAllocHandle.c][493]
Exit:[SQL_SUCCESS]
Output Handle = 0x1857d40
[ODBC][8746][1527046842.073613][SQLDriverConnectW.c][290]
Entry:
Connection = 0x1857d40
Window Hdl = (nil)
Str In = [DSN=MyDSN][length = 15]
Str Out = (nil)
Str Out Max = 0
Str Out Ptr = (nil)
Completion = 0
UNICODE Using encoding ASCII 'ISO8859-1' and UNICODE 'UCS-2LE'
[ODBC][8746][1527046842.208760][__handles.c][460]
Exit:[SQL_SUCCESS]
Environment = 0x1967210
[ODBC][8746][1527046842.208830][SQLGetEnvAttr.c][157]
Entry:
Environment = 0x1967210
Attribute = 65002
Value = 0x7ffe1153fcf0
Buffer Len = 128
StrLen = 0x7ffe1153fc8c
[ODBC][8746][1527046842.208849][SQLGetEnvAttr.c][264]
Exit:[SQL_SUCCESS]
[ODBC][8746][1527046842.208869][SQLFreeHandle.c][219]
Entry:
Handle Type = 1
Input Handle = 0x1967210
Suffice it to say, I'm at my wit's end. Any ideas would be greatly appreciated!

late reply but I hope someone will find it useful.
This very same issue for me was casued by the krb5 package which was coexisting with the system-wide kerberos installation: removing it from the environment solved the problem (I tried to make it work but had no success).

Python 3.4 - Sybase ASE connection

I would like to connect to a Sybase Ase 15 db with Python. Unfortunately I couldnt find any working solution for Windows with Python 3.4. Could anyone refer something? I tried with a few without luck. Can I use OleDb driver (dll) maybe somehow?
It would be great something free which is updated recently. I found solutions from 2012, but there werent working either.
Thank you.

You could use pyodbc:
import pyodbc
DbConnection = pyodbc.connect('DRIVER=freetds;SERVER=%s;PORT=%s;UID=%s;PWD=%s;DATABASE=%s;TDS _Version=5.0;' % (self.ServerAddress, self.ServerPort,
'aselogin', 'loginpwd', DefaultDb),unicode_results=True,autocommit=True)
Prerequisite: installing the driver FreeTDS corresponding to your OS.

First of all I want to point out the following: we can't connect to the Sybase ASE database just using Python; we have to use an external binary (called driver) which is able to manage the connections. In my case, I connect from a Windows machine using "jconn4.jar" driver.
In order to get the driver (jconn4.jar) I had to install Dbeaver application., and I set up a connection to the Sybase database that I want to access via python.
The next step is to test the connection and get the connection parameters. Press Test Connection --> Details.
In the windows that popped up we have all the details we need to configure the Sybase connection from python.
import jaydebeapi
server = "<server IP>"
username = "<username>"
password = "<password>"
database = "<datamase/schema>"
port = <port>
jdbc_driver = r'..\DBeaverData\drivers\drivers\sybase\jconnect\jconn4.jar' // this is the driver path; the path is obtained from the details window
conn = jaydebeapi.connect('com.sybase.jdbc4.jdbc.SybDriver', f'jdbc:sybase:Tds:{server}:{port}/{database}', {'user': username, 'password': password},
jdbc_driver)
cursor = conn.cursor()
cursor.execute("select * from my_table")
result = cursor.fetchall()
print(result)
If you get an error, regarding "JAVA_HOME" is missing
Check JDK is installed on your machine (I have installed JDK 1.8.0_202)
--> If not; install JDK
Add java path to the Environment Variables --> System Variables --> Path --> Edit --> New and paste the path to the Java bin folder (C:\Program Files\Java\jdk1.8.0_202\bin) --> Press OK
Specify the JAVA_HOME environment variable; Environment Variables --> System Variables --> New --> Varialbe Name: JAVA_HOME, Varialbe Value: C:\Program Files\Java\jdk1.8.0_202

SqlAlchemy can't find Teradata ODBC driver on RHEL 7

I am trying to use SQLAlchemy to connect to our Teradata environment and execute a query. I ran the script on a Windows 7 machine using an Anaconda Python 2.7 environment and Jupyter notebook. When I moved this to our Linux server and ran it in an Anaconda Python 2.7 environment it doesn't work. It complains that it can't find the driver Teradata. I have ran this as root and my non-root user.
I am trying to run this in a Jupyter notebook running on the server
from sqlalchemy import create_engine
# connect
td_engine = create_engine('teradata://username:password#teradata_database:22/')
# execute sql
sql="SELECT * FROM sometable"
result = td_engine.execute(sql)
Here is the traceback
Here is what I've done on the RHEL 7 server inside a py27 env
conda install sqlalchemy
pip install teradata
pip install sqlalchemy-teradata
yum install unixODBC
I downloaded the latest Teradata ODBC Driver for Linux
I extracted the file
tar -zxf tdodbc1510__linux_indep_15.10.01.04-1.tar.gz
This gives me three more tar.gz files and a text file of the same name.
I then extract each of these tar.gz files
tar -zxf tdicu1510__linux_indep.15.10.04-1.tar.gz
tar -zxf tdodbc1510__linux_indep.15.10.01.04-1.tar.gz
tar -zxf TeraGSS_linux-x64_linux_indep.15.10.03.02-1.tar.gz
This creates a directory for each root name (tdicu1510, tdodbc1510, TeraGSS)
I cd into each directory and switch to korn shell
/usr/bin/ksh
for the rpm's inside the three directories I install them
rpm -ihv tdicu1510_linux_x64-15.10.03.02-1.noarch.rpm
rpm -ihv TeraGSS_linux_x64-15.10.03.02-1.noarch.rpm
rpm -ihv tdodbc1510_linux_x64-15.10.03.02-1.noarch.rpm
This creates an /opt/teradata directory
I add this to /usr/local/etc/odbcinst.ini
[Teradata]
Driver=/opt/teradata/client/15.10/odbc_64/lib/tdata.so
APILevel=CORE
ConnectFunctions=YYY
DriverODBCVer=3.51
SQLLevel=1
I add this to /usr/local/etc/odbc.ini
[ODBC Data Sources]
TDDSN=tdata.so
[ODBC]
InstallDir=/opt/teradata/client/15.10/odbc_64
Trace=0
TraceDll=/opt/teradata/client/15.10/odbc_64/lib/odbctrac.so
TraceFile=/usr/teradata_logs/odbcusr/trace.log
TraceAutoStop=0
[TDDSN]
Driver=/opt/teradata/client/15.10/odbc_64/lib/tdata.so
Description=Teradata database
DBCName=<MachineName or ip>
LastUser=
Username=
Password=
Database=
Here is my ~/.bashrc
export PATH="/opt/miniconda2/bin:$PATH
export ORACLE_HOME="/usr/lib/oracle/12.1/client64"
export PATH="$PATH:$ORACLE_HOME/bin"
export LD_LIBRARY_PATH="$ORACLE_HOME/lib"
export TNS_ADMIN="$ORACLE_HOME/network/admin"
export TERADATA="/opt/teradata/client/15.10"
export ODBCINI="/opt/teradata/client/15.10/odbc_64/odbc.ini"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$TERADATA/lib64"
echo $LD_LIBRARY_PATH shows /usr/lib/oracle/12.1/client64/lib:/opt/teradata/client/15.10/lib64
Any ideas what I am doing wrong? Why can't SQLAlchemy see the Teradata ODBC driver on the Linux server?

I too had the same issue. I overcame that by using python teradatasqlalchemy module and modifying the connection string from:
create_engine('**teradata**://username:password#teradata_database:22/')
to
create_engine('**teradatasql**://username:password#teradata_database:22/')

How to access Hive on remote server using python client

Case: I have Hive on a cloudera platform. There is a database on Hive that I want to access using python client from my computer. I read a similar SO question but its using pyhs2 which I am unable to install on the remote server. And this SO question too uses Thrift but I cant seem to install it either.
Code: After following the documentation, when I execute the following program it gives me an error.
import pyodbc, sys, os
pyodbc.autocommit=True
con = pyodbc.connect("DSN=default",driver='SQLDriverConnect',autocommit=True)
cursor = con.cursor()
cursor.execute("select * from fb_mpsp")
Error: ssh://ashish#ServerIPAddress/home/ashish/anaconda/bin/python2.7 -u /home/ashish/PyCharm_proj/hdfsConnect/home/ashish/PyCharm_proj/hdfsConnect/Hive_connect/hive_connect.py
Traceback (most recent call last):
File "/home/ashish/PyCharm_proj/hdfsConnect/home/ashish/PyCharm_proj/hdfsConnect/Hive_connect/hive_connect.py", line 5, in
con = pyodbc.connect("DSN=default", driver='SQLDriverConnect',autocommit=True)
pyodbc.Error: ('IM002', '[IM002] [unixODBC][Driver Manager]Data source name not found, and no default driver specified (0) (SQLDriverConnect)')
Process finished with exit code 1
Please suggest how can I solve this problem? Also I am not sure why do I have to specify the driver as SQLDriverConnect when the code will be executed using hadoop hive?
Thanks

This worked for me
oODBC = pyodbc.connect("DSN=Cloudera Hive DSN 64;", autocommit = True, ansi = True )
And now everything works fine.
Be sure anything is fine with you DSN using:
isql -v "Cloudera Hive DSN 64"
and replace "Cloudera Hive DSN 64" with the name you used in your odbc.ini
Also, currently I'm not able to use the kerberos authentication unless I make a ticket by hand. Impala works smoothly using kerberos keytab files
Any help about how to have hive odbc working with keytab files is appreciated.

If you do decide to revisit pyhs2 note that it doesn't need to be installed on the remote server, it's installed on your local client.
If you continue with pyodbc, you need to install the ODBC driver for Hive, which you can get from Cloudera's site.
You don't need to specify the driver in your connection, it should be part of your DSN. The specifics of creating the DSN depend on your OS, but essentially you will create it using Administrative Tools -> Data Sources (Windows), install ODBC and edit /Library/ODBC/odbc.ini (Mac), or edit /etc/odbc.ini (Linux).
Conceptually, think of the DSN as a specification that represents all the information about the connection - it will contain the host, port, and driver information. That way in your code you don't have to specify these things and you can switch details about the database without changing your code.
# Note only the DSN name specifies the connection
import pyodbc
conn = pyodbc.connect("DSN=Hive1")
cursor = conn.cursor()
cursor.execute("select * from YYY")
Alternatively, I've updated the other question you referenced with information about how to install the thrift libraries. I think that's the way to go, if you have that option.

Try this method also to conenct and get data remotely from hive server:
connect remote server with ssh and give the cli command to access data from remote server:
ssh -o UserKnownHostsFile=/dev/null -o ConnectTimeout=90 -o StrictHostKeyChecking=no shashanks#remote_host 'hive -e "select * from DB.testtable limit 5;" >/home/shashanks/testfile'

Problems connecting with Teradata database using pyodbc

Currently, I'm running a simple python script to connect to a database:
import pyodbc
cnxn = pyodbc.connect('DRIVER={Teradata};DBCNAME=(MYDB);UID=(MYUSER); PWD=(MYPASS);QUIETMODE=YES')
With the server and credentials substituted in obviously. However, when running this script, I get the following error:
pyodbc.Error: ('200', '[200] [unixODBC][eaaa[DCTrdt rvr o nuhifraint o n (0) (SQLDriverConnectW)')
The only help I've been able to find is here installed the Teradata ODBC drivers, but I just don't understand why I can't connect. Anyone have any ideas on this?

You have to use like following to connect:
TDCONN = pyodbc.connect('DSN=yourDSNname;',ansi=True, autocommit=True)
You can replace DSN=yourDSNname; with what you have already.

I hit this "non english" error message problem. I think it was due to the wrong versions of libodbc.so and libodbcinst.so being used. Changing the links in /usr/lib/... to point to the teradata installed versions worked for me. Commands using the default install directories in Ubuntu 12.04, 64bit were:
cd /usr/lib/x86_64-linux-gnu
ls -lha | grep odbc (To should see the files below which are to be replaced).
sudo mv libodbc.so.1.0.0 Xlibodbc.so.1.0.0
sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbc.so libodbc.so.1.0.0
sudo mv libodbcinst.so.1.0.0 Xlibodbcinst.so.1.0.0
sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbcinst.so libodbcinst.so.1.0.0
I had also previously installed (through apt-get) odbcinst, so I redirected both files. But possibly only the first is needed.