I am trying to use SQLAlchemy to connect to our Teradata environment and execute a query. I ran the script on a Windows 7 machine using an Anaconda Python 2.7 environment and Jupyter notebook. When I moved this to our Linux server and ran it in an Anaconda Python 2.7 environment it doesn't work. It complains that it can't find the driver Teradata. I have ran this as root and my non-root user.
I am trying to run this in a Jupyter notebook running on the server
from sqlalchemy import create_engine
# connect
td_engine = create_engine('teradata://username:password#teradata_database:22/')
# execute sql
sql="SELECT * FROM sometable"
result = td_engine.execute(sql)
Here is the traceback
Here is what I've done on the RHEL 7 server inside a py27 env
conda install sqlalchemy
pip install teradata
pip install sqlalchemy-teradata
yum install unixODBC
I downloaded the latest Teradata ODBC Driver for Linux
I extracted the file
tar -zxf tdodbc1510__linux_indep_15.10.01.04-1.tar.gz
This gives me three more tar.gz files and a text file of the same name.
I then extract each of these tar.gz files
tar -zxf tdicu1510__linux_indep.15.10.04-1.tar.gz
tar -zxf tdodbc1510__linux_indep.15.10.01.04-1.tar.gz
tar -zxf TeraGSS_linux-x64_linux_indep.15.10.03.02-1.tar.gz
This creates a directory for each root name (tdicu1510, tdodbc1510, TeraGSS)
I cd into each directory and switch to korn shell
/usr/bin/ksh
for the rpm's inside the three directories I install them
rpm -ihv tdicu1510_linux_x64-15.10.03.02-1.noarch.rpm
rpm -ihv TeraGSS_linux_x64-15.10.03.02-1.noarch.rpm
rpm -ihv tdodbc1510_linux_x64-15.10.03.02-1.noarch.rpm
This creates an /opt/teradata directory
I add this to /usr/local/etc/odbcinst.ini
[Teradata]
Driver=/opt/teradata/client/15.10/odbc_64/lib/tdata.so
APILevel=CORE
ConnectFunctions=YYY
DriverODBCVer=3.51
SQLLevel=1
I add this to /usr/local/etc/odbc.ini
[ODBC Data Sources]
TDDSN=tdata.so
[ODBC]
InstallDir=/opt/teradata/client/15.10/odbc_64
Trace=0
TraceDll=/opt/teradata/client/15.10/odbc_64/lib/odbctrac.so
TraceFile=/usr/teradata_logs/odbcusr/trace.log
TraceAutoStop=0
[TDDSN]
Driver=/opt/teradata/client/15.10/odbc_64/lib/tdata.so
Description=Teradata database
DBCName=<MachineName or ip>
LastUser=
Username=
Password=
Database=
Here is my ~/.bashrc
export PATH="/opt/miniconda2/bin:$PATH
export ORACLE_HOME="/usr/lib/oracle/12.1/client64"
export PATH="$PATH:$ORACLE_HOME/bin"
export LD_LIBRARY_PATH="$ORACLE_HOME/lib"
export TNS_ADMIN="$ORACLE_HOME/network/admin"
export TERADATA="/opt/teradata/client/15.10"
export ODBCINI="/opt/teradata/client/15.10/odbc_64/odbc.ini"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$TERADATA/lib64"
echo $LD_LIBRARY_PATH shows /usr/lib/oracle/12.1/client64/lib:/opt/teradata/client/15.10/lib64
Any ideas what I am doing wrong? Why can't SQLAlchemy see the Teradata ODBC driver on the Linux server?
I too had the same issue. I overcame that by using python teradatasqlalchemy module and modifying the connection string from:
create_engine('**teradata**://username:password#teradata_database:22/')
to
create_engine('**teradatasql**://username:password#teradata_database:22/')
Related
Suppose that due to an HTTP 403 Error it's not possible to download the packages from the PyPi repo (nor pip install <package> commands) which causes me to install the pyodbc by cloning the repo from Github (https://github.com/mkleehammer/pyodbc) and by running the next .cmd windows file:
cd "root_folder"
git activate
git clone https://github.com/mkleehammer/pyodbc.git --depth 1
Note that this package is downloaded to the same root folder where my python script is, after this I try to set a connection to Microsoft SQL Server:
import pyodbc as pyodbc
# set connection settings
server="servername"
database="DB1"
user="user1"
password="123"
# establishing connection to db
conn = pyodbc.connect("DRIVER={SQL Server};SERVER="+server+";DATABASE="+database+";UID="+user+";PWD="+password)
cursor=conn.cursor()
print("Succesful connection to sql server")
However, when I run the above code the next traceback error arises:
Traceback (most recent call last):
File "/dcleaner.py", line 47, in
conn = pyodbc.connect("DRIVER={SQL Server};SERVER="+server+";DATABASE="+database+";UID="+user+";PWD="+password)
AttributeError: module 'pyodbc' has no attribute 'connect'
Do you know how can I properly connect from a py script to a sql-server based database?
After you have cloned PYODBC
cd "root_folder"
git activate
git clone https://github.com/mkleehammer/pyodbc.git --depth 1
On your Local Machine, Go into the cloned directory and open terminal and run below command
python setup.py build
if it errors then try to install appropriate C++ compiler (the error might reveal this detail/ on VSCode it gave the URL to open and download, which I have shared below) install it from here link - choose this one
reboot machine and run this again
python setup.py build #if success then continue with below one
python setup.py install
after that you should be able to import and run the below from your local machine
import pyodbc as pyodbc
I have been struggling, and am unable to figure out how to connect to SQL Server using python 3.6 via pydobc and AWS Lambda.
I followed the instructions provided by AWS to create an Amazon Linux AMI on which I was able to install the Microsoft ODBC drivers (v13 and v17), upgrade the unixODBC to a supported version, and get my python code to connect to the AWS RDS SQL Server instance.
However, I have not been able to figure out how to package those changes successfully to deploy this code to AWS Lambda and have it work.
I get one of two errors, depending on how I try to reference the ODBC driver. Using this syntax
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=servername.account.region.rds.amazonaws.com,port;DATABASE=database;UID=user;PWD=password'
I get the error:
"errorMessage": "('01000', \"[01000] [unixODBC][Driver Manager]Can't open lib 'SQL Server' : file not found (0) (SQLDriverConnect)\")",
I have tried using other driver aliases (ODBC Driver 17 for SQL Server, ODBC Driver 13 for SQL Server), with the same results.
Using the syntax:
cnxn = pyodbc.connect('DRIVER=lib/libmsodbcsql-13.so;SERVER=servername.account.region.rds.amazonaws.com,port;DATABASE=database;UID=user;PWD=password')
I get the error:
'IM004', "[IM004] [unixODBC][Driver Manager]Driver's SQLAllocHandle on SQL_HANDLE_HENV failed (0) (SQLDriverConnect)")
I have the following code in my .ZIP deployment file:
simple_db.py - My code to create the connection
pyodbc.so - Lambda version of pyodbc from https://github.com/Miserlou/lambda-packages/tree/master/lambda_packages/pyodbc)
odbcinst.ini - Attempt to use a Linux-version of odbcinst to list the driver(s).
lib/libmsodbcsql-13.so - Copied from the Amazon Linux install
libodbc.so.2 - Copied from Amazon Linux install as well, attempt to deploy unixODBC version.
I've toyed around with directories, and adding more libodbc*.* files from the /usr/lib64 directory, but nothing has worked so far. As well as bringing over the entire msodbcsql directory (with /etc, /include, /lib64, and /share).
Any help would be greatly appreciated!
Based in this answer: AWS Lambda function to connect to SQL Server with Python
You will need to compile unixODBC, take its shared libraries. Then install the MS driver, takes its shared libs and gather together
# Start a container that mimic the lambda environment
docker run -it --rm --entrypoint bash -e ODBCINI=/var/task -e ODBCSYSINI=/var/task -v "$PWD":/var/task lambci/lambda:build-python2.7
# Then, download ODBC source code, compile and take the output
curl ftp://ftp.unixodbc.org/pub/unixODBC/unixODBC-2.3.5.tar.gz -O
tar xvzf unixODBC-2.3.5.tar.gz
cd unixODBC-2.3.5
./configure --sysconfdir=/var/task --disable-gui --disable-drivers --enable-iconv --with-iconv-char-enc=UTF8 --with-iconv-ucode-enc=UTF16LE --prefix=/home
make install
cd ..
mv /home/* .
mv unixODBC-2.3.5 unixODBC-2.3.5.tar.gz /tmp/
# Install MSsql odbc driver
curl https://packages.microsoft.com/config/rhel/6/prod.repo > /etc/yum.repos.d/mssql-release.repo
ACCEPT_EULA=Y yum -y install msodbcsql
export CFLAGS="-I/var/task/include"
export LDFLAGS="-L/var/task/lib"
# Then you can install pyodbc (or pip install -t . -r requirements.txt)
pip install pyodbc -t .
cp -r /opt/microsoft/msodbcsql .
cat <<EOF > odbcinst.ini
[ODBC Driver 13 for SQL Server]
Description=Microsoft ODBC Driver 13 for SQL Server
Driver=/var/task/msodbcsql/lib64/libmsodbcsql-13.1.so.9.2
UsageCount=1
EOF
cat <<EOF > odbc.ini
[ODBC Driver 13 for SQL Server]
Driver = ODBC Driver 13 for SQL Server
Description = My ODBC Driver 13 for SQL Server
Trace = No
EOF
# Test if it works
python -c "import pyodbc; print(pyodbc.drivers());"
python -c 'import pyodbc;conn = pyodbc.connect("DRIVER={ODBC Driver 13 for SQL Server}; SERVER=YOUr_SERVER:ADD;PORT=1443;DATABASE=TestDB;UID=SA;PWD=<YourStrong!Passw0rd>");crsr = conn.cursor();rows = crsr.execute("select ##VERSION").fetchall();print(rows);crsr.close();conn.close()'
How to connect oracle database server from python inside unix server ?
I cant install any packages like cx_Orcale, pyodbc etc.
Please consider even PIP is not available to install.
It my UNIX PROD server, so I have lot of restriction.
I tried to run the sql script from sqlplus command and its working.
Ok, so there is sqlplus and it works, this means that oracle drivers are there.
Try to proceed as follows:
1) create a python virtualenv in your $HOME. In python3
python -m venv $HOME/my_venv
2) activate it
source $HOME/my_venv/bin/activate[.csh] # .csh is for cshell, for bash otherwise
3) install pip using python binary from you new virtualenv, it is well described here: https://pip.pypa.io/en/stable/installing/
TL;DR:
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get_pip.py (this should install pip into your virtualenv as $HOME/my_env/bin/pip[3]
4) install cx_Oracle:
pip install cx_Oracle
Now you should be able to import it in your python code and connect to an oracle DB.
I tried to connect Oracle database via SQLPLUS and I am calling the script with below way :
os.environ['ORACLE_HOME'] = '<ORACEL PATH>'
os.chdir('<DIR NAME>')
VARIBALE=os.popen('./script_to_Call_sql_script.sh select.sql').read()
My shell script: script_to_Call_sql_script.sh
#!/bin/bash
envFile=ENV_FILE_NAME
envFilePath=<LOACTION_OF_ENV>${envFile}
ORACLE_HOME=<ORACLE PATH>
if [[ $# -eq 0 ]]
then
echo "USAGES: Please provide the positional parameter"
echo "`$basename $0` <SQL SCRIPT NAME>"
fi
ECR=`$ORACLE_HOME/bin/sqlplus -s /#<server_name><<EOF
set pages 0
set head off
set feed off
#$1;
exit
EOF`
echo $ECR
Above things help me to do my work done on Production server.
Objective: Connect to a MS SQL Server in AWS Lambda
Error From AWS Lambda:
START RequestId: 37951004-404b-11e7-98fd-5177b3a46ec6 Version: $LATEST
module initialization error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 13 for SQL Server' : file not found (0) (SQLDriverConnect)")
END RequestId: 37951004-404b-11e7-98fd-5177b3a46ec6
My Approach:
refering to
Tesseract OCR on AWS Lambda via virtualenv
Installed unixODBC and ODBC Driver 13 for SQL Server in an aws ec2 instance
Created a deployment package (i.e. pip install -t pyodbc /home/ec2-user/lambda and copied the relevant files to zip root)
mkdir lib and copied all shared libraries by looking up ldd pyodbc.so * ldd libmsodbcsql-13.0.so.1.0
change LD_LIBRARY_PATH to lib in lambda.py and upload the zip to Lambda
Zip File Structure:
.
+-- lambda.py
+-- pyodbc.so
+-- pyodbc-4.0.16-py2.7.egg-info
+-- lib
| +-- libodbc*.so etc.
My Guess:
From the response it looks like everything is running OK except for the unixODBC manager cannot find the driver on AWS Lambda Instance
Question:
How do I bundle the driver with my package for AWS Lambda to use? Or Is there a way to tell ODBC driver manager to look for libmsodbcsql-13.0.so.1.0?
I have tried offline installation of unixODBC and set ./configure --prefix=/home/ec2-user/lambda --libdir=.. and manually changing odbcinst.ini but no hope.
(sorry I am not so familiar with anything outside python environment so I am not sure about all the shared libraries stuff and couldn't solve my problem by just googling for a few days)
Is there a way to tell ODBC driver manager to look for libmsodbcsql-13.0.so.1.0?
You can run odbcinst -j and check where the driver ini file is located (odbcinst.ini) so you can set the path there. Otherwise create the new one at ~/.odbcinst.ini which can define the path to SQL ODBC driver library.
Example INI config file:
[ODBC Driver 13 for SQL Server]
Description=Microsoft ODBC Driver 13 for SQL Server
Driver=/path/to/libmsodbcsql-13.0.so.1.0
Here is the example command to create the config file (~/.odbcinst.ini):
printf "[ODBC Driver 13 for SQL Server]\nDescription=Microsoft ODBC Driver 13 for SQL Server\nDriver=/path/to/libmsodbcsql-13.0.so.1.0\n" >> ~/.odbcinst.ini
Anaconda
If the driver file exists, but still you got file not found, run: conda update libgcc.
See: PyODBC : can't open the driver even if it exists.
Compilation Approach
By default unixODBC looks for odbc.ini inside /etc/. In a Lambda function, you can't modify that. (Well, perhaps you can from the python script at runtime, but that's not ideal.)
If you're compiling unixODBC yourself, you can try adding an extra argument to ./configure, --sysconfdir, as well as --prefix. See here for more detail.
Personally I was putting odbc inside a lambda layer, which goes to /opt. I think the lambda itself is unzipped to /var/task. This means that when you compile on EC2, you should point to /var/task. (You might have to manually create this directory if it doesn't yet exist.)
./configure --sysconfdir=/var/task --prefix=/var/task
Simple Yum Approach
Personally I found that it's easier to just yum install unixODBC, and then /usr/lib64/libodbc* to /lib/ inside the zip, and /opt to / inside the zip, and /usr/bin/odbc* to /bin/ inside the zip. Then the only extra trick was to set environment variables in the lambda.
ODBCINI=/var/task/odbc.ini
ODBCSYSINI=/var/task/
Currently, I'm running a simple python script to connect to a database:
import pyodbc
cnxn = pyodbc.connect('DRIVER={Teradata};DBCNAME=(MYDB);UID=(MYUSER); PWD=(MYPASS);QUIETMODE=YES')
With the server and credentials substituted in obviously. However, when running this script, I get the following error:
pyodbc.Error: ('200', '[200] [unixODBC][eaaa[DCTrdt rvr o nuhifraint o n (0) (SQLDriverConnectW)')
The only help I've been able to find is here installed the Teradata ODBC drivers, but I just don't understand why I can't connect. Anyone have any ideas on this?
You have to use like following to connect:
TDCONN = pyodbc.connect('DSN=yourDSNname;',ansi=True, autocommit=True)
You can replace DSN=yourDSNname; with what you have already.
I hit this "non english" error message problem. I think it was due to the wrong versions of libodbc.so and libodbcinst.so being used. Changing the links in /usr/lib/... to point to the teradata installed versions worked for me. Commands using the default install directories in Ubuntu 12.04, 64bit were:
cd /usr/lib/x86_64-linux-gnu
ls -lha | grep odbc (To should see the files below which are to be replaced).
sudo mv libodbc.so.1.0.0 Xlibodbc.so.1.0.0
sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbc.so libodbc.so.1.0.0
sudo mv libodbcinst.so.1.0.0 Xlibodbcinst.so.1.0.0
sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbcinst.so libodbcinst.so.1.0.0
I had also previously installed (through apt-get) odbcinst, so I redirected both files. But possibly only the first is needed.