Objective: Connect to a MS SQL Server in AWS Lambda
Error From AWS Lambda:
START RequestId: 37951004-404b-11e7-98fd-5177b3a46ec6 Version: $LATEST
module initialization error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 13 for SQL Server' : file not found (0) (SQLDriverConnect)")
END RequestId: 37951004-404b-11e7-98fd-5177b3a46ec6
My Approach:
refering to
Tesseract OCR on AWS Lambda via virtualenv
Installed unixODBC and ODBC Driver 13 for SQL Server in an aws ec2 instance
Created a deployment package (i.e. pip install -t pyodbc /home/ec2-user/lambda and copied the relevant files to zip root)
mkdir lib and copied all shared libraries by looking up ldd pyodbc.so * ldd libmsodbcsql-13.0.so.1.0
change LD_LIBRARY_PATH to lib in lambda.py and upload the zip to Lambda
Zip File Structure:
.
+-- lambda.py
+-- pyodbc.so
+-- pyodbc-4.0.16-py2.7.egg-info
+-- lib
| +-- libodbc*.so etc.
My Guess:
From the response it looks like everything is running OK except for the unixODBC manager cannot find the driver on AWS Lambda Instance
Question:
How do I bundle the driver with my package for AWS Lambda to use? Or Is there a way to tell ODBC driver manager to look for libmsodbcsql-13.0.so.1.0?
I have tried offline installation of unixODBC and set ./configure --prefix=/home/ec2-user/lambda --libdir=.. and manually changing odbcinst.ini but no hope.
(sorry I am not so familiar with anything outside python environment so I am not sure about all the shared libraries stuff and couldn't solve my problem by just googling for a few days)
Is there a way to tell ODBC driver manager to look for libmsodbcsql-13.0.so.1.0?
You can run odbcinst -j and check where the driver ini file is located (odbcinst.ini) so you can set the path there. Otherwise create the new one at ~/.odbcinst.ini which can define the path to SQL ODBC driver library.
Example INI config file:
[ODBC Driver 13 for SQL Server]
Description=Microsoft ODBC Driver 13 for SQL Server
Driver=/path/to/libmsodbcsql-13.0.so.1.0
Here is the example command to create the config file (~/.odbcinst.ini):
printf "[ODBC Driver 13 for SQL Server]\nDescription=Microsoft ODBC Driver 13 for SQL Server\nDriver=/path/to/libmsodbcsql-13.0.so.1.0\n" >> ~/.odbcinst.ini
Anaconda
If the driver file exists, but still you got file not found, run: conda update libgcc.
See: PyODBC : can't open the driver even if it exists.
Compilation Approach
By default unixODBC looks for odbc.ini inside /etc/. In a Lambda function, you can't modify that. (Well, perhaps you can from the python script at runtime, but that's not ideal.)
If you're compiling unixODBC yourself, you can try adding an extra argument to ./configure, --sysconfdir, as well as --prefix. See here for more detail.
Personally I was putting odbc inside a lambda layer, which goes to /opt. I think the lambda itself is unzipped to /var/task. This means that when you compile on EC2, you should point to /var/task. (You might have to manually create this directory if it doesn't yet exist.)
./configure --sysconfdir=/var/task --prefix=/var/task
Simple Yum Approach
Personally I found that it's easier to just yum install unixODBC, and then /usr/lib64/libodbc* to /lib/ inside the zip, and /opt to / inside the zip, and /usr/bin/odbc* to /bin/ inside the zip. Then the only extra trick was to set environment variables in the lambda.
ODBCINI=/var/task/odbc.ini
ODBCSYSINI=/var/task/
Related
I am very new to python code and doing some small tests to verify functionality
Currently trying to establish the connection between an RDS MySQL and a python lambda function.
However, it seems to fail in the code itself and I am not sure why this happens.
There are a couple of guides out there but they all seem to be outdated and fail to work for me.
These are the steps I took to get it working(using MAC-12.3.1 and VS-Studio 1.62.3):
created MYSQL RDS
connected to the MYSQL RDS and created a database called "igor" with table name "lumigor", with 2 columns: id and name (populated with random data).
created on the local machine a folder to contain the code and the package.
installed version Python 3.8.9
created lambda function file app.py with the following code:
import pymysql.cursors
# Connect to the database
connection = pymysql.connect(host='rds end point',
user='user',
password='pswrd',
database='igor',
cursorclass=pymysql.cursors.DictCursor)
with connection:
with connection.cursor() as cursor:
# Read a single record
sql = "SELECT * FROM `Lumigor`"
cursor.execute(sql, ('webmaster#python.org',))
result = cursor.fetchone()
print(result)
```
I added requirement.txt file with the following command
python3 -m pip install PyMySQL && pip3 freeze > requirements.txt --target...
But now I get an error from the visual studio:
"Import "pymysql.cursors" could not be resolved from sourcePylance"
When I zip the file and upload it to lambda, run a test it returns an error
{
"errorMessage": "Unable to import module 'app': No module named 'pymysql.cursors'",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
}
It seems like the dependcies are missing even though installed them and they exist in the directory
The proper way to add pymysql to your lambda is by creating a dedicated layer a as described in the AWS blog:
How do I create a Lambda layer using a simulated Lambda environment with Docker?
Create empty folder, e.g. mylayer.
Go to the folder and create requirements.txt file with the content of
PyMySQL
Run the following docker command (new image from
lambci/docker-lambda for Python 3.9):
docker run --rm --volume "$PWD:/var/task" --workdir /var/task senorcoder/aws-lambda-env:python3.9_build pip install -Ur requirements.txt --target python
Create layer as zip:
zip -r mypymysqllayer.zip python > /dev/null
Create lambda layer based on mypymysqllayer.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.9.
Add the layer to your function:
Alternatively, create your function as Lambda container image
I have been struggling, and am unable to figure out how to connect to SQL Server using python 3.6 via pydobc and AWS Lambda.
I followed the instructions provided by AWS to create an Amazon Linux AMI on which I was able to install the Microsoft ODBC drivers (v13 and v17), upgrade the unixODBC to a supported version, and get my python code to connect to the AWS RDS SQL Server instance.
However, I have not been able to figure out how to package those changes successfully to deploy this code to AWS Lambda and have it work.
I get one of two errors, depending on how I try to reference the ODBC driver. Using this syntax
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=servername.account.region.rds.amazonaws.com,port;DATABASE=database;UID=user;PWD=password'
I get the error:
"errorMessage": "('01000', \"[01000] [unixODBC][Driver Manager]Can't open lib 'SQL Server' : file not found (0) (SQLDriverConnect)\")",
I have tried using other driver aliases (ODBC Driver 17 for SQL Server, ODBC Driver 13 for SQL Server), with the same results.
Using the syntax:
cnxn = pyodbc.connect('DRIVER=lib/libmsodbcsql-13.so;SERVER=servername.account.region.rds.amazonaws.com,port;DATABASE=database;UID=user;PWD=password')
I get the error:
'IM004', "[IM004] [unixODBC][Driver Manager]Driver's SQLAllocHandle on SQL_HANDLE_HENV failed (0) (SQLDriverConnect)")
I have the following code in my .ZIP deployment file:
simple_db.py - My code to create the connection
pyodbc.so - Lambda version of pyodbc from https://github.com/Miserlou/lambda-packages/tree/master/lambda_packages/pyodbc)
odbcinst.ini - Attempt to use a Linux-version of odbcinst to list the driver(s).
lib/libmsodbcsql-13.so - Copied from the Amazon Linux install
libodbc.so.2 - Copied from Amazon Linux install as well, attempt to deploy unixODBC version.
I've toyed around with directories, and adding more libodbc*.* files from the /usr/lib64 directory, but nothing has worked so far. As well as bringing over the entire msodbcsql directory (with /etc, /include, /lib64, and /share).
Any help would be greatly appreciated!
Based in this answer: AWS Lambda function to connect to SQL Server with Python
You will need to compile unixODBC, take its shared libraries. Then install the MS driver, takes its shared libs and gather together
# Start a container that mimic the lambda environment
docker run -it --rm --entrypoint bash -e ODBCINI=/var/task -e ODBCSYSINI=/var/task -v "$PWD":/var/task lambci/lambda:build-python2.7
# Then, download ODBC source code, compile and take the output
curl ftp://ftp.unixodbc.org/pub/unixODBC/unixODBC-2.3.5.tar.gz -O
tar xvzf unixODBC-2.3.5.tar.gz
cd unixODBC-2.3.5
./configure --sysconfdir=/var/task --disable-gui --disable-drivers --enable-iconv --with-iconv-char-enc=UTF8 --with-iconv-ucode-enc=UTF16LE --prefix=/home
make install
cd ..
mv /home/* .
mv unixODBC-2.3.5 unixODBC-2.3.5.tar.gz /tmp/
# Install MSsql odbc driver
curl https://packages.microsoft.com/config/rhel/6/prod.repo > /etc/yum.repos.d/mssql-release.repo
ACCEPT_EULA=Y yum -y install msodbcsql
export CFLAGS="-I/var/task/include"
export LDFLAGS="-L/var/task/lib"
# Then you can install pyodbc (or pip install -t . -r requirements.txt)
pip install pyodbc -t .
cp -r /opt/microsoft/msodbcsql .
cat <<EOF > odbcinst.ini
[ODBC Driver 13 for SQL Server]
Description=Microsoft ODBC Driver 13 for SQL Server
Driver=/var/task/msodbcsql/lib64/libmsodbcsql-13.1.so.9.2
UsageCount=1
EOF
cat <<EOF > odbc.ini
[ODBC Driver 13 for SQL Server]
Driver = ODBC Driver 13 for SQL Server
Description = My ODBC Driver 13 for SQL Server
Trace = No
EOF
# Test if it works
python -c "import pyodbc; print(pyodbc.drivers());"
python -c 'import pyodbc;conn = pyodbc.connect("DRIVER={ODBC Driver 13 for SQL Server}; SERVER=YOUr_SERVER:ADD;PORT=1443;DATABASE=TestDB;UID=SA;PWD=<YourStrong!Passw0rd>");crsr = conn.cursor();rows = crsr.execute("select ##VERSION").fetchall();print(rows);crsr.close();conn.close()'
I am trying to use SQLAlchemy to connect to our Teradata environment and execute a query. I ran the script on a Windows 7 machine using an Anaconda Python 2.7 environment and Jupyter notebook. When I moved this to our Linux server and ran it in an Anaconda Python 2.7 environment it doesn't work. It complains that it can't find the driver Teradata. I have ran this as root and my non-root user.
I am trying to run this in a Jupyter notebook running on the server
from sqlalchemy import create_engine
# connect
td_engine = create_engine('teradata://username:password#teradata_database:22/')
# execute sql
sql="SELECT * FROM sometable"
result = td_engine.execute(sql)
Here is the traceback
Here is what I've done on the RHEL 7 server inside a py27 env
conda install sqlalchemy
pip install teradata
pip install sqlalchemy-teradata
yum install unixODBC
I downloaded the latest Teradata ODBC Driver for Linux
I extracted the file
tar -zxf tdodbc1510__linux_indep_15.10.01.04-1.tar.gz
This gives me three more tar.gz files and a text file of the same name.
I then extract each of these tar.gz files
tar -zxf tdicu1510__linux_indep.15.10.04-1.tar.gz
tar -zxf tdodbc1510__linux_indep.15.10.01.04-1.tar.gz
tar -zxf TeraGSS_linux-x64_linux_indep.15.10.03.02-1.tar.gz
This creates a directory for each root name (tdicu1510, tdodbc1510, TeraGSS)
I cd into each directory and switch to korn shell
/usr/bin/ksh
for the rpm's inside the three directories I install them
rpm -ihv tdicu1510_linux_x64-15.10.03.02-1.noarch.rpm
rpm -ihv TeraGSS_linux_x64-15.10.03.02-1.noarch.rpm
rpm -ihv tdodbc1510_linux_x64-15.10.03.02-1.noarch.rpm
This creates an /opt/teradata directory
I add this to /usr/local/etc/odbcinst.ini
[Teradata]
Driver=/opt/teradata/client/15.10/odbc_64/lib/tdata.so
APILevel=CORE
ConnectFunctions=YYY
DriverODBCVer=3.51
SQLLevel=1
I add this to /usr/local/etc/odbc.ini
[ODBC Data Sources]
TDDSN=tdata.so
[ODBC]
InstallDir=/opt/teradata/client/15.10/odbc_64
Trace=0
TraceDll=/opt/teradata/client/15.10/odbc_64/lib/odbctrac.so
TraceFile=/usr/teradata_logs/odbcusr/trace.log
TraceAutoStop=0
[TDDSN]
Driver=/opt/teradata/client/15.10/odbc_64/lib/tdata.so
Description=Teradata database
DBCName=<MachineName or ip>
LastUser=
Username=
Password=
Database=
Here is my ~/.bashrc
export PATH="/opt/miniconda2/bin:$PATH
export ORACLE_HOME="/usr/lib/oracle/12.1/client64"
export PATH="$PATH:$ORACLE_HOME/bin"
export LD_LIBRARY_PATH="$ORACLE_HOME/lib"
export TNS_ADMIN="$ORACLE_HOME/network/admin"
export TERADATA="/opt/teradata/client/15.10"
export ODBCINI="/opt/teradata/client/15.10/odbc_64/odbc.ini"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$TERADATA/lib64"
echo $LD_LIBRARY_PATH shows /usr/lib/oracle/12.1/client64/lib:/opt/teradata/client/15.10/lib64
Any ideas what I am doing wrong? Why can't SQLAlchemy see the Teradata ODBC driver on the Linux server?
I too had the same issue. I overcame that by using python teradatasqlalchemy module and modifying the connection string from:
create_engine('**teradata**://username:password#teradata_database:22/')
to
create_engine('**teradatasql**://username:password#teradata_database:22/')
I use PHP PDO to connect a MySql Db and it works great. I have something like:
$dsn = 'mysql:host=localhost;dbname=database_name';
$user_db = 'admin';
$password = 'password';
$pdo = new PDO($dsn, $user_db, $password);
Now I need to load same database from a python script and I have to use pypodbc module
But I'm getting some issue:
If I do (on Python):
pyodbc.connect('DRIVER={MySQL};SERVER=localhost;DATABASE=database_name;UID=admin;PWD=password;')
I got en error on log:
pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open
lib '/usr/lib64/libmyodbc5.so' : file not found (0)
(SQLDriverConnect)")
If i check the /etc/odbcinst.ini i can see:
# Driver from the mysql-connector-odbc package
# Setup from the unixODBC package
[MySQL]
Description = ODBC for MySQL
Driver = /usr/lib/libmyodbc5.so
Setup = /usr/lib/libodbcmyS.so
Driver64 = /usr/lib64/libmyodbc5.so
Setup64 = /usr/lib64/libodbcmyS.so
FileUsage = 1
I tried to add the the package mysql-connector-odbc by YUM and i got mysql-connector-odbc.x86_64 0:5.1.5r1144-7.el6
And then running my script I got a new error:
/usr/local/bin/python2.7: relocation error: /usr/lib64/libmyodbc5.so: symbol strmov, version libmysqlclient_16 not defined in file libmysqlclient_r.so.16 with link time reference
It seems this version is not compatible with MySql which I have: 5.5.37-cll - MySQL Community Server (GPL)
I did a YUM REMOVE to restore previous conf.
And now ? Any suggestions ? Thanks!
My configuration:
My Server: CENTOS 6.6 x86_64 virtuozzo
MySql: 5.5.37-cll - MySQL Community Server (GPL)
Finally I fixed it!
yum install unixODBC-devel
yum install mysql-connector-odbc
yum install openssl098e
and than:
rpm -ivh libmysqlclient16-5.1.69-1.w5.x86_64.rpm
Now the
pyodbc.connect('DRIVER={MySQL};SERVER=localhost;DATABASE=database_name;UID=admin;PWD=password;')
works!! yeah!
Currently, I'm running a simple python script to connect to a database:
import pyodbc
cnxn = pyodbc.connect('DRIVER={Teradata};DBCNAME=(MYDB);UID=(MYUSER); PWD=(MYPASS);QUIETMODE=YES')
With the server and credentials substituted in obviously. However, when running this script, I get the following error:
pyodbc.Error: ('200', '[200] [unixODBC][eaaa[DCTrdt rvr o nuhifraint o n (0) (SQLDriverConnectW)')
The only help I've been able to find is here installed the Teradata ODBC drivers, but I just don't understand why I can't connect. Anyone have any ideas on this?
You have to use like following to connect:
TDCONN = pyodbc.connect('DSN=yourDSNname;',ansi=True, autocommit=True)
You can replace DSN=yourDSNname; with what you have already.
I hit this "non english" error message problem. I think it was due to the wrong versions of libodbc.so and libodbcinst.so being used. Changing the links in /usr/lib/... to point to the teradata installed versions worked for me. Commands using the default install directories in Ubuntu 12.04, 64bit were:
cd /usr/lib/x86_64-linux-gnu
ls -lha | grep odbc (To should see the files below which are to be replaced).
sudo mv libodbc.so.1.0.0 Xlibodbc.so.1.0.0
sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbc.so libodbc.so.1.0.0
sudo mv libodbcinst.so.1.0.0 Xlibodbcinst.so.1.0.0
sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbcinst.so libodbcinst.so.1.0.0
I had also previously installed (through apt-get) odbcinst, so I redirected both files. But possibly only the first is needed.