I created an airflow image inside Kubernetes and am having a hard time connecting to my SQL Server database using Pyodbc to make my Python script work.
Here's the part of DockerFile I set up:
FROM python:3.7-slim
# apt-get and system utilities
RUN apt-get update && apt-get install -yqq \
curl gnupg gnupg2 gnupg1 \
&& rm -rf /var/lib/apt/lists/*
# adding custom MS repository
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/19.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN set -ex \
&& apt-get update -yqq \
&& apt-get upgrade -yqq \
&& apt-get install -yqq --no-install-recommends build-essential \
&& export ACCEPT_EULA=Y \
&& apt-get install -yqq msodbcsql17 freetds-dev freetds-bin tdsodbc unixodbc-dev \
&& apt-get update \
&& export ACCEPT_EULA=Y \
&& apt-get install -yqq mssql-tools \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& /bin/bash -c "source ~/.bashrc" \
&& apt-get install -yqq default-libmysqlclient-dev \
&& pip install -U pip setuptools wheel \
&& pip install -r ${AIRFLOW_HOME}/requirements.txt \
&& pip install gcloud \
&& pip install apache-airflow[crypto,postgres,jdbc,gcp_api,mssql,mysql,kubernetes,gcp]
It creates the image without problems. Now I send the part of the python script that tries to connect to Sql Server:
import pyodbc
database='dbd'
uid = 'user'
pwd = 'pass'
server = '192.168.0.1'
driver = "Driver={ODBC Driver 17 for SQL Server};server=" + server + ";database=" + database + ";uid=" + uid + ";pwd=" + pwd
# MySQL Credentials and settings
db = pyodbc.connect(driver)
cursor = db.cursor()
I checked the odbcinst.ini file inside my pod and it looks like this:
[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
Driver=/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.4.so.1.1
UsageCount=1
And finally the error message I have is as follows:
ERROR - ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]SSL Provider: [error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol] (-1) (SQLDriverConnect)')
What else can I do to test connectivity with SQL Server?
I found a much simpler lib to work with Airflow:
import pymssql
driver = pymssql.connect(server='192.168.0.1',user='user',password='pass',database='dbd',port='1433')
cursor = db.cursor()
Related
`I am trying to access DB with google colab using the below code.
`%%sh
apt-get update
apt-get install -y curl apt-transport-https
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
ACCEPT_EULA=Y apt-get install -y msodbcsql17 unixodbc-dev
!sudo apt-get install unixodbc-dev
!pip install pyodbc
!yum install unixODBC-devel
import pyodbc
conn = pyodbc.connect(DRIVER = '{ODBC Driver 17 for SQL Server}',
SERVER = 'tcp:zestlabs.database.windows.net,1433',
DATABASE = *******,
UID = ********,
PWD = **************)`
I have replaced my database name and password with "****" and also commented the code here`
enter image description here
This is the error I am getting when I run the 2nd chunk of code. Can someone help me resolve this issue?
Please Help...
Im getting below error while trying to connect to sql server with databricks runtime 10.4 LTS , while the connection was successfull with databricks runtime 7.4 LTS
Error:
OperationalError: ('08001', '[08001] [unixODBC][Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x2746 (10054) (SQLDriverConnect)')
I'm using below installation steps in an init script
dbutils.fs.put("/databricks/scripts/driversqlodbc.sh", """
#!/bin/bash
sleep 10
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update
ACCEPT_EULA=Y apt-get install msodbcsql17
apt-get -y install unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc """, True)
And my connection string is like this
odbc_driver= '{ODBC Driver 17 for SQL Server}'
conn = pyodbc.connect('DRIVER='+odbc_driver+';SERVER=tcp:'+server+';PORT=1433;DATABASE='+database+';UID='+db_user+';PWD='+ password,autocommit=True)
** EDIT **
Below is the output of cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Now my script looks like this
#!/bin/bash
sleep 10
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18
sudo ACCEPT_EULA=Y apt-get install -y mssql-tools18
echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> ~/.bashrc
source ~/.bashrc
sudo apt-get install -y unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc
And Connection String
odbc_driver= '{ODBC Driver 18 for SQL Server}'
conn = pyodbc.connect('DRIVER='+odbc_driver+';SERVER=tcp:'+server+',1433;DATABASE='+database+';UID='+db_user+';PWD='+ password,autocommit=True)
Maybe an issue with the pyodbc version. We had the same problem. The latest pyodbc update to pyodbc==4.0.34 (5 days ago) causes some issues, see https://github.com/mkleehammer/pyodbc/issues/1079 for example.
So we changed the following line in setup.py:
"pyodbc~=4.0.32",
to
"pyodbc==4.0.32",
Note the double == sign. It works with 4.0.32 for us.
Dockerfile
FROM python:3.8-alpine
WORKDIR /code
RUN echo America/Montevideo > /etc/timezone
RUN echo America/Montevideo > /etc/localtime
RUN ln -snf /usr/share/zoneinfo/America/Montevideo /etc/localtime && echo America/Montevideo > /etc/timezone
RUN apk add curl build-base unixodbc-dev unixodbc freetds-dev && pip install pyodbc==4.0.30 && pip install elasticsearch && pip install python-dateutil && pip install pytz && pip install six && pip install urllib3 && pip install pandas && pip install numpy
RUN apk add --update tzdata
RUN rm -rf /var/cache/apk/*
COPY app/ .
CMD [ "python", "./app.py" ]
app.py
driver = '/usr/lib/libtdsodbc.so'
conn = podbc.connect(f'DRIVER={driver};SERVER={globales.SERVER};PORT={globales.PORT};DATABASE={globales.DATABASE};UID={globales.USERNAME};PWD={globales.PASSWORD};')
cursor = conn.cursor()
I can't use the method executemany with this approach. I'm trying to build my dockerfile so that I can use a better driver and then deploy it to openshift so that it can insert really fast documents in my database.
How can I build this dockerfile? I can use any Python images.
Locally because I'm using Windows I can use {ODBC Driver 17 for SQL Server} and the executemany works perfectly.
I'm trying to connect to an Oracle database at my company through my docker container that contains some of my python scripts with the package cx_Oracle. After i build and run the container, i get the following error:
conn = cx_Oracle.connect("{0}/{1}#{2}".format(configOracle["username"], configOracle["password"],r"ed03:1521/configOracle["servername"]))
cx_Oracle.DatabaseError: DPI-1047: Cannot locate a 64-bit Oracle Client library: "libclntsh.so: cannot open shared object file: No such file or directory". See https://oracle.github.io/odpi/doc/installation.html#linux for help
I have an Oracle config file where the username, password, and server name are coming from and being filled in correctly. I can't seem to get it to work even after downloading the latest client from https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html.
My directory structure looks like this:
--TopDirectory
----instantclient
-------instantclient-basic-linux.x64-19.5.0.0.0dbru.zip
-------instantclient-sdk-linux.x64-19.5.0.0.0dbru.zip
----hello_oracle.py
----Dockerfile
----requirements.txt
----configOracle.json
Here is my Dockerfile:
FROM python:3.7.5
#Oracle Client setup
ENV ORACLE_HOME /opt/oracle/instantclient_19_5
ENV LD_RUN_PATH=$ORACLE_HOME
COPY instantclient/* /tmp/
RUN \
mkdir -p /opt/oracle && \
unzip "/tmp/instantclient*.zip" -d /opt/oracle && \
ln -s $ORACLE_HOME/libclntsh.so.19.1 $ORACLE_HOME/libclntsh.so
# Working directory
WORKDIR /src
# Copying requirements.txt before entire build step
COPY requirements.txt /src/requirements.txt
RUN pip install --upgrade pip
# Installing necessary packages
RUN pip install -r requirements.txt
# Copying rest of files
COPY . /src
CMD ["python3", "/src/hello_oracle.py"]
Here is my requirements.txt file:
pandas
numpy
matplotlib
keras
cx_Oracle
sklearn
tensorflow
pyopenssl
ndg-httpsclient
pyasn1
After many hours trying it, I finally solved it with this Dockerfile
Note I am using python 3.7, Django 3.0, Oracle Database 12c and Pipenv for package management
FROM python:3.7.5-slim-buster
# Installing Oracle instant client
WORKDIR /opt/oracle
RUN apt-get update && apt-get install -y libaio1 wget unzip \
&& wget https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip \
&& unzip instantclient-basiclite-linuxx64.zip \
&& rm -f instantclient-basiclite-linuxx64.zip \
&& cd /opt/oracle/instantclient* \
&& rm -f *jdbc* *occi* *mysql* *README *jar uidrvci genezi adrci \
&& echo /opt/oracle/instantclient* > /etc/ld.so.conf.d/oracle-instantclient.conf \
&& ldconfig
WORKDIR /app
COPY . . # Copy my project folder content into /app container directory
RUN pip3 install pipenv
RUN pipenv install
EXPOSE 8000
# For this statement to work you need to add the next two lines into Pipfilefile
# [scripts]
# server = "python manage.py runserver 0.0.0.0:8000"
ENTRYPOINT ["pipenv", "run", "server"]
The latest release of the Python driver for Oracle got renamed to python-oracledb and is now a 'thin' driver by default. It does not need Instant Client - it's optional. See the release announcement. The Dockerfile can simply be like:
FROM python:3.10-bullseye
RUN python -m pip install oracledb
If you want the option to use the 'Thick' mode of python-oracledb, then you could use a Dockerfile like:
FROM python:3.10-bullseye
WORKDIR /opt/oracle
RUN apt-get update && apt-get install -y libaio1
RUN wget https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip && \
unzip instantclient-basiclite-linuxx64.zip && rm -f instantclient-basiclite-linuxx64.zip && \
cd /opt/oracle/instantclient* && rm -f *jdbc* *occi* *mysql* *README *jar uidrvci genezi adrci && \
echo /opt/oracle/instantclient* > /etc/ld.so.conf.d/oracle-instantclient.conf && ldconfig
RUN python -m pip install oracledb
Oracle has Python cx_Oracle Dockerfiles at https://github.com/oracle/docker-images/tree/master/OracleLinuxDevelopers and cx_Oracle containers at https://github.com/orgs/oracle/packages
There is a two-part blog post series Docker for Oracle Database Applications in Node.js and Python that shows various ways to install. Also there is an Oracle webcast recording discussing cx_Oracle and Docker here.
If you are still using the cx_Oracle namespace, you always need to install Instant Client so a solution is to use:
FROM python:3.10-bullseye
RUN apt-get update && apt-get install -y libaio1
WORKDIR /opt/oracle
RUN wget https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip && \
unzip instantclient-basiclite-linuxx64.zip && rm -f instantclient-basiclite-linuxx64.zip && \
cd /opt/oracle/instantclient* && rm -f *jdbc* *occi* *mysql* *README *jar uidrvci genezi adrci && \
echo /opt/oracle/instantclient* > /etc/ld.so.conf.d/oracle-instantclient.conf && ldconfig
RUN python -m pip install cx_Oracle
If you use a different base image you may need to explicitly install wget and unzip.
I was told to ask this question separately but it is related to this question here.
I am having the exact problem on a docker image I created using official Ubuntu (16.04). It works from isql, but not via pyodbc connection. Below is the odbc trace:
[ODBC][60][1487069096.117665][__handles.c][460]
Exit:[SQL_SUCCESS]
Environment = 0x1458c20
[ODBC][60][1487069096.117687][SQLSetEnvAttr.c][189]
Entry:
Environment = 0x1458c20
Attribute = SQL_ATTR_ODBC_VERSION
Value = 0x3
StrLen = 4
[ODBC][60][1487069096.117695][SQLSetEnvAttr.c][363]
Exit:[SQL_SUCCESS]
[ODBC][60][1487069096.117702][SQLAllocHandle.c][375]
Entry:
Handle Type = 2
Input Handle = 0x1458c20
[ODBC][60][1487069096.117709][SQLAllocHandle.c][493]
Exit:[SQL_SUCCESS]
Output Handle = 0x148ab10
[ODBC][60][1487069096.117719][SQLDriverConnectW.c][290]
Entry:
Connection = 0x148ab10
Window Hdl = (nil)
Str In = [SERVER=server;DATABASE=db;UID=user;PWD=pwd;DRIVER={ODBC Driver 13 for SQL Server};][length = 116]
Str Out = (nil)
Str Out Max = 0
Str Out Ptr = (nil)
Completion = 0
UNICODE Using encoding ASCII 'UTF8' and UNICODE 'UTF16LE'
[ODBC][60][1487069096.118365][SQLConnect.c][1114]Can't open lib '/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.1.so.1.0' : file not found
[ODBC][60][1487069096.118384][SQLDriverConnect.c][726]
Entry:
Connection = 0x148ab10
Window Hdl = (nil)
Str In = [SERVER=server;DATABASE=database;UID=user;PWD=********;DRIVER={ODBC Driver 13 for SQL Server};][length = 116 (SQL_NTS)]
Str Out = 0x7ffc2880f570
Str Out Max = 2048
Str Out Ptr = (nil)
Completion = 0
UNICODE Using encoding ASCII 'UTF8' and UNICODE 'UTF16LE'
[ODBC][60][1487069096.118786][SQLConnect.c][1114]Can't open lib '/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.1.so.1.0' : file not found
[ODBC][60][1487069096.118802][SQLGetDiagRec.c][680]
Entry:
Connection = 0x148ab10
Rec Number = 1
SQLState = 0x7ffc28810160
Native = 0x7ffc2881014c
Message Text = 0x7ffc28810170
Buffer Length = 1023
Text Len Ptr = 0x7ffc2881014a
[ODBC][60][1487069096.118816][SQLGetDiagRec.c][717]
Exit:[SQL_SUCCESS]
SQLState = 01000
Native = 0x7ffc2881014c -> 0
Message Text = [[unixODBC][Driver Manager]Can't open lib '/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.1.so.1.0' : file not found]
[ODBC][60][1487069096.118832][SQLFreeHandle.c][284]
Entry:
Handle Type = 2
Input Handle = 0x148ab10
[ODBC][60][1487069096.118839][SQLFreeHandle.c][333]
Exit:[SQL_SUCCESS]
Here's my dockerfile:
FROM ubuntu:latest
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN apt-get update && \
apt-get clean && \
apt-get -y install curl build-essential \
libssl-dev libldap2-dev libffi-dev libpq-dev apt-transport-https dialog
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - && \
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list && \
apt-get update && \
ACCEPT_EULA=Y apt-get -y install msodbcsql=13.1.1.0-1 mssql-tools && \
apt-get -y install unixodbc-dev-utf16
CMD ["bin", "bash"]
Thanks to Meet and his buddy, Luis, at Microsoft, I was able to use conda distribution with pyodbc in a docker container to connect with SQL server. Below is the dockerfile that they configured for me -
# mssql-python-pyodbc
# Python runtime with pyodbc to connect to SQL Server
FROM ubuntu:16.04
# apt-get and system utilities
RUN apt-get update && apt-get install -y \
curl apt-utils apt-transport-https debconf-utils gcc build-essential g++-5\
&& rm -rf /var/lib/apt/lists/*
# adding custom MS repository
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
# install SQL Server drivers
RUN apt-get update && ACCEPT_EULA=Y apt-get -y install msodbcsql
RUN apt-get -y install unixodbc unixodbc-dev
# install SQL Server tools
RUN apt-get update && ACCEPT_EULA=Y apt-get -y install mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
RUN /bin/bash -c "source ~/.bashrc"
# python libraries
RUN apt-get update && apt-get install -y \
python-pip python-dev python-setuptools \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
# install necessary locales
RUN apt-get update && apt-get install -y locales \
&& echo "en_US.UTF-8 UTF-8" > /etc/locale.gen \
&& locale-gen
RUN pip install --upgrade pip
# install SQL Server Python SQL Server connector module - pyodbc
RUN pip install pyodbc
RUN curl -LO https://repo.continuum.io/archive/Anaconda2-4.3.0-Linux-x86_64.sh && \
bash Anaconda2-4.3.0-Linux-x86_64.sh -p /Anaconda -b && \
rm Anaconda2-4.3.0-Linux-x86_64.sh && \
rm -rf /var/lib/apt/lists/*
ENV PATH $PATH:/Anaconda/bin
RUN conda update -y conda
# add sample code
RUN mkdir /sample
ADD . /sample
WORKDIR /sample
Seems like adding Anaconda to the path is causing the issue. If we add to the end of the PATH it seems to work.