I am trying to run a Python job using a VM on Azure Batch. It's a simple script to add a line to my Azure SQL Database. I downloaded the ODBC connection string straight from my Azure portal yet I get this error. The strange thing is I can run the script perfectly fine on my own machine. I've configured the VM to install the version of Python that I need and then execute my script - I'm at a complete loss. Any ideas?
cnxn = pyodbc.connect('Driver={ODBC Driver 13 for SQL Server};Server=tcp:svr-something.database.windows.net,fakeport232;Database=db-something-prod;Uid=something#svr-something;Pwd{fake_passwd};Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;')
Traceback (most recent call last):
File "D:\batch\tasks\apppackages\batch_python_test1.02018-11-12-14-
30\batch_python_test\python_test.py", line 12, in
r'Driver={ODBC Driver 13 for SQL Server};Server=tcp:svr-
mydatabase.database.windows.net,'
pyodbc.InterfaceError: ('IM002', '[IM002] [Microsoft][ODBC Driver Manager]
Data
source name not found and no default driver specified (0) (SQLDriverConnect)')
Being new to Azure Batch I didn't realise the virtual machines didn't come with ODBC drivers installed. I wrote a .bat file to install drivers on the node when the pool is allocated. Problem solved.
You have to install the ODBC driver in each compute nodes of the pool.
Put the below commands inside a shell script file startup_tasks.sh:
sudo apt-get -y update;
export DEBIAN_FRONTEND=noninteractive;
sudo apt-get install -y python3-pip;
apt-get install -y --no-install-recommends apt-utils apt-transport-https;
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - ;
curl https://packages.microsoft.com/config/debian/9/prod.list >
/etc/apt/sources.list.d/mssql-release.list ;
sudo apt-get -y update ;
ACCEPT_EULA=Y apt-get -y install msodbcsql17 ;
ACCEPT_EULA=Y apt-get -y install mssql-tools ;
echo 'export PATH=\"$PATH:/opt/mssql-tools/bin\"' >> ~/.bash_profile ;
echo 'export PATH=\"$PATH:/opt/mssql-tools/bin\"' >> ~/.bashrc ;
source ~/.bashrc && sudo apt-get install -y unixodbc unixodbc-dev ;
Give bin/bash -c "startup_tasks.sh" as a startup task in azure batch pool.This will install the ODBC driver 17 in each nodes.
And then in your connection string change the ODBC driver version to 17
Related
I have application which deployed via Docker, in my application i am accessing the Azure SQL , so. my connection string look like this
mssql+pyodbc://<user>#<domain>:<password>#<conn_string>.database.windows.net:1433/<db>?driver={'ODBC Driver 17 for SQL Server'}
My Docker File :
# Configure the MS SQL query drivers
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt update
RUN ACCEPT_EULA=Y apt-get install -y msodbcsql17
I am getting below error :
Can't open lib '{'ODBC Driver 17 for SQL Server'' : file not found (0)
Can anyone help me how to properly configure the driver?
Please Help...
Im getting below error while trying to connect to sql server with databricks runtime 10.4 LTS , while the connection was successfull with databricks runtime 7.4 LTS
Error:
OperationalError: ('08001', '[08001] [unixODBC][Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x2746 (10054) (SQLDriverConnect)')
I'm using below installation steps in an init script
dbutils.fs.put("/databricks/scripts/driversqlodbc.sh", """
#!/bin/bash
sleep 10
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update
ACCEPT_EULA=Y apt-get install msodbcsql17
apt-get -y install unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc """, True)
And my connection string is like this
odbc_driver= '{ODBC Driver 17 for SQL Server}'
conn = pyodbc.connect('DRIVER='+odbc_driver+';SERVER=tcp:'+server+';PORT=1433;DATABASE='+database+';UID='+db_user+';PWD='+ password,autocommit=True)
** EDIT **
Below is the output of cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Now my script looks like this
#!/bin/bash
sleep 10
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18
sudo ACCEPT_EULA=Y apt-get install -y mssql-tools18
echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> ~/.bashrc
source ~/.bashrc
sudo apt-get install -y unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc
And Connection String
odbc_driver= '{ODBC Driver 18 for SQL Server}'
conn = pyodbc.connect('DRIVER='+odbc_driver+';SERVER=tcp:'+server+',1433;DATABASE='+database+';UID='+db_user+';PWD='+ password,autocommit=True)
Maybe an issue with the pyodbc version. We had the same problem. The latest pyodbc update to pyodbc==4.0.34 (5 days ago) causes some issues, see https://github.com/mkleehammer/pyodbc/issues/1079 for example.
So we changed the following line in setup.py:
"pyodbc~=4.0.32",
to
"pyodbc==4.0.32",
Note the double == sign. It works with 4.0.32 for us.
Good Morning,
I am having the following issue with my Docker container and pyodbc / unixodbc-dev.
When running my Python API connecting to my Docker container I get the following error message--
(pyodbc.Error) ('01000', "[01000] [unixODBC][Driver
Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)"
Connecting to the same API using my local debug instance everything is working fine -- I can submit a string for searching in the backend database and I get results returned and sent back to the Postman UI.
I see that unixodbc-dev dev 2.3.6-0.1 amd64 installed in the Docker image and I noticed that unixODBC is at 2.3.11 - don't know if there might be any issue with that but that being said our Moonshot instances can't connect to http://deb.debian.org and to get our security group to open it up is next to impossible.
All this being said I'm wondering if I have something configured wrong in my Docker container that is causing my issues. I'm new to the Docker container world so this is definitely a learn as I go.
TIA,
Bill Youngman
I was able to get this figured out - thanks to m.b. for the solution I was looking for.
I was able to take the Debian suggestion from this Install ODBC driver in Alpine Linux Docker Container and modify it for my needs.
This is the code that I used to meet my requirements of downloading unixOdbc as well as downloading and installing the MS Sql ODBC driver.
FROM python:3.8.3
ARG ENV DEBIAN_FRONTEND noninteractive
# install Microsoft SQL Server requirements.
ENV ACCEPT_EULA=Y
RUN apt-get update -y && apt-get update \
&& apt-get install -y --no-install-recommends curl gcc g++ gnupg unixodbc-dev
# Add SQL Server ODBC Driver 17 for Ubuntu 18.04
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends --allow-unauthenticated msodbcsql17 mssql-tools \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
# clean the install.
RUN apt-get -y clean
Once this was accomplished and I built and deployed my container everything worked perfectly.
-Bill
I'm trying to connect a python container with SQL management studio in my host (Windows). I'm using port 1444 that is the port of my database and it give me this error.
Traceback (most recent call last):
File "/app/main.py", line 3, in <module>
connection = pyodbc.connect('Driver={FreeTDS};'
pyodbc.OperationalError: ('08S01', '[08S01] [FreeTDS][SQL Server]Unable to connect: Adaptive Server is unavailable or does not exist (20009) (SQLDriverConnect)')
The main.py code is:
import pyodbc
connection = pyodbc.connect('Driver={FreeTDS};'
'Server=DESKTOP-xxx\SQLEXPRESS,1444;'
'Database=DBxx;'
'UID=username;'
'PWD=password')
cursor = connection.cursor()
cursor.execute("SELECT [Id],[c1] FROM [DBxx].[dbo].[TABLExx]")
for row in cursor.fetchall():
print(row.Name)
Dockerfile:
FROM python
EXPOSE 1444
WORKDIR /app
ADD requirements.txt .
ADD . /app
ADD odbcinst.ini /etc/odbcinst.ini
RUN apt-get update
RUN apt-get install gcc -y
RUN apt-get install -y tdsodbc unixodbc-dev
RUN apt install unixodbc -y
RUN apt-get clean -y
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
RUN echo "[FreeTDS]\n\
Description = FreeTDS Driver\n\
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so\n\
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so" >> /etc/odbcinst.ini
# Run app.py when the container launches
CMD ["python", "main.py"]
And finally the requirements:
pyodbc
I have been using pyodbc on one of my Databricks clusters and have been installing it using this shell command running in the first cell of my notebook:
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update
ACCEPT_EULA=Y apt-get install msodbcsql17
apt-get -y install unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc
This works fine but I have to execute it each time I run the cluster and intend to use pyodbc. I have been doing this by including this piece of code as the first cell of each notebook that uses pyodbc. To fix this I tried to save this code as a .sh file, uploaded it to dbfs, and then added it as one of my cluster's init files. Upon running the code given below:
cnxn1 = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+jdbcHostname+';DATABASE='+jdbcDatabase+';UID='+username1+';PWD='+ password1)
I get the following error:
('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)")
What is it that I am doing wrong with my shell commands/init script that's causing this issue. Any help would be greatly appreciated. Thanks!
This is the recommended way of doing it.
Create the file like this :
dbutils.fs.put("dbfs:/databricks/scripts/pyodbc-install.sh","""
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update
ACCEPT_EULA=Y apt-get install msodbcsql17
apt-get -y install unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc""", True)
Then go to your cluster configuration page.
Click on Edit:
Go down and expand Advanced Options > Init Scripts
There you can add the path of the script :
Then you can click on Confirm.
Now, this script will be executed at the start of your cluster and will make pyodbc available on all notebooks attached to it.
Is it how you did it ?