`I am trying to access DB with google colab using the below code.
`%%sh
apt-get update
apt-get install -y curl apt-transport-https
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
ACCEPT_EULA=Y apt-get install -y msodbcsql17 unixodbc-dev
!sudo apt-get install unixodbc-dev
!pip install pyodbc
!yum install unixODBC-devel
import pyodbc
conn = pyodbc.connect(DRIVER = '{ODBC Driver 17 for SQL Server}',
SERVER = 'tcp:zestlabs.database.windows.net,1433',
DATABASE = *******,
UID = ********,
PWD = **************)`
I have replaced my database name and password with "****" and also commented the code here`
enter image description here
This is the error I am getting when I run the 2nd chunk of code. Can someone help me resolve this issue?
Related
Please Help...
Im getting below error while trying to connect to sql server with databricks runtime 10.4 LTS , while the connection was successfull with databricks runtime 7.4 LTS
Error:
OperationalError: ('08001', '[08001] [unixODBC][Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x2746 (10054) (SQLDriverConnect)')
I'm using below installation steps in an init script
dbutils.fs.put("/databricks/scripts/driversqlodbc.sh", """
#!/bin/bash
sleep 10
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update
ACCEPT_EULA=Y apt-get install msodbcsql17
apt-get -y install unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc """, True)
And my connection string is like this
odbc_driver= '{ODBC Driver 17 for SQL Server}'
conn = pyodbc.connect('DRIVER='+odbc_driver+';SERVER=tcp:'+server+';PORT=1433;DATABASE='+database+';UID='+db_user+';PWD='+ password,autocommit=True)
** EDIT **
Below is the output of cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Now my script looks like this
#!/bin/bash
sleep 10
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18
sudo ACCEPT_EULA=Y apt-get install -y mssql-tools18
echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> ~/.bashrc
source ~/.bashrc
sudo apt-get install -y unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc
And Connection String
odbc_driver= '{ODBC Driver 18 for SQL Server}'
conn = pyodbc.connect('DRIVER='+odbc_driver+';SERVER=tcp:'+server+',1433;DATABASE='+database+';UID='+db_user+';PWD='+ password,autocommit=True)
Maybe an issue with the pyodbc version. We had the same problem. The latest pyodbc update to pyodbc==4.0.34 (5 days ago) causes some issues, see https://github.com/mkleehammer/pyodbc/issues/1079 for example.
So we changed the following line in setup.py:
"pyodbc~=4.0.32",
to
"pyodbc==4.0.32",
Note the double == sign. It works with 4.0.32 for us.
Good Morning,
I am having the following issue with my Docker container and pyodbc / unixodbc-dev.
When running my Python API connecting to my Docker container I get the following error message--
(pyodbc.Error) ('01000', "[01000] [unixODBC][Driver
Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)"
Connecting to the same API using my local debug instance everything is working fine -- I can submit a string for searching in the backend database and I get results returned and sent back to the Postman UI.
I see that unixodbc-dev dev 2.3.6-0.1 amd64 installed in the Docker image and I noticed that unixODBC is at 2.3.11 - don't know if there might be any issue with that but that being said our Moonshot instances can't connect to http://deb.debian.org and to get our security group to open it up is next to impossible.
All this being said I'm wondering if I have something configured wrong in my Docker container that is causing my issues. I'm new to the Docker container world so this is definitely a learn as I go.
TIA,
Bill Youngman
I was able to get this figured out - thanks to m.b. for the solution I was looking for.
I was able to take the Debian suggestion from this Install ODBC driver in Alpine Linux Docker Container and modify it for my needs.
This is the code that I used to meet my requirements of downloading unixOdbc as well as downloading and installing the MS Sql ODBC driver.
FROM python:3.8.3
ARG ENV DEBIAN_FRONTEND noninteractive
# install Microsoft SQL Server requirements.
ENV ACCEPT_EULA=Y
RUN apt-get update -y && apt-get update \
&& apt-get install -y --no-install-recommends curl gcc g++ gnupg unixodbc-dev
# Add SQL Server ODBC Driver 17 for Ubuntu 18.04
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends --allow-unauthenticated msodbcsql17 mssql-tools \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
# clean the install.
RUN apt-get -y clean
Once this was accomplished and I built and deployed my container everything worked perfectly.
-Bill
I have been using pyodbc on one of my Databricks clusters and have been installing it using this shell command running in the first cell of my notebook:
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update
ACCEPT_EULA=Y apt-get install msodbcsql17
apt-get -y install unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc
This works fine but I have to execute it each time I run the cluster and intend to use pyodbc. I have been doing this by including this piece of code as the first cell of each notebook that uses pyodbc. To fix this I tried to save this code as a .sh file, uploaded it to dbfs, and then added it as one of my cluster's init files. Upon running the code given below:
cnxn1 = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+jdbcHostname+';DATABASE='+jdbcDatabase+';UID='+username1+';PWD='+ password1)
I get the following error:
('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)")
What is it that I am doing wrong with my shell commands/init script that's causing this issue. Any help would be greatly appreciated. Thanks!
This is the recommended way of doing it.
Create the file like this :
dbutils.fs.put("dbfs:/databricks/scripts/pyodbc-install.sh","""
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update
ACCEPT_EULA=Y apt-get install msodbcsql17
apt-get -y install unixodbc-dev
sudo apt-get install python3-pip -y
pip3 install --upgrade pyodbc""", True)
Then go to your cluster configuration page.
Click on Edit:
Go down and expand Advanced Options > Init Scripts
There you can add the path of the script :
Then you can click on Confirm.
Now, this script will be executed at the start of your cluster and will make pyodbc available on all notebooks attached to it.
Is it how you did it ?
I created an airflow image inside Kubernetes and am having a hard time connecting to my SQL Server database using Pyodbc to make my Python script work.
Here's the part of DockerFile I set up:
FROM python:3.7-slim
# apt-get and system utilities
RUN apt-get update && apt-get install -yqq \
curl gnupg gnupg2 gnupg1 \
&& rm -rf /var/lib/apt/lists/*
# adding custom MS repository
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/19.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN set -ex \
&& apt-get update -yqq \
&& apt-get upgrade -yqq \
&& apt-get install -yqq --no-install-recommends build-essential \
&& export ACCEPT_EULA=Y \
&& apt-get install -yqq msodbcsql17 freetds-dev freetds-bin tdsodbc unixodbc-dev \
&& apt-get update \
&& export ACCEPT_EULA=Y \
&& apt-get install -yqq mssql-tools \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& /bin/bash -c "source ~/.bashrc" \
&& apt-get install -yqq default-libmysqlclient-dev \
&& pip install -U pip setuptools wheel \
&& pip install -r ${AIRFLOW_HOME}/requirements.txt \
&& pip install gcloud \
&& pip install apache-airflow[crypto,postgres,jdbc,gcp_api,mssql,mysql,kubernetes,gcp]
It creates the image without problems. Now I send the part of the python script that tries to connect to Sql Server:
import pyodbc
database='dbd'
uid = 'user'
pwd = 'pass'
server = '192.168.0.1'
driver = "Driver={ODBC Driver 17 for SQL Server};server=" + server + ";database=" + database + ";uid=" + uid + ";pwd=" + pwd
# MySQL Credentials and settings
db = pyodbc.connect(driver)
cursor = db.cursor()
I checked the odbcinst.ini file inside my pod and it looks like this:
[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
Driver=/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.4.so.1.1
UsageCount=1
And finally the error message I have is as follows:
ERROR - ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]SSL Provider: [error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol] (-1) (SQLDriverConnect)')
What else can I do to test connectivity with SQL Server?
I found a much simpler lib to work with Airflow:
import pymssql
driver = pymssql.connect(server='192.168.0.1',user='user',password='pass',database='dbd',port='1433')
cursor = db.cursor()
I am trying to run a Python job using a VM on Azure Batch. It's a simple script to add a line to my Azure SQL Database. I downloaded the ODBC connection string straight from my Azure portal yet I get this error. The strange thing is I can run the script perfectly fine on my own machine. I've configured the VM to install the version of Python that I need and then execute my script - I'm at a complete loss. Any ideas?
cnxn = pyodbc.connect('Driver={ODBC Driver 13 for SQL Server};Server=tcp:svr-something.database.windows.net,fakeport232;Database=db-something-prod;Uid=something#svr-something;Pwd{fake_passwd};Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;')
Traceback (most recent call last):
File "D:\batch\tasks\apppackages\batch_python_test1.02018-11-12-14-
30\batch_python_test\python_test.py", line 12, in
r'Driver={ODBC Driver 13 for SQL Server};Server=tcp:svr-
mydatabase.database.windows.net,'
pyodbc.InterfaceError: ('IM002', '[IM002] [Microsoft][ODBC Driver Manager]
Data
source name not found and no default driver specified (0) (SQLDriverConnect)')
Being new to Azure Batch I didn't realise the virtual machines didn't come with ODBC drivers installed. I wrote a .bat file to install drivers on the node when the pool is allocated. Problem solved.
You have to install the ODBC driver in each compute nodes of the pool.
Put the below commands inside a shell script file startup_tasks.sh:
sudo apt-get -y update;
export DEBIAN_FRONTEND=noninteractive;
sudo apt-get install -y python3-pip;
apt-get install -y --no-install-recommends apt-utils apt-transport-https;
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - ;
curl https://packages.microsoft.com/config/debian/9/prod.list >
/etc/apt/sources.list.d/mssql-release.list ;
sudo apt-get -y update ;
ACCEPT_EULA=Y apt-get -y install msodbcsql17 ;
ACCEPT_EULA=Y apt-get -y install mssql-tools ;
echo 'export PATH=\"$PATH:/opt/mssql-tools/bin\"' >> ~/.bash_profile ;
echo 'export PATH=\"$PATH:/opt/mssql-tools/bin\"' >> ~/.bashrc ;
source ~/.bashrc && sudo apt-get install -y unixodbc unixodbc-dev ;
Give bin/bash -c "startup_tasks.sh" as a startup task in azure batch pool.This will install the ODBC driver 17 in each nodes.
And then in your connection string change the ODBC driver version to 17