I want to use the Python mariadb connector in a python container. Yet, until now, I've met only troubles and now I segfault when trying to pass arguments to my SQL queries.
here are my dockerfile and a script that highlight the problem.
FROM python:3.11-buster
RUN apt-get install gcc wget
# https://stackoverflow.com/questions/74429209/mariadb-in-docker-mariadb-connector-python-requires-mariadb-connector-c-3-2
RUN wget https://dlm.mariadb.com/2678579/Connectors/c/connector-c-3.3.3/mariadb-connector-c-3.3.3-debian-buster-amd64.tar.gz -O - | tar -zxf - --strip-components=1 -C /usr
WORKDIR /appli
RUN pip3 install pynetdicom>=2.0 mariadb==1.1.4
COPY . .
CMD ["python3", "AEServer.py"]
import mariadb
conn = mariadb.connect(user='myself', password='mypass', host=ip, database='db') # Successfully created connection
cursor = conn.cursor()
data = ('1.2.392', 'NAME^SURNAME^', 'BDAY', 'DX', 1) # Placeholder
insert_stmt = ("INSERT Examens VALUES ('%s', '%s', '%s', '%s', %i, 'COMING')" % data)
segv_stmt = ("INSERT Examens VALUES (?, ?, ?, ?, ?, 'COMING')", data)
segv = "INSERT Examens VALUES (?, ?, ?, ?, ?, 'COMING')"
cursor.execute(insert_stmt) # GOOD
cursor.execute(segv, data) # Segmentation fault
First, I tried adding mariadb in my pip install in the dockerfile. It failed, because I didn't have the mariadb/c connector.
Then, I tried adding it by installing libmariadb3 and libmariadb-dev, according to this page of the documentation : https://mariadb.com/docs/ent/connect/programming-languages/c/install/ but I failed again, because the package fetched by the manager weren't up to date, so I had a version conflit when pip installing.
Then, I tried the following reference : Mariadb in Docker: MariaDB Connector/Python requires MariaDB Connector/C >= 3.2.4, found version 3.1.16, to wget the connector and compiling it myself in the Dockerfile. I succeeded to create my container and I could create mariadb.coonnection and cursor so I thought it was over.
Finally when trying to use insert statement with placeholder (either '%s' or '?'), I am met with a segfault. I tried various associations of sources images (3.11:bullseye, 3.11:buster), connectors (version 3.3.3, 3.2.7) and mariadb (version 1.1.0, 1.1.2, 1.1.4), but all of them segfault.
What I want is a container with python3, my dependancy pynetdicom, and the ability to interract safely with a mariadb Database. A Dockerfile fulfilling the prevous prerequisites should be enough for me to move forward, but I also wish to know why things did happen this way.
Edit : Posted some credentials used in dev, removed them.
Even if you installed a newer MariaDB Connector/C version, the preinstalled 3.1.13 version from 3.11-buster is still installed.
After installation of MariaDB Connector/C 3.3.3 you have 2 versions installed:
/usr/lib/x86_64-linux-gnu/libmariadb.so.3
/usr/lib/mariadb/libmariadb.so.3
Now when running your python script, it crashes in /usr/lib/x86_64-linux-gnu/libmariadb.so.3.
To force Python to use the newer MariaDB Connector/C library just add
ENV LD_PRELOAD=/usr/lib/mariadb/libmariadb.so or ENV LD_LIBRARY_PATH=/usr/lib/mariadb
to your Dockerfile.
Related
I have a table in a sql server database, where time is stored using the datetimeoffset datatype.
All worked fine on windows 10, but when I moved my code to a docker container (ubuntu 1604, python 3.7.6 and pyobdc 4.0.30). OBDC drivers added by Dockerfile with
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql17 unixodbc-dev
I then started getting the following selecting data a datetimetoffset field, eg SELECT date FROM ...:
ProgrammingError: ('ODBC SQL type -155 is not yet supported. column-index=0 type=-155', 'HY106')
My current workaround is the convert the datetime field to a date in sql with SELECT convert(datetime, date) FROM ...
I expect it is a limitation of the OBDC drivers on Linux.
I would prefer not updating the sql, so I wonder if there are alternatives to the OBDC drivers I use or other options
I am trying to;
Run the python code triggered by Cosmos DB when cosmos DB receives the data..
The python code in Azure Functions has code to ingest data from Azure MySQL.
What I have done are;
. Wrote python in Azure Functions and run it with triggered by Cosmos
DB. This was successful.
. Installed mysql.connector referred to
https://prmadi.com/running-python-code-on-azure-functions-app/ and
run the code to connect to Azure MySQL, but It does not work.
Do you know how to install mysql module for Python to Azure Functions and connect to the database?
Thanks!
According to your description ,I think your issue is about how to install the Python third-party module in the Azure function app.
Please refer to the steps as below :
Step 1 :
login kudu : https://Your_APP_NAME.scm.azurewebsites.net/DebugConsole.
Run Below command in d:/home/site/wwwroot/<your function name> folder.(will take some time)
python -m virtualenv myvenv
Step 2 :
Load the env via the below command in env/Scripts folder.
activate.bat
Step 3 :
Your shell should be now prefixed by (env).
Update pip
python -m pip install -U pip
Install what you need
python -m pip install MySQLdb
Step 4 :
In your code, update the sys.path to add this venv:
import sys, os.path
sys.path.append(os.path.abspath(os.path.join(os.path.dirname( __file__ ), 'env/Lib/site-packages')))
Then connect to mysql db via the snippet of code below
#!/usr/bin/python
import MySQLdb
# Connect
db = MySQLdb.connect(host="localhost",
user="appuser",
passwd="",
db="onco")
cursor = db.cursor()
# Execute SQL select statement
cursor.execute("SELECT * FROM location")
# Commit your changes if writing
# In this case, we are only reading data
# db.commit()
# Get the number of rows in the resultset
numrows = cursor.rowcount
# Get and display one row at a time
for x in range(0, numrows):
row = cursor.fetchone()
print row[0], "-->", row[1]
# Close the connection
db.close()
Hope it helps you.
I'm sure this issue has been raised an uncountable number of times before but perhaps, someone could still help me.
I am using pymssql v2.1.3 with Python 2.7.12 and the code that I used several times until yesterday to write data to my Azure SQL DB has somehow decided not to work anymore - for no apparent reason.
The firewall is set, my IP is in the whitelist, I can connect to the database using SQL Server Management Studio and query the data but I still keep getting this error when attempting to connect using pymssql.
The app is a Flask web-app and following is how I connect to the DB:
conn = pymssql.connect(server='myserver.database.windows.net', user='myusername#mydatabase', password='mypassword', database='mydatabase')
This is likely due to the pymssql version. Did you upgrade pymssql? If yes, try reverting back to 2.1.1
sudo pip install pymssql==2.1.1
Not really a solution to the issue I raised, but using pypyodbc instead of pymssql works.
conn = pypyodbc.connect(driver='{SQL Server}',server='tcp:myserver.database.windows.net,1433',database='mydatabase', uid='myusername', pwd='mypassword')
freetds-dev might be missing on linux:
apt-get update && apt-get install freetds-dev
unbelievable the bug is still present...
ENV: WSL2 on Windows 10
Fix -> switch to pyodbc:.
sudo apt-get install unixodbc-dev && pip3 install pyodbc
follow instruction for ubuntu
https://learn.microsoft.com/de-de/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15
import pyodbc
server = 'tcp:myserver.database.windows.net'
database = 'mydb'
username = 'myusername'
password = 'mypassword'
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
I uninstall and install fresh pymssql and it works for me now.
I wrote a python script that communicate with MySQL server on OSX 10.10, it runs fine Mac until I put it on an VPS running Ubuntu 14.04
The problem mainly lies on the pyMySQL module. :I can't even run their example script on the git page here
for running the following code:
import pymysql.cursors
# Connect to the database
connection = pymysql.connect(host='localhost',
user='user',
password='passwd',
db='db',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
try:
with connection.cursor() as cursor:
# Create a new record
sql = "INSERT INTO `users` (`email`, `password`) VALUES (%s, %s)"
cursor.execute(sql, ('webmaster#python.org', 'very-secret'))
# connection is not autocommit by default. So you must commit to save
# your changes.
connection.commit()
with connection.cursor() as cursor:
# Read a single record
sql = "SELECT `id`, `password` FROM `users` WHERE `email`=%s"
cursor.execute(sql, ('webmaster#python.org',))
result = cursor.fetchone()
print(result)
finally:
connection.close()
The result is
Traceback (most recent call last):
File "1.py", line 9, in <module>
cursorclass=pymysql.cursors.DictCursor)
File "build/bdist.linux-x86_64/egg/pymysql/__init__.py", line 93, in Connect
TypeError: __init__() got an unexpected keyword argument 'password'
Environment:
Ubuntu 14.04 x86_64
Python 2.7.10
PyMySQL 0.6.7
Mac is running the above script fine but not the ubuntu.
Thanks in advance.
I didn't do much checking on the error message. Instead I uninstall all the packages on the unbuntu server and follow the package list on the Mac to install them again one by one to the same version. Amazingly, the problem is solved. Unfortunately I can't tell which module is causing the problem
You might want to give it a try if you don't want to spend much time on updating modules one by one.
It seems that this is a ubuntu problem, which is solved if you install the latest pypi version.
I removed the package from ubuntu in dist-packages and installed it in site-packages:
sudo apt-get purge python3-pymysql
sudo pip3 install --upgrade pymysql
You don't (really) want to do this, especially on a server. Alternatively you use a virtualenv:
virtualenv -p python3.4 venv3
source venv3/bin/activate
pip3 install --update pip
pip3 install --update pymysql
You need to learn a bit more about virtualenv's to use them, but the idea is that you have a complete and separate python3.4 environment installed in venv3 which you activate with source venv3/bin/activate. This way it is harder to break your ubuntu (which heavily depends on a working python). Moreover, you can install different versions of packages in different virtualenv's. Also, when debugging, you can add print-statements to python-code in the site-packages folder of the virtualenv, without risking breaking your system severely. This is great, as many good python libraries have poor error reporting.
Faced this error with Python:3.6, PyMySQL3:0.5.
solution: change 'password' parameter to 'passwd'
Below is the parameters list with default values
host="localhost", user=None, passwd="", db=None, port=3306, unix_socket=None, charset='', sql_mode=None, read_default_file=None, conv=decoders, use_unicode=None, client_flag=0, cursorclass=Cursor, init_command=None, connect_timeout=None, ssl=None, read_default_group=None, compress=None, named_pipe=None
Currently, I'm running a simple python script to connect to a database:
import pyodbc
cnxn = pyodbc.connect('DRIVER={Teradata};DBCNAME=(MYDB);UID=(MYUSER); PWD=(MYPASS);QUIETMODE=YES')
With the server and credentials substituted in obviously. However, when running this script, I get the following error:
pyodbc.Error: ('200', '[200] [unixODBC][eaaa[DCTrdt rvr o nuhifraint o n (0) (SQLDriverConnectW)')
The only help I've been able to find is here installed the Teradata ODBC drivers, but I just don't understand why I can't connect. Anyone have any ideas on this?
You have to use like following to connect:
TDCONN = pyodbc.connect('DSN=yourDSNname;',ansi=True, autocommit=True)
You can replace DSN=yourDSNname; with what you have already.
I hit this "non english" error message problem. I think it was due to the wrong versions of libodbc.so and libodbcinst.so being used. Changing the links in /usr/lib/... to point to the teradata installed versions worked for me. Commands using the default install directories in Ubuntu 12.04, 64bit were:
cd /usr/lib/x86_64-linux-gnu
ls -lha | grep odbc (To should see the files below which are to be replaced).
sudo mv libodbc.so.1.0.0 Xlibodbc.so.1.0.0
sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbc.so libodbc.so.1.0.0
sudo mv libodbcinst.so.1.0.0 Xlibodbcinst.so.1.0.0
sudo ln -s /opt/teradata/client/14.10/odbc_64/lib/libodbcinst.so libodbcinst.so.1.0.0
I had also previously installed (through apt-get) odbcinst, so I redirected both files. But possibly only the first is needed.