Develop Reference

python unittests mocking mysql multiple times causing errors

I am writing unittests for a program, the majority of functions are all boilerplate code to do some mysql queries with no real return types, to test these I have written tests to check for the query in the cursor:
#mock.patch('mysql.connector.connect')
def test_query1(self, mock_conn):
test_query_data = 100
import app
a = app.query1(test_query_data)
mock_cursor = mock_conn.return_value.cursor.return_value
self.assertEqual(mock_cursor.execute.call_args[0], ('SELECT id FROM table WHERE data=%s limit 1;', (100,)))
this test on its own works fine but when I have others structured the exact same way the patching of the mysql connection breaks causing an exception in the assert statement
Traceback (most recent call last):
File "c:\users\sirwill\appdata\local\programs\python\python38\lib\site-packages\mock\mock.py", line 1346, in patched
return func(*newargs, **newkeywargs)
File "C:\Users\sirwill\python_project\tests.py", line 69, in test_insert_event
self.assertEqual(mock_cursor.execute.call_args[0], ('SELECT id FROM table WHERE data=%s limit 1;', (100,)))
TypeError: 'NoneType' object is not subscriptable
I have tried to delete the module and reimport with no change in the result

for anyone else having this issue the answer was to reload the library upon importing into the test using
importlib.reload(app)

mysql-connector-python InterfaceError: Failed getting warnings when executing a query with multiple statements with get_warnings=True

Using Python 3.7, I execute a query against a MySQL database, with multiple statements, with get_warnings enabled:
import mysql.connector
cnx = mysql.connector.connect(host='xxx',
user='xxx',
password='xxx',
database='xxx',
use_pure=False,
get_warnings=True)
# Test 1, works:
cur = cnx.cursor()
cur.execute('SELECT "a"+1')
for row in cur:
print(row)
print(cur.fetchwarnings())
cur.close()
# Test 2, InterfaceError:
cur = cnx.cursor()
for rs in cur.execute('SELECT "a"+1; SELECT 2', multi=True):
for row in rs:
print(row)
print(rs.fetchwarnings())
The first test executes a single statement, iterates over the cursor, fetches data, and finally prints warnings. Output as expected:
(1.0,)
[('Warning', 1292, "Truncated incorrect DOUBLE value: 'a'")]
The second test, (you can remove the first test altogether), will execute print(row) once, then an Exception happens. Output:
Traceback (most recent call last):
File "C:\Program Files\Python37\lib\site-packages\mysql\connector\connection_cext.py", line 472, in cmd_query
raw_as_string=raw_as_string)
_mysql_connector.MySQLInterfaceError: Commands out of sync; you can't run this command now
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python37\lib\site-packages\mysql\connector\cursor_cext.py", line 138, in _fetch_warnings
_ = self._cnx.cmd_query("SHOW WARNINGS")
File "C:\Program Files\Python37\lib\site-packages\mysql\connector\connection_cext.py", line 475, in cmd_query
sqlstate=exc.sqlstate)
mysql.connector.errors.DatabaseError: 2014 (HY000): Commands out of sync; you can't run this command now
During handling of the above exception, another exception occurred:
....etc....
Did anyone encounter the same problem? How did you solve it? What am I doing wrong? Could this be a bug in the connector?
Other things I've tried:
If you set get_warnings to False, no error happens and
fetchwarnings() returns None
If you remove the problem from the SQL code, no error happens and fetchwarnings() returns None
use_pure can be True or False, the only difference is a slightly different traceback
Using fetchall() instead of for row in rs gives the same result
Many other variations give the same error.
System:
Connector version is mysql-connector-python-8.0.17 but 8.0.16 has the same issue.
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
MySQL 5.7

The "Commands out of sync" is because MySQL client interface calls are performed in a wrong order. This is not a bug in the connector. This is expected behavior.
Executing that first SELECT returns a MySQL resultset.
Before the client issues another statement that returns a MySQL resultset, we have to do something with the resultset that is already returned. That is, there needs to be calls to either mysql_use_result and mysql_free_result, or a call to mysql_store_result. Once the client does that, then the client can execute another SQL statement that returns a result.
(Note that the execution of the MySQL SHOW WARNINGS statement returns a MySQL resultset.)
Again, this is expected behavior, as documented here:
https://dev.mysql.com/doc/refman/8.0/en/commands-out-of-sync.html
The references to mysql_free_result, mysql_store_result and mysql_use_result aren't specific to a Python interface; these reference the underlying library routines in the MySQL client code. e.g. https://dev.mysql.com/doc/refman/8.0/en/mysql-use-result.html
FOLLOWUP
I suspect the author of the MySQL Python connector didn't anticipate this use case, or if it was anticipated, the observed behavior was judged to be correct.
As far as avoiding the problem, I would avoid the use of the multii=True and do a separate execute for each SQL statement. Following the same pattern as in Test 1, we could add an outer loop to loop through the SQL statements
# Test 1.2
sqls = ['SELECT "a"+1', 'SELECT 2', ]
for sql in sqls:
cur = cnx.cursor()
cur.execute(sql)
for row in cur:
print(row)
print(cur.fetchwarnings())
cur.close()
Another option would be to avoid the call to the fetchwarnings. That is what is causing the SHOW WARNINGS statement to be executed (only after it first verifies that the count of warnings is greater than zero.) We can issue a SHOW WARNINGS statement separately, and loop through the results from that like it were the return from a SELECT.
# Test 1.3
cur = cnx.cursor()
for rs in cur.execute('SELECT "a"+1; SHOW WARNINGS; SELECT 2; SHOW WARNINGS', multi=True):
for row in rs:
print(row)
cur.close()

I get NotImplementedError when trying to do a prepared statement with mysql python connector

I want to use prepared statements to insert data into a MySQL DB (version 5.7) using python, but I keep getting a NotImplementedError.
I'm following the documentation here: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorprepared.html
Using Python 2.7 and version 8.0.11 of mysql-connector-python library:
pip show mysql-connector-python
---
Metadata-Version: 2.1
Name: mysql-connector-python
Version: 8.0.11
Summary: MySQL driver written in Python
Home-page: http://dev.mysql.com/doc/connector-python/en/index.html
This is a cleaned version (no specific hostname, username, password, columns, or tables) of the python script I'm running:
import mysql.connector
from mysql.connector.cursor import MySQLCursorPrepared
connection = mysql.connector.connect(user=username, password=password,
host='sql_server_host',
database='dbname')
print('Connected! getting cursor')
cursor = connection.cursor(cursor_class=MySQLCursorPrepared)
select = "SELECT * FROM table_name WHERE column1 = ?"
param = 'param1'
print('Executing statement')
cursor.execute(select, (param,))
rows = cursor.fetchall()
for row in rows:
value = row.column1
print('value: '+ value)
I get this error when I run this:
Traceback (most recent call last):
File "test.py", line 18, in <module>
cursor.execute(select, (param,))
File "/home/user/.local/lib/python2.7/site-packages/mysql/connector/cursor.py", line 1186, in execute
self._prepared = self._connection.cmd_stmt_prepare(operation)
File "/home/user/.local/lib/python2.7/site-packages/mysql/connector/abstracts.py", line 969, in cmd_stmt_prepare
raise NotImplementedError
NotImplementedError

CEXT will be enabled by default if you have it, and prepared statements are not supported in CEXT at the time of writing.
You can disable the use of CEXT when you connect by adding the keyword argument use_pure=True as follows:
connection = mysql.connector.connect(user=username, password=password,
host='sql_server_host',
database='dbname',
use_pure=True)
Support for prepared statements in CEXT will be included in the upcoming mysql-connector-python 8.0.17 release (according to the MySQL bug report). So once that is available, upgrade to at least 8.0.17 to solve this without needing use_pure=True.

python3 SqlAlchemy error TypeError: Type str doesn't support the buffer API

I am writing a django app that calls a script using SQLAlchemy. The following query runs successfully with Python v2.x:
engine = create_engine('mysql+oursql://ag:529382Ag#localhost/gp?charset=utf8&use_unicode=0',pool_recycle=3600)
query = "select a.gid as entrez, a.source_id as "+id_dict[ids[0]]+", b.source_id as "+id_dict[ids[1]]+" from (select * from gid2source_id where gid2source_id.id_type_id = "+str(ids[0])+") a, (select * from gid2source_id where gid2source_id.id_type_id= "+str(ids[1])+") b where b.gid = a.gid;"
result1 = engine.execute(query)
When run with Python v3.x, the following error is received:
line 105, in get_results result1 = engine.execute(query.decode('utf-8'))
...
line 2453, in initialize self._detect_ansiquotes(connection)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/sqlalchemy/dialects/mysql/base.py",
line 2730, in _detect_ansiquotes
self._server_ansiquotes = 'ANSI_QUOTES' in mode TypeError: Type str doesn't support the buffer API

I guess oursql doesn't interface properly in python3. I switched to mysqlconnector and it works fine.

Error while importing file into DB2 from python script

Getting the below error while trying to import a ^ delimited file into a DB2 database using python 2.4.3.
Error:
Traceback (most recent call last):
File "C:\Python25\Usefulscripts\order.py", line 89, in <module>
load_order_stack()
File "C:\Python25\Usefulscripts\order.py", line 75, in load_order_stack
conn2.execute(importTmp)
ProgrammingError: ('42601', '[42601] [IBM][CLI Driver][DB2/LINUXX8664] SQL0104N An unexpected token "orders_extract"
was found following "import from ".
Code:
import pyodbc
def load_order_stack():
try:
conn2 = pyodbc.connect('DSN=db2Database;UID=ueserid;PWD=password')
importTmp = ("import from orders_extract of del modified by coldel0x5E"
"insert_update into test.ORDERS_Table (ORDER_ID,item,price);")
conn2.execute(importTmp)
conn2.commit()

IMPORT is not an SQL statement. It is a DB2 Command Line Processor (CLP) command and as such can only be run by the said CLP.
There is an SQL interface to some CLP commands via calls to the ADMIN_CMD() stored procedure, please check the manual: IMPORT using ADMIN_CMD

You also have the option of reading the file, line by line, and inserting into your database. This will definitely be slower than any native import operation. Assuming your delimited file structure is, and the file is named input.txt:
ORDER_ID^item^price
1^'bat'^50.00
2^'ball'^25.00
Code:
import csv
import pyodbc
connection = pyodbc.connect('DSN=db2Database;UID=ueserid;PWD=password')
cursor = connection.cursor()
with open('input.txt', 'rb') as f:
rows = csv.reader(f, delimiter='^')
# get column names from header in first line
columns = ','.join(next(rows))
for row in rows:
# build sql with placeholders for insert
placeholders = ','.join('?' * len(row))
sql = 'insert into ({}) values ({});'.format(columns, placeholders)
# execute parameterized database insert
cursor.execute(sql, row)
cursor.commit()
Play around with commit() placement, you probably want to commit in batches to improve performance.