The try/except is not capturing the specified pymssql exception in the following snippet:
import pandas as pd
from fps.databaseconnections import DatabaseConnections
from pymssql import ProgrammingError
DatabaseConnections_instance = DatabaseConnections()
db_cnxn = DatabaseConnections_instance.get_connection("gtaemndprod")
print "\ndb_cnxn is this class: {}\n".format(db_cnxn.__class__)
try:
pd.read_sql("SELECT blah",db_cnxn)
except ProgrammingError:
print "A"
Traceback:
db_cnxn is this class: <class 'sqlalchemy.engine.base.Engine'>
Traceback (most recent call last):
File "<ipython-input-20-9e8944849cfd>", line 11, in <module>
pd.read_sql("SELECT blah",db_cnxn)
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\pandas\io\sql.py", line 515, in read_sql
chunksize=chunksize)
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\pandas\io\sql.py", line 1190, in read_query
result = self.execute(*args)
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\pandas\io\sql.py", line 1081, in execute
return self.connectable.execute(*args, **kwargs)
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\sqlalchemy\engine\base.py", line 1991, in execute
return connection.execute(statement, *multiparams, **params)
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\sqlalchemy\engine\base.py", line 906, in execute
return self._execute_text(object, multiparams, params)
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\sqlalchemy\engine\base.py", line 1054, in _execute_text
statement, parameters
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\sqlalchemy\engine\base.py", line 1146, in _execute_context
context)
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\sqlalchemy\engine\base.py", line 1341, in _handle_dbapi_exception
exc_info
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\sqlalchemy\util\compat.py", line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
context)
File "C:\Users\212476924\AppData\Local\Continuum\Anaconda2\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
cursor.execute(statement, parameters)
File "pymssql.pyx", line 464, in pymssql.Cursor.execute (pymssql.c:7491)
ProgrammingError: (pymssql.ProgrammingError) (207, "Invalid column name 'blah'.DB-Lib error message 20018, severity 16:\nGeneral SQL Server error: Check messages from the SQL Server\n") [SQL: 'SELECT blah']
The DatabaseConnections class is just an easy way to access some commonly used databases that I need to work with. It returns the database connection engine, db_cnxn. As you can see in the traceback, this is a sqlalchemy.engine.base.Engine object.
Importing the entire pymssql package as shown below produces the same traceback:
import pandas as pd
from fps.databaseconnections import DatabaseConnections
import pymssql
DatabaseConnections_instance = DatabaseConnections()
db_cnxn = DatabaseConnections_instance.get_connection("gtaemndprod")
print "\ndb_cnxn is this class: {}\n".format(db_cnxn.__class__)
try:
pd.read_sql("SELECT blah",db_cnxn)
except pymssql.ProgrammingError:
print "A"
Please help! Thank you.
Since you are using sqlalchemy, this is the one you are interested in, not the pymssql one:
from sqlalchemy.exc import ProgrammingError
Related
I have tried multiple ways to connect to MSSQL with sql alchemy but always get the error below:
Traceback (most recent call last):
File "<ipython-input-100-71745b407575>", line 1, in <module>
conn = engine.connect()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 2473, in connect
return self._connection_cls(self, **kwargs)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 878, in __init__
self.__connection = connection or engine.raw_connection()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 2559, in raw_connection
return self.pool.unique_connection()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 184, in unique_connection
return _ConnectionFairy(self).checkout()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 401, in __init__
rec = self._connection_record = pool._do_get()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 746, in _do_get
con = self._create_connection()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 189, in _create_connection
return _ConnectionRecord(self)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 287, in __init__
exec_once(self.connection, self)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\event.py", line 377, in exec_once
self(*args, **kw)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\event.py", line 386, in __call__
fn(*args, **kw)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\strategies.py", line 168, in first_connect
dialect.initialize(c)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\connectors\pyodbc.py", line 135, in initialize
super(PyODBCConnector, self).initialize(connection)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\dialects\mssql\base.py", line 1164, in initialize
super(MSDialect, self).initialize(connection)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\default.py", line 177, in initialize
self._get_default_schema_name(connection)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\dialects\mssql\base.py", line 1180, in _get_default_schema_name
user_name = connection.scalar("SELECT user_name() as user_name;")
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 1383, in scalar
return self.execute(object, *multiparams, **params).scalar()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 1449, in execute
params)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 1628, in _execute_text
statement, parameters
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 1691, in _execute_context
context)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\default.py", line 333, in do_execute
cursor.execute(statement, parameters)
TypeError: The first argument to execute must be a string or unicode query.
Here is my code
import sqlalchemy as sa
import pyodbc
engine =sa.create_engine('mssql+pyodbc://uid:pwd#AWS1DNA01/xxx_Database,driver = SQL+Server')
conn = engine.connect()
In addition, I have tried to pass through the parameters as suggested by the documentation and got the same error.
conn_string = urllib.parse.quote_plus("Driver={SQL Server}; Server=AWS1DNA01; Database=xxx_Database; Integrated Security=False;UID=uid;PWD=pwd")
engine = sa.create_engine('mssql+pyodbc:///?odbc_connect={}'.format(conn_string))
conn = engine.connect()
What does this error mean and how do i fix it?
Also, I have tried to use pyodbc to connect and it works:
conn_string = "Driver={SQL Server}; Server=AWS1DNA01; Database=xxx_Database; Integrated Security=False;UID=uid;PWD=pwd"
connection = pyodbc.connect(conn_string)
Can someone please help with this?
I have finally fixed the issue, although I still don't know the cause of the original issue. Instead of using pyodbc, I used pymssql to create the engine and it works.
engine = sa.create_engine('mssql+pymssql://uid:pdw#server/db?')
I have a database engine as the below:
from sqlalchemy import create_engine
import pydoc
# connect db
engine = create_engine('mssql+pyodbc://xxxx\MARTRNO_EXPRESS/toolDB?driver=SQL+Server+Native+Client+11.0')
connection = engine.connect()
I tried to use something like the below code as to create a database using this connection as the below code:
from database import connec
import pandas as pd
def delete_all_tables_from_db():
delete_all_tables_query = "CREATE DATABASE MyNewDatabase"
delete_all_tables_df = pd.read_sql(delete_all_tables_query, connec.engine)
connec.engine.execute(delete_all_tables_df)
delete_all_tables_from_db()
but I find this error:
Traceback (most recent call last):
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\engine\base.py", line 1245, in _execute_context
self.dialect.do_execute(
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\engine\default.py", line 588, in do_execute
cursor.execute(statement, parameters)
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]CREATE DATABASE statement not allowed within multi-statement transaction. (226) (SQLExecDirectW)')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/Users/haroo501/PycharmProjects/ToolUpdated/database/delete_all_tables_from_db.py", line 10, in <module>
delete_all_tables_from_db()
File "C:/Users/haroo501/PycharmProjects/ToolUpdated/database/delete_all_tables_from_db.py", line 7, in delete_all_tables_from_db
delete_all_tables_df = pd.read_sql(delete_all_tables_query, connec.engine)
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\sql.py", line 432, in read_sql
return pandas_sql.read_query(
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\sql.py", line 1218, in read_query
result = self.execute(*args)
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\sql.py", line 1087, in execute
return self.connectable.execute(*args, **kwargs)
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\engine\base.py", line 2182, in execute
return connection.execute(statement, *multiparams, **params)
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\engine\base.py", line 976, in execute
return self._execute_text(object_, multiparams, params)
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\engine\base.py", line 1143, in _execute_text
ret = self._execute_context(
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\engine\base.py", line 1249, in _execute_context
self._handle_dbapi_exception(
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\engine\base.py", line 1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\util\compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\util\compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\engine\base.py", line 1245, in _execute_context
self.dialect.do_execute(
File "C:\Users\haroo501\AppData\Local\Programs\Python\Python38\lib\site-packages\sqlalchemy\engine\default.py", line 588, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]CREATE DATABASE statement not allowed within multi-statement transaction. (226) (SQLExecDirectW)')
[SQL: CREATE DATABASE MyNewDatabase]
(Background on this error at: http://sqlalche.me/e/f405)
Process finished with exit code 1
I tried to modify this database and works fine but I have to assume the permission for the use.
I am using MicroSoft SQL Managment Studio SQL EXPRESS:
Server Type Database Engine
Authentication Windows Authentication I don't have use name and password for the database
I think now the problem in this part:
'mssql+pyodbc://xxxx\SMARTRNO_EXPRESS/toolDB?driver=SQL+Server+Native+Client+11.0'
That I use this database connection string to connect directly to the toolDB
So I need something like a connection String as the below one:
# connect db
engine = create_engine('mssql+pyodbc://xxxx\SMARTRNO_EXPRESS?driver=SQL+Server+Native+Client+11.0')
connection = engine.connect()
as to able to create a database in this server and able to delete or create or even modify database
Well! I solved this by creating a new connection in database .py file as the below code and by adding the autocommit = True:
conn = pyodbc.connect("driver={SQL Server};server=WINKPN-3B5JTT2\SMARTRNO_EXPRESS; database=master; trusted_connection=true",
autocommit=True)
And Tried to access this connection by the below code:
from database import connec
def create_all_tables_from_db():
create_all_tables_query = "CREATE DATABASE MyNewDatabase"
connec.conn.execute(create_all_tables_query)
create_all_tables_from_db()
I'm trying to connect to a database from python, running in a debian based docker container. I extracted the bit of code that's causing the failure:
import sqlalchemy
from db_tools import generate_sandbox_connection
con = generate_sandbox_connection('churn', 'PASSWORD')
metadata = sqlalchemy.MetaData()
report_lookup = sqlalchemy.Table('REPORT_LOOKUP',
metadata,
schema = 'SCHEMA',
autoload = True,
autoload_with = con)
This causes the following output:
Traceback (most recent call last):
File "sqlalchemytest.py", line 43, in <module>
autoload_with = sandb_con)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 457, in __new__
metadata._remove_table(name, schema)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 452, in __new__
table._init(name, metadata, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 534, in _init
include_columns, _extend_on=_extend_on)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 547, in _autoload
_extend_on=_extend_on
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2056, in run_callable
...
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (cx_Oracle.ProgrammingError) positional and named binds cannot be intermixed [SQL: u'SELECT table_name, compression, compress_for FROM ALL_TABLES WHERE table_name = :table_name AND owner = :owner '] [parameters: {'owner': u'TRIP', 'table_name': u'CHURN_REPORT_LOOKUP'}] (Background on this error at: http://sqlalche.me/e/f405)
Anyone know how to fix this? It runs just fine on my local machine but it's breaking down in the docker instance.
It is a known bug in latest v6.4:
https://github.com/oracle/python-cx_Oracle/issues/199
For now, and until v6.4.1 is available, just downgrade to v6.3.1.
So, I wanted to move a dataframe to a database into a table but I am encountering a few problems. I made the dataframe using the pandas library and wanted to upload the data into a table in my database.
Relevant snippets of my Code and the error details are as follows:
#CONNECTION TO SQL DATABASE
db=sql.connect(host="localhost",user="root",passwd="password",db="database_name")
cur=db.cursor()
#CONVERTING DATAFRAME(df_final) INTO TABLE
df_final.to_sql(con=db, name='finaltable', if_exists='replace', chunksize=5000)
Error details:
Traceback (most recent call last):
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\M
ySQLdb\cursors.py", line 238, in execute
query = query % args
TypeError: not all arguments converted during string formatting
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\io\sql.py", line 1400, in execute
cur.execute(*args)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\M
ySQLdb\cursors.py", line 240, in execute
self.errorhandler(self, ProgrammingError, str(m))
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\M
ySQLdb\connections.py", line 52, in defaulterrorhandler
raise errorclass(errorvalue)
_mysql_exceptions.ProgrammingError: not all arguments converted during string fo
rmatting
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "pt.py", line 77, in <module>
df_final.to_sql(con=db, name='finaltable', if_exists='replace', chunksize=50
00)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\core\generic.py", line 2127, in to_sql
dtype=dtype)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\io\sql.py", line 450, in to_sql
chunksize=chunksize, dtype=dtype)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\io\sql.py", line 1502, in to_sql
table.create()
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\io\sql.py", line 561, in create
if self.exists():
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\io\sql.py", line 549, in exists
return self.pd_sql.has_table(self.name, self.schema)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\io\sql.py", line 1514, in has_table
return len(self.execute(query, [name, ]).fetchall()) > 0
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\io\sql.py", line 1412, in execute
raise_with_traceback(ex)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\compat\__init__.py", line 403, in raise_with_traceback
raise exc.with_traceback(traceback)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\p
andas\io\sql.py", line 1400, in execute
cur.execute(*args)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\M
ySQLdb\cursors.py", line 240, in execute
self.errorhandler(self, ProgrammingError, str(m))
File "C:\Users\DELL\AppData\Local\Programs\Python\Python36\lib\site-packages\M
ySQLdb\connections.py", line 52, in defaulterrorhandler
raise errorclass(errorvalue)
pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT name FROM
sqlite_ma
ster WHERE type='table' AND name=?;': not all arguments converted during string
formatting
So apparently my approach has gone obsolete due to updates and therefore for the connection bit, I used 'create_engine' from 'sqlalchemy' library instead of 'mysqldb.connect()' to access the database:
#relevant import:
from sqlalchemy import create_engine
#accessing database:
engine = create_engine("mysql://root:PASSWORD#host/database_name")
con = engine.connect()
#uploading dataframe to database:
df.to_sql(con=con, name='final_table', if_exists='replace', chunksize=10000)
For me, it seems that you have nulls in the data. Try to fill the nulls in the dataframe before calling .to_sql()
df_final.fillna('default string')
Or to remove the null rows
df_final.dropna(inplace=True)
guys! Hope someone can help me with this issue.
I executing a query through SQLAlchemy that returns ~6kk rows (it's historical data) that I need to process on a python script. I have some functions to read and do some processing on the data using pandas dataframe.Here are the functions:
def consulta_db_cancelamentos(db_con, query):
engine = create_engine(db_con, pool_recycle=3600)
con = engine.connect()
query_result = con.execution_options(stream_results=True).execute(query)
query_result_list = []
while True:
rows = query_result.fetchmany(10000)
if not rows:
break
for row in rows:
data = row['data'],\
row['plano'],\
row['usuario_id'],\
row['timestamp_cancelamentos'],\
row['timestamp'],\
row['status']
query_result_list.append(data)
df = pd.DataFrame()
if df.empty:
df = pd.DataFrame(query_result_list)
else:
df.append(pd.DataFrame(query_result_list))
df_cor = corrige_cancelamentos_df(df, '2017-01-01', '2017-12-15')
con.close()
return df_cor
As you can see, I'm already trying to read the data and process/store it in 10k rows chunk. When I try to execute the whole script I got this error on the function (I'm also including the error raised on the main()):
Traceback (most recent call last):
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1159, in fetchmany
l = self.process_rows(self._fetchmany_impl(size))
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1318, in _fetchmany_impl
row = self._fetchone_impl()
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1308, in _fetchone_impl
self.__buffer_rows()
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1295, in __buffer_rows
self.__rowbuffer = collections.deque(self.cursor.fetchmany(size))
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 485, in fetchmany
row = self.read_next()
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 446, in read_next
return self._conv_row(self._result._read_rowdata_packet_unbuffered())
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1430, in _read_rowdata_packet_unbuffered
packet = self.connection._read_packet()
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1008, in _read_packet
recv_data = self._read_bytes(bytes_to_read)
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1037, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/aiquis/EI/cancelamentos_testes5.py", line 180, in <module>
main()
File "/home/aiquis/EI/cancelamentos_testes5.py", line 164, in main
cancelamentos_df_corrigido = consulta_db_cancelamentos(db_param, query_cancelamentos)
File "/home/aiquis/EI/cancelamentos_testes5.py", line 14, in consulta_db_cancelamentos
rows = query_result.fetchmany(1000)
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1166, in fetchmany
self.cursor, self.context)
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1159, in fetchmany
l = self.process_rows(self._fetchmany_impl(size))
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1318, in _fetchmany_impl
row = self._fetchone_impl()
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1308, in _fetchone_impl
self.__buffer_rows()
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1295, in __buffer_rows
self.__rowbuffer = collections.deque(self.cursor.fetchmany(size))
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 485, in fetchmany
row = self.read_next()
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 446, in read_next
return self._conv_row(self._result._read_rowdata_packet_unbuffered())
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1430, in _read_rowdata_packet_unbuffered
packet = self.connection._read_packet()
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1008, in _read_packet
recv_data = self._read_bytes(bytes_to_read)
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1037, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
Exception ignored in: <bound method MySQLResult.__del__ of <pymysql.connections.MySQLResult object at 0x7f8c543dc198>>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1345, in __del__
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1447, in _finish_unbuffered_query
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 991, in _read_packet
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1022, in _read_bytes
AttributeError: 'NoneType' object has no attribute 'settimeout'
[Finished in 602.4s]
The way I wrote consulta_db_cancelamentos is already a result of some search on SO and SQLAlchemy documentation. Suppose I have no access to my MySQL Server administration.
When I limit my queryy to bring results for only one usuario_id for example (something like ~50 rows) it works fine. I executed the same query on MySQL Workbench and the Duration/Fetch was 251.998 sec/357.541 sec
Solved executing this command in MySQL Server:
set global max_allowed_packet = 67108864;
This solution was suggested here Lost connection to MySQL server during query
I faced a problem like this one. I faced problem after I put stream_results=True to the query.
For me, the parameter net_write_timeout was causing a trouble. When stream_results=True and the app does not immediately read all the data being sent from the MySQL server, then on the MySQL server side, the write communication (sending data packet) to the app gets blocked. net_write_timeout seems to be the parameter which controls how many seconds the connection is allowed for the MySQL server's write to the socket operation being blocked.
So, I've changed net_write_timeout parameter from 60 (default) to 3600 and solved the problem.