I'm running the script wish to save the csv file to MOngoDB, and face pymongo.errors.ServerSelectionTimeoutError: Connection refused
(ps. '1xx.xx.xx.1:27017' is correct mongoDB ip)
the python script
import
import pandas as pd
from pymongo import MongoClient
client = MongoClient('1xx.xx.xx.1:27017')
db = client ['(practice_12_29)img_to_text-001_new']
collection = db ['img_to_text_listOfElems']
data = pd.read_csv('file_listOfElems.csv',encoding = 'UTF-8')
data_json = json.loads(data.to_json(orient='records'))
collection.insert(data_json)
this is the output
jetson#jetson-desktop:~/Desktop/test_12.26$ python3 csv_to_mongoDB.py
Traceback (most recent call last):
File "csv_to_mongoDB.py", line 13, in <module>
collection.insert(data_json)
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/collection.py", line 3182, in insert
check_keys, manipulate, write_concern)
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/collection.py", line 646, in _insert
blk.execute(write_concern, session=session)
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/bulk.py", line 511, in execute
return self.execute_command(generator, write_concern, session)
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/bulk.py", line 344, in execute_command
with client._tmp_session(session) as s:
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1820, in _tmp_session
s = self._ensure_session(session)
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1807, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1760, in __start_session
server_session = self._get_server_session()
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1793, in _get_server_session
return self._topology.get_server_session()
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/topology.py", line 477, in get_server_session
None)
File "/home/jetson/.local/lib/python3.6/site-packages/pymongo/topology.py", line 205, in _select_servers_loop
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: 1xx.xx.xx.1:27017: [Errno 111] Connection refused
(tried 01)
I tried one of the similar issue's solution,but is not work too
Pymongo keeps refusing the connection at 27017
jetson#jetson-desktop:~/Desktop/test_12.26$ sudo rm /var/lib/mongodb/mongod.lock
rm: cannot remove '/var/lib/mongodb/mongod.lock': No such file or directory
(tried 02)
I also find this https://stackoverflow.com/a/68608172/20861658
but how do I know the <username> , <password> and cluster-details , my last time experience with other computer can just upload excel with below code
# this worked fine, that I don't remember I put user name and password
import pandas as pd
from pymongo import MongoClient
aaa = pd.read_excel("T1_new.xls")
print(aaa.head)
client = MongoClient('1xx.xx.xx.1:27017')
db = client['sample_data_in_DB']
collection = db['sample_collection']
collection.insert_many(aaa.to_dict('records'))
if any idea just let me knogw
Related
It is impossible to connect an app in Flask / App Engine to Google Cloud MySQL.
The settings seems correct.
Here are the libraries:
python 38
Flask-SQLAlchemy==2.5.1
pymysql==1.0.2
Here is the mydatabase.py:
unix_socket = "/cloudsql/{conn_name}".format(conn_name=DBCONNECTION)
SQLALCHEMY_DATABASE_URI = ("mysql+pymysql://{user}:{password}#/{database}?unix_socket={socket}").format(user=USER, password=PWD,database=SQLDB, socket=unix_socket)
db = SQLAlchemy()
Here is the main.py:
from mydatabase import db,SQLALCHEMY_DATABASE_URI
db.init_app(app)
The error is strange, it seems that the driver isn't working correctly:
"POST /signin HTTP/1.1" 500
2022-12-06 14:41:34 default[2022120025] [2022-12-06 14:41:34,165] ERROR in app: Exception on /signin [POST]
2022-12-06 14:41:34 default[2022120025] Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/pymysql/connections.py", line 602, in connect sock.connect(self.unix_socket) ConnectionRefusedError: [Errno 111] Connection refused
2022-12-06 14:41:34 default[2022120025] During handling of the above exception, another exception occurred:
2022-12-06 14:41:34 default[2022120025] Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect return fn() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 325, in connect return _ConnectionFairy._checkout(self) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 888, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 491, in checkout rec = pool._do_get() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get self._dec_overflow() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ compat.raise_( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 210, in raise_ raise exception File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get return self._create_connection() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 271, in _create_connection return _ConnectionRecord(self) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 386, in __init__ self.__connect() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 685, in __connect pool.logger.debug("Error on connect(): %s", e) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ compat.raise_( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 210, in raise_ raise exception File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 680, in __connect self.dbapi_connection = connection = pool._invoke_creator(self) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/engine/create.py", line 578, in connect return dialect.connect(*cargs, **cparams) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 598, in connect return self.dbapi.connect(*cargs, **cparams) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/pymysql/connections.py", line 353, in __init__ self.connect() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/pymysql/connections.py", line 664, in connect raise exc pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 111] Connection refused)")
I've added additional Cloud SQL admin rights to the Compute Engine default service account and the App Engine default service account, but it doesn't solve the problem.
I've also tried to use the method with the public IP adding the IP into the the authorized list using curl api.ipify.org, but it doesn't work.
In fact, there was initially some rights and install issue, so I've checked a function and installed again the main components.
I've solved the issue by using the shell command gcloud sql instances describe instance_name that activates some admin rights and installing gcloud components install app-engine-python that probably install everything correctly.
I am attempting to convert a Flask webapp to run on PythonAnywhere instead of my Raspberry Pi on which it is currently hosted and functions perfectly.
One of the core features of the webapp is using Pandas to query an externally hosted MySQL database (NOT hosted on PythonAnywhere).
Previously I have done this using the following method, with no troubles:
import pandas as pd
URI=f"mysql://{user}#{host}:{port}/{schema}"
my_dataframe=pd.read_sql_query(sql="select * from users",con=URI)
Attempting this with the webapp hosted on PythonAnywhere results in a 502-backend error, with an error log of:
File "./webapp.py", line 240, in ages
message_, success=scripts.ages.main()
File "./scripts/ages.py", line 22, in main
my_dataframe=pd.read_sql_query(sql="select * from users",con=URI)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/pandas/io/sql.py", line 383, in read_sql_query
chunksize=chunksize,
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/pandas/io/sql.py", line 1295, in read_query
result = self.execute(*args)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/pandas/io/sql.py", line 1162, in execute
*args, **kwargs
File "<string>", line 2, in execute
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/util/deprecations.py", line 401, in warned
return fn(*args, **kwargs)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3145, in execute
connection = self.connect(close_with_result=True)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3204, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
else engine.raw_connection()
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3283, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3254, in _wrap_pool_connect
e, dialect, self
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2101, in _handle_dbapi_exception_noconnection
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect
return fn()
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 310, in connect
return _ConnectionFairy._checkout(self)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
rec = pool._do_get()
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
self._dec_overflow()
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
with_traceback=exc_tb,
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
return self._create_connection()
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
return _ConnectionRecord(self)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
self.__connect()
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
with_traceback=exc_tb,
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/engine/create.py", line 590, in connect
return dialect.connect(*cargs, **cparams)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 597, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/MySQLdb/__init__.py", line 130, in Connect
return Connection(*args, **kwargs)
File "/home/mywebapp/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/MySQLdb/connections.py", line 185, in __init__
super().__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2003, "Can't connect to MySQL server on '34.105.08.12:8080' (111)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)
(I've changed the IP)
I've take a look at the linked site https://sqlalche.me/e/14/e3q8, but didn't see anything useful on there.
I've also tried googling how to use sqlalchemy within PythonAnywhere. This mostly resulted in questions about connecting to databases hosted by PythonAnywhere, which doesn't help me.
I came across this link:
https://help.pythonanywhere.com/pages/UsingSQLAlchemywithMySQL/
I don't actually know if it's referring to using SQLAlchemy to connect to a PythonAnywhere hosted MySQL database, or an external one, but I tried implementing this anyway. I doubt it makes a difference, but the URI is in a separate file to the Pandas sql query.
#credentials.py
from sqlalchemy import create_engine
URI_string=f"mysql://{user}#{host}:{port}/{schema}"
URI = create_engine(URI_string, pool_recycle=280)
import pandas as pd
import credentials
my_dataframe=pd.read_sql_query(sql="select * from users",con=credentials.URI)
The webapp is still functional when testing on the Pi with this change, however it made no difference in the outcome when testing on PyAnywhere and the same error log output is received.
If anyone has any ideas please let me know!
If you're using a free account, you will only be able to connect out of PythonAnywhere using http(s) to a list of approved sites. So external database connections will not work from a free account.
I have tried multiple ways to connect to MSSQL with sql alchemy but always get the error below:
Traceback (most recent call last):
File "<ipython-input-100-71745b407575>", line 1, in <module>
conn = engine.connect()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 2473, in connect
return self._connection_cls(self, **kwargs)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 878, in __init__
self.__connection = connection or engine.raw_connection()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 2559, in raw_connection
return self.pool.unique_connection()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 184, in unique_connection
return _ConnectionFairy(self).checkout()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 401, in __init__
rec = self._connection_record = pool._do_get()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 746, in _do_get
con = self._create_connection()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 189, in _create_connection
return _ConnectionRecord(self)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\pool.py", line 287, in __init__
exec_once(self.connection, self)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\event.py", line 377, in exec_once
self(*args, **kw)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\event.py", line 386, in __call__
fn(*args, **kw)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\strategies.py", line 168, in first_connect
dialect.initialize(c)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\connectors\pyodbc.py", line 135, in initialize
super(PyODBCConnector, self).initialize(connection)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\dialects\mssql\base.py", line 1164, in initialize
super(MSDialect, self).initialize(connection)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\default.py", line 177, in initialize
self._get_default_schema_name(connection)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\dialects\mssql\base.py", line 1180, in _get_default_schema_name
user_name = connection.scalar("SELECT user_name() as user_name;")
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 1383, in scalar
return self.execute(object, *multiparams, **params).scalar()
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 1449, in execute
params)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 1628, in _execute_text
statement, parameters
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\base.py", line 1691, in _execute_context
context)
File "C:\Users\wantai1\AppData\Roaming\Python\Python37\site-packages\sqlalchemy\engine\default.py", line 333, in do_execute
cursor.execute(statement, parameters)
TypeError: The first argument to execute must be a string or unicode query.
Here is my code
import sqlalchemy as sa
import pyodbc
engine =sa.create_engine('mssql+pyodbc://uid:pwd#AWS1DNA01/xxx_Database,driver = SQL+Server')
conn = engine.connect()
In addition, I have tried to pass through the parameters as suggested by the documentation and got the same error.
conn_string = urllib.parse.quote_plus("Driver={SQL Server}; Server=AWS1DNA01; Database=xxx_Database; Integrated Security=False;UID=uid;PWD=pwd")
engine = sa.create_engine('mssql+pyodbc:///?odbc_connect={}'.format(conn_string))
conn = engine.connect()
What does this error mean and how do i fix it?
Also, I have tried to use pyodbc to connect and it works:
conn_string = "Driver={SQL Server}; Server=AWS1DNA01; Database=xxx_Database; Integrated Security=False;UID=uid;PWD=pwd"
connection = pyodbc.connect(conn_string)
Can someone please help with this?
I have finally fixed the issue, although I still don't know the cause of the original issue. Instead of using pyodbc, I used pymssql to create the engine and it works.
engine = sa.create_engine('mssql+pymssql://uid:pdw#server/db?')
guys! Hope someone can help me with this issue.
I executing a query through SQLAlchemy that returns ~6kk rows (it's historical data) that I need to process on a python script. I have some functions to read and do some processing on the data using pandas dataframe.Here are the functions:
def consulta_db_cancelamentos(db_con, query):
engine = create_engine(db_con, pool_recycle=3600)
con = engine.connect()
query_result = con.execution_options(stream_results=True).execute(query)
query_result_list = []
while True:
rows = query_result.fetchmany(10000)
if not rows:
break
for row in rows:
data = row['data'],\
row['plano'],\
row['usuario_id'],\
row['timestamp_cancelamentos'],\
row['timestamp'],\
row['status']
query_result_list.append(data)
df = pd.DataFrame()
if df.empty:
df = pd.DataFrame(query_result_list)
else:
df.append(pd.DataFrame(query_result_list))
df_cor = corrige_cancelamentos_df(df, '2017-01-01', '2017-12-15')
con.close()
return df_cor
As you can see, I'm already trying to read the data and process/store it in 10k rows chunk. When I try to execute the whole script I got this error on the function (I'm also including the error raised on the main()):
Traceback (most recent call last):
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1159, in fetchmany
l = self.process_rows(self._fetchmany_impl(size))
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1318, in _fetchmany_impl
row = self._fetchone_impl()
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1308, in _fetchone_impl
self.__buffer_rows()
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1295, in __buffer_rows
self.__rowbuffer = collections.deque(self.cursor.fetchmany(size))
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 485, in fetchmany
row = self.read_next()
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 446, in read_next
return self._conv_row(self._result._read_rowdata_packet_unbuffered())
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1430, in _read_rowdata_packet_unbuffered
packet = self.connection._read_packet()
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1008, in _read_packet
recv_data = self._read_bytes(bytes_to_read)
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1037, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/aiquis/EI/cancelamentos_testes5.py", line 180, in <module>
main()
File "/home/aiquis/EI/cancelamentos_testes5.py", line 164, in main
cancelamentos_df_corrigido = consulta_db_cancelamentos(db_param, query_cancelamentos)
File "/home/aiquis/EI/cancelamentos_testes5.py", line 14, in consulta_db_cancelamentos
rows = query_result.fetchmany(1000)
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1166, in fetchmany
self.cursor, self.context)
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1159, in fetchmany
l = self.process_rows(self._fetchmany_impl(size))
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1318, in _fetchmany_impl
row = self._fetchone_impl()
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1308, in _fetchone_impl
self.__buffer_rows()
File "/home/aiquis/.local/lib/python3.5/site-packages/sqlalchemy/engine/result.py", line 1295, in __buffer_rows
self.__rowbuffer = collections.deque(self.cursor.fetchmany(size))
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 485, in fetchmany
row = self.read_next()
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 446, in read_next
return self._conv_row(self._result._read_rowdata_packet_unbuffered())
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1430, in _read_rowdata_packet_unbuffered
packet = self.connection._read_packet()
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1008, in _read_packet
recv_data = self._read_bytes(bytes_to_read)
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1037, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
Exception ignored in: <bound method MySQLResult.__del__ of <pymysql.connections.MySQLResult object at 0x7f8c543dc198>>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1345, in __del__
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1447, in _finish_unbuffered_query
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 991, in _read_packet
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1022, in _read_bytes
AttributeError: 'NoneType' object has no attribute 'settimeout'
[Finished in 602.4s]
The way I wrote consulta_db_cancelamentos is already a result of some search on SO and SQLAlchemy documentation. Suppose I have no access to my MySQL Server administration.
When I limit my queryy to bring results for only one usuario_id for example (something like ~50 rows) it works fine. I executed the same query on MySQL Workbench and the Duration/Fetch was 251.998 sec/357.541 sec
Solved executing this command in MySQL Server:
set global max_allowed_packet = 67108864;
This solution was suggested here Lost connection to MySQL server during query
I faced a problem like this one. I faced problem after I put stream_results=True to the query.
For me, the parameter net_write_timeout was causing a trouble. When stream_results=True and the app does not immediately read all the data being sent from the MySQL server, then on the MySQL server side, the write communication (sending data packet) to the app gets blocked. net_write_timeout seems to be the parameter which controls how many seconds the connection is allowed for the MySQL server's write to the socket operation being blocked.
So, I've changed net_write_timeout parameter from 60 (default) to 3600 and solved the problem.
I am trying to use the in-memory SQLite database using SQLAlchemy with Python 2.7.3 on Windows. I can connect to the engine, but when I try to execute the second statement I am getting the following error:
>>> engine=create_engine('sqlite:///memory:',echo=True)
>>> engine.execute("select 1").scalar()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 2445, in execute
connection = self.contextual_connect(close_with_result=True)
File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 2489, in contextual_connect
self.pool.connect(),
File "C:\Python27\lib\site-packages\sqlalchemy\pool.py", line 236, in connect
return _ConnectionFairy(self).checkout()
File "C:\Python27\lib\site-packages\sqlalchemy\pool.py", line 401, in __init__
rec = self._connection_record = pool._do_get()
File "C:\Python27\lib\site-packages\sqlalchemy\pool.py", line 822, in _do_get
return self._create_connection()
File "C:\Python27\lib\site-packages\sqlalchemy\pool.py", line 189, in _create_connection
return _ConnectionRecord(self)
File "C:\Python27\lib\site-packages\sqlalchemy\pool.py", line 282, in __init__
self.connection = self.__connect()
File "C:\Python27\lib\site-packages\sqlalchemy\pool.py", line 344, in __connect
connection = self.__pool._creator()
File "C:\Python27\lib\site-packages\sqlalchemy\engine\strategies.py", line 80, in connect
return dialect.connect(*cargs, **cparams)
File "C:\Python27\lib\site-packages\sqlalchemy\engine\default.py", line 281, in connect
return self.dbapi.connect(*cargs, **cparams)
sqlalchemy.exc.OperationalError: (OperationalError) unable to open database file
None None
The filename should be :memory:, not memory:. (See the docs for in-memory databases). The relevant SQLAlchemy docs mention that's the default path, so you should use:
engine=create_engine('sqlite://',echo=True)
The error you're getting is presumably because memory: isn't a valid filename on Windows.
The following worked for me:
from sqlalchemy import create_engine
from sqlalchemy.pool import StaticPool
engine = create_engine("sqlite://", poolclass=StaticPool)