Develop Reference

Querying on mysql docker container via python, throwing timeout error after few hours

Inserting via debezium connector to mysql database brought up via docker container.
Trying to query and it is working fine until some number of hours. But, after that, same query is throwing below exception.
export JAVA_HOME=/tmp/tests/artifacts/java-17/jdk-17; export PATH=$PATH:/tmp/tests/artifacts/java-17/jdk-17/bin; docker exec -i mysql_be1e6a mysql --user=demo --password=demo -D demo -e "select count(k) from test_cdc_f0bf84 where uuid = 'd1e5cd6d-8f7a-457c-b2ea-880c2be52f69'"
2023-01-02 16:27:43,812:ERROR: failed to execute query MySQL rows count by uuid:
Traceback (most recent call last):
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/channel.py", line 699, in recv
out = self.in_buffer.read(nbytes, self.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/buffered_pipe.py", line 164, in read
raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/suites/cdc/abstract.py", line 667, in try_query
res = query_function()
^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/suites/cdc/test_cdc.py", line 635, in <lambda>
query = lambda: self.mysql_query(
^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/suites/cdc/abstract.py", line 544, in mysql_query
result = self.ssh.exec_on_host(host, [
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/main/connection.py", line 335, in exec_on_host
return self._exec_on_host(host, commands, fetch, timeout=timeout, limit_output=limit_output)[host]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/main/connection.py", line 321, in _exec_on_host
res = list(out)
^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/file.py", line 125, in __next__
line = self.readline()
^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/file.py", line 291, in readline
new_data = self._read(n)
^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/channel.py", line 1361, in _read
return self.channel.recv(size)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/channel.py", line 701, in recv
raise socket.timeout()
TimeoutError
After some time, logged manually to machine and tried to read, it still reads fine. Not sure, what does this issue mean.
As explained, tried querying from database via python. Expected it will return count of rows, which it was happening until certain time, but after that, it threw timeout error and socket error.

Trying to query and it is working fine until some number of hours. But, after that, same query is throwing below exception.
The default value for interactive_timeout and wait_timeout is 28880 seconds (8 hours). you can disable this behavior by setting this system variable to zero in your MySQL config.
source: Configuring session timeouts

pyhive connection issue after running a query that takes too long

I am using pandas.read_sql function with hive connection to extract a really large data. I have a script like this:
df = pd.read_sql(query_big, hive_connection)
df2 = pd.read_sql(query_simple, hive_connection)
The big query take a long time, and after it is executed, python returns the following error when trying to execute the second line:
raise NotSupportedError("Hive does not have transactions") # pragma: no cover
It seems there is something wrong with the connection.
Moreover, If I replace the second line with multirpocessing.Manager().Queue(), It returns the following error:
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 662, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 554, in _create
conn = self._Client(self._address, authkey=self._authkey)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 493, in Client
answer_challenge(c, authkey)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 732, in answer_challenge
message = connection.recv_bytes(256) # reject large message
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
It seems this kind of error are related to exit function being messed up, in the connection.py. Moreover, when I changed the query in the first command to extract smaller data that doesn't take too long, Everything works fine. So I assume it may be that because it takes too long to execute the first query, something is improperly terminated. which caused the two error, both of which are so different in nature but both are related to broken connection issues.

Python script suddenly throwing timeout exceptions

I have a Python script that downloads product feeds from multiple affiliates in different ways. This didn't give me any problems until last Wednesday, when it started throwing all kinds of timeout exceptions from different locations.
Examples: Here I connect with a FTP service:
ftp = FTP(host=self.host)
threw:
Exception in thread Thread-7:
Traceback (most recent call last):
File "C:\Python27\Lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\Lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Users\Administrator\Documents\Crawler\src\Crawlers\LDLC.py", line 23, in main
ftp = FTP(host=self.host)
File "C:\Python27\Lib\ftplib.py", line 120, in __init__
self.connect(host)
File "C:\Python27\Lib\ftplib.py", line 138, in connect
self.welcome = self.getresp()
File "C:\Python27\Lib\ftplib.py", line 215, in getresp
resp = self.getmultiline()
File "C:\Python27\Lib\ftplib.py", line 201, in getmultiline
line = self.getline()
File "C:\Python27\Lib\ftplib.py", line 186, in getline
line = self.file.readline(self.maxline + 1)
File "C:\Python27\Lib\socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
Or downloading an XML File :
xmlFile = urllib.URLopener()
xmlFile.retrieve(url, self.feedPath + affiliate + "/" + website + '.' + fileType)
xmlFile.close()
throws:
File "C:\Users\Administrator\Documents\Crawler\src\Crawlers\FeedCrawler.py", line 106, in save
xmlFile.retrieve(url, self.feedPath + affiliate + "/" + website + '.' + fileType)
File "C:\Python27\Lib\urllib.py", line 240, in retrieve
fp = self.open(url, data)
File "C:\Python27\Lib\urllib.py", line 208, in open
return getattr(self, name)(url)
File "C:\Python27\Lib\urllib.py", line 346, in open_http
errcode, errmsg, headers = h.getreply()
File "C:\Python27\Lib\httplib.py", line 1139, in getreply
response = self._conn.getresponse()
File "C:\Python27\Lib\httplib.py", line 1067, in getresponse
response.begin()
File "C:\Python27\Lib\httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "C:\Python27\Lib\httplib.py", line 365, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "C:\Python27\Lib\socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
IOError: [Errno socket error] timed out
These are just two examples but there are other methods, like authenticate or other API specific methods where my script throws these timeout errors. It never showed this behavior until Wednesday. Also, it starts throwing them at random times. Sometimes at the beginning of the crawl, sometimes later on. My script has this behavior on both my server and my local machine. I've been struggling with it for two days now but can't seem to figure it out.
This is what I know might have caused this:
On Wednesday one affiliate script broke down with the following error:
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
I didn't change anything to my script but suddenly it stopped crawling that affiliate and threw that error all the time where I tried to authenticate. I looked it up and found that is was due to an OpenSSL error (where did that come from). I fixed it by adding the following before the authenticate method:
if hasattr(ssl, '_create_unverified_context'):
ssl._create_default_https_context = ssl._create_unverified_context
Little did I know, this was just the start of my problems... At that same time, I changed from Python 2.7.8 to Python 2.7.9. It seems that this is the moment that everything broke down and started throwing timeouts.
I tried changing my script in endless ways but nothing worked and like I said, it's not just one method that throws it. Also I switched back to Python 2.7.8, but this didn't do the trick either. Basically everything that makes a request to an external source can throw an error.
Final note: My script is multi threaded. It downloads product feeds from different affiliates at the same time. It used to run 10 threads per affiliate without a problem. Now I tried lowering it to 3 per affiliate, but it still throws these errors. Setting it to 1 is no option because that will take ages. I don't think that's the problem anyway because it used to work fine.
What could be wrong?

pymongo operation error

I am getting a lot of records from my MongoDB and along the way I get an error
File "C:\database\models\mongodb.py", line 228, in __iter__
for result in self.results:
File "C:\Python27\Lib\site-packages\pymongo\cursor.py", line 814, in next
if len(self.__data) or self._refresh():
File "C:\Python27\Lib\site-packages\pymongo\cursor.py", line 776, in _refresh
limit, self.__id))
File "C:\Python27\Lib\site-packages\pymongo\cursor.py", line 720, in __send_message
self.__uuid_subtype)
File "C:\Python27\Lib\site-packages\pymongo\helpers.py", line 99, in _unpack_response
cursor_id)
pymongo.errors.OperationFailure: cursor id '866472135294727793' not valid at server
Exception KeyError: KeyError(38556896,) in <module 'threading' from 'C:\Python27\lib\threading.pyc'> ignored
What does this mean and how do I fix it. I don't know if it matters but I did
use from gevent import monkey; monkey.patch_all() when I opened the connection

When the cursor has been open for a long time with no operations on it, it(the cursor) can timeout -> this leads to the error
you can set timeout=False in your find query to turn the timeout off
reference

Django TemplateSyntaxError: current transaction is aborted, what does this exception mean? Does postgresql 8.4 work fine with django?

The full text of the error is:
TemplateSyntaxError at /
Caught an exception while rendering:
current transaction is aborted,
commands ignored until end of
transaction block
I've recently reinstalled all the software on my computer. The code used to work no problem before. It also still works no problem on a friend's computer and on a development server.
The only thing I can think of, which may have changed is the postgresql server version (and I'm not actually certain whether I tried running it on 8.4 on my old installation or not - it definitely worked on 8.3).
Is anyone able to confirm that Django has problems with postgresql 8.4 and/or has any hints as to why I'm having these errors?
Edit 1
In response to Dominic...
This doesn't happen just on one page or with one tag (though there are some pages which seem to be ok). The only thing the tags and variables that cause errors have in common is that they happen to access the database somewhere along the way (though not all tags that access the database cause the error). Furthermore, the same code does not create TemplateSyntaxError on other computers.
If I remove a variable or custom template tag that is making the error then this is the chain of events that happens:
I remove the variable or tag that is supposedely causing the error and refresh the page.
Django picks a different variable or tag and the same error occurs, so I keep removing them.
Once all the variables and tags that cause the error are removed, I stop getting a proper error page. I get a plain white page with the following traceback:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 279, in run
self.result = application(self.environ, self.start_response)
File "/usr/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 651, in __call__
return self.application(environ, start_response)
File "/usr/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 245, in __call__
response = middleware_method(request, response)
File "/usr/lib/python2.6/site-packages/debug_toolbar/middleware.py", line 90, in process_response
response.content = replace_insensitive(smart_unicode(response.content), u'', smart_unicode(self.debug_toolbars[request].render_toolbar() + u''))
File "/usr/lib/python2.6/site-packages/debug_toolbar/toolbar/loader.py", line 72, in render_toolbar
'BASE_URL': self.request.META.get('SCRIPT_NAME', ''),
File "/usr/lib/python2.6/site-packages/django/template/loader.py", line 108, in render_to_string
return t.render(context_instance)
File "/usr/lib/python2.6/site-packages/django/test/utils.py", line 29, in instrumented_test_render
return self.nodelist.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 779, in render
bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py", line 71, in render_node
result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/defaulttags.py", line 155, in render
nodelist.append(node.render(context))
File "/usr/lib/python2.6/site-packages/django/template/defaulttags.py", line 243, in render
return self.nodelist_true.render(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 779, in render
bits.append(self.render_node(node, context))
File "/usr/lib/python2.6/site-packages/django/template/debug.py", line 81, in render_node
raise wrapped
TemplateSyntaxError: Caught an exception while rendering: current transaction is aborted, commands ignored until end of transaction block
Original Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/django/template/debug.py", line 71, in render_node
result = node.render(context)
File "/usr/lib/python2.6/site-packages/django/template/debug.py", line 87, in render
output = force_unicode(self.filter_expression.resolve(context))
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 546, in resolve
obj = self.var.resolve(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 687, in resolve
value = self._resolve_lookup(context)
File "/usr/lib/python2.6/site-packages/django/template/__init__.py", line 722, in _resolve_lookup
current = current()
File "/usr/lib/python2.6/site-packages/debug_toolbar/panels/template.py", line 64, in content
pformat(k(self.request))) for k in get_standard_processors()
File "/usr/lib/python2.6/site-packages/django/core/context_processors.py", line 27, in auth
'messages': user.get_and_delete_messages(),
File "/usr/lib/python2.6/site-packages/django/contrib/auth/models.py", line 263, in get_and_delete_messages
for m in self.message_set.all():
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 106, in _result_iter
self._fill_cache()
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 692, in _fill_cache
self._result_cache.append(self._iter.next())
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 238, in iterator
for row in self.query.results_iter():
File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py", line 287, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py", line 2369, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.6/site-packages/debug_toolbar/panels/sql.py", line 91, in execute
return self.cursor.execute(sql, params)
InternalError: current transaction is aborted, commands ignored until end of transaction block

That exception means that there was an error in some SQL that is getting executed. Since Django runs all the SQL inside of a database transaction, all the SQL that is being executed after the error gets ignored. So:
BEGIN;
SELECT * FROM table;
SELECT missing_column FROM table WHERE id = 1; -- generates an error because the column is missing
SELECT * FROM another_table; -- this statement and all following statements get ignored until the next COMMIT;
COMMIT;
To figure out the problem, find your log file for PostgreSQL and run tail -f /path/to/postgresql_error.log. Then refresh the page. You should see the error come up in the log file.

If you're moving from a different DB, note that there might be some considerations to bear in mind with the way that Django with Postgres uses transactions. Have a read here:
http://docs.djangoproject.com/en/dev/topics/db/transactions/?from=olddocs#handling-exceptions-within-postgresql-transactions
The quick answer is usually to turn on database level autocommit by adding:
'OPTIONS': {'autocommit': True,}
to your database settings. But it's worth having a read up on it and understanding the coo.
Rolo.