Develop Reference

Spammed with errors from asyncpg when discord bot is turned off

I am trying to integrate asyncpg with discord.py, however I ran into one very anoyying issue.
Whenever I try to stop the bot using ^C, I get spammed with a ton of errors. This is just bothersome as when I'm trying to debug something I often lose the original error.
Here is my code:
loop = asyncio.get_event_loop()
async def connection_init(conn):
await conn.execute("SET CLIENT_ENCODING to 'utf-8';")
conn.client = client
try:
client.pool = loop.run_until_complete(asyncpg.create_pool(
host=os.environ.get("postgres_host"),
database=os.environ.get("postgres_database"),
user=os.environ.get("postgres_user"),
password=os.environ.get("postgres_password"),
connection_class=dbutils.DBUtils,
init=connection_init
))
print('PostgreSQL connection successful')
except Exception as e:
print(e)
# the bot basically cannot function without database
print('PostgreSQL connection failed- aborting')
exit()
client.run(os.environ.get("main"))
Here are the errors that floods my terminal. It's the same error however it pops up like 20 times.
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x033DF148>
Traceback (most recent call last):
File "C:\Users\zghan\AppData\Local\Programs\Python\Python38-32\lib\asyncio\proactor_events.py", line 116, in __del__
File "C:\Users\zghan\AppData\Local\Programs\Python\Python38-32\lib\asyncio\proactor_events.py", line 108, in close
File "C:\Users\zghan\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 719, in call_soon
File "C:\Users\zghan\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 508, in _check_closed
RuntimeError: Event loop is closed

That error happens because the asyncio.get_event_loop() can not retrieve any loop, as at the moment you try to retrieve it, it is already closed.
The correct syntax would be to start a new loop instead of trying to get the current:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(asyncio.new_event_loop())
# ...
loop = asyncio.get_event_loop()
Alternatively, in Python 3.7 asyncio added a new way of managing the loops: using asyncio.run(), which does not require you to create, set or retrieve the current loop as it already manages all that internally.

Why a traceback is not thrown when accessing an element in a dict that does not exist? [duplicate]

This question already has answers here:
Paho MQTT Python Client: No exceptions thrown, just stops
(3 answers)
Closed 2 years ago.
I have a place in my code where I made a mistake in the name of the key of a dict. It took some time to understand why the code was not running past that place because a traceback was not thrown.
The code is below, I put it for completeness, highlighting with →→→ the place where the issue is:
class Alert:
lock = threading.Lock()
sent_alerts = {}
#staticmethod
def start_alert_listener():
# load existing alerts to keep persistancy
try:
with open("sent_alerts.json") as f:
json.load(f)
except FileNotFoundError:
# there is no file, never mind - it will be created at some point
pass
# start the listener
log.info("starting alert listener")
client = paho.mqtt.client.Client()
client.on_connect = Alert.mqtt_connection_alert
client.on_message = Alert.alert
client.connect("mqtt.XXXX", 1883, 60)
client.loop_forever()
#staticmethod
def mqtt_connection_alert(client, userdata, flags, rc):
if rc:
log.critical(f"error connecting to MQTT: {rc}")
sys.exit()
topic = "monitor/+/state"
client.subscribe(topic)
log.info(f"subscribed alert to {topic}")
#staticmethod
def alert(client, userdata, msg):
event = json.loads(msg.payload)
log.debug(f"received alert: {event}")
→→→ if event['ok']:
# remove existing sent flag, not thread safe!
with Alert.lock:
Alert.sent_alerts.pop(msg['id'], None)
return
(...)
The log coming from the line just above is
2021-01-14 22:03:02,617 [monitor] DEBUG received alert: {'full_target_name': 'ThisAlwaysFails → a failure', 'isOK': False, 'why': 'explicit fail', 'extra': None, 'id': '6507a61c9688199a34cb006b354c8433', 'last': '2021-01-14T22:03:02.612912+01:00', 'last_ko': '2021-01-14T22:03:02.612912+01:00'}
This is the dict in which I am trying to erroneously access ok, which should raise an exception and a traceback. But nothing happens. The code does not further than that as if the error was silently discarded (and the method silently fails).
I tried to put a raise Exception("wazaa") between the log.debug() and the if - same thing, the method fails at that point but an exception is not raised.
I am at loss about the reason where an exception could not be visible though a traceback?
The alert() method is called in a separate thread, if this matters. For completeness I tried the follwong code just to make sure threading does not interfere but no (I do not see a reason why it should)
import threading
class Hello:
#staticmethod
def a():
raise Exception("I am an exception")
threading.Thread(target=Hello.a).start()
outputs
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python38\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Python38\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:/Users/yop/AppData/Roaming/JetBrains/PyCharm2020.3/scratches/scratch_1.py", line 7, in a
raise Exception("I am an exception")
Exception: I am an exception

It appears to call your callback within a try, then logs that error:
try:
self.on_message(self, self._userdata, message)
except Exception as err:
self._easy_log(
MQTT_LOG_ERR, 'Caught exception in on_message: %s', err)
if not self.suppress_exceptions:
raise
What I can't explain however is why the exception isn't being raised. I can't see why self.suppress_exceptions would be true for you since you never set it, but try:
Manually setting suppress_exceptions using client.suppress_exceptions = False. This shouldn't be necessary since that appears to be the default, but it's worth a try.
Checking the log that it apparently maintains. You'll need to refer to the docs though on how to do that, since I've never touched this library before.

How to catch custom exception from signal handler in asyncio?

I'm having problems catching a custom exception when thrown from a signal handler callback when using asyncio.
If I throw ShutdownApp from within do_io() below, I am able to properly catch it in run_app(). However, when the exception is raised from handle_sig(), I can't seem to catch it.
Minimal, Reproducible Example Tested using Python 3.8.5:
import asyncio
from functools import partial
import os
import signal
from signal import Signals
class ShutdownApp(BaseException):
pass
os.environ["PYTHONASYNCIODEBUG"] = "1"
class App:
def __init__(self):
self.loop = asyncio.get_event_loop()
def _add_signal_handler(self, signal, handler):
self.loop.add_signal_handler(signal, handler, signal)
def setup_signals(self) -> None:
self._add_signal_handler(signal.SIGINT, self.handle_sig)
def handle_sig(self, signum):
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising error for exit")
raise ShutdownApp("Exiting")
async def do_io(self):
print("io start. Press Ctrl+C now.")
await asyncio.sleep(5)
print("io end")
def run_app(self):
print("Starting Program")
try:
self.loop.run_until_complete(self.do_io())
except ShutdownApp as e:
print("ShutdownApp caught:", e)
# TODO: do other shutdown related items
except:
print("Other error")
finally:
self.loop.close()
if __name__ == "__main__":
my_app = App()
my_app.setup_signals()
my_app.run_app()
print("Finished")
The output after pressing CTRL+C (for SIGINT) with asyncio debug mode:
(env_aiohttp) anav#anav-pc:~/Downloads/test$ python test_asyncio_signal.py
Starting Program
io start. Press Ctrl+C now.
^C
pid: 20359, Received signal: SIGINT, raising error for exit
Exception in callback App.handle_sig(<Signals.SIGINT: 2>)
handle: <Handle App.handle_sig(<Signals.SIGINT: 2>) created at /home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/unix_events.py:99>
source_traceback: Object created at (most recent call last):
File "test_asyncio_signal.py", line 50, in <module>
my_app.setup_signals()
File "test_asyncio_signal.py", line 25, in setup_signals
self._add_signal_handler(signal.SIGINT, self.handle_sig)
File "test_asyncio_signal.py", line 22, in _add_signal_handler
self.loop.add_signal_handler(signal, handler, signal)
File "/home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/unix_events.py", line 99, in add_signal_handler
handle = events.Handle(callback, args, self, None)
Traceback (most recent call last):
File "/home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "test_asyncio_signal.py", line 31, in handle_sig
raise ShutdownApp("Exiting")
ShutdownApp: Exiting
io end
Finished
Expected output:
Starting Program
io start. Press Ctrl+C now.
^C
pid: 20359, Received signal: SIGINT, raising error for exit
ShutdownApp caught: Exiting
io end
Finished
Is it possible to raise a custom exception from a signal handler in asyncio? If so, how do I properly catch/except it?

handle_sig is a callback, so it runs directly off the event loop and its exceptions are just reported to the user via a global hook. If you want the exception raised there to be caught elsewhere in the program, you need to use a future to transfer the exception from handle_sig to where you want it noticed.
To catch the exception at top-level, you probably want to introduce another method, let's call it async_main(), that waits for either self.do_io() or the previously-created future to complete:
def __init__(self):
self.loop = asyncio.get_event_loop()
self.done_future = self.loop.create_future()
async def async_main(self):
# wait for do_io or done_future, whatever happens first
io_task = asyncio.create_task(self.do_io())
await asyncio.wait([self.done_future, io_task],
return_when=asyncio.FIRST_COMPLETED)
if self.done_future.done():
io_task.cancel()
await self.done_future # propagate the exception, if raised
else:
self.done_future.cancel()
To raise the exception from inside handle_sig, you just need to set the exception on the future object:
def handle_sig(self, signum):
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising error for exit")
self.done_future.set_exception(ShutdownApp("Exiting"))
Finally, you modify run_app to pass self.async_main() to run_until_complete, and you're all set:
$ python3 x.py
Starting Program
io start. Press Ctrl+C now.
^C
pid: 2069230, Received signal: SIGINT, raising error for exit
ShutdownApp caught: Exiting
Finished
In closing, note that reliably catching keyboard interrupts is a notoriously tricky undertaking and the above code might not cover all the corner cases.

If I throw ShutdownApp from within do_io() below, I am able to properly catch it in run_app(). However, when the exception is raised from handle_sig(), I can't seem to catch it.
Response to the query given above
Custom Exception Implementation:
class RecipeNotValidError(Exception):
def __init__(self):
self.message = "Your recipe is not valid"
try:
raise RecipeNotValidError
except RecipeNotValidError as e:
print(e.message)
In the method handle_sig, add try and except block. Also, you can customize messages in custom exceptions.
def handle_sig(self, signum):
try:
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising
error for exit")
raise ShutdownApp("Exiting")
except ShutdownApp as e:
print(e.message)
In response to your second query :
Is it possible to raise a custom exception from a signal handler in asyncio? If so, how do I properly catch/except it?
In-built error handling in Asyncio. For more documentation visit https://docs.python.org/3/library/asyncio-exceptions.html
import asyncio
async def f():
try:
while True: await asyncio.sleep(0)
except asyncio.CancelledError:
print('I was cancelled!')
else:
return 111

Tornado: Task exception was never retrieved

I'm getting an Exception in a Tornado WebSocket server but It gives no information in the trace to know which line of code or which step in my program it is originating from. I would like to find out so that I try-catch the origin of the exception.
Error Trace: (No mention of any part of my files)
[E 200527 21:07:19 base_events:1608] Task exception was never retrieved
future: <Task finished coro=<WebSocketProtocol13.write_message.<locals>.wrapper() done, defined at /usr/local/lib/python3.7/site-packages/tornado/websocket.py:1102> exception=WebSocketClosedError()>
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/tornado/websocket.py", line 1104, in wrapper
await fut
tornado.iostream.StreamClosedError: Stream is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/tornado/websocket.py", line 1106, in wrapper
raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError
[E 200527 21:07:19 base_events:1608] Task exception was never retrieved
That Same Group of Traceback repeats over 16 times.
Here is my code:
import tornado.iostream
import tornado.web
import tornado.gen
import tornado.websocket
import asyncio
class SocketHandler(tornado.websocket.WebSocketHandler):
waiters = set()
def initialize(self):
self.client_name = "newly_connected"
def get_compression_options(self):
# Non-None enables compression with default options.
return {}
def open(self):
print('connection opened')
# SocketHandler.waiters.add(self)
def on_close(self):
print("CLOSED!", self.client_name)
try:
SocketHandler.waiters.remove(self)
except KeyError:
print('tried removing new client')
def check_origin(self, origin):
# Override the origin check if needed
return True
#classmethod
async def send_updates(cls, message):
if len(cls.waiters) < 2:
while True:
chat = {}
# Prevent RuntimeError: Set changed size during iteration
waiters_copy = cls.waiters.copy()
for waiter in waiters_copy:
try:
await waiter.write_message(chat)
except tornado.websocket.WebSocketClosedError:
pass
except tornado.iostream.StreamClosedError:
pass
except Exception as e:
print('Exception e:', waiter.client_name)
pass
# sleep a bit
await asyncio.sleep(0.05)
else:
print('broadcast loop already running')
async def on_message(self, message):
print("RECEIVED :", message)
self.client_name = message
self.first_serve_cache_on_connnect()
SocketHandler.waiters.add(self)
await SocketHandler.send_updates(message)
def first_serve_cache_on_connnect(self):
print('serving cache on connect')
temp_calc_results = self.namespace.results
try:
self.write_message(temp_calc_results)
except Exception as e:
pass
I have tried catching the exceptions that may cause any error while sending messages to the websocket clients but this error still happens when clients connect to the server.

The message "task exception was never retrieved" is not about a missing try/except block, it's about a missing await. In first_serve_cache_on_connect, you call write_message without await, so first_serve_cache_on_connect is no longer running by the time the exception is raised and it has nowhere to go but the logs.
This is basically harmless, but if you want to clean up your logs, you need to make first_serve_cache_on_connect an async def coroutine, call it with await in on_message, and call write_message with await.

What does this uWSGI error mean?

from gevent import monkey
monkey.patch_all()
import gevent
from gevent import pywsgi
from gevent import queue
import redis
REDIS_CONNECTION_POOL = redis.ConnectionPool(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)
def redis_wait(environ, body, channel, wait_once):
server = redis.Redis(connection_pool=REDIS_CONNECTION_POOL)
client = server.pubsub()
client.subscribe(channel)
messages = client.listen()
while True:
message = messages.next()
This is the error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
result = self._run(*self.args, **self.kwargs)
File "/home/ubuntu/www/app.wsgi", line 110, in wait_messages
redis_wait(environ, body, channel, False)
File "/home/ubuntu/www/app.wsgi", line 47, in redis_wait
message = messages.next()
StopIteration
<Greenlet at 0x1c190f0: wait_messages({}, ['5386E49C1CEB16573ACBD90566F3B740983768CB,1358532, <Queue at 0x19fd6d0>, '1410290151', None)> failed with StopIteration
I have tried to google the error, but nothing comes up. The error only occurs intermittently. Does anyone know what it means? Is it a timeout of some sort perhaps?

StopIteration is the exception that Python throws when an iterator (such as messages) has reached the end of its values. It is not an error, but a normal, expected condition that will be automatically handled by Python in some circumstances. For example, if you loop over the iterator using a for loop like so:
for message in messages:
print message # Or do something with it
then the StopIteration exception will end the for loop normally.
However, a while loop does not handle StopIteration itself, but lets it continue through to your code so that you can handle it in whatever way you see fit. Your code is currently not handling it, so the exception ends up terminating your program.
Replace your while True: message = messages.next() loop with for message in messages and everything should work.
More reading on iterators, generators, and StopIteration:
http://anandology.com/python-practice-book/iterators.html
http://www.python-course.eu/generators.php
http://chimera.labs.oreilly.com/books/1230000000393/ch04.html