Develop Reference

Why a traceback is not thrown when accessing an element in a dict that does not exist? [duplicate]

This question already has answers here:
Paho MQTT Python Client: No exceptions thrown, just stops
(3 answers)
Closed 2 years ago.
I have a place in my code where I made a mistake in the name of the key of a dict. It took some time to understand why the code was not running past that place because a traceback was not thrown.
The code is below, I put it for completeness, highlighting with →→→ the place where the issue is:
class Alert:
lock = threading.Lock()
sent_alerts = {}
#staticmethod
def start_alert_listener():
# load existing alerts to keep persistancy
try:
with open("sent_alerts.json") as f:
json.load(f)
except FileNotFoundError:
# there is no file, never mind - it will be created at some point
pass
# start the listener
log.info("starting alert listener")
client = paho.mqtt.client.Client()
client.on_connect = Alert.mqtt_connection_alert
client.on_message = Alert.alert
client.connect("mqtt.XXXX", 1883, 60)
client.loop_forever()
#staticmethod
def mqtt_connection_alert(client, userdata, flags, rc):
if rc:
log.critical(f"error connecting to MQTT: {rc}")
sys.exit()
topic = "monitor/+/state"
client.subscribe(topic)
log.info(f"subscribed alert to {topic}")
#staticmethod
def alert(client, userdata, msg):
event = json.loads(msg.payload)
log.debug(f"received alert: {event}")
→→→ if event['ok']:
# remove existing sent flag, not thread safe!
with Alert.lock:
Alert.sent_alerts.pop(msg['id'], None)
return
(...)
The log coming from the line just above is
2021-01-14 22:03:02,617 [monitor] DEBUG received alert: {'full_target_name': 'ThisAlwaysFails → a failure', 'isOK': False, 'why': 'explicit fail', 'extra': None, 'id': '6507a61c9688199a34cb006b354c8433', 'last': '2021-01-14T22:03:02.612912+01:00', 'last_ko': '2021-01-14T22:03:02.612912+01:00'}
This is the dict in which I am trying to erroneously access ok, which should raise an exception and a traceback. But nothing happens. The code does not further than that as if the error was silently discarded (and the method silently fails).
I tried to put a raise Exception("wazaa") between the log.debug() and the if - same thing, the method fails at that point but an exception is not raised.
I am at loss about the reason where an exception could not be visible though a traceback?
The alert() method is called in a separate thread, if this matters. For completeness I tried the follwong code just to make sure threading does not interfere but no (I do not see a reason why it should)
import threading
class Hello:
#staticmethod
def a():
raise Exception("I am an exception")
threading.Thread(target=Hello.a).start()
outputs
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python38\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Python38\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:/Users/yop/AppData/Roaming/JetBrains/PyCharm2020.3/scratches/scratch_1.py", line 7, in a
raise Exception("I am an exception")
Exception: I am an exception

It appears to call your callback within a try, then logs that error:
try:
self.on_message(self, self._userdata, message)
except Exception as err:
self._easy_log(
MQTT_LOG_ERR, 'Caught exception in on_message: %s', err)
if not self.suppress_exceptions:
raise
What I can't explain however is why the exception isn't being raised. I can't see why self.suppress_exceptions would be true for you since you never set it, but try:
Manually setting suppress_exceptions using client.suppress_exceptions = False. This shouldn't be necessary since that appears to be the default, but it's worth a try.
Checking the log that it apparently maintains. You'll need to refer to the docs though on how to do that, since I've never touched this library before.

How to catch custom exception from signal handler in asyncio?

I'm having problems catching a custom exception when thrown from a signal handler callback when using asyncio.
If I throw ShutdownApp from within do_io() below, I am able to properly catch it in run_app(). However, when the exception is raised from handle_sig(), I can't seem to catch it.
Minimal, Reproducible Example Tested using Python 3.8.5:
import asyncio
from functools import partial
import os
import signal
from signal import Signals
class ShutdownApp(BaseException):
pass
os.environ["PYTHONASYNCIODEBUG"] = "1"
class App:
def __init__(self):
self.loop = asyncio.get_event_loop()
def _add_signal_handler(self, signal, handler):
self.loop.add_signal_handler(signal, handler, signal)
def setup_signals(self) -> None:
self._add_signal_handler(signal.SIGINT, self.handle_sig)
def handle_sig(self, signum):
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising error for exit")
raise ShutdownApp("Exiting")
async def do_io(self):
print("io start. Press Ctrl+C now.")
await asyncio.sleep(5)
print("io end")
def run_app(self):
print("Starting Program")
try:
self.loop.run_until_complete(self.do_io())
except ShutdownApp as e:
print("ShutdownApp caught:", e)
# TODO: do other shutdown related items
except:
print("Other error")
finally:
self.loop.close()
if __name__ == "__main__":
my_app = App()
my_app.setup_signals()
my_app.run_app()
print("Finished")
The output after pressing CTRL+C (for SIGINT) with asyncio debug mode:
(env_aiohttp) anav#anav-pc:~/Downloads/test$ python test_asyncio_signal.py
Starting Program
io start. Press Ctrl+C now.
^C
pid: 20359, Received signal: SIGINT, raising error for exit
Exception in callback App.handle_sig(<Signals.SIGINT: 2>)
handle: <Handle App.handle_sig(<Signals.SIGINT: 2>) created at /home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/unix_events.py:99>
source_traceback: Object created at (most recent call last):
File "test_asyncio_signal.py", line 50, in <module>
my_app.setup_signals()
File "test_asyncio_signal.py", line 25, in setup_signals
self._add_signal_handler(signal.SIGINT, self.handle_sig)
File "test_asyncio_signal.py", line 22, in _add_signal_handler
self.loop.add_signal_handler(signal, handler, signal)
File "/home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/unix_events.py", line 99, in add_signal_handler
handle = events.Handle(callback, args, self, None)
Traceback (most recent call last):
File "/home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "test_asyncio_signal.py", line 31, in handle_sig
raise ShutdownApp("Exiting")
ShutdownApp: Exiting
io end
Finished
Expected output:
Starting Program
io start. Press Ctrl+C now.
^C
pid: 20359, Received signal: SIGINT, raising error for exit
ShutdownApp caught: Exiting
io end
Finished
Is it possible to raise a custom exception from a signal handler in asyncio? If so, how do I properly catch/except it?

handle_sig is a callback, so it runs directly off the event loop and its exceptions are just reported to the user via a global hook. If you want the exception raised there to be caught elsewhere in the program, you need to use a future to transfer the exception from handle_sig to where you want it noticed.
To catch the exception at top-level, you probably want to introduce another method, let's call it async_main(), that waits for either self.do_io() or the previously-created future to complete:
def __init__(self):
self.loop = asyncio.get_event_loop()
self.done_future = self.loop.create_future()
async def async_main(self):
# wait for do_io or done_future, whatever happens first
io_task = asyncio.create_task(self.do_io())
await asyncio.wait([self.done_future, io_task],
return_when=asyncio.FIRST_COMPLETED)
if self.done_future.done():
io_task.cancel()
await self.done_future # propagate the exception, if raised
else:
self.done_future.cancel()
To raise the exception from inside handle_sig, you just need to set the exception on the future object:
def handle_sig(self, signum):
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising error for exit")
self.done_future.set_exception(ShutdownApp("Exiting"))
Finally, you modify run_app to pass self.async_main() to run_until_complete, and you're all set:
$ python3 x.py
Starting Program
io start. Press Ctrl+C now.
^C
pid: 2069230, Received signal: SIGINT, raising error for exit
ShutdownApp caught: Exiting
Finished
In closing, note that reliably catching keyboard interrupts is a notoriously tricky undertaking and the above code might not cover all the corner cases.

If I throw ShutdownApp from within do_io() below, I am able to properly catch it in run_app(). However, when the exception is raised from handle_sig(), I can't seem to catch it.
Response to the query given above
Custom Exception Implementation:
class RecipeNotValidError(Exception):
def __init__(self):
self.message = "Your recipe is not valid"
try:
raise RecipeNotValidError
except RecipeNotValidError as e:
print(e.message)
In the method handle_sig, add try and except block. Also, you can customize messages in custom exceptions.
def handle_sig(self, signum):
try:
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising
error for exit")
raise ShutdownApp("Exiting")
except ShutdownApp as e:
print(e.message)
In response to your second query :
Is it possible to raise a custom exception from a signal handler in asyncio? If so, how do I properly catch/except it?
In-built error handling in Asyncio. For more documentation visit https://docs.python.org/3/library/asyncio-exceptions.html
import asyncio
async def f():
try:
while True: await asyncio.sleep(0)
except asyncio.CancelledError:
print('I was cancelled!')
else:
return 111

Catching exception in asyncio stream server

Given the following program:
1 import asyncio
2 async def run():
3 try:
4 server = await asyncio.start_server(on_connected, '127.0.0.1', 15500)
5
6 async with server:
7 await server.serve_forever()
8 except:
9 print("exception!")
10
11 async def on_connected(reader, writer):
12 while True:
13 data = await reader.readline()
14 print(1 / 0)
15
16 asyncio.run(run())
When I run it, and use nc to connect to it & send data, it will raise an exception in line 14. However I am unable to handle it. Instead, I will get an exception printed and the program will hang.
nc test:
$ nc localhost 15500
test
program output:
$ python3 serv
Task exception was never retrieved
future: <Task finished coro=<on_connected() done, defined at serv:14> exception=ZeroDivisionError('division by zero')>
Traceback (most recent call last):
File "serv", line 14, in on_connected
print(1 / 0)
ZeroDivisionError: division by zero
While I of course could just add a try/except around line 14, I want to implement a general exception handling which shall handle all errors which can ever occur inside of on_connected.
How can I do this?

Asyncio provides a method to set a general exception handler
Let's apply it to this example:
Let's just define our loop exception handler: handle_exception
And inside this method let's manage exceptions. Next code just customize logging messages.
import asyncio
import logging
log = logging.getLogger(__name__)
async def run():
loop = asyncio.get_running_loop()
loop.set_exception_handler(handle_exception)
server = await asyncio.start_server(on_connected, '127.0.0.1', 5062)
async with server:
await server.serve_forever()
async def on_connected(reader, writer):
while True:
data = await reader.readline()
raise print(1 / 0)
def handle_exception(loop, context):
# context["message"] will always be there; but context["exception"] may not
msg = context.get("exception", context["message"])
if name := context.get("future").get_coro().__name__ == "on_connected":
if type(msg) == ZeroDivisionError:
log.error(f"Caught ZeroDivisionError from on_connected: {msg}")
return
log.info(f"Caught another minor exception from on_connected: {msg}")
else:
log.error(f"Caught exception from {name}: {msg}")
Output on this case is the following:
Caught ZeroDivisionError from on_connected: division by zero
Reference that I have used to look for further information:
https://www.roguelynn.com/words/asyncio-exception-handling/
https://github.com/econchick/mayhem/blob/master/part-3/mayhem_2.py
Please, let me know if this answer your question.

When developing server code, bug causing an exception often occurs and then the server process just hangs and must be killed, which is super anoying.
I wanted it to exit cleanly as soon as one exception is raised in the protocol code. To do so, I created a handler class that closes the server and registered it as the exception handler. The call to await server.serve_forever() will throw CancelledError, so you may want to catch it to exit cleanly.
import asyncio
class ExcHandler:
def __init__(self, server):
self._server = server
def __call__(self, loop, context):
print("exception occured, closing server")
loop.default_exception_handler(context)
self._server.close()
class EchoServerProtocol(asyncio.Protocol):
def connection_made(self, transport):
raise RuntimeError("oups")
async def main():
loop = asyncio.get_running_loop()
server = await loop.create_server(
lambda: EchoServerProtocol(),
'127.0.0.1', 8888)
loop.set_exception_handler(ExcHandler(server))
async with server:
try:
await server.serve_forever()
except asyncio.exceptions.CancelledError:
print("server cancelled")
asyncio.run(main())
Apparently, the server catching exceptions is by design (See Python issue #42526). The rationale behind that is the server keeps processing other requests, even when one of them crash. I guess this is suitable for production where the show must go on even in case of glitches.

Combining 2 asyncio based code segments

I'm using the Autobahn asyncio system (to talk the Websocket WAMP protocol), which works fine and I can handle incoming RPC calls and pubsub.
My problem is I now have to connect TCP sockets and send information over these sockets as soon as an RPC call comes in through the Autobahn part.
The autobahn part works fine like this :
from autobahn.asyncio.component import Component, run
from asyncio import sleep
from autobahn.asyncio.wamp import ApplicationSession, ApplicationRunner
#comp.on_join
async def joined(session, details):
print("Connected to websocket")
def on_message(msg):
msg = json.loads(msg)
print(msg)
def some_rpc(with_data):
print("Doing something with the data")
return json.dumps({'status': 'OK'})
try:
session.subscribe(on_message, u'some_pubsub_topic')
session.register(some_rpc, u'some_rpc_call')
print("RPC and Pubsub initialized")
except Exception as e:
print("could not subscribe to topic: {0}".format(e))
if __name__ == "__main__":
run([comp])
However now I need to be able to connect to multiple regular TCP sockets :
class SocketClient(asyncio.Protocol):
def __init__(self, loop):
self.data = b''
self.loop = loop
def connection_made(self, transport):
self.transport = transport
print('connected')
def data_received(self, data):
print('Data received: {!r}'.format(data.decode()))
def send(self, data):
self.transport.write(data)
def connection_lost(self, exc):
print('The server closed the connection')
print('Stop the event loop')
self.loop.stop()
loop = asyncio.get_event_loop()
c=loop.create_connection(lambda: SocketClient(loop),
'192.168.0.219', 6773)
loop.run_until_complete(c)
loop.run_forever()
loop.close()
The problem is that, when I combine both and do this :
def some_rpc(with_data):
c.send('test')
return json.dumps({'status': 'OK'})
It barks at me and tells me :
StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"/usr/lib/python3.5/site-packages/autobahn/wamp/websocket.py", line
95, in onMessage
self._session.onMessage(msg) File "/usr/lib/python3.5/site-packages/autobahn/wamp/protocol.py", line
894, in onMessage
on_reply = txaio.as_future(endpoint.fn, *invoke_args, **invoke_kwargs) File "/usr/lib/python3.5/site-packages/txaio/aio.py", line 400, in
as_future
return create_future_error(create_failure()) File "/usr/lib/python3.5/site-packages/txaio/aio.py", line 393, in
create_future_error
reject(f, error) File "/usr/lib/python3.5/site-packages/txaio/aio.py", line 462, in reject
future.set_exception(error.value) File "/usr/lib64/python3.5/asyncio/futures.py", line 365, in set_exception
raise TypeError("StopIteration interacts badly with generators " TypeError: StopIteration interacts badly with generators and cannot be
raised into a Future
Does anyone have any idea on how to call the send function from within the RPC call function ?

In this code:
c=loop.create_connection(lambda: SocketClient(loop),
'192.168.0.219', 6773)
# [...]
def some_rpc(with_data):
c.send('test')
return json.dumps({'status': 'OK'})
create_connection is a coroutine function, so c contains a coroutine object. Such object does have a send method, but one that is entirely unrelated to sending things over the network. After calling create_connection, you probably want to get the resulting transport with something like:
transport, ignore = loop.run_until_complete(c)
and then use transport.write(), not c.send().

How can I close a DatagramTransport as soon one datagram is sent?

I'm trying to close the transport right after sending the UDP packet and I'm getting an Exception in callback _SelectorDatagramTransport._read_ready()
import asyncio
class MyProtocol:
def __init__(self, message, loop):
self.message = message
self.loop = loop
self.transport = None
def connection_made(self, transport):
self.transport = transport
print("Send:", self.message)
self.transport.sendto(self.message.encode())
self.transport.close() # <----------
def error_received(self, exc):
print('Error received', exc)
def connection_lost(self, exc):
print("Socket closed, stop the event loop")
self.loop.stop()
loop = asyncio.get_event_loop()
message = "hello"
connect = loop.create_datagram_endpoint(lambda: MyProtocol(message, loop), remote_addr=('127.0.0.1', 2222))
transport, protocol = loop.run_until_complete(connect)
loop.run_forever()
The full stack trace that I get is while running the snippet above in CPython 3.5.1 is:
Socket closed, stop the event loop
Exception in callback _SelectorDatagramTransport._read_ready()
handle: <Handle _SelectorDatagramTransport._read_ready()>
Traceback (most recent call last):
File "/home/ecerulm/.pyenv/versions/3.5.1/lib/python3.5/asyncio/selector_events.py", line 1002, in _read_ready
data, addr = self._sock.recvfrom(self.max_size)
AttributeError: 'NoneType' object has no attribute 'recvfrom'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ecerulm/.pyenv/versions/3.5.1/lib/python3.5/asyncio/events.py", line 125, in _run
self._callback(*self._args)
File "/home/ecerulm/.pyenv/versions/3.5.1/lib/python3.5/asyncio/selector_events.py", line 1008, in _read_ready
self._fatal_error(exc, 'Fatal read error on datagram transport')
File "/home/ecerulm/.pyenv/versions/3.5.1/lib/python3.5/asyncio/selector_events.py", line 587, in _fatal_error
self._loop.call_exception_handler({
AttributeError: 'NoneType' object has no attribute 'call_exception_handler'
I believe the exception is only generated if the UDP packet is actively refused, with an ICMP Destination Unreachable (which I'm not interested in).
So the question is what is the right way of doing this. I'm not interested in this connection anymore after sending so I want to get rid of the transport as soon as possible. The documentation for DatagramTransport.sendto() just says that the methods doesn't block. But how do I know when the sending is completed? (And by complete I mean when is handed over to the OS, not delivered to the remote).
Is there any other asyncio coroutine to send an UDP packet asynchronously and simple await (maybe even skipping the whole create_datagram_endpoint) ?

Is there any other asyncio coroutine to send an UDP packet asynchronously and simple await?
I would, base on DatagramTransport source, wrap it in Future to be yieldable/awaitable. It will raise exception on error and return True on success. The example PoC code:
import asyncio
import socket
class UDPClient():
def __init__(self, host, port, loop=None):
self._loop = asyncio.get_event_loop() if loop is None else loop
self._sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self._sock.setblocking(False)
self._addr = (host, port)
self._future = None
self._data = None
def sendto(self, data):
self._future = asyncio.Future(loop=self._loop)
self.data = data if isinstance(data, bytes) else str(data).encode('utf-8')
loop.add_writer(self._sock.fileno(), self._sendto)
return self._future
def _sendto(self):
try:
self._sock.sendto(self.data, self._addr)
except (BlockingIOError, InterruptedError):
return
except OSError as exc:
self.abort(exc)
except Exception as exc:
self.abort(exc)
else:
self.close()
self._future.set_result(True)
def abort(self, exc):
self.close()
self._future.set_exception(exc)
def close(self):
self._loop.remove_writer(self._sock.fileno())
self._sock.close()
Than simple example would look like:
#asyncio.coroutine
def test():
yield from UDPClient('127.0.0.1', 1234).sendto('ok')
# or 3.5+ syntax
# async def test():
# await UDPClient('127.0.0.1', 1234).sendto('ok')
loop = asyncio.get_event_loop()
loop.run_until_complete(test())