Develop Reference

How to catch custom exception from signal handler in asyncio?

I'm having problems catching a custom exception when thrown from a signal handler callback when using asyncio.
If I throw ShutdownApp from within do_io() below, I am able to properly catch it in run_app(). However, when the exception is raised from handle_sig(), I can't seem to catch it.
Minimal, Reproducible Example Tested using Python 3.8.5:
import asyncio
from functools import partial
import os
import signal
from signal import Signals
class ShutdownApp(BaseException):
pass
os.environ["PYTHONASYNCIODEBUG"] = "1"
class App:
def __init__(self):
self.loop = asyncio.get_event_loop()
def _add_signal_handler(self, signal, handler):
self.loop.add_signal_handler(signal, handler, signal)
def setup_signals(self) -> None:
self._add_signal_handler(signal.SIGINT, self.handle_sig)
def handle_sig(self, signum):
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising error for exit")
raise ShutdownApp("Exiting")
async def do_io(self):
print("io start. Press Ctrl+C now.")
await asyncio.sleep(5)
print("io end")
def run_app(self):
print("Starting Program")
try:
self.loop.run_until_complete(self.do_io())
except ShutdownApp as e:
print("ShutdownApp caught:", e)
# TODO: do other shutdown related items
except:
print("Other error")
finally:
self.loop.close()
if __name__ == "__main__":
my_app = App()
my_app.setup_signals()
my_app.run_app()
print("Finished")
The output after pressing CTRL+C (for SIGINT) with asyncio debug mode:
(env_aiohttp) anav#anav-pc:~/Downloads/test$ python test_asyncio_signal.py
Starting Program
io start. Press Ctrl+C now.
^C
pid: 20359, Received signal: SIGINT, raising error for exit
Exception in callback App.handle_sig(<Signals.SIGINT: 2>)
handle: <Handle App.handle_sig(<Signals.SIGINT: 2>) created at /home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/unix_events.py:99>
source_traceback: Object created at (most recent call last):
File "test_asyncio_signal.py", line 50, in <module>
my_app.setup_signals()
File "test_asyncio_signal.py", line 25, in setup_signals
self._add_signal_handler(signal.SIGINT, self.handle_sig)
File "test_asyncio_signal.py", line 22, in _add_signal_handler
self.loop.add_signal_handler(signal, handler, signal)
File "/home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/unix_events.py", line 99, in add_signal_handler
handle = events.Handle(callback, args, self, None)
Traceback (most recent call last):
File "/home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "test_asyncio_signal.py", line 31, in handle_sig
raise ShutdownApp("Exiting")
ShutdownApp: Exiting
io end
Finished
Expected output:
Starting Program
io start. Press Ctrl+C now.
^C
pid: 20359, Received signal: SIGINT, raising error for exit
ShutdownApp caught: Exiting
io end
Finished
Is it possible to raise a custom exception from a signal handler in asyncio? If so, how do I properly catch/except it?

handle_sig is a callback, so it runs directly off the event loop and its exceptions are just reported to the user via a global hook. If you want the exception raised there to be caught elsewhere in the program, you need to use a future to transfer the exception from handle_sig to where you want it noticed.
To catch the exception at top-level, you probably want to introduce another method, let's call it async_main(), that waits for either self.do_io() or the previously-created future to complete:
def __init__(self):
self.loop = asyncio.get_event_loop()
self.done_future = self.loop.create_future()
async def async_main(self):
# wait for do_io or done_future, whatever happens first
io_task = asyncio.create_task(self.do_io())
await asyncio.wait([self.done_future, io_task],
return_when=asyncio.FIRST_COMPLETED)
if self.done_future.done():
io_task.cancel()
await self.done_future # propagate the exception, if raised
else:
self.done_future.cancel()
To raise the exception from inside handle_sig, you just need to set the exception on the future object:
def handle_sig(self, signum):
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising error for exit")
self.done_future.set_exception(ShutdownApp("Exiting"))
Finally, you modify run_app to pass self.async_main() to run_until_complete, and you're all set:
$ python3 x.py
Starting Program
io start. Press Ctrl+C now.
^C
pid: 2069230, Received signal: SIGINT, raising error for exit
ShutdownApp caught: Exiting
Finished
In closing, note that reliably catching keyboard interrupts is a notoriously tricky undertaking and the above code might not cover all the corner cases.

If I throw ShutdownApp from within do_io() below, I am able to properly catch it in run_app(). However, when the exception is raised from handle_sig(), I can't seem to catch it.
Response to the query given above
Custom Exception Implementation:
class RecipeNotValidError(Exception):
def __init__(self):
self.message = "Your recipe is not valid"
try:
raise RecipeNotValidError
except RecipeNotValidError as e:
print(e.message)
In the method handle_sig, add try and except block. Also, you can customize messages in custom exceptions.
def handle_sig(self, signum):
try:
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising
error for exit")
raise ShutdownApp("Exiting")
except ShutdownApp as e:
print(e.message)
In response to your second query :
Is it possible to raise a custom exception from a signal handler in asyncio? If so, how do I properly catch/except it?
In-built error handling in Asyncio. For more documentation visit https://docs.python.org/3/library/asyncio-exceptions.html
import asyncio
async def f():
try:
while True: await asyncio.sleep(0)
except asyncio.CancelledError:
print('I was cancelled!')
else:
return 111

Raise an exception in parent thread

I have an issue where a method call is blocking and not releasing. Unfortunately, the bug as to why isn't exactly solvable, so the workaround at the moment is to build in a timeout.
I've tried to do this by registering a timer and have it raise an exception to break from the blocked call. However, that raises the exception in the timer thread, not the main thread.
It looks like this right now:
from threading import Timer
def timeoutSocket():
raise InterruptedError
socketDeadlockDetector = Timer(DEADLOCK_TIMEOUT, timeoutSocket)
socketDeadlockDetector.start()
# receive and unpack data
try:
packet = server.receive()
except InterruptedError:
print("Interrupted socket receive, continuing")
continue
socketDeadlockDetector.cancel()
server.receive() is the method that is blocking when it shouldn't. However, when I run this, the socketDeadlockDetector thread interrupts itself, without affecting the original thread.
Is there a way to pass this exception up to the parent?

Timer creates a thread to run the function. It doesn't do you any good to raise an exception because that's not the thread needing interruption. When you hit the timeout, you need to cancel whatever is blocking in the other thread. In this case its a socket, so killing the socket should do.
import struct
def timeoutSocket():
# enable linger with timeout 0 to send RESET on close
server.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 0))
server.close()

Signal module does not raise exception

I am trying to learn python's signal module. Please consider the example below:
def timeoutFn(func, args=(), kwargs={}, timeout_duration=1, default=None):
import signal
class TimeoutError(Exception):
pass
def handler(signum, frame):
print "Trying to raise exception"
raise TimeoutError
# set the timeout handler
signal.signal(signal.SIGALRM, handler)
signal.alarm(timeout_duration)
try:
result = func(*args, **kwargs)
except TimeoutError as exc:
result = default
finally:
signal.alarm(0)
return result
and,
import time
def foo():
for i in range(10):
time.sleep(0.5)
print "Sleeping"
On calling the function timeoutFn(foo) the following gets printed but it does raise the exception.
Shouldn't it raise the TimeoutError? But, all it prints is
Sleeping
Trying to raise exception
and program stops.

The exception has not been raised, because you catch it. Pay attention to the line:
except TimeoutError as exc:
result = default
The block means, that if the exception TimeoutError had been raised, result will be assigned to default (which is None in your example) and the scripts continues further without showing an exception.
Update:
signal.alarm doesn't stop the flow. So the exception will be raised by timeout and if the script will be by that time in a try block, than the exception will be caught. You can see better how does it work if you increase timeout_duration to 3. Than there will be more time to print several 'sleep' messages. It shows, that by the time when the exception is raised, the interpreter had entered the try block.

python gevent: unexpected output in KeyboardInterrupt

Running this code
import gevent
def f():
while True:
gevent.sleep(1)
if __name__ == '__main__':
tasks = (gevent.spawn(f),)
try:
gevent.wait(tasks)
except KeyboardInterrupt:
print("KeyboardInterrupt trapped")
and then pressing a Ctrl-C, give me this output:
$ python receiver.py
^CKeyboardInterrupt
Tue Aug 8 00:56:04 2017
KeyboardInterrupt trapped
Why?
It seems someone is writing the exit time on output.
How can I prevent that KeyboardInterrupt in the first line and the date in the second?

Those messages are printed by the gevent Hub, which is intercepting the KeyboardInterrupt being raised. Usually you would see a traceback instead of just KeyboardInterrupt and the current date, but because the Hub is special, you get that output.
You have two ways to solve this issue:
Mark KeyboardInterrupt as a non-error:
gevent.get_hub().NOT_ERROR += (KeyboardInterrupt,)
With this trick, the Hub won't print any line when KeyboardInterrupt is caught. This might seem a hack, but it's a short and effective way to stop output pollution.
Register a signal handler for SIGINT:
def handler(signum, frame):
print('SIGINT trapped')
sys.exit(0)
signal.signal(signal.SIGINT, handler)
The default signal handler for SIGINT will raise KeyboardInterrupt, but if you define your own signal handler, you can prevent it and run your cleanup code.
It is important that you exit with an exception from your handler function, otherwise your call to gevent.wait() won't be stopped. The only two exceptions that you can use are SystemExit and GreenletExit (those are the two default exceptions in the NOT_ERROR list above): any other exception will cause gevent to print something on standard error.

asyncore.loop doesn't terminate when there are no more connections

I am following some example code to use asyncore here, only having set a timeout value for asyncore.loop as in the following full example:
import smtpd
import asyncore
class CustomSMTPServer(smtpd.SMTPServer):
def process_message(self, peer, mailfrom, rcpttos, data):
print 'Receiving message from:', peer
print 'Message addressed from:', mailfrom
print 'Message addressed to :', rcpttos
print 'Message length :', len(data)
return
server = CustomSMTPServer(('127.0.0.1', 1025), None)
asyncore.loop(timeout = 1)
I have expected that a timeout occurs after 1 second, but this is not the case. The code runs much longer for than one second. What am I missing here?

The timeout argument to asyncore.loop() is the amount of time the select.select call will wait for data. If there is no data before the timeout runs out it will loop and call select.select again.
Same for the channels idea. This does not mean open sockets but means active asyncore.dispatcher or asynchat.async_chat instances. If you want to stop the loop you will have to call the close() method on ALL instances registered.
In your case server.close() will close the instance/channel and remove it from the asyncore loop. If no more channels are active this loop will then terminate itself.

I really do not know if the timeout argument to asyncore.loop() really is meant to timeout the function call asyncore.loop() after the specified time, but here is a receipt to make that function timeout after a specified time (replacing the line with asyncore.loop() in the example code):
import signal
class TimeoutError(Exception): pass
# define the timeout handler
def handler(signum, frame):
raise TimeoutError()
# set the timeout handler and the signal duration
signal.signal(signal.SIGALRM, handler)
signal.alarm(1)
try:
asyncore.loop()
except TimeoutError as exc:
print "timeout"
finally:
signal.alarm(0)

The timeout of asyncore.loop() is the timeout for select().
It is not useful, because when select() timeout, it loops back, see the pseudo code:
while True:
do_something()
select(...)
do_something_else()
If I do simulation with firewall-ed sockets, in my Python 2.7.3 asyncore.loop() timeout 1 minute after no data is received from some socket.
I found very useful to have following method in asyncore.dispatcher "subclass":
def handle_error(self):
raise
In this way I had "proper" exception dump.
Because I do not wanted to have exception, later I changed it to something like:
def handle_error(self):
print "Error downloading %s" % self.host
pass
Now my code works correct, without exception.
I did not found a way to control the timeout.