Can not catch exception in threaded program - python

The Python program below starts one thread and then continues to perform actions in the main thread. I wrap the whole main thread in a try-except block so I can tear down all running threads if an exception occured.
When I run the script using Python 2.7.5 and invoke a KeyboardInterrupt at a random point during the programs execution, the exception is triggered but not catched. The program continues to run.
$ python test.py
Running server ...
Searching for servers ...
^CTraceback (most recent call last):
File "test.py", line 50, in <module>
main()
File "test.py", line 40, in main
app_main()
File "test.py", line 35, in app_main
searchservers()
File "test.py", line 26, in searchservers
time.sleep(0.0005)
KeyboardInterrupt
I miss a line in the output which is printed in main() when an exception occured.
Code
import time
import threading
thread_pool = []
running = False
def stop():
global running
running = False
def runserver():
print "Running server ..."
global running
running = True
while running:
time.sleep(0.07)
def searchservers():
print "Searching for servers ..."
for i in xrange(256):
for j in xrange(256):
time.sleep(0.0005)
def app_main():
server = threading.Thread(target=runserver)
thread_pool.append(server)
server.start()
time.sleep(0.1)
searchservers()
stop()
def main():
try:
app_main()
except Exception as exc:
stop()
print "%s occured, joining all threads..." % exc.__class__.__name__
for thread in thread_pool:
thread.join()
raise exc
if __name__ == "__main__":
main()
Why is the KeyboardInterrupt not catched? What is the proper way of catching exceptions in a threaded program and tear down the complete process?

KeyboardInterrupt is a special exception; like MemoryError, GeneratorExit and SystemExit, it does not derive from the base Exception class.
Catching just Exception is thus not enough; you'd normally catch it explicitly:
except (Exception, KeyboardInterrupt) as exc:
However, you are also trying to catch exceptions in threads; threads have their own separate stack; you cannot just go and catch exceptions thrown in those stack in your main thread. You'd have to catch exceptions in that thread:
def runserver():
print "Running server ..."
global running
running = True
try:
while running:
time.sleep(0.07)
except (Exception, KeyboardInterrupt) as exc:
print "Error in the runserver thread"
To handle this in a generic way and 'pass' exceptions to the main thread, you'd need some sort of inter-thread communication. See Catch a thread's exception in the caller thread in Python for a full solution to that.

Related

Why a traceback is not thrown when accessing an element in a dict that does not exist? [duplicate]

This question already has answers here:
Paho MQTT Python Client: No exceptions thrown, just stops
(3 answers)
Closed 2 years ago.
I have a place in my code where I made a mistake in the name of the key of a dict. It took some time to understand why the code was not running past that place because a traceback was not thrown.
The code is below, I put it for completeness, highlighting with →→→ the place where the issue is:
class Alert:
lock = threading.Lock()
sent_alerts = {}
#staticmethod
def start_alert_listener():
# load existing alerts to keep persistancy
try:
with open("sent_alerts.json") as f:
json.load(f)
except FileNotFoundError:
# there is no file, never mind - it will be created at some point
pass
# start the listener
log.info("starting alert listener")
client = paho.mqtt.client.Client()
client.on_connect = Alert.mqtt_connection_alert
client.on_message = Alert.alert
client.connect("mqtt.XXXX", 1883, 60)
client.loop_forever()
#staticmethod
def mqtt_connection_alert(client, userdata, flags, rc):
if rc:
log.critical(f"error connecting to MQTT: {rc}")
sys.exit()
topic = "monitor/+/state"
client.subscribe(topic)
log.info(f"subscribed alert to {topic}")
#staticmethod
def alert(client, userdata, msg):
event = json.loads(msg.payload)
log.debug(f"received alert: {event}")
→→→ if event['ok']:
# remove existing sent flag, not thread safe!
with Alert.lock:
Alert.sent_alerts.pop(msg['id'], None)
return
(...)
The log coming from the line just above is
2021-01-14 22:03:02,617 [monitor] DEBUG received alert: {'full_target_name': 'ThisAlwaysFails → a failure', 'isOK': False, 'why': 'explicit fail', 'extra': None, 'id': '6507a61c9688199a34cb006b354c8433', 'last': '2021-01-14T22:03:02.612912+01:00', 'last_ko': '2021-01-14T22:03:02.612912+01:00'}
This is the dict in which I am trying to erroneously access ok, which should raise an exception and a traceback. But nothing happens. The code does not further than that as if the error was silently discarded (and the method silently fails).
I tried to put a raise Exception("wazaa") between the log.debug() and the if - same thing, the method fails at that point but an exception is not raised.
I am at loss about the reason where an exception could not be visible though a traceback?
The alert() method is called in a separate thread, if this matters. For completeness I tried the follwong code just to make sure threading does not interfere but no (I do not see a reason why it should)
import threading
class Hello:
#staticmethod
def a():
raise Exception("I am an exception")
threading.Thread(target=Hello.a).start()
outputs
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python38\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Python38\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:/Users/yop/AppData/Roaming/JetBrains/PyCharm2020.3/scratches/scratch_1.py", line 7, in a
raise Exception("I am an exception")
Exception: I am an exception
It appears to call your callback within a try, then logs that error:
try:
self.on_message(self, self._userdata, message)
except Exception as err:
self._easy_log(
MQTT_LOG_ERR, 'Caught exception in on_message: %s', err)
if not self.suppress_exceptions:
raise
What I can't explain however is why the exception isn't being raised. I can't see why self.suppress_exceptions would be true for you since you never set it, but try:
Manually setting suppress_exceptions using client.suppress_exceptions = False. This shouldn't be necessary since that appears to be the default, but it's worth a try.
Checking the log that it apparently maintains. You'll need to refer to the docs though on how to do that, since I've never touched this library before.

How to catch custom exception from signal handler in asyncio?

I'm having problems catching a custom exception when thrown from a signal handler callback when using asyncio.
If I throw ShutdownApp from within do_io() below, I am able to properly catch it in run_app(). However, when the exception is raised from handle_sig(), I can't seem to catch it.
Minimal, Reproducible Example Tested using Python 3.8.5:
import asyncio
from functools import partial
import os
import signal
from signal import Signals
class ShutdownApp(BaseException):
pass
os.environ["PYTHONASYNCIODEBUG"] = "1"
class App:
def __init__(self):
self.loop = asyncio.get_event_loop()
def _add_signal_handler(self, signal, handler):
self.loop.add_signal_handler(signal, handler, signal)
def setup_signals(self) -> None:
self._add_signal_handler(signal.SIGINT, self.handle_sig)
def handle_sig(self, signum):
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising error for exit")
raise ShutdownApp("Exiting")
async def do_io(self):
print("io start. Press Ctrl+C now.")
await asyncio.sleep(5)
print("io end")
def run_app(self):
print("Starting Program")
try:
self.loop.run_until_complete(self.do_io())
except ShutdownApp as e:
print("ShutdownApp caught:", e)
# TODO: do other shutdown related items
except:
print("Other error")
finally:
self.loop.close()
if __name__ == "__main__":
my_app = App()
my_app.setup_signals()
my_app.run_app()
print("Finished")
The output after pressing CTRL+C (for SIGINT) with asyncio debug mode:
(env_aiohttp) anav#anav-pc:~/Downloads/test$ python test_asyncio_signal.py
Starting Program
io start. Press Ctrl+C now.
^C
pid: 20359, Received signal: SIGINT, raising error for exit
Exception in callback App.handle_sig(<Signals.SIGINT: 2>)
handle: <Handle App.handle_sig(<Signals.SIGINT: 2>) created at /home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/unix_events.py:99>
source_traceback: Object created at (most recent call last):
File "test_asyncio_signal.py", line 50, in <module>
my_app.setup_signals()
File "test_asyncio_signal.py", line 25, in setup_signals
self._add_signal_handler(signal.SIGINT, self.handle_sig)
File "test_asyncio_signal.py", line 22, in _add_signal_handler
self.loop.add_signal_handler(signal, handler, signal)
File "/home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/unix_events.py", line 99, in add_signal_handler
handle = events.Handle(callback, args, self, None)
Traceback (most recent call last):
File "/home/anav/miniconda3/envs/env_aiohttp/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "test_asyncio_signal.py", line 31, in handle_sig
raise ShutdownApp("Exiting")
ShutdownApp: Exiting
io end
Finished
Expected output:
Starting Program
io start. Press Ctrl+C now.
^C
pid: 20359, Received signal: SIGINT, raising error for exit
ShutdownApp caught: Exiting
io end
Finished
Is it possible to raise a custom exception from a signal handler in asyncio? If so, how do I properly catch/except it?
handle_sig is a callback, so it runs directly off the event loop and its exceptions are just reported to the user via a global hook. If you want the exception raised there to be caught elsewhere in the program, you need to use a future to transfer the exception from handle_sig to where you want it noticed.
To catch the exception at top-level, you probably want to introduce another method, let's call it async_main(), that waits for either self.do_io() or the previously-created future to complete:
def __init__(self):
self.loop = asyncio.get_event_loop()
self.done_future = self.loop.create_future()
async def async_main(self):
# wait for do_io or done_future, whatever happens first
io_task = asyncio.create_task(self.do_io())
await asyncio.wait([self.done_future, io_task],
return_when=asyncio.FIRST_COMPLETED)
if self.done_future.done():
io_task.cancel()
await self.done_future # propagate the exception, if raised
else:
self.done_future.cancel()
To raise the exception from inside handle_sig, you just need to set the exception on the future object:
def handle_sig(self, signum):
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising error for exit")
self.done_future.set_exception(ShutdownApp("Exiting"))
Finally, you modify run_app to pass self.async_main() to run_until_complete, and you're all set:
$ python3 x.py
Starting Program
io start. Press Ctrl+C now.
^C
pid: 2069230, Received signal: SIGINT, raising error for exit
ShutdownApp caught: Exiting
Finished
In closing, note that reliably catching keyboard interrupts is a notoriously tricky undertaking and the above code might not cover all the corner cases.
If I throw ShutdownApp from within do_io() below, I am able to properly catch it in run_app(). However, when the exception is raised from handle_sig(), I can't seem to catch it.
Response to the query given above
Custom Exception Implementation:
class RecipeNotValidError(Exception):
def __init__(self):
self.message = "Your recipe is not valid"
try:
raise RecipeNotValidError
except RecipeNotValidError as e:
print(e.message)
In the method handle_sig, add try and except block. Also, you can customize messages in custom exceptions.
def handle_sig(self, signum):
try:
print(f"\npid: {os.getpid()}, Received signal: {Signals(signum).name}, raising
error for exit")
raise ShutdownApp("Exiting")
except ShutdownApp as e:
print(e.message)
In response to your second query :
Is it possible to raise a custom exception from a signal handler in asyncio? If so, how do I properly catch/except it?
In-built error handling in Asyncio. For more documentation visit https://docs.python.org/3/library/asyncio-exceptions.html
import asyncio
async def f():
try:
while True: await asyncio.sleep(0)
except asyncio.CancelledError:
print('I was cancelled!')
else:
return 111

How to catch concurrent.futures._base.TimeoutError

I'm trying to catch exception thrown inside a run_until_complete but whatever I try, I can't seems to catch them properly.
Here's my latest attempt (note, I'm using Pypputeer, a fork of Puppeteer in Python, that uses asyncio):
import asyncio
from pyppeteer.launcher import launch
async def test(instance):
page = await instance.newPage()
await page.goto('http://www.google.com', {'waitUntil': 'load', 'timeout': 1})
await page.pdf({'path': 'example.pdf'})
async def test2():
instance = launch(headless=True)
try:
task = asyncio.ensure_future(test(instance))
print(task)
await task
except:
print("Caught!")
instance.close()
def __main__():
try:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(test2())
except:
print("ERROR")
return 'ok'
The issue I'm having with this code, is two fold:
if I do asyncio.get_event_loop instead, I get the following error:
There is no current event loop in thread 'Thread-1'.
If I change the timeout to a decent value, I get the following error (at loop.run_until_complete(test2())):
RuntimeError: This event loop is already running
If I set the timeout to 1 (to force the error), I get the exception indicated below, shown in the console, and the text "ERROR" is shown. (but not caught).
Here's the stacktrace:
Exception in callback NavigatorWatcher.waitForNavigation.<locals>.watchdog_cb(<Task finishe...> result=None>) at /home/user/www/project/api/env/lib/python3.6/site-packages/pyppeteer/navigator_watcher.py:49
handle: <Handle NavigatorWatcher.waitForNavigation.<locals>.watchdog_cb(<Task finishe...> result=None>) at /home/user/www/project/api/env/lib/python3.6/site-packages/pyppeteer/navigator_watcher.py:49>
Traceback (most recent call last):
File "/usr/lib64/python3.6/asyncio/events.py", line 145, in _run
self._callback(*self._args)
File "/home/user/www/project/api/env/lib/python3.6/site-packages/pyppeteer/navigator_watcher.py", line 52, in watchdog_cb
self._timeout)
File "/home/user/www/project/api/env/lib/python3.6/site-packages/pyppeteer/navigator_watcher.py", line 40, in _raise_error
raise error
concurrent.futures._base.TimeoutError: Navigation Timeout Exceeded: 1 ms exceeded
So, TLDR, how can I catch exceptions thrown inside a run_until_complete call of asyncio?
Thank you so much!
You can't catch this error because it doesn't happen somewhere before event loop finished it's work:
loop.run_until_complete(test2())
print('TEST !!!') # You will see this line, because there was no exception before
But if you look at whole traceback you will see:
Error in atexit._run_exitfuncs:
It means exception happens inside one of the functions Pypputeer registered using atexit handler. You should search how to catch exception there, but I'm not sure if it's possible.
If an exception is raised during execution of the exit handlers, a
traceback is printed (unless SystemExit is raised) and the exception
information is saved. After all exit handlers have had a chance to run
the last exception to be raised is re-raised.
Not related, but never do such thing.
except:

python gevent: unexpected output in KeyboardInterrupt

Running this code
import gevent
def f():
while True:
gevent.sleep(1)
if __name__ == '__main__':
tasks = (gevent.spawn(f),)
try:
gevent.wait(tasks)
except KeyboardInterrupt:
print("KeyboardInterrupt trapped")
and then pressing a Ctrl-C, give me this output:
$ python receiver.py
^CKeyboardInterrupt
Tue Aug 8 00:56:04 2017
KeyboardInterrupt trapped
Why?
It seems someone is writing the exit time on output.
How can I prevent that KeyboardInterrupt in the first line and the date in the second?
Those messages are printed by the gevent Hub, which is intercepting the KeyboardInterrupt being raised. Usually you would see a traceback instead of just KeyboardInterrupt and the current date, but because the Hub is special, you get that output.
You have two ways to solve this issue:
Mark KeyboardInterrupt as a non-error:
gevent.get_hub().NOT_ERROR += (KeyboardInterrupt,)
With this trick, the Hub won't print any line when KeyboardInterrupt is caught. This might seem a hack, but it's a short and effective way to stop output pollution.
Register a signal handler for SIGINT:
def handler(signum, frame):
print('SIGINT trapped')
sys.exit(0)
signal.signal(signal.SIGINT, handler)
The default signal handler for SIGINT will raise KeyboardInterrupt, but if you define your own signal handler, you can prevent it and run your cleanup code.
It is important that you exit with an exception from your handler function, otherwise your call to gevent.wait() won't be stopped. The only two exceptions that you can use are SystemExit and GreenletExit (those are the two default exceptions in the NOT_ERROR list above): any other exception will cause gevent to print something on standard error.

interrupted system call with processing queue

We suddenly started see "Interrupted system call" on Queue operations like this:
Exception in thread Thread-2:
Traceback (most recent call last):
[ . . . ]
result = self.pager.results.get(True, self.WAIT_SECONDS)
File "/usr/lib/python2.5/site-packages/processing-0.52-py2.5-linux-x86_64.egg/processing/queue.py", line 128, in get
if not self._poll(block and (deadline-time.time()) or 0.0):
IOError: [Errno 4] Interrupted system call
This is a Fedora 10 / Python 2.5 machine that recently had a security update. Prior to that our software had run for about a year without incident, now it is crashing daily.
Is it correct/necessary to catch this exception and retry the Queue operation?
We don't have any signal handlers that we set, but this is a Tkinter app maybe it sets some. Is it safe to clear the SIGINT handler, would that solve the problem? Thanks.
Based on this thread on comp.lang.python and this reply from Dan Stromberg I wrote a RetryQueue which is a drop-in replacement for Queue and which does the job for us:
from multiprocessing.queues import Queue
import errno
def retry_on_eintr(function, *args, **kw):
while True:
try:
return function(*args, **kw)
except IOError, e:
if e.errno == errno.EINTR:
continue
else:
raise
class RetryQueue(Queue):
"""Queue which will retry if interrupted with EINTR."""
def get(self, block=True, timeout=None):
return retry_on_eintr(Queue.get, self, block, timeout)
Finally this got fixed in python itself, so the other solution is to update to a newer python:
http://bugs.python.org/issue17097

Categories

Resources