Develop Reference

Handling Timeout Error(python)

import multiprocessing.pool
import functools
import time
import sys
def timeout(max_timeout):
def timeout_decorator(item):
#functools.wraps(item)
def func_wrapper(*args, **kwargs):
pool = multiprocessing.pool.ThreadPool(processes=1)
async_result = pool.apply_async(item, args, kwargs)
return async_result.get(max_timeout)
return func_wrapper
return timeout_decorator
I got this code from SE. This raises the timeout error from system level.
How could i handle this error and do something else when error pops up
Please Note Am a newbie in python.
Thanks in advance

You can wrap an error, using a try/except :
try:
do_something_that_can_raise_error_X()
except X:
do_something_when_error_X_is_raised()

You need to wrap the code that might raise this exception in "try" and then except the exception that you expect.
from multiprocessing import TimeoutError
try:
# actions that raise TimeoutError
except TimeoutError:
# handle TimeoutError
else:
# (optional) actions for when there is no TimeoutError
finally:
# (optional) actions to perform in any case
See this example in the multiprocessing docs.
Note, that using except without listing precise exception types is allowed syntactically but is not a good idea.
Also, you might want to learn how to handle Exceptions in python, see any python tutorial.

Exception that emails stack trace when raised

I'd like to have an exception that always sends an email when raised. As for now, I was planning to put that code into __init():
import sendmail
import configparser
def MailException(BaseException):
"""
sends mail when initiated
"""
def __init__(self, message, config=None):
# Call the base class constructor with the parameters it needs
BaseException.__init__(message)
if config is None:
config = configparser.RawConfigParser()
config.read('mail.ini')
config = config['email']
text = 'exception stack trace + message'
sendmail.sendMail(text=text, config=config['email'], title='Alert')
I'd explicitly like to have the mail-sending here, instead of in every except-block I create. Hence I wonder how to get the stack trace, the code has to be compatible with Python 2.7.
The only thing I could find was traceback, but that apparently only works within the except: realm - or is there a way to implement it within the Exception class? Any other ideas?

Firstly, note that (per the docs) you shouldn't subclass BaseException.
Secondly, rather than implement that logic in the exception itself you could define a new sys.excepthook (see e.g. Python Global Exception Handling), then you get access to the full traceback:
import sys
class MailException(Exception):
pass
def mailing_except_hook(exctype, value, traceback):
if exctype == MailException:
... # do your thing here
else:
sys.__excepthook__(exctype, value, traceback)
sys.excepthook = mailing_except_hook

Do the same change (try-except) on multiple lines - python

There are ways how programmer can make programming and refactoring easier and more simple, python is very good in this area.
I'm curious whether is there a more elegant way to solve my problem than brute-force writing the same code multiple times again and again.
Situation:
I'm writing a code. There are many equal methods calling with different arguments sequentially.
For example - I have this code:
...
...
my_method(1)
my_method(2)
my_method(3)
my_method(4)
...
my_method(10)
...
So I have this code written, everything works fine but suddenly I find out that I need to make a log file so I have to put try-except on everyone of this methods so the code will look like this:
...
...
try:
my_method(3)
except Exception as e:
print_to_file(log.txt,str(e))
...
...
try:
my_method(8)
except Exception as e:
print_to_file(log.txt,str(e))
...
...
Do I have a better option than changing every my_method(x) calling and putting it into try-except clause? I know that it is a mistake of the programmer who had to think about it at the beginning but these situations happens.
EDIT: According to the answer - the code above is the simple example. In real code there are no int arguments given but dates where there is no logic there so I can't put it into the loop. Assume that the arguments can't be generated.

If you're using the logger supplied by python, you can redirect exception output to the log as opposed to have to put a ton of try blocks everywhere:
import os, sys
import logging
logger = logging.getLogger(__name__)
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
def handle_exception(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logger.error("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = handle_exception
if __name__ == "__main__":
raise RuntimeError("Test unhandled")
Now is an exception is thrown, you won't need a try block, it will be written to the log regardless
ref

You can take advantage of the fact that a function, in python, is totally an object, and write a function that takes in another function, runs it, and logs any exceptions
def sloppyRun(func, *args, **kwargs):
"""Runs a function, catching all exceptions
and writing them to a log file."""
try:
return func(*args, **kwargs) #running function here
except:
logging.exception(func.__name__ + str(args) + str(kwargs))
#incidentally, the logging module is wonderful. I'd recommend using it.
#It'll even write the traceback to a file.
And then you can write something like
sloppyRun(my_method, 8) #note the lack of parens for my_method

You could have like a context manager or a decorator to log what you need, when you need to. if you intend to always log an exception when you use that function, I would suggest going the simple decorator rule or even a try and except inside that function. If it is functions not in your code or you dont want them to always log,then I would used a context manager (called as with ..:)
A context manager example code
import functools
class LoggerContext():
def __enter__(self):
# function that is called on enter of the with context
# we dont need this
pass
def __exit__(self, type, value, traceback):
# If there was an exception, it will be passed to the
# exit function.
# type = type of exception
# value = the string arg of the exception
# traceback object for you to extract the traceback if you need to
if traceback:
# do something with exception like log it etc
print(type, value, traceback)
# If the return value of the exit function is not True, python
# interpreter re-raises the exception. We dont want to re-raise
# the exception
return True
def __call__(self, f):
# this is just to make a context manager a decorator
# so that you could use the #context on a function
#functools.wraps(f)
def decorated(*args, **kwds):
with self:
return f(*args, **kwds)
return decorated
#LoggerContext()
def myMethod(test):
raise FileNotFoundError(test)
def myMethod2(test):
raise TypeError(test)
myMethod('asdf')
with LoggerContext():
myMethod2('asdf')
A simple decorator example:
import functools
def LoggerDecorator(f):
#functools.wraps(f)
def decorated(*args, **kwds):
try:
return f(*args, **kwds)
except Exception as e:
# do something with exception
print('Exception:', e)
return decorated
#LoggerDecorator
def myMethod3(test):
raise IOError(test)
myMethod3('asdf')

Exceptions for the whole class

I'm writing a program in Python, and nearly every method im my class is written like this:
def someMethod(self):
try:
#...
except someException:
#in case of exception, do something here
#e.g display a dialog box to inform the user
#that he has done something wrong
As the class grows, it is a little bit annoying to write the same try-except block over and over. Is it possible to create some sort of 'global' exception for the whole class? What's the recommended way in Python to deal with this?

Write one or more exception handler functions that, given a function and the exception raised in it, does what you want to do (e.g. displays an alert). If you need more than one, write them.
def message(func, e):
print "Exception", type(e).__name__, "in", func.__name__
print str(e)
Now write a decorator that applies a given handler to a called function:
import functools
def handle_with(handler, *exceptions):
try:
handler, cleanup = handler
except TypeError:
cleanup = lambda f, e: None
def decorator(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except exceptions or Exception as e:
return handler(func, e)
else:
e = None
finally:
cleanup(func, e)
return wrapper
return decorator
This only captures the exceptions you specify. If you don't specify any, Exception is caught. Additionally, the first argument can be a tuple (or other sequence) of two handler functions; the second handler, if given, is called in a finally clause. The value returned from the primary handler is returned as the value of the function call.
Now, given the above, you can write:
#handle_with(message, TypeError, ValueError)
def add(x, y):
return x + y
You could also do this with a context manager:
from contextlib import contextmanager
#contextmanager
def handler(handler, *exceptions):
try:
handler, cleanup = handler
except TypeError:
cleanup = lambda e: None
try:
yield
except exceptions or Exception as e:
handler(e)
else:
e = None
finally:
cleanup(e)
Now you can write:
def message(e):
print "Exception", type(e).__name__
print str(e)
def add(x, y):
with handler(message, TypeError, ValueError):
return x + y
Note that the context manager doesn't know what function it's in (you can find this out, sorta, using inspect, though this is "magic" so I didn't do it) so it gives you a little less useful information. Also, the context manager doesn't give you the opportunity to return anything in your handler.

I can think of two options:
Write a decorator that can wrap each method in the try block.
Write a "dispatcher" method that calls the appropriate method inside a try block, then call that method instead of the individual ones. That is, instead of calling obj.someMethod(), obj.otherMethod, you call obj.dispatch('someMethod') or obj.dispatch('otherMethod'), where dispatch is a wrapper that contains the try block.
Your approach seems like a bit of a strange design, though. It might make more sense to have the dialog-box stuff in some other part of the code, some higher-level event loop that catches errors and displays messages about them.

Make Python unittest fail on exception from any thread

I am using the unittest framework to automate integration tests of multi-threaded python code, external hardware and embedded C. Despite my blatant abuse of a unittesting framework for integration testing, it works really well. Except for one problem: I need the test to fail if an exception is raised from any of the spawned threads. Is this possible with the unittest framework?
A simple but non-workable solution would be to either a) refactor the code to avoid multi-threading or b) test each thread separately. I cannot do that because the code interacts asynchronously with the external hardware. I have also considered implementing some kind of message passing to forward the exceptions to the main unittest thread. This would require significant testing-related changes to the code being tested, and I want to avoid that.
Time for an example. Can I modify the test script below to fail on the exception raised in my_thread without modifying the x.ExceptionRaiser class?
import unittest
import x
class Test(unittest.TestCase):
def test_x(self):
my_thread = x.ExceptionRaiser()
# Test case should fail when thread is started and raises
# an exception.
my_thread.start()
my_thread.join()
if __name__ == '__main__':
unittest.main()

At first, sys.excepthook looked like a solution. It is a global hook which is called every time an uncaught exception is thrown.
Unfortunately, this does not work. Why? well threading wraps your run function in code which prints the lovely tracebacks you see on screen (noticed how it always tells you Exception in thread {Name of your thread here}? this is how it's done).
Starting with Python 3.8, there is a function which you can override to make this work: threading.excepthook
... threading.excepthook() can be overridden to control how uncaught exceptions raised by Thread.run() are handled
So what do we do? Replace this function with our logic, and voilà:
For python >= 3.8
import traceback
import threading
import os
class GlobalExceptionWatcher(object):
def _store_excepthook(self, args):
'''
Uses as an exception handlers which stores any uncaught exceptions.
'''
self.__org_hook(args)
formated_exc = traceback.format_exception(args.exc_type, args.exc_value, args.exc_traceback)
self._exceptions.append('\n'.join(formated_exc))
return formated_exc
def __enter__(self):
'''
Register us to the hook.
'''
self._exceptions = []
self.__org_hook = threading.excepthook
threading.excepthook = self._store_excepthook
def __exit__(self, type, value, traceback):
'''
Remove us from the hook, assure no exception were thrown.
'''
threading.excepthook = self.__org_hook
if len(self._exceptions) != 0:
tracebacks = os.linesep.join(self._exceptions)
raise Exception(f'Exceptions in other threads: {tracebacks}')
For older versions of Python, this is a bit more complicated.
Long story short, it appears that the threading nodule has an undocumented import which does something along the lines of:
threading._format_exc = traceback.format_exc
Not very surprisingly, this function is only called when an exception is thrown from a thread's run function.
So for python <= 3.7
import threading
import os
class GlobalExceptionWatcher(object):
def _store_excepthook(self):
'''
Uses as an exception handlers which stores any uncaught exceptions.
'''
formated_exc = self.__org_hook()
self._exceptions.append(formated_exc)
return formated_exc
def __enter__(self):
'''
Register us to the hook.
'''
self._exceptions = []
self.__org_hook = threading._format_exc
threading._format_exc = self._store_excepthook
def __exit__(self, type, value, traceback):
'''
Remove us from the hook, assure no exception were thrown.
'''
threading._format_exc = self.__org_hook
if len(self._exceptions) != 0:
tracebacks = os.linesep.join(self._exceptions)
raise Exception('Exceptions in other threads: %s' % tracebacks)
Usage:
my_thread = x.ExceptionRaiser()
# will fail when thread is started and raises an exception.
with GlobalExceptionWatcher():
my_thread.start()
my_thread.join()
You still need to join yourself, but upon exit, the with-statement's context manager will check for any exception thrown in other threads, and will raise an exception appropriately.
THE CODE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED
This is an undocumented, sort-of-horrible hack. I tested it on linux and windows, and it seems to work. Use it at your own risk.

I've come across this problem myself, and the only solution I've been able to come up with is subclassing Thread to include an attribute for whether or not it terminates without an uncaught exception:
from threading import Thread
class ErrThread(Thread):
"""
A subclass of Thread that will log store exceptions if the thread does
not exit normally
"""
def run(self):
try:
Thread.run(self)
except Exception as self.err:
pass
else:
self.err = None
class TaskQueue(object):
"""
A utility class to run ErrThread objects in parallel and raises and exception
in the event that *any* of them fail.
"""
def __init__(self, *tasks):
self.threads = []
for t in tasks:
try:
self.threads.append(ErrThread(**t)) ## passing in a dict of target and args
except TypeError:
self.threads.append(ErrThread(target=t))
def run(self):
for t in self.threads:
t.start()
for t in self.threads:
t.join()
if t.err:
raise Exception('Thread %s failed with error: %s' % (t.name, t.err))

I've been using the accepted answer above for a while now, but since Python 3.8 the solution doesn't work anymore because the threading module doesn't have this _format_exc import anymore.
On the other hand the threading module now has a nice way to register custom except hooks in Python 3.8 so here is a simple solution to run unit tests which assert that some exceptions are raised inside threads:
def test_in_thread():
import threading
exceptions_caught_in_threads = {}
def custom_excepthook(args):
thread_name = args.thread.name
exceptions_caught_in_threads[thread_name] = {
'thread': args.thread,
'exception': {
'type': args.exc_type,
'value': args.exc_value,
'traceback': args.exc_traceback
}
}
# Registering our custom excepthook to catch the exception in the threads
threading.excepthook = custom_excepthook
# dummy function that raises an exception
def my_function():
raise Exception('My Exception')
# running the funciton in a thread
thread_1 = threading.Thread(name='thread_1', target=my_function, args=())
thread_1.start()
thread_1.join()
assert 'thread_1' in exceptions_caught_in_threads # there was an exception in thread 1
assert exceptions_caught_in_threads['thread_1']['exception']['type'] == Exception
assert str(exceptions_caught_in_threads['thread_1']['exception']['value']) == 'My Exception'