My goal is to create a function that I can use to measure the execution and resource use of another function. Using a tutorial, I've create the below using Python's ThreadPoolExecutor:
from resource import *
from time import sleep
from concurrent.futures import ThreadPoolExecutor
class MemoryMonitor:
def __init__(self):
self.keep_measuring = True
def measure_usage(self):
max_usage = 0
u_run_time = 0
s_run_time = 0
while self.keep_measuring:
max_usage = max(max_usage, getrusage(RUSAGE_SELF).ru_maxrss)
u_run_time = max(u_run_time, getrusage(RUSAGE_SELF).ru_utime)
s_run_time = max(s_run_time, getrusage(RUSAGE_SELF).ru_stime)
sleep(0.1) # run this loop every 0.1 seconds
return [max_usage, u_run_time, s_run_time]
def execute(function):
with ThreadPoolExecutor() as executor:
monitor = MemoryMonitor()
stats_thread = executor.submit(monitor.measure_usage)
try:
fn_thread = executor.submit(function)
result = fn_thread.result()
print("print result")
print(result)
print("print result type")
print(type(result))
finally:
monitor.keep_measuring = False
stats = stats_thread.result()
print(stats)
return result
def foo():
i = 0
while i < 3:
print("foo")
i+=1
return 1
def bar(x):
while x < 3:
print("foobar")
x+=1
return 1
var = execute(foo)
print("Var = " + str(var))
var = execute(bar(0))
print("Var = " + str(var))
If I pass the function foo as an argument to the function execute, it prints the correct results and returns the value returned by foo.
If I pass the function bar in the same way, but with bar itself requiring an argument, the function runs (prints 3 times) and then I get the following error:
result = self.fn(*self.args, **self.kwargs)
TypeError: 'int' object is not callable
After some testing, the part where I'm stuck appears to be passing a function as an argument, if that function itself requires an argument. As I understand the ThreadPoolExecutor, the fn_thread object encapsulates the execution of the function submitted. The result object should simply hold the result of that execution - what am I missing that this cannot handle being passed a function with an argument?
You are submitting
bar(0)
instead of
bar, 0
To clarify, look at the submit's signature:
submit(fn, *args, **kwargs)
the result of
bar(0)
is an integer, and the executor cannot call an integer, since it is not 'callable', as the error message suggests.
As you can see, the connect function, is converting the _connect function into a lambda through convert and it's getting passed to run_api function. The exception thrown in _connect is not getting caught by the except in the run_api function. Is anything that needs to be done with respect to lambda?
The code looks good but still am not able to figure out why the exception is not getting caught incase of failure.
Here is my code.
def run_api(function, retry_count):
count = 0
while count < retry_count:
count += 1
try:
function()
return True
except (BleTestFail, BleTestError):
if count == retry_count:
return False
def convert(func):
return lambda: func
def _connect(self, target_id):
result = self.device.ble_central.connect(target_id)
self.logger.debug('Connect output %s', result)
if result['op'] != 'ok':
self.logger.error('Connect command execution failed')
raise ble_utils.BleTestFail('Failed to connect')
return True
def connect(self, target_ids, retry_count=1):
connected = []
unconnected = []
if not isinstance(target_ids, list):
target_ids = [target_ids]
for target_id in target_ids:
connect_function = ble_utils.convert(self._connect(target_id))
connect_status = ble_utils.run_api(connect_function, retry_count,
'connecting device %s' % target_id,
self.logger)
if connect_status:
connected.append(target_id)
else:
unconnected.append(target_id)
if connected:
self.logger.info('Connected to %s devices: %s', len(connected), connected)
if unconnected:
self.logger.error('Unable to connect %s devices: %s', len(unconnected),
unconnected)
return connected, unconnected
So, to clarify, we have the example "API" to which we want to provide a callback:
def run_api(function, retry_count):
count = 0
while count < retry_count:
count += 1
try:
function()
return True
except (BleTestFail, BleTestError):
if count == retry_count:
return False
And a method that we want to be called in that API, with a specific argument:
class Example:
# other stuff omitted...
def _connect(self, target_id):
result = self.device.ble_central.connect(target_id)
self.logger.debug('Connect output %s', result)
if result['op'] != 'ok':
self.logger.error('Connect command execution failed')
raise ble_utils.BleTestFail('Failed to connect')
return True
connection = Example()
So now we want to call run_api with connection._connect, but somehow provide the target_id information.
This is called binding, and the most elegant way to do it is with the standard library functools.partial:
from functools import partial
# This is how we can make the `convert` function from before:
def convert(func, param):
return partial(func, param)
# But there is no point to this, since we can just use `partial` directly.
# There was no hope for the original approach, because you were calling the
# function ahead of time and passing the returned result to `convert`.
# So, the process looks like this:
# target_id = 1, retry_count = 2
run_api(partial(connection._connect, 1), 2)
You can make it work with lambda, but I don't recommend it - functools.partial is more explicit, and elegantly handles more advanced use cases that have some unexpected gotchas (in particular, if you want to make multiple callbacks in a loop; you may find they all unexpectedly bind with the same value, or else you have to use a very ugly workaround). But for the sake of completion, that looks like so:
def convert(func, param):
return lambda: func(param)
I found myself using the following pattern in my tests quite often:
def test(params):
e_list = []
for p in params:
try:
run_test(p) # Or a block of codes that can continue or break
except Exception as e:
e_list.append(e)
assert isEmpty(e_list), 'error encountered: {}'.format(e_list)
I find myself rewriting this pattern quite often, especially with long code block for the loop that has some flow controls with continue and break. I am wondering if there is a python-ic wrapping to this pattern.
I have thought about a wrapper function like this:
def assert_all_tests(test_list):
e_list = []
for t in test_list:
try:
t()
except Exception as e:
e_list.append(e)
assert isEmpty(e_list), 'error encountered: {}'.format(e_list)
def test(params):
assert_all_tests([functools.partial(run_test, p) for p in params])
But I dislike this approach because it wrapped away the loop. There is no way for callable t to do flow control of the loop with continue or break (there is no loop any more, only a list comprehension).
Another approach is to use a context class like this:
def test(params):
ErrorHandler.clearErrorList()
for p in params:
with ErrorHandler():
run_test(p) # or code block that can continue or break
ErrorHandler.assertEmptyErrorList()
where ErrorHandler would be a class with appropriate __enter__ and __exit__ and keep an error list in a class variable. But I feel that at the test function level, this is not any simpler than the original pattern: since there is no way for a ErrorHandler instance to know when a loop has began and ended, I still have to write the pre- and post- loop fixtures.
I'd like to hear idea of approaches to wrap around this pattern. Thanks.
EDIT
Thank you all for your comments.
New approach inspired by #paul-cornelius's answer
class ResultCollector(object):
def __init__(self, raise_on_error=True):
self.result_list = []
self.raise_on_error = raise_on_error
def do(self, func, *args, **kwds):
'''do can only deal with code block that can be wrapped into a function'''
try:
return func(*args, **kwds)
except Exception as e:
if not isinstance(e, AssertionError) and self.raise_on_error:
raise
self.result_list.append(e.message or e)
else:
self.result_list.append(None)
def assertClean(self):
assert not [x for x in self.result_list if x is not None], 'test results: {}'.format(self.result_list)
def __enter__(self):
self.result_list = []
return self
def __exit__(self, exc_t, exc_i, exc_tb):
if exc_t:
return None
self.assertClean()
return True
def test():
def can_be_refactored_into_func(p):
assert p%3, 'failed {}'.format(p)
def condition_for_skip(p):
return p%2
def condition_for_break(p):
return p>5
with ResultCollector() as rc:
for p in range(10):
if condition_for_skip(p):
rc.result_list.append('skipped {}'.format(p))
continue
if condition_for_break(p):
rc.result_list.append('ended {}'.format(p))
break
rc.do(can_be_refactored_into_func, p)
It works pretty well when the code into loop block can be divided up into functions like above.
How about a little class that only does the one thing you find yourself doing over and over:
class TestTracker:
def __init__(self):
self.error_list = []
def do_test(self, f, p):
try:
f(p)
except Exception as e:
self.error_list.append(e)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
if exc_value is not None:
self.error_list.append(exc_value)
return True
def test(params):
tt = TestTracker()
for p in params:
tt.do_test(run_test, p)
assert isEmpty(tt.error_list), 'error encountered: {}'.format(tt.error_list)
def test2(params):
tt = TestTracker()
for p in params:
with tt:
# a block of code with loop control statements
pass
assert isEmpty(tt.error_list), 'error encountered: {}'.format(tt.error_list)
I modified this answer to make the class a context manager. The test2 shows how that can be used with loop control statements. If no exception is raised within the context, the arguments to __exit__ will be None.
You could even mix with statements and calls to do_test.
Python can do anything!
Edits :
Add some convenience to TestTracker
class TestTracker:
def __init__(self):
self.error_list = []
def do_test(self, f, p):
try:
f(p)
except Exception as e:
self.error_list.append(e)
def __bool__(self):
return len(self.error_list) == 0
def __str__(self):
return 'error encountered: {}'.format(self.error_list)
def test(params):
tt = TestTracker()
for p in params:
tt.do_test(run_test, p)
assert tt, str(tt)
I have the following decorator that is supposed to wrap the implementation of test case functions within a try/except block and print the log if an exception occurs.
def print_log(test_case):
#wraps(test_case)
def run_test(self):
try:
test_case(self)
except:
Test_Loop.failure_teardown(self)
raise
return run_test
This however does not seem to work on one of my test cases that calls a yield generator
Please bear with me as this is a basic example:
class Test_Loop:
# ton of implementation here (e.g. initialization, etc)
def runIt(self, name, ip, port):
# code here
#print_log
def test_log_looper(self):
for l in self.links:
# initialize variables seen below and other stuff
for n in names:
# do stuff
for i in ips:
# do stuff
for p in ports:
yield self.runIt, l, n, i, p
From debugging, when the decorator is applied, it seems that it does not even enter the first loop. What am I doing wrong?
You need to iterate over your generator. Modify your decorator like this:
def print_log(test_case):
#wraps(test_case)
def run_test(self):
try:
for _ in test_case(self): pass
except:
Test_Loop.failure_teardown(self)
raise
return run_test
I'd interacting with a lot of deeply nested json I didn't write, and would like to make my python script more 'forgiving' to invalid input. I find myself writing involved try-except blocks, and would rather just wrap the dubious function up.
I understand it's a bad policy to swallow exceptions, but I'd rather prefer they to be printed and analysed later, than to actually stop execution. It's more valuable, in my use-case to continue executing over the loop than to get all keys.
Here's what I'm doing now:
try:
item['a'] = myobject.get('key').METHOD_THAT_DOESNT_EXIST()
except:
item['a'] = ''
try:
item['b'] = OBJECT_THAT_DOESNT_EXIST.get('key2')
except:
item['b'] = ''
try:
item['c'] = func1(ARGUMENT_THAT_DOESNT_EXIST)
except:
item['c'] = ''
...
try:
item['z'] = FUNCTION_THAT_DOESNT_EXIST(myobject.method())
except:
item['z'] = ''
Here's what I'd like, (1):
item['a'] = f(myobject.get('key').get('subkey'))
item['b'] = f(myobject.get('key2'))
item['c'] = f(func1(myobject)
...
or (2):
#f
def get_stuff():
item={}
item['a'] = myobject.get('key').get('subkey')
item['b'] = myobject.get('key2')
item['c'] = func1(myobject)
...
return(item)
...where I can wrap either the single data item (1), or a master function (2), in some function that turns execution-halting exceptions into empty fields, printed to stdout. The former would be sort of an item-wise skip - where that key isn't available, it logs blank and moves on - the latter is a row-skip, where if any of the fields don't work, the entire record is skipped.
My understanding is that some kind of wrapper should be able to fix this. Here's what I tried, with a wrapper:
def f(func):
def silenceit():
try:
func(*args,**kwargs)
except:
print('Error')
return(silenceit)
Here's why it doesn't work. Call a function that doesn't exist, it doesn't try-catch it away:
>>> f(meow())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'meow' is not defined
Before I even add a blank return value, I'd like to get it to try-catch correctly. If the function had worked, this would have printed "Error", right?
Is a wrapper function the correct approach here?
UPDATE
I've had a lot of really useful, helpful answers below, and thank you for them---but I've edited the examples I used above to illustrate that I'm trying to catch more than nested key errors, that I'm looking specifically for a function that wraps a try-catch for...
When a method doesn't exist.
When an object doesn't exist, and is getting a method called on it.
When an object that does not exist is being called as an argument to a function.
Any combination of any of these things.
Bonus, when a function doesn't exist.
There are lots of good answers here, but I didn't see any that address the question of whether you can accomplish this via decorators.
The short answer is "no," at least not without structural changes to your code. Decorators operate at the function level, not on individual statements. Therefore, in order to use decorators, you would need to move each of the statements to be decorated into its own function.
But note that you can't just put the assignment itself inside the decorated function. You need to return the rhs expression (the value to be assigned) from the decorated function, then do the assignment outside.
To put this in terms of your example code, one might write code with the following pattern:
#return_on_failure('')
def computeA():
item['a'] = myobject.get('key').METHOD_THAT_DOESNT_EXIST()
item["a"] = computeA()
return_on_failure could be something like:
def return_on_failure(value):
def decorate(f):
def applicator(*args, **kwargs):
try:
return f(*args,**kwargs)
except:
print('Error')
return value
return applicator
return decorate
You could use a defaultdict and the context manager approach as outlined in Raymond Hettinger's PyCon 2013 presentation
from collections import defaultdict
from contextlib import contextmanager
#contextmanager
def ignored(*exceptions):
try:
yield
except exceptions:
pass
item = defaultdict(str)
obj = dict()
with ignored(Exception):
item['a'] = obj.get(2).get(3)
print item['a']
obj[2] = dict()
obj[2][3] = 4
with ignored(Exception):
item['a'] = obj.get(2).get(3)
print item['a']
It's very easy to achieve using configurable decorator.
def get_decorator(errors=(Exception, ), default_value=''):
def decorator(func):
def new_func(*args, **kwargs):
try:
return func(*args, **kwargs)
except errors, e:
print "Got error! ", repr(e)
return default_value
return new_func
return decorator
f = get_decorator((KeyError, NameError), default_value='default')
a = {}
#f
def example1(a):
return a['b']
#f
def example2(a):
return doesnt_exist()
print example1(a)
print example2(a)
Just pass to get_decorator tuples with error types which you want to silence and default value to return.
Output will be
Got error! KeyError('b',)
default
Got error! NameError("global name 'doesnt_exist' is not defined",)
default
Edit: Thanks to martineau i changed default value of errors to tuples with basic Exception to prevents errors.
It depends on what exceptions you expect.
If your only use case is get(), you could do
item['b'] = myobject.get('key2', '')
For the other cases, your decorator approach might be useful, but not in the way you do it.
I'll try to show you:
def f(func):
def silenceit(*args, **kwargs): # takes all kinds of arguments
try:
return func(*args, **kwargs) # returns func's result
except Exeption, e:
print('Error:', e)
return e # not the best way, maybe we'd better return None
# or a wrapper object containing e.
return silenceit # on the correct level
Nevertheless, f(some_undefined_function())won't work, because
a) f() isn't yet active at the execution time and
b) it is used wrong. The right way would be to wrap the function and then call it: f(function_to_wrap)().
A "layer of lambda" would help here:
wrapped_f = f(lambda: my_function())
wraps a lambda function which in turn calls a non-existing function. Calling wrapped_f() leads to calling the wrapper which calls the lambda which tries to call my_function(). If this doesn't exist, the lambda raises an exception which is caught by the wrapper.
This works because the name my_function is not executed at the time the lambda is defined, but when it is executed. And this execution is protected and wrapped by the function f() then. So the exception occurs inside the lambda and is propagated to the wrapping function provided by the decorator, which handles it gracefully.
This move towards inside the lambda function doesn't work if you try to replace the lambda function with a wrapper like
g = lambda function: lambda *a, **k: function(*a, **k)
followed by a
f(g(my_function))(arguments)
because here the name resolution is "back at the surface": my_function cannot be resolved and this happens before g() or even f() are called. So it doesn't work.
And if you try to do something like
g(print)(x.get('fail'))
it cannot work as well if you have no x, because g() protects print, not x.
If you want to protect x here, you'll have to do
value = f(lambda: x.get('fail'))
because the wrapper provided by f() calls that lambda function which raises an exception which is then silenced.
Extending #iruvar answer - starting with Python 3.4 there is an existing context manager for this in Python standard lib: https://docs.python.org/3/library/contextlib.html#contextlib.suppress
from contextlib import suppress
with suppress(FileNotFoundError):
os.remove('somefile.tmp')
with suppress(FileNotFoundError):
os.remove('someotherfile.tmp')
in your case you first evaluate the value of the meow call (which doesn't exist) and then wrap it in the decorator. this doesn't work that way.
first the exception is raised before it was wrapped, then the wrapper is wrongly indented (silenceit should not return itself). You might want to do something like:
def hardfail():
return meow() # meow doesn't exist
def f(func):
def wrapper():
try:
func()
except:
print 'error'
return wrapper
softfail =f(hardfail)
output:
>>> softfail()
error
>>> hardfail()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in hardfail
NameError: global name 'meow' is not defined
anyway in your case I don't understand why you don't use a simple method such as
def get_subkey(obj, key, subkey):
try:
return obj.get(key).get(subkey, '')
except AttributeError:
return ''
and in the code:
item['a'] = get_subkey(myobject, 'key', 'subkey')
Edited:
In case you want something that will work at any depth. You can do something like:
def get_from_object(obj, *keys):
try:
value = obj
for k in keys:
value = value.get(k)
return value
except AttributeError:
return ''
That you'd call:
>>> d = {1:{2:{3:{4:5}}}}
>>> get_from_object(d, 1, 2, 3, 4)
5
>>> get_from_object(d, 1, 2, 7)
''
>>> get_from_object(d, 1, 2, 3, 4, 5, 6, 7)
''
>>> get_from_object(d, 1, 2, 3)
{4: 5}
And using your code
item['a'] = get_from_object(obj, 2, 3)
By the way, on a personal point of view I also like #cravoori solution using contextmanager. But this would mean having three lines of code each time:
item['a'] = ''
with ignored(AttributeError):
item['a'] = obj.get(2).get(3)
Why not just use cycle?
for dst_key, src_key in (('a', 'key'), ('b', 'key2')):
try:
item[dst_key] = myobject.get(src_key).get('subkey')
except Exception: # or KeyError?
item[dst_key] = ''
Or if you wish write a little helper:
def get_value(obj, key):
try:
return obj.get(key).get('subkey')
except Exception:
return ''
Also you can combine both solutions if you have a few places where you need to get value and helper function would be more reasonable.
Not sure that you actually need a decorator for your problem.
Since you're dealing with lots of broken code, it may be excusable to use eval in this case.
def my_eval(code):
try:
return eval(code)
except: # Can catch more specific exceptions here.
return ''
Then wrap all your potentially broken statements:
item['a'] = my_eval("""myobject.get('key').get('subkey')""")
item['b'] = my_eval("""myobject.get('key2')""")
item['c'] = my_eval("""func1(myobject)""")
How about something like this:
def exception_handler(func):
def inner_function(*args, **kwargs):
try:
func(*args, **kwargs)
except TypeError:
print(f"{func.__name__} error")
return inner_function
then
#exception_handler
def doSomethingExceptional():
a=2/0
all credits go to:https://medium.com/swlh/handling-exceptions-in-python-a-cleaner-way-using-decorators-fae22aa0abec
Try Except Decorator for sync and async functions
Note: logger.error can be replaced with print
Latest version can be found here.