Related
I am writing a script that needs to use a class from an external library, do some
operations on instances of that class, and then repeat with some more instances.
Something like this:
import some_library
work_queue = get_items()
for item in work_queue:
some_object = some_library.SomeClass(item)
operation_1(some_object)
# ...
operation_N(some_object)
However, each of the operations in the loop body can raise some different exceptions.
When these happen I need to log them and skip to the next item. If they raise
some unexpected exception I need to log that before crashing.
I could catch all the exceptions in the main loop, but that would obscure what it does.
So I find myself writing a bunch of wrapper functions that all look kind of similar:
def wrapper_op1(some_object):
try:
some_object.method_1()
except (some_library.SomeOtherError, ValueError) as error_message:
logger.error("Op1 error on {}".format(some_object.friendly_name))
return False
except Exception as error_message:
logger.error("Unknown error during op1 on {} - crashing: {}".format(some_object.friendly_name, error_message))
raise
else:
return True
# Notice there is a different tuple of anticipated exceptions
# and the message formatting is different
def wrapper_opN(some_object):
try:
some_function(some_object.some_attr)
except (RuntimeError, AttributeError) as error_message:
logger.error("OpN error on {} with {}".format(some_object.friendly_name, some_object.some_attr, error_message))
return False
except Exception as error_message:
logger.error("Unknown error during opN on {} with {} - crashing: {}".(some_object.friendly_name, some_object.some_attr, error_message))
raise
else:
return True
And modifying my main loop to be:
for item in work_queue:
some_object = some_library.SomeClass(item)
if not wrapper_op1(some_object):
continue
# ...
if not wrapper_opN(some_object):
continue
This does the job, but it feels like a lot of copy and paste programming with
the wrappers. What would be great is to write a decorator function that could
do all that try...except...else stuff so I could do:
# logged_call(known_exception, known_error_message, unknown_error_message)
def wrapper_op1(some_object):
some_object.method_1()
The wrapper would return True if the operation succeeds, catch the known exceptions
and log with a specified format, and catch any unknown exceptions for logging before re-raising.
However, I can't seem to fathom how to make the error messages work - I can do it with fixed strings:
def logged_call(known_exceptions, s_err, s_fatal):
def decorate(f):
#wraps(f)
def wrapper(*args, **kwargs):
try:
f(*args, **kwargs)
# How to get parameters from args in the message?
except known_exceptions as error:
print(s_err.format(error))
return False
except Exception as error:
print(s_fatal.format(error))
raise
else:
return True
return wrapper
return decorate
However, my error messages need to get attributes that belong to the decorated function.
Is there some Pythonic way to make this work? Or a different pattern to be using
when dealing with might-fail-in-known-ways functions?
The below tryblock function can be used for this purpose, to implement your logged_call concept and reduce the total amount of code, assuming you have enough checks to overcome the decorator implementation. Folks not used to functional programming in Python may actually find this more difficult to understand than simply writing out the try blocks as you have done. Simplicity, like so many things, is in the eye of the beholder.
Python 2.7 using no imports. This uses the exec statement to create a customized try block that works in a functional pattern.
def tryblock(tryf, *catchclauses, **otherclauses):
u'return a general try-catch-else-finally block as a function'
elsef = otherclauses.get('elsef', None)
finallyf = otherclauses.get('finallyf', None)
namespace = {'tryf': tryf, 'elsef': elsef, 'finallyf': finallyf, 'func': []}
for pair in enumerate(catchclauses):
namespace['e%s' % (pair[0],)] = pair[1][0]
namespace['f%s' % (pair[0],)] = pair[1][1]
source = []
add = lambda indent, line: source.append(' ' * indent + line)
add(0, 'def block(*args, **kwargs):')
add(1, "u'generated function executing a try block'")
add(1, 'try:')
add(2, '%stryf(*args, **kwargs)' % ('return ' if otherclauses.get('returnbody', elsef is None) else '',))
for index in xrange(len(catchclauses)):
add(1, 'except e%s as ex:' % (index,))
add(2, 'return f%s(ex, *args, **kwargs)' % (index,))
if elsef is not None:
add(1, 'else:')
add(2, 'return elsef(*args, **kwargs)')
if finallyf is not None:
add(1, 'finally:')
add(2, '%sfinallyf(*args, **kwargs)' % ('return ' if otherclauses.get('returnfinally', False) else '',))
add(0, 'func.append(block)')
exec '\n'.join(source) in namespace
return namespace['func'][0]
This tryblock function is general enough to go into a common library, as it is not specific to the logic of your checks. Add to it your logged_call decorator, implemented as (one import here):
import functools
resultof = lambda func: func() # # token must be followed by an identifier
#resultof
def logged_call():
truism = lambda *args, **kwargs: True
def raisef(ex, *args, **kwargs):
raise ex
def impl(exlist, err, fatal):
return lambda func: \
functools.wraps(func)(tryblock(func,
(exlist, lambda ex, *args, **kwargs: err(ex, *args, **kwargs) and False),
(Exception, lambda ex, *args, **kwargs: fatal(ex, *args, **kwargs) and raisef(ex))),
elsef=truism)
return impl # impl therefore becomes logged_call
Using logged_call as implemented, your two sanity checks look like:
op1check = logged_call((some_library.SomeOtherError, ValueError),
lambda _, obj: logger.error("Op1 error on {}".format(obj.friendly_name)),
lambda ex, obj: logger.error("Unknown error during op1 on {} - crashing: {}".format(obj.friendly_name, ex.message)))
opNcheck = logged_call((RuntimeError, AttributeError),
lambda ex, obj: logger.error("OpN error on {} with {}".format(obj.friendly_name, obj.some_attr, ex.message)),
lambda ex, obj: logger.error("Unknown error during opN on {} with {} - crashing: {}".format(obj.friendly_name, obj.some_attr, ex.message)))
#op1check
def wrapper_op1(obj):
return obj.method_1()
#opNcheck
def wrapper_opN(obj):
return some_function(obj.some_attr)
Neglecting blank lines, this is more compact than your original code by 10 lines, though at the sunk cost of the tryblock and logged_call implementations; whether it is now more readable is a matter of opinion.
You also have the option to define logged_call itself and all distinct decorators derived from it in a separate module, if that's sensible for your code; and therefore to use each derived decorator multiple times.
You may also find more of the logic structure that you can factor into logged_call by tweaking the actual checks.
But in the worst case, where each check has logic that no other does, you may find that it's more readable to just write out each one like you have already. It really depends.
For completeness, here's a unit test for the tryblock function:
import examplemodule as ex
from unittest import TestCase
class TestTryblock(TestCase):
def test_tryblock(self):
def tryf(a, b):
if a % 2 == 0:
raise ValueError
return a + b
def proc_ve(ex, a, b):
self.assertIsInstance(ex, ValueError)
if a % 3 == 0:
raise ValueError
return a + b + 10
def elsef(a, b):
return a + b + 20
def finallyf(a, b):
return a + b + 30
block = ex.tryblock(tryf, (ValueError, proc_ve))
self.assertRaises(ValueError, block, 0, 4)
self.assertRaises(ValueError, block, 6, 4)
self.assertEqual([5, 16, 7, 18, 9], map(lambda v: block(v, 4), xrange(1, 6)))
block = ex.tryblock(tryf, (ValueError, proc_ve), elsef=elsef)
self.assertEqual([25, 16, 27, 18, 29], map(lambda v: block(v, 4), xrange(1, 6)))
block = ex.tryblock(tryf, (ValueError, proc_ve), elsef=elsef, returnbody=True)
self.assertEqual([5, 16, 7, 18, 9], map(lambda v: block(v, 4), xrange(1, 6)))
block = ex.tryblock(tryf, (ValueError, proc_ve), finallyf=finallyf)
self.assertEqual([5, 16, 7, 18, 9], map(lambda v: block(v, 4), xrange(1, 6)))
block = ex.tryblock(tryf, (ValueError, proc_ve), finallyf=finallyf, returnfinally=True)
self.assertEqual([35, 36, 37, 38, 39], map(lambda v: block(v, 4), xrange(1, 6)))
I am writing a function ChrNumber that converts Arab number string to Chinese financial number string. I work out a tree recursion form. But when I tried to get a tail-recursion form, it is really difficult for me to handle the situation bit equals 6,7 or 8 or 10 and bigger ones.
You can see how it works at the end of my question.
Here's the tree-recursion solution. It works:
# -*- coding:utf-8 -*-
unitArab=(2,3,4,5,9)
#unitStr=u'十百千万亿' #this is an alternative
unitStr=u'拾佰仟万亿'
unitDic=dict(zip(unitArab,(list(unitStr))))
numArab=list(u'0123456789')
#numStr=u'零一二三四五六七八九' #this is an alternative
numStr=u'零壹贰叁肆伍陆柒捌玖'
numDic=dict(zip(numArab,list(numStr)))
def ChnNumber(s):
def wrapper(v):
'this is to adapt the string to a abbreviation'
if u'零零' in v:
return wrapper(v.replace(u'零零',u'零'))
return v[:-1] if v[-1]==u'零' else v
def recur(s,bit):
'receives the number sting and its length'
if bit==1:
return numDic[s]
if s[0]==u'0':
return wrapper(u'%s%s' % (u'零',recur(s[1:],bit-1)))
if bit<6 or bit==9:
return wrapper(u'%s%s%s' % (numDic[s[0]],unitDic[bit],recur(s[1:],bit-1)))
'below is the hard part to be converted to tail-recurion'
if bit<9:
return u'%s%s%s' % (recur(s[:-4],bit-4),u"万",recur(s[-4:],4))
if bit>9:
return u'%s%s%s' % (recur(s[:-8],bit-8),u"亿",recur(s[-8:],8))
return recur(s,len(s))
My attempt version is only in recur function, I use a closure res and move the bit inside the recur so there is less arguments.:
res=[]
def recur(s):
bit=len(s)
print s,bit,res
if bit==0:
return ''.join(res)
if bit==1:
res.append(numDic[s])
return recur(s[1:])
if s[0]==u'0':
res.append(u'零')
return recur(s[1:])
if bit<6 or bit==9:
res.append(u'%s%s' %(numDic[s[0]],unitDic[bit]))
return recur(s[1:])
if bit<9:
#...can't work it out
if bit>9:
#...can't work it out
the test code is:
for i in range(17):
v1='9'+'0'*(i+1)
v2='9'+'0'*i+'9'
v3='1'*(i+2)
print '%s->%s\n%s->%s\n%s->%s'% (v1,ChnNumber(v1),v2,ChnNumber(v2),v3,ChnNumber(v3))
which should output:
>>>
90->玖拾
99->玖拾玖
11->壹拾壹
900->玖佰
909->玖佰零玖
111->壹佰壹拾壹
9000->玖仟
9009->玖仟零玖
1111->壹仟壹佰壹拾壹
90000->玖万
90009->玖万零玖
11111->壹万壹仟壹佰壹拾壹
900000->玖拾万
900009->玖拾万零玖
111111->壹拾壹万壹仟壹佰壹拾壹
9000000->玖佰万
9000009->玖佰万零玖
1111111->壹佰壹拾壹万壹仟壹佰壹拾壹
90000000->玖仟万
90000009->玖仟万零玖
11111111->壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
900000000->玖亿
900000009->玖亿零玖
111111111->壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
9000000000->玖拾亿
9000000009->玖拾亿零玖
1111111111->壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
90000000000->玖佰亿
90000000009->玖佰亿零玖
11111111111->壹佰壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
900000000000->玖仟亿
900000000009->玖仟亿零玖
111111111111->壹仟壹佰壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
9000000000000->玖万亿
9000000000009->玖万亿零玖
1111111111111->壹万壹仟壹佰壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
90000000000000->玖拾万亿
90000000000009->玖拾万亿零玖
11111111111111->壹拾壹万壹仟壹佰壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
900000000000000->玖佰万亿
900000000000009->玖佰万亿零玖
111111111111111->壹佰壹拾壹万壹仟壹佰壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
9000000000000000->玖仟万亿
9000000000000009->玖仟万亿零玖
1111111111111111->壹仟壹佰壹拾壹万壹仟壹佰壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
90000000000000000->玖亿亿
90000000000000009->玖亿亿零玖
11111111111111111->壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
900000000000000000->玖拾亿亿
900000000000000009->玖拾亿亿零玖
111111111111111111->壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹亿壹仟壹佰壹拾壹万壹仟壹佰壹拾壹
Python doesn't support tail call elimination nor tail call optimizations. However, there are a number of ways in which you can mimic this approach (Trampolines being the most widely used in other languages.)
Tail call recursive functions should look like the following pseudo code:
def tail_call(*args, acc):
if condition(*args):
return acc
else:
# Operations happen here, producing new_args and new_acc
return tail_call(*new_args, new_acc)
For your example I would not form a closure over anything as your are introducing side-effects and stateful manipulation. Instead, anything that needs to be modified should be modified in isolation of everything else. That makes it easier to reason about.
Copy whatever you're attempting to change (using string.copy for the final output) and pass it in as an argument to the next recursive call. That's where the acc variable comes into play. It's "accumulating" all your changes up to that point.
A classical trampoline can be had from this snippet. There, they are wrapping the function in an object which will eventually either result a result or return another function object which should be called. I prefer this approach as I find it easier to reason about.
This isn't the only way. Take a look at this code snippet. The "magic" occurs when it reaches a point which "solves" the condition and it throws an exception to escape the infinite loop.
Finally, you can read about Trampolines here, here and here.
I keep studying this question off and on these days. and now, I work it out!
NOTE,not just tail-recursion, it's also pure Functional Programming!
The key is to think in a different way (tree-recursion version is processing numbers from left to right while this version is from right to left)
unitDic=dict(zip(range(8),u'拾佰仟万拾佰仟亿'))
numDic=dict(zip('0123456789',u'零壹贰叁肆伍陆柒捌玖'))
wapDic=[(u'零拾',u'零'),(u'零佰',u'零'),(u'零仟',u'零'),
(u'零万',u'万'),(u'零亿',u'亿'),(u'亿万',u'亿'),
(u'零零',u'零'),]
#pure FP
def ChnNumber(s):
def wrapper(s,wd=wapDic):
def rep(s,k,v):
if k in s:
return rep(s.replace(k,v),k,v)
return s
if not wd:
return s
return wrapper(rep(s,*wd[0]),wd[1:])
def recur(s,acc='',ind=0):
if s=='':
return acc
return recur(s[:-1],numDic[s[-1]]+unitDic[ind%8]+acc,ind+1)
def end(s):
if s[-1]!='0':
return numDic[s[-1]]
return ''
def result(start,end):
if end=='' and start[-1]==u'零':
return start[:-1]
return start+end
return result(wrapper(recur(s[:-1])),end(s))
for i in range(18):
v1='9'+'0'*(i+1)
v2='9'+'0'*i+'9'
v3='1'*(i+2)
print ('%s->%s\n%s->%s\n%s->%s'% (v1,ChnNumber(v1),v2,ChnNumber(v2),v3,ChnNumber(v3)))
if any one say that it won't work when facing a huge number(something like a billion-figure number), yeah, I admit that, but this version can solve it(while it will not be pure FP but pure FP won't need this version so..):
class TailCaller(object) :
def __init__(self, f) :
self.f = f
def __call__(self, *args, **kwargs) :
ret = self.f(*args, **kwargs)
while type(ret) is TailCall :
ret = ret.handle()
return ret
class TailCall(object) :
def __init__(self, call, *args, **kwargs) :
self.call = call
self.args = args
self.kwargs = kwargs
def handle(self) :
if type(self.call) is TailCaller :
return self.call.f(*self.args, **self.kwargs)
else :
return self.f(*self.args, **self.kwargs)
def ChnNumber(s):
def wrapper(s,wd=wapDic):
#TailCaller
def rep(s,k,v):
if k in s:
return TailCall(rep,s.replace(k,v),k,v)
return s
if not wd:
return s
return wrapper(rep(s,*wd[0]),wd[1:])
#TailCaller
def recur(s,acc='',ind=0):
if s=='':
return acc
return TailCall(recur,s[:-1],numDic[s[-1]]+unitDic[ind%8]+acc,ind+1)
def end(s):
if s[-1]!='0':
return numDic[s[-1]]
return ''
def result(start,end):
if end=='' and start[-1]==u'零':
return start[:-1]
return start+end
return result(wrapper(recur(s[:-1])),end(s))
I'm still new to python and this is probably going be one of those (stupid) boring questions. However, any help will be much appreciated. I'm programing something that involves many variables and I've decided to use a class to encapsulate all variables (hopefully making it easier to "read" for me as time passes), but it's not working as I thought it will. So, without further ado here is a part of the class that captures the gist.
import numpy as np
class variable:
def __init__(self, length):
self.length = length # time length`
def state_dynamic(self):
length = self.length
return np.zeros((2, np.size(length)))
def state_static(self):
length = self.length
return np.zeros((2, np.size(length)))
def control_dynamic(self):
length = self.length
return np.zeros((2, np.size(length)))
def control_static(self):
length = self.length
return np.zeros((2, np.size(length)))
def scheduling(self):
length = self.length
return np.zeros(np.size(length))
def disturbance(self):
length = self.length
dummy = np.random.normal(0., 0.1, np.size(length))
for i in range(20):
dummy[i+40] = np.random.normal(0., 0.01) + 1.
dummy[80:100] = 0.
return dummy
I've also tried this one:
import numpy as np
class variable:
def __init__(self, type_1, type_2, length):
self.type_1 = type_1 # belongs to set {state, control, scheduling, disturbance}
self.type_2 = type_2 # belongs to set {static, dynamic, none}
self.length = length # time length
def type_v(self):
type_1 = self.type_1
type_2 = self.type_2
length = self.length
if type_1 == 'state' and type_2 == 'dynamic':
return np.zeros((2, np.size(length)))
elif type_1 == 'state' and type_2 == 'static':
return np.zeros((2, np.size(length)))
elif type_1 == 'control' and type_2 == 'dynamic':
return np.zeros((2, np.size(length)))
elif type_1 == 'control' and type_2 == 'static':
return np.zeros((2, np.size(length)))
elif type_1 == 'scheduling' and type_2 == 'none':
return np.zeros(np.size(length))
elif type_1 == 'disturbance' and type_2 == 'none':
dummy = np.random.normal(0., 0.1, np.size(length))
for i in range(20):
dummy[i+40] = np.random.normal(0., 0.01) + 1.
dummy[80:100] = 0.
return dummy
Now, using the first one (the outcome is the same for the second class as well), when I write the following, say:
In [2]: time = np.linspace(0,10,100)
In [5]: v = variable(time)
In [6]: v1 = v.state_dynamic
In [7]: v1.size
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/home/<ipython-input-7-e6a5d17aeb75> in <module>()
----> 1 v1.size
AttributeError: 'function' object has no attribute 'size'
In [8]: v2 = variable(np.size(time)).state_dynamic
In [9]: v2
Out[9]: <bound method variable.state_dynamic of <__main__.variable instance at 0x3ad0a28>>
In [10]: v1[0,0]
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/home/<ipython-input-10-092bc2b9f982> in <module>()
----> 1 v1[0,0]
TypeError: 'instancemethod' object has no attribute '__getitem__'
I was hoping that by writing
variable(length).state_dynamic
I'll access
np.zeros((2, np.size(length)))
Anyway, if I made something utterly stupid please let me know :) and feel free to give any kind of advice. Thank you in advance for your time and kind attention. Best regards.
EDIT #1:
#wheaties:
Thank you for a quick reply and help :)
What I'm currently trying to do is the following. I have to plot several "variables", e.g., state, control, dropouts, scheduling and disturbances. All the variables depend on three parameters, namely, dynamic, static and horizon. Further, state and control are np.zeros((2, np.size(length))), dropouts and scheduling are np.zeros(np.size(length)) and disturbance has specific form (see above). Initially, I declared them in the script and the list is very long and looks ugly. I use these variables to store responses of dynamical systems considered and to plot them. I don't know if this is a good way of doing this and if you have any suggestion please share.
Thanks again for your help.
Do you mean you want named access to a bunch of state information? The ordinary python idiom for class variables would look like this:
class Variable(object):
def __init__ (self, state_dynamic, state_static, control_static, control_dynamic, scheduling):
self.state_dynamic = state_dynamic
self.state_static = state_static
self.control_static = control_static
self.control_dynamic = control_dynamic
self.scheduling = control_dynamic
Which essentially creates a bucket with named fields that hold values you put in via the constructor. You can also create lightweight data classes using the namedtuple factory class, which avoids some of the boilerplate.
The other python idiom that might apply is to use the #property decorator as in #wheaties answer. This basically disguises a function call to make it look like a field. If what you're doing can be reduced to a functional basis this would make sense. This is an example of the idea (not based on your problem set, since I'm not sure I grok what you're doing in detail with all those identical variables) -- in this case I'm making a convenience wrapper for pulling individual flags out that are stored in a python number but really make a bit field:
class Bits(object):
def __init__(self, integer):
self.Integer = integer # pretend this is an integer between 0 and 8 representing 4 flags
#property
def locked(self):
# low bit = locked
return self.Integer & 1 == 1
#property
def available(self):
return self.Integer & 2 == 2
#property
def running_out_of_made_up_names(self):
return self.Integer & 4 == 4
#property
def really_desperate_now(self):
return self.Integer & 8 == 8
example = Bits(7)
print example.locked
# True
print example.really_desperate_now
# False
A method in Python is a function. If you want to get a value from a member function you have to end it with (). That said, some refactoring may help eliminate boilerplate and reduce the problem set size in your head. I'd suggest using a #property for some of these things, combined with a slight refactor
class variable:
def __init__(self, length):
self.length = length # time length`
#property
def state_dynamic(self):
return self.np_length
#property
def state_static(self):
return self.np_length
#property
def control_dynamic(self):
return self.np_length
#property
def control_static(self):
return self.np_length
#property
def scheduling(self):
return self.np_length
#property
def np_length(self):
return np.zeros(2, np.size(self.length))
That way you can use those functions as you would a member variable like you tried before:
var = variable(length).state_dynamic
What I can't tell from all this is what the difference is between all these variables? I don't see a single one. Are you assuming that you have to access them in order? If so, that's bad design and a problem. Never make that assumption.
I'd interacting with a lot of deeply nested json I didn't write, and would like to make my python script more 'forgiving' to invalid input. I find myself writing involved try-except blocks, and would rather just wrap the dubious function up.
I understand it's a bad policy to swallow exceptions, but I'd rather prefer they to be printed and analysed later, than to actually stop execution. It's more valuable, in my use-case to continue executing over the loop than to get all keys.
Here's what I'm doing now:
try:
item['a'] = myobject.get('key').METHOD_THAT_DOESNT_EXIST()
except:
item['a'] = ''
try:
item['b'] = OBJECT_THAT_DOESNT_EXIST.get('key2')
except:
item['b'] = ''
try:
item['c'] = func1(ARGUMENT_THAT_DOESNT_EXIST)
except:
item['c'] = ''
...
try:
item['z'] = FUNCTION_THAT_DOESNT_EXIST(myobject.method())
except:
item['z'] = ''
Here's what I'd like, (1):
item['a'] = f(myobject.get('key').get('subkey'))
item['b'] = f(myobject.get('key2'))
item['c'] = f(func1(myobject)
...
or (2):
#f
def get_stuff():
item={}
item['a'] = myobject.get('key').get('subkey')
item['b'] = myobject.get('key2')
item['c'] = func1(myobject)
...
return(item)
...where I can wrap either the single data item (1), or a master function (2), in some function that turns execution-halting exceptions into empty fields, printed to stdout. The former would be sort of an item-wise skip - where that key isn't available, it logs blank and moves on - the latter is a row-skip, where if any of the fields don't work, the entire record is skipped.
My understanding is that some kind of wrapper should be able to fix this. Here's what I tried, with a wrapper:
def f(func):
def silenceit():
try:
func(*args,**kwargs)
except:
print('Error')
return(silenceit)
Here's why it doesn't work. Call a function that doesn't exist, it doesn't try-catch it away:
>>> f(meow())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'meow' is not defined
Before I even add a blank return value, I'd like to get it to try-catch correctly. If the function had worked, this would have printed "Error", right?
Is a wrapper function the correct approach here?
UPDATE
I've had a lot of really useful, helpful answers below, and thank you for them---but I've edited the examples I used above to illustrate that I'm trying to catch more than nested key errors, that I'm looking specifically for a function that wraps a try-catch for...
When a method doesn't exist.
When an object doesn't exist, and is getting a method called on it.
When an object that does not exist is being called as an argument to a function.
Any combination of any of these things.
Bonus, when a function doesn't exist.
There are lots of good answers here, but I didn't see any that address the question of whether you can accomplish this via decorators.
The short answer is "no," at least not without structural changes to your code. Decorators operate at the function level, not on individual statements. Therefore, in order to use decorators, you would need to move each of the statements to be decorated into its own function.
But note that you can't just put the assignment itself inside the decorated function. You need to return the rhs expression (the value to be assigned) from the decorated function, then do the assignment outside.
To put this in terms of your example code, one might write code with the following pattern:
#return_on_failure('')
def computeA():
item['a'] = myobject.get('key').METHOD_THAT_DOESNT_EXIST()
item["a"] = computeA()
return_on_failure could be something like:
def return_on_failure(value):
def decorate(f):
def applicator(*args, **kwargs):
try:
return f(*args,**kwargs)
except:
print('Error')
return value
return applicator
return decorate
You could use a defaultdict and the context manager approach as outlined in Raymond Hettinger's PyCon 2013 presentation
from collections import defaultdict
from contextlib import contextmanager
#contextmanager
def ignored(*exceptions):
try:
yield
except exceptions:
pass
item = defaultdict(str)
obj = dict()
with ignored(Exception):
item['a'] = obj.get(2).get(3)
print item['a']
obj[2] = dict()
obj[2][3] = 4
with ignored(Exception):
item['a'] = obj.get(2).get(3)
print item['a']
It's very easy to achieve using configurable decorator.
def get_decorator(errors=(Exception, ), default_value=''):
def decorator(func):
def new_func(*args, **kwargs):
try:
return func(*args, **kwargs)
except errors, e:
print "Got error! ", repr(e)
return default_value
return new_func
return decorator
f = get_decorator((KeyError, NameError), default_value='default')
a = {}
#f
def example1(a):
return a['b']
#f
def example2(a):
return doesnt_exist()
print example1(a)
print example2(a)
Just pass to get_decorator tuples with error types which you want to silence and default value to return.
Output will be
Got error! KeyError('b',)
default
Got error! NameError("global name 'doesnt_exist' is not defined",)
default
Edit: Thanks to martineau i changed default value of errors to tuples with basic Exception to prevents errors.
It depends on what exceptions you expect.
If your only use case is get(), you could do
item['b'] = myobject.get('key2', '')
For the other cases, your decorator approach might be useful, but not in the way you do it.
I'll try to show you:
def f(func):
def silenceit(*args, **kwargs): # takes all kinds of arguments
try:
return func(*args, **kwargs) # returns func's result
except Exeption, e:
print('Error:', e)
return e # not the best way, maybe we'd better return None
# or a wrapper object containing e.
return silenceit # on the correct level
Nevertheless, f(some_undefined_function())won't work, because
a) f() isn't yet active at the execution time and
b) it is used wrong. The right way would be to wrap the function and then call it: f(function_to_wrap)().
A "layer of lambda" would help here:
wrapped_f = f(lambda: my_function())
wraps a lambda function which in turn calls a non-existing function. Calling wrapped_f() leads to calling the wrapper which calls the lambda which tries to call my_function(). If this doesn't exist, the lambda raises an exception which is caught by the wrapper.
This works because the name my_function is not executed at the time the lambda is defined, but when it is executed. And this execution is protected and wrapped by the function f() then. So the exception occurs inside the lambda and is propagated to the wrapping function provided by the decorator, which handles it gracefully.
This move towards inside the lambda function doesn't work if you try to replace the lambda function with a wrapper like
g = lambda function: lambda *a, **k: function(*a, **k)
followed by a
f(g(my_function))(arguments)
because here the name resolution is "back at the surface": my_function cannot be resolved and this happens before g() or even f() are called. So it doesn't work.
And if you try to do something like
g(print)(x.get('fail'))
it cannot work as well if you have no x, because g() protects print, not x.
If you want to protect x here, you'll have to do
value = f(lambda: x.get('fail'))
because the wrapper provided by f() calls that lambda function which raises an exception which is then silenced.
Extending #iruvar answer - starting with Python 3.4 there is an existing context manager for this in Python standard lib: https://docs.python.org/3/library/contextlib.html#contextlib.suppress
from contextlib import suppress
with suppress(FileNotFoundError):
os.remove('somefile.tmp')
with suppress(FileNotFoundError):
os.remove('someotherfile.tmp')
in your case you first evaluate the value of the meow call (which doesn't exist) and then wrap it in the decorator. this doesn't work that way.
first the exception is raised before it was wrapped, then the wrapper is wrongly indented (silenceit should not return itself). You might want to do something like:
def hardfail():
return meow() # meow doesn't exist
def f(func):
def wrapper():
try:
func()
except:
print 'error'
return wrapper
softfail =f(hardfail)
output:
>>> softfail()
error
>>> hardfail()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in hardfail
NameError: global name 'meow' is not defined
anyway in your case I don't understand why you don't use a simple method such as
def get_subkey(obj, key, subkey):
try:
return obj.get(key).get(subkey, '')
except AttributeError:
return ''
and in the code:
item['a'] = get_subkey(myobject, 'key', 'subkey')
Edited:
In case you want something that will work at any depth. You can do something like:
def get_from_object(obj, *keys):
try:
value = obj
for k in keys:
value = value.get(k)
return value
except AttributeError:
return ''
That you'd call:
>>> d = {1:{2:{3:{4:5}}}}
>>> get_from_object(d, 1, 2, 3, 4)
5
>>> get_from_object(d, 1, 2, 7)
''
>>> get_from_object(d, 1, 2, 3, 4, 5, 6, 7)
''
>>> get_from_object(d, 1, 2, 3)
{4: 5}
And using your code
item['a'] = get_from_object(obj, 2, 3)
By the way, on a personal point of view I also like #cravoori solution using contextmanager. But this would mean having three lines of code each time:
item['a'] = ''
with ignored(AttributeError):
item['a'] = obj.get(2).get(3)
Why not just use cycle?
for dst_key, src_key in (('a', 'key'), ('b', 'key2')):
try:
item[dst_key] = myobject.get(src_key).get('subkey')
except Exception: # or KeyError?
item[dst_key] = ''
Or if you wish write a little helper:
def get_value(obj, key):
try:
return obj.get(key).get('subkey')
except Exception:
return ''
Also you can combine both solutions if you have a few places where you need to get value and helper function would be more reasonable.
Not sure that you actually need a decorator for your problem.
Since you're dealing with lots of broken code, it may be excusable to use eval in this case.
def my_eval(code):
try:
return eval(code)
except: # Can catch more specific exceptions here.
return ''
Then wrap all your potentially broken statements:
item['a'] = my_eval("""myobject.get('key').get('subkey')""")
item['b'] = my_eval("""myobject.get('key2')""")
item['c'] = my_eval("""func1(myobject)""")
How about something like this:
def exception_handler(func):
def inner_function(*args, **kwargs):
try:
func(*args, **kwargs)
except TypeError:
print(f"{func.__name__} error")
return inner_function
then
#exception_handler
def doSomethingExceptional():
a=2/0
all credits go to:https://medium.com/swlh/handling-exceptions-in-python-a-cleaner-way-using-decorators-fae22aa0abec
Try Except Decorator for sync and async functions
Note: logger.error can be replaced with print
Latest version can be found here.
I am maintaining a little library of useful functions for interacting with my company's APIs and I have come across (what I think is) a neat question that I can't find the answer to.
I frequently have to request large amounts of data from an API, so I do something like:
class Client(object):
def __init__(self):
self.data = []
def get_data(self, offset = 0):
done = False
while not done:
data = get_more_starting_at(offset)
self.data.extend(data)
offset += 1
if not data:
done = True
This works fine and allows me to restart the retrieval where I left off if something goes horribly wrong. However, since python functions are just regular objects, we can do stuff like:
def yo():
yo.hi = "yo!"
return None
and then we can interrogate yo about its properties later, like:
yo.hi => "yo!"
my question is: Can I rewrite my class-based example to pin the data to the function itself, without referring to the function by name. I know I can do this by:
def get_data(offset=0):
done = False
get_data.data = []
while not done:
data = get_more_starting_from(offset)
get_data.data.extend(data)
offset += 1
if not data:
done = True
return get_data.data
but I would like to do something like:
def get_data(offset=0):
done = False
self.data = [] # <===== this is the bit I can't figure out
while not done:
data = get_more_starting_from(offset)
self.data.extend(data) # <====== also this!
offset += 1
if not data:
done = True
return self.data # <======== want to refer to the "current" object
Is it possible to refer to the "current" object by anything other than its name?
Something like "this", "self", or "memememe!" is what I'm looking for.
I don't understand why you want to do this, but it's what a fixed point combinator allows you to do:
import functools
def Y(f):
#functools.wraps(f)
def Yf(*args):
return inner(*args)
inner = f(Yf)
return Yf
#Y
def get_data(f):
def inner_get_data(*args):
# This is your real get data function
# define it as normal
# but just refer to it as 'f' inside itself
print 'setting get_data.foo to', args
f.foo = args
return inner_get_data
get_data(1, 2, 3)
print get_data.foo
So you call get_data as normal, and it "magically" knows that f means itself.
You could do this, but (a) the data is not per-function-invocation, but per function (b) it's much easier to achieve this sort of thing with a class.
If you had to do it, you might do something like this:
def ybother(a,b,c,yrselflambda = lambda: ybother):
yrself = yrselflambda()
#other stuff
The lambda is necessary, because you need to delay evaluation of the term ybother until something has been bound to it.
Alternatively, and increasingly pointlessly:
from functools import partial
def ybother(a,b,c,yrself=None):
#whatever
yrself.data = [] # this will blow up if the default argument is used
#more stuff
bothered = partial(ybother, yrself=ybother)
Or:
def unbothered(a,b,c):
def inbothered(yrself):
#whatever
yrself.data = []
return inbothered, inbothered(inbothered)
This last version gives you a different function object each time, which you might like.
There are almost certainly introspective tricks to do this, but they are even less worthwhile.
Not sure what doing it like this gains you, but what about using a decorator.
import functools
def add_self(f):
#functools.wraps(f)
def wrapper(*args,**kwargs):
if not getattr(f, 'content', None):
f.content = []
return f(f, *args, **kwargs)
return wrapper
#add_self
def example(self, arg1):
self.content.append(arg1)
print self.content
example(1)
example(2)
example(3)
OUTPUT
[1]
[1, 2]
[1, 2, 3]