To start with, my question here is about the semantics and the logic behind why the Python language was designed like this in the case of chained decorators. Please notice the nuance how this is different from the question
How decorators chaining work?
Link: How decorators chaining work? It seems quite a number of other users had the same doubts, about the call order of chained Python decorators. It is not like I can't add a __call__ and see the order for myself. I get this, my point is, why was it designed to start from the bottom, when it comes to chained Python decorators?
E.g.
def first_func(func):
def inner():
x = func()
return x * x
return inner
def second_func(func):
def inner():
x = func()
return 2 * x
return inner
#first_func
#second_func
def num():
return 10
print(num())
Quoting the documentation on decorators:
The decorator syntax is merely syntactic sugar, the following two function definitions are semantically equivalent:
def f(arg):
...
f = staticmethod(f)
#staticmethod
def f(arg):
...
From this it follows that the decoration in
#a
#b
#c
def fun():
...
is equivalent to
fun = a(b(c(fun)))
IOW, it was designed like that because it's just syntactic sugar.
For proof, let's just decorate an existing function and not return a new one:
def dec1(f):
print(f"dec1: got {vars(f)}")
f.dec1 = True
return f
def dec2(f):
print(f"dec2: got {vars(f)}")
f.dec2 = True
return f
#dec1
#dec2
def foo():
pass
print(f"Fully decked out: {vars(foo)}")
prints out
dec2: got {}
dec1: got {'dec2': True}
Fully decked out: {'dec2': True, 'dec1': True}
TL;DR
g(f(x)) means applying f to x first, then applying g to the output.
Omit the parentheses, add # before and line break after each function name:
#g
#f
x
(Syntax only valid if x is the definition of a function/class.)
Abstract explanation
The reasoning behind this design decision becomes fairly obvious IMHO, if you remember what the decorator syntax - in its most abstract and general form - actually means. So I am going to try the abstract approach to explain this.
It is all about syntax
To be clear here, the distinguishing factor in the concept of the "decorator" is not the object underneath it (so to speak) nor the operation it performs. It is the special syntax and the restrictions for it. Thus, a decorator at its core is nothing more than feature of Python grammar.
The decorator syntax requires a target to be decorated. Initially (see PEP 318) the target could only be function definitions; later class definitions were also allowed to be decorated (see PEP 3129).
Minimal valid syntax
Syntactically, this is valid Python:
def f(): pass
#f
class Target: pass # or `def target: pass`
However, this will (perhaps unsuprisingly) cause a TypeError upon execution. As has been reiterated multiple times here and in other posts on this platform, the above is equivalent to this:
def f(): pass
class Target: pass
Target = f(Target)
Minimal working decorator
The TypeError stems from the fact that f lacks a positional argument. This is the obvious logical restriction imposed by what a decorator is supposed to do. Thus, to achieve not only syntactically valid code, but also have it run without errors, this is sufficient:
def f(x): pass
#f
class Target: pass
This is still not very useful, but it is enough for the most general form of a working decorator.
Decoration is just application of a function to the target and assigning the output to the target's name.
Chaining functions ⇒ Chaining decorators
We can ignore the target and what it is or does and focus only on the decorator. Since it merely stands for applying a function, the order of operations comes into play, as soon as we have more than one. What is the order of operation, when we chain functions?
def f(x): pass
def g(x): pass
class Target: pass
Target = g(f(Target))
Well, just like in the composition of purely mathematical functions, this implies that we apply f to Target first and then apply g to the result of f. Despite g appearing first (i.e. further left), it is not what is applied first.
Since stacking decorators is equivalent to nesting functions, it seems obvious to define the order of operation the same way. This time, we just skip the parentheses, add an # symbol in front of the function name and a line break after it.
def f(x): pass
def g(x): pass
#g
#f
class Target: pass
But, why though?
If after the explanation above (and reading the PEPs for historic background), the reasoning behind the order of operation is still not clear or still unintuitive, there is not really any good answer left, other than "because the devs thought it made sense, so get used to it".
PS
I thought I'd add a few things for additional context based on all the comments around your question.
Decoration vs. calling a decorated function
A source of confusion seems to be the distinction between what happens when applying the decorator versus calling the decorated function.
Notice that in my examples above I never actually called target itself (the class or function being decorated). Decoration is itself a function call. Adding #f above the target is calling the f and passing the target to it as the first positional argument.
A "decorated function" might not even be a function
The distinction is very important because nowhere does it say that a decorator actually needs to return a callable (function or class). f being just a function means it can return whatever it wants. This is again valid and working Python code:
def f(x): return 3.14
#f
def target(): return "foo"
try:
target()
except Exception as e:
print(repr(e))
print(target)
Output:
TypeError("'float' object is not callable")
3.14
Notice that the name target does not even refer to a function anymore. It just holds the 3.14 returned by the decorator. Thus, we cannot even call target. The entire function behind it is essentially lost immediately before it is even available to the global namespace. That is because f just completely ignores its first positional argument x.
Replacing a function
Expanding this further, if we want, we can have f return a function. Not doing that seems very strange, considering it is used to decorate a function. But it doesn't have to be related to the target at all. Again, this is fine:
def bar(): return "bar"
def f(x): return bar
#f
def target(): return "foo"
print(target())
print(target is bar)
Output:
bar
True
It comes down to convention
The way decorators are actually overwhelmingly used out in the wild, is in a way that still keeps a reference to the target being decorated around somewhere. In practice it can be as simple as this:
def f(x):
print(f"applied `f({x.__name__})`")
return
#f
def target(): return "foo"
Just running this piece of code outputs applied f(target). Again, notice that we don't call target here, we only called f. But now, the decorated function is still target, so we could add the call print(target()) at the bottom and that would output foo after the other output produced by f.
The fact that most decorators don't just throw away their target comes down to convention. You (as a developer) would not expect your function/class to simply be thrown away completely, when you use a decorator.
Decoration with wrapping
This is why real-life decorators typically either return the reference to the target at the end outright (like in the last example) or they return a different callable, but that callable itself calls the target, meaning a reference to the target is kept in that new callable's local namespace . These functions are what is usually referred to as wrappers:
def f(x):
print(f"applied `f({x.__name__})`")
def wrapper():
print(f"wrapper executing with {locals()=}")
return x()
return wrapper
#f
def target(): return "foo"
print(f"{target()=}")
print(f"{target.__name__=}")
Output:
applied `f(target)`
wrapper executing with locals()={'x': <function target at 0x7f1b2f78f250>}
target()='foo'
target.__name__='wrapper'
As you can see, what the decorator left us is wrapper, not what we originally defined as target. And the wrapper is what we call, when we write target().
Wrapping wrappers
This is the kind of behavior we typically expect, when we use decorators. And therefore it is not surprising that multiple decorators stacked together behave the way they do. The are called from the inside out (as explained above) and each adds its own wrapper around what it receives from the one applied before:
def f(x):
print(f"applied `f({x.__name__})`")
def wrapper_from_f():
print(f"wrapper_from_f executing with {locals()=}")
return x()
return wrapper_from_f
def g(x):
print(f"applied `g({x.__name__})`")
def wrapper_from_g():
print(f"wrapper_from_g executing with {locals()=}")
return x()
return wrapper_from_g
#g
#f
def target(): return "foo"
print(f"{target()=}")
print(f"{target.__name__=}")
Output:
applied `f(target)`
applied `g(wrapper_from_f)`
wrapper_from_g executing with locals()={'x': <function f.<locals>.wrapper_from_f at 0x7fbfc8d64f70>}
wrapper_from_f executing with locals()={'x': <function target at 0x7fbfc8d65630>}
target()='foo'
target.__name__='wrapper_from_g'
This shows very clearly the difference between the order in which the decorators are called and the order in which the wrapped/wrapping functions are called.
After the decoration is done, we are left with wrapper_from_g, which is referenced by our target name in global namespace. When we call it, wrapper_from_g executes and calls wrapper_from_f, which in turn calls the original target.
I am attempting to write a test that checks if a variable holding the bound method of a class is the same as another reference to that method. Normally this is not a problem, but it does not appear to work when done within another method of the same class. Here is a minimal example:
class TestClass:
def sample_method(self):
pass
def test_method(self, method_reference):
print(method_reference is self.sample_method)
I am really using an assert instead of print, but that is neither here nor there since the end result is the same. The test is run as follows:
instance = TestClass()
instance.test_method(instance.sample_method)
The result is False even though I am expecting it to be True. The issue manifests itself in both Python 3.5 and Python 2.7 (running under Anaconda).
I understand that bound methods are closures that are acquired by doing something like TestClass.test_method.__get__(instance, type(instance)). However, I would expect that self.sample_method is already a reference to such a closure, so that self.sample_method and instance.sample_method represent the same reference.
Part of what is confusing me here is the output of the real pytest test that I am running (working on a PR for matplotlib):
assert <bound method TestTransformFormatter.transform1 of <matplotlib.tests.test_ticker.TestTransformFormatter object at 0x7f0101077b70>> is <bound method TestTransformFormatter.transform1 of <matplotlib.tests.test_ticker.TestTransformFormatter object at 0x7f0101077b70>>
E + where <bound method TestTransformFormatter.transform1 of <matplotlib.tests.test_ticker.TestTransformFormatter object at 0x7f0101077b70>> = <matplotlib.ticker.TransformFormatter object at 0x7f0101077e10>.transform
E + and <bound method TestTransformFormatter.transform1 of <matplotlib.tests.test_ticker.TestTransformFormatter object at 0x7f0101077b70>> = <matplotlib.tests.test_ticker.TestTransformFormatter object at 0x7f0101077b70>.transform1
If I understand the output correctly, the actual comparison (the first line) is really comparing the same objects, but somehow turning up False. The only thing I can imagine at this point is that __get__ is in fact being called twice, but I know neither why/where/how, nor how to work around it.
They're not the same reference - the objects representing the two methods occupy different locations in memory:
>>> class TestClass:
... def sample_method(self):
... pass
... def test_method(self, method_reference):
... print(hex(id(method_reference)))
... print(hex(id(self.sample_method)))
...
>>> instance = TestClass()
>>> instance.test_method(instance.sample_method)
0x7fed0cc561c8
0x7fed0cc4e688
Changing to method_reference == self.sample_method will make the assert pass, though.
Edit since question was expanded: seems like a flawed test - probably the actual functionality of the code does not require the references to be the same (is), just equal (==). So your change probably didn't break anything except for the test.
While the accepted answer is in no way incorrect, it seems like it should be noted that methods are bound on attribute lookup. Furthermore, the behavior of unbound methods changes between Python 2.X and Python 3.X.
class A:
def method(self):
pass
a = A()
print(a.method is a.method) # False
print(A.method is A.method) # Python 3.X: True, Python 2.X: False
I'm writing a Python script to parse some data. At the moment I'm trying to make a Class which creates "placeholder" objects. I intend to repeatedly pass each "placeholder" variables, and finally turn it into a dict, float, list or string. For reasons that would take a while to describe, it would be a lot easier if I could replace the instance by calling a method.
Here's a simplified example
class Event( dict ):
def __init__( self ):
self.sumA = 0.0
self.sumB = 0.0
def augmentA( self, i ):
self.sumA += i
def augmentB( self, i ):
self.sumB += i
def seal( self ):
if self.sumA != 0 and self.sumB != 0:
self = [ sumA, sumB ]
elif self.sumA != 0:
self = float( sumA )
elif self.sumB != 0:
self = float( sumB )
And what I want to do is:
e = Event()
e.augmentA( 1 )
e.augmentA( 2 )
e.seal()
...and have 'e' turn into a float.
What I am hoping to avoid is:
e = Event()
e.augmentA( 1 )
e.augmentA( 2 )
e = e.getSealedValue()
I totally understand that "self" in my "seal" method is just a local variable, and won't have any effect on the instance outside scope. I'm unsure however how to achieve what I want, from within the instance, where it would be most convenient for my code. I also understand I could override all the bulit-ins ( getItem, toStr ) but that complicates my code a lot.
I'm a Python noob so I'm unsure if this is even possible. Indulge me, please :)
Under some circunstances, Python allows you to change the class of an object on the fly. However, not any object can be converted to any class, as the example below demonstrates (newlines added for readability):
>>> class A(object):
... pass
...
>>> class B(object):
... pass
...
>>> a = A()
>>> type(a)
<class '__main__.A'>
>>> a.__class__ = B
>>> type(a)
<class '__main__.B'>
>>> a.__class__ = int
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __class__ assignment: only for heap types
(I don't know the exact rules from the top of my head, but if your classes uses __slots__ for instance they must be compatible for the conversion to be possible)
However, as other answerers pointed out, in general it's a very bad idea to do so, even if there was a way to convert every reference to one object to a reference to another. I wouldn't go as far as saying never do that though, there might be legitimate uses of this technique (for instance, I see it as an easy way of implementing the State Design Pattern without creating unnecessary clutter).
Even if there was some sane way to make it work, I would avoid going down this path simply because it changes the type of the object to something completely incompatible with its existing contract.
Now, one could say "but I'm only using it in one place, other code paths won't ever see the old contract", but unfortunately that isn't an argument for this mechanism since you could simply only make the value available for the other code paths once you have the final object.
In short, don't do this and don't even try.
No, you cannot have an variable value replace itself by calling a method on it. The normal way to do this would be what you stated: e = e.getSealedValue()
You can make an object change behavior, but that's generally considered a bad idea and is likely to result in highly unmaintainable code.
How can I write a lambda expression that's equivalent to:
def x():
raise Exception()
The following is not allowed:
y = lambda : raise Exception()
There is more than one way to skin a Python:
y = lambda: (_ for _ in ()).throw(Exception('foobar'))
Lambdas accept statements. Since raise ex is a statement, you could write a general purpose raiser:
def raise_(ex):
raise ex
y = lambda: raise_(Exception('foobar'))
But if your goal is to avoid a def, this obviously doesn't cut it. It does, however allow you to conditionally raise exceptions, e.g.:
y = lambda x: 2*x if x < 10 else raise_(Exception('foobar'))
Alternatively you can raise an exception without defining a named function. All you need is a strong stomach (and 2.x for the given code):
type(lambda:0)(type((lambda:0).func_code)(
1,1,1,67,'|\0\0\202\1\0',(),(),('x',),'','',1,''),{}
)(Exception())
And a python3 strong stomach solution:
type(lambda: 0)(type((lambda: 0).__code__)(
1,0,1,1,67,b'|\0\202\1\0',(),(),('x',),'','',1,b''),{}
)(Exception())
Thanks #WarrenSpencer for pointing out a very simple answer if you don't care which exception is raised: y = lambda: 1/0.
How about:
lambda x: exec('raise(Exception(x))')
I'd like to give an explanation of the UPDATE 3 of the answer provided by Marcelo Cantos:
type(lambda: 0)(type((lambda: 0).__code__)(
1,0,1,1,67,b'|\0\202\1\0',(),(),('x',),'','',1,b''),{}
)(Exception())
Explanation
lambda: 0 is an instance of the builtins.function class.
type(lambda: 0) is the builtins.function class.
(lambda: 0).__code__ is a code object.
A code object is an object which holds the compiled bytecode among other things.
It is defined here in CPython https://github.com/python/cpython/blob/master/Include/code.h.
Its methods are implemented here https://github.com/python/cpython/blob/master/Objects/codeobject.c.
We can run the help on the code object:
Help on code object:
class code(object)
| code(argcount, kwonlyargcount, nlocals, stacksize, flags, codestring,
| constants, names, varnames, filename, name, firstlineno,
| lnotab[, freevars[, cellvars]])
|
| Create a code object. Not for the faint of heart.
type((lambda: 0).__code__) is the code class.
So when we say
type((lambda: 0).__code__)(
1,0,1,1,67,b'|\0\202\1\0',(),(),('x',),'','',1,b'')
we are calling the constructor of the code object with the following arguments:
argcount=1
kwonlyargcount=0
nlocals=1
stacksize=1
flags=67
codestring=b'|\0\202\1\0'
constants=()
names=()
varnames=('x',)
filename=''
name=''
firstlineno=1
lnotab=b''
You can read about what the arguments mean in the definition of the PyCodeObject
https://github.com/python/cpython/blob/master/Include/code.h.
The value of 67 for the flags argument is for example CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE.
The most importand argument is the codestring which contains instruction opcodes.
Let's see what they mean.
>>> import dis
>>> dis.dis(b'|\0\202\1\0')
0 LOAD_FAST 0 (0)
2 RAISE_VARARGS 1
4 <0>
The documentation of opcodes can by found here
https://docs.python.org/3.8/library/dis.html#python-bytecode-instructions.
The first byte is the opcode for LOAD_FAST, the second byte is its argument i.e. 0.
LOAD_FAST(var_num)
Pushes a reference to the local co_varnames[var_num] onto the stack.
So we push the reference to x onto the stack. The varnames is a list of strings containing only 'x'.
We will push the only argument of the function we are defining to the stack.
The next byte is the opcode for RAISE_VARARGS and the next byte is its argument i.e. 1.
RAISE_VARARGS(argc)
Raises an exception using one of the 3 forms of the raise statement, depending on the value of argc:
0: raise (re-raise previous exception)
1: raise TOS (raise exception instance or type at TOS)
2: raise TOS1 from TOS (raise exception instance or type at TOS1 with __cause__ set to TOS)
The TOS is the top-of-stack.
Since we pushed the first argument (x) of our function to the stack and argc is 1 we will raise the
x if it is an exception instance or make an instance of x and raise it otherwise.
The last byte i.e. 0 is not used. It is not a valid opcode. It might as well not be there.
Going back to code snippet we are anylyzing:
type(lambda: 0)(type((lambda: 0).__code__)(
1,0,1,1,67,b'|\0\202\1\0',(),(),('x',),'','',1,b''),{}
)(Exception())
We called the constructor of the code object:
type((lambda: 0).__code__)(
1,0,1,1,67,b'|\0\202\1\0',(),(),('x',),'','',1,b'')
We pass the code object and an empty dictionary to the constructor of a function object:
type(lambda: 0)(type((lambda: 0).__code__)(
1,0,1,1,67,b'|\0\202\1\0',(),(),('x',),'','',1,b''),{}
)
Let's call help on a function object to see what the arguments mean.
Help on class function in module builtins:
class function(object)
| function(code, globals, name=None, argdefs=None, closure=None)
|
| Create a function object.
|
| code
| a code object
| globals
| the globals dictionary
| name
| a string that overrides the name from the code object
| argdefs
| a tuple that specifies the default argument values
| closure
| a tuple that supplies the bindings for free variables
We then call the constructed function passing an Exception instance as an argument.
Consequently we called a lambda function which raises an exception.
Let's run the snippet and see that it indeed works as intended.
>>> type(lambda: 0)(type((lambda: 0).__code__)(
... 1,0,1,1,67,b'|\0\202\1\0',(),(),('x',),'','',1,b''),{}
... )(Exception())
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "", line 1, in
Exception
Improvements
We saw that the last byte of the bytecode is useless. Let's not clutter this
complicated expression needlesly. Let's remove that byte.
Also if we want to golf a little we could omit the instantiation of Exception
and instead pass the Exception class as an argument. Those changes would result
in the following code:
type(lambda: 0)(type((lambda: 0).__code__)(
1,0,1,1,67,b'|\0\202\1',(),(),('x',),'','',1,b''),{}
)(Exception)
When we run it we will get the same result as before. It's just shorter.
Actually, there is a way, but it's very contrived.
You can create a code object using the compile() built-in function. This allows you to use the raise statement (or any other statement, for that matter), but it raises another challenge: executing the code object. The usual way would be to use the exec statement, but that leads you back to the original problem, namely that you can't execute statements in a lambda (or an eval(), for that matter).
The solution is a hack. Callables like the result of a lambda statement all have an attribute __code__, which can actually be replaced. So, if you create a callable and replace it's __code__ value with the code object from above, you get something that can be evaluated without using statements. Achieving all this, though, results in very obscure code:
map(lambda x, y, z: x.__setattr__(y, z) or x, [lambda: 0], ["__code__"], [compile("raise Exception", "", "single"])[0]()
The above does the following:
the compile() call creates a code object that raises the exception;
the lambda: 0 returns a callable that does nothing but return the value 0 -- this is used to execute the above code object later;
the lambda x, y, z creates a function that calls the __setattr__ method of the first argument with the remaining arguments, AND RETURNS THE FIRST ARGUMENT! This is necessary, because __setattr__ itself returns None;
the map() call takes the result of lambda: 0, and using the lambda x, y, z replaces it's __code__ object with the result of the compile() call. The result of this map operation is a list with one entry, the one returned by lambda x, y, z, which is why we need this lambda: if we would use __setattr__ right away, we would lose the reference to the lambda: 0 object!
finally, the first (and only) element of the list returned by the map() call is executed, resulting in the code object being called, ultimately raising the desired exception.
It works (tested in Python 2.6), but it's definitely not pretty.
One last note: if you have access to the types module (which would require to use the import statement before your eval), then you can shorten this code down a bit: using types.FunctionType() you can create a function that will execute the given code object, so you won't need the hack of creating a dummy function with lambda: 0 and replacing the value of its __code__ attribute.
If all you want is a lambda expression that raises an arbitrary exception, you can accomplish this with an illegal expression. For instance, lambda x: [][0] will attempt to access the first element in an empty list, which will raise an IndexError.
PLEASE NOTE: This is a hack, not a feature. Do not use this in any (non code-golf) code that another human being might see or use.
Functions created with lambda forms cannot contain statements.
Every time I have wanted to do this, it was in a test where I wanted to assert that a function was not called.
For this use case, I found it clearer to use a mock with a side effect
from unittest.mock import Mock
MyClass.my_method = Mock(side_effect=AssertionError('we should not reach this method call')
It would also work in other settings, but I'd rather not depend on unittest in my main application
All the solutions above do work but I think this one is the shortest in case you just need any function that raises a random exception:
lambda: 0/0
et voila!