def func():
def nested():
global x
x = 1
x = 2
func()
print(x)
The correct answer would be '2' and the reason why is because func() is not defined. But when I read this it seems that func() is defined as nested(). I would think that when you call func() that would then automatically call nested(). I'm struggling to grasp this and understand why I shouldn't read it that way.
You're defining nested inside func, but you're not calling nested() anywhere in func, so when you call func() it's effectively doing nothing.
To do what you want try defining func as:
def func():
def nested():
global x
x = 1
nested()
UPDATE: After StevenRumbalski's comment, I think a small addition about what exactly is going on in that function can help clarify things around.
Python's functions are themselves objects which can respond to the operator ().
When you define a new function, what you're actually doing is instantiating a function object and giving it a name. In the example above, def func() creates a function instance and gives it name func, so that when you apply operator () to func (i.e. when you call the function with func()) the code of the function associated to that name is executed.
Let's now take one step further and look at what happens with nested.
Nested is defined inside func's scope, so when you exit func's scope the name nested is not defined anymore.
If, however, you return the function object to the caller of func, you can use that object to run nested's code.
A small example:
def returning_func():
def nested():
print("I am nested!")
return nested # note the lack of the () operator!
In this case, if we do
my_reference_to_nested = func()
nothing gets printed, because nested is defined but not executed.
If we call it, however:
my_reference_to_nested()
we execute nested's code and print to the output I am nested!
Related
To start with, my question here is about the semantics and the logic behind why the Python language was designed like this in the case of chained decorators. Please notice the nuance how this is different from the question
How decorators chaining work?
Link: How decorators chaining work? It seems quite a number of other users had the same doubts, about the call order of chained Python decorators. It is not like I can't add a __call__ and see the order for myself. I get this, my point is, why was it designed to start from the bottom, when it comes to chained Python decorators?
E.g.
def first_func(func):
def inner():
x = func()
return x * x
return inner
def second_func(func):
def inner():
x = func()
return 2 * x
return inner
#first_func
#second_func
def num():
return 10
print(num())
Quoting the documentation on decorators:
The decorator syntax is merely syntactic sugar, the following two function definitions are semantically equivalent:
def f(arg):
...
f = staticmethod(f)
#staticmethod
def f(arg):
...
From this it follows that the decoration in
#a
#b
#c
def fun():
...
is equivalent to
fun = a(b(c(fun)))
IOW, it was designed like that because it's just syntactic sugar.
For proof, let's just decorate an existing function and not return a new one:
def dec1(f):
print(f"dec1: got {vars(f)}")
f.dec1 = True
return f
def dec2(f):
print(f"dec2: got {vars(f)}")
f.dec2 = True
return f
#dec1
#dec2
def foo():
pass
print(f"Fully decked out: {vars(foo)}")
prints out
dec2: got {}
dec1: got {'dec2': True}
Fully decked out: {'dec2': True, 'dec1': True}
TL;DR
g(f(x)) means applying f to x first, then applying g to the output.
Omit the parentheses, add # before and line break after each function name:
#g
#f
x
(Syntax only valid if x is the definition of a function/class.)
Abstract explanation
The reasoning behind this design decision becomes fairly obvious IMHO, if you remember what the decorator syntax - in its most abstract and general form - actually means. So I am going to try the abstract approach to explain this.
It is all about syntax
To be clear here, the distinguishing factor in the concept of the "decorator" is not the object underneath it (so to speak) nor the operation it performs. It is the special syntax and the restrictions for it. Thus, a decorator at its core is nothing more than feature of Python grammar.
The decorator syntax requires a target to be decorated. Initially (see PEP 318) the target could only be function definitions; later class definitions were also allowed to be decorated (see PEP 3129).
Minimal valid syntax
Syntactically, this is valid Python:
def f(): pass
#f
class Target: pass # or `def target: pass`
However, this will (perhaps unsuprisingly) cause a TypeError upon execution. As has been reiterated multiple times here and in other posts on this platform, the above is equivalent to this:
def f(): pass
class Target: pass
Target = f(Target)
Minimal working decorator
The TypeError stems from the fact that f lacks a positional argument. This is the obvious logical restriction imposed by what a decorator is supposed to do. Thus, to achieve not only syntactically valid code, but also have it run without errors, this is sufficient:
def f(x): pass
#f
class Target: pass
This is still not very useful, but it is enough for the most general form of a working decorator.
Decoration is just application of a function to the target and assigning the output to the target's name.
Chaining functions ⇒ Chaining decorators
We can ignore the target and what it is or does and focus only on the decorator. Since it merely stands for applying a function, the order of operations comes into play, as soon as we have more than one. What is the order of operation, when we chain functions?
def f(x): pass
def g(x): pass
class Target: pass
Target = g(f(Target))
Well, just like in the composition of purely mathematical functions, this implies that we apply f to Target first and then apply g to the result of f. Despite g appearing first (i.e. further left), it is not what is applied first.
Since stacking decorators is equivalent to nesting functions, it seems obvious to define the order of operation the same way. This time, we just skip the parentheses, add an # symbol in front of the function name and a line break after it.
def f(x): pass
def g(x): pass
#g
#f
class Target: pass
But, why though?
If after the explanation above (and reading the PEPs for historic background), the reasoning behind the order of operation is still not clear or still unintuitive, there is not really any good answer left, other than "because the devs thought it made sense, so get used to it".
PS
I thought I'd add a few things for additional context based on all the comments around your question.
Decoration vs. calling a decorated function
A source of confusion seems to be the distinction between what happens when applying the decorator versus calling the decorated function.
Notice that in my examples above I never actually called target itself (the class or function being decorated). Decoration is itself a function call. Adding #f above the target is calling the f and passing the target to it as the first positional argument.
A "decorated function" might not even be a function
The distinction is very important because nowhere does it say that a decorator actually needs to return a callable (function or class). f being just a function means it can return whatever it wants. This is again valid and working Python code:
def f(x): return 3.14
#f
def target(): return "foo"
try:
target()
except Exception as e:
print(repr(e))
print(target)
Output:
TypeError("'float' object is not callable")
3.14
Notice that the name target does not even refer to a function anymore. It just holds the 3.14 returned by the decorator. Thus, we cannot even call target. The entire function behind it is essentially lost immediately before it is even available to the global namespace. That is because f just completely ignores its first positional argument x.
Replacing a function
Expanding this further, if we want, we can have f return a function. Not doing that seems very strange, considering it is used to decorate a function. But it doesn't have to be related to the target at all. Again, this is fine:
def bar(): return "bar"
def f(x): return bar
#f
def target(): return "foo"
print(target())
print(target is bar)
Output:
bar
True
It comes down to convention
The way decorators are actually overwhelmingly used out in the wild, is in a way that still keeps a reference to the target being decorated around somewhere. In practice it can be as simple as this:
def f(x):
print(f"applied `f({x.__name__})`")
return
#f
def target(): return "foo"
Just running this piece of code outputs applied f(target). Again, notice that we don't call target here, we only called f. But now, the decorated function is still target, so we could add the call print(target()) at the bottom and that would output foo after the other output produced by f.
The fact that most decorators don't just throw away their target comes down to convention. You (as a developer) would not expect your function/class to simply be thrown away completely, when you use a decorator.
Decoration with wrapping
This is why real-life decorators typically either return the reference to the target at the end outright (like in the last example) or they return a different callable, but that callable itself calls the target, meaning a reference to the target is kept in that new callable's local namespace . These functions are what is usually referred to as wrappers:
def f(x):
print(f"applied `f({x.__name__})`")
def wrapper():
print(f"wrapper executing with {locals()=}")
return x()
return wrapper
#f
def target(): return "foo"
print(f"{target()=}")
print(f"{target.__name__=}")
Output:
applied `f(target)`
wrapper executing with locals()={'x': <function target at 0x7f1b2f78f250>}
target()='foo'
target.__name__='wrapper'
As you can see, what the decorator left us is wrapper, not what we originally defined as target. And the wrapper is what we call, when we write target().
Wrapping wrappers
This is the kind of behavior we typically expect, when we use decorators. And therefore it is not surprising that multiple decorators stacked together behave the way they do. The are called from the inside out (as explained above) and each adds its own wrapper around what it receives from the one applied before:
def f(x):
print(f"applied `f({x.__name__})`")
def wrapper_from_f():
print(f"wrapper_from_f executing with {locals()=}")
return x()
return wrapper_from_f
def g(x):
print(f"applied `g({x.__name__})`")
def wrapper_from_g():
print(f"wrapper_from_g executing with {locals()=}")
return x()
return wrapper_from_g
#g
#f
def target(): return "foo"
print(f"{target()=}")
print(f"{target.__name__=}")
Output:
applied `f(target)`
applied `g(wrapper_from_f)`
wrapper_from_g executing with locals()={'x': <function f.<locals>.wrapper_from_f at 0x7fbfc8d64f70>}
wrapper_from_f executing with locals()={'x': <function target at 0x7fbfc8d65630>}
target()='foo'
target.__name__='wrapper_from_g'
This shows very clearly the difference between the order in which the decorators are called and the order in which the wrapped/wrapping functions are called.
After the decoration is done, we are left with wrapper_from_g, which is referenced by our target name in global namespace. When we call it, wrapper_from_g executes and calls wrapper_from_f, which in turn calls the original target.
I'm learning the lambda options to return in python and i have a question:
I need to fill the returns in this function:
def func(n):
if n==0:
print("finished")
else:
return ___
func(5)()()()()()
func(3)()()()
func(8)()()()()()()()()
The output:
finished
finished
finished
I thought this one is a recursive call like return func(n-1) but it doesn't work, and throws an error.
Is there an option to overcome the extra empty brackets? count them? do something, because it should be runnable.
Thanks
You're right about needing to use lambdas and func n-1, specifically
return lambda: func(n-1)
This returns a lambda that doesn't need any parameters passed in, to handle the brackets, and the return of the is the function being called with n-1, which in most calls you're making, is returning the next lambda function call
The operator () is the function call.
So for example:
def foo():
print('Foo executed')
def bar():
# We return the foo FUNCTION itself, without calling it.
return foo
In the above code, if you execute foo(), it'll print "Foo executed". But if you execute bar(), it will return foo, which is the function. So you can execute bar()(), where the first () is executing the function bar, returning foo, and then with the second (), you call the returned foo function.
Edit: When I typed it, you removed the what are those bunch of () because they are new to you... But I just leave it there maybe it'll help.
def test():
x = 99
def nested(x):
print (x)
return nested
a = test()
a()
TypeError: nested() missing 1 required positional argument: 'x'
When I call nested to print argument x assigned in closure function test, the TypeError prompts me to pass a positional argument into nested, but why 'x' assigned in test function didn't be passed into nested?
def test():
x = 99
def nested():
print (x)
return nested
a = test()
a()
The function namespace is inherited automatically, you don’t have to pass it as an argument. If you include an argument into the function definition, then you have to obligatorily pass it a value when you call the function (unless you set a default value, for example:)
def test():
x = 99
def nested(y=100):
print (x,y)
return nested
a = test()
a() # will prinnt 99,100 because x is already inheritted in the namespace,
# and 100 is the fallback value
For the code you provide, I understand you kind of get the functioning of nested functions, but you fail to grasp the idea of namespaces. To tell from a namespace, a scope and a function/method argument, I recommend you some reading:
A scope refers to a region of a program where a namespace can be
directly accessed, i.e. without using a namespace prefix. In other
words: The scope of a name is the area of a program where this name
can be unambiguously used, for example inside of a function. A name's
namespace is identical to it's scope. Scopes are defined statically,
but they are used dynamically.
https://www.python-course.eu/namespaces.php
You may find this other link useful:
https://www.quora.com/What-is-the-difference-between-parameters-and-arguments-in-Python however best is always read Python docs for a precise and complete understanding ;)
When you do a = test(), you have called the test() function, and got the nested function. This function needs an argument, so you can't call it by a() alone (it's like calling it by nested()). You would need to modify this to def nested() for your code to work, and, more importantly, to see that the function "remembers" the x defined outside.
A more typical example would be
def test(x):
def nested(y):
print(f'x is {x}, y is {y}')
return nested
a = test(5)
a(12)
I'm a python beginner and I struggle with a somewhat easy concept of functional programming. I simply can not see why the nested function (see example below) would even be called. This is quite hard to explain.
func3 = func1(func2)
Here I call func1 with the positional argument func2. All fair and well. func1 returns a nested function. But when I run the code the function nest1has been executed even though I've only called func1 (I dont see why this should execute the nested function). func1 should not do more than returning a function without printing anything. Can someone explain this to me?
def func2():
def nest2():
print('Nest2')
nest2()
return nest2
def func1(func):
def nest1():
print('Nest1')
func()
return nest1
func3 = func1(func2)
func3()
If I call a simple function with a nested function the inner function is not executed. See below:
def new_func():
def inner():
print(1)
return inner
new_func()
func1() returns the method nest1 (without executing it), so your call func3 = func1(func2) returns nest1 and assigns it to func3.
The next line you are executing whatever is assigned to func3 - which is the method nest1. In other words, the line func3() is executing the function nest1, and causing the text to print.
But when I run the code the function nest1 has been executed even though I've only called func1 (I dont see why this should execute the nested function). Func 1 should not do more than returning a function without printing anything. Can someone explain this to me?
When you do this:
func3 = func1(func2)
You pass the function func2 into the function func1. Inside of func1, you define a function nest1 that calls the function passed in. This function is then returned and becomes the value of func3. The only thing func1 does is create the nest1 function, and return it. When func3 is called, you actually called the function defined inside of func1, nest1.
If I call a simple function with a nested function the inner function is not executed.
The reason your second example did not work as expected, is because calling new_func does not call inner, it only creates inner and returns it. This is the same case as above. Calling func1 did not call nest1, it simply created nest1 and returned it. You had to explitly call nest1 (func3()). Likewise, you need to call the function returned from new_func explicitly:
new_func()()
In the final example, you need to call the function that is being returned with an extra ():
def new_func():
def inner():
print(1)
return inner
new_func()()
Output:
1
Body of func2 is:
def nest2():
print('Nest2')
nest2()
return nest2
And so body of func1(func2) is:
print('Nest1')
def nest2():
print('Nest2')
nest2()
return nest2
The last piece we get by replacing func with body of func2 in body of nest1.
I'm trying to understand how decorators work, and was wondering if a decorated function can access variables of the decorator. For example, in the following code, how do I make f1 have access to localVariable? Is that possible, and is it even a good way of doing things?
def funcDec(func):
localVariable = "I'm a local string"
def func2Return(*args):
print "Calling localVariable from decorator " + localVariable
func(*args)
print "done with calling f1"
return func2Return
#funcDec
def f1(x, y):
print x + y
print localVariable
f1(2, 3)
Because of Python's scoping rules, a decorated function generally can't access any variables in the decorator. However, since functions can have arbitrary attributes assigned to them, you could do something like the following in the decorator to get a similar effect (due to the same scoping rules):
def funcDec(func):
localVariable = "I'm a local string"
def wrapped(*args):
print("Calling localVariable from funcDec " + localVariable)
func(*args)
print("done with calling f1")
wrapped.attrib = localVariable
return wrapped
#funcDec
def f1(x, y):
print(x + y)
print('f1.attrib: {!r}'.format(f1.attrib))
f1(2, 3)
Which would produce the following output:
Calling localVariable from funcDec I'm a local string
5
f1.attrib: "I'm a local string"
done with calling f1
Someone asked whether this could be applied to methods of a class:
The answer is "yes", but you have to reference the method either through the class itself or the instance of it passed as the self argument. Both techniques are shown below. Using self is preferable since it makes the code independent of the name of the class it's is in.
class Test(object):
#funcDec
def f1(self):
print('{}.f1() called'.format(self.__class__.__name__))
print('self.f1.attrib: {!r}'.format(self.f1.attrib)) # Preferred.
print('Test.f1.attrib: {!r}'.format(Test.f1.attrib)) # Also works.
print()
test = Test()
test.f1()
Output:
Calling localVariable from funcDec I'm a local string
Test.f1() called
self.f1.attrib: "I'm a local string"
Test.f1.attrib: "I'm a local string"
done with calling f1
Update
Another way of doing this that would give the decorated function more direct access to decorator variables would be to temporarily "inject" them into its global namespace (and then remove them afterwards).
I got the idea from #Martijn Pieters' answer to the somewhat related question: How to inject variable into scope with a decorator?
def funcDec(func):
localVariable = "I'm a local string"
# Local variable(s) to inject into wrapped func.
context = {'localVariable': localVariable}
def wrapped(*args):
func_globals = func.__globals__
# Save copy of any global values that will be replaced.
saved_values = {key: func_globals[key] for key in context if key in func_globals}
func_globals.update(context)
print(f'Calling localVariable from funcDec: {localVariable!r}')
try:
func(*args)
finally:
func_globals.update(saved_values) # Restore any replaced globals.
print(f'done with calling {func.__name__}()')
return wrapped
#funcDec
def f1(x, y):
print(x + y)
print(f'Calling funcDec localVariable from f1: {localVariable!r}')
f1(2, 3)
Result from this version:
Calling localVariable from funcDec: "I'm a local string"
5
Calling funcDec localVariable from f1: "I'm a local string"
done with calling f1()
I think it helps if you keep in mind that a decorator
#deco
def f(...): ...
is just syntactic sugar for
def f(...): ...
f = deco(f)
rather than some kind of macro expansion. In Python the scope of a variable is determined statically, so for a global (module-level) function a variable that is neither passed as an argument nor assigned to will be looked up in the global namespace.
Therefore you have to pass on a local variable of func2Return() explicitly. Change the signature of f1 to f1(x, y, localvariable=None) and have the wrapper function fun2Return call it with
f1(*args, localvariable=localvariable)
No, you can't. See this previous question. Just because the function is a decorator doesn't mean functions it calls have special access to its variables. If you do this:
def func():
a = 2
otherFunc()
Then otherFunc doesn't have access to the variable a. That's how it works for all function calls, and it's how it works for decorators too.
Now, the wrapper function you define inside the decorator (func2Return in your example) does have access to the variables, because that function is lexically in the same scope as those variables. So your line print "Calling localVariable from decorator " + localVariable will work. You can use this to some extent to wrap the decorated function with behavior that depends on variables in the decorator. But the function actually being decorated (f1 in your example) doesn't have access to those variables.
A function only has access to local variables from the scope where the function definition actually is. Functions don't get variables from calling scopes. (This is a good thing. If they did, it would be a huge mess.)
I found this question trying to use variables in (I think) the opposite direction, and found the answer from Martineau to apply in that opposite direction. That is, I am using a decorator to wrap some standard error handling around a series of similar functions but want to pass some of that error info up since that's where my logging is.
So, I was able to do something like this:
def error_handler(func):
def wrapper_decorator(*args,**kwargs):
try:
func(*args,**kwargs)
except Exception as e: # I actually handle things better than a general grab here
logger.error(f'func had bad value {error_handler.value}')
return wrapper_decorator
#error_handler
def main():
# blah blah, do stuff, read a dataframe from a web page
error_handler.value = value # value is created within this main() func
# blah blah, do more stuff where we actually end up having an error
This sets the .value attribute on error_handler to a value that is generated within the scope of main(), effectively letting me 'pass up' that value.
In my actual specific example, I'm using a dataframe and getting errors when I save to serve because the data is poorly formatted. main() reads in the dataframe, transforms it, and (tries) to save to server. This setup allows me to pass the dataframe (as the value in the example code) to my actual error_handler function and then save the dataframe as a csv so I can inspect it, and I don't have to have the saving of the dataframe within each individual function.