I came across an interesting issue while trying to achieve dynamic sort.
Given the following code:
>>> l = []
>>> for i in range(2):
>>> def f():
>>> return f.v
>>> f.v = i
>>> l.append(f)
You have to be careful about how to use the functions in l:
>>> l[0]()
1
>>> l[1]()
1
>>> [h() for h in l]
[1, 1]
>>> [f() for f in l]
[0, 1]
>>> f = l[0]
>>> f()
0
>>> k = l[1]
>>> k()
0
>>> f = l[1]
>>> k()
1
>>> del f
>>> k()
NameError: global name 'f' is not defined
The behavior of the function depends on what f currently is.
What should I do to avoid this issue? How can I set a function attribute that does not depends on the function's name?
Update
Reading your comments and answers, here is my actual problem.
I have some data that I want to sort according to user input (so I don't know sorting criteria in advance). User can choose on which part of the data to apply successive sorts, and these sorts can be ascending or descending.
So my first try was to loop over the user inputs, define a function for each criterion, store this function in a list and then use this list for sorted's key like this: key=lambda x: [f(x) for f in functions]. To avoid multiplying conditions into functions themselves, I was computing some needed values before the function definition and binding them to the function (different functions with different pre-computed values).
While debugging, I understood that function attribute was not the solution here, so I indeed wrote a class with a __call__ method.
The issue is due to the fact that return f.v loads the global f, and not the one you intend.1 You can see this by disassembling the code:
>>> dis.dis(l[0])
3 0 LOAD_GLOBAL 0 (f)
3 LOAD_ATTR 1 (v)
6 RETURN_VALUE
After the loop that populates l, f is a reference to the last closure created, as you can see here:
>>> l
[<function f at 0x02594170>, <function f at 0x02594130>]
>>> f
<function f at 0x02594130>
Thus, when you call l[0](), it still loads the f that points to the last function created, and it returns 1. When you redefined f by doing f = l[0], then the global f now points to the first function.
What you seem to want is a function that has a state, which really is a class. You could therefore do something like this:
class MyFunction:
def __init__(self, v):
self.v = v
def __call__(self):
return self.v
l = [MyFunction(i) for i in range(2)]
l[0]() # 0
l[1]() # 1
Though it may be a good idea to explain your actual problem first, as there might be a better solution.
1: Why doesn't it load the global f and not the current instance, you may ask?
Recall that when you create a class, you need to pass a self argument, like so:
# ...
def my_method(self):
return self.value
self is actually a reference to the current instance of your object. That's how Python knows where to load the attribute value. It knows it has to look into the instance referenced by self. So when you do:
a.value = 1
a.my_method()
self is now a reference to a.
So when you do:
def f():
return f.v
There's no way for Python to know what f actually is. It's not a parameter, so it has to load it from elsewhere. In your case, it's loaded from the global variables.
Thus, when you do f.v = i, while you do set an attribute v for the instance of f, there's no way to know which instance you are referring to in the body of your function.
Note that what you are doing here:
def f():
return f.v
is not making a function which returns whatever its own v attribute is. It's returning whatever the f object's v attribute is. So it necessarily depends on the value of f. It's not that your v attribute "depends on the function's name". It really has nothing at all to do with the function's name.
Later, when you do
>>> f = l[0]
>>> k = l[1]
>>> k()
0
What you have done is bound k to the function at l[1]. When you call it, you of course get f.v, because that's what the function does.
But notice:
>>> k.v
1
>>> [h.v for h in l]
[0, 1]
So, a function is an object, and just like most objects, it can have attributes assigned to it (which you can access using dot notation, or the getattr() function, or inspecting the object's dictionary, etc.). But a function is not designed to access its own attributes from within its own code. For that, you want to use a class (as demonstrated by #VincentSavard).
In your particular case, the effect you seem to be after doesn't really need an "attribute" per se; you are apparently looking for a closure. You can implement a closure using a class, but a lighter-weight way is a nested function (one form of which is demonstrated by #TomKarzes; you could also use a named inner function instead of lambda).
Try this:
l = []
for i in range(2):
def f(n):
return lambda: n
l.append(f(i))
This doesn't use attributes, but creates a closure for each value of i. The value of n is then locked once f returns. Here's some sample output:
>>> [f() for f in l]
[0, 1]
As others said, return f.v looks for f name in the current scope which is equal to the last defined function.
To work around this you can simulate functions:
>>> class Function(object):
... def __init__(self, return_value):
... self.return_value = return_value
... def __call__(self):
... return self.return_value
...
>>> l = []
>>> for i in range(2):
... l.append(Function(i))
...
>>> l[0]()
>>> 0
>>> l[1]()
>>> 1
Related
I saw this below piece of code in a tutorial and wondering how it works.
Generally, the lambda takes a input and returns something but here it does not take anything and still it works.
>>> for i in range(3):
... a.append(lambda:i)
...
>>> a
[<function <lambda> at 0x028930B0>, <function <lambda> at 0x02893030>, <function
<lambda> at 0x028930F0>]
lambda:i defines the constant function that returns i.
Try this:
>>> f = lambda:3
>>> f()
You get the value 3.
But there's something more going on. Try this:
>>> a = 4
>>> g = lambda:a
>>> g()
gives you 4. But after a = 5, g() returns 5. Python functions "remember" the environment in which they're executed. This environment is called a "closure". By modifying the data in the closure (e.g. the variable a in the second example) you can change the behavior of the functions defined in that closure.
In this case a is a list of function objects defined in the loop.
Each of which will return 2.
>>> a[0]()
2
To make these function objects remember i values sequentially you should rewrite the code to
>>> for i in range(3):
... a.append(lambda x=i:x)
...
that will give you
>>> a[0]()
0
>>> a[1]()
1
>>> a[2]()
2
but in this case you get side effect that allows you to not to use remembered value
>>> a[0](42)
42
I'm not sure what you mean by "it works". It appears that it doesn't work at all. In the case you have presented, i is a global variable. It changes every time the loop iterates, so after the loop, i == 2. Now, since each lambda function simply says lambda:i each function call will simply return the most recent value of i. For example:
>>> a = []
>>> for i in range(3):
a.append(lambda:1)
>>> print a[0]()
2
>>> print a[1]()
2
>>> print a[2]()
In other words, this does not likely do what you expect it to do.
lambda defines an anonymous inline function. These functions are limited compared to the full functions you can define with def - they can't do assignments, and they just return a result. However, you can run into interesting issues with them, as defining an ordinary function inside a loop is not common, but lambda functions are often put into loops. This can create closure issues.
The following:
>>> a = []
>>> for i in range(3):
... a.append(lambda:i)
adds three functions (which are first-class objects in Python) to a. These functions return the value of i. However, they use the definition of i as it existed at the end of the loop. Therefore, you can call any of these functions:
>>> a[0]()
2
>>> a[1]()
2
>>> a[2]()
2
and they will each return 2, the last iteration of the range object. If you want each to return a different number, use a default argument:
>>> for i in range(3):
... a.append(lambda i=i:i)
This will forcibly give each function an i as it was at that specific point during execution.
>>> a[0]()
0
>>> a[1]()
1
>>> a[2]()
2
Of course, since we're now able to pass an argument to that function, we can do this:
>>> b[0](5)
5
>>> b[0](range(3))
range(0, 3)
It all depends on what you're planning to do with it.
I wanna know if I can prevent my function to work through all its routine if I'm only interested in one (or less than total) of the variables it returns.
To elucidate, suppose I have a function with (a tuple of) multiple returns:
def func_123(n):
x=n+1
y=n+2
z=n+3
return x,y,z
If I'm only interested in the third values, I can just do:
_,_,three = func_123(0)
But I wanna know how it works in the function.
Does my function performs of three calculations and only then chooses to 'drop' the first two and give me the one i want or does it recognise it can do a lot less work if it only performs the subroutines needed to return the value i want? If the first, is there a way around this (besides, of course, creating functions for each calculation and let go of an unique function to organize all subroutines)?
It will calculate, and return, all of the values. For example
def foo(x):
return x+1, x+2
When I call this function
>>> foo(1)
(2, 3)
>>> _, a = foo(1)
>>> a
3
>>> _
2
Note that _ is a perfectly valid, and usable, variable name. It is just used by convention to imply that you do not wish to use that variable.
The closest thing to what you are describing would be to write your function as a generator. For example
def func_123(n):
for i in range(1,4):
yield n + i
>>> a = func_123(1)
>>> next(a)
2
>>> next(a)
3
>>> next(a)
4
In this way, the values are evaluated and returned lazily, or in other words only when they are needed. In this way, you could craft your function so they return in the order that you want.
It doesn't "choose" or "drop" anything. What you're using is tuple assignment; specifically, you're assigning the return value to the tuple (_,_,three). The _ variable is just a convention for a "throw away" variable.
I would like to try something differently using functools builtin module (this may not be exactly what you are looking for but you can rethink of what you are doing.)
>>> import functools
>>> def func_123(n, m):
... return n + m
...
>>> func_dict = dict()
>>> for r in [1,2,3]:
... func_dict[r] = functools.partial(func_123, r)
...
>>> for k in [1,2,3]:
... func_dict[k](10)
...
11
12
13
>>> func_dict[3](20)
23
>>>
OR
>>> func_1 = functools.partial(func_123, 1)
>>> func_2 = functools.partial(func_123, 2)
>>> func_3 = functools.partial(func_123, 3)
>>> func_1(5)
6
>>> func_2(5)
7
>>> func_3(5)
8
>>> func_3(3)
6
>>>
So, you don't need to worry about returning output in tuple and selecting the values you want.
It's only a convention to use _ for unused variables.So all the statements in the function do get evaluated.
For example:
def func(a):
# how to get the name "x"
x = 1
func(x)
If I use inspect module I can get the stack frame object:
import inspect
def func(a):
print inspect.stack()
out:
[
(<frame object at 0x7fb973c7a988>, 'ts.py', 9, 'func', [' stack = inspect.stack()\n'], 0)
(<frame object at 0x7fb973d74c20>, 'ts.py', 18, '<module>', ['func(x)\n'], 0)
]
or use inspect.currentframe() I can get the current frame.
But I can only get the name "a" by inspect the function object.
Use inspect.stack I can get the call stack:"['func(x)\n']",
How can I get the name of actual parameters("x" here) when call the func by parsing the "['func(x)\n']"?
If I call func(x) then I get "x"
if I call func(y) then I get "y"
Edit:
A example:
def func(a):
# Add 1 to acttual parameter
...
x = 1
func(x)
print x # x expected to 2
y = 2
func(y)
print y # y expected to 3
Looking at your comment explanations, the reason you are trying to do this is:
Because I want to impelment Reference Parameters like in C++
You can't do that. In python, everything is passed by value, but that value is a reference to the object you pass. Now, if that object is mutable (like a list), you can modify it, but if it's immutable, a new object is created and set. In your code example, x is an int which is immutable, so if you want to change the value of x it, just return its new value from the function you are invoking:
def func(a):
b = 5 * a
# ... some math
return b
x = 1
x = func(x)
So what if you want to return multiple values? Use a tuple!
def func(a, b):
a, b = 5 * b, 6 * a
# ...
return b, a
a, b = 2, 3
a, b = func(a, b)
That is feasible, up to a point - but nonetheless,useless in any kind of "real code".
You can use it for backyard magic tricks:
Just retrieve the calling stack_frame like you are doing,
then loop linearly through the variables available in the calling frame for
one that references the same object you got as a parameter. The variables are in the
f_locals and f_globals dictionaries which are attributes of the frame object.
Of course, the parameter passed may not be in a variable name to start with:
it might be a constant in the caller, or it might be inside a dict or a list.
Either way, as #Nasser put it in the answer: it is not the way Python works. Objects exist in memory, and variable names, in each scope, just point to those objects. The names themselves are meaningless otherwise.
import inspect
from itertools import chain
def magic(x):
# note that in Python2, 'currentframe' could get a parameter
# to indicate which frame you want on the stack. Not so in Python3
fr = inspect.currentframe().f_back
for var_name, value in chain(fr.f_locals.items(), fr.f_globals.items()):
if value is x:
print ("The variable name in the caller context is {}".format(var_name))
def caler():
y = "My constant"
magic(y)
use lists as references
def func(a):
a[0] = a[0] + 1
x = [1]
func(x)
print x[0] # x expected to 2
y = [2]
func(y)
print y[0] # y expected to 3
This is my final solution. Use ast to parse the function statement and get the args.:
import inspect
import re
import ast
def func(a,b,c):
stack = inspect.stack()
stack_pre = stack[1]
function_call_string = ';'.join( stack_pre[4] ).strip('\n')
ret = re.search(r'(%s[ ]*\(.*\))' % func.func_name, function_call_string)
striped_function_call_string = ret.group()
parser = ast.parse(striped_function_call_string)
frame = inspect.currentframe()
for arg in parser.body[0].value.args:
iter_frame = frame.f_back
while iter_frame:
if arg.id in iter_frame.f_locals:
iter_frame.f_locals[arg.id] += 1
break
iter_frame = iter_frame.f_back
x=1;y=2;z=3;func(x,y,z)
print x,y,z
Out:
2 3 4
I want to have something like
def x():
print get_def_name()
but not necessarily know the name of x.
Ideally it would return 'x' where x would be the name of the function.
You can do this by using Python's built-in inspect library.
You can read more of its documentation if you want to handle more complicated cases, but this snippet will work for you:
from inspect import getframeinfo, currentframe
def test_func_name():
return getframeinfo(currentframe()).function
print(test_func_name())
Functions in Python are objects, and as it happens those objects do have an attribute containing the name they were defined with:
>>> def x():
... pass
...
>>> print x.__name__
x
So, a naïve approach might be this:
>>> def x():
... print x.__name__
...
>>> x()
x
That seems to work. However, since you had to know the name of x inside the function in order to do that, you haven't really gained anything; you might have well just have done this:
def x():
print "x"
In fact, though, it's worse than that, because the __name__ attribute only refers to the name the function was defined with. If it gets bound to another name, it won't behave as you expect:
>>> y = x
>>> y()
x
Even worse, if the original name is no longer around, it won't work at all:
>>> del x
>>> y()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in x
NameError: global name 'x' is not defined
This second problem is one you can actually get around, although it's not pretty. The trick is to write a decorator that can pass the function's name into it as an argument:
>>> from functools import wraps
>>> def introspective(func):
... __name__ = func.__name__
... #wraps(func)
... def wrapper(*args, **kwargs):
... return func(__name__=__name__, *args, **kwargs)
... return wrapper
...
>>> #introspective
... def x(__name__):
... print __name__
...
>>> x()
x
>>> y = x
>>> y()
x
>>> del x
>>> y()
x
... although as you can see, you're still only getting back the name the function was defined with, not the one it happens to be bound to right now.
In practice, the short (and correct) answer is "don't do that". It's a fundamental fact of Python that objects don't know what name (or names) they're bound to - if you think your function needs that information, you're doing something wrong.
This sounds like you want to declare an anonymous function and it would return a reference to the new function object.
In Python, you can get a trivial anonymous function object with lambda but for a complex function it must have a name. But any function object is in fact an object and you can pass references around to it, so the name doesn't matter.
# lambda
sqr = lambda n: n**2
assert sqr(2) == 4
assert sqr(3) == 9
# named function
def f(n):
return n**2
sqr = f
assert sqr(2) == 4
assert sqr(3) == 9
Note that this function does have a name, f, but the name doesn't really matter here. We set the name sqr to the function object reference and use that name. We could put the function reference into a list or other data structure if we wanted to.
You could re-use the name of the function:
def f(n):
return n**2
sqr = f
def f(n):
return n**3
cube = f
So, while Python doesn't really support full anonymous functions, you can get the same effect. It's not really a problem that you have to give functions a name.
If you really don't want the function to have a name, you can unbind the name:
def f(n):
return n**2
lst = [f] # save function reference in a list
del(f) # unbind the name
Now the only way to access this function is through the list; the name of the function is gone.
I found a similar solution as Vazirani's, but I did a step forward to get the function object based on the name. Here is my solution:
import inspect
def named_func():
func_name = inspect.stack()[0].function
func_obj = inspect.stack()[1].frame.f_locals[func_name]
print(func_name, func_obj, func_obj.xxx)
named_func.xxx = 15
named_func()
Output is
named_func <function named_func at 0x7f3bc84622f0> 15
Unfortunately I cannot do this with lambda function. I keep trying.
This is more a curiosity than anything, but I just noticed the following. If I am defining a self-referential lambda, I can do it easily:
>>> f = lambda: f
>>> f() is f
True
But if I am defining a self-referential list, I have to do it in more than one statement:
>>> a = [a]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>> a = []
>>> a.append(a)
>>> a[0] is a
True
>>> a
[[...]]
I also noticed that this is not limited to lists but seems like any other expression other than a lambda can not reference the variable left of the assignment. For example, if you have a cyclic linked-list with one node, you can't simply go:
>>> class Node(object):
... def __init__(self, next_node):
... self.next = next_node
...
>>> n = Node(n)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'n' is not defined
Instead, you have to do it in two statements:
>>> n = Node(None)
>>> n.next = n
>>> n is n.next
True
Does anyone know what the philosophy behind this difference is? I understand that a recursive lambda are used much more frequently, and hence supporting self-reference is important for lambdas, but why not allow it for any assignment?
EDIT: The answers below clarify this quite nicely. The reason is that variables in lambdas in Python are evaluated each time the lambda is called, not when it's defined. In this sense they are exactly like functions defined using def. I wrote the following bit of code to experiment with how this works, both with lambdas and def functions in case it might help clarify it for anyone.
>>> f = lambda: f
>>> f() is f
True
>>> g = f
>>> f = "something else"
>>> g()
'something else'
>>> f = "hello"
>>> g()
'hello'
>>> f = g
>>> g() is f
True
>>> def f():
... print(f)
...
>>> f()
<function f at 0x10d125560>
>>> g = f
>>> g()
<function f at 0x10d125560>
>>> f = "test"
>>> g()
test
>>> f = "something else"
>>> g()
something else
The expression inside a lambda is evaluated when the function is called, not when it is defined.
In other words, Python will not evaluate the f inside your lambda until you call it. And, by then, f is already defined in the current scope (it is the lambda itself). Hence, no NameError is raised.
Note that this is not the case for a line like this:
a = [a]
When Python interprets this type of line (known as an assignment statement), it will evaluate the expression on the right of the = immediately. Moreover, a NameError will be raised for any name used on the right that is undefined in the current scope.
Because a lambda is a function, and the function body is not executed until the function is called.
In other words, the other way to do it is this:
def f():
return f
But you're correct that you can't do it in an expression because def is a statement, so it can't be used in an expression.
We can see when we disassemble the lambda function (this is identical output in Python 2.6 and 3.3)
>>> import dis
>>> f = lambda: f
>>> dis.dis(f)
1 0 LOAD_GLOBAL 0 (f)
3 RETURN_VALUE
We demonstrate that we do not need to load f until it is called, whereupon it is already defined globally, and therefore stored, so this works:
>>> f is f()
True
But when we do:
>>> a = [a]
We have an error (if a is previously undefined), and if we disassemble Python's implementation of this.
>>> def foo():
... a = [a]
...
>>> dis.dis(foo)
2 0 LOAD_FAST 0 (a)
3 BUILD_LIST 1
6 STORE_FAST 0 (a)
9 LOAD_CONST 0 (None)
12 RETURN_VALUE
we see that we attempt to load a before we have stored it.
There's no special-casing required to make this happen; it's just how it works.
A lambda expression is not any different from a normal function, really. Meaning, I can do this:
x = 1
def f():
print x + 2
f()
3
x = 2
f()
4
As you can see, inside the function, the value of x does not have a predefined value - it's looked up when we actually run f. This includes the value of the function itself: We don't look up what f represents until we actually run it, and by then it exists.
Doing that as a lambda doesn't work any differently:
del x
f = lambda: x+2
f()
NameError: global name 'x' is not defined
x = 2
f()
4
works similarly. In this case, I went ahead and deleted x so it was no longer in the scope when f was defined, and running f in this case correctly shows that x doesn't exist. But after we define x, then f works again.
This is different in the list case, because we are actually generating an object right now, and so everything on the right side has to be bound, right now. The way python works (as i understand it, and at least in practice this has been useful) is to consider that everything on the right side is deferenced & bound and then processed, and only after that's all complete are the value(s) on the left side bound and assigned.
Since the same value is on the right side and the left, when python tries to bind the name on the right side, it doesn't exist yet.