How can I get a variable that contains the currently executing function in Python? I don't want the function's name. I know I can use inspect.stack to get the current function name. I want the actual callable object. Can this be done without using inspect.stack to retrieve the function's name and then evaling the name to get the callable object?
Edit: I have a reason to do this, but it's not even a remotely good one. I'm using plac to parse command-line arguments. You use it by doing plac.call(main), which generates an ArgumentParser object from the function signature of "main". Inside "main", if there is a problem with the arguments, I want to exit with an error message that includes the help text from the ArgumentParser object, which means that I need to directly access this object by calling plac.parser_from(main).print_help(). It would be nice to be able to say instead: plac.parser_from(get_current_function()).print_help(), so that I am not relying on the function being named "main". Right now, my implementation of "get_current_function" would be:
import inspect
def get_current_function():
return eval(inspect.stack()[1][3])
But this implementation relies on the function having a name, which I suppose is not too onerous. I'm never going to do plac.call(lambda ...).
In the long run, it might be more useful to ask the author of plac to implement a print_help method to print the help text of the function that was most-recently called using plac, or something similar.
The stack frame tells us what code object we're in. If we can find a function object that refers to that code object in its __code__ attribute, we have found the function.
Fortunately, we can ask the garbage collector which objects hold a reference to our code object, and sift through those, rather than having to traverse every active object in the Python world. There are typically only a handful of references to a code object.
Now, functions can share code objects, and do in the case where you return a function from a function, i.e. a closure. When there's more than one function using a given code object, we can't tell which function it is, so we return None.
import inspect, gc
def giveupthefunc():
frame = inspect.currentframe(1)
code = frame.f_code
globs = frame.f_globals
functype = type(lambda: 0)
funcs = []
for func in gc.get_referrers(code):
if type(func) is functype:
if getattr(func, "__code__", None) is code:
if getattr(func, "__globals__", None) is globs:
funcs.append(func)
if len(funcs) > 1:
return None
return funcs[0] if funcs else None
Some test cases:
def foo():
return giveupthefunc()
zed = lambda: giveupthefunc()
bar, foo = foo, None
print bar()
print zed()
I'm not sure about the performance characteristics of this, but i think it should be fine for your use case.
I recently spent a lot of time trying to do something like this and ended up walking away from it. There's a lot of corner cases.
If you just want the lowest level of the call stack, you can just reference the name that is used in the def statement. This will be bound to the function that you want through lexical closure.
For example:
def recursive(*args, **kwargs):
me = recursive
me will now refer to the function in question regardless of the scope that the function is called from so long as it is not redefined in the scope where the definition occurs. Is there some reason why this won't work?
To get a function that is executing higher up the call stack, I couldn't think of anything that can be reliably done.
This is what you asked for, as close as I can come. Tested in python versions 2.4, 2.6, 3.0.
#!/usr/bin/python
def getfunc():
from inspect import currentframe, getframeinfo
caller = currentframe().f_back
func_name = getframeinfo(caller)[2]
caller = caller.f_back
from pprint import pprint
func = caller.f_locals.get(
func_name, caller.f_globals.get(
func_name
)
)
return func
def main():
def inner1():
def inner2():
print("Current function is %s" % getfunc())
print("Current function is %s" % getfunc())
inner2()
print("Current function is %s" % getfunc())
inner1()
#entry point: parse arguments and call main()
if __name__ == "__main__":
main()
Output:
Current function is <function main at 0x2aec09fe2ed8>
Current function is <function inner1 at 0x2aec09fe2f50>
Current function is <function inner2 at 0x2aec0a0635f0>
Here's another possibility: a decorator that implicitly passes a reference to the called function as the first argument (similar to self in bound instance methods). You have to decorate each function that you want to receive such a reference, but "explicit is better than implicit" as they say.
Of course, it has all the disadvantage of decorators: another function call slightly degrades performance, and the signature of the wrapped function is no longer visible.
import functools
def gottahavethatfunc(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
return func(func, *args, **kwargs)
return wrapper
The test case illustrates that the decorated function still gets the reference to itself even if you change the name to which the function is bound. This is because you're only changing the binding of the wrapper function. It also illustrates its use with a lambda.
#gottahavethatfunc
def quux(me):
return me
zoom = gottahavethatfunc(lambda me: me)
baz, quux = quux, None
print baz()
print zoom()
When using this decorator with an instance or class method, the method should accept the function reference as the first argument and the traditional self as the second.
class Demo(object):
#gottahavethatfunc
def method(me, self):
return me
print Demo().method()
The decorator relies on a closure to hold the reference to the wrapped function in the wrapper. Creating the closure directly might actually be cleaner, and won't have the overhead of the extra function call:
def my_func():
def my_func():
return my_func
return my_func
my_func = my_func()
Within the inner function, the name my_func always refers to that function; its value does not rely on a global name that may be changed. Then we just "lift" that function to the global namespace, replacing the reference to the outer function. Works in a class too:
class K(object):
def my_method():
def my_method(self):
return my_method
return my_method
my_method = my_method()
I just define in the beginning of each function a "keyword" which is just a reference to the actual name of the function. I just do this for any function, if it needs it or not:
def test():
this=test
if not hasattr(this,'cnt'):
this.cnt=0
else:
this.cnt+=1
print this.cnt
The call stack does not keep a reference to the function itself -
although the running frame as a reference to the code object that is the code associated to a given function.
(Functions are objects with code, and some information about their environment, such as closures, name, globals dictionary, doc string, default parameters and so on).
Therefore if you are running a regular function, you are better of using its own name on the globals dictionary to call itself, as has been pointed out.
If you are running some dynamic, or lambda code, in which you can't use the function name, the only solution is to rebuild another function object which re-uses thre currently running code object and call that new function instead.
You will loose a couple of things, like default arguments, and it may be hard to get it working with closures (although it can be done).
I have written a blog post on doing exactly that - calling anonymous functions from within themselves - I hope the code in there can help you:
http://metapython.blogspot.com/2010/11/recursive-lambda-functions.html
On a side note: avoid the use o inspect.stack -- it is too slow, as it rebuilds a lot of information each time it is called. prefer to use inspect.currentframe to deal with code frames instead.
This may sounds complicated, but the code itself is very short - I am pasting it bellow. The post above contains more information on how this works.
from inspect import currentframe
from types import FunctionType
lambda_cache = {}
def myself (*args, **kw):
caller_frame = currentframe(1)
code = caller_frame.f_code
if not code in lambda_cache:
lambda_cache[code] = FunctionType(code, caller_frame.f_globals)
return lambda_cache[code](*args, **kw)
if __name__ == "__main__":
print "Factorial of 5", (lambda n: n * myself(n - 1) if n > 1 else 1)(5)
If you really need the original function itself, the "myself" function above could be made to search on some scopes (like the calling function global dictionary) for a function object which code object would match with the one retrieved from the frame, instead of creating a new function.
sys._getframe(0).f_code returns exactly what you need: the codeobject being executed. Having a code object, you can retrieve a name with codeobject.co_name
OK after reading the question and comments again, I think this is a decent test case:
def foo(n):
""" print numbers from 0 to n """
if n: foo(n-1)
print n
g = foo # assign name 'g' to function object
foo = None # clobber name 'foo' which refers to function object
g(10) # dies with TypeError because function object tries to call NoneType
I tried solving it by using a decorator to temporarily clobber the global namespace and reassigning the function object to the original name of the function:
def selfbind(f):
""" Ensures that f's original function name is always defined as f when f is executed """
oname = f.__name__
def g(*args, **kwargs):
# Clobber global namespace
had_key = None
if globals().has_key(oname):
had_key = True
key = globals()[oname]
globals()[oname] = g
# Run function in modified environment
result = f(*args, **kwargs)
# Restore global namespace
if had_key:
globals()[oname] = key
else:
del globals()[oname]
return result
return g
#selfbind
def foo(n):
if n: foo(n-1)
print n
g = foo # assign name 'g' to function object
foo = 2 # calling 'foo' now fails since foo is an int
g(10) # print from 0..10, even though foo is now an int
print foo # prints 2 (the new value of Foo)
I'm sure I haven't thought through all the use cases. The biggest problem I see is the function object intentionally changing what its own name points to (an operation which would be overwritten by the decorator), but that should be ok as long as the recursive function doesn't redefine its own name in the middle of recursing.
Still not sure I'd ever need to do this, but thinking about was interesting.
Here a variation (Python 3.5.1) of the get_referrers() answer, which tries to distinguish between closures that are using the same code object:
import functools
import gc
import inspect
def get_func():
frame = inspect.currentframe().f_back
code = frame.f_code
return [
referer
for referer in gc.get_referrers(code)
if getattr(referer, "__code__", None) is code and
set(inspect.getclosurevars(referer).nonlocals.items()) <=
set(frame.f_locals.items())][0]
def f1(x):
def f2(y):
print(get_func())
return x + y
return f2
f_var1 = f1(1)
f_var1(3)
# <function f1.<locals>.f2 at 0x0000017235CB2C80>
# 4
f_var2 = f1(2)
f_var2(3)
# <function f1.<locals>.f2 at 0x0000017235CB2BF8>
# 5
def f3():
print(get_func())
f3()
# <function f3 at 0x0000017235CB2B70>
def wrapper(func):
functools.wraps(func)
def wrapped(*args, **kwargs):
return func(*args, **kwargs)
return wrapped
#wrapper
def f4():
print(get_func())
f4()
# <function f4 at 0x0000017235CB2A60>
f5 = lambda: get_func()
print(f5())
# <function <lambda> at 0x0000017235CB2950>
Correction of my previous answer, because the subdict check already works with "<=" called on dict_items and the additional set() calls result in problems, if there are dict-values which are dicts themself:
import gc
import inspect
def get_func():
frame = inspect.currentframe().f_back
code = frame.f_code
return [
referer
for referer in gc.get_referrers(code)
if getattr(referer, "__code__", None) is code and
inspect.getclosurevars(referer).nonlocals.items() <=
frame.f_locals.items()][0]
Related
Is it possible to forward-declare a function in Python? I want to sort a list using my own cmp function before it is declared.
print "\n".join([str(bla) for bla in sorted(mylist, cmp = cmp_configs)])
I've put the definition of cmp_configs method after the invocation. It fails with this error:
NameError: name 'cmp_configs' is not defined
Is there any way to "declare" cmp_configs method before it's used?
Sometimes, it is difficult to reorganize code to avoid this problem. For instance, when implementing some forms of recursion:
def spam():
if end_condition():
return end_result()
else:
return eggs()
def eggs():
if end_condition():
return end_result()
else:
return spam()
Where end_condition and end_result have been previously defined.
Is the only solution to reorganize the code and always put definitions before invocations?
Wrap the invocation into a function of its own so that
foo()
def foo():
print "Hi!"
will break, but
def bar():
foo()
def foo():
print "Hi!"
bar()
will work properly.
The general rule in Python is that a function should be defined before its usage, which does not necessarily mean it needs to be higher in the code.
If you kick-start your script through the following:
if __name__=="__main__":
main()
then you probably do not have to worry about things like "forward declaration". You see, the interpreter would go loading up all your functions and then start your main() function. Of course, make sure you have all the imports correct too ;-)
Come to think of it, I've never heard such a thing as "forward declaration" in python... but then again, I might be wrong ;-)
If you don't want to define a function before it's used, and defining it afterwards is impossible, what about defining it in some other module?
Technically you still define it first, but it's clean.
You could create a recursion like the following:
def foo():
bar()
def bar():
foo()
Python's functions are anonymous just like values are anonymous, yet they can be bound to a name.
In the above code, foo() does not call a function with the name foo, it calls a function that happens to be bound to the name foo at the point the call is made. It is possible to redefine foo somewhere else, and bar would then call the new function.
Your problem cannot be solved because it's like asking to get a variable which has not been declared.
I apologize for reviving this thread, but there was a strategy not discussed here which may be applicable.
Using reflection it is possible to do something akin to forward declaration. For instance lets say you have a section of code that looks like this:
# We want to call a function called 'foo', but it hasn't been defined yet.
function_name = 'foo'
# Calling at this point would produce an error
# Here is the definition
def foo():
bar()
# Note that at this point the function is defined
# Time for some reflection...
globals()[function_name]()
So in this way we have determined what function we want to call before it is actually defined, effectively a forward declaration. In python the statement globals()[function_name]() is the same as foo() if function_name = 'foo' for the reasons discussed above, since python must lookup each function before calling it. If one were to use the timeit module to see how these two statements compare, they have the exact same computational cost.
Of course the example here is very useless, but if one were to have a complex structure which needed to execute a function, but must be declared before (or structurally it makes little sense to have it afterwards), one can just store a string and try to call the function later.
If the call to cmp_configs is inside its own function definition, you should be fine. I'll give an example.
def a():
b() # b() hasn't been defined yet, but that's fine because at this point, we're not
# actually calling it. We're just defining what should happen when a() is called.
a() # This call fails, because b() hasn't been defined yet,
# and thus trying to run a() fails.
def b():
print "hi"
a() # This call succeeds because everything has been defined.
In general, putting your code inside functions (such as main()) will resolve your problem; just call main() at the end of the file.
There is no such thing in python like forward declaration. You just have to make sure that your function is declared before it is needed.
Note that the body of a function isn't interpreted until the function is executed.
Consider the following example:
def a():
b() # won't be resolved until a is invoked.
def b():
print "hello"
a() # here b is already defined so this line won't fail.
You can think that a body of a function is just another script that will be interpreted once you call the function.
Sometimes an algorithm is easiest to understand top-down, starting with the overall structure and drilling down into the details.
You can do so without forward declarations:
def main():
make_omelet()
eat()
def make_omelet():
break_eggs()
whisk()
fry()
def break_eggs():
for egg in carton:
break(egg)
# ...
main()
# declare a fake function (prototype) with no body
def foo(): pass
def bar():
# use the prototype however you see fit
print(foo(), "world!")
# define the actual function (overwriting the prototype)
def foo():
return "Hello,"
bar()
Output:
Hello, world!
No, I don't believe there is any way to forward-declare a function in Python.
Imagine you are the Python interpreter. When you get to the line
print "\n".join([str(bla) for bla in sorted(mylist, cmp = cmp_configs)])
either you know what cmp_configs is or you don't. In order to proceed, you have to
know cmp_configs. It doesn't matter if there is recursion.
You can't forward-declare a function in Python. If you have logic executing before you've defined functions, you've probably got a problem anyways. Put your action in an if __name__ == '__main__' at the end of your script (by executing a function you name "main" if it's non-trivial) and your code will be more modular and you'll be able to use it as a module if you ever need to.
Also, replace that list comprehension with a generator express (i.e., print "\n".join(str(bla) for bla in sorted(mylist, cmp=cmp_configs)))
Also, don't use cmp, which is deprecated. Use key and provide a less-than function.
Import the file itself. Assuming the file is called test.py:
import test
if __name__=='__main__':
test.func()
else:
def func():
print('Func worked')
TL;DR: Python does not need forward declarations. Simply put your function calls inside function def definitions, and you'll be fine.
def foo(count):
print("foo "+str(count))
if(count>0):
bar(count-1)
def bar(count):
print("bar "+str(count))
if(count>0):
foo(count-1)
foo(3)
print("Finished.")
recursive function definitions, perfectly successfully gives:
foo 3
bar 2
foo 1
bar 0
Finished.
However,
bug(13)
def bug(count):
print("bug never runs "+str(count))
print("Does not print this.")
breaks at the top-level invocation of a function that hasn't been defined yet, and gives:
Traceback (most recent call last):
File "./test1.py", line 1, in <module>
bug(13)
NameError: name 'bug' is not defined
Python is an interpreted language, like Lisp. It has no type checking, only run-time function invocations, which succeed if the function name has been bound and fail if it's unbound.
Critically, a function def definition does not execute any of the funcalls inside its lines, it simply declares what the function body is going to consist of. Again, it doesn't even do type checking. So we can do this:
def uncalled():
wild_eyed_undefined_function()
print("I'm not invoked!")
print("Only run this one line.")
and it runs perfectly fine (!), with output
Only run this one line.
The key is the difference between definitions and invocations.
The interpreter executes everything that comes in at the top level, which means it tries to invoke it. If it's not inside a definition.
Your code is running into trouble because you attempted to invoke a function, at the top level in this case, before it was bound.
The solution is to put your non-top-level function invocations inside a function definition, then call that function sometime much later.
The business about "if __ main __" is an idiom based on this principle, but you have to understand why, instead of simply blindly following it.
There are certainly much more advanced topics concerning lambda functions and rebinding function names dynamically, but these are not what the OP was asking for. In addition, they can be solved using these same principles: (1) defs define a function, they do not invoke their lines; (2) you get in trouble when you invoke a function symbol that's unbound.
Python does not support forward declarations, but common workaround for this is use of the the following condition at the end of your script/code:
if __name__ == '__main__': main()
With this it will read entire file first and then evaluate condition and call main() function which will be able to call any forward declared function as it already read the entire file first. This condition leverages special variable __name__ which returns __main__ value whenever we run Python code from current file (when code was imported as a module, then __name__ returns module name).
"just reorganize my code so that I don't have this problem." Correct. Easy to do. Always works.
You can always provide the function prior to it's reference.
"However, there are cases when this is probably unavoidable, for instance when implementing some forms of recursion"
Can't see how that's even remotely possible. Please provide an example of a place where you cannot define the function prior to it's use.
Now wait a minute. When your module reaches the print statement in your example, before cmp_configs has been defined, what exactly is it that you expect it to do?
If your posting of a question using print is really trying to represent something like this:
fn = lambda mylist:"\n".join([str(bla)
for bla in sorted(mylist, cmp = cmp_configs)])
then there is no requirement to define cmp_configs before executing this statement, just define it later in the code and all will be well.
Now if you are trying to reference cmp_configs as a default value of an argument to the lambda, then this is a different story:
fn = lambda mylist,cmp_configs=cmp_configs : \
"\n".join([str(bla) for bla in sorted(mylist, cmp = cmp_configs)])
Now you need a cmp_configs variable defined before you reach this line.
[EDIT - this next part turns out not to be correct, since the default argument value will get assigned when the function is compiled, and that value will be used even if you change the value of cmp_configs later.]
Fortunately, Python being so type-accommodating as it is, does not care what you define as cmp_configs, so you could just preface with this statement:
cmp_configs = None
And the compiler will be happy. Just be sure to declare the real cmp_configs before you ever invoke fn.
Python technically has support for forward declaration.
if you define a function/class then set the body to pass, it will have an empty entry in the global table.
you can then "redefine" the function/class later on to implement the function/class.
unlike c/c++ forward declaration though, this does not work from outside the scope (i.e. another file) as they have their own "global" namespace
example:
def foo(): pass
foo()
def foo(): print("FOOOOO")
foo()
foo is declared both times
however the first time foo is called it does not do anything as the body is just pass
but the second time foo is called. it executes the new body of print("FOOOOO")
but again. note that this does not fix circular dependancies. this is because files have their own name and have their own definitions of functions
example 2:
class bar: pass
print(bar)
this prints <class '__main__.bar'> but if it was declared in another file it would be <class 'otherfile.foo'>
i know this post is old, but i though that this answer would be useful to anyone who keeps finding this post after the many years it has been posted for
One way is to create a handler function. Define the handler early on, and put the handler below all the methods you need to call.
Then when you invoke the handler method to call your functions, they will always be available.
The handler could take an argument nameOfMethodToCall. Then uses a bunch of if statements to call the right method.
This would solve your issue.
def foo():
print("foo")
#take input
nextAction=input('What would you like to do next?:')
return nextAction
def bar():
print("bar")
nextAction=input('What would you like to do next?:')
return nextAction
def handler(action):
if(action=="foo"):
nextAction = foo()
elif(action=="bar"):
nextAction = bar()
else:
print("You entered invalid input, defaulting to bar")
nextAction = "bar"
return nextAction
nextAction=input('What would you like to do next?:')
while 1:
nextAction = handler(nextAction)
In short, the question: Is there a way to prevent Python from looking up variables outside the current scope?
Details:
Python looks for variable definitions in outer scopes if they are not defined in the current scope. Thus, code like this is liable to break when not being careful during refactoring:
def line(x, a, b):
return a + x * b
a, b = 1, 1
y1 = line(1, a, b)
y2 = line(1, 2, 3)
If I renamed the function arguments, but forgot to rename them inside the function body, the code would still run:
def line(x, a0, b0):
return a + x * b # not an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
y2 = line(1, 2, 3) # wrong result
I know it is bad practice to shadow names from outer scopes. But sometimes we do it anyway...
Is there a way to prevent Python from looking up variables outside the current scope? (So that accessing a or b raises an Error in the second example.)
Yes, maybe not in general. However you can do it with functions.
The thing you want to do is to have the function's global to be empty. You can't replace the globals and you don't want to modify it's content (becaus
that would be just to get rid of global variables and functions).
However: you can create function objects in runtime. The constructor looks like types.FunctionType((code, globals[, name[, argdefs[, closure]]]). There you can replace the global namespace:
def line(x, a0, b0):
return a + x * b # will be an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
line = types.FunctionType(line.__code__, {})
y1 = line(1, a, b) # fails since global name is not defined
You can of course clean this up by defining your own decorator:
import types
noglobal = lambda f: types.FunctionType(f.__code__, {}, argdefs=f.__defaults__)
#noglobal
def f():
return x
x = 5
f() # will fail
Strictly speaking you do not forbid it to access global variables, you just make the function believe there is no variables in global namespace. Actually you can also use this to emulate static variables since if it declares an variable to be global and assign to it it will end up in it's own sandbox of global namespace.
If you want to be able to access part of the global namespace then you'll need to populate the functions global sandbox with what you want it to see.
No, you cannot tell Python not to look names up in the global scope.
If you could, you would not be able to use any other classes or functions defined in the module, no objects imported from other modules, nor could you use built-in names. Your function namespace becomes a desert devoid of almost everything it needs, and the only way out would be to import everything into the local namespace. For every single function in your module.
Rather than try to break global lookups, keep your global namespace clean. Don't add globals that you don't need to share with other scopes in the module. Use a main() function for example, to encapsulate what are really just locals.
Also, add unittesting. Refactoring without (even just a few) tests is always prone to create bugs otherwise.
With #skyking's answer, I was unable to access any imports (I could not even use print). Also, functions with optional arguments are broken (compare How can an optional parameter become required?).
#Ax3l's comment improved that a bit. Still I was unable to access imported variables (from module import var).
Therefore, I propose this:
def noglobal(f):
return types.FunctionType(f.__code__, globals().copy(), f.__name__, f.__defaults__, f.__closure__)
For each function decorated with #noglobal, that creates a copy of the globals() defined so far. This keeps imported variables (usually imported at the top of the document) accessible. If you do it like me, defining your functions first and then your variables, this will achieve the desired effect of being able to access imported variables in your function, but not the ones you define in your code. Since copy() creates a shallow copy (Understanding dict.copy() - shallow or deep?), this should be pretty memory-efficient, too.
Note that this way, a function can only call functions defined above itself, so you may need to reorder your code.
For the record, I copy #Ax3l's version from his Gist:
def imports():
for name, val in globals().items():
# module imports
if isinstance(val, types.ModuleType):
yield name, val
# functions / callables
if hasattr(val, '__call__'):
yield name, val
noglobal = lambda fn: types.FunctionType(fn.__code__, dict(imports()))
To discourage global variable lookup, move your function into another module. Unless it inspects the call stack or imports your calling module explicitly; it won't have access to the globals from the module that calls it.
In practice, move your code into a main() function, to avoid creating unnecessary global variables.
If you use globals because several functions need to manipulate shared state then move the code into a class.
As mentioned by #bers the decorator by #skykings breaks most python functionality inside the function, such as print() and the import statement. #bers hacked around the import statement by adding the currently imported modules from globals() at the time of decorator definition.
This inspired me to write yet another decorator that hopefully does what most people who come looking at this post actually want. The underlying problem is that the new function created by the previous decorators lacked the __builtins__ variable which contains all of the standard built-in python functions (e.g. print) available in a freshly opened interpreter.
import types
import builtins
def no_globals(f):
'''
A function decorator that prevents functions from looking up variables in outer scope.
'''
# need builtins in globals otherwise can't import or print inside the function
new_globals = {'__builtins__': builtins}
new_f = types.FunctionType(f.__code__, globals=new_globals, argdefs=f.__defaults__)
new_f.__annotations__ = f.__annotations__ # for some reason annotations aren't copied over
return new_f
Then the usage goes as the following
#no_globals
def f1():
return x
x = 5
f1() # should raise NameError
#no_globals
def f2(x):
import numpy as np
print(x)
return np.sin(x)
x = 5
f2(x) # should print 5 and return -0.9589242746631385
Theoretically you can use your own decorator that removes globals() while a function call. It is some overhead to hide all globals() but, if there are not too many globals() it could be useful. During the operation we do not create/remove global objects, we just overwrites references in dictionary which refers to global objects. But do not remove special globals() (like __builtins__) and modules. Probably you do not want to remove callables from global scope too.
from types import ModuleType
import re
# the decorator to hide global variables
def noglobs(f):
def inner(*args, **kwargs):
RE_NOREPLACE = '__\w+__'
old_globals = {}
# removing keys from globals() storing global values in old_globals
for key, val in globals().iteritems():
if re.match(RE_NOREPLACE, key) is None and not isinstance(val, ModuleType) and not callable(val):
old_globals.update({key: val})
for key in old_globals.keys():
del globals()[key]
result = f(*args, **kwargs)
# restoring globals
for key in old_globals.iterkeys():
globals()[key] = old_globals[key]
return result
return inner
# the example of usage
global_var = 'hello'
#noglobs
def no_globals_func():
try:
print 'Can I use %s here?' % global_var
except NameError:
print 'Name "global_var" in unavailable here'
def globals_func():
print 'Can I use %s here?' % global_var
globals_func()
no_globals_func()
print 'Can I use %s here?' % global_var
...
Can I use hello here?
Name "global_var" in unavailable here
Can I use hello here?
Or, you can iterate over all global callables (i.e. functions) in your module and decorate them dynamically (it's little more code).
The code is for Python 2, I think it's possible to create a very similar code for Python 3.
Is it possible to forward-declare a function in Python? I want to sort a list using my own cmp function before it is declared.
print "\n".join([str(bla) for bla in sorted(mylist, cmp = cmp_configs)])
I've put the definition of cmp_configs method after the invocation. It fails with this error:
NameError: name 'cmp_configs' is not defined
Is there any way to "declare" cmp_configs method before it's used?
Sometimes, it is difficult to reorganize code to avoid this problem. For instance, when implementing some forms of recursion:
def spam():
if end_condition():
return end_result()
else:
return eggs()
def eggs():
if end_condition():
return end_result()
else:
return spam()
Where end_condition and end_result have been previously defined.
Is the only solution to reorganize the code and always put definitions before invocations?
Wrap the invocation into a function of its own so that
foo()
def foo():
print "Hi!"
will break, but
def bar():
foo()
def foo():
print "Hi!"
bar()
will work properly.
The general rule in Python is that a function should be defined before its usage, which does not necessarily mean it needs to be higher in the code.
If you kick-start your script through the following:
if __name__=="__main__":
main()
then you probably do not have to worry about things like "forward declaration". You see, the interpreter would go loading up all your functions and then start your main() function. Of course, make sure you have all the imports correct too ;-)
Come to think of it, I've never heard such a thing as "forward declaration" in python... but then again, I might be wrong ;-)
If you don't want to define a function before it's used, and defining it afterwards is impossible, what about defining it in some other module?
Technically you still define it first, but it's clean.
You could create a recursion like the following:
def foo():
bar()
def bar():
foo()
Python's functions are anonymous just like values are anonymous, yet they can be bound to a name.
In the above code, foo() does not call a function with the name foo, it calls a function that happens to be bound to the name foo at the point the call is made. It is possible to redefine foo somewhere else, and bar would then call the new function.
Your problem cannot be solved because it's like asking to get a variable which has not been declared.
I apologize for reviving this thread, but there was a strategy not discussed here which may be applicable.
Using reflection it is possible to do something akin to forward declaration. For instance lets say you have a section of code that looks like this:
# We want to call a function called 'foo', but it hasn't been defined yet.
function_name = 'foo'
# Calling at this point would produce an error
# Here is the definition
def foo():
bar()
# Note that at this point the function is defined
# Time for some reflection...
globals()[function_name]()
So in this way we have determined what function we want to call before it is actually defined, effectively a forward declaration. In python the statement globals()[function_name]() is the same as foo() if function_name = 'foo' for the reasons discussed above, since python must lookup each function before calling it. If one were to use the timeit module to see how these two statements compare, they have the exact same computational cost.
Of course the example here is very useless, but if one were to have a complex structure which needed to execute a function, but must be declared before (or structurally it makes little sense to have it afterwards), one can just store a string and try to call the function later.
If the call to cmp_configs is inside its own function definition, you should be fine. I'll give an example.
def a():
b() # b() hasn't been defined yet, but that's fine because at this point, we're not
# actually calling it. We're just defining what should happen when a() is called.
a() # This call fails, because b() hasn't been defined yet,
# and thus trying to run a() fails.
def b():
print "hi"
a() # This call succeeds because everything has been defined.
In general, putting your code inside functions (such as main()) will resolve your problem; just call main() at the end of the file.
There is no such thing in python like forward declaration. You just have to make sure that your function is declared before it is needed.
Note that the body of a function isn't interpreted until the function is executed.
Consider the following example:
def a():
b() # won't be resolved until a is invoked.
def b():
print "hello"
a() # here b is already defined so this line won't fail.
You can think that a body of a function is just another script that will be interpreted once you call the function.
Sometimes an algorithm is easiest to understand top-down, starting with the overall structure and drilling down into the details.
You can do so without forward declarations:
def main():
make_omelet()
eat()
def make_omelet():
break_eggs()
whisk()
fry()
def break_eggs():
for egg in carton:
break(egg)
# ...
main()
# declare a fake function (prototype) with no body
def foo(): pass
def bar():
# use the prototype however you see fit
print(foo(), "world!")
# define the actual function (overwriting the prototype)
def foo():
return "Hello,"
bar()
Output:
Hello, world!
No, I don't believe there is any way to forward-declare a function in Python.
Imagine you are the Python interpreter. When you get to the line
print "\n".join([str(bla) for bla in sorted(mylist, cmp = cmp_configs)])
either you know what cmp_configs is or you don't. In order to proceed, you have to
know cmp_configs. It doesn't matter if there is recursion.
You can't forward-declare a function in Python. If you have logic executing before you've defined functions, you've probably got a problem anyways. Put your action in an if __name__ == '__main__' at the end of your script (by executing a function you name "main" if it's non-trivial) and your code will be more modular and you'll be able to use it as a module if you ever need to.
Also, replace that list comprehension with a generator express (i.e., print "\n".join(str(bla) for bla in sorted(mylist, cmp=cmp_configs)))
Also, don't use cmp, which is deprecated. Use key and provide a less-than function.
Import the file itself. Assuming the file is called test.py:
import test
if __name__=='__main__':
test.func()
else:
def func():
print('Func worked')
TL;DR: Python does not need forward declarations. Simply put your function calls inside function def definitions, and you'll be fine.
def foo(count):
print("foo "+str(count))
if(count>0):
bar(count-1)
def bar(count):
print("bar "+str(count))
if(count>0):
foo(count-1)
foo(3)
print("Finished.")
recursive function definitions, perfectly successfully gives:
foo 3
bar 2
foo 1
bar 0
Finished.
However,
bug(13)
def bug(count):
print("bug never runs "+str(count))
print("Does not print this.")
breaks at the top-level invocation of a function that hasn't been defined yet, and gives:
Traceback (most recent call last):
File "./test1.py", line 1, in <module>
bug(13)
NameError: name 'bug' is not defined
Python is an interpreted language, like Lisp. It has no type checking, only run-time function invocations, which succeed if the function name has been bound and fail if it's unbound.
Critically, a function def definition does not execute any of the funcalls inside its lines, it simply declares what the function body is going to consist of. Again, it doesn't even do type checking. So we can do this:
def uncalled():
wild_eyed_undefined_function()
print("I'm not invoked!")
print("Only run this one line.")
and it runs perfectly fine (!), with output
Only run this one line.
The key is the difference between definitions and invocations.
The interpreter executes everything that comes in at the top level, which means it tries to invoke it. If it's not inside a definition.
Your code is running into trouble because you attempted to invoke a function, at the top level in this case, before it was bound.
The solution is to put your non-top-level function invocations inside a function definition, then call that function sometime much later.
The business about "if __ main __" is an idiom based on this principle, but you have to understand why, instead of simply blindly following it.
There are certainly much more advanced topics concerning lambda functions and rebinding function names dynamically, but these are not what the OP was asking for. In addition, they can be solved using these same principles: (1) defs define a function, they do not invoke their lines; (2) you get in trouble when you invoke a function symbol that's unbound.
Python does not support forward declarations, but common workaround for this is use of the the following condition at the end of your script/code:
if __name__ == '__main__': main()
With this it will read entire file first and then evaluate condition and call main() function which will be able to call any forward declared function as it already read the entire file first. This condition leverages special variable __name__ which returns __main__ value whenever we run Python code from current file (when code was imported as a module, then __name__ returns module name).
"just reorganize my code so that I don't have this problem." Correct. Easy to do. Always works.
You can always provide the function prior to it's reference.
"However, there are cases when this is probably unavoidable, for instance when implementing some forms of recursion"
Can't see how that's even remotely possible. Please provide an example of a place where you cannot define the function prior to it's use.
Now wait a minute. When your module reaches the print statement in your example, before cmp_configs has been defined, what exactly is it that you expect it to do?
If your posting of a question using print is really trying to represent something like this:
fn = lambda mylist:"\n".join([str(bla)
for bla in sorted(mylist, cmp = cmp_configs)])
then there is no requirement to define cmp_configs before executing this statement, just define it later in the code and all will be well.
Now if you are trying to reference cmp_configs as a default value of an argument to the lambda, then this is a different story:
fn = lambda mylist,cmp_configs=cmp_configs : \
"\n".join([str(bla) for bla in sorted(mylist, cmp = cmp_configs)])
Now you need a cmp_configs variable defined before you reach this line.
[EDIT - this next part turns out not to be correct, since the default argument value will get assigned when the function is compiled, and that value will be used even if you change the value of cmp_configs later.]
Fortunately, Python being so type-accommodating as it is, does not care what you define as cmp_configs, so you could just preface with this statement:
cmp_configs = None
And the compiler will be happy. Just be sure to declare the real cmp_configs before you ever invoke fn.
Python technically has support for forward declaration.
if you define a function/class then set the body to pass, it will have an empty entry in the global table.
you can then "redefine" the function/class later on to implement the function/class.
unlike c/c++ forward declaration though, this does not work from outside the scope (i.e. another file) as they have their own "global" namespace
example:
def foo(): pass
foo()
def foo(): print("FOOOOO")
foo()
foo is declared both times
however the first time foo is called it does not do anything as the body is just pass
but the second time foo is called. it executes the new body of print("FOOOOO")
but again. note that this does not fix circular dependancies. this is because files have their own name and have their own definitions of functions
example 2:
class bar: pass
print(bar)
this prints <class '__main__.bar'> but if it was declared in another file it would be <class 'otherfile.foo'>
i know this post is old, but i though that this answer would be useful to anyone who keeps finding this post after the many years it has been posted for
One way is to create a handler function. Define the handler early on, and put the handler below all the methods you need to call.
Then when you invoke the handler method to call your functions, they will always be available.
The handler could take an argument nameOfMethodToCall. Then uses a bunch of if statements to call the right method.
This would solve your issue.
def foo():
print("foo")
#take input
nextAction=input('What would you like to do next?:')
return nextAction
def bar():
print("bar")
nextAction=input('What would you like to do next?:')
return nextAction
def handler(action):
if(action=="foo"):
nextAction = foo()
elif(action=="bar"):
nextAction = bar()
else:
print("You entered invalid input, defaulting to bar")
nextAction = "bar"
return nextAction
nextAction=input('What would you like to do next?:')
while 1:
nextAction = handler(nextAction)
In short, the question: Is there a way to prevent Python from looking up variables outside the current scope?
Details:
Python looks for variable definitions in outer scopes if they are not defined in the current scope. Thus, code like this is liable to break when not being careful during refactoring:
def line(x, a, b):
return a + x * b
a, b = 1, 1
y1 = line(1, a, b)
y2 = line(1, 2, 3)
If I renamed the function arguments, but forgot to rename them inside the function body, the code would still run:
def line(x, a0, b0):
return a + x * b # not an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
y2 = line(1, 2, 3) # wrong result
I know it is bad practice to shadow names from outer scopes. But sometimes we do it anyway...
Is there a way to prevent Python from looking up variables outside the current scope? (So that accessing a or b raises an Error in the second example.)
Yes, maybe not in general. However you can do it with functions.
The thing you want to do is to have the function's global to be empty. You can't replace the globals and you don't want to modify it's content (becaus
that would be just to get rid of global variables and functions).
However: you can create function objects in runtime. The constructor looks like types.FunctionType((code, globals[, name[, argdefs[, closure]]]). There you can replace the global namespace:
def line(x, a0, b0):
return a + x * b # will be an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
line = types.FunctionType(line.__code__, {})
y1 = line(1, a, b) # fails since global name is not defined
You can of course clean this up by defining your own decorator:
import types
noglobal = lambda f: types.FunctionType(f.__code__, {}, argdefs=f.__defaults__)
#noglobal
def f():
return x
x = 5
f() # will fail
Strictly speaking you do not forbid it to access global variables, you just make the function believe there is no variables in global namespace. Actually you can also use this to emulate static variables since if it declares an variable to be global and assign to it it will end up in it's own sandbox of global namespace.
If you want to be able to access part of the global namespace then you'll need to populate the functions global sandbox with what you want it to see.
No, you cannot tell Python not to look names up in the global scope.
If you could, you would not be able to use any other classes or functions defined in the module, no objects imported from other modules, nor could you use built-in names. Your function namespace becomes a desert devoid of almost everything it needs, and the only way out would be to import everything into the local namespace. For every single function in your module.
Rather than try to break global lookups, keep your global namespace clean. Don't add globals that you don't need to share with other scopes in the module. Use a main() function for example, to encapsulate what are really just locals.
Also, add unittesting. Refactoring without (even just a few) tests is always prone to create bugs otherwise.
With #skyking's answer, I was unable to access any imports (I could not even use print). Also, functions with optional arguments are broken (compare How can an optional parameter become required?).
#Ax3l's comment improved that a bit. Still I was unable to access imported variables (from module import var).
Therefore, I propose this:
def noglobal(f):
return types.FunctionType(f.__code__, globals().copy(), f.__name__, f.__defaults__, f.__closure__)
For each function decorated with #noglobal, that creates a copy of the globals() defined so far. This keeps imported variables (usually imported at the top of the document) accessible. If you do it like me, defining your functions first and then your variables, this will achieve the desired effect of being able to access imported variables in your function, but not the ones you define in your code. Since copy() creates a shallow copy (Understanding dict.copy() - shallow or deep?), this should be pretty memory-efficient, too.
Note that this way, a function can only call functions defined above itself, so you may need to reorder your code.
For the record, I copy #Ax3l's version from his Gist:
def imports():
for name, val in globals().items():
# module imports
if isinstance(val, types.ModuleType):
yield name, val
# functions / callables
if hasattr(val, '__call__'):
yield name, val
noglobal = lambda fn: types.FunctionType(fn.__code__, dict(imports()))
To discourage global variable lookup, move your function into another module. Unless it inspects the call stack or imports your calling module explicitly; it won't have access to the globals from the module that calls it.
In practice, move your code into a main() function, to avoid creating unnecessary global variables.
If you use globals because several functions need to manipulate shared state then move the code into a class.
As mentioned by #bers the decorator by #skykings breaks most python functionality inside the function, such as print() and the import statement. #bers hacked around the import statement by adding the currently imported modules from globals() at the time of decorator definition.
This inspired me to write yet another decorator that hopefully does what most people who come looking at this post actually want. The underlying problem is that the new function created by the previous decorators lacked the __builtins__ variable which contains all of the standard built-in python functions (e.g. print) available in a freshly opened interpreter.
import types
import builtins
def no_globals(f):
'''
A function decorator that prevents functions from looking up variables in outer scope.
'''
# need builtins in globals otherwise can't import or print inside the function
new_globals = {'__builtins__': builtins}
new_f = types.FunctionType(f.__code__, globals=new_globals, argdefs=f.__defaults__)
new_f.__annotations__ = f.__annotations__ # for some reason annotations aren't copied over
return new_f
Then the usage goes as the following
#no_globals
def f1():
return x
x = 5
f1() # should raise NameError
#no_globals
def f2(x):
import numpy as np
print(x)
return np.sin(x)
x = 5
f2(x) # should print 5 and return -0.9589242746631385
Theoretically you can use your own decorator that removes globals() while a function call. It is some overhead to hide all globals() but, if there are not too many globals() it could be useful. During the operation we do not create/remove global objects, we just overwrites references in dictionary which refers to global objects. But do not remove special globals() (like __builtins__) and modules. Probably you do not want to remove callables from global scope too.
from types import ModuleType
import re
# the decorator to hide global variables
def noglobs(f):
def inner(*args, **kwargs):
RE_NOREPLACE = '__\w+__'
old_globals = {}
# removing keys from globals() storing global values in old_globals
for key, val in globals().iteritems():
if re.match(RE_NOREPLACE, key) is None and not isinstance(val, ModuleType) and not callable(val):
old_globals.update({key: val})
for key in old_globals.keys():
del globals()[key]
result = f(*args, **kwargs)
# restoring globals
for key in old_globals.iterkeys():
globals()[key] = old_globals[key]
return result
return inner
# the example of usage
global_var = 'hello'
#noglobs
def no_globals_func():
try:
print 'Can I use %s here?' % global_var
except NameError:
print 'Name "global_var" in unavailable here'
def globals_func():
print 'Can I use %s here?' % global_var
globals_func()
no_globals_func()
print 'Can I use %s here?' % global_var
...
Can I use hello here?
Name "global_var" in unavailable here
Can I use hello here?
Or, you can iterate over all global callables (i.e. functions) in your module and decorate them dynamically (it's little more code).
The code is for Python 2, I think it's possible to create a very similar code for Python 3.
Is there any way to make possible to use functions in your file before you actually declare their body?
The following code doesn't seem to work:
abc = myFunction
def myFunction():
print "123"
Thanks
You can't use the myFunction variable before it's assigned. Your example code is similar to:
def myFunction():
print abc
abc = 123
To do what you want, either re-arrange the order:
def myFunction():
print "123"
abc = myFunction
Or declare abc as just a proxy:
# Style 1
abc = lambda: myFunction()
# Style 2
def abc():
return myFunction()
def myFunction():
print "123"
If your function takes parameters, use *args and **kwargs:
# Style 1
abc = lambda *args, **kwargs: myFunction(*args, **kwargs)
# Style 2
def abc(*args, **kwargs):
return myFunction(*args, **kwargs)
def myFunction(x):
print x
You can declare functions that use forward declarations, but Python executes the code in your source from top to bottom. So, this would compile and run:
def foo():
print "in foo"
bar()
def bar():
print "in bar"
foo()
foo()
(except it would cause a stack overflow at runtime, of course). In your example, Python is executing the line
abc = myFunction
before myFunction is defined as anything. You could think of this in terms of typing your source code interactively into the interpreter. When you type in your assignment, you wouldn't have even typed in the definition of myFunction yet, so Python wouldn't be able to refer to it.
Another way to look at this might be the following:
>>> myFunction = "hello"
>>> abc = myFunction
>>> def myFunction():
... print "there"
...
>>> abc
'hello'
>>> myFunction
<function myFunction at 0x63270>
>>> myFunction()
there
As you can see, the definition of the myFunction function just changes the binding of the symbol myFunction to a function object.
short answer is no.
In Python, statements are evaluated along as they are parsed - myFunction wasn't parsed so Python doesn't know about it.
You can declare an empty function, use it to set attributes or anything like this, and then modify its code later.
def myFunction():
pass
myFunction.foo = 'bar'
def makeFunction(obj):
def myFunction():
print "123"
f = myFunction
for attr in dir(obj):
if attr not in dir(f):
setattr(f, attr, getattr(obj, attr))
return f
myFunction = makeFunction(myFunction)
myFunction()
print myFunction.foo
Python will raise a NameError as it encounters any reference to any name (token that's a valid variable/class/function/object name) for which there was no previous binding.
The Python interpreter executes the source code of a file as it reads it. Thus def foo(): is actually statement which defines foo() as the code is being loaded.
It's easy to think that forward references are supported. Consider this:
def foo():
return abc
abc="FooBar, Dude"
print foo()
... and you'll see that it can be run without issues. It's best to think of the definition of foo() as being quoted. The contents aren't evaluated until the function is invoked. So the NameError isn't raised (so long as some value has been bound to the name "abc" before the call to the function.
Notice that some of these semantics are sufficiently different than those from languages like Java, C/C++, and Perl that Pythonistas often prefer to use slightly different terminology. Variables and such are called "names" and the process of associating those with values (or, more generally, objects) is referred to as "binding." So instead of "assigning values to variables" you are "binding objects to names."
Informally names, especially for simple numeric or string objects, are called "variables" and statements such as x='foo' are referred to as assignments. The semantics are usually similar enough that we won't care.