Develop Reference

Preventing a function from looking up variables outside it [duplicate]

In short, the question: Is there a way to prevent Python from looking up variables outside the current scope?
Details:
Python looks for variable definitions in outer scopes if they are not defined in the current scope. Thus, code like this is liable to break when not being careful during refactoring:
def line(x, a, b):
return a + x * b
a, b = 1, 1
y1 = line(1, a, b)
y2 = line(1, 2, 3)
If I renamed the function arguments, but forgot to rename them inside the function body, the code would still run:
def line(x, a0, b0):
return a + x * b # not an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
y2 = line(1, 2, 3) # wrong result
I know it is bad practice to shadow names from outer scopes. But sometimes we do it anyway...
Is there a way to prevent Python from looking up variables outside the current scope? (So that accessing a or b raises an Error in the second example.)

Yes, maybe not in general. However you can do it with functions.
The thing you want to do is to have the function's global to be empty. You can't replace the globals and you don't want to modify it's content (becaus
that would be just to get rid of global variables and functions).
However: you can create function objects in runtime. The constructor looks like types.FunctionType((code, globals[, name[, argdefs[, closure]]]). There you can replace the global namespace:
def line(x, a0, b0):
return a + x * b # will be an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
line = types.FunctionType(line.__code__, {})
y1 = line(1, a, b) # fails since global name is not defined
You can of course clean this up by defining your own decorator:
import types
noglobal = lambda f: types.FunctionType(f.__code__, {}, argdefs=f.__defaults__)
#noglobal
def f():
return x
x = 5
f() # will fail
Strictly speaking you do not forbid it to access global variables, you just make the function believe there is no variables in global namespace. Actually you can also use this to emulate static variables since if it declares an variable to be global and assign to it it will end up in it's own sandbox of global namespace.
If you want to be able to access part of the global namespace then you'll need to populate the functions global sandbox with what you want it to see.

No, you cannot tell Python not to look names up in the global scope.
If you could, you would not be able to use any other classes or functions defined in the module, no objects imported from other modules, nor could you use built-in names. Your function namespace becomes a desert devoid of almost everything it needs, and the only way out would be to import everything into the local namespace. For every single function in your module.
Rather than try to break global lookups, keep your global namespace clean. Don't add globals that you don't need to share with other scopes in the module. Use a main() function for example, to encapsulate what are really just locals.
Also, add unittesting. Refactoring without (even just a few) tests is always prone to create bugs otherwise.

With #skyking's answer, I was unable to access any imports (I could not even use print). Also, functions with optional arguments are broken (compare How can an optional parameter become required?).
#Ax3l's comment improved that a bit. Still I was unable to access imported variables (from module import var).
Therefore, I propose this:
def noglobal(f):
return types.FunctionType(f.__code__, globals().copy(), f.__name__, f.__defaults__, f.__closure__)
For each function decorated with #noglobal, that creates a copy of the globals() defined so far. This keeps imported variables (usually imported at the top of the document) accessible. If you do it like me, defining your functions first and then your variables, this will achieve the desired effect of being able to access imported variables in your function, but not the ones you define in your code. Since copy() creates a shallow copy (Understanding dict.copy() - shallow or deep?), this should be pretty memory-efficient, too.
Note that this way, a function can only call functions defined above itself, so you may need to reorder your code.
For the record, I copy #Ax3l's version from his Gist:
def imports():
for name, val in globals().items():
# module imports
if isinstance(val, types.ModuleType):
yield name, val
# functions / callables
if hasattr(val, '__call__'):
yield name, val
noglobal = lambda fn: types.FunctionType(fn.__code__, dict(imports()))

To discourage global variable lookup, move your function into another module. Unless it inspects the call stack or imports your calling module explicitly; it won't have access to the globals from the module that calls it.
In practice, move your code into a main() function, to avoid creating unnecessary global variables.
If you use globals because several functions need to manipulate shared state then move the code into a class.

As mentioned by #bers the decorator by #skykings breaks most python functionality inside the function, such as print() and the import statement. #bers hacked around the import statement by adding the currently imported modules from globals() at the time of decorator definition.
This inspired me to write yet another decorator that hopefully does what most people who come looking at this post actually want. The underlying problem is that the new function created by the previous decorators lacked the __builtins__ variable which contains all of the standard built-in python functions (e.g. print) available in a freshly opened interpreter.
import types
import builtins
def no_globals(f):
'''
A function decorator that prevents functions from looking up variables in outer scope.
'''
# need builtins in globals otherwise can't import or print inside the function
new_globals = {'__builtins__': builtins}
new_f = types.FunctionType(f.__code__, globals=new_globals, argdefs=f.__defaults__)
new_f.__annotations__ = f.__annotations__ # for some reason annotations aren't copied over
return new_f
Then the usage goes as the following
#no_globals
def f1():
return x
x = 5
f1() # should raise NameError
#no_globals
def f2(x):
import numpy as np
print(x)
return np.sin(x)
x = 5
f2(x) # should print 5 and return -0.9589242746631385

Theoretically you can use your own decorator that removes globals() while a function call. It is some overhead to hide all globals() but, if there are not too many globals() it could be useful. During the operation we do not create/remove global objects, we just overwrites references in dictionary which refers to global objects. But do not remove special globals() (like __builtins__) and modules. Probably you do not want to remove callables from global scope too.
from types import ModuleType
import re
# the decorator to hide global variables
def noglobs(f):
def inner(*args, **kwargs):
RE_NOREPLACE = '__\w+__'
old_globals = {}
# removing keys from globals() storing global values in old_globals
for key, val in globals().iteritems():
if re.match(RE_NOREPLACE, key) is None and not isinstance(val, ModuleType) and not callable(val):
old_globals.update({key: val})
for key in old_globals.keys():
del globals()[key]
result = f(*args, **kwargs)
# restoring globals
for key in old_globals.iterkeys():
globals()[key] = old_globals[key]
return result
return inner
# the example of usage
global_var = 'hello'
#noglobs
def no_globals_func():
try:
print 'Can I use %s here?' % global_var
except NameError:
print 'Name "global_var" in unavailable here'
def globals_func():
print 'Can I use %s here?' % global_var
globals_func()
no_globals_func()
print 'Can I use %s here?' % global_var
...
Can I use hello here?
Name "global_var" in unavailable here
Can I use hello here?
Or, you can iterate over all global callables (i.e. functions) in your module and decorate them dynamically (it's little more code).
The code is for Python 2, I think it's possible to create a very similar code for Python 3.

Closing over a local value

I read the following in Google's Python styleguide:
"Avoid nested functions or classes except when closing over a local value".
What does "closing over a local value" mean?
The complete relevant section is below:
2.6 Nested/Local/Inner Classes and Functions
Nested local functions or classes are fine when used to close over a local variable. Inner
classes are fine.
2.6.1 Definition
A class can be defined inside of a method, function, or class. A function can be defined inside a method or function. Nested functions have read-only access to variables defined in
enclosing scopes.
2.6.2 Pros
Allows definition of utility classes and functions that are only used inside of a very limited scope. Very ADT-y. Commonly used
for implementing decorators.
2.6.3 Cons
Instances of nested or local classes cannot be pickled. Nested functions and classes cannot be directly tested. Nesting can make your outer function longer and less readable.
2.6.4 Decision
They are fine with some caveats. Avoid nested functions or classes except when closing over a local value. Do not nest a
function just to hide it from users of a module. Instead, prefix its
name with an _ at the module level so that it can still be accessed by
tests.

It means unless you create a closure. A closure is when a free variable refers to a variable in an enclosing scope. So for example:
def foo():
bar = 42
def baz():
print(bar)
return baz
foo()()
This will print 42, because baz is a closure over the variable bar in the local scope of foo. Note, you can even introspect this:
f = foo()
print(f.__closure__)
So, essentially the guide is telling you to use a nested function only when it is actually useful for something, a small contrived example could be a function factory:
def make_adder(n):
def add(x):
return n + x
return add
add2 = make_adder(2)
add5 = make_adder(5)
print(add2(1), add5(1))
add2 and add5 are closures over n.
Some people might want to nest a function merely to hide it from the global scope, something like:
def foo(n):
def frobnicate(x, y):
return x + y
m = n + 42
return frobnicate(m, 11)
The style guide is saying don't do that, just do:
def frobnicate(x, y):
return x + y
def foo(n):
m = n + 42
return frobnicate(m, 11)

Python 3 - How to exec a string as if it were substituted directly?

Problem Description
I am curious if it is possible to exec a string within a function as if the string were substituted for exec directly (with appropriate indentation). I understand that in 99.9% of cases, you shouldn't be using exec but I'm more interested in if this can be done rather than if it should be done.
The behavior I want is equivalent to:
GLOBAL_CONSTANT = 1
def test_func():
def A():
return GLOBAL_CONSTANT
def B():
return A()
return B
func = test_func()
assert func() == 1
But I am given instead:
GLOBAL_CONSTANT = 1
EXEC_STR = """
def A():
return GLOBAL_CONSTANT
def B():
return A()
"""
def exec_and_extract(exec_str, var_name):
# Insert code here
func = exec_and_extract(EXEC_STR, 'B')
assert func() == 1
Failed Attempts
def exec_and_extract(exec_str, var_name):
exec(EXEC_STR) # equivalent to exec(EXEC_STR, globals(), locals())
return locals()[var_name]
NameError: name 'A' is not defined when calling func() since A and B exist inside exec_and_extract's locals() but the execution context while running A or B is exec_and_extract's globals().
def exec_and_extract(exec_str, var_name):
exec(EXEC_STR, locals()) # equivalent to exec(EXEC_STR, locals(), locals())
return locals()[var_name]
NameError: name 'GLOBAL_CONSTANT' is not defined when calling A from inside func() since the execution context of A is exec_and_extract's locals() which does not contain GLOBAL_CONSTANT.
def exec_and_extract(exec_str, var_name):
exec(EXEC_STR, globals()) # equivalent to exec(EXEC_STR, globals(), globals())
return globals()[var_name]
Works but pollutes global namespace, not equivalent.
def exec_and_extract(exec_str, var_name):
locals().update(globals())
exec(EXEC_STR, locals()) # equivalent to exec(EXEC_STR, locals(), locals())
return locals()[var_name]
Works but requires copying the entire content of exec_and_extract's globals() into its locals() which is a waste of time if globals() is large (of course not applicable in this contrived example). Additionally, is subtly not the same as the "paste in code" version since if one of the arguments to exec_and_extract happened to be GLOBAL_CONSTANT (a terrible argument name), the behavior would be different ("paste in" version would use the argument value while this code would use the global constant value).
Further Constraints
Trying to cover any "loopholes" in the problem statement:
The exec_str value should represent arbitrary code that can access global or local scope variables.
Solution should not require analysis of what global scope variables are accessed within exec_str.
There should be no "pollution" between subsequent calls to exec_and_extract (in global namespace or otherwise). i.e. In this example, execution of EXEC_STR should not leave A around to be referenceable in future calls to exec_and_extract.

This is impossible. exec interacts badly with local variable scope mechanics, and it is far too restricted for anything like this to work. In fact, literally any local variable binding operation in the executed string is undefined behavior, including plain assignment, function definitions, class definitions, imports, and more, if you call exec with the default locals. Quoting the docs:
The default locals act as described for function locals() below: modifications to the default locals dictionary should not be attempted. Pass an explicit locals dictionary if you need to see effects of the code on locals after function exec() returns.
Additionally, code executed by exec cannot return, break, yield, or perform other control flow on behalf of the caller. It can break loops that are part of the executed code, or return from functions defined in the executed code, but it cannot interact with its caller's control flow.
If you're willing to sacrifice the requirement to be able to interact with the calling function's locals (as you mentioned in the comments), and you don't care about interacting with the caller's control flow, then you could insert the code's AST into the body of a new function definition and execute that:
import ast
import sys
def exec_and_extract(code_string, var):
original_ast = ast.parse(code_string)
new_ast = ast.parse('def f(): return ' + var)
fdef = new_ast.body[0]
fdef.body = original_ast.body + fdef.body
code_obj = compile(new_ast, '<string>', 'exec')
gvars = sys._getframe(1).f_globals
lvars = {}
exec(code_obj, gvars, lvars)
return lvars['f']()
I've used an AST-based approach instead of string formatting to avoid problems like accidentally inserting extra indentation into triple-quoted strings in the input.
inspect lets us use the globals of whoever called exec_and_extract, rather than exec_and_extract's own globals, even if the caller is in a different module.
Functions defined in the executed code see the actual globals rather than a copy.
The extra wrapper function in the modified AST avoids some scope issues that would occur otherwise; particularly, B wouldn't be able to see A's definition in your example code otherwise.

Works but pollutes global namespace, not equivalent.
Then how about making a copy of the globals() dict, and retrieving B from that?
def exec_and_extract(exec_str, var_name):
env = dict(globals())
env.update(locals())
exec(EXEC_STR, env)
return env[var_name]
This still works, and doesn't pollute the global namespace.

#user2357112supportsMonica (Responding to comment in thread since this contains code block)
Seems like something like this might work:
def exec_and_extract(exec_str, var_name):
env = {}
modified_exec_str = """def wrapper():
{body}
return {var_name}
""".format(body=textwrap.indent(exec_str, ' '), var_name=var_name)
exec(modified_exec_str, globals(), env)
return env['wrapper']()
This allows accessing of global scope including future changes as well as accessing of other variables defined inside the exec_str.

Forbid the use of global variables inside a function [duplicate]

In short, the question: Is there a way to prevent Python from looking up variables outside the current scope?
Details:
Python looks for variable definitions in outer scopes if they are not defined in the current scope. Thus, code like this is liable to break when not being careful during refactoring:
def line(x, a, b):
return a + x * b
a, b = 1, 1
y1 = line(1, a, b)
y2 = line(1, 2, 3)
If I renamed the function arguments, but forgot to rename them inside the function body, the code would still run:
def line(x, a0, b0):
return a + x * b # not an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
y2 = line(1, 2, 3) # wrong result
I know it is bad practice to shadow names from outer scopes. But sometimes we do it anyway...
Is there a way to prevent Python from looking up variables outside the current scope? (So that accessing a or b raises an Error in the second example.)

Yes, maybe not in general. However you can do it with functions.
The thing you want to do is to have the function's global to be empty. You can't replace the globals and you don't want to modify it's content (becaus
that would be just to get rid of global variables and functions).
However: you can create function objects in runtime. The constructor looks like types.FunctionType((code, globals[, name[, argdefs[, closure]]]). There you can replace the global namespace:
def line(x, a0, b0):
return a + x * b # will be an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
line = types.FunctionType(line.__code__, {})
y1 = line(1, a, b) # fails since global name is not defined
You can of course clean this up by defining your own decorator:
import types
noglobal = lambda f: types.FunctionType(f.__code__, {}, argdefs=f.__defaults__)
#noglobal
def f():
return x
x = 5
f() # will fail
Strictly speaking you do not forbid it to access global variables, you just make the function believe there is no variables in global namespace. Actually you can also use this to emulate static variables since if it declares an variable to be global and assign to it it will end up in it's own sandbox of global namespace.
If you want to be able to access part of the global namespace then you'll need to populate the functions global sandbox with what you want it to see.

No, you cannot tell Python not to look names up in the global scope.
If you could, you would not be able to use any other classes or functions defined in the module, no objects imported from other modules, nor could you use built-in names. Your function namespace becomes a desert devoid of almost everything it needs, and the only way out would be to import everything into the local namespace. For every single function in your module.
Rather than try to break global lookups, keep your global namespace clean. Don't add globals that you don't need to share with other scopes in the module. Use a main() function for example, to encapsulate what are really just locals.
Also, add unittesting. Refactoring without (even just a few) tests is always prone to create bugs otherwise.

With #skyking's answer, I was unable to access any imports (I could not even use print). Also, functions with optional arguments are broken (compare How can an optional parameter become required?).
#Ax3l's comment improved that a bit. Still I was unable to access imported variables (from module import var).
Therefore, I propose this:
def noglobal(f):
return types.FunctionType(f.__code__, globals().copy(), f.__name__, f.__defaults__, f.__closure__)
For each function decorated with #noglobal, that creates a copy of the globals() defined so far. This keeps imported variables (usually imported at the top of the document) accessible. If you do it like me, defining your functions first and then your variables, this will achieve the desired effect of being able to access imported variables in your function, but not the ones you define in your code. Since copy() creates a shallow copy (Understanding dict.copy() - shallow or deep?), this should be pretty memory-efficient, too.
Note that this way, a function can only call functions defined above itself, so you may need to reorder your code.
For the record, I copy #Ax3l's version from his Gist:
def imports():
for name, val in globals().items():
# module imports
if isinstance(val, types.ModuleType):
yield name, val
# functions / callables
if hasattr(val, '__call__'):
yield name, val
noglobal = lambda fn: types.FunctionType(fn.__code__, dict(imports()))

To discourage global variable lookup, move your function into another module. Unless it inspects the call stack or imports your calling module explicitly; it won't have access to the globals from the module that calls it.
In practice, move your code into a main() function, to avoid creating unnecessary global variables.
If you use globals because several functions need to manipulate shared state then move the code into a class.

As mentioned by #bers the decorator by #skykings breaks most python functionality inside the function, such as print() and the import statement. #bers hacked around the import statement by adding the currently imported modules from globals() at the time of decorator definition.
This inspired me to write yet another decorator that hopefully does what most people who come looking at this post actually want. The underlying problem is that the new function created by the previous decorators lacked the __builtins__ variable which contains all of the standard built-in python functions (e.g. print) available in a freshly opened interpreter.
import types
import builtins
def no_globals(f):
'''
A function decorator that prevents functions from looking up variables in outer scope.
'''
# need builtins in globals otherwise can't import or print inside the function
new_globals = {'__builtins__': builtins}
new_f = types.FunctionType(f.__code__, globals=new_globals, argdefs=f.__defaults__)
new_f.__annotations__ = f.__annotations__ # for some reason annotations aren't copied over
return new_f
Then the usage goes as the following
#no_globals
def f1():
return x
x = 5
f1() # should raise NameError
#no_globals
def f2(x):
import numpy as np
print(x)
return np.sin(x)
x = 5
f2(x) # should print 5 and return -0.9589242746631385

Theoretically you can use your own decorator that removes globals() while a function call. It is some overhead to hide all globals() but, if there are not too many globals() it could be useful. During the operation we do not create/remove global objects, we just overwrites references in dictionary which refers to global objects. But do not remove special globals() (like __builtins__) and modules. Probably you do not want to remove callables from global scope too.
from types import ModuleType
import re
# the decorator to hide global variables
def noglobs(f):
def inner(*args, **kwargs):
RE_NOREPLACE = '__\w+__'
old_globals = {}
# removing keys from globals() storing global values in old_globals
for key, val in globals().iteritems():
if re.match(RE_NOREPLACE, key) is None and not isinstance(val, ModuleType) and not callable(val):
old_globals.update({key: val})
for key in old_globals.keys():
del globals()[key]
result = f(*args, **kwargs)
# restoring globals
for key in old_globals.iterkeys():
globals()[key] = old_globals[key]
return result
return inner
# the example of usage
global_var = 'hello'
#noglobs
def no_globals_func():
try:
print 'Can I use %s here?' % global_var
except NameError:
print 'Name "global_var" in unavailable here'
def globals_func():
print 'Can I use %s here?' % global_var
globals_func()
no_globals_func()
print 'Can I use %s here?' % global_var
...
Can I use hello here?
Name "global_var" in unavailable here
Can I use hello here?
Or, you can iterate over all global callables (i.e. functions) in your module and decorate them dynamically (it's little more code).
The code is for Python 2, I think it's possible to create a very similar code for Python 3.

Using global name in a nested function

As I understand the global statement in the code below, it should prevent function_two from rebinding the name test and instead modify test in function_one. However, I get NameError: global name 'test' is not defined.
def function_one():
test = 1
def function_two():
global test
test += 1
function_two()
print test
function_one()
I have looked and I can't find an example like this. What am I missing?

Python 2 does not support the concept of a non-local. Closures (accessing test from a parent function) only support read access, not assignment in Python 2.
The global keyword really does mean global, e.g. that the name lives in the module (global) namespace. The namespace of the function_one() function is not global, it is local (to that function).
In Python 3, you can mark a name as nonlocal, which would make your example work as expected. See PEP 3104 - Access to Names in Outer Scopes.
In Python 2, you'll have to resort to tricks instead. Make the name an attribute of the nested function, for example. 'reading' the function object as a closure is allowed, as is setting attributes on such closed-over objects:
def function_one():
def function_two():
function_two.test += 1
function_two.test = 1
function_two()
print test
Another trick is to use a mutable object, such as a list or a dictionary. Again, you are only reading the closed-over name, then altering the resulting object directly:
def function_one():
test = [1]
def function_two():
test[0] += 1
function_two()
print test[0]