Related
This question already has answers here:
Short description of the scoping rules?
(9 answers)
Closed 2 years ago.
I seem to have misunderstood something about Python variable binding. What are the precise rules for deciding which variable is accessed given a nested scope with shadowing names?
Let me illustrate with some examples. First the basic shadow.
a = 1
def foo():
a = 2
def _foo():
return a
return _foo()
print(foo()) # -> 2
Everything is fine here. The value is overwritten and returned accordingly. However, if the value is changed after the function definition, it is still the inner value:
def bar():
def _bar():
return a
a = 2
return _bar()
print(bar()) # -> 2
What's more, defining a function that references a non-existent variable is possible.
def baz():
def _baz():
return b
return _baz()
Then, if b is defined later, the function can be executed. But not if is defined in another inner scope:
def qux(f):
b = 3
return f()
print(qux(baz())) # -> NameError
Now all of these cases could be explained by having Python know about lines that come later in the program, but that conflicts with my knowledge of Python being an interpreted language, advancing line by line. So are statements parsed at once instead of line by line?
A weird behaviour with shadowing class attributes throws me off a bit more.
class C:
a = 2
b = a
def meth(self):
return a
c = meth
print(C.b, C().meth(), C.c) # -> 2 1 C.meth
Here a is defined as a class attribute and is successfully used in b, but this does not carry over to the method definition. The method itself can be used in later attributes, but not for example in other methods without going through self.
Is my guess about the binding happening all at once correct? And in that case are class bodies an exception by design, or are they not a scope at all? Or is something else going on here?
I think you might be overthinking it.
By default, variables when created are put in the narrowest enclosing function's scope.
Variables from all enclosing scopes are available in a read-only capacity, be that an enclosing function's scope or the global scope. If you try to assign to this, it'll create a new variable in the narrowest enclosing scope, shadowing those outside. Using the global keyword to bring an external variable into the local scope will stop this from happening, allowing you to assign things to the non-local scope.
Additionally, keep in mind that functions are compiled and evaluated at the time when the def statement is interpreted. For nested functions, essentially, every new call re-evaluates the inner functions. This also means that inner functions have read-only access to the scope of the outer functions. Same rules as usual.
Your bar() example works because, by the time python tries to access the variable a, it is present in at least one of the enclosing scopes. Python doesn't check these things until the last possible moment. Your qux() example doesn't work because the scope in which b is declared does not enclose the scope where _baz() is defined, and thus is not accessible.
Class scopes are weird. When the class is evaluated, all variables defined inside it are bound to the class. However, the class doesn't really count as a scope of its own, for the purpose of the methods enclosed inside it. Think of meth() as an unbound function, declared in the global scope, which C.meth refers to (and, now, C.c). Calling a function via dot notation is a syntactic shorthand:
# the following two are identical
C().meth()
C.meth(C())
and while C.meth is technically bound to C, it's not enclosed in C's class-level namespace. Trying to do C().meth() will fail, because a is not defined with respect to the function. (note that if a is defined in the global scope, the function will work as expected - C.meth() has the global scope as a parent, not C's class-level scope).
In short, the question: Is there a way to prevent Python from looking up variables outside the current scope?
Details:
Python looks for variable definitions in outer scopes if they are not defined in the current scope. Thus, code like this is liable to break when not being careful during refactoring:
def line(x, a, b):
return a + x * b
a, b = 1, 1
y1 = line(1, a, b)
y2 = line(1, 2, 3)
If I renamed the function arguments, but forgot to rename them inside the function body, the code would still run:
def line(x, a0, b0):
return a + x * b # not an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
y2 = line(1, 2, 3) # wrong result
I know it is bad practice to shadow names from outer scopes. But sometimes we do it anyway...
Is there a way to prevent Python from looking up variables outside the current scope? (So that accessing a or b raises an Error in the second example.)
Yes, maybe not in general. However you can do it with functions.
The thing you want to do is to have the function's global to be empty. You can't replace the globals and you don't want to modify it's content (becaus
that would be just to get rid of global variables and functions).
However: you can create function objects in runtime. The constructor looks like types.FunctionType((code, globals[, name[, argdefs[, closure]]]). There you can replace the global namespace:
def line(x, a0, b0):
return a + x * b # will be an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
line = types.FunctionType(line.__code__, {})
y1 = line(1, a, b) # fails since global name is not defined
You can of course clean this up by defining your own decorator:
import types
noglobal = lambda f: types.FunctionType(f.__code__, {}, argdefs=f.__defaults__)
#noglobal
def f():
return x
x = 5
f() # will fail
Strictly speaking you do not forbid it to access global variables, you just make the function believe there is no variables in global namespace. Actually you can also use this to emulate static variables since if it declares an variable to be global and assign to it it will end up in it's own sandbox of global namespace.
If you want to be able to access part of the global namespace then you'll need to populate the functions global sandbox with what you want it to see.
No, you cannot tell Python not to look names up in the global scope.
If you could, you would not be able to use any other classes or functions defined in the module, no objects imported from other modules, nor could you use built-in names. Your function namespace becomes a desert devoid of almost everything it needs, and the only way out would be to import everything into the local namespace. For every single function in your module.
Rather than try to break global lookups, keep your global namespace clean. Don't add globals that you don't need to share with other scopes in the module. Use a main() function for example, to encapsulate what are really just locals.
Also, add unittesting. Refactoring without (even just a few) tests is always prone to create bugs otherwise.
With #skyking's answer, I was unable to access any imports (I could not even use print). Also, functions with optional arguments are broken (compare How can an optional parameter become required?).
#Ax3l's comment improved that a bit. Still I was unable to access imported variables (from module import var).
Therefore, I propose this:
def noglobal(f):
return types.FunctionType(f.__code__, globals().copy(), f.__name__, f.__defaults__, f.__closure__)
For each function decorated with #noglobal, that creates a copy of the globals() defined so far. This keeps imported variables (usually imported at the top of the document) accessible. If you do it like me, defining your functions first and then your variables, this will achieve the desired effect of being able to access imported variables in your function, but not the ones you define in your code. Since copy() creates a shallow copy (Understanding dict.copy() - shallow or deep?), this should be pretty memory-efficient, too.
Note that this way, a function can only call functions defined above itself, so you may need to reorder your code.
For the record, I copy #Ax3l's version from his Gist:
def imports():
for name, val in globals().items():
# module imports
if isinstance(val, types.ModuleType):
yield name, val
# functions / callables
if hasattr(val, '__call__'):
yield name, val
noglobal = lambda fn: types.FunctionType(fn.__code__, dict(imports()))
To discourage global variable lookup, move your function into another module. Unless it inspects the call stack or imports your calling module explicitly; it won't have access to the globals from the module that calls it.
In practice, move your code into a main() function, to avoid creating unnecessary global variables.
If you use globals because several functions need to manipulate shared state then move the code into a class.
As mentioned by #bers the decorator by #skykings breaks most python functionality inside the function, such as print() and the import statement. #bers hacked around the import statement by adding the currently imported modules from globals() at the time of decorator definition.
This inspired me to write yet another decorator that hopefully does what most people who come looking at this post actually want. The underlying problem is that the new function created by the previous decorators lacked the __builtins__ variable which contains all of the standard built-in python functions (e.g. print) available in a freshly opened interpreter.
import types
import builtins
def no_globals(f):
'''
A function decorator that prevents functions from looking up variables in outer scope.
'''
# need builtins in globals otherwise can't import or print inside the function
new_globals = {'__builtins__': builtins}
new_f = types.FunctionType(f.__code__, globals=new_globals, argdefs=f.__defaults__)
new_f.__annotations__ = f.__annotations__ # for some reason annotations aren't copied over
return new_f
Then the usage goes as the following
#no_globals
def f1():
return x
x = 5
f1() # should raise NameError
#no_globals
def f2(x):
import numpy as np
print(x)
return np.sin(x)
x = 5
f2(x) # should print 5 and return -0.9589242746631385
Theoretically you can use your own decorator that removes globals() while a function call. It is some overhead to hide all globals() but, if there are not too many globals() it could be useful. During the operation we do not create/remove global objects, we just overwrites references in dictionary which refers to global objects. But do not remove special globals() (like __builtins__) and modules. Probably you do not want to remove callables from global scope too.
from types import ModuleType
import re
# the decorator to hide global variables
def noglobs(f):
def inner(*args, **kwargs):
RE_NOREPLACE = '__\w+__'
old_globals = {}
# removing keys from globals() storing global values in old_globals
for key, val in globals().iteritems():
if re.match(RE_NOREPLACE, key) is None and not isinstance(val, ModuleType) and not callable(val):
old_globals.update({key: val})
for key in old_globals.keys():
del globals()[key]
result = f(*args, **kwargs)
# restoring globals
for key in old_globals.iterkeys():
globals()[key] = old_globals[key]
return result
return inner
# the example of usage
global_var = 'hello'
#noglobs
def no_globals_func():
try:
print 'Can I use %s here?' % global_var
except NameError:
print 'Name "global_var" in unavailable here'
def globals_func():
print 'Can I use %s here?' % global_var
globals_func()
no_globals_func()
print 'Can I use %s here?' % global_var
...
Can I use hello here?
Name "global_var" in unavailable here
Can I use hello here?
Or, you can iterate over all global callables (i.e. functions) in your module and decorate them dynamically (it's little more code).
The code is for Python 2, I think it's possible to create a very similar code for Python 3.
Is scope of a function is same as scope of a variable in python.
Like for example we have global, local and nonlocal scope for a variable, is there something like global, local and non local function ?
There is no difference. A function is a first-class object, and can be bound to a name like any other object. Names have scopes in Python.
A def statement is similar to an assignment statement; it creates a function object, then binds a name to that assignment. For example, the following are very similar:
def foo():
return 3
foo = lambda: 3
def is not indispensable, though, since there is no way to define an arbitrary function using an expression alone. (The body of a lambda expression must be a single expression itself, whereas the body of a def statement can be an arbitrary list of statements.)
In short, the question: Is there a way to prevent Python from looking up variables outside the current scope?
Details:
Python looks for variable definitions in outer scopes if they are not defined in the current scope. Thus, code like this is liable to break when not being careful during refactoring:
def line(x, a, b):
return a + x * b
a, b = 1, 1
y1 = line(1, a, b)
y2 = line(1, 2, 3)
If I renamed the function arguments, but forgot to rename them inside the function body, the code would still run:
def line(x, a0, b0):
return a + x * b # not an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
y2 = line(1, 2, 3) # wrong result
I know it is bad practice to shadow names from outer scopes. But sometimes we do it anyway...
Is there a way to prevent Python from looking up variables outside the current scope? (So that accessing a or b raises an Error in the second example.)
Yes, maybe not in general. However you can do it with functions.
The thing you want to do is to have the function's global to be empty. You can't replace the globals and you don't want to modify it's content (becaus
that would be just to get rid of global variables and functions).
However: you can create function objects in runtime. The constructor looks like types.FunctionType((code, globals[, name[, argdefs[, closure]]]). There you can replace the global namespace:
def line(x, a0, b0):
return a + x * b # will be an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
line = types.FunctionType(line.__code__, {})
y1 = line(1, a, b) # fails since global name is not defined
You can of course clean this up by defining your own decorator:
import types
noglobal = lambda f: types.FunctionType(f.__code__, {}, argdefs=f.__defaults__)
#noglobal
def f():
return x
x = 5
f() # will fail
Strictly speaking you do not forbid it to access global variables, you just make the function believe there is no variables in global namespace. Actually you can also use this to emulate static variables since if it declares an variable to be global and assign to it it will end up in it's own sandbox of global namespace.
If you want to be able to access part of the global namespace then you'll need to populate the functions global sandbox with what you want it to see.
No, you cannot tell Python not to look names up in the global scope.
If you could, you would not be able to use any other classes or functions defined in the module, no objects imported from other modules, nor could you use built-in names. Your function namespace becomes a desert devoid of almost everything it needs, and the only way out would be to import everything into the local namespace. For every single function in your module.
Rather than try to break global lookups, keep your global namespace clean. Don't add globals that you don't need to share with other scopes in the module. Use a main() function for example, to encapsulate what are really just locals.
Also, add unittesting. Refactoring without (even just a few) tests is always prone to create bugs otherwise.
With #skyking's answer, I was unable to access any imports (I could not even use print). Also, functions with optional arguments are broken (compare How can an optional parameter become required?).
#Ax3l's comment improved that a bit. Still I was unable to access imported variables (from module import var).
Therefore, I propose this:
def noglobal(f):
return types.FunctionType(f.__code__, globals().copy(), f.__name__, f.__defaults__, f.__closure__)
For each function decorated with #noglobal, that creates a copy of the globals() defined so far. This keeps imported variables (usually imported at the top of the document) accessible. If you do it like me, defining your functions first and then your variables, this will achieve the desired effect of being able to access imported variables in your function, but not the ones you define in your code. Since copy() creates a shallow copy (Understanding dict.copy() - shallow or deep?), this should be pretty memory-efficient, too.
Note that this way, a function can only call functions defined above itself, so you may need to reorder your code.
For the record, I copy #Ax3l's version from his Gist:
def imports():
for name, val in globals().items():
# module imports
if isinstance(val, types.ModuleType):
yield name, val
# functions / callables
if hasattr(val, '__call__'):
yield name, val
noglobal = lambda fn: types.FunctionType(fn.__code__, dict(imports()))
To discourage global variable lookup, move your function into another module. Unless it inspects the call stack or imports your calling module explicitly; it won't have access to the globals from the module that calls it.
In practice, move your code into a main() function, to avoid creating unnecessary global variables.
If you use globals because several functions need to manipulate shared state then move the code into a class.
As mentioned by #bers the decorator by #skykings breaks most python functionality inside the function, such as print() and the import statement. #bers hacked around the import statement by adding the currently imported modules from globals() at the time of decorator definition.
This inspired me to write yet another decorator that hopefully does what most people who come looking at this post actually want. The underlying problem is that the new function created by the previous decorators lacked the __builtins__ variable which contains all of the standard built-in python functions (e.g. print) available in a freshly opened interpreter.
import types
import builtins
def no_globals(f):
'''
A function decorator that prevents functions from looking up variables in outer scope.
'''
# need builtins in globals otherwise can't import or print inside the function
new_globals = {'__builtins__': builtins}
new_f = types.FunctionType(f.__code__, globals=new_globals, argdefs=f.__defaults__)
new_f.__annotations__ = f.__annotations__ # for some reason annotations aren't copied over
return new_f
Then the usage goes as the following
#no_globals
def f1():
return x
x = 5
f1() # should raise NameError
#no_globals
def f2(x):
import numpy as np
print(x)
return np.sin(x)
x = 5
f2(x) # should print 5 and return -0.9589242746631385
Theoretically you can use your own decorator that removes globals() while a function call. It is some overhead to hide all globals() but, if there are not too many globals() it could be useful. During the operation we do not create/remove global objects, we just overwrites references in dictionary which refers to global objects. But do not remove special globals() (like __builtins__) and modules. Probably you do not want to remove callables from global scope too.
from types import ModuleType
import re
# the decorator to hide global variables
def noglobs(f):
def inner(*args, **kwargs):
RE_NOREPLACE = '__\w+__'
old_globals = {}
# removing keys from globals() storing global values in old_globals
for key, val in globals().iteritems():
if re.match(RE_NOREPLACE, key) is None and not isinstance(val, ModuleType) and not callable(val):
old_globals.update({key: val})
for key in old_globals.keys():
del globals()[key]
result = f(*args, **kwargs)
# restoring globals
for key in old_globals.iterkeys():
globals()[key] = old_globals[key]
return result
return inner
# the example of usage
global_var = 'hello'
#noglobs
def no_globals_func():
try:
print 'Can I use %s here?' % global_var
except NameError:
print 'Name "global_var" in unavailable here'
def globals_func():
print 'Can I use %s here?' % global_var
globals_func()
no_globals_func()
print 'Can I use %s here?' % global_var
...
Can I use hello here?
Name "global_var" in unavailable here
Can I use hello here?
Or, you can iterate over all global callables (i.e. functions) in your module and decorate them dynamically (it's little more code).
The code is for Python 2, I think it's possible to create a very similar code for Python 3.
In short, the question: Is there a way to prevent Python from looking up variables outside the current scope?
Details:
Python looks for variable definitions in outer scopes if they are not defined in the current scope. Thus, code like this is liable to break when not being careful during refactoring:
def line(x, a, b):
return a + x * b
a, b = 1, 1
y1 = line(1, a, b)
y2 = line(1, 2, 3)
If I renamed the function arguments, but forgot to rename them inside the function body, the code would still run:
def line(x, a0, b0):
return a + x * b # not an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
y2 = line(1, 2, 3) # wrong result
I know it is bad practice to shadow names from outer scopes. But sometimes we do it anyway...
Is there a way to prevent Python from looking up variables outside the current scope? (So that accessing a or b raises an Error in the second example.)
Yes, maybe not in general. However you can do it with functions.
The thing you want to do is to have the function's global to be empty. You can't replace the globals and you don't want to modify it's content (becaus
that would be just to get rid of global variables and functions).
However: you can create function objects in runtime. The constructor looks like types.FunctionType((code, globals[, name[, argdefs[, closure]]]). There you can replace the global namespace:
def line(x, a0, b0):
return a + x * b # will be an error
a, b = 1, 1
y1 = line(1, a, b) # correct result by coincidence
line = types.FunctionType(line.__code__, {})
y1 = line(1, a, b) # fails since global name is not defined
You can of course clean this up by defining your own decorator:
import types
noglobal = lambda f: types.FunctionType(f.__code__, {}, argdefs=f.__defaults__)
#noglobal
def f():
return x
x = 5
f() # will fail
Strictly speaking you do not forbid it to access global variables, you just make the function believe there is no variables in global namespace. Actually you can also use this to emulate static variables since if it declares an variable to be global and assign to it it will end up in it's own sandbox of global namespace.
If you want to be able to access part of the global namespace then you'll need to populate the functions global sandbox with what you want it to see.
No, you cannot tell Python not to look names up in the global scope.
If you could, you would not be able to use any other classes or functions defined in the module, no objects imported from other modules, nor could you use built-in names. Your function namespace becomes a desert devoid of almost everything it needs, and the only way out would be to import everything into the local namespace. For every single function in your module.
Rather than try to break global lookups, keep your global namespace clean. Don't add globals that you don't need to share with other scopes in the module. Use a main() function for example, to encapsulate what are really just locals.
Also, add unittesting. Refactoring without (even just a few) tests is always prone to create bugs otherwise.
With #skyking's answer, I was unable to access any imports (I could not even use print). Also, functions with optional arguments are broken (compare How can an optional parameter become required?).
#Ax3l's comment improved that a bit. Still I was unable to access imported variables (from module import var).
Therefore, I propose this:
def noglobal(f):
return types.FunctionType(f.__code__, globals().copy(), f.__name__, f.__defaults__, f.__closure__)
For each function decorated with #noglobal, that creates a copy of the globals() defined so far. This keeps imported variables (usually imported at the top of the document) accessible. If you do it like me, defining your functions first and then your variables, this will achieve the desired effect of being able to access imported variables in your function, but not the ones you define in your code. Since copy() creates a shallow copy (Understanding dict.copy() - shallow or deep?), this should be pretty memory-efficient, too.
Note that this way, a function can only call functions defined above itself, so you may need to reorder your code.
For the record, I copy #Ax3l's version from his Gist:
def imports():
for name, val in globals().items():
# module imports
if isinstance(val, types.ModuleType):
yield name, val
# functions / callables
if hasattr(val, '__call__'):
yield name, val
noglobal = lambda fn: types.FunctionType(fn.__code__, dict(imports()))
To discourage global variable lookup, move your function into another module. Unless it inspects the call stack or imports your calling module explicitly; it won't have access to the globals from the module that calls it.
In practice, move your code into a main() function, to avoid creating unnecessary global variables.
If you use globals because several functions need to manipulate shared state then move the code into a class.
As mentioned by #bers the decorator by #skykings breaks most python functionality inside the function, such as print() and the import statement. #bers hacked around the import statement by adding the currently imported modules from globals() at the time of decorator definition.
This inspired me to write yet another decorator that hopefully does what most people who come looking at this post actually want. The underlying problem is that the new function created by the previous decorators lacked the __builtins__ variable which contains all of the standard built-in python functions (e.g. print) available in a freshly opened interpreter.
import types
import builtins
def no_globals(f):
'''
A function decorator that prevents functions from looking up variables in outer scope.
'''
# need builtins in globals otherwise can't import or print inside the function
new_globals = {'__builtins__': builtins}
new_f = types.FunctionType(f.__code__, globals=new_globals, argdefs=f.__defaults__)
new_f.__annotations__ = f.__annotations__ # for some reason annotations aren't copied over
return new_f
Then the usage goes as the following
#no_globals
def f1():
return x
x = 5
f1() # should raise NameError
#no_globals
def f2(x):
import numpy as np
print(x)
return np.sin(x)
x = 5
f2(x) # should print 5 and return -0.9589242746631385
Theoretically you can use your own decorator that removes globals() while a function call. It is some overhead to hide all globals() but, if there are not too many globals() it could be useful. During the operation we do not create/remove global objects, we just overwrites references in dictionary which refers to global objects. But do not remove special globals() (like __builtins__) and modules. Probably you do not want to remove callables from global scope too.
from types import ModuleType
import re
# the decorator to hide global variables
def noglobs(f):
def inner(*args, **kwargs):
RE_NOREPLACE = '__\w+__'
old_globals = {}
# removing keys from globals() storing global values in old_globals
for key, val in globals().iteritems():
if re.match(RE_NOREPLACE, key) is None and not isinstance(val, ModuleType) and not callable(val):
old_globals.update({key: val})
for key in old_globals.keys():
del globals()[key]
result = f(*args, **kwargs)
# restoring globals
for key in old_globals.iterkeys():
globals()[key] = old_globals[key]
return result
return inner
# the example of usage
global_var = 'hello'
#noglobs
def no_globals_func():
try:
print 'Can I use %s here?' % global_var
except NameError:
print 'Name "global_var" in unavailable here'
def globals_func():
print 'Can I use %s here?' % global_var
globals_func()
no_globals_func()
print 'Can I use %s here?' % global_var
...
Can I use hello here?
Name "global_var" in unavailable here
Can I use hello here?
Or, you can iterate over all global callables (i.e. functions) in your module and decorate them dynamically (it's little more code).
The code is for Python 2, I think it's possible to create a very similar code for Python 3.