Take the following code sample
var = True
def func1():
if var:
print("True")
else:
print("False")
# var = True
func1()
This prints True as one would expect.
However, if I uncomment # var = True, I get the error
UnboundLocalError: local variable 'var' referenced before assignment
Why does writing to a variable make an otherwise accessible variable inaccessible? What was the rationale behind this design choice?
Note I know how to solve it (with the global keyword). My question is why was it decided to behave this way.
Because:
Namespaces exist: the same variable name can be used at module level and inside functions, and have nothing to do with each other.
Python does not require variables to be declared, for ease of use
There still needs to be a way to distinguish between local and global variables
In cases where there is likely unexpected behavior, it is better to throw an error than to silently accept it
So Python chose the rule "if a variable name is assigned to within a function, then that name refers to a local variable" (because if it's never assigned to, it clearly isn't local as it never gets a value).
Your code could have been interpreted as using the module-level variable first (in the if: line), and then using the local variable later for the assignment. But, that will very often not be the expected behavior. So Guido decided that Python would not work like that, and throw the error instead.
Python defaults to implicit variable declaration via assignment, in order to remove the need for additional explicit declarations. Just "implicit declaration" leaves several options what assignment in nested scopes means, most prominently:
Assignment always declares a variable in the inner-most scope.
Assignment always declares a variable in the outer-most scope.
Assignment declares a variable in the inner-most scope, unless declared in any outer scope.
Assignment declares a variable in the inner-most scope, readable only after assignment.
The latter two options mean that a variable does not have a scope well-defined just by the assignment itself. They are "declaration via assignment + X" which can lead to unintended interactions between unrelated code.
That leaves the decision of whether "writing to a variable" should preferably happen to isolated local or shared global variables.
The Python designers consider it more important to explicitly mark writing to a global variable.
Python FAQ: Why am I getting an UnboundLocalError when the variable has a value?
[...]
This explicit declaration is required in order to remind you that (...) you are actually modifying the value of the variable in the outer scope
This is an intentional asymmetry towards purely reading globals, which is considered proper practice.
Python FAQ: What are the rules for local and global variables in Python?
[...]
On one hand, requiring global for assigned variables provides a bar against unintended side-effects. On the other hand, if global was required for all global references, you’d be using global all the time.
This is described in section 4.2.2 Resolution of names
When a name is not found at all, a NameError exception is raised. If the current scope is a function scope, and the name refers to a local variable that has not yet been bound to a value at the point where the name is used, an UnboundLocalError exception is raised. UnboundLocalError is a subclass of NameError.
If a name binding operation occurs anywhere within a code block, all uses of the name within the block are treated as references to the current block. This can lead to errors when a name is used within a block before it is bound. This rule is subtle. Python lacks declarations and allows name binding operations to occur anywhere within a code block. The local variables of a code block can be determined by scanning the entire text of the block for name binding operations.
If a variable name defined in the outer scope is used in a nested scope, it depends on what you do with it in this nested scope:
If you only read a variable, it is the same variable.
If you write to a variable, then Python automatically creates a new, local variable, different from the one in the outer scope.
This local variable prevents access to a variable with the same name in the outer scope.
So writing to a variable don't change its scope, it creates a different, local variable.
You are not able to read this local variable before assigning to it.
Related
I don't know how to explain the case, just try for yourself:
x = 7
def f():
if False:
global x
print("global")
x += 1
f()
print(x)
In my opinion, the above code should result in "UnboundLocalError: local variable 'x' referenced before assignment", but instead it not only works, but it changes the global variable 'x'.
A function gets a local namespace every time it is called. Python needs a way to know which variables are defined in that namespace or a containing namespace. Instead of requiring that all local variables be declared local, python's rule is that any variable that is assigned in a function is local to that function. It figures this out at compile time.
The global keyword does the opposite. It tells python that in this single function, assignment to the "global" variable is really an assignment in the enclosing namespace. Like local variables, this is figured out when the function is compiled. Because of compilation issues, python requires that the global be declared before the first reference to the variable. This is a compile thing, not a runtime thing, so its okay for it to be in a block that isn't ever really run.
As you may know, the scope of a variable is statically determined in python (What are the rules for local and global variables in Python?).
For instance :
a = "global"
def function():
print(a)
a = "local"
function()
# UnboundLocalError: local variable 'a' referenced before assignment
The same rule applies to classes, but it seems to default to the global scope instead of raising an AttributeError:
a = "global"
class C():
print(a)
a = "local"
# 'global'
Moreover, in the case of a nested function, the behavior is the same (without using nonlocal or global) :
a = "global"
def outer_func():
a = "outer"
def inner_func():
print(a)
a = "local"
inner_func()
outer_func()
# UnboundLocalError: local variable 'a' referenced before assignment
But in the case of nested classes, it still defaults to the global scope, and not the outer scope (again without using global or nonlocal) :
a = "global"
def outer_func():
a = "outer"
class InnerClass:
print(a)
a = "local"
outer_func()
# 'global'
The weirdest part is that the nested class default to the outer scope when there is no declaration of a :
a = "global"
def outer_func():
a = "outer"
class InnerClass:
print(a)
outer_func()
# 'outer'
So my questions are :
Why the discrepancy between functions and classes (one raising an exception, the other defaulting to the global scope.
In nested classes, why the default scope has to become global instead of keeping using the outer one when using a variable defined afterward?
The answer is given in great detail in Section 9.2 of the official docs. The crux of the matter is
... On the other hand, the actual search for names is done dynamically, at run time — however, the language definition is evolving towards static name resolution, at “compile” time, so don’t rely on dynamic name resolution! (In fact, local variables are already determined statically.)
When you are in the class definition, which at the moment of its execution is the innermost scope, dynamic name resolution applies. You therefore see printouts of the global value of a.
If the name resolution were static, as in function definitions, the name a would be recognized as a local name even in the print statement. That is why you can't print a in a function before assigning to it.
The rules for class body scoping are alluded to in Section 4.2.2:
Class definition blocks and arguments to exec() and eval() are special in the context of name resolution. A class definition is an executable statement that may use and define names. These references follow the normal rules for name resolution with an exception that unbound local variables are looked up in the global namespace.
Let's parse that last sentence carefully, because it fully covers your last two examples. First off, what is an unbound local variable in this context? A class body creates a new namespace, just like entering a function. If a name is bound somewhere in a class body, it is a local variable. This is determined statically, as mentioned above. If you attempt to reference the name before it is first bound, you have an unbound local variable. Instead of raising an error, as a function call would do, python jumps straight to the global namespace to perform the lookup (and ignores builtins as well). In all other cases (not local variables), normal LEGB lookup order applies.
This is indeed a bit counter-intuitive, and I would argue that it pushes if not outright breaks the rule of least surprise.
Code first:
def another_func(func):
func()
def outer_func(pa=1, pb=2):
def inner_func():
print(pa)
print(type(inner_func))
another_func(inner_func)
if __name__ == '__main__':
outer_func()
#print "1"
I am not sure that, "inner_func" calls a parameter of "outer_func", but that is in the body of "outer_func". How can it "know" there is a "pa" when called by another_func?
I mean, what is actually passed to another_func when its called in "outer_func"? It seems there is something more than a reference of the function object.
Function objects in Python aren't just functions, they're closures:1 they carry around a reference to the local environment where the def statement was executed.
In particular, local variables from inside outer_func can be accessed from inside inner_func. (Even if you return inner_func, those values are kept alive so the closure will still work, for as long as inner_func is alive.)
If you add a nonlocal statement inside inner_func, it can even reassign the local variables from the body of outer_func.
How does this work?
Well, a def statement2 is just a statement, like any other. What it does is something like this:
inner_func = _make_closure(<code from compiling inner_func body>, locals())
That <code from compiling inner_func body> is actually a constant value—the compiler compiles the body of every function in your module into constant code objects at import time.
But the function object that comes back from that _make_closure is a new thing that's created on the fly, and it has a reference to the local variables baked into it. Every time you run outer_func, it creates a new inner_func closure from the same <code>, each one capturing the current local environment.
The details are a little more complicated—and, to some extent, they differ between implementations, so this will be CPython-specific.
Part of the compiler's job is figuring out what kind of variable each name in a function is. You may have read the rules on globals vs. locals (a variable is local if and only if you have an assignment for the name somewhere in the function body, and there's no global statement). But closures make things more complicated.
If a variable would have been local, but a nested function references the variable without assigning to it, or has a nonlocal statement, then it's a cell variable in the outer function, and a free variable in the inner function.3
When the interpreter calls a function, it creates a frame object that holds the local namespace—the references to all of the function's local variables.
But cell variables are special: the interpreter creates a special cell object for each one, and a reference to that cell goes into the namespace, so there's an extra dereference in front of the value every time you access or change it.
And what that _make_closure pseudo-code above does is to copy the cells from the outer function's frame to a special attribute on the nested function called __closure__.
Then, when you call the inner function, the interpreter copies those cells from the __closure__ into the frame for that function.
So, the outer function's frame and the inner function's frame both have references to the same cells, which is how they can share variables.
From more on this, see the inspect module's docs, which show you how to find things like __closure__ and co_freevars in your interactive interpreter, and the dis module which lets you look at the actual bytecode that your functions get compiled to.
1. This is one of those words that has a slew of related but different meanings. "Closure" can mean the technique of capturing the local namespace in a function, or it can mean the captured namespace, or it can mean the function with a captured namespace attached to it, or it can mean one of the variables in the captured namespace. Usually it's obvious which one you mean from context. If not, you have to say something like "closure capture" or "closure function" or "closure variable".
2. In case you're wondering, lambda expressions work exactly the same way as def statements. And class definitions are not identical, but similar.
3. It's actually still more complicated if you have multiple layers of nesting, but let's ignore that.
You seem to be confusing the code if the function with the function object. The code object is evaluated only once, when the source file is read. However, a new function object called inner_func is created every time outer_func is called. This happens because a def statement is a type of assignment: it associates a function object with the specified name.
The function object contains a reference to its code as a matter of course, along with references to all the namespaces it will need to operate, including it's parent's nonlocal namespace and the global module namespace.
So the value of pa in inner_func is going to be whatever it is in outer_func at the time of calling. The reference is to the namespace, not the name itself. If outer_func returns (think decorators), the namespace will be fixed, and only accessible through inner_func's special reference to it.
In Python 3.3.1, this works:
i = 76
def A():
global i
i += 10
print(i) # 76
A()
print(i) # 86
This also works:
def enclosing_function():
i = 76
def A():
nonlocal i
i += 10
print(i) # 76
A()
print(i) # 86
enclosing_function()
But this doesn't work:
i = 76
def A():
nonlocal i # "SyntaxError: no binding for nonlocal 'i' found"
i += 10
print(i)
A()
print(i)
The documentation for the nonlocal keyword states (emphasis added):
The nonlocal statement causes the listed identifiers to refer to
previously bound variables in the nearest enclosing scope.
In the third example, the "nearest enclosing scope" just happens to be the global scope. So why doesn't it work?
PLEASE READ THIS BIT
I do notice that the documentation goes on to state (emphasis added):
The [nonlocal] statement allows encapsulated code to
rebind variables outside of the local scope besides the global
(module) scope.
but, strictly speaking, this doesn't mean that what I'm doing in the third example shouldn't work.
The search order for names is LEGB, i.e Local, Enclosing, Global, Builtin. So the global scope is not an enclosing scope.
EDIT
From the docs:
The nonlocal statement causes the listed identifiers to refer to
previously bound variables in the nearest enclosing scope. This is
important because the default behavior for binding is to search the
local namespace first. The statement allows encapsulated code to
rebind variables outside of the local scope besides the global
(module) scope.
why is a module's scope considered global and not an enclosing one? It's still not global to other modules (well, unless you do from module import *), is it?
If you put some name into module's namespace; it is visible in any module that uses module i.e., it is global for the whole Python process.
In general, your application should use as few mutable globals as possible. See Why globals are bad?:
Non-locality
No Access Control or Constraint Checking
Implicit coupling
Concurrency issues
Namespace pollution
Testing and Confinement
Therefore It would be bad if nonlocal allowed to create globals by accident. If you want to modify a global variable; you could use global keyword directly.
global is the most destructive: may affect all uses of the module anywhere in the program
nonlocal is less destructive: limited by the outer() function scope (the binding is checked at compile time)
no declaration (local variable) is the least destructive option: limited by inner() function scope
You can read about history and motivation behind nonlocal in PEP: 3104
Access to Names in Outer Scopes.
It depends upon the Boundary cases:
nonlocals come with some senstivity areas which we need to be aware of. First, unlike the global statement, nonlocal names really must have previous been assigned in an enclosing def's scope when a nonlocal is evaluated or else you'll get an error-you cannot create them dynamically by assigning them anew in the enclosing scope. In fact, they are checked at function definition time before either or nested function is called
>>>def tester(start):
def nested(label):
nonlocal state #nonlocals must already exist in enclosing def!
state = 0
print(label, state)
return nested
SyntaxError: no binding for nonlocal 'state' found
>>>def tester(start):
def nested(label):
global state #Globals dont have to exits yet when declared
state = 0 #This creates the name in the module now
print(label, state)
return nested
>>> F = tester(0)
>>> F('abc')
abc 0
>>> state
0
Second, nonlocal restricts the scope lookup to just enclosing defs; nonlocals are not looked up in the enclosing module's global scope or the built-in scope outside all def's, even if they are already there:
for example:-
>>>spam = 99
>>>def tester():
def nested():
nonlocal spam #Must be in a def, not the module!
print('current=', spam)
spam += 1
return nested
SyntaxError: no binding for nonlocal 'spam' found
These restrictions make sense once you realize that python would not otherwise generally know enclosing scope to create a brand-new name in. In the prior listing, should spam be assigned in tester, or the module outside? Because this is ambiguous, Python must resolve nonlocals at function creation time, not function call time.
The answer is that the global scope does not enclose anything - it is global to everything. Use the global keyword in such a case.
Historical reasons
In 2.x, nonlocal didn't exist yet. It wasn't considered necessary to be able to modify enclosing, non-global scopes; the global scope was seen as a special case. After all, the concept of a "global variable" is a lot easier to explain than lexical closures.
The global scope works differently
Because functions are objects, and in particular because a nested function could be returned from its enclosing function (producing an object that persists after the call to the enclosing function), Python needs to implement lookup into enclosing scopes differently from lookup into either local or global scopes. Specifically, in the reference implementation of 3.x, Python will attach a __closure__ attribute to the inner function, which is a tuple of cell instances that work like references (in the C++ sense) to the closed-over variables. (These are also references in the reference-counting garbage-collection sense; they keep the call frame data alive so that it can be accessed after the enclosing function returns.)
By contrast, global lookup works by doing a chained dictionary lookup: there's a dictionary that implements the global scope, and if that fails, a separate dictionary for the builtin scope is checked. (Of course, writing a global only writes to the global dict, not the builtin dict; there is no builtin keyword.)
Theoretically, of course, there's no reason why the implementation of nonlocal couldn't fall back on a lookup in the global (and then builtin) scope, in the same way that a lookup in the global scope falls back to builtins. Stack Overflow is not the right place to speculate on the reason behind the design decision. I can't find anything relevant in the PEP, so it may simply not have been considered.
The best I can offer is: like with local variable lookup, nonlocal lookup works by determining at compile time what the scope of the variable will be. If you consider builtins as simply pre-defined, shadow-able globals (i.e. the only real difference between the actual implementation and just dumping them into the global scope ahead of time, is that you can recover access to the builtin with del), then so does global lookup. As they say, "simple is better than complex" and "special cases aren't special enough to break the rules"; so, no fallback behaviour.
I am having troubles using global variables in python...
In my program, i have declared 2 global variables,
global SYNC_DATA and
global SYNC_TOTAL_SIZE
Now in one of my functions, I am able to use the global variable SYNC_DATA without declaring it as global again in the function; however , I am not able to use the other global variable SYNC_TOTAL_SIZE in the same way. I have to declare the latter as global in the function again to use it. I get this error if i use it without declaring as global in the function - "UnboundLocalError: local variable 'SYNC_TOTAL_SIZE' referenced before assignment"
Why is it so that sometimes I can access global variables without declaring them as global in functions and sometimes not?
And why Is it that we have to again declare it as global in the function when it is already declared once in the beginning... Why doesn`t the function just check the variable in the global namespace if it does not find it in its namespace directly?
Assuming SYNC_TOTAL_SIZE is not declared as local variable. It depends on what operations you are performing on global variables, for just reading the global variable there is no need to declare it as global, but if you are modifying the value of global variable you have to declare it with global.
Because SYNC_TOTAL_SIZE is probably used somewhere else in that code as non-global, which makes Python find it and think it isn't global. Another reason may be that you just read it in some function and so it doesn't need a global declaration (see this question for more details). For an exact diagnosis, post a minimal code snippet that demonstrates your problem.
The advice is to always say global when you mean it.