Develop Reference

How scope is determined in Python [duplicate]

This question already has answers here:
UnboundLocalError trying to use a variable (supposed to be global) that is (re)assigned (even after first use)
(14 answers)
Closed 1 year ago.
Why does the first print statement in the second function throw an error that x is not defined?
x = 5
def function_a():
print(x)
def function_b():
print(x)
x = 7
print(x)
Running the first function gives the following result.
>>> function_a()
5
While running the second function throws an error.
UnboundLocalError: local variable 'x' referenced before assignment

Python will infer and use the variable from the inner scope whenever it sees the variable declared within the scope, even if that variable is declared after usage.
To demonstrate this, I created two scenarios.
Variable declared inside the inner scope
A variable is inferred in the following order: local, nonlocal, and global. Since x is declared inside the inner scope, Python will infer and use the x in this scope even if it is declared after usage.
Note that Python cannot distinguish modification from declaration; what was meant to modify x from the global scope was interpreted to declare another variable x within that scope.
No variables declared inside the inner scope
If no variables are declared within the inner scope, Python will switch the inferred scope to nonlocal, and then to global.
Explicitly switching scope
If you explicitly declared the scope of x beforehand, Python would not have to infer.
The following code will not throw an error because the scope it uses is explicitly the global scope instead of the inner one.
x = 5
def scope():
global x
print(x)
x = 7
print(x)
Scopes
By choosing which scope to work with, you are not only using the variables within that specific scope but also modifying the variables. Therefore, you need extra caution when dealing with scopes.
Because Python cannot distinguish variable declaration from variable modification, my advice is that if you want to use a global variable, you should explicitly state it beforehand.
This also applies to nested scopes.
x = 5
def outer():
x = 7
def inner():
nonlocal x
print(x)
x = 3
print(x)
inner()
Running the outer function gives the following result.
>>> outer()
7
3
Try changing the nonlocal keyword to global and see a different result; or remove the line completely to get an error.

In the 2nd method, you have written x = 7, which makes x a local variable for that method. But since, you are trying to access it on the print statement before the line "x = 7", it throws an error saying local variable x is accessed before assignment.
If you remove the line x = 7, it would work just fine.
You can try this
x = 5
def vs_code():
print(x)
def vs_code1():
print(x)
y = 7
print(y)
this will print
5
5
7
Now, since I am not declaring x inside 2nd method, now x is not local for 2nd method. If you declare it inside 2nd method, it will interpret that as a local variable. and you have to use it only after assignment.
Hope you understood.

python is special in variable scope
you can use dis module to see what happeded.
def foo():
print(x)
print(y)
x = ...
import dis
dis.dis(foo)
output
2 0 LOAD_GLOBAL 0 (print)
2 LOAD_FAST 0 (x)
4 CALL_FUNCTION 1
6 POP_TOP
3 8 LOAD_GLOBAL 0 (print)
10 LOAD_GLOBAL 1 (y)
12 CALL_FUNCTION 1
14 POP_TOP
4 16 LOAD_CONST 1 (Ellipsis)
18 STORE_FAST 0 (x)
20 LOAD_CONST 0 (None)
22 RETURN_VALUE
x is LOAD_FAST
y is LOAD_GLOBAL
before running, python thinks x is a local variable because you try to modify it (cannot modify global variable).
this will works fine
def vs_code1():
global x
print(x)
x = 7
print(x)

When a variable is assigned a value inside a function, that variable name is considered local. Unless it is explicitly declared as global with the line:
global x
In the first print of the function the variable x is not defined in the local scope. That-s why throws an error.
So you have to add the line global x at the beggining of the function body for the variable x be considered global despite it is assigned in the function.

When is the existence of nonlocal variables checked?

I am learning Python and right now I am on the topic of scopes and nonlocal statement.
At some point I thought I figured it all out, but then nonlocal came and broke everything down.
Example number 1:
print( "let's begin" )
def a():
def b():
nonlocal x
x = 20
b()
a()
Running it naturally fails.
What is more interesting is that print() does not get executed. Why?.
My understanding was that enclosing def a() is not executed until print() is executed, and nested def b() is executed only when a() is called. I am confused...
Ok, let's try example number 2:
print( "let's begin" )
def a():
if False: x = 10
def b():
nonlocal x
x = 20
b()
a()
Aaand... it runs fine.
Whaaat?! How did THAT fix it? x = 10 in function a is never executed!
My understanding was that nonlocal statement is evaluated and executed at run-time, searching enclosing function's call contexts and binding local name x to some particular "outer" x. And if there is no x in outer functions - raise an exception. Again, at run-time.
But now it looks like this is done at the time of syntax analysis, with pretty dumb check "look in outer functions for x = blah, if there is something like this - we're fine," even if that x = blah is never executed...
Can anybody explain me when and how nonlocal statement is processed?

You can see what the scope of b knows about free variables (available for binding) from the scope of a, like so:
import inspect
print( "let's begin" )
def a():
if False:
x = 10
def b():
print(inspect.currentframe().f_code.co_freevars)
nonlocal x
x = 20
b()
a()
Which gives:
let's begin
('x',)
If you comment out the nonlocal line, and remove the if statement with x inside, the you'll see the free variables available to b is just ().
So let's look at what bytecode instruction this generates, by putting the definition of a into IPython and then using dis.dis:
In [3]: import dis
In [4]: dis.dis(a)
5 0 LOAD_CLOSURE 0 (x)
2 BUILD_TUPLE 1
4 LOAD_CONST 1 (<code object b at 0x7efceaa256f0, file "<ipython-input-1-20ba94fb8214>", line 5>)
6 LOAD_CONST 2 ('a.<locals>.b')
8 MAKE_FUNCTION 8
10 STORE_FAST 0 (b)
10 12 LOAD_FAST 0 (b)
14 CALL_FUNCTION 0
16 POP_TOP
18 LOAD_CONST 0 (None)
20 RETURN_VALUE
So then let's look at how LOAD_CLOSURE is processed in ceval.c.
TARGET(LOAD_CLOSURE) {
PyObject *cell = freevars[oparg];
Py_INCREF(cell);
PUSH(cell);
DISPATCH();
}
So we see it must look up x from freevars of the enclosing scope(s).
This is mentioned in the Execution Model documentation, where it says:
The nonlocal statement causes corresponding names to refer to previously bound variables in the nearest enclosing function scope. SyntaxError is raised at compile time if the given name does not exist in any enclosing function scope.

First, understand that python will check your module's syntax and if it detects something invalid it raises a SyntaxError which stops it from running at all. Your first example raises a SyntaxError but to understand exactly why is pretty complicated although it is easier to understand if you know how __slots__ works so I will quickly introduce that first.
When a class defines __slots__ it is basically saying that the instances should only have those attributes so each object is allocated memory with space for only those, trying to assign other attributes raises an error
class SlotsTest:
__slots__ = ["a", "b"]
x = SlotsTest()
x.a = 1 ; x.b = 2
x.c = 3 #AttributeError: 'SlotsTest' object has no attribute 'c'
The reason x.c = 3 can't work is that there is no memory space to put a .c attribute in.
If you do not specify __slots__ then all instances are created with a dictionary to store the instance variables, dictionaries do not have any limitations on how many values they contain
class DictTest:
pass
y = DictTest()
y.a = 1 ; y.b = 2 ; y.c = 3
print(y.__dict__) #prints {'a': 1, 'b': 2, 'c': 3}
Python functions work similar to slots. When python checks the syntax of your module it finds all variables assigned (or attempted to be assigned) in each function definition and uses that when constructing frames during execution.
When you use nonlocal x it gives an inner function access to a specific variable in the outer function scope but if there is no variable defined in the outer function then nonlocal x has no space to point to.
Global access doesn't run into the same issue since python modules are created with a dictionary to store its attributes. So global x is allowed even if there is no global reference to x

Order of variable reference and assignment in nested function

From the Google Style Guide on lexical scoping:
A nested Python function can refer to variables defined in enclosing
functions, but can not assign to them.
This specification can be seen here:
def toplevel():
a = 5
def nested():
# Tries to print local variable `a`, but `a` is created locally after,
# so `a` is referenced before assignment. You would need `nonlocal a`
print(a + 2)
a = 7
nested()
return a
toplevel()
# UnboundLocalError: local variable 'a' referenced before assignment
Reversing the order of the two statements in nested gets rid of this issue:
def toplevel():
a = 5
def nested():
# Two statements' order reversed, `a` is now locally assigned and can
# be referenced
a = 7
print(a + 2)
nested()
return a
toplevel()
My question is, what is it about Python's implementation that tells the first function that a will be declared locally (after the print statement)? My understanding is that Python is effectively interpreted line by line. So, shouldn't it default to looking for a nonlocal a at that point in the code?
To elaborate, if I was to use just reference (no assignment),
def toplevel():
a = 5
def nested():
print(a + 2)
nested()
return a
toplevel()
somehow the print statement knows to reference the nonlocal a defined in the enclosing function. But if I assign to a local a after that line, the function is almost too smart for its own good.

My understanding is that Python is effectively interpreted line by line.
That's not the right mental model.
The body of the entire function is analysed to determine which names refer to local variables and which don't.
To simplify your example, the following also gives UnboundLocalError:
def func():
print(a)
a = 2
func()
Here, func() compiles to the following bytecodes:
2 0 LOAD_FAST 0 (a)
3 PRINT_ITEM
4 PRINT_NEWLINE
3 5 LOAD_CONST 1 (2)
8 STORE_FAST 0 (a)
11 LOAD_CONST 0 (None)
14 RETURN_VALUE
Compare this with
def gunc():
print(a)
which compiles to
2 0 LOAD_GLOBAL 0 (a)
3 PRINT_ITEM
4 PRINT_NEWLINE
5 LOAD_CONST 0 (None)
8 RETURN_VALUE
Observe how the absence of assignment to a turns the reference from a local to a global one.

My understanding is that Python is effectively interpreted line by line
That's where you're wrong. The whole file is compiled to bytecode before any interpretation begins.
Also, even if the bytecode compilation pass didn't exist, print(a + 2) wouldn't actually be executed before a = 7 is seen, because it's in a function definition. Python would still know about the a = 7 by the time it actually tries to execute print(a + 2).

As per document
A special quirk of Python is that – if no global statement is in effect – assignments to names always go into the innermost scope. Assignments do not copy data — they just bind names to objects.

How do you create new commands in Python when immutable data types are involved? [duplicate]

This question already has answers here:
How do I pass a variable by reference?
(39 answers)
Closed 5 years ago.
In Python, the following code prints '0', not '1'.
def inc(x):
x = x+1
a = 0
inc(a)
print(a)
I understand why this happens; it's because integers are immutable. What I don't understand is how to get around this behaviour when it's undesirable. Suppose we want to create a new command such that the code
a = 0
inc(a)
print(a)
prints '1'.
Obviously, the naive approach won't do it. What can we do instead?

Similar (a bit more general) question can be found here along with a discussion how Python passes params to functions. In short, without making x variable in your code an object, I believe there's nothing we can do. Of course, you can alter your code to e.g. return changed value from function inc() and print that (i.e. print(inc(x))) or just do the printing from inside the inc() method, but that's not what you're essentially looking for.

If I understand correctly, You are trying to increment variable a using function inc(var) and passing 'a' as a external variable to the function inc().
As #Marko Andrijevic stated, variable x passed to function inc() and variable x defined in the function are different . One way to achieve is by returning value of x and collecting externally, which you may not be looking for.
Alternately, Since you have defined variable 'a' outside function ,it can be called global variable.
If you want to pass that to a function, and manipulate it, you need to define that variable ('a' in your case) inside the function as global. Something like below.
def inc(x):
global a
a = x+1
Now when the new value assigned to 'a' after 'x+1', it is retained after execution of 'inc(x)'
>>> a = 0
>>> inc(a)
>>> a
1
EDIT -1
As per comments by #DYZ . Its correct. declaring global a inside inc() function will always increment a.
A better alternative will be , in that case, to return x inside inc() and assign that value to any external variable.
Not an elegant solution, but works as intended.
def inc(x):
return x+1
Result
>>> a
0
>>> a = inc(a)
>>> a
1
>>> a = inc(a)
>>> a
2
>>> b = 0
>>> b = inc(b)
>>> b
1
>>> a
2
>>>

one can use yield to get variable values.
def inc(x,y,z):
x += 1
y+=1
z+=1
yield x,y,z #inc doesn't stop
yield x+y+z
a=b=c=0
gen=inc(a,b,c)
gen=list(gen)
a,b,c,sum=gen[0]+(gen[1],) #however, index must still be known
print a,b,c,sum

When is a new name introduced in Python?

I am asking because of the classic problem where somebody creates a list of lambdas:
foo = []
for i in range(3):
foo.append((lambda: i))
for l in foo:
print(l())
and unexpectedly gets only twos as output.
The commonly proposed solution is to make i a named argument like this:
foo = []
for i in range(3):
foo.append((lambda i=i: i))
for l in foo:
print(l())
Which produces the desired output of 0, 1, 2 but now something magical has happened. It sort of did what is expected because Python is pass-by-reference and you didn't want a reference.
Still, just adding a new name to something, shouldn't that just create another reference?
So the question becomes what are the exact rules for when something is not a reference?
Considering that ints are immutable and the following works:
x = 3
y = x
x = 5
print(x, y) // outputs 5 3
probably explains why adding that named parameter works. A local i with the same value was created and captured.
Now why, in the case of our lambdas was the same i referenced? I pass an int to function and it is refenced and if I store it in a variable it is copied. Hm.
Basically I am looking for the most concise and abstract way possible to remember exactly how this works. When is the same value referenced, when do I get a copy. If it has any common names and there are programming languages were it works the same that would be interesting as well.
Here is my current assumption:
Arguments are always passed to functions by reference.
Assigning to a variable of immutable type creates a copy.
I am asking anyway, just to make sure and hopefully get some background.

The issue here is how you think of names.
In your first example, i is a variable that is assigned to every time the loop iterates. When you use lambda to make a function, you make a function that accesses the name i and returns it's value. This means as the name i changes, the value returned by the functions also changes.
The reason the default argument trick works is that the name is evaluated when the function is defined. This means the default value is the value the i name points to at that time, not the name itself.
i is a label. 0, 1 and 2 are the objects. In the first case, the program assigns 0 to i, then makes a function that returns i - it then does this with 1 and 2. When the function is called, it looks up i (which is now 2) and then returns it.
In the second example, you assign 0 to i, then you make a function with a default argument. That default argument is the value that is gotten by evaluating i - that is the object 0. This is repeated for 1 and 2. When the function is called, it assigns that default value to a new variable i, local to the function and unrelated to the outer i.

Python doesn't exactly pass by reference or by value (at least, not the way you'd think of it, coming from a language like C++).
In many other languages (such as C++), variables can be thought of as synonymous with the values they hold.
However, in Python, variables are names that point to the objects in memory.
(This is a good explanation (with pictures!))
Because of this, you can get multiple names attached to one object, which can lead to interesting effects.
Consider these equivalent program snippets:
// C++:
int x;
x = 10; // line A
x = 20; // line B
and
# Python:
x = 10 # line C
x = 20 # line D
After line A, the int 10 is stored in memory, say, at the memory address 0x1111.
After line B, the memory at 0x1111 is overwritten, so 0x1111 now holds the int 20
However, the way this program works in python is quite different:
After line C, x points to some memory, say, 0x2222, and the value stored at 0x2222 is 10
After line D, x points to some different memory, say, 0x3333, and the value stored at 0x3333 is 20
Eventually, the orphaned memory at 0x2222 is garbage collected by Python.
Hopefully this helps you get a grasp of the subtle differences between variables in Python and most other languages.
(I know I didn't directly answer your question about lambdas, but I think this is good background knowledge to have before reading one of the good explanations here, such as #Lattyware's)
See this question for some more background info.
Here's some final background info, in the form of oft-quoted but instructive examples:
print 'Example 1: Expected:'
x = 3
y = x
x = 2
print 'x =', x
print 'y =', y
print 'Example 2: Surprising:'
x = [3]
y = x
x[0] = 2
print 'x =', x
print 'y =', y
print 'Example 3: Same logic as in Example 1:'
x = [3]
y = x
x = [2]
print 'x =', x
print 'y =', y
The output is:
Example 1: Expected:
x = 2
y = 3
Example 2: Surprising:
x = [2]
y = [2]
Example 3: Same logic as in Example 1:
x = [2]
y = [3]

foo = []
for i in range(3):
foo.append((lambda: i))
Here since all the lambda's were created in the same scope so all of them point to the same global variable variable i. so, whatever value i points to will be returned when they are actually called.
foo = []
for i in range(3):
foo.append((lambda z = i: id(z)))
print id(i) #165618436
print(foo[-1]()) #165618436
Here in each loop we assign the value of i to a local variable z, as default arguments are calculated when the function is parsed so the value z simply points to the values stored by i during the iteration.
Arguments are always passed to functions by reference?
In fact the z in foo[-1] still points to the same object as i of the last iteration, so yes values are passed by reference but as integers are immutable so changing i won't affect z of the foo[-1] at all.
In the example below all lambda's point to some mutable object, so modifying items in lis will also affect the functions in foo:
foo = []
lis = ([], [], [])
for i in lis:
foo.append((lambda z = i: z))
lis[0].append("bar")
print foo[0]() #prints ['bar']
i.append("foo") # `i` still points to lis[-1]
print foo[-1]() #prints ['foo']
Assigning to a variable of immutable type creates a copy?
No values are never copied.
>>> x = 1000
>>> y = x # x and y point to the same object, but an immutable object.
>>> x += 1 # so modifying x won't affect y at all, in fact after this step
# x now points to some different object and y still points to
# the same object 1000
>>> x #x now points to an new object, new id()
1001
>>> y #still points to the same object, same id()
1000
>>> x = []
>>> y = x
>>> x.append("foo") #modify an mutable object
>>> x,y #changes can be seen in all references to the object
(['foo'], ['foo'])

The list of lambdas problem arises because the i referred to in both snippets is the same variable.
Two distinct variables with the same name exist only if they exist in two separate scopes. See the following link for when that happens, but basically any new function (including a lambda) or class establishes its own scope, as do modules, and pretty much nothing else does. See: http://docs.python.org/2/reference/executionmodel.html#naming-and-binding
HOWEVER, when reading the value of a variable, if it is not defined in the current local scope, the enclosing local scopes are searched*. Your first example is of exactly this behaviour:
foo = []
for i in range(3):
foo.append((lambda: i))
for l in foo:
print(l())
Each lambda creates no variables at all, so its own local scope is empty. When execution hits the locally undefined i, it is located in the enclosing scope.
In your second example, each lambda creates its own i variable in the parameter list:
foo = []
for i in range(3):
foo.append((lambda i=i: i))
This is in fact equivalent to lambda a=i: a, because the i inside the body is the same as the i on the left hand side of the assignment, and not the i on the right hand side. The consequence is that i is not missing from the local scope, and so the value of the local i is used by each lambda.
Update: Both of your assumptions are incorrect.
Function arguments are passed by value. The value passed is the reference to the object. Pass-by-reference would allow the original variable to be altered.
No implicit copying ever occurs on function call or assignment, of any language-level object. Under the hood, because this is pass-by-value, the references to the parameter objects are copied when the function is called, as is usual in any language which passes references by value.
Update 2: The details of function evaluation are here: http://docs.python.org/2/reference/expressions.html#calls . See the link above for the details regarding name binding.
* No actual linear search occurs in CPython, because the correct variable to use can be determined at compile time.

The answer is that the references created in a closure (where a function is inside a function, and the inner function accesses variables from the outer one) are special. This is an implementation detail, but in CPython the value is a particular kind of object called a cell and it allows the variable's value to be changed without rebinding it to a new object. More info here.
The way variables work in Python is actually rather simple.
All variables contain references to objects.
Reassigning a variable points it to a different object.
All arguments are passed by value when calling functions (though the values being passed are references).
Some types of objects are mutable, which means they can be changed without changing what any of their variable names point to. Only these types can be changed when passed, since this does not require changing any references to the object.
Values are never copied implicitly. Never.

The behaviour really has very little to do with how parameters are passed (which is always the same way; there is no distinction in Python where things are sometimes passed by reference and sometimes passed by value). Rather the problem is to do with how names themselves are found.
lambda: i
creates a function that is of course equivalent to:
def anonymous():
return i
That i is a name, within the scope of anonymous. But it's never bound within that scope (not even as a parameter). So for that to mean anything i must be a name from some outer scope. To find a suitable name i, Python will look at the scope in which anonymous was defined in the source code (and then similarly out from there), until it finds a definition for i.1
So this loop:
foo = []
for i in range(3):
foo.append((lambda: i))
for l in foo:
print(l())
Is almost exactly as if you had written this:
foo = []
for i in range(3):
def anonymous():
return i
foo.append(anonymous)
for l in foo:
print(l())
So that i in return i (or lambda: i) ends up being the same i from the outer scope, which is the loop variable. Not that they are all references to the same object, but that they are all the same name. So it's simply not possible for the functions stored in foo to return different values; they're all returning the object referred to by a single name.
To prove it, watch what happens when I remove the variable i after the loop:
>>> foo = []
>>> for i in range(3):
foo.append((lambda: i))
>>> del i
>>> for l in foo:
print(l())
Traceback (most recent call last):
File "<pyshell#7>", line 2, in <module>
print(l())
File "<pyshell#3>", line 2, in <lambda>
foo.append((lambda: i))
NameError: global name 'i' is not defined
You can see that the problem isn't that each function has a local i bound to the wrong thing, but rather than each function is returning the value of the same global variable, which I've now removed.
OTOH, when your loop looks like this:
foo = []
for i in range(3):
foo.append((lambda i=i: i))
for l in foo:
print(l())
That is quite like this:
foo = []
for i in range(3):
def anonymous(i=i):
return i
foo.append(anonymous)
for l in foo:
print(l())
Now the i in return i is not the same i as in the outer scope; it's a local variable of the function anonymous. A new function is created in each iteration of the loop (stored temporarily in the outer scope variable anonymous, and then permanently in a slot of foo), so each one has it's own local variables.
As each function is created, the default value of its parameter is set to the value of i (in the scope defining the functions). Like any other "read" of a variable, that pulls out whatever object is referenced by the variable at that time, and thereafter has no connection to the variable.2
So each function gets the default value of i as it is in the outer scope at the time it is created, and then when the function is called without an argument that default value becomes the value of the i in that function's local scope. Each function has no non-local references, so is completely unaffected by what happens outside it.
1 This is done at "compile time" (when the Python file is converted to bytecode), with no regard for what the system is like at runtime; it is almost literally looking for an outer def block with i = ... in the source code. So local variables are actually statically resolved! If that lookup chain falls all the way out to the module global scope, then Python assumes that i will be defined in the global scope at the point that the code will be run, and just treats i as a global variable whether or not there is a statically visible binding for i at module scope, hence why you can dynamically create global variables but not local ones.
2 Confusingly, this means that in lambda i=i: i, the three is refer to three completely different "variables" in two different scopes on the one line.
The leftmost i is the "name" holding the value that will be used for the default value of i, which exists independently of any particular call of the function; it's almost exactly "member data" stored in the function object.
The second i is an expression evaluated as the function is created, to get the default value. So the i=i bit acts very like an independent statement the_function.default_i = i, evaluated in the same scope containing the lambda expression.
And finally the third i is actually the local variable inside the function, which only exists within a call to the anonymous function.