Develop Reference

How does Python know there is a local variable before encountering its declaration?

def f():
print("Before", locals()) # line 2
print(x); # line 3
x = 2 # line 4
print("After", locals()) # line 5
x = 1
f()
I am aware of the LEGB rule for scoping in Python.
For the above code, when I comment out line 4, everything executes normally as expected: for line 3, python does not find variable x in the local scope and therefore searches it in the global scope where it finds it and prints 1.
But when I execute the whole code as it is without commenting, it raises UnboundLocalError: local variable 'x' referenced before assignment.
I do know I can use nonlocal and global, but my question is :
How does python know there is a local variable declaration before it has encountered one?
Even if it does know there is a variable named x in the local scope (although not yet initialised), why doesn't it shows it in locals()?
I tried finding the answer in similar questions suggestions but failed. Please correct if any of my understanding is wrong.

To some extent, the answer is implementation specific, as Python only specifies the expected behavior, not how to implement it.
That said, let's look at the byte code generated for f by the usual implementation, CPython:
>>> import dis
>>> dis.dis(f)
2 0 LOAD_GLOBAL 0 (print)
2 LOAD_CONST 1 ('Before')
4 LOAD_GLOBAL 1 (locals)
6 CALL_FUNCTION 0
8 CALL_FUNCTION 2
10 POP_TOP
3 12 LOAD_GLOBAL 0 (print)
14 LOAD_FAST 0 (x)
16 CALL_FUNCTION 1
18 POP_TOP
4 20 LOAD_CONST 2 (2)
22 STORE_FAST 0 (x)
5 24 LOAD_GLOBAL 0 (print)
26 LOAD_CONST 3 ('After')
28 LOAD_GLOBAL 1 (locals)
30 CALL_FUNCTION 0
32 CALL_FUNCTION 2
34 POP_TOP
36 LOAD_CONST 0 (None)
38 RETURN_VALUE
There are several different LOAD_* op codes used to retrieve various values. LOAD_GLOBAL is used for names in the global scope; LOAD_CONST is used for local values not assigned to any name. LOAD_FAST is used for local variables. Local variables don't even exist by name, but by indices in an array. That's why they are "fast"; they are available in an array rather than a hash table. (LOAD_GLOBAL also uses integer arguments, but that's just an index into an array of names; the name itself still needs to be looked up in whatever mapping provides the global scope.)
You can even see the constants and local values associated with f:
>>> f.__code__.co_consts
(None, 'Before', 2, 'After')
>>> f.__code__.co_varnames
('x',)
LOAD_CONST 1 puts Before on the stack because f.__code__.co_consts[1] == 'Before', and LOAD_FAST 0 puts the value of x on the stack because f.__code__.co_varnames[0] == 'x'.
The key here is that the byte code is generated before f is ever executed. Python isn't simply executing each line the first time it sees it. Executing the def statement involves, among other things:
reading the source code
parsing into an abstract syntax tree (AST)
using the entire AST to generate the byte code stored in the __code__ attribute of the function object.
Part of the code generation is noting that the name x, due to the assignment somewhere in the body of the function (even if that function is logically unreachable), is a local name, and therefore must be accessed with LOAD_FAST.
At the time locals is called (and indeed before LOAD_FAST 0 is used the first time), no assignment to x (i.e., STORE_FAST 0) has yet been made, so there is no local value in slot 0 to look up.

Because you define it before the f() function calling,
let's try this one :
def f(y):
print("Before", locals()) # line 2
print(y); # line 3
y = 2 # line 4
print("After", locals()) # line 5
f(x)
x = 1

Does this way of swapping values in list use extra memory in Python?

The usual way to swap values in a list is to use a temporary variable.
temp = l[i]
l[i] = l[j]
l[j] = temp
But in python you can do this:
l[i], l[j] = l[j], l[i]
How does this second method work? Is it the exact same process? Does it use less / more memory?

import dis
def swap_using_temp(a, b):
temp = a
a = b
b = temp
def swap_using_unpacking(a, b):
a, b = b, a
swap_using_unpacking does not require extra memory.
Explanation:
If you disassemble the code using dis module of both the function described then you will see that in swap_using_unpacking there is a bytecode instruction ROT_TWO which swaps the 2 topmost elements of the stack(which don't require a third variable hence no extra memory is consumed).
dis.dis(swap_using_unpacking)
11 0 LOAD_FAST 1 (b)
2 LOAD_FAST 0 (a)
4 ROT_TWO
6 STORE_FAST 0 (a)
8 STORE_FAST 1 (b)
10 LOAD_CONST 0 (None)
12 RETURN_VALUE
dis.dis(swap_using_temp)
5 0 LOAD_FAST 0 (a)
2 STORE_FAST 2 (temp)
6 4 LOAD_FAST 1 (b)
6 STORE_FAST 0 (a)
7 8 LOAD_FAST 2 (temp)
10 STORE_FAST 1 (b)
12 LOAD_CONST 0 (None)
14 RETURN_VALUE

You are asking the wrong question here. You are not using assembly language but Python, so you should not worry for some extra bytes. What really matters in Python is readability.
Anyway, both versions should be internally implemented more or less the same way, except that the first one creates an additional identifier. It is explicitely named temp so provided you do not use it in that scope, it brings no real problem.
If you use a linting environment that warns you for possible problems (what you should do...) you must be aware that reusing a variable name that hides the same name in an outer scope, while perfectly correct on a language point of view will light some warning on. But as you should not use a temp identifier outside a local scope (readability...) it should not be a problem either.
So it is more of a matter of taste. If you or your team mates often use other languages that do not allow multiple assignments the first way will be more natural. If you mainly use Python, the second way is IMHO more pythonic because it avoids adding an unnecessary identifier in the local scope. But as I have already said, nothing more that a matter of taste...

While loop without content

I'm writing a Python program for a guessing game and the current way I'm implementing the main loop works, but it feels wrong:
# Guess Loop
while not(guess(ans)):
pass
It will work but I was wondering if this is bad practice (I assume it is).

There is nothing against this in PEP-8, Python's de facto coding standard. This is exactly what while loops are for.
I think you're concerned because the while loop is empty, but in a non-trivial program it wouldn't be, and you'd have something to put there.

I'd prefer
while True:
if guess(ans):
break
but your code
while not(guess(ans)):
pass
functions identically. It may even be slightly more performant!
from dis import dis
from random import choice
def foo(): return choice([True, False]) # flip a coin
# My code
def bar1():
while True:
if foo(): break
# Your code
def bar2():
while not(foo()):
pass
>>> dis(bar1)
2 0 SETUP_LOOP 12 (to 14)
3 >> 2 LOAD_GLOBAL 0 (foo)
4 CALL_FUNCTION 0
6 POP_JUMP_IF_FALSE 2
8 BREAK_LOOP
10 JUMP_ABSOLUTE 2
12 POP_BLOCK
>> 14 LOAD_CONST 0 (None)
16 RETURN_VALUE
>>> dis(bar2)
2 0 SETUP_LOOP 10 (to 12)
>> 2 LOAD_GLOBAL 0 (foo)
4 CALL_FUNCTION 0
6 POP_JUMP_IF_TRUE 10
3 8 JUMP_ABSOLUTE 2
>> 10 POP_BLOCK
>> 12 LOAD_CONST 0 (None)
14 RETURN_VALUE
Your snippet can omit the "BREAK_LOOP" there, which might improve performance. Of course the POP_JUMP_IF_TRUE might also be slightly harder than the POP_JUMP_IF_FALSE. Or vice-versa.
Regardless, they function identically. Use whichever one you like. I prefer the look of "Loop forever, and if guess(ans) is true, break" to "Loop until the negation of guess(ans) is false."

There are multiple ways to handle this idea - any issues with looping (specifically the bugs that can occur of a while(true) loop) wouldn't be true here, but maybe would come up in other languages, which is maybe what gave you pause.

Do generators simply make new objects with __iter__ and next functions?

I tried to search for this answer on my own, but there was too much noise.
Are generators in python just a convenience wrapper for the user to make an iterator object?
When you define the generator:
def test():
x = 0
while True:
x += 1
yield x
is python simply making a new object, adding the __iter__ method, then putting the rest of the code into the next function?
class Test(object):
def __init__(self):
self.x = 0
def __iter__(self):
return self
def next(self):
self.x += 1
return self.x

Nope. Like so:
>>> def test():
... x = 0
... while True:
... x += 1
... yield x
...
>>> type(test)
<type 'function'>
So what it returns is a function object. Details from there get hairy; the short course is that the code object belonging to the function (test.func_code) is marked as a generator by one of the flags in test.func_code.co_flags.
You can disassemble the bytecode for test to see that it's just like any other function otherwise, apart from that a generator function always contains a YIELD_VALUE opcode:
>>> import dis
>>> dis.dis(test)
2 0 LOAD_CONST 1 (0)
3 STORE_FAST 0 (x)
3 6 SETUP_LOOP 25 (to 34)
>> 9 LOAD_GLOBAL 0 (True)
12 POP_JUMP_IF_FALSE 33
4 15 LOAD_FAST 0 (x)
18 LOAD_CONST 2 (1)
21 INPLACE_ADD
22 STORE_FAST 0 (x)
5 25 LOAD_FAST 0 (x)
28 YIELD_VALUE
29 POP_TOP
30 JUMP_ABSOLUTE 9
>> 33 POP_BLOCK
>> 34 LOAD_CONST 0 (None)
37 RETURN_VALUE
To do it the way you have in mind, the horrors just start ;-) if you think about how to create an object to mimic just this:
def test():
yield 2
yield 3
yield 4
Now your next() method would have to carry additional hidden state just to remember which yield comes next. Wrap that in some nested loops with some conditionals, and "unrolling" it into a single-entry next() becomes a nightmare.

No -- Generators also provide other methods (.send, .throw, etc) and can be used for more purposes than simply making iterators (e.g. coroutines).
Indeed, generators are an entirely different beast and a core language feature. It'd be very hard (possibly impossible) to create one in vanilla python if they weren't baked into the language.
With that said, one application of generators is to provide an easy syntax for creating an iterator :-).

Are generators in python just a convenience wrapper for the user to make an iterator object?
No. A generator is a function, where as iterators are class. Hence, generator can not be a object of iterator. But in some way you can say that generator is a simplified approach to get iterator like capability. It means:
All Generators are iterators, but not all iterators are generators.
I will strongly suggest you refer below wiki links:
Iterator - traverses a collection one at a time
Generator - generates a sequence, one item at a time
An iterator is typically something that has a next method to get the next element from a stream. A generator is an iterator that is tied to a function.
I will suggest you to refer: Difference between Python's Generators and Iterators.

Behavior of python method in absence of return statement [duplicate]

This question already has answers here:
Why does my recursive function return None?
(4 answers)
Closed 9 years ago.
I have a question related to change in program behavior
the absence of return statement leads to in a python method.
The count method below prints number of digits in a given integer.
With the below chunk of code I get the result as 4, which is
the expected result.
def count(x,acc=0):
if x==0:
return acc
return count(x/10,acc+1)
print "Count is %s" %(count(1234))
Result: Count is 4
If I modify the above method so that the last statement does not contain
the 'return' statement the result I get is 'None'.
def count(x,acc=0):
if x==0:
return acc
count(x/10,acc+1)
print "Count is %s" %(count(1234))
Result: Count is None
(The version of Python I used is: 2.7.3)
Does the above behavior results due to the fact that Python doesn't do tail call optimization or is there any other reasoning involved ?
A similar chunk of code in perl (which AFAIK doesn't do tail call optimization) provides
the expected result without 'return' being part of the last statement.
sub counter {
my ($n,$acc) = #_;
return $acc if ($n==0);
counter(int($n/10), $acc+1);
}
print "Count is:" . counter(1234,0) ."\n"
Result: Count is:4
(The versions of Perl I ran above code chunk are : 5.14.4 and 5.8.5).
My questions are:
Is Tail Call optimization the reason for the behavior shown in above chunk of Python code.
If that were the case then why behavior of perl code differs, which doesn't do TCO either.

Doesn't have to do with the missing of tail optimization at all. Functions in Python need the keyword return explicitly, otherwise is assumed their return None.
I know Ruby doesn't behave that way, it returns the value of the last executed expression. With Perl it must be the same.
It is nothing that clever, just the fact that Python programs behave that way :)
See the disassembly for both Python functions. You may see how the one having the return value actually calls the function and return the value on top of the stack. The one that doesn't have it, see that have two instructions after the funciont call, which load constant None and return it.
def count(x,acc=0):
if x==0:
return acc
return count(x/10,acc+1)
def count2(x,acc=0):
if x==0:
return acc
count(x/10,acc+1)
In [7]: import dis
In [8]: dis.dis(count)
2 0 LOAD_FAST 0 (x)
3 LOAD_CONST 1 (0)
6 COMPARE_OP 2 (==)
9 POP_JUMP_IF_FALSE 16
3 12 LOAD_FAST 1 (acc)
15 RETURN_VALUE
4 >> 16 LOAD_GLOBAL 0 (count)
19 LOAD_FAST 0 (x)
22 LOAD_CONST 2 (10)
25 BINARY_DIVIDE
26 LOAD_FAST 1 (acc)
29 LOAD_CONST 3 (1)
32 BINARY_ADD
33 CALL_FUNCTION 2
36 RETURN_VALUE
In [9]: dis.dis(count2)
2 0 LOAD_FAST 0 (x)
3 LOAD_CONST 1 (0)
6 COMPARE_OP 2 (==)
9 POP_JUMP_IF_FALSE 16
3 12 LOAD_FAST 1 (acc)
15 RETURN_VALUE
4 >> 16 LOAD_GLOBAL 0 (count)
19 LOAD_FAST 0 (x)
22 LOAD_CONST 2 (10)
25 BINARY_DIVIDE
26 LOAD_FAST 1 (acc)
29 LOAD_CONST 3 (1)
32 BINARY_ADD
33 CALL_FUNCTION 2
36 POP_TOP
37 LOAD_CONST 0 (None)
40 RETURN_VALUE

Without the return, the only case where your return something if when acc == 0; for any other call to your count recursive method, you don't return anything, hence the None received.
Unlike other languages (scala, apparently perl as well, probably others), python does not return the last statement by default, you have to call return explicitely.

Tail call elimination does not prevent Perl's semantics (though without Perl semantics, you'd have far fewer tail calls since every function would implicitly end with return;). It's purely a language design choice. The reason Perl and Python differ is that they weren't designed by the same people.
The syntax for returning a value varies by language simply due to choice. The designers of Python opted to require an explicit return statement. In QBasic, you had to assign the value to return to a variable that shared the same name as the function.

Functions in Python without a return statement return the value None.
In this case, you are only returning a value when x == 0, therefore the None that you get. Check the Python tutorial for more info: http://docs.python.org/2/tutorial/controlflow.html#defining-functions