Is there any practical difference between list(iterable) and [*iterable] in versions of Python that support the latter?
list(x) is a function, [*x] is an expression. You can reassign list, and make it do something else (but you shouldn't).
Talking about cPython, b = list(a) translates to this sequence of bytecodes:
LOAD_NAME 1 (list)
LOAD_NAME 0 (a)
CALL_FUNCTION 1
STORE_NAME 2 (b)
Instead, c = [*a] becomes:
LOAD_NAME 0 (a)
BUILD_LIST_UNPACK 1
STORE_NAME 3 (c)
so you can argue that [*a] might be slightly more efficient, but marginally so.
You can use the standard library module dis to investigate the byte code generated by a function. In this case:
import dis
def call_list(x):
return list(x)
def unpacking(x):
return [*x]
dis.dis(call_list)
# 2 0 LOAD_GLOBAL 0 (list)
# 2 LOAD_FAST 0 (x)
# 4 CALL_FUNCTION 1
# 6 RETURN_VALUE
dis.dis(unpacking)
# 2 0 LOAD_FAST 0 (x)
# 2 BUILD_LIST_UNPACK 1
# 4 RETURN_VALUE
So there is a difference and it is not only the loading of the globally defined name list, which does not need to happen with the unpacking. So it boils down to how the built-in list function is defined and what exactly BUILD_LIST_UNPACK does.
Note that both are actually a lot less code than writing a standard list comprehension for this:
def list_comp(x):
return [a for a in x]
dis.dis(list_comp)
# 2 0 LOAD_CONST 1 (<code object <listcomp> at 0x7f65356198a0, file "<ipython-input-46-dd71fb182ec7>", line 2>)
# 2 LOAD_CONST 2 ('list_comp.<locals>.<listcomp>')
# 4 MAKE_FUNCTION 0
# 6 LOAD_FAST 0 (x)
# 8 GET_ITER
# 10 CALL_FUNCTION 1
# 12 RETURN_VALUE
Since [*iterable] is unpacking, it accepts assignment-like syntax, unlike list(iterable):
>>> [*[]] = []
>>> list([]) = []
File "<stdin>", line 1
SyntaxError: can't assign to function call
You can read more about this here (not useful though).
You can also use list(sequence=iterable), i.e. with a key-word argument:
>>> list(sequence=[])
[]
Again not useful.
There's always going to be some differences between two constructs that do the same thing. Thing is, I wouldn't say the differences in this case are actually practical. Both are expressions that take the iterable, iterate through it and then create a list out of it.
The contract is the same: input is an iterable output is a list populated by the iterables elements.
Yes, list can be rebound to a different name; list(it) is a function call while [*it] is a list display; [*it] is faster with smaller iterables but generally performs the same with larger ones. Heck, one could even throw in the fact that [*it] is three less keystrokes.
Are these practical though? Would I think of them when trying to get a list out of an iterable? Well, maybe the keystrokes in order to stay under 79 characters and get the linter to shut it up.
Apparently there’s a performance difference in CPython, where [*a] overallocates and list() doesn’t: What causes [*a] to overallocate?
Related
def f():
print("Before", locals()) # line 2
print(x); # line 3
x = 2 # line 4
print("After", locals()) # line 5
x = 1
f()
I am aware of the LEGB rule for scoping in Python.
For the above code, when I comment out line 4, everything executes normally as expected: for line 3, python does not find variable x in the local scope and therefore searches it in the global scope where it finds it and prints 1.
But when I execute the whole code as it is without commenting, it raises UnboundLocalError: local variable 'x' referenced before assignment.
I do know I can use nonlocal and global, but my question is :
How does python know there is a local variable declaration before it has encountered one?
Even if it does know there is a variable named x in the local scope (although not yet initialised), why doesn't it shows it in locals()?
I tried finding the answer in similar questions suggestions but failed. Please correct if any of my understanding is wrong.
To some extent, the answer is implementation specific, as Python only specifies the expected behavior, not how to implement it.
That said, let's look at the byte code generated for f by the usual implementation, CPython:
>>> import dis
>>> dis.dis(f)
2 0 LOAD_GLOBAL 0 (print)
2 LOAD_CONST 1 ('Before')
4 LOAD_GLOBAL 1 (locals)
6 CALL_FUNCTION 0
8 CALL_FUNCTION 2
10 POP_TOP
3 12 LOAD_GLOBAL 0 (print)
14 LOAD_FAST 0 (x)
16 CALL_FUNCTION 1
18 POP_TOP
4 20 LOAD_CONST 2 (2)
22 STORE_FAST 0 (x)
5 24 LOAD_GLOBAL 0 (print)
26 LOAD_CONST 3 ('After')
28 LOAD_GLOBAL 1 (locals)
30 CALL_FUNCTION 0
32 CALL_FUNCTION 2
34 POP_TOP
36 LOAD_CONST 0 (None)
38 RETURN_VALUE
There are several different LOAD_* op codes used to retrieve various values. LOAD_GLOBAL is used for names in the global scope; LOAD_CONST is used for local values not assigned to any name. LOAD_FAST is used for local variables. Local variables don't even exist by name, but by indices in an array. That's why they are "fast"; they are available in an array rather than a hash table. (LOAD_GLOBAL also uses integer arguments, but that's just an index into an array of names; the name itself still needs to be looked up in whatever mapping provides the global scope.)
You can even see the constants and local values associated with f:
>>> f.__code__.co_consts
(None, 'Before', 2, 'After')
>>> f.__code__.co_varnames
('x',)
LOAD_CONST 1 puts Before on the stack because f.__code__.co_consts[1] == 'Before', and LOAD_FAST 0 puts the value of x on the stack because f.__code__.co_varnames[0] == 'x'.
The key here is that the byte code is generated before f is ever executed. Python isn't simply executing each line the first time it sees it. Executing the def statement involves, among other things:
reading the source code
parsing into an abstract syntax tree (AST)
using the entire AST to generate the byte code stored in the __code__ attribute of the function object.
Part of the code generation is noting that the name x, due to the assignment somewhere in the body of the function (even if that function is logically unreachable), is a local name, and therefore must be accessed with LOAD_FAST.
At the time locals is called (and indeed before LOAD_FAST 0 is used the first time), no assignment to x (i.e., STORE_FAST 0) has yet been made, so there is no local value in slot 0 to look up.
Because you define it before the f() function calling,
let's try this one :
def f(y):
print("Before", locals()) # line 2
print(y); # line 3
y = 2 # line 4
print("After", locals()) # line 5
f(x)
x = 1
The usual way to swap values in a list is to use a temporary variable.
temp = l[i]
l[i] = l[j]
l[j] = temp
But in python you can do this:
l[i], l[j] = l[j], l[i]
How does this second method work? Is it the exact same process? Does it use less / more memory?
import dis
def swap_using_temp(a, b):
temp = a
a = b
b = temp
def swap_using_unpacking(a, b):
a, b = b, a
swap_using_unpacking does not require extra memory.
Explanation:
If you disassemble the code using dis module of both the function described then you will see that in swap_using_unpacking there is a bytecode instruction ROT_TWO which swaps the 2 topmost elements of the stack(which don't require a third variable hence no extra memory is consumed).
dis.dis(swap_using_unpacking)
11 0 LOAD_FAST 1 (b)
2 LOAD_FAST 0 (a)
4 ROT_TWO
6 STORE_FAST 0 (a)
8 STORE_FAST 1 (b)
10 LOAD_CONST 0 (None)
12 RETURN_VALUE
dis.dis(swap_using_temp)
5 0 LOAD_FAST 0 (a)
2 STORE_FAST 2 (temp)
6 4 LOAD_FAST 1 (b)
6 STORE_FAST 0 (a)
7 8 LOAD_FAST 2 (temp)
10 STORE_FAST 1 (b)
12 LOAD_CONST 0 (None)
14 RETURN_VALUE
You are asking the wrong question here. You are not using assembly language but Python, so you should not worry for some extra bytes. What really matters in Python is readability.
Anyway, both versions should be internally implemented more or less the same way, except that the first one creates an additional identifier. It is explicitely named temp so provided you do not use it in that scope, it brings no real problem.
If you use a linting environment that warns you for possible problems (what you should do...) you must be aware that reusing a variable name that hides the same name in an outer scope, while perfectly correct on a language point of view will light some warning on. But as you should not use a temp identifier outside a local scope (readability...) it should not be a problem either.
So it is more of a matter of taste. If you or your team mates often use other languages that do not allow multiple assignments the first way will be more natural. If you mainly use Python, the second way is IMHO more pythonic because it avoids adding an unnecessary identifier in the local scope. But as I have already said, nothing more that a matter of taste...
Why is it that according to the timeit.timeit function the code boolean = True if foo else False runs faster than the code boolean = bool(foo)?
How is it that the if statement is able to determine the trueness of foo faster then the bool function itself?
Why doesn't the bool function simply use the same mechanic?
And what is the purpose of the bool function when it can be outperformed by a factor of four by a different technique?
Or, is it so that I am misusing the timeit function and that bool(foo) is, in fact, faster?
>>> timeit.timeit("boolean = True if foo else False", setup="foo='zon-zero'")
0.021019499999965774
>>> timeit.timeit("boolean = bool(foo)", setup="foo='zon-zero'")
0.0684856000000309
>>> timeit.timeit("boolean = True if foo else False", setup="foo=''")
0.019911300000103438
>>> timeit.timeit("boolean = bool(foo)", setup="foo=''")
0.09232059999999365
Looking at these results, True if foo else False seems to be four to five times faster than bool(foo).
I suspect that the difference in speed is caused by the overhead of calling a function and that does indeed seem to be the case when I use the dis module.
>>> dis.dis("boolean = True if foo else False")
1 0 LOAD_NAME 0 (foo)
2 POP_JUMP_IF_FALSE 8
4 LOAD_CONST 0 (True)
6 JUMP_FORWARD 2 (to 10)
>> 8 LOAD_CONST 1 (False)
>> 10 STORE_NAME 1 (boolean)
12 LOAD_CONST 2 (None)
14 RETURN_VALUE
>>> dis.dis("boolean = bool(foo)")
1 0 LOAD_NAME 0 (bool)
2 LOAD_NAME 1 (foo)
4 CALL_FUNCTION 1
6 STORE_NAME 2 (boolean)
8 LOAD_CONST 0 (None)
10 RETURN_VALUE
According to the dis module, than the difference between the two techniques is:
2 POP_JUMP_IF_FALSE 8
4 LOAD_CONST 0 (True)
6 JUMP_FORWARD 2 (to 10)
>> 8 LOAD_CONST 1 (False)
versus
0 LOAD_NAME 1 (bool)
4 CALL_FUNCTION 1
which makes it look like either the call to a function is far too expensive for something as simple as determining a boolean value or the bool function has been written very inefficiently.
But that actually makes me wonder why anyone would use the bool function when it is this much slower and why the bool function even exists when python does not even seem to use it internally.
So, is the bool function slower because it has been written inefficiently, because of the function overhead, or because of a different reason?
And why would anyone use the bool function when a much faster and equally clear alternative is available?
As per Python documentation :
class bool( [ x ] )
Return a Boolean value, i.e. one of True or False. x is converted using the standard truth testing
procedure. If x is false or omitted, this returns False; otherwise it returns True. The bool class is a
subclass of int (see Numeric Types — int, float, complex). It cannot be subclassed further. Its only
instances are False and True
So, when you directly use the object itself (like foo), the interpreter uses its foo.__bool__ property. But the bool function is a wrapper that again calls foo.__bool__
As you said, calling the function made it expensive.
And the use of bool is, there are certain situations where you need the boolean value of an object and need to refer it by a variable.
x = bool(my_object)
Writing x = my_object doesn't work.
Here its useful.
Sometimes bool(foo) is more readable where you can ignore small time lags.
You might be also interested in knowing that
x = {}
is faster than
x = dict()
Find out why... :)
Please forgive a Python enthusiast a mostly academic question.
I was interested in the cost, if any, of nested functions - not the functionally justified ones that utilize closure etc., but the keep the outer namespace tidy variety.
So I did a simple measurement:
def inner(x):
return x*x
def flat(x):
return inner(x)
def nested(x):
def inner(x):
return x*x
return inner(x)
# just to get a feel of the cost of having two more lines
def fake_nested(x):
y = x
z = x
return inner(x)
from timeit import timeit
print(timeit('f(3)', globals=dict(f=flat)))
print(timeit('f(3)', globals=dict(f=nested)))
print(timeit('f(3)', globals=dict(f=fake_nested)))
# 0.17055258399341255
# 0.23098028398817405
# 0.19381927204085514
So it seems that there is some overhead and it appears to be more than would be explained by having two more lines.
It seems, however, that the inner def statement is not evaluated each time the outer function is called, indeed the inner function object appears to be cached:
def nested(x):
def inner(x):
return x*x
print(id(inner), id(inner.__code__), id(inner.__closure__))
return inner(x)
nested(3)
x = [list(range(i)) for i in range(5000)] # create some memory pressure
nested(3)
# 139876371445960 139876372477824 8845216
# 139876371445960 139876372477824 8845216
Looking for other things that might add to the longer runtime I stumbled over the following nerdgasm:
def nested(x):
def inner(x):
return x*x
print(id(inner), id(inner.__code__), id(inner.__closure__))
return inner
nested(3)
x = [list(range(i)) for i in range(5000)] # create some memory pressure
a = nested(3)
x = [list(range(i)) for i in range(5000)] # create some memory pressure
nested(3)
# 139906265032768 139906264446704 8845216
# 139906265032768 139906264446704 8845216
# 139906264258624 139906264446704 8845216
It seems that if Python detects that there is an outer reference to the cached nested function, then it creates a new function object.
Now - assuming my reasoning so far is not completely off - my question: What is this good for?
My first idea was "Ok, if the user has a reference to the cached function, they may have messsed with it, so better make a clean new one." But on second thoughts that doesn't seem to wash because the copy is not a deep copy and also what if the user messes with the function and then throws the reference away?
Supplementary question: Does Python do any other fiendishly clever things behind the scenes? And is this at all related to the slower execution of nested compared to flat?
Your reasoning is completely off. Python always creates a new function object each time a def is encountered in the normal program flow - no exceptions.
It is just that in CPython the id of the newly created function likely is the same as that of the old. See "Why does id({}) == id({}) and id([]) == id([]) in CPython?".
Now, if you saved a reference to the inner function, it is not deleted before the next function is created, and naturally the new function cannot coexist at the same memory address.
As for the time difference, a look at the bytecode of the two functions provides some hints. Comparison between nested() and fake_nested() shows that whereas fake_nested just loads already defined global function inner(), nested has to create this function. There will be some overhead here whereas the other operations will be relatively fast.
>>> import dis
>>> dis.dis(flat)
2 0 LOAD_GLOBAL 0 (inner)
3 LOAD_FAST 0 (x)
6 CALL_FUNCTION 1
9 RETURN_VALUE
>>> dis.dis(nested)
2 0 LOAD_CONST 1 (<code object inner at 0x7f2958a33830, file "<stdin>", line 2>)
3 MAKE_FUNCTION 0
6 STORE_FAST 1 (inner)
4 9 LOAD_FAST 1 (inner)
12 LOAD_FAST 0 (x)
15 CALL_FUNCTION 1
18 RETURN_VALUE
>>> dis.dis(fake_nested)
2 0 LOAD_FAST 0 (x)
3 STORE_FAST 1 (y)
3 6 LOAD_FAST 0 (x)
9 STORE_FAST 2 (z)
4 12 LOAD_GLOBAL 0 (inner)
15 LOAD_FAST 0 (x)
18 CALL_FUNCTION 1
21 RETURN_VALUE
As for the inner function caching part, the other answer already clarifies that a new inner() function will be created every time nested() is run. To see this more clearly see the following variation on nested(), cond_nested() which creates same functions with two different names based on a flag. First time this runs with a False flag second function inner2() is created. Next when I change the flag to True the first function inner1() is created and the memory occupied by second function inner2() is freed. So if I run again with True flag, the first function is again created and is assigned a memory that was occupied by second function which is free now.
>>> def cond_nested(x, flag=False):
... if flag:
... def inner1(x):
... return x*x
... cond_nested.func = inner1
... print id(inner1)
... return inner1(x)
... else:
... def inner2(x):
... return x*x
... cond_nested.func = inner2
... print id(inner2)
... return inner2(x)
...
>>> cond_nested(2)
139815557561112
4
>>> cond_nested.func
<function inner2 at 0x7f2958a47b18>
>>> cond_nested(2, flag=True)
139815557561352
4
>>> cond_nested.func
<function inner1 at 0x7f2958a47c08>
>>> cond_nested(3, flag=True)
139815557561112
9
>>> cond_nested.func
<function inner1 at 0x7f2958a47b18>
I tried to search for this answer on my own, but there was too much noise.
Are generators in python just a convenience wrapper for the user to make an iterator object?
When you define the generator:
def test():
x = 0
while True:
x += 1
yield x
is python simply making a new object, adding the __iter__ method, then putting the rest of the code into the next function?
class Test(object):
def __init__(self):
self.x = 0
def __iter__(self):
return self
def next(self):
self.x += 1
return self.x
Nope. Like so:
>>> def test():
... x = 0
... while True:
... x += 1
... yield x
...
>>> type(test)
<type 'function'>
So what it returns is a function object. Details from there get hairy; the short course is that the code object belonging to the function (test.func_code) is marked as a generator by one of the flags in test.func_code.co_flags.
You can disassemble the bytecode for test to see that it's just like any other function otherwise, apart from that a generator function always contains a YIELD_VALUE opcode:
>>> import dis
>>> dis.dis(test)
2 0 LOAD_CONST 1 (0)
3 STORE_FAST 0 (x)
3 6 SETUP_LOOP 25 (to 34)
>> 9 LOAD_GLOBAL 0 (True)
12 POP_JUMP_IF_FALSE 33
4 15 LOAD_FAST 0 (x)
18 LOAD_CONST 2 (1)
21 INPLACE_ADD
22 STORE_FAST 0 (x)
5 25 LOAD_FAST 0 (x)
28 YIELD_VALUE
29 POP_TOP
30 JUMP_ABSOLUTE 9
>> 33 POP_BLOCK
>> 34 LOAD_CONST 0 (None)
37 RETURN_VALUE
To do it the way you have in mind, the horrors just start ;-) if you think about how to create an object to mimic just this:
def test():
yield 2
yield 3
yield 4
Now your next() method would have to carry additional hidden state just to remember which yield comes next. Wrap that in some nested loops with some conditionals, and "unrolling" it into a single-entry next() becomes a nightmare.
No -- Generators also provide other methods (.send, .throw, etc) and can be used for more purposes than simply making iterators (e.g. coroutines).
Indeed, generators are an entirely different beast and a core language feature. It'd be very hard (possibly impossible) to create one in vanilla python if they weren't baked into the language.
With that said, one application of generators is to provide an easy syntax for creating an iterator :-).
Are generators in python just a convenience wrapper for the user to make an iterator object?
No. A generator is a function, where as iterators are class. Hence, generator can not be a object of iterator. But in some way you can say that generator is a simplified approach to get iterator like capability. It means:
All Generators are iterators, but not all iterators are generators.
I will strongly suggest you refer below wiki links:
Iterator - traverses a collection one at a time
Generator - generates a sequence, one item at a time
An iterator is typically something that has a next method to get the next element from a stream. A generator is an iterator that is tied to a function.
I will suggest you to refer: Difference between Python's Generators and Iterators.