Suppose I have two lists, A = [1,2,3,4] and B = [4,5,6]
I would like a list which includes the elements from both A and B. (I don't care if A itself gets altered).
A couple things I could do, and my understanding of them (please tell me if I am wrong):
A.extend(B) (elements of B get added in to A; A itself is altered)
C = A + B (makes a brand new object C, which contains the contents of A and B in it.)
I wanted to understand which is more efficient, so I was wondering if someone can someone please tell me if my assumptions below are incorrect.
In the case of A.extend(B), I'm assuming python only has to do 3 list add operations (the 3 elements of B, each of which it appends to A). However, in doing A + B, doesn't python have to iterate through both lists A and B, in that case doing 7 list add operations? (i.e., it has to make a new list, go through A and put all the elements in it, and then go through B and put all the elements in it).
Am I misunderstanding how the interpreter handles these things, or what these operations do in python?
Below is the bytecode analysis of both operations. There are no major performance difference between two. The only difference is that the .extend way involves a CALL_FUNCTION, which is slightly more expensive in Python than the BINARY_ADD.
But this should not be a problem unless of are working on huge data operations.
>>> import dis
>>> a = [1,2,3,4]
>>> b = [4,5,6]
>>> def f1(a,b):
... a.extend(b)
>>> def f2(a,b):
... c = a+ b
>>> dis.dis(f1)
2 0 LOAD_FAST 0 (a)
3 LOAD_ATTR 0 (extend)
6 LOAD_FAST 1 (b)
9 CALL_FUNCTION 1
12 POP_TOP
13 LOAD_CONST 0 (None)
16 RETURN_VALUE
>>> dis.dis(f2)
2 0 LOAD_FAST 0 (a)
3 LOAD_FAST 1 (b)
6 BINARY_ADD
7 STORE_FAST 2 (c)
10 LOAD_CONST 0 (None)
13 RETURN_VALUE
Related
This question already has answers here:
"is" operator behaves unexpectedly with integers
(11 answers)
Closed 6 months ago.
I have this code
l = 40000
m = 40000
print(type(l),type(m))
if l is m:
print("same")
else:
print("nope")
Taking reference from here I was hoping id to be different since the value are not falling in range of (-2,256). Please let me know if I am missing out on something
Myself using Python-3.8.3(32bit) on windows Platform
This is due to a simple optimization at the bytecode compilation stage; if the same constant appears in the same code more than once, it is created only once, and each use of the constant will refer to the same instance.
We can investigate the bytecode with the dis module:
>>> import dis
>>> dis.dis('l = 40000\nm=40000')
1 0 LOAD_CONST 0 (40000)
2 STORE_NAME 0 (l)
2 4 LOAD_CONST 0 (40000)
6 STORE_NAME 1 (m)
8 LOAD_CONST 1 (None)
10 RETURN_VALUE
>>> dis.dis('l = 40000\nm=40001')
1 0 LOAD_CONST 0 (40000)
2 STORE_NAME 0 (l)
2 4 LOAD_CONST 1 (40001)
6 STORE_NAME 1 (m)
8 LOAD_CONST 2 (None)
10 RETURN_VALUE
Note that in the first case where both constants are 40000, the two LOAD_CONST operations both load constant #0, but in the second case, 40000 is constant #0 and 40001 is constant #1.
Also, you will usually get different results if you do this on separate lines in the REPL, since each line in the REPL is compiled and executed as a separate code object, so they cannot share constants:
>>> l = 50000
>>> m = 50000
>>> id(l)
140545755966480
>>> id(m)
140545755966448
But if you do both in one line in the REPL, the same instance of the constant is used again, because it's just one line compiled to one code object, so the constant can be shared:
>>> p = 60000; q = 60000
>>> id(p)
140545755966576
>>> id(q)
140545755966576
I tried in the snippet below:
a, b = a[b] = {}, 5
print('a={0},b={1}'.format(a,b))
The IDE spits out the follows:
a={5: ({...}, 5)},b=5
I have tried S3DEV's advice and execute:
from dis import dis
dis('a, b = a[b] = {}, 5')
And it gives me the follows:
1 0 BUILD_MAP 0
2 LOAD_CONST 0 (5)
4 BUILD_TUPLE 2
6 DUP_TOP
8 UNPACK_SEQUENCE 2
10 STORE_NAME 0 (a)
12 STORE_NAME 1 (b)
14 LOAD_NAME 0 (a)
16 LOAD_NAME 1 (b)
18 STORE_SUBSCR
20 LOAD_CONST 1 (None)
22 RETURN_VALUE
But I still cannot understand why a[b] = a, 5 happened in the step 18 STORE_SUBSCR. Any further explanation?
This is an assignment statement with multiple target_list:s, for which case the docs say that the statement "assigns the single resulting object to each of the target lists, from left to right." Within each target_list, assignments also proceed left to right.
Thus, the statement is equivalent to
a = {}
b = 5
a[b] = a, 5
The reason that the last assignment is a[b]=a,5 and not a,b={},5 is that the value ({}, 5) is only evaluated once, so it's the same dict that gets used throughout. First, a is set to refer to that dict, then the dict — through a — is modified to refer to itself.
EDIT: Perhaps it is clearer to say that the statement is equivalent to
temp1 = {}
temp2 = 5
a = temp1
b = temp2
a[b] = temp1, temp2
Right before the last step, a and temp1 refer to the same object, which thus becomes self-referring after the last step.
This is not code I want to see in production. :)
In [55]: a = 5
In [56]: b = 6
In [57]: (a, b) = (b, a)
In [58]: a
Out[58]: 6
In [59]: b
Out[59]: 5
How does this swapping of values of a and b work internally? Its definitely not using a temp variable.
Python separates the right-hand side expression from the left-hand side assignment. First the right-hand side is evaluated, and the result is stored on the stack, and then the left-hand side names are assigned using opcodes that take values from the stack again.
For tuple assignments with 2 or 3 items, Python just uses the stack directly:
>>> import dis
>>> def foo(a, b):
... a, b = b, a
...
>>> dis.dis(foo)
2 0 LOAD_FAST 1 (b)
3 LOAD_FAST 0 (a)
6 ROT_TWO
7 STORE_FAST 0 (a)
10 STORE_FAST 1 (b)
13 LOAD_CONST 0 (None)
16 RETURN_VALUE
After the two LOAD_FAST opcodes (which push a value from a variable onto the stack), the top of stack holds [a, b]. The ROT_TWO opcode swaps the top two positions on the stack so the stack now has [b, a] at the top. The two STORE_FAST opcodes then takes those two values and store them in the names on the left-hand side of the assignment. The first STORE_FAST pops a value of the top of the stack and puts it into a, the next pops again, storing the value in b. The rotation is needed because Python guarantees that assignments in a target list on the left-hand side are done from left to right.
For a 3-name assignment, ROT_THREE followed by ROT_TWO is executed to reverse the top three items on the stack.
For longer left-hand-side assignments, an explicit tuple is built:
>>> def bar(a, b, c, d):
... d, c, b, a = a, b, c, d
...
>>> dis.dis(bar)
2 0 LOAD_FAST 0 (a)
3 LOAD_FAST 1 (b)
6 LOAD_FAST 2 (c)
9 LOAD_FAST 3 (d)
12 BUILD_TUPLE 4
15 UNPACK_SEQUENCE 4
18 STORE_FAST 3 (d)
21 STORE_FAST 2 (c)
24 STORE_FAST 1 (b)
27 STORE_FAST 0 (a)
30 LOAD_CONST 0 (None)
33 RETURN_VALUE
Here the stack with [d, c, b, a] is used to build a tuple (in reverse order, BUILD_TUPLE pops from the stack again, pushing the resulting tuple onto the stack), and then UNPACK_SEQUENCE pops the tuple from the stack again, pushes all elements back from the tuple back onto the stack again for the STORE_FAST operations.
The latter may seem like a wasteful operation, but the right-hand side of an assignment may be something entirely different, a function call that produces a tuple perhaps, so the Python interpreter makes no assumptions and uses the UNPACK_SEQUENCE opcode always. It does so even for the two and three-name assignment operations, but a later (peephole) optimization step replaces a BUILD_TUPLE / UNPACK_SEQUENCE combination with 2 or 3 arguments with the above ROT_TWO and ROT_THREE opcodes for efficiency.
This question already has an answer here:
Python Assignment Operator Precedence - (a, b) = a[b] = {}, 5
(1 answer)
Closed 4 years ago.
I found the assignment a = a[1:] = [2] in an article. I tried it in python3 and python2; it all works, but I don't understand how it works. = here is not like in C; C processes = by right to left. How does python process the = operator?
Per the language docs on assignment:
An assignment statement evaluates the expression list (remember that this can be a single expression or a comma-separated list, the latter yielding a tuple) and assigns the single resulting object to each of the target lists, from left to right.
In this case, a = a[1:] = [2] has an expression list [2], and two "target lists", a and a[1:], where a is the left-most "target list".
You can see how this behaves by looking at the disassembly:
>>> import dis
>>> dis.dis('a = a[1:] = [2]')
1 0 LOAD_CONST 0 (2)
2 BUILD_LIST 1
4 DUP_TOP
6 STORE_NAME 0 (a)
8 LOAD_NAME 0 (a)
10 LOAD_CONST 1 (1)
12 LOAD_CONST 2 (None)
14 BUILD_SLICE 2
16 STORE_SUBSCR
18 LOAD_CONST 2 (None)
20 RETURN_VALUE
(The last two lines of the disassembly can be ignored, dis is making a function wrapper to disassemble the string)
The important part to note is that when you do x = y = some_val, some_val is loaded on the stack (in this case by the LOAD_CONST and BUILD_LIST), then the stack entry is duplicated and assigned, from left to right, to the targets given.
So when you do:
a = a[1:] = [2]
it makes two references to a brand new list containing 2, and the first action is a STORE one of these references to a. Next, it stores the second reference to a[1:], but since the slice assignment mutates a itself, it has to load a again, which gets the list just stored. Luckily, list is resilient against self-slice-assignment, or we'd have issues (it would be forever reading the value it just added to add to the end until we ran out of memory and crashed); as is, it behaves as a copy of [2] was assigned to replace any and all elements from index one onwards.
The end result is equivalent to if you'd done:
_ = [2]
a = _
a[1:] = _
but it avoids the use of the _ name.
To be clear, the disassembly annotated:
Make list [2]:
1 0 LOAD_CONST 0 (2)
2 BUILD_LIST 1
Make a copy of the reference to [2]:
4 DUP_TOP
Perform store to a:
6 STORE_NAME 0 (a)
Perform store to a[1:]:
8 LOAD_NAME 0 (a)
10 LOAD_CONST 1 (1)
12 LOAD_CONST 2 (None)
14 BUILD_SLICE 2
16 STORE_SUBSCR
The way I understand such assignments is that this is equivalent to
temp = [2]
a = temp
a[1:] = temp
The resulting value of [2, 2] is consistent with this interpretation.
In [55]: a = 5
In [56]: b = 6
In [57]: (a, b) = (b, a)
In [58]: a
Out[58]: 6
In [59]: b
Out[59]: 5
How does this swapping of values of a and b work internally? Its definitely not using a temp variable.
Python separates the right-hand side expression from the left-hand side assignment. First the right-hand side is evaluated, and the result is stored on the stack, and then the left-hand side names are assigned using opcodes that take values from the stack again.
For tuple assignments with 2 or 3 items, Python just uses the stack directly:
>>> import dis
>>> def foo(a, b):
... a, b = b, a
...
>>> dis.dis(foo)
2 0 LOAD_FAST 1 (b)
3 LOAD_FAST 0 (a)
6 ROT_TWO
7 STORE_FAST 0 (a)
10 STORE_FAST 1 (b)
13 LOAD_CONST 0 (None)
16 RETURN_VALUE
After the two LOAD_FAST opcodes (which push a value from a variable onto the stack), the top of stack holds [a, b]. The ROT_TWO opcode swaps the top two positions on the stack so the stack now has [b, a] at the top. The two STORE_FAST opcodes then takes those two values and store them in the names on the left-hand side of the assignment. The first STORE_FAST pops a value of the top of the stack and puts it into a, the next pops again, storing the value in b. The rotation is needed because Python guarantees that assignments in a target list on the left-hand side are done from left to right.
For a 3-name assignment, ROT_THREE followed by ROT_TWO is executed to reverse the top three items on the stack.
For longer left-hand-side assignments, an explicit tuple is built:
>>> def bar(a, b, c, d):
... d, c, b, a = a, b, c, d
...
>>> dis.dis(bar)
2 0 LOAD_FAST 0 (a)
3 LOAD_FAST 1 (b)
6 LOAD_FAST 2 (c)
9 LOAD_FAST 3 (d)
12 BUILD_TUPLE 4
15 UNPACK_SEQUENCE 4
18 STORE_FAST 3 (d)
21 STORE_FAST 2 (c)
24 STORE_FAST 1 (b)
27 STORE_FAST 0 (a)
30 LOAD_CONST 0 (None)
33 RETURN_VALUE
Here the stack with [d, c, b, a] is used to build a tuple (in reverse order, BUILD_TUPLE pops from the stack again, pushing the resulting tuple onto the stack), and then UNPACK_SEQUENCE pops the tuple from the stack again, pushes all elements back from the tuple back onto the stack again for the STORE_FAST operations.
The latter may seem like a wasteful operation, but the right-hand side of an assignment may be something entirely different, a function call that produces a tuple perhaps, so the Python interpreter makes no assumptions and uses the UNPACK_SEQUENCE opcode always. It does so even for the two and three-name assignment operations, but a later (peephole) optimization step replaces a BUILD_TUPLE / UNPACK_SEQUENCE combination with 2 or 3 arguments with the above ROT_TWO and ROT_THREE opcodes for efficiency.