I need to assign values to a bunch of variables. If the value is None, the variable should stay put, but if there is a value, it should get assigned. The obvious way is
if v is not None:
x = v
but repeating this construct over and over again uglifies the code. Doing this works
x = v if v is not None else x
but it does an unnecessary assignment operation and this is a frequently executed code path.
Is there a better way? Or does python optimize something like this and there is no assignment?
Using the dis module, we can examine the assembled python.
import dis
def a(v):
if v is not None:
x = v
def b(v):
x = v if v is not None else x
It would appear that the second method is actually slightly faster, although extremely marginally.
>>> dis.dis(a)
2 0 LOAD_FAST 0 (v)
2 LOAD_CONST 0 (None)
4 COMPARE_OP 9 (is not)
6 POP_JUMP_IF_FALSE 12
8 LOAD_FAST 0 (v)
10 JUMP_FORWARD 2 (to 14)
>> 12 LOAD_FAST 1 (x)
>> 14 STORE_FAST 1 (x)
16 LOAD_CONST 0 (None)
18 RETURN_VALUE
>>> dis.dis(b)
4 0 LOAD_FAST 0 (v)
2 LOAD_CONST 0 (None)
4 COMPARE_OP 9 (is not)
6 POP_JUMP_IF_FALSE 12
5 8 LOAD_FAST 0 (v)
10 STORE_FAST 1 (x)
>> 12 LOAD_CONST 0 (None)
14 RETURN_VALUE
That being said, pick whatever is more readable, or more accepted. I don't think two instructions is noticeable on any scale.
Related
In this trivial example, I want to factor out the i < 5 condition of a list comprehension into it's own function. I also want to eat my cake and have it too, and avoid the overhead of the CALL_FUNCTION bytecode/creating a new frame in the python virtual machine.
Is there any way to factor out the conditions inside of a list comprehension into a new function but somehow get a disassembled result that avoids the large overhead of CALL_FUNCTION?
import dis
import sys
import timeit
def my_filter(n):
return n < 5
def a():
# list comprehension with function call
return [i for i in range(10) if my_filter(i)]
def b():
# list comprehension without function call
return [i for i in range(10) if i < 5]
assert a() == b()
>>> sys.version_info[:]
(3, 6, 5, 'final', 0)
>>> timeit.timeit(a)
1.2616060493517098
>>> timeit.timeit(b)
0.685117881097812
>>> dis.dis(a)
3 0 LOAD_CONST 1 (<code object <listcomp> at 0x0000020F4890B660, file "<stdin>", line 3>)
# ...
>>> dis.dis(b)
3 0 LOAD_CONST 1 (<code object <listcomp> at 0x0000020F48A42270, file "<stdin>", line 3>)
# ...
# list comprehension with function call
# big overhead with that CALL_FUNCTION at address 12
>>> dis.dis(a.__code__.co_consts[1])
3 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 16 (to 22)
6 STORE_FAST 1 (i)
8 LOAD_GLOBAL 0 (my_filter)
10 LOAD_FAST 1 (i)
12 CALL_FUNCTION 1
14 POP_JUMP_IF_FALSE 4
16 LOAD_FAST 1 (i)
18 LIST_APPEND 2
20 JUMP_ABSOLUTE 4
>> 22 RETURN_VALUE
# list comprehension without function call
>>> dis.dis(b.__code__.co_consts[1])
3 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 16 (to 22)
6 STORE_FAST 1 (i)
8 LOAD_FAST 1 (i)
10 LOAD_CONST 0 (5)
12 COMPARE_OP 0 (<)
14 POP_JUMP_IF_FALSE 4
16 LOAD_FAST 1 (i)
18 LIST_APPEND 2
20 JUMP_ABSOLUTE 4
>> 22 RETURN_VALUE
I'm willing to take a hacky solution that I would never use in production, like somehow replacing the bytecode at run time.
In other words, is it possible to replace a's addresses 8, 10, and 12 with b's 8, 10, and 12 at runtime?
Consolidating all of the excellent answers in the comments into one.
As georg says, this sounds like you are looking for a way to inline a function or an expression, and there is no such thing in CPython attempts have been made: https://bugs.python.org/issue10399
Therefore, along the lines of "metaprogramming", you can build the lambda's inline and eval:
from typing import Callable
import dis
def b():
# list comprehension without function call
return [i for i in range(10) if i < 5]
def gen_list_comprehension(expr: str) -> Callable:
return eval(f"lambda: [i for i in range(10) if {expr}]")
a = gen_list_comprehension("i < 5")
dis.dis(a.__code__.co_consts[1])
print("=" * 10)
dis.dis(b.__code__.co_consts[1])
which when run under 3.7.6 gives:
6 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 16 (to 22)
6 STORE_FAST 1 (i)
8 LOAD_FAST 1 (i)
10 LOAD_CONST 0 (5)
12 COMPARE_OP 0 (<)
14 POP_JUMP_IF_FALSE 4
16 LOAD_FAST 1 (i)
18 LIST_APPEND 2
20 JUMP_ABSOLUTE 4
>> 22 RETURN_VALUE
==========
1 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 16 (to 22)
6 STORE_FAST 1 (i)
8 LOAD_FAST 1 (i)
10 LOAD_CONST 0 (5)
12 COMPARE_OP 0 (<)
14 POP_JUMP_IF_FALSE 4
16 LOAD_FAST 1 (i)
18 LIST_APPEND 2
20 JUMP_ABSOLUTE 4
>> 22 RETURN_VALUE
From a security standpoint "eval" is dangerous, athough here it is less so because what you can do inside a lambda. And what can be done in an IfExp expression is even more limited, but still dangerous like call a function that does evil things.
However, if you want the same effect that is more secure, instead of working with strings you can modify AST's. I find that a lot more cumbersome though.
A hybrid approach would be the call ast.parse() and check the result. For example:
import ast
def is_cond_str(s: str) -> bool:
try:
mod_ast = ast.parse(s)
expr_ast = isinstance(mod_ast.body[0])
if not isinstance(expr_ast, ast.Expr):
return False
compare_ast = expr_ast.value
if not isinstance(compare_ast, ast.Compare):
return False
return True
except:
return False
This is a little more secure, but there still may be evil functions in the condition so you could keep going. Again, I find this a little tedious.
Coming from the other direction of starting off with bytecode, there is my cross-version assembler; see https://pypi.org/project/xasm/
This question already has answers here:
Simplify Chained Comparison
(2 answers)
How do chained comparisons in Python actually work?
(1 answer)
Closed 4 years ago.
Probably some of you might think this is duplicate, and yes, I found a lot of examples similar to this:
But Pycharm says that I could simply this:
if y > x and x != -1:
# do something
And I did some searching and could not find something similar.
My question is it correct to simplify this function like this:
if y > x != -1:
# do something
And if so, is it safe and is there is any difference at all between this version and not simplified version except that it's shorter?
If it is not correct way to simplify it, what is then?
this is functionnaly equivalent, but when this:
10 < x < 40
is nice to read, mixing different operators to use chained comparisons isn't the best choice.
Is that really the same? let's disassemble to find out:
def f1(x,y):
if y > x and x != -1:
return 0
def f2(x,y):
if y > x != -1:
return 0
import dis
print("function 1")
dis.dis(f1)
print("function 2")
dis.dis(f2)
result:
function 1
2 0 LOAD_FAST 1 (y)
3 LOAD_FAST 0 (x)
6 COMPARE_OP 4 (>)
9 POP_JUMP_IF_FALSE 28
12 LOAD_FAST 0 (x)
15 LOAD_CONST 3 (-1)
18 COMPARE_OP 3 (!=)
21 POP_JUMP_IF_FALSE 28
3 24 LOAD_CONST 2 (0)
27 RETURN_VALUE
>> 28 LOAD_CONST 0 (None)
31 RETURN_VALUE
function 2
6 0 LOAD_FAST 1 (y)
3 LOAD_FAST 0 (x)
6 DUP_TOP
7 ROT_THREE
8 COMPARE_OP 4 (>)
11 JUMP_IF_FALSE_OR_POP 23
14 LOAD_CONST 3 (-1)
17 COMPARE_OP 3 (!=)
20 JUMP_FORWARD 2 (to 25)
>> 23 ROT_TWO
24 POP_TOP
>> 25 POP_JUMP_IF_FALSE 32
7 28 LOAD_CONST 2 (0)
31 RETURN_VALUE
>> 32 LOAD_CONST 0 (None)
35 RETURN_VALUE
>>>
Surprisingly they aren't the same, and the chained version has more instructions.
Not sure what's going on here (some took some more time to explain it better: How do chained comparisons in Python actually work?), but really I'd stick to the and version which shortcuts and is so much readable (think of future maintainers too...).
That said, one interesting thing about chained comparisons would be if the central argument is computed/takes a long time to compute/has a side effect in the computation and you don't want to store it in a variable:
if y > super_long_computation(x) != -1:
In that case, the central argument is only evaluated once. In the case of and, you'd have to store it beforehand.
In Python, while assigning a value to a variable, we can either do:
variable = variable + 20
or
variable += 20.
While I do understand that both the operations are semantically same, i.e., they achieve the same goal of increasing the previous value of variable by 20, I was wondering if there are subtle run-time performance differences between the two, or any other slight differences which might deem one better than the other.
Is there any such difference, or are they exactly the same?
If there is any difference, is it the same for other languages such as C++?
Thanks.
Perhaps this can help you understand better:
import dis
def a():
x = 0
x += 20
return x
def b():
x = 0
x = x + 20
return x
print 'In place add'
dis.dis(a)
print 'Binary add'
dis.dis(b)
We get the following outputs:
In place add
4 0 LOAD_CONST 1 (0)
3 STORE_FAST 0 (x)
5 6 LOAD_FAST 0 (x)
9 LOAD_CONST 2 (20)
12 INPLACE_ADD
13 STORE_FAST 0 (x)
6 16 LOAD_FAST 0 (x)
19 RETURN_VALUE
Binary add
9 0 LOAD_CONST 1 (0)
3 STORE_FAST 0 (x)
10 6 LOAD_FAST 0 (x)
9 LOAD_CONST 2 (20)
12 BINARY_ADD
13 STORE_FAST 0 (x)
11 16 LOAD_FAST 0 (x)
19 RETURN_VALUE
You could do a loop a thousand or so times using a timer to compare perfomance, but the main difference is that one. I suppose binary add should be faster tho.
When using the in operator on a literal, is it most idiomatic for that literal to be a list, set, or tuple?
e.g.
for x in {'foo', 'bar', 'baz'}:
doSomething(x)
...
if val in {1, 2, 3}:
doSomethingElse(val)
I don't see any benefit to the list, but the tuple's immutably means it could be hoisted or reused by an efficient interpreter. And in the case of the if, if it's reused, there's an efficiency benefit.
Which is the most idiomatic, and which is most performant in cpython?
Python provides a disassembler, so you can often just check the bytecode:
In [4]: def checktup():
...: for _ in range(10):
...: if val in (1, 2, 3):
...: print("foo")
...:
In [5]: def checkset():
...: for _ in range(10):
...: if val in {1, 2, 3}:
...: print("foo")
...:
In [6]: import dis
For the tuple literal:
In [7]: dis.dis(checktup)
2 0 SETUP_LOOP 32 (to 34)
2 LOAD_GLOBAL 0 (range)
4 LOAD_CONST 1 (10)
6 CALL_FUNCTION 1
8 GET_ITER
>> 10 FOR_ITER 20 (to 32)
12 STORE_FAST 0 (_)
3 14 LOAD_GLOBAL 1 (val)
16 LOAD_CONST 6 ((1, 2, 3))
18 COMPARE_OP 6 (in)
20 POP_JUMP_IF_FALSE 10
4 22 LOAD_GLOBAL 2 (print)
24 LOAD_CONST 5 ('foo')
26 CALL_FUNCTION 1
28 POP_TOP
30 JUMP_ABSOLUTE 10
>> 32 POP_BLOCK
>> 34 LOAD_CONST 0 (None)
36 RETURN_VALUE
For the set-literal:
In [8]: dis.dis(checkset)
2 0 SETUP_LOOP 32 (to 34)
2 LOAD_GLOBAL 0 (range)
4 LOAD_CONST 1 (10)
6 CALL_FUNCTION 1
8 GET_ITER
>> 10 FOR_ITER 20 (to 32)
12 STORE_FAST 0 (_)
3 14 LOAD_GLOBAL 1 (val)
16 LOAD_CONST 6 (frozenset({1, 2, 3}))
18 COMPARE_OP 6 (in)
20 POP_JUMP_IF_FALSE 10
4 22 LOAD_GLOBAL 2 (print)
24 LOAD_CONST 5 ('foo')
26 CALL_FUNCTION 1
28 POP_TOP
30 JUMP_ABSOLUTE 10
>> 32 POP_BLOCK
>> 34 LOAD_CONST 0 (None)
36 RETURN_VALUE
You'll notice that in both cases, the function will LOAD_CONST, i.e., both times it has been optimized. Even better, in the case of the set literal, the compiler has saved a frozenset, which during the construction of the function, the peephole-optimizer has managed to figure out can become the immutable equivalent of a set.
Note, on Python 2, the compiler builds a set every time!:
In [1]: import dis
In [2]: def checkset():
...: for _ in range(10):
...: if val in {1, 2, 3}:
...: print("foo")
...:
In [3]: dis.dis(checkset)
2 0 SETUP_LOOP 49 (to 52)
3 LOAD_GLOBAL 0 (range)
6 LOAD_CONST 1 (10)
9 CALL_FUNCTION 1
12 GET_ITER
>> 13 FOR_ITER 35 (to 51)
16 STORE_FAST 0 (_)
3 19 LOAD_GLOBAL 1 (val)
22 LOAD_CONST 2 (1)
25 LOAD_CONST 3 (2)
28 LOAD_CONST 4 (3)
31 BUILD_SET 3
34 COMPARE_OP 6 (in)
37 POP_JUMP_IF_FALSE 13
4 40 LOAD_CONST 5 ('foo')
43 PRINT_ITEM
44 PRINT_NEWLINE
45 JUMP_ABSOLUTE 13
48 JUMP_ABSOLUTE 13
>> 51 POP_BLOCK
>> 52 LOAD_CONST 0 (None)
55 RETURN_VALUE
IMO, there's essentially no such thing as "idiomatic" usage of literal values as shown in the question. Such values look like "magic numbers" to me. Using literals for "performance" is probably misguided because it sacrifices readability for marginal gains. In cases where performance really matters, using literals is unlikely to help much and there are better options regardless.
I think the idiomatic thing to do would be to store such values in a global or class variable, especially if you're using them in multiple places (but also even if you aren't). This provides some documentation as to what a value's purpose is and makes it easier to update. You can then memomize these values in function/method definitions to improve performance if necessary.
As to what type of data structure is most appropriate, that would depend on what your program does and how it uses the data. For example, does ordering matter? With an if x in y, it won't, but maybe you're using the data in a for and an if. Without context, it's hard to say what the best choice would be.
Here's an example I think is readable, extensible, and also efficient. Memoizing the global ITEMS in the function definitions makes lookup fast because items is in the local namespace of the function. If you look at the disassembled code, you'll see that items is looked up via LOAD_FAST instead of LOAD_GLOBAL. This approach also avoids making multiple copies of the list of items, which might be relevant if it's big enough (although, if it was big enough, you probably wouldn't try to inline it anyway). Personally, I wouldn't bother with these kinds of optimizations most of the time, but they can be useful in some cases.
# In real code, this would have a domain-specific name instead of the
# generic `ITEMS`.
ITEMS = {'a', 'b', 'c'}
def filter_in_items(values, items=ITEMS):
matching_items = []
for value in values:
if value in items:
matching_items.append(value)
return matching_items
def filter_not_in_items(values, items=ITEMS):
non_matching_items = []
for value in values:
if value not in items:
non_matching_items.append(value)
return non_matching_items
print(filter_in_items(('a', 'x'))) # -> ['a']
print(filter_not_in_items(('a', 'x'))) # -> ['x']
import dis
dis.dis(filter_in_items)
Say I have a list called list, which is comprised of boolean values. Also say that I have some (valid) index i which is the index of list where I want to switch the value.
Currently, I have: list[i] = not list[i].
But my question is, doesn't this iterate through list twice? If so is there are way to setup a temp value through aliasing to only iterate through the list once?
I tried the following:
temp = list[i]
temp = not temp
But this has not worked for me, it has only switched the value of temp, and not the value of list[i].
you can look a little ways 'under the hood' using the dis module https://docs.python.org/3/library/dis.html
import dis
boolst = [True, True, True, True, True]
dis.dis('boolst[2] = not boolst[2]')
1 0 LOAD_NAME 0 (boolst)
2 LOAD_CONST 0 (2)
4 BINARY_SUBSCR
6 UNARY_NOT
8 LOAD_NAME 0 (boolst)
10 LOAD_CONST 0 (2)
12 STORE_SUBSCR
14 LOAD_CONST 1 (None)
16 RETURN_VALUE
dis.dis('boolst[2] ^= True')
1 0 LOAD_NAME 0 (boolst)
2 LOAD_CONST 0 (2)
4 DUP_TOP_TWO
6 BINARY_SUBSCR
8 LOAD_CONST 1 (True)
10 INPLACE_XOR
12 ROT_THREE
14 STORE_SUBSCR
16 LOAD_CONST 2 (None)
18 RETURN_VALUE