Develop Reference

Is it possible to call a function from within a list comprehension without the overhead of calling the function?

In this trivial example, I want to factor out the i < 5 condition of a list comprehension into it's own function. I also want to eat my cake and have it too, and avoid the overhead of the CALL_FUNCTION bytecode/creating a new frame in the python virtual machine.
Is there any way to factor out the conditions inside of a list comprehension into a new function but somehow get a disassembled result that avoids the large overhead of CALL_FUNCTION?
import dis
import sys
import timeit
def my_filter(n):
return n < 5
def a():
# list comprehension with function call
return [i for i in range(10) if my_filter(i)]
def b():
# list comprehension without function call
return [i for i in range(10) if i < 5]
assert a() == b()
>>> sys.version_info[:]
(3, 6, 5, 'final', 0)
>>> timeit.timeit(a)
1.2616060493517098
>>> timeit.timeit(b)
0.685117881097812
>>> dis.dis(a)
3 0 LOAD_CONST 1 (<code object <listcomp> at 0x0000020F4890B660, file "<stdin>", line 3>)
# ...
>>> dis.dis(b)
3 0 LOAD_CONST 1 (<code object <listcomp> at 0x0000020F48A42270, file "<stdin>", line 3>)
# ...
# list comprehension with function call
# big overhead with that CALL_FUNCTION at address 12
>>> dis.dis(a.__code__.co_consts[1])
3 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 16 (to 22)
6 STORE_FAST 1 (i)
8 LOAD_GLOBAL 0 (my_filter)
10 LOAD_FAST 1 (i)
12 CALL_FUNCTION 1
14 POP_JUMP_IF_FALSE 4
16 LOAD_FAST 1 (i)
18 LIST_APPEND 2
20 JUMP_ABSOLUTE 4
>> 22 RETURN_VALUE
# list comprehension without function call
>>> dis.dis(b.__code__.co_consts[1])
3 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 16 (to 22)
6 STORE_FAST 1 (i)
8 LOAD_FAST 1 (i)
10 LOAD_CONST 0 (5)
12 COMPARE_OP 0 (<)
14 POP_JUMP_IF_FALSE 4
16 LOAD_FAST 1 (i)
18 LIST_APPEND 2
20 JUMP_ABSOLUTE 4
>> 22 RETURN_VALUE
I'm willing to take a hacky solution that I would never use in production, like somehow replacing the bytecode at run time.
In other words, is it possible to replace a's addresses 8, 10, and 12 with b's 8, 10, and 12 at runtime?

Consolidating all of the excellent answers in the comments into one.
As georg says, this sounds like you are looking for a way to inline a function or an expression, and there is no such thing in CPython attempts have been made: https://bugs.python.org/issue10399
Therefore, along the lines of "metaprogramming", you can build the lambda's inline and eval:
from typing import Callable
import dis
def b():
# list comprehension without function call
return [i for i in range(10) if i < 5]
def gen_list_comprehension(expr: str) -> Callable:
return eval(f"lambda: [i for i in range(10) if {expr}]")
a = gen_list_comprehension("i < 5")
dis.dis(a.__code__.co_consts[1])
print("=" * 10)
dis.dis(b.__code__.co_consts[1])
which when run under 3.7.6 gives:
6 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 16 (to 22)
6 STORE_FAST 1 (i)
8 LOAD_FAST 1 (i)
10 LOAD_CONST 0 (5)
12 COMPARE_OP 0 (<)
14 POP_JUMP_IF_FALSE 4
16 LOAD_FAST 1 (i)
18 LIST_APPEND 2
20 JUMP_ABSOLUTE 4
>> 22 RETURN_VALUE
==========
1 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 16 (to 22)
6 STORE_FAST 1 (i)
8 LOAD_FAST 1 (i)
10 LOAD_CONST 0 (5)
12 COMPARE_OP 0 (<)
14 POP_JUMP_IF_FALSE 4
16 LOAD_FAST 1 (i)
18 LIST_APPEND 2
20 JUMP_ABSOLUTE 4
>> 22 RETURN_VALUE
From a security standpoint "eval" is dangerous, athough here it is less so because what you can do inside a lambda. And what can be done in an IfExp expression is even more limited, but still dangerous like call a function that does evil things.
However, if you want the same effect that is more secure, instead of working with strings you can modify AST's. I find that a lot more cumbersome though.
A hybrid approach would be the call ast.parse() and check the result. For example:
import ast
def is_cond_str(s: str) -> bool:
try:
mod_ast = ast.parse(s)
expr_ast = isinstance(mod_ast.body[0])
if not isinstance(expr_ast, ast.Expr):
return False
compare_ast = expr_ast.value
if not isinstance(compare_ast, ast.Compare):
return False
return True
except:
return False
This is a little more secure, but there still may be evil functions in the condition so you could keep going. Again, I find this a little tedious.
Coming from the other direction of starting off with bytecode, there is my cross-version assembler; see https://pypi.org/project/xasm/

How to keep variable value if new value is None

I need to assign values to a bunch of variables. If the value is None, the variable should stay put, but if there is a value, it should get assigned. The obvious way is
if v is not None:
x = v
but repeating this construct over and over again uglifies the code. Doing this works
x = v if v is not None else x
but it does an unnecessary assignment operation and this is a frequently executed code path.
Is there a better way? Or does python optimize something like this and there is no assignment?

Using the dis module, we can examine the assembled python.
import dis
def a(v):
if v is not None:
x = v
def b(v):
x = v if v is not None else x
It would appear that the second method is actually slightly faster, although extremely marginally.
>>> dis.dis(a)
2 0 LOAD_FAST 0 (v)
2 LOAD_CONST 0 (None)
4 COMPARE_OP 9 (is not)
6 POP_JUMP_IF_FALSE 12
8 LOAD_FAST 0 (v)
10 JUMP_FORWARD 2 (to 14)
>> 12 LOAD_FAST 1 (x)
>> 14 STORE_FAST 1 (x)
16 LOAD_CONST 0 (None)
18 RETURN_VALUE
>>> dis.dis(b)
4 0 LOAD_FAST 0 (v)
2 LOAD_CONST 0 (None)
4 COMPARE_OP 9 (is not)
6 POP_JUMP_IF_FALSE 12
5 8 LOAD_FAST 0 (v)
10 STORE_FAST 1 (x)
>> 12 LOAD_CONST 0 (None)
14 RETURN_VALUE
That being said, pick whatever is more readable, or more accepted. I don't think two instructions is noticeable on any scale.

Performance of assigning multiple return values from function call [duplicate]

This question already has answers here:
How can I return two values from a function in Python?
(8 answers)
Closed 4 years ago.
I'm wondering if there's a way to assign the outputs of a function to variables, while only having to run the function once. For example, say I have a function like this:
def my_func(x):
return x+1, x**2, x**3
And then I want to store each output as a variable, so normally I would code this afterwards:
var1 = my_func(2)[0]
var2 = my_func(2)[1]
var3 = my_func(2)[2]
But from my understanding, each line above requires the function to run again. If I have a longer function, it would be more convenient to assign them all at the same time, only running the function once. Is there a way to do this?

Two possibilities:
assign the function's return value (which is a tuple) to a variable, and access its elements:
var = my_func(2)
var1 = var[0]
var2 = var[1]
var3 = var[2]
or use unpacking:
var1, var2, var3 = my_func(2)
Re: Which method runs faster in most cases?
Let's compare the disassemble (I hilighted with > the differences):
dis.dis('t=my_func(2);d=t[0]+t[1]+t[2]')
1 0 LOAD_NAME 0 (my_func)
2 LOAD_CONST 0 (2)
4 CALL_FUNCTION 1
> 6 STORE_NAME 1 (t)
8 LOAD_NAME 1 (t)
> 10 LOAD_CONST 1 (0)
> 12 BINARY_SUBSCR
14 LOAD_NAME 1 (t)
> 16 LOAD_CONST 2 (1)
> 18 BINARY_SUBSCR
20 BINARY_ADD
22 LOAD_NAME 1 (t)
> 24 LOAD_CONST 0 (2)
> 26 BINARY_SUBSCR
28 BINARY_ADD
30 STORE_NAME 2 (d)
32 LOAD_CONST 3 (None)
34 RETURN_VALUE
dis.dis('a,b,c=my_func(2);d=a+b+c')
1 0 LOAD_NAME 0 (my_func)
2 LOAD_CONST 0 (2)
4 CALL_FUNCTION 1
> 6 UNPACK_SEQUENCE 3
> 8 STORE_NAME 1 (a)
> 10 STORE_NAME 2 (b)
> 12 STORE_NAME 3 (c)
14 LOAD_NAME 1 (a)
16 LOAD_NAME 2 (b)
18 BINARY_ADD
20 LOAD_NAME 3 (c)
22 BINARY_ADD
24 STORE_NAME 4 (d)
26 LOAD_CONST 1 (None)
28 RETURN_VALUE
So, the first approach takes two more instructions each time a variable is used (LOAD_CONST, BINARY_SUBSCR), while the second approach takes one more instruction when the variable is created.
The difference is practically meaningless.

Yes, you can do this via sequence unpacking:
def my_func(x):
return x+1, x**2, x**3
var1, var2, var3 = my_func(2)
This feature is described in the docs: Tuples and Sequences

What's the difference between in-place assignment and assignment using the variable's name again?

In Python, while assigning a value to a variable, we can either do:
variable = variable + 20
or
variable += 20.
While I do understand that both the operations are semantically same, i.e., they achieve the same goal of increasing the previous value of variable by 20, I was wondering if there are subtle run-time performance differences between the two, or any other slight differences which might deem one better than the other.
Is there any such difference, or are they exactly the same?
If there is any difference, is it the same for other languages such as C++?
Thanks.

Perhaps this can help you understand better:
import dis
def a():
x = 0
x += 20
return x
def b():
x = 0
x = x + 20
return x
print 'In place add'
dis.dis(a)
print 'Binary add'
dis.dis(b)
We get the following outputs:
In place add
4 0 LOAD_CONST 1 (0)
3 STORE_FAST 0 (x)
5 6 LOAD_FAST 0 (x)
9 LOAD_CONST 2 (20)
12 INPLACE_ADD
13 STORE_FAST 0 (x)
6 16 LOAD_FAST 0 (x)
19 RETURN_VALUE
Binary add
9 0 LOAD_CONST 1 (0)
3 STORE_FAST 0 (x)
10 6 LOAD_FAST 0 (x)
9 LOAD_CONST 2 (20)
12 BINARY_ADD
13 STORE_FAST 0 (x)
11 16 LOAD_FAST 0 (x)
19 RETURN_VALUE
You could do a loop a thousand or so times using a timer to compare perfomance, but the main difference is that one. I suppose binary add should be faster tho.

Python bytecode compiler; removes unnecessary variables?

Given the following:
def foo():
x = a_method_returning_a_long_list()
y = a_method_which_filters_a_list(x)
return y
will Python's bytecode compiler keep x & y in memory, or is it clever enough to reduce it to the following?
def foo():
return a_method_which_filters_a_list(a_method_returning_a_long_list())

It keeps x and y in memory:
import dis
dis.dis(foo)
2 0 LOAD_GLOBAL 0 (a_method_returning_a_long_list)
3 CALL_FUNCTION 0
6 STORE_FAST 0 (x)
3 9 LOAD_GLOBAL 1 (a_method_which_filters_a_list)
12 LOAD_FAST 0 (x)
15 CALL_FUNCTION 1
18 STORE_FAST 1 (y)
4 21 LOAD_FAST 1 (y)
24 RETURN_VALUE
The whole operation is quite efficient, as it is done using the LOAD_FAST and STORE_FAST codes.
As Roadrunner-EX remarks in one of the comments, the amount of memory used by your two versions of foo is basically the same, as x and y are just references (i.e., pointers) to the results.

In [1]: import dis
In [2]: def f():
...: x = f1()
...: y = f2(x)
...: return y
...:
In [3]: dis.dis(f)
2 0 LOAD_GLOBAL 0 (f1)
3 CALL_FUNCTION 0
6 STORE_FAST 0 (x)
3 9 LOAD_GLOBAL 1 (f2)
12 LOAD_FAST 0 (x)
15 CALL_FUNCTION 1
18 STORE_FAST 1 (y)
4 21 LOAD_FAST 1 (y)
24 RETURN_VALUE
So it looks like both variables are held separately.

I'm not certain, but I would guess it would keep them in memory, for 2 reasons. First, it's probably more effort than its worth to do that. There wouldn't be a huge performance change either way. And second, the variables x and y are probably themselves taking up memory (in the form of pointers/references), which the compiler would not touch, due to the explicit nature of the assignment.