Given a Python interpreter (CPython, Jython, etc), is the bytecode generated deterministic?
That is, if I compile 2 different scripts that differ only in whitespace, but otherwise syntactically equivalent, would the chosen compiler generate exactly the same bytecodes?
It is not clear what you are looking for, exactly. Syntactically the same code is going to result in the same instructions being executed, certainly. But even syntactically equivalent python files can generate different .pyc cached bytecode files. Adding or removing newlines will result in different line offsets:
>>> import dis
>>> def foo():
... # in the interpreter, comments will do the same job as newlines
... baz
... # extra newlines or comments push the bytecode offsets
... return 42
...
>>> def bar():
... baz
... return 42
...
>>> dis.dis(foo)
3 0 LOAD_GLOBAL 0 (baz)
3 POP_TOP
5 4 LOAD_CONST 1 (42)
7 RETURN_VALUE
>>> dis.dis(bar)
2 0 LOAD_GLOBAL 0 (baz)
3 POP_TOP
3 4 LOAD_CONST 1 (42)
7 RETURN_VALUE
Note the different values in the left-hand column; the interpreter will still behave exactly the same, but the offsets differ.
The bytecode and offsets can be accessed separately leaving what the interpreter executes equal:
>>> foo.__code__.co_lnotab
'\x00\x02\x04\x02'
>>> bar.__code__.co_lnotab
'\x00\x01\x04\x01'
>>> foo.__code__.co_code == bar.__code__.co_code
True
Related
def f():
print("Before", locals()) # line 2
print(x); # line 3
x = 2 # line 4
print("After", locals()) # line 5
x = 1
f()
I am aware of the LEGB rule for scoping in Python.
For the above code, when I comment out line 4, everything executes normally as expected: for line 3, python does not find variable x in the local scope and therefore searches it in the global scope where it finds it and prints 1.
But when I execute the whole code as it is without commenting, it raises UnboundLocalError: local variable 'x' referenced before assignment.
I do know I can use nonlocal and global, but my question is :
How does python know there is a local variable declaration before it has encountered one?
Even if it does know there is a variable named x in the local scope (although not yet initialised), why doesn't it shows it in locals()?
I tried finding the answer in similar questions suggestions but failed. Please correct if any of my understanding is wrong.
To some extent, the answer is implementation specific, as Python only specifies the expected behavior, not how to implement it.
That said, let's look at the byte code generated for f by the usual implementation, CPython:
>>> import dis
>>> dis.dis(f)
2 0 LOAD_GLOBAL 0 (print)
2 LOAD_CONST 1 ('Before')
4 LOAD_GLOBAL 1 (locals)
6 CALL_FUNCTION 0
8 CALL_FUNCTION 2
10 POP_TOP
3 12 LOAD_GLOBAL 0 (print)
14 LOAD_FAST 0 (x)
16 CALL_FUNCTION 1
18 POP_TOP
4 20 LOAD_CONST 2 (2)
22 STORE_FAST 0 (x)
5 24 LOAD_GLOBAL 0 (print)
26 LOAD_CONST 3 ('After')
28 LOAD_GLOBAL 1 (locals)
30 CALL_FUNCTION 0
32 CALL_FUNCTION 2
34 POP_TOP
36 LOAD_CONST 0 (None)
38 RETURN_VALUE
There are several different LOAD_* op codes used to retrieve various values. LOAD_GLOBAL is used for names in the global scope; LOAD_CONST is used for local values not assigned to any name. LOAD_FAST is used for local variables. Local variables don't even exist by name, but by indices in an array. That's why they are "fast"; they are available in an array rather than a hash table. (LOAD_GLOBAL also uses integer arguments, but that's just an index into an array of names; the name itself still needs to be looked up in whatever mapping provides the global scope.)
You can even see the constants and local values associated with f:
>>> f.__code__.co_consts
(None, 'Before', 2, 'After')
>>> f.__code__.co_varnames
('x',)
LOAD_CONST 1 puts Before on the stack because f.__code__.co_consts[1] == 'Before', and LOAD_FAST 0 puts the value of x on the stack because f.__code__.co_varnames[0] == 'x'.
The key here is that the byte code is generated before f is ever executed. Python isn't simply executing each line the first time it sees it. Executing the def statement involves, among other things:
reading the source code
parsing into an abstract syntax tree (AST)
using the entire AST to generate the byte code stored in the __code__ attribute of the function object.
Part of the code generation is noting that the name x, due to the assignment somewhere in the body of the function (even if that function is logically unreachable), is a local name, and therefore must be accessed with LOAD_FAST.
At the time locals is called (and indeed before LOAD_FAST 0 is used the first time), no assignment to x (i.e., STORE_FAST 0) has yet been made, so there is no local value in slot 0 to look up.
Because you define it before the f() function calling,
let's try this one :
def f(y):
print("Before", locals()) # line 2
print(y); # line 3
y = 2 # line 4
print("After", locals()) # line 5
f(x)
x = 1
Is there any practical difference between list(iterable) and [*iterable] in versions of Python that support the latter?
list(x) is a function, [*x] is an expression. You can reassign list, and make it do something else (but you shouldn't).
Talking about cPython, b = list(a) translates to this sequence of bytecodes:
LOAD_NAME 1 (list)
LOAD_NAME 0 (a)
CALL_FUNCTION 1
STORE_NAME 2 (b)
Instead, c = [*a] becomes:
LOAD_NAME 0 (a)
BUILD_LIST_UNPACK 1
STORE_NAME 3 (c)
so you can argue that [*a] might be slightly more efficient, but marginally so.
You can use the standard library module dis to investigate the byte code generated by a function. In this case:
import dis
def call_list(x):
return list(x)
def unpacking(x):
return [*x]
dis.dis(call_list)
# 2 0 LOAD_GLOBAL 0 (list)
# 2 LOAD_FAST 0 (x)
# 4 CALL_FUNCTION 1
# 6 RETURN_VALUE
dis.dis(unpacking)
# 2 0 LOAD_FAST 0 (x)
# 2 BUILD_LIST_UNPACK 1
# 4 RETURN_VALUE
So there is a difference and it is not only the loading of the globally defined name list, which does not need to happen with the unpacking. So it boils down to how the built-in list function is defined and what exactly BUILD_LIST_UNPACK does.
Note that both are actually a lot less code than writing a standard list comprehension for this:
def list_comp(x):
return [a for a in x]
dis.dis(list_comp)
# 2 0 LOAD_CONST 1 (<code object <listcomp> at 0x7f65356198a0, file "<ipython-input-46-dd71fb182ec7>", line 2>)
# 2 LOAD_CONST 2 ('list_comp.<locals>.<listcomp>')
# 4 MAKE_FUNCTION 0
# 6 LOAD_FAST 0 (x)
# 8 GET_ITER
# 10 CALL_FUNCTION 1
# 12 RETURN_VALUE
Since [*iterable] is unpacking, it accepts assignment-like syntax, unlike list(iterable):
>>> [*[]] = []
>>> list([]) = []
File "<stdin>", line 1
SyntaxError: can't assign to function call
You can read more about this here (not useful though).
You can also use list(sequence=iterable), i.e. with a key-word argument:
>>> list(sequence=[])
[]
Again not useful.
There's always going to be some differences between two constructs that do the same thing. Thing is, I wouldn't say the differences in this case are actually practical. Both are expressions that take the iterable, iterate through it and then create a list out of it.
The contract is the same: input is an iterable output is a list populated by the iterables elements.
Yes, list can be rebound to a different name; list(it) is a function call while [*it] is a list display; [*it] is faster with smaller iterables but generally performs the same with larger ones. Heck, one could even throw in the fact that [*it] is three less keystrokes.
Are these practical though? Would I think of them when trying to get a list out of an iterable? Well, maybe the keystrokes in order to stay under 79 characters and get the linter to shut it up.
Apparently there’s a performance difference in CPython, where [*a] overallocates and list() doesn’t: What causes [*a] to overallocate?
From some of the answers on Stackoverflow, I came to know that from -5 to 256 same memory location is referenced thus we get true for:
>>> a = 256
>>> a is 256
True
Now comes the twist (see this line before marking duplicate):
>>> a = 257
>>> a is 257
False
This is completely understood, but now if I do:
>>> a = 257; a is 257
True
>>> a = 12345; a is 12345
True
Why?
What you're seeing is an optimization in the compiler in CPython (which compiles your source code into the bytecode that the interpreter runs). Whenever the same immutable constant value is used in several different places within the a chunk of code that is being compiled in one step, the compiler will try to use a reference to same object for each place.
So if you do multiple assignments on the same line in an interactive session, you'll get two references to the same object, but you won't if you use two separate lines:
>>> x = 257; y = 257 # multiple statements on the same line are compiled in one step
>>> print(x is y) # prints True
>>> x = 257
>>> y = 257
>>> print(x is y) # prints False this time, since the assignments were compiled separately
Another place this optimization comes up is in the body of a function. The whole function body will be compiled together, so any constants used anywhere in the function can be combined, even if they're on separate lines:
def foo():
x = 257
y = 257
return x is y # this will always return True
While it's interesting to investigate optimizations like this one, you should never rely upon this behavior in your normal code. Different Python interpreters, and even different versions of CPython may do these optimizations differently or not at all. If your code depends on a specific optimization, it may be completely broken for somebody else who tries to run it on their own system.
As an example, the two assignments on the same line I show in my first code block above doesn't result in two references to the same object when I do it in the interactive shell inside Spyder (my preferred IDE). I have no idea why that specific situation doesn't work the same way it does in a conventional interactive shell, but the different behavior is my fault, since my code relies upon implementation-specific behavior.
Generally speaking, numbers outside the range -5 to 256 will not necessarily have the optimization applied to numbers within that range. However, Python is free to apply other optimizations as appropriate. In your cause, you're seeing that the same literal value used multiple times on one line is stored in a single memory location no matter how many times it's used on that line. Here are some other examples of this behavior:
>>> s = 'a'; s is 'a'
True
>>> s = 'asdfghjklzxcvbnmsdhasjkdhskdja'; s is 'asdfghjklzxcvbnmsdhasjkdhskdja'
True
>>> x = 3.14159; x is 3.14159
True
>>> t = 'a' + 'b'; t is 'a' + 'b'
True
>>>
From python2 docs:
The operators is and is not test for object identity: x is y is true
if and only if x and y are the same object. x is not y yields the
inverse truth value. [6]
From python3 docs:
The operators is and is not test for object identity: x is y is true
if and only if x and y are the same object. Object identity is
determined using the id() function. x is not y yields the inverse
truth value. [4]
So basically the key to understand those tests you've run on the repl console is by using
accordingly the id() function, here's an example that will show you what's going on behind the curtains:
>>> a=256
>>> id(a);id(256);a is 256
2012996640
2012996640
True
>>> a=257
>>> id(a);id(257);a is 257
36163472
36162032
False
>>> a=257;id(a);id(257);a is 257
36162496
36162496
True
>>> a=12345;id(a);id(12345);a is 12345
36162240
36162240
True
That said, usually a good way to understand what's going on behind the curtains with these type of snippets is by using either dis.dis or dis.disco, let's take a look for instance what this snippet would look like:
import dis
import textwrap
dis.disco(compile(textwrap.dedent("""\
a=256
a is 256
a=257
a is 257
a=257;a is 257
a=12345;a is 12345\
"""), '', 'exec'))
the output would be:
1 0 LOAD_CONST 0 (256)
2 STORE_NAME 0 (a)
2 4 LOAD_NAME 0 (a)
6 LOAD_CONST 0 (256)
8 COMPARE_OP 8 (is)
10 POP_TOP
3 12 LOAD_CONST 1 (257)
14 STORE_NAME 0 (a)
4 16 LOAD_NAME 0 (a)
18 LOAD_CONST 1 (257)
20 COMPARE_OP 8 (is)
22 POP_TOP
5 24 LOAD_CONST 1 (257)
26 STORE_NAME 0 (a)
28 LOAD_NAME 0 (a)
30 LOAD_CONST 1 (257)
32 COMPARE_OP 8 (is)
34 POP_TOP
6 36 LOAD_CONST 2 (12345)
38 STORE_NAME 0 (a)
40 LOAD_NAME 0 (a)
42 LOAD_CONST 2 (12345)
44 COMPARE_OP 8 (is)
46 POP_TOP
48 LOAD_CONST 3 (None)
50 RETURN_VALUE
As we can see in this case the asm output doesn't tell us very much, we can see than lines 3-4 are basically the "same" instructions than line 5. So my recommendation would be once again to use id() smartly so you'll know what's is will compare. In case you want to know exactly the type of optimizations cpython is doing I'm afraid you'd need to dig out in its source code
After discussion and testing in various versions, the final conclusions can be drawn.
Python will interpret and compile instructions in blocks. Depending on the syntax used, Python version, Operating System, distribution, different results may be achieved depending on what instructions Python takes in one block.
The general rules are:
(from official documentation)
The current implementation keeps an array of integer objects for all
integers between -5 and 256
Therefore:
a = 256
id(a)
Out[2]: 1997190544
id(256)
Out[3]: 1997190544 # int actually stored once within Python
a = 257
id(a)
Out[5]: 2365489141456
id(257)
Out[6]: 2365489140880 #literal, temporary. as you see the ids differ
id(257)
Out[7]: 2365489142192 # literal, temporary. as you see it gets a new id everytime
# since it is not pre-stored
The part below returns False in Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 17 2017, 23:26:12) [MSC v.1900 64 bit (AMD64)]
a = 257; a is 257
Out[8]: False
But
a=257; print(a is 257) ; a=258; print(a is 257)
>>>True
>>>False
As it is evident, whatever Python takes in "one block" is non deterministic and can be swayed depending on how it is written, single line or not, as well as the version, operating system and distribution used.
I was wondering, what code runs faster? For example, we have variable x:
if x!=0 : return
or
if x: return
I tried to check with timeit, and here are results:
>>> def a():
... if 0 == 0: return
...
>>> def b():
... if 0: return
...>>> timeit(a)
0.18059834650234943
>>> timeit(b)
0.13115053638194007
>>>
I can't quite understand it.
This is too hard to show in a comment: there's more (or less ;-) ) going on here than any of the comments so far noted. With a() and b() defined as you showed, let's go on:
>>> from dis import dis
>>> dis(b)
2 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
What happens is that when the CPython compiler sees if 0: or if 1:, it evaluates them at compile time, and doesn't generate any code to do the testing at run time. So the code for b() just loads None and returns it.
But the code generated for a() is much more involved:
>>> dis(a)
2 0 LOAD_CONST 1 (0)
3 LOAD_CONST 1 (0)
6 COMPARE_OP 2 (==)
9 POP_JUMP_IF_FALSE 16
12 LOAD_CONST 0 (None)
15 RETURN_VALUE
>> 16 LOAD_CONST 0 (None)
19 RETURN_VALUE
Nothing is evaluated at compile time in this case - it's all done at run time. That's why a() is much slower.
Beyond that, I endorse #Charles Duffy's comment: worrying about micro-optimization is usually counterproductive in Python. But, if you must ;-) , learn how to use dis.dis so you're not fooled by gross differences in generated code, as happened in this specific case.
When testing for membership, we can use:
x not in y
Or alternatively:
not x in y
There can be many possible contexts for this expression depending on x and y. It could be for a substring check, list membership, dict key existence, for example.
Are the two forms always equivalent?
Is there a preferred syntax?
They always give the same result.
In fact, not 'ham' in 'spam and eggs' appears to be special cased to perform a single "not in" operation, rather than an "in" operation and then negating the result:
>>> import dis
>>> def notin():
'ham' not in 'spam and eggs'
>>> dis.dis(notin)
2 0 LOAD_CONST 1 ('ham')
3 LOAD_CONST 2 ('spam and eggs')
6 COMPARE_OP 7 (not in)
9 POP_TOP
10 LOAD_CONST 0 (None)
13 RETURN_VALUE
>>> def not_in():
not 'ham' in 'spam and eggs'
>>> dis.dis(not_in)
2 0 LOAD_CONST 1 ('ham')
3 LOAD_CONST 2 ('spam and eggs')
6 COMPARE_OP 7 (not in)
9 POP_TOP
10 LOAD_CONST 0 (None)
13 RETURN_VALUE
>>> def not__in():
not ('ham' in 'spam and eggs')
>>> dis.dis(not__in)
2 0 LOAD_CONST 1 ('ham')
3 LOAD_CONST 2 ('spam and eggs')
6 COMPARE_OP 7 (not in)
9 POP_TOP
10 LOAD_CONST 0 (None)
13 RETURN_VALUE
>>> def noteq():
not 'ham' == 'spam and eggs'
>>> dis.dis(noteq)
2 0 LOAD_CONST 1 ('ham')
3 LOAD_CONST 2 ('spam and eggs')
6 COMPARE_OP 2 (==)
9 UNARY_NOT
10 POP_TOP
11 LOAD_CONST 0 (None)
14 RETURN_VALUE
I had thought at first that they always gave the same result, but that not on its own was simply a low precedence logical negation operator, which could be applied to a in b just as easily as any other boolean expression, whereas not in was a separate operator for convenience and clarity.
The disassembly above was revealing! It seems that while not obviously is a logical negation operator, the form not a in b is special cased so that it's not actually using the general operator. This makes not a in b literally the same expression as a not in b, rather than merely an expression that results in the same value.
No, there is no difference.
The operator not in is defined to have the inverse true value of in.
—Python documentation
I would assume not in is preferred because it is more obvious and they added a special case for it.
They are identical in meaning, but the pycodestyle Python style guide checker (formerly called pep8) prefers the not in operator in rule E713:
E713: test for membership should be not in
See also "Python if x is not None or if not x is None?" for a very similar choice of style.
Others have already made it very clear that the two statements are, down to a quite low level, equivalent.
However, I don't think that anyone yet has stressed enough that since this leaves the choice up to you, you should
choose the form that makes your code as readable as possible.
And not necessarily as readable as possible to anyone, even if that's of course a nice thing to aim for. No, make sure the code is as readable as possible to you, since you are the one who is the most likely to come back to this code later and try to read it.
In Python, there is no difference. And there is no preference.
Syntactically they're the same statement. I would be quick to state that 'ham' not in 'spam and eggs' conveys clearer intent, but I've seen code and scenarios in which not 'ham' in 'spam and eggs' conveys a clearer meaning than the other.