I've been using the dis module to observe CPython bytecode. But lately, I've noticed some inconvenient behavior of dis.dis().
Take this example for instance: I first define a function multiplier with a nested function inside of it inner:
>>> def multiplier(n):
def inner(multiplicand):
return multiplicand * n
return inner
>>>
I then use dis.dis() to disassemble it:
>>> from dis import dis
>>> dis(multiplier)
2 0 LOAD_CLOSURE 0 (n)
3 BUILD_TUPLE 1
6 LOAD_CONST 1 (<code object inner at 0x7ff6a31d84b0, file "<pyshell#12>", line 2>)
9 LOAD_CONST 2 ('multiplier.<locals>.inner')
12 MAKE_CLOSURE 0
15 STORE_FAST 1 (inner)
4 18 LOAD_FAST 1 (inner)
21 RETURN_VALUE
>>>
As you can see, it disassembled the top-level code object fine. However, it did not disassemble inner. It simply showed that it created a code object named inner and displayed the default (uninformative) __repr__() for code objects.
Is there a way I can make dis.dis() print the code objects recursively? That is, if I have nested code objects, it will print the bytecode for all of the code objects out, rather than stopping at the top-level code object. I'd mainly like this feature for things such as decorators, closures, or generator comprehensions.
It appears that the latest version of Python - 3.7 alpha 1 - has exactly the behavior I want from dis.dis():
>>> def func(a):
def ifunc(b):
return b + 10
return ifunc
>>> dis(func)
2 0 LOAD_CONST 1 (<code object ifunc at 0x7f199855ac90, file "python", line 2>)
2 LOAD_CONST 2 ('func.<locals>.ifunc')
4 MAKE_FUNCTION 0
6 STORE_FAST 1 (ifunc)
4 8 LOAD_FAST 1 (ifunc)
10 RETURN_VALUE
Disassembly of <code object ifunc at 0x7f199855ac90, file "python", line 2>:
3 0 LOAD_FAST 0 (b)
2 LOAD_CONST 1 (10)
4 BINARY_ADD
6 RETURN_VALUE
The What’s New In Python 3.7 article makes note of this:
The dis() function now is able to disassemble nested code objects (the code of comprehensions, generator expressions and nested functions, and the code used for building nested classes). (Contributed by Serhiy Storchaka in bpo-11822.)
However, besides Python 3.7 not being formally released yet, what if you don't want or cannot use Python 3.7? Are there ways to accomplish this in earlier versions of Python such as 3.5 or 2.7 using the old dis.dis()?
You could do something like this (Python 3):
import dis
def recursive_dis(code):
print(code)
dis.dis(code)
for obj in code.co_consts:
if isinstance(obj, type(code)):
print()
recursive_dis(obj)
https://repl.it/#solly_ucko/Recursive-dis
Note that you have to call it with f.__code__ instead of just f. For example:
def multiplier(n):
def inner(multiplicand):
return multiplicand * n
return inner
recursive_dis(multiplier.__code__)
First off, if you need this for anything other than interactive use, I would recommend just copying the code from the Python 3.7 sources and backporting it (hopefully that isn't difficult).
For interactive use, an idea would be to use one of the ways to access an object by its memory value to grab the code object by its memory address, which is printed in the dis output.
For example:
>>> def func(a):
... def ifunc(b):
... return b + 10
... return ifunc
>>> import dis
>>> dis.dis(func)
2 0 LOAD_CONST 1 (<code object ifunc at 0x10cabda50, file "<stdin>", line 2>)
3 LOAD_CONST 2 ('func.<locals>.ifunc')
6 MAKE_FUNCTION 0
9 STORE_FAST 1 (ifunc)
4 12 LOAD_FAST 1 (ifunc)
15 RETURN_VALUE
Here I copy-paste the memory address of the code object printed above
>>> import ctypes
>>> c = ctypes.cast(0x10cabda50, ctypes.py_object).value
>>> dis.dis(c)
3 0 LOAD_FAST 0 (b)
3 LOAD_CONST 1 (10)
6 BINARY_ADD
7 RETURN_VALUE
WARNING: the ctypes.cast line will segfault the interpreter if you pass it something that doesn't exist in memory (say, because it's been garbage collected). Some of the other solutions from the above referenced question may work better (I tried the gc one but it didn't seem to be able to find code objects).
This also means that this won't work if you pass dis a string, because the internal code objects will already be garbage collected by the time you try to access them. You need to either pass it a real Python object, or, if you have a string, compile() it first.
Related
How can one use type comments in Python to change or narrow the type of an already declared variable, in such a way as to make pycharm or other type-aware systems understand the new type.
For instance, I might have two classes:
class A:
is_b = False
...
class B(A):
is_b = True
def flummox(self):
return '?'
and another function elsewhere:
def do_something_to_A(a_in: A):
...
if a_in.is_b:
assert isinstance(a_in, B) # THIS IS THE LINE...
a_in.flummox()
As long as I have the assert statement, PyCharm will understand that I've narrowed a_in to be of class B, and not complain about .flummox(). Without it, errors/warnings such as a_in has no method flummox will appear.
The question I have is, is there a PEP 484 (or successor) way of showing that a_in (which might have originally been of type A or B or something else) is now of type B without having the assert statement. The statement b_in : B = a_in also gives type errors.
In TypeScript I could do something like this:
if a_in.is_b:
const b_in = <B><any> a_in;
b_in.flummox()
// or
if a_in.is_b:
(a_in as B).flummox()
There are two main reasons I don't want to use the assert line is (1) speed is very important to this part of code, and having an extra is_instance call for every time the line is run slows it down too much, and (2) a project code style that forbids bare assert statements.
So long as you are using Python 3.6+, you can "re-annotate" the type of a variable arbitrarily using the same syntax as you would use to "declare" the type of a variable without initializing it (PEP 526).
In the example you have provided, the following snippet has the behavior you expect:
def do_something_to_A(a_in: A):
...
if a_in.is_b:
a_in: B
a_in.flummox()
I have tested that this technique is properly detected by PyCharm 2019.2.
It is worth noting that this incurs no runtime cost since the same bytecode is generated with or without this added annotation statement. Given the following defintions,
def do_something_with_annotation(a_in: A):
if a_in.is_b:
a_in: B
a_in.flummox()
def do_something_without_annotation(a_in: A):
if a_in.is_b:
a_in.flummox()
dis produce the following bytecode:
>>> dis.dis(do_something_with_annotation)
3 0 LOAD_FAST 0 (a_in)
2 LOAD_ATTR 0 (is_b)
4 POP_JUMP_IF_FALSE 14
5 6 LOAD_FAST 0 (a_in)
8 LOAD_ATTR 1 (flummox)
10 CALL_FUNCTION 0
12 POP_TOP
>> 14 LOAD_CONST 0 (None)
16 RETURN_VALUE
>>> dis.dis(do_something_without_annotation)
3 0 LOAD_FAST 0 (a_in)
2 LOAD_ATTR 0 (is_b)
4 POP_JUMP_IF_FALSE 14
4 6 LOAD_FAST 0 (a_in)
8 LOAD_ATTR 1 (flummox)
10 CALL_FUNCTION 0
12 POP_TOP
>> 14 LOAD_CONST 0 (None)
16 RETURN_VALUE
As a side note, you could also keep the assertion statements and disable assertions in your production environment by invoking the interpreter with the -O flag. This may or may not be considered more readable by your colleagues, depending on their familiarity with type hinting in Python.
Imagine this simple function creating a modified value of a variable default, modified:
default = 0
def modify():
modified = default + 1
print(modified) # replace with OS call, I can't see the output
modify() # 1
default # 0
disassembled:
import dis
dis.dis(modify)
2 0 LOAD_GLOBAL 0 (default)
3 LOAD_CONST 1 (1)
6 BINARY_ADD
7 STORE_FAST 0 (modified)
3 10 LOAD_GLOBAL 1 (print)
13 LOAD_FAST 0 (modified)
16 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
19 POP_TOP
20 LOAD_CONST 0 (None)
23 RETURN_VALUE
I can't change the function modify(), but I know what's in it either directly (I can see the code) or indirectly (disassembly). What I need it is to get a value of the modified variable, so I though maybe there is a way how to remove specific parts (print(modified)) of the function through dis module, but I didn't find anything.
Is there any way how to remove probably everything except return_value after 16 CALL_FUNCTION and replace it with e.g. return modified? Or is there any other way how to pull a local variable out without actually executing the last line(s)?
As a possible solution I see 3 ways:
pulling disassembled codes and creating my own function (or inplace) according to them with removing the code I don't want (everything after 16 ...)
modifying the function's return value, so that it returns modified (that unfortunately calls the OS function)
manually recreating the function according to the source code
I'd like to avoid the second way, which is probably easier than the first one, but I must avoid the third way, so... is there any way how to solve my problem?
There is a 4th option: replace the print() global:
printed = []
print = lambda *args: printed.extend(args)
modify()
del print
modified = printed[0]
It is otherwise possible to produce modified bytecode, but this can easily lead to bugs that blow up the interpreter (there is zero protection from invalid bytecode), so be warned.
You can create a new function object with a new code object with updated bytecode; based on the offsets in the dis you showed, I manually created new bytecode that would return the local variable at index 0:
>>> altered_bytecode = modify.__code__.co_code[:8] + bytes(
... [dis.opmap['LOAD_FAST'], 0, # load local variable 0 onto the stack
... dis.opmap['RETURN_VALUE']])) # and return it.
>>> dis.dis(altered_bytecode)
0 LOAD_GLOBAL 0 (0)
2 LOAD_CONST 1 (1)
4 BINARY_ADD
6 STORE_FAST 0 (0)
8 LOAD_FAST 0 (0)
10 RETURN_VALUE
RETURN_VALUE returns the object at the top of the stack; all I did was inject a LOAD_FAST opcode to load what modified references onto the stack.
You'd have to create a new code object, then a new function object wrapping the code object, to make this callable:
>>> code = type(modify.__code__)
>>> function = type(modify)
>>> ocode = modify.__code__
>>> new_modify = function(
... code(ocode.co_argcount, ocode.co_kwonlyargcount, ocode.co_nlocals, ocode.co_stacksize,
... ocode.co_flags, altered_bytecode,
... ocode.co_consts, ocode.co_names, ocode.co_varnames, ocode.co_filename,
... 'new_modify', ocode.co_firstlineno, ocode.co_lnotab, ocode.co_freevars,
... ocode.co_cellvars),
... modify.__globals__, 'new_modify', modify.__defaults__, modify.__closure__)
>>> new_modify()
1
This does, obviously, require some understanding of how Python bytecode works in the first place; the dis module does contain descriptions of the various codes, and the dis.opmap dictionary lets you map back to byte values.
There are a few modules out there that try to make this easier; take a look at byteplay, the bytecode module of the pwnypack project or several others, if you want to explore this further.
I can also heartily recommend you watch the Playing with Python Bytecode presentation given by Scott Sanderson, Joe Jevnik at PyCon 2016, and play with their codetransformer module. Highly entertaining and very informative.
I'm using python 2.7 and I have a very simple script
def simple():
print("It's simple!")
x = "Come on"
Then I import this script in one project like this in order to disassemble him
import marshal
import dis
pyc_file = open('./simple.pyc', 'rb')
magic = pyc_file.read(4)
date = pyc_file.read(4)
code_object = marshal.load(pyc_file)
pyc_file.close()
dis.dis(code_object)
and get an output
1 0 LOAD_CONST 0 (<code object simple at 0x7efc5d1bfc30, file "/home/svintsov/PycharmProjects/www.artour.com/simple.py", line 1>)
3 MAKE_FUNCTION 0
6 STORE_NAME 0 (simple)
4 9 LOAD_CONST 1 ('Come on')
12 STORE_NAME 1 (x)
15 LOAD_CONST 2 (None)
18 RETURN_VALUE
But I also tried another way of disassembling in another project:
import dis
s = __import__("simple")
dis.dis(s)
which gives another output
Disassembly of simple:
2 0 LOAD_CONST 1 ("It's simple!")
3 PRINT_ITEM
4 PRINT_NEWLINE
5 LOAD_CONST 0 (None)
8 RETURN_VALUE
What's the reason these outputs are different? One of them doesn't seem to recognize string literal.
Ok I'm actually gonna answer my question myself.
As python 2.x documentation says
dis.dis([bytesource]) Disassemble the bytesource object. bytesource
can denote either a module, a class, a method, a function, or a code
object. For a module, it disassembles all functions. For a class, it
disassembles all methods. For a single code sequence, it prints one
line per bytecode instruction. If no object is provided, it
disassembles the last traceback.
So this function does different kinds of disassembly with respect to "raw" code objects and modules, despite the fact that the bytecode (.pyc file) is the same.
I was wondering how to modify byte code, then recompile that code so I can use it in python as a function? I've been trying:
a = """
def fact():
a = 8
a = 0
"""
c = compile(a, '<string>', 'exec')
w = c.co_consts[0].co_code
dis(w)
which decompiles to:
0 LOAD_CONST 1 (1)
3 STORE_FAST 1 (1)
6 LOAD_CONST 2 (2)
9 STORE_FAST 1 (1)
12 LOAD_CONST 0 (0)
15 RETURN_VALUE
supposing I want to get rid of lines 0 and 3, I call:
x = c.co_consts[0].co_code[6:16]
dis(x)
which results in :
0 LOAD_CONST 2 (2)
3 STORE_FAST 1 (1)
6 LOAD_CONST 0 (0)
9 RETURN_VALUE
my problem is what to do with x, if I try exec x I get an 'expected string without nullbytes and I get the same for exec w,
trying to compile x results in: compile() expected string without null bytes.
I'm not sure what the best way to proceed, except maybe I need to create some kind of code-object, but I'm not sure how, but I'm assuming it must be
possible aka byteplay, python assemblers et al
I'm using python 2.7.10, but I'd like it to be future compatible (Eg python 3) if it's possible.
Update: For sundry reasons I have started writing a Cross-Python-version assembler. See https://github.com/rocky/python-xasm. It is still in very early beta. See also bytecode.
As far as I know there is no other currently-maintained Python assembler. PEAK's Bytecode Disassembler was developed for Python 2.6, and later modified to support early Python 2.7.
It is pretty cool from the documentation. But it relies on other PEAK libraries which might be problematic.
I'll go through the whole example to give you a feel for what you'd have to do. It is not pretty, but then you should expect that.
Basically after modifying the bytecode, you need to create a new types.CodeType object. You need a new one because many of the objects in the code type, for good reason, you can't change. For example the interpreter may have some of these object values cached.
After creating code, you can use this in functions that use a code type which can be used in exec or eval.
Or you can write this to a bytecode file. Alas the code format has changed between Python versions 1.3, 1,5, 2.0, 3.0, 3.8, and 3.10. And by the way so has the optimization and bytecodes. In fact, in Python 3.6 they will be word codes not bytecodes.
So here is what you'd have to do for your example:
a = """
def fact():
a = 8
a = 0
return a
"""
c = compile(a, '<string>', 'exec')
fn_code = c.co_consts[0] # Pick up the function code from the main code
from dis import dis
dis(fn_code)
print("=" * 30)
x = fn_code.co_code[6:16] # modify bytecode
import types
opt_fn_code = types.CodeType(fn_code.co_argcount,
# c.co_kwonlyargcount, Add this in Python3
# c.co_posonlyargcount, Add this in Python 3.8+
fn_code.co_nlocals,
fn_code.co_stacksize,
fn_code.co_flags,
x, # fn_code.co_code: this you changed
fn_code.co_consts,
fn_code.co_names,
fn_code.co_varnames,
fn_code.co_filename,
fn_code.co_name,
fn_code.co_firstlineno,
fn_code.co_lnotab, # In general, You should adjust this
fn_code.co_freevars,
fn_code.co_cellvars)
dis(opt_fn_code)
print("=" * 30)
print("Result is", eval(opt_fn_code))
# Now let's change the value of what's returned
co_consts = list(opt_fn_code.co_consts)
co_consts[-1] = 10
opt_fn_code = types.CodeType(fn_code.co_argcount,
# c.co_kwonlyargcount, Add this in Python3
# c.co_posonlyargcount, Add this in Python 3.8+
fn_code.co_nlocals,
fn_code.co_stacksize,
fn_code.co_flags,
x, # fn_code.co_code: this you changed
tuple(co_consts), # this is now changed too
fn_code.co_names,
fn_code.co_varnames,
fn_code.co_filename,
fn_code.co_name,
fn_code.co_firstlineno,
fn_code.co_lnotab, # In general, You should adjust this
fn_code.co_freevars,
fn_code.co_cellvars)
dis(opt_fn_code)
print("=" * 30)
print("Result is now", eval(opt_fn_code))
When I ran this here is what I got:
3 0 LOAD_CONST 1 (8)
3 STORE_FAST 0 (a)
4 6 LOAD_CONST 2 (0)
9 STORE_FAST 0 (a)
5 12 LOAD_FAST 0 (a)
15 RETURN_VALUE
==============================
3 0 LOAD_CONST 2 (0)
3 STORE_FAST 0 (a)
4 6 LOAD_FAST 0 (a)
9 RETURN_VALUE
==============================
('Result is', 0)
3 0 LOAD_CONST 2 (10)
3 STORE_FAST 0 (a)
4 6 LOAD_FAST 0 (a)
9 RETURN_VALUE
==============================
('Result is now', 10)
Notice that the line numbers haven't changed even though I removed in code a couple of lines. That is because I didn't update fn_code.co_lnotab.
If you want to now write a Python bytecode file from this. Here is what you'd do:
co_consts = list(c.co_consts)
co_consts[0] = opt_fn_code
c1 = types.CodeType(c.co_argcount,
# c.co_posonlyargcount, Add this in Python 3.8+
# c.co_kwonlyargcount, Add this in Python3
c.co_nlocals,
c.co_stacksize,
c.co_flags,
c.co_code,
tuple(co_consts),
c.co_names,
c.co_varnames,
c.co_filename,
c.co_name,
c.co_firstlineno,
c.co_lnotab, # In general, You should adjust this
c.co_freevars,
c.co_cellvars)
from struct import pack
with open('/tmp/testing.pyc', 'w') as fp:
fp.write(pack('Hcc', 62211, '\r', '\n')) # Python 2.7 magic number
import time
fp.write(pack('I', int(time.time())))
# In Python 3.7+ you need to PEP 552 bits
# In Python 3 you need to write out the size mod 2**32 here
import marshal
fp.write(marshal.dumps(c1))
To simplify writing the boilerplate bytecode above, I've added a routine to xasm called write_pycfile().
Now to check the results:
$ uncompyle6 /tmp/testing.pyc
# uncompyle6 version 2.9.2
# Python bytecode 2.7 (62211)
# Disassembled from: Python 2.7.12 (default, Jul 26 2016, 22:53:31)
# [GCC 5.4.0 20160609]
# Embedded file name: <string>
# Compiled at: 2016-10-18 05:52:13
def fact():
a = 0
# okay decompiling /tmp/testing.pyc
$ pydisasm /tmp/testing.pyc
# pydisasm version 3.1.0
# Python bytecode 2.7 (62211) disassembled from Python 2.7
# Timestamp in code: 2016-10-18 05:52:13
# Method Name: <module>
# Filename: <string>
# Argument count: 0
# Number of locals: 0
# Stack size: 1
# Flags: 0x00000040 (NOFREE)
# Constants:
# 0: <code object fact at 0x7f815843e4b0, file "<string>", line 2>
# 1: None
# Names:
# 0: fact
2 0 LOAD_CONST 0 (<code object fact at 0x7f815843e4b0, file "<string>", line 2>)
3 MAKE_FUNCTION 0
6 STORE_NAME 0 (fact)
9 LOAD_CONST 1 (None)
12 RETURN_VALUE
# Method Name: fact
# Filename: <string>
# Argument count: 0
# Number of locals: 1
# Stack size: 1
# Flags: 0x00000043 (NOFREE | NEWLOCALS | OPTIMIZED)
# Constants:
# 0: None
# 1: 8
# 2: 10
# Local variables:
# 0: a
3 0 LOAD_CONST 2 (10)
3 STORE_FAST 0 (a)
4 6 LOAD_CONST 0 (None)
9 RETURN_VALUE
$
An alternate approach for optimization is to optimize at the Abstract Syntax Tree level (AST). The compile, eval and exec functions can start from an AST, or you can dump the AST. You could also write this back out as Python source using the Python module astor
Note however that some kinds of optimization like tail-recursion elimination might leave bytecode in a form that it can't be transformed in a truly faithful way to source code. See my pycon2018 Columbia Lightning Talk for a video I made which eliminates tail recursion in bytecode to get an idea of what I'm talking about here.
If you want to be able to debug and single step bytecode instructions. See my bytecode interpreter and its bytecode debugger.
If I have a class of the following format:
class TestClass:
def __init__(self):
self.value = 0
def getNewObject(self):
return TestClass()
Is there a limitation to the amount of times I can call the function? For example:
obj = TestClass()
obj.getNewObject().getNewObject()
Is there a limitation to how many times I can call getNewObject() on the return value of getNewObject()? If so what factors affect this?
I doubt it. One reason that makes me doubt it is that if we have this function:
def test(obj):
obj.getNewObject().getNewObject().getNewObject()
And we disassemble it:
import dis
dis.dis(test)
We get this:
2 0 LOAD_FAST 0 (obj)
3 LOAD_ATTR 0 (getNewObject)
6 CALL_FUNCTION 0
9 LOAD_ATTR 0 (getNewObject)
12 CALL_FUNCTION 0
15 LOAD_ATTR 0 (getNewObject)
18 CALL_FUNCTION 0
21 POP_TOP
22 LOAD_CONST 0 (None)
25 RETURN_VALUE
That's just repetitions of LOAD_ATTR followed by CALL_FUNCTION. I can't imagine that that would require much memory or other resources to manage. As such, there is probably no limit.
There is a recursion limit in Python (adjustable), but that is unrelated. Each call is made after the previous call has completed, so they're all called from the same level of the stack frame (i.e. from your user code). Now, you might hit a line-length limit or something, especially for an interactive Python shell.