I am trying to find the line number of the start and the end of a context. In Python 2.7 I am able to successfully do so as follows:
1 from contextlib import contextmanager
2 import sys
3
4 #contextmanager
5 def print_start_end_ctx():
6 frame = sys._getframe(2)
7 start_line = frame.f_lineno
8 yield
9 end_line = frame.f_lineno
10 print("start_line={}\nend_line={}".format(start_line, end_line))
11
12 with print_start_end_ctx():
13 100
14 (200,
15 300)
Output in Python 2.7:
start_line=12
end_line=15
However, extracting line numbers from frame object fails in Python 3.7:
start_line=12
end_line=14
Related
I am aware of how to find the location of a python function in its source code with the inspect module via
import inspect
inspect.getsourcefile(random_function)
However, while a python function is running, or after it has run, how would one find all of the pieces of the source code it utilized/referenced during its individual run?
For ex., if I ran random_function(arg1=1, arg2=2) vs. random_function(arg1=1, arg5=3.5), I would like to know which different parts of the module got used each time.
Is there anything like the example here?
You can do this by getting the information of bytecode using dis module. For instance:
import dis
import numpy as np
def main():
array = np.zeros(shape = 5)
if __name__ == '__main__':
print(dis.dis(main))
result:
5 0 LOAD_GLOBAL 0 (np)
2 LOAD_ATTR 1 (zeros)
4 LOAD_CONST 1 (5)
6 LOAD_CONST 2 (('shape',))
8 CALL_FUNCTION_KW 1
10 STORE_FAST 0 (array)
12 LOAD_CONST 0 (None)
14 RETURN_VALUE
I was wondering how to modify byte code, then recompile that code so I can use it in python as a function? I've been trying:
a = """
def fact():
a = 8
a = 0
"""
c = compile(a, '<string>', 'exec')
w = c.co_consts[0].co_code
dis(w)
which decompiles to:
0 LOAD_CONST 1 (1)
3 STORE_FAST 1 (1)
6 LOAD_CONST 2 (2)
9 STORE_FAST 1 (1)
12 LOAD_CONST 0 (0)
15 RETURN_VALUE
supposing I want to get rid of lines 0 and 3, I call:
x = c.co_consts[0].co_code[6:16]
dis(x)
which results in :
0 LOAD_CONST 2 (2)
3 STORE_FAST 1 (1)
6 LOAD_CONST 0 (0)
9 RETURN_VALUE
my problem is what to do with x, if I try exec x I get an 'expected string without nullbytes and I get the same for exec w,
trying to compile x results in: compile() expected string without null bytes.
I'm not sure what the best way to proceed, except maybe I need to create some kind of code-object, but I'm not sure how, but I'm assuming it must be
possible aka byteplay, python assemblers et al
I'm using python 2.7.10, but I'd like it to be future compatible (Eg python 3) if it's possible.
Update: For sundry reasons I have started writing a Cross-Python-version assembler. See https://github.com/rocky/python-xasm. It is still in very early beta. See also bytecode.
As far as I know there is no other currently-maintained Python assembler. PEAK's Bytecode Disassembler was developed for Python 2.6, and later modified to support early Python 2.7.
It is pretty cool from the documentation. But it relies on other PEAK libraries which might be problematic.
I'll go through the whole example to give you a feel for what you'd have to do. It is not pretty, but then you should expect that.
Basically after modifying the bytecode, you need to create a new types.CodeType object. You need a new one because many of the objects in the code type, for good reason, you can't change. For example the interpreter may have some of these object values cached.
After creating code, you can use this in functions that use a code type which can be used in exec or eval.
Or you can write this to a bytecode file. Alas the code format has changed between Python versions 1.3, 1,5, 2.0, 3.0, 3.8, and 3.10. And by the way so has the optimization and bytecodes. In fact, in Python 3.6 they will be word codes not bytecodes.
So here is what you'd have to do for your example:
a = """
def fact():
a = 8
a = 0
return a
"""
c = compile(a, '<string>', 'exec')
fn_code = c.co_consts[0] # Pick up the function code from the main code
from dis import dis
dis(fn_code)
print("=" * 30)
x = fn_code.co_code[6:16] # modify bytecode
import types
opt_fn_code = types.CodeType(fn_code.co_argcount,
# c.co_kwonlyargcount, Add this in Python3
# c.co_posonlyargcount, Add this in Python 3.8+
fn_code.co_nlocals,
fn_code.co_stacksize,
fn_code.co_flags,
x, # fn_code.co_code: this you changed
fn_code.co_consts,
fn_code.co_names,
fn_code.co_varnames,
fn_code.co_filename,
fn_code.co_name,
fn_code.co_firstlineno,
fn_code.co_lnotab, # In general, You should adjust this
fn_code.co_freevars,
fn_code.co_cellvars)
dis(opt_fn_code)
print("=" * 30)
print("Result is", eval(opt_fn_code))
# Now let's change the value of what's returned
co_consts = list(opt_fn_code.co_consts)
co_consts[-1] = 10
opt_fn_code = types.CodeType(fn_code.co_argcount,
# c.co_kwonlyargcount, Add this in Python3
# c.co_posonlyargcount, Add this in Python 3.8+
fn_code.co_nlocals,
fn_code.co_stacksize,
fn_code.co_flags,
x, # fn_code.co_code: this you changed
tuple(co_consts), # this is now changed too
fn_code.co_names,
fn_code.co_varnames,
fn_code.co_filename,
fn_code.co_name,
fn_code.co_firstlineno,
fn_code.co_lnotab, # In general, You should adjust this
fn_code.co_freevars,
fn_code.co_cellvars)
dis(opt_fn_code)
print("=" * 30)
print("Result is now", eval(opt_fn_code))
When I ran this here is what I got:
3 0 LOAD_CONST 1 (8)
3 STORE_FAST 0 (a)
4 6 LOAD_CONST 2 (0)
9 STORE_FAST 0 (a)
5 12 LOAD_FAST 0 (a)
15 RETURN_VALUE
==============================
3 0 LOAD_CONST 2 (0)
3 STORE_FAST 0 (a)
4 6 LOAD_FAST 0 (a)
9 RETURN_VALUE
==============================
('Result is', 0)
3 0 LOAD_CONST 2 (10)
3 STORE_FAST 0 (a)
4 6 LOAD_FAST 0 (a)
9 RETURN_VALUE
==============================
('Result is now', 10)
Notice that the line numbers haven't changed even though I removed in code a couple of lines. That is because I didn't update fn_code.co_lnotab.
If you want to now write a Python bytecode file from this. Here is what you'd do:
co_consts = list(c.co_consts)
co_consts[0] = opt_fn_code
c1 = types.CodeType(c.co_argcount,
# c.co_posonlyargcount, Add this in Python 3.8+
# c.co_kwonlyargcount, Add this in Python3
c.co_nlocals,
c.co_stacksize,
c.co_flags,
c.co_code,
tuple(co_consts),
c.co_names,
c.co_varnames,
c.co_filename,
c.co_name,
c.co_firstlineno,
c.co_lnotab, # In general, You should adjust this
c.co_freevars,
c.co_cellvars)
from struct import pack
with open('/tmp/testing.pyc', 'w') as fp:
fp.write(pack('Hcc', 62211, '\r', '\n')) # Python 2.7 magic number
import time
fp.write(pack('I', int(time.time())))
# In Python 3.7+ you need to PEP 552 bits
# In Python 3 you need to write out the size mod 2**32 here
import marshal
fp.write(marshal.dumps(c1))
To simplify writing the boilerplate bytecode above, I've added a routine to xasm called write_pycfile().
Now to check the results:
$ uncompyle6 /tmp/testing.pyc
# uncompyle6 version 2.9.2
# Python bytecode 2.7 (62211)
# Disassembled from: Python 2.7.12 (default, Jul 26 2016, 22:53:31)
# [GCC 5.4.0 20160609]
# Embedded file name: <string>
# Compiled at: 2016-10-18 05:52:13
def fact():
a = 0
# okay decompiling /tmp/testing.pyc
$ pydisasm /tmp/testing.pyc
# pydisasm version 3.1.0
# Python bytecode 2.7 (62211) disassembled from Python 2.7
# Timestamp in code: 2016-10-18 05:52:13
# Method Name: <module>
# Filename: <string>
# Argument count: 0
# Number of locals: 0
# Stack size: 1
# Flags: 0x00000040 (NOFREE)
# Constants:
# 0: <code object fact at 0x7f815843e4b0, file "<string>", line 2>
# 1: None
# Names:
# 0: fact
2 0 LOAD_CONST 0 (<code object fact at 0x7f815843e4b0, file "<string>", line 2>)
3 MAKE_FUNCTION 0
6 STORE_NAME 0 (fact)
9 LOAD_CONST 1 (None)
12 RETURN_VALUE
# Method Name: fact
# Filename: <string>
# Argument count: 0
# Number of locals: 1
# Stack size: 1
# Flags: 0x00000043 (NOFREE | NEWLOCALS | OPTIMIZED)
# Constants:
# 0: None
# 1: 8
# 2: 10
# Local variables:
# 0: a
3 0 LOAD_CONST 2 (10)
3 STORE_FAST 0 (a)
4 6 LOAD_CONST 0 (None)
9 RETURN_VALUE
$
An alternate approach for optimization is to optimize at the Abstract Syntax Tree level (AST). The compile, eval and exec functions can start from an AST, or you can dump the AST. You could also write this back out as Python source using the Python module astor
Note however that some kinds of optimization like tail-recursion elimination might leave bytecode in a form that it can't be transformed in a truly faithful way to source code. See my pycon2018 Columbia Lightning Talk for a video I made which eliminates tail recursion in bytecode to get an idea of what I'm talking about here.
If you want to be able to debug and single step bytecode instructions. See my bytecode interpreter and its bytecode debugger.
I've built a tree using ete2 package. Now I'm trying to write a piece of code that takes the data from the tree and a csv file and does some data analysis through the function fre.
Here is an example of the csv file I've used:
PID Code Value
1 A1... 6
1 A2... 5
2 A.... 4
2 D.... 1
2 A1... 2
3 D.... 5
3 D1... 3
3 D2... 5
Here is a simplified version of the code
from ete2 import Tree
import pandas as pd
t= Tree("((A1...,A2...)A...., (D1..., D2...)D....).....;", format=1)
data= pd.read_csv('/data_2.csv', names=['PID','Code', 'Value'])
code_count = data.groupby('Code').sum()
total_patients= len(list (set(data['PID'])))
del code_count['PID']
############
def fre(code1,code2):
code1_ancestors=[]
code2_ancestors=[]
for i in t.search_nodes(name=code1)[0].get_ancestors():
code1_ancestors.append(i.name)
for i in t.search_nodes(name=code2)[0].get_ancestors():
code2_ancestors.append(i.name)
common_ancestors = []
for i in code1_ancestors:
for j in code2_ancestors:
if i==j:
common_ancestors.append(i)
print common_ancestors
####
for i in patients_list:
a= list (data.Code[data.PID==patients_list[i-1]])
#print a
for j in patients_list:
b= list (data.Code[data.PID==patients_list[j-1]])
for k in a:
for t in b:
fre (k,t)
However, an error is raising which is:
AttributeError Traceback (most recent call last)
<ipython-input-12-f9b47fcec010> in <module>()
38 for k in a:
39 for t in b:
---> 40 fre (k,t)
<ipython-input-12-f9b47fcec010> in fre(code1, code2)
12 code1_ancestors=[]
13 code2_ancestors=[]
---> 14 for i in t.search_nodes(name=code1)[0].get_ancestors():
15 code1_ancestors.append(i.name)
16 for i in t.search_nodes(name=code2)[0].get_ancestors():
AttributeError: 'str' object has no attribute 'search_nodes'
I've tried to manually pass all possible values to the function and it works! However, When I'm using the last section of the code, it raises the error.
You're changing your global variable 't' with your for loop.
If you print out its value before each call to your function, you will find that you have assigned it to a string at some point.
I have a python script Test.py that runs an R script Test.R below:
import subprocess
import pandas
import pyper
#Run Simple R Code and Print Output
proc = subprocess.Popen(['Path/To/Rscript.exe',
'Path/To/Test.R'],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()
print stdout
print stderr
The R script is below:
library("methods")
x <- c(1,2,3,4,5,6,7,8,9,10)
y <- c(2,4,6,8,10,12,14,16,18,20)
data <- data.frame(x,y)
How can I pass the R data frame (or any R object for that matter) to Python? I've had great difficulty getting Rpy2 to work on windows, and I've seen this link to use PypeR but it's using a lot of in-line R code in the Python code and I'd really like to keep the code on separate files (or is this practice considered acceptable?) Thanks.
I've experienced issues getting Rpy2 to work on a mac too and use the same workaround calling R directly from python via subprocess; also agree that keeping files separate helps manage complexity.
First export your data as a .csv from R (again script called through subprocess):
write.table(data, file = 'test.csv')
After that, you can import as a python pandas data frame (Pandas):
import pandas as pd
dat = pd.read_csv('test.csv')
dat
row x y
1 1 2
2 2 4
3 3 6
4 4 8
5 5 10
6 6 12
7 7 14
8 8 16
9 9 18
10 10 20
I am very new to Networkx. I am trying to import layout position generated by random_layout() function. I dont know how to do proceed with it.
Code to generate layout position:
G = nx.random_geometric_graph(10, 0.5)
pos = nx.random_layout(G)
nx.set_node_attributes(G, 'pos', pos)
f = open("graphLayout.txt", 'wb')
f.write("%s" % pos)
f.close()
print pos
filename = "ipRandomGrid.txt"
fh = open(filename, 'wb')
nx.write_adjlist(G, fh)
#nx.write_graphml(G, sys.stdout)
nx.draw(G)
plt.show()
fh.close()
File: ipRandomGrid.txt
# GMT Tue Dec 06 04:28:27 2011
# Random Geometric Graph
0 1 3 4 6 8 9
1 3 4 6 8 9
2 4 7
3 8 6
4 5 6 7 8
5 8 9 6 7
6 7 8 9
7 9
8 9
9
I am storing both node adjlist and the layout in files. Now I want to generate the graph with the same layout and adjlist from other file. I tried to generate it with the below code. Can anyone help me to figure out what is wrong over here.
Code while importing:
Pseudo Code
G = nx.Graph()
G = nx.read_adjlist("ipRandomGrid.txt")
# load POS value from file
nx.draw(G)
nx.draw_networkx_nodes(G, pos, nodelist=['1','2'], node_color='b')
plt.show()
The nx.random_layout function returns a dictionary mapping nodes to positions. As pos is a Python object, you don't want to just store the printed string version of it to a file, as you did in f.write("%s" % pos). This gives you a file containing your dictionary, but reading it back in isn't as easy.
Instead, serialize pos using one of the standard library modules designed for that task, for example, json or pickle. Their interfaces are basically the same, so I'll just show how to do it with pickle. Storing is:
with open("graphLayout.txt", 'wb') as f:
pickle.dump(pos, f)
Reloading is:
with open("graphLayout.txt", 'rb') as f:
pos = pickle.load(f)