Develop Reference

Why local variables go missing in Python exec while running bytecode?

I've built a function called foo to alter a function's code at bytecode level and execute it before returning to regular function execution flow.
import sys
from types import CodeType
def foo():
frame = sys._getframe(1) # get main's frame
main_code: CodeType = do_something(frame.f_code) # modify function code
# copy globals & locals
main_globals: dict = frame.f_globals.copy()
main_locals: dict = frame.f_locals.copy()
# execute altered bytecode before returning to regular code
exec(main_code, main_globals, main_locals)
return
def main():
bar: list = []
# run altered code
foo()
# return to regular code
bar.append(0)
return bar
if __name__ == '__main__':
main()
Though, there is a problem with the evaluation of the local variable during exec:
Traceback (most recent call last):
File "C:\Users\Pedro\main.py", line 31, in <module>
main()
File "C:\Users\Pedro\main.py", line 23, in main
foo()
File "C:\Users\Pedro\main.py", line 15, in foo
exec(main_code, main_globals, main_locals)
File "C:\Users\Pedro\main.py", line 26, in main
bar.append(0)
UnboundLocalError: local variable 'bar' referenced before assignment
If I print main_locals before the call to exec it shows exactly the same contents as if it was done before calling foo. I wonder if it has to do with any of the frame.f_code.co_* arguments passed to the CodeType constructor. They are pretty much the same, except for the actual bytecode frame.f_code.co_code, to which I made a few operations.
I need help to understand why the evaluation of the code under these globals and locals fail to reference main's local variables.
Note: I'm pretty sure that the changes made to main's bytecode prevent the process from going into unwanted recursion.
Edit: As asked in the comments, the basic behaviour of do_something can be resumed to remove all of main's code before call to foo. Some additional steps would involve applying changes to local variables i.e. bar.
import copy
import dis
## dump opcodes into global scope
globals().update(dis.opmap)
NULL = 0
def do_something(f_code) -> CodeType:
bytecode = f_code.co_code
f_consts = copy.deepcopy(f_code.co_consts)
for i in range(0, len(bytecode), 2):
cmd, arg = bytecode[i], bytecode[i+1]
# watch for the first occurence of calling 'foo'
if cmd == LOAD_GLOBAL and f_code.co_names[arg] == 'foo':
break # use 'i' variable later
else:
raise NameError('foo is not defined.')
f_bytelist = list(bytecode)
f_bytelist[i:i+4] = [
NOP, NULL, ## LOAD
LOAD_CONST, len(f_consts) ## CALL
# Constant 'None' will be added to 'f_consts'
]
f_bytelist[-2:] = [NOP, NULL] # 'main' function RETURN
# This piece of code removes all code before
# calling 'foo' (except for JUMP_ABSOLUTE) so
# it can be usend inside while loops.
null_code = [True] * i
j = i + 2
while j < len(f_bytelist):
if j >= i:
cmd, arg = f_bytelist[j], f_bytelist[j+1]
if cmd == JUMP_ABSOLUTE and arg < i and null_code[arg]:
j = arg
else:
j += 2
else:
null_code[j] = False
j += 2
else:
for j in range(0, i, 2):
if null_code[j]:
f_bytelist[j:j+2] = [NOP, NULL] # skip instruction
else:
continue
f_bytecode = bytes(f_bytelist)
f_consts = f_consts + (None,) ## Add constant to return
return CodeType(
f_code.co_argcount,
f_code.co_kwonlyargcount,
f_code.co_posonlyargcount, # Remove this if Python < 3.8
f_code.co_nlocals,
f_code.co_stacksize,
f_code.co_flags,
f_bytecode,
f_consts,
f_code.co_names,
f_code.co_varnames,
f_code.co_filename,
f_code.co_name,
f_code.co_firstlineno,
f_code.co_lnotab,
f_code.co_freevars,
f_code.co_cellvars
)

Creating functions dynamically in Python -- f(2) cannot resolve (f1)

I am trying to dynamically create 2 functions defined within a string. Code:
def main():
fns = '''
def plus_one(x):
return x + 1
def plus_two(x):
return plus_one(x) + 1
'''
exec(fns)
result = eval('plus_two(11)')
print(result)
if __name__ == '__main__':
main()
Saving this code to a file called dyn_code.py and running it gives me the following error:
python dyn_code.py
Traceback (most recent call last):
File "dyn_code.py", line 19, in <module>
main()
File "dyn_code.py", line 14, in main
result = eval('plus_two(11)')
File "<string>", line 1, in <module>
File "<string>", line 7, in plus_two
NameError: name 'plus_one' is not defined
Problem here is that plus_one cannot be resolved inside plus_two.
plus_one on its own is fine here and can be called with the correct result.
Can anybody please give me an idea on how to inject code like this into the local namespace? Specifically, I want to create 2 functions, with one referring to the other.
I have intentionally used the most open form of both exec and eval, I do know how to restrict them, etc. I have also verified that after the call to exec both functions are present in the local namespace.
What makes this more frustrating is that the code works fine in an interpreter session! That is, after injecting these 2 functions into the interpreter namespace via exec, plus_two runs without any issues.
Ideally, I would like to avoid a function-in-function scenario i.e.
def plus_two(x):
def plus_one(x):
return x + 1
return plus_one(x) + 1
This technique actually works but I want 2 explicitly named and standalone functions.

indentations of your function in fns matters! and you have to pass globals() optional argument for mapping!
def main():
fns = '''def plus_one(x):
return x + 1
def plus_two(x):
return plus_one(x) + 1
'''
exec(fns,globals())
result = eval('plus_two(11)')
print(result)
if __name__ == '__main__':
main()
Output:
13
Hope it helps!

You need to add the globals() dictionary in your call to exec(). You can also omit the eval call for plus_two, like so:
def main():
exec('def plus_one(x):\n return x + 1\n\ndef plus_two(x): return plus_one(x) + 1', globals())
print(plus_two(11))
if __name__ == '__main__':
main()

How can I split a long function into separate steps while maintaining the relationship between said steps?

I have a very long function func which takes a browser handle and performs a bunch of requests and reads a bunch of responses in a specific order:
def func(browser):
# make sure we are logged in otherwise log in
# make request to /search and check that the page has loaded
# fill form in /search and submit it
# read table of response and return the result as list of objects
Each operation require a large amount of code due to the complexity of the DOM and they tend to grow really fast.
What would be the best way to refactor this function into smaller components so that the following properties still hold:
the execution flow of the operations and/or their preconditions is guaranteed just like in the current version
the preconditions are not checked with asserts against the state, as this is a very costly operation
func can be called multiple times on the browser
?

Just wrap the three helper methods in a class, and track which methods are allowed to run in an instance.
class Helper(object):
def __init__(self):
self.a = True
self.b = False
self.c = False
def funcA(self):
if not self.A:
raise Error("Cannot run funcA now")
# do stuff here
self.a = False
self.b = True
return whatever
def funcB(self):
if not self.B:
raise Error("Cannot run funcB now")
# do stuff here
self.b = False
self.c = True
return whatever
def funcC(self):
if not self.C:
raise Error("Cannot run funcC now")
# do stuff here
self.c = False
self.a = True
return whatever
def func(...):
h = Helper()
h.funcA()
h.funcB()
h.funcC()
# etc
The only way to call a method is if its flag is true, and each method clears its own flag and sets the next method's flag before exiting. As long as you don't touch h.a et al. directly, this ensures that each method can only be called in the proper order.
Alternately, you can use a single flag that is a reference to the function currently allowed to run.
class Helper(object):
def __init__(self):
self.allowed = self.funcA
def funcA(self):
if self.allowed is not self.funcA:
raise Error("Cannot run funcA now")
# do stuff
self.allowed = self.funcB
return whatever
# etc

Here's the solution I came up with. I used a decorator (closely related to the one in this blog post) which only allows for a function to be called once.
def call_only_once(func):
def new_func(*args, **kwargs):
if not new_func._called:
try:
return func(*args, **kwargs)
finally:
new_func._called = True
else:
raise Exception("Already called this once.")
new_func._called = False
return new_func
#call_only_once
def stateA():
print 'Calling stateA only this time'
#call_only_once
def stateB():
print 'Calling stateB only this time'
#call_only_once
def stateC():
print 'Calling stateC only this time'
def state():
stateA()
stateB()
stateC()
if __name__ == "__main__":
state()
You'll see that if you re-call any of the functions, the function will throw an Exception stating that the functions have already been called.
The problem with this is that if you ever need to call state() again, you're hosed. Unless you implement these functions as private functions, I don't think you can do exactly what you want due to the nature of Python's scoping rules.
Edit
You can also remove the else in the decorator and your function will always return None.

Here a snippet I used once for my state machine
class StateMachine(object):
def __init__(self):
self.handlers = {}
self.start_state = None
self.end_states = []
def add_state(self, name, handler, end_state=0):
name = name.upper()
self.handlers[name] = handler
if end_state:
self.end_states.append(name)
def set_start(self, name):
# startup state
self.start_state = name
def run(self, **kw):
"""
Run
:param kw:
:return:
"""
# the first .run call call the first handler with kw keywords
# each registered handler should returns the following handler and the needed kw
try:
handler = self.handlers[self.start_state]
except:
raise InitializationError("must call .set_start() before .run()")
while True:
(new_state, kw) = handler(**kw)
if isinstance(new_state, str):
if new_state in self.end_states:
print("reached ", new_state)
break
else:
handler = self.handlers[new_state]
elif hasattr(new_state, "__call__"):
handler = new_state
else:
return
The use
class MyParser(StateMachine):
def __init__(self):
super().__init__()
# define handlers
# we can define many handler as we want
self.handlers["begin_parse"] = self.begin_parse
# define the startup handler
self.set_start("begin_parse")
def end(self, **kw):
logging.info("End of parsing ")
# no callable handler => end
return None, None
def second(self, **kw):
logging.info("second ")
# do something
# if condition is reach the call `self.end` handler
if ...:
return self.end, {}
def begin_parse(self, **kw):
logging.info("start of parsing ")
# long process until the condition is reach then call the `self.second` handler with kw new keywords
while True:
kw = {}
if ...:
return self.second, kw
# elif other cond:
# return self.other_handler, kw
# elif other cond 2:
# return self.other_handler 2, kw
else:
return self.end, kw
# start the state machine
MyParser().run()
will print
INFO:root:start of parsing
INFO:root:second
INFO:root:End of parsing

You could use local functions in your func function. Ok, they are still declared inside one single global function, but Python is nice enough to still give you access to them for tests.
Here is one example of one function declaring and executing 3 (supposedly heavy) subfunctions. It takes one optional parameter test that when set to TEST prevent actual execution but instead gives external access to individual sub-functions and to a local variable:
def func(test=None):
glob = []
def partA():
glob.append('A')
def partB():
glob.append('B')
def partC():
glob.append('C')
if (test == 'TEST'):
global testA, testB, testC, testCR
testA, testB, testC, testCR = partA, partB, partC, glob
return None
partA()
partB()
partC()
return glob
When you call func, the 3 parts are executed in sequence. But if you first call func('TEST'), you can then access the local glob variable as testCR, and the 3 subfunctions as testA, testB and testC. This way you can still test individually the 3 parts with well defined input and control their output.

I would insist on the suggestion given by #user3159253 in his comment on the original question:
If the sole purpose is readability I would split the func into three "private" > or "protected" ones (i.e. _func1 or __func1) and a private or protected property > which keeps the state shared between the functions.
This makes a lot of sense to me and seems more usual amongst object oriented programming than the other options. Consider this example as an alternative:
Your class (teste.py):
class Test:
def __init__(self):
self.__environment = {} # Protected information to be shared
self.public_stuff = 'public info' # Accessible to outside callers
def func(self):
print "Main function"
self.__func_a()
self.__func_b()
self.__func_c()
print self.__environment
def __func_a(self):
self.__environment['function a says'] = 'hi'
def __func_b(self):
self.__environment['function b says'] = 'hello'
def __func_c(self):
self.__environment['function c says'] = 'hey'
Other file:
from teste import Test
t = Test()
t.func()
This will output:
Main function says hey guys
{'function a says': 'hi', 'function b says': 'hello', 'function c says': 'hey'}
If you try to call one of the protected functions, an error occurs:
Traceback (most recent call last):
File "C:/Users/Lucas/PycharmProjects/testes/other.py", line 6, in <module>
t.__func_a()
AttributeError: Test instance has no attribute '__func_a'
Same thing if you try to access the protected environment variable:
Traceback (most recent call last):
File "C:/Users/Lucas/PycharmProjects/testes/other.py", line 5, in <module>
print t.__environment
AttributeError: Test instance has no attribute '__environment'
In my view this is the most elegant, simple and readable way to solve your problem, let me know if it fits your needs :)

Function into a Python class

I am new in Python and I wrote the following code:
class Frazione:
def __init__(self, Numeratore, Denominatore=1):
mcd=MCD(Numeratore,Denominatore)
self.Numeratore=Numeratore/mcd
self.Denominatore=Denominatore/mcd
def MCD(m,n):
if m%n==0:
return n
else:
return MCD(n,m%n)
def __str__(self):
return "%d/%d" %(self.Numeratore, self.Denominatore)
def __mul__(self, AltraFrazione):
if type(AltraFrazione)==type(5):
AltraFrazione=Frazione(AltraFrazione)
return Frazione(self.Numeratore*AltraFrazione.Numeratore, self.Denominatore*AltraFrazione.Denominatore)
__rmul__=__mul__
Open shell at the same folder of Frazione.py:
>>> from Frazione import Frazione
end then
>>> f=Frazione(10,5)
When I press Enter, I receive this output:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".\Frazione.py", line 5, in __init__
mcd=MCD(Numeratore,Denominatore)
NameError: global name 'MCD' is not defined
PS. I apologize for my english!

MCD is a method of Frazione, but you're calling it as if it were a global function. The easiest (and cleanest, IMHO) fix is to just move it outside the class, because it doesn't need to access any class or instance members.
So:
def MCD(m, n):
if m % n == 0:
return n
else:
return MCD(n, m % n)
class Frazione:
# as before but without MCD
If you do want to keep it in the class, then you might rewrite it to be iterative instead of recursive and call it as self.MCD in __init__. That's a good idea anyway, as Python's support for recursion is rather weak.

Passing uncertain amount of inputs to one function's sub-functions

Sorry if I did not explain myself clearly.
I would like to create a wrapper to call pre-defined functions with different number of inputs. Of course, I can create an individual wrapper for each function, but I am wondering if there is a way to create a generic wrapper for all cases.
The functions that should be called are named 'fun1' and 'fun2' with different number of inputs. I need to create a wrapper 'fun_wrap(func_name, uncertain amount of inputs)', which only needs the function name to be called and its associated amount of inputs.
One more thing, I need to change the input names by adding '_in' and make them global variables first. Below is my broken code. Thanks for any suggestions!
def fun1(a,b):
return a+b
def fun2(a,b,c):
return a*b/c
def set_globals(**kwargs):
for argname in kwargs:
globals()['%s_in' % argname] = kwargs[argname]
def fun_wrap(func_name, uncertain amount of inputs):
ffunc_name(set_globals(uncertain amount of inputs))
In this way, if I can call final_fun with arguments like:
fun_wrap(fun1,a,b)
fun_wrap(fun2,a,b)
UPDATE
I tried to use *arg, but failed...
def fun1(a,b):
return a+b
def fun2(a,b,c):
return a*b/c
def set_globals(**kwargs):
for argname in kwargs:
globals()['%s_in' % argname] = kwargs[argname]
def fun_wrap(func_name, *arg):
func_name(set_globals(*arg))
fun_wrap(fun2,a=1,b=2,c=3)
got error:
Traceback (most recent call last):
File "D:\Dropbox\AppPest\rice\try.py", line 19, in <module>
fun_wrap(fun2,a=1,b=2,c=3)
TypeError: fun_wrap() got an unexpected keyword argument 'a'

def fun1(a,b):
return a + b
def fun2(a,b,c):
return a * b / c
def set_globals(**kwargs):
for argname in kwargs:
globals()['%s_in' % argname] = kwargs[argname]
def fun_wrap(func, **kwargs):
set_globals(**kwargs) # made the call to set_globals before calling your function
return func(**kwargs) # return the value returned by the functions called