I want to replace some builtin functions inside the code that I run with exec. It is possible by passing it as a dictionary entry in the second exec argument. But when I try to import a module inside the executed code, the functions are as in original bultins, when called inside imported module.
This is the example of what I'm trying to achieve:
from inspect import cleandoc
def new_print(val):
print('Hello', val)
code_inner = cleandoc("""
def bar():
print('Inner')
""")
with open('inner.py', 'w') as f:
f.write(code_inner)
code_outer = cleandoc("""
import inner
print('Outer')
inner.bar()
""")
exec(code_outer, {'print': new_print}, {})
This is the response that I receive:
Hello Outer
Inner
And this is what I would like to have:
Hello Outer
Hello Inner
Is there any way to pass new globals, or builtins, or maybe variable list to the module beeing imported?
I'm not sure if it's quite what you want, but passing a dictionary parameter to the function and updating its modules globals works.
code_inner = cleandoc("""
def bar(d):
globals().update(d)
print('Inner')
""")
code_outer = cleandoc("""
import inner
print('Outer')
inner.bar({'print': print})
""")
Alternatively, without modification of the bar function you can pass its module a global like so:
code_outer = cleandoc("""
import inner
inner.print = print
print('Outer')
inner.bar()
""")
Related
If I import a module that uses exec or eval, is it possible to give it access to the main program?
myExec.py
def myExec(code):
exec(code)
main.py
import myExec
def test():
print("ok")
myExec.myExec("test()")
Yes!
exec has a few optional parameters, globals and locals. These basically tell it what global and local variables its allowed to use, in the form of a dictionary. Calling globals() or locals() functions returns the dictionary with all the global and local variables where you are calling from, so you can use:
myExec.py:
def myExec(code, globals_=None, locals_=None): # the trailing underscore is so that there are no name conflicts
exec(code, globals_, locals_)
main.py:
import myExec
def test():
print("ok")
myExec.myExec("test()", globals())
I want python to show me which function has been executed and from what file... so I have the following as test1.py
import sys,os, test2
def some_function():
print (sys._getframe().f_code.co_name +" "+ os.path.basename(__file__) , 'executed')
print (test2.function_details(), 'executed')
test2.py is:
import sys,os
def function_details():
return sys._getframe().f_code.co_name + " " +os.path.basename(__file__)
now when I run it
import test1
test1.some_function()
I get the following output:
('some_function test1.pyc', 'executed')
('function_details test2.pyc', 'executed')
When I try to make a function for calling the file and function of the executed, it tells me the sub function I made instead of the original.
My question is how do I modify test2.py so it will output
('some_function test1.pyc', 'executed')
('some_function test1.pyc', 'executed')
So there are two issues with function_details:
The current frame is function_details so you need to go up one frame to get to the calling frame. To do this you pass 1 to sys._getframe.
__file__ is the name of the current file for whatever module you happen to be in (if defined), this needs to be replaced with the co_filename attribute of the f_code object associated with the correct frame.
Correcting both of these things, we redefine function_details as:
def function_details():
code_obj = sys._getframe(1).f_code
return " ".join([code_obj.co_name, os.path.basename(code_obj.co_filename)])
Which produces the desired result. In my opinion, the inspect module accomplishes this same thing far better than using sys._getframe directly. Here's the new function_details written using inspect:
import inspect
def function_details():
frame = inspect.getouterframes(inspect.currentframe())[1]
return " ".join([frame[3], os.path.basename(frame[1])])
I am designing a module, say mymodule.py and I write the code for the module as follows:
def charCount(my_string, my_char):
a = my_string.count(my_char)
return a
def aCount(my_string):
a = charCount(my_string, 'a')
return a
Inside Python shell, I use the following command:
import mymodule as mm
and then,
mString = 'ghghghghgaaaaa'
and then
a = mm.aCount(mString)
It is seen that there is an error. Apparently, the function is not able to be called from the same module. How can this be avoided?
You need to put return statement in both the functions and it would work fine.
Try this:
def charCount(my_string, my_char):
a = my_string.count(my_char)
return a
def aCount(my_string):
a = charCount(my_string, 'a')
return a
I have a number of functions that need to get called from various imported files.
The functions are formated along the lines of this:
a.foo
b.foo2
a.bar.foo4
a.c.d.foo5
and they are passed in to my script as a raw string.
I'm looking for a clean way to run these, with arguments, and get the return values
Right now I have a messy system of splitting the strings then feeding them to the right getattr call but this feels kind of clumsy and is very un-scalable. Is there a way I can just pass the object portion of getattr as a string? Or some other way of doing this?
import a, b, a.bar, a.c.d
if "." in raw_script:
split_script = raw_script.split(".")
if 'a' in raw_script:
if 'a.bar' in raw_script:
out = getattr(a.bar, split_script[-1])(args)
if 'a.c.d' in raw_script:
out = getattr(a.c.d, split_script[-1])(args)
else:
out = getattr(a, split_script[-1])(args)
elif 'b' in raw_script:
out = getattr(b, split_script[-1])(args)
It's hard to tell from your question, but it sounds like you have a command line tool you run as my-tool <function> [options]. You could use importlib like this, avoiding most of the getattr calls:
import importlib
def run_function(name, args):
module, function = name.rsplit('.', 1)
module = importlib.import_module(module)
function = getattr(module, function)
function(*args)
if __name__ == '__main__':
# Elided: retrieve function name and args from command line
run_function(name, args)
Try this:
def lookup(path):
obj = globals()
for element in path.split('.'):
try:
obj = obj[element]
except KeyError:
obj = getattr(obj, element)
return obj
Note that this will handle a path starting with ANY global name, not just your a and b imported modules. If there are any possible concerns with untrusted input being provided to the function, you should start with a dict containing the allowed starting points, not the entire globals dict.
I decided to try to preprocess function text before it's compilation into byte-code and following execution. This is merely for training. I hardly imagine situations where it'll be a satisfactory solution to be used. I have faced one problem which I wanted to solve in this way, but eventually a better way was found. So this is just for training and to learn something new, not for real usage.
Assume we have a function, which source code we want to be modified quite a bit before compilation:
def f():
1;a()
print('Some statements 1')
1;a()
print('Some statements 2')
Let, for example, mark some lines of it with 1;, for them to be sometimes commented and sometimes not. I just take it for example, modifications of the function may be different.
To comment these lines I made a decorator. The whole code it bellow:
from __future__ import print_function
def a():
print('a()')
def comment_1(s):
lines = s.split('\n')
return '\n'.join(line.replace(';','#;',1) if line.strip().startswith('1;') else line for line in lines)
def remove_1(f):
import inspect
source = inspect.getsource(f)
new_source = comment_1(source)
with open('temp.py','w') as file:
file.write(new_source)
from temp import f as f_new
return f_new
def f():
1;a()
print('Some statements 1')
1;a()
print('Some statements 2')
f = remove_1(f) #If decorator #remove is used above f(), inspect.getsource includes #remove inside the code.
f()
I used inspect.getsourcelines to retrieve function f code. Then I made some text-processing (in this case commenting lines starting with 1;). After that I saved it to temp.py module, which is then imported. And then a function f is decorated in the main module.
The output, when decorator is applied, is this:
Some statements 1
Some statements 2
when NOT applied is this:
a()
Some statements 1
a()
Some statements 2
What I don't like is that I have to use hard drive to load compiled function. Can it be done without writing it to temporary module temp.py and importing from it?
The second question is about placing decorator above f: #replace. When I do this, inspect.getsourcelines returns f text with this decorator. I could manually be deleted from f's text. but that would be quite dangerous, as there may be more than one decorator applied. So I resorted to the old-style decoration syntax f = remove_1(f), which does the job. But still, is it possible to allow normal decoration technique with #replace?
One can avoid creating a temporary file by invoking the exec statement on the source. (You can also explicitly call compile prior to exec if you want additional control over compilation, but exec will do the compilation for you, so it's not necessary.) Correctly calling exec has the additional benefit that the function will work correctly if it accesses global variables from the namespace of its module.
The problem described in the second question can be resolved by temporarily blocking the decorator while it is running. That way the decorator remains, along all the other ones, but is a no-op.
Here is the updated source.
from __future__ import print_function
import sys
def a():
print('a()')
def comment_1(s):
lines = s.split('\n')
return '\n'.join(line.replace(';','#;',1) if line.strip().startswith('1;') else line for line in lines)
_blocked = False
def remove_1(f):
global _blocked
if _blocked:
return f
import inspect
source = inspect.getsource(f)
new_source = comment_1(source)
env = sys.modules[f.__module__].__dict__
_blocked = True
try:
exec new_source in env
finally:
_blocked = False
return env[f.__name__]
#remove_1
def f():
1;a()
print('Some statements 1')
1;a()
print('Some statements 2')
f()
def remove_1(f):
import inspect
source = inspect.getsource(f)
new_source = comment_1(source)
env = sys.modules[f.__module__].__dict__.copy()
exec new_source in env
return env[f.__name__]
I'll leave a modified version of the solution given in the answer by user4815162342. It uses ast module to delete some parts of f, as was suggested in the comment to the question. To make it I majorly relied on the information in this article.
This implementation deletes all occurrences of a as standalone expression.
from __future__ import print_function
import sys
import ast
import inspect
def a():
print('a() is called')
_blocked = False
def remove_1(f):
global _blocked
if _blocked:
return f
import inspect
source = inspect.getsource(f)
a = ast.parse(source) #get ast tree of f
class Transformer(ast.NodeTransformer):
'''Will delete all expressions containing 'a' functions at the top level'''
def visit_Expr(self, node): #visit all expressions
try:
if node.value.func.id == 'a': #if expression consists of function with name a
return None #delete it
except(ValueError):
pass
return node #return node unchanged
transformer = Transformer()
a_new = transformer.visit(a)
f_new_compiled = compile(a_new,'<string>','exec')
env = sys.modules[f.__module__].__dict__
_blocked = True
try:
exec(f_new_compiled,env)
finally:
_blocked = False
return env[f.__name__]
#remove_1
def f():
a();a()
print('Some statements 1')
a()
print('Some statements 2')
f()
The output is:
Some statements 1
Some statements 2