I have a YAML script where the users can define some parameters. They can also state in the YAML file a custom function they want to apply. For example:
processes:
args1: 10
args2: 5
funct: np.sum
This YAML will be passed to a function like this:
def custom_func(args1, args2, funct):
return funct(args1, args2)
With the YAML example above, I expect custom_func() to execute np.sum(10,5). How can I make the data in YAML file to be parsed as Callable? eval() probably does the job but it might have security issue. Is there any proper way? Thanks!
You can also load the function given its Python dotted path.
However, this only gets the function and calls it, it does not try to figure out calling signature, which is np.sum([5,10]), not np.sum(10,5)
from yaml import safe_load as yload
from importlib import import_module
def load_func(dotpath : str):
""" load function in module. function is right-most segment """
module_, func = dotpath.rsplit(".", maxsplit=1)
m = import_module(module_)
return getattr(m, func)
yaml = """
processes:
args1: 10
args2: 5
#gotta give real package name, not `np`
funct: numpy.sum
"""
di = yload(yaml)
pr = di["processes"]
func = load_func(pr["funct"])
array = [pr["args1"], pr["args2"]]
print (f"{func=} {func(array)=}")
output:
func=<function sum at 0x106645e50> func(array)=15
Yes, eval and exec are both bad practices. Good that you know that :D
You can use dictionaries to do this.
What you can do is define a dictionary and make the key the funct value in YAML. The value will be an uncalled function. This is similar to a variable.
funct_parser = {
"np.sum": np.sum # Notice how this doesn't have ()
}
You can then use funct against the dictionary to call the proper method. Something like the following:
def custom_func(args1, args2, funct):
my_func = funct_parser[str(funct)] # This will cause my_func to be a copy of `np.sum`.
return my_func(args1, args2) # Call my_func
BTW, you don't really have to do custom_func on multiple lines, I just did so to explain it better.
Alternative choice:
If you don't feel like hardcoding every single part in a dictionary, you can use getattr().
# This code assumes you have numpy imported as np (You can always change it)
def custom_func(args1, args2, funct):
funct = funct.split("np.")[0] # Will work for np. You can always change to other modules when needed.
return getattr(np, funct)
Related
I am attempting to implement a decorator that receives a function, parses it into an AST, eventually will do something to the AST, then reconstruct the original (or modified) function from the AST and return it. My current approach is, once I have the AST, compile it to a code <module> object, then get the constant in it with the name of the function, convert it to FunctionType, and return it. I have the following:
import ast, inspect, types
def as_ast(f):
source = inspect.getsource(f)
source = '\n'.join(source.splitlines()[1:]) # Remove as_ast decoration, pretend there can be no other decorations for now
tree = ast.parse(source)
print(ast.dump(tree, indent=4)) # Debugging log
# I would modify the AST somehow here
filename = f.__code__.co_filename
code = compile(tree, filename, 'exec')
func_code = next(
filter(
lambda x: isinstance(x, types.CodeType) and x.co_name == f.__name__,
code.co_consts)) # Get function object
func = types.FunctionType(func_code, {})
return func
#as_ast
def test(arg: int=4):
print(f'{arg=}')
Now, I would expect that calling test later in this source code will simply have the effect of calling test if the decorator were absent, which is what I observe, so long as I pass an argument for arg. However, if I pass no argument, instead of using the default I gave (4), it throws a TypeError for the missing argument. This makes it pretty clear that my approach for getting a callable function from the AST is not quite correct, as the default argument is not applied, and there may be other details that would slip through as it is now. How might I be able to correctly recreate the function from the AST? The way I currently go from the code module object to the function code object also seems... off intuitively, but I do not know how else one might achieve this.
The root node of the AST is a Module. Calling compile() on the AST, results in a code object for a module. Looking at the compiled code object returned using dis.dis(), from the standard library, shows the module level code builds the function and stores it in the global name space. So the easiest thing to do is exec the compiled code and then get the function from the 'global' environment of the exec call.
The AST node for the function includes a list of the decorators to be applied to the function. Any decorators that haven't been applied yet should be deleted from the list so they don't get applied twice (once when this decorator compiles the code, and once after this decorator returns). And delete this decorator from the list or you'll get an infinite recursion. The question is what to do with any decorators that came before this one. They have already run, but their result is tossed out because this decorator (as_ast) goes back to the source code. You can leave them in the list so they get rerun, or delete them if they don't matter.
In the code below, all the decorators are deleted from the parse tree, under the assumption that the as_ast decorator is applied first. The call to exec() uses a copy of globals() so the decorator has access to any other globally visible names (variables, functions, etc). See the docs for exec() for other considerations. Uncommented the print statements to see what is going on.
import ast
import dis
import inspect
import types
def as_ast(f):
source = inspect.getsource(f)
#print(f"=== source ===\n{source}")
tree = ast.parse(source)
#print(f"\n=== original ===\n{ast.dump(tree, indent=4)}")
# Remove the decorators from the AST, because the modified function will
# be passed to them anyway and we don't want them to be called twice.
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef):
node.decorator_list.clear()
# Make modifications to the AST here
#print(f"\n=== revised ===\n{ast.dump(tree, indent=4)}")
name = f.__code__.co_name
code = compile(tree, name, 'exec')
#print("\n=== byte code ===")
#dis.dis(code)
#print()
temp_globals = dict(globals())
exec(code, temp_globals)
return temp_globals[name]
Note: this decorator has not been tested much and has not been tested at all on methods or nested functions.
An interesting idea would be to for as_ast to return the AST. Then subsequent decorators could manipulate the AST. Lastly, a from_ast decorator could compile the modified AST into a function.
I recently found some code that calls a function from a different directory like this:
modules["module_name"].function(parameters)
Then I had a thought about calling a function (but not a module; I already know how to do that) through lists in that format like so:
[function_name](parameters)
You can't call a function like that in a list. I don't know what that syntax you have there. t's not possible, but you can call method references from dicts or variables below
If you're function is stored in a dictionary like so
def add(x, y):
return x + y
some_dict = {
'add' : add
}
Then you can call it like so
some_dict['add'](10, 10)
Calling a function using list... you can have a list of functions and call it out from index
def foo(arg):
print(arg)
list_of_functions = [foo]
list_of_functions[0](3)
This also works with dictionaries, since functions are just pointers, as anything else.
But i believe you wanted to find the function by the name from variable, in that case, probably (idk, never went this route) use globals for that one
def foo(arg):
print(arg)
function_name = 'foo'
globals()[function_name](3)
P.S. I still wonder about 2.7 though
If the list is created in other module :
module.py it should looks like this:
def add(a,b):
return a+b
lista = [add]
And you could call the fuction like this in your own module:
from module import lista
res = lista[0](2,3)
print(res)
The important here is note that the function should be define before to list it and in the same module where the list is created. In other way your code will be broken.
Don't ask why but I have python 2.7 too
I have a function foo that calls another function get_info_from_tags.
Here's the get_info_from_tags implementation:
def get_info_from_tags(*args):
instance_id = get_instance_id()
proc = subprocess.Popen(["aws", "ec2", "describe-tags", "--filters", f"Name=resource-id,Values={instance_id}"],
stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()
em_json = json.loads(out.decode('utf8'))
tags = em_json['Tags'] # tags list
results = []
for arg in args:
for tag in tags:
if tag['Key'] == arg:
results.append(tag['Value'])
return results
There is a set of 10 possible args that can be passed to get_info_from_tags, and I need to return the correct array (I don't want to make a call to aws services, that's the point of my mock, I will manually set the values in a dictionary).
How can I mock get_info_from_tags so that when I call
get_info_from_tags('key1', 'key2' ...)
inside the foo function, I get the results I want?
I've already tried some functions of pytest but, as it seems, I didn't quite understand.
A possible solution would be to create another function:
def mocked_get_info_from_tags(*args):
values = []
for arg in args:
values.append(my_dictionary[arg])
return values
But I don't know how to implement this override within a test environment.
Thank you.
unittest.mock.patch is your friend.
You didn't specify the module names, so I put some <placeholders> there.
from unittest.mock import patch
from <module_with_foo> import foo
from <module_with_mock> import mocked_get_info_from_tags
with patch('<module_with_foo>.get_info_from_tags', mocked_get_info_from_tags):
foo()
This will replace get_info_from_tags with your mocked version of this function. The replacement is done on a module level, so everything in module <module_with_foo> that calls get_info_from_tags will now call your mock.
Note about the path given to patch:
patch replaces values of module attributes. So, if you have a module moo with a function foo, which calls bar from module moo2:
# moo module
from moo2 import bar
def foo():
bar()
...from the point of view of patch, moo.foo calls moo.bar, not moo2.bar. That's why you have to patch the module where the function is used, not where it is defined.
You could call one function from the other, and then replace when you have the db setup. That way you can still call foo.get_info_from_db('key1', 'key2' ...) everywhere in your code and when you add the proper database connection, all you have to change is the one main get_info_from_db function implementation and remove the mock
import db_connector
def mocked_get_info_from_db(*args):
values = []
for arg in args:
values.append(my_dictionary[arg])
return values
def get_info_from_db(*args):
# remove this once your database is setup
return mocked_get_info_from_db(*args)
# values = []
# for arg in args:
# values.append(db_connector.get(arg))
# return values
I'm writing a wrapper or pipeline to create a tfrecords dataset to which I would like to supply a function to apply to the dataset.
I would like to make it possible for the user to inject a function defined in another python file which is called in my script to transform the data.
Why? The only thing the user has to do is write the function which brings his data into the right format, then the existing code does the rest.
I'm aware of the fact that I could have the user write the function in the same file and call it, or to have an import statement etc.
So as a minimal example, I would like to have file y.py
def main(argv):
# Parse args etc, let's assume it is there.
dataset = tf.data.TFRecordDataset(args.filename)
dataset = dataset.map(args.function)
# Continue with doing stuff that is independent from actual content
So what I'd like to be able to do is something like this
python y.py --func x.py my_func
And use the function defined in x.py my_func in dataset.map(...)
Is there a way to do this in python and if yes, which is the best way to do it?
Pass the name of the file as an argument to your script (and function name)
Read the file into a string, possibly extracting the given function
use Python exec() to execute the code
An example:
file = "def fun(*args): \n return args"
func = "fun(1,2,3)"
def execute(func, file):
program = file + "\nresult = " + func
local = {}
exec(program, local)
return local['result']
r = execute(func, file)
print(r)
Similar to here however we must use locals() as we are not calling exec in global scope.
Note: the use of exec is somewhat dangerous, you should be sure that the function is safe - if you are using it then its fine!
Hope this helps.
Ok so I have composed the answer myself now using the information from comments and this answer.
import importlib, inspect, sys, os
# path is given path to file, funcion_name is name of function and args are the function arguments
# Create package and module name from path
package = os.path.dirname(path).replace(os.path.sep,'.')
module_name = os.path.basename(path).split('.')[0]
# Import module and get members
module = importlib.import_module(module_name, package)
members = inspect.getmembers(module)
# Find matching function
function = [t[1] for t in members if t[0] == function_name][0]
function(args)
This exactly solves the question, since I get a callable function object which I can call, pass around, use it as a normal function.
I have a file that contains several python functions, each with some statements.
def func1():
codeX...
def func2():
codeY...
codeX and codeY can be multiple statements. I want to be able to parse the file, find a function by name, then evaluate the code in that function.
With the ast module, I can parse the file, find the FunctionDef objects, and get the list of Stmt objects, but how do I turn this into bytecode that I can pass to eval? Should I use the compile module, or the parser module instead?
Basically, the function defs are just used to create separate blocks of code. I want to be able to grab any block of code given the name and then execute that code in eval (providing my own local/global scope objects). If there is a better way to do this than what I described that would be helpful too.
Thanks
I want to be able to grab any block of code given the name and then execute that code ... (providing my own local/global scope objects).
A naive solution looks like this. This is based on the assumption that the functions don't all depend on global variables.
from file_that_contains_several_python_functions import *
Direction = some_value
func1()
func2()
func3()
That should do exactly what you want.
However, if all of your functions rely on global variables -- a design that calls to mind 1970's-era FORTRAN -- then you have to do something slightly more complex.
from file_that_contains_several_python_functions import *
Direction = some_value
func1( globals() )
func2( globals() )
func3( globals() )
And you have to rewrite all of your global-using functions like this.
def func1( context )
globals().update( context )
# Now you have access to all kinds of global variables
This seems ugly because it is. Functions which rely entirely on global variables are not really the best idea.
Using Python 2.6.4:
text = """
def fun1():
print 'fun1'
def fun2():
print 'fun2'
"""
import ast
tree = ast.parse(text)
# tree.body[0] contains FunctionDef for fun1, tree.body[1] for fun2
wrapped = ast.Interactive(body=[a.body[1]])
code = compile(wrapped, 'yourfile', 'single')
eval(code)
fun2() # prints 'fun2'
Take a look at grammar in ast doc: http://docs.python.org/library/ast.html#abstract-grammar. Top-level statement must be either Module, Interactive or Expression, so you need to wrap function def in one of those.
If you're using Python 2.6 or later, then the compile() function accepts AST objects in addition to source code.
>>> import ast
>>> a = ast.parse("print('hello world')")
>>> x = compile(a, "(none)", "exec")
>>> eval(x)
hello world
These modules have all been rearranged for Python 3.