I'm making a script parser in python and I'm a little stuck. I am not quite sure how to parse a line for all its functions (or even just one function at a time) and then search for a function with that name, and if it exists, execute that function short of writing a massive list if elif else block....
EDIT
This is for my own scripting language that i'm making. its nothing very complex, but i have a standard library of 8 functions or so that i need to be able to be run, how can i parse a line and run the function named in the line?
Once you get the name of the function, use a dispatch dict to run the function:
def mysum(...): ...
def myotherstuff(...): ...
# create dispatch dict:
myfunctions = {'sum': mysum, 'stuff': myotherstuff}
# run your parser:
function_name, parameters = parse_result(line)
# run the function:
myfunctions[function_name](parameters)
Alternatively create a class with the commands:
class Commands(object):
def do_sum(self, ...): ...
def do_stuff(self, ...): ...
def run(self, funcname, params):
getattr(self, 'do_' + funcname)(params)
cmd = Commands()
function_name, parameters = parse_result(line)
cmd.run(function_name, parameters)
You could also look at the cmd module in the stdlib to do your class. It can provide you with a command-line interface for your language, with tab command completion, automatically.
Check out PyParsing, it allows for definition of the grammar directly in Python code:
Assuming a function call is just somename():
>>> from pyparsing import *
>>> grammar = Word(alphas + "_", alphanums + "_")("func_name") + "()" + StringEnd()
>>> grammar.parseString("ab()\n")["func_name"]
"ab"
Take a look at PLY. It should help you keep your parser specification clean.
It all depends on what code you are parsing.
If you are parsing Python syntax, use the parser module from Python:
http://docs.python.org/library/parser.html
A quite complete list of parser libraries available for Python you can find at: http://nedbatchelder.com/text/python-parsers.html
Related
I have a YAML script where the users can define some parameters. They can also state in the YAML file a custom function they want to apply. For example:
processes:
args1: 10
args2: 5
funct: np.sum
This YAML will be passed to a function like this:
def custom_func(args1, args2, funct):
return funct(args1, args2)
With the YAML example above, I expect custom_func() to execute np.sum(10,5). How can I make the data in YAML file to be parsed as Callable? eval() probably does the job but it might have security issue. Is there any proper way? Thanks!
You can also load the function given its Python dotted path.
However, this only gets the function and calls it, it does not try to figure out calling signature, which is np.sum([5,10]), not np.sum(10,5)
from yaml import safe_load as yload
from importlib import import_module
def load_func(dotpath : str):
""" load function in module. function is right-most segment """
module_, func = dotpath.rsplit(".", maxsplit=1)
m = import_module(module_)
return getattr(m, func)
yaml = """
processes:
args1: 10
args2: 5
#gotta give real package name, not `np`
funct: numpy.sum
"""
di = yload(yaml)
pr = di["processes"]
func = load_func(pr["funct"])
array = [pr["args1"], pr["args2"]]
print (f"{func=} {func(array)=}")
output:
func=<function sum at 0x106645e50> func(array)=15
Yes, eval and exec are both bad practices. Good that you know that :D
You can use dictionaries to do this.
What you can do is define a dictionary and make the key the funct value in YAML. The value will be an uncalled function. This is similar to a variable.
funct_parser = {
"np.sum": np.sum # Notice how this doesn't have ()
}
You can then use funct against the dictionary to call the proper method. Something like the following:
def custom_func(args1, args2, funct):
my_func = funct_parser[str(funct)] # This will cause my_func to be a copy of `np.sum`.
return my_func(args1, args2) # Call my_func
BTW, you don't really have to do custom_func on multiple lines, I just did so to explain it better.
Alternative choice:
If you don't feel like hardcoding every single part in a dictionary, you can use getattr().
# This code assumes you have numpy imported as np (You can always change it)
def custom_func(args1, args2, funct):
funct = funct.split("np.")[0] # Will work for np. You can always change to other modules when needed.
return getattr(np, funct)
I'm writing a wrapper or pipeline to create a tfrecords dataset to which I would like to supply a function to apply to the dataset.
I would like to make it possible for the user to inject a function defined in another python file which is called in my script to transform the data.
Why? The only thing the user has to do is write the function which brings his data into the right format, then the existing code does the rest.
I'm aware of the fact that I could have the user write the function in the same file and call it, or to have an import statement etc.
So as a minimal example, I would like to have file y.py
def main(argv):
# Parse args etc, let's assume it is there.
dataset = tf.data.TFRecordDataset(args.filename)
dataset = dataset.map(args.function)
# Continue with doing stuff that is independent from actual content
So what I'd like to be able to do is something like this
python y.py --func x.py my_func
And use the function defined in x.py my_func in dataset.map(...)
Is there a way to do this in python and if yes, which is the best way to do it?
Pass the name of the file as an argument to your script (and function name)
Read the file into a string, possibly extracting the given function
use Python exec() to execute the code
An example:
file = "def fun(*args): \n return args"
func = "fun(1,2,3)"
def execute(func, file):
program = file + "\nresult = " + func
local = {}
exec(program, local)
return local['result']
r = execute(func, file)
print(r)
Similar to here however we must use locals() as we are not calling exec in global scope.
Note: the use of exec is somewhat dangerous, you should be sure that the function is safe - if you are using it then its fine!
Hope this helps.
Ok so I have composed the answer myself now using the information from comments and this answer.
import importlib, inspect, sys, os
# path is given path to file, funcion_name is name of function and args are the function arguments
# Create package and module name from path
package = os.path.dirname(path).replace(os.path.sep,'.')
module_name = os.path.basename(path).split('.')[0]
# Import module and get members
module = importlib.import_module(module_name, package)
members = inspect.getmembers(module)
# Find matching function
function = [t[1] for t in members if t[0] == function_name][0]
function(args)
This exactly solves the question, since I get a callable function object which I can call, pass around, use it as a normal function.
I've been given a Python code, together with the modules it imports. I would like to build a tree indicating which function calls what other functions. How can I do that?
you can use the ast (abstract syntax tree) module from the python standard library
# foo.py
def func(x):
print('hello')
parsing the file using ast.parse:
import ast
tree = ast.parse(open('foo.py').read())
print(ast.dump(tree)) # dumps the whole tree
# get the function from the tree body (i.e. from the file's content)
func = tree.body[0]
# get the function argument names
arguments = [a.arg for a in func.args.args]
print('the functions is: %s(%s)' % (func.name, ', '.join(arguments)))
outputs:
"Module(body=[FunctionDef(name='func', args=arguments(args=[arg(arg='x', annotation=None)], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[]), body=[Expr(value=Call(func=Name(id='print', ctx=Load()), args=[Str(s='hello')], keywords=[]))], decorator_list=[], returns=None)])"
the functions is: func(x)
You should begin from the main function of the program and at the first layer link all functions that are called from the main this would provide a start point and then you can link all the functions below it.
I'm doing genetic programming framework and I need to be able to execute some string representing complete python programs. I'm using Python 2.7. I have a config class in which the primitive sets are defined. Lets say
class Foo():
def a(self,x):
return x
def b(self,y):
return y
I'm extracting the functions with the python inspect module and I want to create some executable source code with imports and everything. I end up with a string that looks like this
import sys
def a(x,y):
return x
def b(y):
return y
def main(x,y)
lambda x,y: a(b(y),a(x,y))
main(*sys.argv)
My problem is that I don't know how to pass command line arguments to the string I'm running with eval(). How can I pass command line arguments to a source file I want to run with eval()?
Edit: There are millions of individuals so writing to a file is not a great option.
Edit: I made a mistake. The eval() method is used only for expressions and not statements so using exec() is the correct approach
eval("function_name")(arg1, arg2)
or if you have a list of arguments:
arguments= [arg1,arg2,arg3,something]
eval("function_name")(*arguments)
You have three options, roughly speaking. You can keep going with eval(),you could actually write the string as a file and execute it with subprocess.Popen(), or you could call the function something besides main() and call it after defining it with eval().
exec() way:
In the string you want to exec
main(#REPLACE_THIS#)
Function to evaluate
import string
def exec_with_args(exec_string,args):
arg_string=reduce(lambda x,y:x+','+y,args)
exec_string.replace("#REPLACE_THIS#", arg_string)
Subprocess way:
import subprocess
#Write string to a file
exec_file=open("file_to_execute","w")
exec_file.write(string_to_execute)
#Run the python file as a separate process
output=subprocess.Popen(["python","file_to_execute"].extend(argument_list),
stdout=subprocess.PIPE)
Function Definition Way
In the string you want to exec
def function_name(*args):
import sys
def a(x,y):
return x
def b(y):
return y
def inner_main(x,y):
lambda x,y: a(b(y),a(x,y))
inner_main(*args)
Outer code
exec(program_string)
function_name(*args)
I have a file that contains several python functions, each with some statements.
def func1():
codeX...
def func2():
codeY...
codeX and codeY can be multiple statements. I want to be able to parse the file, find a function by name, then evaluate the code in that function.
With the ast module, I can parse the file, find the FunctionDef objects, and get the list of Stmt objects, but how do I turn this into bytecode that I can pass to eval? Should I use the compile module, or the parser module instead?
Basically, the function defs are just used to create separate blocks of code. I want to be able to grab any block of code given the name and then execute that code in eval (providing my own local/global scope objects). If there is a better way to do this than what I described that would be helpful too.
Thanks
I want to be able to grab any block of code given the name and then execute that code ... (providing my own local/global scope objects).
A naive solution looks like this. This is based on the assumption that the functions don't all depend on global variables.
from file_that_contains_several_python_functions import *
Direction = some_value
func1()
func2()
func3()
That should do exactly what you want.
However, if all of your functions rely on global variables -- a design that calls to mind 1970's-era FORTRAN -- then you have to do something slightly more complex.
from file_that_contains_several_python_functions import *
Direction = some_value
func1( globals() )
func2( globals() )
func3( globals() )
And you have to rewrite all of your global-using functions like this.
def func1( context )
globals().update( context )
# Now you have access to all kinds of global variables
This seems ugly because it is. Functions which rely entirely on global variables are not really the best idea.
Using Python 2.6.4:
text = """
def fun1():
print 'fun1'
def fun2():
print 'fun2'
"""
import ast
tree = ast.parse(text)
# tree.body[0] contains FunctionDef for fun1, tree.body[1] for fun2
wrapped = ast.Interactive(body=[a.body[1]])
code = compile(wrapped, 'yourfile', 'single')
eval(code)
fun2() # prints 'fun2'
Take a look at grammar in ast doc: http://docs.python.org/library/ast.html#abstract-grammar. Top-level statement must be either Module, Interactive or Expression, so you need to wrap function def in one of those.
If you're using Python 2.6 or later, then the compile() function accepts AST objects in addition to source code.
>>> import ast
>>> a = ast.parse("print('hello world')")
>>> x = compile(a, "(none)", "exec")
>>> eval(x)
hello world
These modules have all been rearranged for Python 3.