I am attempting to implement a decorator that receives a function, parses it into an AST, eventually will do something to the AST, then reconstruct the original (or modified) function from the AST and return it. My current approach is, once I have the AST, compile it to a code <module> object, then get the constant in it with the name of the function, convert it to FunctionType, and return it. I have the following:
import ast, inspect, types
def as_ast(f):
source = inspect.getsource(f)
source = '\n'.join(source.splitlines()[1:]) # Remove as_ast decoration, pretend there can be no other decorations for now
tree = ast.parse(source)
print(ast.dump(tree, indent=4)) # Debugging log
# I would modify the AST somehow here
filename = f.__code__.co_filename
code = compile(tree, filename, 'exec')
func_code = next(
filter(
lambda x: isinstance(x, types.CodeType) and x.co_name == f.__name__,
code.co_consts)) # Get function object
func = types.FunctionType(func_code, {})
return func
#as_ast
def test(arg: int=4):
print(f'{arg=}')
Now, I would expect that calling test later in this source code will simply have the effect of calling test if the decorator were absent, which is what I observe, so long as I pass an argument for arg. However, if I pass no argument, instead of using the default I gave (4), it throws a TypeError for the missing argument. This makes it pretty clear that my approach for getting a callable function from the AST is not quite correct, as the default argument is not applied, and there may be other details that would slip through as it is now. How might I be able to correctly recreate the function from the AST? The way I currently go from the code module object to the function code object also seems... off intuitively, but I do not know how else one might achieve this.
The root node of the AST is a Module. Calling compile() on the AST, results in a code object for a module. Looking at the compiled code object returned using dis.dis(), from the standard library, shows the module level code builds the function and stores it in the global name space. So the easiest thing to do is exec the compiled code and then get the function from the 'global' environment of the exec call.
The AST node for the function includes a list of the decorators to be applied to the function. Any decorators that haven't been applied yet should be deleted from the list so they don't get applied twice (once when this decorator compiles the code, and once after this decorator returns). And delete this decorator from the list or you'll get an infinite recursion. The question is what to do with any decorators that came before this one. They have already run, but their result is tossed out because this decorator (as_ast) goes back to the source code. You can leave them in the list so they get rerun, or delete them if they don't matter.
In the code below, all the decorators are deleted from the parse tree, under the assumption that the as_ast decorator is applied first. The call to exec() uses a copy of globals() so the decorator has access to any other globally visible names (variables, functions, etc). See the docs for exec() for other considerations. Uncommented the print statements to see what is going on.
import ast
import dis
import inspect
import types
def as_ast(f):
source = inspect.getsource(f)
#print(f"=== source ===\n{source}")
tree = ast.parse(source)
#print(f"\n=== original ===\n{ast.dump(tree, indent=4)}")
# Remove the decorators from the AST, because the modified function will
# be passed to them anyway and we don't want them to be called twice.
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef):
node.decorator_list.clear()
# Make modifications to the AST here
#print(f"\n=== revised ===\n{ast.dump(tree, indent=4)}")
name = f.__code__.co_name
code = compile(tree, name, 'exec')
#print("\n=== byte code ===")
#dis.dis(code)
#print()
temp_globals = dict(globals())
exec(code, temp_globals)
return temp_globals[name]
Note: this decorator has not been tested much and has not been tested at all on methods or nested functions.
An interesting idea would be to for as_ast to return the AST. Then subsequent decorators could manipulate the AST. Lastly, a from_ast decorator could compile the modified AST into a function.
Related
Is it possible to use with statement in Python anonymous functions? For example, I have a function that writes 1 to a file:
def write_one(filename):
with open(filename, 'wt') as fp:
fp.write('1')
But this function is to be organized in a dict:
my_functions = {
....
}
Obviously I can write this statement to add this function to the dict:
my_functions['write_one'] = write_one
But the problem is the name write_one still exists in the current scope. How can I introduce an anonymous function without polluting the current namespace?
For simple functions, I can use lambda. For slightly complicated functions, I can return a tuple to execute multiple statements (to be precise, expressions). But I didn't find a way to cleverly use lambda so that it can work with with statements. If this is impossible, where it says so in its documentation?
The solution with a del write_one doesn't look good to me. I don't want this name to be introduced at all in the current namespace.
In a word, what I want is something like this:
my_functions['write_one'] = def(filename):
with open(filename, 'wt') as fp:
fp.write('1')
This is kind of awkward with Python's indentation-based rules, I know. But it does its job.
Lambda expressions are quite restricted in what they can do. From the docs
Note that functions created with lambda expressions cannot contain statements or annotations.
Just use a full function definition. If you really want to avoid polluting the namespace, just del the name afterwards.
Or if you simply want to avoid the module namespace from having a bunch of these small functions for code-completion purposes, use _ in the front of the function name.
If you truly want to avoid it, you could use the function constructor and dynamically compile code, etc. Or use some other kind of dynamic code execution, e.g. using eval or exec. But that is almost certainly not worth the trouble.
Especially if you can just del the name after you are done using it.
Or perhaps the best approach is to put all these functions in another namespace, like another module.
There are not many namespaces in Python - a function is basically the local one. That gives a good hint to a possible solution:
def gen_functions():
def f1(i):
print(i)
def f2(i):
print(i+1)
return f1,f2
my_funcs = dict(zip(('a','b'),gen_functions()))
my_funcs['a'](2)
my_funcs['b'](7)
f1(3)
Run this:
>py bla.py
2
8
Traceback (most recent call last):
File "bla.py", line 11, in <module>
f1(3)
NameError: name 'f1' is not defined
So you can make arbitrarily complex functions - to hide them from the global namespace you just enclose them in another function.
I have a body of python code that contains inline functions within functions. I'd like to unit test the make_exciting inner function, so I'm trying to figure out how to invoke it directly.
def say_something_exciting(name, phrase):
def make_exciting(phrase):
return phrase + "!"
return "%s says '%s'" % (name, make_exciting(phrase))
Function say_something_exciting is written at the top level of a .py file, and is not inside a class. The py file is in the org.something module. Tried:
Invoking the function directly via org.something.say_something_exciting.make_exciting("Hello") - error: 'function' object has no attribute 'make_exciting'
Inspecting dir(org.something.say_something_exciting) and org.something.say_something_exciting.__dict__ for any paths to traverse, didn't see make_exciting anywhere.
internal_function = org.something.say_something_exciting.__dict__.get('make_exciting'), but internal_function is None.
How can I access (unit test) this inner function? This may suggest what I'm asking isn't possible. I'm generally familiar with unit testing and how to use the unittest module; accessing the function is the problem. If it's not possible, how should I re-write this code to support testing (if other than promote the inner function to a top-level function)?. Thanks!
UPDATE: In Java I often give class methods default/package visibility so they're less visible but still accessible to unit tests, looking for a python equivalent.
I don't think you can.
You'll need to return the inner function, or bind it elsewhere. It gets ugly though. You can bind the declared function to somewhere else, from inside the function:
import testmodule
testmodule.testfunc = make_exciting
Or you can return it, for example when a keyword-argument is given, such as export=True.
I want to get the line number of a python function in the source code.
What i have in runtime is module, class, method objects
Had a look at inspect
inspect.getsourcelines(object)
which also gives line number in result.
I see that for methods with decorators, line no. returned from above inspect function points to the actual decorator's source code rather than desired function's source code.
So any way to workaround this?
(i understand that interpreter does something like wrapping function inside decorator in runtime, but i might be wrong)
There is no easy solution in the general case.
A decorator is a function that given a function returns a function, normally by "wrapping" it in a closure that performs the operation for which the decorator has been designed.
The file and line number information are not however in the function object itself and you cannot "fix" them by copying this information from the wrapped function to the wrapper. That data is instead contained in the code object of the function (available with .func_code), and it is shared among all closures you are going to create.
>>> def bar(x):
... def foo():
... return x
... return foo
...
>>> f1 = bar(1)
>>> f2 = bar(2)
>>> f1()
1
>>> f2()
2
>>> f1.func_code is f2.func_code
True
>>>
The wrapt module solves this problem by allowing you to write decorators which preserve the necessary metadata to find the source of a function as well as perform other introspection. It's like an improved functools.wraps.
I've got a decorator, but I'd like to store the original function in the namespace for future reference, such that I end up with a decorated and non-decorated version of the function.
I'm inserting that original function into the namespace like that:
def crossover(cross):
def ecspy_crossover(random, candidates, args):
# ... decorator magic
globals()['single_'+cross.func_name] = cross
However, when I import the module, the single_* functions do not show up.
When I launch pdb in this module, this function is present in the namespace...
[ there is no __all__statement in this module... ]
Any ideas?
Why not just add the original function as an attribute of the wrapped function? In fact that's exactly what functools.wraps does in Python 3.2, it stores the original function in the __wrapped__ attribute.
Regarding your question, I would guess that your problem is that you are adding the function into the global namespace of the module containing the decorator rather than that of the decorated function. You can use f.__globals__ to get at the correct global namespace.
I have a file that contains several python functions, each with some statements.
def func1():
codeX...
def func2():
codeY...
codeX and codeY can be multiple statements. I want to be able to parse the file, find a function by name, then evaluate the code in that function.
With the ast module, I can parse the file, find the FunctionDef objects, and get the list of Stmt objects, but how do I turn this into bytecode that I can pass to eval? Should I use the compile module, or the parser module instead?
Basically, the function defs are just used to create separate blocks of code. I want to be able to grab any block of code given the name and then execute that code in eval (providing my own local/global scope objects). If there is a better way to do this than what I described that would be helpful too.
Thanks
I want to be able to grab any block of code given the name and then execute that code ... (providing my own local/global scope objects).
A naive solution looks like this. This is based on the assumption that the functions don't all depend on global variables.
from file_that_contains_several_python_functions import *
Direction = some_value
func1()
func2()
func3()
That should do exactly what you want.
However, if all of your functions rely on global variables -- a design that calls to mind 1970's-era FORTRAN -- then you have to do something slightly more complex.
from file_that_contains_several_python_functions import *
Direction = some_value
func1( globals() )
func2( globals() )
func3( globals() )
And you have to rewrite all of your global-using functions like this.
def func1( context )
globals().update( context )
# Now you have access to all kinds of global variables
This seems ugly because it is. Functions which rely entirely on global variables are not really the best idea.
Using Python 2.6.4:
text = """
def fun1():
print 'fun1'
def fun2():
print 'fun2'
"""
import ast
tree = ast.parse(text)
# tree.body[0] contains FunctionDef for fun1, tree.body[1] for fun2
wrapped = ast.Interactive(body=[a.body[1]])
code = compile(wrapped, 'yourfile', 'single')
eval(code)
fun2() # prints 'fun2'
Take a look at grammar in ast doc: http://docs.python.org/library/ast.html#abstract-grammar. Top-level statement must be either Module, Interactive or Expression, so you need to wrap function def in one of those.
If you're using Python 2.6 or later, then the compile() function accepts AST objects in addition to source code.
>>> import ast
>>> a = ast.parse("print('hello world')")
>>> x = compile(a, "(none)", "exec")
>>> eval(x)
hello world
These modules have all been rearranged for Python 3.