I'm trying to use python's ast.parse to generate the AST of a function I defined, but the parse function takes in strings as parameter. How can I use it on a function / code object that is defined in the same file? For instance:
def foo(bar):
print(bar)
import ast
ast.parse(foo) # this doesn't work
Related
Is it possible to obtain a string containing the whole namespace from which a function was imported?
I would like some code like the following:
import math
import numpy
whole1 = get_whole_namespace(math.sin)
whole2 = get_whole_namespace(numpy.sin)
The resulting whole1 should be the string "math.sin", and the resulting whole2 should be the string "numpy.sin". Is there any standard library (or third party) functionality equivalent to the get_whole_namespace function in my example?
I originally thought that my desired result should be available in some dunder method or attribute. But the __name__ attribute of both math.sin and numpy.sin is the string "sin".
The math.sin function also has a __qualname__ attribute (which is the string "sin"), and a __module__ attribute (which is the string "math").
The numpy.sin function does not have either __qualname__ or __math__ attributes.
I am using Python 3.10.6 and NumPy 1.23.4.
I am attempting to implement a decorator that receives a function, parses it into an AST, eventually will do something to the AST, then reconstruct the original (or modified) function from the AST and return it. My current approach is, once I have the AST, compile it to a code <module> object, then get the constant in it with the name of the function, convert it to FunctionType, and return it. I have the following:
import ast, inspect, types
def as_ast(f):
source = inspect.getsource(f)
source = '\n'.join(source.splitlines()[1:]) # Remove as_ast decoration, pretend there can be no other decorations for now
tree = ast.parse(source)
print(ast.dump(tree, indent=4)) # Debugging log
# I would modify the AST somehow here
filename = f.__code__.co_filename
code = compile(tree, filename, 'exec')
func_code = next(
filter(
lambda x: isinstance(x, types.CodeType) and x.co_name == f.__name__,
code.co_consts)) # Get function object
func = types.FunctionType(func_code, {})
return func
#as_ast
def test(arg: int=4):
print(f'{arg=}')
Now, I would expect that calling test later in this source code will simply have the effect of calling test if the decorator were absent, which is what I observe, so long as I pass an argument for arg. However, if I pass no argument, instead of using the default I gave (4), it throws a TypeError for the missing argument. This makes it pretty clear that my approach for getting a callable function from the AST is not quite correct, as the default argument is not applied, and there may be other details that would slip through as it is now. How might I be able to correctly recreate the function from the AST? The way I currently go from the code module object to the function code object also seems... off intuitively, but I do not know how else one might achieve this.
The root node of the AST is a Module. Calling compile() on the AST, results in a code object for a module. Looking at the compiled code object returned using dis.dis(), from the standard library, shows the module level code builds the function and stores it in the global name space. So the easiest thing to do is exec the compiled code and then get the function from the 'global' environment of the exec call.
The AST node for the function includes a list of the decorators to be applied to the function. Any decorators that haven't been applied yet should be deleted from the list so they don't get applied twice (once when this decorator compiles the code, and once after this decorator returns). And delete this decorator from the list or you'll get an infinite recursion. The question is what to do with any decorators that came before this one. They have already run, but their result is tossed out because this decorator (as_ast) goes back to the source code. You can leave them in the list so they get rerun, or delete them if they don't matter.
In the code below, all the decorators are deleted from the parse tree, under the assumption that the as_ast decorator is applied first. The call to exec() uses a copy of globals() so the decorator has access to any other globally visible names (variables, functions, etc). See the docs for exec() for other considerations. Uncommented the print statements to see what is going on.
import ast
import dis
import inspect
import types
def as_ast(f):
source = inspect.getsource(f)
#print(f"=== source ===\n{source}")
tree = ast.parse(source)
#print(f"\n=== original ===\n{ast.dump(tree, indent=4)}")
# Remove the decorators from the AST, because the modified function will
# be passed to them anyway and we don't want them to be called twice.
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef):
node.decorator_list.clear()
# Make modifications to the AST here
#print(f"\n=== revised ===\n{ast.dump(tree, indent=4)}")
name = f.__code__.co_name
code = compile(tree, name, 'exec')
#print("\n=== byte code ===")
#dis.dis(code)
#print()
temp_globals = dict(globals())
exec(code, temp_globals)
return temp_globals[name]
Note: this decorator has not been tested much and has not been tested at all on methods or nested functions.
An interesting idea would be to for as_ast to return the AST. Then subsequent decorators could manipulate the AST. Lastly, a from_ast decorator could compile the modified AST into a function.
I've been given a Python code, together with the modules it imports. I would like to build a tree indicating which function calls what other functions. How can I do that?
you can use the ast (abstract syntax tree) module from the python standard library
# foo.py
def func(x):
print('hello')
parsing the file using ast.parse:
import ast
tree = ast.parse(open('foo.py').read())
print(ast.dump(tree)) # dumps the whole tree
# get the function from the tree body (i.e. from the file's content)
func = tree.body[0]
# get the function argument names
arguments = [a.arg for a in func.args.args]
print('the functions is: %s(%s)' % (func.name, ', '.join(arguments)))
outputs:
"Module(body=[FunctionDef(name='func', args=arguments(args=[arg(arg='x', annotation=None)], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[]), body=[Expr(value=Call(func=Name(id='print', ctx=Load()), args=[Str(s='hello')], keywords=[]))], decorator_list=[], returns=None)])"
the functions is: func(x)
You should begin from the main function of the program and at the first layer link all functions that are called from the main this would provide a start point and then you can link all the functions below it.
This is a example in lxml doc:
>>> regexpNS = "http://exslt.org/regular-expressions"
>>> find = etree.XPath("//*[re:test(., '^abc$', 'i')]",
... namespaces={'re':regexpNS})
>>> root = etree.XML("<root><a>aB</a><b>aBc</b></root>")
>>> print(find(root)[0].text)
aBc
I want to import re:test() function to default namespace, so that I can call it without prefix re:. How can I do it? Thanks!
You can put a function in the empty function namespace:
functionNS = etree.FunctionNamespace(None)
functionNS['test'] = lambda context, nodes, *args: print(context, nodes, args)
By doing so, the new test function is already registered with the empty namespace prefix, that means you can use it like this:
root.xpath("//*[test(., 'arg1', 'arg2')]")
Unfortunately the function that is called for "{http://exslt.org/regular-expressions}test" isn't available from python, only from within the lxml extension implemented in C, so you can't simply assign it to functionNS['test'].
That means you'd need to reimplement it in python to assign it to the empty function namespace...
If that's not worth the trouble for you to spare you typing three characters, you could use this trick to make the re prefix for the namespace global:
etree.FunctionNamespace("http://exslt.org/regular-expressions").prefix = 're'
Then at least you don't need to pass the namespaces dict for each xpath expression.
How can I get the int(), float(), dict(), etc. callables from their names? For example, I'm trying to save Python values to xml and storing the variable type as a string. Is there a way to get the callable from the string when converting from string back to the Python type?
Normally I would do something like getattr(myobj, 'str'), but there is no module to use as the first argument for these built-in conversion functions. I've also tried getattr(object, 'str'), but this doesn't work either since these functions are not part of the base 'object' type, merely globals to the language.
Normally I would do something like getattr(myobj, 'str'), but there is no module to use as the first argument for these built-in conversion functions.
Wrong, there is:
import __builtin__
my_str = getattr(__builtin__, "str")
(In Python 3.x: import builtins)
You don't need to import anything
vars(__builtins__)['dict']
vars(__builtins__)['float']
vars(__builtins__)['int']
etc.
One quick way is to invoke it from the __builtin__ module. For example
>>> import __builtin__
>>> __builtin__.__dict__['str'](10)
'10'