Inside a customised python interpreter, if run single command, I can see the command output.
But if I put commands into a file and run with execfile('file.py'), except print command, I can't see anything. It's hard to debug.
How can I check command output when run with execfile command?
Tried below commands to confirm the python version.
import sys
print sys.version_info[0],sys.version_info[1]
2 7
'command output' is not really a valid term. The Python interactive interpreter will echo the result of any expression statement, unless that expression produced None. An expression statement is a top-level statement in a module body that only contains a single expression (as opposed to other simple statements such as assignment or import or compound statements such as if or while. From the linked documentation:
In interactive mode, if the value is not None, it is converted to a string using the built-in repr() function and the resulting string is written to standard output [...] on a line by itself. (Expression statements yielding None are not written, so that procedure calls do not cause any output.)
Doing so in a script would not be practical; you'd never be able to run any expression statement without the user of the script having to then decipher every little thing printed to the screen. Scripts usually want to control what is printed (to the point of not printing anything at all, but using logging instead, for example).
If you are using execfile() to run scripts, you could switch to parsing the script into an abstract syntax tree, at which point you can transform the expression statement nodes to add printing. The resulting transformed AST can be fed directly to the compile() function, and the exec() function can take the output of the compilation step.
I'd use an extra global function passed to exec() here:
import ast
class ExprStatementPrinter(ast.NodeTransformer):
def visit_Expr(self, node): # expression statement node
# call the function __print_expr_result__, passing in the expression
# contents; replace the original expression contents with that call.
new_value = ast.Call(
ast.Name('__print_expr_result__', ast.Load()), # load as a global
[node.value], [], None, None) # node.value is the only argument
# to be able to execute, we need to set the lineno and coll_offset; just
# reuse the values of the expr node, then fix up any children
node.value = ast.copy_location(new_value, node.value)
return ast.fix_missing_locations(node)
def print_expr_result(result):
if result is not None:
print repr(result)
def execfile(filename):
with open(filename) as source:
tree = ast.parse(source.read())
ExprStatementPrinter().visit(tree) # updates the tree in-place
codeobj = compile(tree, filename, 'exec')
exec(codeobj, {'__print_expr_result__': print_expr_result})
I used a double-underscore 'system' defined name for the print function to avoid collisions with anything the script might have defined.
Demo:
>>> example = '''\
... import sys
... sys.version_info[0],sys.version_info[1]
... 'Hello world!'
... None # should not be printed
... '''
>>> exec(example) # no output
>>> tree = ast.parse(example)
>>> ExprStatementPrinter().visit(tree)
<_ast.Module object at 0x1024c14d0>
>>> codeobj = compile(tree, '', 'exec')
>>> exec(codeobj, {'__print_expr_result__': print_expr_result})
(2, 7)
'Hello world!'
Note: this only applies to the top-level file you execute. Any imported modules are still going to be run without transformation (just like they would in the Python interactive interpreter). You'd have to hook into the __import__ global to intercept the normal importing machinery.
Related
I am attempting to implement a decorator that receives a function, parses it into an AST, eventually will do something to the AST, then reconstruct the original (or modified) function from the AST and return it. My current approach is, once I have the AST, compile it to a code <module> object, then get the constant in it with the name of the function, convert it to FunctionType, and return it. I have the following:
import ast, inspect, types
def as_ast(f):
source = inspect.getsource(f)
source = '\n'.join(source.splitlines()[1:]) # Remove as_ast decoration, pretend there can be no other decorations for now
tree = ast.parse(source)
print(ast.dump(tree, indent=4)) # Debugging log
# I would modify the AST somehow here
filename = f.__code__.co_filename
code = compile(tree, filename, 'exec')
func_code = next(
filter(
lambda x: isinstance(x, types.CodeType) and x.co_name == f.__name__,
code.co_consts)) # Get function object
func = types.FunctionType(func_code, {})
return func
#as_ast
def test(arg: int=4):
print(f'{arg=}')
Now, I would expect that calling test later in this source code will simply have the effect of calling test if the decorator were absent, which is what I observe, so long as I pass an argument for arg. However, if I pass no argument, instead of using the default I gave (4), it throws a TypeError for the missing argument. This makes it pretty clear that my approach for getting a callable function from the AST is not quite correct, as the default argument is not applied, and there may be other details that would slip through as it is now. How might I be able to correctly recreate the function from the AST? The way I currently go from the code module object to the function code object also seems... off intuitively, but I do not know how else one might achieve this.
The root node of the AST is a Module. Calling compile() on the AST, results in a code object for a module. Looking at the compiled code object returned using dis.dis(), from the standard library, shows the module level code builds the function and stores it in the global name space. So the easiest thing to do is exec the compiled code and then get the function from the 'global' environment of the exec call.
The AST node for the function includes a list of the decorators to be applied to the function. Any decorators that haven't been applied yet should be deleted from the list so they don't get applied twice (once when this decorator compiles the code, and once after this decorator returns). And delete this decorator from the list or you'll get an infinite recursion. The question is what to do with any decorators that came before this one. They have already run, but their result is tossed out because this decorator (as_ast) goes back to the source code. You can leave them in the list so they get rerun, or delete them if they don't matter.
In the code below, all the decorators are deleted from the parse tree, under the assumption that the as_ast decorator is applied first. The call to exec() uses a copy of globals() so the decorator has access to any other globally visible names (variables, functions, etc). See the docs for exec() for other considerations. Uncommented the print statements to see what is going on.
import ast
import dis
import inspect
import types
def as_ast(f):
source = inspect.getsource(f)
#print(f"=== source ===\n{source}")
tree = ast.parse(source)
#print(f"\n=== original ===\n{ast.dump(tree, indent=4)}")
# Remove the decorators from the AST, because the modified function will
# be passed to them anyway and we don't want them to be called twice.
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef):
node.decorator_list.clear()
# Make modifications to the AST here
#print(f"\n=== revised ===\n{ast.dump(tree, indent=4)}")
name = f.__code__.co_name
code = compile(tree, name, 'exec')
#print("\n=== byte code ===")
#dis.dis(code)
#print()
temp_globals = dict(globals())
exec(code, temp_globals)
return temp_globals[name]
Note: this decorator has not been tested much and has not been tested at all on methods or nested functions.
An interesting idea would be to for as_ast to return the AST. Then subsequent decorators could manipulate the AST. Lastly, a from_ast decorator could compile the modified AST into a function.
When defining code dynamically in Python (e.g. through exec or loading it from some other medium other than import), I am unable to get to the source of the defined function.
inspect.getsource seems to look for a loaded module from where it was loaded.
import inspect
code = """
def my_function():
print("Hello dears")
"""
exec(code)
my_function() #Works, as expected
print(inspect.getsource(my_function)) ## Fails with OSError('could not get source code')
Is there any other way to get at the source of a dynamically interpreted function (or other object, for that matter)?
Is there any other way to get at the source of a dynamically interpreted function (or other object, for that matter)?
One option would be to dump the source to a file and exec from there, though that litters your filesystem with garbage you need to cleanup.
A somewhat less reliable but less garbagey alternative would be to rebuild the source (-ish) from the bytecode, using astor.to_source() for instance. It will give you a "corresponding" source but may alter formatting or lose metadata compared to the original.
The simplest would be to simply attach your original source to the created function object:
code = """
def my_function():
print("Hello dears")
"""
exec(code)
my_function.__source__ = code # has nothing to do with getsource
One more alternative (though probably not useful here as I assume you want the body to be created dynamically from a template for instance) would be to swap the codeobject for one you've updated with the correct / relevant firstlineno (and optionally filename though you can set that as part of the compile statement). That's only useful if for some weird reason you have your python code literally embedded in an other file but can't or don't want to extract it to its own module for a normal evaluation.
You can do it almost like below
import inspect
source = """
def foo():
print("Hello World")
"""
file_name ='/tmp/foo.py' # you can use any hash_function
with open(file_name, 'w') as f:
f.write(source)
code = compile(source, file_name, 'exec')
exec(code)
foo() # Works, as expected
print(inspect.getsource(foo))
If I have a string of Python code, how do I tell if it is valid, i.e., if entered at the Python prompt, it would raise a SyntaxError or not? I thought that using compiler.parse would work, but apparently that module has been removed in Python 3. Is there a way to do it that also works in Python 3. Obviously, I don't want to execute the code, just check its syntax.
Use ast.parse:
import ast
def is_valid_python(code):
try:
ast.parse(code)
except SyntaxError:
return False
return True
>>> is_valid_python('1 // 2')
True
>>> is_valid_python('1 /// 2')
False
The compiler module is now a built-in.
compile(source, filename, mode[, flags[, dont_inherit]])
Compile the source into a code or AST object. Code objects can be
executed by an exec statement or evaluated by a call to eval(). source
can either be a string or an AST object. Refer to the ast module
documentation for information on how to work with AST objects.
The AST parser is now a seperate module.
ast.parse(expr, filename='<unknown>', mode='exec')
Parse an expression into an AST node. Equivalent to compile(expr, filename, mode, ast.PyCF_ONLY_AST).
Here is a python module,
#a.py
def bar():
print x #x not defined, apparently will result in an error
def foo():
pass
if __name__ == '__main__':
foo()
The above module can be run ($ python a.py) without any error. Why? Just because bar is not used in __main__?
But bar's definition is executed, isn't it?
Yes, bar's definition is executed, but the definition doesn't contain an error. It is valid Python to define a function that refers to globals that don't yet exist, so long as they exist when the function is called. Consider this:
def bar():
print x
x = 10
if __name__ == '__main__':
bar()
This does not result in an error. And this is only sensible, since even if x exists at the time the function is defined, there is nothing to stop you using del on it later. The point when x needs to be defined is when bar is called (if ever), not when bar is defined.
If Python did work the way you are suggesting, then it would be impossible to define mutually recursive functions without weird hacks like temporarily binding one name to None, then defining both functions.
EDIT: To elaborate on Ignacio's answer to the Alcott's question in the comments, yes syntax errors are caught before the function can be executed, but they're actually caught before it can be defined either.
When Python loads a file, it parses the entire contents into statements and then executes the statements one at a time. A syntax error means it was unable to successfully figure out what statements the file contains, so it can't execute anything. So the error will occur when the file is loaded, which means either when you directly run it with the interpreter, or when you import it.
This pre-processing step is known as "compile time", even though Python is not normally thought of as a compiled language; it is technically compiled to a byte code format, but this is almost entirely uninteresting because the byte code pretty much just directly represents the source code statements.
Python resolves name lookups at runtime.
def bar():
print x
x = 3
bar()
It's true that the definition of bar is executed when you run the script. However, Python can't determine whether a global variable named x actually exists until the whole script is run.
For example, you could do:
if __name__ == '__main__':
if random.random() < 0.5:
x = 5
foo()
The compiler wouldn't be able to determine at compile time whether x is going to exist or not.
I have a file that contains several python functions, each with some statements.
def func1():
codeX...
def func2():
codeY...
codeX and codeY can be multiple statements. I want to be able to parse the file, find a function by name, then evaluate the code in that function.
With the ast module, I can parse the file, find the FunctionDef objects, and get the list of Stmt objects, but how do I turn this into bytecode that I can pass to eval? Should I use the compile module, or the parser module instead?
Basically, the function defs are just used to create separate blocks of code. I want to be able to grab any block of code given the name and then execute that code in eval (providing my own local/global scope objects). If there is a better way to do this than what I described that would be helpful too.
Thanks
I want to be able to grab any block of code given the name and then execute that code ... (providing my own local/global scope objects).
A naive solution looks like this. This is based on the assumption that the functions don't all depend on global variables.
from file_that_contains_several_python_functions import *
Direction = some_value
func1()
func2()
func3()
That should do exactly what you want.
However, if all of your functions rely on global variables -- a design that calls to mind 1970's-era FORTRAN -- then you have to do something slightly more complex.
from file_that_contains_several_python_functions import *
Direction = some_value
func1( globals() )
func2( globals() )
func3( globals() )
And you have to rewrite all of your global-using functions like this.
def func1( context )
globals().update( context )
# Now you have access to all kinds of global variables
This seems ugly because it is. Functions which rely entirely on global variables are not really the best idea.
Using Python 2.6.4:
text = """
def fun1():
print 'fun1'
def fun2():
print 'fun2'
"""
import ast
tree = ast.parse(text)
# tree.body[0] contains FunctionDef for fun1, tree.body[1] for fun2
wrapped = ast.Interactive(body=[a.body[1]])
code = compile(wrapped, 'yourfile', 'single')
eval(code)
fun2() # prints 'fun2'
Take a look at grammar in ast doc: http://docs.python.org/library/ast.html#abstract-grammar. Top-level statement must be either Module, Interactive or Expression, so you need to wrap function def in one of those.
If you're using Python 2.6 or later, then the compile() function accepts AST objects in addition to source code.
>>> import ast
>>> a = ast.parse("print('hello world')")
>>> x = compile(a, "(none)", "exec")
>>> eval(x)
hello world
These modules have all been rearranged for Python 3.