I currently have the problem that I cant use inspect.getsource (OSError: could not get source code) inside code executed through exec().
For example when i feed the following code as a string into exec(code)
import inspect
def sample(p1):
print(p1)
return 1
print(inspect.getsource(sample))
So does anyone know how i could get the source inside the exec()?
I use python 3.8 64 Bit
Thank you for your answers.
Edit: First of all, thanks for the answers and the help to get the right formating for my question, but the function and the inspect.getsource() have to be inside the exec() string.
Regrettably, as I'm sure you've realised, this simply is not possible using exec...but it is possible using exec_module.
If you are not required to use exec specifically, this may work for you:
import tempfile
from importlib import util
import importlib.machinery
raw = """
def sample(p1):
print(p1)
return 1
import inspect
print(inspect.getsource(sample))
"""
# Create a temporary source code file
with tempfile.NamedTemporaryFile(suffix='.py') as tmp:
tmp.write(raw.encode())
tmp.flush()
# Now load that file as a module
spec = util.spec_from_file_location('tmp', tmp.name)
module = util.module_from_spec(spec)
spec.loader.exec_module(module)
# ...or, while the tmp file exists, you can query it externally
import inspect
print(inspect.getsource(module.sample))
The TLDR is, without a source file, it is just flat out not-possible to use getsource to read the source code of a function, because the implementation reads the source code from the file defined in f.__code__.co_filename on the function (read-only, so monkey patching is impossible).
However, if you write the source code to a temporary file, you can load that file as a module (exec_module is practically identical to the effect of exec, you just have to do a few other steps first).
Note: tmp.flush() <-- don't forget this, tmp.write() doesn't write until you flush, and you'll be 'loading' an empty file if you don't flush it first.
If your code is like this:
exec("""\
import inspect
def sample(p1):
print(p1)
return 1
print(inspect.getsource(sample))\
""")
And the exception is OSError: could not get source code then that probably means that inspect.getsource() requires a
file's source to inspect. exec is a dynamic execution of python code so the sample function that's being defined from exec is not on the file's source!
Something like this:
def sample(p1):
print(p1)
return 1
exec("""\
import inspect
print(inspect.getsource(sample))\
""")
would work because the function sample is defined in the source!
Hope that makes sense and answers your question!
~ PanTrakX
Related
I have a bytecode document that declares functions and a logo. I also have a .py file where I call the bytecode to output the logo and strings in the functions. How do I go about actually executing the bytecode? I was able to dissemble it and see the assembly code. How can I actually run it?
question.py
import dis
import logo
def work_here():
# execute the bytecode
def main():
work_here()
if __name__ == '__main__':
main()
Try something like:
import dis
code = 'some byte code'
b_code = dis.Bytecode(code)
exec(b.codeobj)
To import a .pyc file, you just do the same thing you do with a .py file: import spam will find an appropriately-placed spam.pyc (or rather, something like __pycache__/spam.cpython-36.pyc) just as it will find an appropriately-placed spam.py. Its top-level code gets run, any functions and classes get defined so you can call them, etc., exactly the same as with a .py file; the only difference is that there isn't source text to show for things like tracebacks or debugger stepping.
If you want to programmatically import a .pyc file by explicit path, or execute one without importing it, you again do the same thing you do with a .py file.
Look at the Examples in importlib. For example:
path = 'bytecoderepo/myfile.pyc'
spec = importlib.util.spec_from_file('myfile', path)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
And now, the code in bytecoderepo/myfile.pyc has been executed, and the resulting module is available in the variable mod, but it isn't in sys.modules or stored as a global.
If you actually need to dig into the .pyc format and, e.g., extract the bytecode of some function so you can exec it (or build a function object out of it) without executing the main module code, the details are only documented in the source, and subject to change between Python versions. Start with importlib; being able to (validate and) skip over the header and marshal.loads the body may be as far as you need to learn, but probably not (since ultimately, that's what the module loader already does for you in the sample code above, so if that's not good enough, you need to get deeper into the internals).
When writing throwaway scripts it's often needed to load a configuration file, image, or some such thing from the same directory as the script. Preferably this should continue to work correctly regardless of the directory the script is executed from, so we may not want to simply rely on the current working directory.
Something like this works fine if defined within the same file you're using it from:
from os.path import abspath, dirname, join
def prepend_script_directory(s):
here = dirname(abspath(__file__))
return join(here, s)
It's not desirable to copy-paste or rewrite this same function into every module, but there's a problem: if you move it into a separate library, and import as a function, __file__ is now referencing some other module and the results are incorrect.
We could perhaps use this instead, but it seems like the sys.argv may not be reliable either.
def prepend_script_directory(s):
here = dirname(abspath(sys.argv[0]))
return join(here, s)
How to write prepend_script_directory robustly and correctly?
I would personally just os.chdir into the script's directory whenever I execute it. It is just:
import os
os.chdir(os.path.split(__file__)[0])
However if you did want to refactor this thing into a library, you are in essence wanting a function that is aware of its caller's state. You thus have to make it
prepend_script_directory(__file__, blah)
If you just wanted to write
prepend_script_directory(blah)
you'd have to do cpython-specific tricks with stack frames:
import inspect
def getCallerModule():
# gets globals of module called from, and prints out __file__ global
print(inspect.currentframe().f_back.f_globals['__file__'])
I think the reason it doesn't smell right is that $PYTHONPATH (or sys.path) is the proper general mechanism to use.
You want pkg_resources
import pkg_resources
foo_fname = pkg_resources.resource_filename(__name__, "foo.txt")
I have a requirement where I need to parse the functions defined in a python file from another python file.
e.g. I have a python file with following contents:
a.py
import os, sys
def function1('text'):
pass
def function2('text'):
pass
Another file is:
b.py
interested_func = 'function2'
function_list = <code to fetch the functions defined in a.py>
if interested_func in function_list:
print 'match found'
How can I get the functions from a.py into b.py so that I can compare the same with the 'interested_func' data and can do specific task based on the match.
Please note that I have 100s of files with different functions defined inside them, so I do not want to import the file.
Please help, thanks in advance!
You should probably use the importlib module:
import importlib
obj = importlib.import_module(module)
print(dir(obj))
You can read more about importlib over in the Python docs.
If that doesn't work for you, then you'll probably want to look at some static code analysis tools such as pylint that might give you a clue into how to do this sort of thing. Another place to look would be to check out PyDev's source code and see how it does code analysis.
I'm writing a python application in which I want to make use of dynamic, one-time-runnable plugins.
By this I mean that at various times during the running of this application, it looks for python source files with special names in specific locations. If any such source file is found, I want my application to load it, run a pre-named function within it (if such a function exists), and then forget about that source file.
Later during the running of the application, that file might have changed, and I want my python application to reload it afresh, execute its method, and then forget about it, like before.
The standard import system keeps the module resident after the initial load, and this means that subsequent "import" or "__import__" calls won't reload the same module after its initial import. Therefore, any changes to the python code within this source file are ignored during its second through n-th imports.
In order for such packages to be loaded uniquely each time, I came up with the following procedure. It works, but it seems kind of "hacky" to me. Are there any more elegant or preferred ways of doing this? (note that the following is an over-simplified, illustrative example)
import sys
import imp
# The following module name can be anything, as long as it doesn't
# change throughout the life of the application ...
modname = '__whatever__'
def myimport(path):
'''Dynamically load python code from "path"'''
# get rid of previous instance, if it exists
try:
del sys.modules[modname]
except:
pass
# load the module
try:
return imp.load_source(modname, path)
except Exception, e:
print 'exception: {}'.format(e)
return None
mymod = myimport('/path/to/plugin.py')
if mymod is not None:
# call the plugin function:
try:
mymod.func()
except:
print 'func() not defined in plugin: {}'.format(path)
Addendum: one problem with this is that func() runs within a separate module context, and it has no access to any functions or variables within the caller's space. I therefore have to do inelegant things like the following if I
want func_one(), func_two() and abc to be accessible within the invocation
of func():
def func_one():
# whatever
def func_two():
# whatever
abc = '123'
# Load the module as shown above, but before invoking mymod.func(),
# the following has to be done ...
mymod.func_one = func_one
mymod.func_two = func_two
mymod.abc = abc
# This is a PITA, and I'm hoping there's a better way to do all of
# this.
Thank you very much.
I use the following code to do this sort of thing.
Note that I don't actually import the code as a module, but instead execute the code in a particular context. This lets me define a bunch of api functions automatically available to the plugins without users having to import anything.
def load_plugin(filename, context):
source = open(filename).read()
code = compile(source, filename, 'exec')
exec(code, context)
return context['func']
context = { 'func_one': func_one, 'func_two': func_two, 'abc': abc }
func = load_plugin(filename, context)
func()
This method works in python 2.6+ and python 3.3+
The approach you use is totally fine. For this question
one problem with this is that func() runs within a separate module context, and it has no access to any functions or variables within the caller's space.
It may be better to use execfile function:
# main.py
def func1():
print ('func1 called')
exec(open('trackableClass.py','r').read(),globals()) # this is similar to import except everything is done in the current module
#execfile('/path/to/plugin.py',globals()) # python 2 version
func()
Test it:
#/path/to/plugin.py
def func():
func1()
Result:
python main.py
# func1 called
One potential problem with this approach is namespace pollution because every file is run in the current namespace which increase the chance of name conflict.
I have a vaguely defined function of a class Graph of a module I call gt (it is graph-tool). so i declare g = gt.graph() then want to use g.degree_property_map but do not know how. Therefore I want to see where in code g.degree_property_map or in this case just the function, is defined. How can I find that? I'm working on command line on a vm.
Thanks
For reference the library in question is graph-tool - http://projects.skewed.de/graph-tool/
Also I am currently importing it using from graph_tool.all import * . that is of course somewhat of a problem.
You could use inspect.getsource(gt.Graph.degree_property_map). (You have to import inspect.)
Of course, what you pass into getsource() will change depending on how you imported Graph. So if you used from graphtools.all import *, you'd just need to use inspect.getsource(Graph.degree_property_map).
If you open interactive python (type python and hit ENTER on the command line), you should be able to run the command help(<graph's module name>), then, under the FILE section of the help documentation that is generated, you should see the absolute path to the code you are interested in.
For example, I just ran:
import numpy
help(numpy)
# Returned documentation containing:
# FILE
# /usr/lib/python2.7/dist-packages/numpy/__init__.py
Also,
import my_module # A module I just created that contains the "Line" class
help(my_module)
# Returned documentation containing:
# FILE
# /home/<my user name>/Programming/Python/my_module.py
If it is a normal function (not a builtin, ufunc, etc) you can try using the func_code attribute
For example:
>>> inspect.iscode
<function iscode at 0x02EAEF30>
>>> inspect.iscode.func_code
<code object iscode at 02EB2B60, file "C:\Python27\lib\inspect.py", line 209>
Never mind I just did help(graph_tool) and manually jumped through code. Thanks for the help though!