I'm writing a python application in which I want to make use of dynamic, one-time-runnable plugins.
By this I mean that at various times during the running of this application, it looks for python source files with special names in specific locations. If any such source file is found, I want my application to load it, run a pre-named function within it (if such a function exists), and then forget about that source file.
Later during the running of the application, that file might have changed, and I want my python application to reload it afresh, execute its method, and then forget about it, like before.
The standard import system keeps the module resident after the initial load, and this means that subsequent "import" or "__import__" calls won't reload the same module after its initial import. Therefore, any changes to the python code within this source file are ignored during its second through n-th imports.
In order for such packages to be loaded uniquely each time, I came up with the following procedure. It works, but it seems kind of "hacky" to me. Are there any more elegant or preferred ways of doing this? (note that the following is an over-simplified, illustrative example)
import sys
import imp
# The following module name can be anything, as long as it doesn't
# change throughout the life of the application ...
modname = '__whatever__'
def myimport(path):
'''Dynamically load python code from "path"'''
# get rid of previous instance, if it exists
try:
del sys.modules[modname]
except:
pass
# load the module
try:
return imp.load_source(modname, path)
except Exception, e:
print 'exception: {}'.format(e)
return None
mymod = myimport('/path/to/plugin.py')
if mymod is not None:
# call the plugin function:
try:
mymod.func()
except:
print 'func() not defined in plugin: {}'.format(path)
Addendum: one problem with this is that func() runs within a separate module context, and it has no access to any functions or variables within the caller's space. I therefore have to do inelegant things like the following if I
want func_one(), func_two() and abc to be accessible within the invocation
of func():
def func_one():
# whatever
def func_two():
# whatever
abc = '123'
# Load the module as shown above, but before invoking mymod.func(),
# the following has to be done ...
mymod.func_one = func_one
mymod.func_two = func_two
mymod.abc = abc
# This is a PITA, and I'm hoping there's a better way to do all of
# this.
Thank you very much.
I use the following code to do this sort of thing.
Note that I don't actually import the code as a module, but instead execute the code in a particular context. This lets me define a bunch of api functions automatically available to the plugins without users having to import anything.
def load_plugin(filename, context):
source = open(filename).read()
code = compile(source, filename, 'exec')
exec(code, context)
return context['func']
context = { 'func_one': func_one, 'func_two': func_two, 'abc': abc }
func = load_plugin(filename, context)
func()
This method works in python 2.6+ and python 3.3+
The approach you use is totally fine. For this question
one problem with this is that func() runs within a separate module context, and it has no access to any functions or variables within the caller's space.
It may be better to use execfile function:
# main.py
def func1():
print ('func1 called')
exec(open('trackableClass.py','r').read(),globals()) # this is similar to import except everything is done in the current module
#execfile('/path/to/plugin.py',globals()) # python 2 version
func()
Test it:
#/path/to/plugin.py
def func():
func1()
Result:
python main.py
# func1 called
One potential problem with this approach is namespace pollution because every file is run in the current namespace which increase the chance of name conflict.
Related
I want to be able to get the path to the file which is importing my python library. How can I do that?
For example:
The user creates a file at C:\Users\Bob\bobsproject\main.py. From within the library, I want to be able to get the path to the file, and read it as a txt. How can I do that?
If you want to get the name of the driver script that is (possibly indirectly) loading your library, you can use the fact that python runs a script under the name __main__. You can get it from sys.modules just like any other module and access its __file__ attribute if it exists:
import sys
try:
print(sys.modules['__main__'].__file__)
except KeyError:
print('libray not loaded from script')
except AttributeError:
print('script not loaded from file')
The KeyError is unlikely to ever occur (not even if you run the script with python -m), but it's useful to be safe. The AttributeError is much more likely, and can easily be demonstrated with something like python -c.
If you want something more complex, like the file containing the code that actually called your library function, you will likely have to use the inspect module or similar. This will be even less robust as a matter of course, but may still suit your needs:
import inspect
module = inspect.getmodule(inspect.stack()[1][0])
try:
print(module.__file__)
except AttributeError:
print(f'module "{module.__name__}" not loaded from file')
Notice that inspect.getmodule explicitly uses the word "guess" in its official documentation, while inspect.stack can be a fidgety beast sometimes.
Code for second part referenced from here: https://stackoverflow.com/a/1095621/2988730.
Remember that there are two options here. If you place this code directly in your library module, it will be executed exactly once, when the module is first imported. If you place it in a function that the user can call directly, you will see the printouts every time. If you place the second snippet it in a utility function that you then call from your public module functions, don't forget to increment the frame index to reflect that:
module = inspect.getmodule(inspect.stack()[2][0])
When writing throwaway scripts it's often needed to load a configuration file, image, or some such thing from the same directory as the script. Preferably this should continue to work correctly regardless of the directory the script is executed from, so we may not want to simply rely on the current working directory.
Something like this works fine if defined within the same file you're using it from:
from os.path import abspath, dirname, join
def prepend_script_directory(s):
here = dirname(abspath(__file__))
return join(here, s)
It's not desirable to copy-paste or rewrite this same function into every module, but there's a problem: if you move it into a separate library, and import as a function, __file__ is now referencing some other module and the results are incorrect.
We could perhaps use this instead, but it seems like the sys.argv may not be reliable either.
def prepend_script_directory(s):
here = dirname(abspath(sys.argv[0]))
return join(here, s)
How to write prepend_script_directory robustly and correctly?
I would personally just os.chdir into the script's directory whenever I execute it. It is just:
import os
os.chdir(os.path.split(__file__)[0])
However if you did want to refactor this thing into a library, you are in essence wanting a function that is aware of its caller's state. You thus have to make it
prepend_script_directory(__file__, blah)
If you just wanted to write
prepend_script_directory(blah)
you'd have to do cpython-specific tricks with stack frames:
import inspect
def getCallerModule():
# gets globals of module called from, and prints out __file__ global
print(inspect.currentframe().f_back.f_globals['__file__'])
I think the reason it doesn't smell right is that $PYTHONPATH (or sys.path) is the proper general mechanism to use.
You want pkg_resources
import pkg_resources
foo_fname = pkg_resources.resource_filename(__name__, "foo.txt")
I was wondering if there are any sort of python codeing etc that will displays the files imports/used locations in a python file?
Eg. TestA.py contains 3 files from 3 different directory
Import01 : /u/ext/TestA/UI
Import02 : /u/ext/TestA/src
Import03 : /user_data/setup/localImports
And hence, while executing the coding, it will displays the list of directories used in the python file?
I am asking as I am working on several (and maybe tons, in the future) scripts that are heavily involved in Maya, there are times in which when I located the path but they are the wrong ones (with same name) and is actually located in another path
Add this code to module
import inspect
frame = inspect.currentframe()
if frame and frame.f_back:
print('module "{}" is imported by "{}"'.format(__file__, frame.f_back.f_locals['__file__']))
If module_a.py contains the code above, and main.py imports it. the output is
module "/path/to/module_a.py" is imported by "/path/to/main.py"
As documented, this answer may not be an exact solution. Because if not supported, returns None.
CPython implementation detail: This function relies on Python stack frame support in the interpreter, which isn’t guaranteed to exist in all implementations of Python. If running in an implementation without Python stack frame support this function returns None.
At any point in when the code is running, you can determine the origin of a module by checking it's file attribute:
import sys
for name, each_mod in sys.modules.items():
try:
print name, each_mod.__file__
except AttributeError: # = built in module or dll
print "?"
To check the imports without running the code, you'd need do more complex analysis: Here's an example method that could probably be adapted to figure it out :http://www.tarind.com/depgraph.html
You could also create a custom ModuleFinder that printed out file sources as imports are processed. Something like this, which prints out name of py/pyc files when trying to load them.
import os
import sys
import imp
import ihooks
class ReportingFinder(object):
"""Find modules collected in a shelve archive."""
def __init__(self, path_entry):
self.path_entry = path_entry
if not os.path.isdir(path_entry):
raise ImportError
def find_module(self, fullname, path=None):
for suffix in (".py", ".pyc"):
test_path = os.path.join(self.path_entry, fullname + suffix)
print test_path
if os.path.exists(test_path):
print "attemnpting to load from %s" % test_path
return self
return None
def load_module(self, name):
stuff = imp.find_module(name)
return ihooks.FancyModuleLoader(verbose=1).load_module(name, stuff)
sys.path_hooks.insert(0, ReportingFinder)
HACK WARNING!!!! Please be aware this code is a quick diagnostic hack! Don't use it for production :) Among other flaws, it will print out py path names even if the code comes from the pyc, and it's dumb about packages -- I only provided it because it sounds like you're using single-file scripts rather than packages. It is handy for catching imported modules as they get loaded. It won't print out the names of zip files.
It sounds like the real problem is having too many competing paths: you should try to get down to as few as you can so that there are fewer suprises.
I imported a module as below:
filename = "email"
mymodule = __import__('actions.'+filename)
the problem I have with this is, that the file is immediatly executing, and I would much rather execute a specific function from the file (that way I can send variables through it).
I am basically working with plugins, so it works.
Edit:
for the time being, I am not concerned with whether or not the script executes when I add the line below:
mymodule = __import__('actions.'+filename)
but what I would like to work is when I add the line below, I would like the function to execute. But instead I get an error that the module dosn't have that function even though it exisits in the script.
mymodule.dosomething(n)
Edit:
I personally don't think that the function has anything to do with it but here is one python files that I am trying to open.
import webbrowser
def OpenEmail():
handle = webbrowser.get()
handle.open('http://gmail.google.com')
OpenEmail()
print "Your email has been opened"
The functions don't exist unless the module executes. You can't have it both ways. Perhaps you need to add a main stanza to the module.
The problem is, that you get the actions module returned. Try this:
mymodule = __import__('actions.'+filename)
for submodule in filename.split('.'):
mymodule = getattr(mymodule, submodule)
This happens when you try importing a submodule, i.e. module.something.somethingelse, you get module returned.
Summary: when a certain python module is imported, I want to be able to intercept this action, and instead of loading the required class, I want to load another class of my choice.
Reason: I am working on some legacy code. I need to write some unit test code before I start some enhancement/refactoring. The code imports a certain module which will fail in a unit test setting, however. (Because of database server dependency)
Pseduo Code:
from LegacyDataLoader import load_me_data
...
def do_something():
data = load_me_data()
So, ideally, when python excutes the import line above in a unit test, an alternative class, says MockDataLoader, is loaded instead.
I am still using 2.4.3. I suppose there is an import hook I can manipulate
Edit
Thanks a lot for the answers so far. They are all very helpful.
One particular type of suggestion is about manipulation of PYTHONPATH. It does not work in my case. So I will elaborate my particular situation here.
The original codebase is organised in this way
./dir1/myapp/database/LegacyDataLoader.py
./dir1/myapp/database/Other.py
./dir1/myapp/database/__init__.py
./dir1/myapp/__init__.py
My goal is to enhance the Other class in the Other module. But since it is legacy code, I do not feel comfortable working on it without strapping a test suite around it first.
Now I introduce this unit test code
./unit_test/test.py
The content is simply:
from myapp.database.Other import Other
def test1():
o = Other()
o.do_something()
if __name__ == "__main__":
test1()
When the CI server runs the above test, the test fails. It is because class Other uses LegacyDataLoader, and LegacydataLoader cannot establish database connection to the db server from the CI box.
Now let's add a fake class as suggested:
./unit_test_fake/myapp/database/LegacyDataLoader.py
./unit_test_fake/myapp/database/__init__.py
./unit_test_fake/myapp/__init__.py
Modify the PYTHONPATH to
export PYTHONPATH=unit_test_fake:dir1:unit_test
Now the test fails for another reason
File "unit_test/test.py", line 1, in <module>
from myapp.database.Other import Other
ImportError: No module named Other
It has something to do with the way python resolves classes/attributes in a module
You can intercept import and from ... import statements by defining your own __import__ function and assigning it to __builtin__.__import__ (make sure to save the previous value, since your override will no doubt want to delegate to it; and you'll need to import __builtin__ to get the builtin-objects module).
For example (Py2.4 specific, since that's what you're asking about), save in aim.py the following:
import __builtin__
realimp = __builtin__.__import__
def my_import(name, globals={}, locals={}, fromlist=[]):
print 'importing', name, fromlist
return realimp(name, globals, locals, fromlist)
__builtin__.__import__ = my_import
from os import path
and now:
$ python2.4 aim.py
importing os ('path',)
So this lets you intercept any specific import request you want, and alter the imported module[s] as you wish before you return them -- see the specs here. This is the kind of "hook" you're looking for, right?
There are cleaner ways to do this, but I'll assume that you can't modify the file containing from LegacyDataLoader import load_me_data.
The simplest thing to do is probably to create a new directory called testing_shims, and create LegacyDataLoader.py file in it. In that file, define whatever fake load_me_data you like. When running the unit tests, put testing_shims into your PYTHONPATH environment variable as the first directory. Alternately, you can modify your test runner to insert testing_shims as the first value in sys.path.
This way, your file will be found when importing LegacyDataLoader, and your code will be loaded instead of the real code.
The import statement just grabs stuff from sys.modules if a matching name is found there, so the simplest thing is to make sure you insert your own module into sys.modules under the target name before anything else tries to import the real thing.
# in test code
import sys
import MockDataLoader
sys.modules['LegacyDataLoader'] = MockDataLoader
import module_under_test
There are a handful of variations on the theme, but that basic approach should work fine to do what you describe in the question. A slightly simpler approach would be this, using just a mock function to replace the one in question:
# in test code
import module_under_test
def mock_load_me_data():
# do mock stuff here
module_under_test.load_me_data = mock_load_me_data
That simply replaces the appropriate name right in the module itself, so when you invoke the code under test, presumably do_something() in your question, it calls your mock routine.
Well, if the import fails by raising an exception, you could put it in a try...except loop:
try:
from LegacyDataLoader import load_me_data
except: # put error that occurs here, so as not to mask actual problems
from MockDataLoader import load_me_data
Is that what you're looking for? If it fails, but doesn't raise an exception, you could have it run the unit test with a special command line tag, like --unittest, like this:
import sys
if "--unittest" in sys.argv:
from MockDataLoader import load_me_data
else:
from LegacyDataLoader import load_me_data