I'd like to use a plugin system within my code. I've looked around for simple (yet powerful) python modules, and found Yapsy (among some others).
It is quite what I was looking for, but the way Yapsy discover plugins is not very flexible and require a plugin info file to be present. I'd like to get rid of it, without having to fork the code (if I start relying on Yapsy, I want to be sure I'll get all the updates from it without having to refork it each time).
I came out with this quick and dirty solution which is working fine, but do not improve the flexibility of the "discovering" process:
#!/usr/bin/env python
import os
import logging
from cStringIO import StringIO
from yapsy.PluginManager import PluginManager
from yapsy.IPlugin import IPlugin
from yapsy.PluginInfo import PluginInfo
class MyPluginManager(PluginManager):
"""
My attempt to get rid of the plugin info file...
"""
def __init__(self,
categories_filter={"Default":IPlugin},
directories_list=None,
plugin_info_ext="plugin.py"):
"""
Initialize the mapping of the categories and set the list of
directories where plugins may be. This can also be set by
direct call the methods:
- ``setCategoriesFilter`` for ``categories_filter``
- ``setPluginPlaces`` for ``directories_list``
- ``setPluginInfoExtension`` for ``plugin_info_ext``
You may look at these function's documentation for the meaning
of each corresponding arguments.
"""
self.setPluginInfoClass(PluginInfo)
self.setCategoriesFilter(categories_filter)
self.setPluginPlaces(directories_list)
self.setPluginInfoExtension(plugin_info_ext)
def _gatherCorePluginInfo(self, directory, filename):
"""
Gather the core information (name, and module to be loaded)
about a plugin described by it's info file (found at
'directory/filename').
Return an instance of ``self.plugin_info_cls`` and the
config_parser used to gather the core data *in a tuple*, if the
required info could be localised, else return ``(None,None)``.
.. note:: This is supposed to be used internally by subclasses
and decorators.
"""
# now we can consider the file as a serious candidate
candidate_infofile = os.path.join(directory,filename)
print candidate_infofile
# My hack : just create a StringIO file with basic plugin info
_fname = filename.rstrip(".py")
_file = StringIO()
_file.write("""[Core]
Name = %s
Module = %s
""" % (_fname, _fname))
_file.seek(0)
# parse the information file to get info about the plugin
name,moduleName,config_parser = self._getPluginNameAndModuleFromStream(_file, candidate_infofile)
print name, moduleName, config_parser
if (name,moduleName,config_parser)==(None,None,None):
return (None,None)
# start collecting essential info
plugin_info = self._plugin_info_cls(name,os.path.join(directory,moduleName))
return (plugin_info,config_parser)
This hack just assumes that the plugin has an extension ".plugin.py" (or ".plugin" for directory, but I did not test it). Then I create a cSringIO file to fool Yapsy and make it think he found a plugin info file. (One can still provide additional informations in the plugin by setting the proper variables: author, description...).
I'm wondering if there is a better way or if people have already done that. This hack is clearly too rough to be really useful, and I'd like to have something more flexible: a plugin may be discovered by its plugin info file (as in the original code) or by a pattern for the plugin name (probably using re, allowing the usage of prefix, suffix...). As far as I see, having these ideas implemented would require a much more complex hack than what I've already done...
Ok, I've implemented a fork of the Yapsy plugin manager, and am actually in touch with the author of the package. As soon as the documentation and tests are done, I think this may be included in the next release of Yapsy.
Related
I am trying to upgrade a 10 year old event listener that I didn't write from Python 2.7 to python 3.7. The basic issue I'm running into is the way the original script was importing its plugins. The idea behind the original script was that any python file put into a "plugins" folder, with a "registerCallbacks" function inside it would auto-load itself into the event listener and run. It's been working great for lots of studios for years, but Python 3.7 is not liking it at all.
The folder structure for the original code is as follows:
EventListenerPackage
src
event_listener.py
plugins
plugin_1.py
plugin_2.py
From this, you can see that both the event listener and the plugins are held in folders that are parallel to each other, not nested.
The original code read like this:
# Python 2.7 implementation
import imp
class Plugin(object):
def __init__(self, path):
self._path = 'c:/full/path/to/EventListenerPackage/plugins/plugin_1.py'
self._pluginName = 'plugin_1'
def load(self):
try:
plugin = imp.load_source(self._pluginName, self._path)
except:
self._active = False
self.logger.error('Could not load the plugin at %s.\n\n%s', self._path, traceback.format_exc())
return
regFunc = getattr(plugin, 'registerCallbacks', None)
Due to the nature of the changes (as I understand them) in the way that Python 3 imports modules, none of the other message boards seem to be getting me to the answer.
I have tried several different approaches, the best so far being:
How to import a module given the full path?
I've tried several different methods, including adding the full path to the sys.path, but I always get "ModuleNotFoundError".
Here is roughly where I'm at now.
import importlib.util
import importlib.abc
import importlib
class Plugin(object):
def __init__(self, path):
self._path = 'c:/full/path/to/EventListenerPackage/plugins/plugin_1.py'
self._pluginName = 'plugin_1'
def load(self):
try:
spec = importlib.util.spec_from_file_location('plugins.%s' % self._pluginName, self._path)
plugin = importlib.util.module_from_spec(spec)
# OR I HAVE ALSO TRIED
plugin = importlib.import_module(self._path)
except:
self._active = False
self.logger.error('Could not load the plugin at %s.\n\n%s', self._path, traceback.format_exc())
return
regFunc = getattr(plugin, 'registerCallbacks', None)
Does anyone have any insights into how I can actually import these modules with the given folder structure?
Thanks in advance.
You're treating plugins like it's a package. It's not. It's just a folder you happen to have your plugin source code in.
You need to stop putting plugins. in front of the module name argument in spec_from_file_location:
spec = importlib.util.spec_from_file_location(self._pluginName, self._path)
Aside from that, you're also missing the part that actually executes the module's code:
spec.loader.exec_module(plugin)
Depending on how you want your plugin system to interact with regular modules, you could alternatively just stick the plugin directory onto the import path:
sys.path.append(plugin_directory)
and then import your plugins with import or importlib.import_module. Probably importlib.import_module, since it sounds like the plugin loader won't know plugin names in advance:
plugin = importlib.import_module(plugin_name)
If you do this, plugins will be treated as ordinary modules, with consequences like not being able to safely pick a plugin name that collides with an installed module.
As an entirely separate issue, it's pretty weird that your Plugin class completely ignores its path argument.
Is it possible to import a Python module from over the internet using the http(s), ftp, smb or any other protocol? If so, how? If not, why?
I guess it's about making Python use more the one protocol(reading the filesystem) and enabling it to use others as well. Yes I agree it would be many folds slower, but some optimization and larger future bandwidths would certainly balance it out.
E.g.:
import site
site.addsitedir("https://bitbucket.org/zzzeek/sqlalchemy/src/e8167548429b9d4937caaa09740ffe9bdab1ef61/lib")
import sqlalchemy
import sqlalchemy.engine
Another version,
I like this answer. when applied it, i simplified it a bit - similar to the look and feel of javascript includes over HTTP.
This is the result:
import os
import imp
import requests
def import_cdn(uri, name=None):
if not name:
name = os.path.basename(uri).lower().rstrip('.py')
r = requests.get(uri)
r.raise_for_status()
codeobj = compile(r.content, uri, 'exec')
module = imp.new_module(name)
exec (codeobj, module.__dict__)
return module
Usage:
redisdl = import_cdn("https://raw.githubusercontent.com/p/redis-dump-load/master/redisdl.py")
# Regular usage of the dynamic included library
json_text = redisdl.dumps(host='127.0.0.1')
Tip - place the import_cdn function in a common library, this way you could re-use this small function
Bear in mind It will fail when no connectivity to that file over http
In principle, yes, but all of the tools built-in which kinda support this go through the filesystem.
To do this, you're going to have to load the source from wherever, compile it with compile, and exec it with the __dict__ of a new module. See below.
I have left the actually grabbing text from the internet, and parsing uris etc as an exercise for the reader (for beginners: I suggest using requests)
In pep 302 terms, this would be the implementation behind a loader.load_module function (the parameters are different). See that document for details on how to integrate this with the import statement.
import imp
modulesource = 'a=1;b=2' #load from internet or wherever
def makemodule(modulesource,sourcestr='http://some/url/or/whatever',modname=None):
#if loading from the internet, you'd probably want to parse the uri,
# and use the last part as the modulename. It'll come up in tracebacks
# and the like.
if not modname: modname = 'newmodulename'
#must be exec mode
# every module needs a source to be identified, can be any value
# but if loading from the internet, you'd use the URI
codeobj = compile(modulesource, sourcestr, 'exec')
newmodule = imp.new_module(modname)
exec(codeobj,newmodule.__dict__)
return newmodule
newmodule = makemodule(modulesource)
print(newmodule.a)
At this point newmodule is already a module object in scope, so you don't need to import it or anything.
modulesource = '''
a = 'foo'
def myfun(astr):
return a + astr
'''
newmod = makemodule(modulesource)
print(newmod.myfun('bat'))
Ideone here: http://ideone.com/dXGziO
Tested with python 2, should work with python 3 (textually compatible print used;function-like exec syntax used).
This seems to be a use case for a self-written import hook. Look up in PEP 302 how exactly they work.
Essentially, you'll have to provide a finder object which, in turn, provides a loader object. I don't understand the process at the very first glance (otherwise I'd be more explicit), but the PEP contains all needed details for implementing the stuff.
As glglgl's has it this import hook has been implemented for Python2 and Python3 in a module called httpimport.
It uses a custom finder/loader object to locate resources using HTTP/S.
Additionally, the import_cdn function in Jossef Harush's answer is almost identically implemented in httpimport's github_repo, and bitbucket_repo functions.
#Marcin's answer contains a good portion of the code of the httpimport's loader class.
I was wondering if there are any sort of python codeing etc that will displays the files imports/used locations in a python file?
Eg. TestA.py contains 3 files from 3 different directory
Import01 : /u/ext/TestA/UI
Import02 : /u/ext/TestA/src
Import03 : /user_data/setup/localImports
And hence, while executing the coding, it will displays the list of directories used in the python file?
I am asking as I am working on several (and maybe tons, in the future) scripts that are heavily involved in Maya, there are times in which when I located the path but they are the wrong ones (with same name) and is actually located in another path
Add this code to module
import inspect
frame = inspect.currentframe()
if frame and frame.f_back:
print('module "{}" is imported by "{}"'.format(__file__, frame.f_back.f_locals['__file__']))
If module_a.py contains the code above, and main.py imports it. the output is
module "/path/to/module_a.py" is imported by "/path/to/main.py"
As documented, this answer may not be an exact solution. Because if not supported, returns None.
CPython implementation detail: This function relies on Python stack frame support in the interpreter, which isn’t guaranteed to exist in all implementations of Python. If running in an implementation without Python stack frame support this function returns None.
At any point in when the code is running, you can determine the origin of a module by checking it's file attribute:
import sys
for name, each_mod in sys.modules.items():
try:
print name, each_mod.__file__
except AttributeError: # = built in module or dll
print "?"
To check the imports without running the code, you'd need do more complex analysis: Here's an example method that could probably be adapted to figure it out :http://www.tarind.com/depgraph.html
You could also create a custom ModuleFinder that printed out file sources as imports are processed. Something like this, which prints out name of py/pyc files when trying to load them.
import os
import sys
import imp
import ihooks
class ReportingFinder(object):
"""Find modules collected in a shelve archive."""
def __init__(self, path_entry):
self.path_entry = path_entry
if not os.path.isdir(path_entry):
raise ImportError
def find_module(self, fullname, path=None):
for suffix in (".py", ".pyc"):
test_path = os.path.join(self.path_entry, fullname + suffix)
print test_path
if os.path.exists(test_path):
print "attemnpting to load from %s" % test_path
return self
return None
def load_module(self, name):
stuff = imp.find_module(name)
return ihooks.FancyModuleLoader(verbose=1).load_module(name, stuff)
sys.path_hooks.insert(0, ReportingFinder)
HACK WARNING!!!! Please be aware this code is a quick diagnostic hack! Don't use it for production :) Among other flaws, it will print out py path names even if the code comes from the pyc, and it's dumb about packages -- I only provided it because it sounds like you're using single-file scripts rather than packages. It is handy for catching imported modules as they get loaded. It won't print out the names of zip files.
It sounds like the real problem is having too many competing paths: you should try to get down to as few as you can so that there are fewer suprises.
Is it possible to import a Python module from over the internet using the http(s), ftp, smb or any other protocol? If so, how? If not, why?
I guess it's about making Python use more the one protocol(reading the filesystem) and enabling it to use others as well. Yes I agree it would be many folds slower, but some optimization and larger future bandwidths would certainly balance it out.
E.g.:
import site
site.addsitedir("https://bitbucket.org/zzzeek/sqlalchemy/src/e8167548429b9d4937caaa09740ffe9bdab1ef61/lib")
import sqlalchemy
import sqlalchemy.engine
Another version,
I like this answer. when applied it, i simplified it a bit - similar to the look and feel of javascript includes over HTTP.
This is the result:
import os
import imp
import requests
def import_cdn(uri, name=None):
if not name:
name = os.path.basename(uri).lower().rstrip('.py')
r = requests.get(uri)
r.raise_for_status()
codeobj = compile(r.content, uri, 'exec')
module = imp.new_module(name)
exec (codeobj, module.__dict__)
return module
Usage:
redisdl = import_cdn("https://raw.githubusercontent.com/p/redis-dump-load/master/redisdl.py")
# Regular usage of the dynamic included library
json_text = redisdl.dumps(host='127.0.0.1')
Tip - place the import_cdn function in a common library, this way you could re-use this small function
Bear in mind It will fail when no connectivity to that file over http
In principle, yes, but all of the tools built-in which kinda support this go through the filesystem.
To do this, you're going to have to load the source from wherever, compile it with compile, and exec it with the __dict__ of a new module. See below.
I have left the actually grabbing text from the internet, and parsing uris etc as an exercise for the reader (for beginners: I suggest using requests)
In pep 302 terms, this would be the implementation behind a loader.load_module function (the parameters are different). See that document for details on how to integrate this with the import statement.
import imp
modulesource = 'a=1;b=2' #load from internet or wherever
def makemodule(modulesource,sourcestr='http://some/url/or/whatever',modname=None):
#if loading from the internet, you'd probably want to parse the uri,
# and use the last part as the modulename. It'll come up in tracebacks
# and the like.
if not modname: modname = 'newmodulename'
#must be exec mode
# every module needs a source to be identified, can be any value
# but if loading from the internet, you'd use the URI
codeobj = compile(modulesource, sourcestr, 'exec')
newmodule = imp.new_module(modname)
exec(codeobj,newmodule.__dict__)
return newmodule
newmodule = makemodule(modulesource)
print(newmodule.a)
At this point newmodule is already a module object in scope, so you don't need to import it or anything.
modulesource = '''
a = 'foo'
def myfun(astr):
return a + astr
'''
newmod = makemodule(modulesource)
print(newmod.myfun('bat'))
Ideone here: http://ideone.com/dXGziO
Tested with python 2, should work with python 3 (textually compatible print used;function-like exec syntax used).
This seems to be a use case for a self-written import hook. Look up in PEP 302 how exactly they work.
Essentially, you'll have to provide a finder object which, in turn, provides a loader object. I don't understand the process at the very first glance (otherwise I'd be more explicit), but the PEP contains all needed details for implementing the stuff.
As glglgl's has it this import hook has been implemented for Python2 and Python3 in a module called httpimport.
It uses a custom finder/loader object to locate resources using HTTP/S.
Additionally, the import_cdn function in Jossef Harush's answer is almost identically implemented in httpimport's github_repo, and bitbucket_repo functions.
#Marcin's answer contains a good portion of the code of the httpimport's loader class.
I have a module that defines a class which instantiates a class from one of two (or more) other modules. Below are a couple of code examples. In the first example, two modules are imported, but only one is used (one per instance of MyIo). In the second example, only the required module is imported. There may be one or more instances of MyIo in a higher level module.
I like that the second example only imports what is used, but I don't really like that the import is taking place in a 'code execution' section.
My questions are:
Which of the examples is a better architectural choice, and why?
Is there a penalty for importing modules that are not eventually
used?
Are imports in code execution sections in Python considered 'bad form?'
This example imports both modules, but only uses one...
''' MyIo.py '''
...
...
from DevSerial import Device as DeviceSerial
from DevUSB import Device as DeviceUSB
class MyIo:
def __init__(self, port)
if port.lower() == 'usb':
self.device=DeviceUSB()
else:
self.device=DeviceSerial(port)
...
...
The following imports only the module being used...
''' MyIo.py '''
...
...
class MyIo:
def __init__(self, port)
if port.lower() == 'usb':
from DevUSB import Device
self.device=Device()
else:
from DevSerial import Device
self.device=Device(port)
...
...
As per PEP 8, all imports should be together at the top of the file. Having them spread throughout the file leads to hard to maintain and debug software.
The only performance overhead I can think of is at program startup - it has to load more modules. Once the program is running there shouldn't be any extra overhead.
To answer your questions:
The former. It is clearly obvious what other files are used, whereas you have to dig through the second to find all the dependencies.
Yes, but only at startup.
Yes.
Actually, even tho you are importing the modules into a function, they will still exists into sys.modules once your function is done executing unless your are deleting them manually. So yeah, there's no point to don't import them directly at the top of your code (like example #1).
The most common use for imports that are not just jammed up at the top of the page is for situations where sibling modules represent different, mutually exclusive options: the best example is os.path, which is automatically swapped for the appropriate module. Even there its common to do the differential import up at the top and not down in the code.