Develop Reference

How to enable the Python program to load and use a few different packages with the same name?

I'm developing a Python application that controls hardware. The hardware consists of FPGA-based systems. Their internal structure from the control point of view is managed with an automated system AGWB ( http://github.com/wzab/agwb.git ).
The description of each system is compiled into a Python module "agwb" containing a few submodules.
After the module is loaded, it is used to produce an object handling further communication with the associated system.
When my application works with a few different systems, it has to load a few different "agwb" modules (of course, each one from a different directory).
Loading of the module from a specified directory can be easily achieved by adding that directory at the beginning of the sys.path (and removing it, after the module is loaded).
Unfortunately, Python imports the module "agwb" only once. Later attempts to import it results in using of the previously loaded one.
Using the importlib.reload also does not help.
After some reading of the https://docs.python.org/3/reference/import.html I have found the following solution:
import sys
import importlib
def load_agwb(lddir,top):
sys.path.insert(0,lddir)
importlib.import_module('agwb')
# Get the class for the top entity
res = getattr(sys.modules['agwb'],top)()
sys.path.remove(lddir)
sys.modules.pop('agwb')
mods_to_remove = []
for mod in sys.modules.keys():
if mod.find('agwb.') == 0:
mods_to_remove.append(mod)
for mod in mods_to_remove:
sys.modules.pop(mod)
return res
at1 = load_agwb('./t1','top_v0')
at2 = load_agwb('./t2','top_v0')
print("Call at1.show")
at1.show()
print("Call at2.show")
at2.show()
The directories "./t1", and "./t2" contain two different versions of "agwb" module.
After executing the script we get the following output proving that indeed two different versions of "agwb" have been loaded and are used simultaneously:
Call at1.show
object from t1, desc= That's version from directory t1
type ID= 1111
Call at2.show
object from t2, desc= That's version from directory t2
type ID= 2222
The whole minimal demonstrator is available (as a shar archive) at https://pastebin.com/ZGZj8qV1 (unfortunately, I can't attach a file to my question).
And here come my questions:
Is the above described method a reliable, legal solution in Python?
Maybe it works just by chance and may stop working in the next version of Python?
Are there other, more "official" methods of solving my problem?

I have created an answer based on pavel's comment (thanks a lot!).
I've used the temporary directory and symlinks to solve the problem of long, and complex paths. Below is the result:
import sys
import os
import importlib
import tempfile
# Directory with description of the first board
dir1 = './xt1/very/long/path'
# Directory with description of the second board
dir2 = './xt2/completely/different/path'
# Name of the top entity in the design
top = 'top_v0'
# Create a temporary directory
tmpdir = tempfile.TemporaryDirectory()
# Add it to the Python path
sys.path.insert(0,tmpdir.name)
# Create symlinks to the directories with descriptions of the boards
os.symlink(os.path.abspath(dir1),tmpdir.name+"/t1")
os.symlink(os.path.abspath(dir2),tmpdir.name+"/t2")
# Import description of the first board as agwb
import t1.agwb as agwb
# use the imported agwb
at1 = getattr(agwb,top)()
# if the name of the top entity is constant, you can do it easier
bt1 = agwb.top_v0()
# Import description of the second board as agwb
import t2.agwb as agwb
# use the imported agwb
at2 = getattr(agwb,top)()
# if the name of the top entity is constant, you can do it easier
bt2 = agwb.top_v0()
# Now we can remove the temporary directory from the python path
sys.path.remove(tmpdir.name)
# We can even delete it (otherwise it will be deleted when Python session ends)
tmpdir.cleanup()
# Check if both versions are loaded and available
print("Call at1.show")
at1.show()
print("Call at2.show")
at2.show()
print("Call bt1.show")
bt1.show()
print("Call bt2.show")
bt2.show()
As previously, the whole minimal reproducer is available in the shar format at https://pastebin.com/GrRUh8U9 .

Is this the approved way to acess data adjacent to/packaged with a Python script?

I have a Python script that needs some data that's stored in a file that will always be in the same location as the script. I have a setup.py for the script, and I want to make sure it's pip installable in a wide variety of environments, and can be turned into a standalone executable if necessary.
Currently the script runs with Python 2.7 and Python 3.3 or higher (though I don't have a test environment for 3.3 so I can't be sure about that).
I came up with this method to get the data. This script isn't part of a module directory with __init__.py or anything, it's just a standalone file that will work if just run with python directly, but also has an entry point defined in the setup.py file. It's all one file. Is this the correct way?
def fetch_wordlist():
wordlist = 'wordlist.txt'
try:
import importlib.resources as res
return res.read_binary(__file__, wordlist)
except ImportError:
pass
try:
import pkg_resources as resources
req = resources.Requirement.parse('makepw')
wordlist = resources.resource_filename(req, wordlist)
except ImportError:
import os.path
wordlist = os.path.join(os.path.dirname(__file__), wordlist)
with open(wordlist, 'rb') as f:
return f.read()
This seems ridiculously complex. Also, it seems to rely on the package management system in ways I'm uncomfortable with. The script no longer works unless it's been pip-installed, and that also doesn't seem desirable.

Resources living on the filesystem
The standard way to read a file adjacent to your python script would be:
a) If you've got python>=3.4 I'd suggest you use the pathlib module, like this:
from pathlib import Path
def fetch_wordlist(filename="wordlist.txt"):
return (Path(__file__).parent / filename).read_text()
if __name__ == '__main__':
print(fetch_wordlist())
b) And if you're still using a python version <3.4 or you still want to use the good old os.path module you should do something like this:
import os
def fetch_wordlist(filename="wordlist.txt"):
with open(os.path.join(os.path.dirname(__file__), filename)) as f:
return f.read()
if __name__ == '__main__':
print(fetch_wordlist())
Also, I'd suggest you capture exceptions in the outer callers, the above methods are standard way to read files in python so you don't need wrap them in a function like fetch_wordlist, said otherwise, reading files in python is an "atomic" operation.
Now, it may happen that you've frozen your program using some freezer such as cx_freeze, pyinstaller or similars... in that case you'd need to detect that, here's a simple way to check it out:
a) using os.path:
if getattr(sys, 'frozen', False):
app_path = os.path.dirname(sys.executable)
elif __file__:
app_path = os.path.dirname(__file__)
b) using pathlib:
if getattr(sys, 'frozen', False):
app_path = Path(sys.executable).parent
elif __file__:
app_path = Path(__file__).parent
Resources living inside a zip file
The above solutions would work if the code lives on the file system but it wouldn't work if the package is living inside a zip file, when that happens you could use either importlib.resources (new in version 3.7) or pkg_resources combo as you've already shown in the question (or you could wrap up in some helpers) or you could use a nice 3rd party library called importlib_resources that should work with the old&modern python versions:
pypi: https://pypi.org/project/importlib_resources/
documentation: https://importlib-resources.readthedocs.io/en/latest/
Specifically for your particular problem I'd suggest you take a look to this https://importlib-resources.readthedocs.io/en/latest/using.html#file-system-or-zip-file.
If you want to know what that library is doing behind the curtains because you're not willing to install any 3rd party library you can find the code for py2 here and py3 here in case you wanted to get the relevant bits for your particular problem

I'm going to go out on a limb and make an assumption because it may drastically simplify your problem. The only way I can imagine that you can claim that this data is "stored in a file that will always be in the same location as the script" is because you created this data, once, and put it in a file in the source code directory. Even though this data is binary, have you considered making the data a literal byte-string in a python file, and then simply importing it as you would anything else?

You're right about the fact that your method of reading a file is a bit unnecessarily complex. Unless you have got a really specific reason to use the importlib and pkg_resources modules, it's rather simple.
import os
def fetch_wordlist():
if not os.path.exists('wordlist.txt'):
raise FileNotFoundError
with open('wordlist.txt', 'rb') as wordlist:
return wordlist.read()
You haven't given much information regarding your script, so I cannot comment on why it doesn't work unless it's installed using pip. My best guess: your script is probably packed into a python package.

Display file imports/usage used in a python file

I was wondering if there are any sort of python codeing etc that will displays the files imports/used locations in a python file?
Eg. TestA.py contains 3 files from 3 different directory
Import01 : /u/ext/TestA/UI
Import02 : /u/ext/TestA/src
Import03 : /user_data/setup/localImports
And hence, while executing the coding, it will displays the list of directories used in the python file?
I am asking as I am working on several (and maybe tons, in the future) scripts that are heavily involved in Maya, there are times in which when I located the path but they are the wrong ones (with same name) and is actually located in another path

Add this code to module
import inspect
frame = inspect.currentframe()
if frame and frame.f_back:
print('module "{}" is imported by "{}"'.format(__file__, frame.f_back.f_locals['__file__']))
If module_a.py contains the code above, and main.py imports it. the output is
module "/path/to/module_a.py" is imported by "/path/to/main.py"
As documented, this answer may not be an exact solution. Because if not supported, returns None.
CPython implementation detail: This function relies on Python stack frame support in the interpreter, which isn’t guaranteed to exist in all implementations of Python. If running in an implementation without Python stack frame support this function returns None.

At any point in when the code is running, you can determine the origin of a module by checking it's file attribute:
import sys
for name, each_mod in sys.modules.items():
try:
print name, each_mod.__file__
except AttributeError: # = built in module or dll
print "?"
To check the imports without running the code, you'd need do more complex analysis: Here's an example method that could probably be adapted to figure it out :http://www.tarind.com/depgraph.html
You could also create a custom ModuleFinder that printed out file sources as imports are processed. Something like this, which prints out name of py/pyc files when trying to load them.
import os
import sys
import imp
import ihooks
class ReportingFinder(object):
"""Find modules collected in a shelve archive."""
def __init__(self, path_entry):
self.path_entry = path_entry
if not os.path.isdir(path_entry):
raise ImportError
def find_module(self, fullname, path=None):
for suffix in (".py", ".pyc"):
test_path = os.path.join(self.path_entry, fullname + suffix)
print test_path
if os.path.exists(test_path):
print "attemnpting to load from %s" % test_path
return self
return None
def load_module(self, name):
stuff = imp.find_module(name)
return ihooks.FancyModuleLoader(verbose=1).load_module(name, stuff)
sys.path_hooks.insert(0, ReportingFinder)
HACK WARNING!!!! Please be aware this code is a quick diagnostic hack! Don't use it for production :) Among other flaws, it will print out py path names even if the code comes from the pyc, and it's dumb about packages -- I only provided it because it sounds like you're using single-file scripts rather than packages. It is handy for catching imported modules as they get loaded. It won't print out the names of zip files.
It sounds like the real problem is having too many competing paths: you should try to get down to as few as you can so that there are fewer suprises.

Bundle a Python app as a single file to support add-ons or extensions?

There are several utilities — all with different procedures, limitations, and target operating systems — for getting a Python package and all of its dependencies and turning them into a single binary program that is easy to ship to customers:
http://wiki.python.org/moin/Freeze
http://www.pyinstaller.org/
http://www.py2exe.org/
http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.html
My situation goes one step further: third-party developers will be wanting to write plug-ins, extensions, or add-ons for my application. It is, of course, a daunting question how users on platforms like Windows would most easily install plugins or addons in such a way that my app can easily discover that they have been installed. But beyond that basic question is another: how can a third-party developer bundle their extension with whatever libraries the extension itself needs (which might be binary modules, like lxml) in such a way that the plugin's dependencies become available for import at the same time that the plugin becomes available.
How can this be approached? Will my application need its own plug-in area on disk and its own plug-in registry to make this tractable? Or are there general mechanisms, that I could avoid writing myself, that would allow an app that is distributed as a single executable to look around and find plugins that are also installed as single files?

You should be able to have a plugins directory that your application scans at runtime (or later) to import the code in question. Here's an example that should work with regular .py or .pyc code that even works with plugins stored inside zip files (so users could just drop someplugin.zip in the 'plugins' directory and have it magically work):
import re, os, sys
class Plugin(object):
"""
The base class from which all plugins are derived. It is used by the
plugin loading functions to find all the installed plugins.
"""
def __init__(self, foo):
self.foo = foo
# Any useful base plugin methods would go in here.
def get_plugins(plugin_dir):
"""Adds plugins to sys.path and returns them as a list"""
registered_plugins = []
#check to see if a plugins directory exists and add any found plugins
# (even if they're zipped)
if os.path.exists(plugin_dir):
plugins = os.listdir(plugin_dir)
pattern = ".py$"
for plugin in plugins:
plugin_path = os.path.join(plugin_dir, plugin)
if os.path.splitext(plugin)[1] == ".zip":
sys.path.append(plugin_path)
(plugin, ext) = os.path.splitext(plugin) # Get rid of the .zip extension
registered_plugins.append(plugin)
elif plugin != "__init__.py":
if re.search(pattern, plugin):
(shortname, ext) = os.path.splitext(plugin)
registered_plugins.append(shortname)
if os.path.isdir(plugin_path):
plugins = os.listdir(plugin_path)
for plugin in plugins:
if plugin != "__init__.py":
if re.search(pattern, plugin):
(shortname, ext) = os.path.splitext(plugin)
sys.path.append(plugin_path)
registered_plugins.append(shortname)
return registered_plugins
def init_plugin_system(cfg):
"""
Initializes the plugin system by appending all plugins into sys.path and
then using load_plugins() to import them.
cfg - A dictionary with two keys:
plugin_path - path to the plugin directory (e.g. 'plugins')
plugins - List of plugin names to import (e.g. ['foo', 'bar'])
"""
if not cfg['plugin_path'] in sys.path:
sys.path.insert(0, cfg['plugin_path'])
load_plugins(cfg['plugins'])
def load_plugins(plugins):
"""
Imports all plugins given a list.
Note: Assumes they're all in sys.path.
"""
for plugin in plugins:
__import__(plugin, None, None, [''])
if plugin not in Plugin.__subclasses__():
# This takes care of importing zipped plugins:
__import__(plugin, None, None, [plugin])
So lets say I have a plugin named "foo.py" in a directory called 'plugins' (that is in the base dir of my app) that will add a new capability to my application. The contents might look like this:
from plugin_stuff import Plugin
class Foo(Plugin):
"""An example plugin."""
self.menu_entry = {'Tools': {'Foo': self.bar}}
def bar(self):
return "foo plugin!"
I could initialize my plugins when I launch my app like so:
plugin_dir = "%s/plugins" % os.getcwd()
plugin_list = get_plugins(plugin_dir)
init_plugin_system({'plugin_path': plugin_dir, 'plugins': plugin_list})
plugins = find_plugins()
plugin_menu_entries = []
for plugin in plugins:
print "Enabling plugin: %s" % plugin.__name__
plugin_menu_entries.append(plugin.menu_entry))
add_menu_entries(plugin_menu_entries) # This is an imaginary function
That should work as long as the plugin is either a .py or .pyc file (assuming it is byte-compiled for the platform in question). It can be standalone file or inside of a directory with an init.py or inside of a zip file with the same rules.
How do I know this works? It is how I implemented plugins in PyCI. PyCI is a web application but there's no reason why this method wouldn't work for a regular ol' GUI. For the example above I chose to use an imaginary add_menu_entries() function in conjunction with a Plugin object variable that could be used to add a plugin's methods to your GUI's menus.
Hopefully this answer will help you build your own plugin system. If you want to see precisely how it is implemented I recommend you download the PyCI source code and look at plugin_utils.py and the Example plugin in the plugins_enabled directory.

Here is another example of a Python app that uses plugins: OpenSTV. Here, the plugins can only be Python modules.

In Python, how can I efficiently manage references between script files?

I have a fair number of Python scripts that contain reusable code that are used and referenced by other Python scripts. However, these scripts tend to be scattered across different directories and I find it to be somewhat tedious to have to include (most often multiple) calls to sys.path.append on my top-level scripts. I just want to provide the 'import' statements without the additional file references in the same script.
Currently, I have this:
import sys
sys.path.append('..//shared1//reusable_foo')
import Foo
sys.path.append('..//shared2//reusable_bar')
import Bar
My preference would be the following:
import Foo
import Bar
My background is primarily in the .NET platform so I am accustomed to having meta files such as *.csproj, *.vbproj, *.sln, etc. to manage and contain the actual file path references outside of the source files. This allows me to just provide 'using' directives (equivalent to Python's import) without exposing all of the references and allowing for reuse of the path references themselves across multiple scripts.
Does Python have equivalent support for this and, if not, what are some techniques and approaches?

The simple answer is to put your reusable code in your site-packages directory, which is in your sys.path.
You can also extend the search path by adding .pth files somewhere in your path.
See https://docs.python.org/2/install/#modifying-python-s-search-path for more details
Oh, and python 2.6/3.0 adds support for PEP370, Per-user site-packages Directory

If your reusable files are packaged (that is, they include an __init__.py file) and the path to that package is part of your PYTHONPATH or sys.path then you should be able to do just
import Foo
This question provides a few more details.
(Note: As Jim said, you could also drop your reusable code into your site-packages directory.)

You can put the reusable stuff in site-packages. That's completely transparent, since it's in sys.path by default.
You can put someName.pth files in site-packages. These files have the directory in which your actual reusable stuff lives. This is also completely transparent. And doesn't involve the extra step of installing a change in site-packages.
You can put the directory of the reusable stuff on PYTHONPATH. That's a little less transparent, because you have to make sure it's set. Not rocket science, but not completely transparent.

In one project, I wanted to make sure that the user could put python scripts (that could basically be used as plugins) anywhere. My solution was to put the following in the config file for that project:
[server]
PYPATH_APPEND: /home/jason:/usr/share/some_directory
That way, this would add /home/jason and /usr/share/some_directory to the python path at program launch.
Then, it's just a simple matter of splitting the string by the colons and adding those directories to the end of the sys.path. You may want to consider putting a module in the site-packages directory that contains a function to read in that config file and add those directories to the sys.path (unfortunately, I don't have time at the moment to write an example).
As others have mentioned, it's a good idea to put as much in site-packages as possible and also using .pth files. But this can be a good idea if you have a script that needs to import a bunch of stuff that's not in site-packages that you wouldn't want to import from other scripts.
(there may also be a way to do this using .pth files, but I like being able to manipulate the python path in the same place as I put the rest of my configuration info)

The simplest way is to set (or add to) PYTHONPATH, and put (or symlink) your modules and packages into a path contained in PYTHONPATH.

My solution was to package up one utility that would import the module:
my_util is in site packages
import my_util
foo = myutil.import_script('..//shared1//reusable_foo')
if foo == None:
sys.exit(1)
def import_script(script_path, log_status = True):
"""
imports a module and returns the handle
"""
lpath = os.path.split(script_path)
if lpath[1] == '':
log('Error in script "%s" in import_script' % (script_path))
return None
#check if path is already in sys.path so we don't repeat
npath = None
if lpath[0] == '':
npath = '.'
else:
if lpath[0] not in sys.path:
npath = lpath[0]
if npath != None:
try:
sys.path.append(npath)
except:
if log_status == True:
log('Error adding path "%s" in import_script' % npath)
return None
try:
mod = __import__(lpath[1])
except:
error_trace,error_reason = FormatExceptionInfo()
if log_status == True:
log('Error importing "%s" module in import_script: %s' % (script_path, error_trace + error_reason))
sys.path.remove(npath)
return None
return mod