Present situation:
I have a script: program.py.
I have a directory: /imports. Which contains an arbitrary amount of modules in subdirectories.
Python's package directory syntax is being used.
__init__.py
program.py
/imports
__init__.py
/module a
__init__.py
other_files.py
/module b
__init__.py
other
Intention:
program.py should import every subdirectory in /imports.
program.py needs object access to the modules. For example, imports['name'].Service().
Remark: I don't feel like illustrating my purpose or questioning the design choice. I'm using the above workaround.
The problem is described below.
The existing code:
import os
modules = []
IMPORT_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'imports')
to_import = [name for name in os.listdir(IMPORT_PATH) if os.path.isdir(os.path.join(IMPORT_PATH, name))]
modules = [__import__('imports.' + x) for x in to_import]
print modules
Output:
[<module 'imports' from '/code/test/imports/__init__.pyc'>, <module 'imports' from '/code/test/imports/__init__.pyc'>]
Desired output:
[<module 'B' from '/code/test/imports/B'>, <module 'C' from '/code/test/imports/C'>]
Problem identified: The fromlist=[] argument for __import__() is missing. See Why does Python's __import__ require fromlist? for elaboration.
Corrected example:
modules = {}
IMPORT_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'imports')
to_import = [name for name in os.listdir(IMPORT_PATH) if os.path.isdir(os.path.join(IMPORT_PATH, name))]
for x in to_import:
modules[x] = __import__('imports.' + x, fromlist=['*'])
Output:
{'C': <module 'imports.C' from '/code/test/imports/C/__init__.pyc'>,
'B': <module 'imports.B' from '/code/test/imports/B/__init__.pyc'>}
Achieved result:
Now modules[dir_name].Service() can be used in program.py.
Related
I'm writing a Python application that takes a command as an argument, for example:
$ python myapp.py command1
I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like:
myapp/
__init__.py
commands/
__init__.py
command1.py
command2.py
foo.py
bar.py
So I want the application to find the available command modules at runtime and execute the appropriate one.
Python defines an __import__() function, which takes a string for a module name:
__import__(name, globals=None, locals=None, fromlist=(), level=0)
The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name.
Source: https://docs.python.org/3/library/functions.html#__import__
So currently I have something like:
command = sys.argv[1]
try:
command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"])
except ImportError:
# Display error message
command_module.run()
This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code.
Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added.
See also: How do I import a module given the full path?
With Python older than 2.7/3.1, that's pretty much how you do it.
For newer versions, see importlib.import_module for Python 2 and Python 3.
Or using __import__ you can import a list of modules by doing this:
>>> moduleNames = ['sys', 'os', 're', 'unittest']
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__, moduleNames)
Ripped straight from Dive Into Python.
The recommended way for Python 2.7 and 3.1 and later is to use importlib module:
importlib.import_module(name, package=None)
Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).
e.g.
my_module = importlib.import_module('os.path')
Note: imp is deprecated since Python 3.4 in favor of importlib
As mentioned the imp module provides you loading functions:
imp.load_source(name, path)
imp.load_compiled(name, path)
I've used these before to perform something similar.
In my case I defined a specific class with defined methods that were required.
Once I loaded the module I would check if the class was in the module, and then create an instance of that class, something like this:
import imp
import os
def load_from_file(filepath):
class_inst = None
expected_class = 'MyClass'
mod_name,file_ext = os.path.splitext(os.path.split(filepath)[-1])
if file_ext.lower() == '.py':
py_mod = imp.load_source(mod_name, filepath)
elif file_ext.lower() == '.pyc':
py_mod = imp.load_compiled(mod_name, filepath)
if hasattr(py_mod, expected_class):
class_inst = getattr(py_mod, expected_class)()
return class_inst
Using importlib
Importing a source file
Here is a slightly adapted example from the documentation:
import sys
import importlib.util
file_path = 'pluginX.py'
module_name = 'pluginX'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# Verify contents of the module:
print(dir(module))
From here, module will be a module object representing the pluginX module (the same thing that would be assigned to pluginX by doing import pluginX). Thus, to call e.g. a hello function (with no parameters) defined in pluginX, use module.hello().
To get the effect "importing" functionality from the module instead, store it in the in-memory cache of loaded modules, and then do the corresponding from import:
sys.modules[module_name] = module
from pluginX import hello
hello()
Importing a package
To import a package instead, calling import_module is sufficient. Suppose there is a package folder pluginX in the current working directory; then just do
import importlib
pkg = importlib.import_module('pluginX')
# check if it's all there..
print(dir(pkg))
Use the imp module, or the more direct __import__() function.
You can use exec:
exec("import myapp.commands.%s" % command)
If you want it in your locals:
>>> mod = 'sys'
>>> locals()['my_module'] = __import__(mod)
>>> my_module.version
'2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]'
same would work with globals()
Similar as #monkut 's solution but reusable and error tolerant described here http://stamat.wordpress.com/dynamic-module-import-in-python/:
import os
import imp
def importFromURI(uri, absl):
mod = None
if not absl:
uri = os.path.normpath(os.path.join(os.path.dirname(__file__), uri))
path, fname = os.path.split(uri)
mname, ext = os.path.splitext(fname)
if os.path.exists(os.path.join(path,mname)+'.pyc'):
try:
return imp.load_compiled(mname, uri)
except:
pass
if os.path.exists(os.path.join(path,mname)+'.py'):
try:
return imp.load_source(mname, uri)
except:
pass
return mod
The below piece worked for me:
>>>import imp;
>>>fp, pathname, description = imp.find_module("/home/test_module");
>>>test_module = imp.load_module("test_module", fp, pathname, description);
>>>print test_module.print_hello();
if you want to import in shell-script:
python -c '<above entire code in one line>'
The following worked for me:
import sys, glob
sys.path.append('/home/marc/python/importtest/modus')
fl = glob.glob('modus/*.py')
modulist = []
adapters=[]
for i in range(len(fl)):
fl[i] = fl[i].split('/')[1]
fl[i] = fl[i][0:(len(fl[i])-3)]
modulist.append(getattr(__import__(fl[i]),fl[i]))
adapters.append(modulist[i]())
It loads modules from the folder 'modus'. The modules have a single class with the same name as the module name. E.g. the file modus/modu1.py contains:
class modu1():
def __init__(self):
self.x=1
print self.x
The result is a list of dynamically loaded classes "adapters".
Problem
I have a folder structure like this:
- modules
- root
- abc
hello.py
__init__.py
- xyz
hi.py
__init__.py
blah.py
__init__.py
foo.py
bar.py
__init_.py
Here is the same thing in string format:
"modules",
"modues/__init__.py",
"modules/foo.py",
"modules/bar.py",
"modules/root",
"modules/root/__init__.py",
"modules/root/blah,py",
"modules/root/abc",
"modules/root/abc/__init__.py",
"modules/root/abc/hello.py",
"modules/root/xyz",
"modules/root/xyz/__init__.py",
"modules/root/xyz/hi.py"
I am trying to print out all the modules in the python import style format.
An example output would like this:
modules.foo
modules.bar
modules.root.blah
modules.root.abc.hello
modules.root.xyz.hi
How can I do this is in python(if possible without third party libraries) easily?
What I tried
Sample Code
import pkgutil
import modules
absolute_modules = []
def find_modules(module_path):
for package in pkgutil.walk_packages(module_path):
print(package)
if package.ispkg:
find_modules([package.name])
else:
absolute_modules.append(package.name)
if __name__ == "__main__":
find_modules(modules.__path__)
for module in absolute_modules:
print(module)
However, this code will only print out 'foo' and 'bar'. But not 'root' and it's sub packages. I'm also having trouble figuring out how to convert this to preserve it's absolute import style. The current code only gets the package/module name and not the actual absolute import.
This uses setuptools.find_packages (for the packages) and pkgutil.iter_modules for their submodules. Python2 is supported as well. No need for recursion, it's all handled by these two functions used together.
import sys
from setuptools import find_packages
from pkgutil import iter_modules
def find_modules(path):
modules = set()
for pkg in find_packages(path):
modules.add(pkg)
pkgpath = path + '/' + pkg.replace('.', '/')
if sys.version_info.major == 2 or (sys.version_info.major == 3 and sys.version_info.minor < 6):
for _, name, ispkg in iter_modules([pkgpath]):
if not ispkg:
modules.add(pkg + '.' + name)
else:
for info in iter_modules([pkgpath]):
if not info.ispkg:
modules.add(pkg + '.' + info.name)
return modules
So I finally figured out how to do this cleanly and get pkgutil to take care of all the edge case for you. This code was based off python's help() function which only displays top level modules and packages.
import importlib
import pkgutil
import sys
import modules
def find_abs_modules(module):
path_list = []
spec_list = []
for importer, modname, ispkg in pkgutil.walk_packages(module.__path__):
import_path = f"{module.__name__}.{modname}"
if ispkg:
spec = pkgutil._get_spec(importer, modname)
importlib._bootstrap._load(spec)
spec_list.append(spec)
else:
path_list.append(import_path)
for spec in spec_list:
del sys.modules[spec.name]
return path_list
if __name__ == "__main__":
print(sys.modules)
print(find_abs_modules(modules))
print(sys.modules)
This will work even for builtin packages.
The below code will give you the relative package module from the codes current working directory.
import os
import re
for root,dirname,filename in os.walk(os.getcwd()):
pth_build=""
if os.path.isfile(root+"/__init__.py"):
for i in filename:
if i <> "__init__.py" and i <> "__init__.pyc":
if i.split('.')[1] == "py":
slot = list(set(root.split('\\')) -set(os.getcwd().split('\\')))
pth_build = slot[0]
del slot[0]
for j in slot:
pth_build = pth_build+"."+j
print pth_build +"."+ i.split('.')[0]
This code will display:
modules.foo
modules.bar
modules.root.blah
modules.root.abc.hello
modules.root.xyz.hi
If you run it outside the modules folder.
I defined a package that include a dynamically growing set of modules:
- mypackage
- __init__.py
- module1.py
- module2.py
- module3.py
... many more .py files will be added
I could expose every name in every module in __init__.py like this:
from module1 import *
from module2 import *
from module3 import *
Now when I import mypackage in client code, I get all the names defined in the sub-modules:
# suppose funcA is defined in module1, class B is defined in module2
import mypackage
mypackage.funcA() # call module1.funcA
b = mypackage.B() # module2.B
The problem is, I could define many new modules in mypackage, and I don't want to add an extra line from modulex import * to __init__.py, every time I add a new module to the package.
What is the best way to dynamically export names in all submodules?
In Python 3 the solution is straightforward using importlib and pkgutil.
Placing the following code in __init__.py is the same typing from submodule import * for all submodules (even nested ones).
import importlib
import pkgutil
for mod_info in pkgutil.walk_packages(__path__, __name__ + '.'):
mod = importlib.import_module(mod_info.name)
# Emulate `from mod import *`
try:
names = mod.__dict__['__all__']
except KeyError:
names = [k for k in mod.__dict__ if not k.startswith('_')]
globals().update({k: getattr(mod, k) for k in names})
If you only want to include immediate submodules (e.g. pkg.mod but not pkg.mod.submod), replace walk_packages with iter_modules.
I emulated from mod import * based on this answer: How to do from module import * using importlib?
I'm not sure if this is what you mean:
but if i've understood correctly - do this in your __init__.py file.
import os
__all__ = []
for module in os.listdir(os.path.dirname(__file__)):
if module != '__init__.py' and module[-3:] == '.py':
__all__.append(module[:-3])
You're adding all files in the same package into the __all__
Simply adding module names to __all__ will not always serve the purpose. I came across such an issue and also required them to be imported in addition to adding them to __all__. This is the code I came up with to make it work in my case. I didn't have any sub packages, so this code works only at the top level.
modules = glob.glob(os.path.dirname(__file__) + '/*.py')
__all__ = []
for mod in modules:
if not str(mod).endswith('__init__.py'):
package_prefix = __name__ + '.'
module_name = str(mod)[str(mod).rfind('\\') + 1:-3]
__all__.append(module_name)
__import__(package_prefix + module_name, globals(), locals(), [''])
I'm writing a Python application that takes a command as an argument, for example:
$ python myapp.py command1
I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like:
myapp/
__init__.py
commands/
__init__.py
command1.py
command2.py
foo.py
bar.py
So I want the application to find the available command modules at runtime and execute the appropriate one.
Python defines an __import__() function, which takes a string for a module name:
__import__(name, globals=None, locals=None, fromlist=(), level=0)
The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name.
Source: https://docs.python.org/3/library/functions.html#__import__
So currently I have something like:
command = sys.argv[1]
try:
command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"])
except ImportError:
# Display error message
command_module.run()
This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code.
Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added.
See also: How do I import a module given the full path?
With Python older than 2.7/3.1, that's pretty much how you do it.
For newer versions, see importlib.import_module for Python 2 and Python 3.
Or using __import__ you can import a list of modules by doing this:
>>> moduleNames = ['sys', 'os', 're', 'unittest']
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__, moduleNames)
Ripped straight from Dive Into Python.
The recommended way for Python 2.7 and 3.1 and later is to use importlib module:
importlib.import_module(name, package=None)
Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).
e.g.
my_module = importlib.import_module('os.path')
Note: imp is deprecated since Python 3.4 in favor of importlib
As mentioned the imp module provides you loading functions:
imp.load_source(name, path)
imp.load_compiled(name, path)
I've used these before to perform something similar.
In my case I defined a specific class with defined methods that were required.
Once I loaded the module I would check if the class was in the module, and then create an instance of that class, something like this:
import imp
import os
def load_from_file(filepath):
class_inst = None
expected_class = 'MyClass'
mod_name,file_ext = os.path.splitext(os.path.split(filepath)[-1])
if file_ext.lower() == '.py':
py_mod = imp.load_source(mod_name, filepath)
elif file_ext.lower() == '.pyc':
py_mod = imp.load_compiled(mod_name, filepath)
if hasattr(py_mod, expected_class):
class_inst = getattr(py_mod, expected_class)()
return class_inst
Using importlib
Importing a source file
Here is a slightly adapted example from the documentation:
import sys
import importlib.util
file_path = 'pluginX.py'
module_name = 'pluginX'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# Verify contents of the module:
print(dir(module))
From here, module will be a module object representing the pluginX module (the same thing that would be assigned to pluginX by doing import pluginX). Thus, to call e.g. a hello function (with no parameters) defined in pluginX, use module.hello().
To get the effect "importing" functionality from the module instead, store it in the in-memory cache of loaded modules, and then do the corresponding from import:
sys.modules[module_name] = module
from pluginX import hello
hello()
Importing a package
To import a package instead, calling import_module is sufficient. Suppose there is a package folder pluginX in the current working directory; then just do
import importlib
pkg = importlib.import_module('pluginX')
# check if it's all there..
print(dir(pkg))
Use the imp module, or the more direct __import__() function.
You can use exec:
exec("import myapp.commands.%s" % command)
If you want it in your locals:
>>> mod = 'sys'
>>> locals()['my_module'] = __import__(mod)
>>> my_module.version
'2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]'
same would work with globals()
Similar as #monkut 's solution but reusable and error tolerant described here http://stamat.wordpress.com/dynamic-module-import-in-python/:
import os
import imp
def importFromURI(uri, absl):
mod = None
if not absl:
uri = os.path.normpath(os.path.join(os.path.dirname(__file__), uri))
path, fname = os.path.split(uri)
mname, ext = os.path.splitext(fname)
if os.path.exists(os.path.join(path,mname)+'.pyc'):
try:
return imp.load_compiled(mname, uri)
except:
pass
if os.path.exists(os.path.join(path,mname)+'.py'):
try:
return imp.load_source(mname, uri)
except:
pass
return mod
The below piece worked for me:
>>>import imp;
>>>fp, pathname, description = imp.find_module("/home/test_module");
>>>test_module = imp.load_module("test_module", fp, pathname, description);
>>>print test_module.print_hello();
if you want to import in shell-script:
python -c '<above entire code in one line>'
The following worked for me:
import sys, glob
sys.path.append('/home/marc/python/importtest/modus')
fl = glob.glob('modus/*.py')
modulist = []
adapters=[]
for i in range(len(fl)):
fl[i] = fl[i].split('/')[1]
fl[i] = fl[i][0:(len(fl[i])-3)]
modulist.append(getattr(__import__(fl[i]),fl[i]))
adapters.append(modulist[i]())
It loads modules from the folder 'modus'. The modules have a single class with the same name as the module name. E.g. the file modus/modu1.py contains:
class modu1():
def __init__(self):
self.x=1
print self.x
The result is a list of dynamically loaded classes "adapters".
i have a 2.6 python script and library in the following directory structure:
+ bin
\- foo.py
+ lib
\+ foo
\- bar.py
i would like users to run bin/foo.py to instantiate the classes within lib/foo.py. to achieve this, in my bin/foo.py script i have the following code:
from __future__ import absolute_import
import foo
klass = foo.bar.Klass()
however, this results in:
AttributeError: 'module' object has no attribute 'bar'
ie it thinks that foo is itself rather than the library foo - renaming bin/foo.py to bin/foo-script.py works as expected.
is there a way i can keep the bin/foo.py script and import lib/foo.py?
The current directory is on the path by default, so you need to remove that before you import the other foo module:
import sys
sys.path = [dir for dir in sys.path if dir != '']
Alternatively, prepend the lib directory so that it takes precedence:
import sys
sys.path = ['../lib'] + sys.path
If you just write import foo, it will definitely load the foo module in the current scope. Assuming lib and foo as packages, won't you need to write something like this in order to make it work?
import lib.foo.bar as foobar
klass = foobar.Klass()