Could someone provide me with a good way of importing a whole directory of modules?
I have a structure like this:
/Foo
bar.py
spam.py
eggs.py
I tried just converting it to a package by adding __init__.py and doing from Foo import * but it didn't work the way I had hoped.
List all python (.py) files in the current folder and put them as __all__ variable in __init__.py
from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]
Add the __all__ Variable to __init__.py containing:
__all__ = ["bar", "spam", "eggs"]
See also http://docs.python.org/tutorial/modules.html
Update in 2017: you probably want to use importlib instead.
Make the Foo directory a package by adding an __init__.py. In that __init__.py add:
import bar
import eggs
import spam
Since you want it dynamic (which may or may not be a good idea), list all py-files with list dir and import them with something like this:
import os
for module in os.listdir(os.path.dirname(__file__)):
if module == '__init__.py' or module[-3:] != '.py':
continue
__import__(module[:-3], locals(), globals())
del module
Then, from your code do this:
import Foo
You can now access the modules with
Foo.bar
Foo.eggs
Foo.spam
etc. from Foo import * is not a good idea for several reasons, including name clashes and making it hard to analyze the code.
Python, include all files under a directory:
For newbies who just can't get it to work who need their hands held.
Make a folder /home/el/foo and make a file main.py under /home/el/foo Put this code in there:
from hellokitty import *
spam.spamfunc()
ham.hamfunc()
Make a directory /home/el/foo/hellokitty
Make a file __init__.py under /home/el/foo/hellokitty and put this code in there:
__all__ = ["spam", "ham"]
Make two python files: spam.py and ham.py under /home/el/foo/hellokitty
Define a function inside spam.py:
def spamfunc():
print("Spammity spam")
Define a function inside ham.py:
def hamfunc():
print("Upgrade from baloney")
Run it:
el#apollo:/home/el/foo$ python main.py
spammity spam
Upgrade from baloney
Expanding on Mihail's answer, I believe the non-hackish way (as in, not handling the file paths directly) is the following:
create an empty __init__.py file under Foo/
Execute
import pkgutil
import sys
def load_all_modules_from_dir(dirname):
for importer, package_name, _ in pkgutil.iter_modules([dirname]):
full_package_name = '%s.%s' % (dirname, package_name)
if full_package_name not in sys.modules:
module = importer.find_module(package_name
).load_module(full_package_name)
print module
load_all_modules_from_dir('Foo')
You'll get:
<module 'Foo.bar' from '/home/.../Foo/bar.pyc'>
<module 'Foo.spam' from '/home/.../Foo/spam.pyc'>
I know I'm updating a quite old post, and I tried using automodinit, but found out it's setup process is broken for python3. So, based on Luca's answer, I came up with a simpler answer - which might not work with .zip - to this issue, so I figured I should share it here:
within the __init__.py module from yourpackage:
#!/usr/bin/env python
import os, pkgutil
__all__ = list(module for _, module, _ in pkgutil.iter_modules([os.path.dirname(__file__)]))
and within another package below yourpackage:
from yourpackage import *
Then you'll have all the modules that are placed within the package loaded, and if you write a new module, it'll be automagically imported as well. Of course, use that kind of things with care, with great powers comes great responsibilities.
I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/.
Usage is like this:
Include the automodinit package into your setup.py dependencies.
Replace all __init__.py files like this:
__all__ = ["I will get rewritten"]
# Don't modify the line above, or this line!
import automodinit
automodinit.automodinit(__name__, __file__, globals())
del automodinit
# Anything else you want can go after here, it won't get modified.
That's it! From now on importing a module will set __all__ to
a list of .py[co] files in the module and will also import each
of those files as though you had typed:
for x in __all__: import x
Therefore the effect of "from M import *" matches exactly "import M".
automodinit is happy running from inside ZIP archives and is therefore ZIP safe.
Niall
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
for imp, module, ispackage in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
__import__(module)
I have also encountered this problem and this was my solution:
import os
def loadImports(path):
files = os.listdir(path)
imps = []
for i in range(len(files)):
name = files[i].split('.')
if len(name) > 1:
if name[1] == 'py' and name[0] != '__init__':
name = name[0]
imps.append(name)
file = open(path+'__init__.py','w')
toWrite = '__all__ = '+str(imps)
file.write(toWrite)
file.close()
This function creates a file (in the provided folder) named __init__.py, which contains an __all__ variable that holds every module in the folder.
For example, I have a folder named Test
which contains:
Foo.py
Bar.py
So in the script I want the modules to be imported into I will write:
loadImports('Test/')
from Test import *
This will import everything from Test and the __init__.py file in Test will now contain:
__all__ = ['Foo','Bar']
When from . import * isn't good enough, this is an improvement over the answer by ted. Specifically, the use of __all__ is not necessary with this approach.
"""Import all modules that exist in the current directory."""
# Ref https://stackoverflow.com/a/60861023/
from importlib import import_module
from pathlib import Path
for f in Path(__file__).parent.glob("*.py"):
module_name = f.stem
if (not module_name.startswith("_")) and (module_name not in globals()):
import_module(f".{module_name}", __package__)
del f, module_name
del import_module, Path
Note that module_name not in globals() is intended to avoid reimporting the module if it's already imported, as this can risk cyclic imports.
This is the best way i've found so far:
from os.path import dirname, join, isdir, abspath, basename
from glob import glob
pwd = dirname(__file__)
for x in glob(join(pwd, '*.py')):
if not x.startswith('__'):
__import__(basename(x)[:-3], globals(), locals())
Anurag Uniyal answer with suggested improvements!
#!/usr/bin/python
# -*- encoding: utf-8 -*-
import os
import glob
all_list = list()
for f in glob.glob(os.path.dirname(__file__)+"/*.py"):
if os.path.isfile(f) and not os.path.basename(f).startswith('_'):
all_list.append(os.path.basename(f)[:-3])
__all__ = all_list
Using importlib the only thing you've got to add is
from importlib import import_module
from pathlib import Path
__all__ = [
import_module(f".{f.stem}", __package__)
for f in Path(__file__).parent.glob("*.py")
if "__" not in f.stem
]
del import_module, Path
I've created a module for that, which doesn't rely on __init__.py (or any other auxiliary file) and makes me type only the following two lines:
import importdir
importdir.do("Foo", globals())
Feel free to re-use or contribute: http://gitlab.com/aurelien-lourot/importdir
Anurag's example with a couple of corrections:
import os, glob
modules = glob.glob(os.path.join(os.path.dirname(__file__), "*.py"))
__all__ = [os.path.basename(f)[:-3] for f in modules if not f.endswith("__init__.py")]
I'd like to add to Anurag Uniyal's answer.
You can make it even simpler and get rid of a lot of the imports.
Contents of the __init__.py file:
from os import listdir
from os.path import dirname
__all__ = [i[:-3] for i in listdir(dirname(__file__)) if not i.startswith('__') and i.endswith('.py')]
See that your __init__.py defines __all__. The modules - packages doc says
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.
...
The only solution is for the package author to provide an explicit index of the package. The import statement uses the following convention: if a package’s __init__.py code defines a list named __all__, it is taken to be the list of module names that should be imported when from package import * is encountered. It is up to the package author to keep this list up-to-date when a new version of the package is released. Package authors may also decide not to support it, if they don’t see a use for importing * from their package. For example, the file sounds/effects/__init__.py could contain the following code:
__all__ = ["echo", "surround", "reverse"]
This would mean that from sound.effects import * would import the three named submodules of the sound package.
Look at the pkgutil module from the standard library. It will let you do exactly what you want as long as you have an __init__.py file in the directory. The __init__.py file can be empty.
Just import them by importlib and add them to __all__ (add action is optional) in recurse in the __init__.py of package.
/Foo
bar.py
spam.py
eggs.py
__init__.py
# __init__.py
import os
import importlib
pyfile_extes = ['py', ]
__all__ = [importlib.import_module('.%s' % filename, __package__) for filename in [os.path.splitext(i)[0] for i in os.listdir(os.path.dirname(__file__)) if os.path.splitext(i)[1] in pyfile_extes] if not filename.startswith('__')]
del os, importlib, pyfile_extes
I had a nested directory structure i.e. I had multiple directories inside the main directory that contained the python modules.
I added the following script to my __init__.py file to import all the modules
import glob, re, os
module_parent_directory = "path/to/the/directory/containing/__init__.py/file"
owd = os.getcwd()
if not owd.endswith(module_parent_directory): os.chdir(module_parent_directory)
module_paths = glob.glob("**/*.py", recursive = True)
for module_path in module_paths:
if not re.match( ".*__init__.py$", module_path):
import_path = module_path[:-3]
import_path = import_path.replace("/", ".")
exec(f"from .{import_path} import *")
os.chdir(owd)
Probably not the best way to achieve this, but I couldn't make anything else work for me.
Here is a solution, with which you do not have to write the file name. Just add this code snippet to your __init__.py
from inspect import isclass
from pkgutil import iter_modules
from pathlib import Path
from importlib import import_module
# iterate through the modules in the current package
package_dir = Path(__file__).resolve().parent
for (_, module_name, _) in iter_modules([package_dir]):
# import the module and iterate through its attributes
module = import_module(f"{__name__}.{module_name}")
for attribute_name in dir(module):
attribute = getattr(module, attribute_name)
if isclass(attribute):
# Add the class to this package's variables
globals()[attribute_name] = attribute
Source
None of the solutions was working for me in Python 3.9.5, Flask 2.2.2, the module being a directory 2 levels down the cwd.
This is my solution:
import importlib
import pathlib
import re
path = pathlib.Path(__file__).parent.absolute()
names = [x.name[:-3] for x in path.iterdir() if x.is_file() and re.search("^[a-z]*\.py$", x.name)]
for name in names:
importlib.import_module(f".{name}", __name__)
Related
I have created a package with the following file structure:
- package
- __init__.py
- load.py
- train.py
- test.py
My __init__.py file is simply an import of classes for these files:
from package.load import Load
from package.train import Train
from package.test import Test
Most of the time, I want to load all three, however on occasion I only want to load one of these classes specifically. For example in an ad hoc script (outside of the package) I want to be able to call only the Load class like so:
from package import Load
While all of the above works in this design, I have an issue where dependencies from train/test are also loaded when I import Load like the above. How can I setup the __init__.py file such that I can make the same import call without getting the dependency to load from train/test?
Additional explanation:
Why I am doing this: I have an issue where I want some people to be able to use the Load class, which only uses base python, however the Train/Test files include specialized dependencies which users of just the Load class wont want to utilize or even install.
Here's a way to do something very close what you want. Instead of unconditionally importing all the package's classes in your __init__.py, you can define a function in it to explicitly import any of the ones desired (or all of them if none are specified).
__init__.py:
from pathlib import Path
import sys
print(f'In {Path(__file__).name}')
package_name = Path(__file__).parent.name
package_prefix = package_name + '.'
class_to_module_map = {'Load': 'load', 'Train': 'train', 'Test': 'test'}
def import_classes(*class_names):
namespace = sys._getframe(1).f_globals # Caller's globals.
if not class_names:
class_names = class_to_module_map.keys() # Import them all.
for class_name in class_names:
module = class_to_module_map[class_name]
temp = __import__(package_prefix+module, globals(), locals(), [class_name])
namespace[class_name] = getattr(temp, class_name) # Add to caller's namespace.
For testing purposes, here's what I put in the load.py script:
(I also put something similar in the other two modules in order to verify whether or not they were getting imported.)
load.py:
from pathlib import Path
print(f'In {Path(__file__).name}')
class Load: pass
And finally here's a example of using it to only import the Load class:
ad_hoc.py:
from my_package import import_classes
#from my_package import Load
import_classes('Load')
test = Load()
print(test)
Along with the output produced:
In __init__.py
In load.py
<my_package.load.Load object at 0x001FE4A8>
Inside a folder oranges\, this is our __init__.py file:
__all__ = []
from pathlib import Path
from importlib import import_module
from sys import modules
package = modules[__name__]
initfile = Path(__file__)
for entry in initfile.parent.iterdir():
is_file = entry.is_file()
is_pyfile = entry.name.endswith('.py')
is_not_initpy = (entry != initfile)
if is_file and is_pyfile and is_not_initpy:
module_name = entry.name.removesuffix('.py')
module_path = __name__ + '.' + module_name
module = import_module(module_path)
setattr(package, module_name, module)
__all__.append(module_name)
When we do from oranges import *, the code inside oranges\__init__.py cycles through the *.py files inside oranges\ (except __init__.py), and for each .py file does the following:
imports the .py file as a module into the variable module using Python's importlib.import_module
sets the module as a variable within the __init__.py file (or, more correctly, within the oranges package) using Python's setattr
finally, appends the module to the __all__ list
I define parameters for my programs in a python module for simplicity. These parameters are then loaded using import. Therefore, I have to make sure to always load from the working directory, and nowhere else (independent form the location of the executed script or available modules in python path).
I found two solutions. First modifying the path:
import sys
from os import getcwd
import_path = sys.path
sys.path = [str(getcwd(), ]
import xxx
sys.path = import_path
or using importlib
from pathlib import Path
from importlib.util import module_from_spec, spec_from_file_location
spec = spec_from_file_location('xxx', str(Path('.').expanduser()/'xxx.py'))
xxx = module_from_spec(spec)
spec.loader.exec_module(xxx)
Of course this can be wrapped into a context manager or a function respectively.
What would the pythonic way be to do this? Do those two approaches have advantages and disadvantages?
I checked How can I import a Python library located in the current working directory?
as well as Import python package from local directory into interpreter, but they lack the focus on robustness.
Local imports can be achieved by modifying the path. A context manager is a suitable solution:
import sys
from pathlib import Path
from contextlib import contextmanager
#contextmanager
def local_import(dir_=None):
"""Only import modules within `dir_` (default: cwd)."""
if dir_ is None:
dir_ = Path.cwd()
else:
dir_ = Path(dir_).absolute().resolve(strict=True)
import_path0 = sys.path[0]
sys.path[0] = str(dir_)
try:
yield
finally:
sys.path[0] = import_path0
Then the local import can be done using the standard import syntax
with local_import():
import xxx
This solution relys on the order, in which the paths are scanned, thus we temporarily replace sys.path[0]. We replace it, instead of prepending to avoid import conflicts with the script directory.
Note
You have to be careful to avoid name conflicts, as the import statement is used, modules with identical names will be imported only once. Thus if different modules with the same name exist in the working directory and in the original sys.path[0], only one of them will be imported. Thus, local_import should only be used for scripts, that only use the standard library or installed third party libraries, but not for scripts that import other scripts from the directory. For the unlikely case that you want to import different files with the same name, the following function can be used:
import uuid
from importlib.util import module_from_spec, spec_from_file_location
def import_file(file, content=None):
"""Try importing `file` as module avoiding name clashes.
If `content` is given `content = import_file('file.py', 'content')`
roughly corresponds to `from file import content`
else `file = import_file('file.py')`
roughly corresponds to `import file`.
Parameters
----------
file : str or Path
The Python file corresponding to the module.
content : str, optional
What to import from the module (optional).
"""
file = Path(file).expanduser().resolve(strict=True)
print(file)
spec = spec_from_file_location(file.stem + str(uuid.uuid4()), str(file))
module = module_from_spec(spec)
spec.loader.exec_module(module)
if content:
print(module)
return getattr(module, content)
else:
return module
I am trying to get a list of the imported custom modules (modules I created by myself) in my current file but I could not find a proper way to achieve it.
For example:
<test.py>
import sys
import foo
import bar
I would like to get a list of [foo, bar] excluding the sys module.
Lets say, that if file located near curent file, or in some subfolder, when it's "our" module, not system one.
For this example, I've created three files: main.py, other.py and another.py, and placed whem into one folder.
The code of main.py is:
# os and sys are needed to work
import sys
import os
import shutil
import io
import datetime
import other
import another
def get_my_modules():
# Get list of modules loaded
modules = list(sys.modules.keys())
mymodules = []
# Get current dir
curdir = os.path.realpath(os.path.dirname(__file__))
for m in modules:
try:
# if some module's file path located in current folder or in some subfolder
# lets sey, that it's our self-made module
path = sys.modules[m].__file__
if path.startswith(curdir):
mymodules.append(m)
except Exception:
# Exception could happen if module doesn't have any __file__ property
pass
# Return list of our moudles
return mymodules
print(get_my_modules())
And this code actually outputs ["other", "another"]
This approach has a problem: If you import module, that somehow located in upper folder, it woudln't be detected.
My code starts with the following:
from module import *
How can I get a list of all submodules imported from this module.
For example module has:
module.submodule
module.submodule2
module.anothersubmodule
I want to get after from module import *:
[submodule, submodule2, anothersubmodule]
(not strings with name of submodules, but a list with all submodules themselves)
UPD: I understood that I asked about XY problem.
So here's what i'm trying to achieve:
I have a folder called modules that will have a bunch of scripts following the same pattern. They will all have a function like main(). In my main script i want to import them all and iterate like that:
for i in modules:
i.main(*some_args)
Use the pkgutil module. For example:
import pkgutil
import module
modualList = []
for importer, modname, ispkg in pkgutil.iter_modules(module.__path__):
modualList.append(modname)
What about importlib?
import importlib
import os
directory = './module'
for filename in os.listdir(directory):
filepath = os.path.join(directory, filename)
if not os.path.isfile(filepath):
continue
modulename = os.path.splitext(filename)[0]
args = [filename, filepath]
module = importlib.import_module('module.{}'.format(modulename))
module.main(*args)
With 3 differents Python files in ./modules, all like:
def main(*args):
print('module one: {}'.format(', '.join(args)))
It gives:
module three: three.py, ./module/three.py
module two: two.py, ./module/two.py
module one: one.py, ./module/one.py
I have this problem and do not know how to solve it efficiently.
I've this file structure
THE NUMBER OF FOLDERS NOR THE NAMES ARE GIVEN, IT'S ALL UNKNOWN
app/
__init__.py
modules/
__init__.py
ModuleA/
__init__.py
file.py
otherfile.py
config.ini
ModuleB/
__init__.py
file.py
otherfile.py
config.ini
ModuleC/
__init__.py
file.py
otherfile.py
config.ini
**arbitrary number of modules with same structure*
As you can notiche, app is the main package of my app, but I need an efficient way to import the mods folder and its' content
* My actual solution *
from app import modules ad mods
def load_modules_from_packages(self, pkg = mods):
pkgname = pkg.__name__
pkgpath = dirname(pkg.__file__)
for loader,name,ispkg in pkgutil.walk_packages(pkg.__path__, pkgname+'.'):
if ispkg is True:
__import__(name,globals(),locals(),[],0)
elif ispkg is False:
__import__(name,globals(),locals(),[],0)
This works since pkgutil iterate the structure with the dot notation for names, so import works well.
But now I want load infos in the config file if I am in one of the somemodule folder(the one with own init.py and config.ini
I want to do this to recreate the structure of module package and output it in a JSON rapresentation for another thing
* my other solution does not works*
def load_modules_from_packages(directory)
dir_path = dirname(directory.__file__)
dir_name = directory.__name__
for filename in glob.glob(dir_path + '/**/*.ini', recursive=True):
plugin = {}
plugin['name'] = filename.split('/')[-2]
plugin['path'] = dirname(filename)
plugin['config_file'] = filename
for pyname in glob.glob(dirname(filename)+ '/**/*.py', recursive=True):
importlib.import_module(pyname)
I cant use the solution posted in this thread
How to import a module given the full path?
since I do not know the module name, ad pointed without solutions in the comment.
spec = importlib.util.spec_from_file_location('what.ever', 'foo.py')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
I know 'foo.py' but I cant figure out 'what.ever' like pkgutil.walk_package does.
In fact the modules imported in this way have the package and name entry wrong. With this approach I cant figure out where I am in the file structure to create modules dictionary and the relative modules (for the JSON output)
Any help?