I have a Python project in which I have the following folder structure:
> root
> download_module
> __init__.py
> downloadProcess.py
> sharedFunctions.py
> someHelper.py
> useSharedFunction.py
The download_module/__init__.py has the following code:
from .sharedFunctions import stringArgumentToDate
from .downloadProcess import downloadProcessMethod
The sharedFunctions.py file contains the following function:
def stringArgumentToDate(arg):
dateformat = "%m/%d/%Y"
date = None
if arg.isnumeric():
date = datetime.fromtimestamp(int(arg))
if date == None:
date = datetime.strptime(arg, dateformat)
return date
Then on the useSharedFunction.py I try to import the shared function and use it like this.
from download_module import stringArgumentToDate
from download_module import downloadProcessMethod
def main():
arg = '03/14/2022'
dateArg = stringArgumentToDate(arg)
if __name__ == '__main__':
main()
When I try to run this by using python3 useSharedFunction.py I got the following error:
Traceback (most recent call last):
File "useSharedFunction.py", line 4, in <module>
from download_module import stringArgumentToDate
File "/Users/jacobo/Documents/project/download_module/__init__.py", line 2, in <module>
from .download_module import downloadAndProcessMethod
File "/Users/jacobo/Documents/project/download_module/downloadProcess.py", line 10, in <module>
from sharedFunctions import stringArgumentToDate, otherFunction
ModuleNotFoundError: No module named 'sharedFunctions'
I do believe the error is in downloadProcess since at the beggining of the file we got this import:
from sharedFunctions import stringArgumentToDate, otherFunction
from someHelper import Helper
Which refers to sibling files.
However I'm unsure what will be a proper fix to allow to run the downloadProcess.py main independently but also, being able to call it one of its method from a root or any other file out of the module.
Consider this structure:
┬ module
| ├ __init__.py
| ├ importing_submodule.py
| └ some_submodule.py
├ __main__.py
├ some_submodule.py
└ module_in_parent_dir.py
with content:
__main__.py
import module
/module/__init__.py
from . import importing_submodule
/module/importing_submodule.py
from some_submodule import SomeClass
/module/some_submodule.py
print("you imported from module")
class SomeClass:
pass
/some_submodule.py
print("you imported from root")
class SomeClass:
pass
/module_in_parent_dir.py
class SomeOtherClass:
pass
How sibling import works
(skip this section if you know already)
Now lets run __main__.py and it will say "you imported from root".
But if we change code a bit..
/module/importing_submodule.py
from module.some_submodule import SomeClass
It now says "You imported from module" as we wanted, probably with scary red line in IDE saying "Unresolved reference" if you didn't config working directory in IDE.
How this happen is simple: script root(Current working directory) is decided by main script(first script that's running), and python uses namespaces.
Python's import system uses 2 import method, and for convenience let's call it absolute import and relative import.
Absolute import: Import from dir listed in sys.path and current working directory
Relative import: Import relative to the very script that called import
And what decide the behavior is whether we use . at start of module name or not.
Since we imported by from some_submodule without preceeding dot, python take it as 'Absolute import'(the term we decided earlier).
And then when we also specified module name like from module.some_submodule python looks for module in path list or in current working directory.
Of course, this is never a good idea; script root can change via calls like os.chdir() then submodules inside module folder may get lost.
Therefore, the best practices for sibling import is using relative import inside module folder.
/module/importing_submodule.py
from .some_submodule import SomeClass
Making script that work in both way
To make submodule import it's siblings when running as main script, yet still work as submodule when imported by other script, then use try - except and look for ImportError.
For importing_submodule.py as an example:
/module/importing_submodule.py
try:
from .some_submodule import SomeClass
except ImportError:
# attempted relative import with no known parent package
# because this is running as main script, there's no parent package.
from some_submodule import SomeClass
Importing modules from parent directory is a bit more tricky.
Since submodule is now main script, relative import to parent level directory doesn't work.
So we need to add the parent directory to sys.path, when the script is running as main script.
/module/importing_submodule.py
try:
from .some_submodule import SomeClass
except ImportError:
# attempted relative import with no known parent package
# because this is running as main script, there's no parent package.
from some_submodule import SomeClass
# now since we don't have parent package, we just append the path.
from sys import path
import pathlib
path.append(pathlib.Path(__file__).parent.parent.as_posix())
print("Importing module_in_parent_dir from sys.path")
else:
print("Importing module_in_parent_dir from working directory")
# Now either case we have parent directory of `module_in_parent_dir`
# in working dir or path, we can import it
# might need to suppress false IDE warning this case.
# noinspection PyUnresolvedReferences
from module_in_parent_dir import SomeOtherClass
Output:
"C:\Program Files\Python310\python.exe" .../module/importing_module.py
you imported from module
Importing module_in_parent_dir from sys.path
Process finished with exit code 0
"C:\Program Files\Python310\python.exe" .../__main__.py
you imported from module
Importing module_in_parent_dir from working directory
Process finished with exit code 0
using pyscafold in order to build a module I get an structure as follows
scr
---module
------__init__.py
------file.py (containing func inside)
tests
---fileTest.py
How is the right way to import file.py in the test file fileTest.py?
So far many of this variations dont work:
import pytest
from ../src/module import func
including this in init.py does not help:
import os, sys
sys.path.append(os.path.dirname(os.path.realpath(__file__)))
These do not help:
from .src/module/myfile import func
from ..src/module/myfile import func
There should not be forward slashes in your imports.
Also:
add a __init__.py file in tests directory
in fileTest.py, you should have:
import pytest
from src.module.file import func
Then, make sure the current working directory is the parent directory of src and run pytest .
I want to add an external package to my program which should include all recipes and third-party packages it uses. I don't want to force anyone (including myself) to install those packages, not to mention the version incopatibilities. I just
want to put them into their own subfolders and use them. They,
of course, come from various sources.
The folder structure should look like this:
| main.py
|---[external]
|---[networkx]
| | ...
|---[xlrd]
| | ...
| __init__.py
| recipe1.py
| recipe2.py
I want to reach the packages and recipes in the following ways in my program:
import external
import external.xlrd
from external import networkx as nx
import external.recipe1
from external import recipe2 as magic
from external import *
Any package however may contain absolute imports which can result ImportError assuming an external has an empty __init__.py:
>>> import external.networkx
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".\external\networkx\__init__.py", line 46, in <module>
from networkx import release
ImportError: No module named 'networkx'
Is there an easy way to make such an external package?
I made a function which does this. All you need to do is to put this line to the __init__.py of the external folder:
__all__ = import_all(__path__)
This function can use generally to import all submodules, even if they are nested, just put this to all __init__.py involved.
If you know a cleaner solution to the question, please share it with us! I won't accept my own answer.
from os import listdir
from os.path import abspath, basename, exists, isdir, join, \
relpath, sep, splitext
from sys import path
def import_all(path__here):
"""Imports all subpackages and submodules, excluding .py
files starting with an underscore. The directories have to
have an __init__.py to get imported.
Add this line to the __init__.py:
__all__ = import_all(__path__)"""
result, packagepath = [], relpath(path__here[0])
for e in listdir(packagepath):
mod_path = join(packagepath, e)
mod_name = splitext(basename(e))[0]
file_to_import = (e.endswith(".py")
and not mod_name.startswith("_"))
dir_to_import = (isdir(mod_path)
and exists(join(mod_path, "__init__.py")))
if not file_to_import and not dir_to_import:
continue
im_str = ".".join(mod_path.split(sep)[:-1] + [mod_name])
try:
__import__(im_str)
except ImportError as err:
# In case of a subpackage countains absolute imports
# assuming it is in the root, we put it into the
# system path and try again.
if abspath(packagepath) not in path:
path.insert(0, abspath(packagepath))
__import__(im_str)
else:
raise err from None
result.append(mod_name)
return result
i have a 2.6 python script and library in the following directory structure:
+ bin
\- foo.py
+ lib
\+ foo
\- bar.py
i would like users to run bin/foo.py to instantiate the classes within lib/foo.py. to achieve this, in my bin/foo.py script i have the following code:
from __future__ import absolute_import
import foo
klass = foo.bar.Klass()
however, this results in:
AttributeError: 'module' object has no attribute 'bar'
ie it thinks that foo is itself rather than the library foo - renaming bin/foo.py to bin/foo-script.py works as expected.
is there a way i can keep the bin/foo.py script and import lib/foo.py?
The current directory is on the path by default, so you need to remove that before you import the other foo module:
import sys
sys.path = [dir for dir in sys.path if dir != '']
Alternatively, prepend the lib directory so that it takes precedence:
import sys
sys.path = ['../lib'] + sys.path
If you just write import foo, it will definitely load the foo module in the current scope. Assuming lib and foo as packages, won't you need to write something like this in order to make it work?
import lib.foo.bar as foobar
klass = foobar.Klass()
I have a directory structure as follows:
| main.py
| scripts
|--| __init__.py
| script1.py
| script2.py
| script3.py
From main.py, the module scripts is imported. I tried using pkgutils.walk_packages in combination with __all__, but using that, I can only import all the submodules directly under main using from scripts import *. I would like to get them all under scripts. What would be the cleanest way to import all the submodules of scripts so that I could access scripts.script1 from main?
EDIT: I am sorry that I was a bit vague. I would like to import the submodules on run-time without specifying them explicitly in __init__.py. I can use pkgutils.walk_packages to get the submodule names (unless someone knows of a better way), but I am not sure of the cleanest way to use these names (or maybe the ImpImporters that walk_packages returns?) to import them.
Edit: Here's one way to recursively import everything at runtime...
(Contents of __init__.py in top package directory)
import pkgutil
__all__ = []
for loader, module_name, is_pkg in pkgutil.walk_packages(__path__):
__all__.append(module_name)
_module = loader.find_module(module_name).load_module(module_name)
globals()[module_name] = _module
I'm not using __import__(__path__+'.'+module_name) here, as it's difficult to properly recursively import packages using it. If you don't have nested sub-packages, and wanted to avoid using globals()[module_name], though, it's one way to do it.
There's probably a better way, but this is the best I can do, anyway.
Original Answer (For context, ignore othwerwise. I misunderstood the question initially):
What does your scripts/__init__.py look like? It should be something like:
import script1
import script2
import script3
__all__ = ['script1', 'script2', 'script3']
You could even do without defining __all__, but things (pydoc, if nothing else) will work more cleanly if you define it, even if it's just a list of what you imported.
This is based on the answer that kolypto provided, but his answer does not perform recursive import of packages, whereas this does. Although not required by the main question, I believe recursive import applies and can be very useful in many similar situations. I, for one, found this question when searching on the topic.
This is a nice, clean way of performing the import of the subpackage's modules, and should be portable as well, and it uses the standard lib for python 2.7+ / 3.x.
import importlib
import pkgutil
def import_submodules(package, recursive=True):
""" Import all submodules of a module, recursively, including subpackages
:param package: package (name or actual module)
:type package: str | module
:rtype: dict[str, types.ModuleType]
"""
if isinstance(package, str):
package = importlib.import_module(package)
results = {}
for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):
full_name = package.__name__ + '.' + name
results[full_name] = importlib.import_module(full_name)
if recursive and is_pkg:
results.update(import_submodules(full_name))
return results
Usage:
# from main.py, as per the OP's project structure
import scripts
import_submodules(scripts)
# Alternatively, from scripts.__init__.py
import_submodules(__name__)
Simply works, and allows relative import inside packages:
def import_submodules(package_name):
""" Import all submodules of a module, recursively
:param package_name: Package name
:type package_name: str
:rtype: dict[types.ModuleType]
"""
package = sys.modules[package_name]
return {
name: importlib.import_module(package_name + '.' + name)
for loader, name, is_pkg in pkgutil.walk_packages(package.__path__)
}
Usage:
__all__ = import_submodules(__name__).keys()
Not nearly as clean as I would like, but none of the cleaner methods worked for me. This achieves the specified behaviour:
Directory structure:
| pkg
|--| __init__.py
| main.py
| scripts
|--| __init__.py
| script1.py
| script2.py
| script3.py
Where pkg/scripts/__init__.py is empty, and pkg/__init__.py contains:
import importlib as _importlib
import pkgutil as _pkgutil
__all__ = [_mod[1].split(".")[-1] for _mod in
filter(lambda _mod: _mod[1].count(".") == 1 and not
_mod[2] and __name__ in _mod[1],
[_mod for _mod in _pkgutil.walk_packages("." + __name__)])]
__sub_mods__ = [".".join(_mod[1].split(".")[1:]) for _mod in
filter(lambda _mod: _mod[1].count(".") > 1 and not
_mod[2] and __name__ in _mod[1],
[_mod for _mod in
_pkgutil.walk_packages("." + __name__)])]
from . import *
for _module in __sub_mods__:
_importlib.import_module("." + _module, package=__name__)
Although it's messy, it should be portable. I've used this code for several different packages.
I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/. Usage is like this:
Include the automodinit package into your setup.py dependencies.
Add the following to the beginning of the __init__.py file:
__all__ = ["I will get rewritten"]
# Don't modify the line above, or this line!
import automodinit
automodinit.automodinit(__name__, __file__, globals())
del automodinit
# Anything else you want can go after here, it won't get modified.
That's it! From now on importing a module will set __all__ to
a list of .py[co] files in the module and will also import each
of those files as though you had typed:
for x in __all__: import x
Therefore the effect of from M import * matches exactly import M.
automodinit is happy running from inside ZIP archives and is therefore ZIP safe.
To just load all submodules of a package, you can use this simple function:
import importlib
import pkgutil
def import_submodules(module):
"""Import all submodules of a module, recursively."""
for loader, module_name, is_pkg in pkgutil.walk_packages(
module.__path__, module.__name__ + '.'):
importlib.import_module(module_name)
Use case: load all database models of a Flask app, so that Flask-Migrate could detect changes to the schema. Usage:
import myproject.models
import_submodules(myproject.models)
I was writing a small personal library and adding new modules all the time so I wrote a shell script to look for scripts and create the __init__.py's. The script is executed just outside of the main directory for my package, pylux.
I know it probably isn't the answer you're looking for, but it servered its purpose for me and it might be useful to someone else, too.
#!/bin/bash
echo 'Traversing folder hierarchy...'
CWD=`pwd`
for directory in `find pylux -type d -exec echo {} \;`;
do
cd $directory
#echo Entering $directory
echo -n "" > __init__.py
for subdirectory in `find . -type d -maxdepth 1 -mindepth 1`;
do
subdirectory=`echo $subdirectory | cut -b 3-`
#echo -n ' ' ...$subdirectory
#echo -e '\t->\t' import $subdirectory
echo import $subdirectory >> __init__.py
done
for pyfile in *.py ;
do
if [ $pyfile = $(echo __init__.py) ]; then
continue
fi
#echo -n ' ' ...$pyfile
#echo -e '\t->\t' import `echo $pyfile | cut -d . -f 1`
echo import `echo $pyfile | cut -d . -f 1` >> __init__.py
done
cd $CWD
done
for directory in `find pylux -type d -exec echo {} \;`;
do
echo $directory/__init__.py:
cat $directory/__init__.py | awk '{ print "\t"$0 }'
done
I've played around with Joe Kington's Answer and have built a solution that uses globals and get/setattr and thus doesn't need eval. A slight modification is that instead of directly using the packages __path__ for walk_packages, I use the packages parent directory and then only import modules starting with __name__ + ".". This was done to reliably get all subpackages from walk_packages - in my use case I had a subpackage named test which caused pkgutil to iterate over the test package from python's library; furthermore, using __path__ would not recurse into the packages subdirectories. All these issues were observed using jython and python2.5, the code below is only tested in jython thus far.
Also note that OPs question only talks about importing all modules from a package, this code recursively imports all packages too.
from pkgutil import walk_packages
from os import path
__all__ = []
__pkg_prefix = "%s." % __name__
__pkg_path = path.abspath(__path__[0]).rsplit("/", 1)[0] #parent directory
for loader, modname, _ in walk_packages([__pkg_path]):
if modname.startswith(__pkg_prefix):
#load the module / package
module = loader.find_module(modname).load_module(modname)
modname = modname[len(__pkg_prefix):] #strip package prefix from name
#append all toplevel modules and packages to __all__
if not "." in modname:
__all__.append(modname)
globals()[modname] = module
#set everything else as an attribute of their parent package
else:
#get the toplevel package from globals()
pkg_name, rest = modname.split(".", 1)
pkg = globals()[pkg_name]
#recursively get the modules parent package via getattr
while "." in rest:
subpkg, rest = rest.split(".", 1)
pkg = getattr(pkg, subpkg)
#set the module (or package) as an attribute of its parent package
setattr(pkg, rest, module)
As a future improvement I'll try to make this dynamic with a __getattr__ hook on the package, so the actual modules are only imported when they are accessed...
This works nicely for me in Python 3.3. Note that this works only for submodules which are in files in the same directory as the __init__.py. With some work however it can be enhanced for supporting submodules in directories too.
from glob import iglob
from os.path import basename, relpath, sep, splitext
def import_submodules(__path__to_here):
"""Imports all submodules.
Import this function in __init__.py and put this line to it:
__all__ = import_submodules(__path__)"""
result = []
for smfile in iglob(relpath(__path__to_here[0]) + "/*.py"):
submodule = splitext(basename(smfile))[0]
importstr = ".".join(smfile.split(sep)[:-1])
if not submodule.startswith("_"):
__import__(importstr + "." + submodule)
result.append(submodule)
return result