I'm writing a python script to import a setting file from another project.
This is the structure of the project:
- root
- ...
- folder_1
- setting_folder
- __init__.py
- setting_1.py
- setting_2.py
- setting_3.py
Here the file contents:
init.py
from .setting_1 import *
from .setting_2 import *
setting_1.py
foo = "foo1"
setting_2.py
foo = "foo2"
try:
from .setting_3 import *
except ImportError:
pass
setting_3.py
foo = "foo3"
What I need to do is, from a script outside the project, load setting_2.py and get the value of foo variable (foo3 due to relative import).
Suppose that I run my script from directory "C:\Users\bar\Desktop".
My idea to achieve this goal is to copy setting_2.py in another directory outside the project (let's say a), create an empty file init.py in a, append to PYTHONPATH "C:\Users\bar\Desktop" and then import the module.
Here the code:
import os
import importlib.util
with open(setting_2_path, "r") as f:
test_file_content = f.read()
setting_tmp_path = r"C:\Users\bar\Desktop\a\setting_2.py"
with open(setting_tmp_path, "w") as f:
f.write(test_file_content)
init_tmp_path = r"C:\Users\bar\Desktop\a\__init__.py"
with open(init_tmp_path, "w") as f:
f.write("")
current_env = os.environ.copy()
current_env.update({'PYTHONPATH': r"C:\Users\bar\Desktop"})
spec_module = importlib.import_module('a.setting_2')
print(getattr(spec_module, "foo"))
This is working well, but in production this script will be in another project and I cannot create a folder at the same level of the script.
I can create the folder but in another directory.
To simulate this scenario, suppose that I run the script from C:\Users\bar\Desktop\bar2 and the folder is C:\Users\bar\Desktop\a.
In this case I have the following error:
ImportError: No module named 'a'
How can I fix the problem?
import os
import sys
cwd = os.getcwd() # This is current working directory, as in your assumption it is: C:\Users\bar\Desktop\bar2
path_to_settings_files = os.path.join(os.path.dirname(cwd), 'a') # This is where settings are, as in your assumption it is: C:\Users\bar\Desktop\a
sys.path.append(path_to_settings_files) # This line guides interpreter to find settings.py in loading modules
import setting_1
import setting_2
import setting_3
print(setting_1.foo)
print(setting_2.foo)
print(setting_3.foo)
I have created a package with the following file structure:
- package
- __init__.py
- load.py
- train.py
- test.py
My __init__.py file is simply an import of classes for these files:
from package.load import Load
from package.train import Train
from package.test import Test
Most of the time, I want to load all three, however on occasion I only want to load one of these classes specifically. For example in an ad hoc script (outside of the package) I want to be able to call only the Load class like so:
from package import Load
While all of the above works in this design, I have an issue where dependencies from train/test are also loaded when I import Load like the above. How can I setup the __init__.py file such that I can make the same import call without getting the dependency to load from train/test?
Additional explanation:
Why I am doing this: I have an issue where I want some people to be able to use the Load class, which only uses base python, however the Train/Test files include specialized dependencies which users of just the Load class wont want to utilize or even install.
Here's a way to do something very close what you want. Instead of unconditionally importing all the package's classes in your __init__.py, you can define a function in it to explicitly import any of the ones desired (or all of them if none are specified).
__init__.py:
from pathlib import Path
import sys
print(f'In {Path(__file__).name}')
package_name = Path(__file__).parent.name
package_prefix = package_name + '.'
class_to_module_map = {'Load': 'load', 'Train': 'train', 'Test': 'test'}
def import_classes(*class_names):
namespace = sys._getframe(1).f_globals # Caller's globals.
if not class_names:
class_names = class_to_module_map.keys() # Import them all.
for class_name in class_names:
module = class_to_module_map[class_name]
temp = __import__(package_prefix+module, globals(), locals(), [class_name])
namespace[class_name] = getattr(temp, class_name) # Add to caller's namespace.
For testing purposes, here's what I put in the load.py script:
(I also put something similar in the other two modules in order to verify whether or not they were getting imported.)
load.py:
from pathlib import Path
print(f'In {Path(__file__).name}')
class Load: pass
And finally here's a example of using it to only import the Load class:
ad_hoc.py:
from my_package import import_classes
#from my_package import Load
import_classes('Load')
test = Load()
print(test)
Along with the output produced:
In __init__.py
In load.py
<my_package.load.Load object at 0x001FE4A8>
Inside a folder oranges\, this is our __init__.py file:
__all__ = []
from pathlib import Path
from importlib import import_module
from sys import modules
package = modules[__name__]
initfile = Path(__file__)
for entry in initfile.parent.iterdir():
is_file = entry.is_file()
is_pyfile = entry.name.endswith('.py')
is_not_initpy = (entry != initfile)
if is_file and is_pyfile and is_not_initpy:
module_name = entry.name.removesuffix('.py')
module_path = __name__ + '.' + module_name
module = import_module(module_path)
setattr(package, module_name, module)
__all__.append(module_name)
When we do from oranges import *, the code inside oranges\__init__.py cycles through the *.py files inside oranges\ (except __init__.py), and for each .py file does the following:
imports the .py file as a module into the variable module using Python's importlib.import_module
sets the module as a variable within the __init__.py file (or, more correctly, within the oranges package) using Python's setattr
finally, appends the module to the __all__ list
I have this problem and do not know how to solve it efficiently.
I've this file structure
THE NUMBER OF FOLDERS NOR THE NAMES ARE GIVEN, IT'S ALL UNKNOWN
app/
__init__.py
modules/
__init__.py
ModuleA/
__init__.py
file.py
otherfile.py
config.ini
ModuleB/
__init__.py
file.py
otherfile.py
config.ini
ModuleC/
__init__.py
file.py
otherfile.py
config.ini
**arbitrary number of modules with same structure*
As you can notiche, app is the main package of my app, but I need an efficient way to import the mods folder and its' content
* My actual solution *
from app import modules ad mods
def load_modules_from_packages(self, pkg = mods):
pkgname = pkg.__name__
pkgpath = dirname(pkg.__file__)
for loader,name,ispkg in pkgutil.walk_packages(pkg.__path__, pkgname+'.'):
if ispkg is True:
__import__(name,globals(),locals(),[],0)
elif ispkg is False:
__import__(name,globals(),locals(),[],0)
This works since pkgutil iterate the structure with the dot notation for names, so import works well.
But now I want load infos in the config file if I am in one of the somemodule folder(the one with own init.py and config.ini
I want to do this to recreate the structure of module package and output it in a JSON rapresentation for another thing
* my other solution does not works*
def load_modules_from_packages(directory)
dir_path = dirname(directory.__file__)
dir_name = directory.__name__
for filename in glob.glob(dir_path + '/**/*.ini', recursive=True):
plugin = {}
plugin['name'] = filename.split('/')[-2]
plugin['path'] = dirname(filename)
plugin['config_file'] = filename
for pyname in glob.glob(dirname(filename)+ '/**/*.py', recursive=True):
importlib.import_module(pyname)
I cant use the solution posted in this thread
How to import a module given the full path?
since I do not know the module name, ad pointed without solutions in the comment.
spec = importlib.util.spec_from_file_location('what.ever', 'foo.py')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
I know 'foo.py' but I cant figure out 'what.ever' like pkgutil.walk_package does.
In fact the modules imported in this way have the package and name entry wrong. With this approach I cant figure out where I am in the file structure to create modules dictionary and the relative modules (for the JSON output)
Any help?
Could someone provide me with a good way of importing a whole directory of modules?
I have a structure like this:
/Foo
bar.py
spam.py
eggs.py
I tried just converting it to a package by adding __init__.py and doing from Foo import * but it didn't work the way I had hoped.
List all python (.py) files in the current folder and put them as __all__ variable in __init__.py
from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]
Add the __all__ Variable to __init__.py containing:
__all__ = ["bar", "spam", "eggs"]
See also http://docs.python.org/tutorial/modules.html
Update in 2017: you probably want to use importlib instead.
Make the Foo directory a package by adding an __init__.py. In that __init__.py add:
import bar
import eggs
import spam
Since you want it dynamic (which may or may not be a good idea), list all py-files with list dir and import them with something like this:
import os
for module in os.listdir(os.path.dirname(__file__)):
if module == '__init__.py' or module[-3:] != '.py':
continue
__import__(module[:-3], locals(), globals())
del module
Then, from your code do this:
import Foo
You can now access the modules with
Foo.bar
Foo.eggs
Foo.spam
etc. from Foo import * is not a good idea for several reasons, including name clashes and making it hard to analyze the code.
Python, include all files under a directory:
For newbies who just can't get it to work who need their hands held.
Make a folder /home/el/foo and make a file main.py under /home/el/foo Put this code in there:
from hellokitty import *
spam.spamfunc()
ham.hamfunc()
Make a directory /home/el/foo/hellokitty
Make a file __init__.py under /home/el/foo/hellokitty and put this code in there:
__all__ = ["spam", "ham"]
Make two python files: spam.py and ham.py under /home/el/foo/hellokitty
Define a function inside spam.py:
def spamfunc():
print("Spammity spam")
Define a function inside ham.py:
def hamfunc():
print("Upgrade from baloney")
Run it:
el#apollo:/home/el/foo$ python main.py
spammity spam
Upgrade from baloney
Expanding on Mihail's answer, I believe the non-hackish way (as in, not handling the file paths directly) is the following:
create an empty __init__.py file under Foo/
Execute
import pkgutil
import sys
def load_all_modules_from_dir(dirname):
for importer, package_name, _ in pkgutil.iter_modules([dirname]):
full_package_name = '%s.%s' % (dirname, package_name)
if full_package_name not in sys.modules:
module = importer.find_module(package_name
).load_module(full_package_name)
print module
load_all_modules_from_dir('Foo')
You'll get:
<module 'Foo.bar' from '/home/.../Foo/bar.pyc'>
<module 'Foo.spam' from '/home/.../Foo/spam.pyc'>
I know I'm updating a quite old post, and I tried using automodinit, but found out it's setup process is broken for python3. So, based on Luca's answer, I came up with a simpler answer - which might not work with .zip - to this issue, so I figured I should share it here:
within the __init__.py module from yourpackage:
#!/usr/bin/env python
import os, pkgutil
__all__ = list(module for _, module, _ in pkgutil.iter_modules([os.path.dirname(__file__)]))
and within another package below yourpackage:
from yourpackage import *
Then you'll have all the modules that are placed within the package loaded, and if you write a new module, it'll be automagically imported as well. Of course, use that kind of things with care, with great powers comes great responsibilities.
I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/.
Usage is like this:
Include the automodinit package into your setup.py dependencies.
Replace all __init__.py files like this:
__all__ = ["I will get rewritten"]
# Don't modify the line above, or this line!
import automodinit
automodinit.automodinit(__name__, __file__, globals())
del automodinit
# Anything else you want can go after here, it won't get modified.
That's it! From now on importing a module will set __all__ to
a list of .py[co] files in the module and will also import each
of those files as though you had typed:
for x in __all__: import x
Therefore the effect of "from M import *" matches exactly "import M".
automodinit is happy running from inside ZIP archives and is therefore ZIP safe.
Niall
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
for imp, module, ispackage in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
__import__(module)
I have also encountered this problem and this was my solution:
import os
def loadImports(path):
files = os.listdir(path)
imps = []
for i in range(len(files)):
name = files[i].split('.')
if len(name) > 1:
if name[1] == 'py' and name[0] != '__init__':
name = name[0]
imps.append(name)
file = open(path+'__init__.py','w')
toWrite = '__all__ = '+str(imps)
file.write(toWrite)
file.close()
This function creates a file (in the provided folder) named __init__.py, which contains an __all__ variable that holds every module in the folder.
For example, I have a folder named Test
which contains:
Foo.py
Bar.py
So in the script I want the modules to be imported into I will write:
loadImports('Test/')
from Test import *
This will import everything from Test and the __init__.py file in Test will now contain:
__all__ = ['Foo','Bar']
When from . import * isn't good enough, this is an improvement over the answer by ted. Specifically, the use of __all__ is not necessary with this approach.
"""Import all modules that exist in the current directory."""
# Ref https://stackoverflow.com/a/60861023/
from importlib import import_module
from pathlib import Path
for f in Path(__file__).parent.glob("*.py"):
module_name = f.stem
if (not module_name.startswith("_")) and (module_name not in globals()):
import_module(f".{module_name}", __package__)
del f, module_name
del import_module, Path
Note that module_name not in globals() is intended to avoid reimporting the module if it's already imported, as this can risk cyclic imports.
This is the best way i've found so far:
from os.path import dirname, join, isdir, abspath, basename
from glob import glob
pwd = dirname(__file__)
for x in glob(join(pwd, '*.py')):
if not x.startswith('__'):
__import__(basename(x)[:-3], globals(), locals())
Anurag Uniyal answer with suggested improvements!
#!/usr/bin/python
# -*- encoding: utf-8 -*-
import os
import glob
all_list = list()
for f in glob.glob(os.path.dirname(__file__)+"/*.py"):
if os.path.isfile(f) and not os.path.basename(f).startswith('_'):
all_list.append(os.path.basename(f)[:-3])
__all__ = all_list
Using importlib the only thing you've got to add is
from importlib import import_module
from pathlib import Path
__all__ = [
import_module(f".{f.stem}", __package__)
for f in Path(__file__).parent.glob("*.py")
if "__" not in f.stem
]
del import_module, Path
I've created a module for that, which doesn't rely on __init__.py (or any other auxiliary file) and makes me type only the following two lines:
import importdir
importdir.do("Foo", globals())
Feel free to re-use or contribute: http://gitlab.com/aurelien-lourot/importdir
Anurag's example with a couple of corrections:
import os, glob
modules = glob.glob(os.path.join(os.path.dirname(__file__), "*.py"))
__all__ = [os.path.basename(f)[:-3] for f in modules if not f.endswith("__init__.py")]
I'd like to add to Anurag Uniyal's answer.
You can make it even simpler and get rid of a lot of the imports.
Contents of the __init__.py file:
from os import listdir
from os.path import dirname
__all__ = [i[:-3] for i in listdir(dirname(__file__)) if not i.startswith('__') and i.endswith('.py')]
See that your __init__.py defines __all__. The modules - packages doc says
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.
...
The only solution is for the package author to provide an explicit index of the package. The import statement uses the following convention: if a package’s __init__.py code defines a list named __all__, it is taken to be the list of module names that should be imported when from package import * is encountered. It is up to the package author to keep this list up-to-date when a new version of the package is released. Package authors may also decide not to support it, if they don’t see a use for importing * from their package. For example, the file sounds/effects/__init__.py could contain the following code:
__all__ = ["echo", "surround", "reverse"]
This would mean that from sound.effects import * would import the three named submodules of the sound package.
Look at the pkgutil module from the standard library. It will let you do exactly what you want as long as you have an __init__.py file in the directory. The __init__.py file can be empty.
Just import them by importlib and add them to __all__ (add action is optional) in recurse in the __init__.py of package.
/Foo
bar.py
spam.py
eggs.py
__init__.py
# __init__.py
import os
import importlib
pyfile_extes = ['py', ]
__all__ = [importlib.import_module('.%s' % filename, __package__) for filename in [os.path.splitext(i)[0] for i in os.listdir(os.path.dirname(__file__)) if os.path.splitext(i)[1] in pyfile_extes] if not filename.startswith('__')]
del os, importlib, pyfile_extes
I had a nested directory structure i.e. I had multiple directories inside the main directory that contained the python modules.
I added the following script to my __init__.py file to import all the modules
import glob, re, os
module_parent_directory = "path/to/the/directory/containing/__init__.py/file"
owd = os.getcwd()
if not owd.endswith(module_parent_directory): os.chdir(module_parent_directory)
module_paths = glob.glob("**/*.py", recursive = True)
for module_path in module_paths:
if not re.match( ".*__init__.py$", module_path):
import_path = module_path[:-3]
import_path = import_path.replace("/", ".")
exec(f"from .{import_path} import *")
os.chdir(owd)
Probably not the best way to achieve this, but I couldn't make anything else work for me.
Here is a solution, with which you do not have to write the file name. Just add this code snippet to your __init__.py
from inspect import isclass
from pkgutil import iter_modules
from pathlib import Path
from importlib import import_module
# iterate through the modules in the current package
package_dir = Path(__file__).resolve().parent
for (_, module_name, _) in iter_modules([package_dir]):
# import the module and iterate through its attributes
module = import_module(f"{__name__}.{module_name}")
for attribute_name in dir(module):
attribute = getattr(module, attribute_name)
if isclass(attribute):
# Add the class to this package's variables
globals()[attribute_name] = attribute
Source
None of the solutions was working for me in Python 3.9.5, Flask 2.2.2, the module being a directory 2 levels down the cwd.
This is my solution:
import importlib
import pathlib
import re
path = pathlib.Path(__file__).parent.absolute()
names = [x.name[:-3] for x in path.iterdir() if x.is_file() and re.search("^[a-z]*\.py$", x.name)]
for name in names:
importlib.import_module(f".{name}", __name__)