I am trying to get a list of the imported custom modules (modules I created by myself) in my current file but I could not find a proper way to achieve it.
For example:
<test.py>
import sys
import foo
import bar
I would like to get a list of [foo, bar] excluding the sys module.
Lets say, that if file located near curent file, or in some subfolder, when it's "our" module, not system one.
For this example, I've created three files: main.py, other.py and another.py, and placed whem into one folder.
The code of main.py is:
# os and sys are needed to work
import sys
import os
import shutil
import io
import datetime
import other
import another
def get_my_modules():
# Get list of modules loaded
modules = list(sys.modules.keys())
mymodules = []
# Get current dir
curdir = os.path.realpath(os.path.dirname(__file__))
for m in modules:
try:
# if some module's file path located in current folder or in some subfolder
# lets sey, that it's our self-made module
path = sys.modules[m].__file__
if path.startswith(curdir):
mymodules.append(m)
except Exception:
# Exception could happen if module doesn't have any __file__ property
pass
# Return list of our moudles
return mymodules
print(get_my_modules())
And this code actually outputs ["other", "another"]
This approach has a problem: If you import module, that somehow located in upper folder, it woudln't be detected.
There are multiple threads about getting the current Python's script directory, for example:
import os
dir = os.path.dirname(os.path.abspath(__file__))
The question is, what should I do if I want to add this function into some utility file, while I want the returned value to be the directory of the calling file, in this case, the file executed by the user?
What would happen here?
// utils.py
def get_script_dir():
import os
return os.path.dirname(os.path.abspath(__file__))
// main_app.py
from utils import get_script_dir
print(get_script_dir())
// user's shell
python c:\path\to\somewhere\main_app.py
Will it print the directory of main_app.py or the directory of utils.py? What is an elegant solution for placing the function in the utils file while getting the directory of the file that was actually executed by the user?
Please try following and let me know whether it's exactly what you want:
# utils.py
def get_script_dir():
import sys
import os
return os.path.dirname(sys.modules['__main__'].__file__)
# main_app.py
from utils import get_script_dir
print(get_script_dir())
Supposing that your project directory tree is like the following:
Python/
main.py
modules/
utils.py
__init__.py
Being Python/modules/utils.py:
import os
get_script_dir = lambda file: os.path.dirname(os.path.abspath(file))
and Python/main.py:
from modules import utils
import os
script_dir = utils.get_script_dir(__file__)
print("[i] Currently executing file {} located at {}".format(os.path.basename(__file__), script_dir))
Executing Python/main.py is going to output something like:
[i] Currently executing file main.py located at C:\Users\BlackVirusScript\Desktop\Python
My flask application is one big lists of files right now and I am trying to divide it up into two primary folders, app and spiders so that it looks like this:
- app
-- __init__.py
-- app.py
-- models.py
- spiders
-- __init__.py
-- scrape.py
Procfile
requirements.txt
The problem I am having is when trying to import my models into the scrape.py file.
When I try to import it absolutely with:
from app.models import Rate, Hotel, Location I get the error 'no module named app'.
When I try to import relatively:
from ..app.models import Rate, Hotel, Location I get the error SystemError: Parent module '' not loaded, cannot perform relative import.
What can I do to fix this?
This should work:
import sys
from os.path import dirname, abspath, sep
sys.path.append(abspath(dirname(__file__) + sep + ".."))
import app.models as models
Note: This is really an ugly solution as it messes up your import block in the file. I suggest to add a main code file in the root folder like main.py and import spiders and app modules in there. You should not spread your flow controllers among files in sub folders
I have a directory which has all the files:
myDirectory/
directory1/
importantFile.py
Output.py
How can I import Output.py from importantFile.py without having to put in the same directory?
importantFile.py
import Output
Output.write('This worked!')
Output.py
class Output():
def writeOutput(s):
print s
if "call" is import, in Output.py
import sys
import os.path
# change how import path is resolved by adding the subdirectory
sys.path.append(os.path.abspath(os.getcwd()+'/directory1'))
import importantFile
importantFile.f()
sys.path contains the list of path where to look for modules, details in https://docs.python.org/2/library/sys.html
The other way is to use the relative notation, for which the python file you want to import should be in a package.
You have to make the directory a python package by putting an init.py file.
Look for the packages section in this link.
https://docs.python.org/2/tutorial/modules.html
import sys
sys.path.append('/full/path/to/use')
global exist_importedname
exist_importedname = True
try:
import myimport
except ImportError as e:
exist_importedname = False
print (e.message)
Could someone provide me with a good way of importing a whole directory of modules?
I have a structure like this:
/Foo
bar.py
spam.py
eggs.py
I tried just converting it to a package by adding __init__.py and doing from Foo import * but it didn't work the way I had hoped.
List all python (.py) files in the current folder and put them as __all__ variable in __init__.py
from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]
Add the __all__ Variable to __init__.py containing:
__all__ = ["bar", "spam", "eggs"]
See also http://docs.python.org/tutorial/modules.html
Update in 2017: you probably want to use importlib instead.
Make the Foo directory a package by adding an __init__.py. In that __init__.py add:
import bar
import eggs
import spam
Since you want it dynamic (which may or may not be a good idea), list all py-files with list dir and import them with something like this:
import os
for module in os.listdir(os.path.dirname(__file__)):
if module == '__init__.py' or module[-3:] != '.py':
continue
__import__(module[:-3], locals(), globals())
del module
Then, from your code do this:
import Foo
You can now access the modules with
Foo.bar
Foo.eggs
Foo.spam
etc. from Foo import * is not a good idea for several reasons, including name clashes and making it hard to analyze the code.
Python, include all files under a directory:
For newbies who just can't get it to work who need their hands held.
Make a folder /home/el/foo and make a file main.py under /home/el/foo Put this code in there:
from hellokitty import *
spam.spamfunc()
ham.hamfunc()
Make a directory /home/el/foo/hellokitty
Make a file __init__.py under /home/el/foo/hellokitty and put this code in there:
__all__ = ["spam", "ham"]
Make two python files: spam.py and ham.py under /home/el/foo/hellokitty
Define a function inside spam.py:
def spamfunc():
print("Spammity spam")
Define a function inside ham.py:
def hamfunc():
print("Upgrade from baloney")
Run it:
el#apollo:/home/el/foo$ python main.py
spammity spam
Upgrade from baloney
Expanding on Mihail's answer, I believe the non-hackish way (as in, not handling the file paths directly) is the following:
create an empty __init__.py file under Foo/
Execute
import pkgutil
import sys
def load_all_modules_from_dir(dirname):
for importer, package_name, _ in pkgutil.iter_modules([dirname]):
full_package_name = '%s.%s' % (dirname, package_name)
if full_package_name not in sys.modules:
module = importer.find_module(package_name
).load_module(full_package_name)
print module
load_all_modules_from_dir('Foo')
You'll get:
<module 'Foo.bar' from '/home/.../Foo/bar.pyc'>
<module 'Foo.spam' from '/home/.../Foo/spam.pyc'>
I know I'm updating a quite old post, and I tried using automodinit, but found out it's setup process is broken for python3. So, based on Luca's answer, I came up with a simpler answer - which might not work with .zip - to this issue, so I figured I should share it here:
within the __init__.py module from yourpackage:
#!/usr/bin/env python
import os, pkgutil
__all__ = list(module for _, module, _ in pkgutil.iter_modules([os.path.dirname(__file__)]))
and within another package below yourpackage:
from yourpackage import *
Then you'll have all the modules that are placed within the package loaded, and if you write a new module, it'll be automagically imported as well. Of course, use that kind of things with care, with great powers comes great responsibilities.
I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/.
Usage is like this:
Include the automodinit package into your setup.py dependencies.
Replace all __init__.py files like this:
__all__ = ["I will get rewritten"]
# Don't modify the line above, or this line!
import automodinit
automodinit.automodinit(__name__, __file__, globals())
del automodinit
# Anything else you want can go after here, it won't get modified.
That's it! From now on importing a module will set __all__ to
a list of .py[co] files in the module and will also import each
of those files as though you had typed:
for x in __all__: import x
Therefore the effect of "from M import *" matches exactly "import M".
automodinit is happy running from inside ZIP archives and is therefore ZIP safe.
Niall
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
for imp, module, ispackage in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
__import__(module)
I have also encountered this problem and this was my solution:
import os
def loadImports(path):
files = os.listdir(path)
imps = []
for i in range(len(files)):
name = files[i].split('.')
if len(name) > 1:
if name[1] == 'py' and name[0] != '__init__':
name = name[0]
imps.append(name)
file = open(path+'__init__.py','w')
toWrite = '__all__ = '+str(imps)
file.write(toWrite)
file.close()
This function creates a file (in the provided folder) named __init__.py, which contains an __all__ variable that holds every module in the folder.
For example, I have a folder named Test
which contains:
Foo.py
Bar.py
So in the script I want the modules to be imported into I will write:
loadImports('Test/')
from Test import *
This will import everything from Test and the __init__.py file in Test will now contain:
__all__ = ['Foo','Bar']
When from . import * isn't good enough, this is an improvement over the answer by ted. Specifically, the use of __all__ is not necessary with this approach.
"""Import all modules that exist in the current directory."""
# Ref https://stackoverflow.com/a/60861023/
from importlib import import_module
from pathlib import Path
for f in Path(__file__).parent.glob("*.py"):
module_name = f.stem
if (not module_name.startswith("_")) and (module_name not in globals()):
import_module(f".{module_name}", __package__)
del f, module_name
del import_module, Path
Note that module_name not in globals() is intended to avoid reimporting the module if it's already imported, as this can risk cyclic imports.
This is the best way i've found so far:
from os.path import dirname, join, isdir, abspath, basename
from glob import glob
pwd = dirname(__file__)
for x in glob(join(pwd, '*.py')):
if not x.startswith('__'):
__import__(basename(x)[:-3], globals(), locals())
Anurag Uniyal answer with suggested improvements!
#!/usr/bin/python
# -*- encoding: utf-8 -*-
import os
import glob
all_list = list()
for f in glob.glob(os.path.dirname(__file__)+"/*.py"):
if os.path.isfile(f) and not os.path.basename(f).startswith('_'):
all_list.append(os.path.basename(f)[:-3])
__all__ = all_list
Using importlib the only thing you've got to add is
from importlib import import_module
from pathlib import Path
__all__ = [
import_module(f".{f.stem}", __package__)
for f in Path(__file__).parent.glob("*.py")
if "__" not in f.stem
]
del import_module, Path
I've created a module for that, which doesn't rely on __init__.py (or any other auxiliary file) and makes me type only the following two lines:
import importdir
importdir.do("Foo", globals())
Feel free to re-use or contribute: http://gitlab.com/aurelien-lourot/importdir
Anurag's example with a couple of corrections:
import os, glob
modules = glob.glob(os.path.join(os.path.dirname(__file__), "*.py"))
__all__ = [os.path.basename(f)[:-3] for f in modules if not f.endswith("__init__.py")]
I'd like to add to Anurag Uniyal's answer.
You can make it even simpler and get rid of a lot of the imports.
Contents of the __init__.py file:
from os import listdir
from os.path import dirname
__all__ = [i[:-3] for i in listdir(dirname(__file__)) if not i.startswith('__') and i.endswith('.py')]
See that your __init__.py defines __all__. The modules - packages doc says
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.
...
The only solution is for the package author to provide an explicit index of the package. The import statement uses the following convention: if a package’s __init__.py code defines a list named __all__, it is taken to be the list of module names that should be imported when from package import * is encountered. It is up to the package author to keep this list up-to-date when a new version of the package is released. Package authors may also decide not to support it, if they don’t see a use for importing * from their package. For example, the file sounds/effects/__init__.py could contain the following code:
__all__ = ["echo", "surround", "reverse"]
This would mean that from sound.effects import * would import the three named submodules of the sound package.
Look at the pkgutil module from the standard library. It will let you do exactly what you want as long as you have an __init__.py file in the directory. The __init__.py file can be empty.
Just import them by importlib and add them to __all__ (add action is optional) in recurse in the __init__.py of package.
/Foo
bar.py
spam.py
eggs.py
__init__.py
# __init__.py
import os
import importlib
pyfile_extes = ['py', ]
__all__ = [importlib.import_module('.%s' % filename, __package__) for filename in [os.path.splitext(i)[0] for i in os.listdir(os.path.dirname(__file__)) if os.path.splitext(i)[1] in pyfile_extes] if not filename.startswith('__')]
del os, importlib, pyfile_extes
I had a nested directory structure i.e. I had multiple directories inside the main directory that contained the python modules.
I added the following script to my __init__.py file to import all the modules
import glob, re, os
module_parent_directory = "path/to/the/directory/containing/__init__.py/file"
owd = os.getcwd()
if not owd.endswith(module_parent_directory): os.chdir(module_parent_directory)
module_paths = glob.glob("**/*.py", recursive = True)
for module_path in module_paths:
if not re.match( ".*__init__.py$", module_path):
import_path = module_path[:-3]
import_path = import_path.replace("/", ".")
exec(f"from .{import_path} import *")
os.chdir(owd)
Probably not the best way to achieve this, but I couldn't make anything else work for me.
Here is a solution, with which you do not have to write the file name. Just add this code snippet to your __init__.py
from inspect import isclass
from pkgutil import iter_modules
from pathlib import Path
from importlib import import_module
# iterate through the modules in the current package
package_dir = Path(__file__).resolve().parent
for (_, module_name, _) in iter_modules([package_dir]):
# import the module and iterate through its attributes
module = import_module(f"{__name__}.{module_name}")
for attribute_name in dir(module):
attribute = getattr(module, attribute_name)
if isclass(attribute):
# Add the class to this package's variables
globals()[attribute_name] = attribute
Source
None of the solutions was working for me in Python 3.9.5, Flask 2.2.2, the module being a directory 2 levels down the cwd.
This is my solution:
import importlib
import pathlib
import re
path = pathlib.Path(__file__).parent.absolute()
names = [x.name[:-3] for x in path.iterdir() if x.is_file() and re.search("^[a-z]*\.py$", x.name)]
for name in names:
importlib.import_module(f".{name}", __name__)