Develop Reference

Set path of dynamically loaded module in Python 3.7

OK, so this will be a little tricky to explan well. I will try my best.
So consider my naive implementation of "load python module by filename" as follows:
def load_module_by_filepath(module_name, module_filepath):
module=None
failure=None
try:
spec = importlib.util.spec_from_file_location(module_name, module_filepath)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
except Exception as e:
failure=f"Import of module '{module_name}'from file '{module_filepath}' had errors ({e})"
module=None
return module, failure
Let's say I have the following code in a file my_mod.py:
from my_other_mod import something
def somefunc():
something()
Then I load it using my loader like this:
module, failure = load_module_by_filepath('my_mod', 'my_mod.py')
At this point I will be able to call somefunc from my loaded module like so:
my_callable = getattr(module, 'somefunc')
my_callable()
All good, except, if you look carefully there is an import statement in my_mod.py for something from my_other_mod.
So what happens when I run this code is that I get a failure to load the module saying "No module 'my_other_mod'".
I am guessing this is because my_other_mod was installed in a location that is not available to my newly loaded module. They don't belong to the same environment or context if you will.
So my question is, how can I change my loader to add my_other_mod to the path of my_mod so that it will load and run?

How do I use 'from /path/module import function' where my /path/module is stored inside of a variable in python? [duplicate]

I'm working on a documentation (personal) for nested matplotlib (MPL) library, which differs from MPL own provided, by interested submodule packages. I'm writing Python script which I hope will automate document generation from future MPL releases.
I selected interested submodules/packages and want to list their main classes from which I'll generate list and process it with pydoc
Problem is that I can't find a way to instruct Python to load submodule from string. Here is example of what I tried:
import matplotlib.text as text
x = dir(text)
.
i = __import__('matplotlib.text')
y = dir(i)
.
j = __import__('matplotlib')
z = dir(j)
And here is 3 way comparison of above lists through pprint:
I don't understand what's loaded in y object - it's base matplotlib plus something else, but it lack information that I wanted and that is main classes from matplotlib.text package. It's top blue coloured part on screenshot (x list)
Please don't suggest Sphinx as different approach.

The __import__ function can be a bit hard to understand.
If you change
i = __import__('matplotlib.text')
to
i = __import__('matplotlib.text', fromlist=[''])
then i will refer to matplotlib.text.
In Python 3.1 or later, you can use importlib:
import importlib
i = importlib.import_module("matplotlib.text")
Some notes
If you're trying to import something from a sub-folder e.g. ./feature/email.py, the code will look like importlib.import_module("feature.email")
Before Python 3.3 you could not import anything if there was no __init__.py in the folder with file you were trying to import (see caveats before deciding if you want to keep the file for backward compatibility e.g. with pytest).

importlib.import_module is what you are looking for. It returns the imported module.
import importlib
# equiv. of your `import matplotlib.text as text`
text = importlib.import_module('matplotlib.text')
You can thereafter access anything in the module as text.myclass, text.myfunction, etc.

spent some time trying to import modules from a list, and this is the thread that got me most of the way there - but I didnt grasp the use of ___import____ -
so here's how to import a module from a string, and get the same behavior as just import. And try/except the error case, too. :)
pipmodules = ['pycurl', 'ansible', 'bad_module_no_beer']
for module in pipmodules:
try:
# because we want to import using a variable, do it this way
module_obj = __import__(module)
# create a global object containging our module
globals()[module] = module_obj
except ImportError:
sys.stderr.write("ERROR: missing python module: " + module + "\n")
sys.exit(1)
and yes, for python 2.7> you have other options - but for 2.6<, this works.

Apart from using the importlib one can also use exec method to import a module from a string variable.
Here I am showing an example of importing the combinations method from itertools package using the exec method:
MODULES = [
['itertools','combinations'],
]
for ITEM in MODULES:
import_str = "from {0} import {1}".format(ITEM[0],', '.join(str(i) for i in ITEM[1:]))
exec(import_str)
ar = list(combinations([1, 2, 3, 4], 2))
for elements in ar:
print(elements)
Output:
(1, 2)
(1, 3)
(1, 4)
(2, 3)
(2, 4)
(3, 4)

Module auto-install & import from list
Below script works fine with both submodules and pseudo submodules.
# PyPI imports
import pkg_resources, subprocess, sys
modules = {'lxml.etree', 'pandas', 'screeninfo'}
required = {m.split('.')[0] for m in modules}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', 'pip'])
subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])
for module in set.union(required, modules):
globals()[module] = __import__(module)
Tests:
print(pandas.__version__)
print(lxml.etree.LXML_VERSION)

I developed these 3 useful functions:
def loadModule(moduleName):
module = None
try:
import sys
del sys.modules[moduleName]
except BaseException as err:
pass
try:
import importlib
module = importlib.import_module(moduleName)
except BaseException as err:
serr = str(err)
print("Error to load the module '" + moduleName + "': " + serr)
return module
def reloadModule(moduleName):
module = loadModule(moduleName)
moduleName, modulePath = str(module).replace("' from '", "||").replace("<module '", '').replace("'>", '').split("||")
if (modulePath.endswith(".pyc")):
import os
os.remove(modulePath)
module = loadModule(moduleName)
return module
def getInstance(moduleName, param1, param2, param3):
module = reloadModule(moduleName)
instance = eval("module." + moduleName + "(param1, param2, param3)")
return instance
And everytime I want to reload a new instance I just have to call getInstance() like this:
myInstance = getInstance("MyModule", myParam1, myParam2, myParam3)
Finally I can call all the functions inside the new Instance:
myInstance.aFunction()
The only specificity here is to customize the params list (param1, param2, param3) of your instance.

You can also use exec built-in function that execute any string as a Python code.
In [1]: module = 'pandas'
...: function = 'DataFrame'
...: alias = 'DF'
In [2]: exec(f"from {module} import {function} as {alias}")
In [3]: DF
Out[3]: pandas.core.frame.DataFrame
For me this was the most readable way to solve my problem.

How to Import Exceptions from Package (Openpyxl)?

I am trying to catch a sheet error exception for the package I am using (Openpyxl). I tried importing the exception like so from openpyxl.utils import SheetTitleException but I get the error "ImportError: cannot import name SheetTitleException". When I tried importing it just with from openpyxl.utils import *, I get the error NameError: global name 'SheetTitleException' is not defined.
I'm sure I'm importing it incorrectly, but I'm not sure where I'm going wrong.
Here is the documentation on exceptions for Openpyxl.
And here is the code I am using to catch the exception:
try:
bdws = bdwb[finalBDSheetName]
except SheetTitleException:
messageBox("Invalid sheet title. Check your sheet title and try again.")
return

The title of the page you linked to says "openpyxl.utils.exceptions".
Therefore you should be doing:
from openpyxl.utils.exceptions import SheetTitleException

If it's anything like other module exception handling that i've done it should be
from openpyxl.utils.exceptions import SheetTitleException
then to use it
except SheetTitleException as e:
# do something

How to import modules given a list of their names?

I am trying to import modules from a list. This would be to allow easier editing of imported modules, cleaner error message, and better error handling. Here is basically what I am trying to do:
imports = ['sys', 'itertools', 'datetime', 'os']
for x in imports:
try:
import x
print "Successfully imported ", x, '.'
except ImportError:
print "Error importing ", x, '.'
The issue here is that it tries importing x, not the value x should hold. I realize that to import from the list I could do something like below, but I do not see a way to handle the errors with it:
imports = ['sys', 'itertools', 'datetime', 'os']
modules = map(__import__, imports)
Is there a way to integrate the error handling with this method or should I try a different approach?

Instead of mapping them to ___import__ all at once, just append each module to the list modules one at a time inside the for-loop:
imports = ['sys', 'itertools', 'datetime', 'os']
modules = []
for x in imports:
try:
modules.append(__import__(x))
print "Successfully imported ", x, '.'
except ImportError:
print "Error importing ", x, '.'
Note however that most Python programmers prefer the use of importlib.import_module rather than __import__ for this task.
Note too that it might be better to make modules a dictionary instead of a list:
imports = ['sys', 'itertools', 'datetime', 'os']
modules = {}
for x in imports:
try:
modules[x] = __import__(x)
print "Successfully imported ", x, '.'
except ImportError:
print "Error importing ", x, '.'
Now, instead of by index:
modules[0].version
modules[3].path
you can access the modules by name:
modules["sys"].version
modules["os"].path

None of the most voted options worked for me. They seemed to be imported successfully, but I couldn't use them later. In case you experienced the same issue, this tutorial solved it for me.
modules = ['sys', 'itertools', 'datetime', 'os']
for lib in modules:
globals()[lib] = __import__(lib)
PS: I guess they were not added to my global variables before

you can import programatically achieving the same effect as import module as module_shortname by using globals():
packages_to_import = [{'name': 'numpy', 'as': 'np'},
{'name': 'pandas', 'as': 'pd'}]
for package in packages_to_import:
package_name = package['name']
import_as = package.get('as', package_name)
globals()[import_as] = __import__(package_name)
print(np.version.full_version)
print(pd.__version__)

This worked for me on Python 3.7
modules = ["sys","os","platform","random","time","functools"]
for library in modules:
try:
exec("import {module}".format(module=library))
except Exception as e:
print(e)
print(sys.argv)
Importing a submodule:
modules = ["PyQt5"] # pip install PyQt5
submodules = ["QtCore"]
for library in modules:
for sublibrary in submodules:
try:
exec("from {m} import {s}".format(m=library, s=sublibrary))
except Exception as e:
print(e)
print(dir()) # Includes QtCore
print(dir(QtCore)) # All functions, classes and variables are exactly correct as with "from PyQt5 import QtCore"
Importing everything:
modules = ["sys","os","platform","random","time","functools"]
for library in modules:
try:
exec("from {module} import *".format(module=library))
except Exception as e:
print(e)
print(dir()) # Exactly working as thought
Importing a instance or something:
modules = ["PyQt5"] # pip install PyQt5
submodules = ["QtCore"]
func = ["QCoreApplication"]
for library in modules:
for f in func:
for sublibrary in submodules:
try:
exec("from {m}.{s} import {f}".format(m=library, s=sublibrary, f=f))
except Exception as e:
print(e)
print(dir()) # Includes QCoreApplication instance
Importing everything from a modules's submodule:
modules = ["PyQt5"] # pip install PyQt5
submodules = ["QtCore"]
for library in modules:
for sublibrary in submodules:
try:
exec("from {m}.{s} import *".format(m=library, s=sublibrary)) # Didn't mention f"" strings all the times. But for beginners .format is better.
except Exception as e:
print(e)
print(dir()) # Includes all PyQt5.QtCore stuff

You can modify your import x line to use the __import__(x) format
imports = ['sys', 'itertools', 'datetime', 'os','doesntexit']
for x in imports:
try:
__import__(x)
print "Success"
except ImportError:
print "Error ", x
Outputs:
Success
Success
Success
Success
Error doesntexit

Pesky pseudo submodules
This works with pesky pseudo submodules like lxml.etree:
# PyPI imports
import pkg_resources, subprocess, sys
modules = {'lxml.etree', 'pandas', 'screeninfo'}
required = {m.split('.')[0] for m in modules}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', 'pip'])
subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])
for module in set.union(required, modules):
globals()[module] = __import__(module)
Tests:
print(pandas.__version__)
print(lxml.etree.LXML_VERSION)

How to check if a Python module exists without importing it

How can I know if a Python module exists, without importing it?
Importing something that might not exist (not what I want) results in:
try:
import eggs
except ImportError:
pass

TL;DR) Use importlib.util.find_spec(module_name) (Python 3.4+).
Python2: imp.find_module
To check if import can find something in Python 2, using imp:
import imp
try:
imp.find_module('eggs')
found = True
except ImportError:
found = False
To find dotted imports, you need to do more:
import imp
try:
spam_info = imp.find_module('spam')
spam = imp.load_module('spam', *spam_info)
imp.find_module('eggs', spam.__path__) # __path__ is already a list
found = True
except ImportError:
found = False
You can also use pkgutil.find_loader (more or less the same as the Python 3 part:
import pkgutil
eggs_loader = pkgutil.find_loader('eggs')
found = eggs_loader is not None
Python 3
Python 3 ≤ 3.3: importlib.find_loader
You should use importlib. I went about doing this like:
import importlib
spam_loader = importlib.find_loader('spam')
found = spam_loader is not None
My expectation being, if you can find a loader for it, then it exists. You can also be a bit more smart about it, like filtering out what loaders you will accept. For example:
import importlib
spam_loader = importlib.find_loader('spam')
# only accept it as valid if there is a source file for the module - no bytecode only.
found = issubclass(type(spam_loader), importlib.machinery.SourceFileLoader)
Python 3 ≥ 3.4: importlib.util.find_spec
In Python 3.4 importlib.find_loader Python documentation was deprecated in favour of importlib.util.find_spec. The recommended method is the importlib.util.find_spec. There are others like importlib.machinery.FileFinder, which is useful if you're after a specific file to load. Figuring out how to use them is beyond the scope of this.
import importlib
spam_spec = importlib.util.find_spec("spam")
found = spam_spec is not None
This also works with relative imports, but you must supply the starting package, so you could also do:
import importlib
spam_spec = importlib.util.find_spec("..spam", package="eggs.bar")
found = spam_spec is not None
spam_spec.name == "eggs.spam"
While I'm sure there exists a reason for doing this - I'm not sure what it would be.
Warning
When trying to find a submodule, it will import the parent module (for ALL of the above methods)!
food/
|- __init__.py
|- eggs.py
## __init__.py
print("module food loaded")
## eggs.py
print("module eggs")
were you then to run
>>> import importlib
>>> spam_spec = importlib.util.find_spec("food.eggs")
module food loaded
ModuleSpec(name='food.eggs', loader=<_frozen_importlib.SourceFileLoader object at 0x10221df28>, origin='/home/user/food/eggs.py')
Comments are welcome on getting around this
Acknowledgements
#rvighne for importlib
#lucas-guido for Python 3.3+ deprecating find_loader
#enpenax for pkgutils.find_loader behaviour in Python 2.7

Python 3 >= 3.6: ModuleNotFoundError
The ModuleNotFoundError has been introduced in Python 3.6 and can be used for this purpose:
try:
import eggs
except ModuleNotFoundError:
# Error handling
pass
The error is raised when a module or one of its parents cannot be found. So
try:
import eggs.sub
except ModuleNotFoundError as err:
# Error handling
print(err)
would print a message that looks like No module named 'eggs' if the eggs module cannot be found; but it would print something like No module named 'eggs.sub' if only the sub module couldn't be found, but the eggs package could be found.
See the documentation of the import system for more information on the ModuleNotFoundError.

After using yarbelk's response, I've made this so I don't have to import ìmp.
try:
__import__('imp').find_module('eggs')
# Make things with a supposed existing module
except ImportError:
pass
It is useful in Django's settings.py file, for example.

Python 2, without relying on ImportError
Until the current answer is updated, here is the way for Python 2
import pkgutil
import importlib
if pkgutil.find_loader(mod) is not None:
return importlib.import_module(mod)
return None
Why another answer?
A lot of answers make use of catching an ImportError. The problem with that is, that we cannot know what throws the ImportError.
If you import your existent module and there happens to be an ImportError in your module (e.g., typo on line 1), the result will be that your module does not exist.
It will take you quite the amount of backtracking to figure out that your module exists and the ImportError is caught and makes things fail silently.

go_as's answer as a one-liner:
python -c "help('modules');" | grep module

Use one of the functions from pkgutil, for example:
from pkgutil import iter_modules
def module_exists(module_name):
return module_name in (name for loader, name, ispkg in iter_modules())

I wrote this helper function:
def is_module_available(module_name):
if sys.version_info < (3, 0):
# python 2
import importlib
torch_loader = importlib.find_loader(module_name)
elif sys.version_info <= (3, 3):
# python 3.0 to 3.3
import pkgutil
torch_loader = pkgutil.find_loader(module_name)
elif sys.version_info >= (3, 4):
# python 3.4 and above
import importlib
torch_loader = importlib.util.find_spec(module_name)
return torch_loader is not None

Here is a way to check if a module is loaded from the command line:
Linux/UNIX script file method: make a file module_help.py:
#!/usr/bin/env python
help('modules')
Then make sure it's executable: chmod u+x module_help.py
And call it with a pipe to grep:
./module_help.py | grep module_name
Invoke the built-in help system. (This function is intended for interactive use.) If no argument is given, the interactive help system starts on the interpreter console. If the argument is a string, then the string is looked up as the name of a module, function, class, method, keyword, or documentation topic, and a help page is printed on the console. If the argument is any other kind of object, a help page on the object is generated.
Interactive method: in the console, load python
>>> help('module_name')
If found, quit reading by typing q.
To exit the Python interpreter interactive session, press Ctrl + D
Windows script file method, also Linux/UNIX compatible, and better overall:
#!/usr/bin/env python
import sys
help(sys.argv[1])
Calling it from the command like:
python module_help.py site
Would output:
Help on module site:
NAME
site - Append module search paths for third-party packages to sys.path.
FILE
/usr/lib/python2.7/site.py
MODULE DOCS
http://docs.python.org/library/site
DESCRIPTION
...
:
And you'd have to press q to exit interactive mode.
Using it for an unknown module, e.g.,
python module_help.py lkajshdflkahsodf
Would output:
no Python documentation found for 'lkajshdflkahsodf'
and exit.

You could just write a little script that would try to import all the modules and tell you which ones are failing and which ones are working:
import pip
if __name__ == '__main__':
for package in pip.get_installed_distributions():
pack_string = str(package).split(" ")[0]
try:
if __import__(pack_string.lower()):
print(pack_string + " loaded successfully")
except Exception as e:
print(pack_string + " failed with error code: {}".format(e))
Output:
zope.interface loaded successfully
zope.deprecation loaded successfully
yarg loaded successfully
xlrd loaded successfully
WMI loaded successfully
Werkzeug loaded successfully
WebOb loaded successfully
virtualenv loaded successfully
...
A word of warning: this will try to import everything, so you'll see things like PyYAML failed with error code: No module named pyyaml, because the actual import name is just yaml. So as long as you know your imports, this should do the trick for you.

There isn't any way to reliably check if "dotted module" is importable without importing its parent package. Saying this, there are many solutions to problem "how to check if a Python module exists".
The below solution addresses the problem that an imported module can raise an ImportError even if it exists. We want to distinguish that situation from such in which the module does not exist.
Python 2:
import importlib
import pkgutil
import sys
def find_module(full_module_name):
"""
Returns module object if module `full_module_name` can be imported.
Returns None if module does not exist.
Exception is raised if (existing) module raises exception during its import.
"""
module = sys.modules.get(full_module_name)
if module is None:
module_path_tail = full_module_name.split('.')
module_path_head = []
loader = True
while module_path_tail and loader:
module_path_head.append(module_path_tail.pop(0))
module_name = ".".join(module_path_head)
loader = bool(pkgutil.find_loader(module_name))
if not loader:
# Double check if module realy does not exist
# (case: full_module_name == 'paste.deploy')
try:
importlib.import_module(module_name)
except ImportError:
pass
else:
loader = True
if loader:
module = importlib.import_module(full_module_name)
return module
Python 3:
import importlib
def find_module(full_module_name):
"""
Returns module object if module `full_module_name` can be imported.
Returns None if module does not exist.
Exception is raised if (existing) module raises exception during its import.
"""
try:
return importlib.import_module(full_module_name)
except ImportError as exc:
if not (full_module_name + '.').startswith(exc.name + '.'):
raise

In django.utils.module_loading.module_has_submodule:
import sys
import os
import imp
def module_has_submodule(package, module_name):
"""
check module in package
django.utils.module_loading.module_has_submodule
"""
name = ".".join([package.__name__, module_name])
try:
# None indicates a cached miss; see mark_miss() in Python/import.c.
return sys.modules[name] is not None
except KeyError:
pass
try:
package_path = package.__path__ # No __path__, then not a package.
except AttributeError:
# Since the remainder of this function assumes that we're dealing with
# a package (module with a __path__), so if it's not, then bail here.
return False
for finder in sys.meta_path:
if finder.find_module(name, package_path):
return True
for entry in package_path:
try:
# Try the cached finder.
finder = sys.path_importer_cache[entry]
if finder is None:
# Implicit import machinery should be used.
try:
file_, _, _ = imp.find_module(module_name, [entry])
if file_:
file_.close()
return True
except ImportError:
continue
# Else see if the finder knows of a loader.
elif finder.find_module(name):
return True
else:
continue
except KeyError:
# No cached finder, so try and make one.
for hook in sys.path_hooks:
try:
finder = hook(entry)
# XXX Could cache in sys.path_importer_cache
if finder.find_module(name):
return True
else:
# Once a finder is found, stop the search.
break
except ImportError:
# Continue the search for a finder.
continue
else:
# No finder found.
# Try the implicit import machinery if searching a directory.
if os.path.isdir(entry):
try:
file_, _, _ = imp.find_module(module_name, [entry])
if file_:
file_.close()
return True
except ImportError:
pass
# XXX Could insert None or NullImporter
else:
# Exhausted the search, so the module cannot be found.
return False

In case you know the location of file and want to check that the respective Python code file has that module or not, you can simply check via the astor package in Python. Here is a quick example:
"""
Check if a module function exists or not without importing a Python package file
"""
import ast
import astor
tree = astor.parse_file('handler.py')
method_to_check = 'handle'
for item in tree.body:
if isinstance(item, ast.FunctionDef):
if item.name == method_to_check:
print('method exists')
break

A simpler if statement from Ask Ubuntu, How do I check whether a module is installed in Python?:
import sys
print('eggs' in sys.modules)

You can also use importlib.util directly
import importlib.util
def module_exists_without_import(module_name):
spec = importlib.util.find_spec(module_name)
return spec is not None