I am trying to write a simple function as follows:
def lexical_diversity(text,word):
from nltk.book import text
return 100*text.count(word)/len(set(text))
I understand that I can import a text before the function. But, I was wondering why I get the following error
ImportError: cannot import name 'text' from 'nltk.book'
It is telling me that "text" as a corpus does not exist in nltk--it is true. But, I want the user to identify the text to be text1, text2, or text3.
In order to import a submodule of a library in python, you can use importlib module.
import importlib
def lexical_diversity(nltk_sub_module,word):
imported_model_module = importlib.import_module( "nltk.book")
text = getattr(imported_model_module,nltk_sub_module)
return 100*text.count(word)/len(set(text))
lexical_diversity("text3", "book")
I'm trying to test a tool I'm building which uses some jMetalPy functionality. I had/have a previous version working but I am now trying to refactor out some external dependencies (such as the aforementioned jMetalPy).
Project Code & Structure
Here is a minimalist structure of my project.
MyToolDirectory
¦--/MyTool
¦----/__init__.py
¦----/_jmetal
¦------/__init__.py
¦------/core
¦--------/quality_indicator.py
¦----/core
¦------/__init__.py
¦------/run_manager.py
¦----/tests
¦------/__init__.py
¦------/test_run_manager.py
The _jmetal directory is to remove external dependency on the jMetalPy package - and I have copied only the necessary packages/modules that I need.
Minimal contents of run_manager.py
# MyTool\core\run_manager.py
import jmetal
# from jmetal.core.quality_indicators import HyperVolume # old working version
class RunManager:
def __init__(self):
pass
#staticmethod
def calculate_hypervolume(front, ref_point):
if front is None or len(front) < 1:
return 0.
hv = jmetal.core.quality_indicator.HyperVolume(ref_point)
# hv = HyperVolume(ref_point)
hypervolume = hv.compute(front)
return hypervolume
Minimal contents of test_run_manager.py
# MyTool\tests\test_run_manager.py
import unittest
from unittest.mock import MagicMock, Mock, patch
from MyTool import core
class RunManagerTest(unittest.TestCase):
def setUp(self):
self.rm = core.RunManager()
def test_calculate_hypervolume(self):
ref_points = [0.0, 57.5]
front = [None, None]
# with patch('MyTool.core.run_manager.HyperVolume') as mock_HV: # old working version
with patch('MyTool.core.run_manager.jmetal.core.quality_indicator.HyperVolume') as mock_HV:
mock_HV.return_value = MagicMock()
res = self.rm.calculate_hypervolume(front, ref_points)
mock_HV.assert_called_with(ref_points)
mock_HV().compute.assert_called_with(front)
Main Question
When I run a test with the code as-is, I get this error message:
E ModuleNotFoundError: No module named 'MyTool.core.run_manager.jmetal'; 'MyTool.core.run_manager' is not a package
But when I change it to:
with patch('MyTool.core.run_manager.jmetal.core') as mock_core:
mock_HV = mock_core.quality_indicator.HyperVolume
mock_HV.return_value = MagicMock()
res = self.rm.calculate_hypervolume(front, ref_points)
mock_HV.assert_called_with(ref_points)
mock_HV().compute.assert_called_with(front)
... now the test passes. What gives?!
Why can't (or rather, how can) I surgically patch the exact class I want (i.e., HyperVolume) without patching out an entire sub-package as well? Is there a way around this? There may be code in jmetal.core that needs to run normally.
Is the reason this isn't working only because there is no from . import quality_indicator statement in jMetalPy's jmetal\core\__init__.py ?
Because even with patch('MyTool.core.run_manager.jmetal.core.quality_indicator) throws:
E AttributeError: <module 'jmetal.core' from 'path\\to\\venv\\lib\\site-packages\\jmetal\\core\\__init__.py'> does not have the attribute 'quality_indicator'
Or is there something I'm doing wrong?
In the case that it is just about adding those import statements, I could do that in my _jmetal sub-package, but I was hoping to let the user default to their own jMetalPy installation if they already had one by adding this to MyTool\__init__.py:
try:
import jmetal
except ModuleNotFoundError:
from . import _jmetal as jmetal
and then replacing all instances of import jmetal with from MyTool import jmetal. However, I'd run into the same problem all over again.
I feel that there is some core concept I am not grasping. Thanks for the help.
Im trying to import files on Flask app in base of url route. I started to coding python few days ago so i havent idea if i doing it well. I write this on :
#app.route('/<file>')
def call(file):
__import__('controller.'+file)
hello = Example('Hello world')
return hello.msg
And i have other file called example.py into a controller folder that contains this:
class Example:
def __init__(self, msg):
self.msg = msg
So i start from terminal the app and i try to enter to localhost:5000/example.
Im trying to show in screen Hello world but give me the next error:
NameError: global name 'Example' is not defined
Thanks for all!
__import__ returns the newly imported module; names from that module are not added to your globals, so you need to get the Example class as an attribute from the returned module:
module = __import__('controller.'+file)
hello = module.Example('Hello world')
__import__ is rather low-level, you probably want to use importlib.import_module() instead:
import importlib
module = importlib.import_module('controller.'+file)
hello = module.Example('Hello world')
If you need to dynamically get the classname too, use getattr():
class_name = 'Example'
hello_class = getattr(module, class_name)
hello = hello_class('Hello world')
The Werkzeug package (used by Flask) offers a helpful functions here: werkzeug.utils.import_string() imports an object dynamically:
from werkzeug.utils import import_string
object_name = 'controller.{}:Example'.format(file)
hello_class = import_string(object_name)
This encapsulates the above process.
You'll need to be extremely careful with accepting names from web requests and using those as module names. Please do sanitise the file argument and only allow alphanumerics to prevent relative imports from being used.
You could use the werkzeug.utils.find_modules() function to limit the possible values for file here:
from werkzeug.utils import find_modules, import_string
module_name = 'controller.{}'.format(file)
if module_name not in set(find_modules('controller')):
abort(404) # no such module in the controller package
hello_class = import_string(module_name + ':Example')
I think you might not add the directory to the file, add the following code into the previous python program
# Add another directory
import sys
sys.path.insert(0, '/your_directory')
from Example import Example
There are two ways for you to do imports in Python:
import example
e = example.Example('hello world')
or
from example import Example
e = Example('hello world')
I need to dynamically import a module and create a class.
This is my working code in Python 3.2:
klass = {}
mod = __import__('sites.' + self.className + '_login', fromlist=[self.className])
klass[self.className] = getattr(mod, self.className)
klass[self.className](**self.args)
The module is inside the sites folder. It's called my_site_login and the class within that module my_site.
Since I upgrade to Python 3.3, the code stopped working. I read that _____import_____ was replace by importlib.import_module. I tried several ways to do it but I can't get it to work.
This is what I tried:
https://stackoverflow.com/a/8790051/1095101
mod = getattr(import_module('sites.' + self.className + '_login'), self.className)
I can't remember what else I've tried. What I can say, is that none of the print() I put right after any import attempt, was showing. I don't get any error message. It just does nothing.
You want to restructure this; in the sites package __init__.py file, import all modules that need to be imported dynamically.
Then just use attribute access instead:
import sites
mod = getattr(sites, self.className + '_login')
klass[self.className] = getattr(mod, self.className)
klass[self.className](**self.args)
If your my_site_login.py modules are dynamically generated, the importlib.import_module() callable works very simply:
importlib
mod = importlib.import_module('sites.{}_login'.format(self.className))
klass[self.className] = getattr(mod, self.className)
klass[self.className](**self.args)
From the interactive command prompt this works fine:
>>> import importlib
>>> importlib.import_module('sites.my_site_login')
<module 'sites.my_site_login' from './sites/my_site_login.py'>
Tested that in Python 2.7, 3.2 and 3.3.
How can I know if a Python module exists, without importing it?
Importing something that might not exist (not what I want) results in:
try:
import eggs
except ImportError:
pass
TL;DR) Use importlib.util.find_spec(module_name) (Python 3.4+).
Python2: imp.find_module
To check if import can find something in Python 2, using imp:
import imp
try:
imp.find_module('eggs')
found = True
except ImportError:
found = False
To find dotted imports, you need to do more:
import imp
try:
spam_info = imp.find_module('spam')
spam = imp.load_module('spam', *spam_info)
imp.find_module('eggs', spam.__path__) # __path__ is already a list
found = True
except ImportError:
found = False
You can also use pkgutil.find_loader (more or less the same as the Python 3 part:
import pkgutil
eggs_loader = pkgutil.find_loader('eggs')
found = eggs_loader is not None
Python 3
Python 3 ≤ 3.3: importlib.find_loader
You should use importlib. I went about doing this like:
import importlib
spam_loader = importlib.find_loader('spam')
found = spam_loader is not None
My expectation being, if you can find a loader for it, then it exists. You can also be a bit more smart about it, like filtering out what loaders you will accept. For example:
import importlib
spam_loader = importlib.find_loader('spam')
# only accept it as valid if there is a source file for the module - no bytecode only.
found = issubclass(type(spam_loader), importlib.machinery.SourceFileLoader)
Python 3 ≥ 3.4: importlib.util.find_spec
In Python 3.4 importlib.find_loader Python documentation was deprecated in favour of importlib.util.find_spec. The recommended method is the importlib.util.find_spec. There are others like importlib.machinery.FileFinder, which is useful if you're after a specific file to load. Figuring out how to use them is beyond the scope of this.
import importlib
spam_spec = importlib.util.find_spec("spam")
found = spam_spec is not None
This also works with relative imports, but you must supply the starting package, so you could also do:
import importlib
spam_spec = importlib.util.find_spec("..spam", package="eggs.bar")
found = spam_spec is not None
spam_spec.name == "eggs.spam"
While I'm sure there exists a reason for doing this - I'm not sure what it would be.
Warning
When trying to find a submodule, it will import the parent module (for ALL of the above methods)!
food/
|- __init__.py
|- eggs.py
## __init__.py
print("module food loaded")
## eggs.py
print("module eggs")
were you then to run
>>> import importlib
>>> spam_spec = importlib.util.find_spec("food.eggs")
module food loaded
ModuleSpec(name='food.eggs', loader=<_frozen_importlib.SourceFileLoader object at 0x10221df28>, origin='/home/user/food/eggs.py')
Comments are welcome on getting around this
Acknowledgements
#rvighne for importlib
#lucas-guido for Python 3.3+ deprecating find_loader
#enpenax for pkgutils.find_loader behaviour in Python 2.7
Python 3 >= 3.6: ModuleNotFoundError
The ModuleNotFoundError has been introduced in Python 3.6 and can be used for this purpose:
try:
import eggs
except ModuleNotFoundError:
# Error handling
pass
The error is raised when a module or one of its parents cannot be found. So
try:
import eggs.sub
except ModuleNotFoundError as err:
# Error handling
print(err)
would print a message that looks like No module named 'eggs' if the eggs module cannot be found; but it would print something like No module named 'eggs.sub' if only the sub module couldn't be found, but the eggs package could be found.
See the documentation of the import system for more information on the ModuleNotFoundError.
After using yarbelk's response, I've made this so I don't have to import ìmp.
try:
__import__('imp').find_module('eggs')
# Make things with a supposed existing module
except ImportError:
pass
It is useful in Django's settings.py file, for example.
Python 2, without relying on ImportError
Until the current answer is updated, here is the way for Python 2
import pkgutil
import importlib
if pkgutil.find_loader(mod) is not None:
return importlib.import_module(mod)
return None
Why another answer?
A lot of answers make use of catching an ImportError. The problem with that is, that we cannot know what throws the ImportError.
If you import your existent module and there happens to be an ImportError in your module (e.g., typo on line 1), the result will be that your module does not exist.
It will take you quite the amount of backtracking to figure out that your module exists and the ImportError is caught and makes things fail silently.
go_as's answer as a one-liner:
python -c "help('modules');" | grep module
Use one of the functions from pkgutil, for example:
from pkgutil import iter_modules
def module_exists(module_name):
return module_name in (name for loader, name, ispkg in iter_modules())
I wrote this helper function:
def is_module_available(module_name):
if sys.version_info < (3, 0):
# python 2
import importlib
torch_loader = importlib.find_loader(module_name)
elif sys.version_info <= (3, 3):
# python 3.0 to 3.3
import pkgutil
torch_loader = pkgutil.find_loader(module_name)
elif sys.version_info >= (3, 4):
# python 3.4 and above
import importlib
torch_loader = importlib.util.find_spec(module_name)
return torch_loader is not None
Here is a way to check if a module is loaded from the command line:
Linux/UNIX script file method: make a file module_help.py:
#!/usr/bin/env python
help('modules')
Then make sure it's executable: chmod u+x module_help.py
And call it with a pipe to grep:
./module_help.py | grep module_name
Invoke the built-in help system. (This function is intended for interactive use.) If no argument is given, the interactive help system starts on the interpreter console. If the argument is a string, then the string is looked up as the name of a module, function, class, method, keyword, or documentation topic, and a help page is printed on the console. If the argument is any other kind of object, a help page on the object is generated.
Interactive method: in the console, load python
>>> help('module_name')
If found, quit reading by typing q.
To exit the Python interpreter interactive session, press Ctrl + D
Windows script file method, also Linux/UNIX compatible, and better overall:
#!/usr/bin/env python
import sys
help(sys.argv[1])
Calling it from the command like:
python module_help.py site
Would output:
Help on module site:
NAME
site - Append module search paths for third-party packages to sys.path.
FILE
/usr/lib/python2.7/site.py
MODULE DOCS
http://docs.python.org/library/site
DESCRIPTION
...
:
And you'd have to press q to exit interactive mode.
Using it for an unknown module, e.g.,
python module_help.py lkajshdflkahsodf
Would output:
no Python documentation found for 'lkajshdflkahsodf'
and exit.
You could just write a little script that would try to import all the modules and tell you which ones are failing and which ones are working:
import pip
if __name__ == '__main__':
for package in pip.get_installed_distributions():
pack_string = str(package).split(" ")[0]
try:
if __import__(pack_string.lower()):
print(pack_string + " loaded successfully")
except Exception as e:
print(pack_string + " failed with error code: {}".format(e))
Output:
zope.interface loaded successfully
zope.deprecation loaded successfully
yarg loaded successfully
xlrd loaded successfully
WMI loaded successfully
Werkzeug loaded successfully
WebOb loaded successfully
virtualenv loaded successfully
...
A word of warning: this will try to import everything, so you'll see things like PyYAML failed with error code: No module named pyyaml, because the actual import name is just yaml. So as long as you know your imports, this should do the trick for you.
There isn't any way to reliably check if "dotted module" is importable without importing its parent package. Saying this, there are many solutions to problem "how to check if a Python module exists".
The below solution addresses the problem that an imported module can raise an ImportError even if it exists. We want to distinguish that situation from such in which the module does not exist.
Python 2:
import importlib
import pkgutil
import sys
def find_module(full_module_name):
"""
Returns module object if module `full_module_name` can be imported.
Returns None if module does not exist.
Exception is raised if (existing) module raises exception during its import.
"""
module = sys.modules.get(full_module_name)
if module is None:
module_path_tail = full_module_name.split('.')
module_path_head = []
loader = True
while module_path_tail and loader:
module_path_head.append(module_path_tail.pop(0))
module_name = ".".join(module_path_head)
loader = bool(pkgutil.find_loader(module_name))
if not loader:
# Double check if module realy does not exist
# (case: full_module_name == 'paste.deploy')
try:
importlib.import_module(module_name)
except ImportError:
pass
else:
loader = True
if loader:
module = importlib.import_module(full_module_name)
return module
Python 3:
import importlib
def find_module(full_module_name):
"""
Returns module object if module `full_module_name` can be imported.
Returns None if module does not exist.
Exception is raised if (existing) module raises exception during its import.
"""
try:
return importlib.import_module(full_module_name)
except ImportError as exc:
if not (full_module_name + '.').startswith(exc.name + '.'):
raise
In django.utils.module_loading.module_has_submodule:
import sys
import os
import imp
def module_has_submodule(package, module_name):
"""
check module in package
django.utils.module_loading.module_has_submodule
"""
name = ".".join([package.__name__, module_name])
try:
# None indicates a cached miss; see mark_miss() in Python/import.c.
return sys.modules[name] is not None
except KeyError:
pass
try:
package_path = package.__path__ # No __path__, then not a package.
except AttributeError:
# Since the remainder of this function assumes that we're dealing with
# a package (module with a __path__), so if it's not, then bail here.
return False
for finder in sys.meta_path:
if finder.find_module(name, package_path):
return True
for entry in package_path:
try:
# Try the cached finder.
finder = sys.path_importer_cache[entry]
if finder is None:
# Implicit import machinery should be used.
try:
file_, _, _ = imp.find_module(module_name, [entry])
if file_:
file_.close()
return True
except ImportError:
continue
# Else see if the finder knows of a loader.
elif finder.find_module(name):
return True
else:
continue
except KeyError:
# No cached finder, so try and make one.
for hook in sys.path_hooks:
try:
finder = hook(entry)
# XXX Could cache in sys.path_importer_cache
if finder.find_module(name):
return True
else:
# Once a finder is found, stop the search.
break
except ImportError:
# Continue the search for a finder.
continue
else:
# No finder found.
# Try the implicit import machinery if searching a directory.
if os.path.isdir(entry):
try:
file_, _, _ = imp.find_module(module_name, [entry])
if file_:
file_.close()
return True
except ImportError:
pass
# XXX Could insert None or NullImporter
else:
# Exhausted the search, so the module cannot be found.
return False
In case you know the location of file and want to check that the respective Python code file has that module or not, you can simply check via the astor package in Python. Here is a quick example:
"""
Check if a module function exists or not without importing a Python package file
"""
import ast
import astor
tree = astor.parse_file('handler.py')
method_to_check = 'handle'
for item in tree.body:
if isinstance(item, ast.FunctionDef):
if item.name == method_to_check:
print('method exists')
break
A simpler if statement from Ask Ubuntu, How do I check whether a module is installed in Python?:
import sys
print('eggs' in sys.modules)
You can also use importlib.util directly
import importlib.util
def module_exists_without_import(module_name):
spec = importlib.util.find_spec(module_name)
return spec is not None