Is there a way to do something (like print "funkymodule imported" for example) every time a module is imported from any other module? Not only the first time it's imported to the runtime or reloaded?
One possibility would be to monkey patch __import__:
>>> old_import = __import__
>>> def my_import(module,*args,**kwargs):
... print module, 'loaded'
... return old_import(module,*args,**kwargs)
...
>>> __builtins__.__import__ = my_import
>>> import datetime
datetime loaded
>>> import datetime
datetime loaded
>>> import django
django loaded
It worked fine on command line (using Python 2.7.3 on Windows XP), but I dunno if would work in other environments.
To access the module object (instead of just the module name - so you can do something useful with it) just intercept the return value instead of the argument:
>>> def my_import(*args,**kwargs):
... ret = old_import(*args,**kwargs)
... print ret
... return ret
...
>>> __builtins__.__import__ = my_import
>>> import datetime
<module 'datetime' (built-in)>
>>> import django
<module 'django' from 'C:\Python27\lib\site-packages\django\__init__.pyc'>
Update: Just confirmed it works if used inside a python file too - though in this case, the correct way of assigning it is __builtins__['__import__'] = my_import.
Related
How can I import an arbitrary python source file (whose filename could contain any characters, and does not always ends with .py) in Python 3.3+?
I used imp.load_module as follows:
>>> import imp
>>> path = '/tmp/a-b.txt'
>>> with open(path, 'U') as f:
... mod = imp.load_module('a_b', f, path, ('.py', 'U', imp.PY_SOURCE))
...
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
It still works in Python 3.3, but according to imp.load_module documentation, it is deprecated:
Deprecated since version 3.3: Unneeded as loaders should be used to
load modules and find_module() is deprecated.
and imp module documentation recommends to use importlib:
Note New programs should use importlib rather than this module.
What is the proper way to load an arbitrary python source file in Python 3.3+ without using the deprecated imp.load_module function?
Found a solution from importlib test code.
Using importlib.machinery.SourceFileLoader:
>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = loader.load_module()
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
NOTE: only works in Python 3.3+.
UPDATE Loader.load_module is deprecated since Python 3.4. Use Loader.exec_module instead:
>>> import types
>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = types.ModuleType(loader.name)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b'>
>>> import importlib.machinery
>>> import importlib.util
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> spec = importlib.util.spec_from_loader(loader.name, loader)
>>> mod = importlib.util.module_from_spec(spec)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
Updated for Python >= 3.8:
Short version:
>>> # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
>>> import importlib.util, sys
>>> spec = importlib.util.spec_from_file_location(modname, fname)
>>> module = importlib.util.module_from_spec(spec)
>>> sys.modules[modname] = module
>>> spec.loader.exec_module(module)
Full version:
>>> import importlib.util
>>> import sys
>>> from pathlib import Path
>>> from typing import TYPE_CHECKING
>>>
>>>
>>> if TYPE_CHECKING:
... import types
...
...
>>> def import_source_file(fname: str | Path, modname: str) -> "types.ModuleType":
... """
... Import a Python source file and return the loaded module.
... Args:
... fname: The full path to the source file. It may container characters like `.`
... or `-`.
... modname: The name for the loaded module. It may contain `.` and even characters
... that would normally not be allowed (e.g., `-`).
... Return:
... The imported module
... Raises:
... ImportError: If the file cannot be imported (e.g, if it's not a `.py` file or if
... it does not exist).
... Exception: Any exception that is raised while executing the module (e.g.,
... :exc:`SyntaxError). These are errors made by the author of the module!
... """
... # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
... spec = importlib.util.spec_from_file_location(modname, fname)
... if spec is None:
... raise ImportError(f"Could not load spec for module '{modname}' at: {fname}")
... module = importlib.util.module_from_spec(spec)
... sys.modules[modname] = module
... try:
... spec.loader.exec_module(module)
... except FileNotFoundError as e:
... raise ImportError(f"{e.strerror}: {fname}") from e
... return module
...
>>> import_source_file(Path("/tmp/my_mod.py"), "my_mod")
<module 'my_mod' from '/tmp/my_mod.py'>
Original answer for Python 3.5 and 3.6
Shorter version of #falsetru 's solution:
>>> import importlib.util
>>> spec = importlib.util.spec_from_file_location('a_b', '/tmp/a-b.py')
>>> mod = importlib.util.module_from_spec(spec)
>>> spec.loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
I tested it with Python 3.5 and 3.6.
According to the comments, it does not work with arbitrary file extensions.
Similar to #falsetru but for Python 3.5+ and accounting for what the importlib doc states on using importlib.util.module_from_spec over types.ModuleType:
This function [importlib.util.module_from_spec] is preferred over using types.ModuleType to create a new
module as spec is used to set as many import-controlled attributes on
the module as possible.
We are able to import any file with importlib alone by modifying the importlib.machinery.SOURCE_SUFFIXES list.
import importlib
importlib.machinery.SOURCE_SUFFIXES.append('') # empty string to allow any file
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# if desired: importlib.machinery.SOURCE_SUFFIXES.pop()
importlib helper function
Here is a convenient, ready-to-use helper to replace imp, with an example. The technique is the same as that of https://stackoverflow.com/a/19011259/895245 , this is just providing a more convenient function.
main.py
#!/usr/bin/env python3
import os
import importlib
def import_path(path):
module_name = os.path.basename(path).replace('-', '_')
spec = importlib.util.spec_from_loader(
module_name,
importlib.machinery.SourceFileLoader(module_name, path)
)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
sys.modules[module_name] = module
return module
notmain = import_path('not-main')
print(notmain)
print(notmain.x)
not-main
x = 1
Run:
python3 main.py
Output:
<module 'not_main' from 'not-main'>
1
I replace - with _ because my importable Python executables without extension have hyphens as in my-cmd. This is not mandatory, but produces better module names like my_cmd.
This pattern is also mentioned in the docs at: https://docs.python.org/3.7/library/importlib.html#importing-a-source-file-directly
I ended up moving to it because after updating to Python 3.7, import imp prints:
DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
and I don't know how to turn that off, this was asked at:
The imp module is deprecated
How to ignore deprecation warnings in Python
Tested in Python 3.7.3.
after many failure solutions this one works for me
def _import(func,*args):
import os
from importlib import util
module_name = "my_module"
BASE_DIR = "wanted module directory path"
path = os.path.join(BASE_DIR,module_name)
spec = util.spec_from_file_location(func, path)
mod = util.module_from_spec(spec)
spec.loader.exec_module(mod)
return getattr(mod,func)(*args)
and to call it just write the function name and it's parameters _import("function",*args)
I would like to know if it is possible to obtain the module name and directory path from a class instance of that module.
I need that because I save a class instance with pickle and I need to know from where this class come from when I load it from another software (on the same computer).
update
So I tried the aswer of #albar
>>> from Models import Model_physio_moyen
>>> b = Model_physio_moyen.pop_Sigmoid_outside_noise
>>> module_name = sys.modules[b.__module__].__name__
>>> module_name
'Models.Model_physio_moyen'
So that working except if the class has been compiled with jitclassof numba:
>>> from Models.Pops_Stim import ATN
>>> a = ATN.pop_ATN()
>>> module_name = sys.modules[a.__module__].__name__
>>> module_name
'numba.jitclass.boxing'
In this condition, I wish I could find Models.Pops_Stim but I'm getting 'numba.jitclass.boxing'instead.
Assuming your instance is called a:
import sys
module_name = sys.modules[a.__module__].__name__
module_file = sys.modules[a.__module__].__file__
I have my Python unittest script like below. It takes an argument '-a' to determine whether the testcases should load the base module from foo_PC_A.py or foo_PC_B.py. I use shutil.move() to rename either .py file to foo.py, so all the testcase modules (e.g. tm1.py, tm2.py) can simply import foo. Though this looks like a workaround and not Pythonic. Is there any better way to do this? Or a better design to fundamentally resolve this issue.
(run_all_unittest.py)
if sys.argv[1] = '-a':
shutil.move('foo_PC_A.py', 'foo.py')
else:
shutil.move('foo_PC_B.py', 'foo.py')
test_module_list = ['tm1', 'tm2', ...]
for test_module_name in test_module_list:
test_module = __import__(test_module_name)
test_suites.append(unittest.TestLoader().loadTestsFromModule(test_module))
alltests = unittest.TestSuite(test_suites)
unittest.TextTestRunner().run(alltests)
if sys.argv[1] = '-a':
shutil.move('foo.py', 'foo_PC_A.py')
else:
shutil.move('foo.py', 'foo_PC_B.py')
(tm1.py)
from foo import MyTestCase
...
(foo_PC_A.py)
import <some module only available on PC A>
class MyTestCase(unittest.TestCase):
...
(foo_PC_B.py)
# since the PC A modules are not available on PC B,
# just call the pre-built executable via subprocess
import subprocess
class MyTestCase(unittest.TestCase):
...
def test_run(self):
subprocess.call(...)
You can fool Python into thinking the module has already been loaded. Just import the module dynamically and use sys.modules:
import sys
import importlib
if sys.argv[1] = '-a':
sys.modules['foo'] = importlib.import_module('foo_PC_A')
else:
sys.modules['foo'] = importlib.import_module('foo_PC_A')
When any module runs import foo or from foo import ..., Python will use that path.
Note that if foo is moved to a package, the full Python path must be specified, as in:
sys.modules['path.to.foo'] = ...
PyCharm shows me that imp is deprecated so I wonder if there any analogue of imp.new_module for importlib.
Quoting from documentation (Emphasis mine) -
imp.new_module(name)
Return a new empty module object called name. This object is not inserted in sys.modules.
Deprecated since version 3.4: Use types.ModuleType instead.
Example -
>>> import types
>>> types.ModuleType('name')
<module 'name'>
To show how they are synonymous -
>>> import imp
>>> imp.new_module('name')
<module 'name'>
Python documentation suggests to us:
Note Use importlib.util.module_from_spec() to create a new module if
you wish to set the various import-controlled attributes.
importlib.util.module_from_spec(spec) is preferred over using types.ModuleType to create a new module as spec is used to set as many import-controlled attributes on the module as possible.
To import a Python source file directly, we can use the following snippet:
import importlib.util
import sys
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)
The example below, we can use to implement lazy imports:
import importlib.util
import sys
def lazy_import(name):
spec = importlib.util.find_spec(name)
loader = importlib.util.LazyLoader(spec.loader)
spec.loader = loader
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
loader.exec_module(module)
return module
>>> lazy_typing = lazy_import("typing")
>>> #lazy_typing is a real module object,
>>> #but it is not loaded in memory yet.
>>> lazy_typing.TYPE_CHECKING
False
import v_framework as framework
framework.loadModules(["Maintenance"])
framework.Maintenance.showPage()
In framework I have:
def loadModules(aModules):
d_utility = {"Maintenance":"COOl_M_PAGE"}
for module in a_aModules:
exec("import " + d_utility[module] + " as " + module)
When loadModules is executed, it imports the modules in the v_framework namespace. Since I am importing v_framework as framework, I think I should be able to use the imported module using framework.Maintenance. But it does not work that way.
Is there a way to do way to do what I'm trying to do? Alternatively, is there any way to import modules in a namespace other than the one where exec is executed?
There are libraries for importing modules dynamically. You could use importlib (and another one that might be useful is pkgutil). Now, for your case, I guess this would do the job:
import importlib
mods = {}
def loadModules(aModule):
global mods
mods[module] = importlib.import_module(d_utility[module])
# or maybe globals()[module] = ... would work also (exactly as you expect it to
UPDATE: exec modifies the function's local namespace, not the global one (I think).
Hope it helps. :)
When you import inside a function, the module is imported/executed as normal, but the name you import under is local to the function, just like any other variable assigned inside a function.
>>> def test_import():
... import os
... print os
...
>>> test_import()
<module 'os' from '/usr/lib/python2.7/os.pyc'>
>>> os
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'os' is not defined
os has been imported though, and you can still access it through sys.modules:
>>> import sys
>>> sys.modules['os']
<module 'os' from '/usr/lib/python2.7/os.pyc'>
>>> os = sys.modules['os']
>>> os
<module 'os' from '/usr/lib/python2.7/os.pyc'>
A quick and dirty way to do what you want would be something like this; exec takes an optional mapping to be used as the local and global variables. So you could do
def loadModules(aModules):
d_utility = {"Maintenance":"COOl_M_PAGE"}
for module in aModules:
exec ('import %s as %s' % (d_utility[module], module)) in globals()
Though this is ugly and probably has security implications or something. As jadkik94 mentions, there are libraries that provide cleaner ways to deal with this.