I have found many variations for importing dynamically created/named modules by reference to their names as text, but all import the module as a whole and do not seem to facilitate importing all * ....
In my case, the objects within the file are dynamically created and named, so their identities cannot be discovered beforehand.
This works, but is there a better way perhaps using importlib ?
PREFIX = "my_super_new"
active_data_module = "{0}_data_module".format(PREFIX)
exec("from {0} import *".format(active_data_module))
You could use vars with the module. This would return a dictionary of all attributes on the module (I think). Then you can assign the dictionary to the globals dictionary to make it accessible in the current module:
import importlib
PREFIX = "my_super_new"
active_data_module = "{0}_data_module".format(PREFIX)
module = importlib.import_module(active_data_module)
globals().update(vars(module))
Using Peter Wood's answer, I created a small utility function:
import importlib
def import_everything_from_module_by_name(module_name):
globals().update(vars(importlib.import_module(module_name)))
modules_for_import = [
module_a,
module_b,
module_c
]
for module_name in modules_for_import:
import_everything_from_module_by_name(module_name)
Related
I'd like to make the following line dynamic :
from my_package import my_class as my_custom_name
I know how to dynamically import modules via string
import importlib
module_name = "my_package"
my_module = importlib.import_module(module_name)
as suggested here. However it still doesn't let me specify the class I want to import (my_class) and the alias I want to assign to the class name (my_custom_name). I'm using python 3.6
Two steps. Number one, you can reference a module directly using importlib:
importlib.import_module('my_package.my_module') # You can use '.'.join((my_package, my_module))
Your class will be contained in the module itself as an attribute, as in any import. As such, just use
my_custom_name = importlib.import_module('my_package.my_module').__dict__['my_class']
or even better
my_custom_name = getattr(importlib.import_module('my_package.my_module'), 'my_class')
I wish to dynamically import a module in Python (3.7), whereby the module's code is defined within a string.
Below is a working example that uses the imp module, which is deprecated in favour of importlib (as of version 3.4):
import imp
def import_code(code, name):
# create blank module
module = imp.new_module(name)
# populate the module with code
exec(code, module.__dict__)
return module
code = """
def testFunc():
print('spam!')
"""
m = import_code(code, 'test')
m.testFunc()
Python's documentation states that importlib.util.module_from_spec() should be used instead of imp.new_module(). However, there doesn't seem to be a way to create a blank module object using the importlib module, like I could with imp.
How can I use importlib instead of imp to achieve the same result?
You can simply instantiate types.Module:
import types
mod = types.ModuleType("mod")
Then you can populate it with exec just like you did:
exec(code, mod.__dict__)
mod.testFunc() # will print 'spam!'
So your code will look like this:
import types
def import_code(code, name):
# create blank module
module = types.ModuleType(name)
# populate the module with code
exec(code, module.__dict__)
return module
code = """
def testFunc():
print('spam!')
"""
m = import_code(code, 'test')
m.testFunc()
As commented by #Error - Syntactical Remorse, you should keep in mind that exec basically executes whatever code is contained in the string you give it, so you should use it with extra care.
At least check what you're given, but it'd be good to use exclusively predefined strings.
According to Python documentation module_from_spec()
importlib.util.module_from_spec(spec)
...
This function is preferred over using types.ModuleType to create a new module as spec is used to set as many import-controlled attributes on the module as possible.
Here is what I came up with to load the module from source code located in github repo. It is a way without writing the file to disk.
import requests
url = "https://github.com/udacity/deep-learning-v2-pytorch/raw/master/intro-to-pytorch/helper.py"
r = requests.get(url)
import importlib.util
spec = importlib.util.spec_from_loader('helper', loader=None, origin=url)
helper = importlib.util.module_from_spec(spec)
exec(r.content, helper.__dict__)
helper.view_classify() # executes function from github file
I have a system that collects all classes that derive from certain base classes and stores them in a dictionary. I want to avoid having to specify which classes are available (I would like to discover them programatically), so have used a from ModuleName import * statement. The user is then directed to place all tests to be collected in the ModuleName module. However, I cannot find a way to programatically determine what symbols were imported with that import statement. I have tried using dir() and __dict__ as indicated in the following example, but to no avail. How does one programatically find symbols imported in this manner (with import *)? I am unable to find them with the above methods.
testTypeFigureOuterrer.py:
from testType1 import *
from testType2 import *
class TestFigureOuterrer(object):
def __init__(self):
self.existingTests = {'type1':{},'type2':{}}
def findAndSortTests(self):
for symbol in dir(): # Also tried: dir(self) and __dict__
try:
thing = self.__getattribute__(symbol)
except AttributeError:
continue
if issubclass(thing,TestType1):
self.existingTests['type1'].update( dict(symbol,thing) )
elif issubclass(thing,TestType3):
self.existingTests['type2'].update( dict(symbol,thing) )
else:
continue
if __name__ == "__main__":
testFigureOuterrer = TestFigureOuterrer()
testFigureOuterrer.findAndSortTests()
testType1.py:
class TestType1(object):
pass
class TestA(TestType1):
pass
class TestB(TestType1):
pass
testType2.py:
class TestType2:
pass
class TestC(TestType2):
pass
class TestD(TestType2):
pass
Since you know the imports yourself, you should just import the module manually again, and then check the contents of the module. If an __all__ property is defined, its contents are imported as names when you do from module import *. Otherwise, just use all its members:
def getImportedNames (module):
names = module.__all__ if hasattr(module, '__all__') else dir(module)
return [name for name in names if not name.startswith('_')]
This has the benefit that you do not need to go through the globals, and filter everything out. And since you know the modules you import from at design time, you can also check them directly.
from testType1 import *
from testType2 import *
import testType1, testType2
print(getImportedNames(testType1))
print(getImportedNames(testType2))
Alternatively, you can also look up the module by its module name from sys.modules, so you don’t actually need the extra import:
import sys
def getImportedNames (moduleName):
module = sys.modules[moduleName]
names = module.__all__ if hasattr(module, '__all__') else dir(module)
return [name for name in names if not name.startswith('_')]
print(getImportedNames('testType1'))
print(getImportedNames('testType2'))
Take a look at this SO answer, which describes how to determine the name of loaded classes, you can get the name of all the classes defined within the context of the module.
import sys, inspect
clsmembers = inspect.getmembers(sys.modules['testType1'], inspect.isclass)
which is now defined as
[('TestA', testType1.TestA),
('TestB', testType1.TestB),
('TestType1', testType1.TestType1)]
You can also replace testType1 with __name__ when you're within the function of interest.
Don't use the * form of import. This dumps the imported names into your script's global namespace. Not only could they clobber some important bit of data by using the same name, you don't have any easy way to fish out the names you just imported. (Easiest way is probably to take a snapshot of globals().keys() before and after.)
Instead, import just the module:
import testType1
import testType2
Now you can easily get a list of what's in each module:
tests = dir(testType1)
And access each using getattr() on the module object:
for testname in tests:
test = getattr(testType1, testname)
if callable(test):
# do something with it
A Python namespace package can be spread over many directories, and zip files or custom importers. What's the correct way to iterate over all the importable submodules of a namespace package?
Here is a way that works well for me. Create a new submodule all.py, say, in one of the packages in the namespace.
If you write
import mynamespace.all
you are given the object for the mynamespace module. This object contains all of the loaded modules in the namespace, irrespective of where they were loaded, since there is only one instance of mynamespace around.
So, just load all the packages in the namespace in all.py!
# all.py
from pkgutil import iter_modules
# import this module's namespace (= parent) package
pkg = __import__(__package__)
# iterate all modules in pkg's paths,
# prefixing the returned module names with namespace-dot,
# and import the modules by name
for m in iter_modules(pkg.__path__, __package__ + '.'):
__import__(m.name)
Or in a one-liner that keeps the all module empty, if you care about that sort of thing:
# all.py
(lambda: [__import__(_.name) for _ in __import__('pkgutil').iter_modules(__import__(__package__).__path__, __package__ + '.')])() # noqa
After importing the all module from your namespace, you then actually receive a fully populated namespace module:
import mynamespace.all
mynamespace.mymodule1 # works
mynamespace.mymodule2 # works
...
Of course, you can use the same mechanism to enumerate or otherwise process the modules in the namespace, if you do not want to import them immediately.
Please read import confusion.
It very clearly distinguishes all the different ways you can import packages and its sub modules and in the process answers your question. When you need a certain submodule from a package, it’s often much more convenient to write from io.drivers import zip than import io.drivers.zip, since the former lets you refer to the module simply as zip instead of its full name.
from modname import *, this provides an easy way to import all the items from a module into the current namespace; however, this statement should be used sparingly.
Is there a way to load a module twice in the same python session?
To fill this question with an example: Here is a module:
Mod.py
x = 0
Now I would like to import that module twice, like creating two instances of a class to have actually two copies of x.
To already answer the questions in the comments, "why anyone would want to do that if they could just create a class with x as a variable":
You are correct, but there exists some huge amount of source that would have to be rewritten, and loading a module twice would be a quick fix^^.
Yes, you can load a module twice:
import mod
import sys
del sys.modules["mod"]
import mod as mod2
Now, mod and mod2 are two instances of the same module.
That said, I doubt this is ever useful. Use classes instead -- eventually it will be less work.
Edit: In Python 2.x, you can also use the following code to "manually" import a module:
import imp
def my_import(name):
file, pathname, description = imp.find_module(name)
code = compile(file.read(), pathname, "exec", dont_inherit=True)
file.close()
module = imp.new_module(name)
exec code in module.__dict__
return module
This solution might be more flexible than the first one. You no longer have to "fight" the import mechanism since you are (partly) rolling your own one. (Note that this implementation doesn't set the __file__, __path__ and __package__ attributes of the module -- if these are needed, just add code to set them.)
Deleting an entry from sys.modules will not necessarily work (e.g. it will fail when importing recurly twice, if you want to work with multiple recurly accounts in the same app etc.)
Another way to accomplish this is:
>>> import importlib
>>> spec = importlib.util.find_spec(module_name)
>>> instance_one = importlib.util.module_from_spec(spec)
>>> instance_two = importlib.util.module_from_spec(spec)
>>> instance_one == instance_two
False
You could use the __import__ function.
module1 = __import__("module")
module2 = __import__("module")
Edit: As it turns out, this does not import two separate versions of the module, instead module1 and module2 will point to the same object, as pointed out by Sven.