Related
The objective:
I have a package with submodules that I would like to be accessible in the most straightforward way possible. The submodules contain classes to take advantage of the class structure, but don't need to be initialized (as they contain static and class methods). So, ideally, I would like to access them as follows:
from myPackage.subModule import someMethod
print (someMethod)
from myPackage import subModule
print (subModule.someMethod)
import myPackage
print(myPackage.subModule.someMethod)
Here is the package structure:
myPackage ─┐
__init__.py
subModule
subModule2
etc.
Example of a typical submodule:
# submodule.py
class SomeClass():
someAttr = list(range(10))
#classmethod
def someMethod(cls):
pass
#staticmethod
def someMethod2():
pass
Here is the code I have in my '__init __.py': In order to achieve the above; it attempts to set attributes for each class at the package level, and the same for it's methods at the sub-module level.
# __init__.py
def import_submodules(package, filetypes=('py', 'pyc', 'pyd'), ignoreStartingWith='_'):
'''Import submodules to the given package, expose any classes at the package level
and their respective class methods at submodule level.
:Parameters:
package (str)(obj) = A python package.
filetypes (str)(tuple) = Filetype extension(s) to include.
ignoreStartingWith (str)(tuple) = Ignore submodules starting with given chars.
'''
if isinstance(package, str):
package = sys.modules[package]
if not package:
return
pkg_dir = os.path.dirname(os.path.abspath(package.__file__))
sys.path.append(pkg_dir) #append this dir to the system path.
for mod_name in os.listdir(pkg_dir):
if mod_name.startswith(ignoreStartingWith):
continue
elif os.path.isfile(os.path.join(pkg_dir, mod_name)):
mod_name, *mod_ext = mod_name.rsplit('.', 1)
if filetypes:
if not mod_ext or mod_ext[0] not in filetypes:
continue
mod = importlib.import_module(mod_name)
vars(package)[mod_name] = mod
classes = inspect.getmembers(mod, inspect.isclass)
for cls_name, clss in classes:
vars(package)[cls_name] = clss
methods = inspect.getmembers(clss, inspect.isfunction)
for method_name, method in methods:
vars(mod)[method_name] = method
del mod_name
import_submodules(__name__)
At issue is this line:
vars(mod)[method_name] = method
Which ultimately results in: (indicating that the attribute was not set)
from myPackage.subModule import someMethod
ImportError: cannot import name 'someMethod' from 'myPackage.subModule'
I am able to set the methods as attributes to the module within that module, but setting them from outside (ie. in the package __init __), isn't working as written. I understand this isn't ideal to begin with, but my current logic is; that the ease of use, outweighs any perceived issues with namespace pollution. I am, of course, always open to counter-arguments.
I just checked it on my machine.
Created a package myPackage with a module subModule that has a function someMethod.
I run a python shell with working directory in the same directory that the myPackage is in, and to get these 3 import statements to work:
from myPackage.subModule import someMethod
from myPackage import subModule
import myPackage
All I had to do was to create an __init__.py with this line in it:
from . import subModule
Found a nice "hacky" solution -
subModule.py:
class myClass:
#staticmethod
def someMethod():
print("I have a bad feeling about this")
myInstance = myClass()
someMethod = myInstance.someMethod
init.py is empty
Still scratching my head of why I am unable to do this from the package __init __, but this solution works with the caveat it has to be called at the end of each submodule. Perhaps someone, in the future, someone can chime in as to why this wasn't working when completely contained in the __init __.
def addMembers(module, ignoreStartingWith='_'):
'''Expose class members at module level.
:Parameters:
module (str)(obj) = A python module.
ignoreStartingWith (str)(tuple) = Ignore class members starting with given chars.
ex. call: addMembers(__name__)
'''
if isinstance(module, str):
module = sys.modules[module]
if not module:
return
classes = inspect.getmembers(module, inspect.isclass)
for cls_name, clss in classes:
cls_members = [(o, getattr(clss, o)) for o in dir(clss) if not o.startswith(ignoreStartingWith)]
for name, mem in cls_members:
vars(module)[name] = mem
This is the solution I ended up going with. It needs to be put at the end of each submodule of your package. But, it is simple and in addition to all the standard ways of importing, allows you to import a method directly:
def __getattr__(attr):
'''Attempt to get a class attribute.
:Parameters:
attr (str): A name of a class attribute.
:Return:
(obj) The attribute.
'''
try:
return getattr(Someclass, attr)
except AttributeError as error:
raise AttributeError(f'{__file__} in __getattr__\n\t{error} ({type(attr).__name__})')
from somePackage.someModule import someMethod
In python, I have to instantiate certain class, knowing its name in a string, but this class 'lives' in a dynamically imported module. An example follows:
loader-class script:
import sys
class loader:
def __init__(self, module_name, class_name): # both args are strings
try:
__import__(module_name)
modul = sys.modules[module_name]
instance = modul.class_name() # obviously this doesn't works, here is my main problem!
except ImportError:
# manage import error
some-dynamically-loaded-module script:
class myName:
# etc...
I use this arrangement to make any dynamically-loaded-module to be used by the loader-class following certain predefined behaviours in the dyn-loaded-modules...
You can use getattr
getattr(module, class_name)
to access the class. More complete code:
module = __import__(module_name)
class_ = getattr(module, class_name)
instance = class_()
As mentioned below, we may use importlib
import importlib
module = importlib.import_module(module_name)
class_ = getattr(module, class_name)
instance = class_()
tl;dr
Import the root module with importlib.import_module and load the class by its name using getattr function:
# Standard import
import importlib
# Load "module.submodule.MyClass"
MyClass = getattr(importlib.import_module("module.submodule"), "MyClass")
# Instantiate the class (pass arguments to the constructor, if needed)
instance = MyClass()
explanations
You probably don't want to use __import__ to dynamically import a module by name, as it does not allow you to import submodules:
>>> mod = __import__("os.path")
>>> mod.join
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'join'
Here is what the python doc says about __import__:
Note: This is an advanced function that is not needed in everyday
Python programming, unlike importlib.import_module().
Instead, use the standard importlib module to dynamically import a module by name. With getattr you can then instantiate a class by its name:
import importlib
my_module = importlib.import_module("module.submodule")
MyClass = getattr(my_module, "MyClass")
instance = MyClass()
You could also write:
import importlib
module_name, class_name = "module.submodule.MyClass".rsplit(".", 1)
MyClass = getattr(importlib.import_module(module_name), class_name)
instance = MyClass()
This code is valid in python ≥ 2.7 (including python 3).
Copy-paste snippet:
import importlib
def str_to_class(module_name, class_name):
"""Return a class instance from a string reference"""
try:
module_ = importlib.import_module(module_name)
try:
class_ = getattr(module_, class_name)()
except AttributeError:
logging.error('Class does not exist')
except ImportError:
logging.error('Module does not exist')
return class_ or None
Use getattr to get an attribute from a name in a string. In other words, get the instance as
instance = getattr(modul, class_name)()
One can simply use the pydoc.locate function.
from pydoc import locate
my_class = locate("module.submodule.myclass")
instance = my_class()
If you want this sentence from foo.bar import foo2 to be loaded dynamically, you should do this
foo = __import__("foo")
bar = getattr(foo,"bar")
foo2 = getattr(bar,"foo2")
instance = foo2()
If you want to import a class and method from string, you should do this:
dynamic_import
│ my_class.py
│
└───subfolder
│ │ my_subfolder_module.py
│ │
my_subfolder_module.py
class MySubfolderClass():
def test_method(self):
print ("Hello World")
main.py
import importlib
module = importlib.import_module('subfolder.my_subfolder_module')
class_ = getattr(module, "MySubfolderClass")
method_instance = getattr(class_(),"test_method")
method_instance()
#it will output the result of the test method, which is "Hello World"
I couldn't quite get there in my use case from the examples above, but Ahmad got me the closest (thank you). For those reading this in the future, here is the code that worked for me.
def get_class(fully_qualified_path, module_name, class_name, *instantiation):
"""
Returns an instantiated class for the given string descriptors
:param fully_qualified_path: The path to the module eg("Utilities.Printer")
:param module_name: The module name eg("Printer")
:param class_name: The class name eg("ScreenPrinter")
:param instantiation: Any fields required to instantiate the class
:return: An instance of the class
"""
p = __import__(fully_qualified_path)
m = getattr(p, module_name)
c = getattr(m, class_name)
instance = c(*instantiation)
return instance
Use this snippet code:
def to_class(path:str):
try:
from pydoc import locate
class_instance = locate(path)
except ImportError:
print('Module does not exist')
return class_instance or None
Usage:
if your class name is MyClass and located in my_app.models.MyClass then:
path = "my_app.models.MyClass"
my_class = to_class(path)
I'm using Python for a weeks now and i'm confronted to an issue with dynamic import.
I have a file Test.py that in which a class is defined. I would like to use this class after the dynamic import of Test.py from another file.
My final goal is more complex but I simplified it but i still get the same problem.
File : Test.py
class Test :
def __init__ ( self ) :
print ( "instance" )
File : Main.py
def allImports ( ) :
__import__ ( "Test" )
What i get :
>>> import Main
>>> Main.allImports()
>>> myInstance = Test ()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'Test' is not defined
I cannot specify in the fromlist which element from Test.py i have to import because i'm not supposed to know them.
What should i do ?
For a solution closer to your intent:
import importlib
def allImports(globals):
mod = importlib.import_module('Test', globals['__name__'])
try:
keys = mod.__all__
except AttributeError:
keys = dir(mod)
for key in keys:
if not key.startswith('_'):
globals[key] = getattr(mod, key)
# …
allImports(globals())
Test # should work, you can also look into dir(Test) to find the class.
If your module doesn't have an __all__ the code above /will/ clobber your namespace something fierce. Either make sure you define __all__, or modify allImports() to only import the things you want. (E.g. only classes, or only classes defined in the module. This part really depends on your use case.)
When using __import__() to load a module, you have to look it up in sys.modules:
>>> import sys
>>> import Main
>>> Main.allImports()
>>> myInstance = sys.modules['Test'].Test()
instance
>>>
More information in the documentation and here, here, and here.
this code makes __import__ ( "Test" ) a local variable, so you can't access it outside the function.
def allImports ( ) :
__import__ ( "Test" )
try:
def allImports ( ) :
test= __import__ ( "Test" )
return test #return the module
>>> import Main
>>> x=Main.allImports() #store the returned module in x
>>> myInstance = x.Test ()
instance
>>>myInstance
<Test.Test instance at 0x011D7F80>
__import__ doesn't modify magically neither global nor local namespaces.
Modules and classes are first class citizens in Python i.e., you can use them as any other object in Python (bind to a name, pass as a parameter to a function, return as a value from a function).
def allImports():
return __import__("Test")
Test_module = allImports()
Test = Test_module.Test # class
test_instance = Test()
If the above code is inside a function then to put Test into global namespace: globals()['Test'] = Test. Note most probably you don't need it and there are better ways to do whatever you want without modifying global namespace inside a function.
Usage of __import__() is discouraged use importlib.import_module() instead.
If the name of the module and the class are known you could just write at the module level:
from Test import Test
With my python application, I have 40 modules (classes) which contain a parser for some text. In my function I only want to instanciate and use a particular module. These are all sorted in the database.
I am at the point now where I know my parser, and have both the python file name and class I want to import and create
However.... How do you actually do this in python?
eg;
file_name = 'lex_parser'
class_name = 'LexParser'
how can I do....
from {file_name} import {class_name}
Parser = {class_name}()
Follow what I mean?
Try this:
file_name = 'lex_parser'
class_name = 'LexParser'
Parser = getattr(__import__(file_name), class_name)
Note that file_name must not contain .py.
This won't work if the module is within a package because __import__ would return the top level package. In that case you can do this:
import sys
file_name = 'parsers.lex_parser'
class_name = 'LexParser'
__import__(file_name)
Parser = getattr(sys.modules[file_name], class_name)
This will work in both cases and is recommeded by the __import__ function documentation.
In both examples Parser is a class which you have to instantiate as normally:
parser = Parser()
How about something like this:
module = __import__('my_module')
if hasattr(module, 'ClassName'):
ClassName = module.ClassName
my_object = ClassName()
Given a string of a Python class, e.g. my_package.my_module.MyClass, what is the best possible way to load it?
In other words I am looking for a equivalent Class.forName() in Java, function in Python. It needs to work on Google App Engine.
Preferably this would be a function that accepts the FQN of the class as a string, and returns a reference to the class:
my_class = load_class('my_package.my_module.MyClass')
my_instance = my_class()
From the python documentation, here's the function you want:
def my_import(name):
components = name.split('.')
mod = __import__(components[0])
for comp in components[1:]:
mod = getattr(mod, comp)
return mod
The reason a simple __import__ won't work is because any import of anything past the first dot in a package string is an attribute of the module you're importing. Thus, something like this won't work:
__import__('foo.bar.baz.qux')
You'd have to call the above function like so:
my_import('foo.bar.baz.qux')
Or in the case of your example:
klass = my_import('my_package.my_module.my_class')
some_object = klass()
EDIT: I was a bit off on this. What you're basically wanting to do is this:
from my_package.my_module import my_class
The above function is only necessary if you have a empty fromlist. Thus, the appropriate call would be like this:
mod = __import__('my_package.my_module', fromlist=['my_class'])
klass = getattr(mod, 'my_class')
If you don't want to roll your own, there is a function available in the pydoc module that does exactly this:
from pydoc import locate
my_class = locate('my_package.my_module.MyClass')
The advantage of this approach over the others listed here is that locate will find any python object at the provided dotted path, not just an object directly within a module. e.g. my_package.my_module.MyClass.attr.
If you're curious what their recipe is, here's the function:
def locate(path, forceload=0):
"""Locate an object by name or dotted path, importing as necessary."""
parts = [part for part in split(path, '.') if part]
module, n = None, 0
while n < len(parts):
nextmodule = safeimport(join(parts[:n+1], '.'), forceload)
if nextmodule: module, n = nextmodule, n + 1
else: break
if module:
object = module
else:
object = __builtin__
for part in parts[n:]:
try:
object = getattr(object, part)
except AttributeError:
return None
return object
It relies on pydoc.safeimport function. Here are the docs for that:
"""Import a module; handle errors; return None if the module isn't found.
If the module *is* found but an exception occurs, it's wrapped in an
ErrorDuringImport exception and reraised. Unlike __import__, if a
package path is specified, the module at the end of the path is returned,
not the package at the beginning. If the optional 'forceload' argument
is 1, we reload the module from disk (unless it's a dynamic extension)."""
import importlib
module = importlib.import_module('my_package.my_module')
my_class = getattr(module, 'MyClass')
my_instance = my_class()
If you're using Django you can use import_string.
Yes i'm aware OP did not ask for django, but i ran across this question looking for a Django solution, didn't find one, and put it here for the next boy/gal that looks for it.
# It's available for v1.7+
# https://github.com/django/django/blob/stable/1.7.x/django/utils/module_loading.py
from django.utils.module_loading import import_string
Klass = import_string('path.to.module.Klass')
func = import_string('path.to.module.func')
var = import_string('path.to.module.var')
Keep in mind, if you want to import something that doesn't have a ., like re or argparse use:
re = __import__('re')
def import_class(cl):
d = cl.rfind(".")
classname = cl[d+1:len(cl)]
m = __import__(cl[0:d], globals(), locals(), [classname])
return getattr(m, classname)
Here is to share something I found on __import__ and importlib while trying to solve this problem.
I am using Python 3.7.3.
When I try to get to the class d in module a.b.c,
mod = __import__('a.b.c')
The mod variable refer to the top namespace a.
So to get to the class d, I need to
mod = getattr(mod, 'b') #mod is now module b
mod = getattr(mod, 'c') #mod is now module c
mod = getattr(mod, 'd') #mod is now class d
If we try to do
mod = __import__('a.b.c')
d = getattr(mod, 'd')
we are actually trying to look for a.d.
When using importlib, I suppose the library has done the recursive getattr for us. So, when we use importlib.import_module, we actually get a handle on the deepest module.
mod = importlib.import_module('a.b.c') #mod is module c
d = getattr(mod, 'd') #this is a.b.c.d
OK, for me that is the way it worked (I am using Python 2.7):
a = __import__('file_to_import', globals(), locals(), ['*'], -1)
b = a.MyClass()
Then, b is an instance of class 'MyClass'
If you happen to already have an instance of your desired class, you can use the 'type' function to extract its class type and use this to construct a new instance:
class Something(object):
def __init__(self, name):
self.name = name
def display(self):
print(self.name)
one = Something("one")
one.display()
cls = type(one)
two = cls("two")
two.display()
Python has an inbuilt library importlib to get the job done. :, How to access module method and class method dynamically bypassing package name as a param. An example is given below.
Module 1:
def get_scenario_data():
return "module1 scenario data"
class Module1:
def module1_function1(self):
return "module1_function"
def module1_function2(self):
return "module2_function"
Module 2:
def get_scenario_data():
return "module2 scenario data"
class Module2:
def module2_function1(self):
return "module2_function1"
def module2_function2(self):
return "module2_function2"
ModuleTest:
Will access the module methods dynamically based on the package name as param
Will access the class methods dynamically based on the package name as param.
ModuleTest
import importlib
module = importlib.import_module('pack1.nestedpack1.module1')
print(module.get_scenario_data())
modul1_cls_obj = getattr(module, 'Module1')()
print(modul1_cls_obj.module1_function1())
print(modul1_cls_obj.module1_function2())
module = importlib.import_module('pack1.nestedpack1.module2')
modul2_cls_obj = getattr(module, 'Module2')()
print(modul2_cls_obj.module2_function1())
print(modul2_cls_obj.module2_function2())
print(module.get_scenario_data())
Results
module1 scenario data
module1_function
module2_function
module2_function1
module2_function2
module2 scenario data
PyPI module autoloader & import
# PyPI imports
import pkg_resources, subprocess, sys
modules = {'lxml.etree', 'pandas', 'screeninfo'}
required = {m.split('.')[0] for m in modules}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', 'pip'])
subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])
for module in set.union(required, modules):
globals()[module] = __import__(module)
Tests:
print(pandas.__version__)
print(lxml.etree.LXML_VERSION)
Adding a bit of sophistication to the existing answers....
Depending on the use case, it may be somewhat inconvenient to have to explicitly specify the full path (E.g. package.subpackage.module...) of the class/method you want to import. On top of importlib, we can leverage __init__.py to make things even cleaner.
Let's say I have a python package, like so:
├── modes
│ ├── __init__.py
│ ├── bar.py
│ ├── foo.py
│ ├── modes.py
foo.py, say, have some class/functions we'd like to use somewhere else in our program:
from modes.modes import Mode
class Foo(Mode):
def __init__(self, *arg, **kwargs):
super(Foo, self).__init__(*arg, **kwargs)
def run(self):
self.LOG.info(f"This is FOO!")
With a command line argument, I can pass an argument that corresponds to a mode that I want to run. I'd like to be able to so something like this:
def set_mode(mode):
""" """
import importlib
module = importlib.import_module('modes.foo')
getattr(module, mode)().run()
which outputs:
>> set_mode("Foo")
>> engine_logger:INFO - This is FOO!
That works fine, however what we'd REALLY want to get at is this:
def set_mode(mode):
""" """
import importlib
module = importlib.import_module('modes') # only import the package, not modules explicitely
getattr(module, mode)().run()
Which raises an error:
>> set_mode("Foo")
>> AttributeError: module 'modes' has no attribute 'Foo'
However, we can add the following to /modes/__init__.py:
from .foo import Foo
from .bar import Bar
Then, we can do:
>> set_mode("Foo")
>> engine_logger:INFO - This is FOO!
>> set_mode("Bar")
>> engine_logger:INFO - This is BAR!
In other worlds, all sub modules/functions/classes we import in init.py will be found directly with importlib.import_module(...), without having to specify the full path from outside.
In Google App Engine there is a webapp2 function called import_string. For more info see here:https://webapp-improved.appspot.com/api/webapp2.html
So,
import webapp2
my_class = webapp2.import_string('my_package.my_module.MyClass')
For example this is used in the webapp2.Route where you can either use a handler or a string.
module = __import__("my_package/my_module")
the_class = getattr(module, "MyClass")
obj = the_class()