Related
When I load the source code of a class from a module directly, it's fine:
import arg_master
inspect.getsource(func)
When I load a module with spec_from_file_location and go for a function it's fine.
When I load a module with spec_from_file_location and go for a class,
it fails with:
TypeError: <class 'mymod.ArgMaster'> is a built-in class
(it's not. I wrote it.)
Here is my full source:
import os, inspect, importlib
filename = 'arg_master.py'
spec = importlib.util.spec_from_file_location("mymod", filename)
mymod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mymod)
func = vars(mymod)['ArgMaster']
inspect.getsource(func) #<<< Fails
Load method number 2 also fails:
import importlib, types
filename = 'arg_master.py'
loader = importlib.machinery.SourceFileLoader('mod', filename)
mod = types.ModuleType(loader.name)
loader.exec_module(mod)
func = vars(mod)['ArgMaster']
inspect.getsource(func)
Edit: I found a hackish solution:
import inspect
filename = 'arg_master.py'
name = os.path.basename(filename)
name = os.path.splitext(name)[0]
importlib.import_module(name)
func = vars(mod)['ArgMaster']
inspect.getsource(func)
I get the same error with your code, which oddly enough works for functions, but not classes. This below is essentially the same as your solution, just slightly less "hacky" as you don't have to mess with file paths:
import inspect, importlib
cls = getattr(importlib.import_module('arg_master'), 'ArgMaster')
print(inspect.getsource(cls))
This will work if the two modules in the same dir. If you need to do relative imports, might have to mess around with package=__pckage__ or similar
I am trying to instantiate a class dynamically from a string that is fed to a method.
from pydoc import locate
name = "folder.subfolder.classname"
my_class = locate(name)
instance = my_class()
Doing this results in an error:
TypeError: 'module' object is not callable
My question is: how can I dinamically know which class I need to call? If I hardcode the value and do my_class.classname() I can instantiate it, but I want that last part to be dynamic, instead of hardcoded. How can I do that?
Edit 2:
This solution will gather all classes available in the module the user provides, and then simply instantiates the first class.
import inspect
from pydoc import locate
name = "folder.subfolder.classname"
my_module = locate(name)
all_classes = inspect.getmembers(my_module, inspect.isclass)
if len(all_classes) > 0:
instance = all_classes[0][1]()
Edit:
Gathered from the comments I believe the following code will work for your use case:
from pydoc import locate
name = "folder.subfolder.classname"
my_module = locate(name)
class_name = name.rsplit('.')[-1]
my_class = getattr(my_module, class_name)
instance = my_class()
You do seem to be a bit confused as to what modules are which resulted in our miscommunication. class.py doesn't point to a class, but to a module. In that module (your class.py file) there can be multiple classes. For the answer above I just assume that in your class.py file you have a class with the same name as the file.
Original Answer:
Well as your error shows, your my_class variable is actually a module object. The name you provide doesn't actually point to your class but to the module containing the class.
To get a class from a module object by string you can do:
my_class = getattr(module, class_name)
and then like in your example code you can instantiate it like this:
instance = my_class()
To bring it all together:
module_name = "folder.subfolder.module_name"
my_module = locate(module_name)
my_class = getattr(my_module, class_name)
instance = my_class()
You didn't locate the class inside of the file:
from pydoc import locate
name = "folder.subfolder.filename.classname"
my_class = locate(name)
instance = my_class()
I have a string, say: abc.def.ghi.jkl.myfile.mymethod. How do I dynamically import mymethod?
Here is how I went about it:
def get_method_from_file(full_path):
if len(full_path) == 1:
return map(__import__,[full_path[0]])[0]
return getattr(get_method_from_file(full_path[:-1]),full_path[-1])
if __name__=='__main__':
print get_method_from_file('abc.def.ghi.jkl.myfile.mymethod'.split('.'))
I am wondering if the importing individual modules is required at all.
Edit: I am using Python version 2.6.5.
From Python 2.7 you can use the importlib.import_module() function. You can import a module and access an object defined within it with the following code:
from importlib import import_module
p, m = name.rsplit('.', 1)
mod = import_module(p)
met = getattr(mod, m)
met()
You don't need to import the individual modules. It is enough to import the module you want to import a name from and provide the fromlist argument:
def import_from(module, name):
module = __import__(module, fromlist=[name])
return getattr(module, name)
For your example abc.def.ghi.jkl.myfile.mymethod, call this function as
import_from("abc.def.ghi.jkl.myfile", "mymethod")
(Note that module-level functions are called functions in Python, not methods.)
For such a simple task, there is no advantage in using the importlib module.
For Python < 2.7 the builtin method __ import__ can be used:
__import__('abc.def.ghi.jkl.myfile.mymethod', fromlist=[''])
For Python >= 2.7 or 3.1 the convenient method importlib.import_module has been added. Just import your module like this:
importlib.import_module('abc.def.ghi.jkl.myfile.mymethod')
Update: Updated version according to comments (I must admit I didn't read the string to be imported till the end and I missed the fact that a method of a module should be imported and not a module itself):
Python < 2.7 :
mymethod = getattr(__import__("abc.def.ghi.jkl.myfile", fromlist=["mymethod"]))
Python >= 2.7:
mymethod = getattr(importlib.import_module("abc.def.ghi.jkl.myfile"), "mymethod")
from importlib import import_module
name = "file.py".strip('.py')
# if Path like : "path/python/file.py"
# use name.replaces("/",".")
imp = import_module(name)
# get Class From File.py
model = getattr(imp, "classNameImportFromFile")
NClass = model() # Class From file
It's unclear what you are trying to do to your local namespace. I assume you want just my_method as a local, typing output = my_method()?
# This is equivalent to "from a.b.myfile import my_method"
the_module = importlib.import_module("a.b.myfile")
same_module = __import__("a.b.myfile")
# import_module() and __input__() only return modules
my_method = getattr(the_module, "my_method")
# or, more concisely,
my_method = getattr(__import__("a.b.myfile"), "my_method")
output = my_method()
While you only add my_method to the local namespace, you do load the chain of modules. You can look at changes by watching the keys of sys.modules before and after the import. I hope this is clearer and more accurate than your other answers.
For completeness, this is how you add the whole chain.
# This is equivalent to "import a.b.myfile"
a = __import__("a.b.myfile")
also_a = importlib.import_module("a.b.myfile")
output = a.b.myfile.my_method()
# This is equivalent to "from a.b import myfile"
myfile = __import__("a.b.myfile", fromlist="a.b")
also_myfile = importlib.import_module("a.b.myfile", "a.b")
output = myfile.my_method()
And, finally, if you are using __import__() and have modified you search path after the program started, you may need to use __import__(normal args, globals=globals(), locals=locals()). The why is a complex discussion.
This website has a nice solution: load_class. I use it like this:
foo = load_class(package.subpackage.FooClass)()
type(foo) # returns FooClass
As requested, here is the code from the web link:
import importlib
def load_class(full_class_string):
"""
dynamically load a class from a string
"""
class_data = full_class_string.split(".")
module_path = ".".join(class_data[:-1])
class_str = class_data[-1]
module = importlib.import_module(module_path)
# Finally, we retrieve the Class
return getattr(module, class_str)
Use importlib (2.7+ only).
The way I tend to to this (as well as a number of other libraries, such as pylons and paste, if my memory serves me correctly) is to separate the module name from the function/attribute name by using a ':' between them. See the following example:
'abc.def.ghi.jkl.myfile:mymethod'
This makes the import_from(path) function below a little easier to use.
def import_from(path):
"""
Import an attribute, function or class from a module.
:attr path: A path descriptor in the form of 'pkg.module.submodule:attribute'
:type path: str
"""
path_parts = path.split(':')
if len(path_parts) < 2:
raise ImportError("path must be in the form of pkg.module.submodule:attribute")
module = __import__(path_parts[0], fromlist=path_parts[1])
return getattr(module, path_parts[1])
if __name__=='__main__':
func = import_from('a.b.c.d.myfile:mymethod')
func()
How about this :
def import_module(name):
mod = __import__(name)
for s in name.split('.')[1:]:
mod = getattr(mod, s)
return mod
I have a package of plug-in style modules. It looks like this:
/Plugins
/Plugins/__init__.py
/Plugins/Plugin1.py
/Plugins/Plugin2.py
etc...
Each .py file contains a class that derives from PluginBaseClass. So I need to list every module in the Plugins package and then search for any classes that implement PluginBaseClass. Ideally I want to be able to do something like this:
for klass in iter_plugins(project.Plugins):
action = klass()
action.run()
I have seen some other answers out there, but my situation is different. I have an actual import to the base package (ie: import project.Plugins) and I need to find the classes after discovering the modules.
Edit: here's a revised solution. I realised I was making a mistake while testing my previous one, and it doesn't really work the way you would expect. So here is a more complete solution:
import os
from imp import find_module
from types import ModuleType, ClassType
def iter_plugins(package):
"""Receives package (as a string) and, for all of its contained modules,
generates all classes that are subclasses of PluginBaseClass."""
# Despite the function name, "find_module" will find the package
# (the "filename" part of the return value will be None, in this case)
filename, path, description = find_module(package)
# dir(some_package) will not list the modules within the package,
# so we explicitly look for files. If you need to recursively descend
# a directory tree, you can adapt this to use os.walk instead of os.listdir
modules = sorted(set(i.partition('.')[0]
for i in os.listdir(path)
if i.endswith(('.py', '.pyc', '.pyo'))
and not i.startswith('__init__.py')))
pkg = __import__(package, fromlist=modules)
for m in modules:
module = getattr(pkg, m)
if type(module) == ModuleType:
for c in dir(module):
klass = getattr(module, c)
if (type(klass) == ClassType and
klass is not PluginBaseClass and
issubclass(klass, PluginBaseClass)):
yield klass
My previous solution was:
You could try something like:
from types import ModuleType
import Plugins
classes = []
for item in dir(Plugins):
module = getattr(Plugins, item)
# Get all (and only) modules in Plugins
if type(module) == ModuleType:
for c in dir(module):
klass = getattr(module, c)
if isinstance(klass, PluginBaseClass):
classes.append(klass)
Actually, even better, if you want some modularity:
from types import ModuleType
def iter_plugins(package):
# This assumes "package" is a package name.
# If it's the package itself, you can remove this __import__
pkg = __import__(package)
for item in dir(pkg):
module = getattr(pkg, item)
if type(module) == ModuleType:
for c in dir(module):
klass = getattr(module, c)
if issubclass(klass, PluginBaseClass):
yield klass
You may (and probably should) define __all__ in __init__.py as a list of the submodules in your package; this is so that you support people doing from Plugins import *. If you have done so, you can iterate over the modules with
import Plugins
import sys
modules = { }
for module in Plugins.__all__:
__import__( module )
modules[ module ] = sys.modules[ module ]
# iterate over dir( module ) as above
The reason another answer posted here fails is that __import__ imports the lowest-level module, but returns the top-level one (see the docs). I don't know why.
Scanning modules isn't good idea. If you need class registry you should look at metaclasses or use existing solutions like zope.interface.
Simple solution through metaclasses may look like that:
from functools import reduce
class DerivationRegistry(type):
def __init__(cls,name,bases,cls_dict):
type.__init__(cls,name,bases,cls_dict)
cls._subclasses = set()
for base in bases:
if isinstance(base,DerivationRegistry):
base._subclasses.add(cls)
def getSubclasses(cls):
return reduce( set.union,
( succ.getSubclasses() for succ in cls._subclasses if isinstance(succ,DerivationRegistry)),
cls._subclasses)
class Base(object):
__metaclass__ = DerivationRegistry
class Cls1(object):
pass
class Cls2(Base):
pass
class Cls3(Cls2,Cls1):
pass
class Cls4(Cls3):
pass
print(Base.getSubclasses())
If you don't know what's going to be in Plugins ahead of time, you can get a list of python files in the package's directory, and import them like so:
# compute a list of modules in the Plugins package
import os
import Plugins
plugin_modules = [f[:-3] for f in os.listdir(os.path.dirname(Plugins.__file__))
if f.endswith('.py') and f != '__init__.py']
Sorry, that comprehension might be a mouthful for someone relatively new to python. Here's a more verbose version (might be easier to follow):
plugin_modules = []
package_path = Plugins.__file__
file_list = os.listdir(os.path.dirname(package_path))
for file_name in file_list:
if file_name.endswith('.py') and file_name != '__init__.py':
plugin_modules.append(file_name)
Then you can use __import__ to get the module:
# get the first one
plugin = __import__('Plugins.' + plugin_modules[0])
Given a string of a Python class, e.g. my_package.my_module.MyClass, what is the best possible way to load it?
In other words I am looking for a equivalent Class.forName() in Java, function in Python. It needs to work on Google App Engine.
Preferably this would be a function that accepts the FQN of the class as a string, and returns a reference to the class:
my_class = load_class('my_package.my_module.MyClass')
my_instance = my_class()
From the python documentation, here's the function you want:
def my_import(name):
components = name.split('.')
mod = __import__(components[0])
for comp in components[1:]:
mod = getattr(mod, comp)
return mod
The reason a simple __import__ won't work is because any import of anything past the first dot in a package string is an attribute of the module you're importing. Thus, something like this won't work:
__import__('foo.bar.baz.qux')
You'd have to call the above function like so:
my_import('foo.bar.baz.qux')
Or in the case of your example:
klass = my_import('my_package.my_module.my_class')
some_object = klass()
EDIT: I was a bit off on this. What you're basically wanting to do is this:
from my_package.my_module import my_class
The above function is only necessary if you have a empty fromlist. Thus, the appropriate call would be like this:
mod = __import__('my_package.my_module', fromlist=['my_class'])
klass = getattr(mod, 'my_class')
If you don't want to roll your own, there is a function available in the pydoc module that does exactly this:
from pydoc import locate
my_class = locate('my_package.my_module.MyClass')
The advantage of this approach over the others listed here is that locate will find any python object at the provided dotted path, not just an object directly within a module. e.g. my_package.my_module.MyClass.attr.
If you're curious what their recipe is, here's the function:
def locate(path, forceload=0):
"""Locate an object by name or dotted path, importing as necessary."""
parts = [part for part in split(path, '.') if part]
module, n = None, 0
while n < len(parts):
nextmodule = safeimport(join(parts[:n+1], '.'), forceload)
if nextmodule: module, n = nextmodule, n + 1
else: break
if module:
object = module
else:
object = __builtin__
for part in parts[n:]:
try:
object = getattr(object, part)
except AttributeError:
return None
return object
It relies on pydoc.safeimport function. Here are the docs for that:
"""Import a module; handle errors; return None if the module isn't found.
If the module *is* found but an exception occurs, it's wrapped in an
ErrorDuringImport exception and reraised. Unlike __import__, if a
package path is specified, the module at the end of the path is returned,
not the package at the beginning. If the optional 'forceload' argument
is 1, we reload the module from disk (unless it's a dynamic extension)."""
import importlib
module = importlib.import_module('my_package.my_module')
my_class = getattr(module, 'MyClass')
my_instance = my_class()
If you're using Django you can use import_string.
Yes i'm aware OP did not ask for django, but i ran across this question looking for a Django solution, didn't find one, and put it here for the next boy/gal that looks for it.
# It's available for v1.7+
# https://github.com/django/django/blob/stable/1.7.x/django/utils/module_loading.py
from django.utils.module_loading import import_string
Klass = import_string('path.to.module.Klass')
func = import_string('path.to.module.func')
var = import_string('path.to.module.var')
Keep in mind, if you want to import something that doesn't have a ., like re or argparse use:
re = __import__('re')
def import_class(cl):
d = cl.rfind(".")
classname = cl[d+1:len(cl)]
m = __import__(cl[0:d], globals(), locals(), [classname])
return getattr(m, classname)
Here is to share something I found on __import__ and importlib while trying to solve this problem.
I am using Python 3.7.3.
When I try to get to the class d in module a.b.c,
mod = __import__('a.b.c')
The mod variable refer to the top namespace a.
So to get to the class d, I need to
mod = getattr(mod, 'b') #mod is now module b
mod = getattr(mod, 'c') #mod is now module c
mod = getattr(mod, 'd') #mod is now class d
If we try to do
mod = __import__('a.b.c')
d = getattr(mod, 'd')
we are actually trying to look for a.d.
When using importlib, I suppose the library has done the recursive getattr for us. So, when we use importlib.import_module, we actually get a handle on the deepest module.
mod = importlib.import_module('a.b.c') #mod is module c
d = getattr(mod, 'd') #this is a.b.c.d
OK, for me that is the way it worked (I am using Python 2.7):
a = __import__('file_to_import', globals(), locals(), ['*'], -1)
b = a.MyClass()
Then, b is an instance of class 'MyClass'
If you happen to already have an instance of your desired class, you can use the 'type' function to extract its class type and use this to construct a new instance:
class Something(object):
def __init__(self, name):
self.name = name
def display(self):
print(self.name)
one = Something("one")
one.display()
cls = type(one)
two = cls("two")
two.display()
Python has an inbuilt library importlib to get the job done. :, How to access module method and class method dynamically bypassing package name as a param. An example is given below.
Module 1:
def get_scenario_data():
return "module1 scenario data"
class Module1:
def module1_function1(self):
return "module1_function"
def module1_function2(self):
return "module2_function"
Module 2:
def get_scenario_data():
return "module2 scenario data"
class Module2:
def module2_function1(self):
return "module2_function1"
def module2_function2(self):
return "module2_function2"
ModuleTest:
Will access the module methods dynamically based on the package name as param
Will access the class methods dynamically based on the package name as param.
ModuleTest
import importlib
module = importlib.import_module('pack1.nestedpack1.module1')
print(module.get_scenario_data())
modul1_cls_obj = getattr(module, 'Module1')()
print(modul1_cls_obj.module1_function1())
print(modul1_cls_obj.module1_function2())
module = importlib.import_module('pack1.nestedpack1.module2')
modul2_cls_obj = getattr(module, 'Module2')()
print(modul2_cls_obj.module2_function1())
print(modul2_cls_obj.module2_function2())
print(module.get_scenario_data())
Results
module1 scenario data
module1_function
module2_function
module2_function1
module2_function2
module2 scenario data
PyPI module autoloader & import
# PyPI imports
import pkg_resources, subprocess, sys
modules = {'lxml.etree', 'pandas', 'screeninfo'}
required = {m.split('.')[0] for m in modules}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', 'pip'])
subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])
for module in set.union(required, modules):
globals()[module] = __import__(module)
Tests:
print(pandas.__version__)
print(lxml.etree.LXML_VERSION)
Adding a bit of sophistication to the existing answers....
Depending on the use case, it may be somewhat inconvenient to have to explicitly specify the full path (E.g. package.subpackage.module...) of the class/method you want to import. On top of importlib, we can leverage __init__.py to make things even cleaner.
Let's say I have a python package, like so:
├── modes
│ ├── __init__.py
│ ├── bar.py
│ ├── foo.py
│ ├── modes.py
foo.py, say, have some class/functions we'd like to use somewhere else in our program:
from modes.modes import Mode
class Foo(Mode):
def __init__(self, *arg, **kwargs):
super(Foo, self).__init__(*arg, **kwargs)
def run(self):
self.LOG.info(f"This is FOO!")
With a command line argument, I can pass an argument that corresponds to a mode that I want to run. I'd like to be able to so something like this:
def set_mode(mode):
""" """
import importlib
module = importlib.import_module('modes.foo')
getattr(module, mode)().run()
which outputs:
>> set_mode("Foo")
>> engine_logger:INFO - This is FOO!
That works fine, however what we'd REALLY want to get at is this:
def set_mode(mode):
""" """
import importlib
module = importlib.import_module('modes') # only import the package, not modules explicitely
getattr(module, mode)().run()
Which raises an error:
>> set_mode("Foo")
>> AttributeError: module 'modes' has no attribute 'Foo'
However, we can add the following to /modes/__init__.py:
from .foo import Foo
from .bar import Bar
Then, we can do:
>> set_mode("Foo")
>> engine_logger:INFO - This is FOO!
>> set_mode("Bar")
>> engine_logger:INFO - This is BAR!
In other worlds, all sub modules/functions/classes we import in init.py will be found directly with importlib.import_module(...), without having to specify the full path from outside.
In Google App Engine there is a webapp2 function called import_string. For more info see here:https://webapp-improved.appspot.com/api/webapp2.html
So,
import webapp2
my_class = webapp2.import_string('my_package.my_module.MyClass')
For example this is used in the webapp2.Route where you can either use a handler or a string.
module = __import__("my_package/my_module")
the_class = getattr(module, "MyClass")
obj = the_class()