Main Goal: Automatically register classes (by a string) in a factory to be created dynamically at run time using that string, classes can be in their own file and not grouped in one file.
I have couple of classes which all inherit from the same base class and they define a string as their type.
A user wants to get an instance of one of these classes but only knows the type at run time.
Therefore I have a factory to create an instance given a type.
I didn't want to hard code an "if then statements" so I have a meta class to register all the sub classes of the base class:
class MetaRegister(type):
# we use __init__ rather than __new__ here because we want
# to modify attributes of the class *after* they have been
# created
def __init__(cls, name, bases, dct):
if not hasattr(cls, 'registry'):
# this is the base class. Create an empty registry
cls.registry = {}
else:
# this is a derived class. Add cls to the registry
interface_id = cls().get_model_type()
cls.registry[interface_id] = cls
super(MetaRegister, cls).__init__(name, bases, dct)
The problem is that for this to work the factory has to import all the subclass (So the meta class runs).
To fix this you can use from X import *
But for this to work you need to define an __all__ var in the __init__.py file of the package to include all the sub classes.
I don't want to hard code the sub classes because it beats the purpose of using the meta class.
I can go over the file in the package using:
import glob
from os.path import dirname, basename, isfile
modules = glob.glob(dirname(__file__) + "/*.py")
__all__ = [basename(f)[:-3] for f in modules if isfile(f)]
Which works great, but the project needs to compile to a single .so file, which nullifies the use of the file system.
So how could I achieve my main goal of creating instances at run time without hard codding the type?
Is there a way to populate an __all__ var at run time without touching the filesystem?
In Java I'd probably decorate the class with an annotation and then get all the classes with that annotation at run time, is there something similar on python?
I know there are decorators in python but I'm not sure I can use them in this way.
Edit 1:
Each subclass must be in a file:
- Models
-- __init__.py
-- ModelFactory.py
-- Regression
--- __init__.py
--- Base.py
--- Subclass1.py
--- Subclass2ExtendsSubclass1.py
Edit 2: Some code to Illustrate the problem:
+ main.py
|__ Models
|__ __init__.py
|__ ModelFactory.py
|__ Regression
|__ init__.py
|__ Base.py
|__ SubClass.py
|__ ModelRegister.py
main.py
from models.ModelFactory import ModelFactory
if __name__ == '__main__':
ModelFactory()
ModelFactory.py
from models.regression.Base import registry
import models.regression
class ModelFactory(object):
def get(self, some_type):
return registry[some_type]
ModelRegister.py
class ModelRegister(type):
# we use __init__ rather than __new__ here because we want
# to modify attributes of the class *after* they have been
# created
def __init__(cls, name, bases, dct):
print cls.__name__
if not hasattr(cls, 'registry'):
# this is the base class. Create an empty registry
cls.registry = {}
else:
# this is a derived class. Add cls to the registry
interface_id = cls().get_model_type()
cls.registry[interface_id] = cls
super(ModelRegister, cls).__init__(name, bases, dct)
Base.py
from models.regression.ModelRegister import ModelRegister
class Base(object):
__metaclass__ = ModelRegister
def get_type(self):
return "BASE"
SubClass.py
from models.regression.Base import Base
class SubClass(Base):
def get_type(self):
return "SUB_CLASS"
Running it you can see only "Base" it printed.
Using a decorator gives the same results.
A simple way to register classes as runtime is to use decorators:
registry = {}
def register(cls):
registry[cls.__name__] = cls
return cls
#register
class Foo(object):
pass
#register
class Bar(object):
pass
This will work if all of your classes are defined in the same module, and if that module is imported at runtime. Your situation, however, complicates things. First, you want to define your classes in different modules. This means that we must be able to dynamically determine which modules exist within our package at runtime. This would be straightforward using Python's pkgutil module, however, you also state that you are using Nuitka to compile your package into an extension module. pkgutil doesn't work with such extension modules.
I cannot find any documented way of determining the modules contained within an Nuitka extension module from within Python. If one does exist, the decorator approach above would work after dynamically importing each submodule.
As it is, I believe the most straightforward solution is to write a script to generate an __init__.py before compiling. Suppose we have the following package structure:
.
├── __init__.py
├── plugins
│ ├── alpha.py
│ └── beta.py
└── register.py
The "plugins" are contained within the plugins directory. The contents of the files are:
# register.py
# -----------
registry = {}
def register(cls):
registry[cls.__name__] = cls
return cls
# __init__.py
# -----------
from . import plugins
from . import register
# ./plugins/alpha.py
# ------------------
from ..register import register
#register
class Alpha(object):
pass
# ./plugins/beta.py
# ------------------
from ..register import register
#register
class Beta(object):
pass
As it stands, importing the package above will not result in any of the classes being registered. This is because the class definitions are never run, since the modules containing them are never imported. The remedy is to automatically generate an __init__.py for the plugins folder. Below is a script which does exactly this -- this script can be made part of your compilation process.
import pathlib
root = pathlib.Path('./mypkg/plugins')
exclude = {'__init__.py'}
def gen_modules(root):
for entry in root.iterdir():
if entry.suffix == '.py' and entry.name not in exclude:
yield entry.stem
with (root / '__init__.py').open('w') as fh:
for module in gen_modules(root):
fh.write('from . import %s\n' % module)
Placing this script one directory above your package root (assuming your package is called mypkg) and running it yields:
from . import alpha
from . import beta
Now for the test: we compile the package:
nuitka --module mypkg --recurse-to=mypkg
and try importing it, checking to see if all of the classes were properly registered:
>>> import mypkg
>>> mypkg.register.registry
{'Beta': <class 'mypkg.plugins.beta.Beta'>,
'Alpha': <class 'mypkg.plugins.alpha.Alpha'>}
Note that the same approach will work with using metaclasses to register the plugin classes, I simply preferred to use decorators here.
If the reflected classes are using your metaclass, you don't need to use from X import * to get them registered. Only import X should be enough. As soon as the module containing the classes is imported, the classes will be created and available in your metaclass registry.
I would do this with dynamic imports.
models/regression/base.py:
class Base(object):
def get_type(self):
return "BASE"
models/regression/subclass.py:
from models.regression.base import Base
class SubClass(Base):
def get_type(self):
return "SUB_CLASS"
__myclass__ = SubClass
loader.py:
from importlib import import_module
class_name = "subclass"
module = import_module("models.regression.%s" % class_name)
model = module.__myclass__()
print(model.get_type())
And empty __init__.py files in models/ and models/regression/
With:
nuitka --recurse-none --recurse-directory models --module loader.py
The resulting loader.so contains all the modules under the models/ subdirectory.
Related
In order to get a code easier to maintain, I wanted to split classes in different files, classified per role.
Everyting is placed into the same directory, with a init.py file to obtain a submodule.
Something like that :
Resources
|- __init__.py
|- main_code.py
|- resources.py
|--- classes
|- __init__.py
|- solutions.py
|- tests.py
But as in some classes of the tests.py, I inherit from other classes located in solutions.py, I've imported the solutions.py using :
from . import solutions
Here is an example of my code in tests.py :
from . import solutions
class snapshot(solutions.device):
def __init__(self, d):
solutions.device.__init__(self, d)
self.ip = d
But doing that, I've got the following error:
AttributeError: module 'solutions' has no attribute 'device'
I've also tried with :
from resources.classes import solutions
But I've got the same result.
Thanks for your help,
EDIT
Here is the solutions.py :
class device:
def __init__(self, d: str, **kwargs):
self.info = d
username = kwargs.get("username", None)
password = kwargs.get("password", None)
action = kwargs.get("action", None)
vault = kwargs.get("vault", None)
self.init_connection(username, password, vault)
<--- ommitted for visibility --->
When everything was located into the same classes.py file, it worked perfectly.
Maybe you can do
from .solutions import device
in tests.py
In a long-running app I need to dynamically modify static class members based on path to the class' module and the class name.
Ex. I have a class pack1.mod1.Person and by definition I know it has a age property. So utilizing the importlib and inspect I try to load the class using the module path and class name and update the age property. It all seems fine until I read the the age property from my naturally imported Person class and find it's not updated.
Here are some more details:
.
├── app.py
└── pack1
├── __init__.py
└── mod1.py
mod1.py
class Person:
age = 42
app.py
import inspect
import os
from importlib import util
from pack1.mod1 import Person
if __name__ == '__main__':
Person.age = 3
print(Person.age) # => 3
spec = util.spec_from_file_location('pack1.mod1', os.path.join('pack1', 'mod1.py'))
module = util.module_from_spec(spec)
spec.loader.exec_module(module)
members = inspect.getmembers(module)
for x, member in inspect.getmembers(module, lambda i: inspect.isclass(i) and i.__name__ == Person.__name__):
print('Person:', Person.age) # => Person: 3
print('Person from inspect:', member.age) # => Person from inspect: 42
Person.age = 11
member.age = 66
print('Person:', Person.age) # => Person: 11
print('Person from inspect:', member.age) # => Person from inspect: 66
In the app.py I would expect member and Person to be the same thing but as the example shows they aren't.
What am I missing and how to achieve such an update on the static members of a class?
Python has no way of knowing that the regularly imported module and the manually module are "the same": Using util.spec_from_file_location up to spec.loader.exec_module side-steps Python's module registry and explicitly creates a new instance of the module.
Instead, use the native operations of the interpreter (import, ...) or their programmatic equivalents (importlib.load_module, ...)
If the module/class are well-known, one can import it regularly and directly inspect it.
import pack1.mod1
pack1.mod1.Person.age = 66
If module and class are only known by name, one can look them up from the existing modules.
import importlib
module_name, qualname, attribute, value = 'pack1.mod1', 'Person', 'age', 66
obj = importlib.import_module(module_name) # same as `import {module_name}`
for part in qualname.split('.'):
obj = getattr(obj, part) # same as `{obj}.{part}
setattr(obj, attribute, value) # same as `{obj}.{attribute} = {value}`
I'm trying to use sphinx and autodoc for a large set of python modules. How can it document a class from a module, that has been imported and insatiated in another module:
# module1.py
class Class1():
def method1():
pass
# module2.py
import module1
class Class2():
class1 = module1.Class1()
I want the class1 instance in Class2() to show up in the docs, and refer back to the module1 document.
I have a module that should have a #property, I solved this by setting a class as the module. I got the idea from this answer: Lazy module variables--can it be done?
I wanted this to be repeatable and easy to use so I made a metaclass for it. This works like a charm.
The problem is that when using Sphinx to generate documentation properties don't get documented. Everything else is documented as expected. I have no idea how to fix this, maybe this is a problem with Sphinx?
The module:
import sys
import types
class ClassAsModule(type):
def __new__(cls, name, bases, attrs):
# Make sure the name of the class is the module name.
name = attrs.pop('__module__')
# Create a class.
cls = type.__new__(cls, name, bases, attrs)
# Instantiate the class and register it.
sys.modules[name] = cls = cls(name)
# Update the dict so dir works properly
cls.__dict__.update(attrs)
class TestClass(types.ModuleType):
"""TestClass docstring."""
__metaclass__ = ClassAsModule
#property
def some_property(self):
"""Property docstring."""
pass
def meth():
"""meth doc"""
pass
And a copy-paste to generate/view Sphinx documentation:
sphinx-apidoc . -o doc --full
sphinx-build doc html
xdg-open html/module.html
The most essential part is to document the class' properties. Bonus points to also document original module members.
EDIT: The class should be documented as the module it is in. The class is used this way and should thus appear this way in Sphinx.
Example of desired output:
Module Foo
TestClass docstring.
some_property
Property docstring.
meth()
meth doc
EDIT 2: I found something that may aid in finding a solution. When having a regular module foo with the following content:
#: Property of foo
prop = 'test'
Sphinx documents this like:
foo.prop = 'test'
Property of foo
The same works if prop is an attribute of a class. I haven't figured out why it doesn't work in my special case.
Here's my understanding.
The theory is: making a mutant your class act like a module this (a bit hacky) way makes sphinx think that he doesn't need (to parse) properties from modules (because it's a class-level paradigm). So, for sphinx, TestClass is a module.
First of all, to make sure that the culprit is the code for making a class act like a module - let's remove it:
class ClassAsModule(type):
pass
we'll see in docs:
package Package
script Module
class package.script.ClassAsModule
Bases: type
class package.script.TestClass
Bases: module
TestClass docstring.
meth()
meth doc
some_property
Property docstring.
As you see, sphinx read the property without any problems. Nothing special here.
Possible solution for your problem is to avoid using #property decorator and replace it with calling property class constructor. E.g.:
import sys
import types
class ClassAsModule(type):
def __new__(cls, name, bases, attrs):
# Make sure the name of the class is the module name.
name = attrs.pop('__module__')
# Create a class.
cls = type.__new__(cls, name, bases, attrs)
# Instantiate the class and register it.
sys.modules[name] = cls = cls(name)
# Update the dict so dir works properly
cls.__dict__.update(attrs)
class TestClass(types.ModuleType):
"""TestClass docstring."""
__metaclass__ = ClassAsModule
def get_some_property(self):
"""Property docstring."""
pass
some_property = property(get_some_property)
def meth(self):
"""meth doc"""
pass
For this code sphinx generates:
package Package
script Module
TestClass docstring.
package.script.get_some_property(self)
Property docstring.
package.script.meth(self)
meth doc
May be the answer is a piece of nonsense, but I hope it'll point you to the right direction.
The way I've found that works best is to keep the file contents the same as if you were writing a regular module, then at the end replace the embryonic module in sys.modules:
"""Module docstring. """
import sys
import types
def _some_property(self):
pass
some_property = property(_some_property)
"""Property docstring."""
def meth():
"""meth doc"""
pass
def _make_class_module(name):
mod = sys.modules[name]
cls = type('ClassModule', (types.ModuleType,), mod.__dict__)
clsmod = cls(name)
clsmod.__dict__.update(mod.__dict__)
clsmod.__wrapped__ = mod
sys.modules[name] = clsmod
_make_class_module(__name__)
Text documentation:
mymod Module
************
Module docstring.
mymod.meth()
meth doc
mymod.some_property = None
Property docstring.
For the version of Sphinx I'm using (v1.1.3), it looks like you have to apply the property constructor explicitly (you can't use it as a decorator), and the docstring has to go in the file at the top level, on the line after the constructor call that creates the property (it doesn't work as a docstring inside the property getter). The source is still fairly readable, though.
I want convert any string to my existing Entiy. Is it possible writing a convertToEntity() function as below?
class Personel(db.Model):
name=db.StringProperty()
class IsEntityExists(webapp.RequestHandler):
def get(self):
entity="Personal"
Entity=entity.convertToEntity()
Entity.all()
I wonder if the question is just asking to somehow look up the model class given its name, when it has already been imported. You can do this easily (but only when it has already been imported!), as follows:
cls = db.class_for_kind("Personel")
... cls.all() ...
The equivalent in NDB:
cls = ndb.Model._kind_map["Personel"]
... cls.query() ...
Good luck!
PS. No, it won't do spell correction. :-)
Only if you build loader for models... for example:
from app import model_loader
class IsEntityExists(webapp.RequestHandler):
def get(self):
Entity=model_loader("Personal")
Entity.all()
while the model_loader function would search the folder structure (python modules) for defined model.. for example you have folder structure:
models/
personal.py
other_model.py
user.py
So the model_loader("Personal") would import personal.py and extract "Personal" class from that module, allowing you to perform whatever you want with that class - if it finds it and loads it.
Of course you would have to code the loader.
However if the class (defined model) is in the same file as the code, you could search over locals() for "Personal"
def load_model(name):
local = locals()
try:
return local[name]
except KeyError:
return None