Develop Reference

How to avoid circular imports in class hierarchy located in several files?

I want to create instances of several classes at runtime, based on configuration xml files which contain the specific class type and other stuff. Let's start with a simple example:
class ParentClass:
#staticmethod
def create_subclass(class_type: str, xml):
if class_type == 'childA':
return ChildClassA(xml)
elif class_type == 'childB':
return ChildClassB(xml)
else:
return ParentClass(xml)
def __init__(self, xml):
print('\nParent stuff done.')
class ChildClassA(ParentClass):
def __init__(self, xml):
super().__init__(xml)
print('Class A stuff done.')
class ChildClassB(ParentClass):
def __init__(self, xml):
super().__init__(xml)
print('Class B stuff done.')
ParentClass.create_subclass('childA', None)
ParentClass.create_subclass('childB', None)
ParentClass.create_subclass('unknown', None)
The output is as expected:
Parent stuff done.
Class A stuff done.
Parent stuff done.
Class B stuff done.
Parent stuff done.
As the hierarchy grows larger I want to have the parent class in a separate module and each child class in a separate module. Obviously, I need to import the parent module to the child class modules. Due to the create subclass method, the parent class module needs to know about the child classes. Importing those to the parent module would cause a circular import which is considered bad design.
What would be a good solution to solve this issue? Are there better ways to create the subclass instances?

You can use some form of Registry:
from typing import Dict
class Registry:
_instance: 'Registry' = None
def __init__(self):
self._classes: Dict[str, type] = {}
self._default: type = object
#classmethod
def get_instance(cls) -> 'Registry':
if cls._instance is None:
cls._instance = Registry()
return cls._instance
def register(self, name: str, cls: type) -> None:
self._classes[name] = cls
def set_default(self, cls: type) -> None:
self._default = cls
def access(self, name: str) -> type:
cls = self._classes.get(name)
if cls is None:
return self._default
return cls
then you can set the parent class as its default class.
from .registry import Registry
class ParentClass:
def __init__(self, xml):
print('\nParent stuff done.')
registry = Registry.get_instance()
registry.set_default(ParentClass)
and submit child classes to your registry after defining them.
from .registry import Registry
class ChildClassA(ParentClass):
def __init__(self, xml):
super().__init__(xml)
print('Class A stuff done.')
registry = Registry.get_instance()
registry.register('childA', ChildClassA)
from .registry import Registry
class ChildClassB(ParentClass):
def __init__(self, xml):
super().__init__(xml)
print('Class B stuff done.')
registry = Registry.get_instance()
registry.register('childB', ChildClassB)
Cause your registry doesn't depend on your classes, you can use it without worrying about circulaNoticerts. notice that the access method returns a type, and you must call the constructor yourself.
from .registry import Registry
registry = Registry.get_instance()
registry.access('childA')(xml=None)
# Parent stuff done.
# Class A stuff done.
registry.access('childB')(xml=None)
# Parent stuff done.
# Class B stuff done.
registry.access('unknown')(xml=None)
# Parent stuff done.
You can also access subclasses and their names dynamically (as answered here) but since you didn't use subclass names as your key when creating subclasses, you probably need some mapping between names and subclasses. If you want your subclasses to tell your parent class the name they wish to be called by, you are implementing Registry in your parent class again.
accessing subclass names:
print([cls.__name__ for cls in ParentClass.__subclasses__()])
# ['ChildClassA', 'ChildClassB']
accessing subclasses themselves:
print(ParentClass.__subclasses__())
# [<class '__main__.ChildClassA'>, <class '__main__.ChildClassB'>]

How to create an abstract class attribute (potentially read-only)

I have spent a lot of time researching this, but none of the answers seem to work how I would like.
I have an abstract class with a class attribute I want each subclass to be forced to implement
class AbstractFoo():
forceThis = 0
So that when I do this
class RealFoo(AbstractFoo):
pass
it throws an error telling me it can't create the class until I implement forceThis.
How can I do that?
(I don't want the attribute to be read-only, but if that's the only solution, I'll accept it.)
For a class method, I've discovered I can do
from abc import ABCMeta, abstractmethod
class AbstractFoo(metaclass=ABCMeta):
#classmethod
#abstractmethod
def forceThis():
"""This must be implemented"""
so that
class RealFoo(AbstractFoo):
pass
at least throws the error TypeError: Can't instantiate abstract class EZ with abstract methods forceThis
(Although it doesn't force forceThis to be a class method.)
How can I get a similar error to pop up for the class attribute?

You can do this by defining your own metaclass. Something like:
class ForceMeta(type):
required = ['foo', 'bar']
def __new__(mcls, name, bases, namespace):
cls = super().__new__(mcls, name, bases, namespace)
for prop in mcls.required:
if not hasattr(cls, prop):
raise NotImplementedError('must define {}'.format(prop))
return cls
Now you can use this as the metaclass of your own classes:
class RequiredClass(metaclass=ForceMeta):
foo = 1
which will raise the error 'must define bar'.

I came up with a solution based on those posted earlier. (Thank you #Daniel Roseman and #martineau)
I created a metaclass called ABCAMeta (the last 'A' stands for 'Attributes').
The class has two ways of working.
A class which just uses ABCAMeta as a metaclass must have a property called required_attributes which should contain a list of the names of all the attributes you want to require on future subclasses of that class
A class whose parent's metaclass is ABCAMeta must have all the required attributes specified by its parent class(es).
For example:
class AbstractFoo(metaclass=ABCAMeta):
required_attributes = ['force_this']
class RealFoo(AbstractFoo):
pass
will throw an error:
NameError: Class 'RealFoo' has not implemented the following attributes: 'force_this'
Exactly how I wanted.
from abc import ABCMeta
class NoRequirements(RuntimeError):
def __init__(self, message):
RuntimeError.__init__(self, message)
class ABCAMeta(ABCMeta):
def __init__(mcls, name, bases, namespace):
ABCMeta.__init__(mcls, name, bases, namespace)
def __new__(mcls, name, bases, namespace):
def get_requirements(c):
"""c is a class that should have a 'required_attributes' attribute
this function will get that list of required attributes or
raise a NoRequirements error if it doesn't find one.
"""
if hasattr(c, 'required_attributes'):
return c.required_attributes
else:
raise NoRequirements(f"Class '{c.__name__}' has no 'required_attributes' property")
cls = super().__new__(mcls, name, bases, namespace)
# true if no parents of the class being created have ABCAMeta as their metaclass
basic_metaclass = True
# list of attributes the class being created must implement
# should stay empty if basic_metaclass stays True
reqs = []
for parent in bases:
parent_meta = type(parent)
if parent_meta==ABCAMeta:
# the class being created has a parent whose metaclass is ABCAMeta
# the class being created must contain the requirements of the parent class
basic_metaclass=False
try:
reqs.extend(get_requirements(parent))
except NoRequirements:
raise
# will force subclasses of the created class to define
# the attributes listed in the required_attributes attribute of the created class
if basic_metaclass:
get_requirements(cls) # just want it to raise an error if it doesn't have the attributes
else:
missingreqs = []
for req in reqs:
if not hasattr(cls, req):
missingreqs.append(req)
if len(missingreqs)!=0:
raise NameError(f"Class '{cls.__name__}' has not implemented the following attributes: {str(missingreqs)[1:-1]}")
return cls
Any suggestions for improvement are welcome in the comments.

Although you can do something very similar with a metaclass, as illustrated in #Daniel Roseman's answer, it can also be done with a class decorator. A couple of advantages they have are that errors will occur when the class is defined, instead of when an instance of one is created, and the syntax for specifying them is the same in both Python 2 and 3. Some folks also find them simpler and easier to understand.
def check_reqs(cls):
requirements = 'must_have',
missing = [req for req in requirements if not hasattr(cls, req)]
if missing:
raise NotImplementedError(
'class {} did not define required attribute{} named {}'.format(
cls.__name__, 's' if len(missing) > 1 else '',
', '.join('"{}"'.format(name) for name in missing)))
return cls
#check_reqs
class Foo(object): # OK
must_have = 42
#check_reqs
class Bar(object): # raises a NotImplementedError
pass

How to incorporate type checking in an abstract base class in Python

When I define a class, I like to include type checking (using assert) of the input variables. I am now defining a 'specialized' class Rule which inherits from an abstract base class (ABC) BaseRule, similar to the following:
import abc
class BaseRule(object):
__metaclass__ = abc.ABCMeta
#abc.abstractproperty
def resources(self):
pass
class Rule(BaseRule):
def __init__(self, resources):
assert all(isinstance(resource, Resource) for resource in resources) # type checking
self._resources = resources
#property
def resources(self):
return self._resources
class Resource(object):
def __init__(self, domain):
self.domain = domain
if __name__ == "__main__":
resources = [Resource("facebook.com")]
rule = Rule(resources)
The assert statement in the __init__ function of the Rule class ensures that the resources input is a list (or other iterable) of Resource objects. However, this would also be the case for other classes which inherit from BaseRule, so I would like to incorporate this assertion in the abstractproperty somehow. How might I go about this?

See this documentation on abc Type annotations with mypy-lang https://mypy.readthedocs.io/en/latest/class_basics.html#abstract-base-classes-and-multiple-inheritance

Make your base class have a non-abstract property that calls separate abstract getter and setter methods. The property can do the validation you want before calling the setter. Other code (such as the __init__ method of a derived class) that wants to trigger the validation can do so by doing its assignment via the property:
class BaseRule(object):
__metaclass__ = abc.ABCMeta
#property
def resources(self): # this property isn't abstract and shouldn't be overridden
return self._get_resources()
#resources.setter
def resources(self, value):
assert all(isinstance(resource, Resources) for resource in value)
self._set_resources(value)
#abstractmethod
def _get_resources(self): # these methods should be, instead
pass
#abstractmethod
def _set_resources(self, value):
pass
class Rule(BaseRule):
def __init__(self, resources):
self.resources = resources # assign via the property to get type-checking!
def _get_resources(self):
return self._resources
def _set_resources(self, value):
self._resources = value
You might even consider moving the __init__ method from Rule into the BaseRule class, since it doesn't need any knowledge about Rule's concrete implementation.

How can one locate where an inherited variable comes from in Python?

If you have multiple layers of inheritance and know that a particular variable exists, is there a way to trace back to where the variable originated? Without having to navigate backwards by looking through each file and classes. Possibly calling some sort of function that will do it?
Example:
parent.py
class parent(object):
def __init__(self):
findMe = "Here I am!"
child.py
from parent import parent
class child(parent):
pass
grandson.py
from child import child
class grandson(child):
def printVar(self):
print self.findMe
Try to locate where the findMe variable came from with a function call.

If the "variable" is an instance variable - , so , if at any point in chain of __init__ methods you do:
def __init__(self):
self.findMe = "Here I am!"
It is an instance variable from that point on, and cannot, for all effects, be made distinct of any other instance variable. (Unless you put in place a mechanism, like a class with a special __setattr__ method, that will keep track of attributes changing, and introspect back which part of the code set the attribute - see last example on this answer)
Please also note that on your example,
class parent(object):
def __init__(self):
findMe = "Here I am!"
findMe is defined as a local variable to that method and does not even exist after __init__ is finished.
Now, if your variable is set as a class attribute somewhere on the inheritance chain:
class parent(object):
findMe = False
class childone(parent):
...
It is possible to find the class where findMe is defined by introspecting each class' __dict__ in the MRO (method resolution order) chain . Of course, there is no way, and no sense, in doing that without introspecting all classes in the MRO chain - except if one keeps track of attributes as defined, like in the example bellow this - but introspecting the MRO itself is a oneliner in Python:
def __init__(self):
super().__init__()
...
findme_definer = [cls for cls in self.__class__.__mro__ if "findMe" in cls.__dict__][0]
Again - it would be possible to have a metaclass to your inheritance chain which would keep track of all defined attributes in the inheritance tree, and use a dictionary to retrieve where each attribute is defined. The same metaclass could also auto-decorate all __init__ (or all methods), and set a special __setitem__ so that it could track instance attributes as they are created, as listed above.
That can be done, is a bit complicated, would be hard to maintain, and probably is a signal you are taking the wrong approach to your problem.
So, the metaclass to record just class attributes could simply be (python3 syntax - define a __metaclass__ attribute on the class body if you are still using Python 2.7):
class MetaBase(type):
definitions = {}
def __init__(cls, name, bases, dct):
for attr in dct.keys():
cls.__class__.definitions[attr] = cls
class parent(metaclass=MetaBase):
findMe = 5
def __init__(self):
print(self.__class__.definitions["findMe"])
Now, if one wants to find which of the superclasses defined an attribute of the currentclass, just a "live" tracking mechanism, wrapping each method in each class can work - it is a lot trickier.
I've made it - even if you won't need this much, this combines both methods - keeping track of class attributes in the class'class definitions and on an instance _definitions dictionary - since in each created instance an arbitrary method might have been the last to set a particular instance attribute: (This is pure Python3, and maybe not that straighforward porting to Python2 due to the "unbound method" that Python2 uses, and is a simple function in Python3)
from threading import current_thread
from functools import wraps
from types import MethodType
from collections import defaultdict
def method_decorator(func, cls):
#wraps(func)
def wrapper(self, *args, **kw):
self.__class__.__class__.current_running_class[current_thread()].append(cls)
result = MethodType(func, self)(*args, **kw)
self.__class__.__class__.current_running_class[current_thread()].pop()
return result
return wrapper
class MetaBase(type):
definitions = {}
current_running_class = defaultdict(list)
def __init__(cls, name, bases, dct):
for attrname, attr in dct.items():
cls.__class__.definitions[attr] = cls
if callable(attr) and attrname != "__setattr__":
setattr(cls, attrname, method_decorator(attr, cls))
class Base(object, metaclass=MetaBase):
def __setattr__(self, attr, value):
if not hasattr(self, "_definitions"):
super().__setattr__("_definitions", {})
self._definitions[attr] = self.__class__.current_running_class[current_thread()][-1]
return super().__setattr__(attr,value)
Example Classes for the code above:
class Parent(Base):
def __init__(self):
super().__init__()
self.findMe = 10
class Child1(Parent):
def __init__(self):
super().__init__()
self.findMe1 = 20
class Child2(Parent):
def __init__(self):
super().__init__()
self.findMe2 = 30
class GrandChild(Child1, Child2):
def __init__(self):
super().__init__()
def findall(self):
for attr in "findMe findMe1 findMe2".split():
print("Attr '{}' defined in class '{}' ".format(attr, self._definitions[attr].__name__))
And on the console one will get this result:
In [87]: g = GrandChild()
In [88]: g.findall()
Attr 'findMe' defined in class 'Parent'
Attr 'findMe1' defined in class 'Child1'
Attr 'findMe2' defined in class 'Child2'

Python metaclasses: Why isn't __setattr__ called for attributes set during class definition?

I have the following python code:
class FooMeta(type):
def __setattr__(self, name, value):
print name, value
return super(FooMeta, self).__setattr__(name, value)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
I would have expected __setattr__ of the meta class being called for both FOO and a. However, it is not called at all. When I assign something to Foo.whatever after the class has been defined the method is called.
What's the reason for this behaviour and is there a way to intercept the assignments that happen during the creation of the class? Using attrs in __new__ won't work since I'd like to check if a method is being redefined.

A class block is roughly syntactic sugar for building a dictionary, and then invoking a metaclass to build the class object.
This:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Comes out pretty much as if you'd written:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Only without the namespace pollution (and in reality there's also a search through all the bases to determine the metaclass, or whether there's a metaclass conflict, but I'm ignoring that here).
The metaclass' __setattr__ can control what happens when you try to set an attribute on one of its instances (the class object), but inside the class block you're not doing that, you're inserting into a dictionary object, so the dict class controls what's going on, not your metaclass. So you're out of luck.
Unless you're using Python 3.x! In Python 3.x you can define a __prepare__ classmethod (or staticmethod) on a metaclass, which controls what object is used to accumulate attributes set within a class block before they're passed to the metaclass constructor. The default __prepare__ simply returns a normal dictionary, but you could build a custom dict-like class that doesn't allow keys to be redefined, and use that to accumulate your attributes:
from collections import MutableMapping
class SingleAssignDict(MutableMapping):
def __init__(self, *args, **kwargs):
self._d = dict(*args, **kwargs)
def __getitem__(self, key):
return self._d[key]
def __setitem__(self, key, value):
if key in self._d:
raise ValueError(
'Key {!r} already exists in SingleAssignDict'.format(key)
)
else:
self._d[key] = value
def __delitem__(self, key):
del self._d[key]
def __iter__(self):
return iter(self._d)
def __len__(self):
return len(self._d)
def __contains__(self, key):
return key in self._d
def __repr__(self):
return '{}({!r})'.format(type(self).__name__, self._d)
class RedefBlocker(type):
#classmethod
def __prepare__(metacls, name, bases, **kwargs):
return SingleAssignDict()
def __new__(metacls, name, bases, sad):
return super().__new__(metacls, name, bases, dict(sad))
class Okay(metaclass=RedefBlocker):
a = 1
b = 2
class Boom(metaclass=RedefBlocker):
a = 1
b = 2
a = 3
Running this gives me:
Traceback (most recent call last):
File "/tmp/redef.py", line 50, in <module>
class Boom(metaclass=RedefBlocker):
File "/tmp/redef.py", line 53, in Boom
a = 3
File "/tmp/redef.py", line 15, in __setitem__
'Key {!r} already exists in SingleAssignDict'.format(key)
ValueError: Key 'a' already exists in SingleAssignDict
Some notes:
__prepare__ has to be a classmethod or staticmethod, because it's being called before the metaclass' instance (your class) exists.
type still needs its third parameter to be a real dict, so you have to have a __new__ method that converts the SingleAssignDict to a normal one
I could have subclassed dict, which would probably have avoided (2), but I really dislike doing that because of how the non-basic methods like update don't respect your overrides of the basic methods like __setitem__. So I prefer to subclass collections.MutableMapping and wrap a dictionary.
The actual Okay.__dict__ object is a normal dictionary, because it was set by type and type is finicky about the kind of dictionary it wants. This means that overwriting class attributes after class creation does not raise an exception. You can overwrite the __dict__ attribute after the superclass call in __new__ if you want to maintain the no-overwriting forced by the class object's dictionary.
Sadly this technique is unavailable in Python 2.x (I checked). The __prepare__ method isn't invoked, which makes sense as in Python 2.x the metaclass is determined by the __metaclass__ magic attribute rather than a special keyword in the classblock; which means the dict object used to accumulate attributes for the class block already exists by the time the metaclass is known.
Compare Python 2:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Where the metaclass to invoke is determined from the dictionary, versus Python 3:
class Foo(metaclass=FooMeta):
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = FooMeta.__prepare__('Foo', ())
d['Foo'] = 123
def a(self):
pass
d['a'] = a
Foo = FooMeta('Foo', (), d)
Where the dictionary to use is determined from the metaclass.

There are no assignments happening during the creation of the class. Or: they are happening, but not in the context you think they are. All class attributes are collected from class body scope and passed to metaclass' __new__, as the last argument:
class FooMeta(type):
def __new__(self, name, bases, attrs):
print attrs
return type.__new__(self, name, bases, attrs)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
Reason: when the code in the class body executes, there's no class yet. Which means there's no opportunity for metaclass to intercept anything yet.

Class attributes are passed to the metaclass as a single dictionary and my hypothesis is that this is used to update the __dict__ attribute of the class all at once, e.g. something like cls.__dict__.update(dct) rather than doing setattr() on each item. More to the point, it's all handled in C-land and simply wasn't written to call a custom __setattr__().
It's easy enough to do whatever you want to the attributes of the class in your metaclass's __init__() method, since you're passed the class namespace as a dict, so just do that.

During the class creation, your namespace is evaluated to a dict and passed as an argument to the metaclass, together with the class name and base classes. Because of that, assigning a class attribute inside the class definition wouldn't work the way you expect. It doesn't create an empty class and assign everything. You also can't have duplicated keys in a dict, so during class creation attributes are already deduplicated. Only by setting an attribute after the class definition you can trigger your custom __setattr__.
Because the namespace is a dict, there's no way for you to check duplicated methods, as suggested by your other question. The only practical way to do that is parsing the source code.