Develop Reference

Python generate a list of classes from folders and subfolders only of a certain type [duplicate]

I need a working approach of getting all classes that are inherited from a base class in Python.

New-style classes (i.e. subclassed from object, which is the default in Python 3) have a __subclasses__ method which returns the subclasses:
class Foo(object): pass
class Bar(Foo): pass
class Baz(Foo): pass
class Bing(Bar): pass
Here are the names of the subclasses:
print([cls.__name__ for cls in Foo.__subclasses__()])
# ['Bar', 'Baz']
Here are the subclasses themselves:
print(Foo.__subclasses__())
# [<class '__main__.Bar'>, <class '__main__.Baz'>]
Confirmation that the subclasses do indeed list Foo as their base:
for cls in Foo.__subclasses__():
print(cls.__base__)
# <class '__main__.Foo'>
# <class '__main__.Foo'>
Note if you want subsubclasses, you'll have to recurse:
def all_subclasses(cls):
return set(cls.__subclasses__()).union(
[s for c in cls.__subclasses__() for s in all_subclasses(c)])
print(all_subclasses(Foo))
# {<class '__main__.Bar'>, <class '__main__.Baz'>, <class '__main__.Bing'>}
Note that if the class definition of a subclass hasn't been executed yet - for example, if the subclass's module hasn't been imported yet - then that subclass doesn't exist yet, and __subclasses__ won't find it.
You mentioned "given its name". Since Python classes are first-class objects, you don't need to use a string with the class's name in place of the class or anything like that. You can just use the class directly, and you probably should.
If you do have a string representing the name of a class and you want to find that class's subclasses, then there are two steps: find the class given its name, and then find the subclasses with __subclasses__ as above.
How to find the class from the name depends on where you're expecting to find it. If you're expecting to find it in the same module as the code that's trying to locate the class, then
cls = globals()[name]
would do the job, or in the unlikely case that you're expecting to find it in locals,
cls = locals()[name]
If the class could be in any module, then your name string should contain the fully-qualified name - something like 'pkg.module.Foo' instead of just 'Foo'. Use importlib to load the class's module, then retrieve the corresponding attribute:
import importlib
modname, _, clsname = name.rpartition('.')
mod = importlib.import_module(modname)
cls = getattr(mod, clsname)
However you find the class, cls.__subclasses__() would then return a list of its subclasses.

If you just want direct subclasses then .__subclasses__() works fine. If you want all subclasses, subclasses of subclasses, and so on, you'll need a function to do that for you.
Here's a simple, readable function that recursively finds all subclasses of a given class:
def get_all_subclasses(cls):
all_subclasses = []
for subclass in cls.__subclasses__():
all_subclasses.append(subclass)
all_subclasses.extend(get_all_subclasses(subclass))
return all_subclasses

The simplest solution in general form:
def get_subclasses(cls):
for subclass in cls.__subclasses__():
yield from get_subclasses(subclass)
yield subclass
And a classmethod in case you have a single class where you inherit from:
#classmethod
def get_subclasses(cls):
for subclass in cls.__subclasses__():
yield from subclass.get_subclasses()
yield subclass

Python 3.6 - __init_subclass__
As other answer mentioned you can check the __subclasses__ attribute to get the list of subclasses, since python 3.6 you can modify this attribute creation by overriding the __init_subclass__ method.
class PluginBase:
subclasses = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.subclasses.append(cls)
class Plugin1(PluginBase):
pass
class Plugin2(PluginBase):
pass
This way, if you know what you're doing, you can override the behavior of of __subclasses__ and omit/add subclasses from this list.

Note: I see that someone (not #unutbu) changed the referenced answer so that it no longer uses vars()['Foo'] — so the primary point of my post no longer applies.
FWIW, here's what I meant about #unutbu's answer only working with locally defined classes — and that using eval() instead of vars() would make it work with any accessible class, not only those defined in the current scope.
For those who dislike using eval(), a way is also shown to avoid it.
First here's a concrete example demonstrating the potential problem with using vars():
class Foo(object): pass
class Bar(Foo): pass
class Baz(Foo): pass
class Bing(Bar): pass
# unutbu's approach
def all_subclasses(cls):
return cls.__subclasses__() + [g for s in cls.__subclasses__()
for g in all_subclasses(s)]
print(all_subclasses(vars()['Foo'])) # Fine because Foo is in scope
# -> [<class '__main__.Bar'>, <class '__main__.Baz'>, <class '__main__.Bing'>]
def func(): # won't work because Foo class is not locally defined
print(all_subclasses(vars()['Foo']))
try:
func() # not OK because Foo is not local to func()
except Exception as e:
print('calling func() raised exception: {!r}'.format(e))
# -> calling func() raised exception: KeyError('Foo',)
print(all_subclasses(eval('Foo'))) # OK
# -> [<class '__main__.Bar'>, <class '__main__.Baz'>, <class '__main__.Bing'>]
# using eval('xxx') instead of vars()['xxx']
def func2():
print(all_subclasses(eval('Foo')))
func2() # Works
# -> [<class '__main__.Bar'>, <class '__main__.Baz'>, <class '__main__.Bing'>]
This could be improved by moving the eval('ClassName') down into the function defined, which makes using it easier without loss of the additional generality gained by using eval() which unlike vars() is not context-sensitive:
# easier to use version
def all_subclasses2(classname):
direct_subclasses = eval(classname).__subclasses__()
return direct_subclasses + [g for s in direct_subclasses
for g in all_subclasses2(s.__name__)]
# pass 'xxx' instead of eval('xxx')
def func_ez():
print(all_subclasses2('Foo')) # simpler
func_ez()
# -> [<class '__main__.Bar'>, <class '__main__.Baz'>, <class '__main__.Bing'>]
Lastly, it's possible, and perhaps even important in some cases, to avoid using eval() for security reasons, so here's a version without it:
def get_all_subclasses(cls):
""" Generator of all a class's subclasses. """
try:
for subclass in cls.__subclasses__():
yield subclass
for subclass in get_all_subclasses(subclass):
yield subclass
except TypeError:
return
def all_subclasses3(classname):
for cls in get_all_subclasses(object): # object is base of all new-style classes.
if cls.__name__.split('.')[-1] == classname:
break
else:
raise ValueError('class %s not found' % classname)
direct_subclasses = cls.__subclasses__()
return direct_subclasses + [g for s in direct_subclasses
for g in all_subclasses3(s.__name__)]
# no eval('xxx')
def func3():
print(all_subclasses3('Foo'))
func3() # Also works
# -> [<class '__main__.Bar'>, <class '__main__.Baz'>, <class '__main__.Bing'>]

Here is a simple but efficient version of code:
def get_all_subclasses(cls):
subclass_list = []
def recurse(klass):
for subclass in klass.__subclasses__():
subclass_list.append(subclass)
recurse(subclass)
recurse(cls)
return set(subclass_list)
Its time complexity is O(n) where n is the number of all subclasses if there's no multiple inheritance.
It's more efficient than the functions that recursively create lists or yield classes with generators, whose complexity could be (1) O(nlogn) when the class hierarchy is a balanced tree or (2) O(n^2) when the class hierarchy is a biased tree.

A much shorter version for getting a list of all subclasses:
from itertools import chain
def subclasses(cls):
return list(
chain.from_iterable(
[list(chain.from_iterable([[x], subclasses(x)])) for x in cls.__subclasses__()]
)
)

Here's a version without recursion:
def get_subclasses_gen(cls):
def _subclasses(classes, seen):
while True:
subclasses = sum((x.__subclasses__() for x in classes), [])
yield from classes
yield from seen
found = []
if not subclasses:
return
classes = subclasses
seen = found
return _subclasses([cls], [])
This differs from other implementations in that it returns the original class.
This is because it makes the code simpler and:
class Ham(object):
pass
assert(issubclass(Ham, Ham)) # True
If get_subclasses_gen looks a bit weird that's because it was created by converting a tail-recursive implementation into a looping generator:
def get_subclasses(cls):
def _subclasses(classes, seen):
subclasses = sum(*(frozenset(x.__subclasses__()) for x in classes))
found = classes + seen
if not subclasses:
return found
return _subclasses(subclasses, found)
return _subclasses([cls], [])

How can I find all subclasses of a class given its name?
We can certainly easily do this given access to the object itself, yes.
Simply given its name is a poor idea, as there can be multiple classes of the same name, even defined in the same module.
I created an implementation for another answer, and since it answers this question and it's a little more elegant than the other solutions here, here it is:
def get_subclasses(cls):
"""returns all subclasses of argument, cls"""
if issubclass(cls, type):
subclasses = cls.__subclasses__(cls)
else:
subclasses = cls.__subclasses__()
for subclass in subclasses:
subclasses.extend(get_subclasses(subclass))
return subclasses
Usage:
>>> import pprint
>>> list_of_classes = get_subclasses(int)
>>> pprint.pprint(list_of_classes)
[<class 'bool'>,
<enum 'IntEnum'>,
<enum 'IntFlag'>,
<class 'sre_constants._NamedIntConstant'>,
<class 'subprocess.Handle'>,
<enum '_ParameterKind'>,
<enum 'Signals'>,
<enum 'Handlers'>,
<enum 'RegexFlag'>]

This isn't as good an answer as using the special built-in __subclasses__() class method which #unutbu mentions, so I present it merely as an exercise. The subclasses() function defined returns a dictionary which maps all the subclass names to the subclasses themselves.
def traced_subclass(baseclass):
class _SubclassTracer(type):
def __new__(cls, classname, bases, classdict):
obj = type(classname, bases, classdict)
if baseclass in bases: # sanity check
attrname = '_%s__derived' % baseclass.__name__
derived = getattr(baseclass, attrname, {})
derived.update( {classname:obj} )
setattr(baseclass, attrname, derived)
return obj
return _SubclassTracer
def subclasses(baseclass):
attrname = '_%s__derived' % baseclass.__name__
return getattr(baseclass, attrname, None)
class BaseClass(object):
pass
class SubclassA(BaseClass):
__metaclass__ = traced_subclass(BaseClass)
class SubclassB(BaseClass):
__metaclass__ = traced_subclass(BaseClass)
print subclasses(BaseClass)
Output:
{'SubclassB': <class '__main__.SubclassB'>,
'SubclassA': <class '__main__.SubclassA'>}

While I'm very partial to the __init_subclass__ approach, this will preserve definition order, and avoid combinatorial order of growth if you have a very dense hierarchy with multiple inheritance everywhere:
def descendents(cls):
'''Does not return the class itself'''
R = {}
def visit(cls):
for subCls in cls.__subclasses__():
R[subCls] = True
visit(subCls)
visit(cls)
return list(R.keys())
This works because dictionaries remember the insertion order of their keys. A list approach would also work.

Why does a metaclass not have access to the attributes inhereited from a subclass of a class defined by the metaclass?

Class Foo is defined with a metaclass Meta. The metaclass loops over the class attributes and prints them to screen.
Class Bar subclasses Foo. However, the metaclass does not print the inherited attributes from Bar.
Why doesn't the metaclass have access to Foo's attributes inherited in Bar? What am I not understanding about python's metaclass system?
Here is the sample code in 2.7:
class Meta(type):
def __init__(cls, name, bases, attrs):
print "bases = {}".format(bases)
items = {k:v for k,v in attrs.iteritems() if not k.startswith('__')}
for k,v in items.iteritems():
print k, v
class Foo(object):
__metaclass__ = Meta
hi = 1
# This prints:
# bases = (<type 'object'>,)
# hi 1
class Bar(Foo):
pass
# This prints:
# bases = (<class '__main__.Foo'>,)
Foo.hi
#prints 1
Bar.hi
#prints 1

The attrs parameter to __init__ only contains the attributes for that class, not for its bases.
A Bar object does not have an attribute hi. Instead, when you ask for Bar.hi the lookup will start at Bar, find out that it doesn't have hi, then look in base Foo to find it.

As #orlp says, attrs contains only the class dictionary for the class being created. You still have access to hi, however, because it's in the __dict__ attribute of one of Foo's bases. That is, you could do something similar to what you have, but recurse through the base classes and print out the entries in each base class dictionary.
Another approach is to use dir(), which should roughly return a list of all attributes a class has. I say roughly because a class can implement __getattr__ or __getattribute__ to return attributes "on the fly", meaning that the class may not have a well-defined set of attributes for dir() to return -- see the full disclaimer here. But in many common cases, something like the following will work:
class Meta(type):
def __init__(cls, name, bases, attrs):
print "bases = {}".format(bases)
for attr in dir(cls):
if not attr.startswith('_'):
print attr, getattr(cls, attr)
class Foo(object):
__metaclass__ = Meta
hi = 1
class Bar(Foo):
pass
Which prints:
bases = (<type 'object'>,)
hi 1
bases = (<class '__main__.Foo'>,)
hi 1

Abstract attribute (not property)?

What's the best practice to define an abstract instance attribute, but not as a property?
I would like to write something like:
class AbstractFoo(metaclass=ABCMeta):
#property
#abstractmethod
def bar(self):
pass
class Foo(AbstractFoo):
def __init__(self):
self.bar = 3
Instead of:
class Foo(AbstractFoo):
def __init__(self):
self._bar = 3
#property
def bar(self):
return self._bar
#bar.setter
def setbar(self, bar):
self._bar = bar
#bar.deleter
def delbar(self):
del self._bar
Properties are handy, but for simple attribute requiring no computation they are an overkill. This is especially important for abstract classes which will be subclassed and implemented by the user (I don't want to force someone to use #property when he just could have written self.foo = foo in the __init__).
Abstract attributes in Python question proposes as only answer to use #property and #abstractmethod: it doesn't answer my question.
The ActiveState recipe for an abstract class attribute via AbstractAttribute may be the right way, but I am not sure. It also only works with class attributes and not instance attributes.

A possibly a bit better solution compared to the accepted answer:
from better_abc import ABCMeta, abstract_attribute # see below
class AbstractFoo(metaclass=ABCMeta):
#abstract_attribute
def bar(self):
pass
class Foo(AbstractFoo):
def __init__(self):
self.bar = 3
class BadFoo(AbstractFoo):
def __init__(self):
pass
It will behave like this:
Foo() # ok
BadFoo() # will raise: NotImplementedError: Can't instantiate abstract class BadFoo
# with abstract attributes: bar
This answer uses same approach as the accepted answer, but integrates well with built-in ABC and does not require boilerplate of check_bar() helpers.
Here is the better_abc.py content:
from abc import ABCMeta as NativeABCMeta
class DummyAttribute:
pass
def abstract_attribute(obj=None):
if obj is None:
obj = DummyAttribute()
obj.__is_abstract_attribute__ = True
return obj
class ABCMeta(NativeABCMeta):
def __call__(cls, *args, **kwargs):
instance = NativeABCMeta.__call__(cls, *args, **kwargs)
abstract_attributes = {
name
for name in dir(instance)
if getattr(getattr(instance, name), '__is_abstract_attribute__', False)
}
if abstract_attributes:
raise NotImplementedError(
"Can't instantiate abstract class {} with"
" abstract attributes: {}".format(
cls.__name__,
', '.join(abstract_attributes)
)
)
return instance
The nice thing is that you can do:
class AbstractFoo(metaclass=ABCMeta):
bar = abstract_attribute()
and it will work same as above.
Also one can use:
class ABC(ABCMeta):
pass
to define custom ABC helper. PS. I consider this code to be CC0.
This could be improved by using AST parser to raise earlier (on class declaration) by scanning the __init__ code, but it seems to be an overkill for now (unless someone is willing to implement).
2021: typing support
You can use:
from typing import cast, Any, Callable, TypeVar
R = TypeVar('R')
def abstract_attribute(obj: Callable[[Any], R] = None) -> R:
_obj = cast(Any, obj)
if obj is None:
_obj = DummyAttribute()
_obj.__is_abstract_attribute__ = True
return cast(R, _obj)
which will let mypy highlight some typing issues
class AbstractFooTyped(metaclass=ABCMeta):
#abstract_attribute
def bar(self) -> int:
pass
class FooTyped(AbstractFooTyped):
def __init__(self):
# skipping assignment (which is required!) to demonstrate
# that it works independent of when the assignment is made
pass
f_typed = FooTyped()
_ = f_typed.bar + 'test' # Mypy: Unsupported operand types for + ("int" and "str")
FooTyped.bar = 'test' # Mypy: Incompatible types in assignment (expression has type "str", variable has type "int")
FooTyped.bar + 'test' # Mypy: Unsupported operand types for + ("int" and "str")
and for the shorthand notation, as suggested by #SMiller in the comments:
class AbstractFooTypedShorthand(metaclass=ABCMeta):
bar: int = abstract_attribute()
AbstractFooTypedShorthand.bar += 'test' # Mypy: Unsupported operand types for + ("int" and "str")

Just because you define it as an abstractproperty on the abstract base class doesn't mean you have to make a property on the subclass.
e.g. you can:
In [1]: from abc import ABCMeta, abstractproperty
In [2]: class X(metaclass=ABCMeta):
...: #abstractproperty
...: def required(self):
...: raise NotImplementedError
...:
In [3]: class Y(X):
...: required = True
...:
In [4]: Y()
Out[4]: <__main__.Y at 0x10ae0d390>
If you want to initialise the value in __init__ you can do this:
In [5]: class Z(X):
...: required = None
...: def __init__(self, value):
...: self.required = value
...:
In [6]: Z(value=3)
Out[6]: <__main__.Z at 0x10ae15a20>
Since Python 3.3 abstractproperty is deprecated. So Python 3 users should use the following instead:
from abc import ABCMeta, abstractmethod
class X(metaclass=ABCMeta):
#property
#abstractmethod
def required(self):
raise NotImplementedError

If you really want to enforce that a subclass define a given attribute, you can use metaclasses:
class AbstractFooMeta(type):
def __call__(cls, *args, **kwargs):
"""Called when you call Foo(*args, **kwargs) """
obj = type.__call__(cls, *args, **kwargs)
obj.check_bar()
return obj
class AbstractFoo(object):
__metaclass__ = AbstractFooMeta
bar = None
def check_bar(self):
if self.bar is None:
raise NotImplementedError('Subclasses must define bar')
class GoodFoo(AbstractFoo):
def __init__(self):
self.bar = 3
class BadFoo(AbstractFoo):
def __init__(self):
pass
Basically the meta class redefine __call__ to make sure check_bar is called after the init on an instance.
GoodFoo()  # ok
BadFoo ()  # yield NotImplementedError

As Anentropic said, you don't have to implement an abstractproperty as another property.
However, one thing all answers seem to neglect is Python's member slots (the __slots__ class attribute). Users of your ABCs required to implement abstract properties could simply define them within __slots__ if all that's needed is a data attribute.
So with something like,
class AbstractFoo(abc.ABC):
__slots__ = ()
bar = abc.abstractproperty()
Users can define sub-classes simply like,
class Foo(AbstractFoo):
__slots__ = 'bar', # the only requirement
# define Foo as desired
def __init__(self):
self.bar = ...
Here, Foo.bar behaves like a regular instance attribute, which it is, just implemented differently. This is simple, efficient, and avoids the #property boilerplate that you described.
This works whether or not ABCs define __slots__ at their class' bodies. However, going with __slots__ all the way not only saves memory and provides faster attribute accesses but also gives a meaningful descriptor instead of having intermediates (e.g. bar = None or similar) in sub-classes.1
A few answers suggest doing the "abstract" attribute check after instantiation (i.e. at the meta-class __call__() method) but I find that not only wasteful but also potentially inefficient as the initialization step could be a time-consuming one.
In short, what's required for sub-classes of ABCs is to override the relevant descriptor (be it a property or a method), it doesn't matter how, and documenting to your users that it's possible to use __slots__ as implementation for abstract properties seems to me as the more adequate approach.
1 In any case, at the very least, ABCs should always define an empty __slots__ class attribute because otherwise sub-classes are forced to have __dict__ (dynamic attribute access) and __weakref__ (weak reference support) when instantiated. See the abc or collections.abc modules for examples of this being the case within the standard library.

The problem isn't what, but when:
from abc import ABCMeta, abstractmethod
class AbstractFoo(metaclass=ABCMeta):
#abstractmethod
def bar():
pass
class Foo(AbstractFoo):
bar = object()
isinstance(Foo(), AbstractFoo)
#>>> True
It doesn't matter that bar isn't a method! The problem is that __subclasshook__, the method of doing the check, is a classmethod, so only cares whether the class, not the instance, has the attribute.
I suggest you just don't force this, as it's a hard problem. The alternative is forcing them to predefine the attribute, but that just leaves around dummy attributes that just silence errors.

I've searched around for this for awhile but didn't see anything I like. As you probably know if you do:
class AbstractFoo(object):
#property
def bar(self):
raise NotImplementedError(
"Subclasses of AbstractFoo must set an instance attribute "
"self._bar in it's __init__ method")
class Foo(AbstractFoo):
def __init__(self):
self.bar = "bar"
f = Foo()
You get an AttributeError: can't set attribute which is annoying.
To get around this you can do:
class AbstractFoo(object):
#property
def bar(self):
try:
return self._bar
except AttributeError:
raise NotImplementedError(
"Subclasses of AbstractFoo must set an instance attribute "
"self._bar in it's __init__ method")
class OkFoo(AbstractFoo):
def __init__(self):
self._bar = 3
class BadFoo(AbstractFoo):
pass
a = OkFoo()
b = BadFoo()
print a.bar
print b.bar # raises a NotImplementedError
This avoids the AttributeError: can't set attribute but if you just leave off the abstract property all together:
class AbstractFoo(object):
pass
class Foo(AbstractFoo):
pass
f = Foo()
f.bar
You get an AttributeError: 'Foo' object has no attribute 'bar' which is arguably almost as good as the NotImplementedError. So really my solution is just trading one error message from another .. and you have to use self._bar rather than self.bar in the init.

Following https://docs.python.org/2/library/abc.html you could do something like this in Python 2.7:
from abc import ABCMeta, abstractproperty
class Test(object):
__metaclass__ = ABCMeta
#abstractproperty
def test(self): yield None
def get_test(self):
return self.test
class TestChild(Test):
test = None
def __init__(self, var):
self.test = var
a = TestChild('test')
print(a.get_test())

How to keep track of class instances?

Toward the end of a program I'm looking to load a specific variable from all the instances of a class into a dictionary.
For example:
class Foo():
def __init__(self):
self.x = {}
foo1 = Foo()
foo2 = Foo()
...
Let's say the number of instances will vary and I want the x dict from each instance of Foo() loaded into a new dict. How would I do that?
The examples I've seen in SO assume one already has the list of instances.

One way to keep track of instances is with a class variable:
class A(object):
instances = []
def __init__(self, foo):
self.foo = foo
A.instances.append(self)
At the end of the program, you can create your dict like this:
foo_vars = {id(instance): instance.foo for instance in A.instances}
There is only one list:
>>> a = A(1)
>>> b = A(2)
>>> A.instances
[<__main__.A object at 0x1004d44d0>, <__main__.A object at 0x1004d4510>]
>>> id(A.instances)
4299683456
>>> id(a.instances)
4299683456
>>> id(b.instances)
4299683456

#JoelCornett's answer covers the basics perfectly. This is a slightly more complicated version, which might help with a few subtle issues.
If you want to be able to access all the "live" instances of a given class, subclass the following (or include equivalent code in your own base class):
from weakref import WeakSet
class base(object):
def __new__(cls, *args, **kwargs):
instance = object.__new__(cls, *args, **kwargs)
if "instances" not in cls.__dict__:
cls.instances = WeakSet()
cls.instances.add(instance)
return instance
This addresses two possible issues with the simpler implementation that #JoelCornett presented:
Each subclass of base will keep track of its own instances separately. You won't get subclass instances in a parent class's instance list, and one subclass will never stumble over instances of a sibling subclass. This might be undesirable, depending on your use case, but it's probably easier to merge the sets back together than it is to split them apart.
The instances set uses weak references to the class's instances, so if you del or reassign all the other references to an instance elsewhere in your code, the bookkeeping code will not prevent it from being garbage collected. Again, this might not be desirable for some use cases, but it is easy enough to use regular sets (or lists) instead of a weakset if you really want every instance to last forever.
Some handy-dandy test output (with the instances sets always being passed to list only because they don't print out nicely):
>>> b = base()
>>> list(base.instances)
[<__main__.base object at 0x00000000026067F0>]
>>> class foo(base):
... pass
...
>>> f = foo()
>>> list(foo.instances)
[<__main__.foo object at 0x0000000002606898>]
>>> list(base.instances)
[<__main__.base object at 0x00000000026067F0>]
>>> del f
>>> list(foo.instances)
[]

You would probably want to use weak references to your instances. Otherwise the class could likely end up keeping track of instances that were meant to have been deleted. A weakref.WeakSet will automatically remove any dead instances from its set.
One way to keep track of instances is with a class variable:
import weakref
class A(object):
instances = weakref.WeakSet()
def __init__(self, foo):
self.foo = foo
A.instances.add(self)
#classmethod
def get_instances(cls):
return list(A.instances) #Returns list of all current instances
At the end of the program, you can create your dict like this:
foo_vars = {id(instance): instance.foo for instance in A.instances}
There is only one list:
>>> a = A(1)
>>> b = A(2)
>>> A.get_instances()
[<inst.A object at 0x100587290>, <inst.A object at 0x100587250>]
>>> id(A.instances)
4299861712
>>> id(a.instances)
4299861712
>>> id(b.instances)
4299861712
>>> a = A(3) #original a will be dereferenced and replaced with new instance
>>> A.get_instances()
[<inst.A object at 0x100587290>, <inst.A object at 0x1005872d0>]

You can also solve this problem using a metaclass:
When a class is created (__init__ method of metaclass), add a new instance registry
When a new instance of this class is created (__call__ method of metaclass), add it to the instance registry.
The advantage of this approach is that each class has a registry - even if no instance exists. In contrast, when overriding __new__ (as in Blckknght's answer), the registry is added when the first instance is created.
class MetaInstanceRegistry(type):
"""Metaclass providing an instance registry"""
def __init__(cls, name, bases, attrs):
# Create class
super(MetaInstanceRegistry, cls).__init__(name, bases, attrs)
# Initialize fresh instance storage
cls._instances = weakref.WeakSet()
def __call__(cls, *args, **kwargs):
# Create instance (calls __init__ and __new__ methods)
inst = super(MetaInstanceRegistry, cls).__call__(*args, **kwargs)
# Store weak reference to instance. WeakSet will automatically remove
# references to objects that have been garbage collected
cls._instances.add(inst)
return inst
def _get_instances(cls, recursive=False):
"""Get all instances of this class in the registry. If recursive=True
search subclasses recursively"""
instances = list(cls._instances)
if recursive:
for Child in cls.__subclasses__():
instances += Child._get_instances(recursive=recursive)
# Remove duplicates from multiple inheritance.
return list(set(instances))
Usage: Create a registry and subclass it.
class Registry(object):
__metaclass__ = MetaInstanceRegistry
class Base(Registry):
def __init__(self, x):
self.x = x
class A(Base):
pass
class B(Base):
pass
class C(B):
pass
a = A(x=1)
a2 = A(2)
b = B(x=3)
c = C(4)
for cls in [Base, A, B, C]:
print cls.__name__
print cls._get_instances()
print cls._get_instances(recursive=True)
print
del c
print C._get_instances()
If using abstract base classes from the abc module, just subclass abc.ABCMeta to avoid metaclass conflicts:
from abc import ABCMeta, abstractmethod
class ABCMetaInstanceRegistry(MetaInstanceRegistry, ABCMeta):
pass
class ABCRegistry(object):
__metaclass__ = ABCMetaInstanceRegistry
class ABCBase(ABCRegistry):
__metaclass__ = ABCMeta
#abstractmethod
def f(self):
pass
class E(ABCBase):
def __init__(self, x):
self.x = x
def f(self):
return self.x
e = E(x=5)
print E._get_instances()

Another option for quick low-level hacks and debugging is to filter the list of objects returned by gc.get_objects() and generate the dictionary on the fly that way. In CPython that function will return you a (generally huge) list of everything the garbage collector knows about, so it will definitely contain all of the instances of any particular user-defined class.
Note that this is digging a bit into the internals of the interpreter, so it may or may not work (or work well) with the likes of Jython, PyPy, IronPython, etc. I haven't checked. It's also likely to be really slow regardless. Use with caution/YMMV/etc.
However, I imagine that some people running into this question might eventually want to do this sort of thing as a one-off to figure out what's going on with the runtime state of some slice of code that's behaving strangely. This method has the benefit of not affecting the instances or their construction at all, which might be useful if the code in question is coming out of a third-party library or something.

Here's a similar approach to Blckknght's, which works with subclasses as well. Thought this might be of interest, if someone ends up here. One difference, if B is a subclass of A, and b is an instance of B, b will appear in both A.instances and B.instances. As stated by Blckknght, this depends on the use case.
from weakref import WeakSet
class RegisterInstancesMixin:
instances = WeakSet()
def __new__(cls, *args, **kargs):
o = object.__new__(cls, *args, **kargs)
cls._register_instance(o)
return o
#classmethod
def print_instances(cls):
for instance in cls.instances:
print(instance)
#classmethod
def _register_instance(cls, instance):
cls.instances.add(instance)
for b in cls.__bases__:
if issubclass(b, RegisterInstancesMixin):
b._register_instance(instance)
def __init_subclass__(cls):
cls.instances = WeakSet()
class Animal(RegisterInstancesMixin):
pass
class Mammal(Animal):
pass
class Human(Mammal):
pass
class Dog(Mammal):
pass
alice = Human()
bob = Human()
cannelle = Dog()
Animal.print_instances()
Mammal.print_instances()
Human.print_instances()
Animal.print_instances() will print three objects, whereas Human.print_instances() will print two.

Using the answer from #Joel Cornett I've come up with the following, which seems to work. i.e. i'm able to total up object variables.
import os
os.system("clear")
class Foo():
instances = []
def __init__(self):
Foo.instances.append(self)
self.x = 5
class Bar():
def __init__(self):
pass
def testy(self):
self.foo1 = Foo()
self.foo2 = Foo()
self.foo3 = Foo()
foo = Foo()
print Foo.instances
bar = Bar()
bar.testy()
print Foo.instances
x_tot = 0
for inst in Foo.instances:
x_tot += inst.x
print x_tot
output:
[<__main__.Foo instance at 0x108e334d0>]
[<__main__.Foo instance at 0x108e334d0>, <__main__.Foo instance at 0x108e33560>, <__main__.Foo instance at 0x108e335a8>, <__main__.Foo instance at 0x108e335f0>]
5
10
15
20

(For Python)
I have found a way to record the class instances via the "dataclass" decorator while defining a class. Define a class attribute 'instances' (or any other name) as a list of the instances you want to record. Append that list with the 'dict' form of created objects via the dunder method __dict__. Thus, the class attribute 'instances' will record instances in the dict form, which you want.
For example,
from dataclasses import dataclass
#dataclass
class player:
instances=[]
def __init__(self,name,rank):
self.name=name
self.rank=rank
self.instances.append(self.__dict__)

Reverse mapping class attributes to classes in Python

I have some code in Python where I'll have a bunch of classes, each of which will have an attribute _internal_attribute. I would like to be able to generate a mapping of those attributes to the original class. Essentially I would like to be able to do this:
class A(object):
_internal_attribute = 'A attribute'
class B(object):
_internal_attribute = 'B attribute'
a_instance = magic_reverse_mapping['A attribute']()
b_instance = magic_reverse_mapping['B attribute']()
What I'm missing here is how to generate magic_reverse_mapping dict. I have a gut feeling that having a metaclass generate A and B is the correct way to go about this; does that seem right?

You can use a meta class to automatically register your classes in magic_reverse_mapping:
magic_reverse_mapping = {}
class MagicRegister(type):
def __new__(meta, name, bases, dict):
cls = type.__new__(meta, name, bases, dict)
magic_reverse_mapping[dict['_internal_attribute']] = cls
return cls
class A(object):
__metaclass__ = MagicRegister
_internal_attribute = 'A attribute'
afoo = magic_reverse_mapping['A attribute']()
Alternatively you can use a decorator on your classes to register them. I think this is more readable and easier to understand:
magic_reverse_mapping = {}
def magic_register(cls):
magic_reverse_mapping[cls._internal_attribute] = cls
return cls
#magic_register
class A(object):
_internal_attribute = 'A attribute'
afoo = magic_reverse_mapping['A attribute']()
Or you could even do it by hand. It's not that much more work without using any magic:
reverse_mapping = {}
class A(object):
_internal_attribute = 'A attribute'
reverse_mapping[A._internal_attribute] = A
Looking at the different variants I think the decorator version would be the most pleasant to use.

You need some data structure to store the list of applicable classes in the first place, but you don't have to generate it in the first place. You can read classes from globals instead. This naturally assumes that your classes extend object, as they do in your first post.
def magic_reverse_mapping(attribute_name, attribute_value):
classobjects = [val for val in globals().values() if isinstance(val, object)]
attrobjects = [cls for cls in classobjects if hasattr(cls, attribute_name)]
resultobjects = [cls for cls in attrobjects if object.__getattribute__(cls, attribute_name) == attribute_value]
return resultobjects
magic_reverse_mapping('_internal_attribute', 'A attribute')
#output: [<class '__main__.A'>]
Note that this returns a list of classes with that attribute value, because there may be more than one. If you wanted to instantiate the first one:
magic_reverse_mapping('_internal_attribute', 'A attribute')[0]()
#output: <__main__.A object at 0xb7ce486c>
Unlike in sth's answer, you don't have to add a decorator to your classes (neat solution, though). However, there's no way to exclude any classes that are in the global namespace.