Develop Reference

Understanding Abstract Base Classes in Python

I was reading about abstract base class and came across https://www.python-course.eu/python3_abstract_classes.php website. I got general idea about them but I found two statement contradictory of each other.
Subclasses of an abstract class in Python are not required to implement abstract methods of the parent class.
and
A class that is derived from an abstract class cannot be instantiated unless all of its abstract methods are overridden.
My understanding of first statement is, derived class are not required to implement abstract method of the parent class which is wrong. I made a sample program to check that.
from abc import ABC, abstractmethod
class AbstractClassExample(ABC):
#abstractmethod
def do_something(self):
print("Some implementation!")
class AnotherSubclass(AbstractClassExample):
def just_another_method(self):
super().do_something()
print("The enrichment from AnotherSubclass")
x = AnotherSubclass() # TypeError: Can't instantiate abstract class AnotherSubclass with abstract methods do_something
x.do_something()
I would like an explanation of what the first statement means(preferably with examples).

Your code demonstrates that the second statement is true. It doesn't show that the first statement is false.
In your code, you are trying to instantiate AnotherSubclass, which is not allowed because AnotherSubclass does not implement all the abstract methods. The second statement says this.
However, if you delete the last two lines, i.e. not instantiating AnotherSubclass, then your code will produce no errors when you try to run it. This shows that the first statement is true - subclasses of abstract classes that doesn't implement all its abstract methods are allowed to exist.
You can write another subclass of AnotherSubclass called YetAnotherClass, this time implementing the abstract method, and you will be able to instantiate YetAnotherClass. Note that your program now does something, and AnotherSubclass is still allowed to exist.

Inheritance - proper way to create a new instance of a class with a class method

I'm God (or evolution, or whatever you believe in). I am trying to create all living things with Python.
I have defined a class that is able to "reproduce", i.e is able to create a new instance of itself (ignore the fact that it looks more like cloning than reproduction, this is for a beta version of Earth) :
class Animal:
def __init__(self, **kwargs):
self.characteristics = kwargs
def reproduce(self):
return Animal(**self.characteristics)
This works fine in the case of a base class, but what happens when I create a class that inherits from Animal ?
class Fox (Animal):
def __init__ (self, **kwargs):
self.color = 'red'
super().__init__(dict(self.color, **kwargs))
If a Fox tries to reproduce, I will have an instance of type Animal rather than Fox (even though it still has the color 'red').
I could overload the method for a fox to be able to reproduce:
def reproduce(self):
return Fox(self.characteristics)
However, I would have to do that for every new creature I define!
How can I create a class from which I could make all my creatures inherit so when they reproduce an object of the same class is created? So that I could be sure that:
parent = Fox()
child = parent.reproduce()
assert type(parent) == type(child)
I know I can use type to make reproduce return type(self)(self.characteristics) or self.__class__(**self.characteristics), but it does not seem very pytonic to me. Is there a more proper way to do this ?

Note: you changed your question from one where your subclasses took different numbers of arguments. If you stick to such a design, then you have no choice here but to override reproduce(), because there is no consistent API to create a new instance of the 'current class'.
If you were to standardise your class API, you can then also standardise creating new instances, at which point you can write a reproduce() method that just takes type(self) to reference the current class and then proceed to make a new instance of this class.
Note that having to write a new reproduce() method for each subclass is a good option too, because that's a good way of delegating creating new instances of specialised subclasses. You give each subclass the responsibility of handling the details of reproduction.
But if you don't want to do that, then you take away that responsibility from the subclass and put it in the base class, at which point the base design of how you create instances also is the responsibility of that base class.
There are middle grounds between those two options, of course, but all of them come down to some form of delegation. You could have the base classes provide some kind of structure that details what attributes should be copied across when creating an instance, you could have subclasses implement the __copy__ or __deepcopy__ hooks to handle 'reproduction' via copy.copy() or copy.deepcopy() calls, etc.
Your updated question structure is just another example of that delegation; you added a characteristics dictionary, so subclasses are responsible of keeping that dictionary updated so that the base class can implement reproduction as:
def reproduce(self):
return type(self)(**self.characteristics)
That's perfectly Pythonic, but more because this is a decent OO design where you have made choices to minimise what subclasses are responsible for and have the base class do as much of the reproducing as possible.

How to tell if a class is abstract in Python 3?

I wrote a metaclass that automatically registers its classes in a dict at runtime. In order for it to work properly, it must be able to ignore abstract classes.
The code works really well in Python 2, but I've run into a wall trying to make it compatible with Python 3.
Here's what the code looks like currently:
def AutoRegister(registry, base_type=ABCMeta):
class _metaclass(base_type):
def __init__(self, what, bases=None, attrs=None):
super(_metaclass, self).__init__(what, bases, attrs)
# Do not register abstract classes.
# Note that we do not use `inspect.isabstract` here, as
# that only detects classes with unimplemented abstract
# methods - which is a valid approach, but not what we
# want here.
# :see: http://stackoverflow.com/a/14410942/
metaclass = attrs.get('__metaclass__')
if not (metaclass and issubclass(metaclass, ABCMeta)):
registry.register(self)
return _metaclass
Usage in Python 2 looks like this:
# Abstract classes; these are not registered.
class BaseWidget(object): __metaclass__ = AutoRegister(widget_registry)
class BaseGizmo(BaseWidget): __metaclass__ = ABCMeta
# Concrete classes; these get registered.
class AlphaWidget(BaseWidget): pass
class BravoGizmo(BaseGizmo): pass
What I can't figure out, though, is how to make this work in Python 3.
How can a metaclass determine if it is initializing an abstract class in Python 3?

PEP3119 describes how the ABCMeta metaclass "marks" abstract methods and creates an __abstractmethods__ frozenset that contains all methods of a class that are still abstract. So, to check if a class cls is abstract, check if cls.__abstractmethods__ is empty or not.
I also found this relevant post on abstract classes useful.

I couldn't shake the feeling as I was posting this question that I was dealing with an XY Problem. As it turns out, that's exactly what was going on.
The real issue here is that the AutoRegister metaclass, as implemented, relies on a flawed understanding of what an abstract class is. Python or not, one of the most important criteria of an abstract class is that it is not instanciable.
In the example posted in the question, BaseWidget and BaseGizmo are instanciable, so they are not abstract.
Aren't we just bifurcating rabbits here?
Well, why was I having so much trouble getting AutoRegister to work in Python 3? Because I was trying to build something whose behavior contradicts the way classes work in Python.
The fact that inspect.isabstract wasn't returning the result I wanted should have been a major red flag: AutoRegister is a warranty-voider.
So what's the real solution then?
First, we have to recognize that BaseWidget and BaseGizmo have no reason to exist. They do not provide enough functionality to be instantiable, nor do they declare abstract methods that describe the functionality that they are missing.
One could argue that they could be used to "categorize" their sub-classes, but a) that's clearly not what's going on in this case, and b) quack.
Instead, we could embrace Python's definition of "abstract":
Modify BaseWidget and BaseGizmo so that they define one or more abstract methods.
If we can't come up with any abstract methods, then can we remove them entirely?
If we can't remove them but also can't make them properly abstract, it might be worthwhile to take a step back and see if there are other ways we might solve this problem.
Modify the definition of AutoRegister so that it uses inspect.isabstract to decide if a class is abstract: see final implementation.
That's cool and all, but what if I can't change the base classes?
Or, if you have to maintain backwards compatibility with existing code (as was the case for me), a decorator is probably easier:
#widget_registry.register
class AlphaWidget(object):
pass
#widget_registry.register
class BravoGizmo(object):
pass

Class factories and abstract base classes

I am trying to define a number of classes based on an abstract base class. Each of these classes basically defines a cell shape for a visualisation package. The cell is comprised of a number of vertices (points) and each subclass will require a different number of points. Each class can be thought of as a container for a fixed number of point coordinates.
As an example, consider the base class Shape, which is simply a container for a list of coordinates:
class Shape(object):
"""Cell shape base class."""
def __init__(self, sequence):
self.points = sequence
#property
def points(self):
return self._points
#points.setter
def points(self, sequence):
# Error checking goes here, e.g. check that `sequence` is a
# sequence of numeric values.
self._points = sequence
Ideally I want to be able to define, say, a Square class, where the points.setter method checks that sequence is of length four. Furthermore I would like a user to not be able to instantiate Shape. Is there a way I can define Shape to be an abstract base class? I have tried changing the definition of shape to the following:
import abc
class Shape(object):
"""Cell shape base class."""
__metaclass__ = abc.ABCMeta
def __init__(self, sequence):
self.points = sequence
#abc.abstractproperty
def npoints(self):
pass
#property
def points(self):
return self._points
#points.setter
def points(self, sequence):
# Error checking goes here...
if len(sequence) != self.npoints:
raise TypeError('Some descriptive error message!')
self._points = sequence
This requires subclasses to define the property npoints. I can then define a class Square as
class Square(Shape):
#property
def npoints(self):
return 4
However, this would be rather tedious to implement for a large number of sublcasses (and with more than one property to implement). I was hoping to define a class factory which would create my subclasses for me, something along the lines of:
def Factory(name, npoints):
return type(name, (Shape,), dict(npoints=npoints))
Triangle = Factory('Triangle', 3)
Square = Factory('Square', 4)
# etc...
Is this class factory function a valid approach to take, or am I clobbering the npoints property? Is it better to replace the call to type with something more verbose like:
def Factory(name, _npoints):
class cls(Shape):
#property
def npoints(self):
return _npoints
cls.__name__ = name
return cls
An alternative approach would be to define a class attribute _NPOINTS and change the npoints
property of Shape to
#property
def npoints(self):
return _NPOINTS
However, then I loose the benefit of using an abstract base class since:
I can't see how to define a class attribute using type, and
I don't know how to define an abstract class attribute.
Does anyone have any thoughts on the best way to implement this abstract base class and class factory function, or even an altogether better design?

Without knowing more about your project, I cannot give specific advice on the general design. I will just provide a few more general hints and thoughts.
Dynamically generated classes are often a sign that you don't need separate classes at all – simply write a single class that incorparates all the functionality. What's the problem with a Shape class that gets it's properties at instantiation time? (Of course there are reasons to use dynamically generated classes – the namedtuple() factory function is one example. I couldn't find any specific reasons in your question, however.)
Instead of using abstract base classes, you often simply document the intended interface, and than write classes conforming to this interface. Due to the dynamic nature of Python, you don't strictly need a common base class. There are often other advantages to a common base class – for example shared functionality.
Only check for application code errors if not doing so leads to strange errors in unrelated places. If, say, your function expects an iterable, simply assume you got an iterable. If the user passed in something else, you code will fail when it tries to iterate the passed in object anyway, and the error message will usually be enough for the application developer to understand the error.

In Python, when should I use a meta class?

I have gone through this: What is a metaclass in Python?
But can any one explain more specifically when should I use the meta class concept and when it's very handy?
Suppose I have a class like below:
class Book(object):
CATEGORIES = ['programming','literature','physics']
def _get_book_name(self,book):
return book['title']
def _get_category(self, book):
for cat in self.CATEGORIES:
if book['title'].find(cat) > -1:
return cat
return "Other"
if __name__ == '__main__':
b = Book()
dummy_book = {'title':'Python Guide of Programming', 'status':'available'}
print b._get_category(dummy_book)
For this class.
In which situation should I use a meta class and why is it useful?
Thanks in advance.

You use metaclasses when you want to mutate the class as it is being created. Metaclasses are hardly ever needed, they're hard to debug, and they're difficult to understand -- but occasionally they can make frameworks easier to use. In our 600Kloc code base we've used metaclasses 7 times: ABCMeta once, 4x models.SubfieldBase from Django, and twice a metaclass that makes classes usable as views in Django. As #Ignacio writes, if you don't know that you need a metaclass (and have considered all other options), you don't need a metaclass.

Conceptually, a class exists to define what a set of objects (the instances of the class) have in common. That's all. It allows you to think about the instances of the class according to that shared pattern defined by the class. If every object was different, we wouldn't bother using classes, we'd just use dictionaries.
A metaclass is an ordinary class, and it exists for the same reason; to define what is common to its instances. The default metaclass type provides all the normal rules that make classes and instances work the way you're used to, such as:
Attribute lookup on an instance checks the instance followed by its class, followed by all superclasses in MRO order
Calling MyClass(*args, **kwargs) invokes i = MyClass.__new__(MyClass, *args, **kwargs) to get an instance, then invokes i.__init__(*args, **kwargs) to initialise it
A class is created from the definitions in a class block by making all the names bound in the class block into attributes of the class
Etc
If you want to have some classes that work differently to normal classes, you can define a metaclass and make your unusual classes instances of the metaclass rather than type. Your metaclass will almost certainly be a subclass of type, because you probably don't want to make your different kind of class completely different; just as you might want to have some sub-set of Books behave a bit differently (say, books that are compilations of other works) and use a subclass of Book rather than a completely different class.
If you're not trying to define a way of making some classes work differently to normal classes, then a metaclass is probably not the most appropriate solution. Note that the "classes define how their instances work" is already a very flexible and abstract paradigm; most of the time you do not need to change how classes work.
If you google around, you'll see a lot of examples of metaclasses that are really just being used to go do a bunch of stuff around class creation; often automatically processing the class attributes, or finding new ones automatically from somewhere. I wouldn't really call those great uses of metaclasses. They're not changing how classes work, they're just processing some classes. A factory function to create the classes, or a class method that you invoke immediately after class creation, or best of all a class decorator, would be a better way to implement this sort of thing, in my opinion.
But occasionally you find yourself writing complex code to get Python's default behaviour of classes to do something conceptually simple, and it actually helps to step "further out" and implement it at the metaclass level.
A fairly trivial example is the "singleton pattern", where you have a class of which there can only be one instance; calling the class will return an existing instance if one has already been created. Personally I am against singletons and would not advise their use (I think they're just global variables, cunningly disguised to look like newly created instances in order to be even more likely to cause subtle bugs). But people use them, and there are huge numbers of recipes for making singleton classes using __new__ and __init__. Doing it this way can be a little irritating, mainly because Python wants to call __new__ and then call __init__ on the result of that, so you have to find a way of not having your initialisation code re-run every time someone requests access to the singleton. But wouldn't be easier if we could just tell Python directly what we want to happen when we call the class, rather than trying to set up the things that Python wants to do so that they happen to do what we want in the end?
class Singleton(type):
def __init__(self, *args, **kwargs):
super(Singleton, self).__init__(*args, **kwargs)
self.__instance = None
def __call__(self, *args, **kwargs):
if self.__instance is None:
self.__instance = super(Singleton, self).__call__(*args, **kwargs)
return self.__instance
Under 10 lines, and it turns normal classes into singletons simply by adding __metaclass__ = Singleton, i.e. nothing more than a declaration that they are a singleton. It's just easier to implement this sort of thing at this level, than to hack something out at the class level directly.
But for your specific Book class, it doesn't look like you have any need to do anything that would be helped by a metaclass. You really don't need to reach for metaclasses unless you find the normal rules of how classes work are preventing you from doing something that should be simple in a simple way (which is different from "man, I wish I didn't have to type so much for all these classes, I wonder if I could auto-generate the common bits?"). In fact, I have never actually used a metaclass for something real, despite using Python every day at work; all my metaclasses have been toy examples like the above Singleton or else just silly exploration.

A metaclass is used whenever you need to override the default behavior for classes, including their creation.
A class gets created from the name, a tuple of bases, and a class dict. You can intercept the creation process to make changes to any of those inputs.
You can also override any of the services provided by classes:
__call__ which is used to create instances
__getattribute__ which is used to lookup attributes and methods on a class
__setattr__ which controls setting attributes
__repr__ which controls how the class is diplayed
In summary, metaclasses are used when you need to control how classes are created or when you need to alter any of the services provided by classes.

If you for whatever reason want to do stuff like Class[x], x in Class etc., you have to use metaclasses:
class Meta(type):
def __getitem__(cls, x):
return x ** 2
def __contains__(cls, x):
return int(x ** (0.5)) == x ** 0.5
# Python 2.x
class Class(object):
__metaclass__ = Meta
# Python 3.x
class Class(metaclass=Meta):
pass
print Class[2]
print 4 in Class

check the link Meta Class Made Easy to know how and when to use meta class.