I'm editing the original question because we're all focusing on SHOULD you ever want to do this. My question is simply CAN I do this and HOW (understanding that there may be several solutions). So I'm just going to leave the actual question and cut out the background.
Suppose I have a base class and a child class. Is there anything I can do in the base class to prevent __init__ from being called on the child class - or at least throw an exception or even log if __init__ exists or is called on the child class? I do want the __init__ method to be called on the parent class.
Edit/Conclusion - After exploring the options presented in the answers, I decided that doing this would be bad style. I will solve my problem a different way. Nonetheless, hopefully the answers below are helpful in case someone else wants to do this.
" Whether or not I should or need to do it is a separate discussion :)"
Please, keep that in mind.
But it can be done - when a class is instantiated, not only the syntax is just like
a method call - with the class object name followed y a parenthesis - the class itself (which is a Python object), is called - as a callable object.
Calling an object in Python invokes the __call__ magic method in its class. Therefore, instantiating a class, invokes the __call__ method on its metaclass.
What is inside this __call__ method in the standard metaclass (which is "type") is roughly equivalent to:
def __call__(cls, *args, **kw):
self = cls.__new__(cls, *args, **kw)
cls.__init__(self, *args, **kw)
return self
So, if you write a metaclass, overriding __call__ and suppress the call to __init__ in these, it won't be called at all:
class Meta(type):
def __call__(cls, *args, **kw):
return cls.__new__(cls, *args, **kw)
class NoInit(object):
__metaclass__ = Meta
def __init__(self):
print "Hello!"
NoInit()
If you want just to avoid that sublcasses have __init__ instead of not calling it, you can do a much simpler metaclass that would just raise an exception at class instantiation time:
class Meta(type):
def __new__(metacls, name, bases, dct):
if "__init__" in dct:
raise NameError("Classes in this hierarchy should not have an __init__ method")
return type.__new__(metacls, name, bases, dct)
That's quite doable, but I don't think you should.
Tell the users how to use your class and they should obey. Also, if someone is subclassing he should know how to call the parent's initialization method.
As a proof of concept, here's how it can be done with metaclasses (Python 2.x syntax):
>>> class WhoMovedMyInit(object):
class __metaclass__(type):
def __init__(self, *args, **kw):
super(type,self).__init__(*args, **kw)
if self.__init__ is not WhoMovedMyInit.__init__:
raise Exception('Dude, I told not to override my __init__')
>>> class IAmOk(WhoMovedMyInit):
pass
>>> class Lol(WhoMovedMyInit):
def __init__(self):
pass
Traceback (most recent call last):
File "<pyshell#35>", line 1, in <module>
class Lol(WhoMovedMyInit):
File "<pyshell#31>", line 6, in __init__
raise Exception('Dude, I told not to override my __init__')
Exception: Dude, I told not to override my __init__
You can also replace the subclass __init__ method to one which warns the user or raises an error on "runtime".
Most of these answers are outdated.
This can be easily done since python 3.6. It was defined in PEP487.
class Base:
def __init_subclass__(cls):
if Base.__init__ is not cls.__init__:
raise Exception(f'Do not override {cls}.__init__')
class Good(Base):
pass
class Bad(Base):
def __init__(self):
pass
Good() # No problem
Bad() # Raises Exception
Related
So I am trying to override __new__ and let it exist as a factory to create
derived instances. After reading a bit on SO, I am under the impression that I should be calling __new__ on the derived instance as well.
BaseThing
class BaseThing:
def __init(self, name, **kwargs):
self.name = name
# methods to be derived
ThingFactory
class Thing(BaseThing):
def __new__(cls, name, **kwargs):
if name == 'A':
return A.__new__(name, **kwargs)
if name == 'B':
return B.__new__(name, **kwargs)
def __init__(self, *args, **kwargs):
super().__init__(name, **kwargs)
# methods to be implemented by concrete class (same as those in base)
A
class A(BaseThing):
def __init__(self, name, **kwargs):
super().__init__(name, **kwargs)
B
class B(BaseThing):
def __init__(self, name, **kwargs):
super().__init__(name, **kwargs)
what I am expecting was that it'd just work.
>>> a = Thing('A')
gives me TypeError: object.__new__(X): X is not a type object (str)
I am bit confused by this; when I just return a concrete instance of derived classes, it just worked. i.e.
def __new__(cls, name, **kwargs):
if name == 'A':
return A(name)
if name == 'B':
return B(name)
I don't think this is the correct way to return in __new__; it may duplicate the calls to __init__.
when I am checking signatures of __new__ in object it seems be this one:
#staticmethod # known case of __new__
def __new__(cls, *more): # known special case of object.__new__
""" Create and return a new object. See help(type) for accurate signature. """
pass
I didn't expect this was the one; I'd expect it came with args and kwargs as well. I must have done something wrong here.
it seems to me that I need to inherit object directly in my base but could anyone explain the correct way of doing it?
You're calling __new__ wrong. If you want your __new__ to create an instance of a subclass, you don't call the subclass's __new__; you call the superclass's __new__ as usual, but pass it the subclass as the first argument:
instance = super().__new__(A)
I can't guarantee that this will be enough to fix your problems, since the code you've posted wouldn't reproduce the error you claim; it has other problems that would have caused a different error first (infinite recursion). Particularly, if A and B don't really descend from Thing, that needs different handling.
I have a master class for a planet:
class Planet:
def __init__(self,name):
self.name = name
(...)
def destroy(self):
(...)
I also have a few classes that inherit from Planet and I want to make one of them unable to be destroyed (not to inherit the destroy function)
Example:
class Undestroyable(Planet):
def __init__(self,name):
super().__init__(name)
(...)
#Now it shouldn't have the destroy(self) function
So when this is run,
Undestroyable('This Planet').destroy()
it should produce an error like:
AttributeError: Undestroyable has no attribute 'destroy'
The mixin approach in other answers is nice, and probably better for most cases. But nevertheless, it spoils part of the fun - maybe obliging you to have separate planet-hierarchies - like having to live with two abstract classes each ancestor of "destroyable" and "non-destroyable".
First approach: descriptor decorator
But Python has a powerful mechanism, called the "descriptor protocol", which is used to retrieve any attribute from a class or instance - it is even used to ordinarily retrieve methods from instances - so, it is possible to customize the method retrieval in a way it checks if it "should belong" to that class, and raise attribute error otherwise.
The descriptor protocol mandates that whenever you try to get any attribute from an instance object in Python, Python will check if the attribute exists in that object's class, and if so, if the attribute itself has a method named __get__. If it has, __get__ is called (with the instance and class where it is defined as parameters) - and whatever it returns is the attribute. Python uses this to implement methods: functions in Python 3 have a __get__ method that when called, will return another callable object that, in turn, when called will insert the self parameter in a call to the original function.
So, it is possible to create a class whose __get__ method will decide whether to return a function as a bound method or not depending on the outer class been marked as so - for example, it could check an specific flag non_destrutible. This could be done by using a decorator to wrap the method with this descriptor functionality
class Muteable:
def __init__(self, flag_attr):
self.flag_attr = flag_attr
def __call__(self, func):
"""Called when the decorator is applied"""
self.func = func
return self
def __get__(self, instance, owner):
if instance and getattr(instance, self.flag_attr, False):
raise AttributeError('Objects of type {0} have no {1} method'.format(instance.__class__.__name__, self.func.__name__))
return self.func.__get__(instance, owner)
class Planet:
def __init__(self, name=""):
pass
#Muteable("undestroyable")
def destroy(self):
print("Destroyed")
class BorgWorld(Planet):
undestroyable = True
And on the interactive prompt:
In [110]: Planet().destroy()
Destroyed
In [111]: BorgWorld().destroy()
...
AttributeError: Objects of type BorgWorld have no destroy method
In [112]: BorgWorld().destroy
AttributeError: Objects of type BorgWorld have no destroy method
Perceive that unlike simply overriding the method, this approach raises the error when the attribute is retrieved - and will even make hasattr work:
In [113]: hasattr(BorgWorld(), "destroy")
Out[113]: False
Although, it won't work if one tries to retrieve the method directly from the class, instead of from an instance - in that case the instance parameter to __get__ is set to None, and we can't say from which class it was retrieved - just the owner class, where it was declared.
In [114]: BorgWorld.destroy
Out[114]: <function __main__.Planet.destroy>
Second approach: __delattr__ on the metaclass:
While writting the above, it occurred me that Pythn does have the __delattr__ special method. If the Planet class itself implements __delattr__ and we'd try to delete the destroy method on specifc derived classes, it wuld nt work: __delattr__ gards the attribute deletion of attributes in instances - and if you'd try to del the "destroy" method in an instance, it would fail anyway, since the method is in the class.
However, in Python, the class itself is an instance - of its "metaclass". That is usually type . A proper __delattr__ on the metaclass of "Planet" could make possible the "disinheitance" of the "destroy" method by issuing a `del UndestructiblePlanet.destroy" after class creation.
Again, we use the descriptor protocol to have a proper "deleted method on the subclass":
class Deleted:
def __init__(self, cls, name):
self.cls = cls.__name__
self.name = name
def __get__(self, instance, owner):
raise AttributeError("Objects of type '{0}' have no '{1}' method".format(self.cls, self.name))
class Deletable(type):
def __delattr__(cls, attr):
print("deleting from", cls)
setattr(cls, attr, Deleted(cls, attr))
class Planet(metaclass=Deletable):
def __init__(self, name=""):
pass
def destroy(self):
print("Destroyed")
class BorgWorld(Planet):
pass
del BorgWorld.destroy
And with this method, even trying to retrieve or check for the method existense on the class itself will work:
In [129]: BorgWorld.destroy
...
AttributeError: Objects of type 'BorgWorld' have no 'destroy' method
In [130]: hasattr(BorgWorld, "destroy")
Out[130]: False
metaclass with a custom __prepare__ method.
Since metaclasses allow one to customize the object that contains the class namespace, it is possible to have an object that responds to a del statement within the class body, adding a Deleted descriptor.
For the user (programmer) using this metaclass, it is almost the samething, but for the del statement been allowed into the class body itself:
class Deleted:
def __init__(self, name):
self.name = name
def __get__(self, instance, owner):
raise AttributeError("No '{0}' method on class '{1}'".format(self.name, owner.__name__))
class Deletable(type):
def __prepare__(mcls,arg):
class D(dict):
def __delitem__(self, attr):
self[attr] = Deleted(attr)
return D()
class Planet(metaclass=Deletable):
def destroy(self):
print("destroyed")
class BorgPlanet(Planet):
del destroy
(The 'deleted' descriptor is the correct form to mark a method as 'deleted' - in this method, though, it can't know the class name at class creation time)
As a class decorator:
And given the "deleted" descriptor, one could simply inform the methods to be removed as a class decorator - there is no need for a metaclass in this case:
class Deleted:
def __init__(self, cls, name):
self.cls = cls.__name__
self.name = name
def __get__(self, instance, owner):
raise AttributeError("Objects of type '{0}' have no '{1}' method".format(self.cls, self.name))
def mute(*methods):
def decorator(cls):
for method in methods:
setattr(cls, method, Deleted(cls, method))
return cls
return decorator
class Planet:
def destroy(self):
print("destroyed")
#mute('destroy')
class BorgPlanet(Planet):
pass
Modifying the __getattribute__ mechanism:
For sake of completeness - what really makes Python reach methods and attributes on the super-class is what happens inside the __getattribute__ call. n the object version of __getattribute__ is where the algorithm with the priorities for "data-descriptor, instance, class, chain of base-classes, ..." for attribute retrieval is encoded.
So, changing that for the class is an easy an unique point to get a "legitimate" attribute error, without need for the "non-existent" descritor used on the previous methods.
The problem is that object's __getattribute__ does not make use of type's one to search the attribute in the class - if it did so, just implementing the __getattribute__ on the metaclass would suffice. One have to do that on the instance to avoid instance lookp of an method, and on the metaclass to avoid metaclass look-up. A metaclass can, of course, inject the needed code:
def blocker_getattribute(target, attr, attr_base):
try:
muted = attr_base.__getattribute__(target, '__muted__')
except AttributeError:
muted = []
if attr in muted:
raise AttributeError("object {} has no attribute '{}'".format(target, attr))
return attr_base.__getattribute__(target, attr)
def instance_getattribute(self, attr):
return blocker_getattribute(self, attr, object)
class M(type):
def __init__(cls, name, bases, namespace):
cls.__getattribute__ = instance_getattribute
def __getattribute__(cls, attr):
return blocker_getattribute(cls, attr, type)
class Planet(metaclass=M):
def destroy(self):
print("destroyed")
class BorgPlanet(Planet):
__muted__=['destroy'] # or use a decorator to set this! :-)
pass
If Undestroyable is a unique (or at least unusual) case, it's probably easiest to just redefine destroy():
class Undestroyable(Planet):
# ...
def destroy(self):
cls_name = self.__class__.__name__
raise AttributeError("%s has no attribute 'destroy'" % cls_name)
From the point of view of the user of the class, this will behave as though Undestroyable.destroy() doesn't exist … unless they go poking around with hasattr(Undestroyable, 'destroy'), which is always a possibility.
If it happens more often that you want subclasses to inherit some properties and not others, the mixin approach in chepner's answer is likely to be more maintainable. You can improve it further by making Destructible an abstract base class:
from abc import abstractmethod, ABCMeta
class Destructible(metaclass=ABCMeta):
#abstractmethod
def destroy(self):
pass
class BasePlanet:
# ...
pass
class Planet(BasePlanet, Destructible):
def destroy(self):
# ...
pass
class IndestructiblePlanet(BasePlanet):
# ...
pass
This has the advantage that if you try to instantiate the abstract class Destructible, you'll get an error pointing you at the problem:
>>> Destructible()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class Destructible with abstract methods destroy
… similarly if you inherit from Destructible but forget to define destroy():
class InscrutablePlanet(BasePlanet, Destructible):
pass
>>> InscrutablePlanet()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class InscrutablePlanet with abstract methods destroy
Rather than remove an attribute that is inherited, only inherit destroy in the subclasses where it is applicable, via a mix-in class. This preserves the correct "is-a" semantics of inheritance.
class Destructible(object):
def destroy(self):
pass
class BasePlanet(object):
...
class Planet(BasePlanet, Destructible):
...
class IndestructiblePlanet(BasePlanet): # Does *not* inherit from Destructible
...
You can provide suitable definitions for destroy in any of Destructible, Planet, or any class that inherits from Planet.
Metaclasses and descriptor protocols are fun, but perhaps overkill. Sometimes, for raw functionality, you can't beat good ole' __slots__.
class Planet(object):
def __init__(self, name):
self.name = name
def destroy(self):
print("Boom! %s is toast!\n" % self.name)
class Undestroyable(Planet):
__slots__ = ['destroy']
def __init__(self,name):
super().__init__(name)
print()
x = Planet('Pluto') # Small, easy to destroy
y = Undestroyable('Jupiter') # Too big to fail
x.destroy()
y.destroy()
Boom! Pluto is toast!
Traceback (most recent call last):
File "planets.py", line 95, in <module>
y.destroy()
AttributeError: destroy
You cannot inherit only a portion of a class. Its all or nothing.
What you can do is to put the destroy function in a second level of the class, such you have the Planet-class without the destry-function, and then you make a DestroyablePlanet-Class where you add the destroy-function, which all the destroyable planets use.
Or you can put a flag in the construct of the Planet-Class which determines if the destroy function will be able to succeed or not, which is then checked in the destroy-function.
I'm interested in using the __new__ functionality to inject code into the __init__ function of subclasses. My understanding from the documentation is that python will call __init__ on the instance returned by __new__. However, my efforts to change the value of __init__ in the instance before returning it from __new__ don't seem to work.
class Parent(object):
def __new__(cls, *args, **kwargs):
new_object = super(Parent, cls).__new__(cls)
user_init = new_object.__init__
def __init__(self, *args, **kwargs):
print("New __init__ called")
user_init(self, *args, **kwargs)
self.extra()
print("Replacing __init__")
setattr(new_object, '__init__', __init__)
return new_object
def extra(self):
print("Extra called")
class Child(Parent):
def __init__(self):
print("Original __init__ called")
super(Child, self).__init__()
c = Child()
The above code prints:
Replacing __init__
Original __init__ called
but I would expect it to print
Replacing __init__
New __init__ called
Original __init__ called
Extra called
Why not?
I feel like Python is calling the original value of __init__, regardless of what I set it to in __new__. Running introspection on c.__init__ shows that the new version is in place, but it hasn't been called as part of the object creation.
Well, the new object is expected to be empty before the __init__ is called. So probably python, as optimization, does not bother to query the object and goes to fetch __init__ straight from the class.
Therefore you'll have to modify __init__ of the subclasses themselves. Fortunately Python has a tool for that, metaclasses.
In Python 2, you set metaclass by setting special member:
class Parent(object):
__metaclass__ = Meta
...
See Python2 documentation
In Python 3, you set metaclass via keyword attribute in the parent list, so
class Parent(metaclass=Meta):
...
See Python3 documentation
The metaclass is a base class for the class instance. It has to be derived from type and in it's __new__ it can modify the class being created (I believe the __init__ should be called too, but the examples override __new__, so I'll go with it). The __new__ will be similar to what you have:
class Meta(type):
def __new__(mcs, name, bases, namespace, **kwargs):
new_cls = super(Meta, mcs).__new__(mcs, name, bases, namespace, **kwargs)
user_init = new_cls.__init__
def __init__(self, *args, **kwargs):
print("New __init__ called")
user_init(self, *args, **kwargs)
self.extra()
print("Replacing __init__")
setattr(new_cls, '__init__', __init__)
return new_cls
(using the Python 3 example, but the signature in Python 2 seems to be the same except there are no **kwargs, but adding them shouldn't hurt; I didn't test it).
I suspect the answer is that __init__ is a special function, internally it is defined as a class method, and as a result cannot be replaced by reassigning it in an instance of the object.
In Python, all objects are represented by the PyObject in C, which has a pointer to a PyTypeObject. This contains a member called tp_init that I believe contains a pointer to the __init__ function.
The other solution works, because we are modifying the class, not an instance of the object.
I experiment with metaclasses to generate the class with the custom special method - particularly, __call__. The generation of the class depends on the parameters the constructor was called with. I've faced a strange effect, simplified example is below:
def trick(self, *args, **kwargs):
print "Works!"
class Test1Factory(type):
def __new__(mcls, name, bases, namespace):
namespace['__call__'] = trick
return type.__new__(mcls, name, bases, namespace)
class Test1(object):
__metaclass__ = Test1Factory
def __init__(self, value):
self._value = value
t1 = Test1(1)
t1() # "Works!"
It works, but it is not really useful, because there is no access to constructor arguments within __new__. type.__call__ should do the trick:
import types
class Test2Factory(type):
def __call__(self, *args, **kwargs):
obj = type.__call__(self, *args, **kwargs)
setattr(obj, '__call__', types.MethodType(trick, obj, Test2))
return obj
class Test2(object):
__metaclass__ = Test2Factory
def __init__(self, value):
self._value = value
t2 = Test2(2)
t2.__call__() # "Works!"
t2() # TypeError: 'Test2' object is not callable
As far as I understand, instance() is similar to instance.__call__(), but it is not the case here. Using __new__ static method of the class does the same. I have a workaround that does not use metaclasses at all, but just want to understand the phenomena. Python version is 2.7.5
The wrong assumption may be in “instance() is similar to instance.__call__()”, as __call__ is not looked up for in instance, but in instance's type. That is, the __call__ used is not that of instance, but that of instance.__class__ or type(instance).
Any __call__ attribute defined on the instance solely, may be accessed regularly as any other attribute, but will not be used when instance is called as in instance(). That's part of Python's semantic.
Try to define a __call__ both on an instance and on its type, and see what you get.
If I understand the question correctly, the question has the same background as another I had, and which gets an answer (summarized, with demonstrations by experiments, in the question's post) : “How do Python tell “this is called as a function”?”.
Part 1
I have a setup where I have a set of classes that I want to mock, my idea was that in the cases where I want to do this I pass a mock keyword argument into the constructor and in __new__ intercept this and instead pass back a mocked version of that object.
It looks like this (Edited the keyword lookup after #mgilsons suggestion):
class RealObject(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock', None):
return MockRealObject()
return super(RealObect, cls).__new__(cls, *args, **kwargs)
def __init__(self, whatever = None):
'''
Constructor
'''
#stuff happens
I then call the constructor like this:
ro = RealObject(mock = bool)
The issue I have here is that I get the following error when bool is False:
TypeError: __init__() got an unexpected keyword argument 'mock'
This works if I add mock as a keyword argument to __init__ but what I am asking if this is possible to avoid. I even pop the mock from the kwargs dict.
This is also a question about the design. Is there a better way to do this? (of course!) I wanted to try doing it this way, without using a factory or a superclass or anything. But still, should I use another keyword maybe? __call__?
Part 2 based on jsbueno's answer
So I wanted to extract the metaclass and the __new__ function into a separate module. I did this:
class Mockable(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock', None):
mock_cls = eval('{0}{1}'.format('Mock',cls.__name__))
return super(mock_cls, mock_cls).__new__(mock_cls)
return super(cls, cls).__new__(cls,*args, **kwargs)
class MockableMetaclass(type):
def __call__(self, *args, **kwargs):
obj = self.__new__(self, *args, **kwargs)
if "mock" in kwargs:
del kwargs["mock"]
obj.__init__(*args, **kwargs)
return obj
And I have defined in a separate module the classes RealObject and MockRealObject.
I have two problems now:
If MockableMetaclass and Mockable are not in the same module as the RealObject class the eval will raise a NameError if I provide mock = True.
If mock = False the code will enter into an endless recursion that ends in an impressive RuntimeError: maximum recursion depth exceeded while calling a Python objec. I'm guessing this is due to RealObject's superclass no longer being object but instead Mockable.
How can I fix these problems? is my approach incorrect? Should I instead have Mockable as a decorator? I tried that but that didn't seem to work since __new__ of an instance is only read-only it seems.
This is a job for the metaclass! :-)
The code responsible to call both __new__ and __init__ when instantiating a Python new-style object lies in the __call__method for the class metaclass. (or the semantically equivalent to that).
In other words - when you do:
RealObject() - what is really called is the RealObject.__class__.__call__ method.
Since without declaring a explicit metaclass, the metaclass is type, it is type.__call__ which is called.
Most recipes around dealing with metaclasses deal with subclassing the __new__ method - automating actions when the class is created. But overriding __call__ we can take actions when the class is instantiated, instead.
In this case, all that is needed is to remove the "mock" keyword parameter, if any, before calling __init__:
class MetaMock(type):
def __call__(cls, *args, **kw):
obj = cls.__new__(cls, *args, **kw)
if "mock" in kw:
del kw["mock"]
obj.__init__(*args, **kw)
return obj
class RealObject(metaclass=MetaMock):
...
A subclass is pretty much essential, since __new__ always passes the arguments to the constructor call to the __init__ method. If you add a subclass via a class decorator as a mixin then you can intercept the mock argument in the subclass __init__:
def mock_with(mock_cls):
class MockMixin(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock'):
return mock_cls()
return super(MockMixin, cls).__new__(cls, *args, **kwargs)
def __init__(self, *args, **kwargs):
kwargs.pop('mock')
super(MockMixin, self).__init__(*args, **kwargs)
def decorator(real_cls):
return type(real_cls.__name__, (MockMixin, real_cls), {})
return decorator
class MockRealObject(object):
pass
#mock_with(MockRealObject)
class RealObject(object):
def __init__(self, whatever=None):
pass
r = RealObject(mock=False)
assert isinstance(r, RealObject)
m = RealObject(mock=True)
assert isinstance(m, MockRealObject)
The alternative is for the subclass __new__ method to return RealObject(cls, *args, **kwargs); in that case, since the returned object isn't an instance of the subclass. However in that case the isinstance check will fail.