Why did the Python designers decide that subclasses' __init__() methods don't automatically call the __init__() methods of their superclasses, as in some other languages? Is the Pythonic and recommended idiom really like the following?
class Superclass(object):
def __init__(self):
print 'Do something'
class Subclass(Superclass):
def __init__(self):
super(Subclass, self).__init__()
print 'Do something else'
The crucial distinction between Python's __init__ and those other languages constructors is that __init__ is not a constructor: it's an initializer (the actual constructor (if any, but, see later;-) is __new__ and works completely differently again). While constructing all superclasses (and, no doubt, doing so "before" you continue constructing downwards) is obviously part of saying you're constructing a subclass's instance, that is clearly not the case for initializing, since there are many use cases in which superclasses' initialization needs to be skipped, altered, controlled -- happening, if at all, "in the middle" of the subclass initialization, and so forth.
Basically, super-class delegation of the initializer is not automatic in Python for exactly the same reasons such delegation is also not automatic for any other methods -- and note that those "other languages" don't do automatic super-class delegation for any other method either... just for the constructor (and if applicable, destructor), which, as I mentioned, is not what Python's __init__ is. (Behavior of __new__ is also quite peculiar, though really not directly related to your question, since __new__ is such a peculiar constructor that it doesn't actually necessarily need to construct anything -- could perfectly well return an existing instance, or even a non-instance... clearly Python offers you a lot more control of the mechanics than the "other languages" you have in mind, which also includes having no automatic delegation in __new__ itself!-).
I'm somewhat embarrassed when people parrot the "Zen of Python", as if it's a justification for anything. It's a design philosophy; particular design decisions can always be explained in more specific terms--and they must be, or else the "Zen of Python" becomes an excuse for doing anything.
The reason is simple: you don't necessarily construct a derived class in a way similar at all to how you construct the base class. You may have more parameters, fewer, they may be in a different order or not related at all.
class myFile(object):
def __init__(self, filename, mode):
self.f = open(filename, mode)
class readFile(myFile):
def __init__(self, filename):
super(readFile, self).__init__(filename, "r")
class tempFile(myFile):
def __init__(self, mode):
super(tempFile, self).__init__("/tmp/file", mode)
class wordsFile(myFile):
def __init__(self, language):
super(wordsFile, self).__init__("/usr/share/dict/%s" % language, "r")
This applies to all derived methods, not just __init__.
Java and C++ require that a base class constructor is called because of memory layout.
If you have a class BaseClass with a member field1, and you create a new class SubClass that adds a member field2, then an instance of SubClass contains space for field1 and field2. You need a constructor of BaseClass to fill in field1, unless you require all inheriting classes to repeat BaseClass's initialization in their own constructors. And if field1 is private, then inheriting classes can't initialise field1.
Python is not Java or C++. All instances of all user-defined classes have the same 'shape'. They're basically just dictionaries in which attributes can be inserted. Before any initialisation has been done, all instances of all user-defined classes are almost exactly the same; they're just places to store attributes that aren't storing any yet.
So it makes perfect sense for a Python subclass not to call its base class constructor. It could just add the attributes itself if it wanted to. There's no space reserved for a given number of fields for each class in the hierarchy, and there's no difference between an attribute added by code from a BaseClass method and an attribute added by code from a SubClass method.
If, as is common, SubClass actually does want to have all of BaseClass's invariants set up before it goes on to do its own customisation, then yes you can just call BaseClass.__init__() (or use super, but that's complicated and has its own problems sometimes). But you don't have to. And you can do it before, or after, or with different arguments. Hell, if you wanted you could call the BaseClass.__init__ from another method entirely than __init__; maybe you have some bizarre lazy initialization thing going.
Python achieves this flexibility by keeping things simple. You initialise objects by writing an __init__ method that sets attributes on self. That's it. It behaves exactly like a method, because it is exactly a method. There are no other strange and unintuitive rules about things having to be done first, or things that will automatically happen if you don't do other things. The only purpose it needs to serve is to be a hook to execute during object initialisation to set initial attribute values, and it does just that. If you want it to do something else, you explicitly write that in your code.
To avoid confusion it is useful to know that you can invoke the base_class __init__() method if the child_class does not have an __init__() class.
Example:
class parent:
def __init__(self, a=1, b=0):
self.a = a
self.b = b
class child(parent):
def me(self):
pass
p = child(5, 4)
q = child(7)
z= child()
print p.a # prints 5
print q.b # prints 0
print z.a # prints 1
In fact the MRO in python will look for __init__() in the parent class when can not find it in the children class. You need to invoke the parent class constructor directly if you have already an __init__() method in the children class.
For example the following code will return an error:
class parent:
def init(self, a=1, b=0):
self.a = a
self.b = b
class child(parent):
def __init__(self):
pass
def me(self):
pass
p = child(5, 4) # Error: constructor gets one argument 3 is provided.
q = child(7) # Error: constructor gets one argument 2 is provided.
z= child()
print z.a # Error: No attribute named as a can be found.
"Explicit is better than implicit." It's the same reasoning that indicates we should explicitly write 'self'.
I think in in the end it is a benefit-- can you recite all of the rules Java has regarding calling superclasses' constructors?
Right now, we have a rather long page describing the method resolution order in case of multiple inheritance: http://www.python.org/download/releases/2.3/mro/
If constructors were called automatically, you'd need another page of at least the same length explaining the order of that happening. That would be hell...
Often the subclass has extra parameters which can't be passed to the superclass.
Maybe __init__ is the method that the subclass needs to override. Sometimes subclasses need the parent's function to run before they add class-specific code, and other times they need to set up instance variables before calling the parent's function. Since there's no way Python could possibly know when it would be most appropriate to call those functions, it shouldn't guess.
If those don't sway you, consider that __init__ is Just Another Function. If the function in question were dostuff instead, would you still want Python to automatically call the corresponding function in the parent class?
i believe the one very important consideration here is that with an automatic call to super.__init__(), you proscribe, by design, when that initialization method is called, and with what arguments. eschewing automatically calling it, and requiring the programmer to explicitly do that call, entails a lot of flexibility.
after all, just because class B is derived from class A does not mean A.__init__() can or should be called with the same arguments as B.__init__(). making the call explicit means a programmer can have e.g. define B.__init__() with completely different parameters, do some computation with that data, call A.__init__() with arguments as appropriate for that method, and then do some postprocessing. this kind of flexibility would be awkward to attain if A.__init__() would be called from B.__init__() implicitly, either before B.__init__() executes or right after it.
As Sergey Orshanskiy pointed out in the comments, it is also convenient to write a decorator to inherit the __init__ method.
You can write a decorator to inherit the __init__ method, and even perhaps automatically search for subclasses and decorate them. – Sergey Orshanskiy Jun 9 '15 at 23:17
Part 1/3: The implementation
Note: actually this is only useful if you want to call both the base and the derived class's __init__ since __init__ is inherited automatically. See the previous answers for this question.
def default_init(func):
def wrapper(self, *args, **kwargs) -> None:
super(type(self), self).__init__(*args, **kwargs)
return wrapper
class base():
def __init__(self, n: int) -> None:
print(f'Base: {n}')
class child(base):
#default_init
def __init__(self, n: int) -> None:
pass
child(42)
Outputs:
Base: 42
Part 2/3: A warning
Warning: this doesn't work if base itself called super(type(self), self).
def default_init(func):
def wrapper(self, *args, **kwargs) -> None:
'''Warning: recursive calls.'''
super(type(self), self).__init__(*args, **kwargs)
return wrapper
class base():
def __init__(self, n: int) -> None:
print(f'Base: {n}')
class child(base):
#default_init
def __init__(self, n: int) -> None:
pass
class child2(child):
#default_init
def __init__(self, n: int) -> None:
pass
child2(42)
RecursionError: maximum recursion depth exceeded while calling a Python object.
Part 3/3: Why not just use plain super()?
But why not just use the safe plain super()? Because it doesn't work since the new rebinded __init__ is from outside the class, and super(type(self), self) is required.
def default_init(func):
def wrapper(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
return wrapper
class base():
def __init__(self, n: int) -> None:
print(f'Base: {n}')
class child(base):
#default_init
def __init__(self, n: int) -> None:
pass
child(42)
Errors:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-9-6f580b3839cd> in <module>
13 pass
14
---> 15 child(42)
<ipython-input-9-6f580b3839cd> in wrapper(self, *args, **kwargs)
1 def default_init(func):
2 def wrapper(self, *args, **kwargs) -> None:
----> 3 super().__init__(*args, **kwargs)
4 return wrapper
5
RuntimeError: super(): __class__ cell not found
Background - We CAN AUTO init a parent AND child class!
A lot of answers here and say "This is not the python way, use super().__init__() from the subclass". The question is not asking for the pythonic way, it's comparing to the expected behavior from other languages to python's obviously different one.
The MRO document is pretty and colorful but it's really a TLDR situation and still doesn't quite answer the question, as is often the case in these types of comparisons - "Do it the Python way, because.".
Inherited objects can be overloaded by later declarations in subclasses, a pattern building on #keyvanrm's (https://stackoverflow.com/a/46943772/1112676) answer solves the case where I want to AUTOMATICALLY init a parent class as part of calling a class without explicitly calling super().__init__() in every child class.
In my case where a new team member might be asked to use a boilerplate module template (for making extensions to our application without touching the core application source) which we want to make as bare and easy to adopt without them needing to know or understand the underlying machinery - to only need to know of and use what is provided by the application's base interface which is well documented.
For those who will say "Explicit is better than implicit." I generally agree, however, when coming from many other popular languages inherited automatic initialization is the expected behavior and it is very useful if it can be leveraged for projects where some work on a core application and others work on extending it.
This technique can even pass args/keyword args for init which means pretty much any object can be pushed to the parent and used by the parent class or its relatives.
Example:
class Parent:
def __init__(self, *args, **kwargs):
self.somevar = "test"
self.anothervar = "anothertest"
#important part, call the init surrogate pass through args:
self._init(*args, **kwargs)
#important part, a placeholder init surrogate:
def _init(self, *args, **kwargs):
print("Parent class _init; ", self, args, kwargs)
def some_base_method(self):
print("some base method in Parent")
self.a_new_dict={}
class Child1(Parent):
# when omitted, the parent class's __init__() is run
#def __init__(self):
# pass
#overloading the parent class's _init() surrogate
def _init(self, *args, **kwargs):
print(f"Child1 class _init() overload; ",self, args, kwargs)
self.a_var_set_from_child = "This is a new var!"
class Child2(Parent):
def __init__(self, onevar, twovar, akeyword):
print(f"Child2 class __init__() overload; ", self)
#call some_base_method from parent
self.some_base_method()
#the parent's base method set a_new_dict
print(self.a_new_dict)
class Child3(Parent):
pass
print("\nRunning Parent()")
Parent()
Parent("a string", "something else", akeyword="a kwarg")
print("\nRunning Child1(), keep Parent.__init__(), overload surrogate Parent._init()")
Child1()
Child1("a string", "something else", akeyword="a kwarg")
print("\nRunning Child2(), overload Parent.__init__()")
#Child2() # __init__() requires arguments
Child2("a string", "something else", akeyword="a kwarg")
print("\nRunning Child3(), empty class, inherits everything")
Child3().some_base_method()
Output:
Running Parent()
Parent class _init; <__main__.Parent object at 0x7f84a721fdc0> () {}
Parent class _init; <__main__.Parent object at 0x7f84a721fdc0> ('a string', 'something else') {'akeyword': 'a kwarg'}
Running Child1(), keep Parent.__init__(), overload surrogate Parent._init()
Child1 class _init() overload; <__main__.Child1 object at 0x7f84a721fdc0> () {}
Child1 class _init() overload; <__main__.Child1 object at 0x7f84a721fdc0> ('a string', 'something else') {'akeyword': 'a kwarg'}
Running Child2(), overload Parent.__init__()
Child2 class __init__() overload; <__main__.Child2 object at 0x7f84a721fdc0>
some base method in Parent
{}
Running Child3(), empty class, inherits everything, access things set by other children
Parent class _init; <__main__.Child3 object at 0x7f84a721fdc0> () {}
some base method in Parent
As one can see, the overloaded definition(s) take the place of those declared in Parent class but can still be called BY the Parent class thereby allowing one to emulate the classical implicit inheritance initialization behavior Parent and Child classes both initialize without needing to explicitly invoke the Parent's init() from the Child class.
Personally, I call the surrogate _init() method main() because it makes sense to me when switching between C++ and Python for example since it is a function that will be automatically run for any subclass of Parent (the last declared definition of main(), that is).
Related
The case is such that I have an abstract class and a few child classes implementing it.
class Parent(metaclass=ABCMeta):
#abstract_method
def first_method(self, *args, **kwargs):
raise NotImplementedError()
#abstract_method
def second_method(self, *args, **kwargs):
raise NotImplementedError()
class Child(Parent):
def first_method(self, *args, **kwargs):
print('First method of the child class called!')
def second_method(self, *args, **kwargs):
print('Second method of the child class called!')
My goal is to make some kind of decorator, which will be used on methods of any child of the Parent class. I need this because every method make some kind of preparation before actually doing something, and this preparation is absolutely the same in all methods of all childs of the Parent class. Like:
class Child(Parent):
def first_method(self, *args, **kwargs):
print('Preparation!')
print('First method of the child class called!')
def second_method(self, *args, **kwargs):
print('Preparation!')
print('Second method of the child class called!')
The first thing came to my mind is to use Parent class method implementation: just remove "raise NotImplementedError()" and put some functionality, and then in child classes I would call, for example, super().first_method(self, *args, **kwargs) in the beginning of each method. It is good, but I also would want to return some data from the Parent method, and it would look weird when parent method and child method return something different in declaration. Not to mention that I would probably want to do some post-processing work after the method, so then I would need 2 different functions: for the beginning and after the performing the script.
The next thing I came up with is making MetaClass.
Just implement all the decoration of methods in the new MetaClass during creating a class, and pass the newly generated data which is used in child methods to them in kwargs.
This is the closest solution to my goal, but it feels wrong anyway. Because it is not explicit that some kwargs will be passed to child methods, and if you are new to this code, then you need to do some researches to understand how it works. I feel like I overengineering or so.
So the question: is there any pattern or something along these lines to implement this functionality?
Probably you can advise something better for my case?
Thank you a lot in advance!
So, existing patterns apart: I won't know if this has an specific name, what you need, that would be a "pattern" is the use of "slots": that is - you document special named methods that will be called as part of the execution of another method. This other method then performs its setup code, checks if the slotted method (usually identifiable by name) exists, call them, with a plain simple method call, which will run the most specialized version of it, even if the special method that calls the slots is in the base class, and you are on a big class-inheritance hierarchy.
One plain example of this pattern is the way Python instantiates objects: what one actually invokes calling the class with the same syntax that is used for function calls (MyClass()) is that class's class (its metaclass) __call__ method. (Usally type.__call__). In Python's code for type.__call__ the class' __new__ method is called, then the class' __init__ method is called and finally the value returned by the first call, to __new__ is returned. A custom metaclass can modify __call__ to run whatever code it wants before, between, or after these two calls.
So, if this was not Python, all you'd need is to spec down this, and document that these methods should not be called directly, but rather through an "entry point" method - which could simply feature an "ep_" prefix. These would have to be fixed and hardcoded on a baseclass, and you'd need one for each of the methods you want to prefix/postfix code to.
class Base(ABC):
def ep_first_method(self, *args, **kw);
# prefix code...
ret_val = self.first_method(*args, **kw)
# postfix code...
return ret_val
#abstractmethod
def first_method(self):
pass
class Child(Base):
def first_method(self, ...):
...
This being Python, it is easier to add some more magic to avoid code repetition and keep things concise.
One possible thing is to have a special class that, when detecting a method in a child class that should be called as a slot of a wrapper method, like above, to automatically rename that method: this way the entry point methods can feature the same name as the child methods - and better yet, a simple decorator can mark the methods that are meant to be "entrypoints", and inheritance would even work for them.
Basically, when building a new class we check all methods: if any of them has a correspondent part in the calling hierarchy which is marked as an entrypoint, the renaming takes place.
It is more practical if any entrypoint method will take as second parameter (the first being self), a reference for the slotted method to be called.
After some fiddling: the good news is that a custommetaclass is not needed - the __init_subclass__ special method in a baseclass is enough to enable the decorator.
The bad news: due to re-entry iterations in the entry-point triggered by potential calls to "super()" on the final methods, a somewhat intricate heuristic to call the original method in the intermediate classes is needed. I also took care to put some multi-threading protections - although this is not 100% bullet-proof.
import sys
import threading
from functools import wraps
def entrypoint(func):
name = func.__name__
slotted_name = f"_slotted_{name}"
recursion_control = threading.local()
recursion_control.depth = 0
lock = threading.Lock()
#wraps(func)
def wrapper(self, *args, **kw):
slotted_method = getattr(self, slotted_name, None)
if slotted_method is None:
# this check in place of abstractmethod errors. It is only raised when the method is called, though
raise TypeError("Child class {type(self).__name__} did not implement mandatory method {func.__name__}")
# recursion control logic: also handle when the slotted method calls "super",
# not just straightforward recursion
with lock:
recursion_control.depth += 1
if recursion_control.depth == 1:
normal_course = True
else:
normal_course = False
try:
if normal_course:
# runs through entrypoint
result = func(self, slotted_method, *args, **kw)
else:
# we are within a "super()" call - the only way to get the renamed method
# in the correct subclass is to recreate the callee's super, by fetching its
# implicit "__class__" variable.
try:
callee_super = super(sys._getframe(1).f_locals["__class__"], self)
except KeyError:
# callee did not make a "super" call, rather it likely is a recursive function "for real"
callee_super = type(self)
slotted_method = getattr(callee_super, slotted_name)
result = slotted_method(*args, **kw)
finally:
recursion_control.depth -= 1
return result
wrapper.__entrypoint__ = True
return wrapper
class SlottedBase:
def __init_subclass__(cls, *args, **kw):
super().__init_subclass__(*args, **kw)
for name, child_method in tuple(cls.__dict__.items()):
#breakpoint()
if not callable(child_method) or getattr(child_method, "__entrypoint__", None):
continue
for ancestor_cls in cls.__mro__[1:]:
parent_method = getattr(ancestor_cls, name, None)
if parent_method is None:
break
if not getattr(parent_method, "__entrypoint__", False):
continue
# if the code reaches here, this is a method that
# at some point up has been marked as having an entrypoint method: we rename it.
delattr (cls, name)
setattr(cls, f"_slotted_{name}", child_method)
break
# the chaeegs above are inplace, no need to return anything
class Parent(SlottedBase):
#entrypoint
def meth1(self, slotted, a, b):
print(f"at meth 1 entry, with {a=} and {b=}")
result = slotted(a, b)
print("exiting meth1\n")
return result
class Child(Parent):
def meth1(self, a, b):
print(f"at meth 1 on Child, with {a=} and {b=}")
class GrandChild(Child):
def meth1(self, a, b):
print(f"at meth 1 on grandchild, with {a=} and {b=}")
super().meth1(a,b)
class GrandGrandChild(GrandChild):
def meth1(self, a, b):
print(f"at meth 1 on grandgrandchild, with {a=} and {b=}")
super().meth1(a,b)
c = Child()
c.meth1(2, 3)
d = GrandChild()
d.meth1(2, 3)
e = GrandGrandChild()
e.meth1(2, 3)
I have a class inherited from project.task named ProjectTask
The class has a copy method that overrides the copy function from project.task it's named Task
I need to run the base copy function from my class instead of the one of the parents class
this is my class code:
#api.multi
#api.returns('self', lambda value: value.id)
def copy(self, default=None):
if default is None:
default = {}
if not default.get('name'):
default['name'] = self.name.id
return super(ProjectTask, self).copy(default) #<-- I don't want to call the inherited class method I want to call the base class method instead
This is the copy method from the base class (Task)
#api.multi
#api.returns('self', lambda value: value.id)
def copy(self, default=None):
if default is None:
default = {}
if not default.get('name'):
default['name'] = _("%s (copy)") % self.name
return super(Task, self).copy(default) # <-- I want to run this method from my class (ProjectTask) which is the child class
Any advice will be more than welcome
With the parent class implementation you show, calling it with your own default should do what you want, as it will just pass it through to its own parent with no changes. (At least, that's true with the bare method code, I don't know what the odoo decorators do to change things.)
But if you really do need to skip over it for some non-obvious reason, you probably can do it. Generally speaking, these approaches will only work as intended if you don't expect your class to ever be used with multiple inheritance. If your MRO gets complicated, then you really want to be doing the normal thing with super and making all your methods play nicely together.
One option for skipping an inherited method is to directly name the class you want your call to go to (i.e. your grandparent class).
class Base():
def foo(self):
print("Base")
class Parent(Base):
def foo(self):
print("Parent")
super().foo() # super() in Python 3 is equivalent to super(Parent, self)
class Child(Parent):
def foo(self):
print("Child")
Base.foo(self) # call Base.foo directly, we need to pass the self argument ourselves
Another option would be to change the argument you give to super to name the parent class instead of your own class. Usually that's a newbie error, but if that's really what you want, it's allowed (though I'd strongly recommend adding a comment to the code explaining that you really do want that behavior!
class Child(Parent):
def foo(self):
print("Child")
super(Parent, self).foo() # Note: Deliberately skipping over Parent.foo here!
A final note: If you find yourself wanting to skip a parent class's implementation of some of its methods, perhaps you should reconsider if you should really be inheriting from it at all. It may be that you really want to be inheriting from the same base class as it instead, and skipping the middle class altogether. Obviously, this has its own limitations (maybe some library code does type checking for that class), but if you find yourself fighting the inheritance machinery, it may be that you're doing things the hard way, and there's an easier alternative.
I have the following base class and subclass:
class Event:
def __init__(self, sr1=None, foobar=None):
self.sr1 = sr1
self.foobar = foobar
self.state = STATE_NON_EVENT
# Event class wrappers to provide syntatic sugar
class TypeTwoEvent(Event):
def __init__(self, level=None):
self.sr1 = level
self.state = STATE_EVENT_TWO
Further on in my code, I am inspecting an instance of a TypeTwoEvent class, checking for a field I know exists in the base class - I expected it to be defaulted to value None. However, my code raises the following exception:
AttributeError: 'TypeTwoEvent' object has no attribute 'foobar'
I was under the impression that the base class fields would be inherited by the subclass and that creating an instance of a subclass will instantiate the base class (and thus invoke its constructor) ...
What am I missing here? Why does TypeTwoEvent not have a foobar attribute - when the base class from which it is derived has a foobar attribute?
Your subclass should be:
class TypeTwoEvent(Event):
def __init__(self, level=None, *args, **kwargs):
super().__init__(*args, **kwargs)
self.sr1 = level
self.state = STATE_EVENT_TWO
Because you override the __init__ method, so you need to call the parent method if you want the parent behavior to happen.
Remember, __init__ is not a special method dispite its strange name. It's just the method automatically called after the object is created. Otherwise it's an ordinary method, and ordinary inheritance rules apply.
super().__init__(arguments, that, goes, to, parents)
is the syntax to call the parent version of the method.
For *args and **kwargs, it just ensures we catch all additional arguments passed to __init__ and pass it to the parent method, as you child method signature didn't do it and the parent need these arguments to work.
You're overriding the constructor (__init__) of the parent class. To extend it, you need to explicitly call the constructor of the parent with a super() call.
class TypeTwoEvent(Event):
def __init__(self, level=None, **kwargs):
# the super call to set the attributes in the parent class
super().__init__(**kwargs)
# now, extend other attributes
self.sr1 = level
self.state = STATE_EVENT_TWO
Note that the super call is not always at the top of the __init__ method in your sub-class. Its location depends on your situation and logic.
When the instance is created, its __init__ method is called. In this case, that is TypeTwoEvent.__init__. Superclass methods will not be called automatically because that would be immensely confusing.
You should call Event.__init__(self, ...) from TypeTwoEvent.__init__ (or use super, but if you're not familiar with it, read up on it first so you know what you're doing).
You need to call the __init__ method of the base class from the __init__ method of the inherited class.
See here for how to do this.
I've had the same problem, but in my case I put super().__init__() on the bottom of my derived class and that's why it doesn't work. Because I tried to use attributes that are not initialized.
I'm just trying to streamline one of my classes and have introduced some functionality in the same style as the flyweight design pattern.
However, I'm a bit confused as to why __init__ is always called after __new__. I wasn't expecting this. Can anyone tell me why this is happening and how I can implement this functionality otherwise? (Apart from putting the implementation into the __new__ which feels quite hacky.)
Here's an example:
class A(object):
_dict = dict()
def __new__(cls):
if 'key' in A._dict:
print "EXISTS"
return A._dict['key']
else:
print "NEW"
return super(A, cls).__new__(cls)
def __init__(self):
print "INIT"
A._dict['key'] = self
print ""
a1 = A()
a2 = A()
a3 = A()
Outputs:
NEW
INIT
EXISTS
INIT
EXISTS
INIT
Why?
Use __new__ when you need to control
the creation of a new instance.
Use
__init__ when you need to control initialization of a new instance.
__new__ is the first step of instance creation. It's called first, and is
responsible for returning a new
instance of your class.
In contrast,
__init__ doesn't return anything; it's only responsible for initializing the
instance after it's been created.
In general, you shouldn't need to
override __new__ unless you're
subclassing an immutable type like
str, int, unicode or tuple.
From April 2008 post: When to use __new__ vs. __init__? on mail.python.org.
You should consider that what you are trying to do is usually done with a Factory and that's the best way to do it. Using __new__ is not a good clean solution so please consider the usage of a factory. Here's a good example: ActiveState Fᴀᴄᴛᴏʀʏ ᴘᴀᴛᴛᴇʀɴ Recipe.
__new__ is static class method, while __init__ is instance method.
__new__ has to create the instance first, so __init__ can initialize it. Note that __init__ takes self as parameter. Until you create instance there is no self.
Now, I gather, that you're trying to implement singleton pattern in Python. There are a few ways to do that.
Also, as of Python 2.6, you can use class decorators.
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance
#singleton
class MyClass:
...
In most well-known OO languages, an expression like SomeClass(arg1, arg2) will allocate a new instance, initialise the instance's attributes, and then return it.
In most well-known OO languages, the "initialise the instance's attributes" part can be customised for each class by defining a constructor, which is basically just a block of code that operates on the new instance (using the arguments provided to the constructor expression) to set up whatever initial conditions are desired. In Python, this corresponds to the class' __init__ method.
Python's __new__ is nothing more and nothing less than similar per-class customisation of the "allocate a new instance" part. This of course allows you to do unusual things such as returning an existing instance rather than allocating a new one. So in Python, we shouldn't really think of this part as necessarily involving allocation; all that we require is that __new__ comes up with a suitable instance from somewhere.
But it's still only half of the job, and there's no way for the Python system to know that sometimes you want to run the other half of the job (__init__) afterwards and sometimes you don't. If you want that behavior, you have to say so explicitly.
Often, you can refactor so you only need __new__, or so you don't need __new__, or so that __init__ behaves differently on an already-initialised object. But if you really want to, Python does actually allow you to redefine "the job", so that SomeClass(arg1, arg2) doesn't necessarily call __new__ followed by __init__. To do this, you need to create a metaclass, and define its __call__ method.
A metaclass is just the class of a class. And a class' __call__ method controls what happens when you call instances of the class. So a metaclass' __call__ method controls what happens when you call a class; i.e. it allows you to redefine the instance-creation mechanism from start to finish. This is the level at which you can most elegantly implement a completely non-standard instance creation process such as the singleton pattern. In fact, with less than 10 lines of code you can implement a Singleton metaclass that then doesn't even require you to futz with __new__ at all, and can turn any otherwise-normal class into a singleton by simply adding __metaclass__ = Singleton!
class Singleton(type):
def __init__(self, *args, **kwargs):
super(Singleton, self).__init__(*args, **kwargs)
self.__instance = None
def __call__(self, *args, **kwargs):
if self.__instance is None:
self.__instance = super(Singleton, self).__call__(*args, **kwargs)
return self.__instance
However this is probably deeper magic than is really warranted for this situation!
To quote the documentation:
Typical implementations create a new instance of the class by invoking
the superclass's __new__() method using "super(currentclass,
cls).__new__(cls[, ...])"with appropriate arguments and then
modifying the newly-created instance as necessary before returning it.
...
If __new__() does not return an instance of cls, then the new
instance's __init__() method will not be invoked.
__new__() is intended mainly to allow subclasses of immutable
types (like int, str, or tuple) to customize instance creation.
I realize that this question is quite old but I had a similar issue.
The following did what I wanted:
class Agent(object):
_agents = dict()
def __new__(cls, *p):
number = p[0]
if not number in cls._agents:
cls._agents[number] = object.__new__(cls)
return cls._agents[number]
def __init__(self, number):
self.number = number
def __eq__(self, rhs):
return self.number == rhs.number
Agent("a") is Agent("a") == True
I used this page as a resource http://infohost.nmt.edu/tcc/help/pubs/python/web/new-new-method.html
When __new__ returns instance of the same class, __init__ is run afterwards on returned object. I.e. you can NOT use __new__ to prevent __init__ from being run. Even if you return previously created object from __new__, it will be double (triple, etc...) initialized by __init__ again and again.
Here is the generic approach to Singleton pattern which extends vartec answer above and fixes it:
def SingletonClass(cls):
class Single(cls):
__doc__ = cls.__doc__
_initialized = False
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Single, cls).__new__(cls, *args, **kwargs)
return cls._instance
def __init__(self, *args, **kwargs):
if self._initialized:
return
super(Single, self).__init__(*args, **kwargs)
self.__class__._initialized = True # Its crucial to set this variable on the class!
return Single
Full story is here.
Another approach, which in fact involves __new__ is to use classmethods:
class Singleton(object):
__initialized = False
def __new__(cls, *args, **kwargs):
if not cls.__initialized:
cls.__init__(*args, **kwargs)
cls.__initialized = True
return cls
class MyClass(Singleton):
#classmethod
def __init__(cls, x, y):
print "init is here"
#classmethod
def do(cls):
print "doing stuff"
Please pay attention, that with this approach you need to decorate ALL of your methods with #classmethod, because you'll never use any real instance of MyClass.
I think the simple answer to this question is that, if __new__ returns a value that is the same type as the class, the __init__ function executes, otherwise it won't. In this case your code returns A._dict('key') which is the same class as cls, so __init__ will be executed.
class M(type):
_dict = {}
def __call__(cls, key):
if key in cls._dict:
print 'EXISTS'
return cls._dict[key]
else:
print 'NEW'
instance = super(M, cls).__call__(key)
cls._dict[key] = instance
return instance
class A(object):
__metaclass__ = M
def __init__(self, key):
print 'INIT'
self.key = key
print
a1 = A('aaa')
a2 = A('bbb')
a3 = A('aaa')
outputs:
NEW
INIT
NEW
INIT
EXISTS
NB As a side effect M._dict property automatically becomes accessible from A as A._dict so take care not to overwrite it incidentally.
An update to #AntonyHatchkins answer, you probably want a separate dictionary of instances for each class of the metatype, meaning that you should have an __init__ method in the metaclass to initialize your class object with that dictionary instead of making it global across all the classes.
class MetaQuasiSingleton(type):
def __init__(cls, name, bases, attibutes):
cls._dict = {}
def __call__(cls, key):
if key in cls._dict:
print('EXISTS')
instance = cls._dict[key]
else:
print('NEW')
instance = super().__call__(key)
cls._dict[key] = instance
return instance
class A(metaclass=MetaQuasiSingleton):
def __init__(self, key):
print 'INIT'
self.key = key
print()
I have gone ahead and updated the original code with an __init__ method and changed the syntax to Python 3 notation (no-arg call to super and metaclass in the class arguments instead of as an attribute).
Either way, the important point here is that your class initializer (__call__ method) will not execute either __new__ or __init__ if the key is found. This is much cleaner than using __new__, which requires you to mark the object if you want to skip the default __init__ step.
__new__ should return a new, blank instance of a class. __init__ is then called to initialise that instance. You're not calling __init__ in the "NEW" case of __new__, so it's being called for you. The code that is calling __new__ doesn't keep track of whether __init__ has been called on a particular instance or not nor should it, because you're doing something very unusual here.
You could add an attribute to the object in the __init__ function to indicate that it's been initialised. Check for the existence of that attribute as the first thing in __init__ and don't proceed any further if it has been.
Digging little deeper into that!
The type of a generic class in CPython is type and its base class is Object (Unless you explicitly define another base class like a metaclass). The sequence of low level calls can be found here. The first method called is the type_call which then calls tp_new and then tp_init.
The interesting part here is that tp_new will call the Object's (base class) new method object_new which does a tp_alloc (PyType_GenericAlloc) which allocates the memory for the object :)
At that point the object is created in memory and then the __init__ method gets called. If __init__ is not implemented in your class then the object_init gets called and it does nothing :)
Then type_call just returns the object which binds to your variable.
One should look at __init__ as a simple constructor in traditional OO languages. For example, if you are familiar with Java or C++, the constructor is passed a pointer to its own instance implicitly. In the case of Java, it is the this variable. If one were to inspect the byte code generated for Java, one would notice two calls. The first call is to an "new" method, and then next call is to the init method (which is the actual call to the user defined constructor). This two step process enables creation of the actual instance before calling the constructor method of the class which is just another method of that instance.
Now, in the case of Python, __new__ is a added facility that is accessible to the user. Java does not provide that flexibility, due to its typed nature. If a language provided that facility, then the implementor of __new__ could do many things in that method before returning the instance, including creating a totally new instance of a unrelated object in some cases. And, this approach also works out well for especially for immutable types in the case of Python.
However, I'm a bit confused as to why __init__ is always called after __new__.
I think the C++ analogy would be useful here:
__new__ simply allocates memory for the object. The instance variables of an object needs memory to hold it, and this is what the step __new__ would do.
__init__ initialize the internal variables of the object to specific values (could be default).
Referring to this doc:
When subclassing immutable built-in types like numbers and strings,
and occasionally in other situations, the static method __new__ comes
in handy. __new__ is the first step in instance construction, invoked
before __init__.
The __new__ method is called with the class as its
first argument; its responsibility is to return a new instance of that
class.
Compare this to __init__: __init__ is called with an instance
as its first argument, and it doesn't return anything; its
responsibility is to initialize the instance.
There are situations
where a new instance is created without calling __init__ (for example
when the instance is loaded from a pickle). There is no way to create
a new instance without calling __new__ (although in some cases you can
get away with calling a base class's __new__).
Regarding what you wish to achieve, there also in same doc info about Singleton pattern
class Singleton(object):
def __new__(cls, *args, **kwds):
it = cls.__dict__.get("__it__")
if it is not None:
return it
cls.__it__ = it = object.__new__(cls)
it.init(*args, **kwds)
return it
def init(self, *args, **kwds):
pass
you may also use this implementation from PEP 318, using a decorator
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance
#singleton
class MyClass:
...
Now I've got the same problem, and for some reasons I decided to avoid decorators, factories and metaclasses. I did it like this:
Main file
def _alt(func):
import functools
#functools.wraps(func)
def init(self, *p, **k):
if hasattr(self, "parent_initialized"):
return
else:
self.parent_initialized = True
func(self, *p, **k)
return init
class Parent:
# Empty dictionary, shouldn't ever be filled with anything else
parent_cache = {}
def __new__(cls, n, *args, **kwargs):
# Checks if object with this ID (n) has been created
if n in cls.parent_cache:
# It was, return it
return cls.parent_cache[n]
else:
# Check if it was modified by this function
if not hasattr(cls, "parent_modified"):
# Add the attribute
cls.parent_modified = True
cls.parent_cache = {}
# Apply it
cls.__init__ = _alt(cls.__init__)
# Get the instance
obj = super().__new__(cls)
# Push it to cache
cls.parent_cache[n] = obj
# Return it
return obj
Example classes
class A(Parent):
def __init__(self, n):
print("A.__init__", n)
class B(Parent):
def __init__(self, n):
print("B.__init__", n)
In use
>>> A(1)
A.__init__ 1 # First A(1) initialized
<__main__.A object at 0x000001A73A4A2E48>
>>> A(1) # Returned previous A(1)
<__main__.A object at 0x000001A73A4A2E48>
>>> A(2)
A.__init__ 2 # First A(2) initialized
<__main__.A object at 0x000001A7395D9C88>
>>> B(2)
B.__init__ 2 # B class doesn't collide with A, thanks to separate cache
<__main__.B object at 0x000001A73951B080>
Warning: You shouldn't initialize Parent, it will collide with other classes - unless you defined separate cache in each of the children, that's not what we want.
Warning: It seems a class with Parent as grandparent behaves weird. [Unverified]
Try it online!
The __init__ is called after __new__ so that when you override it in a subclass, your added code will still get called.
If you are trying to subclass a class that already has a __new__, someone unaware of this might start by adapting the __init__ and forwarding the call down to the subclass __init__. This convention of calling __init__ after __new__ helps that work as expected.
The __init__ still needs to allow for any parameters the superclass __new__ needed, but failing to do so will usually create a clear runtime error. And the __new__ should probably explicitly allow for *args and '**kw', to make it clear that extension is OK.
It is generally bad form to have both __new__ and __init__ in the same class at the same level of inheritance, because of the behavior the original poster described.
However, I'm a bit confused as to why __init__ is always called after __new__.
Not much of a reason other than that it just is done that way. __new__ doesn't have the responsibility of initializing the class, some other method does (__call__, possibly-- I don't know for sure).
I wasn't expecting this. Can anyone tell me why this is happening and how I implement this functionality otherwise? (apart from putting the implementation into the __new__ which feels quite hacky).
You could have __init__ do nothing if it's already been initialized, or you could write a new metaclass with a new __call__ that only calls __init__ on new instances, and otherwise just returns __new__(...).
The simple reason is that the new is used for creating an instance, while init is used for initializing the instance. Before initializing, the instance should be created first. That's why new should be called before init.
When instantiating a class, first, __new__() is called to create the instance of a class, then __init__() is called to initialize the instance.
__new__():
Called to create a new instance of class cls. ...
If __new__() is invoked during object construction and it returns an
instance of cls, then the new instance’s __init__() method will be
invoked like __init__(self[, ...]), ...
__init__():
Called after the instance has been created (by __new__()), ...
Because __new__() and __init__() work together in constructing objects
(__new__() to create it, and __init__() to customize it), ...
For example, when instantiating Teacher class, first, __new__() is called to create the instance of Teacher class, then __init__() is called to initialize the instance as shown below:
class Teacher:
def __init__(self, name):
self.name = name
class Student:
def __init__(self, name):
self.name = name
obj = Teacher("John") # Instantiation
print(obj.name)
This is the output:
<class '__main__.Teacher'>
John
And, using __new__() of the instance of Teacher class, we can create the instance of Student class as shown below:
# ...
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student) # Creates the instance of "Student" class
print(type(obj))
Now, the instance of Student class is created as shown below:
<class '__main__.Teacher'>
<__main__.Teacher object at 0x7f4e3950bf10>
<class '__main__.Student'> # Here
Next, if we try to get the value of name variable from **the instance of Student class as shown below:
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student)
print(type(obj))
print(obj.name) # Tries to get the value of "name" variable
The error below occurs because the instance of Student class has not been initialized by __init__() yet:
AttributeError: 'Student' object has no attribute 'name'
So, we initialize the instance of Student class as shown below:
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student)
print(type(obj))
obj.__init__("Tom") # Initializes the instance of "Student" class
print(obj.name)
Then, we can get the value of name variable from the instance of Student class as shown below:
<class '__main__.Teacher'>
John
<class '__main__.Student'>
Tom # Here
People have already detailed the question and answer both use some examples like singleton etc. See the code below:
__instance = None
def __new__(cls):
if cls.__instance is None:
cls.__instance = object.__new__(cls)
return cls.__instance
I got the above code from this link, it has detailed overview of new vs init. Worth reading!
I'm just trying to streamline one of my classes and have introduced some functionality in the same style as the flyweight design pattern.
However, I'm a bit confused as to why __init__ is always called after __new__. I wasn't expecting this. Can anyone tell me why this is happening and how I can implement this functionality otherwise? (Apart from putting the implementation into the __new__ which feels quite hacky.)
Here's an example:
class A(object):
_dict = dict()
def __new__(cls):
if 'key' in A._dict:
print "EXISTS"
return A._dict['key']
else:
print "NEW"
return super(A, cls).__new__(cls)
def __init__(self):
print "INIT"
A._dict['key'] = self
print ""
a1 = A()
a2 = A()
a3 = A()
Outputs:
NEW
INIT
EXISTS
INIT
EXISTS
INIT
Why?
Use __new__ when you need to control
the creation of a new instance.
Use
__init__ when you need to control initialization of a new instance.
__new__ is the first step of instance creation. It's called first, and is
responsible for returning a new
instance of your class.
In contrast,
__init__ doesn't return anything; it's only responsible for initializing the
instance after it's been created.
In general, you shouldn't need to
override __new__ unless you're
subclassing an immutable type like
str, int, unicode or tuple.
From April 2008 post: When to use __new__ vs. __init__? on mail.python.org.
You should consider that what you are trying to do is usually done with a Factory and that's the best way to do it. Using __new__ is not a good clean solution so please consider the usage of a factory. Here's a good example: ActiveState Fᴀᴄᴛᴏʀʏ ᴘᴀᴛᴛᴇʀɴ Recipe.
__new__ is static class method, while __init__ is instance method.
__new__ has to create the instance first, so __init__ can initialize it. Note that __init__ takes self as parameter. Until you create instance there is no self.
Now, I gather, that you're trying to implement singleton pattern in Python. There are a few ways to do that.
Also, as of Python 2.6, you can use class decorators.
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance
#singleton
class MyClass:
...
In most well-known OO languages, an expression like SomeClass(arg1, arg2) will allocate a new instance, initialise the instance's attributes, and then return it.
In most well-known OO languages, the "initialise the instance's attributes" part can be customised for each class by defining a constructor, which is basically just a block of code that operates on the new instance (using the arguments provided to the constructor expression) to set up whatever initial conditions are desired. In Python, this corresponds to the class' __init__ method.
Python's __new__ is nothing more and nothing less than similar per-class customisation of the "allocate a new instance" part. This of course allows you to do unusual things such as returning an existing instance rather than allocating a new one. So in Python, we shouldn't really think of this part as necessarily involving allocation; all that we require is that __new__ comes up with a suitable instance from somewhere.
But it's still only half of the job, and there's no way for the Python system to know that sometimes you want to run the other half of the job (__init__) afterwards and sometimes you don't. If you want that behavior, you have to say so explicitly.
Often, you can refactor so you only need __new__, or so you don't need __new__, or so that __init__ behaves differently on an already-initialised object. But if you really want to, Python does actually allow you to redefine "the job", so that SomeClass(arg1, arg2) doesn't necessarily call __new__ followed by __init__. To do this, you need to create a metaclass, and define its __call__ method.
A metaclass is just the class of a class. And a class' __call__ method controls what happens when you call instances of the class. So a metaclass' __call__ method controls what happens when you call a class; i.e. it allows you to redefine the instance-creation mechanism from start to finish. This is the level at which you can most elegantly implement a completely non-standard instance creation process such as the singleton pattern. In fact, with less than 10 lines of code you can implement a Singleton metaclass that then doesn't even require you to futz with __new__ at all, and can turn any otherwise-normal class into a singleton by simply adding __metaclass__ = Singleton!
class Singleton(type):
def __init__(self, *args, **kwargs):
super(Singleton, self).__init__(*args, **kwargs)
self.__instance = None
def __call__(self, *args, **kwargs):
if self.__instance is None:
self.__instance = super(Singleton, self).__call__(*args, **kwargs)
return self.__instance
However this is probably deeper magic than is really warranted for this situation!
To quote the documentation:
Typical implementations create a new instance of the class by invoking
the superclass's __new__() method using "super(currentclass,
cls).__new__(cls[, ...])"with appropriate arguments and then
modifying the newly-created instance as necessary before returning it.
...
If __new__() does not return an instance of cls, then the new
instance's __init__() method will not be invoked.
__new__() is intended mainly to allow subclasses of immutable
types (like int, str, or tuple) to customize instance creation.
I realize that this question is quite old but I had a similar issue.
The following did what I wanted:
class Agent(object):
_agents = dict()
def __new__(cls, *p):
number = p[0]
if not number in cls._agents:
cls._agents[number] = object.__new__(cls)
return cls._agents[number]
def __init__(self, number):
self.number = number
def __eq__(self, rhs):
return self.number == rhs.number
Agent("a") is Agent("a") == True
I used this page as a resource http://infohost.nmt.edu/tcc/help/pubs/python/web/new-new-method.html
When __new__ returns instance of the same class, __init__ is run afterwards on returned object. I.e. you can NOT use __new__ to prevent __init__ from being run. Even if you return previously created object from __new__, it will be double (triple, etc...) initialized by __init__ again and again.
Here is the generic approach to Singleton pattern which extends vartec answer above and fixes it:
def SingletonClass(cls):
class Single(cls):
__doc__ = cls.__doc__
_initialized = False
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Single, cls).__new__(cls, *args, **kwargs)
return cls._instance
def __init__(self, *args, **kwargs):
if self._initialized:
return
super(Single, self).__init__(*args, **kwargs)
self.__class__._initialized = True # Its crucial to set this variable on the class!
return Single
Full story is here.
Another approach, which in fact involves __new__ is to use classmethods:
class Singleton(object):
__initialized = False
def __new__(cls, *args, **kwargs):
if not cls.__initialized:
cls.__init__(*args, **kwargs)
cls.__initialized = True
return cls
class MyClass(Singleton):
#classmethod
def __init__(cls, x, y):
print "init is here"
#classmethod
def do(cls):
print "doing stuff"
Please pay attention, that with this approach you need to decorate ALL of your methods with #classmethod, because you'll never use any real instance of MyClass.
I think the simple answer to this question is that, if __new__ returns a value that is the same type as the class, the __init__ function executes, otherwise it won't. In this case your code returns A._dict('key') which is the same class as cls, so __init__ will be executed.
class M(type):
_dict = {}
def __call__(cls, key):
if key in cls._dict:
print 'EXISTS'
return cls._dict[key]
else:
print 'NEW'
instance = super(M, cls).__call__(key)
cls._dict[key] = instance
return instance
class A(object):
__metaclass__ = M
def __init__(self, key):
print 'INIT'
self.key = key
print
a1 = A('aaa')
a2 = A('bbb')
a3 = A('aaa')
outputs:
NEW
INIT
NEW
INIT
EXISTS
NB As a side effect M._dict property automatically becomes accessible from A as A._dict so take care not to overwrite it incidentally.
An update to #AntonyHatchkins answer, you probably want a separate dictionary of instances for each class of the metatype, meaning that you should have an __init__ method in the metaclass to initialize your class object with that dictionary instead of making it global across all the classes.
class MetaQuasiSingleton(type):
def __init__(cls, name, bases, attibutes):
cls._dict = {}
def __call__(cls, key):
if key in cls._dict:
print('EXISTS')
instance = cls._dict[key]
else:
print('NEW')
instance = super().__call__(key)
cls._dict[key] = instance
return instance
class A(metaclass=MetaQuasiSingleton):
def __init__(self, key):
print 'INIT'
self.key = key
print()
I have gone ahead and updated the original code with an __init__ method and changed the syntax to Python 3 notation (no-arg call to super and metaclass in the class arguments instead of as an attribute).
Either way, the important point here is that your class initializer (__call__ method) will not execute either __new__ or __init__ if the key is found. This is much cleaner than using __new__, which requires you to mark the object if you want to skip the default __init__ step.
__new__ should return a new, blank instance of a class. __init__ is then called to initialise that instance. You're not calling __init__ in the "NEW" case of __new__, so it's being called for you. The code that is calling __new__ doesn't keep track of whether __init__ has been called on a particular instance or not nor should it, because you're doing something very unusual here.
You could add an attribute to the object in the __init__ function to indicate that it's been initialised. Check for the existence of that attribute as the first thing in __init__ and don't proceed any further if it has been.
Digging little deeper into that!
The type of a generic class in CPython is type and its base class is Object (Unless you explicitly define another base class like a metaclass). The sequence of low level calls can be found here. The first method called is the type_call which then calls tp_new and then tp_init.
The interesting part here is that tp_new will call the Object's (base class) new method object_new which does a tp_alloc (PyType_GenericAlloc) which allocates the memory for the object :)
At that point the object is created in memory and then the __init__ method gets called. If __init__ is not implemented in your class then the object_init gets called and it does nothing :)
Then type_call just returns the object which binds to your variable.
One should look at __init__ as a simple constructor in traditional OO languages. For example, if you are familiar with Java or C++, the constructor is passed a pointer to its own instance implicitly. In the case of Java, it is the this variable. If one were to inspect the byte code generated for Java, one would notice two calls. The first call is to an "new" method, and then next call is to the init method (which is the actual call to the user defined constructor). This two step process enables creation of the actual instance before calling the constructor method of the class which is just another method of that instance.
Now, in the case of Python, __new__ is a added facility that is accessible to the user. Java does not provide that flexibility, due to its typed nature. If a language provided that facility, then the implementor of __new__ could do many things in that method before returning the instance, including creating a totally new instance of a unrelated object in some cases. And, this approach also works out well for especially for immutable types in the case of Python.
However, I'm a bit confused as to why __init__ is always called after __new__.
I think the C++ analogy would be useful here:
__new__ simply allocates memory for the object. The instance variables of an object needs memory to hold it, and this is what the step __new__ would do.
__init__ initialize the internal variables of the object to specific values (could be default).
Referring to this doc:
When subclassing immutable built-in types like numbers and strings,
and occasionally in other situations, the static method __new__ comes
in handy. __new__ is the first step in instance construction, invoked
before __init__.
The __new__ method is called with the class as its
first argument; its responsibility is to return a new instance of that
class.
Compare this to __init__: __init__ is called with an instance
as its first argument, and it doesn't return anything; its
responsibility is to initialize the instance.
There are situations
where a new instance is created without calling __init__ (for example
when the instance is loaded from a pickle). There is no way to create
a new instance without calling __new__ (although in some cases you can
get away with calling a base class's __new__).
Regarding what you wish to achieve, there also in same doc info about Singleton pattern
class Singleton(object):
def __new__(cls, *args, **kwds):
it = cls.__dict__.get("__it__")
if it is not None:
return it
cls.__it__ = it = object.__new__(cls)
it.init(*args, **kwds)
return it
def init(self, *args, **kwds):
pass
you may also use this implementation from PEP 318, using a decorator
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance
#singleton
class MyClass:
...
Now I've got the same problem, and for some reasons I decided to avoid decorators, factories and metaclasses. I did it like this:
Main file
def _alt(func):
import functools
#functools.wraps(func)
def init(self, *p, **k):
if hasattr(self, "parent_initialized"):
return
else:
self.parent_initialized = True
func(self, *p, **k)
return init
class Parent:
# Empty dictionary, shouldn't ever be filled with anything else
parent_cache = {}
def __new__(cls, n, *args, **kwargs):
# Checks if object with this ID (n) has been created
if n in cls.parent_cache:
# It was, return it
return cls.parent_cache[n]
else:
# Check if it was modified by this function
if not hasattr(cls, "parent_modified"):
# Add the attribute
cls.parent_modified = True
cls.parent_cache = {}
# Apply it
cls.__init__ = _alt(cls.__init__)
# Get the instance
obj = super().__new__(cls)
# Push it to cache
cls.parent_cache[n] = obj
# Return it
return obj
Example classes
class A(Parent):
def __init__(self, n):
print("A.__init__", n)
class B(Parent):
def __init__(self, n):
print("B.__init__", n)
In use
>>> A(1)
A.__init__ 1 # First A(1) initialized
<__main__.A object at 0x000001A73A4A2E48>
>>> A(1) # Returned previous A(1)
<__main__.A object at 0x000001A73A4A2E48>
>>> A(2)
A.__init__ 2 # First A(2) initialized
<__main__.A object at 0x000001A7395D9C88>
>>> B(2)
B.__init__ 2 # B class doesn't collide with A, thanks to separate cache
<__main__.B object at 0x000001A73951B080>
Warning: You shouldn't initialize Parent, it will collide with other classes - unless you defined separate cache in each of the children, that's not what we want.
Warning: It seems a class with Parent as grandparent behaves weird. [Unverified]
Try it online!
The __init__ is called after __new__ so that when you override it in a subclass, your added code will still get called.
If you are trying to subclass a class that already has a __new__, someone unaware of this might start by adapting the __init__ and forwarding the call down to the subclass __init__. This convention of calling __init__ after __new__ helps that work as expected.
The __init__ still needs to allow for any parameters the superclass __new__ needed, but failing to do so will usually create a clear runtime error. And the __new__ should probably explicitly allow for *args and '**kw', to make it clear that extension is OK.
It is generally bad form to have both __new__ and __init__ in the same class at the same level of inheritance, because of the behavior the original poster described.
However, I'm a bit confused as to why __init__ is always called after __new__.
Not much of a reason other than that it just is done that way. __new__ doesn't have the responsibility of initializing the class, some other method does (__call__, possibly-- I don't know for sure).
I wasn't expecting this. Can anyone tell me why this is happening and how I implement this functionality otherwise? (apart from putting the implementation into the __new__ which feels quite hacky).
You could have __init__ do nothing if it's already been initialized, or you could write a new metaclass with a new __call__ that only calls __init__ on new instances, and otherwise just returns __new__(...).
The simple reason is that the new is used for creating an instance, while init is used for initializing the instance. Before initializing, the instance should be created first. That's why new should be called before init.
When instantiating a class, first, __new__() is called to create the instance of a class, then __init__() is called to initialize the instance.
__new__():
Called to create a new instance of class cls. ...
If __new__() is invoked during object construction and it returns an
instance of cls, then the new instance’s __init__() method will be
invoked like __init__(self[, ...]), ...
__init__():
Called after the instance has been created (by __new__()), ...
Because __new__() and __init__() work together in constructing objects
(__new__() to create it, and __init__() to customize it), ...
For example, when instantiating Teacher class, first, __new__() is called to create the instance of Teacher class, then __init__() is called to initialize the instance as shown below:
class Teacher:
def __init__(self, name):
self.name = name
class Student:
def __init__(self, name):
self.name = name
obj = Teacher("John") # Instantiation
print(obj.name)
This is the output:
<class '__main__.Teacher'>
John
And, using __new__() of the instance of Teacher class, we can create the instance of Student class as shown below:
# ...
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student) # Creates the instance of "Student" class
print(type(obj))
Now, the instance of Student class is created as shown below:
<class '__main__.Teacher'>
<__main__.Teacher object at 0x7f4e3950bf10>
<class '__main__.Student'> # Here
Next, if we try to get the value of name variable from **the instance of Student class as shown below:
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student)
print(type(obj))
print(obj.name) # Tries to get the value of "name" variable
The error below occurs because the instance of Student class has not been initialized by __init__() yet:
AttributeError: 'Student' object has no attribute 'name'
So, we initialize the instance of Student class as shown below:
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student)
print(type(obj))
obj.__init__("Tom") # Initializes the instance of "Student" class
print(obj.name)
Then, we can get the value of name variable from the instance of Student class as shown below:
<class '__main__.Teacher'>
John
<class '__main__.Student'>
Tom # Here
People have already detailed the question and answer both use some examples like singleton etc. See the code below:
__instance = None
def __new__(cls):
if cls.__instance is None:
cls.__instance = object.__new__(cls)
return cls.__instance
I got the above code from this link, it has detailed overview of new vs init. Worth reading!