I'm trying to look under the hood in idle to wrap my head around python custom classes and how they are stored in memory. Suppose I have the following code:
class Point:
pass
x=Point()
print(x)
Given the following output:
<__main__.Point object at 0x000002A3A071DF60>
I know that since my class consists of no code, when I create an object of type Point, an object of type object is implicitly created from which the Point object x inherits such methods as __str__ etc. However, I cant seem to see the connection ie. when I type dir(x), I dont see any attribute that stores a reference to an object of type object. Am I misunderstanding how it works or is there some attribute that I am unaware of?
There is no (direct) instance of type object involved. When you call Point(), a Point instance is created, and nothing else. Inheritance is a mechanism that operates on classes, not on instances.
Python objects have two properties: Their type and their value. The type of x is Point. The value of x is represented by its __dict__, which is empty (because it has no attributes):
>>> type(x)
<class '__main__.Point'>
>>> x.__dict__
{}
There is no reference to an object instance, because no such instance exists. The inheritance from object manifests in the class Point:
>>> Point.__base__
<class 'object'>
>>> Point.mro()
[<class '__main__.Point'>, <class 'object'>]
So the reason why x can use object methods is because Point inherits from object, not because there's a hidden object instance stored in x. You can see that Point has access to all object methods:
>>> Point.__str__ is object.__str__
True
>>> Point.__init__ is object.__init__
True
# etc...
So, by proxy, x has access to all object methods as well.
you may be looking for Point.__mro__? the method resolution order of the class Point. it tells you in what order python will go looking for methods if you call x.method. in your case this will be:
print(Point.__mro__) # or Point.mro()
# (<class '__main__.Point'>, <class 'object'>)
I'm doing Code Academy's tutorials on Python, and I'm a bit confused about the definition of method and function. From the tutorial:
You already know about some of the built-in functions we've used on (or to create) strings, such as .upper(), .lower(), str(), and len().
Coming from C++, I would think .upper() and .lower() would be called methods and len() and str() functions. In the tutorial, the terms seem to be used interchangeably.
Does Python distinguish between methods and functions in the way C++ does?
Unlike Difference between a method and a function, I'm asking about the particulars of Python. The terms 'method' and 'function' do not seem to always follow the definition given in the accepted answer of the linked question.
Needs Attention: This answer seems to be outdated. Check this
A function is a callable object in Python, i.e. can be called using the call operator (though other objects can emulate a function by implementing __call__). For example:
>>> def a(): pass
>>> a
<function a at 0x107063aa0>
>>> type(a)
<type 'function'>
A method is a special class of function, one that can be bound or unbound.
>>> class A:
... def a(self): pass
>>> A.a
<unbound method A.a>
>>> type(A.a)
<type 'instancemethod'>
>>> A().a
<bound method A.a of <__main__.A instance at 0x107070d88>>
>>> type(A().a)
<type 'instancemethod'>
Of course, an unbound method cannot be called (at least not directly without passing an instance as an argument):
>>> A.a()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method a() must be called with A instance as first argument (got nothing instead)
In Python, in most cases, you won't notice the difference between a bound method, a function, or a callable object (i.e. an object that implements __call__), or a class constructor. They all look the same, they just have different naming conventions. Under the hood, the objects may look vastly different though.
This means that a bound method can be used as a function, this is one of the many small things that makes Python so powerful
>>> b = A().a
>>> b()
It also means that even though there is a fundamental difference between len(...) and str(...) (the latter is a type constructor), you won't notice the difference until you dig a little deeper:
>>> len
<built-in function len>
>>> str
<type 'str'>
If you still don’t understand how methods work, a look at the
implementation can perhaps clarify matters. When an instance attribute
is referenced that isn’t a data attribute, its class is searched. If
the name denotes a valid class attribute that is a function object, a
method object is created by packing (pointers to) the instance object
and the function object just found together in an abstract object:
this is the method object. When the method object is called with an
argument list, a new argument list is constructed from the instance
object and the argument list, and the function object is called with
this new argument list.
http://docs.python.org/2/tutorial/classes.html#method-objects
Read carefully this excerpt.
It means :
1) An instance doesn't really hold an object being a method that would be its attribute.
In fact, there is not at all a "method" attribute in the __dict__ of an instance (__dict__ is the namespace of an object)
2) The fact that an instance seems to have a "method" when a "method" attribute is called, is due to a process, not the presence of a method object inside the namespace of an instance
3) Also, there doesn't really exist a method object in the namespace of a class.
But there's a difference with an instance, because there must be somewhere something that leads to a real method object when such a call is done, must not ?
What is called a "method" attribute of a class, for easiness of wording, is in reality a function object being attribute in the namespace of the class.
That is to say, a pair (identifier of the function, function) is a member of the __dict__ of a class, and this attribute allows the intepreter to construct a method object when a method call is performed.
4) Again, the fact that a class seems to have a "method" when a "method" attribute is called, is due to a process, not to the presence of a method object inside the namespace of a class
EDIT I'm no more sure of that; see at the end
5) A method object (not "method" object; I mean the real object being really a method`, what is described in the excerpt) is created at the moment of the call, it doesn't exists before.
It is a kind of wrapper : it packs pointers to the instance object and the function object on which the method is based.
So, a method is based on a function. This function is for me the real attribute of the class holding the said "method", because this function really belongs to the namespace ( __dict__ ) of the class: this function is described as a <function ......> when the __dict__ is printed.
This function can be reached from the method object using the alias im_func or __func__ (see the below code)
.
I believe that these notions are not very commonly known and understood. But the following code proves what I said.
class A(object):
def __init__(self,b=0):
self.b = b
print 'The __init__ object :\n',__init__
def addu(self):
self.b = self.b + 10
print '\nThe addu object :\n',addu
print '\nThe A.__dict__ items :\n',
print '\n'.join(' {0:{align}11} : {1}'.format(*it,align='^')
for it in A.__dict__.items())
a1 = A(101)
a2 = A(2002)
print '\nThe a1.__dict__ items:'
print '\n'.join(' {0:{align}11} : {1}'.format(*it,align='^')
for it in a1.__dict__.items())
print '\nThe a2.__dict__ items:'
print '\n'.join(' {0:{align}11} : {1}'.format(*it,align='^')
for it in a2.__dict__.items())
print '\nA.addu.__func__ :',A.addu.__func__
print id(A.addu.__func__),'==',hex(id(A.addu.__func__))
print
print 'A.addu :\n ',
print A.addu,'\n ',id(A.addu),'==',hex(id(A.addu))
print 'a1.addu :\n ',
print a1.addu,'\n ',id(a1.addu),'==',hex(id(a1.addu))
print 'a2.addu :\n ',
print a2.addu,'\n ',id(a2.addu),'==',hex(id(a2.addu))
a2.addu()
print '\na2.b ==',a2.b
print '\nThe A.__dict__ items :\n',
print '\n'.join(' {0:{align}11} : {1}'.format(*it,align='^')
for it in A.__dict__.items())
result
The __init__ object :
<function __init__ at 0x011E54B0>
The addu object :
<function addu at 0x011E54F0>
The A.__dict__ items :
__module__ : __main__
addu : <function addu at 0x011E54F0>
__dict__ : <attribute '__dict__' of 'A' objects>
__weakref__ : <attribute '__weakref__' of 'A' objects>
__doc__ : None
__init__ : <function __init__ at 0x011E54B0>
The a1.__dict__ items:
b : 101
The a2.__dict__ items:
b : 2002
A.addu.__func__ : <function addu at 0x011E54F0>
18765040 == 0x11e54f0
A.addu :
<unbound method A.addu>
18668040 == 0x11cda08
a1.addu :
<bound method A.addu of <__main__.A object at 0x00CAA850>>
18668040 == 0x11cda08
a2.addu :
<bound method A.addu of <__main__.A object at 0x011E2B90>>
18668040 == 0x11cda08
a2.b == 2012
The A.__dict__ items :
__module__ : __main__
addu : <function addu at 0x011E54F0>
__dict__ : <attribute '__dict__' of 'A' objects>
__weakref__ : <attribute '__weakref__' of 'A' objects>
__doc__ : None
__init__ : <function __init__ at 0x011E54B0>
.
EDIT
Something is troubling me and I don't know the deep innards of the subject:
The above code shows that A.addu , a1.addu and a2.addu are all the same method object, with a unique identity.
However A.addu is said an unbound method because it doesn't have any information concerning an particular instance,
and a1.addu and a2.addu are said bound methods because each one has information designating the instance that must be concerned by the operations of the method.
Logically, for me, that would mean that the method should be different for each of these 3 cases.
BUT the identity is the same for all three, and moreover this identity is different from the identity of the function on which the method is based.
It leads to the conclusion that the method is really an object living in the memory, and that it doesn't change from one call from an instance to another cal from another instance.
HOWEVER , printing the namespace __dict__ of the class, even after the creation of instances and the call of the "method" addu(), this namespace doesn't exposes a new object that could be identified to the method object different from the addu function.
What does it mean ?
It gives me the impression that as soon as a method object is created, it isn't destroyed, it lives in the memory (RAM).
But it lives hidden and only the processes that form the interpeter's functionning know how and where to find it.
This hidden object, the real method object, must have the ability to change the reference to the instance to which the function must be applied, or to reference None if it is called as an unbound method. That's what it seems to me, but it's only brain-storming hypothesis.
Does anybody know something on this interrogation ?
To answer to the question, it can be considered correct to call .upper and .lower functions , since in reality they are based on functionsas every method of a class.
However, the following result is special, probably because they are builtin methods/functions, not user's methods/functions as in my code.
x = 'hello'
print x.upper.__func__
result
print x.upper.__func__
AttributeError: 'builtin_function_or_method' object has no attribute '__func__'
In the following class definition:
class MyClass:
"""A simple example class"""
def f(self):
return 'hello world'
Class : MyClass
Function: f()
Method: None (Actually, not applicable)
Lets create an instance of the above class. We'll do this by assigning class object, i.e. MyClass() to var x
x = MyClass()
Here,
Function: None
Method: x.f()
And lets not forget that the function object MyClass.f was used to define (internally) the method object x.f when we assigned x to MyClass()
Basically, yes, Python does distinguish them, but in Python it is common to view methods as a subset of functions. Methods are associated with a class or instance, and "standalone functions" are not. Something that is a method is also a function, but there can be functions that are not methods.
As Jon Clements mentioned in his comment, though, the distinction is not so ironclad as in C++. Standalone functions can be "converted" into methods at runtime, and methods can be assigned to variables in such a way that they effectively behave no differently than standalone functions. So the boundary between methods and functions is permeable.
Took snippet from this answer
Following are the changes
Unbound methods (methods bound to a class object) are no longer available.
unbound method -> function
type instancemethod are removed from Python 3
More about methods on this answer
About methods on Python docs
More about classes on Python docs
import sys
print(sys.version)
# 3.9.0rc2 (tags/v3.9.0rc2:2bd31b5, Sep 17 2020, 00:58:12) [MSC v.1927 64 bit (AMD64)]
class A:
def a(self): pass
print(A.a)
# <unbound method A.a>
# <function A.a at 0x00000200FBE121F0>
print(type(A.a))
# <type 'instancemethod'>
# <class 'function'>
print(A().a)
# <bound method A.a of <__main__.A instance at 0x107070d88>>
# <bound method A.a of <__main__.A object at 0x00000200FBB12640>>
print(type(A().a))
# <type 'instancemethod'>
# <class 'method'>
Short answer:
Methods are bounded functions. Functions become methods when function is bound to an object.
To understand the difference between function and method in python you need to understand
Everything is an object in python. Even function is an object
I searched python.org to find exact definition of object in python. But I could not find exact definition.
In your python console define an integer
my_int = 1
my_int.__class__
output:
<class 'int'>
Define a lambda function
my_lambda = lambda x:x+1
my_lambda.__class__
output
<class 'function'>
Define a function.
def mycounter(x):
return x+1
mycounter.__class__
output
<class 'function'>
From this we can surmise that everything is an object in python. Even function is an object.
Class object and instance object are different. When you define a class, class object is created. When you instantiate a class, instance object is created.
In python console define a class.
class MyClass:
"""A simple example class"""
i = 12345
def f(self):
return 'hello world'
In your python console you can check MyClass object is created
MyClass.__class__
output
<class 'type'>
Create instance object of the class
mc = MyClass()
mc.__class__
output
<class 'MyClass'>
Check attribute i for instance object
mc.i
output
12345
Check attribute i for class object
MyClass.i
output
12345
Check attribute f for instance object.
mc.f
output
<bound method MyClass.f of <MyClass object at 0x7fbea6c18d00>>
Check attribute f for class object.
MyClass.f
output
<function MyClass.f at 0x7f1730f9e1f0>
We can see that for Class object f is a function object and for instance object, f is a bound method object.
Call f() on instance object
mc.f()
output
'hello world'
Call f() on class object.
MyClass.f()
output
Traceback (most recent call last):
File "<console>", line 1, in <module>
TypeError: f() missing 1 required positional argument: 'self'
Pass an instance object when calling class object f function.
MyClass.f(mc)
output
'hello world'
From above examples we can say that -
MyClass is a Class object and mc is an instance object and they are different from one another.
Functions become methods when functions are bound to an object. These objects need not be just class instance object. They can any other python objects.
References:
As per python documentation for function -
Blockquote
A method is a function that ‘belongs’ to an object and is named obj.methodname, where obj is some object (this may be an expression), and methodname is the name of a method that is defined by the object’s type. Different types define different methods. Methods of different types may have the same name without causing ambiguity.
As per python documentation for class definition -
When a class definition is entered, a new namespace is created, and used as the local scope — thus, all assignments to local variables go into this new namespace. In particular, function definitions bind the name of the new function here. When a class definition is left normally (via the end), a class object is created.
As per python documentation for class objects
Class objects support two kinds of operations: attribute references and instantiation.
As per python documentation for instance objects
The only operations understood by instance objects are attribute references. There are two kinds of valid attribute names: data attributes and methods.
A method is a function that “belongs to” an object. (In Python, the term method is not unique to class instances: other object types can have methods as well. For example, list objects have methods called append, insert, remove, sort, and so on. However, in the following discussion, we’ll use the term method exclusively to mean methods of class instance objects, unless explicitly stated otherwise.)
By definition, all attributes of a class that are function objects define corresponding methods of its instances. Valid method names of an instance object depend on its class. By definition, all attributes of a class that are function objects define corresponding methods of its instances. So in our example, x.f is a valid method reference, since MyClass.f is a function,
but x.i is not, since MyClass.i is not. But x.f is not the same thing as MyClass.f — it is a method object, not a function object.
I've never seen anything else work like this before.
Is there anything else that does this?
>>> class NothingSpecial:
#classmethod
def meth(cls): pass
>>> NothingSpecial.meth
<bound method classobj.meth of <class __main__.NothingSpecial at 0x02C68C70>>
>>> NothingSpecial.__dict__['meth']
<classmethod object at 0x03F15FD0>
>>> getattr(NothingSpecial, 'meth')
<bound method NothingSpecial.meth of <class '__main__.NothingSpecial'>>
>>> object.__getattribute__(NothingSpecial, 'meth')
<classmethod object at 0x03FAFE90>
>>> type.__getattribute__(NothingSpecial, 'meth')
<bound method NothingSpecial.meth of <class '__main__.NothingSpecial'>>
Getattr Uses Descriptor Logic
The main difference is that the dictionary lookup does no extra processing while the attribute fetch incorporates extra logic (see my Descriptor How-To Guide for all the details).
There Are Two Different Underlying Methods
1) The call NothingSpecial.__dict__['meth'] uses the square brackets operator which dispatches to dict.__getitem__ which does a simple hash table lookup or raises KeyError if not found.
2) The call NothingSpecial.meth uses the dot operator which dispatches to type.__getattribute__ which does a simple lookup followed by a special case for descriptors. If the lookup fails, an AttributeError is raised.
How It Works
The overall logic is documented here and here.
In general, a descriptor is an object attribute with “binding
behavior”, one whose attribute access has been overridden by methods
in the descriptor protocol: __get__(), __set__(), and/or __delete__(). If
any of those methods are defined for an object, it is said to be a
descriptor.
The default behavior for attribute access is to get, set, or delete
the attribute from an object’s dictionary. For instance, a.x has a
lookup chain starting with a.__dict__['x'], then
type(a).__dict__['x'], and continuing through the base classes of
type(a) excluding metaclasses.
However, if the looked-up value is an object defining one of the
descriptor methods, then Python may override the default behavior and
invoke the descriptor method instead. Where this occurs in the
precedence chain depends on which descriptor methods were defined and
how they were called
Hope you've found all of this to be helpful. The kind of exploring you're doing is a great way to learn about Python :-)
P.S. You might also enjoy reading the original Whatsnew in Python 2.2 entry for descriptors or looking at PEP 252 where Guido van Rossum originally proposed the idea.
object.__getattribute__(NothingSpecial, 'meth')
and
NothingSpecial.__dict__['meth']
return the same object in this case. You can quickly check it by doing:
NothingSpecial.__dict__['meth'] is object.__getattribute__(NothingSpecial, 'meth')
$True
Both of them points to the same descriptor object
on the other hand:
object.__getattribute__(NothingSpecial, 'meth') is getattr(NothingSpecial, 'meth')
$False
Basically, they aren't they are not the same object nand the same type:
type(object.__getattribute__(NothingSpecial, 'meth'))
$<class 'classmethod'>
type(getattr(NothingSpecial, 'meth'))
$<class 'method'>
So the answer is that getattr will automagically invoke an object's __get__ method if it has one, whereas object.__getattribute__ and the objects __dict__ lookup do not. The following function proves that:
class Nothing:
#classmethod
def a(cls):
return cls()
#staticmethod
def b():
return 'b'
def c(self):
return 'c'
def gitter(obj, name):
value = object.__getattribute__(obj, name)
if hasattr(value, '__get__'):
if isclass(obj):
instance, cls = None, obj
else:
instance, cls = obj, type(obj)
return value.__get__(instance, cls)
return value
>>> gitter(Nothing, 'a')()
<__main__.Nothing object at 0x03E97930>
>>> gitter(Nothing, 'b')()
'b'
>>> gitter(Nothing(), 'c')()
'c'
However, gitter(Nothing(), 'b') doesn't work currently because it's not detecting that the objtype default value is None, but this is enough.
From what I understand, when calling pickle.dumps on an object, it will call the object's __getstate__ method (if it has one) to determine what to pickle.
If I create a class such as:
class DictClass(dict):
def __getstate__(self):
print "pickling"
return self
I get this result:
>>> pickle.dumps(DictClass())
pickling
'ccopy_reg\n_reconstructor\np0...'
I can do the same thing, replacing 'dict' with 'list':
class ListClass(list):
def __getstate__(self):
print "pickling"
return self
>>> pickle.dumps(ListClass())
pickling
'ccopy_reg\n_reconstructor\np0...'
But if I use 'set', something different happens:
class SetClass(set):
def __getstate__(self):
print "pickling"
return self
>>> pickle.dumps(SetClass())
'c__main__\nSetClass...'
The __getstate__ method doesn't get called. Why is this, and is it possible to specify what part of a subclass of a set to pickle?
list does not implement __reduce__(), whereas set does:
>>> list().__reduce__()
...
TypeError: can't pickle list objects
>>> set().__reduce__()
(<type 'set'>, ([],), None)
It's the last tuple in the above example that gets pickled, so SetClass.__getstate__() never enters the picture.
I've little bit test to fully understand metaclass in python.
class Test(object):
pass
print Test.__class__
print Test.__class__.__class__
print Test.__class__.__class__.__class__
All of result is same type. but each of their address is not same
I can't really understand why metaclass has a metaclass recursively.
Explain me please?
Actually, addresses are the same:
>>> id(Test.__class__)
6384576
>>> id(Test.__class__.__class__)
6384576
>>> id(Test.__class__.__class__.__class__)
6384576
Everything is an object in Python, and each object must have a class (it should belong to some type). You can access that class/type reference by __class__ attribute, e.g.:
>>> (1).__class__
<type 'int'>
Everything includes classes itself, which are of class/type called type:
>>> (1).__class__.__class__
<type 'type'>
In the same time type 'type'> is also an object and should reference to some class/type. But since this is kind of special object, its __class__ attribute refers to itself:
>>> (1).__class__.__class__.__class__ is (1).__class__.__class__
True
When you do Test.__class__, you get back the type of Test, which is type (because Test is a class identifier).
type itself is again a class identifier, so you can call __class__ (which is inherited from object) on it and you get back that its type is, again, type because it is a class identifier.
Because you will always get back type which is a class itself, you can do this infinitely many times and will always get back that the current object's type is type.
All classes are classes which means they are derived from a class called class...
All the python's class object is build by the built-in function type(). You could also try this.
>>> T.__class__ == type
True
>>> type(type)
<type 'type'>
The T.class is equal to the build-in function type which is also an object implemented the call function. It's a attribute will be interpret as class(T). As your T class have no base class so type() is used which will return the type object.
You could check the python doc about customizing class creation to get detail about class creation.
To determining the appropriate metaclass
if no bases and no explicit metaclass are given, then type() is used
if an explicit metaclass is given and it is not an instance of type(), then it is used directly as the metaclass
if an instance of type() is given as the explicit metaclass, or bases are defined, then the most derived metaclass is used