Develop Reference

Invisible argument python [duplicate]

This question already has answers here:
What is the purpose of the `self` parameter? Why is it needed?
(26 answers)
Closed 6 months ago.
When defining a method on a class in Python, it looks something like this:
class MyClass(object):
def __init__(self, x, y):
self.x = x
self.y = y
But in some other languages, such as C#, you have a reference to the object that the method is bound to with the "this" keyword without declaring it as an argument in the method prototype.
Was this an intentional language design decision in Python or are there some implementation details that require the passing of "self" as an argument?

I like to quote Peters' Zen of Python. "Explicit is better than implicit."
In Java and C++, 'this.' can be deduced, except when you have variable names that make it impossible to deduce. So you sometimes need it and sometimes don't.
Python elects to make things like this explicit rather than based on a rule.
Additionally, since nothing is implied or assumed, parts of the implementation are exposed. self.__class__, self.__dict__ and other "internal" structures are available in an obvious way.

It's to minimize the difference between methods and functions. It allows you to easily generate methods in metaclasses, or add methods at runtime to pre-existing classes.
e.g.
>>> class C:
... def foo(self):
... print("Hi!")
...
>>>
>>> def bar(self):
... print("Bork bork bork!")
...
>>>
>>> c = C()
>>> C.bar = bar
>>> c.bar()
Bork bork bork!
>>> c.foo()
Hi!
>>>
It also (as far as I know) makes the implementation of the python runtime easier.

I suggest that one should read Guido van Rossum's blog on this topic - Why explicit self has to stay.
When a method definition is decorated, we don't know whether to automatically give it a 'self' parameter or not: the decorator could turn the function into a static method (which has no 'self'), or a class method (which has a funny kind of self that refers to a class instead of an instance), or it could do something completely different (it's trivial to write a decorator that implements '#classmethod' or '#staticmethod' in pure Python). There's no way without knowing what the decorator does whether to endow the method being defined with an implicit 'self' argument or not.
I reject hacks like special-casing '#classmethod' and '#staticmethod'.

Python doesn't force you on using "self". You can give it whatever name you want. You just have to remember that the first argument in a method definition header is a reference to the object.

Also allows you to do this: (in short, invoking Outer(3).create_inner_class(4)().weird_sum_with_closure_scope(5) will return 12, but will do so in the craziest of ways.
class Outer(object):
def __init__(self, outer_num):
self.outer_num = outer_num
def create_inner_class(outer_self, inner_arg):
class Inner(object):
inner_arg = inner_arg
def weird_sum_with_closure_scope(inner_self, num)
return num + outer_self.outer_num + inner_arg
return Inner
Of course, this is harder to imagine in languages like Java and C#. By making the self reference explicit, you're free to refer to any object by that self reference. Also, such a way of playing with classes at runtime is harder to do in the more static languages - not that's it's necessarily good or bad. It's just that the explicit self allows all this craziness to exist.
Moreover, imagine this: We'd like to customize the behavior of methods (for profiling, or some crazy black magic). This can lead us to think: what if we had a class Method whose behavior we could override or control?
Well here it is:
from functools import partial
class MagicMethod(object):
"""Does black magic when called"""
def __get__(self, obj, obj_type):
# This binds the <other> class instance to the <innocent_self> parameter
# of the method MagicMethod.invoke
return partial(self.invoke, obj)
def invoke(magic_self, innocent_self, *args, **kwargs):
# do black magic here
...
print magic_self, innocent_self, args, kwargs
class InnocentClass(object):
magic_method = MagicMethod()
And now: InnocentClass().magic_method() will act like expected. The method will be bound with the innocent_self parameter to InnocentClass, and with the magic_self to the MagicMethod instance. Weird huh? It's like having 2 keywords this1 and this2 in languages like Java and C#. Magic like this allows frameworks to do stuff that would otherwise be much more verbose.
Again, I don't want to comment on the ethics of this stuff. I just wanted to show things that would be harder to do without an explicit self reference.

I think it has to do with PEP 227:
Names in class scope are not accessible. Names are resolved in the
innermost enclosing function scope. If a class definition occurs in a
chain of nested scopes, the resolution process skips class
definitions. This rule prevents odd interactions between class
attributes and local variable access. If a name binding operation
occurs in a class definition, it creates an attribute on the resulting
class object. To access this variable in a method, or in a function
nested within a method, an attribute reference must be used, either
via self or via the class name.

I think the real reason besides "The Zen of Python" is that Functions are first class citizens in Python.
Which essentially makes them an Object. Now The fundamental issue is if your functions are object as well then, in Object oriented paradigm how would you send messages to Objects when the messages themselves are objects ?
Looks like a chicken egg problem, to reduce this paradox, the only possible way is to either pass a context of execution to methods or detect it. But since python can have nested functions it would be impossible to do so as the context of execution would change for inner functions.
This means the only possible solution is to explicitly pass 'self' (The context of execution).
So i believe it is a implementation problem the Zen came much later.

As explained in self in Python, Demystified
anything like obj.meth(args) becomes Class.meth(obj, args). The calling process is automatic while the receiving process is not (its explicit). This is the reason the first parameter of a function in class must be the object itself.
class Point(object):
def __init__(self,x = 0,y = 0):
self.x = x
self.y = y
def distance(self):
"""Find distance from origin"""
return (self.x**2 + self.y**2) ** 0.5
Invocations:
>>> p1 = Point(6,8)
>>> p1.distance()
10.0
init() defines three parameters but we just passed two (6 and 8). Similarly distance() requires one but zero arguments were passed.
Why is Python not complaining about this argument number mismatch?
Generally, when we call a method with some arguments, the corresponding class function is called by placing the method's object before the first argument. So, anything like obj.meth(args) becomes Class.meth(obj, args). The calling process is automatic while the receiving process is not (its explicit).
This is the reason the first parameter of a function in class must be the object itself. Writing this parameter as self is merely a convention. It is not a keyword and has no special meaning in Python. We could use other names (like this) but I strongly suggest you not to. Using names other than self is frowned upon by most developers and degrades the readability of the code ("Readability counts").
...
In, the first example self.x is an instance attribute whereas x is a local variable. They are not the same and lie in different namespaces.
Self Is Here To Stay
Many have proposed to make self a keyword in Python, like this in C++ and Java. This would eliminate the redundant use of explicit self from the formal parameter list in methods. While this idea seems promising, it's not going to happen. At least not in the near future. The main reason is backward compatibility. Here is a blog from the creator of Python himself explaining why the explicit self has to stay.

The 'self' parameter keeps the current calling object.
class class_name:
class_variable
def method_name(self,arg):
self.var=arg
obj=class_name()
obj.method_name()
here, the self argument holds the object obj. Hence, the statement self.var denotes obj.var

There is also another very simple answer: according to the zen of python, "explicit is better than implicit".

How to create a static class property that returns an instance of the class itself?

I wrote a class that can handle integers with arbitrary precision (just for learning purposes). The class takes a string representation of an integer and converts it into an instance of BigInt for further calculations.
Often times you need the numbers Zero and One, so I thought it would be helpfull if the class could return these. I tried the following:
class BigInt():
zero = BigInt("0")
def __init__(self, value):
####yada-yada####
This doesn't work. Error: "name 'BigInt' is not defined"
Then I tried the following:
class BigInt():
__zero = None
#staticmethod
def zero():
if BigInt.__zero is None:
BigInt.__zero = BigInt('0')
return BigInt.__zero
def __init__(self, value):
####yada-yada####
This actually works very well. What I don't like is that zero is a method (and thus has to be called with BigInt.zero()) which is counterintuitive since it should just refer to a fixed value.
So I tried changing zero to become a property, but then writing BigInt.zero returns an instance of the class property instead of BigInt because of the decorator used. That instance cannot be used for calculations because of the wrong type.
Is there a way around this issue?

A static property...? We call a static property an "attribute". This is not Java, Python is a dynamically typed language and such a construct would be really overcomplicating matters.
Just do this, setting a class attribute:
class BigInt:
def __init__(self, value):
...
BigInt.zero = BigInt("0")
If you want it to be entirely defined within the class, do it using a decorator (but be aware it's just a more fancy way of writing the same thing).
def add_zero(cls):
cls.zero = cls("0")
return cls
#add_zero
class BigInt:
...

The question is contradictory: static and property don't go together in this way. Static attributes in Python are simply ones that are only assigned once, and the language itself includes a very large number of these. (Most strings are interred, all integers < a certain value are pre-constructed, etc. E.g. the string module.). Easiest approach is to statically assign the attributes after construction as wim illustrates:
class Foo:
...
Foo.first = Foo()
...
Or, as he further suggested, using a class decorator to perform the assignments, which is functionally the same as the above. A decorator is, effectively, a function that is given the "decorated" function as an argument, and must return a function to effectively replace the original one. This may be the original function, say, modified with some annotations, or may be an entirely different function. The original (decorated) function may or may not be called as appropriate for the decorator.
def preload(**values):
def inner(cls):
for k, v in values.items():
setattr(cls, k, cls(v))
return cls
return inner
This can then be used dynamically:
#preload(zero=0, one=1)
class Foo:
...
If the purpose is to save some time on common integer values, a defaultdict mapping integers to constructed BigInts could be useful as a form of caching and streamlined construction / singleton storage. (E.g. BigInt.numbers[27])
However, the problem of utilizing #property at the class level intrigued me, so I did some digging. It is entirely possible to make use of "descriptor protocol objects" (which the #property decorator returns) at the class level if you punt the attribute up the object model hierarchy, to the metaclass.
class Foo(type):
#property
def bar(cls):
print("I'm a", cls)
return 27
class Bar(metaclass=Foo):
...
>>> Bar.bar
I'm a <class '__main__.Bar'>
<<< 27
Notably, this attribute is not accessible from instances:
>>> Bar().bar
AttributeError: 'Bar' object has no attribute 'bar'
Hope this helps!

Is it possible to display an objects instance variables in IPython like Matlab does?

I'm trying to move away from Matlab to Python. While the magic ? in IPython is nice, one very nice feature of Matlab is that you can see on the command line (by omitting the ;) the instance variables (called properties in Matlab) of the object in question. Is this possible in python (I guess via IPython)?
Ideally a class like this:
class MyClass(object):
_x = 5
#property
def x(self):
return self._x + 100
#x.setter
def x(self, value):
self._x = value + 1
def myFunction(self, y):
return self.x ** 2 + y
Would display something like:
mc = Myclass()
mc
<package.MyClass> <superclass1> <superclass2>
Attributes:
_x: 5
x: 105
Method Attributes:
myFunction(self, y)
Is that possible via overriding the print method (if such a thing exits) of the class? Or via a magic method in ipython?

The short answer is that there is no way to get a list of all attributes of an object in Python, because the attributes could be generated dynamically. For an extreme example, consider this class:
>>> class Spam(object):
... def __getattr__(self, attr):
... if attr.startswith('x'):
... return attr[1:]
>>> spam = Spam()
>>> spam.xeggs
'eggs'
Even if the interpreter could someone figure out a list of all attributes, that list would be infinite.
For simple classes, spam.__dict__ is often good enough. It doesn't handle dynamic attributes, __slots__-based attributes, class attributes, C extension classes, attributes inherited from most of the above, and all kinds of other things. But it's at least something—and sometimes, it's the something you want. To a first approximation, it's exactly the stuff you explicitly assigned in __init__ or later, and nothing else.
For a best effort aimed at "everything" aimed at human readability, use dir(spam).
For a best effort aimed at "everything" for programmatic use, use inspect.getmembers(spam). (Although in fact the implementation is just a wrapper around dir in CPython 2.x, it could do more—and in fact does in CPython 3.2+.)
These will both handle a wide range of things that __dict__ cannot, and may skip things that are in the __dict__ but that you don't want to see. But they're still inherently incomplete.
Whatever you use, to get the values as well as the keys is easy. If you're using __dict__ or getmembers, it's trivial; the __dict__ is, normally, either a dict, or something that acts close enough to a dict for your purposes, and getmembers explicitly returns key-value pairs. If you're using dir, you can get a dict very easily:
{key: getattr(spam, key) for key in dir(spam)}
One last thing: "object" is a bit of an ambiguous term. It can mean "any instance of a class derived from object", "any instance of a class", "any instance of a new-style class", or "any value of any type at all" (modules, classes, functions, etc.). You can use dir and getmembers on just about anything; the exact details of what that means are described in the docs.
One even-last-er thing: You may notice that getmembers returns things like ('__str__', <method-wrapper '__str__' of Spam object at 0x1066be790>), which you probably aren't interested in. Since the results are just name-value pairs, if you just want to remove __dunder__ methods, _private variables, etc., that's easy. But often, you want to filter on the "kind of member". The getmembers function takes a filter parameter, but the docs don't do a great job explaining how to use it (and, on top of that, expect that you understand how descriptors work). Basically, if you want a filter, it's usually callable, lambda x: not callable(x), or a lambda made up of a combination of inspect.isfoo functions.
So, this is common enough you may want to write it up as a function:
def get_public_variables(obj):
return [(name, value) for name, value
in inspect.getmembers(obj, lambda x: not callable(x))
if not name.startswith('_')]
You can turn that into a custom IPython %magic function, or just make a %macro out of it, or just leave it as a regular function and call it explicitly.
In a comment, you asked whether you can just package this up into a __repr__ function instead of trying to create a %magic function or whatever.
If you've already got all of your classes inheriting from a single root class, this is a great idea. You can write a single __repr__ that works for all of your classes (or, if it works for 99% of them, you can override that __repr__ in the other 1%), and then every time you evaluate any of your objects in the interpreter or print them out, you'll get what you want.
However, a few things to keep in mind:
Python has both __str__ (what you get if you print an object) and __repr__ (what you get if you just evaluate an object at the interactive prompt) for a reason. Usually, the former is a nice human-readable representation, while the latter is something that's either eval-able (or typable-into-the-interactive-prompt), or the concise angle-bracket form that gives you just enough to distinguish the type and identity of an object.
That's just a convention rather than a rule, so you can feel free to break it. However, if you are going to break it, you may still want to make use of the str/repr distinction—e.g., make repr give you a complete dump of all the internals, while str shows just the useful public values.
More seriously, you have to consider how repr values are composed. For example, if you print or repr a list, you get, effectively, '[' + ', '.join(map(repr, item))) + ']'. This is going to look pretty odd with a multi-line repr. And it'll be even worse if you use any kind of pretty-printer that tries to indent nested collections, like the one that's built into IPython. The result probably won't be unreadable, it'll just defeat the benefits that the pretty-printer is meant to provide.
As for the specific stuff you want to display: That's all pretty easy. Something like this:
def __repr__(self):
lines = []
classes = inspect.getmro(type(self))
lines.append(' '.join(repr(cls) for cls in classes))
lines.append('')
lines.append('Attributes:')
attributes = inspect.getmembers(self, callable)
longest = max(len(name) for name, value in attributes)
fmt = '{:>%s}: {}' % (longest, )
for name, value in attributes:
if not name.startswith('__'):
lines.append(fmt.format(name, value))
lines.append('')
lines.append('Methods:')
methods = inspect.getmembers(self, negate(callable))
for name, value in methods:
if not name.startswith('__'):
lines.append(name)
return '\n'.join(lines)
Right-justifying the attribute names is the hardest part here. (And I probably got it wrong, since this is untested code…) Everything else is either easy, or fun (playing with different filters to getmembers to see what they do).

I was able to achieve what I wanted with IPython (at least roughly) by implementing _repr_pretty_:
def get_public_variables(obj):
from inspect import getmembers
return [(name, value) for name, value in
getmembers(obj, lambda x: not callable(x)) if
not name.startswith('__')]
class MySuperClass(object):
def _repr_pretty_(self, p, cycle):
for (name, value) in get_public_variables(self):
f = '{:>12}{} {:<} \n'
line = f.format(str(name), ':', str(value))
# p.text(str(name) + ': ' + str(value) + '\n')
p.text(line)
class MyClass(MySuperClass):
_x = 5
#property
def x(self):
return self._x + 100
gives me out of:
mc = MyClass()
mc
Out[15]:
_x: 5
x: 105
clearly there is some fine tuning to be done in terms of whitespace, etc. But this was roughly what I was trying to accomplish

You can get to an object's instance variable using obj.__dict__, e.g.:
class Dog:
def __init__(self, name, age):
self.name = name
self.age = age
d1 = Dog("Fido", 7)
for key, val in d1.__dict__.items():
print(key, ": ", val)
Output:
age : 7
name : Fido
You might find this doesn't work well with real objects that might have a large number of instance variables and methods though.

Different instance-method behavior between Python 2.5 and 2.6

Trying to change the __unicode__ method on an instance after it's created produces different results on Python 2.5 and 2.6.
Here's a test script:
class Dummy(object):
def __unicode__(self):
return u'one'
def two(self):
return u'two'
d = Dummy()
print unicode(d)
d.__unicode__ = d.two
print unicode(d)
print d.__unicode__()
On Python 2.5, this produces
one
two
two
That is, changing the instance's __unicode__ also changes unicode(instance)
On Python 2.6, this produces
one
one
two
So, after a change, unicode(instance) and instance.__unicode__() return different results.
Why? How can I get this working on Python 2.6?
(For what it's worth, the use case here is that I want to append something to the output of __unicode__ for all subclasses of a given class, without having to modify the code of the subclasses.)
Edit to make the use case a little clearer
I have Class A, which has many subclasses. Those subclasses define simple __unicode__ methods. I want to add logic so that, for instances of a Class A subclass, unicode(instance) gets something tacked on to the end. To keep the code simple, and because there are many subclasses I don't want to change, I'd prefer to avoid editing subclass code.
This is actually existing code that works in Python 2.5. It's something like this:
class A(object):
def __init__(self):
self._original_unicode = self.__unicode__
self.__unicode__ = self.augmented_unicode
def augmented_unicode(self):
return self._original_unicode() + u' EXTRA'
It's this code that no longer works on 2.6. Any suggestions on how to achieve this without modifying subclass code? (If the answer involves metaclasses, note that class A is itself a subclass of another class -- django.db.models.Model -- with a pretty elaborate metaclass.)

It appears that you are not allowed to monkey-patch protocol methods (those that begin and end with double underscores) :
Note
In practise there is another exception
that we haven't handled here. Although
you can override methods with instance
attributes (very useful for monkey
patching methods for test purposes)
you can't do this with the Python
protocol methods. These are the 'magic
methods' whose names begin and end
with double underscores. When invoked
by the Python interpreter they are
looked up directly on the class and
not on the instance (however if you
look them up directly - e.g.
x.repr - normal attribute lookup
rules apply).
That being the case, you may be stuck unless you can go with ~unutbu's answer.
EDIT: Or, you can have the base class __unicode__ method search the instance object's dict for a __unicode__ attribute. If it's present, then __unicode__ is defined on the instance object, and the class method calls the instance method. Otherwise, we fall back to the class definition of __unicode__.
I think that this could allow your existing subclass code to work without any changes. However, it gets ugly if the derived class wants to invoke the class implementation -- you need to be careful to avoid infinite loops. I haven't implemented such hacks in this example; merely commented about them.
import types
class Dummy(object):
def __unicode__(self):
func = self.__dict__.get("__unicode__", None)
if func:
// WARNING: if func() invokes this __unicode__ method directly,
// an infinite loop could result. You may need an ugly hack to guard
// against this. (E.g., set a flag on entry / unset the flag on exit,
// using a try/finally to protect against exceptions.)
return func()
return u'one'
def two(self):
return u'two'
d = Dummy()
print unicode(d)
funcType = type(Dummy.__unicode__)
d.__unicode__ = types.MethodType(Dummy.two, d)
print unicode(d)
print d.__unicode__()
Testing with Python 2.6 produces the following output:
> python dummy.py
one
two
two

Edit: In response to the OP's comment: Adding a layer of indirection can allow you to change the behavior of unicode on a per-instance basis:
class Dummy(object):
def __unicode__(self):
return self._unicode()
def _unicode(self):
return u'one'
def two(self):
return u'two'
d = Dummy()
print unicode(d)
# one
d._unicode = d.two
print unicode(d)
# two
print d.__unicode__()
# two

Looks like Dan is correct about monkey-patching protocol methods, and that this was a change between Python 2.5 and Python 2.6.
My fix ended up being making the change on the classes rather the instances:
class A(object):
def __init__(self):
self.__class__.__unicode__ = self.__class__.augmented_unicode

is it ever useful to define a class method with a reference to self not called 'self' in Python?

I'm teaching myself Python and I see the following in Dive into Python section 5.3:
By convention, the first argument of any Python class method (the reference to the current instance) is called self. This argument fills the role of the reserved word this in C++ or Java, but self is not a reserved word in Python, merely a naming convention. Nonetheless, please don't call it anything but self; this is a very strong convention.
Considering that self is not a Python keyword, I'm guessing that it can sometimes be useful to use something else. Are there any such cases? If not, why is it not a keyword?

No, unless you want to confuse every other programmer that looks at your code after you write it. self is not a keyword because it is an identifier. It could have been a keyword and the fact that it isn't one was a design decision.

As a side observation, note that Pilgrim is committing a common misuse of terms here: a class method is quite a different thing from an instance method, which is what he's talking about here. As wikipedia puts it, "a method is a subroutine that is exclusively associated either with a class (in which case it is called a class method or a static method) or with an object (in which case it is an instance method).". Python's built-ins include a staticmethod type, to make static methods, and a classmethod type, to make class methods, each generally used as a decorator; if you don't use either, a def in a class body makes an instance method. E.g.:
>>> class X(object):
... def noclass(self): print self
... #classmethod
... def withclass(cls): print cls
...
>>> x = X()
>>> x.noclass()
<__main__.X object at 0x698d0>
>>> x.withclass()
<class '__main__.X'>
>>>
As you see, the instance method noclass gets the instance as its argument, but the class method withclass gets the class instead.
So it would be extremely confusing and misleading to use self as the name of the first parameter of a class method: the convention in this case is instead to use cls, as in my example above. While this IS just a convention, there is no real good reason for violating it -- any more than there would be, say, for naming a variable number_of_cats if the purpose of the variable is counting dogs!-)

The only case of this I've seen is when you define a function outside of a class definition, and then assign it to the class, e.g.:
class Foo(object):
def bar(self):
# Do something with 'self'
def baz(inst):
return inst.bar()
Foo.baz = baz
In this case, self is a little strange to use, because the function could be applied to many classes. Most often I've seen inst or cls used instead.

I once had some code like (and I apologize for lack of creativity in the example):
class Animal:
def __init__(self, volume=1):
self.volume = volume
self.description = "Animal"
def Sound(self):
pass
def GetADog(self, newvolume):
class Dog(Animal):
def Sound(this):
return self.description + ": " + ("woof" * this.volume)
return Dog(newvolume)
Then we have output like:
>>> a = Animal(3)
>>> d = a.GetADog(2)
>>> d.Sound()
'Animal: woofwoof'
I wasn't sure if self within the Dog class would shadow self within the Animal class, so I opted to make Dog's reference the word "this" instead. In my opinion and for that particular application, that was more clear to me.

Because it is a convention, not language syntax. There is a Python style guide that people who program in Python follow. This way libraries have a familiar look and feel. Python places a lot of emphasis on readability, and consistency is an important part of this.

I think that the main reason self is used by convention rather than being a Python keyword is because it's simpler to have all methods/functions take arguments the same way rather than having to put together different argument forms for functions, class methods, instance methods, etc.
Note that if you have an actual class method (i.e. one defined using the classmethod decorator), the convention is to use "cls" instead of "self".