I am trying to implement a class in which an attempt to access any attributes that do not exist in the current class or any of its ancestors will attempt to access those attributes from a member. Below is a trivial version of what I am trying to do.
class Foo:
def __init__(self, value):
self._value = value
def __getattr__(self, name):
return getattr(self._value, name)
if __name__ == '__main__':
print(Foo(5) > Foo(4)) # should do 5 > 4 (or (5).__gt__(4))
However, this raises a TypeError. Even using the operator module's attrgetter class does the same thing. I was taking a look at the documentation regarding customizing attribute access, but I didn't find it an easy read. How can I get around this?
If I understand you correctly, what you are doing is correct, but it still won't work for what you're trying to use it for. The reason is that implicit magic-method lookup does not use __getattr__ (or __getattribute__ or any other such thing). The methods have to actually explicitly be there with their magic names. Your approach will work for normal attributes, but not magic methods. (Note that if you do Foo(5).__lt__(4) explicitly, it will work; it's only the implicit "magic" lookup --- e.g., calling __lt__ when < is used) --- that is blocked.)
This post describes an approach for autogenerating magic methods using a metaclass. If you only need certain methods, you can just define them on the class manually.
__*__ methods will not work unless they actually exist - so neither __getattr__ nor __getattribute__ will allow you to proxy those calls. You must create every single methods manually.
Yes, this does involve quite a bit of copy&paste. And yes, it's perfectly fine in this case.
You might be able to use the werkzeug LocalProxy class as a base or instead of your own class; your code would look like this when using LocalProxy:
print(LocalProxy(lambda: 5) > LocalProxy(lambda: 4))
Related
I just can't see why do we need to use #staticmethod. Let's start with an exmaple.
class test1:
def __init__(self,value):
self.value=value
#staticmethod
def static_add_one(value):
return value+1
#property
def new_val(self):
self.value=self.static_add_one(self.value)
return self.value
a=test1(3)
print(a.new_val) ## >>> 4
class test2:
def __init__(self,value):
self.value=value
def static_add_one(self,value):
return value+1
#property
def new_val(self):
self.value=self.static_add_one(self.value)
return self.value
b=test2(3)
print(b.new_val) ## >>> 4
In the example above, the method, static_add_one , in the two classes do not require the instance of the class(self) in calculation.
The method static_add_one in the class test1 is decorated by #staticmethod and work properly.
But at the same time, the method static_add_one in the class test2 which has no #staticmethod decoration also works properly by using a trick that provides a self in the argument but doesn't use it at all.
So what is the benefit of using #staticmethod? Does it improve the performance? Or is it just due to the zen of python which states that "Explicit is better than implicit"?
The reason to use staticmethod is if you have something that could be written as a standalone function (not part of any class), but you want to keep it within the class because it's somehow semantically related to the class. (For instance, it could be a function that doesn't require any information from the class, but whose behavior is specific to the class, so that subclasses might want to override it.) In many cases, it could make just as much sense to write something as a standalone function instead of a staticmethod.
Your example isn't really the same. A key difference is that, even though you don't use self, you still need an instance to call static_add_one --- you can't call it directly on the class with test2.static_add_one(1). So there is a genuine difference in behavior there. The most serious "rival" to a staticmethod isn't a regular method that ignores self, but a standalone function.
Today I suddenly find a benefit of using #staticmethod.
If you created a staticmethod within a class, you don't need to create an instance of the class before using the staticmethod.
For example,
class File1:
def __init__(self, path):
out=self.parse(path)
def parse(self, path):
..parsing works..
return x
class File2:
def __init__(self, path):
out=self.parse(path)
#staticmethod
def parse(path):
..parsing works..
return x
if __name__=='__main__':
path='abc.txt'
File1.parse(path) #TypeError: unbound method parse() ....
File2.parse(path) #Goal!!!!!!!!!!!!!!!!!!!!
Since the method parse is strongly related to the classes File1 and File2, it is more natural to put it inside the class. However, sometimes this parse method may also be used in other classes under some circumstances. If you want to do so using File1, you must create an instance of File1 before calling the method parse. While using staticmethod in the class File2, you may directly call the method by using the syntax File2.parse.
This makes your works more convenient and natural.
I will add something other answers didn't mention. It's not only a matter of modularity, of putting something next to other logically related parts. It's also that the method could be non-static at other point of the hierarchy (i.e. in a subclass or superclass) and thus participate in polymorphism (type based dispatching). So if you put that function outside the class you will be precluding subclasses from effectively overriding it. Now, say you realize you don't need self in function C.f of class C, you have three two options:
Put it outside the class. But we just decided against this.
Do nothing new: while unused, still keep the self parameter.
Declare you are not using the self parameter, while still letting other C methods to call f as self.f, which is required if you wish to keep open the possibility of further overrides of f that do depend on some instance state.
Option 2 demands less conceptual baggage (you already have to know about self and methods-as-bound-functions, because it's the more general case). But you still may prefer to be explicit about self not being using (and the interpreter could even reward you with some optimization, not having to partially apply a function to self). In that case, you pick option 3 and add #staticmethod on top of your function.
Use #staticmethod for methods that don't need to operate on a specific object, but that you still want located in the scope of the class (as opposed to module scope).
Your example in test2.static_add_one wastes its time passing an unused self parameter, but otherwise works the same as test1.static_add_one. Note that this extraneous parameter can't be optimized away.
One example I can think of is in a Django project I have, where a model class represents a database table, and an object of that class represents a record. There are some functions used by the class that are stand-alone and do not need an object to operate on, for example a function that converts a title into a "slug", which is a representation of the title that follows the character set limits imposed by URL syntax. The function that converts a title to a slug is declared as a staticmethod precisely to strongly associate it with the class that uses it.
The question refers to which one is preferable to be used in which use case, not about the technical background.
In python, you can control the access of attributes via a property, a descriptor, or magic methods. Which one is most pythonic in which use case? All of them seem to have the same effect (see the examples below).
I am looking for an answer like:
Property: Should be used in case of …
Descriptor: In the case of … it should be used instead of a property.
Magic method: Only use if ….
Example
A use case would be an attribute that might not be able to be set in the __init__ method, for example because the object is not present in the database yet, but at a later time. Each time the attribute is accessed, it should be tried to be set and returned.
As an example that works with Copy&Paste in the Python shell, there is a class that wants to present its attribute only the second time it is asked for it. So, which one is the best way, or are there different situations one of them is preferable? Here are the three ways to implement it:
With Property::
class ContactBook(object):
intents = 0
def __init__(self):
self.__first_person = None
def get_first_person(self):
ContactBook.intents += 1
if self.__first_person is None:
if ContactBook.intents > 1:
value = 'Mr. First'
self.__first_person = value
else:
return None
return self.__first_person
def set_first_person(self, value):
self.__first_person = value
first_person = property(get_first_person, set_first_person)
With __getattribute__::
class ContactBook(object):
intents = 0
def __init__(self):
self.first_person = None
def __getattribute__(self, name):
if name == 'first_person' \
and object.__getattribute__(self, name) is None:
ContactBook.intents += 1
if ContactBook.intents > 1:
value = 'Mr. First'
self.first_person = value
else:
value = None
else:
value = object.__getattribute__(self, name)
return value
Descriptor::
class FirstPerson(object):
def __init__(self, value=None):
self.value = None
def __get__(self, instance, owner):
if self.value is None:
ContactBook.intents += 1
if ContactBook.intents > 1:
self.value = 'Mr. First'
else:
return None
return self.value
class ContactBook(object):
intents = 0
first_person = FirstPerson()
Each one of it has this behavior::
book = ContactBook()
print(book.first_person)
# >>None
print(book.first_person)
# >>Mr. First
Basically, use the simplest one you can. Roughly speaking, the order of complexity/heavy-duty-ness goes: regular attribute, property, __getattr__, __getattribute__/descriptor. (__getattribute__ and custom descriptors are both things you probably won't need to do very often.) This leads to some simple rules of thumb:
Don't use a property if a normal attribute will work.
Don't write your own descriptor if a property will work.
Don't use __getattr__ if a property will work.
Don't use __getattribute__ if __getattr__ will work.
Stated a bit more specifically: use a property to customize handling of one or a small set of attributes; use __getattr__ to customize handling of all attributes, or all except a small set; use __getattribute__ if you were hoping to use __getattr__ but it doesn't quite work; write your own descriptor class if you are doing something very complicated.
You use a property when you have one or a small set of attributes whose getting/setting you want to hook into. That is, you want things like obj.prop and obj.prop = 2 to secretly call a function that you write to customize what happens.
You would use __getattr__ when you want to do that for so many attributes that you don't actually want to define them all individually, but rather want to customize the whole attribute-access process as a whole. In other words, instead of hooking into obj.prop1, and obj.prop2, etc., you have so many that you want to be able to hook into obj.<anything>, and handle that in general.
However, __getattr__ still won't let you override what happens for attributes that really do exist, it just lets you hook in with a blanket handling for any use of attributes that would otherwise raise an AttributeError. Using __getattribute__ lets you hook in to handle everything, even normal attributes that would have worked without messing with __getattribute__. Because of this, using __getattribute__ has the potential to break fairly basic behavior, so you should only use it if you considered using __getattr__ and it wasn't enough. It also can have a noticeable performance impact. You might for instance need to use __getattribute__ if you're wrapping a class that defines some attributes, and you want to be able to wrap those attributes in a custom way, so that they work as usual in some situations but get custom behavior in other situations.
Finally, I would say writing your own descriptor is a fairly advanced task. property is a descriptor, and for probably 95% of cases it's the only one you'll need. A good simple example of why you might write your own descriptor is given here: basically, you might do it if you would otherwise have to write several propertys with similar behavior; a descriptor lets you factor out the common behavior to avoid code repetition. Custom descriptors are used, for instance, to drive systems like like Django and SQLAlchemy. If you find yourself writing something at that level of complexity you might need to write a custom descriptor.
In your example, property would be the best choice. It is usually (not always) a red flag if you're doing if name == 'somespecificname' inside __getattribute__. If you only need to specially handle one specific name, you can probably do it without stooping to the level of __getattribute__. Likewise, it doesn't make sense to write your own descriptor if all you write for its __get__ is something you could have written in a property's getter method.
__getattribute__ is the hook that enables property (and other descriptors) to work in the first place and is called for all attribute access on an object. Consider it a lower-level API when a property or even a custom descriptor is not enough for your needs. __getattr__ is called by __getattribute__ when no attribute has been located through other means, as a fallback.
Use property for dynamic attributes with a fixed name, __getattr__ for attributes of a more dynamic nature (e.g. a series of attributes that map to values in an algorithmic manner).
Descriptors are used when you need to bind arbitrary objects to an instance. When you need to replace method objects with something more advanced for example; a recent example is a class-based decorator wrapping methods that needed to support additional attributes and methods on the method object. Generally, when you are still thinking in terms of scalar attributes, you don't need descriptors.
I have an instance of a cx_Oracle.Connection called x and I'm trying to print x.clientinfo or x.module and getting:
attribute 'module' of 'cx_Oracle.Connection' objects is not readable
(What's weird is that I can do print x.username)
I can still do dir(x) with success and I don't have time to look at the source code of cx_Oracle (lots of it implemented in C) so I'm wondering how the implementer was able to do this? Was it by rolling descriptors? or something related to __getitem__? What would be the motivation for this?
You can do this pretty easily in Python with a custom descriptor.
Look at the Descriptor Example in the HOWTO. If you just change the __get__ method to raise an AttributeError… that's it. We might as well rename it and strip out the logging stuff to make it simpler.
class WriteOnly(object):
"""A data descriptor that can't be read.
"""
def __init__(self, initval=None, name='var'):
self.val = initval
self.name = name
def __get__(self, obj, objtype):
raise AttributeError("No peeking at attribute '{}'!".format(self.name))
def __set__(self, obj, val):
self.val = val
class MyClass(object):
x = WriteOnly(0, 'x')
m = MyClass()
m.x = 20 # works
print(m.x) # raises AttributeError
Note that in 2.x, if you forget the (object) and create a classic class, descriptors won't work. (I believe descriptors themselves can actually be classic classes… but don't do that.) In 3.x, there are no classic classes, so that's not a problem.
So, if the value is write-only, how would you ever read it?
Well, this toy example is useless. But you could, e.g., set some private attribute on obj rather than on yourself, at which point code that knows where the data are stored can find it, but casual introspection can't.
But you don't even need descriptors. If you want an attribute that's write-only no matter what class you attach it to, that's one thing, but if you just want to block read access to certain members of a particular class, there's an easier way:
class MyClass(object):
def __getattribute__(self, name):
if name in ('x', 'y', 'z'):
raise AttributeError("No! Bad user! You cannot see my '{}'!".format(name))
return super().__getattribute__(self, name)
m = MyClass()
m.x = 20
m.x # same exception
For more details, see the __getattr__ and __getattribute__ documentation from the data model chapter in the docs.
In 2.x, if you leave the (object) off and create a classic class, the rules for attribute lookup are completely different, and not completely documented, and you really don't want to learn them unless you're planning to spend a lot of time in the 90s, so… don't do that. Also, 2.x will obviously need the 2.x-style explicit super call instead of the 3.x-style magic super().
From the C API side, you've got most of the same hooks, but they're a bit different. See PyTypeObjects for details, but basically:
tp_getset lets you automatically build descriptors out of getter and setter functions, which is similar to #property but not identical.
tp_descr_get and tp_descr_set are for building descriptors separately.
tp_getattro and tp_setattro are similar to __getattr__ and __setattr__, except that the rules for when they get called are a little different, and you typically call PyObject_GenericGetAttr instead of delegating to super() when you know you have no base classes that need to hook attribute access.
Still, why would you do that?
Personally, I've done stuff like this to learn more about the Python data model and descriptors, but that's hardly a reason to put it in a published library.
I'm guessing that more often than not, someone does it because they're trying to force a mistaken notion of OO encapsulation (based on the traditional C++ model) on Python—or, worse, trying to build Java-style security-by-encapsulation (which doesn't work without a secure class loader and all that comes with it).
But there could be cases where there's some generic code that uses these objects via introspection, and "tricking" that code could be useful in a way that trying to trick human users isn't. For example, imagine a serialization library that tried to pickle or JSON-ify or whatever all of the attributes. You could easily write it ignore non-readable attributes. (Of course you could just as easily make it, say, ignore attributes prefixed with a _…)
As for why cx_Oracle did it… I've never even looked at it, so I have no idea.
class Foo(object):
pass
foo = Foo()
def bar(self):
print 'bar'
Foo.bar = bar
foo.bar() #bar
Coming from JavaScript, if a "class" prototype was augmented with a certain attribute. It is known that all instances of that "class" would have that attribute in its prototype chain, hence no modifications has to be done on any of its instances or "sub-classes".
In that sense, how can a Class-based language like Python achieve Monkey patching?
The real question is, how can it not? In Python, classes are first-class objects in their own right. Attribute access on instances of a class is resolved by looking up attributes on the instance, and then the class, and then the parent classes (in the method resolution order.) These lookups are all done at runtime (as is everything in Python.) If you add an attribute to a class after you create an instance, the instance will still "see" the new attribute, simply because nothing prevents it.
In other words, it works because Python doesn't cache attributes (unless your code does), because it doesn't use negative caching or shadowclasses or any of the optimization techniques that would inhibit it (or, when Python implementations do, they take into account the class might change) and because everything is runtime.
I just read through a bunch of documentation, and as far as I can tell, the whole story of how foo.bar is resolved, is as follows:
Can we find foo.__getattribute__ by the following process? If so, use the result of foo.__getattribute__('bar').
(Looking up __getattribute__ will not cause infinite recursion, but the implementation of it might.)
(In reality, we will always find __getattribute__ in new-style objects, as a default implementation is provided in object - but that implementation is of the following process. ;) )
(If we define a __getattribute__ method in Foo, and access foo.__getattribute__, foo.__getattribute__('__getattribute__') will be called! But this does not imply infinite recursion - if you are careful ;) )
Is bar a "special" name for an attribute provided by the Python runtime (e.g. __dict__, __class__, __bases__, __mro__)? If so, use that. (As far as I can tell, __getattribute__ falls into this category, which avoids infinite recursion.)
Is bar in the foo.__dict__ dict? If so, use foo.__dict__['bar'].
Does foo.__mro__ exist (i.e., is foo actually a class)? If so,
For each base-class base in foo.__mro__[1:]:
(Note that the first one will be foo itself, which we already searched.)
Is bar in base.__dict__? If so:
Let x be base.__dict__['bar'].
Can we find (again, recursively, but it won't cause a problem) x.__get__?
If so, use x.__get__(foo, foo.__class__).
(Note that the function bar is, itself, an object, and the Python compiler automatically gives functions a __get__ attribute which is designed to be used this way.)
Otherwise, use x.
For each base-class base of foo.__class__.__mro__:
(Note that this recursion is not a problem: those attributes should always exist, and fall into the "provided by the Python runtime" case. foo.__class__.__mro__[0] will always be foo.__class__, i.e. Foo in our example.)
(Note that we do this even if foo.__mro__ exists. This is because classes have a class, too: its name is type, and it provides, among other things, the method used to calculate __mro__ attributes in the first place.)
Is bar in base.__dict__? If so:
Let x be base.__dict__['bar'].
Can we find (again, recursively, but it won't cause a problem) x.__get__?
If so, use x.__get__(foo, foo.__class__).
(Note that the function bar is, itself, an object, and the Python compiler automatically gives functions a __get__ attribute which is designed to be used this way.)
Otherwise, use x.
If we still haven't found something to use: can we find foo.__getattr__ by the preceding process? If so, use the result of foo.__getattr__('bar').
If everything failed, raise AttributeError.
bar.__get__ is not really a function - it's a "method-wrapper" - but you can imagine it being implemented vaguely like this:
# Somewhere in the Python internals
class __method_wrapper(object):
def __init__(self, func):
self.func = func
def __call__(self, obj, cls):
return lambda *args, **kwargs: func(obj, *args, **kwargs)
# Except it actually returns a "bound method" object
# that uses cls for its __repr__
# and there is a __repr__ for the method_wrapper that I *think*
# uses the hashcode of the underlying function, rather than of itself,
# but I'm not sure.
# Automatically done after compiling bar
bar.__get__ = __method_wrapper(bar)
The "binding" that happens within the __get__ automatically attached to bar (called a descriptor), by the way, is more or less the reason why you have to specify self parameters explicitly for Python methods. In Javascript, this itself is magical; in Python, it is merely the process of binding things to self that is magical. ;)
And yes, you can explicitly set a __get__ method on your own objects and have it do special things when you set a class attribute to an instance of the object and then access it from an instance of that other class. Python is extremely reflective. :) But if you want to learn how to do that, and get a really full understanding of the situation, you have a lot of reading to do. ;)
I am used to that Python allows some neat tricks to delegate functionality to other objects. One example is delegation to contained objects.
But it seams, that I don't have luck, when I want to delegate __contains __:
class A(object):
def __init__(self):
self.mydict = {}
self.__contains__ = self.mydict.__contains__
a = A()
1 in a
I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: argument of type 'A' is not iterable
What I am making wrong? When I call a.__contains __(1), everything goes smooth. I even tried to define an __iter __ method in A to make A more look like an iterable, but it did not help. What I am missing out here?
Special methods such as __contains__ are only special when defined on the class, not on the instance (except in legacy classes in Python 2, which you should not use anyway).
So, do your delegation at class level:
class A(object):
def __init__(self):
self.mydict = {}
def __contains__(self, other):
return self.mydict.__contains__(other)
I'd actually prefer to spell the latter as return other in self.mydict, but that's a minor style issue.
Edit: if and when "totally dynamic per-instance redirecting of special methods" (like old-style classes offered) is indispensable, it's not hard to implement it with new-style classes: you just need each instance that has such peculiar need to be wrapped in its own special class. For example:
class BlackMagic(object):
def __init__(self):
self.mydict = {}
self.__class__ = type(self.__class__.__name__, (self.__class__,), {})
self.__class__.__contains__ = self.mydict.__contains__
Essentially, after the little bit of black magic reassigning self.__class__ to a new class object (which behaves just like the previous one but has an empty dict and no other instances except this one self), anywhere in an old-style class you would assign to self.__magicname__, assign to self.__class__.__magicname__ instead (and make sure it's a built-in or staticmethod, not a normal Python function, unless of course in some different case you do want it to receive the self when called on the instance).
Incidentally, the in operator on an instance of this BlackMagic class is faster, as it happens, than with any of the previously proposed solutions -- or at least so I'm measuring with my usual trusty -mtimeit (going directly to the built-in method, instead of following normal lookup routes involving inheritance and descriptors, shaves a bit of the overhead).
A metaclass to automate the self.__class__-per-instance idea would not be hard to write (it could do the dirty work in the generated class's __new__ method, and maybe also set all magic names to actually assign on the class if assigned on the instance, either via __setattr__ or many, many properties). But that would be justified only if the need for this feature was really widespread (e.g. porting a huge ancient Python 1.5.2 project that liberally use "per-instance special methods" to modern Python, including Python 3).
Do I recommend "clever" or "black magic" solutions? No, I don't: almost invariably it's better to do things in simple, straightforward ways. But "almost" is an important word here, and it's nice to have at hand such advanced "hooks" for the rare, but not non-existent, situations where their use may actually be warranted.