I have a class that has a numpy.ndarray as a member and behaves similar to ndarray by overloading __getitem__ and __getattr__:
class Foo(object):
def __init__(values):
# numpy.ndarray
self._values = values
def __getitem__(self, key):
return self._values[key]
def __getattr__(self, name):
return getattr(self._values, name)
Thus I can use the numpy method like shape, size, ... directly on an object of this class. I can also do things like obj.__add__(1), which will add 1 to obj._values. However, if I try obj + 1 it raises "unsupported operand type(s)". I would like to get the same behaviour for obj + 1 as obj.__add__(1). Is this possible without adding __add__ to Foo?
I can see what you're trying to do here, but it's not going to work the way you think it should. This is a very non-obvious subtlety in Python.
What you're thinking is that what you do obj + 1 Python is actually calling obj.__add__(1) and that, failing to find an __add__ attribute on obj it will fall through to its __getattr__.
But this is not exactly how it works for arithmetic operators, the implementation of which is actually significantly more complicated. In this case if obj does not have an __add__ method, it will attempt to call the right-hand operand's __radd__ (for "right add") method to see if 1 knows how to add with the left-hand operator. It does not so you get an exception.
There are other subtleties involving type slots that I won't get into.
If you want your class to act as a proxy for an ndarray you have a few options. It really depends on what you're actually trying to accomplish, which you might consider asking in a separate question. You might just be able to subclass ndarray directly and implement your additional functionality in the subclass.
If you don't want a subclass of ndarray you might also consider using a proxy, such as the ObjectProxy from wrapt. This may or may not be what you want. It will make an object that walks, talks, quacks like, and is even named ndarray, though you can still subclass ObjectProxy to override methods that you don't want proxied.
Otherwise there's the tedious manual method.
Related
Suppose I want to have a wrapper class Image for numpy array. My goal is to let it behave just like a 2D array but with some additional functionality (which is not important here). I am doing so because inheriting numpy array is way more troublesome.
import numpy as np
class Image(object):
def __init__(self, data: np.ndarray):
self._data = np.array(data)
def __getitem__(self, item):
return self._data.__getitem__(item)
def __setitem__(self, key, value):
self._data.__setitem__(key, value)
def __getattr__(self, item):
# delegates array's attributes and methods, except dunders.
try:
return getattr(self._data, item)
except AttributeError:
raise AttributeError()
# binary operations
def __add__(self, other):
return Image(self._data.__add__(other))
def __sub__(self, other):
return Image(self._data.__sub__(other))
# many more follow ... How to avoid this redundancy?
As you can see, I want to have all the magic methods for numeric operations, just like a normal numpy array, but with the return values as Image type. So the implementations of these magic methods, i.e. the __add__, __sub__, __truediv__ and so on, are almost the same and it's kind of silly. My question is if there is a way to avoid this redundancy?
Beyond what specifically I am doing here, is there a way to code up the magic methods in one place via some meta-programming technique, or it's just impossible? I searched some about python metaclass, but it's still not clear to me.
Notice __getattr__ won't handle delegates for magic methods. See this.
Edit
Just to clarify, I understand inheritance is a general solution for a problem like this, though my experience is very limited. But I feel inheriting numpy array really isn't a good idea. Because numpy array needs to handle view-casting and ufuncs (see this). And when you use your subclass in other py-libs, you also need to think how your array subclass gets along with other array subclasses. See my stupid gh-issue. That's why I am looking for alternatives.
The magic methods are always looked up in the class and bypass getattribute entirely so you must define them in the class. https://docs.python.org/3/reference/datamodel.html#special-lookup
However, you can save yourself some typing:
import operator
def make_bin_op(oper):
def op(self, other):
if isinstance(other, Image):
return Image(oper(self._data, other._data))
else:
return Image(oper(self._data, other))
return op
class Image:
...
__add__ = make_bin_op(operator.add)
__sub__ = make_bin_op(operator.sub)
If you want you could make a dict of operator dunder names and the corresponding operators and add them with a decorator. e.g.
OPER_DICT = {'__add__' : operator.add, '__sub__' : operator.sub, ...}
def add_operators(cls):
for k,v in OPER_DICT.items():
setattr(cls, k, make_bin_op(v))
#add_operators
class Image:
...
You could use a metaclass to do the same thing. However, you probably don't want use a metaclass unless you really understand what's going on.
What you're after is the concept called inheritance, a key part of object-oriented programming (see Wikipedia here.
When you define your class with class Image(object):, what that means is that Image is a subclass of object, which is a built-in type that does very little. Your functionality is added on to that more-or-less blank concept. But if instead you defined your class with class Image(np.array):, then Image would be a subclass of array, which means it would inherit all the default functionality of the array class. Essentially, any class method you want to leave as is you simply shouldn't redefine. If you don't write a __getitem__ function, it uses the one defined in array.
If you need to add additional functionality in any of those functions, you can still redefine them (called overriding) and then use super().__getitem__ (or whatever) to access the function defined in the inherited class. This often happens with __init__ for example.
For a more thorough explanation, take a look at the chapter on inheritance in Think Python.
Was just thinking about Python's dict "function" and starting to realize that dict isn't really a function at all. For example, if we do dir(dict), we get all sorts of methods that aren't include in the usual namespace of an user defined function. Extending that thought, its similar to dir(list) and dir(len). They aren't function, but really types. But then I'm confused about the documentation page, http://docs.python.org/2/library/functions.html, which clearly says functions. (I guess it should really just says builtin callables)
So what gives? (Starting to seem that making the distinction of classes and functions is trivial)
It's a callable, as are classes in general. Calling dict() is effectively to call the dict constructor. It is like when you define your own class (C, say) and you call C() to instantiate it.
One way that dict is special, compared to, say, sum, is that though both are callable, and both are implemented in C (in cpython, anyway), dict is a type; that is, isinstance(dict, type) == True. This means that you can use dict as the base class for other types, you can write:
class MyDictSubclass(dict):
pass
but not
class MySumSubclass(sum):
pass
This can be useful to make classes that behave almost like a builtin object, but with some enhancements. For instance, you can define a subclass of tuple that implements + as vector addition instead of concatenation:
class Vector(tuple):
def __add__(self, other):
return Vector(x + y for x, y in zip(self, other))
Which brings up another interesting point. type is also implemented in C. It's also callable. Like dict (and unlike sum) it's an instance of type; isinstance(type, type) == True. Because of this weird, seemingly impossible cycle, type can be used to make new classes of classes, (called metaclasses). You can write:
class MyTypeSubclass(type):
pass
class MyClass(object):
__metaclass__ = MyTypeSubclass
or, in Python 3:
class MyClass(metaclass=MyTypeSubclass):
pass
Which give the interesting result that isinstance(MyClass, MyTypeSubclass) == True. How this is useful is a bit beyond the scope of this answer, though.
dict() is a constructor for a dict instance. When you do dir(dict) you're looking at the attributes of class dict. When you write a = dict() you're setting a to a new instance of type dict.
I'm assuming here that dict() is what you're referring to as the "dict function". Or are you calling an indexed instance of dict, e.g. a['my_key'] a function?
Note that calling dir on the constructor dict.__init__
dir(dict.__init__)
gives you what you would expect, including the same stuff as you'd get for any other function. Since a call to the dict() constructor results in a call to dict.__init__(instance), that explains where those function attributes went. (Of course there's a little extra behind-the-scenes work in any constructor, but that's the same for dicts as for any object.)
I am trying to implement a class in which an attempt to access any attributes that do not exist in the current class or any of its ancestors will attempt to access those attributes from a member. Below is a trivial version of what I am trying to do.
class Foo:
def __init__(self, value):
self._value = value
def __getattr__(self, name):
return getattr(self._value, name)
if __name__ == '__main__':
print(Foo(5) > Foo(4)) # should do 5 > 4 (or (5).__gt__(4))
However, this raises a TypeError. Even using the operator module's attrgetter class does the same thing. I was taking a look at the documentation regarding customizing attribute access, but I didn't find it an easy read. How can I get around this?
If I understand you correctly, what you are doing is correct, but it still won't work for what you're trying to use it for. The reason is that implicit magic-method lookup does not use __getattr__ (or __getattribute__ or any other such thing). The methods have to actually explicitly be there with their magic names. Your approach will work for normal attributes, but not magic methods. (Note that if you do Foo(5).__lt__(4) explicitly, it will work; it's only the implicit "magic" lookup --- e.g., calling __lt__ when < is used) --- that is blocked.)
This post describes an approach for autogenerating magic methods using a metaclass. If you only need certain methods, you can just define them on the class manually.
__*__ methods will not work unless they actually exist - so neither __getattr__ nor __getattribute__ will allow you to proxy those calls. You must create every single methods manually.
Yes, this does involve quite a bit of copy&paste. And yes, it's perfectly fine in this case.
You might be able to use the werkzeug LocalProxy class as a base or instead of your own class; your code would look like this when using LocalProxy:
print(LocalProxy(lambda: 5) > LocalProxy(lambda: 4))
class Foo(object):
pass
foo = Foo()
def bar(self):
print 'bar'
Foo.bar = bar
foo.bar() #bar
Coming from JavaScript, if a "class" prototype was augmented with a certain attribute. It is known that all instances of that "class" would have that attribute in its prototype chain, hence no modifications has to be done on any of its instances or "sub-classes".
In that sense, how can a Class-based language like Python achieve Monkey patching?
The real question is, how can it not? In Python, classes are first-class objects in their own right. Attribute access on instances of a class is resolved by looking up attributes on the instance, and then the class, and then the parent classes (in the method resolution order.) These lookups are all done at runtime (as is everything in Python.) If you add an attribute to a class after you create an instance, the instance will still "see" the new attribute, simply because nothing prevents it.
In other words, it works because Python doesn't cache attributes (unless your code does), because it doesn't use negative caching or shadowclasses or any of the optimization techniques that would inhibit it (or, when Python implementations do, they take into account the class might change) and because everything is runtime.
I just read through a bunch of documentation, and as far as I can tell, the whole story of how foo.bar is resolved, is as follows:
Can we find foo.__getattribute__ by the following process? If so, use the result of foo.__getattribute__('bar').
(Looking up __getattribute__ will not cause infinite recursion, but the implementation of it might.)
(In reality, we will always find __getattribute__ in new-style objects, as a default implementation is provided in object - but that implementation is of the following process. ;) )
(If we define a __getattribute__ method in Foo, and access foo.__getattribute__, foo.__getattribute__('__getattribute__') will be called! But this does not imply infinite recursion - if you are careful ;) )
Is bar a "special" name for an attribute provided by the Python runtime (e.g. __dict__, __class__, __bases__, __mro__)? If so, use that. (As far as I can tell, __getattribute__ falls into this category, which avoids infinite recursion.)
Is bar in the foo.__dict__ dict? If so, use foo.__dict__['bar'].
Does foo.__mro__ exist (i.e., is foo actually a class)? If so,
For each base-class base in foo.__mro__[1:]:
(Note that the first one will be foo itself, which we already searched.)
Is bar in base.__dict__? If so:
Let x be base.__dict__['bar'].
Can we find (again, recursively, but it won't cause a problem) x.__get__?
If so, use x.__get__(foo, foo.__class__).
(Note that the function bar is, itself, an object, and the Python compiler automatically gives functions a __get__ attribute which is designed to be used this way.)
Otherwise, use x.
For each base-class base of foo.__class__.__mro__:
(Note that this recursion is not a problem: those attributes should always exist, and fall into the "provided by the Python runtime" case. foo.__class__.__mro__[0] will always be foo.__class__, i.e. Foo in our example.)
(Note that we do this even if foo.__mro__ exists. This is because classes have a class, too: its name is type, and it provides, among other things, the method used to calculate __mro__ attributes in the first place.)
Is bar in base.__dict__? If so:
Let x be base.__dict__['bar'].
Can we find (again, recursively, but it won't cause a problem) x.__get__?
If so, use x.__get__(foo, foo.__class__).
(Note that the function bar is, itself, an object, and the Python compiler automatically gives functions a __get__ attribute which is designed to be used this way.)
Otherwise, use x.
If we still haven't found something to use: can we find foo.__getattr__ by the preceding process? If so, use the result of foo.__getattr__('bar').
If everything failed, raise AttributeError.
bar.__get__ is not really a function - it's a "method-wrapper" - but you can imagine it being implemented vaguely like this:
# Somewhere in the Python internals
class __method_wrapper(object):
def __init__(self, func):
self.func = func
def __call__(self, obj, cls):
return lambda *args, **kwargs: func(obj, *args, **kwargs)
# Except it actually returns a "bound method" object
# that uses cls for its __repr__
# and there is a __repr__ for the method_wrapper that I *think*
# uses the hashcode of the underlying function, rather than of itself,
# but I'm not sure.
# Automatically done after compiling bar
bar.__get__ = __method_wrapper(bar)
The "binding" that happens within the __get__ automatically attached to bar (called a descriptor), by the way, is more or less the reason why you have to specify self parameters explicitly for Python methods. In Javascript, this itself is magical; in Python, it is merely the process of binding things to self that is magical. ;)
And yes, you can explicitly set a __get__ method on your own objects and have it do special things when you set a class attribute to an instance of the object and then access it from an instance of that other class. Python is extremely reflective. :) But if you want to learn how to do that, and get a really full understanding of the situation, you have a lot of reading to do. ;)
Might be a n00b question, but I currently have a class that implements an iterator so I can do something like
for i in class():
but I want to be able to access the class by index as well like
class()[1]
How can I do that?
Thanks!
The current accepted answer from #Ignacio Vazquez-Abrams is sufficient. However, others interested in this question may want to consider inheriting their class from an abstract base class (ABC) (such as those found in the standard module collections.abc). This does a number of things (there are probably others as well):
ensures that all of the methods you need to treat your object "like a ____" are there
it is self-documenting, in that someone reading your code is able to instantly know that you intend your object to "act like a ____".
allows isinstance(myobject,SomeABC) to work correctly.
often provides methods auto-magically so we don't have to define them ourselves
(Note that, in addition to the above, creating your own ABC can allow you to test for the presence of a specific method or set of methods in any object, and based on this to declare that object to be a subclass of the ABC, even if the object does not inherit from the ABCdirectly. See this answer for more information.)
Example: implement a read-only, list-like class using ABC
Now as an example, let's choose and implement an ABC for the class in the original question. There are two requirements:
the class is iterable
access the class by index
Obviously, this class is going to be some kind of collection. So what we will do is look at our menu of collection ABC's to find the appropriate ABC (note that there are also numeric ABCs). The appropriate ABC is dependent upon which abstract methods we wish to use in our class.
We see that an Iterable is what we are after if we want to use the method __iter__(), which is what we need in order to do things like for o in myobject:. However, an Iterable does not include the method __getitem__(), which is what we need in order to do things like myobject[i]. So we'll need to use a different ABC.
On down the collections.abc menu of abstract base classes, we see that a Sequence is the simplest ABC to offer the functionality we require. And - would you look at that - we get Iterable functionality as a mixin method - which means we don't have to define it ourselves - for free! We also get __contains__, __reversed__, index, and count. Which, if you think about it, are all things that should be included in any indexed object. If you had forgotten to include them, users of your code (including, potentially, yourself!) might get pretty annoyed (I know I would).
However, there is a second ABC that also offers this combination of functionality (iterable, and accessible by []): a Mapping. Which one do we want to use?
We recall that the requirement is to be able to access the object by index (like a list or a tuple), i.e. not by key (like a dict). Therefore, we select Sequence instead of Mapping.
Sidebar: It's important to note that a Sequence is read-only (as is a Mapping), so it will not allow us to do things like myobject[i] = value, or random.shuffle(myobject). If we want to be able do things like that, we need to continue down the menu of ABCs and use a MutableSequence (or a MutableMapping), which will require implementing several additional methods.
Example Code
Now we are able to make our class. We define it, and have it inherit from Sequence.
from collections.abc import Sequence
class MyClass(Sequence):
pass
If we try to use it, the interpreter will tell us which methods we need to implement before it can be used (note that the methods are also listed on the Python docs page):
>>> myobject = MyClass()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyClass with abstract methods __getitem__, __len__
This tells us that if we go ahead and implement __getitem__ and __len__, we'll be able to use our new class. We might do it like this in Python 3:
from collections.abc import Sequence
class MyClass(Sequence):
def __init__(self,L):
self.L = L
super().__init__()
def __getitem__(self, i):
return self.L[i]
def __len__(self):
return len(self.L)
# Let's test it:
myobject = MyClass([1,2,3])
try:
for idx,_ in enumerate(myobject):
print(myobject[idx])
except Exception:
print("Gah! No good!")
raise
# No Errors!
It works!
Implement both __iter__() and __getitem__() et alia methods.