I am implementing an array-like object that should be interoperable with standard numpy arrays. I just hit an annoying problem that narrows down to the following:
class MyArray( object ):
def __rmul__( self, other ):
return MyArray() # value not important for current purpose
from numpy import array
print array([1,2,3]) * MyArray()
This yields the following output:
[<__main__.MyArray instance at 0x91903ec>
<__main__.MyArray instance at 0x919038c>
<__main__.MyArray instance at 0x919042c>]
Clearly, rather than calling MyArray().__rmul__( array([1,2,3]) ) as I had hoped, __rmul__ is called for every individual element of the array, and the result wrapped in an object array. This seems to me non compliant with python's coercion rules. More importantly, it renders my left multiplication useless.
Does anybody know a way around this?
(I thought a could fix it using __coerce__ but the linked document explains that that one is no longer invoked in response to binary operators...)
It turns out that numpy offers a simple fix for this problem. The following code works as intended.
class MyArray( object ):
__array_priority__ = 1. # <- fixes the problem
def __rmul__( self, other ):
return MyArray()
More information can be found here.
Related
The below code snippet seems to be returning a handle instead of a value while getting the c_int attribute of my ctypes structure.
How can the method get_val be changed to return the value of either o_num or o_str?
import ctypes
class my_struct(ctypes.Structure):
_fields_=[('o_num',ctypes.c_int),('o_str',ctypes.c_wchar_p),('o_bool',ctypes.c_int),('o_err',ctypes.c_int)]
class my_substruct(ctypes.Structure):
_fields_=[('rows',ctypes.c_int),('columns',ctypes.c_int),('next',ctypes.POINTER(my_struct))]
my_struct._fields_.append(('array',my_substruct))
class oper(ctypes.Union):
_fields_=[('val',my_struct),('type',ctypes.c_wchar_p)]
_mapping_={'num':'o_num','string':'o_str'}
def get_val(self):
return getattr(self.val,self._mapping_[self.type])
aa=my_struct(o_str='hello')
a=oper(aa,'string')
print(a.get_val())
bb=my_struct(o_num=33)
b=oper(bb,'num')
print(b.get_val())
print(bb.o_num)
and for some reasons i dont understand this returns
hello
263314768 ===> this looks like a handle but WHY ???
33
Further to Mark's answer i've also tested with the below change:
class oper(ctypes.Union):
_fields_=[('val',my_struct),('type',ctypes.c_int)]
_mapping_={0:'o_num',1:'o_str'}
but this provides me with an equally weird result:
hello
0 ===> this is equally weird
33
oper is declared as a Union, so val and type occupy the same memory. When you create b=oper(bb,'num') both are written to the same memory location. Only one will be correct. What you see in o_num is probably the c_wchar_p pointing to the 'num' string.
Thanks to Mark's answer, i realised that my problem was coming from the fact that Union [as opposed to Structure] uses a single memory space of the size of the biggest type in that Union. Structures on the other hand, allocates a dedicated memory space for each type in the structure. So, changing my Union to Structure does correct the problem and prevent this automatic casting from happening.
How can we make a class represent itself as a slice when appropriate?
This didn't work:
class MyThing(object):
def __init__(self, start, stop, otherstuff):
self.start = start
self.stop = stop
self.otherstuff = otherstuff
def __index__(self):
return slice(self.start, self.stop)
Expected output:
>>> thing = MyThing(1, 3, 'potato')
>>> 'hello world'[thing]
'el'
Actual output:
TypeError: __index__ returned non-(int,long) (type slice)
Inheriting from slice doesn't work either.
TLDR: It's impossible to make custom classes replace slice for builtins types such as list and tuple.
The __index__ method exists purely to provide an index, which is by definition an integer in python (see the Data Model). You cannot use it for resolving an object to a slice.
I'm afraid that slice seems to be handled specially by python. The interface requires an actual slice; providing its signature (which also includes the indices method) is not sufficient. As you've found out, you cannot inherit from it, so you cannot create new types of slices. Even Cython will not allow you to inherit from it.
So why is slice special? Glad you asked. Welcome to the innards of CPython. Please wash your hands after reading this.
So slice objects are described in slice.rst. Note these two guys:
.. c:var:: PyTypeObject PySlice_Type
The type object for slice objects. This is the same as :class:slice in the
Python layer.
.. c:function:: int PySlice_Check(PyObject *ob)
Return true if ob is a slice object; ob must not be NULL.
Now, this is actually implemented in sliceobject.h as :
#define PySlice_Check(op) (Py_TYPE(op) == &PySlice_Type)
So only the slice type is allowed here. This check is actually used in list_subscript (and tuple subscript, ...) after attempting to use the index protocol (so having __index__ on a slice is a bad idea). A custom container class is free to overwrite __getitem__ and use its own rules, but that's how list (and tuple, ...) does it.
Now, why is it not possible to subclass slice? Well, type actually has a flag indicating whether something can be subclassed. It is checked here and generates the error you have seen:
if (!PyType_HasFeature(base_i, Py_TPFLAGS_BASETYPE)) {
PyErr_Format(PyExc_TypeError,
"type '%.100s' is not an acceptable base type",
base_i->tp_name);
return NULL;
}
I haven't been able to track down how slice (un)sets this value, but the fact that one gets this error means it does. This means you cannot subclass it.
Closing remarks: After remembering some long-forgotten C-(non)-skills, I'm fairly sure this is not about optimization in the strict sense. All existing checks and tricks would still work (at least those I've found).
After washing my hands and digging around in the internet, I've found a few references to similar "issues". Tim Peters has said all there is to say:
Nothing implemented in C is subclassable unless somebody volunteers the work
to make it subclassable; nobody volunteered the work to make the [insert name here]
type subclassable. It sure wasn't at the top of my list wink.
Also see this thread for a short discussion on non-subclass'able types.
Practically all alternative interpreters replicate the behavior to various degrees: Jython, Pyston, IronPython and PyPy (didn't find out how they do it, but they do).
I'M SORRY FOR THE DARK MAGIC
Using Forbiddenfruit and the python's builtin new method I was able to do this:
from forbiddenfruit import curse
class MyThing(int):
def __new__(cls, *args, **kwargs):
magic_slice = slice(args[0], args[1])
curse(slice, 'otherstuff', args[2])
return magic_slice
thing = MyThing(1, 3, 'thing')
print 'hello world'[thing]
print thing.otherstuff
output:
>>> el
>>> thing
I wrote it as a challenge just because everybody said it is impossible, I would never use it on production code IT HAS SO MANY SIDE EFFECTS, you should think again on your structure and needs
A slice can't be in your return type as the method just doesn't support this. You can read more about the __index__ special method here.
I could only come up with a workaround that directly calls the function in your class:
class MyThing(object):
def __init__(self, start, stop, otherstuff):
self.start = start
self.stop = stop
self.otherstuff = otherstuff
def __index__(self):
return slice(self.start, self.stop)
thing = MyThing(1, 3, 'potato')
print 'Hello World'[thing.__index__()]
This will return el.
Basically I have a class which subclasses ndarray and has additional information. When I call np.asarray() on my object, it returns just the numpy array and destroys my additional information.
My question is then this: Is there a way in Python to change how np.asarray() acts on my subclass of ndarray from within my subclass? I don't want to change numpy of course, and I do not want to go through every instance where np.asarray() is called to take care of this.
Thanks in advance!
Chris
Short answer: No. Numpy's asarray() doesn't check e.g. if a special method on the class of its argument exists and so doesn't provide a way to override its behaviour.
Long answer: It's not possible from your subclass, but you can hotpatch the numpy module in your module level code to replace the asarray function with your own wrapper. This is a very hacky solution and I don't recommend it, but it may work for you.
_real_asarray = np.asarray
def _new_asarray(a, dtype=None, order=None):
if isinstance(a, MyClass):
# special handling here
else:
return _real_asarray(a, dtype, order)
np.asarray = _new_asarray
No. Numpy's asarray() is coded to instantiate a regular numpy array, and you can't change that without editing asarray() or changing the caller's code to call your special method instead of asarray()
I am trying to create a subclass of numpy.ndarray. It is very simple, and is just a numpy array with some extra attributes and methods that manipulate those attributes. For the most part, it works fine, however I have a problem when using reductions like np.sum.
First off, I have read both Subclassing ndarray and Zero-Rank Arrays.
It seems that when I create a subclass of ndarray it behaves differently with respect to zero-rank array -> scalar conversion.
In this example I just use the simplest possible derived class, one that doesn't actually do anything:
class XArray(np.ndarray):
pass
x = np.eye(2)
y = x.view(type=XArray)
print type(np.sum(x)), type(np.sum(y))
<type 'numpy.float64'> <type '__main__.XArray'>
The former is a numpy scalar, the latter is a zero-rank array of my subclass. Overriding __new__ and __array_finalize__ as documented in the array subclassing guide doesn't change this behavior.
First, my problem: this breaks object oriented-ness. XArray instances cannot be substituted for ndarray instances transparently without breaking lots of code.
I can fix this by overriding the __array_wrap__ method:
class XArray(np.ndarray):
def __array_wrap__(self, obj):
if len(obj.shape)==0:
return obj[()]
else:
return np.ndarray.__array_wrap__(obj)
a = np.sum(np.eye(2).view(XArray))
print type(a)
<type 'numpy.float64'>
I am fine with this, except for two questions:
Is this the right place to do this special case? I can't figure out where this conversion is happening for normal numpy arrays, so I can't tell where it should happen to my derived class.
Is this enough to make my subclass work, or am I going to continue having compatibility problems. Should I just abandon the idea of subclassing ndarray?
The goal here is to be 100 % compatible with regular numpy arrays. It is OK and expected that some operations will lose the derived type information and return an ndarray base class. I am fine with that, I just can't have code written to operate on ndarray's break.
[Python 3]
I like ndarray but I find it annoying to use.
Here's one problem I face. I want to write class Array that will inherit much of the functionality of ndarray, but has only one way to be instantiated: as a zero-filled array of a certain size. I was hoping to write:
class Array(numpy.ndarray):
def __init__(size):
# What do here?
I'd like to call super().__init__ with some parameters to create a zero-filled array, but it won't work since ndarray uses a global function numpy.zeros (rather than a constructor) to create a zero-filled array.
Questions:
Why does ndarray use global (module) functions instead of constructors in many cases? It is a big annoyance if I'm trying to reuse them in an object-oriented setting.
What's the best way to define class Array that I need? Should I just manually populate ndarray with zeroes, or is there any way to reuse the zeros function?
Why does ndarray use global (module) functions instead of constructors in many cases?
To be compatible/similar to Matlab, where functions like zeros or ones originally came from.
Global factory functions are quick to write and easy to understand. What should the semantics of a constructor be, e.g. how would you express a simple zeros or empty or ones with one single constructor? In fact, such factory functions are quite common, also in other programming languages.
What's the best way to define class Array that I need?
import numpy
class Array(numpy.ndarray):
def __new__(cls, size):
result = numpy.ndarray.__new__(Array, size)
result.fill(0)
return result
arr = Array(5)
def test(a):
print type(a), a
test(arr)
test(arr[2:4])
test(arr.view(int))
arr[2:4] = 5.5
test(arr)
test(arr[2:4])
test(arr.view(int))
Note that this is Python 2, but it would require only small modifications to work with Python 3.
If you don't like ndarray interface then don't inherit it. You can define your own interface and delegate the rest to ndarray and numpy.
import functools
import numpy as np
class Array(object):
def __init__(self, size):
self._array = np.zeros(size)
def __getattr__(self, attr):
try: return getattr(self._array, attr)
except AttributeError:
# extend interface to all functions from numpy
f = getattr(np, attr, None)
if hasattr(f, '__call__'):
return functools.partial(f, self._array)
else:
raise AttributeError(attr)
def allzero(self):
return np.allclose(self._array, 0)
a = Array(10)
# ndarray doesn't have 'sometrue()' that is the same as 'any()' that it has.
assert a.sometrue() == a.any() == False
assert a.allzero()
try: a.non_existent
except AttributeError:
pass
else:
assert 0
Inheritance of ndarray is little bit tricky. ndarray does not even have method __init(self, )___, so it can't be called from subclass, but there are reasons for that. Please see numpy documentation of subclassing.
By the way could you be more specific of your particular needs? It's still quite easy to cook up a class (utilizing ndarray) for your own needs, but a subclass of ndarray to pass all the numpy machinery is quite different issue.
It seems that I can't comment my own post, odd
#Philipp: It will be called by Python, but not by numpy. There are three ways to instantiate ndarray, and the guidelines how to handle all cases is given on that doc.