I have a class in python that acts as a front-end to a c-library. This library performs simulations and handles very large arrays of data. This library passes forward a ctype array and my wrapper converts it into a proper numpy.ndarray.
class SomeClass(object):
#property
def arr(self):
return numpy.array(self._lib.get_arr())
However, in order to make sure that memory problems don't occur, I keep the ndarray data separate from the library data, so changing the ndarray does not cause a change in the true array being used by the library. However, I can pass along a new array of the same shape and overwrite the library's held array.
#arr.setter
def arr(self, new_arr):
self._lib.set_arr(new_arr.ctypes)
So, I can interact with the array like so:
x = SomeClass()
a = x.arr
a[0] += 1
x.arr = a
My desire is to simplify this even more by allowing syntax to simply be x.arr[0] += 1, which would be more readable and have less variables. I am not exactly sure how to go about creating such a wrapper (I have very little experience making wrapper classes/functions) that mimics properties but allows item access as my example.
How would I go about making such a wrapper class? Is there a better way to accomplish this goal? If you have any advice or reading that could help I would appreciate it very much.
This could work. Array is a proxy for the Numpy/C array:
class Array(object):
def __init__(self):
#self.__lib = ...
self.np_array = numpy.array(self._lib.get_arr())
def __getitem__(self, key):
self.np_array = numpy.array(self._lib.get_arr())
return self.np_array.__getitem__(key)
def __setitem__(self, key, value):
self.np_array.__setitem__(key, value)
self._lib.set_arr(new_arr.ctypes)
def __getattr__(self, name):
"""Delegate to NumPy array."""
try:
return getattr(self.np_array, name)
except AttributeError:
raise AttributeError(
"'Array' object has no attribute {}".format(name))
Should behave like this:
>>> a = Array()
>>> a[1]
1
>>> a[1] = 10
>>> a[1]
10
The 10 should end up in your C array too.
I think your descriptor should return Instance of list-like class which knows about self._lib and will update it during normal operation append, __setitem__, __getitem__, etc.
Related
In Python you can use a tuple as the content of the brackets
matrix = np.zeros((3, 4, 8, 9))
coords = (2, 3)
element = matrix[coords]
I would like to do the same with a custom class CustomDuple
duple = CustomDuple(2, 3)
matrix[duple]
But I receive the IndexError: only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices
Is this currently implemented in Python? Without using a method like CustomDuple.to_tuple(self)->tuple
You didn't provide any example code to show what CustomDuple actually is or does, but given that you want to be able to use it directly as a tuple, we have to assume it at least is most of what a tuple is.
Having said that, inheritance seems like the best solution. However, the tuple __init__ won't allow you to pass more than one argument, even if you subclass it. You can create a factory method to produce a new tuple though, which could take any number of arguments (which is what you seem to be trying to achieve):
import numpy as np
class CustomDuple(tuple):
#classmethod
def new(cls, *args):
if len(args) > 0:
# this is the behaviour you seem to want (for any size tuple)
return cls([*args])
else:
try:
# try and stay close to the normal constructor behaviour
return cls(*args)
except TypeError:
# however, if a single non-iterable was passed, wrap it
return cls([*args])
def first_element(self):
return None if not len(self) else self[0]
duple = CustomDuple.new(2, 3) # creating a duple tuple with its factory method
matrix = np.zeros((3, 4, 8, 9))
print(matrix[duple])
print(duple.first_element()) # your duple can have additional behaviour
Note that this wouldn't allow you to call CustomDuple.new([1]) and expect ([1],) as a result - you'd get (1,). You could of course just remove the bit of code that tries to stay as close as possible to tuple() for CustomDuple.new():
class CustomDuple(tuple):
#classmethod
def new(cls, *args):
return cls([*args])
However, although either solution works as you ask, I wouldn't recommend doing this unless you have very, very good reasons not to want to use a normal tuple.
After all, you can just create it like this:
duple = tuple([2, 3])
User #blckknght asked in the comments why not override __new__ - you definitely can, if you are OK with specifically passing 2 arguments, instead of any number:
class CustomDuple(tuple):
def __new__(cls, arg1, arg2):
return super().__new__(cls, [arg1, arg2])
Note however, that neither of these then work:
duple = CustomDuple()
duple = CustomDuple([1, 2])
I have a three dimensional dataset where the 1st dimension gives the type of the variable and the 2nd and 3rd dimensions are spatial indexes. I am attempting to make this data more user friendly by creating a subclass of ndarray containing the data, but with attributes that have sensible names that point to the appropriate variable dimension. One of the variable types is temperature, which I would like to represent with the attribute .T. I attempt to set it like this:
self.T = self[8,:,:]
However, this clashes with the underlying numpy attribute for transposing an array. Normally, overriding a class attribute is trivial, however in this case I get an exception when I try to re-write the attribute. The following is a minimal example of the same problem:
import numpy as np
class foo(np.ndarray):
def __new__(cls, input_array):
obj = np.asarray(input_array).view(cls)
obj.T = 100.0
return obj
foo([1,2,3,4])
results in:
Traceback (most recent call last):
File "tmp.py", line 9, in <module>
foo([1,2,3,4])
File "tmp.py", line 6, in __new__
obj.T = 100.0
AttributeError: attribute 'T' of 'numpy.ndarray' objects is not writable
I have tried using setattr(obj, 'T', 100.0) to set the attribute, but the result is the same.
Obviously, I could just give up and name my attribute .temperature, or something else. However .T will be much more eloquent for the subsequent mathematical expressions which will be done with these data objects. How can I force python/numpy to override this attribute?
For np.matrix subclass, as defined in np.matrixlib.defmatrix:
#property
def T(self):
"""
Returns the transpose of the matrix.
....
"""
return self.transpose()
T is not a conventional attribute that lives in a __dict__ or __slots__. In fact, you can see this immediately because the result of T changes if you modify the shape or contents of an array.
Since ndarray is a class written in C, it has special descriptors for the dynamic attributes it exposes. T is one of these dynamic attributes, defined as a PyGetSetDef structure. You can't override it by simple assignment, because there is nothing to assign to, but you can make a descriptor that overrides it at the class level.
As #hpaulj's answer suggests, the simplest solution may be to use a property to implement the descriptor protocol for you:
import numpy as np
class foo(np.ndarray):
#property
def T(self):
return self[8, :, :]
More complicated alternatives would be to make your own descriptor type, or even to extend the class in C and write your own PyGetSetDef structure. It all depends on what you are trying to achieve.
Following Mad Physicist and hpaulj's lead, the solution to my minimal working example is:
import numpy as np
class foo(np.ndarray):
def __new__(cls, input_array):
obj = np.asarray(input_array).view(cls)
return obj
#property
def T(self):
return 100.0
x = foo([1,2,3,4])
print("T is", x.T)
Which results in:
T is [1 2 3 4]
I'm trying to model a collection of objects in python (2). The collection should make a certain attribute (an integer, float or any immutable object) of the objects available via a list interface.
(1)
>>> print (collection.attrs)
[1, 5, 3]
>>> collection.attrs = [4, 2, 3]
>>> print (object0.attr == 4)
True
I especially expect this list interface in the collection to allow for reassigning a single object's attribute, e.g.
(2)
>>> collection.attrs[2] = 8
>>> print (object2.attr == 8)
True
I am sure this is a quite frequently occurring situation, unfortunately I was not able to find a satisfying answer on how to implement it on stackoverflow / google etc.
Behind the scenes, I expect the object.attr to be implemented as a mutable object. Somehow I also expect the collection to hold a "list of references" to the object.attr and not the respectively referenced (immutable) values themselves.
I ask for your suggestion how to solve this in an elegant and flexible way.
A possible implementation that allows for (1) but not for (2) is
class Component(object):
"""One of many components."""
def __init__(self, attr):
self.attr = attr
class System(object):
"""One System object contains and manages many Component instances.
System is the main interface to adjusting the components.
"""
def __init__(self, attr_list):
self._components = []
for attr in attr_list:
new = Component(attr)
self._components.append(new)
#property
def attrs(self):
# !!! this breaks (2):
return [component.attr for component in self._components]
#attrs.setter
def attrs(self, new_attrs):
for component, new_attr in zip(self._components, new_attrs):
component.attr = new_attr
The !!! line breaks (2) because we create a new list whose entries are references to the values of all Component.attr and not references to the attributes themselves.
Thanks for your input.
TheXMA
Just add another proxy inbetween:
class _ListProxy:
def __init__(self, system):
self._system = system
def __getitem__(self, index):
return self._system._components[index].attr
def __setitem__(self, index, value):
self._system._components[index].attr = value
class System:
...
#property
def attrs(self):
return _ListProxy(self)
You can make the proxy fancier by implementing all the other list methods, but this is enough for your use-case.
#filmor thanks a lot for your answer, this solves the problem perfectly! I made it a bit more general:
class _ListProxy(object):
"""Is a list of object attributes. Accessing _ListProxy entries
evaluates the object attributes each time it is accessed,
i.e. this list "proxies" the object attributes.
"""
def __init__(self, list_of_objects, attr_name):
"""Provide a list of object instances and a name of a commonly
shared attribute that should be proxied by this _ListProxy
instance.
"""
self._list_of_objects = list_of_objects
self._attr_name = attr_name
def __getitem__(self, index):
return getattr(self._list_of_objects[index], self._attr_name)
def __setitem__(self, index, value):
setattr(self._list_of_objects[index], self._attr_name, value)
def __repr__(self):
return repr(list(self))
def __len__(self):
return len(self._list_of_objects)
Are there any important list methods missing?
And what if I want some of the components (objects) to be garbage collected?
Do I need to use something like a WeakList to prevent memory leakage?
I have a class that wraps around python deque from collections. When I go and create a deque x=deque(), and I want to reference the first variable....
In[78]: x[0]
Out[78]: 0
My question is how can use the [] for referencing in the following example wrapper
class deque_wrapper:
def __init__(self):
self.data_structure = deque()
def newCustomAddon(x):
return len(self.data_structure)
def __repr__(self):
return repr(self.data_structure)
Ie, continuing from above example:
In[75]: x[0]
Out[76]: TypeError: 'deque_wrapper' object does not support indexing
I want to customize my own referencing, is that possible?
You want to implement the __getitem__ method:
class DequeWrapper:
def __init__(self):
self.data_structure = deque()
def newCustomAddon(x):
return len(self.data_structure)
def __repr__(self):
return repr(self.data_structure)
def __getitem__(self, index):
# etc
Whenever you do my_obj[x], Python will actually call my_obj.__getitem__(x).
You may also want to consider implementing the __setitem__ method, if applicable. (When you write my_obj[x] = y, Python will actually run my_obj.__setitem__(x, y).
The documentation on Python data models will contain more information on which methods you need to implement in order to make custom data structures in Python.
I am looking for a way to create a basic python "object" which I can externally assign attributes to.
Currently I am doing it the following way:
I define an empty class with
class C(object):
pass
and then I instantiate an object and assign attributes like this:
c = C()
c.attr = 2
Coming to my question
Is there a way to instantiate an empty class object, which I can then assign attributes like shown above without defining a class C?
Is there maybe an other better way to accomplish what I am after?
It looks like you are looking for a flexible container that has no methods and can take attributes with arbitrary names. That's a dict.
d = dict()
d['myattr'] = 42
If you prefer the attribute syntax that you get with a class (c.myattr = 42), then use a class just as per the code in your question.
Is there a way to instantiate an empty class object, which I can then assign attributes like shown above without defining a class C?
Yes:
>>> C = type("C", (object,), {})
>>> c = C()
>>> c.attr = 2
But as you can see, it's not much of an improvement, and the end result is the same -- it's just another way of creating the same class C.
Addendum:
You can make it prettier by "hiding" it in a function:
def attr_holder(cls=type("C", (object,), {})):
return cls()
c = attr_holder()
c.attr = 2
Though this is just reinventing the wheel -- replace the two line function with
class attr_holder(object):
pass
and it'll work exactly the same, and we've come full circle. So: go with what David or Reorx suggests.
I had come to the same question long ago, and then create this class to use in many of my projects:
class DotDict(dict):
"""
retrieve value of dict in dot style
"""
def __getattr__(self, key):
try:
return self[key]
except KeyError:
raise AttributeError('has no attribute %s' % key)
def __setattr__(self, key, value):
self[key] = value
def __delattr__(self, key):
try:
del self[key]
except KeyError:
raise AttributeError(key)
def __str__(self):
return '<DotDict %s >' % self.__to_dict()
def __to_dict(self):
return dict(self)
When I want a object to store data or want to retrieve value easily from a dict, I always use this class.
Additionally, it can help me serialize the attributes that I set in the object, and reversely get the original dict when needed.
So I think this may be a good solution in many situations, though other tricks look simple,
they are not very helpful further.