Develop Reference

block mean of 2D numpy array (in both dimensions)

This question is related to Block mean of numpy 2D array (in fact the title is almost the same!) except that my case is a generalization. I want to divide a 2D array into a sub-blocks in both directions and take the mean over the blocks. (The linked example only divides the array in one dimension).
Thus if my array is this:
import numpy as np
a=np.arange(16).reshape((4,4))
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
If my sub-blocks have a size 2x2, then my desired answer is
array([[ 2.5, 4.5],
[10.5, 12.5]])
The only way I could think of doing this was to carefully reshape on one dimension at a time:
np.mean(np.mean(a.reshape((2,2,-1)),axis=1).reshape((-1,2,2)),axis=2)
This gives the correct solution but is a bit of a convoluted mess, and I was wondering if there is a cleaner easier code to do the same thing, maybe some numpy blocking function that I am unaware of ?

You can do:
# sample data
a=np.arange(24).reshape((6,4))
rows, cols = a.shape
a.reshape(rows//2, 2, cols//2, 2).mean(axis=(1,-1))
Output:
array([[ 2.5, 4.5],
[10.5, 12.5],
[18.5, 20.5]])

getting multiple array after performing subtraction operation within array elements

import numpy as np
m = []
k = []
a = np.array([[1,2,3,4,5,6],[50,51,52,40,20,30],[60,71,82,90,45,35]])
for i in range(len(a)):
m.append(a[i, -1:])
for j in range(len(a[i])-1):
n = abs(m[i] - a[i,j])
k.append(n)
k.append(m[i])
print(k)
Expected Output in k:
[5,4,3,2,1,6],[20,21,22,10,10,30],[25,36,47,55,10,35]
which is also a numpy array.
But the output that I am getting is
[array([5]), array([4]), array([3]), array([2]), array([1]), array([6]), array([20]), array([21]), array([22]), array([10]), array([10]), array([30]), array([25]), array([36]), array([47]), array([55]), array([10]), array([35])]
How can I solve this situation?

You want to subtract the last column of each sub array from themselves. Why don't you use a vectorized approach? You can do all the subtractions at once by subtracting the last column from the rest of the items and then column_stack together with unchanged version of the last column. Also note that you need to change the dimension of the last column inorder to be subtractable from the 2D array. For that sake we can use broadcasting.
In [71]: np.column_stack((abs(a[:, :-1] - a[:, None, -1]), a[:,-1]))
Out[71]:
array([[ 5, 4, 3, 2, 1, 6],
[20, 21, 22, 10, 10, 30],
[25, 36, 47, 55, 10, 35]])

Numpy Problems with Arrays of poly1d Objects

I'd like to first start this out with the fact that it is possible, in numpy, to create an array of poly1d objects:
random_poly = np.frompyfunc(lambda i, j: np.poly1d(np.random.randint(1, 4, 3)), 2, 1)
def random_poly_array(shape):
return np.fromfunction(random_poly, shape)
a1 = random_poly_array((3,3))
This works just fine, and we can even multiply matrices made from this form using np.dot:
a2 = random_poly_array((3,3))
a1_x_a2 = np.dot(a1, a2)
However, most other methods fail to work. For example, you can't take a list of certain poly1d objects and convert it into an array:
np.array([np.poly1d([1,2,3]), np.poly1d([1,2,3])])
As that will raise ValueError: cannot copy sequence with size 2 to array axis with dimension 3. To add to the confusion,
np.array([np.poly1d([1,2]), np.poly1d([1,2])])
will not raise an error, but instead create a 2x2 array of just 2's. Adding dtype=object has no affect, and numpy will still try to convert the poly1d objects to arrays.
The reason why this is problematic is that one cannot take an array of dimension d and convert it to an array of poly1d objects of dimension d-1. I would have expected
arr = np.arange(1, 10).reshape(3,3)
np.apply_along_axis(np.poly1d, 0, arr)
To return an array of poly1d objects, but instead it returns an unalterated array. Even worse, if arr=np.arange(9).reshape(3,3), it will throw an error, as the first poly1d object created will have a length of 2 instead of 3 due to the zero coefficient. Thus, my question is this: is there a feasible method to create poly1d arrays in numpy? If not, why not?

Using the concept of None forcing numpy to not broadcast an object into an array, something brought to my attention by Paul Panzer, I created a function which will transform the last axis into a poly1d object:
def array_to_poly(arr):
return np.apply_along_axis(lambda poly: [None, np.poly1d(poly)], -1, arr)[..., 1]
However, if we're okay with abusing more than one system in a single function, we can make it apply over arbitrary axes:
def array_to_poly(arr, axis=-1):
temp_arr = np.apply_along_axis(lambda poly: [None, np.poly1d(poly)], axis, arr)
n = temp_arr.ndim
s = [slice(None) if i != axis%n else 1 for i in range(n)]
return temp_arr[s]
Testing it with arr = np.arange(1, 25).reshape(2,3,4), we obtain:
In [ ]: array_to_poly(arr, 0)
Out[ ]:
array([[poly1d([ 1, 13]), poly1d([ 2, 14]), poly1d([ 3, 15]),
poly1d([ 4, 16])],
[poly1d([ 5, 17]), poly1d([ 6, 18]), poly1d([ 7, 19]),
poly1d([ 8, 20])],
[poly1d([ 9, 21]), poly1d([10, 22]), poly1d([11, 23]),
poly1d([12, 24])]], dtype=object)
In [ ]: array_to_poly(arr, 1)
Out[ ]:
array([[poly1d([1, 5, 9]), poly1d([ 2, 6, 10]), poly1d([ 3, 7, 11]),
poly1d([ 4, 8, 12])],
[poly1d([13, 17, 21]), poly1d([14, 18, 22]), poly1d([15, 19, 23]),
poly1d([16, 20, 24])]], dtype=object)
In [ ]: array_to_poly(arr, 2)
Out[ ]:
array([[poly1d([1, 2, 3, 4]), poly1d([5, 6, 7, 8]),
poly1d([ 9, 10, 11, 12])],
[poly1d([13, 14, 15, 16]), poly1d([17, 18, 19, 20]),
poly1d([21, 22, 23, 24])]], dtype=object)
as expected.

Apply function to an array of tuples

I have a function that I would like to apply to an array of tuples and I am wondering if there is a clean way to do it.
Normally, I could use np.vectorize to apply the function to each item in the array, however, in this case "each item" is a tuple so numpy interprets the array as a 3d array and applies the function to each item within the tuple.
So I can assume that the incoming array is one of:
tuple
1 dimensional array of tuples
2 dimensional array of tuples
I can probably write some looping logic but it seems like numpy most likely has something that does this more efficiently and I don't want to reinvent the wheel.
This is an example. I am trying to apply the tuple_converter function to each tuple in the array.
array_of_tuples1 = np.array([
[(1,2,3),(2,3,4),(5,6,7)],
[(7,2,3),(2,6,4),(5,6,6)],
[(8,2,3),(2,5,4),(7,6,7)],
])
array_of_tuples2 = np.array([
(1,2,3),(2,3,4),(5,6,7),
])
plain_tuple = (1,2,3)
# Convert each set of tuples
def tuple_converter(tup):
return tup[0]**2 + tup[1] + tup[2]
# Vectorizing applies the formula to each integer rather than each tuple
tuple_converter_vectorized = np.vectorize(tuple_converter)
print(tuple_converter_vectorized(array_of_tuples1))
print(tuple_converter_vectorized(array_of_tuples2))
print(tuple_converter_vectorized(plain_tuple))
Desired Output for array_of_tuples1:
[[ 6 11 38]
[54 14 37]
[69 13 62]]
Desired Output for array_of_tuples2:
[ 6 11 38]
Desired Output for plain_tuple:
6
But the code above produces this error (because it is trying to apply the function to an integer rather than a tuple.)
<ipython-input-209-fdf78c6f4b13> in tuple_converter(tup)
10
11 def tuple_converter(tup):
---> 12 return tup[0]**2 + tup[1] + tup[2]
13
14
IndexError: invalid index to scalar variable.

array_of_tuples1 and array_of_tuples2 are not actually arrays of tuples, but just 3- and 2-dimensional arrays of integers:
In [1]: array_of_tuples1 = np.array([
...: [(1,2,3),(2,3,4),(5,6,7)],
...: [(7,2,3),(2,6,4),(5,6,6)],
...: [(8,2,3),(2,5,4),(7,6,7)],
...: ])
In [2]: array_of_tuples1
Out[2]:
array([[[1, 2, 3],
[2, 3, 4],
[5, 6, 7]],
[[7, 2, 3],
[2, 6, 4],
[5, 6, 6]],
[[8, 2, 3],
[2, 5, 4],
[7, 6, 7]]])
So, instead of vectorizing your function, because it then will basically for-loop through the elements of the array (integers), you should apply it on the suitable axis (the axis of the "tuples") and not care about the type of the sequence:
In [6]: np.apply_along_axis(tuple_converter, 2, array_of_tuples1)
Out[6]:
array([[ 6, 11, 38],
[54, 14, 37],
[69, 13, 62]])
In [9]: np.apply_along_axis(tuple_converter, 1, array_of_tuples2)
Out[9]: array([ 6, 11, 38])

The other answer above is certainly correct, and probably what you're looking for. But I noticed you put the word "clean" into your question, and so I'd like to add this answer as well.
If we can make the assumption that all the tuples are 3 element tuples (or that they have some constant number of elements), then there's a nice little trick you can do so that the same piece of code will work on any single tuple, 1d array of tuples, or 2d array of tuples without an if/else for the 1d/2d cases. I'd argue that avoiding switches is always cleaner (although I suppose this could be contested).
import numpy as np
def map_to_tuples(x):
x = np.array(x)
flattened = x.flatten().reshape(-1, 3)
return np.array([tup[0]**2 + tup[1] + tup[2] for tup in flattened]).reshape(x.shape[:-1])
Outputs the following for your inputs (respectively), as desired:
[[ 6 11 38]
[54 14 37]
[69 13 62]]
[ 6 11 38]
6

If you are serious about the tuples bit, you could define a structured dtype.
In [535]: dt=np.dtype('int,int,int')
In [536]: x1 = np.array([
[(1,2,3),(2,3,4),(5,6,7)],
[(7,2,3),(2,6,4),(5,6,6)],
[(8,2,3),(2,5,4),(7,6,7)],
], dtype=dt)
In [537]: x1
Out[537]:
array([[(1, 2, 3), (2, 3, 4), (5, 6, 7)],
[(7, 2, 3), (2, 6, 4), (5, 6, 6)],
[(8, 2, 3), (2, 5, 4), (7, 6, 7)]],
dtype=[('f0', '<i4'), ('f1', '<i4'), ('f2', '<i4')])
Note that the display uses tuples. x1 is a 3x3 array of type dt. The elements, or records, are displayed as tuples. This more useful if the tuple elements differ - float, integer, string etc.
Now define a function that works with fields of such an array:
In [538]: def foo(tup):
return tup['f0']**2 + tup['f1'] + tup['f2']
It applies neatly to x1.
In [539]: foo(x1)
Out[539]:
array([[ 6, 11, 38],
[54, 14, 37],
[69, 13, 62]])
It also applies to a 1d array of the same dtype.
In [540]: x2=np.array([(1,2,3),(2,3,4),(5,6,7) ],dtype=dt)
In [541]: foo(x2)
Out[541]: array([ 6, 11, 38])
And a 0d array of matching type:
In [542]: foo(np.array(plain_tuple,dtype=dt))
Out[542]: 6
But foo(plain_tuple) won't work, since the function is written to work with named fields, not indexed ones.
The function could be modified to cast the input to the correct dtype if needed:
In [545]: def foo1(tup):
temp = np.asarray(tup, dtype=dt)
.....: return temp['f0']**2 + temp['f1'] + temp['f2']
In [548]: plain_tuple
Out[548]: (1, 2, 3)
In [549]: foo1(plain_tuple)
Out[549]: 6
In [554]: foo1([(1,2,3),(2,3,4),(5,6,7)]) # list of tuples
Out[554]: array([ 6, 11, 38])

Numpy - Stacked memory view of two 1D arrays

I know that I can do the following:
import numpy as np
c = np.random.randn(20, 2)
a = c[:, 0]
b = c[:, 1]
Here, a and b are pointers to c's first and second column respectively. Modifying a or b will change c (same reciprocally).
However, what I want to achieve is exactly the opposite. I want to create a 2D memory view where each column (or row) will point to a memory of a different 1D array. Assume that I already have two 1D arrays, is it possible to create a 2D view to these arrays where each row/column points to each of them?
I can create c from a and b in the following way:
c = np.c_[a, b]
However, this copies a's and b memory onto c. Can I just somehow create c as 'view' of [a b], where, by modifying an element of c this reflects in the respective a or b 1D array?

I don't think it is possible.
In your first example, the values of the a and b views are interwoven, as can be seen from this variation:
In [51]: c=np.arange(10).reshape(5,2)
In [52]: a, b = c[:,0], c[:,1]
In [53]: a
Out[53]: array([0, 2, 4, 6, 8])
In [54]: c.flatten()
Out[54]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
The data buffer for c and a start at the same memory point; b starts at 4 bytes into that buffer.
In [55]: c.__array_interface__
Out[55]:
{'strides': None,
'data': (172552624, False),...}
In [56]: a.__array_interface__
Out[56]:
{'strides': (8,),
'data': (172552624, False),...}
In [57]: b.__array_interface__
Out[57]:
{'strides': (8,),
'data': (172552628, False),...}
Even if the a,b split were by rows, b would start just further along in the same shared data buffer.
From the .flags we see that c is C-contiguous, b is not. But b values are accessed with constant strides in that shared data buffer.
When a and b are created separately, their data buffers are entirely separate. The numpy striding mechanism cannot step back and forth between these two data buffers. A 2d composite of a and b has to work with its own data buffer.
I can imagine writing a class that ends up looking like what you want. The indexing_tricks file that defines np.c_ might give you ideas (e.g. a class with a custom __getitem__ method). But it wouldn't have the speed advantages of a regular 2d array. And it might be hard to implement all of the ndarray functionality.

While #hpaulj's answer is the correct one, for your particular case, and more as an exercise in understanding numpy memory layout than as anything with practical applications, here's how you can get a view of two 1-D arrays as columns of a common array:
>>> from numpy.lib.stride_tricks import as_strided
>>> a = np.arange(10)
>>> b = np.arange(20, 30)
>>> col_stride = (b.__array_interface__['data'][0] -
a.__array_interface__['data'][0])
>>> c = as_strided(a, shape=(10, 2), strides=(a.strides[0], col_stride))
>>> c
array([[ 0, 20],
[ 1, 21],
[ 2, 22],
[ 3, 23],
[ 4, 24],
[ 5, 25],
[ 6, 26],
[ 7, 27],
[ 8, 28],
[ 9, 29]])
>>> c[4, 1] = 0
>>> c[6, 0] = 0
>>> a
array([0, 1, 2, 3, 4, 5, 0, 7, 8, 9])
>>> b
array([20, 21, 22, 23, 0, 25, 26, 27, 28, 29])
There are many things that can go wrong here, mainly that the array b has not had its reference count increased, so if you delete it its memory will be released, but the view will still be accessing it. It can also not be extended to more than two 1-D arrays, and requires that both 1-D arrays have the same stride.
Of course, just because you can do it doesn't mean you should do it! And you should definitely not do this.