Develop Reference

Build a list of array without ',' with sliding window

I have an array a as follow:
import numpy as np
a= np.array([[1, 3, 5, 7, 8, 7, 1],
[11, 13, 51, 17, 18, 17, 10]])
I want to build a list of that array with a sliding window. Here is the output that I want:
I have using the following code, however it does not provide the output which I want:
lag = 3
out = []
for i in range(2):
eachrow =[]
for col in range(a.shape[1]-lag):
X_row = []
xtmp = a[i, col:col+lag]
X_row.append(xtmp)
ytmp = a[i, col+lag]
X_row.append(ytmp)
eachrow.append(X_row)
out.append(eachrow)
Any help appreciated. Thanks.

You can use numpy.lib.stride_tricks.sliding_window_view and numpy.apply_along_axis like below:
numpy.lib.stride_tricks.sliding_window_view is New in version 1.20.0. you need update your numpy
a = np.array([[1, 3, 5, 7, 8, 7, 1],[11, 13, 51, 17, 18, 17, 10]])
b = np.lib.stride_tricks.sliding_window_view(a.ravel(),4)
def create_array(row):
return np.array([row[:3],np.array(row[-1])], dtype=object)
c = np.apply_along_axis(create_array, 1, b)
print(c)
Output:
[[array([1, 3, 5]) array(7)]
[array([3, 5, 7]) array(8)]
[array([5, 7, 8]) array(7)]
[array([7, 8, 7]) array(1)]
[array([8, 7, 1]) array(11)]
[array([ 7, 1, 11]) array(13)]
[array([ 1, 11, 13]) array(51)]
[array([11, 13, 51]) array(17)]
[array([13, 51, 17]) array(18)]
[array([51, 17, 18]) array(17)]
[array([17, 18, 17]) array(10)]]

Your code produces:
In [3]: out
Out[3]:
[[[array([1, 3, 5]), 7],
[array([3, 5, 7]), 8],
[array([5, 7, 8]), 7],
[array([7, 8, 7]), 1]],
[[array([11, 13, 51]), 17],
[array([13, 51, 17]), 18],
[array([51, 17, 18]), 17],
[array([17, 18, 17]), 10]]]
That's a list of length 2. Within that lists.
If we make an array from that - with object dtype, we get a 3d array, where some elements are arrays, and some are integers:
In [6]: arr = np.array(out, object)
In [7]: arr
Out[7]:
array([[[array([1, 3, 5]), 7],
[array([3, 5, 7]), 8],
[array([5, 7, 8]), 7],
[array([7, 8, 7]), 1]],
[[array([11, 13, 51]), 17],
[array([13, 51, 17]), 18],
[array([51, 17, 18]), 17],
[array([17, 18, 17]), 10]]], dtype=object)
In [8]: arr.shape
Out[8]: (2, 4, 2)
Change one line of your code to
X_row.append(np.array([ytmp]))
In [11]: np.array(out,object)
Out[11]:
array([[[array([1, 3, 5]), array([7])],
[array([3, 5, 7]), array([8])],
[array([5, 7, 8]), array([7])],
[array([7, 8, 7]), array([1])]],
[[array([11, 13, 51]), array([17])],
[array([13, 51, 17]), array([18])],
[array([51, 17, 18]), array([17])],
[array([17, 18, 17]), array([10])]]], dtype=object)
or displayed with the str/print array formatting:
In [12]: print(_)
[[[array([1, 3, 5]) array([7])]
[array([3, 5, 7]) array([8])]
[array([5, 7, 8]) array([7])]
[array([7, 8, 7]) array([1])]]
[[array([11, 13, 51]) array([17])]
[array([13, 51, 17]) array([18])]
[array([51, 17, 18]) array([17])]
[array([17, 18, 17]) array([10])]]]
We could reshape that to a (8,2) array (still object dtype):
In [14]: print(Out[11].reshape(-1,2))
[[array([1, 3, 5]) array([7])]
[array([3, 5, 7]) array([8])]
[array([5, 7, 8]) array([7])]
[array([7, 8, 7]) array([1])]
[array([11, 13, 51]) array([17])]
[array([13, 51, 17]) array([18])]
[array([51, 17, 18]) array([17])]
[array([17, 18, 17]) array([10])]]
Since the inner most arrays have a mixed size - some 3 some 1, the result can only be object dtype - or list of lists. That's isn't optimal for array calculations.
Commas are part of the display, along with [] and words like array. Together they give us clues as to the underlying objects, whether they are lists or arrays. Equally important are the shape and dtype (if the object is an array) or length if a list.
===
Another answer uses a striding_tricks function. Here's that method in more detail. While x is a view, slicing and reshaping will make copies, so it's hard to say whether this is any faster. For this small example I bet your code is faster, but for larger case it might not be.
In [16]: np.lib.stride_tricks.sliding_window_view(a,(1,4))
Out[16]:
array([[[[ 1, 3, 5, 7]],
[[ 3, 5, 7, 8]],
[[ 5, 7, 8, 7]],
[[ 7, 8, 7, 1]]],
[[[11, 13, 51, 17]],
[[13, 51, 17, 18]],
[[51, 17, 18, 17]],
[[17, 18, 17, 10]]]])
In [17]: x = np.lib.stride_tricks.sliding_window_view(a,(1,4))
In [18]: x.shape
Out[18]: (2, 4, 1, 4)
That's a 4d view of the original 1d array.
Your size 3 'arrays' can be sliced from that:
In [19]: x[:,:,0,:3]
Out[19]:
array([[[ 1, 3, 5],
[ 3, 5, 7],
[ 5, 7, 8],
[ 7, 8, 7]],
[[11, 13, 51],
[13, 51, 17],
[51, 17, 18],
[17, 18, 17]]])
In [20]: x[:,:,0,:3].reshape(-1,3)
Out[20]:
array([[ 1, 3, 5],
[ 3, 5, 7],
[ 5, 7, 8],
[ 7, 8, 7],
[11, 13, 51],
[13, 51, 17],
[51, 17, 18],
[17, 18, 17]])
and the 1 element column:
In [21]: x[:,:,0,-1].reshape(-1,1)
Out[21]:
array([[ 7],
[ 8],
[ 7],
[ 1],
[17],
[18],
[17],
[10]])
These 2 arrays may be more useful than your object out.
The arrays shown in [14] could be split into 2 similar arrays:
In [27]: np.stack(arr.reshape(-1,2)[:,0])
Out[27]:
array([[ 1, 3, 5],
[ 3, 5, 7],
[ 5, 7, 8],
[ 7, 8, 7],
[11, 13, 51],
[13, 51, 17],
[51, 17, 18],
[17, 18, 17]])
In [28]: arr.reshape(-1,2)[:,1].astype(int)
Out[28]: array([ 7, 8, 7, 1, 17, 18, 17, 10])

There are two issues I see:
The last element in the window isn't enclosed in an np.array.
Then, a quick fix would be to change this line:
X_row.append(ytmp)
to
X_row.append(np.array([ytmp]))
which produces the desired output.
The result is two dimensions, since you create a separate sublist for each row in the array, and then append that sublist to the result. To resolve, change:
out.append(eachrow)
to
out.extend(eachrow)

You can use numpy.lib.stride_tricks.sliding_window_view for a fast, vectorized solution:
x = np.lib.stride_tricks.sliding_window_view(a, (1,3))[:, :-1]
x.shape = (*x.shape[:2], *x.shape[3:])
y = a[:, -x.shape[1]:, None]
Output:
>>> x
array([[[ 1, 3, 5],
[ 3, 5, 7],
[ 5, 7, 8],
[ 7, 8, 7]],
[[11, 13, 51],
[13, 51, 17],
[51, 17, 18],
[17, 18, 17]]])
>>> y
array([[[ 7],
[ 8],
[ 7],
[ 1]],
[[17],
[18],
[17],
[10]]])
Now, just use zip + list:
out = [list(zip(x[i], y[i])) for i in range(len(y))]
Output:
>>> out
[[(array([1, 3, 5]), array([7])),
(array([3, 5, 7]), array([8])),
(array([5, 7, 8]), array([7])),
(array([7, 8, 7]), array([1]))],
[(array([11, 13, 51]), array([17])),
(array([13, 51, 17]), array([18])),
(array([51, 17, 18]), array([17])),
(array([17, 18, 17]), array([10]))]]

How to convert a NumPy (k,n,m)-size ndarray into a (k)-size dttype=object ndarray of (n,m)-size ndarrays?

I have a NumPy ndarray A of size (k,n,m) of int, representing k images each of size nxm pixels.
I would like to convert it into a dtype=object ndarray B of size (k,) containing each of the individual images as ndarrays of size (n,m).
I can do it with a for-loop (below), but is there a more elegant/straightforward way?
A = np.arange(2*3*4).reshape(2,3,4)
B = np.empty(A.shape[0],dtype=object)
for i in range(0,A.shape[0]):
B[i] = A[i]
print(B)
array([array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]]), array([[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]])], dtype=object)

Your arrays:
In [37]: A = np.arange(2*3*4).reshape(2,3,4)
...:
...: B = np.empty(A.shape[0],dtype=object)
...: for i in range(0,A.shape[0]):
...: B[i] = A[i]
...:
In [38]: B
Out[38]:
array([array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]]), array([[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]])], dtype=object)
Alternate way of assigning A to B. Shorter, but not necessarily faster.
In [39]: B[:]=list(A)
In [40]: B
Out[40]:
array([array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]]), array([[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]])], dtype=object)
Direct assignment does not work; it has to be a list of arrays, not an array:
In [41]: B[:]=A
Traceback (most recent call last):
File "<ipython-input-41-b3ca91787565>", line 1, in <module>
B[:]=A
ValueError: could not broadcast input array from shape (2,3,4) into shape (2,)
The other answer does not work:
In [42]: np.array([*A], dtype=object)
Out[42]:
array([[[0, 1, 2, 3],
[4, 5, 6, 7],
[8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]], dtype=object)

You could use unpacking instead to get cleaner code:
B = np.array([*A], dtype=object)
EDIT: This does not work as the inner elements also gets turned into object type.

How do we keep specific dimension unchanged while reshaping in numpy?

I have a huge (N*20) matrix where every 5 rows is a valid sample, ie. every (5*20) matrix. I'm trying to reshape it into a (N/5,1,20,5) matrix where the dimension 20 is kept unchanged. I could do it in tensroflow using keep_dim, but how can I achieve this in numpy?
Thanks in advance.

Reshape and then swap the axes around:
arr1 = arr.reshape(N/5,5,1,20)
arr2 = arr1.transpose(0,2,3,1)
for example
In [476]: arr = np.arange(24).reshape(6,4)
In [477]: arr
Out[477]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]])
In [478]: arr1 = arr.reshape(2,3,1,4)
In [479]: arr2 = arr1.transpose(0,2,3,1)
In [480]: arr2.shape
Out[480]: (2, 1, 4, 3)
In [482]: arr2
Out[482]:
array([[[[ 0, 4, 8],
[ 1, 5, 9],
[ 2, 6, 10],
[ 3, 7, 11]]],
[[[12, 16, 20],
[13, 17, 21],
[14, 18, 22],
[15, 19, 23]]]])

Shuffle ordering of some rows in numpy array

I want to shuffle the ordering of only some rows in a numpy array. These rows will always be continuous (e.g. shuffling rows 23-80). The number of elements in each row can vary from 1 (such that the array is actually 1D) to 100.
Below is example code to demonstrate how I see the method shuffle_rows() could work. How would I design such a method to do this shuffling efficiently?
import numpy as np
>>> a = np.arange(20).reshape(4, 5)
>>> a
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
>>> shuffle_rows(a, [1, 3]) # including rows 1, 2 and 3 in the shuffling
array([[ 0, 1, 2, 3, 4],
[15, 16, 17, 18, 19],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14]])

You can use np.random.shuffle. This shuffles the rows themselves, not the elements within the rows.
From the docs:
This function only shuffles the array along the first index of a multi-dimensional array
As an example:
import numpy as np
def shuffle_rows(arr,rows):
np.random.shuffle(arr[rows[0]:rows[1]+1])
a = np.arange(20).reshape(4, 5)
print(a)
# array([[ 0, 1, 2, 3, 4],
# [ 5, 6, 7, 8, 9],
# [10, 11, 12, 13, 14],
# [15, 16, 17, 18, 19]])
shuffle_rows(a,[1,3])
print(a)
#array([[ 0, 1, 2, 3, 4],
# [10, 11, 12, 13, 14],
# [15, 16, 17, 18, 19],
# [ 5, 6, 7, 8, 9]])
shuffle_rows(a,[1,3])
print(a)
#array([[ 0, 1, 2, 3, 4],
# [10, 11, 12, 13, 14],
# [ 5, 6, 7, 8, 9],
# [15, 16, 17, 18, 19]])

How to slice and extend a 2D numpy array?

I have a numpy array of size nxm. I want the number of columns to be limited to k and rest of the columns to be extended in new rows. Following is the scenario -
Initial array: nxm
Final array: pxk
where p = (m/k)*n
Eg. n = 2, m = 6, k = 2
Initial array:
[[1, 2, 3, 4, 5, 6,],
[7, 8, 9, 10, 11, 12]]
Final array:
[[1, 2],
[7, 8],
[3, 4],
[9, 10],
[5, 6],
[11, 12]]
I tried using reshape but not getting the desired result.

Here's one way to do it
q=array([[1, 2, 3, 4, 5, 6,],
[7, 8, 9, 10, 11, 12]])
r=q.T.reshape(-1,2,2)
s=r.swapaxes(1,2)
t=s.reshape(-1,2)
as a one liner,
q.T.reshape(-1,2,2).swapaxes(1,2).reshape(-1,2)
array([[ 1, 2],
[ 7, 8],
[ 3, 4],
[ 9, 10],
[ 5, 6],
[11, 12]])
EDIT: for the general case, use
q=arange(1,1+n*m).reshape(n,m) #example input
r=q.T.reshape(-1,k,n)
s=r.swapaxes(1,2)
t=s.reshape(-1,k)
one liner is:
q.T.reshape(-1,k,n).swapaxes(1,2).reshape(-1,k)
example for n=3,m=12,k=4
q=array([[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12],
[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24],
[25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36]])
result is
array([[ 1, 2, 3, 4],
[13, 14, 15, 16],
[25, 26, 27, 28],
[ 5, 6, 7, 8],
[17, 18, 19, 20],
[29, 30, 31, 32],
[ 9, 10, 11, 12],
[21, 22, 23, 24],
[33, 34, 35, 36]])

Using numpy.vstack and numpy.hsplit:
a = np.array([[1, 2, 3, 4, 5, 6,],
[7, 8, 9, 10, 11, 12]])
n, m, k = 2, 6, 2
np.vstack(np.hsplit(a, m/k))
result array:
array([[ 1, 2],
[ 7, 8],
[ 3, 4],
[ 9, 10],
[ 5, 6],
[11, 12]])
UPDATE As flebool commented, above code is very slow, because hsplit returns a python list, and then vstack reconstructs the final array from a list of arrays.
Here's alternative solution that is much faster.
a.reshape(-1, m/k, k).transpose(1, 0, 2).reshape(-1, k)
or
a.reshape(-1, m/k, k).swapaxes(0, 1).reshape(-1, k)