How can I convert the a array into the b array as they are specified below in Python and using numpy library? I am looking for a very efficient way since my actual array that I want to use this method on is very big. I should mention that the numbers can be any number and there is no relationship among the numbers. Also, I tried to show in the below picture how I want to slice the array.
import numpy as np
a = np.arange(1, 49).reshape(6, 8)
a = [[ 1, 2, 3, 4, 5, 6, 7, 8],
[ 9, 10, 11, 12, 13, 14, 15, 16],
[17, 18, 19, 20, 21, 22, 23, 24],
[25, 26, 27, 28, 29, 30, 31, 32],
[33, 34, 35, 36, 37, 38, 39, 40],
[41, 42, 43, 44, 45, 46, 47, 48]]
b =[[1, 2, 9, 10], [2, 3, 10, 11], [3, 4, 11, 12], [4, 5, 12, 13],
[5, 6, 13, 14], [6, 7, 14, 15], [7, 8, 15, 16], [9, 10, 17, 18],
[10, 11, 18, 19], [11, 12, 19, 20], [12, 13, 20, 21], [13, 14, 21, 22],
[14, 15, 22, 23], [15, 16, 23, 24], [17, 18, 25, 26], [18, 19, 26, 27],
[19, 20, 27, 28], [20, 21, 28, 29], [21, 22, 29, 30], [22, 23, 30, 31],
[23, 24, 31, 32], [25, 26, 33, 34], [26, 27, 34, 35], [27, 28, 35, 36],
[28, 29, 36, 37], [29, 30, 37, 38], [30, 31, 38, 39], [31, 32, 39, 40],
[33, 34, 41, 42], [34, 35, 42, 43], [35, 36, 43, 44], [36, 37, 44, 45],
[37, 38, 45, 46], [38, 39, 46, 47], [39, 40, 47, 48]]
I was trying to find a way with reshape and transpose function but the problem is that I could not find a way to include the boundaries. c shows what I was thinking about the solution.
c = a.reshape(3, 2, 4, 2).transpose(0, 2, 3, 1).reshape(3*4, 2*2).
The picture: https://ibb.co/QC7tkPM.
With numpy 1.20 or higher you can use np.lib.stride_tricks.sliding_window_view:
import numpy as np
a = np.arange(12).reshape(3, 4)
print(a)
Gives:
[[ 0 1 2 3]
[ 4 5 6 7]
[ 8 9 10 11]]
Below (2,2) is the shape of the sliding window:
windows = np.lib.stride_tricks.sliding_window_view(a, (2,2))
print(windows)
It gives:
[[[[ 0 1]
[ 4 5]]
[[ 1 2]
[ 5 6]]
[[ 2 3]
[ 6 7]]]
[[[ 4 5]
[ 8 9]]
[[ 5 6]
[ 9 10]]
[[ 6 7]
[10 11]]]]
In earlier versions of numpy a similar result can be obtained using np.lib.index_tricks.as_strided:
w = 2 # width and height of the sliding window
r, c = a.shape
size = a.itemsize
ast = np.lib.index_tricks.as_strided
windows_ast = ast(a,
shape=(r - w + 1, c - w + 1, w, w),
strides=(c * size, size, c * size, size))
print(windows_ast)
It gives:
[[[[ 0 1]
[ 4 5]]
[[ 1 2]
[ 5 6]]
[[ 2 3]
[ 6 7]]]
[[[ 4 5]
[ 8 9]]
[[ 5 6]
[ 9 10]]
[[ 6 7]
[10 11]]]]
Note that the numpy documentation warns that np.lib.stride_tricks.as_strided should be avoided if possible, since its results may lead to several issues. The above code may also fail if the input array does not have a contiguous memory layout.
In any case, you can reshape the result to the desired shape:
windows.reshape(-1, 4)
It gives:
array([[ 0, 1, 4, 5],
[ 1, 2, 5, 6],
[ 2, 3, 6, 7],
[ 4, 5, 8, 9],
[ 5, 6, 9, 10],
[ 6, 7, 10, 11]])
Thank you all for your solutions and insights. It just gave me a reason to update my numpy to 1.21.2. The reshape/transpose combo also can work when we are willing to repeat the rows and columns at the boundaries using np.repeat. However, it would be more costly than np.lib.stride_tricks.sliding_window_view. Here is an example:
a = np.arange(1,49).reshape(6,8)
aa = np.repeat(a, repeats=[1,2,2,2,2,1], axis=0)
aa = np.repeat(aa, repeats=[1,2,2,2,2,2,2,1], axis=1)
b = aa.reshape(5,2,7,2).transpose(0,2,3,1).reshape(5*7,2*2)
Related
Say I have an array that looks like
array([[ 0, 1, 2, 3, 4, 5, 6, 7],
[ 8, 9, 10, 11, 12, 13, 14, 15],
[16, 17, 18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29, 30, 31],
[32, 33, 34, 35, 36, 37, 38, 39],
[40, 41, 42, 43, 44, 45, 46, 47],
[48, 49, 50, 51, 52, 53, 54, 55],
[56, 57, 58, 59, 60, 61, 62, 63]])
And I wanted to extract the following array:
array([[ 0, 1, 4, 5],
[ 8, 9, 12, 13],
[32, 33, 36, 37],
[40, 41, 44, 45]])
Essentially it's the top-left 2x2 block in every 4x4 macro-block.
I saw this example in 1D, but couldn't figure out the 2D case.
Another way I can think of is:
h, w = full.shape
X, Y = np.meshgrid(np.arange(w), np.arange(h))
tl = full[(X%4<2) & (Y%4<2)].reshape((h//2,-1))
But I wonder if there's a cleaner way of doing this.
Here's one way you could do it:
In [73]: a
Out[73]:
array([[ 0, 1, 2, 3, 4, 5, 6, 7],
[ 8, 9, 10, 11, 12, 13, 14, 15],
[16, 17, 18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29, 30, 31],
[32, 33, 34, 35, 36, 37, 38, 39],
[40, 41, 42, 43, 44, 45, 46, 47],
[48, 49, 50, 51, 52, 53, 54, 55],
[56, 57, 58, 59, 60, 61, 62, 63]])
In [74]: nr, nc = [s // 2 for s in a.shape] # Shape of the new array
In [75]: b = a.reshape((nr, 2, nc, 2))[::2, :, ::2, :].reshape(nr, nc)
In [76]: b
Out[76]:
array([[ 0, 1, 4, 5],
[ 8, 9, 12, 13],
[32, 33, 36, 37],
[40, 41, 44, 45]])
By specifying the index array, so if a.shape[0] % 2 == 0 (even number):
Note: these methods can handle not only when a.shape[0] % 4 == 0, but also a.shape[0] % 2 == 0 (for all even numbers).
First method:
using advance indexing:
w = 2
ind = np.arange(a.shape[1]).reshape(-1, w)[::2].ravel() # [0 1 4 5]
b = a[ind[:, None], ind[None, :]]
Second method:
by np.delete:
w = 2
ind = np.arange(a.shape[1]).reshape(-1, w)[1::2].ravel() # [2 3 6 7]
b = np.delete(a, ind, axis=0)
b = np.delete(b, ind, axis=1)
Third method:
by splitting and stacking as:
b = np.asarray(np.hsplit(a, a.shape[0] // 2)[::2])
# [[[ 0 1] [[ 4 5]
# [ 8 9] [12 13]
# [16 17] [20 21]
# [24 25] , [28 29]
# [32 33] [36 37]
# [40 41] [44 45]
# [48 49] [52 53]
# [56 57]] [60 61]]]
b = np.asarray(np.vsplit(np.hstack(b), a.shape[0] // 2)[::2])
# [[[ 0 1 4 5]
# [ 8 9 12 13]]
# ,
# [[32 33 36 37]
# [40 41 44 45]]]
b = np.vstack(b).squeeze()
I am building a neural network. where I have to flatten my training dataset.
I have two options.
1 is:
train_x_flatten = train_x_orig.reshape(train_x_orig.shape[0], -1).T
and 2nd one is:
train_x_flatten = train_x_orig.reshape(train_x_orig.shape[1]*train_x_orig.shape[2]*train_x_orig.shape[3], 209)
both gave the same shape but I found difference while computing cost?
why is that? thank you
Your original tensor is of at least rank 4 based on the second example. The first example pulls each element, ordered by increasing the right-most index, and inserts the elements into rows the length of the zeroth shape. Then transposes.
The second example again pull elements from by incrementing from the right-most index, i.e.:
element = train_x_orig[0, 0, 0, 0]
new_row.append(element)
element = train_x_orig[0, 0, 0, 1]
new_row.append(element)
but the size of the row is different. It is now the dimension of everything else in the tensor.
Here is an example to illustrate.
First we create an ordered array and reshape it to rank 4.
import numpy as np
x = np.arange(36).reshape(3,2,3,2)
x
# returns:
array([[[[ 0, 1],
[ 2, 3],
[ 4, 5]],
[[ 6, 7],
[ 8, 9],
[10, 11]]],
[[[12, 13],
[14, 15],
[16, 17]],
[[18, 19],
[20, 21],
[22, 23]]],
[[[24, 25],
[26, 27],
[28, 29]],
[[30, 31],
[32, 33],
[34, 35]]]])
Here is the output of the first example
x.reshape(x.shape[0], -1).T
# returns:
array([[ 0, 12, 24],
[ 1, 13, 25],
[ 2, 14, 26],
[ 3, 15, 27],
[ 4, 16, 28],
[ 5, 17, 29],
[ 6, 18, 30],
[ 7, 19, 31],
[ 8, 20, 32],
[ 9, 21, 33],
[10, 22, 34],
[11, 23, 35]])
And here is the second example
x.reshape(x.shape[1]*x.shape[2]*x.shape[3], -1)
# returns:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23],
[24, 25, 26],
[27, 28, 29],
[30, 31, 32],
[33, 34, 35]])
How the elements get reordered is fundamentally different.
Python 2.7.10 and NumPy. I have a matrix like this:
[[[ 0 1 2]
[ 3 4 5]
[ 6 7 8]
[ 9 10 11]]
[[12 13 14]
[15 16 17]
[18 19 20]
[21 22 23]]
[[24 25 26]
[27 28 29]
[30 31 32]
[33 34 35]]
[[36 37 38]
[39 40 41]
[42 43 44]
[45 46 47]]]
Note: The real matrix will have real data, and not consecutive numbers.
I need to rotate, flip, or something (I have tried them all) so as to end up with this:
[[[ 2 5 8 11]
[ 1 4 7 10]
[ 0 3 6 9]
[[14 17 20 23]
[13 16 19 22]
[12 15 18 21]
[[26 29 32 35]
[25 28 31 34]
[24 27 30 33]
[[38 41 44 47]
[37 40 43 46]
[36 39 42 45]]]
Basically, I need the entire columns of the matrix to become the rows.
Thanks.
Flip the positions of columns with [:,:,::-1] and use np.transpose to swap rows with columns -
In [25]: A
Out[25]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[30, 31, 32],
[33, 34, 35]]])
In [26]: A[:,:,::-1].transpose(0,2,1)
Out[26]:
array([[[ 2, 5, 8, 11],
[ 1, 4, 7, 10],
[ 0, 3, 6, 9]],
[[14, 17, 20, 23],
[13, 16, 19, 22],
[12, 15, 18, 21]],
[[26, 29, 32, 35],
[25, 28, 31, 34],
[24, 27, 30, 33]]])
Here's a simpler way to do it:
a=numpy.arange(48).reshape((4,4,3)
numpy.fliplr(a.swapaxes(1,2))
#or you could do
numpy.fliplr(a.transpose(0,2,1))
From what I can tell, flipud flips the last dimension, while fliplr flips the second to last dimension. In three dimensions, the last dimension is Z, while the second to last dimension is Y. Hence transposing the data, and flipping the Y dimension works.
Enjoy.
For each 2d subarray in your super-array you can apply the numpy function:
np.rot90() http://docs.scipy.org/doc/numpy/reference/generated/numpy.rot90.html
so:
import numpy as np
array= np.array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[30, 31, 32],
[33, 34, 35]],
[[36, 37, 38],
[39, 40, 41],
[42, 43, 44],
[45, 46, 47]]])
desired_output = np.array([np.rot90(sub_array) for sub_array in array])
transpose and flipud are what you are looking for; the swapaxes can also function as transpose
Note that transpose has a version that operates on multiple dimensions.
There may be a simpler expression for this, but this has the advantage of not using elaborate indexing. Example, done in Python 2.7.3 with numpy
f=numpy.flipud
a=numpy.arange(48).reshape((4,4,3))
result=f(f(f(a).T).T).transpose(0,2,1)
In [2]: a=numpy.arange(48).reshape((4,4,3))
Out[3]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[30, 31, 32],
[33, 34, 35]],
[[36, 37, 38],
[39, 40, 41],
[42, 43, 44],
[45, 46, 47]]])
In [5]: f(f(f(a).T).T).transpose(0,2,1)
Out[5]:
array([[[ 2, 5, 8, 11],
[ 1, 4, 7, 10],
[ 0, 3, 6, 9]],
[[14, 17, 20, 23],
[13, 16, 19, 22],
[12, 15, 18, 21]],
[[26, 29, 32, 35],
[25, 28, 31, 34],
[24, 27, 30, 33]],
[[38, 41, 44, 47],
[37, 40, 43, 46],
[36, 39, 42, 45]]])
.
I have a 2D array of shape (M*N,N) which in fact consists of M, N*N arrays. I would like to transpose all of these elements(N*N matrices) in a vectorized fashion. As an example,
import numpy as np
A=np.arange(1,28).reshape((9,3))
print "A before transposing:\n", A
for i in range(3):
A[i*3:(i+1)*3,:]=A[i*3:(i+1)*3,:].T
print "A after transposing:\n", A
This code generates the following output:
A before transposing:
[[ 1 2 3]
[ 4 5 6]
[ 7 8 9]
[10 11 12]
[13 14 15]
[16 17 18]
[19 20 21]
[22 23 24]
[25 26 27]]
A after transposing:
[[ 1 4 7]
[ 2 5 8]
[ 3 6 9]
[10 13 16]
[11 14 17]
[12 15 18]
[19 22 25]
[20 23 26]
[21 24 27]]
Which I expect. But I want the vectorized version.
Here's a nasty way to do it in one line!
A.reshape((-1, 3, 3)).swapaxes(-1, 1).reshape(A.shape)
Step by step. Reshape to (3, 3, 3)
>>> A.reshape((-1, 3, 3))
array([[[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9]],
[[10, 11, 12],
[13, 14, 15],
[16, 17, 18]],
[[19, 20, 21],
[22, 23, 24],
[25, 26, 27]]])
Then perform a transpose-like operation swapaxes on each sub-array
>>> A.reshape((-1, 3, 3)).swapaxes(-1, 1)
array([[[ 1, 4, 7],
[ 2, 5, 8],
[ 3, 6, 9]],
[[10, 13, 16],
[11, 14, 17],
[12, 15, 18]],
[[19, 22, 25],
[20, 23, 26],
[21, 24, 27]]])
Finally reshape to (9, 3).
>>> A.reshape((-1, 3, 3)).swapaxes(-1, 1).reshape(A.shape)
array([[ 1, 4, 7],
[ 2, 5, 8],
[ 3, 6, 9],
[10, 13, 16],
[11, 14, 17],
[12, 15, 18],
[19, 22, 25],
[20, 23, 26],
[21, 24, 27]])
>>>
I think that with any method, data must be copied since there's no 2d strides/shape that can generate the result from:
array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27])
(is there?) In my version I think data is copied in the final reshape step
In [42]: x = np.arange(1,28).reshape((9,3))
In [43]: x
Out[43]:
array([[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9],
[10, 11, 12],
[13, 14, 15],
[16, 17, 18],
[19, 20, 21],
[22, 23, 24],
[25, 26, 27]])
In [31]: r,c = x.shape
In [39]: z = np.vstack(np.hsplit(x.T,r/c))
In [45]: z
Out[45]:
array([[ 1, 4, 7],
[ 2, 5, 8],
[ 3, 6, 9],
[10, 13, 16],
[11, 14, 17],
[12, 15, 18],
[19, 22, 25],
[20, 23, 26],
[21, 24, 27]])
I have the following slicing problem in numpy.
a = np.arange(36).reshape(-1,4)
a
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23],
[24, 25, 26, 27],
[28, 29, 30, 31],
[32, 33, 34, 35]])
In my problem always three rows represent one sample, in my case coordinates.
I want to access this matrix in a way that if I use a[0:2] to get the following:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]
These are the first two coordinate samples.
I have to extract a large amount of these coordinate sets from an array.
Thanks
Based on How do you split a list into evenly sized chunks?, I found the following solution, which gives me the desired result.
def chunks(l, n, indices):
return np.vstack([l[idx*n:idx*n+n] for idx in indices])
chunks(a,3,[0,2])
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[24, 25, 26, 27],
[28, 29, 30, 31],
[32, 33, 34, 35]])
Probably this solution could be improved and somebody won't need the stacking.
If three rows are a sample, you can reshape your array to reflect that, use fancy indexing to retrieve your samples, then undo the shape change:
>>> a = a.reshape(-1, 3, 4)
>>> a[[0, 2]].reshape(-1, 4)
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[24, 25, 26, 27],
[28, 29, 30, 31],
[32, 33, 34, 35]])