I have a 3d array of shape (3, 5, 5). I need to slice using different indices along the 2nd and 3rd axis for each of the 3 elements
ts = np.arange(25*3).reshape(3,5,5)
print(ts)
newr1 = np.array([1,0,2])
newr2 = np.array([3,2,4])
newc1 = np.array([1,2,0])
newc2 = np.array([3,4,2])
I want something like ts[:, newr1:newr2, newc1:newc2] but this type of slicing only works for scalar indexes. The output should look like below. Please advise
array([[[ 6, 7, 8],
[11, 12, 13],
[16, 17, 18]],
[[27, 28, 29],
[32, 33, 34],
[37, 38, 39]],
[[60, 61, 62],
[65, 66, 67],
[70, 71, 72]]])
OK, my curiosity got the better of me. I'll work out an answer even though you didn't provide as much information as you should have.
In [10]: ts = np.arange(25*3).reshape(3,5,5)
...: print(ts)
...: newr1 = np.array([1,0,2])
...: newr2 = np.array([3,2,4])
...:
...: newc1 = np.array([1,2,0])
...: newc2 = np.array([3,4,2])
...:
[[[ 0 1 2 3 4]
[ 5 6 7 8 9]
[10 11 12 13 14]
[15 16 17 18 19]
[20 21 22 23 24]]
[[25 26 27 28 29]
[30 31 32 33 34]
[35 36 37 38 39]
[40 41 42 43 44]
[45 46 47 48 49]]
[[50 51 52 53 54]
[55 56 57 58 59]
[60 61 62 63 64]
[65 66 67 68 69]
[70 71 72 73 74]]]
My guess is that what you want is:
In [11]: for i in range(3):
...: x = ts[i,newr1[i]:newr2[i],newc1[i]:newc2[i]]
...: print(x)
...:
[[ 6 7]
[11 12]]
[[27 28]
[32 33]]
[[60 61]
[65 66]]
OK, that's close, though you didn't treat the end points of slices right.
linspace is able to generate 'slices/aranges', for arrays of inputs:
In [13]: np.linspace(newr1,newr2,3)
Out[13]:
array([[1., 0., 2.],
[2., 1., 3.],
[3., 2., 4.]])
trying to use such matrices as index gives an error:
In [14]: I=np.linspace(newr1,newr2,3)
In [15]: J=np.linspace(newc1,newc2,3)
In [16]: ts[np.arange(3)[:,None,None], I[:,:,None], J[:,None,:]]
Traceback (most recent call last):
File "<ipython-input-16-5bc9e51832b8>", line 1, in <module>
ts[np.arange(3)[:,None,None], I[:,:,None], J[:,None,:]]
IndexError: arrays used as indices must be of integer (or boolean) type
In [17]: I=np.linspace(newr1,newr2,3).astype(int)
In [18]: J=np.linspace(newc1,newc2,3).astype(int)
In [19]: I
Out[19]:
array([[1, 0, 2],
[2, 1, 3],
[3, 2, 4]])
In [20]: J
Out[20]:
array([[1, 2, 0],
[2, 3, 1],
[3, 4, 2]])
After a couple of mistakes, I arrived at:
In [23]: ts[np.arange(3)[:,None,None], I.T[:,:,None], J.T[:,None,:]]
Out[23]:
array([[[ 6, 7, 8],
[11, 12, 13],
[16, 17, 18]],
[[27, 28, 29],
[32, 33, 34],
[37, 38, 39]],
[[60, 61, 62],
[65, 66, 67],
[70, 71, 72]]])
So if the slice sizes are consistent across 'planes', we can make this selection with an appropriate set of index arrays.
I have an nxn numpy array, and I would like to divide it evenly into nxn tiles and randomly shuffle these, while retaining the pattern inside the tiles.
For example, if I have an array that's size (200,200), I want to be able to divide this into say 16 arrays of size (50,50), or even 64 arrays of size (25,25), and randomly shuffle these, while retaining the same shape of the original array (200,200) and retaining the order of numbers inside of the smaller arrays.
I have looked up specific numpy functions, and I found the numpy.random.shuffle(x) function, but this will randomly shuffle the individual elements of an array. I would only like to shuffle these smaller arrays within the larger array.
Is there any numpy function or quick way that will do this? I'm not sure where to begin.
EDIT: To further clarify exactly what I want:
Let's say I have an input 2D array of shape (10,10) of values:
0 1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19
20 21 22 23 24 25 26 27 28 29
30 31 32 33 34 35 36 37 38 39
40 41 42 43 44 45 46 47 48 49
50 51 52 53 54 55 56 57 58 59
60 61 62 63 64 65 66 67 68 69
70 71 72 73 74 75 76 77 78 79
80 81 82 83 84 85 86 87 88 89
90 91 92 93 94 95 96 97 98 99
I choose a tile size such that it fits evenly into this array, so since this array has shape (10,10), I can either choose to split this into 4 (5,5) tiles, or 25 (2,2) tiles. So if I choose 4 (5,5) tiles, I want to randomly shuffle these tiles that results in an output array that could look like this:
50 51 52 53 54 0 1 2 3 4
60 61 62 63 64 10 11 12 13 14
70 71 72 73 74 20 21 22 23 24
80 81 82 83 84 30 31 32 33 34
90 91 92 93 94 40 41 42 43 44
55 56 57 58 59 5 6 7 8 9
65 66 67 68 69 15 16 17 18 19
75 76 77 78 79 25 26 27 28 29
85 86 87 88 89 35 36 37 38 39
95 96 97 98 99 45 46 47 48 49
Every array (both the input array, the output array, and the separate tiles) would be squares, so that when randomly shuffled the size and dimension of the main array stays the same (10,10).
here is my solution using loop
import numpy as np
arr = np.arange(36).reshape(6,6)
def suffle_section(arr, n_sections):
assert arr.shape[0]==arr.shape[1], "arr must be square"
assert arr.shape[0]%n_sections == 0, "arr size must divideable into equal n_sections"
size = arr.shape[0]//n_sections
new_arr = np.empty_like(arr)
## randomize section's row index
rand_indxes = np.random.permutation(n_sections*n_sections)
for i in range(n_sections):
## randomize section's column index
for j in range(n_sections):
rand_i = rand_indxes[i*n_sections + j]//n_sections
rand_j = rand_indxes[i*n_sections + j]%n_sections
new_arr[i*size:(i+1)*size, j*size:(j+1)*size] = \
arr[rand_i*size:(rand_i+1)*size, rand_j*size:(rand_j+1)*size]
return new_arr
result = suffle_section(arr, 3)
display(arr)
display(result)
array([[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35]])
array([[ 4, 5, 16, 17, 24, 25],
[10, 11, 22, 23, 30, 31],
[14, 15, 2, 3, 0, 1],
[20, 21, 8, 9, 6, 7],
[26, 27, 12, 13, 28, 29],
[32, 33, 18, 19, 34, 35]])
If you have access to skimage (it comes with Spyder) you could use view_as_blocks:
from skimage.util import view_as_blocks
def shuffle_tiles(arr, m, n):
a_= view_as_blocks(arr,(m,n)).reshape(-1,m,n)
# shuffle works along 1st dimension and in-place
np.random.shuffle(a_)
return a_
We will use np.random.shuffle alongwith axes permutations to achieve the desired results. There are two interpretations to it. Hence, two solutions.
Shuffle randomly within each block
Elements in each block are randomized and that same randomized order is maintaiined in all blocks.
def randomize_tiles_shuffle_within(a, M, N):
# M,N are the height and width of the blocks
m,n = a.shape
b = a.reshape(m//M,M,n//N,N).swapaxes(1,2).reshape(-1,M*N)
np.random.shuffle(b.T)
return b.reshape(m//M,n//N,M,N).swapaxes(1,2).reshape(a.shape)
Shuffle randomly blocks w.r.t each other
Blocks are randomized w.r.t each other, while keeping the order within each block same as in the original array.
def randomize_tiles_shuffle_blocks(a, M, N):
m,n = a.shape
b = a.reshape(m//M,M,n//N,N).swapaxes(1,2).reshape(-1,M*N)
np.random.shuffle(b)
return b.reshape(m//M,n//N,M,N).swapaxes(1,2).reshape(a.shape)
Sample runs -
In [47]: a
Out[47]:
array([[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35]])
In [48]: randomize_tiles_shuffle_within(a, 3, 3)
Out[48]:
array([[ 1, 7, 13, 4, 10, 16],
[14, 8, 12, 17, 11, 15],
[ 0, 6, 2, 3, 9, 5],
[19, 25, 31, 22, 28, 34],
[32, 26, 30, 35, 29, 33],
[18, 24, 20, 21, 27, 23]])
In [49]: randomize_tiles_shuffle_blocks(a, 3, 3)
Out[49]:
array([[ 3, 4, 5, 18, 19, 20],
[ 9, 10, 11, 24, 25, 26],
[15, 16, 17, 30, 31, 32],
[ 0, 1, 2, 21, 22, 23],
[ 6, 7, 8, 27, 28, 29],
[12, 13, 14, 33, 34, 35]])
Here is an approach that tries hard to avoid unnecessary copies:
import numpy as np
def f_pp(a,bs):
i,j = a.shape
k,l = bs
esh = i//k,k,j//l,l
bc = esh[::2]
sh1,sh2 = np.unravel_index(np.random.permutation(bc[0]*bc[1]),bc)
ns1,ns2 = np.unravel_index(np.arange(bc[0]*bc[1]),bc)
out = np.empty_like(a)
out.reshape(esh)[ns1,:,ns2] = a.reshape(esh)[sh1,:,sh2]
return out
Timings:
pp 0.41529153706505895
dv 1.3133141631260514
br 1.6034217830747366
Test script (continued)
# Divakar
def f_dv(a,bs):
M,N = bs
m,n = a.shape
b = a.reshape(m//M,M,n//N,N).swapaxes(1,2).reshape(-1,M*N)
np.random.shuffle(b)
return b.reshape(m//M,n//N,M,N).swapaxes(1,2).reshape(a.shape)
from skimage.util import view_as_blocks
# Brenlla shape fixed by pp
def f_br(arr,bs):
m,n = bs
a_= view_as_blocks(arr,(m,n))
sh = a_.shape
a_ = a_.reshape(-1,m,n)
# shuffle works along 1st dimension and in-place
np.random.shuffle(a_)
return a_.reshape(sh).swapaxes(1,2).reshape(arr.shape)
ex = np.arange(100000).reshape(1000,100)
bs = 10,10
tst = np.tile(np.arange(np.prod(bs)).reshape(bs),np.floor_divide(ex.shape,bs))
from timeit import timeit
for n,f in list(globals().items()):
if n.startswith('f_'):
assert (tst==f(tst,bs)).all()
print(n[2:],timeit(lambda:f(ex,bs),number=1000))
Here's code to shuffle row order but keep row items exactly as is:
import numpy as np
np.random.seed(0)
#creates a 6x6 array
a = np.random.randint(0,100,(6,6))
a
array([[44, 47, 64, 67, 67, 9],
[83, 21, 36, 87, 70, 88],
[88, 12, 58, 65, 39, 87],
[46, 88, 81, 37, 25, 77],
[72, 9, 20, 80, 69, 79],
[47, 64, 82, 99, 88, 49]])
#creates a number for each row index, 0,1,2,3,4,5
order = np.arange(6)
#shuffle index array
np.random.shuffle(order)
#make new array in shuffled order
shuffled = np.array([a[y] for y in order])
shuffled
array([[46, 88, 81, 37, 25, 77],
[88, 12, 58, 65, 39, 87],
[83, 21, 36, 87, 70, 88],
[47, 64, 82, 99, 88, 49],
[44, 47, 64, 67, 67, 9],
[72, 9, 20, 80, 69, 79]])
As a disclaimer I'm very new to python and numpy arrays. Reading some of the answers to similar questions and trying their solutions for my own data hasn't been very helpful so I thought I'd just post my own question. For example, Reshaping 3D Numpy Array to a 2D array. Its entirely believable though that I've just implemented those other solutions wrong.
I have a 3D numpy array "C"
C = np.reshape(np.arange(3*3*4),(3,3,4))
print(C)
[[[ 0 1 2 3]
[ 4 5 6 7]
[ 8 9 10 11]]
[[12 13 14 15]
[16 17 18 19]
[20 21 22 23]]
[[24 25 26 27]
[28 29 30 31]
[32 33 34 35]]]
I would like to reshape it into something like:
[0 12 14], [1,13,25], [2,24,26] ..... etc
where the first elements of each of the 3 arrays gets put into its own array, then the second elements of each array get put into a new array, and so on.
It seems trivial, but I'm stumped. I've tried different types combinations of .reshape, just for example,
output=C.reshape(12,3)
I've tried changing the order from "C" to "F", playing around with different .reshape() parameters, but can't seem to actually get the final result in the desired structure
Any tips would be much appreciated.
I think this is what you want:
C = np.reshape(np.arange(3*3*4),(3,3,4))
C.reshape(3,12).T
array([[ 0, 12, 24],
[ 1, 13, 25],
[ 2, 14, 26],
[ 3, 15, 27],
[ 4, 16, 28],
[ 5, 17, 29],
[ 6, 18, 30],
[ 7, 19, 31],
[ 8, 20, 32],
[ 9, 21, 33],
[10, 22, 34],
[11, 23, 35]])
Hi there I have the following matrix
[[ 47 43 51 81 54 81 52 54 31 46]
[ 35 21 30 16 37 11 35 30 39 37]
[ 8 17 11 2 5 4 11 9 17 10]
[ 5 9 4 0 1 1 0 3 9 3]
[ 2 7 2 0 0 0 0 1 2 1]
[215 149 299 199 159 325 179 249 249 199]
[ 27 49 24 4 21 8 35 15 45 25]
[100 100 100 100 100 100 100 100 100 100]]
I need to iterate over the matrix summing all elements in rows 0,1,2,3,4 only
example: I need
row_0_sum = 47+43+51+81....46
Furthermore I need to store each rows sum in an array like this
[row0_sum, row1_sum, row2_sum, row3_sum, row4_sum]
So far I have tried this code but its not doing the job:
mu = np.zeros(shape=(1,6))
#get an average
def standardize_ratings(matrix):
sum = 0
for i, eli in enumerate(matrix):
for j, elj in enumerate(eli):
if(i<5):
sum = sum + matrix[i][j]
if(j==elj.len -1):
mu[i] = sum
sum = 0
print "mu[i]="
print mu[i]
This just gives me an Error: numpy.int32 object has no attribute 'len'
So can someone help me. What's the best way to do this and which type of array in Python should I use to store this. Im new to Python but have done programming....
Thannks
Make your data, matrix, a numpy.ndarray object, instead of a list of lists, and then just do matrix.sum(axis=1).
>>> matrix = np.asarray([[ 47, 43, 51, 81, 54, 81, 52, 54, 31, 46],
[ 35, 21, 30, 16, 37, 11, 35, 30, 39, 37],
[ 8, 17, 11, 2, 5, 4, 11, 9, 17, 10],
[ 5, 9, 4, 0, 1, 1, 0, 3, 9, 3],
[ 2, 7, 2, 0, 0, 0, 0, 1, 2, 1],
[215, 149, 299, 199, 159, 325, 179, 249, 249, 199],
[ 27, 49, 24, 4, 21, 8, 35, 15, 45, 25],
[100, 100, 100, 100, 100, 100, 100, 100, 100, 100]])
>>> print matrix.sum(axis=1)
[ 540 291 94 35 15 2222 253 1000]
To get the first five rows from the result, you can just do:
>>> row_sums = matrix.sum(axis=1)
>>> rows_0_through_4_sums = row_sums[:5]
>>> print rows_0_through_4_sums
[540 291 94 35 15]
Or, you can alternatively sub-select only those rows to begin with and only apply the summation to them:
>>> rows_0_through_4 = matrix[:5,:]
>>> print rows_0_through_4.sum(axis=1)
[540 291 94 35 15]
Some helpful links will be:
NumPy for Matlab Users, if you are familiar with these things in Matlab/Octave
Slicing/Indexing in NumPy