Related
I made some numpy array np3
np1 = np.array(range(2*3*5))
np3 = np1.reshape(2,3,5)
and np3 has shape like this:
[[[ 0 1 2 3 4]
[ 5 6 7 8 9]
[10 11 12 13 14]]
[[15 16 17 18 19]
[20 21 22 23 24]
[25 26 27 28 29]]]
then, I made new numpy array np_55
np_55 = np.full((3,1),55)
and np_55 has shape like this:
[[55]
[55]
[55]]
I want make numpy array like below using both numpy arrays np3 and np_55 (I'll call that 'ANSWER'):
[[[ 0 1 2 3 4 55]
[ 5 6 7 8 9 55]
[10 11 12 13 14 55]]
[[15 16 17 18 19 55]
[20 21 22 23 24 55]
[25 26 27 28 29 55]]]
but I can't make it using both numpy arrays np3 and np_55. Of course I can make hard code like this:
a = np.append((np3[0]), np3_55, axis=1)
b = np.append((np3[1]), np3_55, axis=1)
a = a.reshape(1,3,6)
b = b.reshape(1,3,6)
np.append(a, b, axis=0)
but I don't know how can I solve ANSWER simply.
You can try the following:
import numpy as np
a = np.arange(2*3*5).reshape(2, 3, 5)
b = np.full((3,1),55)
np.c_[a, np.broadcast_to(b, (a.shape[0], *b.shape))]
It gives:
array([[[ 0, 1, 2, 3, 4, 55],
[ 5, 6, 7, 8, 9, 55],
[10, 11, 12, 13, 14, 55]],
[[15, 16, 17, 18, 19, 55],
[20, 21, 22, 23, 24, 55],
[25, 26, 27, 28, 29, 55]]])
I'm having troubles writing this piece of code.
I need to create a list to only have 3 values every 3 values :
The expected output must be something like :
output1 = [1,2,3,7,8,9,13,14,15,....67,68,69]
output2 = [4,5,6,10,11,12...70,71,72]
Any ideas how can I reach that ?
Use two loops -- one for each group of three, and one for each item within that group. For example:
>>> [i*6 + j for i in range(12) for j in range(1, 4)]
[1, 2, 3, 7, 8, 9, 13, 14, 15, 19, 20, 21, 25, 26, 27, 31, 32, 33, 37, 38, 39, 43, 44, 45, 49, 50, 51, 55, 56, 57, 61, 62, 63, 67, 68, 69]
>>> [i*6 + j for i in range(12) for j in range(4, 7)]
[4, 5, 6, 10, 11, 12, 16, 17, 18, 22, 23, 24, 28, 29, 30, 34, 35, 36, 40, 41, 42, 46, 47, 48, 52, 53, 54, 58, 59, 60, 64, 65, 66, 70, 71, 72]
Suppose you want n values every n values of total sets starting with start. Just change the start and number of sets you need. In below example list start with 1, so first set [1,2,3] and we need 12 sets each containing 3 consecutive element
Method 1
n = 3
start = 1
total = 12
# 2*n*i + start is first element of every set of n tuples (Arithmetic progression)
print([j for i in range(total) for j in range(2*n*i + start, 2*n*i + start+n)])
# Or
print(sum([list(range(2*n*i + start, 2*n*i + start+n)) for i in range(total)], []))
Method 2 (Numpy does operation in C, so fast)
import numpy as np
n = 3
start = 1
total = 12
# One liner
print(
(np.arange(start, start + n, step=1)[:, np.newaxis] + np.arange(0, total, 1) * 2*n).transpose().reshape(-1)
)
##############EXPLAINATION OF ABOVE ONE LINEAR########################
# np.arange start, start+1, ... start + n - 1
first_set = np.arange(start, start + n, step=1)
# [1 2 3]
# np.arange 0, 2*n, 4*n, 6*n, ....
multiple_to_add = np.arange(0, total, 1) * 2*n
print(multiple_to_add)
# broadcast first set using np.newaxis and repeatively add to each element in multiple_to_add
each_set_as_col = first_set[:, np.newaxis] + multiple_to_add
# [[ 1 7 13 19 25 31 37 43 49 55 61 67]
# [ 2 8 14 20 26 32 38 44 50 56 62 68]
# [ 3 9 15 21 27 33 39 45 51 57 63 69]]
# invert rows and columns
each_set_as_row = each_set_as_col.transpose()
# [[ 1 2 3]
# [ 7 8 9]
# [13 14 15]
# [19 20 21]
# [25 26 27]
# [31 32 33]
# [37 38 39]
# [43 44 45]
# [49 50 51]
# [55 56 57]
# [61 62 63]
# [67 68 69]]
merge_all_set_in_single_row = each_set_as_row.reshape(-1)
# array([ 1, 2, 3, 7, 8, 9, 13, 14, 15, 19, 20, 21, 25, 26, 27, 31, 32,
# 33, 37, 38, 39, 43, 44, 45, 49, 50, 51, 55, 56, 57, 61, 62, 63, 67,
# 68, 69])
To make the logic understandable, because sometimes the Pythonic methods look 'magic'
Here's a naive algorithm to do that:
output1 = []
output2 = []
for i in range(1, 100): # change as you like:
if (i-1) % 6 < 3:
output1.append(i)
else:
output2.append(i)
What's going on here:
Initializing two empty lists.
Iterate through integers in a range.
How to tell if i should go to output1 or output2:
I can see that 3 consecutive numbers go to output1, then 3 consecutive to output2.
This tells me I can use the modulo % operator, (doing % 6)
The rest is simple logic to get the exact result wanted.
Consider I have a 2D array in the following form
D = [
[A11,A21,A31,A22,A23,A33],
[B11,B21,B31,B22,B23,B33],
[C11,C21,C31,C22,C23,C33]
]
in which each D[i] is a representation of a symmetric matrix.
The symmetric matrix can be reshaped as
[
[[A11,A21,A31
A21,A22,A23
A31,A23,A33]],
[[B11,B21,B31
B21,B22,B23
B31,B23,B33]],
[[C11,C21,C31
C21,C22,C23
C31,C23,C33]]
]
So D[i] is a list of values for the lower triangular part of i-th symmetric matrix (with diagonals )
It's easy to just perform iterative loop by starts with result = np.zeros(3,3,3) then we fill the entries.
Note that I don't need to compute correlation or etc since the values for covariance matrix is already given. I simply want to reshape the 2D into 3D with certain constraint (symmetric and correct indexed)
I'm wondering if there is more efficient way without using loop ? Thanks
You can achieve this in 3 steps (starting with one symmetric matrix for simplicity):
Assuming there is a single vector d0 = D[0]
d0 = D[0] # [A11,A21,A31,A22,A23,A33]
First create an empty matrix
r = np.zeros([3, 3]) # note: any size will do
Assign d0 to the upper part of the matrix
upper_tri = ~np.tri(3, 3, -1, dtype=bool)
# [[ True, True, True],
# [False, True, True],
# [False, False, True]]
r[upper_tri] = d0
# [[A11,A21,A31],
# [ 0 ,A22,A23],
# [ 0 , 0 ,A33]]
Then transpose the result and assign it to itself, but apply the mask that matches only the lower triangle:
lower_tri = ~upper_tri
r[lower_tri] = r.T[lower_tri]
# [[A11,A21,A31
# A21,A22,A23
# A31,A23,A33]]
You can than extend this approach using broadcasting, but it's quite tricky. You need to transpose each input and output matrix. This is because the approach that applies to scalars (e.g. A21 is a single scalar here) would also apply to vectors
d0 = D.T # [ [A11, B11, C11], [A21, B21, C21], [A31, B31, C31]... ]
N = 3 # as batch size to avoid confusion
r = np.zeros([3, 3, N])
upper_tri = ~np.tri(3, 3, -1, dtype=bool) # same as before
r[upper_tri] = d0
lower_tri = ~upper_tri
r[lower_tri] = r.transpose([1, 0, 2])[lower_tri]
r = r.transpose([2, 0, 1])
If I understand correctly, I actually had a similar problem a while ago. I saw your question and decided to take another crack at it but with a general solution (useful for dimensions of 3 or greater). I could not find a way to do it without any loops (sorry), but it is pretty simple and could be defined as a function with the array and dimension as parameters
Solution
Here is the code I used to produce matrices of the type you are asking for from an array like the one you have (note that I just removed the letters and am using ints for purposes of demoing). It does use nested loops still.
import numpy as np
D = [
[11,21,31,22,23,33],
[11,21,31,22,23,33],
[11,21,31,22,23,33]
]
d = 3 # dimension
N = 3 # number of sets (A, B, C) or len(D)
# index offset matrix to index from D
offsets = np.zeros((d, d), dtype=int)
# adjustments to offset matrix at each i,j index
adj = np.arange(d-2, 0, -1)
for i in range(1, d-1):
offsets[i:, i:] += adj[i-1]
cov = np.empty((N, d, d), dtype=int)
# iterate over A, B, C
for n in range(N):
for i in range(d):
for j in range(d):
cov[n, i, j] = D[n][i+j+offsets[i, j]]
print(cov)
This prints
[[[11 21 31]
[21 22 23]
[31 23 33]]
[[11 21 31]
[21 22 23]
[31 23 33]]
[[11 21 31]
[21 22 23]
[31 23 33]]]
And if you have a larger set:
D = [
[11, 21, 31, 41, 51, 61, 22, 23, 24, 25, 26, 33, 34, 35, 36, 44, 45, 46, 55, 56, 66],
[11, 21, 31, 41, 51, 61, 22, 23, 24, 25, 26, 33, 34, 35, 36, 44, 45, 46, 55, 56, 66],
[11, 21, 31, 41, 51, 61, 22, 23, 24, 25, 26, 33, 34, 35, 36, 44, 45, 46, 55, 56, 66]
]
d = 6
N = 3
offsets = np.zeros((d, d), dtype=int)
adj = np.arange(d-2, 0, -1)
for i in range(1, d-1):
offsets[i:, i:] += adj[i-1]
cov = np.empty((N, d, d), dtype=int)
for n in range(N):
for i in range(d):
for j in range(d):
cov[n, i, j] = D[n][i+j+offsets[i, j]]
print(cov)
You get:
[[[11 21 31 41 51 61]
[21 22 23 24 25 26]
[31 23 33 34 35 36]
[41 24 34 44 45 46]
[51 25 35 45 55 56]
[61 26 36 46 56 66]]
[[11 21 31 41 51 61]
[21 22 23 24 25 26]
[31 23 33 34 35 36]
[41 24 34 44 45 46]
[51 25 35 45 55 56]
[61 26 36 46 56 66]]
[[11 21 31 41 51 61]
[21 22 23 24 25 26]
[31 23 33 34 35 36]
[41 24 34 44 45 46]
[51 25 35 45 55 56]
[61 26 36 46 56 66]]]
Notes
This requires your input array D follow the pattern
A11, A12, A13, A14, A22, A23, A24, A33, A34, A44
as in your 3 dimensional question.
I found this solution by mapping out the indices from D onto the matrix desired, and found that they were matrix indices plus some offset for submatrices within:
# [[i+j, i+j, i+j, i+j, i+j ],
# [i+j, i+j+3, i+j+3, i+j+3, i+j+3],
# [i+j, i+j+3, i+j+5, i+j+5, i+j+5],
# [i+j, i+j+3, i+j+5, i+j+6, i+j+6],
# [i+j, i+j+3, i+j+5, i+j+6, i+j+6]]
These offsets are start at 0, then as i,j increase they increase by 3, then 2, then 1. This pattern scales with higher dimensions.
Long answer but I hope it helps, I definitely have seen and had this problem before.
Cheers
I have an nxn numpy array, and I would like to divide it evenly into nxn tiles and randomly shuffle these, while retaining the pattern inside the tiles.
For example, if I have an array that's size (200,200), I want to be able to divide this into say 16 arrays of size (50,50), or even 64 arrays of size (25,25), and randomly shuffle these, while retaining the same shape of the original array (200,200) and retaining the order of numbers inside of the smaller arrays.
I have looked up specific numpy functions, and I found the numpy.random.shuffle(x) function, but this will randomly shuffle the individual elements of an array. I would only like to shuffle these smaller arrays within the larger array.
Is there any numpy function or quick way that will do this? I'm not sure where to begin.
EDIT: To further clarify exactly what I want:
Let's say I have an input 2D array of shape (10,10) of values:
0 1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19
20 21 22 23 24 25 26 27 28 29
30 31 32 33 34 35 36 37 38 39
40 41 42 43 44 45 46 47 48 49
50 51 52 53 54 55 56 57 58 59
60 61 62 63 64 65 66 67 68 69
70 71 72 73 74 75 76 77 78 79
80 81 82 83 84 85 86 87 88 89
90 91 92 93 94 95 96 97 98 99
I choose a tile size such that it fits evenly into this array, so since this array has shape (10,10), I can either choose to split this into 4 (5,5) tiles, or 25 (2,2) tiles. So if I choose 4 (5,5) tiles, I want to randomly shuffle these tiles that results in an output array that could look like this:
50 51 52 53 54 0 1 2 3 4
60 61 62 63 64 10 11 12 13 14
70 71 72 73 74 20 21 22 23 24
80 81 82 83 84 30 31 32 33 34
90 91 92 93 94 40 41 42 43 44
55 56 57 58 59 5 6 7 8 9
65 66 67 68 69 15 16 17 18 19
75 76 77 78 79 25 26 27 28 29
85 86 87 88 89 35 36 37 38 39
95 96 97 98 99 45 46 47 48 49
Every array (both the input array, the output array, and the separate tiles) would be squares, so that when randomly shuffled the size and dimension of the main array stays the same (10,10).
here is my solution using loop
import numpy as np
arr = np.arange(36).reshape(6,6)
def suffle_section(arr, n_sections):
assert arr.shape[0]==arr.shape[1], "arr must be square"
assert arr.shape[0]%n_sections == 0, "arr size must divideable into equal n_sections"
size = arr.shape[0]//n_sections
new_arr = np.empty_like(arr)
## randomize section's row index
rand_indxes = np.random.permutation(n_sections*n_sections)
for i in range(n_sections):
## randomize section's column index
for j in range(n_sections):
rand_i = rand_indxes[i*n_sections + j]//n_sections
rand_j = rand_indxes[i*n_sections + j]%n_sections
new_arr[i*size:(i+1)*size, j*size:(j+1)*size] = \
arr[rand_i*size:(rand_i+1)*size, rand_j*size:(rand_j+1)*size]
return new_arr
result = suffle_section(arr, 3)
display(arr)
display(result)
array([[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35]])
array([[ 4, 5, 16, 17, 24, 25],
[10, 11, 22, 23, 30, 31],
[14, 15, 2, 3, 0, 1],
[20, 21, 8, 9, 6, 7],
[26, 27, 12, 13, 28, 29],
[32, 33, 18, 19, 34, 35]])
If you have access to skimage (it comes with Spyder) you could use view_as_blocks:
from skimage.util import view_as_blocks
def shuffle_tiles(arr, m, n):
a_= view_as_blocks(arr,(m,n)).reshape(-1,m,n)
# shuffle works along 1st dimension and in-place
np.random.shuffle(a_)
return a_
We will use np.random.shuffle alongwith axes permutations to achieve the desired results. There are two interpretations to it. Hence, two solutions.
Shuffle randomly within each block
Elements in each block are randomized and that same randomized order is maintaiined in all blocks.
def randomize_tiles_shuffle_within(a, M, N):
# M,N are the height and width of the blocks
m,n = a.shape
b = a.reshape(m//M,M,n//N,N).swapaxes(1,2).reshape(-1,M*N)
np.random.shuffle(b.T)
return b.reshape(m//M,n//N,M,N).swapaxes(1,2).reshape(a.shape)
Shuffle randomly blocks w.r.t each other
Blocks are randomized w.r.t each other, while keeping the order within each block same as in the original array.
def randomize_tiles_shuffle_blocks(a, M, N):
m,n = a.shape
b = a.reshape(m//M,M,n//N,N).swapaxes(1,2).reshape(-1,M*N)
np.random.shuffle(b)
return b.reshape(m//M,n//N,M,N).swapaxes(1,2).reshape(a.shape)
Sample runs -
In [47]: a
Out[47]:
array([[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35]])
In [48]: randomize_tiles_shuffle_within(a, 3, 3)
Out[48]:
array([[ 1, 7, 13, 4, 10, 16],
[14, 8, 12, 17, 11, 15],
[ 0, 6, 2, 3, 9, 5],
[19, 25, 31, 22, 28, 34],
[32, 26, 30, 35, 29, 33],
[18, 24, 20, 21, 27, 23]])
In [49]: randomize_tiles_shuffle_blocks(a, 3, 3)
Out[49]:
array([[ 3, 4, 5, 18, 19, 20],
[ 9, 10, 11, 24, 25, 26],
[15, 16, 17, 30, 31, 32],
[ 0, 1, 2, 21, 22, 23],
[ 6, 7, 8, 27, 28, 29],
[12, 13, 14, 33, 34, 35]])
Here is an approach that tries hard to avoid unnecessary copies:
import numpy as np
def f_pp(a,bs):
i,j = a.shape
k,l = bs
esh = i//k,k,j//l,l
bc = esh[::2]
sh1,sh2 = np.unravel_index(np.random.permutation(bc[0]*bc[1]),bc)
ns1,ns2 = np.unravel_index(np.arange(bc[0]*bc[1]),bc)
out = np.empty_like(a)
out.reshape(esh)[ns1,:,ns2] = a.reshape(esh)[sh1,:,sh2]
return out
Timings:
pp 0.41529153706505895
dv 1.3133141631260514
br 1.6034217830747366
Test script (continued)
# Divakar
def f_dv(a,bs):
M,N = bs
m,n = a.shape
b = a.reshape(m//M,M,n//N,N).swapaxes(1,2).reshape(-1,M*N)
np.random.shuffle(b)
return b.reshape(m//M,n//N,M,N).swapaxes(1,2).reshape(a.shape)
from skimage.util import view_as_blocks
# Brenlla shape fixed by pp
def f_br(arr,bs):
m,n = bs
a_= view_as_blocks(arr,(m,n))
sh = a_.shape
a_ = a_.reshape(-1,m,n)
# shuffle works along 1st dimension and in-place
np.random.shuffle(a_)
return a_.reshape(sh).swapaxes(1,2).reshape(arr.shape)
ex = np.arange(100000).reshape(1000,100)
bs = 10,10
tst = np.tile(np.arange(np.prod(bs)).reshape(bs),np.floor_divide(ex.shape,bs))
from timeit import timeit
for n,f in list(globals().items()):
if n.startswith('f_'):
assert (tst==f(tst,bs)).all()
print(n[2:],timeit(lambda:f(ex,bs),number=1000))
Here's code to shuffle row order but keep row items exactly as is:
import numpy as np
np.random.seed(0)
#creates a 6x6 array
a = np.random.randint(0,100,(6,6))
a
array([[44, 47, 64, 67, 67, 9],
[83, 21, 36, 87, 70, 88],
[88, 12, 58, 65, 39, 87],
[46, 88, 81, 37, 25, 77],
[72, 9, 20, 80, 69, 79],
[47, 64, 82, 99, 88, 49]])
#creates a number for each row index, 0,1,2,3,4,5
order = np.arange(6)
#shuffle index array
np.random.shuffle(order)
#make new array in shuffled order
shuffled = np.array([a[y] for y in order])
shuffled
array([[46, 88, 81, 37, 25, 77],
[88, 12, 58, 65, 39, 87],
[83, 21, 36, 87, 70, 88],
[47, 64, 82, 99, 88, 49],
[44, 47, 64, 67, 67, 9],
[72, 9, 20, 80, 69, 79]])
I am trying to cummatively add a value to the previous value and each time, store the value in an array.
This code is just part of a larger project. For simplicity i am going to define my variables as follows:
ele_ini = [12]
smb = [2, 5, 7, 8, 9, 10]
val = ele_ini
for i in range(len(smb)):
val += smb[i]
print(val)
elevation_smb.append(val)
Problem
Each time, the previous value stored in elevation_smb is replaced by the current value such that the result i obtain is:
elevation_smb = [22, 22, 22, 22, 22, 22]
The result i am expecting however is
elevation_smb = [14, 19, 26, 34, 43, 53]
NOTE:
ele_ini is a vector with n elements. I am only using 1 element just for simplicity.
Don use loops, because slow. Better is fast vectorized solution below.
I think need numpy.cumsum and add vector ele_ini for 2d numpy array:
ele_ini = [12, 10, 1, 0]
smb = [2, 5, 7, 8, 9, 10]
elevation_smb = np.cumsum(np.array(smb)) + np.array(ele_ini)[:, None]
print (elevation_smb)
[[14 19 26 34 43 53]
[12 17 24 32 41 51]
[ 3 8 15 23 32 42]
[ 2 7 14 22 31 41]]
It seems vector in your case is using pointers. That's why it is not creating new values. Try adding copy() which copies the value.
elevation_smb.append(val.copy())
Do with reduce,
In [6]: reduce(lambda c, x: c + [c[-1] + x], smb, ele_ini)
Out[6]: [12, 14, 19, 26, 34, 43, 53]