In numpy I have a 3d array and I would ike to remove some of the 2d subarrays. Think about it like this:
r = range(27)
arr = np.reshape(r, (3,3,3))
del = [[0,1,2],[0,0,2]]
flatSeam = np.ravel_multi_index(del, arr.shape)
arr = np.delete(arr, flatSeam)
So at the end I would like to have an array of the shape (3,2,3) without the elements 00, 10, 22 from the original array. My problem is that I acn not use ravel_multi_index for this, because my indices are 2d and the array shape is 3d, so the wrong indices are calculated (the code above also does not execute because the indices array and the shape have to be the same size).
Do you have any ideas how I can achieve this?
Here's an approach using advanced-indexing -
# arr: Input array, rm_idx : 2-row list/array of indices to be removed
m,n,p = arr.shape
mask = np.asarray(rm_idx[1])[:,None] != np.arange(n)
out = arr[np.arange(m)[:,None],np.where(mask)[1].reshape(m,-1)]
Alternatively, with boolean-indexing -
out = arr.reshape(-1,p)[mask.ravel()].reshape(m,-1,p)
A bit less memory-intensive approach as we try to avoid creating 2D mask -
vmask = ~np.in1d(np.arange(m*n),rm_idx[1] + n*np.arange(m))
out = arr.reshape(-1,p)[vmask].reshape(m,-1,p)
Related
I have a 5D numpy array containing values, and would like to obtain a subarray with one less dimension, where the values have been selected based on a 3D array that contains indices of the forth dimension of the first array. E.g., I have the following arrays:
values = np.random.randn(3,4,5,10,2)
indices = np.random.randint(0,values.shape[3],size=values.shape[:3])
I found one solution, but find it rather complicated, and would prefer a one-liner:
x = np.arange(values.shape[0])
y = np.arange(values.shape[1])
z = np.arange(values.shape[2])
result = values[x[:,None,None],y[None,:,None],z[None, None,:],indices,:]
Is there any better solution to get this array?
You can try the following:
indices = indices[..., None, None]
result = np.take_along_axis(values, indices, axis=3).squeeze(axis=3)
As the question says I have a 2D matrix (1000, 2000), in which I apply a condition with the numpy where function, in this way:
import numpy as np
A = np.random.randn(1000, 2000)
print(A.shape)
(1000, 2000)
mask = np.where((A >=0.1) & (A <= 0.5))
A = A[mask]
print(A.shape)
(303112,)
and I get a flattened matrix which I use as input in a Fortran program which only supports 1D matrices, the output of this program has the same dimension as the input 1D matrix (303112,), is there any method or function to reconstruct the flattened matrix to its original 2D form. I was thinking of saving the indexes in a boolean matrix and use these to reconstruct the matrix, if someone knows of any numpy method or any suggestion would be of great help.
Greetings.
IIUC you need to maintain the 1D indexes and 2D indexes of the mask so that when you try to update those values using a FORTRAN program, you can switch to 1D for input and then switch back to 2D to update the original array.
You can use np.ravel_multi_index to convert 2D indexes to 1D. Then you can use these 1D indexes to convert them back to 2D using np.unravel_index (though since you already have the 2D mask, you don't need to convert 1D to 2D again.)
import numpy as np
A = np.random.randn(1000, 2000)
mask = np.where((A >=0.1) & (A <= 0.5))
idx_flat = np.ravel_multi_index(mask, (1000,2000)) #FLAT 1D indexes using original mask
idx_2d = np.unravel_index(idx_flat, (1000,2000)) #2D INDEXES using the FLAT 1D indexes
#Comparing the method of using flat indexes and A[mask]
print(np.allclose(A.ravel()[idx_flat],A[mask]))
### True
#Comparing the freshly created 2d indexes to the original mask
print(np.allclose(idx_2d,mask))
### True
Here is a dummy test case with end to end code for a (3,3) matrix.
import numpy as np
#Dummy matrix A and mask
A = np.random.randn(3, 3) #<---- shape (3,3)
mask = np.where(A <= 0.5)
mask[0].shape #Number of indexes in 2D mask
###Output: (6,)
#########################################################
#Flatten 2D indexes to 1D
idx_flat = np.ravel_multi_index(mask, (3,3)) #<--- shape (3,3)
idx_flat.shape #Number of indexes in flattened mask
###Output: (6,)
#########################################################
#Feed the 6 length array to fortran function
def fortran_function(x):
return x**2
flat_array = A.ravel()[idx_flat]
fortran_output = fortran_function(flat_array)
#Number of values in fortran_output
fortran_output.shape
###Output: (6,)
#########################################################
#Create a empty array
new_arr = np.empty((3,3)) #<---- shape (3,3)
new_arr[:] = np.nan
new_arr[mask] = fortran_output #Feed the 1D array to the 2D masked empty array
new_arr
array([[5.63399114e-04, nan, 7.86255167e-01],
[3.94992857e+00, 4.88932044e-02, 2.45489069e+00],
[3.51957270e-02, nan, nan]])
I am trying to sample with replacement a base 2D numpy array with shape of (4,2) by rows, say 10 times. The final output should be a 3D numpy array.
Have tried the code below, it works. But is there a way to do it without the for loop?
base=np.array([[20,30],[50,60],[70,80],[10,30]])
print(np.shape(base))
nsample=10
tmp=np.zeros((np.shape(base)[0],np.shape(base)[1],10))
for i in range(nsample):
id_pick = np.random.choice(np.shape(base)[0], size=(np.shape(base)[0]))
print(id_pick)
boot1=base[id_pick,:]
tmp[:,:,i]=boot1
print(tmp)
Here's one vectorized approach -
m,n = base.shape
idx = np.random.randint(0,m,(m,nsample))
out = base[idx].swapaxes(1,2)
Basic idea is that we generate all the possible indices with np.random.randint as idx. That would an array of shape (m,nsample). We use this array to index into the input array along the first axis. Thus, it selects random rows off base. To get the final output with a shape (m,n,nsample), we need to swap last two axes.
You can use the stack function from numpy. Your code would then look like:
base=np.array([[20,30],[50,60],[70,80],[10,30]])
print(np.shape(base))
nsample=10
tmp = []
for i in range(nsample):
id_pick = np.random.choice(np.shape(base)[0], size=(np.shape(base)[0]))
print(id_pick)
boot1=base[id_pick,:]
tmp.append(boot1)
tmp = np.stack(tmp, axis=-1)
print(tmp)
Based on #Divakar 's answer, if you already know the shape of this 2D-array, you can treat it as an (8,) 1D array while bootstrapping, and then reshape it:
m, n = base.shape
flatbase = np.reshape(base, (m*n,))
idxs = np.random.choice(range(8), (numReps, m*n))
bootflats = flatbase[idx]
boots = np.reshape(flatbase, (numReps, m, n))
I am trying to compare a 1D array element-wise to a 2D array, and returns the elements of the 2D array which fulfils the condition in a 2D array form without using a for loop. Preferably using numpy or quicker method.
a = range(1,10)
Tna = np.random.choice(a, size=[250,10,1000], replace=True)
sum_Ta = np.sum(Tna, axis = 1)
percent = np.percentile(sum_Ta, 5, axis =0)
Now I would like to get a 2D array which contains the elements of sum_Ta if the elements are smaller the percent. Such that 250 elements of sum_Ta are comparing with 1 element of percent for 1000 times. Originally I can do, ES = sum_Ta[sum_Ta < percent[:,None]], but it only gives me a 1D array, not a 2D array.
Assuming you mean that for each row, you want the element of the row to be included if it is less than the percentage associated with its column.
Try the following:
mask = sum_Ta < (percent * np.ones((250,1)))
ES = np.zeros((250, 1000))
ES[mask] = sum_Ta[mask]
I have a 2d array and a 1d array and I need to multiply each element in the 1d array x each element in the 2d array columns. It's basically a matrix multiplication but numpy won't allow matrix multiplication because of the 1d array. This is because matrices are inherently 2d in numpy. How can I get around this problem? This is an example of what I want:
FrMtx = np.zeros(shape=(24,24)) #2d array
elem = np.zeros(24, dtype=float) #1d array
Result = np.zeros(shape=(24,24), dtype=float) #2d array to store results
some_loop to increment i:
some_other_loop to increment j:
Result[i][j] = (FrMtx[i][j] x elem[j])
Numerous efforts have given me errors such as arrays used as indices must be of integer or boolean type
Due to the NumPy broadcasting rules, a simple
Result = FrMtx * elem
Will give the desired result.
You should be able to just multiply your arrays together, but its not immediately obvious what 'direction' the arrays will be multiplied since the matrix is square. To be more explicit about which axes are being multiplied, I find it is helpful to always multiply arrays that have the same number of dimensions.
For example, to multiply the columns:
mtx = np.zeros(shape=(5,7))
col = np.zeros(shape=(5,))
result = mtx * col.reshape((5, 1))
By reshaping col to (5,1), we guarantee that axis 0 of mtx is multiplied against axis 0 of col. To multiply rows:
mtx = np.zeros(shape=(5,7))
row = np.zeros(shape=(7,))
result = mtx * row.reshape((1, 7))
This guarantees that axis 1 in mtx is multiplied by axis 0 in row.