Develop Reference

Numpy Split array to sub arrays by specific number of elements

for example
I have a point list
a = np.array([[0,0,0],
[1,1,1],
[2,2,2],
[3,3,3],
[4,4,4],
[5,5,5],
[6,6,6],
[7,7,7],
[8,8,8],
[9,9,9]])
and I have another array represents the number of elements
b = np.array([2,0,3,5])
how can I split array a according the number of elements of array b so that I can get the output
[[[0,0,0],[1,1,1]],
[],
[[2,2,2],[3,3,3],[4,4,4]],
[[5,5,5],[6,6,6],[7,7,7],[8,8,8],[9,9,9]]]

You can use numpy.split using cumsum on b to get the split points:
out = np.split(a, b.cumsum()[:-1])
output:
[array([[0, 0, 0],
[1, 1, 1]]),
array([], shape=(0, 3), dtype=int64),
array([[2, 2, 2],
[3, 3, 3],
[4, 4, 4]]),
array([[5, 5, 5],
[6, 6, 6],
[7, 7, 7],
[8, 8, 8],
[9, 9, 9]])]
If you want lists:
out = [x.tolist() for x in np.split(a, b.cumsum()[:-1])]
output:
[[[0, 0, 0], [1, 1, 1]],
[],
[[2, 2, 2], [3, 3, 3], [4, 4, 4]],
[[5, 5, 5], [6, 6, 6], [7, 7, 7], [8, 8, 8], [9, 9, 9]]]
intermediate:
b.cumsum()[:-1]
# array([2, 2, 5])

Append an element to Array of Arrays

Suppose I have an array of arrays.
import numpy as np
x = np.array([ [1, 2], [3, 4], [5, 6]])
I want to add 10 as the first element of each of those arrays without running for loop. Result should look like
array([[10, 1, 2],
[10, 3, 4],
[10, 5, 6]])
Plain append does not work.
np.append(10, x)
array([10, 1, 2, 3, 4, 5, 6])
My original problem has 100K arrays. So I need to find an efficient way to do this.

You are looking for np.insert.
https://numpy.org/doc/stable/reference/generated/numpy.insert.html
np.insert(x, 0, [10,10,10], axis=1)

np.insert is your choice
>>> import numpy as np
x = np.array([ [1, 2], [3, 4], [5, 6]])
>>> x
array([[1, 2],
[3, 4],
[5, 6]])
>>> np.insert(x, 0, 10, axis=1)
array([[10, 1, 2],
[10, 3, 4],
[10, 5, 6]])
you also can insert different values
>>> np.insert(x, 0, [10,11,12] , axis=1)
array([[10, 1, 2],
[11, 3, 4],
[12, 5, 6]])

row-wise Cartesian product between a 1d array and a 2d array

I think I'm missing something obvious. I want to find a cartesian product of arr1 (a 1d numpy array), and the ROWS of arr2 (a 2d numpy array). So, if arr1 has 4 elements and arr2 has shape (5,2), the output should have shape (20,3). (see below)
import numpy as np
arr1 = np.array([1, 4, 7, 3])
arr2 = np.array([[0, 1],
[2, 3],
[4, 5],
[4, 0],
[9, 9]])
The desired output is:
arr3 = np.array([[1, 0, 1],
[1, 2, 3],
[1, 4, 5],
[1, 4, 0],
[1, 9, 9],
[4, 0, 1],
[4, 2, 3],
[4, 4, 5],
[4, 4, 0],
[4, 9, 9],
[7, 0, 1],
[7, 2, 3],
[7, 4, 5],
[7, 4, 0],
[7, 9, 9],
[3, 0, 1],
[3, 2, 3],
[3, 4, 5],
[3, 4, 0],
[3, 9, 9]])
I've been trying to use transpose and reshape with code like np.array(np.meshgrid(arr1,arr2)), but no success yet.
I'm hoping the solution can be generalized because I also need to deal with situations like this: Get all combinations of the ROWS of a 2d (10,2) array and the ROWS of a 2d array (20, 5) to get an output array (200,7).

Here is a vectorized solution that works for your general case as well:
arr1 = np.array([[1, 4],
[7, 3]])
arr2 = np.array([[0, 1],
[2, 3],
[4, 5],
[4, 0],
[9, 9]])
np.hstack((np.repeat(arr1,len(arr2),0),np.stack((arr2,)*len(arr1)).reshape(-1,arr2.shape[1])))
output of shape (2,2)*(5,2)->(10,4):
[[1 4 0 1]
[1 4 2 3]
[1 4 4 5]
[1 4 4 0]
[1 4 9 9]
[7 3 0 1]
[7 3 2 3]
[7 3 4 5]
[7 3 4 0]
[7 3 9 9]]

You can use hstack to add columns to arr2, and vstack to get the final array.
np.vstack(np.apply_along_axis(lambda x: np.hstack([np.repeat(x[0], arr2.shape[0]).reshape(-1, 1),
arr2]),
1,
arr1[:, None]))

I think this should do it:
import numpy as np
arr0 = np.array([1, 4, 7, 3])
arr1 = np.reshape(arr0, (len(arr0),1))
arr2 = np.array([[0, 1],
[2, 3],
[4, 5],
[4, 0],
[9, 9]])
r1,c1 = arr1.shape
r2,c2 = arr2.shape
arrOut = np.zeros((r1,r2,c1+c2), dtype=arr1.dtype)
arrOut[:,:,:c1] = arr1[:,None,:]
arrOut[:,:,c1:] = arr2
arrOut.reshape(-1,c1+c2)
The output is:
array([[1, 0, 1],
[1, 2, 3],
[1, 4, 5],
[1, 4, 0],
[1, 9, 9],
[4, 0, 1],
[4, 2, 3],
[4, 4, 5],
[4, 4, 0],
[4, 9, 9],
[7, 0, 1],
[7, 2, 3],
[7, 4, 5],
[7, 4, 0],
[7, 9, 9],
[3, 0, 1],
[3, 2, 3],
[3, 4, 5],
[3, 4, 0],
[3, 9, 9]])

Applying tf.gather to all rows of two tensors

I want to apply tf.gather() to all the rows of a given parameters tensor and an indices tensor.
I can apply tf.gather() on two 1D tensors to extract a 1D tensor:
# params == array([3, 8, 9, 7, 6])
# inds == array([1, 2, 3])
>>> tf.gather(params, inds).eval()
array([8, 9, 7])
Now what if I have two 2D tensors, and want to apply tf.gather() on them row-wise? I want something like this:
# params == array([[3, 8, 9, 7, 6],
# [6, 1, 7, 0, 7],
# [7, 4, 4, 5, 8]])
# inds == array([[1, 2, 3],
# [2, 3, 4],
# [0, 1, 2]])
>>> row_wise_gather(params, inds)
array([[8, 9, 7],
[7, 0, 7],
[7, 4, 4]]
The closest I've come so far is using tf.gather() with axis=1, which yields a 3D tensor, and then index the result with gather_nd():
>>> gathered3d = tf.gather(params, inds, axis=1)
# gathered3d == array([[[8, 9, 7],
# [9, 7, 6],
# [3, 8, 9]],
#
# [[1, 7, 0],
# [7, 0, 7],
# [6, 1, 7]],
#
# [[4, 4, 5],
# [4, 5, 8],
# [7, 4, 4]]])
>>> tf.gather_nd(gathered3d, [[0, 0], [1, 1], [2, 2]]).eval()
array([[8, 9, 7],
[7, 0, 7],
[7, 4, 4]])
(I'd call other functions instead of giving literal values, but that's beside the point and not an issue)
This is very clumsy. Is there a more efficient way to do this?
By the way, the indices I use are always values increasing one by one; each row just has a different start and end value. That might make the problem easier.

numpy optimize way(remove loops)

I want to use this code on very huge array. this code take long time to execute and it is not efficient.
is there any way to remove loop and convert this code to optimum way?
>>> import numpy as np
>>> x=np.random.randint(10, size=(4,5,3))
>>> x
array([[[3, 2, 6],
[4, 6, 6],
[3, 7, 9],
[6, 4, 2],
[9, 0, 1]],
[[9, 0, 4],
[1, 8, 9],
[6, 8, 1],
[9, 4, 5],
[1, 5, 2]],
[[6, 1, 6],
[1, 8, 8],
[3, 8, 3],
[7, 1, 0],
[7, 7, 0]],
[[5, 6, 6],
[8, 3, 1],
[0, 5, 4],
[6, 1, 2],
[5, 6, 1]]])
>>> y=[]
>>> for i in range(x.shape[1]):
for j in range(x.shape[2]):
y.append(x[:, i, j].tolist())
>>> y
[[3, 9, 6, 5], [2, 0, 1, 6], [6, 4, 6, 6], [4, 1, 1, 8], [6, 8, 8, 3], [6, 9, 8, 1], [3, 6, 3, 0], [7, 8, 8, 5], [9, 1, 3, 4], [6, 9, 7, 6], [4, 4, 1, 1], [2, 5, 0, 2], [9, 1, 7, 5], [0, 5, 7, 6], [1, 2, 0, 1]]

You could permute axes with np.transpose and then reshape to 2D -
y = x.transpose(1,2,0).reshape(-1,x.shape[0])
Append with .tolist() for list output.

yes, either use np.reshape(x, shape) or try it with np.ndarray.flatten(x, order='F') (F for Fortran style, column first, according to your example).
read the documentation to find out which parameters fit the best. IMHO, I think ndarray.flatten is the better and more elegant option for you here. However, depending on your exact wanted solution, you might have to reshape the array first.