Given a NumPy array of shape (X, Y, 2) representing an array of "frames" including "points" and every point having an (x,y) coordinate, I'd like to consolidate the first and second dimensions to an (X*Y, 4) array that now represents all of the points, and indexes of the X and Y dimensions.
For example, If my array is:
[
[ # Frame 0
[1, 2], # Point 0
[2, 3] # Point 1
],
[ # Frame 1
[4, 5], # Point 0
[6, 7] # Point 1
]
]
I'd like to get the array:
[
[0, 0, 1, 2], # Frame 0, Point 0
[0, 1, 2, 3] # Frame 0, Point 1
[1, 0, 4, 5], # Frame 1, Point 0
[1, 1, 6, 7] # Frame 1, Point 1
]
Slow solution:
arr = np.array([[[1, 2],[2, 3]],[[4, 5],[6, 7]]])
new_arr = []
for i, points in enumerate(arr):
for j, point in enumerate(points):
new_arr.append([i, j] + point.tolist())
Is there a faster way?
You can solve each part separately using numpy.ndindex to get indices and .reshape(). Then you can use numpy.c_ to stack them.
a = np.array([[[1, 2],[2, 3]],[[4, 5],[6, 7]]])
c = a.reshape(-1, a.shape[-1])
print(c)
# [[1 2]
# [2 3]
# [4 5]
# [6 7]]
indices = list(np.ndindex(a.shape[:-1]))
print(indices)
# [(0, 0), (0, 1), (1, 0), (1, 1)]
print(np.c_[indices, c])
# [[0 0 1 2]
# [0 1 2 3]
# [1 0 4 5]
# [1 1 6 7]]
A larger example array is used in this code so that it could be tested with different sizes in each dimension:
import numpy as np
arr = np.array(
[
[
[1, 2],
[2, 3],
[3, 4]
],
[
[4, 5],
[6, 7],
[8, 7]
],
[
[14, 5],
[16, 7],
[18, 7]
],
[
[24, 5],
[26, 7],
[28, 7]
]
]
)
x, y = arr.shape[:2]
assert(arr.shape[2] == 2)
ay, ax = (a.reshape(x, y, 1) for a in np.meshgrid(np.arange(y), np.arange(x)))
new_array = np.concatenate([ax, ay, arr], axis=2).reshape(x * y, 4)
print(repr(new_array))
gives the following:
array([[ 0, 0, 1, 2],
[ 0, 1, 2, 3],
[ 0, 2, 3, 4],
[ 1, 0, 4, 5],
[ 1, 1, 6, 7],
[ 1, 2, 8, 7],
[ 2, 0, 14, 5],
[ 2, 1, 16, 7],
[ 2, 2, 18, 7],
[ 3, 0, 24, 5],
[ 3, 1, 26, 7],
[ 3, 2, 28, 7]])
And using your original example array gives:
array([[0, 0, 1, 2],
[0, 1, 2, 3],
[1, 0, 4, 5],
[1, 1, 6, 7]])
There are no explicit loops, so it ought to be faster. (Any looping is inside numpy and will be implemented in optimised C code.)
I'm also new to NumPy but I think this should work (someone correct me if I'm wrong): arr.reshape(-1,4)
Related
Suppose I have an array a representing 3 center points of 3 rectangles. I want to create four other copied points based on each of points in array a by add 1 or minus 1 in x, y coordinates like shown in the picture.
a = np.arange(9).reshape(3,3)
>>>a
>>>out:[[ 0 1 2]
[ 3 4 5]
[ 6 7 8]]
I'm very new to numpy. What I can think of is that I can make 4 coupies of a, for the first do a[:,0]+=1, then a[:,1]+=1. for the second do a[:,0]+=1,then a[:,1]-=1.for the third do a[:,0]-=1 then a[:,1]+=1, for the fourth do a[:,0]-=1then a[:,1]-=1. But I know it's stupid. So I 'm wondering if there is a clearer way to do it in numpy?
My expected outputs:
array_1 = [[ 1 2 2]
[ 4 5 5]
[ 7 8 8]]
array_2 = [[ 1 0 2]
[ 4 3 5]
[ 7 6 8]]
array_3 = [[ -1 2 2]
[ 2 5 5]
[ 5 8 8]]
array_4 = [[ -1 0 2]
[ 2 3 5]
[ 5 6 8]]
You can generate a 3D array:
a = np.arange(9).reshape(3,3)
b = np.array([[ 1, 1,0],
[ 1,-1,0],
[-1, 1,0],
[-1,-1,0]])
# or programmatically
from itertools import product
b = np.array(list(product([1,-1], [1,-1], [0])))
out = np.tile(a, (4,1,1))+b[:,None,:]
array([[[ 1, 2, 2],
[ 4, 5, 5],
[ 7, 8, 8]],
[[ 1, 0, 2],
[ 4, 3, 5],
[ 7, 6, 8]],
[[-1, 2, 2],
[ 2, 5, 5],
[ 5, 8, 8]],
[[-1, 0, 2],
[ 2, 3, 5],
[ 5, 6, 8]]])
Subsetting:
out[0]
array([[1, 2, 2],
[4, 5, 5],
[7, 8, 8]])
it seems that what you need is to loop over a Cartesian product, there are many ways of doing so, one is to use itertools, here goes:
import numpy as np
import itertools
a = np.arange(9).reshape(3,3)
list_of_arrays = []
for seq in itertools.product([1, -1], repeat=2):
b = a.copy()
b[:,0]+=seq[0]
b[:,1]+=seq[1]
list_of_arrays.append(b)
list_of_arrays:
[array([[1, 2, 2],
[4, 5, 5],
[7, 8, 8]]),
array([[1, 0, 2],
[4, 3, 5],
[7, 6, 8]]),
array([[-1, 2, 2],
[ 2, 5, 5],
[ 5, 8, 8]]),
array([[-1, 0, 2],
[ 2, 3, 5],
[ 5, 6, 8]])]
Using numpy broadcasting and itertools to generate the shifts:
import itertools
import numpy as np
a = np.arange(9).reshape(3, 3)
shifts = np.array([(dx, dy, 0) for dx, dy in itertools.product([1, -1], repeat=2)])
shifted_a = a + shifts[:, None]
Suppose I have a matrix A with some arbitrary values:
array([[ 2, 4, 5, 3],
[ 1, 6, 8, 9],
[ 8, 7, 0, 2]])
And a matrix B which contains indices of elements in A:
array([[0, 0, 1, 2],
[0, 3, 2, 1],
[3, 2, 1, 0]])
How do I select values from A pointed by B, i.e.:
A[B] = [[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]]
EDIT: np.take_along_axis is a builtin function for this use case implemented since numpy 1.15. See #hpaulj 's answer below for how to use it.
You can use NumPy's advanced indexing -
A[np.arange(A.shape[0])[:,None],B]
One can also use linear indexing -
m,n = A.shape
out = np.take(A,B + n*np.arange(m)[:,None])
Sample run -
In [40]: A
Out[40]:
array([[2, 4, 5, 3],
[1, 6, 8, 9],
[8, 7, 0, 2]])
In [41]: B
Out[41]:
array([[0, 0, 1, 2],
[0, 3, 2, 1],
[3, 2, 1, 0]])
In [42]: A[np.arange(A.shape[0])[:,None],B]
Out[42]:
array([[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]])
In [43]: m,n = A.shape
In [44]: np.take(A,B + n*np.arange(m)[:,None])
Out[44]:
array([[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]])
More recent versions have added a take_along_axis function that does the job:
A = np.array([[ 2, 4, 5, 3],
[ 1, 6, 8, 9],
[ 8, 7, 0, 2]])
B = np.array([[0, 0, 1, 2],
[0, 3, 2, 1],
[3, 2, 1, 0]])
np.take_along_axis(A, B, 1)
Out[]:
array([[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]])
There's also a put_along_axis.
I know this is an old question, but another way of doing it using indices is:
A[np.indices(B.shape)[0], B]
output:
[[2 2 4 5]
[1 9 8 6]
[2 0 7 8]]
Following is the solution using for loop:
outlist = []
for i in range(len(B)):
lst = []
for j in range(len(B[i])):
lst.append(A[i][B[i][j]])
outlist.append(lst)
outarray = np.asarray(outlist)
print(outarray)
Above can also be written in more succinct list comprehension form:
outlist = [ [A[i][B[i][j]] for j in range(len(B[i]))]
for i in range(len(B)) ]
outarray = np.asarray(outlist)
print(outarray)
Output:
[[2 2 4 5]
[1 9 8 6]
[2 0 7 8]]
I have a numpy array. lets say
x=([8, 9, 0, 1, 2, 3, 4, 5, 6, 7,12,13])
x2 = np.reshape(x, (2,6))
now
x2= [[ 8 9 0 1 2 3]
[ 4 5 6 7 12 13]]
I need to shift x2 in such a way that the final result be
X3=[[2 3 0 1 8 9]
[12 13 6 7 4 5]]
A fancy index and swap
x2[:, 0:2], x2[:, -2:] = x2[:, -2:].copy(), x2[:, 0:2].copy()
Out[117]:
array([[ 2, 3, 0, 1, 8, 9],
[12, 13, 6, 7, 4, 5]])
You don't need to copy anything; just slice once and pass both lists of indices.
import numpy as np
x = np.array([8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 12, 13])
x = x.reshape(x, [2, 6])
x = x[:, [[0, -2], [1, -1]]] = x[:, [[-2, 0], [-1, 1]]]
x
# array([
# [ 2, 3, 0, 1, 8, 9],
# [12, 13, 6, 7, 4, 5],
# ])
I noticed you had a tensorflow tag on your question. This is a bit more involved in tensorflow.
import tensorflow as tf
x = tf.constant([
[8, 9, 0, 1, 2, 3],
[4, 5, 6, 7, 12, 13],
])
idx = tf.constant([
[[0, 4], [0, 5], [0, 2], [0, 3], [0, 0], [0, 1]],
[[1, 4], [1, 5], [1, 2], [1, 3], [1, 0], [1, 1]],
])
shp = tf.constant([2, 6])
swapped = tf.scatter_nd(indices=idx, updates=x, shape=shp)
with tf.Session() as sess:
print(swapped.eval(session=sess))
# [[ 2 3 0 1 8 9]
# [12 13 6 7 4 5]]
Let's say that I have a (x, y, z) sized matrix. Now, I wish to create a new matrix of dimension (x, y, i, i), where the (i, i) matrix is upper diagonal and constructed from the values on the z-dimension. Is there some easy way of doing this in numpy without using more than 1 for-loop (looping over x)? Thanks.
EDIT
original = np.array([
[
[0, 1, 3],
[4, 5, 6]
],
[
[7, 8, 9],
[3, 2, 1]
],
])
new = np.array([
[
[
[0, 1],
[0, 3]
],
[
[4, 5],
[0, 6]
]
],
[
[
[7, 8],
[0, 9]
],
[
[3, 2],
[0, 1]
]
]
])
So, using the above we see that
original[0, 0, :] = [0 1 3]
new[0, 0, :, :] = [[0 1]
[0 3]]
Here's an approach using boolean-indexing -
n = 2 # This would depend on a.shape[-1]
out = np.zeros(a.shape[:2] + (n,n,),dtype=a.dtype)
out[:,:,np.arange(n)[:,None] <= np.arange(n)] = a
Sample run -
In [247]: a
Out[247]:
array([[[0, 1, 3],
[4, 5, 6]],
[[7, 8, 9],
[3, 2, 1]]])
In [248]: out
Out[248]:
array([[[[0, 1],
[0, 3]],
[[4, 5],
[0, 6]]],
[[[7, 8],
[0, 9]],
[[3, 2],
[0, 1]]]])
Another approach could be suggested using subscripted-indexing to replace the last step -
r,c = np.triu_indices(n)
out[:,:,r,c] = a
Note : As stated earlier, n would depend on a.shape[-1]. Here, we had a.shape[-1] as 3, so n was 2. If a.shape[-1] were 6, n would be 3 and so on. The relationship is : (n*(n+1))//2 == a.shape[-1].
I have a 4D array training images, whose dimensions correspond to (image_number,channels,width,height). I also have a 2D target labels，whose dimensions correspond to (image_number,class_number). When training, I want to randomly shuffle the data by using random.shuffle, but how can I keep the labels shuffled by the same order of my images? Thx!
from sklearn.utils import shuffle
import numpy as np
X = np.array([[0, 0, 0], [1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4]])
y = np.array([0, 1, 2, 3, 4])
X, y = shuffle(X, y)
print(X)
print(y)
[[1 1 1]
[3 3 3]
[0 0 0]
[2 2 2]
[4 4 4]]
[1 3 0 2 4]
There is another easy way to do that. Let us suppose that there are total N images. Then we can do the following:
from random import shuffle
ind_list = [i for i in range(N)]
shuffle(ind_list)
train_new = train[ind_list, :,:,:]
target_new = target[ind_list,]
If you want a numpy-only solution, you can just reindex the second array on the first, assuming you've got the same image numbers in both:
In [67]: train = np.arange(20).reshape(4,5).T
In [68]: target = np.hstack([np.arange(5).reshape(5,1), np.arange(100, 105).reshape(5,1)])
In [69]: train
Out[69]:
array([[ 0, 5, 10, 15],
[ 1, 6, 11, 16],
[ 2, 7, 12, 17],
[ 3, 8, 13, 18],
[ 4, 9, 14, 19]])
In [70]: target
Out[70]:
array([[ 0, 100],
[ 1, 101],
[ 2, 102],
[ 3, 103],
[ 4, 104]])
In [71]: np.random.shuffle(train)
In [72]: target[train[:,0]]
Out[72]:
array([[ 2, 102],
[ 3, 103],
[ 1, 101],
[ 4, 104],
[ 0, 100]])
In [73]: train
Out[73]:
array([[ 2, 7, 12, 17],
[ 3, 8, 13, 18],
[ 1, 6, 11, 16],
[ 4, 9, 14, 19],
[ 0, 5, 10, 15]])
If you're looking for a sync/ unison shuffle you can use the following func.
def unisonShuffleDataset(a, b):
assert len(a) == len(b)
p = np.random.permutation(len(a))
return a[p], b[p]
the one above is only for 2 numpy. One can extend to more than 2 by adding the number of input vars on the func. and also on the return of the function.
Depending on what you want to do, you could also randomly generate a number for each dimension of your array with
random.randint(a, b) #a and b are the extremes of your array
which would select randomly amongst your objects.
Use the same seed to build the random generator multiple times to shuffle different arrays:
>>> seed = np.random.SeedSequence()
>>> arrays = [np.arange(10).repeat(i).reshape(10, -1) for i in range(1, 4)]
>>> for ar in arrays:
... np.random.default_rng(seed).shuffle(ar)
...
>>> arrays
[array([[1],
[2],
[7],
[8],
[0],
[4],
[3],
[6],
[9],
[5]]),
array([[1, 1],
[2, 2],
[7, 7],
[8, 8],
[0, 0],
[4, 4],
[3, 3],
[6, 6],
[9, 9],
[5, 5]]),
array([[1, 1, 1],
[2, 2, 2],
[7, 7, 7],
[8, 8, 8],
[0, 0, 0],
[4, 4, 4],
[3, 3, 3],
[6, 6, 6],
[9, 9, 9],
[5, 5, 5]])]