I have a 4D array training images, whose dimensions correspond to (image_number,channels,width,height). I also have a 2D target labels，whose dimensions correspond to (image_number,class_number). When training, I want to randomly shuffle the data by using random.shuffle, but how can I keep the labels shuffled by the same order of my images? Thx!
from sklearn.utils import shuffle
import numpy as np
X = np.array([[0, 0, 0], [1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4]])
y = np.array([0, 1, 2, 3, 4])
X, y = shuffle(X, y)
print(X)
print(y)
[[1 1 1]
[3 3 3]
[0 0 0]
[2 2 2]
[4 4 4]]
[1 3 0 2 4]
There is another easy way to do that. Let us suppose that there are total N images. Then we can do the following:
from random import shuffle
ind_list = [i for i in range(N)]
shuffle(ind_list)
train_new = train[ind_list, :,:,:]
target_new = target[ind_list,]
If you want a numpy-only solution, you can just reindex the second array on the first, assuming you've got the same image numbers in both:
In [67]: train = np.arange(20).reshape(4,5).T
In [68]: target = np.hstack([np.arange(5).reshape(5,1), np.arange(100, 105).reshape(5,1)])
In [69]: train
Out[69]:
array([[ 0, 5, 10, 15],
[ 1, 6, 11, 16],
[ 2, 7, 12, 17],
[ 3, 8, 13, 18],
[ 4, 9, 14, 19]])
In [70]: target
Out[70]:
array([[ 0, 100],
[ 1, 101],
[ 2, 102],
[ 3, 103],
[ 4, 104]])
In [71]: np.random.shuffle(train)
In [72]: target[train[:,0]]
Out[72]:
array([[ 2, 102],
[ 3, 103],
[ 1, 101],
[ 4, 104],
[ 0, 100]])
In [73]: train
Out[73]:
array([[ 2, 7, 12, 17],
[ 3, 8, 13, 18],
[ 1, 6, 11, 16],
[ 4, 9, 14, 19],
[ 0, 5, 10, 15]])
If you're looking for a sync/ unison shuffle you can use the following func.
def unisonShuffleDataset(a, b):
assert len(a) == len(b)
p = np.random.permutation(len(a))
return a[p], b[p]
the one above is only for 2 numpy. One can extend to more than 2 by adding the number of input vars on the func. and also on the return of the function.
Depending on what you want to do, you could also randomly generate a number for each dimension of your array with
random.randint(a, b) #a and b are the extremes of your array
which would select randomly amongst your objects.
Use the same seed to build the random generator multiple times to shuffle different arrays:
>>> seed = np.random.SeedSequence()
>>> arrays = [np.arange(10).repeat(i).reshape(10, -1) for i in range(1, 4)]
>>> for ar in arrays:
... np.random.default_rng(seed).shuffle(ar)
...
>>> arrays
[array([[1],
[2],
[7],
[8],
[0],
[4],
[3],
[6],
[9],
[5]]),
array([[1, 1],
[2, 2],
[7, 7],
[8, 8],
[0, 0],
[4, 4],
[3, 3],
[6, 6],
[9, 9],
[5, 5]]),
array([[1, 1, 1],
[2, 2, 2],
[7, 7, 7],
[8, 8, 8],
[0, 0, 0],
[4, 4, 4],
[3, 3, 3],
[6, 6, 6],
[9, 9, 9],
[5, 5, 5]])]
Related
I have a 3D array consist of 2D arrays, I want to rotate only the 2D arrays inside the 3D array without changing the order, so it will become a 3D array consist of rotated 3D arrays.
For example, I have a 3D array like this.
foo = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]])
print(foo)
>>> array([[[ 1, 2, 3],
[ 4, 5, 6]],
[[ 7, 8, 9],
[10, 11, 12]]])
foo.shape
>>> (2, 2, 3)
I want to rotate it into this.
rotated_foo = np.array([[[4, 1], [5, 2], [6, 3]], [[10, 7], [11, 8], [12, 9]]])
print(rotated_foo)
>>> array([[[ 4, 1],
[ 5, 2],
[ 6, 3]],
[[10, 7],
[11, 8],
[12, 9]]])
rotated_foo.shape
>>> (2, 3, 2)
I've tried it using numpy's rot90 but I got something like this.
rotated_foo = np.rot90(foo)
print(rotated_foo)
>>> array([[[ 4, 5, 6],
[10, 11, 12]],
[[ 1, 2, 3],
[ 7, 8, 9]]])
rotated_foo.shape
>>> (2, 2, 3)
You can use numpy.rot90 by setting axes that you want to rotate.
foo = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]])
rotated_foo = np.rot90(foo, axes=(2,1))
print(rotated_foo)
Output:
array([[[ 4, 1],
[ 5, 2],
[ 6, 3]],
[[10, 7],
[11, 8],
[12, 9]]])
Try np.transpose and np.flip:
print(np.flip(np.transpose(foo, (0, 2, 1)), axis=2))
Prints:
[[[ 4 1]
[ 5 2]
[ 6 3]]
[[10 7]
[11 8]
[12 9]]]
You want to rotate 90 in the opposite direction of np.rot90, or equivalently rotate by 270 = 3 * 90 in the np.rot90 direction:
>>> np.rot90(foo, k=3, axes=(1, 2))
array([[[ 4, 1],
[ 5, 2],
[ 6, 3]],
[[10, 7],
[11, 8],
[12, 9]]])
I have a number of row vectors which I would like to batch as column vectors and use as input for Conv1d. As an example I'd like to reshape the tensor x into y i.e. making two groups of two column vectors.
# size = [4, 3]
x = torch.tensor([
[0, 1, 2],
[3, 4, 5],
[6, 7, 8],
[9, 10, 11]
])
# size = [2, 3, 2]
y = torch.tensor([
[[0, 3],
[1, 4],
[2, 5]],
[[6, 9],
[7, 10],
[8, 11]]
])
Is there a way to do this with just reshape and similar functions? The only way I can think of is using loops and copying into a new tensor.
You need to use permute as well as reshape:
x.reshape(2, 2, 3).permute(0, 2, 1)
Out[*]:
tensor([[[ 0, 3],
[ 1, 4],
[ 2, 5]],
[[ 6, 9],
[ 7, 10],
[ 8, 11]]])
First, you split the vectors into 2 x.reshape(2,2,3) placing the extra dimension in the middle. Then using permute you change the order of dimensions to be as you expected.
You can also use torch.split and torch.stack like
torch.stack(x.split(2), dim=2) # or torch.stack(x.T.split(2, dim=1))
tensor([[[ 0, 3],
[ 1, 4],
[ 2, 5]],
[[ 6, 9],
[ 7, 10],
[ 8, 11]]])
Suppose I have a matrix A with some arbitrary values:
array([[ 2, 4, 5, 3],
[ 1, 6, 8, 9],
[ 8, 7, 0, 2]])
And a matrix B which contains indices of elements in A:
array([[0, 0, 1, 2],
[0, 3, 2, 1],
[3, 2, 1, 0]])
How do I select values from A pointed by B, i.e.:
A[B] = [[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]]
EDIT: np.take_along_axis is a builtin function for this use case implemented since numpy 1.15. See #hpaulj 's answer below for how to use it.
You can use NumPy's advanced indexing -
A[np.arange(A.shape[0])[:,None],B]
One can also use linear indexing -
m,n = A.shape
out = np.take(A,B + n*np.arange(m)[:,None])
Sample run -
In [40]: A
Out[40]:
array([[2, 4, 5, 3],
[1, 6, 8, 9],
[8, 7, 0, 2]])
In [41]: B
Out[41]:
array([[0, 0, 1, 2],
[0, 3, 2, 1],
[3, 2, 1, 0]])
In [42]: A[np.arange(A.shape[0])[:,None],B]
Out[42]:
array([[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]])
In [43]: m,n = A.shape
In [44]: np.take(A,B + n*np.arange(m)[:,None])
Out[44]:
array([[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]])
More recent versions have added a take_along_axis function that does the job:
A = np.array([[ 2, 4, 5, 3],
[ 1, 6, 8, 9],
[ 8, 7, 0, 2]])
B = np.array([[0, 0, 1, 2],
[0, 3, 2, 1],
[3, 2, 1, 0]])
np.take_along_axis(A, B, 1)
Out[]:
array([[2, 2, 4, 5],
[1, 9, 8, 6],
[2, 0, 7, 8]])
There's also a put_along_axis.
I know this is an old question, but another way of doing it using indices is:
A[np.indices(B.shape)[0], B]
output:
[[2 2 4 5]
[1 9 8 6]
[2 0 7 8]]
Following is the solution using for loop:
outlist = []
for i in range(len(B)):
lst = []
for j in range(len(B[i])):
lst.append(A[i][B[i][j]])
outlist.append(lst)
outarray = np.asarray(outlist)
print(outarray)
Above can also be written in more succinct list comprehension form:
outlist = [ [A[i][B[i][j]] for j in range(len(B[i]))]
for i in range(len(B)) ]
outarray = np.asarray(outlist)
print(outarray)
Output:
[[2 2 4 5]
[1 9 8 6]
[2 0 7 8]]
I would like to use a numpy function in a daily report, because my data is quite large.
Let consider i have a numpy 2d-array
A = array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4],
[3, 4, 5],
[4, 5, 6],
[5, 6, 7],
[6, 7, 8],
[7, 8, 9]])
I want to do something like this
abs(array([0, 1, 2]) - array([[3, 4, 5], [4, 5, 6], ..., [7, 8, 9]])).sum()
abs(array([1, 2, 3]) - array([[4, 5, 6], [5, 6, 7], ..., [7, 8, 9]])).sum()
...
abs(array([3, 4, 5]) - array([[0, 1, 2], [6, 7, 8], [7, 8, 9]])).sum()
abs(array([4, 5, 6]) - array([[0, 1, 2], [1, 2, 3], [7, 8, 9]])).sum()
...
abs(array([7, 8, 9]) - array([[0, 1, 2], [1, 2, 3], ..., [4, 5, 6]])).sum()
I have tried this, but cannot skip arrays with any of elements on the left side that are in array on the right side.
for i in range(len(A)):
temp = np.roll(A, -i, axis=0)
print(abs(temp[0] - temp[3:]).sum())
This is the expected results
results = [75, 54, ..., 30, 30, ...75]
Sorry for my poor english explanation, thank you.
If you wish to have a simple one-liner solution involving only NumPy functionality, I propose this:
import numpy as np
results = np.apply_along_axis(arr=A,
axis=1,
func1d=lambda x:
np.abs(x - A[~np.isin(A, x).any(axis=1),:]).sum()
)
The results is as expected:
array([75, 54, 36, 30, 30, 36, 54, 75])
Here You go:
=^..^=
import numpy as np
A = np.array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4],
[3, 4, 5],
[4, 5, 6],
[5, 6, 7],
[6, 7, 8],
[7, 8, 9]])
def sum_data(select_row):
# roll data
rolled_data = np.roll(A, -select_row, axis=0)
drop_numbers = []
for item in rolled_data[0]:
drop_numbers.append(item)
# find rows to drop
drop_rows = []
for item in drop_numbers:
# get rows
gg = np.unique(np.where(rolled_data == item)[0])
for number in gg:
drop_rows.append(number)
# get unique rows numbers
unique_rows = list(set(drop_rows))
del unique_rows[0] # delete first number that is selected row
# delete rows
rolled_data = np.delete(rolled_data, unique_rows, axis=0)
# calculate
difference_value = 0
for i in range(1, len(rolled_data), 1):
difference_value += abs(rolled_data[0] - rolled_data[i]).sum()
return difference_value
# loop over each row
collect_values = []
for j in range(len(A)):
collect_values.append(sum_data(j))
Output:
[75, 54, 36, 30, 30, 36, 54, 75]
I'm having trouble finding a way to clone a 1-D numpy array to create a 3-D array. Say for example, I have
z = np.array([0, 2, 3, 5, 7, 9, 10])
This represents a vertical column in 3-D space (let's say it's in meters, for example). I want to create a horizontal dimension (x, y), so that the final array has dimensions (len(z), len(x), len(y)) , where each column at each x,y point is the same. I'm doing this so that I can match the spatial dimensions to other 3-D data that I have.
So with the array z as input, and given horizontal dimensions of, say, ndimx=3, ndimy=2, I want to find a function that outputs the array
np.array([ [[0, 0], [0, 0], [0, 0]],
[[2, 2], [2, 2], [2, 2]],
[[3, 3], [3, 3], [3, 3]],
[[5, 5], [5, 5], [5, 5]],
[[7, 7], [7, 7], [7, 7]],
[[9, 9], [9, 9], [9, 9]],
[[10, 10], [10, 10], [10, 10]] ])
which has a shape of (7, 3, 2). This seemed trivial to me at first, but after ages of experimenting with np.dstack(), np.astype(), np.repeat(), and transposes with ().T, I can't get the dimensional order right.
That's the critical thing here, that the vertical column is the first dimension. I'm willing to bet the answer is indeed trivial, I just can't find the magic numpy function that will do it.
Any suggestions?
We can simply get a view into input z with np.broadcast_to with no additional memory overhead and as such virtually free on performance -
np.broadcast_to(z[:,None,None],(len(z),ndimx,ndimy))
Sample run -
In [23]: z
Out[23]: array([ 0, 2, 3, 5, 7, 9, 10])
In [24]: ndimx=3; ndimy=2
In [25]: np.broadcast_to(z[:,None,None],(len(z),ndimx,ndimy))
Out[25]:
array([[[ 0, 0],
[ 0, 0],
[ 0, 0]],
[[ 2, 2],
[ 2, 2],
[ 2, 2]],
[[ 3, 3],
[ 3, 3],
[ 3, 3]],
[[ 5, 5],
[ 5, 5],
[ 5, 5]],
[[ 7, 7],
[ 7, 7],
[ 7, 7]],
[[ 9, 9],
[ 9, 9],
[ 9, 9]],
[[10, 10],
[10, 10],
[10, 10]]])
In [360]: z = np.array([0, 2, 3, 5, 7, 9, 10])
In [361]: z1 = np.stack([z,z], axis=1)
In [362]: z2 = np.stack([z1,z1,z1],axis=1)
In [363]: z2
Out[363]:
array([[[ 0, 0],
[ 0, 0],
[ 0, 0]],
[[ 2, 2],
[ 2, 2],
[ 2, 2]],
[[ 3, 3],
[ 3, 3],
[ 3, 3]],
...
[[10, 10],
[10, 10],
[10, 10]]])