Related
I would like to know if there is any fast way to sum each row of a first array with all rows of a second array. In this case both arrays have the same number of colulmns. For instance if array1.shape = (n,c) and array2.shape = (m,c), the resulting array would be an array3.shape = ((n*m), c)
Look at the example below:
array1 = np.array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
array2 = np.array([[0, 1, 2],
[3, 4, 5]])
The result would be:
array3 = np.array([[0, 2, 4],
[3, 5, 7]
[3, 5, 7]
[6, 8, 10]
[6, 8, 10]
[9, 11, 13]])
The only way I see I can do this is to repeat each row of one of the arrays the number of rows of the other array. For instance, by doing np.repeat(array1, len(array2), axis=0) and then sum this array with array2. This is not very practical however if the number of rows is too big. The other way would be with a for loop but this is too slow.
Any other better way to do it..?
Thanks in advance.
Extend array1 to 3D so that it becomes broadcastable against 2D array2 and then perform broadcasted addition and a final reshape is needed for desired output -
In [30]: (array1[:,None,:] + array2).reshape(-1,array1.shape[1])
Out[30]:
array([[ 0, 2, 4],
[ 3, 5, 7],
[ 3, 5, 7],
[ 6, 8, 10],
[ 6, 8, 10],
[ 9, 11, 13]])
You could try the following inline code if you haven't already. This is the simplest and probably also the quickest on a single thread.
>>> import numpy as np
>>> array1 = np.array([[0, 1, 2],
... [3, 4, 5],
... [6, 7, 8]])
>>>
>>> array2 = np.array([[0, 1, 2],
... [3, 4, 5]])
>>> array3 = np.array([i+j for i in array1 for j in array2])
>>> array3
array([[ 0, 2, 4],
[ 3, 5, 7],
[ 3, 5, 7],
[ 6, 8, 10],
[ 6, 8, 10],
[ 9, 11, 13]])
>>>
If you are looking for speed up by treading, you could consider using CUDA or multithreading. This suggestion goes a bit out of scope of your question but gives you an idea of what can be done to speed up matrix operations.
I'm having trouble finding a way to clone a 1-D numpy array to create a 3-D array. Say for example, I have
z = np.array([0, 2, 3, 5, 7, 9, 10])
This represents a vertical column in 3-D space (let's say it's in meters, for example). I want to create a horizontal dimension (x, y), so that the final array has dimensions (len(z), len(x), len(y)) , where each column at each x,y point is the same. I'm doing this so that I can match the spatial dimensions to other 3-D data that I have.
So with the array z as input, and given horizontal dimensions of, say, ndimx=3, ndimy=2, I want to find a function that outputs the array
np.array([ [[0, 0], [0, 0], [0, 0]],
[[2, 2], [2, 2], [2, 2]],
[[3, 3], [3, 3], [3, 3]],
[[5, 5], [5, 5], [5, 5]],
[[7, 7], [7, 7], [7, 7]],
[[9, 9], [9, 9], [9, 9]],
[[10, 10], [10, 10], [10, 10]] ])
which has a shape of (7, 3, 2). This seemed trivial to me at first, but after ages of experimenting with np.dstack(), np.astype(), np.repeat(), and transposes with ().T, I can't get the dimensional order right.
That's the critical thing here, that the vertical column is the first dimension. I'm willing to bet the answer is indeed trivial, I just can't find the magic numpy function that will do it.
Any suggestions?
We can simply get a view into input z with np.broadcast_to with no additional memory overhead and as such virtually free on performance -
np.broadcast_to(z[:,None,None],(len(z),ndimx,ndimy))
Sample run -
In [23]: z
Out[23]: array([ 0, 2, 3, 5, 7, 9, 10])
In [24]: ndimx=3; ndimy=2
In [25]: np.broadcast_to(z[:,None,None],(len(z),ndimx,ndimy))
Out[25]:
array([[[ 0, 0],
[ 0, 0],
[ 0, 0]],
[[ 2, 2],
[ 2, 2],
[ 2, 2]],
[[ 3, 3],
[ 3, 3],
[ 3, 3]],
[[ 5, 5],
[ 5, 5],
[ 5, 5]],
[[ 7, 7],
[ 7, 7],
[ 7, 7]],
[[ 9, 9],
[ 9, 9],
[ 9, 9]],
[[10, 10],
[10, 10],
[10, 10]]])
In [360]: z = np.array([0, 2, 3, 5, 7, 9, 10])
In [361]: z1 = np.stack([z,z], axis=1)
In [362]: z2 = np.stack([z1,z1,z1],axis=1)
In [363]: z2
Out[363]:
array([[[ 0, 0],
[ 0, 0],
[ 0, 0]],
[[ 2, 2],
[ 2, 2],
[ 2, 2]],
[[ 3, 3],
[ 3, 3],
[ 3, 3]],
...
[[10, 10],
[10, 10],
[10, 10]]])
If I define a array X with shape (2, 2):
X = np.array([[1, 2], [3, 4]])
and take the kronecker product, then reshape the output using
np.kron(X, X).reshape((2, 2, 2, 2))
I get a resulting matrix:
array([[[[ 1, 2],
[ 2, 4]],
[[ 3, 4],
[ 6, 8]]],
[[[ 3, 6],
[ 4, 8]],
[[ 9, 12],
[12, 16]]]])
However, when I use np.tensordot(X, X, axes=0) the following matrix is output
array([[[[ 1, 2],
[ 3, 4]],
[[ 2, 4],
[ 6, 8]]],
[[[ 3, 6],
[ 9, 12]],
[[ 4, 8],
[12, 16]]]])
which is different from the first output. Why is this the case? I found this while searching for answers, however I don't understand why that solution works or how to generalise to higher dimensions.
My first question is, why do you expect them to be same?
Let's do the kron without reshaping:
In [403]: X = np.array([[1, 2],
...: [3, 4]])
...:
In [404]: np.kron(X,X)
Out[404]:
array([[ 1, 2, 2, 4],
[ 3, 4, 6, 8],
[ 3, 6, 4, 8],
[ 9, 12, 12, 16]])
It's easy to visualize the action.
[X*1, X*2
X*3, X*4]
tensordot normally is thought of as a generalization of np.dot, able to handle more complex situations than the common matrix product (i.e. sum of products on one or more axes). But here there's no summing.
In [405]: np.tensordot(X,X, axes=0)
Out[405]:
array([[[[ 1, 2],
[ 3, 4]],
[[ 2, 4],
[ 6, 8]]],
[[[ 3, 6],
[ 9, 12]],
[[ 4, 8],
[12, 16]]]])
When axes is an integer rather than a tuple, the action is a little tricky to understand. The docs say:
``axes = 0`` : tensor product :math:`a\otimes b`
I just tried to explain what is happening when axes is a scalar (it's not trivial)
How does numpy.tensordot function works step-by-step?
Specifying axes=0 is equivalent to providing this tuple:
np.tensordot(X,X, axes=([],[]))
In any case it's evident from the output that this tensordot is producing the same numbers - but the layout is different from the kron.
I can replicate the kron layout with
In [424]: np.tensordot(X,X,axes=0).transpose(0,2,1,3).reshape(4,4)
Out[424]:
array([[ 1, 2, 2, 4],
[ 3, 4, 6, 8],
[ 3, 6, 4, 8],
[ 9, 12, 12, 16]])
That is I swap the middle 2 axes.
And omitting the reshape, I get the same (2,2,2,2) you get from kron:
np.tensordot(X,X,axes=0).transpose(0,2,1,3)
I like the explicitness of np.einsum:
np.einsum('ij,kl->ijkl',X,X) # = tensordot(X,X,0)
np.einsum('ij,kl->ikjl',X,X) # = kron(X,X).reshape(2,2,2,2)
Or using broadcasting, the 2 products are:
X[:,:,None,None]*X[None,None,:,:] # tensordot 0
X[:,None,:,None]*X[None,:,None,:] # kron
I am looking for a way to apply a function n items at the time along an axis. E.g.
array([[ 1, 2],
[ 3, 4],
[ 5, 6],
[ 7, 8]])
If I apply sum across the rows 2 items at a time I get:
array([[ 4, 6],
[ 12, 14]])
Which is the sum of 1st 2 rows and the last 2 rows.
NB: I am dealing with much larger array and I have to apply the function to n items which I can be decided at runtime.
The data extends along different axis. E.g.
array([[... [ 1, 2, ...],
[ 3, 4, ...],
[ 5, 6, ...],
[ 7, 8, ...],
...], ...])
This is a reduction:
numpy.add.reduceat(a, [0,2])
>>> array([[ 4, 6],
[12, 14]], dtype=int32)
As long as by "larger" you mean longer in the "y" axis, you can extend:
a = numpy.array([[ 1, 2],
[ 3, 4],
[ 5, 6],
[ 7, 8],
[ 9, 10],
[11, 12]])
numpy.add.reduceat(a, [0,2,4])
>>> array([[ 4, 6],
[12, 14],
[20, 22]], dtype=int32)
EDIT: actually, this works fine for "larger in both dimensions", too:
a = numpy.arange(24).reshape(6,4)
numpy.add.reduceat(a, [0,2,4])
>>> array([[ 4, 6, 8, 10],
[20, 22, 24, 26],
[36, 38, 40, 42]], dtype=int32)
I will leave it up to you to adapt the indices to your specific case.
Reshape splitting the first axis into two axes, such that the second split axis is of length n to have a 3D array and then sum along that split axis, like so -
a.reshape(a.shape[0]//n,n,a.shape[1]).sum(1)
It should be pretty efficient as reshaping just creates a view into input array.
Sample run -
In [55]: a
Out[55]:
array([[2, 8, 0, 0],
[1, 5, 3, 3],
[6, 1, 4, 7],
[0, 4, 0, 7],
[8, 0, 8, 1],
[8, 3, 3, 8]])
In [56]: n = 2 # Sum every two rows
In [57]: a.reshape(a.shape[0]//n,n,a.shape[1]).sum(1)
Out[57]:
array([[ 3, 13, 3, 3],
[ 6, 5, 4, 14],
[16, 3, 11, 9]])
How about something like this?
n = 2
# calculate the cumsum along axis 0 and take one row from every n rows
cumarr = arr.cumsum(axis = 0)[(n-1)::n]
# calculate the difference of the resulting numpy array along axis 0
np.vstack((cumarr[0][None, :], np.diff(cumarr, axis=0)))
# array([[ 4, 6],
# [12, 14]])
I have a 4D array training images, whose dimensions correspond to (image_number,channels,width,height). I also have a 2D target labels，whose dimensions correspond to (image_number,class_number). When training, I want to randomly shuffle the data by using random.shuffle, but how can I keep the labels shuffled by the same order of my images? Thx!
from sklearn.utils import shuffle
import numpy as np
X = np.array([[0, 0, 0], [1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4]])
y = np.array([0, 1, 2, 3, 4])
X, y = shuffle(X, y)
print(X)
print(y)
[[1 1 1]
[3 3 3]
[0 0 0]
[2 2 2]
[4 4 4]]
[1 3 0 2 4]
There is another easy way to do that. Let us suppose that there are total N images. Then we can do the following:
from random import shuffle
ind_list = [i for i in range(N)]
shuffle(ind_list)
train_new = train[ind_list, :,:,:]
target_new = target[ind_list,]
If you want a numpy-only solution, you can just reindex the second array on the first, assuming you've got the same image numbers in both:
In [67]: train = np.arange(20).reshape(4,5).T
In [68]: target = np.hstack([np.arange(5).reshape(5,1), np.arange(100, 105).reshape(5,1)])
In [69]: train
Out[69]:
array([[ 0, 5, 10, 15],
[ 1, 6, 11, 16],
[ 2, 7, 12, 17],
[ 3, 8, 13, 18],
[ 4, 9, 14, 19]])
In [70]: target
Out[70]:
array([[ 0, 100],
[ 1, 101],
[ 2, 102],
[ 3, 103],
[ 4, 104]])
In [71]: np.random.shuffle(train)
In [72]: target[train[:,0]]
Out[72]:
array([[ 2, 102],
[ 3, 103],
[ 1, 101],
[ 4, 104],
[ 0, 100]])
In [73]: train
Out[73]:
array([[ 2, 7, 12, 17],
[ 3, 8, 13, 18],
[ 1, 6, 11, 16],
[ 4, 9, 14, 19],
[ 0, 5, 10, 15]])
If you're looking for a sync/ unison shuffle you can use the following func.
def unisonShuffleDataset(a, b):
assert len(a) == len(b)
p = np.random.permutation(len(a))
return a[p], b[p]
the one above is only for 2 numpy. One can extend to more than 2 by adding the number of input vars on the func. and also on the return of the function.
Depending on what you want to do, you could also randomly generate a number for each dimension of your array with
random.randint(a, b) #a and b are the extremes of your array
which would select randomly amongst your objects.
Use the same seed to build the random generator multiple times to shuffle different arrays:
>>> seed = np.random.SeedSequence()
>>> arrays = [np.arange(10).repeat(i).reshape(10, -1) for i in range(1, 4)]
>>> for ar in arrays:
... np.random.default_rng(seed).shuffle(ar)
...
>>> arrays
[array([[1],
[2],
[7],
[8],
[0],
[4],
[3],
[6],
[9],
[5]]),
array([[1, 1],
[2, 2],
[7, 7],
[8, 8],
[0, 0],
[4, 4],
[3, 3],
[6, 6],
[9, 9],
[5, 5]]),
array([[1, 1, 1],
[2, 2, 2],
[7, 7, 7],
[8, 8, 8],
[0, 0, 0],
[4, 4, 4],
[3, 3, 3],
[6, 6, 6],
[9, 9, 9],
[5, 5, 5]])]