Is there a way to slice the array below without having to define the row indices i.e. not having to write range(len(X))?
X = np.arange(10*2).reshape((10,2))
L = np.random.randint(0,2,10)
Xs = X[range(len(X)),L]
I thought it was possible to slice with X[:,L] but looks like it's not.
You're probably looking for np.choose:
In [25]: X = np.arange(10*2).reshape((10,2)); X
Out[25]:
array([[ 0, 1],
[ 2, 3],
[ 4, 5],
[ 6, 7],
[ 8, 9],
[10, 11],
[12, 13],
[14, 15],
[16, 17],
[18, 19]])
In [26]: L = np.random.randint(0,2,10); L
Out[26]: array([1, 1, 1, 1, 1, 0, 0, 0, 0, 1])
In [27]: L.choose(X.T)
Out[27]: array([ 1, 3, 5, 7, 9, 10, 12, 14, 16, 19])
In [28]: # or otherwise
In [29]: np.choose(L, X.T)
Out[29]: array([ 1, 3, 5, 7, 9, 10, 12, 14, 16, 19])
Performance note: while this solution is a direct answer to the question, it's quickly becomes not the most optimal with increase of len(X). As of numpy 1.9.0, np.arange approach is faster:
In [17]: %timeit X[range(len(X)), L]
1000 loops, best of 3: 629 µs per loop
In [18]: %timeit X[np.arange(len(X)), L]
10000 loops, best of 3: 78.8 µs per loop
In [19]: %timeit L.choose(X.T)
10000 loops, best of 3: 146 µs per loop
In [20]: X.shape, L.shape
Out[20]: ((10000, 2), (10000,))
You take the diagonal elements of X[:,L] using diag (or diagonal):
np.diag(X[:,L])
Another way to do it is with where:
np.where(L,X[:,1],X[:,0])
Note that
In [9]: X[:, L]
Out[9]:
array([[ 1, 1, 0, 0, 1, 0, 1, 0, 1, 0],
[ 3, 3, 2, 2, 3, 2, 3, 2, 3, 2],
[ 5, 5, 4, 4, 5, 4, 5, 4, 5, 4],
[ 7, 7, 6, 6, 7, 6, 7, 6, 7, 6],
[ 9, 9, 8, 8, 9, 8, 9, 8, 9, 8],
[11, 11, 10, 10, 11, 10, 11, 10, 11, 10],
[13, 13, 12, 12, 13, 12, 13, 12, 13, 12],
[15, 15, 14, 14, 15, 14, 15, 14, 15, 14],
[17, 17, 16, 16, 17, 16, 17, 16, 17, 16],
[19, 19, 18, 18, 19, 18, 19, 18, 19, 18]])
And you want the diagonal elements:
So just do:
In [14]: X[:, L].diagonal()
Out[14]: array([ 1, 3, 4, 6, 9, 10, 13, 14, 17, 18])
Related
I have a tensor x, that looks like this:
x = tensor([ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10]
[ 11, 12, 13, 14, 15])
I'm trying to switch the first two and last two numbers of each tensor, like this:
x = tensor([ 4, 5, 3, 1, 2],
[ 9, 10, 8, 6, 7],
[ 14, 15, 13, 11, 12])
How could I do this with torch.roll()? How would I switch 3 instead of 1?
Not sure if that can be done with torch.roll alone... However, you can expect the desired result by using a temporary tensor and a pair assignment:
>>> x = torch.arange(1, 16).reshape(3,-1)
tensor([[ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10],
[11, 12, 13, 14, 15]])
>>> tmp = x.clone()
# swap the two sets of columns
>>> x[:,:2], x[:,-2:] = tmp[:,-2:], tmp[:,:2]
Such that tensor x has been mutated as:
>>> x
tensor([[ 4, 5, 3, 1, 2],
[ 9, 10, 8, 6, 7],
[14, 15, 13, 11, 12]])
You can pull off this operation with torch.roll and some indexing:
>>> x = torch.arange(1, 21).reshape(4,-1)
tensor([[ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10],
[11, 12, 13, 14, 15],
[16, 17, 18, 19, 20]])
>>> rolled = x.roll(-2,0)
tensor([[11, 12, 13, 14, 15],
[16, 17, 18, 19, 20],
[ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10]])
# overwrite columns [1,-1[ from rolled with those from x
>>> rolled[:, 1:-1] = x[:, 1:-1]
Such that at this end you get:
>>> rolled
tensor([[11, 2, 3, 4, 15],
[16, 7, 8, 9, 20],
[ 1, 12, 13, 14, 5],
[ 6, 17, 18, 19, 10]])
I have an array
xx = np.arange(24).reshape(2, 12)
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]])
and I would like to reshape it, to obtain
array([[[ 0, 1, 2, 3],
[12, 13, 14, 15]],
[[ 4, 5, 6, 7],
[16, 17, 18, 19]],
[[ 8, 9, 10, 11],
[20, 21, 22, 23]]])
I can achieve it via
xx.T.reshape(3, 4, 2).transpose(0, 2, 1)
But it has to be transposed twice, which seems unnecessary to me. So could somebody confirm that this is the only way of doing it or provide more readable solution otherwise?
Thanks!
It is possible to do a single transpose:
data = np.arange(24).reshape(2, 12)
data = data.reshape(2, 3, 4).transpose(1, 0, 2)
Edit:
I checked this using itertools.permutations and itertools.product:
import itertools
import numpy as np
data = np.arange(24).reshape(2, 12)
desired_data = np.array([[[ 0, 1, 2, 3],
[12, 13, 14, 15]],
[[ 4, 5, 6, 7],
[16, 17, 18, 19]],
[[ 8, 9, 10, 11],
[20, 21, 22, 23]]])
shapes = [2, 3, 4]
transpose_dims = [0, 1, 2]
shape_permutations = itertools.permutations(shapes)
transpose_permutations = itertools.permutations(transpose_dims)
for shape, transpose in itertools.product(
list(shape_permutations),
list(transpose_permutations),
):
new_data = data.reshape(*shape).transpose(*transpose)
try:
np.allclose(new_data, desired_data)
except ValueError as e:
pass
else:
break
print(f"{shape=}, {transpose=}")
shape=(2, 3, 4), transpose=(1, 0, 2)
I would do it this way: first, generate two arrays (shown separated for the sake of decomposition):
xx.reshape(2, -1, 4)
# Output:
# array([[[ 0, 1, 2, 3],
# [ 4, 5, 6, 7],
# [ 8, 9, 10, 11]],
#
# [[12, 13, 14, 15],
# [16, 17, 18, 19],
# [20, 21, 22, 23]]])
From here, I would then stack along the second dimension in order to combine them like you want:
np.stack(xx.reshape(2, -1, 4), axis=1)
# Output:
# array([[[ 0, 1, 2, 3],
# [12, 13, 14, 15]],
#
# [[ 4, 5, 6, 7],
# [16, 17, 18, 19]],
#
# [[ 8, 9, 10, 11],
# [20, 21, 22, 23]]])
You'd avoid the transposition. Hopefully it's more readable, but in the end, that's highly subjective, right? '^^
To add on top of #Paul's answer, there is some speedup from removing one of the transpose. The time gain is of ~15%:
I'm considering a numpy array:
import numpy as np
b = np.empty((10,11,12))
Now I would expect the following shapes to be the same, but they apparently are not:
>>> b[0,:,1].shape
>>> (11,)
and
>>> b[0][:][1].shape
>>> (12,)
Can somebody explain to me why the shapes are different? I read the Numpy documentation on indexing but there it says that writing a[k][l] is the same as a[k,l].
This happens because b[0][:] is a view of b[0], so that b[0][:][1] is really b[0, 1, :]. A numeric example may help to highlight what is going on:
In [5]: b = np.arange(3*4*5).reshape((3, 4, 5))
In [6]: b[0]
Out[6]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
In [7]: b[0, :]
Out[7]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
In [8]: b[0, :, 1]
Out[8]: array([ 1, 6, 11, 16])
In [10]: b[0][:]
Out[10]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
In [11]: b[0][:][1]
Out[11]: array([5, 6, 7, 8, 9])
In [13]: b[0, 1, :]
Out[13]: array([5, 6, 7, 8, 9])
In [32]: b[0][:, 1]
Out[32]: array([ 1, 6, 11, 16])
I want to shuffle the ordering of only some rows in a numpy array. These rows will always be continuous (e.g. shuffling rows 23-80). The number of elements in each row can vary from 1 (such that the array is actually 1D) to 100.
Below is example code to demonstrate how I see the method shuffle_rows() could work. How would I design such a method to do this shuffling efficiently?
import numpy as np
>>> a = np.arange(20).reshape(4, 5)
>>> a
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
>>> shuffle_rows(a, [1, 3]) # including rows 1, 2 and 3 in the shuffling
array([[ 0, 1, 2, 3, 4],
[15, 16, 17, 18, 19],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14]])
You can use np.random.shuffle. This shuffles the rows themselves, not the elements within the rows.
From the docs:
This function only shuffles the array along the first index of a multi-dimensional array
As an example:
import numpy as np
def shuffle_rows(arr,rows):
np.random.shuffle(arr[rows[0]:rows[1]+1])
a = np.arange(20).reshape(4, 5)
print(a)
# array([[ 0, 1, 2, 3, 4],
# [ 5, 6, 7, 8, 9],
# [10, 11, 12, 13, 14],
# [15, 16, 17, 18, 19]])
shuffle_rows(a,[1,3])
print(a)
#array([[ 0, 1, 2, 3, 4],
# [10, 11, 12, 13, 14],
# [15, 16, 17, 18, 19],
# [ 5, 6, 7, 8, 9]])
shuffle_rows(a,[1,3])
print(a)
#array([[ 0, 1, 2, 3, 4],
# [10, 11, 12, 13, 14],
# [ 5, 6, 7, 8, 9],
# [15, 16, 17, 18, 19]])
so this is a question regarding the use of reshape and how this functions uses each axis on a multidimensional scale.
Suppose I have the following array that contains matrices indexed by the first index. What I want to achieve is to instead index the columns of each matrix with the first index. In order to illustrate this problem, consider the following example where the given numpy array that indexes matrices with its first index is z.
x = np.arange(9).reshape((3, 3))
y = np.arange(9, 18).reshape((3, 3))
z = np.dstack((x, y)).T
Where z looks like:
array([[[ 0, 3, 6],
[ 1, 4, 7],
[ 2, 5, 8]],
[[ 9, 12, 15],
[10, 13, 16],
[11, 14, 17]]])
And its shape is (2, 3, 3). Here, the first index are the two images and the three x three is a matrix.
The question more specifically phrased then, is how to use reshape to obtain the following desired output:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]])
Whose shape is (6, 3). This achieves that the dimension of the array indexes the columns of the matrix x and y as presented above. My natural inclination was to use reshape directly on z in the following way:
out = z.reshape(2 * 3, 3)
But its output is the following which indexes the rows of the matrices and not the columns:
array([[ 0, 3, 6],
[ 1, 4, 7],
[ 2, 5, 8],
[ 9, 12, 15],
[10, 13, 16],
[11, 14, 17]]
Could reshape be used to obtain the desired output above? Or more general, can you control how each axis is used when you use the reshape function?
Two things:
I know how to solve the problem. I can go through each element of the big matrix (z) transposed and then apply reshape in the way above. This increases computation time a little bit and is not really problematic. But it does not generalize and it does not feel python. So I was wondering if there is a standard enlightened way of doing this.
I was not clear on how to phrase this question. If anyone has suggestion on how to better phrase this problem I am all ears.
Every array has a natural (1D flattened) order to its elements. When you reshape an array, it is as though it were flattened first (thus obtaining the natural order), and then reshaped:
In [54]: z.ravel()
Out[54]:
array([ 0, 3, 6, 1, 4, 7, 2, 5, 8, 9, 12, 15, 10, 13, 16, 11, 14,
17])
In [55]: z.ravel().reshape(2*3, 3)
Out[55]:
array([[ 0, 3, 6],
[ 1, 4, 7],
[ 2, 5, 8],
[ 9, 12, 15],
[10, 13, 16],
[11, 14, 17]])
Notice that in the "natural order", 0 and 1 are far apart. However you reshape it, 0 and 1 will not be next to each other along the last axis, which is what you want in the desired array:
desired = np.array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]])
This requires some reordering, which in this case can be done by swapaxes:
In [53]: z.swapaxes(1,2).reshape(2*3, 3)
Out[53]:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]])
because swapaxes(1,2) places the values in the desired order
In [56]: z.swapaxes(1,2).ravel()
Out[56]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17])
In [57]: desired.ravel()
Out[57]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17])
Note that the reshape method also has a order parameter which can be used to control the (C- or F-) order with which the elements are read from the array and placed in the reshaped array. However, I don't think this helps in your case.
Another way to think about the limits of reshape is to say that all reshapes followed by ravel are the same:
In [71]: z.reshape(3,3,2).ravel()
Out[71]:
array([ 0, 3, 6, 1, 4, 7, 2, 5, 8, 9, 12, 15, 10, 13, 16, 11, 14,
17])
In [72]: z.reshape(3,2,3).ravel()
Out[72]:
array([ 0, 3, 6, 1, 4, 7, 2, 5, 8, 9, 12, 15, 10, 13, 16, 11, 14,
17])
In [73]: z.reshape(3*2,3).ravel()
Out[73]:
array([ 0, 3, 6, 1, 4, 7, 2, 5, 8, 9, 12, 15, 10, 13, 16, 11, 14,
17])
In [74]: z.reshape(3*3,2).ravel()
Out[74]:
array([ 0, 3, 6, 1, 4, 7, 2, 5, 8, 9, 12, 15, 10, 13, 16, 11, 14,
17])
So if the ravel of the desired array is different, there is no way to obtain it only be reshaping.
The same goes for reshaping with order='F', provided you also ravel with order='F':
In [109]: z.reshape(2,3,3, order='F').ravel(order='F')
Out[109]:
array([ 0, 9, 1, 10, 2, 11, 3, 12, 4, 13, 5, 14, 6, 15, 7, 16, 8,
17])
In [110]: z.reshape(2*3*3, order='F').ravel(order='F')
Out[110]:
array([ 0, 9, 1, 10, 2, 11, 3, 12, 4, 13, 5, 14, 6, 15, 7, 16, 8,
17])
In [111]: z.reshape(2*3,3, order='F').ravel(order='F')
Out[111]:
array([ 0, 9, 1, 10, 2, 11, 3, 12, 4, 13, 5, 14, 6, 15, 7, 16, 8,
17])
It is possible to obtain the desired array using two reshapes:
In [83]: z.reshape(2, 3*3, order='F').reshape(2*3, 3)
Out[83]:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]])
but I stumbled upon this serendipidously.
If I've totally misunderstood your question and x and y are the givens (not z) then you could obtain the desired array using row_stack instead of dstack:
In [88]: z = np.row_stack([x, y])
In [89]: z
Out[89]:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]])
It you look at dstack code you'll discover that
np.dstack((x, y)).T
is effectively:
np.concatenate([i[:,:,None] for i in (x,y)],axis=2).transpose([2,1,0])
It reshapes each component array and then joins them along this new axis. Finally it transposes axes.
Your target is the same as (row stack)
np.concatenate((x,y),axis=0)
So with a bit of reverse engineering we can create it from z with
np.concatenate([i[...,0] for i in np.split(z.T,2,axis=2)],axis=0)
np.concatenate([i.T[:,:,0] for i in np.split(z,2,axis=0)],axis=0)
or
np.concatenate(np.split(z.T,2,axis=2),axis=0)[...,0]
or with a partial transpose we can keep the split-and-rejoin axis first, and just use concatenate:
np.concatenate(z.transpose(0,2,1),axis=0)
or its reshape equivalent
(z.transpose(0,2,1).reshape(-1,3))