I'm trying to index the maximum elements along the last dimension in a multidimensional tensor. For example, say I have a tensor
A = torch.randn((5, 2, 3))
_, idx = torch.max(A, dim=2)
Here idx stores the maximum indices, which may look something like
>>>> A
tensor([[[ 1.0503, 0.4448, 1.8663],
[ 0.8627, 0.0685, 1.4241]],
[[ 1.2924, 0.2456, 0.1764],
[ 1.3777, 0.9401, 1.4637]],
[[ 0.5235, 0.4550, 0.2476],
[ 0.7823, 0.3004, 0.7792]],
[[ 1.9384, 0.3291, 0.7914],
[ 0.5211, 0.1320, 0.6330]],
[[ 0.3292, 0.9086, 0.0078],
[ 1.3612, 0.0610, 0.4023]]])
>>>> idx
tensor([[ 2, 2],
[ 0, 2],
[ 0, 0],
[ 0, 2],
[ 1, 0]])
I want to be able to access these indices and assign to another tensor based on them. Meaning I want to be able to do
B = torch.new_zeros(A.size())
B[idx] = A[idx]
where B is 0 everywhere except where A is maximum along the last dimension. That is B should store
>>>>B
tensor([[[ 0, 0, 1.8663],
[ 0, 0, 1.4241]],
[[ 1.2924, 0, 0],
[ 0, 0, 1.4637]],
[[ 0.5235, 0, 0],
[ 0.7823, 0, 0]],
[[ 1.9384, 0, 0],
[ 0, 0, 0.6330]],
[[ 0, 0.9086, 0],
[ 1.3612, 0, 0]]])
This is proving to be much more difficult than I expected, as the idx does not index the array A properly. Thus far I have been unable to find a vectorized solution to use idx to index A.
Is there a good vectorized way to do this?
You can use torch.meshgrid to create an index tuple:
>>> index_tuple = torch.meshgrid([torch.arange(x) for x in A.size()[:-1]]) + (idx,)
>>> B = torch.zeros_like(A)
>>> B[index_tuple] = A[index_tuple]
Note that you can also mimic meshgrid via (for the specific case of 3D):
>>> index_tuple = (
... torch.arange(A.size(0))[:, None],
... torch.arange(A.size(1))[None, :],
... idx
... )
Bit more explanation:
We will have the indices something like this:
In [173]: idx
Out[173]:
tensor([[2, 1],
[2, 0],
[2, 1],
[2, 2],
[2, 2]])
From this, we want to go to three indices (since our tensor is 3D, we need three numbers to retrieve each element). Basically we want to build a grid in the first two dimensions, as shown below. (And that's why we use meshgrid).
In [174]: A[0, 0, 2], A[0, 1, 1]
Out[174]: (tensor(0.6288), tensor(-0.3070))
In [175]: A[1, 0, 2], A[1, 1, 0]
Out[175]: (tensor(1.7085), tensor(0.7818))
In [176]: A[2, 0, 2], A[2, 1, 1]
Out[176]: (tensor(0.4823), tensor(1.1199))
In [177]: A[3, 0, 2], A[3, 1, 2]
Out[177]: (tensor(1.6903), tensor(1.0800))
In [178]: A[4, 0, 2], A[4, 1, 2]
Out[178]: (tensor(0.9138), tensor(0.1779))
In the above 5 lines, the first two numbers in the indices are basically the grid that we build using meshgrid and the third number is coming from idx.
i.e. the first two numbers form a grid.
(0, 0) (0, 1)
(1, 0) (1, 1)
(2, 0) (2, 1)
(3, 0) (3, 1)
(4, 0) (4, 1)
An ugly hackaround is to create a binary mask out of idx and use it to index the arrays. The basic code looks like this:
import torch
torch.manual_seed(0)
A = torch.randn((5, 2, 3))
_, idx = torch.max(A, dim=2)
mask = torch.arange(A.size(2)).reshape(1, 1, -1) == idx.unsqueeze(2)
B = torch.zeros_like(A)
B[mask] = A[mask]
print(A)
print(B)
The trick is that torch.arange(A.size(2)) enumerates the possible values in idx and mask is nonzero in places where they equal the idx. Remarks:
If you really discard the first output of torch.max, you can use torch.argmax instead.
I assume that this is a minimal example of some wider problem, but be aware that you are currently reinventing torch.nn.functional.max_pool3d with kernel of size (1, 1, 3).
Also, be aware that in-place modification of tensors with masked assignment can cause issues with autograd, so you may want to use torch.where as shown here.
I would expect that somebody comes up with a cleaner solution (avoiding the intermedia allocation of the mask array), likely making use of torch.index_select, but I can't get it to work right now.
could use torch.scatter here
>>> import torch
>>> a = torch.randn(4,2,3)
>>> a
tensor([[[ 0.1583, 0.1102, -0.8188],
[ 0.6328, -1.9169, -0.5596]],
[[ 0.5335, 0.4069, 0.8403],
[-1.2537, 0.9868, -0.4947]],
[[-1.2830, 0.4386, -0.0107],
[ 1.3384, 0.5651, 0.2877]],
[[-0.0334, -1.0619, -0.1144],
[ 0.1954, -0.7371, 1.7001]]])
>>> ind = torch.max(a,1,keepdims=True)[1]
>>> ind
tensor([[[1, 0, 1]],
[[0, 1, 0]],
[[1, 1, 1]],
[[1, 1, 1]]])
>>> torch.zeros_like(a).scatter(1,ind,a)
tensor([[[ 0.0000, 0.1102, 0.0000],
[ 0.1583, 0.0000, -0.8188]],
[[ 0.5335, 0.0000, 0.8403],
[ 0.0000, 0.4069, 0.0000]],
[[ 0.0000, 0.0000, 0.0000],
[-1.2830, 0.4386, -0.0107]],
[[ 0.0000, 0.0000, 0.0000],
[-0.0334, -1.0619, -0.1144]]])
Related
I have a 2d numpy array consisting of 1s and 0s.
I want to club up the 1s and 0s of each row.
arr =
[[0 1 0]
[ 0 0 0]
[ 1 1 1]
[ 0 1 1]]
Desired output (each element is dtype str, to make sure leading zeros are not omitted)
[ 010 , 000 , 111 , 011 ]
How can I manipulate the 2d array to get this output? Is it possible in numpy or regex packages, by using their functions? Can a for loop be avoided to do this array transformation?
Using strings:
import numpy as np
arr = np.array([[0, 1, 0], [ 0, 0, 0], [ 1, 1, 1], [ 0, 1, 1]])
binaries = []
for idx, row in enumerate(arr):
strings = [str(integer) for integer in row]
a_string = "".join(strings)
binaries.append(a_string)
>>> binaries
>>> ['010', '000', '111', '011']
The question is quite unclear, assuming integers in and out, you could use:
a = np.array([[0, 1, 0],
[0, 0, 0],
[1, 1, 1],
[0, 1, 1]])
out = (a[:,::-1]*(10**np.arange(a.shape[1]))).sum(1)
But you won't have leading zeros…
output:
array([ 10, 0, 111, 11])
Assuming you really want to convert from binary, you should probably use np.packbits:
out = np.packbits(np.pad(a, ((0,0), (8-a.shape[1],0))), axis=1, bitorder='big')
output:
array([[2],
[0],
[7],
[3]], dtype=uint8)
or as flat version:
out = (np.packbits(np.pad(a, ((0,0), (8-a.shape[1],0))), axis=1, bitorder='big')
.ravel()
)
# array([2, 0, 7, 3], dtype=uint8)
I'm trying to get the index values out of a numpy array, I've tried using intersects instead to no avail. I'm simply trying to find like values in 2 arrays. One is 2D and I'm selecting a column, and the other is 1D, just a list of values to search for, so effectively just 2 1D arrays.
We'll call this array a:
array([[ 1, 97553, 1],
[ 1, 97587, 1],
[ 1, 97612, 1],
[ 1, 97697, 1],
[ 1, 97826, 3],
[ 1, 97832, 1],
[ 1, 97839, 1],
[ 1, 97887, 1],
[ 1, 97944, 1],
[ 1, 97955, 2]])
And we're searching say, values = numpy.array([97612, 97633, 97697, 97999, 97943, 97944])
So I try:
numpy.where(a[:, 1] == values)
And I'd expect a bunch of indices of the values, but instead I get back an array that's empty, it spits out [(array([], dtype=int64),)].
If I try this though:
numpy.where(a[:, 1] == 97697)
It gives me back (array([2]),), which is what I would expect.
What weirdness of arrays am I missing here? Or is there maybe even an easier way to do this? Finding array indices and matching arrays seems to not work as I expect at all. When I want to find the unions or intersects of arrays, by indice or unique value it just doesn't seem to function. Any help would be super. Thanks.
Edit:
As per Warrens request:
import numpy
a = numpy.array([[ 1, 97553, 1],
[ 1, 97587, 1],
[ 1, 97612, 1],
[ 1, 97697, 1],
[ 1, 97826, 3],
[ 1, 97832, 1],
[ 1, 97839, 1],
[ 1, 97887, 1],
[ 1, 97944, 1],
[ 1, 97955, 2]])
values = numpy.array([97612, 97633, 97697, 97999, 97943, 97944])
I've found that numpy.in1d will give me a correct truth table of booleans for the operation, with a 1d array of the same length that should map to the original data. My only issue here is now how to act with that, for instance deleting or modifying the original array at those indices. I could do it laboriously with a loop, but as far as I know there are better ways in numpy. Truth tables as masks are supposed to be quite powerful with numpy from what I have been able to find.
np.where with a single argument is equivalent to np.nonzero. It gives you the indices where a condition, the input array, is True.
In your example you are checking for element-wise equality between a[:,1] and values
a[:, 1] == values
False
So it's giving you the correct result: no index in the input is True.
You should use np.isin instead
np.isin(a[:,1], values)
array([False, False, True, True, False, False, False, False, True, False], dtype=bool)
Now you can use np.where to get the indices
np.where(np.isin(a[:,1], values))
(array([2, 3, 8]),)
and use those to address the original array
a[np.where(np.isin(a[:,1], values))]
array([[ 1, 97612, 1],
[ 1, 97697, 1],
[ 1, 97944, 1]])
Your initial solution with a simple equality check could indeed have worked with proper broadcasting:
np.where(a[:,1] == values[..., np.newaxis])[1]
array([2, 3, 8])
EDIT: given you seem to have issues with using the above results to index and manipulate your array here's a couple of simple examples
Now you should have two ways of accessing your matching elements in the original array, either the binary mask or the indices from np.where.
mask = np.isin(a[:,1], values) # np.in1d if np.isin is not available
idx = np.where(mask)
Let's say you want to set all matching rows to zero
a[mask] = 0 # or a[idx] = 0
array([[ 1, 97553, 1],
[ 1, 97587, 1],
[ 0, 0, 0],
[ 0, 0, 0],
[ 1, 97826, 3],
[ 1, 97832, 1],
[ 1, 97839, 1],
[ 1, 97887, 1],
[ 0, 0, 0],
[ 1, 97955, 2]])
Or you want to multiply the third column of matching rows by 100
a[mask, 2] *= 100
array([[ 1, 97553, 1],
[ 1, 97587, 1],
[ 1, 97612, 100],
[ 1, 97697, 100],
[ 1, 97826, 3],
[ 1, 97832, 1],
[ 1, 97839, 1],
[ 1, 97887, 1],
[ 1, 97944, 100],
[ 1, 97955, 2]])
Or you want to delete matching rows (here using indices is more convenient than masks)
np.delete(a, idx, axis=0)
array([[ 1, 97553, 1],
[ 1, 97587, 1],
[ 1, 97826, 3],
[ 1, 97832, 1],
[ 1, 97839, 1],
[ 1, 97887, 1],
[ 1, 97955, 2]])
Just a thought:
Try to flatten the 2D array and compare using numpy.intersect1d.
https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.ndarray.flatten.html
https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.intersect1d.html
Lets say I have the following data set
[[0 1994]
[0 1965]
[0, 1943]
[1, 1994]
[1,1965]
[2, 1948]]
I want to achieve the following output by separating out the data into individual arrays based on the value in the first row using numpy or Pandas methods instead of loops.
[
[[0 1994]
[0 1965]
[0 1943]]
[[1 1994]
[1 1965]]
[[2, 1948]]
]
Find the expected indices based on the differences of the firs columns' items and then split the array based on those indices:
In [22]: inds = np.where(np.diff(a[:,0]) != 0)[0] + 1
In [23]: np.split(a, inds)
Out[23]:
[array([[ 0, 1994],
[ 0, 1965],
[ 0, 1943]]), array([[ 1, 1994],
[ 1, 1965]]), array([[ 2, 1948]])]
The pandas way to do is to pass the data to DataFrame and then groupby operation:
df = pd.DataFrame(data)
df.groupby(0).apply(lambda x: x.values).values
Output:
[array([[ 0, 1994],
[ 0, 1965],
[ 0, 1943]]), array([[ 1, 1994],
[ 1, 1965]]), array([[ 2, 1948]])]
if i have an array x, which value is as follow with shape (2,3,4):
array([[[ 0.15845319, 0.57808432, 0.05638804, 0.56237656],
[ 0.73164208, 0.80562342, 0.64561066, 0.15397456],
[ 0.34734043, 0.88063258, 0.4863103 , 0.09881028]],
[[ 0.35823078, 0.71260357, 0.49410944, 0.94909165],
[ 0.02730397, 0.67890392, 0.74340148, 0.47434223],
[ 0.02494292, 0.59827256, 0.20550867, 0.30859339]]])
and i have an index array y, which shape is (2, 3, 3), and the value is:
array([[[0, 2, 2],
[2, 0, 2],
[0, 0, 2]],
[[1, 2, 1],
[1, 1, 1],
[1, 2, 2]]])
so i could use x[0,0,y[0][0]] to index the array x, and it will generate the output as follow:
array([ 0.15845319, 0.05638804, 0.05638804])
my question is: is there any simple way to do this? i had already tried with
x[y], it did not work.
You could use fancy-indexing -
m,n = y.shape[:2]
out = x[np.arange(m)[:,None,None],np.arange(n)[:,None],y]
I have two 2D numpy arrays like this, representing the x/y distances between three points. I need the x/y distances as tuples in a single array.
So from:
x_dists = array([[ 0, -1, -2],
[ 1, 0, -1],
[ 2, 1, 0]])
y_dists = array([[ 0, -1, -2],
[ 1, 0, -1],
[ 2, 1, 0]])
I need:
dists = array([[[ 0, 0], [-1, -1], [-2, -2]],
[[ 1, 1], [ 0, 0], [-1, -1]],
[[ 2, 2], [ 1, 1], [ 0, 0]]])
I've tried using various permutations of dstack/hstack/vstack/concatenate, but none of them seem to do what I want. The actual arrays in code are liable to be gigantic, so iterating over the elements in python and doing the rearrangement "manually" isn't an option speed-wise.
Edit:
This is what I came up with in the end: https://gist.github.com/807656
import numpy as np
dists = np.vstack(([x_dists.T], [y_dists.T])).T
returns dists like you wanted them. Afterwards it is not "a single 2D array of 2-tuples", but a normal 3D array where the third axis is the concatenation of the two original arrays.
You see:
dists.shape # (3, 3, 2)
numpy.rec.fromarrays([x_dists, y_dists], names='x,y')