I am trying to use the scatter_nd function in TensorFlow to reorder elements within rows of a Matrix. For example, suppose I have the code:
indices = tf.constant([[1],[0]])
updates = tf.constant([ [5, 6, 7, 8],
[1, 2, 3, 4] ])
shape = tf.constant([2, 4])
scatter1 = tf.scatter_nd(indices, updates, shape)
$ print(scatter1) = [[1,2,3,4]
[5,6,7,8]]
This reorders the rows of the updates matrix.
Instead of only being able to reorder the rows, I'd like to reorder the individual elements within each row as well. If I just have a vector (Tensor of rank 1), then this example works:
indices = tf.constant([[1],[0],[2],[3]])
updates = tf.constant([5, 6, 7, 8])
shape = tf.constant([4])
scatter2 = tf.scatter_nd(indices, updates, shape)
$ print(scatter2) = [6,5,7,8]
What I really care about is to be able to swap elements within each row in scatter1, as I had done in scatter2, but do it for each row of scatter1. I've tried various combinations of indices but keep getting errors that the sizes are inconsistent thrown by the scatter_nd function.
The following swaps the elements of each row of each row using scatter_nd
indices = tf.constant([[[0, 1], [0, 0], [0, 2], [0, 3]],
[[1, 1], [1, 0], [1, 2], [1, 3]]])
updates = tf.constant([ [5, 6, 7, 8],
[1, 2, 3, 4] ])
shape = tf.constant([2, 4])
scatter1 = tf.scatter_nd(indices, updates, shape)
with tf.Session() as sess:
print(sess.run(scatter1))
Giving an output of:
[[6 5 7 8]
[2 1 3 4]]
The locations of the coordinate in indices define where the values are being taken from in updates and the actual cordinates define where the values will be placed in scatter1.
This answer is a few months late but hopefully still helpful.
Suppose you want to swap elements in the second dimension either keeping the first dimension order or not.
import tensorflow as tf
sess = tf.InteractiveSession()
def prepare_fd(fd_indices, sd_dims):
fd_indices = tf.expand_dims(fd_indices, 1)
fd_indices = tf.tile(fd_indices, [1, sd_dims])
return fd_indices
# define the updates
updates = tf.constant([[11, 12, 13, 14],
[21, 22, 23, 24],
[31, 32, 33, 34]])
sd_dims = tf.shape(updates)[1]
sd_indices = tf.constant([[1, 0, 2, 3], [0, 2, 1, 3], [0, 1, 3, 2]])
fd_indices_range = tf.range(0, limit=tf.shape(updates)[0])
fd_indices_custom = tf.constant([2, 0, 1])
# define the indices
indices1 = tf.stack((prepare_fd(fd_indices_range, sd_dims), sd_indices), axis=2)
indices2 = tf.stack((prepare_fd(fd_indices_custom, sd_dims), sd_indices), axis=2)
# define the shape
shape = tf.shape(updates)
scatter1 = tf.scatter_nd(indices1, updates, shape)
scatter2 = tf.scatter_nd(indices2, updates, shape)
print(scatter1.eval())
# array([[12, 11, 13, 14],
# [21, 23, 22, 24],
# [31, 32, 34, 33]], dtype=int32)
print(scatter2.eval())
# array([[21, 23, 22, 24],
# [31, 32, 34, 33],
# [12, 11, 13, 14]], dtype=int32)
May this example help.
Related
I am new to numpy and python and I am trying to understand the usage of transpose function of numpy. The code below works fine but I am still not be able to understand the effect of transpose function and also the use of the arguments inside it. It would be great help if someone can explain the usage and effect of transpose function in below code.
import numpy as np
my_list = [[[[[[1,2],[3,4]],[[1,2],[3,4]]], [[[1,2],[3,4]],[[1,2],[3,4]]]],[[[[1,2],[3,4]],[[1,2],[3,4]]], [[[1,2],[3,4]],[[1,2],[3,4]]]]], [[[[[1,2],[3,4]],[[1,2],[3,4]]], [[[1,0],[1,1]],[[1,0],[1,1]]]],[[[[1,0],[1,1]],[[1,0],[1,1]]], [[[1,0],[1,1]],[[1,0],[1,1]]]]]]
arr = np.array(my_list)
perm_testing = [0,1,2,3,4,5]
testing = arr.transpose(perm_testing)
print(testing)
Edit
import numpy as np
my_list = [[1,2],[3,4]]
arr = np.array(my_list)
perm_testing = [1,0]
testing = arr.transpose(perm_testing)
print(testing)
[[1 3]
[2 4]]
Here's an attempt to visually explain for a 3d-array. I hope it'll help you better understand what's happening:
a=np.arange(24).reshape(2,4,3)
# array([[[ 0, 1, 2],
# [ 3, 4, 5],
# [ 6, 7, 8],
# [ 9, 10, 11]],
#
# [[12, 13, 14],
# [15, 16, 17],
# [18, 19, 20],
# [21, 22, 23]]])
And a visual 3d representation of a (axis 0 corresponds to the first bracket level and to the first size in the shape, and so on for axis 1 and 2):
a.transpose(1,0,2) # swapping axis 0 and 1
# array([[[ 0, 1, 2],
# [12, 13, 14]],
#
# [[ 3, 4, 5],
# [15, 16, 17]],
#
# [[ 6, 7, 8],
# [18, 19, 20]],
#
# [[ 9, 10, 11],
# [21, 22, 23]]])
Visual 3d representation of the new array (sorry, my drawing skills are quite limited):
I am trying to access a pytorch tensor by a matrix of indices and I recently found this bit of code that I cannot find the reason why it is not working.
The code below is split into two parts. The first half proves to work, whilst the second trips an error. I fail to see the reason why. Could someone shed some light on this?
import torch
import numpy as np
a = torch.rand(32, 16)
m, n = a.shape
xx, yy = np.meshgrid(np.arange(m), np.arange(m))
result = a[xx] # WORKS for a torch.tensor of size M >= 32. It doesn't work otherwise.
a = torch.rand(16, 16)
m, n = a.shape
xx, yy = np.meshgrid(np.arange(m), np.arange(m))
result = a[xx] # IndexError: too many indices for tensor of dimension 2
and if I change a = np.random.rand(16, 16) it does work as well.
To whoever comes looking for an answer: it looks like its a bug in pyTorch.
Indexing using numpy arrays is not well defined, and it works only if tensors are indexed using tensors. So, in my example code, this works flawlessly:
a = torch.rand(M, N)
m, n = a.shape
xx, yy = torch.meshgrid(torch.arange(m), torch.arange(m), indexing='xy')
result = a[xx] # WORKS
I made a gist to check it, and it's available here
First, let me give you a quick insight into the idea of indexing a tensor with a numpy array and another tensor.
Example: this is our target tensor to be indexed
numpy_indices = torch.tensor([[0, 1, 2, 7],
[0, 1, 2, 3]]) # numpy array
tensor_indices = torch.tensor([[0, 1, 2, 7],
[0, 1, 2, 3]]) # 2D tensor
t = torch.tensor([[1, 2, 3, 4], # targeted tensor
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16],
[17, 18, 19, 20],
[21, 22, 23, 24],
[25, 26, 27, 28],
[29, 30, 31, 32]])
numpy_result = t[numpy_indices]
tensor_result = t[tensor_indices]
Indexing using a 2D numpy array: the index is read like pairs (x,y) tensor[row,column] e.g. t[0,0], t[1,1], t[2,2], and t[7,3].
print(numpy_result) # tensor([ 1, 6, 11, 32])
Indexing using a 2D tensor: walks through the index tensor in a row-wise manner and each value is an index of a row in the targeted tensor.
e.g. [ [t[0],t[1],t[2],[7]] , [[0],[1],[2],[3]] ] see the example below, the new shape of tensor_result after indexing is (tensor_indices.shape[0],tensor_indices.shape[1],t.shape[1])=(2,4,4).
print(tensor_result) # tensor([[[ 1, 2, 3, 4],
# [ 5, 6, 7, 8],
# [ 9, 10, 11, 12],
# [29, 30, 31, 32]],
# [[ 1, 2, 3, 4],
# [ 5, 6, 7, 8],
# [ 9, 10, 11, 12],
# [ 13, 14, 15, 16]]])
If you try to add a third row in numpy_indices, you will get the same error you have because the index will be represented by 3D e.g., (0,0,0)...(7,3,3).
indices = np.array([[0, 1, 2, 7],
[0, 1, 2, 3],
[0, 1, 2, 3]])
print(numpy_result) # IndexError: too many indices for tensor of dimension 2
However, this is not the case with indexing by tensor and the shape will be bigger (3,4,4).
Finally, as you see the outputs of the two types of indexing are completely different. To solve your problem, you can use
xx = torch.tensor(xx).long() # convert a numpy array to a tensor
What happens in the case of advanced indexing (rows of numpy_indices > 3 ) as your situation is still ambiguous and unsolved and you can check 1 , 2, 3.
I have a 3D numpy array of shape (i, j, k). I have an array of length i which contains indices in k. I would like to index the array to get a shape (i, j).
Here is an example of what I am trying to achieve:
import numpy as np
arr = np.arange(2 * 3 * 4).reshape(2, 3, 4)
# array([[[ 0, 1, 2, 3],
# [ 4, 5, 6, 7],
# [ 8, 9, 10, 11]],
#
# [[12, 13, 14, 15],
# [16, 17, 18, 19],
# [20, 21, 22, 23]]])
indices = np.array([1, 3])
# I want to mask `arr` using `indices`
# Desired output is equivalent to
# np.stack((arr[0, :, 1], arr[1, :, 3]))
# array([[ 1, 5, 9],
# [15, 19, 23]])
I tried reshaping the indices array to be able to broadcast with arr but this raises an IndexError.
arr[indices[np.newaxis, np.newaxis, :]]
# IndexError: index 3 is out of bounds for axis 0 with size 2
I also tried creating a 3D mask and applying it to arr. This seems closer to the correct answer to me but I still end up with an IndexError.
mask = np.stack((np.arange(arr.shape[0]), indices), axis=1)
arr[mask.reshape(2, 1, 2)]
# IndexError: index 3 is out of bounds for axis 0 with size 2
From what I understand in your example, you can simply pass indices as your second dimension slice, and a range of length corresponding to your indices for the zeroth dimension slice, like this:
import numpy as np
arr = np.arange(2 * 3 * 4).reshape(2, 3, 4)
indices = np.array([1, 3])
print(arr[range(len(indices)), :, indices])
# array([[ 1, 5, 9],
# [15, 19, 23]])
This works:
sub = arr[[0,1], :, [1,3]]
Output:
>>> sub
array([[ 1, 5, 9],
[15, 19, 23]])
A more dynamic version by #Psidom:
>>> sub = arr[np.arange(len(arr)), :, [1,3]]
array([[ 1, 5, 9],
[15, 19, 23]])
I have a problem, which seems to be easy but it is causing me a lot of headache.
The problem is that I'm programming in python (I'm relative new to it) and I'm looking for an aquivalent of the function max (min) of a matrix in matlab but using numpy.
What I want to do is to get the minimum value and its index in a matrix
Just to keep it as easiest as possible with an example, let's say this is the matrix:
arr2D = np.array([[11, 12, 13, 34],
[14, 15, 16, 3],
[17, 15, 11, 1],
[7, 5, 11, 4],
[1, 12, 4, 4],
[12, 14, 15,-3]])
in matlab I would do:
[local_max, index] = min(arr2D)
and I would get the min value and its index for every column in the matrix.
Trying to repeat the same in python (after looking here and here) with the following code:
print(np.where(arr2D == np.amin(arr2D, axis = 0))) # axis 0 is for columns
I get the following output:
(array([3, 4, 4, 5]), array([1, 0, 2, 3]))
which is not really what I want to get!
The expected output should be:
[1, 4] # Meaning the minimum value is 1 and it is in row 4 for the first column
[5, 3] # Meaning the minimum value is 5 and it is in row 3 for the second column
[4, 4] # Meaning the minimum value is 4 and it is in row 4 for the third column
[-3, 5] # Meaning the minimum value is -3 and it is in row 5 for the last column
I cannot use the output I get by:
print(np.where(arr2D == np.amin(arr2D, axis = 0)))
Or I don't understand the output or that's not the right way to get the aquivalent function max (min) of matlab.
Could you help me?
UPDATE:
I forgot to say that the matrix is float and not integer. I used integer just for the example
np.amin or np.min returns the min values along an axis
np.amin(arr2D, axis=0)
Out:
array([ 1, 5, 4, -3])
np.argmin returns the indices
np.argmin(arr2D, axis=0)
Out:
array([4, 3, 4, 5])
To get the desired output you can use np.vstack and transpose the array
np.vstack([np.amin(arr2D, axis=0), np.argmin(arr2D, axis=0)]).T
Out:
array([[ 1, 4],
[ 5, 3],
[ 4, 4],
[-3, 5]])
Use this code (you can simply make a function out of it):
import numpy as np
arr2D = np.array([[11, 12, 13, 34],
[14, 15, 16, 3],
[17, 15, 11, 1],
[7, 5, 11, 4],
[1, 12, 4, 4],
[12, 14, 15,-3]])
flat = arr2D.flatten()
arrayIndex = flat.tolist().index(min(flat))
// results
rowIndex = int(minIndex/arr2D.shape[0])
columnIndex = minIndex % arr2D.shape[1]
The situation
I have 2D array representing dual-channel audio. I want to create a function that returns slices of this array at arbitrary locations (e.g. speech only parts). I already know how to do it when I explicitly write the values into np.r_:
Sample data
arr = np.arange(0,24).reshape((2, -1))
# array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
# [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]])
Input
A x length array of width 2. E.g.
selector = np.array([[0, 2], [6, 9]])
# array([[0, 2],
# [6, 9]])
Desired output
# create an indexed arrays
selection_indices = np.r_[0:2, 6:9]
# array([0, 1, 6, 7, 8])
# use indices to select 2D
arr[:, selection_indices]
# array([[ 0, 1, 6, 7, 8],
# [12, 13, 18, 19, 20]])
Goal
A function that takes a X length array of width 2 (shape: X, 2), representing the start and end of a slice, and use that to return a selection of an array. Effectively the np.r_[0:2, 6:9], but then from an argument.
arr = np.arange(0,24).reshape((2, -1))
def slice_returner(arr, selector):
# something like this (broken); should be like: np.r_[0:2, 6:9]
selection_indices = np.r_[[row[0]:row[1]] for row in selector]
# return 2D slice
return arr[:, selection_indices]
selector = np.array([[0, 2], [6, 9]])
sliced_arr = slice_returner(arr, selector)
How do I turn the input into selection slices? Preferably with minimal array creation / copying.
Think boolean-indexing could be one efficient way. Hence, we can create a mask and then index cols and get our output -
# Generate mask for cols
mask = np.zeros(arr.shape[1],dtype=bool)
for (i,j) in selector:
mask[i:j] = True
# Boolean index into cols for final o/p
out = arr[:,mask]
The memory-overhead is just the mask, which being a boolean array should be minimal and the final output, which is required anyway.
Vectorized mask creation
If there are many entries in selector, there's a broadcasting-based vectorized way to create the mask for cols, like so -
r = np.arange(arr.shape[1])
mask = ((selector[:,0,None]<=r) & (selector[:,1,None]>r)).any(0)
You can just create an indexing array from individual aranges
slices = [[0, 2], [6, 9]]
np.concatenate([np.arange(*i) for i in slices])
# array([0, 1, 6, 7, 8])
and use it to extract the data
arr[:, np.concatenate([np.arange(*i) for i in slices])]
# array([[ 0, 1, 6, 7, 8],
# [12, 13, 18, 19, 20]])