I am new to numpy and python and I am trying to understand the usage of transpose function of numpy. The code below works fine but I am still not be able to understand the effect of transpose function and also the use of the arguments inside it. It would be great help if someone can explain the usage and effect of transpose function in below code.
import numpy as np
my_list = [[[[[[1,2],[3,4]],[[1,2],[3,4]]], [[[1,2],[3,4]],[[1,2],[3,4]]]],[[[[1,2],[3,4]],[[1,2],[3,4]]], [[[1,2],[3,4]],[[1,2],[3,4]]]]], [[[[[1,2],[3,4]],[[1,2],[3,4]]], [[[1,0],[1,1]],[[1,0],[1,1]]]],[[[[1,0],[1,1]],[[1,0],[1,1]]], [[[1,0],[1,1]],[[1,0],[1,1]]]]]]
arr = np.array(my_list)
perm_testing = [0,1,2,3,4,5]
testing = arr.transpose(perm_testing)
print(testing)
Edit
import numpy as np
my_list = [[1,2],[3,4]]
arr = np.array(my_list)
perm_testing = [1,0]
testing = arr.transpose(perm_testing)
print(testing)
[[1 3]
[2 4]]
Here's an attempt to visually explain for a 3d-array. I hope it'll help you better understand what's happening:
a=np.arange(24).reshape(2,4,3)
# array([[[ 0, 1, 2],
# [ 3, 4, 5],
# [ 6, 7, 8],
# [ 9, 10, 11]],
#
# [[12, 13, 14],
# [15, 16, 17],
# [18, 19, 20],
# [21, 22, 23]]])
And a visual 3d representation of a (axis 0 corresponds to the first bracket level and to the first size in the shape, and so on for axis 1 and 2):
a.transpose(1,0,2) # swapping axis 0 and 1
# array([[[ 0, 1, 2],
# [12, 13, 14]],
#
# [[ 3, 4, 5],
# [15, 16, 17]],
#
# [[ 6, 7, 8],
# [18, 19, 20]],
#
# [[ 9, 10, 11],
# [21, 22, 23]]])
Visual 3d representation of the new array (sorry, my drawing skills are quite limited):
Related
I am trying to access a pytorch tensor by a matrix of indices and I recently found this bit of code that I cannot find the reason why it is not working.
The code below is split into two parts. The first half proves to work, whilst the second trips an error. I fail to see the reason why. Could someone shed some light on this?
import torch
import numpy as np
a = torch.rand(32, 16)
m, n = a.shape
xx, yy = np.meshgrid(np.arange(m), np.arange(m))
result = a[xx] # WORKS for a torch.tensor of size M >= 32. It doesn't work otherwise.
a = torch.rand(16, 16)
m, n = a.shape
xx, yy = np.meshgrid(np.arange(m), np.arange(m))
result = a[xx] # IndexError: too many indices for tensor of dimension 2
and if I change a = np.random.rand(16, 16) it does work as well.
To whoever comes looking for an answer: it looks like its a bug in pyTorch.
Indexing using numpy arrays is not well defined, and it works only if tensors are indexed using tensors. So, in my example code, this works flawlessly:
a = torch.rand(M, N)
m, n = a.shape
xx, yy = torch.meshgrid(torch.arange(m), torch.arange(m), indexing='xy')
result = a[xx] # WORKS
I made a gist to check it, and it's available here
First, let me give you a quick insight into the idea of indexing a tensor with a numpy array and another tensor.
Example: this is our target tensor to be indexed
numpy_indices = torch.tensor([[0, 1, 2, 7],
[0, 1, 2, 3]]) # numpy array
tensor_indices = torch.tensor([[0, 1, 2, 7],
[0, 1, 2, 3]]) # 2D tensor
t = torch.tensor([[1, 2, 3, 4], # targeted tensor
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16],
[17, 18, 19, 20],
[21, 22, 23, 24],
[25, 26, 27, 28],
[29, 30, 31, 32]])
numpy_result = t[numpy_indices]
tensor_result = t[tensor_indices]
Indexing using a 2D numpy array: the index is read like pairs (x,y) tensor[row,column] e.g. t[0,0], t[1,1], t[2,2], and t[7,3].
print(numpy_result) # tensor([ 1, 6, 11, 32])
Indexing using a 2D tensor: walks through the index tensor in a row-wise manner and each value is an index of a row in the targeted tensor.
e.g. [ [t[0],t[1],t[2],[7]] , [[0],[1],[2],[3]] ] see the example below, the new shape of tensor_result after indexing is (tensor_indices.shape[0],tensor_indices.shape[1],t.shape[1])=(2,4,4).
print(tensor_result) # tensor([[[ 1, 2, 3, 4],
# [ 5, 6, 7, 8],
# [ 9, 10, 11, 12],
# [29, 30, 31, 32]],
# [[ 1, 2, 3, 4],
# [ 5, 6, 7, 8],
# [ 9, 10, 11, 12],
# [ 13, 14, 15, 16]]])
If you try to add a third row in numpy_indices, you will get the same error you have because the index will be represented by 3D e.g., (0,0,0)...(7,3,3).
indices = np.array([[0, 1, 2, 7],
[0, 1, 2, 3],
[0, 1, 2, 3]])
print(numpy_result) # IndexError: too many indices for tensor of dimension 2
However, this is not the case with indexing by tensor and the shape will be bigger (3,4,4).
Finally, as you see the outputs of the two types of indexing are completely different. To solve your problem, you can use
xx = torch.tensor(xx).long() # convert a numpy array to a tensor
What happens in the case of advanced indexing (rows of numpy_indices > 3 ) as your situation is still ambiguous and unsolved and you can check 1 , 2, 3.
I have following array:
b=np.zeros((5,5)).astype('int32')
I wish to populate each element of above array with a list using below two arrays:
x=np.linspace(11, 15, 5)
`y=np.linspace(6, 10, 5)`
The output i am looking at:
`array([[11,6], [11,7], [11,8], [11,9], [11,10]],
[[12,6], [12,7], [12,8], [12,9], [12,10]],
[[13,6], [13,7], [13,8], [13,9], [13,10]],
[[14,6], [14,7], [14,8], [14,9], [14,10]],
[[15,6], [15,7], [15,8], [15,9], [15,10]])`
Like #DocDriven said, you'll have to adjust the shape of the b array first, to (5, 5, 2).
After that, note that you can set a whole row of y values by doing b[row,:,1] = y and a whole column of x values by doing b[:,col,0] = x.
Numpy also broadcasts shapes, which means you can use a 1d array to fill a 2d array; b[:,:,1] = y will fill all the y values in a single operation, while b[:,:,0] = x.reshape((5, 1)) will do the same for the x values.
In short, you can get what you want by doing just:
b = np.zeros((5, 5, 2)).astype('int32')
b[:,:,1] = y
b[:,:,0] = x.reshape((5, 1))
Another way is to use np.meshgrid():
b = np.array(np.meshgrid(x, y)).T.astype('int32')
I slightly adjusted your original numpy array because you cannot replace a single integer with a sequence.
import numpy as np
b = np.zeros((5,5,2)).astype('int32')
x = np.linspace(11, 15, 5).astype('int32')
y = np.linspace(6, 10, 5).astype('int32')
idx_x = 0
idx_y = 0
for row in b:
for _ in row:
b[idx_x, idx_y] = [x[idx_x], y[idx_y]]
idx_y += 1
idx_y = 0
idx_x += 1
print(b.tolist())
Output:
[[[11, 6], [11, 7], [11, 8], [11, 9], [11, 10]],
[[12, 6], [12, 7], [12, 8], [12, 9], [12, 10]],
[[13, 6], [13, 7], [13, 8], [13, 9], [13, 10]],
[[14, 6], [14, 7], [14, 8], [14, 9], [14, 10]],
[[15, 6], [15, 7], [15, 8], [15, 9], [15, 10]]]
If you want to keep it as a numpy array, do not cast it via tolist().
I tried to use reshape
import numpy as np
d = np.arange(30).reshape(1,3)
It is not working cannot reshape array of size 30 into shape (1,3)
but when I tried to use
d = np.arange(30).reshape(-1,3) # This works
Why do we have to use -1?.
It's really confusing and I'm can't seem to figure out how reshape works. I would really appreciate if someone can help me figure out how this works. I tried docs and other posts in SO but it wasn't much helpful.
I am new to ML and Python.
A reshape means that you order your elements of the array, according to other dimensions. For example arange(27) will produce a vector containing 27 elements. But with .reshape(9, 3) you specify here that you want to transform it into a two dimensional array, where the first dimension contains 9 elements, and the second three elements. So the result will look like:
>>> np.arange(27).reshape(9, 3)
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23],
[24, 25, 26]])
But we can also make it a 3×3×3 array:
>>> np.arange(27).reshape(3, 3, 3)
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
-1 is used as a value that will numpy derive the dimension.
So if you have an array containing 30 elements, and you reshape these to m×3, then m is 10. -1 is thus not the real value, it is used for programmer convenience if for example you do not know the number of elements, but know that it is divisable by three.
The following two are (under the assumption that m contains 30 elements equivalent:
m.reshape(10, 3)
m.reshape(-1, 3)
Note that you can specify at most one -1, since otherwise there are multiple possibilities, and it becomes also harder to find a valid configuration.
I am trying to use the scatter_nd function in TensorFlow to reorder elements within rows of a Matrix. For example, suppose I have the code:
indices = tf.constant([[1],[0]])
updates = tf.constant([ [5, 6, 7, 8],
[1, 2, 3, 4] ])
shape = tf.constant([2, 4])
scatter1 = tf.scatter_nd(indices, updates, shape)
$ print(scatter1) = [[1,2,3,4]
[5,6,7,8]]
This reorders the rows of the updates matrix.
Instead of only being able to reorder the rows, I'd like to reorder the individual elements within each row as well. If I just have a vector (Tensor of rank 1), then this example works:
indices = tf.constant([[1],[0],[2],[3]])
updates = tf.constant([5, 6, 7, 8])
shape = tf.constant([4])
scatter2 = tf.scatter_nd(indices, updates, shape)
$ print(scatter2) = [6,5,7,8]
What I really care about is to be able to swap elements within each row in scatter1, as I had done in scatter2, but do it for each row of scatter1. I've tried various combinations of indices but keep getting errors that the sizes are inconsistent thrown by the scatter_nd function.
The following swaps the elements of each row of each row using scatter_nd
indices = tf.constant([[[0, 1], [0, 0], [0, 2], [0, 3]],
[[1, 1], [1, 0], [1, 2], [1, 3]]])
updates = tf.constant([ [5, 6, 7, 8],
[1, 2, 3, 4] ])
shape = tf.constant([2, 4])
scatter1 = tf.scatter_nd(indices, updates, shape)
with tf.Session() as sess:
print(sess.run(scatter1))
Giving an output of:
[[6 5 7 8]
[2 1 3 4]]
The locations of the coordinate in indices define where the values are being taken from in updates and the actual cordinates define where the values will be placed in scatter1.
This answer is a few months late but hopefully still helpful.
Suppose you want to swap elements in the second dimension either keeping the first dimension order or not.
import tensorflow as tf
sess = tf.InteractiveSession()
def prepare_fd(fd_indices, sd_dims):
fd_indices = tf.expand_dims(fd_indices, 1)
fd_indices = tf.tile(fd_indices, [1, sd_dims])
return fd_indices
# define the updates
updates = tf.constant([[11, 12, 13, 14],
[21, 22, 23, 24],
[31, 32, 33, 34]])
sd_dims = tf.shape(updates)[1]
sd_indices = tf.constant([[1, 0, 2, 3], [0, 2, 1, 3], [0, 1, 3, 2]])
fd_indices_range = tf.range(0, limit=tf.shape(updates)[0])
fd_indices_custom = tf.constant([2, 0, 1])
# define the indices
indices1 = tf.stack((prepare_fd(fd_indices_range, sd_dims), sd_indices), axis=2)
indices2 = tf.stack((prepare_fd(fd_indices_custom, sd_dims), sd_indices), axis=2)
# define the shape
shape = tf.shape(updates)
scatter1 = tf.scatter_nd(indices1, updates, shape)
scatter2 = tf.scatter_nd(indices2, updates, shape)
print(scatter1.eval())
# array([[12, 11, 13, 14],
# [21, 23, 22, 24],
# [31, 32, 34, 33]], dtype=int32)
print(scatter2.eval())
# array([[21, 23, 22, 24],
# [31, 32, 34, 33],
# [12, 11, 13, 14]], dtype=int32)
May this example help.
I am trying to copy a section of an input 2d array "img" and mirroring that section and copying it into the 2d array "out"
The following code does what I need
a = numpy.zeros(shape=(pad, pad))
a[:,:]=img[0:pad,0:pad]
out[0:pad,0:pad]=a[::-1,::-1]
But simply doing the following does not
out[0:pad,0:pad]=img[0:pad:-1,0:pad:-1]
and instead returnsValueError: could not broadcast input array from shape (0,0) into shape (2,2) for pad=2 and I am not sure why.
img[0:pad:-1,0:pad:-1]
should be
img[pad-1::-1, pad-1::-1]
since you want the index to start at pad-1 and step down to 0. See here for the complete rules governing NumPy basic slicing.
For example,
import numpy as np
img = np.arange(24).reshape(6,4)
# array([[ 0, 1, 2, 3],
# [ 4, 5, 6, 7],
# [ 8, 9, 10, 11],
# [12, 13, 14, 15],
# [16, 17, 18, 19],
# [20, 21, 22, 23]])
pad = 2
out = img[pad-1::-1, pad-1::-1]
print(out)
yields
[[5 4]
[1 0]]