Related
I am new to numpy and python and I am trying to understand the usage of transpose function of numpy. The code below works fine but I am still not be able to understand the effect of transpose function and also the use of the arguments inside it. It would be great help if someone can explain the usage and effect of transpose function in below code.
import numpy as np
my_list = [[[[[[1,2],[3,4]],[[1,2],[3,4]]], [[[1,2],[3,4]],[[1,2],[3,4]]]],[[[[1,2],[3,4]],[[1,2],[3,4]]], [[[1,2],[3,4]],[[1,2],[3,4]]]]], [[[[[1,2],[3,4]],[[1,2],[3,4]]], [[[1,0],[1,1]],[[1,0],[1,1]]]],[[[[1,0],[1,1]],[[1,0],[1,1]]], [[[1,0],[1,1]],[[1,0],[1,1]]]]]]
arr = np.array(my_list)
perm_testing = [0,1,2,3,4,5]
testing = arr.transpose(perm_testing)
print(testing)
Edit
import numpy as np
my_list = [[1,2],[3,4]]
arr = np.array(my_list)
perm_testing = [1,0]
testing = arr.transpose(perm_testing)
print(testing)
[[1 3]
[2 4]]
Here's an attempt to visually explain for a 3d-array. I hope it'll help you better understand what's happening:
a=np.arange(24).reshape(2,4,3)
# array([[[ 0, 1, 2],
# [ 3, 4, 5],
# [ 6, 7, 8],
# [ 9, 10, 11]],
#
# [[12, 13, 14],
# [15, 16, 17],
# [18, 19, 20],
# [21, 22, 23]]])
And a visual 3d representation of a (axis 0 corresponds to the first bracket level and to the first size in the shape, and so on for axis 1 and 2):
a.transpose(1,0,2) # swapping axis 0 and 1
# array([[[ 0, 1, 2],
# [12, 13, 14]],
#
# [[ 3, 4, 5],
# [15, 16, 17]],
#
# [[ 6, 7, 8],
# [18, 19, 20]],
#
# [[ 9, 10, 11],
# [21, 22, 23]]])
Visual 3d representation of the new array (sorry, my drawing skills are quite limited):
I am trying to access a pytorch tensor by a matrix of indices and I recently found this bit of code that I cannot find the reason why it is not working.
The code below is split into two parts. The first half proves to work, whilst the second trips an error. I fail to see the reason why. Could someone shed some light on this?
import torch
import numpy as np
a = torch.rand(32, 16)
m, n = a.shape
xx, yy = np.meshgrid(np.arange(m), np.arange(m))
result = a[xx] # WORKS for a torch.tensor of size M >= 32. It doesn't work otherwise.
a = torch.rand(16, 16)
m, n = a.shape
xx, yy = np.meshgrid(np.arange(m), np.arange(m))
result = a[xx] # IndexError: too many indices for tensor of dimension 2
and if I change a = np.random.rand(16, 16) it does work as well.
To whoever comes looking for an answer: it looks like its a bug in pyTorch.
Indexing using numpy arrays is not well defined, and it works only if tensors are indexed using tensors. So, in my example code, this works flawlessly:
a = torch.rand(M, N)
m, n = a.shape
xx, yy = torch.meshgrid(torch.arange(m), torch.arange(m), indexing='xy')
result = a[xx] # WORKS
I made a gist to check it, and it's available here
First, let me give you a quick insight into the idea of indexing a tensor with a numpy array and another tensor.
Example: this is our target tensor to be indexed
numpy_indices = torch.tensor([[0, 1, 2, 7],
[0, 1, 2, 3]]) # numpy array
tensor_indices = torch.tensor([[0, 1, 2, 7],
[0, 1, 2, 3]]) # 2D tensor
t = torch.tensor([[1, 2, 3, 4], # targeted tensor
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16],
[17, 18, 19, 20],
[21, 22, 23, 24],
[25, 26, 27, 28],
[29, 30, 31, 32]])
numpy_result = t[numpy_indices]
tensor_result = t[tensor_indices]
Indexing using a 2D numpy array: the index is read like pairs (x,y) tensor[row,column] e.g. t[0,0], t[1,1], t[2,2], and t[7,3].
print(numpy_result) # tensor([ 1, 6, 11, 32])
Indexing using a 2D tensor: walks through the index tensor in a row-wise manner and each value is an index of a row in the targeted tensor.
e.g. [ [t[0],t[1],t[2],[7]] , [[0],[1],[2],[3]] ] see the example below, the new shape of tensor_result after indexing is (tensor_indices.shape[0],tensor_indices.shape[1],t.shape[1])=(2,4,4).
print(tensor_result) # tensor([[[ 1, 2, 3, 4],
# [ 5, 6, 7, 8],
# [ 9, 10, 11, 12],
# [29, 30, 31, 32]],
# [[ 1, 2, 3, 4],
# [ 5, 6, 7, 8],
# [ 9, 10, 11, 12],
# [ 13, 14, 15, 16]]])
If you try to add a third row in numpy_indices, you will get the same error you have because the index will be represented by 3D e.g., (0,0,0)...(7,3,3).
indices = np.array([[0, 1, 2, 7],
[0, 1, 2, 3],
[0, 1, 2, 3]])
print(numpy_result) # IndexError: too many indices for tensor of dimension 2
However, this is not the case with indexing by tensor and the shape will be bigger (3,4,4).
Finally, as you see the outputs of the two types of indexing are completely different. To solve your problem, you can use
xx = torch.tensor(xx).long() # convert a numpy array to a tensor
What happens in the case of advanced indexing (rows of numpy_indices > 3 ) as your situation is still ambiguous and unsolved and you can check 1 , 2, 3.
I have a 3D numpy array of shape (i, j, k). I have an array of length i which contains indices in k. I would like to index the array to get a shape (i, j).
Here is an example of what I am trying to achieve:
import numpy as np
arr = np.arange(2 * 3 * 4).reshape(2, 3, 4)
# array([[[ 0, 1, 2, 3],
# [ 4, 5, 6, 7],
# [ 8, 9, 10, 11]],
#
# [[12, 13, 14, 15],
# [16, 17, 18, 19],
# [20, 21, 22, 23]]])
indices = np.array([1, 3])
# I want to mask `arr` using `indices`
# Desired output is equivalent to
# np.stack((arr[0, :, 1], arr[1, :, 3]))
# array([[ 1, 5, 9],
# [15, 19, 23]])
I tried reshaping the indices array to be able to broadcast with arr but this raises an IndexError.
arr[indices[np.newaxis, np.newaxis, :]]
# IndexError: index 3 is out of bounds for axis 0 with size 2
I also tried creating a 3D mask and applying it to arr. This seems closer to the correct answer to me but I still end up with an IndexError.
mask = np.stack((np.arange(arr.shape[0]), indices), axis=1)
arr[mask.reshape(2, 1, 2)]
# IndexError: index 3 is out of bounds for axis 0 with size 2
From what I understand in your example, you can simply pass indices as your second dimension slice, and a range of length corresponding to your indices for the zeroth dimension slice, like this:
import numpy as np
arr = np.arange(2 * 3 * 4).reshape(2, 3, 4)
indices = np.array([1, 3])
print(arr[range(len(indices)), :, indices])
# array([[ 1, 5, 9],
# [15, 19, 23]])
This works:
sub = arr[[0,1], :, [1,3]]
Output:
>>> sub
array([[ 1, 5, 9],
[15, 19, 23]])
A more dynamic version by #Psidom:
>>> sub = arr[np.arange(len(arr)), :, [1,3]]
array([[ 1, 5, 9],
[15, 19, 23]])
I tried to use reshape
import numpy as np
d = np.arange(30).reshape(1,3)
It is not working cannot reshape array of size 30 into shape (1,3)
but when I tried to use
d = np.arange(30).reshape(-1,3) # This works
Why do we have to use -1?.
It's really confusing and I'm can't seem to figure out how reshape works. I would really appreciate if someone can help me figure out how this works. I tried docs and other posts in SO but it wasn't much helpful.
I am new to ML and Python.
A reshape means that you order your elements of the array, according to other dimensions. For example arange(27) will produce a vector containing 27 elements. But with .reshape(9, 3) you specify here that you want to transform it into a two dimensional array, where the first dimension contains 9 elements, and the second three elements. So the result will look like:
>>> np.arange(27).reshape(9, 3)
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23],
[24, 25, 26]])
But we can also make it a 3×3×3 array:
>>> np.arange(27).reshape(3, 3, 3)
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
-1 is used as a value that will numpy derive the dimension.
So if you have an array containing 30 elements, and you reshape these to m×3, then m is 10. -1 is thus not the real value, it is used for programmer convenience if for example you do not know the number of elements, but know that it is divisable by three.
The following two are (under the assumption that m contains 30 elements equivalent:
m.reshape(10, 3)
m.reshape(-1, 3)
Note that you can specify at most one -1, since otherwise there are multiple possibilities, and it becomes also harder to find a valid configuration.
I am currently monkey-patching a Scikit-Learn function and one of the lines requires a NumPy array with 2 dimensions. However, the data I am working with is a NumPy array with 3 dimensions, which raises the error "too many values to unpack."
I am calling the K-Means function fit to cluster the data. My problem boils down to this following line of code, assuming X is the ndarray that I pass in:
n_samples, n_features = X.shape
X is an array with 3 dimensions, like the following:
X = np.array([[[1, 2, 3],
[4, 5, 6]],
[[7, 8, 9],
[10, 11, 12]],
[[13, 14, 15],
[16, 17, 18]]])
The data represents a group of time series of data points that have 6 dimensions. For example, the first element, [[1, 2, 3], [4, 5, 6]] would represent a time series with samples from 2 time periods, each sample with 3 dimensions.
And I have monkey-patched the k_means_ code to allow me to perform clustering on an ndarray of ndarrays. My goal is to perform k-means on 2D arrays.
Is it possible to set the shape of the 3D ndarray to 2 elements? For example, I tried converting the 3D array to a 2D array of objects but it ends up getting converted back to a 3D array.
np.array([[x.astype(object) for x in c] for c in combined])
Likewise, the following code is also converted back to a 3D array.
np.array([[np.array(x) for x in c] for c in combined])
The list comprehension [[x.astype(object) for x in c] for c in combined] looks like it creates the correct array, but because it is of type list, it no longer works in the function.
I am looking for some way to "convert" a 3D NumPy array into 2 dimensions. Any help would be greatly appreciated!
Note: I am not looking for a way to reshape the array. I need to keep all the dimensions but change the shape to ignore one of the dimensions.
To make an array of arrays, we have to play some tricks, because np.array tries to make an as-high dimensional array as it can. If the subarrays vary in size that is ok, but if they are all the same we have to fight that.
Here's one way:
start with a 3d array:
In [812]: arr = np.arange(24).reshape(2,3,4)
and an empty object array of the right size (but flattened)
In [813]: A = np.empty((6,),object)
copy values (again with flattening), and reshape to the target shape
In [814]: A[:]=list(arr.reshape(-1,4))
In [815]: A=A.reshape(2,3)
In [816]: A
Out[816]:
array([[array([0, 1, 2, 3]), array([4, 5, 6, 7]), array([ 8, 9, 10, 11])],
[array([12, 13, 14, 15]), array([16, 17, 18, 19]),
array([20, 21, 22, 23])]], dtype=object)
So now we have a (2,3) array, who's shape can be unpacked.
I tried to start with np.empty((2,3),object), but couldn't get the A[:,:]=... assignment to work. For this object reshaping to work we have to split arr into a list of arrays. An object array is, like a list, an array of pointers.
But will the scikit functions accept such an array? (after passing the shape hurdle). I suspect the object reshaping is a short sighted solution.
In [824]: [[x.astype(object) for x in c] for c in arr]
Out[824]:
[[array([0, 1, 2, 3], dtype=object),
array([4, 5, 6, 7], dtype=object),
array([8, 9, 10, 11], dtype=object)],
[array([12, 13, 14, 15], dtype=object),
array([16, 17, 18, 19], dtype=object),
array([20, 21, 22, 23], dtype=object)]]
In [825]: _[0][0].shape
Out[825]: (4,)
This creates a nested list of lists, with the inner elements being (4,) object array. Wrap that in np.array and it recreates a 3d array with dtype object.
reshaping, which for some unknown reason, you don't want to do preserves the numeric dtype
In [828]: arr.reshape(2,-1)
Out[828]:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]])
In [829]: arr.reshape(-1,4)
Out[829]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]])