On the link
http://scipy-lectures.github.io/advanced/advanced_numpy/
there is statement ,
there is no way to represent the array c given one stride and the block of memory for a. Therefore, the reshape operation needs to make a copy here.
How this copy works inside numpy ? What is time complexity of that ?
How numpy gets to know further array can't be represented using same array ?
If you have two dimensions of an array, with shapes sh0 and sh1 and strides st0 and st1, that you want to merge into a single dimension of shape sh0*sh1, the condition to be able to do so without a copy is that st0 == sh1*st1. Notice that the order the dimensions come up in the shape is relevant, and in a C order array with positive strides, dimension 0 precedes dimension 1.
There may be situations that require a more subtle analysis, about which stride and shape to multiply to compare to which stride, if you have Fortran order arrays, or negative strides, but the basic premise is still mostly the same.
Related
I have a tensor of shape "torch.Size([2, 2, 3])" and another tensor of shape "torch.Size([2, 1, 3])". I want a concatenated tensor of shape "torch.Size([2, 2, 6])".
For example :
a=torch.tensor([[[2,3,5],[12,13,15]],[[20,30,50],[120,130,150]]])
b=torch.tensor([[[99,99,99]],[[999,999,999]]])
I want the output as : [[[99,99,99,2,3,5],[99,99,99,12,13,15]],[[999,999,999,20,30,50],[999,999,999,120,130,150]]]
I have written a O(n2) solution using two for loops but,
This is taking a lot of time with millions of calculation, Does anyone help me in doing this efficiently ?? May be some matrix calculation trick for tensors ??
To exactly match the example you have provided:
c = torch.cat([b.repeat([1,a.shape[1]//b.shape[1],1]),a],2)
The reasoning behind this is that the concatenate operation in pytorch (and numpy and other libraries) will complain if the dimensions of the two tensors in the non-specified axes (in this case 0 and 1) do not match. Therefore, you have to repeat the tensor along the non-matching axis (the first axis, therefore the second element of the repeat list) in order to make the dimensions align. Note that the solution here will only work if the middle dimension of a is evenly divisible by the middle dimension of b.
In newer versions of pytorch, this can also be done using the torch.tile() function.
I am looking for a nice way to "clean up" the dimensions of two arrays which I would like to combine together using broadcasting.In particular I would like to broadcast a one dimensional array up to the shape of a multidimensional array and then add the two arrays. My understanding of the broadcasting rules tells me that this should work find if the last dimension of the multidimensional array matches that of the one dimensional array. For example, arrays with shapes (,3) and (10,3) would add fine
My problem is, given how the array I have is built the matching dimension happens to be the first dimension of the array so the broadcasting rules are not met. For reference my one d array has shape (,3) and the multi-dimensional array is (3,10,10,50).
I could correct this by reshaping the multi-dimensional array so that the compatible dimension is the last dimension but I'd like to avoid this as I find the logic of reshaping tricky to follow when the different dimensions have specific meaning.
I can also add empty dimensions to the one dimensional array as in the code below until the one dimensional array has as many dimensions as the high dimensional array as in the code snippet below.
>>> import numpy as np
>>> a = np.array([[1,2],
>>> [3,4],
>>> [5,6]])
>>> b = np.array([10,20,30])
>>> a+b[:,None]
array([[11, 12],
[23, 24],
[35, 36]])
This gives me my desired output however in my case my high dimensional array has 4 different axes so I would need to add in multiple empty dimensions which starts to feel inelegant. I can do something like
b = b[[slice(None)] + 3*[np.newaxis]]
and then proceed but that doesn't seem great. More generally one could imagine needing an arbitrary number of new axes on both sides of the original one dimension and writing a helper function to generalize the above logic. Is there a nicer/clearer way to achieve this?
I have a 4x4x256 tensor and a 128x256 matrix. I need to multiply each 256-d depth-wise vector of the tensor by the matrix, such that I get a 4x4x128 tensor as a result.
Working in Numpy it's not clear to me how to do this. In their current shape it doesn't look like any variant of np.dot exists to do this. Manipulating the shapes to take advantage of broadcasting rules doesn't seem to provide any help. np.tensordot and np.einsum may be useful but looking at the documentation is going right over my head.
Is there an efficient way to do this?
You can use np.einsum to do this operation. An example with random values:
a = np.arange(4096.).reshape(4,4,256)
b = np.arange(32768.).reshape(128,256)
c = np.einsum('ijk,lk->ijl',a,b)
print(c.shape)
Here, the subscripts argument is: ijk,lk->ijl
From your requirement, i=4, j=4, k=256, l=128
The comma separates the subscripts for two operands, and the subscripts state that the multiplication should be performed over the last subscript in each tensor (the subscript k which is common to both the tensors).
The tensor subscript after the -> states that the resultant tensor should have the shape (i,j,l). Now depending on the type of operation you are performing, you might have to retain this subscript or change this subscript to jil, but the rest of the subscripts remains the same.
I have two images , image 1 of dimension (32,43,3) and image2 of dimension (67,86,3) . How can i store this in a numpy array , Whenever i try to append the array
image=cv2.imread(image1,0)
image=cv2.resize(image,(32,43))
x_train=np.array(image.flatten())
x_train=x_train.reshape(-1,3,32,43)
X_train =np.append(X_train,x_train) #X_train is my array
image=cv2.imread(image2,0)
image=cv2.resize(image,(67,86))
x_train=np.array(image.flatten())
x_train=x_train.reshape(-1,3,67,86)
X_train =np.append(X_train,x_train)
Value Error: total size of new array must be unchanged.
i want the X_train in shape (-1,depth,height,width).So that i can feed it into my neural network. Is there any way to store images of different dimension in array and feed into neural network ?
Don't use np.append. If you must join arrays, start with np.concatenate. It'll force you to pay more attention to the compatibility of dimensions.
You can't join 2 arrays with shapes (32,43,3) (67,86,3) to make a larger array of some compatible shape. The only dimension they share is the last.
These reshapes don't make sense either: (-1,3,32,43), (-1,3,67,86).
It works, but it also messes up the 'image'. You aren't just adding a 4th dimension. It looks like you want to do some axis swapping or transpose as well. Practice with some small arrays so you can see what's happening, e.g. (2,4,3).
What final shape do you expect for Xtrain?
You can put these two images in a object dtype array, which is basically the same as the list [image1, image2]. But I doubt if your neuralnet can do anything practical with that.
If you reshaped the (32,43,3) array to (16,86,3) you could concatenate that with (67,86,3) on axis=0 to produce a (83,86,3) array. If you needed the 3 to be first, I'd use np.transpose(..., (2,0,1)).
Conversely reshape (67,86,3) to (2*67,43,3).
Passing the (32,43,3) to (32,86,3) is another option.
Joining them on a new 4th dimension, requires that the number of 'rows' match as well as the number of 'columns'.
My understanding is that 1-D arrays in numpy can be interpreted as either a column-oriented vector or a row-oriented vector. For instance, a 1-D array with shape (8,) can be viewed as a 2-D array of shape (1,8) or shape (8,1) depending on context.
The problem I'm having is that the functions I write to manipulate arrays tend to generalize well in the 2-D case to handle both vectors and matrices, but not so well in the 1-D case.
As such, my functions end up doing something like this:
if arr.ndim == 1:
# Do it this way
else:
# Do it that way
Or even this:
# Reshape the 1-D array to a 2-D array
if arr.ndim == 1:
arr = arr.reshape((1, arr.shape[0]))
# ... Do it the 2-D way ...
That is, I find I can generalize code to handle 2-D cases (r,1), (1,c), (r,c), but not the 1-D cases without branching or reshaping.
It gets even uglier when the function operates on multiple arrays as I would check and convert each argument.
So my question is: am I missing some better idiom? Is the pattern I've described above common to numpy code?
Also, as a related matter of API design principles, if the caller passes a 1-D array to some function that returns a new array, and the return value is also a vector, is it common practice to reshape a 2-D vector (r,1) or (1,c) back to a 1-D array or simply document that the function returns a 2-D array regardless?
Thanks
I think in general NumPy functions that require an array of shape (r,c) make no special allowance for 1-D arrays. Instead, they expect the user to either pass an array of shape (r,c) exactly, or for the user to pass a 1-D array that broadcasts up to shape (r,c).
If you pass such a function a 1-D array of shape (c,) it will broadcast to shape (1,c), since broadcasting adds new axes on the left. It can also broadcast to shape (r,c) for an arbitrary r (depending on what other array it is being combined with).
On the other hand, if you have a 1-D array, x, of shape (r,) and you need it to broadcast up to shape (r,c), then NumPy expects the user to pass an array of shape (r,1) since broadcasting will not add the new axes on the right for you.
To do that, the user must pass x[:,np.newaxis] instead of just x.
Regarding return values: I think it better to always return a 2-D array. If the user knows the output will be of shape (1,c), and wants a 1-D array, let her slice off the 1-D array x[0] herself.
By making the return value always the same shape, it will be easier to understand code that uses this function, since it is not always immediately apparent what the shape of the inputs are.
Also, broadcasting blurs the distinction between a 1-D array of shape (c,) and a 2-D array of shape (r,c). If your function returns a 1-D array when fed 1-D input, and a 2-D array when fed 2-D input, then your function makes the distinction strict instead of blurred. Stylistically, this reminds me of checking if isinstance(obj,type), which goes against the grain of duck-typing. Don't do it if you don't have to.
unutbu's explanation is good, but I disagree on the return dimension.
The function internal pattern depends on the type of function.
Reduce operations with an axis argument can often be written so that the number of dimensions doesn't matter.
Numpy has also an atleast_2d (and atleast_1d) function that is also commonly used if you need an explicit 2d array. In statistics, I sometimes use a function like atleast_2d_cols, that reshapes 1d (r,) to 2d (r,1) for code that expects 2d, or if the input array is 1d, then the interpretation and linear algebra requires a column vector. (reshaping is cheap so this is not a problem)
In a third case, I might have different code paths if the lower dimensional case can be done cheaper or simpler than the higher dimensional case. (example: if 2d requires several dot products.)
return dimension
I think not following the numpy convention with the return dimension can be very confusing to users for general functions. (topic specific functions can be different.)
For example, reduce operations loose one dimension.
For many other functions the output dimension matches the input dimension. I think a 1d input should have a 1d output and not an extra redundant dimension. Except for functions in linalg, I don't remember any functions that would return a redundant extra dimension. (The scalar versus 1-element array case is not always consistent.)
Stylistically this reminds me of an isinstance check:
Try without it if you allow for example for numpy matrices and masked arrays. You will get funny results that are not easy to debug. Although, for most numpy and scipy functions the user has to know whether the array type will work with them, since there are few isinstance checks and asarray might not always do the right thing.
As a user, I always know what kind of "array_like" I have, a list, tuple or which array subclass, especially when I use multiplication.
np.array(np.eye(3).tolist()*3)
np.matrix(range(3)) * np.eye(3)
np.arange(3) * np.eye(3)
another example: What does this do?
>>> x = np.array(tuple(range(3)), [('',int)]*3)
>>> x
array((0, 1, 2),
dtype=[('f0', '<i4'), ('f1', '<i4'), ('f2', '<i4')])
>>> x * np.eye(3)
This question has already very good answers. Here I just want to add what I usually do (which somehow summarizes responses by others) when I want to write functions that accept a wide range of inputs while the operations I do on them require a 2d row or column vector.
If I know the input is always 1d (array or list):
a. if I need a row: x = np.asarray(x)[None,:]
b. if I need a column: x = np.asarray(x)[:,None]
If the input can be either 2d (array or list) with the right shape or 1d (which needs to be converted to 2d row/column):
a. if I need a row: x = np.atleast_2d(x)
b. if I need a column: x = np.atleast_2d(np.asarray(x).T).T or x = np.reshape(x, (len(x),-1)) (the latter seems faster)
This is a good use for decorators
def atmost_2d(func):
def wrapr(x):
return func(np.atleast_2d(x)).squeeze()
return wrapr
For example, this function will pick out the last column of its input.
#atmost_2d
def g(x):
return x[:,-1]
But: it works for:
1d:
In [46]: b
Out[46]: array([0, 1, 2, 3, 4, 5])
In [47]: g(b)
Out[47]: array(5)
2d:
In [49]: A
Out[49]:
array([[0, 1],
[2, 3],
[4, 5]])
In [50]: g(A)
Out[50]: array([1, 3, 5])
0d:
In [51]: g(99)
Out[51]: array(99)
This answer builds on the previous two.