This question already has an answer here:
why there is deference between the output type of this two Numpy slice commands
(1 answer)
Closed last year.
I'm a little bit new to Python & Numpy, but I've noticed that when you call operator [] on a numpy array A, if it's a single index that is used (e.g., A[1]), the resulting sub-array is 1 dimension smaller, but if it's a range of indices (e.g., A[1:]) the dimension of the subarray remains unchanged, even if the range of indices covers only a single index, e.g., in this above case, if A was 2x2, A[1:] is effectively just a single index, but the resulting size is not the same as A[1].
My question is: is this always true in that if you supply a range of indices when extracting a subarray, the dimension doesn't change, and that a single index always reduces the dimension by 1? Are there edge cases?
That is always the case. When you use one index-value, e.g. A[1], you are effectively saying "give me the subarray A[1]", which, by definition, has a dimensionality smaller (by 1).
When you request a range of indices, e.g., A[1:] you are "cropping" A, to get everything but the first slice (A[0]). See, the range of indices define the axis you "lost" in the previous case (A[1]).
The following docs should be helpful to understand numpy arrays (indexing):
Arrays: https://numpy.org/doc/stable/user/absolute_beginners.html#more-information-about-arrays
Indexing: https://numpy.org/doc/stable/user/basics.indexing.html
Related
I have a 5 dimension array like this
a=np.random.randint(10,size=[2,3,4,5,600])
a.shape #(2,3,4,5,600)
I want to get the first element of the 2nd dimension, and several elements of the last dimension
b=a[:,0,:,:,[1,3,5,30,17,24,30,100,120]]
b.shape #(9,2,4,5)
as you can see, the last dimension was automatically converted to the first dimension.
why? and how to avoid that?
This behavior is described in the numpy documentation. In the expression
a[:,0,:,:,[1,3,5,30,17,24,30,100,120]]
both 0 and [1,3,5,30,17,24,30,100,120] are advanced indexes, separated by slices. As the documentation explains, in such case dimensions coming from advanced indexes will be first in the resulting array.
If we replace 0 by the slice 0:1 it will change this situation (since it will leave only one advanced index), and then the order of dimensions will be preserved. Thus one way to fix this issue is to use the 0:1 slice and then squeeze the appropriate axis:
a[:,0:1,:,:,[1,3,5,30,17,24,30,100,120]].squeeze(axis=1)
Alternatively, one can keep both advanced indexes, and then rearrange axes:
np.moveaxis(a[:,0,:,:,[1,3,5,30,17,24,30,100,120]], 0, -1)
There is this great Question/Answer about slicing the last dimension:
Numpy slice of arbitrary dimensions: for slicing a numpy array to obtain the i-th index in the last dimension, one can use ... or Ellipsis,
slice = myarray[...,i]
What if the first N dimensions are needed ?
For 3D myarray, N=2:
slice = myarray[:,:,0]
For 4D myarray, N=2:
slice = myarray[:,:,0,0]
Does this can be generalized to an arbitrary dimension?
I don't think there's any built-in syntactic sugar for that, but slices are just objects like anything else. The slice(None) object is what is created from :, and otherwise just picking the index 0 works fine.
myarray[(slice(None),)*N+(0,)*(myarray.ndim-N)]
Note the comma in (slice(None),). Python doesn't create tuples from parentheses by default unless the parentheses are empty. The comma signifies that don't just want to compute whatever's on the inside.
Slices are nice because they give you a view into the object instead of a copy of the object. You can use the same idea to, e.g., iterate over everything except the N-th dimension on the N-th dimension. There have been some stackoverflow questions about that, and they've almost unanimously resorted to rolling the indices and other things that I think are hard to reason about in high-dimensional spaces. Slice tuples are your friend.
From the comments, #PaulPanzer points out another technique that I rather like.
myarray.T[(myarray.ndim-N)*(0,)].T
First, transposes in numpy are view-operations instead of copy-operations. This isn't inefficient in the slightest. Here's how it works:
Start with myarray with dimensions (0,...,k)
The transpose myarray.T reorders those to (k,...,0)
The whole goal is to fix the last myarray.ndim-N dimensions from the original array, so we select those with [(myarray.ndim-N)*(0,)], which grabs the first myarray.ndim-N dimensions from this array.
They're in the wrong order. We have dimensions (N-1,...,0). Use another transpose with .T to get the ordering (0,...,N-1) instead.
I'm messing around with 2-dimensional slicing and don't understand why leaving out some defaults grabs the same values from the original array but produces different output. What's going on with the double brackets and shape changing?
x = np.arange(9).reshape(3,3)
y = x[2]
z = x[2:,:]
print y
print z
print shape(y)
print shape(z)
[6 7 8]
[[6 7 8]]
(3L,)
(1L, 3L)
x is a two dimensional array, an instance of NumPy's ndarray object. You can index/slice these objects in essentially two ways: basic and advanced.
y[2] fetches the row at index 2 of the array, returning the array [6 7 8]. You're doing basic slicing because you've specified only an integer. You can also specify a tuple of slice objects and integers for basic slicing, e.g. x[:,2] to select the right-hand column.
With basic slicing, you're also reducing the number of dimensions of the returned object (in this case from two to just one):
An integer, i, returns the same values as i:i+1 except the dimensionality of the returned object is reduced by 1.
So when you ask for the shape of y, this is why you only get back one dimension (from your two-dimensional x).
Advanced slicing occurs when you specify an ndarray: or a tuple with at least one sequence object or ndarray. This is the case with x[2:,:] since 2: counts as a sequence object.
You get back an ndarray. When you ask for its shape, you will get back all of the dimensions (in this case two):
The shape of the output (or the needed shape of the object to be used for setting) is the broadcasted shape.
In a nutshell, as soon as you start slicing along any dimension of your array with :, you're doing advanced slicing and not basic slicing.
One brief point worth mentioning: basic slicing returns a view onto the original array (changes made to y will be reflected in x). Advanced slicing returns a brand new copy of the array.
You can read about array indexing and slicing in much more detail here.
Suppose there is an array
(1) x=np.array([[1,2],[1,2],[1,2]])
and a second array
(2) y=np.array([[1],[1,2],[1,2,3]])
The command size(x) returns the total count of all elements along every axis. In this case 6. However, size(y) returns 3. This must be because numpy interprets (2) in this case as three elements (the three subarrays) along one axis, although shape(y) returns (3, ). My question is now: how can I get numpy to interpret (2) as an array with three axes, so that size(y) returns the total count of all atomic elemets, which is 6?
I don't think it's possible to get the number of elements from y without looping over the objects.
The problem is that the elements of y are not numbers, they are objects (lists). Numpy does not support lists of lists and therefore it stores it as a 1-dimensional array of objects. I don't think there are Numpy methods to get the total number of elements in y.
I have a bit of code that attempts to find the contents of an array at indices specified by another, that may specify indices that are out of range of the former array.
input = np.arange(0, 5)
indices = np.array([0, 1, 2, 99])
What I want to do is this:
print input[indices]
and get
[0 1 2]
But this yields an exception (as expected):
IndexError: index 99 out of bounds 0<=index<5
So I thought I could use masked arrays to hide the out of bounds indices:
indices = np.ma.masked_greater_equal(indices, 5)
But still:
>print input[indices]
IndexError: index 99 out of bounds 0<=index<5
Even though:
>np.max(indices)
2
So I'm having to fill the masked array first, which is annoying, since I don't know what fill value I could use to not select any indices for those that are out of range:
print input[np.ma.filled(indices, 0)]
[0 1 2 0]
So my question is: how can you use numpy efficiently to select indices safely from an array without overstepping the bounds of the input array?
Without using masked arrays, you could remove the indices greater or equal to 5 like this:
print input[indices[indices<5]]
Edit: note that if you also wanted to discard negative indices, you could write:
print input[indices[(0 <= indices) & (indices < 5)]]
It is a VERY BAD idea to index with masked arrays. There was a (very short) time with using MaskedArrays for indexing would have thrown an exception, but it was a bit too harsh...
In your test, you're filtering indices to find the entries matching a condition. What should you do with the missing entries of your MaskedArray ? Is the condition False ? True ? Should you use a default ? It's up to you, the user, to decide what to do.
Using indices.filled(0) means that when an item of indices is masked (as in, undefined), you want to take the first index (0) as default. Probably not what you wanted.
Here, I would have simply used input[indices.compressed()] : the compressed method flattens your MaskedArray, keeping only the unmasked entries.
But as you realized, you probably didn't need MaskedArrays in the first place