Develop Reference

How to reorder dstack

I have 6 files with shape (6042,) or 1 column. I used dstack to stack the 6 files in hopes of getting a shape (6042, 1, 6). But after I stack it I get shape (1, 6042, 6). Then I tried to change the order using
new_train = np.reshape(train_x,(train_x[1],1,train_x[2]))
error appears:
IndexError: index 1 is out of bounds for axis 0 with size 1
This is my dstack code:
train_x = dstack([train_data['gx'],train_data['gy'], train_data['gz'], train_data['ax'],train_data['ay'], train_data['az']])

error is because
train_x[1]
tries looking 2nd row of train_x but it has only 1 row as you said shape 1, 6042, 6). So you need to look shape and index it
new_train = np.reshape(train_x, (train_x.shape[1], 1, train_x.shape[2]))
but this can be also doable with transpose
new_train = train_x.transpose(1, 0, 2)
so this changes axes 0 and 1's positions.
Other solution is fixing dstack's way. It gives "wrong" shape because your datas shape not (6042, 1) but (6042,) as you say. So if you reshape the datas before dstack it should also work:
datas = [train_data['gx'],train_data['gy'], train_data['gz'],
train_data['ax'],train_data['ay'], train_data['az']]
#this list comprehension makes all shape (6042, 1) now
new_datas = [td[:, np.newaxis] for td in datas]
new_train = dstack(new_datas)

You can use np.moveaxis(X, 0, -2), where X is your (1,6042,6) array.
This function swaps the axis. 0 for your source axis and -2 is your destination axis.

np.dstack uses:
arrs = atleast_3d(*tup)
to convert the list of arrays to a list of 3d arrays.
In [51]: alist = [np.ones(3,int),np.zeros(3,int)]
In [52]: alist
Out[52]: [array([1, 1, 1]), array([0, 0, 0])]
In [53]: np.atleast_3d(*alist)
Out[53]:
[array([[[1],
[1],
[1]]]),
array([[[0],
[0],
[0]]])]
In [54]: _[0].shape
Out[54]: (1, 3, 1)
Concatenating those on the last dimension produces the (1,n,6) kind of result.
With expand_dims we can adjust the shape of all arrays to (n,1,1), and then do the concatenate:
In [62]: np.expand_dims(alist[0],[1,2]).shape
Out[62]: (3, 1, 1)
In [63]: np.concatenate([np.expand_dims(a,[1,2]) for a in alist], axis=2)
Out[63]:
array([[[1, 0]],
[[1, 0]],
[[1, 0]]])
In [64]: _.shape
Out[64]: (3, 1, 2)
direct reshape or newaxis would work just as well:
In [65]: np.concatenate([a[:,None,None] for a in alist], axis=2).shape
Out[65]: (3, 1, 2)
stack is another cover that adjusts shapes before concatenate:
In [67]: np.stack(alist,1).shape
Out[67]: (3, 2)
In [68]: np.stack(alist,1)[:,None].shape
Out[68]: (3, 1, 2)
So there are lots of ways to get what you want, whether it means adjusting shapes before the concatenate, or after.

How do I transpose an empty numpy array?

I have an 'empty' 2D array in numpy as
arr = np.array([[[], [], []], [[], [], []]]).
When I do np.transpose(arr), I get the result: [], instead of the expected:
[[[],[]],[[],[]],[[],[]]].

You get an empty array [] with the right shape. Mind that also arr is an empty array [].
arr = np.array([[[], [], []], [[], [], []]])
print(arr, arr.shape)
t = arr.T
print(t, t.shape)
[] (2, 3, 0)
[] (0, 3, 2)

Look at what your expression produces:
In [41]: arr = np.array([[[], [], []], [[], [], []]])
In [42]: arr
Out[42]: array([], shape=(2, 3, 0), dtype=float64)
In [43]: print(arr)
[]
In [44]: print(repr(arr))
array([], shape=(2, 3, 0), dtype=float64)
The print shows the str display, while repr is a fuller one that tells us shape and dtype. np.array has followed the [] all the way down, making a 3d array that has float elements. But since the lowest level is created from [], it has size 0 dimension, and overall the array has 0 elements.
What you want, based on the comment, is a (2,3) array with object dtype. This can hold objects such as lists. But making that with np.array is tricky. A more general tool is to make one with the right shape and dtype.
I like to use empty for this, since is fills the object array with None elements. (In a numeric dtype np.empty has other problems, but for object it's nice.)
In [45]: arr = np.empty((2,3), dtype=object)
In [46]: arr
Out[46]:
array([[None, None, None],
[None, None, None]], dtype=object)
But trying to assign a list to elements of such an array can be tricky:
In [47]: arr[:]=[]
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-47-b5ed8d639464> in <module>
----> 1 arr[:]=[]
ValueError: could not broadcast input array from shape (0) into shape (2,3)
In [48]: np.full((2,3),[])
...
ValueError: could not broadcast input array from shape (0) into shape (2,3)
full has the same problem. In addition assignment like this, if it worked, would put the same list in each slot, the equivalent of making a list with [[]]*3. We want a new [] in each slot.
We can do this in the 2d arr, but the iteration is simpler with a 1d (which can be reshaped later):
In [49]: arr = np.empty(6, object)
In [50]: arr
Out[50]: array([None, None, None, None, None, None], dtype=object)
In [51]: for i in range(6): arr[i]=[]
In [52]: arr
Out[52]:
array([list([]), list([]), list([]), list([]), list([]), list([])],
dtype=object)
In [53]: arr = np.reshape(arr, (2,3))
In [54]: arr
Out[54]:
array([[list([]), list([]), list([])],
[list([]), list([]), list([])]], dtype=object)
Obvious we could transpose that, but could just as well use (3,2) in the reshape.
Note that this display of arr clearly shows that it contains list objects.
But do you really need such an array? Is it worth the extra work?

For a given condition, get indices of values in 2D tensor A, use those to index a 3D tensor B

For a given 2D tensor I want to retrieve all indices where the value is 1. I expected to be able to simply use torch.nonzero(a == 1).squeeze(), which would return tensor([1, 3, 2]). However, instead, torch.nonzero(a == 1) returns a 2D tensor (that's okay), with two values per row (that's not what I expected). The returned indices should then be used to index the second dimension (index 1) of a 3D tensor, again returning a 2D tensor.
import torch
a = torch.Tensor([[12, 1, 0, 0],
[4, 9, 21, 1],
[10, 2, 1, 0]])
b = torch.rand(3, 4, 8)
print('a_size', a.size())
# a_size torch.Size([3, 4])
print('b_size', b.size())
# b_size torch.Size([3, 4, 8])
idxs = torch.nonzero(a == 1)
print('idxs_size', idxs.size())
# idxs_size torch.Size([3, 2])
print(b.gather(1, idxs))
Evidently, this does not work, leading to aRunTimeError:
RuntimeError: invalid argument 4: Index tensor must have same
dimensions as input tensor at
C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorEvenMoreMath.cpp:453
It seems that idxs is not what I expect it to be, nor can I use it the way I thought. idxs is
tensor([[0, 1],
[1, 3],
[2, 2]])
but reading through the documentation I don't understand why I also get back the row indices in the resulting tensor. Now, I know I can get the correct idxs by slicing idxs[:, 1] but then still, I cannot use those values as indices for the 3D tensor because the same error as before is raised. Is it possible to use the 1D tensor of indices to select items across a given dimension?

You could simply slice them and pass it as the indices as in:
In [193]: idxs = torch.nonzero(a == 1)
In [194]: c = b[idxs[:, 0], idxs[:, 1]]
In [195]: c
Out[195]:
tensor([[0.3411, 0.3944, 0.8108, 0.3986, 0.3917, 0.1176, 0.6252, 0.4885],
[0.5698, 0.3140, 0.6525, 0.7724, 0.3751, 0.3376, 0.5425, 0.1062],
[0.7780, 0.4572, 0.5645, 0.5759, 0.5957, 0.2750, 0.6429, 0.1029]])
Alternatively, an even simpler & my preferred approach would be to just use torch.where() and then directly index into the tensor b as in:
In [196]: b[torch.where(a == 1)]
Out[196]:
tensor([[0.3411, 0.3944, 0.8108, 0.3986, 0.3917, 0.1176, 0.6252, 0.4885],
[0.5698, 0.3140, 0.6525, 0.7724, 0.3751, 0.3376, 0.5425, 0.1062],
[0.7780, 0.4572, 0.5645, 0.5759, 0.5957, 0.2750, 0.6429, 0.1029]])
A bit more explanation about the above approach of using torch.where(): It works based on the concept of advanced indexing. That is, when we index into the tensor using a tuple of sequence objects such as tuple of tensors, tuple of lists, tuple of tuples etc.
# some input tensor
In [207]: a
Out[207]:
tensor([[12., 1., 0., 0.],
[ 4., 9., 21., 1.],
[10., 2., 1., 0.]])
For basic slicing, we would need a tuple of integer indices:
In [212]: a[(1, 2)]
Out[212]: tensor(21.)
To achieve the same using advanced indexing, we would need a tuple of sequence objects:
# adv. indexing using a tuple of lists
In [213]: a[([1,], [2,])]
Out[213]: tensor([21.])
# adv. indexing using a tuple of tuples
In [215]: a[((1,), (2,))]
Out[215]: tensor([21.])
# adv. indexing using a tuple of tensors
In [214]: a[(torch.tensor([1,]), torch.tensor([2,]))]
Out[214]: tensor([21.])
And the dimension of the returned tensor would always be one dimension less than the dimension of the input tensor.

Assuming that b's three dimensions are batch_size x sequence_length x features (b x s x feats), the expected results can be achieved as follows.
import torch
a = torch.Tensor([[12, 1, 0, 0],
[4, 9, 21, 1],
[10, 2, 1, 0]])
b = torch.rand(3, 4, 8)
print(b.size())
# b x s x feats
idxs = torch.nonzero(a == 1)[:, 1]
print(idxs.size())
# b
c = b[torch.arange(b.size(0)), idxs]
print(c.size())
# b x feats

import torch
a = torch.Tensor([[12, 1, 0, 0],
[4, 9, 21, 1],
[10, 2, 1, 0]])
b = torch.rand(3, 4, 8)
print('a_size', a.size())
# a_size torch.Size([3, 4])
print('b_size', b.size())
# b_size torch.Size([3, 4, 8])
#idxs = torch.nonzero(a == 1, as_tuple=True)
idxs = torch.nonzero(a == 1)
#print('idxs_size', idxs.size())
print(torch.index_select(b,1,idxs[:,1]))

As a supplementary of #kmario23's solution, you can still achieve the same results like
b[torch.nonzero(a==1,as_tuple=True)]

suppress numpy array creation protocol for numpy arrays of objects

I am trying to build a library which reads complex HDF5 data files in python.
I am running into a problem where, an HDF5 Dataset somehow implements the default array protocol (sometimes), such that when a numpy array is created from it, it casts to the particular array type.
In [8]: ds
Out[8]: <HDF5 dataset "two_by_zero_empty_matrix": shape (2,), type "<u8">
In [9]: ds.value
Out[9]: array([2, 0], dtype=uint64)
This Dataset object, implements the numpy array protocol, and when the dataset consists of numbers, it supplies a default array type.
In [10]: np.array(ds)
Out[10]: array([2, 0], dtype=uint64)
However, if the dataset doesn't consist of numbers, but some other objects, as you would expect, it just uses a numpy array of type np.object:
In [43]: ds2
Out[43]: <HDF5 dataset "somecells": shape (2, 3), type "|O8">
In [44]: np.array(ds2)
Out[44]:
array([[<HDF5 object reference>, <HDF5 object reference>,
<HDF5 object reference>],
[<HDF5 object reference>, <HDF5 object reference>,
<HDF5 object reference>]], dtype=object)
This behavior might seem convenient but in my case it's actually inconvenient since it interferes with my recursive traversal of the data file. Working around it really turns out to be difficult since there a lot of different possible data types which have to be special-cased a little differently depending on whether they are children of objects or arrays of numbers.
My question is this: is there a way to suppress the default array creation protocol, such that I could create an object array out of dataset objects that want to cast to their natural duck types?
That is, I want something like: np.array(ds, dtype=object), which will produce an array of [<Dataset object of type int>, dtype=object] and not [3 4 5, dtype=int].
But np.array(ds, dtype=np.object) throws IOError: Can't read data (No appropriate function for conversion path)
I tried in earnest to google some documentation about the numpy array protocol works, and found a lot, but it doesn't really appear to me that anyone considered the possibility that someone might want this behavior.

I can understand where the Out[44] is coming from. It's an array containing pointers to objects, in this case h5py references to objects on the file (I think).
With np.array(ds, dtype=object) are you trying to create something more like this, rather than the 'normal' array that you get with np.array(ds)? array([2, 0], dtype=uint64).
But what is the parallel array? A single element array with a pointer to ds? Or a 2 element array with pointers to 2 and 0 somewhere on the file? What if they aren't <HDF5 object reference>?
In numpy, without any h5py stuff, I can create an object array from a list of values:
In [104]: np.array([2,0], dtype=object)
Out[104]: array([2, 0], dtype=object)
Or I can start with an empty array (filled with None) and assign values:
In [105]: x=np.empty((2,), dtype=object)
In [106]: x[0]=2
In [107]: x[1]=0
In [108]: x
Out[108]: array([2, 0], dtype=object)
I guess you could try:
x[0] = ds[0]
or
x[:] = ds[:]
Or make a single element object array
x = np.empty((), dtype=object)
x[()] = ds
I don't have a h5py test file open on my Ipython session to test this.
But I can do something weird like make an object array that contains itself. I can work with, but I can't display it without getting a recursion error.
In [118]: x=np.empty((),dtype=object)
In [119]: x[()]=x
In [120]: x1=x[()]
In [121]: x1==x
Out[121]: True
I have a small h5py file open on another terminal:
In [315]: list(f.keys())
Out[315]: ['d', 'x', 'y']
In [317]: f['d'] # the group
Out[317]: <HDF5 group "/d" (2 members)>
x is a string:
In [318]: f['x'] # a single element (a string)
Out[318]: <HDF5 dataset "x": shape (), type "|O4">
In [330]: f['x'].value
Out[330]: 'astring'
In [331]: np.array(f['x'])
Out[331]: array('astring', dtype=object)
y is an array:
In [320]: f['y'][:]
Out[320]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
In [321]: f['y'].value
Out[321]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
In [322]: np.array(f['y'])
Out[322]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
In [323]: timeit np.array(f['y'])
1000 loops, best of 3: 364 µs per loop
In [324]: timeit f['y'].value
1000 loops, best of 3: 380 µs per loop
So access with value and array is equivalent.
Access as object array gives the same sort of error as you got.
In [325]: np.array(f['y'],dtype=object)
...
OSError: can't read data (Dataset: Read failed)
Conversion to float works fine:
In [326]: np.array(f['y'],dtype=float)
Out[326]: array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
And the assignment to a predefined object array works:
In [327]: x=np.empty((),dtype=object)
In [328]: x[()]=f['y']
In [329]: x
Out[329]: array(<HDF5 dataset "y": shape (10,), type "<i4">, dtype=object)
Trying to create a 10 element array to take y:
In [332]: y1=np.empty((10,),dtype=object)
In [333]: y1[:]=f['y']
...
OSError: can't read data (Dataset: Read failed)
In [334]: y1[:]=f['y'].value
In [335]: y1
Out[335]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object)
y1[:]=f['y'][:] also works
I can't assign dataset to y1 (same error as when I tried np.array(f['y'],dtype=object). But I can assign its values. I can even assign the dataset to one element of y1
In [338]: y1[-1]=f['y']
In [339]: y1
Out[339]:
array([0, 1, 2, 3, 4, 5, 6, 7, 8,
<HDF5 dataset "y": shape (10,), type "<i4">], dtype=object)
I keep coming back to the basic idea that an object array is just a collection of pointers, essentially a list in an array wrapper.

Numpy multi-dimensional slicing with multiple boolean arrays

I'm trying to use individual 1-dimensional boolean arrays to slice a multi-dimension array. For some reason, this code doesn't work:
>>> a = np.ones((100, 200, 300, 2))
>>> a.shape
(100, 200, 300, 2)
>>> m1 = np.asarray([True]*200)
>>> m2 = np.asarray([True]*300)
>>> m2[-1] = False
>>> a[:,m1,m2,:]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (200,) (299,)
>>> m2 = np.asarray([True]*300) # try again with all 300 dimensions True
>>> a[:,m1,m2,:]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (200,) (300,)
But this works just fine:
>>> a = np.asarray([[[1, 2], [3, 4], [5, 6]], [[11, 12], [13, 14], [15, 16]]])
>>> a.shape
(2, 3, 2)
>>> m1 = np.asarray([True, False, True])
>>> m2 = np.asarray([True, False])
>>> a[:,m1,m2]
array([[ 1, 5],
[11, 15]])
Any idea of what I might be doing wrong in the first example?

Short answer: The number of True elements in m1 and m2 must match, unless one of them has only one True term.
Also distinguish between 'diagonal' indexing and 'rectangular' indexing. This is about indexing, not slicing. The dimensions with : are just along for the ride.
Initial ideas
I can get your first case working with:
In [137]: a=np.ones((100,200,300,2))
In [138]: m1=np.ones((200,),bool)
In [139]: m2=np.ones((300,),bool)
In [140]: m2[-1]=False
In [141]: I,J=np.ix_(m1,m2)
In [142]: a[:,I,J,:].shape
Out[142]: (100, 200, 299, 2)
np.ix_ turns the 2 boolean arrays into broadcastable index arrays
In [143]: I.shape
Out[143]: (200, 1)
In [144]: J.shape
Out[144]: (1, 299)
Note that this picks 200 'rows' in one dimension, and 299 in the other.
I'm not sure why this kind of reworking of the arrays is needed in this case, but not in the 2nd
In [154]: b=np.arange(2*3*2).reshape((2,3,2))
In [155]: n1=np.array([True,False,True])
In [156]: n2=np.array([True,False])
In [157]: b[:,n1,n2]
Out[157]:
array([[ 0, 4], # shape (2,2)
[ 6, 10]])
Taking the same ix_ strategy produces the same values but a different shape:
In [164]: b[np.ix_(np.arange(b.shape[0]),n1,n2)]
# or I,J=np.ix_(n1,n2);b[:,I,J]
Out[164]:
array([[[ 0],
[ 4]],
[[ 6],
[10]]])
In [165]: _.shape
Out[165]: (2, 2, 1)
Both cases use all rows of the 1st dimension. The ix one picks 2 'rows' of the 2nd dim, and 1 column of the last, resulting the (2,2,1) shape. The other picks b[:,0,0] and b[0,2,0] terms, resulting (2,2) shape.
(see my addenda as to why both are simply broadcasting).
These are all cases of advanced indexing, with boolean and numeric indexes. One can study the docs, or one can play around. Sometimes it's more fun to do the later. :)
(I knew that ix_ was good for adding the necessary np.newaxis to arrays so can be broadcast together, but didn't realize that worked with boolean arrays as well - it uses np.nonzero() to convert boolean to indices.)
Resolution
Underlying this is, I think, a confusion over 2 modes of indexing. which might called 'diagonal' and 'rectangular' (or element-by-element selection versus block selection). To illustrate look at a small 2d array
In [73]: M=np.arange(6).reshape(2,3)
In [74]: M
Out[74]:
array([[0, 1, 2],
[3, 4, 5]])
and 2 simple numeric indexes
In [75]: m1=np.arange(2); m2=np.arange(2)
They can be used 2 ways:
In [76]: M[m1,m2]
Out[76]: array([0, 4])
and
In [77]: M[m1[:,None],m2]
Out[77]:
array([[0, 1],
[3, 4]])
The 1st picks 2 points, the M[0,0] and M[1,1]. This kind of indexing lets us pick out the diagonals of an array.
The 2nd picks 2 rows and from that 2 columns. This is the kind of indexing the np.ix_ produces. The 1st picks 2 points, the M[0,0] and M[1,1]. This a 'rectangular' form of indexing.
Change m2 to 3 values:
In [78]: m2=np.arange(3)
In [79]: M[m1[:,None],m2] # returns a 2x3
Out[79]:
array([[0, 1, 2],
[3, 4, 5]])
In [80]: M[m1,m2] # produces an error
...
ValueError: shape mismatch: objects cannot be broadcast to a single shape
But if m2 has just one element, we don't get the broadcast error - because the size 1 dimension can be expanded during broadcasting:
In [81]: m2=np.arange(1)
In [82]: M[m1,m2]
Out[82]: array([0, 3])
Now change the index arrays to boolean, each matching the length of the respective dimensions, 2 and 3.
In [91]: m1=np.ones(2,bool); m2=np.ones(3,bool)
In [92]: M[m1,m2]
...
ValueError: shape mismatch: objects cannot be broadcast to a single shape
In [93]: m2[2]=False # m1 and m2 each have 2 True elements
In [94]: M[m1,m2]
Out[94]: array([0, 4])
In [95]: m2[0]=False # m2 has 1 True element
In [96]: M[m1,m2]
Out[96]: array([1, 4])
With 2 and 3 True terms we get an error, but with 2 and 2 or 2 and 1 it runs - just as though we'd used the indices of the True elements: np.nonzero(m2).
To apply this to your examples. In the first, m1 and m2 have 200 and 299 True elements. a[:,m1,m2,:] fails because of a mismatch in the number of True terms.
In the 2nd, they have 2 and 1 True terms, with nonzero indices of [0,2] and [0], which can be broadcast to [0,0]. So it runs.
http://docs.scipy.org/doc/numpy-1.10.0/reference/arrays.indexing.html
explains boolean array indexing in terms of nonzero and ix_.
Combining multiple Boolean indexing arrays or a Boolean with an integer indexing array can best be understood with the obj.nonzero() analogy. The function ix_ also supports boolean arrays and will work without any surprises.
Addenda
On further thought the distinction between 'diagonal' and 'block/rectangular' indexing might be more my mental construct that numpys. Underlying both is the concept of broadcasting.
Take the n1 and n2 booleans, and get their nonzero equivalents:
In [107]: n1
Out[107]: array([ True, False, True], dtype=bool)
In [108]: np.nonzero(n1)
Out[108]: (array([0, 2], dtype=int32),)
In [109]: n2
Out[109]: array([ True, False], dtype=bool)
In [110]: np.nonzero(n2)
Out[110]: (array([0], dtype=int32),)
Now try broadcasting in the 'diagonal' and 'rectangular' modes:
In [105]: np.broadcast_arrays(np.array([0,2]),np.array([0]))
Out[105]: [array([0, 2]),
array([0, 0])]
In [106]: np.broadcast_arrays(np.array([0,2])[:,None],np.array([0]))
Out[106]:
[array([[0],
[2]]),
array([[0],
[0]])]
One produces (2,) arrays, the other (2,1).

This might be a simple workaround:
a[:,m1,:,:][:,:,m2,:]