Develop Reference

Pytorch: Index with tensor along multiple axes OR scatter to more than one index at once

I am trying to update very specific indices of a multidimensional tensor in Pytorch, and I am not sure how to access the correct indices. I can do this in a very straightforward way in Numpy:
import numpy as np
#set up the array containing the data
data = 100*np.ones((10,10,2))
data[5:,:,:] = 0
#select the data points that I want to update
idxs = np.nonzero(data.sum(2))
#generate the updates that I am going to do
updates = np.random.randint(5,size=(idxs[0].shape[0],2))
#update the data
data[idxs[0],idxs[1],:] = updates
I need to implement this in Pytorch but I am not sure how to do this. It seems like I need the scatter function but that only works along a single dimension instead of the multiple dimensions that I need. How can I do this?

These operations work exactly the same in their PyTorch counterparts, except for torch.nonzero, which by default returns a tensor of size [z, n] (where z is the number of non-zero elements and n the number of dimensions) instead of a tuple of n tensors with size [z] (as NumPy does), but that behaviour can be changed by setting as_tuple=True.
Other than that you can directly translate it to PyTorch, but you need to make sure that the types match, because you cannot assign a tensor of type torch.long (default of torch.randint) to a tensor of type torch.float (default of torch.ones). In this case, data is probably meant to have type torch.long:
#set up the array containing the data
data = 100*torch.ones((10,10,2), dtype=torch.long)
data[5:,:,:] = 0
#select the data points that I want to update
idxs = torch.nonzero(data.sum(2), as_tuple=True)
#generate the updates that I am going to do
updates = torch.randint(5,size=(idxs[0].shape[0],2))
#update the data
data[idxs[0],idxs[1],:] = updates

Reshaping numpy arrays of differing dimensions inside an array

So the task is to optimise a Neural Network with a PSO. The PSO needs a one-dimensional list of all the weights and biases, like so [0.1 0.244 ... 0.214]. The NN needs an array of arrays with different dimensions, like so [[x,y], [m,n], ...(all the hidden layer matrices)... ,[p,q]] X and y are the dimensions for the input layer, then all the hidden layers and finally p and q - the dimensions of the output layer.
I can easily flatten the array to pass it to the PSO, but I need a method that takes the modified array and reshapes it back into the same array of arrays with the same dimensions as the starting one from the NN.
The dimensions depend on the amount of neurons in a layer, we have that information from the start.
I have tried to keep track of the shapes array and create an indices array to know when to stop but it doesn't seem to work. I am trying something with slicing now but no cigar yet. A modification to the NN is also possible but how to create it so it takes a predefined list of weights? There might be a very nice and efficient way to do it but I just haven't thought of it yet... Any suggestions?
Example:
a = np.array([1,2,3])
b = np.array([7,8,9,10])
c = np.array([12,13,14,15,16])
b.reshape(2,2)
arr = []
arr.append(a)
arr.append(b)
arr.append(c)
This is a very simple example of what the list of weights is as the NN works with it - a list of multi-dimensional array. Arr can be converted into a numpy array of objects if necessary with np.asarray(arr).
Flattening is easy, here is how I do it (there might be a better that doesn't need a loop, if you know, I'd be thankful if you shared).
Flattening:
new_arr = np.array([])
for i in range(len(arr)):
new_arr = np.append(arr, arr[i].flatten())
My question is how to take new_arr and put it back together to look like arr and is there a beautiful and fast way to do it.

You can save the shape in a variable (it's just a tuple). Try something like:
...
old_shape = arr.shape
# ... do flattening here
new_arr.reshape(old_shape)

new_arr = np.array([])
shapes=[]
for i in range(len(arr)):
new_arr = np.append(new_arr, arr[i].flatten())
shapes.append(arr[i].shape)
#do whatever
restoredArray =[]
offset=0
for i in range(len(shapes)):
s = shapes[i]
n = np.prod(s)
restoredArray.append(new_arr[offset:(offset+n)].reshape(s))
offset+=n

How to use tf.data.Dataset.padded_batch with a nested shape?

I am building a dataset with two tensors of shape [batch,width,heigh,3] and [batch,class] for each element. For simplicity lets say class = 5.
What shape do you feed to dataset.padded_batch(1000,shape) such that image is padded along the width/height/3 axis?
I have tried the following:
tf.TensorShape([[None,None,None,3],[None,5]])
[tf.TensorShape([None,None,None,3]),tf.TensorShape([None,5])]
[[None,None,None,3],[None,5]]
([None,None,None,3],[None,5])
(tf.TensorShape([None,None,None,3]),tf.TensorShape([None,5])‌​)
Each raising TypeError
The docs state:
padded_shapes: A nested structure of tf.TensorShape or tf.int64 vector
tensor-like objects representing the shape to which the respective
component of each input element should be padded prior to batching.
Any unknown dimensions (e.g. tf.Dimension(None) in a tf.TensorShape or
-1 in a tensor-like object) will be padded to the maximum size of that dimension in each batch.
The relevant code:
dataset = tf.data.Dataset.from_generator(generator,tf.float32)
shapes = (tf.TensorShape([None,None,None,3]),tf.TensorShape([None,5]))
batch = dataset.padded_batch(1,shapes)

Thanks to mrry for finding the solution. Turns out that the type in from_generator has to match the number of tensors in the entries.
new code:
dataset = tf.data.Dataset.from_generator(generator,(tf.float32,tf.float32))
shapes = (tf.TensorShape([None,None,None,3]),tf.TensorShape([None,5]))
batch = dataset.padded_batch(1,shapes)

TensorShape doesn't accept nested lists. tf.TensorShape([None, None, None, 3, None, 5]) and TensorShape(None) (note no []) are legal.
Combining these two tensors sounds odd to me, though. I'm not sure what you're trying to accomplish, but I'd recommend trying to do it without combining tensors of different dimensions.

Feeding Numpy Arrays to CNTK LSTM Model

I'm looking to see if there is a way to feed sequence data as Numpy arrays to a text LSTM model defined in CTNK. Each instance in my dataset is a sequence of integers mapping back to words, and the length of each sequence is different. It seems like one can convert their raw text data to the CTF format and feed this data to a model by creating a reader function which generates mini-batches as in this example. However, I'm wondering if there is a way to feed Numpy arrays to this same model.
Further down in this example, there is a discussion of feeding sequences with Numpy, which I was hoping would solve my problem. However, the example deals with sequences of images instead of variable-length sequences of words. In the case of the example, we'll end up with a tensor of n elements that are each 3 x 32 x 32, and we can set up an input variable expecting these dimensions. However, in the case of sequences of words where each sequence has a different length, this example breaks down.
Any help on interop between CTNK and Numpy for text-based LSTM's / RNN's would be greatly appreciated.

You are probably looking for:
x = cntk.sequence.input_variable(shape=())
Here is a sample little program that demonstrates how it works with a variable sequence length:
import numpy as np
import cntk
# define the model
x = cntk.sequence.input_variable(shape=())
z = cntk.sequence.last(x)
# define the data
a = [[1,2,3], [4,5], [6,7,8,9], [0]]
b = [np.array(i, dtype=np.float32) for i in a]
# evaluate
res = z.eval({x: b})
print(res)

How to create numpy ndarray from numpy ndarrays?

I used the MNIST dataset for training a neural network, where the training data is returned as a tuple with two entries. The first entry contains the actual training images. This is a numpy ndarray with 50,000 entries. Each entry is, in turn, a numpy ndarray with 784 values, representing the 28 * 28 = 784 pixels in a single MNIST image.
I would like to create a new training set, however I do not know how to create an ndarray from other ndarrays. For instance, if I have the following two ndarrays:
a = np.ndarray((3,1), buffer=np.array([0.9,1.0,1.0]), dtype=float)
b = np.ndarray((3,1), buffer=np.array([0.8,1.0,1.0]), dtype=float)
how to make a third one containing these two?
I tried the following but it creates only one entry.
c = np.ndarray((1,6,1), buffer=np.array(([a],[b])), dtype=float)
I would need it to be two entries.

Thanks, in the meanwhile I figured out it is simply:
c = np.array((a, b))