Develop Reference

Pytorch - extracting predicted values to an array

I'm a new pytorch user and moderate experience with Tensorflow/Keras. The pytorch examples are fantastic. I've worked through the demand forecasting lab using the Temporal Fusion Transform (https://pytorch-forecasting.readthedocs.io/en/latest/tutorials/stallion.html).
All makes sense but haven't figured out how to save the predicted values in notebook section #20 to a numpy array.
Section #20,
*new_raw_predictions, new_x = best_tft.predict(new_prediction_data, mode="raw", return_x=True)*
I see the values in the tensors, print(new_raw_predictions) ,
like this --
*{'prediction': tensor([[[3.4951e+00, 1.7341e+01, 2.7446e+01, ..., 6.3175e+01,
9.0240e+01, 1.2589e+02],
[1.1698e+01, 2.3643e+01, 3.3291e+01, ..., 6.6374e+01,
9.1148e+01, 1.3173e+02],
I've seen some similar questions asked here but none seem to work. All attempts result in a similar error so I'm missing something fundamental about pytorch and the output tensor; I always get 'AttributeError: 'dict' object has no attribute new_raw_predictions'
A few examples of what's been tried:
*new_raw_predictions.cpu().numpy() new_raw_predictions.detach().cpu().numpy() new_raw_predictions.numpy()*
Goal is to save the predicted output so I can compare changes to the model. Thanks in advance!

It all depends on how you've created your model, because pytorch can return values however you specify. In your case, it looks like it returns a dictionary, of which 'prediction' is a key. You can convert to numpy using the command you supplied above, but with one change:
preds = new_raw_predictions['prediction'].detach().cpu().numpy()
of course if it's not on the GPU you don't need to use .detach().cpu(), just .numpy()

Tensorflow Datasets, padded_batch, why allow different output_shapes, and is there a better way?

I'm trying to write Tensorflow 2.0 code which is good enough to share with other people. I have run into a problem with tf.data.Dataset. I have solved it, but I dislike my solutions.
Here is working Python code which generates padded batches from irregular data, two different ways. In one case, I re-use a global variable to supply the shape information. I dislike the global variable, especially because I know that the Dataset knows its own output shapes, and in the future I may have Dataset objects with several different output shapes.
In the other case, I extract the shape information from the Dataset object itself. But I have to jump through hoops to do it.
import numpy as np
import tensorflow as tf
print("""
Create a data set with the desired shape: 1 input per sub-element,
3 targets per sub-element, 8 elements of varying lengths.
""")
def gen():
lengths = np.tile(np.arange(4,8), 2)
np.random.shuffle(lengths)
for length in lengths:
inp = np.random.randint(1, 51, length)
tgt = np.random.random((length, 3))
yield inp, tgt
output_types = (tf.int64, tf.float64)
output_shapes = ([None], [None, 3])
dataset = tf.data.Dataset.from_generator(gen, output_types, output_shapes)
print("""
Using the global variable, output_shapes, allows the retrieval
of padded batches.
""")
for inp, tgt in dataset.padded_batch(3, output_shapes):
print(inp)
print(tgt)
print()
print("""
Obtaining the shapes supplied to Dataset.from_generator()
is possible, but hard.
""")
default_shapes = tuple([[y.value for y in x.shape.dims] for x in dataset.element_spec]) # Crazy!
for inp, tgt in dataset.padded_batch(3, default_shapes):
print(inp)
print(tgt)
I don't quite understand why one might want to pad the data in a batch of unevenly-sized elements to any shapes other than the output shapes which were used to define the Dataset elements in the first place. Does anyone know of a use case?
Also, there is no default value for the padded_shapes argument. I show how to retrieve what I think is the sensible default value for padded_shapes. That one-liner works... but why is it so difficult?
I'm currently trying to subclass Dataset to provide the Dataset default shapes as a Python property. Tensorflow is fighting me, probably because the underlying Dataset is a C++ object while I'm working in Python.
All this trouble makes me wonder whether there is a cleaner approach than what I have tried.
Thanks for your suggestions.

Answering my own question. I asked this same question on Reddit. A Tensorflow contributor replied that TF 2.2 will provide a default value for the padded_shapes argument. I am glad to see that the development team has recognized the same need that I identified.

How to apply the torch.inverse() function of PyTorch to every sample in the batch?

This may seem like a basic question, but I am unable to work it through.
In the forward pass of my neural network, I have an output tensor of shape 8x3x3, where 8 is my batch size. We can assume each 3x3 tensor to be a non-singular matrix. I need to find the inverse of these matrices.
The PyTorch inverse() function only works on square matrices. Since I now have 8x3x3, how do I apply this function to every matrix in the batch in a differentiable manner?
If I iterate through the samples and append the inverses to a python list, which I then convert to a PyTorch tensor, should it be a problem during backprop? (I am asking since converting PyTorch tensors to numpy to perform some operations and then back to a tensor won't compute gradients during backprop for such operations)
I also get the following error when I try to do something like that.
a = torch.arange(0,8).view(-1,2,2)
b = [m.inverse() for m in a]
c = torch.FloatTensor(b)
TypeError: 'torch.FloatTensor' object does not support indexing

EDIT:
As of Pytorch version 1.0, torch.inverse now supports batches of tensors. See here. So you can simply use the built-in function torch.inverse
OLD ANSWER
There are plans to implement batched inverse soon. For discussion, see for example issue 7500 or issue 9102. However, as of the time of writing, the current stable version (0.4.1), no batch inverse operation is available.
Having said that, recently batch support for torch.gesv was added. This can be (ab)used to define your own batched inverse operation along the following lines:
def b_inv(b_mat):
eye = b_mat.new_ones(b_mat.size(-1)).diag().expand_as(b_mat)
b_inv, _ = torch.gesv(eye, b_mat)
return b_inv
I found that this gives good speed-ups over a for loop when running on GPU.

You could split the tensor using torch.functional.unbind(), apply inverse to every element of the result, and then stack back:
a = torch.arange(0,8).view(-1,2,2)
b = [t.inverse() for t in torch.functional.unbind(a)]
c = torch.functional.stack(b)

CNTK & python: How to pass input data to the eval func?

With CNTK I have created a network with 2 input neurons and 1 output neuron.
A line in the training file looks like
|features 1.567518 2.609619 |labels 1.000000
Then the network was trained with brain script. Now I want to use the network for predicting values. For example: Input data is [1.82, 3.57]. What ist the output from the net?
I have tried Python with the following code, but here I am new. Code does not work. So my question is: How to pass the input data [1.82, 3.57] to the eval function?
On stackoverflow there are some hints, here and here, but this is too abstract for me.
Thank you.
import cntk as ct
import numpy as np
z = ct.load_model("LR_reg.dnn", ct.device.cpu())
input_data= np.array([1.82, 3.57], dtype=np.float32)
pred = z.eval({ z.arguments[0] : input_data })
print(pred)

Here's the most defensive way of doing it. CNTK can be forgiving if you omit some of this when the network is specified with V2 constructs. Not sure about a network that was created with V1 code.
Basically you need a pair of braces for each axis. Which axes exist in Brainscript? There's a batch axis, a sequence axis and then the static axes of your network. You have one dimensional data so that means the following should work:
input_data= np.array([[[1.82, 3.57]]], dtype=np.float32)
This specifies a batch of one sequence, of length one, containing one 1d vector of two elements. You can also try omitting the outermost braces and see if you are getting the same result.
Update based on more information from the comment below, we should not forget that the V1 code also saved the part of the network that computes things like loss and accuracy. If we provide only the features, CNTK will complain that the labels have not been provided. There are two ways to deal with this issue. One possibility is to provide some fake labels, so that the network can evaluate these auxiliary operations. Another possibility is to identify the prediction and use that. If the prediction was called 'p' in V1, this python code
p = z.find_by_name('p')
should create a CNTK function that only needs the features in order to compute the prediction.

Pymc3: Optimizing parameters with multiple data?

I've designed a model using Pymc3, and I have some trouble optimizing it with multiple data.
The model is a bit similar to the coal-mining disaster (as in the Pymc3 tutorial for those who know it), except there are multiple switchpoints.
The output of the network is a serie of real numbers for instance:
[151,152,150,20,19,18,0,0,0]
with Model() as accrochage_model:
time=np.linspace(0,n_cycles*data_length,n_cycles*data_length)
poisson = [Normal('poisson_0',5,1), Normal('poisson_1',10,1)]
variance=3
t = [Normal('t_0',0.5,0.01), Normal('t_1',0.7,0.01)]
taux = [Bernoulli('taux_{}'.format(i),t[i]) for i in range(n_peaks)]
switchpoint = [Poisson('switchpoint_{}'.format(i),poisson[i])*taux[i] for i in range(n_peaks)]
peak=[Normal('peak_0',150,2),Normal('peak_1',50,2),Normal('peak_2',0,2)]
z_init=switch(switchpoint[0]>=time%n_cycles,0,peak[0])
z_list=[switch(sum(switchpoint[j] for j in range(i))>=time%n_cycles,0,peak[i]-peak[i-1]) for i in range(1,n_peaks)]
z=(sum(z_list[i] for i in range(len(z_list))))
z+=z_init
m =Normal('m', z, variance,observed=data)
I have multiple realisations of the true distribution and I'd like taking all of them into account while performing optimization of the parameters of the system.
Right now my "data" that appears in observed=data is just one list of results , such as:
[151,152,150,20,19,18,0,0,0]
What I would like to do is give not just one but several lists of results,
for instance:
data=([151,152,150,20,19,18,0,0,0],[145,152,150,21,17,19,1,0,0],[151,149,153,17,19,18,0,0,1])
I tried using the shape parameter and making data an array of results but none of it seemed to work.
Does anyone have an idea of how it's possible to do the inference so that the network is optimized for an entire dataset and not a single output?