I want to build a neural network using neupy.
Therefore I consturcted the following architecture:
network = layers.join(
layers.Input(10),
layers.Linear(500),
layers.Relu(),
layers.Linear(300),
layers.Relu(),
layers.Linear(10),
layers.Softmax(),
)
My data is shaped as follwoing:
x_train.shape = (32589,10)
y_train.shape = (32589,1)
When I try to train this network using:
model.train(x_train, y_trian)
I get the follwoing error:
ValueError: Input dimension mis-match. (input[0].shape[1] = 10, input[1].shape[1] = 1)
Apply node that caused the error: Elemwise{sub,no_inplace}(SoftmaxWithBias.0, algo:network/var:network-output)
Toposort index: 26
Inputs types: [TensorType(float64, matrix), TensorType(float64, matrix)]
Inputs shapes: [(32589, 10), (32589, 1)]
Inputs strides: [(80, 8), (8, 8)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Elemwise{Composite{((i0 * i1) / i2)}}(TensorConstant{(1, 1) of 2.0}, Elemwise{sub,no_inplace}.0, Elemwise{mul,no_inplace}.0), Elemwise{Sqr}[(0, 0)](Elemwise{sub,no_inplace}.0)]]
How do I have to edit my network to map this kind of data?
Thank you a lot!
Your architecture has 10 outputs instead of 1. I assume that your y_train function is a 0-1 class identifier. If so, than you need to change your structure to this:
network = layers.join(
layers.Input(10),
layers.Linear(500),
layers.Relu(),
layers.Linear(300),
layers.Relu(),
layers.Linear(1), # Single output
layers.Sigmoid(), # Sigmoid works better for 2-class classification
)
You can make it even simpler
network = layers.join(
layers.Input(10),
layers.Relu(500),
layers.Relu(300),
layers.Sigmoid(1),
)
The reason why it works is because layers.Liner(10) > layers.Relu() is the same as layers.Relu(10). You can learn more in official documentation: http://neupy.com/docs/layers/basics.html#mutlilayer-perceptron-mlp
Related
I know this question has been asked a bunch of times but I just can't resolve this error using any of the answers at SO. I am trying to build embeddings and my data is shaped: (20, 7, 12) i.e. 20 training samples, that have 7 words each, with one-hot encoded to 12 dimensions.
When I fit my model using the below specs, I get the error:
Error when checking input: expected embedding_24_input to have 2
dimensions, but got array with shape (20, 7, 12)
embedding_dims = 10
model = Sequential()
model.add(Embedding(12, embedding_dims,input_length=7))
I then tried to Flatten before Embedding, but that failed complaining that "input_length" is 7, but received input has shape (None, 84)". I then changed the input_length on the embedding layer to match that but no luck with that either:
Error when checking target: expected embedding_26 to have 3
dimensions, but got array with shape (20, 12)
model = Sequential()
model.add(Flatten())
model.add(Embedding(12, embedding_dims,input_length=84))
I would really appreciate any help with some explanation, please!
Embedding layer don't expect the one hot encoding in input data, you should using function like numpy.argmax or tf.argmax to convert data to integer matrix of size (batch, input_length) before feed data in Embedding layer.
Following example working without any error:
import tensorflow as tf
import numpy as np
embedding_dims = 10
batch_size = 20
vocabulary_size = 12
# The model will take as input an integer matrix of size (batch, input_length),
# and the largest integer (i.e. word index) in the input should be no larger than 11 (vocabulary size - 1)
model = tf.keras.Sequential()
model.add(tf.keras.layers.Embedding(vocabulary_size, embedding_dims, input_length=7))
model.compile('rmsprop', 'mse')
raw_input_array = np.random.randint(2, size=(batch_size, 7, vocabulary_size))
input_array = np.argmax(raw_input_array, axis=-1)
output_array = model.predict(input_array)
print("One hot encodeing data shape: {}\ninput integer matrix shape: {}\noutput shape: {}\n".format(raw_input_array.shape, input_array.shape, output_array.shape))
Outputs:
One hot encodeing data shape: (20, 7, 12)
input integer matrix shape: (20, 7)
output shape: (20, 7, 10)
Using PyTorch, I have an ANN model (for a classification task) below:
import torch
import torch.nn as nn
# Setting up artifical neural net model which separates out categorical
# from continuous features, so that embedding could be applied to
# categorical features
class TabularModel(nn.Module):
# Initialize parameters embeds, emb_drop, bn_cont and layers
def __init__(self, emb_szs, n_cont, out_sz, layers, p=0.5):
super().__init__()
self.embeds = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in emb_szs])
self.emb_drop = nn.Dropout(p)
self.bn_cont = nn.BatchNorm1d(n_cont)
# Create empty list for each layer in the neural net
layerlist = []
# Number of all embedded columns for categorical features
n_emb = sum((nf for ni, nf in emb_szs))
# Number of inputs for each layer
n_in = n_emb + n_cont
for i in layers:
# Set the linear function for the weights and biases, wX + b
layerlist.append(nn.Linear(n_in, i))
# Using ReLu activation function
layerlist.append(nn.ReLU(inplace=True))
# Normalised all the activation function output values
layerlist.append(nn.BatchNorm1d(i))
# Set some of the normalised activation function output values to zero
layerlist.append(nn.Dropout(p))
# Reassign number of inputs for the next layer
n_in = i
# Append last layer
layerlist.append(nn.Linear(layers[-1], out_sz))
# Create sequential layers
self.layers = nn.Sequential(*layerlist)
# Function for feedforward
def forward(self, x_cat_cont):
x_cat = x_cat_cont[:,0:cat_train.shape[1]].type(torch.int64)
x_cont = x_cat_cont[:,cat_train.shape[1]:].type(torch.float32)
# Create empty list for embedded categorical features
embeddings = []
# Embed categorical features
for i, e in enumerate(self.embeds):
embeddings.append(e(x_cat[:,i]))
# Concatenate embedded categorical features
x = torch.cat(embeddings, 1)
# Apply dropout rates to categorical features
x = self.emb_drop(x)
# Batch normalize continuous features
x_cont = self.bn_cont(x_cont)
# Concatenate categorical and continuous features
x = torch.cat([x, x_cont], 1)
# Feed categorical and continuous features into neural net layers
x = self.layers(x)
return x
I am trying to use this model with skorch's GridSearchCV, as below:
from skorch import NeuralNetBinaryClassifier
# Random seed chosen to ensure results are reproducible by using the same
# initial random weights and biases, and applying dropout rates to the same
# random embedded categorical features and neurons in the hidden layers
torch.manual_seed(0)
net = NeuralNetBinaryClassifier(module=TabularModel,
module__emb_szs=emb_szs,
module__n_cont=con_train.shape[1],
module__out_sz=2,
module__layers=[30],
module__p=0.0,
criterion=nn.CrossEntropyLoss,
criterion__weight=cls_wgt,
optimizer=torch.optim.Adam,
optimizer__lr=0.001,
max_epochs=150,
device='cuda'
)
from sklearn.model_selection import GridSearchCV
param_grid = {'module__layers': [[30], [50,20]],
'module__p': [0.0, 0.2, 0.4],
'max_epochs': [150, 175, 200, 225]
}
models = GridSearchCV(net, param_grid, scoring='roc_auc').fit(cat_con_train.cpu(), y_train.cpu())
models.best_params_
But when I ran the code, I am getting this error message below:
/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_validation.py:536: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details:
ValueError: Expected module output to have shape (n,) or (n, 1), got (128, 2) instead
FitFailedWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_validation.py:536: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details:
ValueError: Expected module output to have shape (n,) or (n, 1), got (128, 2) instead
FitFailedWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_validation.py:536: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details:
ValueError: Expected module output to have shape (n,) or (n, 1), got (128, 2) instead
FitFailedWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_validation.py:536: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details:
ValueError: Expected module output to have shape (n,) or (n, 1), got (128, 2) instead
FitFailedWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_validation.py:536: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details:
ValueError: Expected module output to have shape (n,) or (n, 1), got (128, 2) instead
FitFailedWarning)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-86-c408d65e2435> in <module>()
98
---> 99 models = GridSearchCV(net, param_grid, scoring='roc_auc').fit(cat_con_train.cpu(), y_train.cpu())
100
101 models.best_params_
11 frames
/usr/local/lib/python3.6/dist-packages/skorch/classifier.py in infer(self, x, **fit_params)
303 raise ValueError(
304 "Expected module output to have shape (n,) or "
--> 305 "(n, 1), got {} instead".format(tuple(y_infer.shape)))
306
307 y_infer = y_infer.reshape(-1)
ValueError: Expected module output to have shape (n,) or (n, 1), got (128, 2) instead
I am not sure what is wrong or how to fix this. Any help on this would really be appreciated.
Many thanks in advance!
To quote ptrblck on the pytorch forum who outlined the solution already:
I guess the NeuralNetBinaryClassifier expects the output to have one logit, since it’s used for a binary use case.
If you want to use two output units for a binary classification (which would be a multi-class classification with 2 classes), you would have to use another wrapper I guess.
I’m not deeply familiar with skorch, but think that NeuralNetClassifier might work.
His assesment was correct. skorch's NeuralNetBinaryClassifier expects the y to have one dimension, so a shape of (x, 1) or (x,) where the value of y is either 0 or 1. So a valid y would be:
y = torch.tensor([0, 1, 0]) # shape is (3,)
y = torch.tensor([[0],[1],[0]) # shape is (3, 1)
I try to determine the confusion_matrix of my neural network model which
is written in python by using google tensorflow.
By using this piece of code:
cm = tf.zeros(shape=[2,2], dtype=tf.int32)
for i in range(0, validation_data.shape[0], batch_size_validation):
batched_val_data = np.array(validation_data[i:i+batch_size_validation, :, :], dtype='float')
batched_val_labels = np.array(validation_labels[i:i+batch_size_validation, :], dtype='float')
batched_val_data = batched_val_data.reshape((-1, n_chunks, chunk_size))
_acc, _c, _p = sess.run([accuracy, correct, pred], feed_dict=({x:batched_val_data, y:batched_val_labels}))
#batched_val_labels.shape ==> (2048, 2)
#_p.shape ==> (2048, 2)
#this piece of code throws the error!
cm = tf.confusion_matrix(labels=batched_val_labels, predictions=_p)
I get the following error:
ValueError: Shape (2, 2048, 2) must have rank 2
At least you should know that the array for the validation labels batched_val_labels is an one hot array.
Can someone help me pls? Thanks in advance!
The problem was that I am using an one hot array.
By following this instruction: Tensorflow confusion matrix using one-hot code
I changed this piece of code:
cm = tf.math.confusion_matrix(labels=batched_val_labels, predictions=_p)
into:
cm = tf.math.confusion_matrix(labels=tf.argmax(batched_val_labels, 1), predictions=tf.argmax(_p, 1))
When I run my code, I get a value error with the following message:
ValueError: Input dimension mis-match. (input[0].shape[1] = 1, input[2].shape[1] = 20)
Apply node that caused the error: Elemwise{Composite{((i0 + i1) - i2)}}[(0, 0)](Dot22.0, InplaceDimShuffle{x,0}.0, InplaceDimShuffle{x,0}.0)
Toposort index: 18
Inputs types: [TensorType(float64, matrix), TensorType(float64, row), TensorType(float64, row)]
Inputs shapes: [(20, 1), (1, 1), (1, 20)]
Inputs strides: [(8, 8), (8, 8), (160, 8)]
Inputs values: ['not shown', array([[ 0.]]), 'not shown']
Outputs clients: [[Elemwise{Composite{((i0 * i1) / i2)}}(TensorConstant{(1, 1) of 2.0}, Elemwise{Composite{((i0 + i1) - i2)}}[(0, 0)].0, Elemwise{mul,no_inplace}.0), Elemwise{Sqr}[(0, 0)](Elemwise{Composite{((i0 + i1) - i2)}}[(0, 0)].0)]]
My training data is a matrix of entries such as ..
[ 815.257786 320.447 310.841]
And the batches I'm inputting to my training function have a shape of (BATCH_SIZE, 3) and type TensorType(float64, matrix)
My neural net is very simple:
self.inpt = T.dmatrix('inpt')
self.out = T.dvector('out')
self.network_in = nnet.layers.InputLayer(shape=(BATCH_SIZE, 3), input_var=self.inpt)
self.l0 = nnet.layers.DenseLayer(self.network_in, num_units=40,
nonlinearity=nnet.nonlinearities.rectify,
)
self.network = nnet.layers.DenseLayer(self.l0, num_units=1,
nonlinearity=nnet.nonlinearities.linear
)
My loss function is:
pred = nnet.layers.get_output(self.network)
loss = nnet.objectives.squared_error(pred, self.out)
loss = loss.mean()
I'm a bit confused as to why I'm getting a dimension mismatch. I'm passing in the correct input and label types (as per my symbolic variables), and the shape of my input data corresponds to the expected 'shape' parameter that I'm giving my InputLayer. I believe it's a problem with how I'm specifying the batch size, as when I use a batch size of 1 then my network can train without any problem, and the input[2].shape[1] value from the error message is my batch size. I'm quite new to machine learning, and any help would be greatly appreciated!
Turns out the problem was that my labels had the wrong dimensionality.
My data had shapes:
x_train.shape == (batch_size, 3)
y_train.shape == (batch_size,)
And the symbolic inputs to my net were:
self.inpt = T.dmatrix('inpt')
self.out = T.dvector('out')
I was able to solve my problem by reshaping y_train. I then changed the symbolic output variable to a matrix to account for these changes.
y_train = np.reshape(y_train, y_train.shape + (1,))
# y_train.shape == (batch_size, 1)
self.out = T.dmatrix('out')
I'm trying to use Theano's autoencoders to discover context-specific features from two different types of data.
The first type has 13 features and the second one has 60.
n_ins=[13,60],
n_hiddens=[20, 20, 20],
Both have their own independent stack of autoencoders.
I merge the outputs of the topmost layers and feed these into a regression layer for supervised training.
self.logLayer = LogisticRegression(
input=(self.sigmoid_layers[0][-1].output+self.sigmoid_layers[1][-1].output),
n_in=self.n_modes*n_hiddens[-1],
n_out=n_outs
)
Pre-training for each context seems to work correctly, however I hit a snag during finetuning using the standard training function in the tutorials.
train_fn = theano.function(
inputs=[index],
outputs=self.finetune_cost,
updates=updates,
givens={
self.x: train_set_x[
index * batch_size: (index + 1) * batch_size
],
self.y: train_set_y[
index * batch_size: (index + 1) * batch_size
]
},
name='train'
)
I get the following error:
ValueError: dimension mismatch in args to gemm (5,73)x(13,20)->(5,20)
Apply node that caused the error: GpuDot22(GpuSubtensor{int64:int64:}.0, W)
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix)]
Inputs shapes: [(5, 73), (13, 20)]
Inputs strides: [(73, 1), (20, 1)]
Inputs values: ['not shown', 'not shown']
I believe this has something to do with how the Theano nodes are processed during training. Seems like the training batch (5, 73), is being applied straight into the output nodes separately starting with the 1st context (13, 20).