Develop Reference

NLP Keras - Dimension of Embedding and Global Average Pooling Layers

I am trying to trace the calculation of Tensorflow’s NLP neural network pipeline below.
embedding_dim=16
model = Sequential([
vectorize_layer,
Embedding(vocab_size, embedding_dim, name="embedding"),
GlobalAveragePooling1D(),
Dense(16, activation='relu'),
Dense(1)
])
I am a little bit confused on the dimensions of the embedding and global average pooling layers.
For example, if I have a sample sentence vector [‘I’, ‘like’, ‘Chinese’, ‘food’]. The embedding layer will expand this vector from 4x1 dimensions to a matrix of 4x16 dimensions right? Then the global average pooling 1D will take an average of each feature, and return a vector of 16x1, where each value represents an average feature value of my sentence, am I correct?

Understanding of Basic Neural Network Structure

Let's say I want to code this basic Neural Network Structure in Keras which has 10 units in Input Layer and 3 units in Output layer.
Now if I am using Keras, and give input_shape of more then 10, how it will adjust in it.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
model = Sequential()
model.add(Dense(10, activation = 'relu', input_shape = (64,)))
model.add(Dense(3, activation = 'sigmoid'))
model.summary()
You see, here input_shape is of size 64, but how will it adjust in model whose first layer has 10 units because for what I have learned that size of input shape/vector should be equal to number of units in the input layer.
Or Am I not implementing this neural network right?

That would not be a problem. The weight matrix of shape (10,64) would be used in input layer. your input has shape 64 and first hidden layer has 10 units giving a output of 3 units. Seems fine to me.
But your input layer itself is 64. So what you are getting is a 3-layer network with a hidden layer of 10 units.

If the shape of your input vector is 64, then you really need to have an input layer with size 64. The input layer of a neural network doesn't perform any computations. It just passes the inputs forward to the first hidden layer. This one, on the other hand, performs the computations for all neurons contained in it (linear combination of input vector and weights, later served as an input to the activation function, which is the ReLU in your case).
In your code, you are building a neural net with 64 input neurons (which again don't perform any computations), 10 neurons in the first (and only) hidden layer and 3 neurons in the output layer.

Replicated Convolutional Bi-Directional LSTM implementation in Keras diverging

This is the model I am trying to replicate (more information in linked paper):
In our models, we adopted one dropout layer between LSTM models and the first fully-connected layer and another dropout layer between the first fully-connected layer and the second fully-connected layer. Their masking probabilities are both set to 0.5.
...
For our proposed CBLSTM, one-layer CNN is firstly designed, whose filter number, filter size and pooling size are set to 150, 10 and 5. Therefore, the shape of the raw sensory sequence is changed from 100 x 12 to 19 x 150 after CNN. Then, a two-layer bi-directional LSTM is built on top of the CNN.
Backward and forward LSTMs share the same layer sizes as [150, 200]. Therefore, the output of the LSTM module is the concatenated vector of the representations learned by backward and forward LSTMs, and its dimensionality is 400. Then, before feeding the representation into the linear regression layer, two fully-connected layers with a size of [500, 600] are adopted. The nonlinearity activation functions in our proposed CBLSTM are all set to ReLu.
Source: Zhao, R., Yan, R., Wang, J., & Mao, K. (2017). Learning to monitor machine health with convolutional bi-directional LSTM networks. Sensors, 17(2), 273. link to paper
The input is 630 samples x 100 timesteps x 12 features.
How my model looks at the moment:
model = Sequential()
model.add(Conv1D(filters=150, kernel_size=10, activation='relu', input_shape=(100,12)))
model.add(MaxPooling1D(pool_size=5, strides=None, padding='valid'))
model.add(Bidirectional(LSTM(150, return_sequences=True), merge_mode='concat'))
model.add(Bidirectional(LSTM(200, return_sequences=False), merge_mode='concat'))
model.add(Dropout(0.5))
model.add(Dense(500, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(600, activation='relu'))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['mae'])
While the training loss steadily decreases per epoch, the validation set does not and diverges pretty quickly. This indicates that there is a mistake in my model which I have not yet been able to find. Any ideas as to what is wrong?
Side note: I am using the same data as input as the authors.

How to use Conv1D and Bidirectional LSTM in keras to do multiclass classification of each timestep?

I am trying to use a Conv1D and Bidirectional LSTM in keras (much like in this question) for signal processing, but doing a multiclass classification of each time step.
The problem is that even though the shapes used by Conv1D and LSTM are somewhat equivalent:
Conv1D: (batch, length, channels)
LSTM: (batch, timeSteps, features)
The output of the Conv1D is = (length - (kernel_size - 1)/strides), and therefore doesn't match the LSTM shape anymore, even without using MaxPooling1D and Dropout.
To be more specific, my training set X has n samples with 1000 time steps and one channel (n_samples, 1000, 1), and I used LabelEncoder and OneHotEncoder so y has n samples, 1000 time steps and 5 one hot encoded classes (n_samples, 1000, 5).
Since one class is much more prevalent than the others (is actually the absence of signal), I am using loss='sparse_categorical_crossentropy', sample_weight_mode="temporal" and sample_weight to give a higher weight to time steps containing meaningful classes.
model = Sequential()
model.add(Conv1D(128, 3, strides=1, input_shape = (1000, 1), activation = 'relu'))
model.add(Bidirectional(LSTM(128, return_sequences=True)))
model.add(TimeDistributed(Dense(5, activation='softmax')))
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['categorical_accuracy'], sample_weight_mode="temporal")
print(model.summary())
Model
When I try to fit the model I get this error message:
Error when checking target: expected time_distributed_1 to have shape
(None, 998, 1) but got array with shape (100, 1000, 5).
Is there a way to make such a neural network configuration work?

Your convolution is cutting the tips of the sequence. Use padding='same' in the convolutional layers.
The message, though, seems not to fit your model. Your model clearly has 5 output features (because of Dense(5)), but the massage says it expects 1. Maybe this is happening because of "sparse" crossentropy. You should probably, by the format of your data, use a "categorical_crossentropy".

Keras word embedding in four gram model

I am following coursera neural network class and I am trying to pass the assignments using python+keras instead of octave.
I want to predict the fourth word given the previous three ones. My input documents total 250 unique words.
The model should have an embedding layer that maps each word to a 50-d vector space, a hidden layer with 200 neurons with sigmoid activation function and an output layer of 250 units scoring the probability of the forth word to be equal to those in my vocabulary through a softmax activation.
I am having troubles with dimensions. Here is my code:
from keras.models import Sequential
from keras.layers import Dense, Activation, Embedding
model = Sequential([Embedding(250,50),
Dense(200, activation='sigmoid'),
Dense(250, activation='softmax')
])
model.compile(optimizer='rmsprop',
loss='categorical_crossentropy',
metrics=['accuracy'])
Yet I never get to compile the model since I am encountering the following error:
Exception: Input 0 is incompatible with layer dense_1: expected ndim=2, found ndim=3
Any hint will be much appreciated. Thanks in advance

From https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html
"All that the Embedding layer does is to map the integer inputs to the vectors found at the corresponding index in the embedding matrix, i.e. the sequence [1, 2] would be converted to [embeddings[1], embeddings[2]]. This means that the output of the Embedding layer will be a 3D tensor of shape (samples, sequence_length, embedding_dim)."
Your embedding layer outputs 3 dimension vectors while the dense layers expects 2 dim vecs.
You can follow the links tutorial and with some mods it will fit your problem.