Develop Reference

Adding additional hidden layer and attention layer to LSTM model

sequence_input = Input(shape=(MAX_LENGTH_SEQUENCE,), dtype='int32')
embedded_sequences = embedding_layer(sequence_input)
l_lstm = Bidirectional(LSTM(10))(embedded_sequences)
preds = Dense(len(macronum), activation='softmax')(l_lstm)
model = Model(sequence_input, preds)
model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['acc'])
I need to add additional hidden layer and an attention layer to the above LSTM model , usually i construct the model in this way:
model = tensorflow.keras.Sequential()
model.add(tensorflow.keras.layers.LSTM(128, dropout=0.3,
recurrent_dropout=0.2,input_shape=(N, K), return_sequences=True))
#model.add(tensorflow.keras.layers.LSTM(128, dropout=0.3,
recurrent_dropout=0.2,input_shape=(N, K), return_sequences=True))
model.add(Attention(name='attention_weight'))
model.add(tensorflow.keras.layers.Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())
print(_x_train.shape)
model.fit(_x_train, _y_train, epochs=10, batch_size=1)
scores = model.evaluate(_x_test, _y_test, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))
as shown in the second code i normaly stack and add as many layer as i need using
model.add
but for the first code i'm not familiar with this approach,
if i want to add extra LSTM layer and Attention layer to the first code, where should i include them inside the code? what is the right syntax ?

getting incompatible shape error in tensorflow functional model

I am trying to implement a deep model using tensorflow.keras which contains an embedding layer + Conv1D + 2 BiLstm layers. This is the implementation in sequential mode:
model = models.Sequential()
model.add(layers.Embedding(vocab_size, embedding_dim, weights=[embedding_matrix], input_length=limit_on_length, trainable=False))
model.add(layers.Conv1D(50, 4, padding='same', activation='relu'))
model.add(layers.Bidirectional(layers.LSTM(units=200, return_sequences=True, dropout=0.2, recurrent_dropout=0.2)))
model.add(layers.Bidirectional(layers.LSTM(units=200, return_sequences=False,dropout=0.2, recurrent_dropout=0.2)))
model.add(layers.Dense(len(set(tr_intents)), activation='softmax'))
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
And I fit the model like this:
model.fit(train_padded, to_categorical(tr_intents), epochs=15, batch_size=32, validation_split=0.1)
Everything goes on very good in this sequential mode. But when I implement the model in functional mode, I get this kind of error:
ValueError: Shapes (32, 22) and (32, 11, 22) are incompatible
And here is my implementation in functional structure:
input_layer = layers.Input(shape=(None,))
x = layers.Embedding(vocab_size, embedding_dim, weights=[embedding_matrix], input_length=limit_on_length, trainable=False)(input_layer)
x = layers.Conv1D(50, 4, padding='same', activation='relu')(x)
x = layers.Bidirectional(layers.LSTM(units=200, return_sequences=True, dropout=0.2, recurrent_dropout=0.2))(x)
x = layers.Bidirectional(layers.LSTM(units=200, return_sequences=True, dropout=0.2, recurrent_dropout=0.2))(x)
intents_out = layers.Dense(n_intents, activation='softmax')(x)
model = models.Model(input_layer, intents_out)
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
Can anybody help me with this error? I need to implement the model in functional mode cause I have to add one more output layer to it.
Number of my intents(or labels) is 22 and each sentence has length of 11.

text binary classification error:logits and labels must have the same shape

I'm trying to bulid a model to classify my text to hate (1) or not (0) using nn.
Information about the data, it's consists of tweets and class label (hate (1) or not (0)):
sentences = df['comment']
y = df['isHate']
sentences_train, sentences_test, train_y, test_y = train_test_split(sentences, y, test_size=0.25, random_state=42)
the text get through a lot of Word Embeddings and I applied pad sequences on the tweets and LabelEncoder on the labels.
the problem is when I do the run I get this error:
ValueError: logits and labels must have the same shape ((None, 1) vs (None, 2))
the code of the model:
emb_dim = 16
model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim= emb_dim, input_length=maxlen))
model.add(Flatten())
model.add(Dense(2, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
model.summary()
the problem happened in this part:
history = model.fit(X_train, y_train,
batch_size=32,
epochs=15,
validation_data=(X_test, y_test))
Any help?

In your code:
model.add(Dense(1, activation='sigmoid'))
Your last dense layer has only 1 unit but your labels are one hot encoded which consist of 2 classes. So you need to change:
model.add(Dense(2, activation='softmax'))
You also need to change your loss function, because they are one-hot-encoded:
loss='categorical_crossentropy'

How to build a 1D CNN

I am trying to use a CNN for classification. My training data is shown in the picture below and has 9923 pieces of data with each piece containing 1k numeric values.
My current model has only around 10 percent accuracy and I am wondering if anyone knows if I am doing something wrong.
model = Sequential()
model.add(Conv1D(64,3, activation ='relu', input_shape= (1000, 1)))
model.add(MaxPooling1D(2))
model.add(Conv1D(64,3, activation ='relu'))
model.add(MaxPooling1D(pool_size=(2)))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(28, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X, Y, epochs = 30, validation_split = 0.1)

Identical results with different model parameters

I'm making a neural network model to assign sentiments to phrases. My problem is no matter what I change in my model, the accuracy score is always identical.
I have tried adding or removing layers, changing batch size, changing number of epochs, changing activators etc.
tokenizer = Tokenizer()
total_phrases = phrases_train.append([phrases_valid, phrases_test],
ignore_index = True)
tokenizer.fit_on_texts(total_phrases)
maxlen = max([len(s.split()) for s in total_phrases])
vocab_size = len(tokenizer.word_index) + 1
phrases_train_tokens = tokenizer.texts_to_sequences(phrases_train)
phrases_valid_tokens = tokenizer.texts_to_sequences(phrases_valid)
phrases_test_tokens = tokenizer.texts_to_sequences(phrases_test)
pad_phrases_train = pad_sequences(phrases_train_tokens, maxlen=maxlen,
padding='post')
pad_phrases_valid = pad_sequences(phrases_valid_tokens, maxlen=maxlen,
padding='post')
pad_phrases_test = pad_sequences(phrases_valid_tokens, maxlen=maxlen,
padding='post')
model = Sequential()
model.add(Embedding(vocab_size, 100, input_length=maxlen))
model.add(Dense(50, kernel_initializer='truncated_normal',
activation='relu'))
model.add(GRU(units=32, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=
['accuracy'])
model.fit(pad_phrases_train, sents_train, batch_size=250, epochs=50,
validation_data=(pad_phrases_valid, sents_valid), verbose=2)
model.evaluate(pad_phrases_test, sents_test)
Evaluate always shows exactly 0.2057 score. I want to achieve around 50% accuracy