Develop Reference

Keras model with fasttext word embedding

I am trying to learn a language model to predict the last word of a sentence given all the previous words using keras. I would like to embed my inputs using a learned fasttext embedding model.
I managed to preprocess my text data and embed the using fasttext. My training data is comprised of sentences of 40 tokens each. I created 2 np arrays, X and y as inputs, with y what I want to predict.
X is of shape (44317, 39, 300) with 44317 the number of example sentences, 39 the number of tokens in each sentence, and 300 the dimension of the word embedding.
y is of shape (44317, 300) is for each example the embedding of the last token of the sentence.
My code for the keras model goes as follow (inspired by this)
#importing all the needed tensorflow.keras components
model = Sequential()
model.add(InputLayer((None, 300)))
model.add(LSTM(100, return_sequences=True))
model.add(LSTM(100))
model.add(Dense(100, activation='relu'))
model.add(Dense(300, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X, y, batch_size=128, epochs=20)
model.save('model.h5')
However, the accuracy I get while training on this model is extremely low (around 1.5%). I think there is some component of the keras model that I misundertood, as if I don't embed my inputs and add an extra embedding layer instead of the InputLayer I get an accuracy of about 60 percents.
My main doubt is the value of "300" on my second Dense layer, as I read that this should correspond the vocabulary size of my word embedding model (which is 48000), however if I put anything else than 300 I get a dimension error. So I understand that I'm doing something wrong, but I can't find how to fix it.
PS :
I have also tried y = to_categorical(y, num_classes=vocab_size) with vocab_size the vocabulary size of my word embedding, and by changing 300 by this same value in the second Dense, however then it tries to create an array of shape(13295100, 48120) instead of what I expect : (44317, 48120).

If you really want to use the word vectors from Fasttext, you will have to incorporate them into your model using a weight matrix and Embedding layer. The goal of the embedding layer is to map each integer sequence representing a sentence to its corresponding 300-dimensional vector representation:
import gensim.downloader as api
import numpy as np
import tensorflow as tf
def load_doc(filename):
file = open(filename, 'r')
text = file.read()
file.close()
return text
fasttext = api.load("fasttext-wiki-news-subwords-300")
embedding_dim = 300
in_filename = 'data.txt'
doc = load_doc(in_filename)
lines = doc.split('\n')
tokenizer = tf.keras.preprocessing.text.Tokenizer()
tokenizer.fit_on_texts(lines)
text_sequences = tokenizer.texts_to_sequences(lines)
text_sequences = tf.keras.preprocessing.sequence.pad_sequences(text_sequences, padding='post')
vocab_size = len(tokenizer.word_index) + 1
text_sequences = np.array(text_sequences)
X, y = text_sequences[:, :-1], text_sequences[:, -1]
y = tf.keras.utils.to_categorical(y, num_classes=vocab_size)
max_length = X.shape[1]
weight_matrix = np.zeros((vocab_size, embedding_dim))
for word, i in tokenizer.word_index.items():
try:
embedding_vector = fasttext[word]
weight_matrix[i] = embedding_vector
except KeyError:
weight_matrix[i] = np.random.uniform(-5, 5, embedding_dim)
sentence_input = tf.keras.layers.Input(shape=(max_length,))
x = tf.keras.layers.Embedding(vocab_size, embedding_dim, weights=[weight_matrix],
input_length=max_length)(sentence_input)
x = tf.keras.layers.LSTM(100, return_sequences=True)(x)
x = tf.keras.layers.LSTM(100)(x)
x = tf.keras.layers.Dense(100, activation='relu')(x)
output = tf.keras.layers.Dense(vocab_size, activation='softmax')(x)
model = tf.keras.Model(sentence_input, output)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X, y, batch_size=5, epochs=20)
Note that I am using the dataset and preprocessing steps from the tutorial you linked.

It's very difficult to train RNN models in next sentence prediction task. LSTM/GRU do not have enough resources to extract enough features from text.
There are 2 ways to solve issue:
predict chars instead of word class
use transformer model. For example, Bert is good for features extracting and predicting masked word

How do I normalize labels using tf.data?

I am trying to build a regression model using Keras. Because I have a lot of data that cannot be loaded in memory, I am using tf.data.experimental.make_csv_dataset to create a dataset object.
The data needs to be normalized, and I think I understood how to normalize the features, but I can't find a proper way to normalize the labels.
I have the following code so far, with my training data being in csv files in the training_data folder. In the csv files, columns 'a' and 'b'are the features, and 'labels' are the labels, all numeric.
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense, Flatten
# Create the dataset
dataset = tf.data.experimental.make_csv_dataset(
file_pattern = "training_data/*.csv",
select_columns=['a', 'b', 'labels'],
label_name='labels',
batch_size=5, num_epochs=1,
num_parallel_reads=20,
shuffle_buffer_size=10000)
# Create a function for feature normalization
def get_normalization_layer(name, dataset):
# Create a Normalization layer for our feature.
normalizer = preprocessing.Normalization()
# Prepare a Dataset that only yields our feature.
feature_ds = dataset.map(lambda x, y: x[name])
# Learn the statistics of the data.
normalizer.adapt(feature_ds)
return normalizer
# Create a preprocessing layer for input
numerical_columns = []
for feature in ['a', 'b']:
normalizer = get_normalization_layer(feature, dataset)
num_col = tf.feature_column.numeric_column(feature, normalizer_fn=normalizer)
numerical_columns.append(num_col)
preprocessing_layer = tf.keras.layers.DenseFeatures(numerical_columns)
# Create and compile the model
model = Sequential()
model.add(preprocessing_layer)
model.add(Dense(20, activation='relu'))
model.add(Dense(20, activation='relu'))
model.compile(loss='mse', optimizer='adam', metrics=['mse'])
So in short, how do I normalize labels in a PrefetchDataset?

You can do the following, you can use the from_generator API for normalizing the labels. If you want you can normalize your features also with this approach. I am providing a pseudo code as I don't have your complete code with me, but you will get the gist of where I am going.
def gen():
# initialize enumerator
dataset_enum = dataset.enumerate()
i = 0
x = []
y = []
for element in dataset_enum.as_numpy_iterator():
# I am supposing that element[0] is x and element[1] is y.
if i % BATCH_SIZE == 0:
# normalize y
normalized_y = normalization_function(y)
x_features = x
x = []
y = []
yield x_features, normalized_y
x.append(element[0])
y.append(element[1])
new_dataset = tf.data.Dataaset.from_generator(
gen,
output_signature=(
# here what you expect to have as an output.
# Look at this to have a better idea https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_generator
)
)
Feel free to ask questions.

This may seem a bit unorthodox but you could make use of the multiply and add layer to reverse the normalization.
If you were using the functional api you would just add the two layers at the end:
x = multiply([x, np.sqrt(normalizer.variance)])
x = add([x, normalizer.mean])

Keras Normalization for a 2d input array

I am new to machine learning and trying to apply it to my problem.
I have a training dataset with 44000 rows of features with shape 6, 25. I want to build a sequential model. I was wondering if there is a way to use the features without flattening it. Currently, I flatten the features to 1d array and normalize for training (see the code below). I could not find a way to normalize 2d features.
dataset2d = dataset2d.reshape(dataset2d.shape[0],
dataset2d.shape[1]*dataset2d.shape[2])
normalizer = preprocessing.Normalization()
normalizer.adapt(dataset2d)
print(normalizer.mean.numpy())
x_train, x_test, y_train, y_test = train_test_split(dataset2d, flux_val,
test_size=0.2)
# %% DNN regression multiple parameter
def build_and_compile_model(norm):
inputs = Input(shape=(x_test.shape[1],))
x = norm(inputs)
x = layers.Dense(128, activation="selu")(x)
x = layers.Dense(64, activation="relu")(x)
x = layers.Dense(32, activation="relu")(x)
x = layers.Dense(1, activation="linear")(x)
model = Model(inputs, x)
model.compile(loss='mean_squared_error',
optimizer=keras.optimizers.Adam(learning_rate=1e-3))
return model
dnn_model = build_and_compile_model(normalizer)
dnn_model.summary()
# interrupt training when model is no longer imporving
path_checkpoint = "model_checkpoint.h5"
modelckpt_callback = keras.callbacks.ModelCheckpoint(monitor="val_loss",
filepath=path_checkpoint,
verbose=1,
save_weights_only=True,
save_best_only=True)
es_callback = keras.callbacks.EarlyStopping(monitor="val_loss",
min_delta=0, patience=10)
history = dnn_model.fit(x_train, y_train, validation_split=0.2,
epochs=120, callbacks=[es_callback, modelckpt_callback])
I also tried to modify my model input layer to the following, such that I do not need to reshape my input
inputs = Input(shape=(x_test.shape[-1], x_test.shape[-2], ))
and modify the normalization to the following
normalizer = preprocessing.Normalization(axis=1)
normalizer.adapt(dataset2d)
print(normalizer.mean.numpy())
But this does not seem to help. The normalization adapts to a 1d array of length 6, while I want it to adapt to a 2d array of shape 25, 6.
Sorry for the long question. You help will be much appreciated.

I'm not sure if I understood your issue. The normalizer layer can take N-D tensor and it produces an output with the same shape, for example:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
t = tf.constant(np.arange(2*3*4).reshape(2,3,4) , dtype=tf.float32)
tf.print("\n",t)
normalizer_layer = tf.keras.layers.LayerNormalization(axis=1)
output = normalizer_layer(t)
tf.print("\n",output)

How to shape TFRecordDataset to meet Model API?

I am building a model based on this code for noise suppression. My problem with the vanilla implementation is that it loads all data at once, which is not the best idea when the training data gets really large; my input file, denoted in the linked code as training.h5, is over 30 GB.
I decided to instead go with tf.data interface that should allow me to work with large data sets; my problem here is that I don't know how to properly shape TFRecordDataset so that it meets what's required by the Model API.
If you check model.fit(x_train, [y_train, vad_train], it essentially requires the following:
x_train, shape [nb_sequences, window, 42]
y_train, shape [nb_sequences, window, 22]
vad_train, shape [nb_sequences, window, 1]
window one typically fixes (in the code: 2000), so the only variable nb_sequences that stems from how large is your data set. However, with tf.data, we don't supply x and y, but only x (see Model API docs).
Saving tfrecord to file
In an effort to make the code reproducible, I created the input file with the following code:
writer = tf.io.TFRecordWriter(path='example.tfrecord')
for record in data:
feature = {}
feature['X'] = tf.train.Feature(float_list=tf.train.FloatList(value=record[:42]))
feature['y'] = tf.train.Feature(float_list=tf.train.FloatList(value=record[42:64]))
feature['vad'] = tf.train.Feature(float_list=tf.train.FloatList(value=[record[64]]))
example = tf.train.Example(features=tf.train.Features(feature=feature))
serialized = example.SerializeToString()
writer.write(serialized)
writer.close()
data is our training data with shape [10000, 65]. My example.tfrecord is available here. It's 3 MB, in reality it would be 30 GB+.
You might notice that in the linked code, numpy array has shape [x, 87], while mine is [x, 65]. That's OK - the remainder is not used anywhere.
Loading the dataset with tf.data.TFRecordDataset
I would like to use tf.data to load "on demand" the data with some prefetching, there's no need to keep it all in memory. My attempt:
import datetime
import numpy as np
import h5py
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import GRU
from tensorflow.keras import regularizers
from tensorflow.keras.constraints import Constraint
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras import backend as K
from tensorflow.keras.layers import concatenate
def load_dataset(path):
def _parse_function(example_proto):
keys_to_features = {
'X': tf.io.FixedLenFeature([42], tf.float32),
'y': tf.io.FixedLenFeature([22], tf.float32),
'vad': tf.io.FixedLenFeature([1], tf.float32)
}
features = tf.io.parse_single_example(example_proto, keys_to_features)
return (features['X'], (features['y'], features['vad']))
dataset = tf.data.TFRecordDataset(path).map(_parse_function)
return dataset
def my_crossentropy(y_true, y_pred):
return K.mean(2 * K.abs(y_true - 0.5) * K.binary_crossentropy(y_pred, y_true), axis=-1)
def mymask(y_true):
return K.minimum(y_true + 1., 1.)
def msse(y_true, y_pred):
return K.mean(mymask(y_true) * K.square(K.sqrt(y_pred) - K.sqrt(y_true)), axis=-1)
def mycost(y_true, y_pred):
return K.mean(mymask(y_true) * (10 * K.square(K.square(K.sqrt(y_pred) - K.sqrt(y_true))) + K.square(
K.sqrt(y_pred) - K.sqrt(y_true)) + 0.01 * K.binary_crossentropy(y_pred, y_true)), axis=-1)
def my_accuracy(y_true, y_pred):
return K.mean(2 * K.abs(y_true - 0.5) * K.equal(y_true, K.round(y_pred)), axis=-1)
class WeightClip(Constraint):
'''Clips the weights incident to each hidden unit to be inside a range
'''
def __init__(self, c=2.0):
self.c = c
def __call__(self, p):
return K.clip(p, -self.c, self.c)
def get_config(self):
return {'name': self.__class__.__name__,
'c': self.c}
def build_model():
reg = 0.000001
constraint = WeightClip(0.499)
main_input = Input(shape=(None, 42), name='main_input')
tmp = Dense(24, activation='tanh', name='input_dense', kernel_constraint=constraint, bias_constraint=constraint)(
main_input)
vad_gru = GRU(24, activation='tanh', recurrent_activation='sigmoid', return_sequences=True, name='vad_gru',
kernel_regularizer=regularizers.l2(reg), recurrent_regularizer=regularizers.l2(reg),
kernel_constraint=constraint, recurrent_constraint=constraint, bias_constraint=constraint)(tmp)
vad_output = Dense(1, activation='sigmoid', name='vad_output', kernel_constraint=constraint,
bias_constraint=constraint)(vad_gru)
noise_input = concatenate([tmp, vad_gru, main_input])
noise_gru = GRU(48, activation='relu', recurrent_activation='sigmoid', return_sequences=True, name='noise_gru',
kernel_regularizer=regularizers.l2(reg), recurrent_regularizer=regularizers.l2(reg),
kernel_constraint=constraint, recurrent_constraint=constraint, bias_constraint=constraint)(noise_input)
denoise_input = concatenate([vad_gru, noise_gru, main_input])
denoise_gru = GRU(96, activation='tanh', recurrent_activation='sigmoid', return_sequences=True, name='denoise_gru',
kernel_regularizer=regularizers.l2(reg), recurrent_regularizer=regularizers.l2(reg),
kernel_constraint=constraint, recurrent_constraint=constraint, bias_constraint=constraint)(
denoise_input)
denoise_output = Dense(22, activation='sigmoid', name='denoise_output', kernel_constraint=constraint,
bias_constraint=constraint)(denoise_gru)
model = Model(inputs=main_input, outputs=[denoise_output, vad_output])
model.compile(loss=[mycost, my_crossentropy],
metrics=[msse],
optimizer='adam', loss_weights=[10, 0.5])
return model
model = build_model()
dataset = load_dataset('example.tfrecord')
My dataset has now the following shape:
<MapDataset shapes: ((42,), ((22,), (1,))), types: (tf.float32, (tf.float32, tf.float32))>
which I thought is what Model API expects (spoiler: it doesn't).
model.fit(dataset.batch(10))
gives following error:
ValueError: Error when checking input: expected main_input to have 3 dimensions, but got array with shape (None, 42)
Makes sense, I don't have the window here. At the same time it seems like it's not getting correct shape expected by Model(inputs=main_input, outputs=[denoise_output, vad_output]).
How to modify load_dataset so that it matches what's expected by the Model API for the tf.data?

Given that your model has 1 input and 2 outputs, your tf.data.Dataset should have two entries:
1) Input array of shape (window, 42)
2) Tuple of two arrays each of shape (window, 22) and (window, 1)
EDIT: Updated answer - you already return two element tuple
I just noticed that your dataset has these two entries (similar to those described above) and the only thing that differs is the shape.
The only operations you need to perfom is to batch your data twice:
First - to restore the window parameter.
Second - to pass a batch to a model.
window_size = 1
batch_size = 10
dataset = load_dataset('example.tfrecord')
model.fit(dataset.batch(window_size).batch(batch_size)
And that should work.
Below is an old answer, where I wrongfully assumed your dataset shape:
Old Answer, where I assumed you are returning three element tuple:
Assuming that you are starting from three element tuple of shapes (42,), (22,) and (1,), this can be achieved in the same batching operations, enriched with a custom_reshape function to return two-element tuple:
window_size = 1
batch_size = 10
dataset = load_dataset('example.tfrecord')
dataset = dataset.batch(window_size).batch(batch_size)
# Change output format
def custom_reshape(x, y, vad):
return x, (y, vad)
dataset = dataset.map(custom_reshape)
In short, given this dataset shape, you could just call:
model.fit(dataset.batch(window_size).batch(10).map(custom_reshape)
and it should work too.
Best of luck. And sorry again for the fuss.

Two parallel but different datasets in Keras as multiple inputs?

I have been googling all day trying to find an example of the functional input for two parallel datasets in Keras but I can't find one.
My problem is that I have dataset 1, a set of images of people performing different actions. It is formatted as a csv as follows:
image_url,class
example1.png,BRUSH_TEETH
example2,BRUSH_TEETH
...
example10000.png,DANCING
I will preprocess these and make them all 64x64 in size. My second dataset will be leap motion data where every row is information captured at the same time as the corresponding row in dataset 1
(ignore the column names and values, I'm not sure how they will look like yet as I haven't gathered the data, but they will be one row and parallel to the above dataset1)
x,y,z,a,b,c,d,class
1,2,3,4,5,6,7,BRUSH_TEETH
8,9,10,3,1,3,4,BRUSH_TEETH
...
1,2,3,4,5,6,7,DANCING
I have been reading about the functional API and it seems as if I can run the data object from dataset1 through a CNN while running the same data object from dataset2 through, for example, a deep MLP. Then, using merge or concatenate, bring the two outputs from their final layers to another deep MLP and then finally link this final merged model to an output
Forgetting about the CNN for a minute, a simple example of merging is given by the API as follows:
import keras
input1 = keras.layers.Input(shape=(16,))
x1 = keras.layers.Dense(8, activation='relu')(input1)
input2 = keras.layers.Input(shape=(32,))
x2 = keras.layers.Dense(8, activation='relu')(input2)
# equivalent to added = keras.layers.add([x1, x2])
added = keras.layers.Add()([x1, x2])
out = keras.layers.Dense(4)(added)
model = keras.models.Model(inputs=[input1, input2], outputs=out)
My problem is that I need to feed input1 (when in the form of a CNN) the image contained in the csv while at the same time feeding input2 with the correlating row in the second dataset containing Leap Motion data. PS: how in the above would I continue the model after merging with two dense layers before output? Would it be simply this:
x3 = keras.layers.Dense(100)(added)
x3 = keras.layers.Dense(50)(x3)
out = keras.layers.Dense(4)(x3)
Is this possible to perform? If so, I would MASSIVELY appreciate a helping hand, I'm losing my mind trying to get my head around how the two datasets would be kept in sync with one another!
A sample script that I can try out and play with would be excellent, as I'm relatively new to the Keras framework
Thank you very much!

Please check if this is useful. Tested with Keras 2.2.4.
from keras.layers import Conv2D, MaxPooling2D, Input, Dense, Flatten, concatenate
from keras.models import Model
import numpy as np
img_input = Input(shape=(64, 64, 1)) ## branch 1 with image input
x = Conv2D(64, (3, 3))(img_input)
x = Conv2D(64, (3, 3))(x)
x = MaxPooling2D((2, 2))(x)
x = Flatten()(x)
out_a = Dense(64)(x)
num_input = Input(shape=(7,)) ## branch 2 with numerical input
x1 = Dense(8, activation='relu')(num_input)
out_b = Dense(16, activation='relu')(x1)
concatenated = concatenate([out_a, out_b]) ## concatenate the two branches
out = Dense(4, activation='softmax')(concatenated)
model = Model([img_input, num_input], out)
print(model.summary())
model.compile('sgd', 'categorical_crossentropy', ['accuracy'])
### Just for sanity check
X = [np.zeros((1,64,64,1)), np.zeros((1,7))]
y = np.ones((1,4))
model.fit(X, y)
print(model.predict(X))
You can read the input data using Pandas
from PIL import Image
import pandas as pd
def get_num_input():
df = pd.read_csv('num.csv')
columns = list(df.columns)
features = columns[:-1]
cls_name = columns[-1]
X = np.zeros((len(df), len(features)))
Y = list()
for i, row in df.iterrows():
X[i] = row[features]
Y.append(row[cls_name])
return (X, Y)
def get_img_input():
df = pd.read_csv('img.csv')
X_img = np.zeros((len(df), 28, 28)) # change as per image size
Y = list()
for i, row in df.iterrows():
X_img[i] = np.array(Image.open(row['image_url']))
Y.append(row['class'])
return (X_img, Y)
X_num, Y = get_num_input()
X_img, _ = get_img_input() # use one of the Ys
# X feature normalization, convert Y to one-hot representation
model.fit() has a 'validation_split' parameter which can be set to 0.3 to create a 70:30 split. model.fit() returns a History object which can be used to plot the accuracy curves or you can use TensorBoard callback for live tracking.
https://chrisalbon.com/deep_learning/keras/visualize_loss_history/
https://keras.io/callbacks/#tensorboard