Develop Reference

Keras Sequential Neural Network : Bounce in Train and Test loss on learning curve

I'm working of the Online-News-Popularity dataset. I'm trying to predict the numbers of shares with a keras sequential network.
However, a weird bounce in mae for train and test happens on my learning curve and I don't know how to interprete it. Do you have any idea how it means ? (Link Below)
Learning curve of my model based on mae
Here is my code for the model :
x = m_news.loc[:, m_news.columns != "shares"]
y = np.ravel(m_news.loc[:, m_news.columns == "shares"])
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33, random_state=42) #data is already scaled
model = Sequential()
model.add(Dense(24, activation='relu', input_shape=(55,)))
model.add(Dense(32, activation='relu'))
model.add(Dense(1))
model.compile(loss='mse',
metrics=['mae', 'mse'],
optimizer = optimizers.RMSprop(0.001))
history = model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=15, batch_size=200)
plt.plot(history.history['mae'], label='train')
plt.plot(history.history['val_mae'], label='test')
_=plt.legend()
plt.xlabel("no epochs")
plt.ylabel("value loss")
Thanks for reading and trying to help me :)

how can i use cross validation in ensembling models (like voting)?

I am trying to use voting model in the ensembling methods, how can i add a cross validation to it?
Thanks
model_1 = SGD_cls=SGDClassifier(random_state=0)
model_2 = DecisionTreeClassifier(criterion='entropy', max_depth=12,max_leaf_nodes=35,splitter='best')
model_3 = GradientBoostingClassifier(n_estimators=100, random_state=42)
model_4 = MLPClassifier(random_state=42)
# model_3 = KNeighborsClassifier(n_neighbors=2)
X = df1.drop('product', axis = 1)
y = df1['product']
X_new = res_fit.transform(X)
#X_new =pd.DataFrame(X_new,columns = X.iloc[:,res_fit.support_].columns)
y_pred=cross_val_predict(model,X_new,y,cv=10)
X_train, X_test, y_train, y_test = train_test_split(X_new, y, test_size=0.25, random_state=42)
model_5 = VotingClassifier([('ADA', model_1),
('Tree', model_2),
('GradBoost',model_3),
('MLP',model_4)],
voting='hard')
for model in (model_1, model_2, model_3,model_4,model_5):
model.fit(X_train, y_train)
print(model.__class__.__name__, model.score(X_test, y_test))

Why does passing a list of metrics when compiling Tensorflow models result in an InvalidArgumentError?

I'm learning Tensorflow and I'm trying to pass a list of metrics to be evaluated when I compile classification models in a loop, like the example in the documentation.
However, when I pass a list like:
METRICS = [
keras.metrics.AUC(name='auc'),
keras.metrics.AUC(name='prc', curve='PR')
]
def compile_model(model, X_train, y_train, X_val=None, y_val=None, callbacks=None, batch_size=1000, epochs = 10, optimizer='adam', loss_func=keras.losses.BinaryCrossentropy(), metrics=['accuracy']):
# Copy the model
mdl = model
mdl.compile(optimizer = optimizer, loss = loss_func, metrics = metrics)
# Fit model
if (X_val is not None) and (y_val is not None):
mdl.fit(X_train, y_train, epochs = epochs, batch_size=batch_size, callbacks=callbacks,
validation_data = (X_val, y_val))
else:
mdl.fit(X_train, y_train, epochs = epochs, batch_size=batch_size, callbacks=callbacks)
return mdl
# Compile model
model = compile_model(model, X_train, y_train, X_test, y_test, callbacks=callbacks, batch_size=1000, epochs = 10, metrics=METRICS)
I get the error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'metrics/tp/AssignAddVariableOp/resource_1' with dtype resource
[[node metrics/tp/AssignAddVariableOp/resource_1 (defined at /opt/anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_keras_scratch_graph_6163]
If I try to compile the model with a list of metrics like
model = compile_model(model, X_train, y_train, X_test, y_test, callbacks=callbacks, batch_size=1000, \
epochs = 10, metrics=[keras.metrics.AUC(name='auc'), keras.metrics.AUC(name='prc', curve='PR')] )
the models are able to compile without any issues. None of the related questions I've seen about these placeholder tensors involve Keras' metrics. Why does the error only occur when I try to pass the metrics that way?

Build a neural network using output of another neural network, problem with tensors

I'm trying do some NLP on stack overflow posts to predict tags according to what's in the title.
I have a constraint which is I have to embed my sentences using the framework of Sentence transformers
The idea was to embed the sentences and use them as an input to a neural network I built.
I'm not an expert in neural network so there are probably a lot of things that I'm missing
The problem I encounter is it failed to convert to a tensor.
I have tried solving this with this post on SO , but still have the same issue...
Below is my code :
title_list = df.Title.tolist()
model = SentenceTransformer('paraphrase-distilroberta-base-v1')
embeddings = model.encode(title_list)
embeddings_list = [elem for elem in embeddings_ex]
df_embed = df
df_embed['Embeddings'] = embeddings_list
df_embed.Embeddings = [np.asarray(x).astype('float32') for x in df_embed.Embeddings]
X = df_embed['Embeddings'].values
y = df_embed.Tags
mlb = MultiLabelBinarizer(classes=top_tags)
y_mlb = pd.DataFrame(mlb.fit_transform(y),columns=mlb.classes_, index=y.index)
from sklearn.model_selection import train_test_split
X_train, X_val, y_train, y_val = train_test_split(X, y_mlb, test_size = 0.3, random_state = 0)
X_val, X_test, y_val, y_test = train_test_split(X_val, y_val, test_size = 0.4, random_state = 0)
model = Sequential()
# Input - Layer
model.add(Dense(100, activation = "relu"))
# Hidden - Layers
model.add(Dropout(0.3, noise_shape=None, seed=None))
# Output- Layer
model.add(Dense(50, activation = "sigmoid"))
model.compile(loss='binary_crossentropy',
optimizer=Adam(0.01),
metrics=['accuracy'])
hist = model.fit(X_train, y_train, batch_size=8, epochs=10, validation_split=0.1)
I got this error:
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type numpy.ndarray).

Are these 2 keras deep learning code the same for multiple outputs?

I've a problem involving airfoil velocity and pressure prediction, given the AOA,x,y. I'm using keras with MLP. I have 3 inputs (AOA,x,y) and I have to predict 3 outputs (u,v,p). I initially have a code which outputs the MSE loss as a single value. However, I modified the code so that I have MSE for each output. However, I don't get the avg MSE of the 3 outputs (u_mean_squared_error: 73.63%,v_mean_squared_error: 1.13%,p_mean_squared_error: 2.16%) equal to the earlier single MSE loss (mean_squared_error: 5.81%). Hence, I'm wondering if my new code is wrong. Or whether I'm doing it the right way. Can someone help?
Old code:
# load pima indians dataset
dataset = numpy.loadtxt("S1020_data.csv", delimiter=",")
# split into input and output variables
X = dataset[:,0:3]
Y = dataset[:,3:6]
# split into 67% for train and 33% for test
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, random_state=seed)
# create model
input_data = layers.Input(shape=(3,))
#create the layers and pass them the input tensor to get the output tensor:
hidden1Out = Dense(units=12, activation='relu')(input_data)
hidden2Out = Dense(units=8, activation='relu')(hidden1Out)
finalOut = Dense(units=3, activation='relu')(hidden2Out)
#define the model's start and end points
model = Model(input_data, finalOut)
# Compile model
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mean_squared_error'])
# Fit the model
model.fit(X_train, y_train, validation_data=(X_test,y_test), epochs=10, batch_size=1000)
# evaluate the model
scores = model.evaluate(X, Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
New code:
# load pima indians dataset
dataset = numpy.loadtxt("S1020_data.csv", delimiter=",")
# split into input and output variables
X = dataset[:,0:3]
Y = dataset[:,3:6]
# split into 67% for train and 33% for test
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, random_state=seed)
# create model
input_data = layers.Input(shape=(3,))
#create the layers and pass them the input tensor to get the output tensor:
hidden1Out = Dense(units=12, activation='relu')(input_data)
hidden2Out = Dense(units=8, activation='relu')(hidden1Out)
u_out = Dense(1, activation='relu', name='u')(hidden2Out)
v_out = Dense(1, activation='relu', name='v')(hidden2Out)
p_out = Dense(1, activation='relu', name='p')(hidden2Out)
#define the model's start and end points
model = Model(input_data,outputs = [u_out, v_out, p_out])
# Compile model
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mean_squared_error'])
# Fit the model
model.fit(X_train, [y_train[:,0], y_train[:,1], y_train[:,2]], validation_data=(X_test,[y_test[:,0], y_test[:,1], y_test[:,2]]), epochs=10, batch_size=1000)
# evaluate the model
scores = model.evaluate(X, [Y[:,0], Y[:,1], Y[:,2]])
for i in range(7):
print("\n%s: %.2f%%" % (model.metrics_names[i], scores[i]*100))

I think the difference comes from the optimization objective.
In your old code, the objective was:
sqrt( (u_true - u_pred)^2 + (v_true - v_pred)^2 + (p_true - p_pred)^2 )
which minimizes the 2-norm of the [u_pred,v_pred,p_pred] vector with respect to its target.
But in the new one, the objective became:
sqrt((u_true - u_pred)^2) + sqrt((v_true - v_pred)^2) + sqrt((p_true - p_pred)^2)
which is quite different from the previous one.