after a few days where my code was reproducible everytime - it is now not! i don't know what happend, i just changed some lines of codes and i don't know how to fix it!
# Code reproduzierbar machen
import numpy as np
import os
import random as rn
import tensorflow as tf
import keras
from keras import backend as K
#-----------------------------Keras reproducible------------------#
SEED = 1234
tf.set_random_seed(SEED)
os.environ['PYTHONHASHSEED'] = str(SEED)
np.random.seed(SEED)
rn.seed(SEED)
session_conf = tf.ConfigProto(
intra_op_parallelism_threads=1,
inter_op_parallelism_threads=1
)
sess = tf.Session(
graph=tf.get_default_graph(),
config=session_conf
)
K.set_session(sess)
# Importiere Datasets (Training und Test)
import pandas as pd
poker_train = pd.read_csv("C:/Users/elihe/Documents/Studium Master/WS 19 und 20/Softwareprojekt/poker-hand-training-true.data")
poker_test = pd.read_csv("C:/Users/elihe/Documents/Studium Master/WS 19 und 20/Softwareprojekt/poker-hand-testing.data")
X_tr = poker_train.iloc[:, 0:10]
y_train = poker_train.iloc[:, 10:11]
X_te = poker_test.iloc[:, 0:10]
y_test = poker_test.iloc[:, 10:11]
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_tr)
X_test = sc.transform(X_te)
# NN mit Keras erstellen
import keras
from keras.models import Sequential
from keras.layers import Dense
nen = Sequential()
nen.add(Dense(100, input_dim = 10, activation = 'relu'))
nen.add(Dense(50, activation = 'relu'))
nen.add(Dense(10, activation = 'softmax'))
# Kompilieren
from keras import metrics
nen.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
nen_fit = nen.fit(X_train, y_train,epochs=2, batch_size=50, verbose=1, validation_split = 0.2, shuffle = False)
I took just 2 epochs so that i can see immediately if my Output is the same, normally there would be 500 epochs. The first lines of code made it until today reproducible, but now it is not! I changed the part with the X_te and X_tr, because first i made a OneHotEncoding with the classes y_train and y_test, but now i am not doing it. Also i changed the activation functions from sigmoid to relu and the optimizer from RMSprop to adam. I don't know what to do:(
Related
I'm trying to learn autoencoder by implementing it but model.fit crashes my vscode when the batch_size or epochs is set at a high number. I can also see that my memory reaches 100% and vscode becomes unresponsive
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from sklearn import metrics
#define input and output
n = 19
fib = [1,1]
for i in np.arange(2, n+1):
fib.append(fib[i-1] + fib[i-2])
x = np.array([fib])
y = np.array([fib])
print(x)
print(x.shape)
print(y.shape)
plt.scatter(x,y)
#building neural network
model = Sequential()
model.add(Dense(2, input_dim= x.shape[1], activation = 'relu'))
model.add(Dense(x.shape[1]))
model.compile(loss= 'mean_squared_error', optimizer = 'adam')
model.summary()
#model fit
model.fit(y,y, verbose = 1, batch_size = 4, epochs = 1000)
pred = model.predict(y)
I've been trying to create a model that recognizes different singing techniques. I have got good results but I want to do different tests with different optimizers, layers, etc. However, I can't get reproducible results. By running twice this model training:
num_epochs = 100
batch_size = 128
history = modelo.fit(X_train_f, Y_train, validation_data=(X_test_f,Y_test), epochs=num_epochs, batch_size=batch_size, verbose=2)
I can get 25% accuracy the first run and then 34% the second. Then if I change the optimizer from "sgd" to "adam", I would get a 99%. If I come back to the previous "sgd" optimizer that got me 34% the second run, I would get 100% or something crazy like that. I don't understand why.
I've tried many things I've read in similar questions. The following lines show how I am trying to make my code to be reproducible, and these are actually the first lines of my whole code:
import numpy as np
import tensorflow as tf
import random as rn
import os
#https://stackoverflow.com/questions/57305909/tensorflow-keras-reproducibility-problem-on-google-colab
os.environ['PYTHONHASHSEED']=str(5)
np.random.seed(5)
rn.seed(12345)
session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,
inter_op_parallelism_threads=1)
tf.compat.v1.set_random_seed(1234)
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
tf.compat.v1.keras.backend.set_session(sess)
Question is, what am I doing wrong with the code above that is not working (as I mentioned)?
Here's where I create the training sets:
from keras.datasets import mnist
from keras.utils import np_utils
from keras.models import Sequential
from keras.layers.convolutional import Conv1D, MaxPooling1D
from keras.layers.core import Dense, Flatten
from keras.layers import BatchNormalization,Activation
from keras.optimizers import SGD, Adam
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size=0.2, random_state=2)
My model:
from tensorflow.keras import layers
from tensorflow.keras import initializers
input_dim = X_train_f.shape[1]
output_dim = Y_train.shape[1]
modelo = Sequential()
modelo.add(Conv1D(filters=6, kernel_initializer=initializers.glorot_uniform(seed=5), kernel_size=5, activation='relu', input_shape=(40, 1))) # 6
modelo.add(MaxPooling1D(pool_size=2))
modelo.add(Conv1D(filters=16, kernel_initializer=initializers.glorot_uniform(seed=5), kernel_size=5, activation='relu')) # 16
modelo.add(MaxPooling1D(pool_size=2))
modelo.add(Flatten())
modelo.add(Dense(120, kernel_initializer=initializers.glorot_uniform(seed=5), activation='relu')) # 120
modelo.add(Dense(84, kernel_initializer=initializers.glorot_uniform(seed=5), activation='relu')) # 84
modelo.add(Dense(nclases, kernel_initializer=initializers.glorot_uniform(seed=5), activation='softmax'))
sgd = SGD(lr=0.1)
#modelo.compile(loss='categorical_crossentropy',
# optimizer='adam',
# metrics=['accuracy'])
modelo.compile(loss='categorical_crossentropy',
optimizer=sgd,
metrics=['accuracy'])
modelo.summary()
modelo.input_shape
It is a normal situation. Adam optimizer is much more powerful comparing to SGD. Adam implicitly performs coordinate-wise gradient clipping and can hence, unlike SGD, tackle heavy-tailed noise.
I have been trying to implement a transfer learning model using the Xception model and fine-tune it. But when I tried to train the model in the last section of the code it is showing the following error - AttributeError: 'numpy.ndarray' object has no attribute '_in_multi_worker_mode'.Can someone help me solve this error or any code to train the model.My code is given below.
# Install TensorFlow
try:
# %tensorflow_version only exists in Colab.
%tensorflow_version 2.x
except Exception:
pass
import tensorflow as to
print(tf.__version__)
from tensorflow import keras
tf.random.set_seed(42)
import numpy as np
np.random.seed(42)
# Pandas and Numpy for data structures and util functions
import numpy as np
import pandas as pd
from numpy.random import rand
pd.options.display.max_colwidth = 600
# Scikit Imports
from sklearn.model_selection import train_test_split
# Matplot Imports
import matplotlib.pyplot as plt
params = {'legend.fontsize': 'x-large',
'figure.figsize': (15, 5),
'axes.labelsize': 'x-large',
'axes.titlesize':'x-large',
'xtick.labelsize':'x-large',
'ytick.labelsize':'x-large'}
import glob
import PIL
from PIL import Image
plt.rcParams.update(params)
%matplotlib inline
# pandas display data frames as tables
from IPython.display import display, HTML
import warnings
warnings.filterwarnings('ignore')
import sys
import os
from tensorflow.keras import utils as np_utils
from tensorflow.keras.utils import multi_gpu_model
from tensorflow.keras.utils import Sequence
from tensorflow.keras.models import Model
from tensorflow.keras import layers
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping
from tensorflow.keras import regularizers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.layers import GlobalAveragePooling2D
from tensorflow.keras.layers import Dense, Dropout, Activation,
BatchNormalization, Flatten
from tensorflow.keras.models import Sequential,load_model
from tensorflow.python.keras.utils.data_utils import Sequence
from tensorflow.keras.preprocessing.image import ImageDataGenerator,
img_to_array, load_img
from tensorflow.keras.preprocessing.image import NumpyArrayIterator
from keras.applications import Xception
from tensorflow.keras.preprocessing import image
from tensorflow.keras import backend as K
imgFiles = glob.glob("dataset/*/*.jpg")
for items in imgFiles[:8]:
print(items)
X = []
y = []
for fName in imgFiles:
X_i = Image.open(fName)
X_i = X_i.resize((299,299))
X_i = np.array(X_i) / 255.0
X.append(X_i)
label = fName.split("/")
y_i = label[-2]
y.append(y_i)
print(set(y))
from sklearn.preprocessing import LabelEncoder
lEncoder = LabelEncoder()
y = lEncoder.fit_transform(y)
print(set(y))
print(lEncoder.classes_)
X = np.array(X)
y = np.array(y)
print(X.shape)
print(y.shape)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
stratify=y,
random_state=42)#splitting train and test
images to 70% and 30% reps.
print("X_train_shape: {}".format(X_train.shape))
print("X_test_shape: {}".format(X_test.shape))
mu = X_train.mean()
std = X_train.std()
X_train_std = (X_train-mu)/std
X_test_std = (X_test-mu)/std
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train,
test_size=0.15, stratify=y_train,
random_state=42)#splitting 15% of
train images for validation
print("X_val_shape: {}".format(X_val.shape))
# hyper parameters for model
nb_classes = 6 # number of classes
based_model_last_block_layer_number = 126
img_width, img_height = 299, 299
num_channels= 3
batch_size = 32
nb_epoch = 15 # number of iteration the algorithm gets trained.
transformation_ratio = .05 # how aggressive will be the data
augmentation/transformation
#data augmentation
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=30,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip='true',
vertical_flip='true')
train_generator = train_datagen.flow(X,y, shuffle=False,
batch_size=batch_size, seed=1)
validation_datagen = ImageDataGenerator(rescale=1. / 255)
val_generator = train_datagen.flow(X,y, shuffle=False,
batch_size=batch_size, seed=1)
# Pre-Trained CNN Model using imagenet dataset for pre-trained weights
# Transfer Learning!!
# Importing Xception pre trained model on ImageNet
base_model = keras.applications.xception.Xception(include_top=False,
weights='imagenet',
input_shape=(img_width, img_height, num_channels))
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all layers of the based model that is already pre-trained.
for layer in base_model.layers:
layer.trainable = False
# Top Model Block which is to be stacked over xception model
out = base_model.output
out = GlobalAveragePooling2D()(out)
out = Dense(1024, activation='relu')(out)
out = Dense(512, activation='relu')(out)
total_classes = y.shape[0]
predictions = Dense(total_classes, activation='softmax')(out)
model = keras.models.Model(inputs=base_model.input, outputs=predictions)
model.compile(Adam(lr=.0001), loss='categorical_crossentropy', metrics=
['accuracy'])
model.summary()
callbacks_list = [keras.callbacks.ModelCheckpoint("bestTL.h5",
save_best_only=True)]
# Train the model
batch_size = batch_size
train_steps_per_epoch = X_train.shape[0] // batch_size
val_steps_per_epoch = X_val.shape[0] // batch_size
#training cnn up to 15 epoch
history = Model.fit(X_train_std,y_train,
steps_per_epoch=train_steps_per_epoch,
validation_data=val_generator,
validation_steps=val_steps_per_epoch,
callbacks=callback_list
epochs=15,
verbose=1)
How about using only tensorflow.keras.xxxx or keras.xxxx instead of using both of them at one time?
I use keras for time series prediction. My code can predict next 6 months by predict next one month and then get it to be input for predict next month again untill complete 6 months. That means predict one month 6 times. Can I predict next 6 month in one time.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from keras.layers import LSTM
from pandas.tseries.offsets import MonthEnd
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense, Dropout
import keras.backend as K
from keras.layers import Bidirectional
from keras.layers import Embedding
from keras.layers import GRU
df = pd.read_csv('D://data.csv',
engine='python')
df['DATE_'] = pd.to_datetime(df['DATE_']) + MonthEnd(1)
df = df.set_index('DATE_')
df.head()
split_date = pd.Timestamp('03-01-2015')
train = df.loc[:split_date, ['data']]
test = df.loc[split_date:, ['data']]
sc = MinMaxScaler()
train_sc = sc.fit_transform(train)
test_sc = sc.transform(test)
X_train = train_sc[:-1]
y_train = train_sc[1:]
X_test = test_sc[:-1]
y_test = test_sc[1:]
K.clear_session()
model = Sequential()
model.add(Dense(12, input_dim=1, activation='relu'))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.summary()
model.fit(X_train, y_train, epochs=200, batch_size=2)
y_pred = model.predict(X_test)
real_pred = sc.inverse_transform(y_pred)
real_test = sc.inverse_transform(y_test)
print("Predict Value")
print(real_pred)
print("Test Value")
print(real_test)
Yes, by changing your output layer (the last layer) from Dense(1) to Dense(6). Of course you also have to change your y_train and y_test to have shape (1,6) instead of (1,1).
Best of luck.
I'm using a combination of sklearn and Keras running with Theano as its back-end. I'm using the following code-
import numpy as np
import pandas as pd
from pandas import Series, DataFrame
import keras
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.constraints import maxnorm
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import SGD
from keras.wrappers.scikit_learn import KerasClassifier
from keras.constraints import maxnorm
from keras.utils.np_utils import to_categorical
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from datetime import datetime
import time
from datetime import timedelta
from __future__ import division
seed = 7
np.random.seed(seed)
Y = data['Genre']
del data['Genre']
X = data
encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)
X = X.as_matrix().astype("float")
calls=[EarlyStopping(monitor='acc', patience=10), ModelCheckpoint('C:/Users/1383921/Documents/NNs/model', monitor='acc', save_best_only=True, mode='auto', period=1)]
def create_baseline():
# create model
model = Sequential()
model.add(Dense(18, input_dim=9, init='normal', activation='relu'))
model.add(Dense(9, init='normal', activation='relu'))
model.add(Dense(12, init='normal', activation='softmax'))
# Compile model
sgd = SGD(lr=0.01, momentum=0.8, decay=0.0, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
return model
np.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasClassifier(build_fn=create_baseline, nb_epoch=300, batch_size=16, verbose=2)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
results = cross_val_score(pipeline, X, encoded_Y, cv=kfold, fit_params={'mlp__callbacks':calls})
print("Baseline: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
The result when I start running this last part is-
Epoch 1/10
...
Epoch 2/10
etc.
It's supposed to be Epoch 1/300 and it works just fine when I run it on a different notebook.
What do you guys think is happening? np_epoch=300...
What Keras version is this? If its greater than 2.0, then nb_epoch was changed to just epochs. Else it defaults to 10.
In Keras 2.0 the nb_epoch parameter was renamed to epochs so when you set epochs=300 it runs 300 epochs. If you use nb_epoch=300 it will default to 10 instead.
Another solution to your problem: Forget about nb_epoch (doesn't work). Pass epochs inside fit_params:
results = cross_val_score(pipeline, X, encoded_Y, cv=kfold,
fit_params={'epochs':300,'mlp__callbacks':calls})
And that would work. fit_params goes straight into the Fit method and it will get the right epochs.
The parameter name in your function should be epochs instead of nb_epochs.
Be very careful though. For example, I trained my ANN with the old fashioned way of declaring the parameters (nb_epochs = number), and it worked (the iPython console only showed me some warnings), but when I plugged the same parameter names in the cross_val_score function, it did not work.
I think that what sklearn calls "Epoch" is one step of your crossvalidation. So it does 300 epochs of training 10 times :-) is that possible? Try with verbose=1