I am working on an LSTM for a final project. I've been following TensorFlow's tutorial here: https://www.tensorflow.org/tutorials/sequences/text_generation for most of it, especially for how to save and load the models. However, it's coming up with this error:
Traceback (most recent call last):
File "D:\xxx\Documents\Class Coding\Artificial Intelligence\Shelley>\Writerbot.py", line 187, in
restore_progress()
File "D:\xxx\Documents\Class Coding\Artificial Intelligence\Shelley\Writerbot.py", line 141, in restore_progress
shelley.load_weights(weights)
File "C:\Users\xxx\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\network.py", line 1508, in load_weights
if _is_hdf5_filepath(filepath):
File "C:\Users\xxx\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\network.py", line 1648, in _is_hdf5_filepath
return filepath.endswith('.h5') or filepath.endswith('.keras')
AttributeError: 'NoneType' object has no attribute 'endswith'
And here is my code related to loading and restoring weights, as best as I can tell, since the rest of the error's coming from keras:
def create_shelley(vocab, embedding, numunits, batch):
"""This is what actually creates a neural network."""
shelley = tf.keras.Sequential([
tf.keras.layers.Embedding(vocab, embedding,
batch_input_shape=[batch, None]),
lstm(numunits,
return_sequences=True,
recurrent_initializer='glorot_uniform',
stateful=True),
tf.keras.layers.Dense(vocab)
])
return shelley
def train():
"""We create weight checkpoints as we train our neural network on files fed into it."""
checkpoints = 'D:\\xxx\\Documents\\Class Coding\\Artificial Intelligence\\Shelley\\trainingcheckpoints'
prefix = os.path.join(checkpoints, "ckpt_{epoch}")
callback=tf.keras.callbacks.ModelCheckpoint(
filepath=prefix,
save_weights_only=True)
print(epochsteps)
history = shelley.fit(botfeed.repeat(), epochs=epochs, steps_per_epoch=epochsteps, callbacks=[callback])
def restore_progress():
"""Load the most recent weight checkpoint."""
trainingcheckpoints = "D:\\Robin Pegau\\Documents\\Class Coding\\Artificial Intelligence\\Shelley\\trainingcheckpoints\\checkpoint"
weights = tf.train.latest_checkpoint(trainingcheckpoints)
shelley = create_shelley(vocab, embed, totalunits, batch = 1)
shelley.load_weights(weights)
shelley.build(tf.TensorShape([1, None]))
restore_progress()
There is a "checkpoint" file that has no filetype. There are also files that look like "ckpt_[x].index" and "ckpt_[x].data-00000-of-00001
Thank you all for your help in advance.
Related
I have downloaded strong texta pretrained model, and im trying to transfer learn it.
therefore I'm loading the model which is saved as a 'xray_model.h5' file, and set it as untrainable:
model = tf.keras.models.load_model('xray_model.h5')
model.trainable = False
later I take the start layer and end layer and build my addings on it:
base_input = model.layers[0].input
base_output = model.get_layer(name="flatten").output
base_output = build_model()(base_output)
new_model = keras.Model(inputs=base_input, outputs=base_output)
since I want to train my layers (and after some games, I realized that I might need to train the old layers too) I want to set the model as trainable:
for i in range(len(new_model.layers)):
new_model._layers[i].trainable = True
BUT, when I start training it, with the callback:
METRICS = ['accuracy',
tf.keras.metrics.Precision(name='precision'),
tf.keras.metrics.Recall(name='recall'),
lr_metric]
reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=2, min_lr=0.00001, verbose=1)
save_callback = tf.keras.callbacks.ModelCheckpoint("new_xray_model.h5",
save_best_only=True,
monitor='accuracy')
history = new_model.fit(train_generator,
verbose=1,
steps_per_epoch=BATCH_SIZE,
epochs=EPOCHS,
validation_data=test_generator,
callbacks=[save_callback, reduce_lr])
I get the next error:
File "C:\Users\jm10o\AppData\Local\Programs\Python\Python38\lib\site-packages\h5py\_hl\group.py", line 373, in __setitem__
h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)
File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5o.pyx", line 202, in h5py.h5o.link
OSError: Unable to create link (name already exists)
Process finished with exit code 1
I noticed that it happens only when I'm trying to further train the model which I loaded.
I couldn't find any solution for it.
The problem came from the Model_checkpoint callback. for each epoch, you save the model with the same name.
use the following format
ModelCheckpoint('your_model_name{epoch:0d}.h5',
monitor='accuracy')
I am learning TensorFlow and was going through this step-by-step guide. The below code is the exact same as on the website. However, when running it, I get an error when trying to fit the model. The full traceback I get is as follows:
Traceback (most recent call last):
File "C:\users\name\desktop\python ml tutorial\embedding.py", line 49, in <module>
model.fit(x=padded_docs, y=labels, epochs=50, verbose=0)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\engine\training.py", line 1213, in fit
self._make_train_function()
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\engine\training.py", line 316, in _make_train_function
loss=self.total_loss)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\optimizer_v2\optimizer_v2.py", line 506, in get_updates
return [self.apply_gradients(grads_and_vars)]
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\optimizer_v2\optimizer_v2.py", line 441, in apply_gradients
kwargs={"name": name})
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\distribute\distribute_lib.py", line 1917, in merge_call
return self._merge_call(merge_fn, args, kwargs)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\distribute\distribute_lib.py", line 1924, in _merge_call
return merge_fn(self._strategy, *args, **kwargs)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\optimizer_v2\optimizer_v2.py", line 494, in _distributed_apply
with ops.control_dependencies(update_ops):
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 5257, in control_dependencies
return get_default_graph().control_dependencies(control_inputs)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\func_graph.py", line 356, in control_dependencies
return super(FuncGraph, self).control_dependencies(filtered_control_inputs)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 4691, in control_dependencies
c = self.as_graph_element(c)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3610, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3699, in _as_graph_element_locked
(type(obj).__name__, types_str))
TypeError: Can not convert a NoneType into a Tensor or Operation.
And the full code is below:
from numpy import array
from keras.preprocessing.text import one_hot
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.embeddings import Embedding
# define document
docs = ['Well done!',
'Good work',
'Great effort',
'nice work',
'Excellent!',
'Weak',
'Poor effort!',
'not good',
'poor work',
'Could have done better.']
# define class labels
labels = array([1,1,1,1,1,0,0,0,0,0])
# integer-encode the documents
vocab_size = 50
encoded_docs = [one_hot(d, vocab_size) for d in docs]
print(encoded_docs)
# padding
max_length = 4
padded_docs = pad_sequences(encoded_docs, maxlen = max_length, padding = 'post')
print(padded_docs)
# define model
model = Sequential()
model.add(Embedding(vocab_size, 8, input_length=max_length))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
# compile the model
model.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics=['accuracy'])
# summarize
print(model.summary())
# fit the model
model.fit(x=padded_docs, y=labels, epochs=50, verbose=0)
# evaluate the model
loss, accuracy = model.evaluate(x=padded_docs, y=labels, verbose=0)
print("Accuracy: {}".format(accuracy))
What's going on here? The article was originally written back in 2017 but its last revision and update was just a week ago. I imagine they are constantly tweaking TensorFlow since it is still state-of-the-art and needs a lot of improvement.
Any ideas on how to circumvent this?
Edit:
I began trying to figure out where the script could have gone wrong. I will be listing what I found here, hopefully it will help us spot something:
I found that in ops.py's control_dependencies() function, control_inputs parameter has the following values: Tensor("Adam/gradients/gradients/loss/dense_1_loss/binary_crossentropy/logistic_loss_grad/Reshape_1:0", shape=(None, 1), dtype=float32), Tensor("Adam/gradients/gradients/loss/dense_1_loss/binary_crossentropy/logistic_loss/Log1p_grad/mul:0", shape=(None, 1), dtype=float32), and None. When it becomes None, the program crashes.
DataSet
import keras
print(keras.__version__)
mnist = keras.datasets.mnist
(x_train,y_train),(x_test,y_test) = mnist.load_data()
normalizing
x_train = keras.utils.normalize(x_train,axis=1)
x_test = keras.utils.normalize(x_test,axis=1)
model
model = keras.models.Sequential()
model.add(keras.layers.Flatten(x_train))
model.add(keras.layers.Dense(128,activation= keras.nn.relu))
model.add(keras.layers.Dense(128,activation= keras.nn.relu))
model.add(keras.layers.Dense(10,activation= keras.nn.softmax))
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics = ['accuracy']
)
model.fit(x_train,y_train,epochs=3)
ERROR:
Using TensorFlow backend.
2.3.1
Traceback (most recent call last):
File "/Users/aditya/Desktop/Desktop/dataScience/Practice/OpenCV/FaceDetect/Hackathon/classMnist.py", line 28, in <module>
model.add(keras.layers.Flatten(x_train))
File "/usr/local/lib/python3.7/site-packages/keras/layers/core.py", line 495, in __init__
self.data_format = K.normalize_data_format(data_format)
File "/usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 311, in normalize_data_format
data_format = value.lower()
AttributeError: 'numpy.ndarray' object has no attribute 'lower'
The problem is that the Keras can't flatten the x_train. So Do you know why throw this error?
model.add(keras.layers.Flatten(x_train))
The Keras create a network that can't load data.
model.fit(x_train,y_train,epochs=3)
There are data loaded.
So you should edit the first code:
model.add(keras.layers.Flatten())
And your codes have other error:
# wrong
model.add(keras.layers.Dense(128,activation= keras.nn.relu))
# right
model.add(keras.layers.Dense(128,activation= keras.backend.relu))
I have two threads, one handling the training, and the other one is handling the estimations. I have several entities and I would like to have a model for every entity, so I load and save models "on the fly" (I know this is quite slow).
If I load the model every time I want to call to the predict function, this works. However, if I just want to load the model once, and make several predicts in a row, I have the following exceptions:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 1182, in run
self.function(*self.args, **self.kwargs)
File "/home/arroyadr/Proyectos/iot-ai-engine/src/trainer.py", line 388, in train
self.update_prediction_historics_all()
File "/home/arroyadr/Proyectos/iot-ai-engine/src/trainer.py", line 413, in update_prediction_historics_all
self.update_prediction_historics_dataset(new_dataset, loadModel=True)
File "/home/arroyadr/Proyectos/iot-ai-engine/src/trainer.py", line 444, in update_prediction_historics_dataset
loadModel=False)[0]
File "/home/arroyadr/Proyectos/iot-ai-engine/src/estimator.py", line 207, in get_predictions_sequential
prediction = model.predict(new_data)
File "/home/arroyadr/.local/lib/python3.6/site-packages/keras/engine/training.py", line 1169, in predict
steps=steps)
File "/home/arroyadr/.local/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop
batch_outs = f(ins_batch)
File "/home/arroyadr/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "/home/arroyadr/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2671, in _call
session)
File "/home/arroyadr/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2623, in _make_callable
callable_fn = session._make_callable_from_options(callable_opts)
File "/home/arroyadr/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1431, in _make_callable_from_options
return BaseSession._Callable(self, callable_options)
File "/home/arroyadr/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1385, in __init__
session._session, options_ptr, status)
File "/home/arroyadr/.local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 526, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Tensor lstm_1_input:0, specified in either feed_devices or fetch_devices was not found in the Graph
> Exception ignored in: <bound method BaseSession._Callable.__del__ of <tensorflow.python.client.session.BaseSession._Callable object at 0x7f1ca2b33748>>
Traceback (most recent call last):
File "/home/arroyadr/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1415, in __del__
self._session._session, self._handle, status)
File "/home/arroyadr/.local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 526, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.CancelledError: Session has been closed.
My code to load the model and predict is like the following:
def load_model_file(self, path=None):
"""
Load the model given in path
:param path: path of the model. If it is None it loads a default model
:return:
"""
lock = Lock()
lock.acquire()
model = None
if (path is not None):
if (os.path.isfile(path)):
if (not sklearn):
model = load_model(path)
# model = self.pred.get_model([1, self.num_previous_measures, 1, 1], activation=self.activation)
# model.load_weights(path)
model._make_predict_function()
self.graph = tf.get_default_graph()
# Load scalers
scalers = []
for i in range(self.num_features_dataset):
scaler = joblib.load(
'../rsc/datasets/scalers/' + path.split("/")[3].split(".")[0] + str(i) + '.pkl')
scalers.append(scaler)
lock.release()
if (model is None):
self.logger.error("No model could be found for " + str(path))
self.model_predict = None
self.scalers_predict = None
return None, None
else:
self.model_predict = model
self.scalers_predict = scalers
return model, scalers
def get_predictions_sequential(self, data, num_pred, column_data, path=None, loadModel=True):
"""
Predicts a list of values from the data given as param.
:param data: data (time series) from which predict the next value
:param num_pred: number of predictions
:param path: path where to read the model
:return: list of predictions
"""
# Load model, if there is no model, then it will try to train and set the scaler
if (loadModel):
# with filelock.FileLock(path + ".lock"):
model, scalers = self.load_model_file(path)
else:
model = self.model_predict
scalers = self.scalers_predict
# model._make_predict_function()
# Scale prediction data
data = np.reshape(np.array(data), (1, self.num_previous_measures, self.num_features_dataset))
for i in range(self.num_features_dataset):
data2 = data[:, :, i].copy().reshape(1, self.num_previous_measures)
data2 = np.insert(data2, self.num_previous_measures, data.mean())
data2 = np.reshape(data2, (1, self.num_previous_measures + 1))
data2 = scalers[i].transform(data2)
data[:, :, i] = data2[0][:-1]
predictions = []
new_data = data.copy()
for i in range(num_pred):
if (not sklearn):
with self.graph.as_default():
# with tf.Session(graph=self.graph) as sess:
# sess = tf.Session()
# K.set_session(sess)
prediction = model.predict(new_data)
# self.logger.info("Pred not scaled: "+ str(prediction[0]))
prediction_rescaled = self.invert_scale(scalers[column_data - 1],
new_data[0, :, column_data - 1],
prediction[0][0])
return predictions
I have read and tracked this issue, but I do not manage to find any proper solution.
Does anyone have had this problem?
After investigating and more trial/error, I have found a solution.
About the Session Cancelled Error, the answer 1 for this question could be useful:
K.clear_session() is useful when you're creating multiple models in succession, such as during hyperparameter search or cross-validation. Each model you train adds nodes (potentially numbering in the thousands) to the graph. TensorFlow executes the entire graph whenever you (or Keras) call tf.Session.run() or tf.Tensor.eval(), so your models will become slower and slower to train, and you may also run out of memory. Clearing the session removes all the nodes left over from previous models, freeing memory and preventing slowdown.
So what I did was to add the call K.clear_session() before loading the new model, this way we can avoid several models loading on the same graph.
On the other hand, I did NOT have to call
with self.graph.as_default():
with tf.Session(graph=self.graph) as sess:
backend.set_session(sess)
So loading the model function looks like this now:
def load_model_file(self, path=None):
"""
Load the model given in path
:param path: path of the model. If it is None it loads a default model
:return:
"""
model = None
if (path is not None):
if (os.path.isfile(path)):
if (not sklearn):
K.clear_session()
model = load_model(path)
# model = self.pred.get_model([1, self.num_previous_measures, 1, 1], activation=self.activation)
# model.load_weights(path)
model._make_predict_function()
self.graph = tf.get_default_graph()
else:
model = joblib.load(path)
# Load scalers
scalers = []
for i in range(self.num_features_dataset):
scaler = joblib.load(
'../rsc/datasets/scalers/' + path.split("/")[3].split(".")[0] + str(i) + '.pkl')
scalers.append(scaler)
if (model is None):
self.logger.error("No model could be found for " + str(path))
self.model_predict = None
self.scalers_predict = None
return None, None
else:
self.model_predict = model
self.scalers_predict = scalers
return model, scalers
For multithreading problems and loading different models consequently please keep this in mind:
Whenever I call the function get_predictions_sequential from other classes or threads, instead, I should just use directly in function get_predictions_sequential:
with self.graph.as_default():
prediction = model.predict(new_data)
And in this other classes try NOT to call any tf graph-related function or session-setting function, as you will be mixing the graph in one of the classes with the graph in the class with get_predictions_sequential function.
Cheers
I'm trying to create a LSTM model. Before passing the data to the first LSTM layer, I want to add a Masking layer. I am able to do this using Sequential approach in Keras. See example. However when I try to code it differently I get a value error (see below). Any Idea on how to fix this?
import keras
def network_structure(window_len, n_features, lstm_neurons):
masking = keras.layers.Masking(
mask_value=0.0, input_shape=(window_len, n_features)
)
lstm_h1 = keras.layers.LSTM(lstm_neurons)(masking)
lstm_h2 = keras.layers.LSTM(lstm_neurons)(lstm_h1)
cte = keras.layers.Dense(
1,
activation='linear',
name='CTE',
)(lstm_h2)
ate = keras.layers.Dense(
1,
activation='linear',
name='ATE',
)(lstm_h2)
pae = keras.layers.Dense(
1,
activation='linear',
name='PAE',
)(lstm_h2)
model = keras.models.Model(
inputs=masking,
outputs=[cte, ate, pae]
)
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mae'])
model.summary()
return model
model = network_structure(32, 44, 125)
Error Message:
Using TensorFlow backend.
Traceback (most recent call last):
File "C:\Python35\lib\site-packages\keras\engine\topology.py", line 442, in assert_input_compatibility
K.is_keras_tensor(x)
File "C:\Python35\lib\site-packages\keras\backend\tensorflow_backend.py", line 468, in is_keras_tensor
raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) + '`. '
ValueError: Unexpectedly found an instance of type `<class 'keras.layers.core.Masking'>`. Expected a symbolic tensor instance.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/Master Tk/PycharmProjects/FPL/testcompile.py", line 46, in <module>
model = network_structure(32, 44, 125)
File "C:/Users/Master Tk/PycharmProjects/FPL/testcompile.py", line 12, in network_structure
lstm_h1 = keras.layers.LSTM(lstm_neurons)(masking)
File "C:\Python35\lib\site-packages\keras\layers\recurrent.py", line 499, in __call__
return super(RNN, self).__call__(inputs, **kwargs)
File "C:\Python35\lib\site-packages\keras\engine\topology.py", line 575, in __call__
self.assert_input_compatibility(inputs)
File "C:\Python35\lib\site-packages\keras\engine\topology.py", line 448, in assert_input_compatibility
str(inputs) + '. All inputs to the layer '
ValueError: Layer lstm_1 was called with an input that isn't a symbolic tensor. Received type: <class 'keras.layers.core.Masking'>. Full input: [<keras.layers.core.Masking object at 0x000002224683A780>]. All inputs to the layer should be tensors.
You have forgotten to create an input layer. First define the input layer and then pass the placeholder tensor to the Masking layer:
inp = Input(shape=(window_len, n_features))
masking = keras.layers.Masking(mask_value=0.0)(inp)
lstm_h1 = keras.layers.LSTM(lstm_neurons)(masking)
And don't forget to change the model definition accordingly by passing the input tensor as the inputs argument:
model = keras.models.Model(inputs=inp, outputs=[cte, ate, pae])