I'm training models with a genetic algorithm, and the fitness function first builds the model with certain parameters, and then runs inference on the dataset with that model.
Obviously, genetic algorithms are very parallelizable, but I am running into problems loading and running multiple models at once. I am using Python's multiprocessing library and running the function that loads the model and runs inference using the Pool method. I get an error when doing this:
Blas GEMM launch failed : a.shape=(32, 50), b.shape=(50, 200), m=32, n=200, k=50
[[{{node lstm_1/while/body/_1/MatMul_1}}]] [Op:__inference_keras_scratch_graph_494]
Function call stack:
keras_scratch_graph
Not sure what's happening here, but the error isn't thrown when the models are not parallelized.
Any help is super appreciated.
Here is the code:
import tensorflow as tf
from keras import regularizers
from keras.optimizers import SGD
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Flatten, ELU, LSTM, PReLU, GRU, CuDNNGRU, CuDNNLSTM
from keras.callbacks import EarlyStopping, ModelCheckpoint
import numpy as np
from sklearn import preprocessing
from sklearn.preprocessing import minmax_scale
import matplotlib.pyplot as plt
import math, random, copy, pickle, time, statistics
import pandas as pd
import multiprocessing as mp
def buildModel(windowSize):
#model to use
sample_model = Sequential()
sample_model.add(CuDNNLSTM(50, input_shape = (windowSize, 5), return_sequences=False))
sample_model.add(Dense(50, activation="relu"))
sample_model.add(Dense(3, activation="softmax"))
sample_model.compile(optimizer="adam", loss="categorical_crossentropy")
sample_model.build()
#record weight and bias shape
modelShape = []
for i in range(len(sample_model.layers)):
layerShape = []
for x in range(len(sample_model.layers[i].get_weights())):
layerShape.append(sample_model.layers[i].get_weights()[x].shape)
modelShape.append(layerShape)
return(sample_model, modelShape)
model = BuildModel(120)
pool = mp.Pool(mp.cpu_count())
results = [pool.apply(model.predict, args=(np.array(features[x]), batch_size=len(features[i]))) for x in range(len(features))]
pool.close()
The features I'm using aren't important, they could all be lists of 120 random numbers or something. I didn't actually add the features I use because they are huge and from a really big file.
I just want to be able to run model.predict inside the pool.apply[] multiprocessing function so that I can run multiple predictions concurrently.
Related
I have an issue with calling Python code in MATLAB. My Python code involves predicting the battery state of charge using LSTM with attention ANN based on the inputs sent from MATLAB. The prediction is then sent back to MATLAB. I already have previously trained weights and biases saved in an h5 file, which is loaded and used in the Python code. Below is the Python code:
import pandas as pd
import numpy as np
import tensorflow as tf
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from tensorflow import keras
from tensorflow.keras import Sequential
from tensorflow.keras.layers import LSTM
from tensorflow.keras.layers import Dense
from tensorflow.keras import optimizers
import matplotlib.pyplot as plt
from tensorflow.keras.layers import *
from tensorflow.keras.models import *
from tensorflow.keras import backend as K
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
from sklearn.model_selection import train_test_split
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.models import load_model
from tensorflow.keras.layers import Dropout, InputLayer
import h5py
#to create sequential data
def create_inout_sequences(input_data, tw):
inout_seq = []
L = len(input_data)
for i in range(L-tw):
train_seq = input_data[i:i+tw]
#train_out = output_data[i:i+tw]
inout_seq.append(train_seq)
return inout_seq
def search(inputs):
class attention(Layer):
def __init__(self, return_sequences=True,**kwargs):
self.return_sequences = return_sequences
super(attention,self).__init__()
def build(self, input_shape):
self.W=self.add_weight(name="att_weight", shape=(input_shape[-1],1),
initializer="normal")
self.b=self.add_weight(name="att_bias", shape=(input_shape[1],1),
initializer="zeros")
super(attention,self).build(input_shape)
def call(self, x):
e=(K.dot(x,self.W)+self.b)
a = K.softmax(e, axis=1)
output = x*a
if self.return_sequences:
return output
return K.sum(output, axis=1)
def get_config(self):
# For serialization with 'custom_objects'
config = super().get_config()
config['return_sequences'] = self.return_sequences
return config
#convert test data to sequential form
inputs=np.array(inputs)
inputs=np.tile(inputs, (36, 1))
inputs_new=create_inout_sequences(inputs, 35)
inputs_new=np.array(inputs_new)
model1 = Sequential()
model1.add(InputLayer(input_shape=(35,5)))
model1.add((LSTM(22, return_sequences=True)))
model1.add(attention(return_sequences=False))
model1.add(Dense(104, activation="relu"))
model1.add(Dropout(0.2))
model1.add(Dense(1, activation="sigmoid"))
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
initial_learning_rate=0.01,
decay_steps=10000,
decay_rate=0.99)
model1.compile(optimizer=tf.keras.optimizers.Adam(epsilon=1e-08,learning_rate=lr_schedule),loss='mse')
#call previously trained weights
model1.load_weights('SOC_weights.h5')
x=float(model1.predict(inputs_new, batch_size=100,verbose=0))
return x # send prediction to Matlab
Note: I am using Python 3.6, tensorflow version: 2.5, keras version: 2.4.3, h5py version: 3.1.0, cython version: 0.28
I am able to run this code without any error on Python, but have issues when used in MATLAB 2020a... below is my MATLAB code:
pyenv('Version','3.6');
py.importlib.import_module('tensorflow');
py.importlib.import_module('testingSOC'); % file containing the Python codes
inputs=[0.555555556,0.435139205,0.68313128,0.499987472,0.241225578];% test inputs
SOC_output=py.testingSOC.search(inputs)
Below is the error received on Matlab:
Error using training>load_weights (line 2312)
Python Error: ImportError: `load_weights` requires h5py when loading weights from HDF5.
Error in testingSOC>search (line 87)
the error looks like h5py is not identified by MATLAB, so I have tried reinstalling h5py by using the command prompt (I am using Windows 10):
pip uninstall h5py
pip install h5py
but no changes...
I have also tried with tensorflow version: 2.2, keras version 2.4.3, h5py version 2.10 and cython version 0.29 but still get the same error.
I would really appreciate if you guys can provide an insight in solving this issue, and if there are any fundamental parts that I have missed. I would be glad to share more details if required.
Thanks!
Thanks to #TimRoberts for pointing out about including 'py.importlib.import_module('h5py')' which helped me in resolving this issue.Below is my solution, for those who would like to refer:
When I included 'py.importlib.import_module('h5py')' in my matlab codes, I received the following error:
Error using h5>init h5py.h5 (line 1)
Python Error: ImportError: DLL load failed: The specified procedure could not be found.
It looks like Python environment seems to use Matlab's h5 library in my case, which does not have the same features as Python's h5 library...I found that there is an option of running Python codes as a separate process which seems to be working for me (as seen in this link):
https://www.mathworks.com/help/matlab/matlab_external/out-of-process-execution-of-python-functionality.html?searchHighlight=out%20of%20process%20python&s_tid=srchtitle
I have constructed, fitted, and saved the following model:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras import preprocessing
from tensorflow.keras.models import Sequential
import config
from tensorflow.keras import applications
model = Sequential()
model.add(layers.Flatten(input_shape=input_shape.shape[1:]))
model.add(layers.Dense(100, activation=keras.layers.LeakyReLU(alpha=0.3)))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(50, activation=keras.layers.LeakyReLU(alpha=0.3)))
model.add(layers.Dropout(0.3))
model.add(layers.Dense(num_classes, activation='softmax'))
I am using the load_model function for evaluation, and I have not had any trouble up until now, but I am now getting the following error:
ValueError: Unknown activation function: LeakyReLU
Are there any syntactic changes to the architecture I should make, or is there a deeper issue here? Any advice would be appreciated, as I had already tried setting some custom objects as described here: https://github.com/BBQuercus/deepBlink/issues/107
Edit:
My imports in the file where I am calling load_model are the following:
import config
import numpy as np
from tensorflow.keras.preprocessing.image import img_to_array, load_img
from models.create_image_model import make_vgg
import argparse
from tensorflow.keras.models import load_model
import time
from tensorflow import keras
from tensorflow.keras import layers
There seem to be some issues when saving & loading models with such "non-standard" activations, as implied also in this SO thread; the safest way would seem to be to re-write your model with the LeakyReLU as a layer, and not as an activation:
model = Sequential()
model.add(layers.Flatten(input_shape=input_shape.shape[1:]))
model.add(layers.Dense(100)) # no activation here
model.add(layers.LeakyReLU(alpha=0.3)) # activation layer here instead
model.add(layers.Dropout(0.5))
model.add(layers.Dense(50)) # no activation here
model.add(layers.LeakyReLU(alpha=0.3)) # activation layer here instead
model.add(layers.Dropout(0.3))
model.add(layers.Dense(num_classes, activation='softmax'))
This is exactly equivalent to your own model, and more consistent with the design choices of Keras - which, for good or bad, includes LeakyReLU as a layer, and not as a standard activation function.
I have completed a udacity nanodegree for NLP. I used the udacity platform for the project, but I am now trying to use my own local machine to train models etc.
I've finally gotten my GPU/tensorflow issues worked out(I think), but I'm running into some problems that I believe are related to the versions of tensorflow that udacity was using.
I am currently using TensorFlow 2.2
Specifically, I am getting an error from a validation step the project uses to list the loss function.
def _test_model(model, input_shape, output_sequence_length, french_vocab_size):
if isinstance(model, Sequential):
model = model.model
print(model.loss_functions)
When this is called I get the "'Model' object has no attribute 'loss_functions'" error.
The model is built with the below code.
def simple_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
"""
Build and train a basic RNN on x and y
:param input_shape: Tuple of input shape
:param output_sequence_length: Length of output sequence
:param english_vocab_size: Number of unique English words in the dataset
:param french_vocab_size: Number of unique French words in the dataset
:return: Keras model built, but not trained
"""
# TODO: Build the layers
learning_rate = 0.01
#Config Model
inputs = Input(shape=input_shape[1:])
hidden_layer = GRU(output_sequence_length, return_sequences=True)(inputs)
outputs = TimeDistributed(Dense(french_vocab_size, activation='softmax'))(hidden_layer)
#Create Model from parameters defined above
model = keras.Model(inputs=inputs, outputs=outputs)
#loss_function = 'sparse_categorical_crossentropy'
loss_fn = keras.losses.SparseCategoricalCrossentropy()
model.compile(loss=loss_fn,optimizer=Adam(learning_rate),metrics=['accuracy'])
I am using the below libraries along the way
import tensorflow as tf
from tensorflow.keras.losses import sparse_categorical_crossentropy
from tensorflow.keras.optimizers import Adam
from tensorflow import keras
import collections
import helper
import numpy as np
import project_tests as tests
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.layers import GRU, Input, Dense, TimeDistributed, Activation, RepeatVector, Bidirectional, Dropout
from tensorflow.keras.layers.embeddings import Embedding
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import sparse_categorical_crossentropy
I can just comment out this check for the loss functions, but I would really like to understand what happened.
Thanks
I think the API changed in Tensorflow 2, does the following work:
model.compiled_loss._get_loss_object(model.compiled_loss._losses).fn
I'm lost as to how to import Tensorflow 2's built in datasets. Their docs aren't very intuitive and I'm used to working with csvs.
How do I get the 'Titanic' dataset to work with the basic model?
is there a good resource to learn Tensorflow's API for pipelining their datasets?
for the below code I get the error: ValueError: Layer sequential_54 expects 1 inputs, but it received 13 input tensors
import tensorflow as tf
import tensorflow_datasets as tfds
from tensorflow.keras.optimizers import Adam
data = tfds.load("titanic",split='train', as_supervised=True).map(lambda x,y: (x,y)).batch(10)
model = tf.keras.models.Sequential([tf.keras.layers.Dense(2,activation='relu'),
tf.keras.layers.Dense(13, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')])
model.compile(optimizer=Adam(learning_rate=0.01), loss='categorical_crossentropy', metrics=
['accuracy'])
model.fit(data,epochs=30)
The import of data seems to be correct, but you are using categorical_crossentropy which naturally needs one-hot encoded labels (targets), which can be generated using:
from keras.utils import to_categorical
labels = to_categorical(labels)
But for two class (binary) problem, you need to use binary_crossentropy and you can still maintain your dense layer:
tf.keras.layers.Dense(1, activation='sigmoid')])
Lastly, you need to add the labels (targets) here to train the network and possibly add batch size
model.fit(data, labels, epochs=30, batch_size=80)
I have issues with saving a sequential model produced by Keras to SavedModel format.
As been said in https://www.tensorflow.org/guide/keras/save_and_serialize#export_to_savedmodel ,
to save the Keras model to the format that could be used by TensorFlow, I need to use model.save() and provide save_format='tf', but what I have is:
Traceback (most recent call last):
File "load_file2.py", line 14, in <module>
classifier.save('/tmp/keras-model.pb', save_format='tf')
My code example is:
import pandas as pd
import tensorflow as tf;
import keras;
from keras import Sequential
from keras.layers import Dense
import json;
import numpy as np;
classifier = Sequential()
classifier.add(Dense(4, activation='relu', kernel_initializer='random_normal', input_dim=4))
classifier.add(Dense(1, activation='sigmoid', kernel_initializer='random_normal'))
classifier.compile(optimizer ='adam',loss='binary_crossentropy', metrics = ['accuracy'])
classifier.save('/tmp/keras-model.pb', save_format='tf')
My python is 3.6.10.
My tensorflow is 1.14 and 2.0 (I tested on both, my result is the same).
My keras is 2.3.1.
What is wrong there or what should I change to make my model saved and then used by tensorflow?
Or, maybe, there is another way of saving models from Keras with TensorFlow2 as backend?
Thanks.
I ran your code. With tensorflow 1.15 I got type error saying save_format is not a known parameter. With tensorflow 2 I got the suggesstion to use tf.keras instead of native keras. So, I tried tf.keras instead of keras. This time the code ran with no error.
Also, I don't see a fit method before saving the model.
With TF2.0:
import pandas as pd
import tensorflow as tf;
##Change.
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
import json;
import numpy as np;
classifier = Sequential()
classifier.add(Dense(4, activation='relu', kernel_initializer='random_normal', input_dim=4))
classifier.add(Dense(1, activation='sigmoid', kernel_initializer='random_normal'))
classifier.compile(optimizer ='adam',loss='binary_crossentropy', metrics = ['accuracy'])
classifier.save('/tmp/keras-model.pb', save_format='tf')
Result:
INFO:tensorflow:Assets written to: /tmp/keras-model.pb/assets