Develop Reference

BioBERT Inferencing Using TF_Record file and GRPC

I have been trying to perform inference using bioBERT(https://github.com/dmis-lab/biobert)
Model and TF serve to perform QA task.
I have succesfully exported the model: My serving function looks like this:
feature_columns = [
tf.feature_column.numeric_column("unique_ids", shape=(FLAGS.max_seq_length,), dtype=tf.int64),
tf.feature_column.numeric_column("input_ids", shape=(FLAGS.max_seq_length,), dtype=tf.int64),
tf.feature_column.numeric_column("input_mask", shape=(FLAGS.max_seq_length,), dtype=tf.int64),
tf.feature_column.numeric_column("segment_ids", shape=(FLAGS.max_seq_length,), dtype=tf.int64)
]
serving_input_fn=tf.estimator.export.build_parsing_serving_input_receiver_fn(tf.feature_column.make_parse_example_spec(feature_columns))
estimator._export_to_tpu = False
estimator_path = estimator.export_saved_model(estimator_base_path, serving_input_fn, checkpoint_path)
##############################################
I am also able to generate a TFrecord File and trying to utilize TFrecordIterator to iterate over tht tf records file and call the GRPC generated stub.
#record_path is the path to TF_record filw
Function below....
all_results = [ ]
record_iterator = tf.python_io.tf_record_iterator(pathToTfRecordFile)
for string_record in record_iterator:
model_request.inputs['examples'].CopyFrom(
tf.contrib.util.make_tensor_proto(string_record,
dtype=tf.string,
shape=[batch_size])
)
result_future = stub.Predict.future(model_request, 30.0)
result = result_future.result().outputs
all_results.append(process_result(result))
The Error I am getting is as follows:
_MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Name: <unknown>, Key: unique_ids, Index: 0. Number of int64 values != expected. Values size: 1 but output shape: [384]
Any help on this issue is appreciated.

Try shaping the unique_ids like this in the service_inputfn.
tf.feature_column.numeric_column("unique_ids", shape=(1,), dtype=tf.int64),

Tensorflow/tfjs "When inputs is an array, neither initialState or constants should be provided"

I am trying to make a ChatBot using python (tensorflow/keras) for making, training and converting the neural network and then using it in my Angular app with tensorflow/tfjs. I was following the example found here: https://github.com/tensorflow/tfjs-examples/tree/master/translation but trying to add an embedding layer as well.
Creating the model:
latent_dim = 200
encoder_inputs = Input(shape=(encoder_max_length, ), dtype='int32', )
encoder_embedding = Embedding(num_tokens,
embedd_size,
weights=[word2em],
input_length=encoder_max_length,
mask_zero=True,
trainable=False
)(encoder_inputs)
encoder_outputs, state_h, state_c = LSTM(latent_dim, return_state=True)(encoder_embedding)
encoder_states = [state_h, state_c]
decoder_inputs = Input(shape=(decoder_max_length, ), dtype='int32', )
decoder_embedding = Embedding(num_tokens,
embedd_size,
weights=[word2em],
input_length=decoder_max_length,
mask_zero=True,
trainable=False
)(decoder_inputs)
decoder_LSTM = LSTM(latent_dim, return_state=True, return_sequences=True)
decoder_outputs, _, _ = decoder_LSTM(decoder_embedding, initial_state=encoder_sates)
decoder_dense = Dense(num_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
Creating encoder / decoder models (tfjs):
If I remove 'initialState' from LSTM.apply() the error disappears but the result is the same sentence no matter the input.
prepareEncoder(model) {
const encInputs = model.input[0];
const stateH = model.layers[4].output[1];
const stateC = model.layers[4].output[2];
const encoderStates = [stateH, stateC];
this.encoder = tf.model({inputs: encInputs, outputs: encoderStates});
}
prepareDecoder(model) {
const tmp = model.layers[4].output[1];
const latentDim = tmp.shape[tmp.shape.length - 1];
const decoderStateInputH = tf.input({shape: [latentDim], name: 'decoder_state_input_h'});
const decoderStateInputC = tf.input({shape: [latentDim], name: 'decoder_state_input_c'});
const decoderStateInputs = [decoderStateInputH, decoderStateInputC];
const decoderLSTM = model.layers[5];
const decoderInputs = model.input[1];
const decoderEmbedding = decoderLSTM.input[0];
const applyOutputs = decoderLSTM.apply(decoderEmbedding, {initialState: decoderStateInputs});
let decoderOutputs = applyOutputs[0];
const decoderStateH = applyOutputs[1];
const decoderStateC = applyOutputs[2];
const decoderStates = [decoderStateH, decoderStateC];
const decoderDense = model.layers[6];
decoderOutputs = decoderDense.apply(decoderOutputs);
this.decoder = tf.model({
inputs: [decoderInputs].concat(decoderStateInputs),
outputs: [decoderOutputs].concat(decoderStates)
});
}
Predicting with the model:
botReply(input_seq) {
let states_value = this.encoder.predict(input_seq);
let target_seq = tf.buffer([1, data['dec_max_length']], 'int32');
target_seq.set(dict['<START>'], 0, 0);
let stop_condition = false;
let decoded_sentence = '';
let word_count = 1;
while (!stop_condition) {
let predict_outputs = this.decoder.predict([target_seq.toTensor()].concat(states_value));
And this is where it all fails and I get the following error:
ERROR Error: When inputs is an array, neither initialState or constants should be provided
at standardizeArgs (recurrent.js:54)
at LSTM.apply (recurrent.js:465)
at execute (executor.js:275)
at training.js:856
at engine.js:307
at Engine.scopedRun (engine.js:317)
at Engine.tidy (engine.js:306)
at Module.tidy (globals.js:166)
at training.js:839
at engine.js:307
Things that might be worth mentioning.
The problem only appears in tfjs. In python it works fine and the results are decent given the dataset and training epochs. And the inference models are made in the exact same way is in tfjs
Back when i wasn't using the embedding layer and only one-hot encoded vectors for inputs and outputs it worked so i doubt it's a version problem or sth.
The embedding is made with word2vec on the same dataset as the training of the model.
I convert the keras model with tfjsconverter.
Any help will be deeply appreciated because I really am running out of ideas.

Try this - remove 'mask_zero=True' from your Embedding layers and see if this resolves the problem.

Sequential model always predicts 5

So I've got a sequential model from mostly following along with this video. I've also got a drawing app done in JS and am sending the contents of that canvas to a flask server using JQuery. All of that seems to be working fine but when I give my model the image from the canvas and call predict it always gives me back the same exact response. The values at each index are always the same.
Am I missing something simple? Am I calling the predict wrong in the server or is it something to do with the model itself? It gets the predictions correct in the notebook.
Thanks for your time!
Flask server:
model = load_model('MyModel.h5')
imageWidth = 28
imageHeight = 28
dim = (imageWidth, imageHeight)
# Create the flask web application
app = fl.Flask(__name__)
#app.route('/uploadimage', methods = ['GET', 'POST'])
def uploadimage():
# Get the image from the request
theImage = fl.request.values.get("theImage", "")
#Decode the string to an image
decodedimg = base64.b64decode(theImage[22:])
# Save the image
with open ("theImage.png", "wb") as f:
f.write(decodedimg)
# Open the image from the request as originalImage
originalImage = Image.open("theImage.png")
# Resize it
resizedImage = ImageOps.fit(originalImage, dim, Image.ANTIALIAS)
# Confirm the dimensions of the resized image
w1, h1 = resizedImage.size
print(w1, h1)
# Save it locally
resizedImage.save("resizedImage.png", quality=100, optimize=True)
# Convert to grayscale and then convert that to an array
grayscaleImage = ImageOps.grayscale(resizedImage)
grayscaleArray = np.array(grayscaleImage)
grayscaleArray.shape # Gives (20, 20)
grayscaleArray = grayscaleArray.reshape(1, 28, 28)
setPrediction = model.predict(grayscaleArray)
print(setPrediction) # Always gives back same values
getPrediction = np.array(setPrediction[0])
predictedNumber = str(np.argmax(getPrediction))
print(predictedNumber) # Always '5'
return predictedNumber
Model:
model = kr.models.Sequential() # Create a new sequential neural network
model.add(kr.layers.Flatten()) # Input layer
model.add(kr.layers.Dense(128, activation="relu")) # 128 neurons and the 'basic' activation function.
model.add(kr.layers.Dense(128, activation="relu"))
model.add(kr.layers.Dense(10, activation="softmax"))
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"]) # Played around with 'sgd' and 'rmsporp' optimizer also.
model.fit(X_train, y_train, epochs=3)
val_loss, val_acc = model.evaluate(X_test, y_test)
print(val_loss, val_acc)

Predictions do not end in Tensorflow

I wrote a neural network using the Tensorflow framework but, when I try to make some predictions the program doesn't end.
The program contains a lot of files and I will try to post here the main ones but, in short, I have a custom model function and an Experimenter to train and to evaluate it.
I have to use the Experimenter API because the program needs to run both locally and in cloud but, in the first case, I don't want use Tensorflow Serving to run my predictions.
Note: The project is similar to the Google Cloud Platform sample that you can find at this address
My model function
def generate_model_fn(...):
def _model_fn(features, labels, mode):
# Pop the name of the signal.
if 'FN' in features:
names = features.pop('FN')
if 'FT' in features:
labels = features.pop('FT')
columns = [layers.real_valued_column(key) for key, value in features.items()]
inputs = layers.input_from_feature_columns(features, columns)
hidden_layers = None
# Iterate all over the hidden units.
for unit in hidden_units:
hidden_layers = tf.layers.dense(
inputs=inputs if hidden_layers is None else hidden_layers,
activation=tf.nn.relu,
units=unit,
)
dropout_layer = layers.dropout(inputs=hidden_layers,keep_prob=1.0 - dropout)
logits = tf.layers.dense(inputs=dropout_layer, activation=None, units=2)
if mode in (ModeKeys.PREDICT, ModeKeys.EVAL):
probabilities = tf.nn.softmax(logits)
predictions = tf.argmax(logits, 1)
if mode == ModeKeys.PREDICT:
predictions = {
'classes': predictions,
'scores': probabilities,
}
export_outputs = {
'prediction': tf.estimator.export.PredictOutput(predictions)
}
return tf.estimator.EstimatorSpec(
mode=mode,
predictions=predictions,
export_outputs=export_outputs
)
return _model_fn
The experimenter
def generate_experimenter_fn(**args):
def _experimenter_fn(run_config, hparams):
training_fn = lambda: generate_input_fn(...)
evaluating_fn = lambda: generate_input_fn(...)
return learn.Experiment(
tf.estimator.Estimator(
generate_model_fn(
learning_rate=hparams.learning_rate,
hidden_units=hparams.hidden_units,
dropout=hparams.dropout,
weights=hparams.weights,
),
config=run_config,
),
train_input_fn=training_fn,
eval_input_fn=evaluating_fn,
**args
)
return _experimenter_fn
learn_runner.run(
generate_experimenter_fn(
train_steps=args.train_steps,
eval_steps=args.eval_steps,
),
run_config=run_config.RunConfig(model_dir=args.job_dir),
hparams=hparam.HParams(**args.__dict__),
)
The predictions
Everything posted before works perfectly, now it's time to make some predictions. As I said I don't want use Tensorflow Serving locally so, I'm trying to create an Estimator using the same model and the same configurations.
classifier = tf.estimator.Estimator(
generate_model_fn(
learning_rate=args.learning_rate,
hidden_units=args.hidden_units,
dropout=args.dropout,
weights=args.weights,
),
config=run_config.RunConfig(model_dir=args.job_dir),
)
The model seems restore correctly according to the log file, so I want try to make a simple prediction with just one record. This program will never end and it is running until my machine will be completely tilted.
def predict_input_fn():
x = {
'FN': tf.constant(['A']),
'SS': tf.constant([1]),
'SN': tf.constant([2]),
'SL': tf.constant([3]),
'NS': tf.constant([4]),
'NN': tf.constant([5]),
'NL': tf.constant([6]),
'LS': tf.constant([7]),
'LN': tf.constant([8]),
'LL': tf.constant([9]),
'FT': tf.constant([0])
}
y = x['FT']
return x,y
predictions = classifier.predict(input_fn=predict_input_fn)

Tensor model input - nvalidArgumentError (see above for traceback): Shape in shape_and_slice spec

Hey I am trying to set an input point for the model that i have writen in tensorflow
this is the code for the classification
n_dim = training_features.shape[1]
x = tf.placeholder(tf.float32, [None,n_dim])
classifier = (...)
init_op = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init_op)
classifier.fit(training_features, training_labels, steps=100)
accuracy_score = classifier.evaluate(testing_features, testing_labels, steps=100)["accuracy"]
print('Accuracy', accuracy_score)
pred_a = np.asarray([x])
prediction = format(list(classifier.predict(pred_a)))
prediction_result = np.array(prediction)
output = tf.convert_to_tensor(prediction_result,dtype=None,name="output", preferred_dtype=None)
and here is my building code
export_path_base = sys.argv[-1]
export_path = os.path.join(
compat.as_bytes(export_path_base),
compat.as_bytes(str(FLAGS.model_version)))
print('Exporting trained model to', export_path)
builder = saved_model_builder.SavedModelBuilder(export_path)
classification_inputs = utils.build_tensor_info(y)
classification_outputs_classes = utils.build_tensor_info(output)
print('classification_signature...')
classification_signature = signature_def_utils.build_signature_def(
inputs={signature_constants.CLASSIFY_INPUTS: classification_inputs},
outputs={
signature_constants.CLASSIFY_OUTPUT_CLASSES:
classification_outputs_classes
},
method_name=signature_constants.CLASSIFY_METHOD_NAME)
tensor_info_x = utils.build_tensor_info(x)
print('prediction_signature...')
prediction_signature = signature_def_utils.build_signature_def(
inputs={'input': tensor_info_x},
outputs={
'classes' : classification_outputs_classes
},
method_name=signature_constants.PREDICT_METHOD_NAME)
print('Exporting...')
legacy_init_op = tf.group(tf.tables_initializer(), name='legacy_init_op')
builder.add_meta_graph_and_variables(
sess, [tag_constants.SERVING],
signature_def_map={
'predict_sound':
prediction_signature,
signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
classification_signature,
},
legacy_init_op=legacy_init_op)
builder.save()
print('Saved...')
I have tried manually passing dummy data before the build and that works but i am trying to have the client stub pass data into the model dynamically.
When i try run that code to build i get this error
InvalidArgumentError (see above for traceback): Shape in
shape_and_slice spec [1,280] does not match the shape stored in
checkpoint: [193,280] [[Node: save/RestoreV2_1 =
RestoreV2[dtypes=[DT_FLOAT],
_device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_1/tensor_names, save/RestoreV2_1/shape_and_slices)]]
May main goal is to have x as an input and output return the results, the output works but cant get the input to work.

Edit: If you just put the np.array as input without going through an input function, it will work but you also give up the chance to check the input.
Tensorflow won't check your input even if it's in the wrong shape or type or somehow corrupted but to throw such an error in the middle of session. Since you can run it with your test data successfully, the problem should be your actual data. Thus, it's recommended to write an input function to check your data before put it into classifier. Note that input function should return tf.Tensor with the shape of [x,1] (x is the number of your features) instead of an np.array.
Please refer to https://www.tensorflow.org/get_started/input_fn to see how to write your own input function and pass it to the classifier.
An example of input function:
def input_fn_predict(): # returns x, None
#do your check here or you can just print it out
feature_tensor = tf.constant(pred_a,shape=[1,pred_a.size])
return feature_tensor,None