Looking at this tutorial here, they use a starting sequence of “Romeo: “.
int(generate_text(model, start_string=u"ROMEO: "))
However, looking at the actual generation step, is it fair to say it’s only using the last character “ “? So it’s the same whether we use “ROMEO: “ or just “ “? It’s hard to test as it samples from the output distribution ...
Relatedly, it’s unclear how it would predict from such a short string as the original training sequence is much longer. I understand if we trained on a history of 100 chars we predict the 101st and then use 2-101 to predict 102... but how does it start with just 7 characters?
EDIT
As a specific example, I reworked my model to be of the following form:
model = tf.keras.Sequential()
model.add(tf.keras.layers.SimpleRNN(units=512, input_shape = (seq_len, 1), activation="tanh"))
model.add(tf.keras.layers.Dense(len(vocab)))
model.compile(loss=loss, optimizer='adam')
model.summary()
Notice, I use a simpleRNN instead of a GRU and drop the embedding step. Both of those changes are to simplify the model but that shouldn't matter.
My training and output data is as follows:
>>> input_array_reshaped
array([[46., 47., 53., ..., 39., 58., 1.],
[ 8., 0., 20., ..., 33., 31., 10.],
[63., 1., 44., ..., 58., 46., 43.],
...,
[47., 41., 47., ..., 0., 21., 57.],
[59., 58., 1., ..., 1., 61., 43.],
[52., 57., 43., ..., 1., 63., 53.]])
>>> input_array_reshaped.shape
(5000, 100)
>>> output_array_reshaped.shape
(5000, 1, 1)
>>> output_array_reshaped
array([[[40.]],
[[ 0.]],
[[56.]],
...,
[[ 1.]],
[[56.]],
[[59.]]])
However, if I try to predict on a string less than 100 characters I get:
ValueError: Error when checking input: expected simple_rnn_1_input to have shape (100, 1) but got array with shape (50, 1)
Below is my prediction function if needed. If I change the required_training_length to anything but 100 it crashes. It requires "specifically" time_steps of length 100.
Can someone tell me how to adjust the model to make it more flexible as in the example? What subtlety am I missing?
def generateText(starting_corpus, num_char_to_generate = 1000, required_training_length = 100):
random_starting_int = random.sample(range(len(text)),1)[0]
ending_position = random_starting_int+required_training_length
starting_string = text[random_starting_int:ending_position]
print("Starting string is: " + starting_string)
numeric_starting_string = [char2idx[x] for x in starting_string]
reshaped_numeric_string = np.reshape(numeric_starting_string, (1, len(numeric_starting_string), 1)).astype('float32')
output_numeric_vector = []
for i in range(num_char_to_generate):
if i%50 == 0:
print("Processing character index: "+str(i))
predicted_values = model.predict(reshaped_numeric_string)
selected_predicted_value = tf.random.categorical(predicted_values, num_samples = 1)[0][0].numpy().astype('float32') #sample from the predicted values
#temp = reshaped_numeric_string.copy()
output_numeric_vector.append(selected_predicted_value)
reshaped_numeric_string = np.append(reshaped_numeric_string[:,1:,:], np.reshape(selected_predicted_value, (1,1,1)), axis = 1)
predicted_chars = [idx2char[x] for x in output_numeric_vector]
final_text = ''.join(predicted_chars)
return(final_text)
However, looking at the actual generation step, is it fair to say
it’s only using the last character “ “? So it’s the same whether we
use “ROMEO: “ or just “ “? It’s hard to test as it samples from the
output distribution ...
No, it is taking all characters into consideration. You can easily
verify that by using a fixed random seed:
from numpy.random import seed
from tensorflow.random import set_seed
seed(1)
set_seed(1)
print('======')
print(generate_text(m, 'ROMEO: '))
seed(1)
set_seed(1)
print('======')
print(generate_text(m, ' '))
Relatedly, it’s unclear how it would predict from such a short
string as the original training sequence is much longer. I
understand if we trained on a history of 100 chars we predict the
101st and then use 2-101 to predict 102... but how does it start
with just 7 characters?
Internally it runs the sequence in a loop. It takes the first
character and predicts the second. Then the second to predict the
third and so on. While doing so it updates its hidden state so that
its predictions becomes better and better. Eventually it plateaus
because it cannot remember arbitrary long sequences.
Related
Please see the following code associated with output,
import torch
import torch.nn as nn
inputTensor = torch.tensor([1.0, 2.0, 3, 4, 5])
outplace_dropout = nn.Dropout(p=0.4)
print(inputTensor)
output_afterDropout = outplace_dropout(inputTensor)
print(output_afterDropout)
print(inputTensor)
The output is:
tensor([1., 2., 3., 4., 5.])
tensor([1.6667, 3.3333, 0.0000, 6.6667, 0.0000])
tensor([1., 2., 3., 4., 5.])
Could you please elaborate why the input tensor values are still unchanged?
From the documentation of torch.nn.Dropout, you can see that the inplace argument defaults to False. If you wish to change the input tensor in place, change the initialization to:
outplace_dropout = nn.Dropout(p=0.4, inplace=True)
I understand that Flatten removes all of the dimensions except for one. For example, I understand flatten():
> t = torch.ones(4, 3)
> t
tensor([[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]])
> flatten(t)
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
However, I don't get Flatten, especially I don't get meaning of this snippet from the doc:
>>> input = torch.randn(32, 1, 5, 5)
>>> m = nn.Sequential(
>>> nn.Conv2d(1, 32, 5, 1, 1),
>>> nn.Flatten()
>>> )
>>> output = m(input)
>>> output.size()
torch.Size([32, 288])
I felt the output should have size [160], because 32*5=160.
Q1. So why it outputted size [32,288]?
Q2. I also don't get meaning of shape information given in the doc:
Q3. And also meaning of parameters:
It is a difference in the default behaviour. torch.flatten flattens all dimensions by default, while torch.nn.Flatten flattens all dimensions starting from the second dimension (index 1) by default.
You can see this behaviour in the default values of the start_dim and end_dim arguments. The start_dim argument denotes the first dimension to be flattened (zero-indexed), and the end_dim argument denotes the last dimension to be flattened. So, when start_dim=1, which is the default for torch.nn.Flatten, the first dimension (index 0) is not flattened, but it is included when start_dim=0, which is the default for torch.flatten.
The reason behind this difference is probably because torch.nn.Flatten is intended to be used with torch.nn.Sequential, where typically a series of operations are performed on a batch of inputs, where each input is treated independently of the others. For example, if you have a batch of images and you call torch.nn.Flatten, the typical use case would be to flatten each image separately, and not flatten the whole batch.
If you do want to flatten all dimensions using torch.nn.Flatten, you can simply create the object as torch.nn.Flatten(start_dim=0).
Finally, the shape information in the docs just covers how the shape of the tensor will be affected, illustrating that the first (index 0) dimension is left as it is. So, if you have an input tensor of shape (N, *dims), where *dims is an arbitrary sequence of dimensions, the output tensor will have the shape (N, product of *dims), since all dimensions except the batch dimension are flattened. For example, an input of shape (3,10,10) will have an output of shape (3, 10 x 10) = (3, 100).
I want to feed a batch_size integer as a placeholder in Tensorflow. But it does not act as an integer. Consider the following example:
import tensorflow as tf
max_length = 5
batch_size = 3
batch_size_placeholder = tf.placeholder(dtype=tf.int32)
mask_0 = tf.one_hot(indices=[0]*batch_size_placeholder, depth=max_length, on_value=0., off_value=1.)
mask_1 = tf.one_hot(indices=[0]*batch_size, depth=max_length, on_value=0., off_value=1.)
# new session
with tf.Session() as sess:
feed = {batch_size_placeholder : 3}
batch, mask0, mask1 = sess.run([
batch_size_placeholder, mask_0, mask_1
], feed_dict=feed)
When I print the values of batch, mask0 and mask1 I have the following:
print(batch)
>>> array(3, dtype=int32)
print(mask0)
>>> array([[0., 1., 1., 1., 1.]], dtype=float32)
print(mask1)
>>> array([[0., 1., 1., 1., 1.],
[0., 1., 1., 1., 1.],
[0., 1., 1., 1., 1.]], dtype=float32)
Indeed I thought mask0 and mask1 must be the same, but it seems that Tensorflow does not treat batch_size_placeholder as an integer. I believe it would be a tensor, but is there anyway that I can use it as an integer in my computations?
Is there anyway I can fix this problem? Just FYI, I used tf.one_hot as just an example, I want to run train/validation during training in my code where I will need a lot of other computations with different values for batch_size in training and in validation steps.
Any help would be appreciated.
In pure python usage, [0]*3 will be [0,0,0]. However, batch_size_placeholder is a placeholder, during the graph execution, it will be a tensor. [0]*tensor will be regarded as tensor multiplication. In your case, it will be a 1-d tensor which has 0 value. To correctly use batch_size_placeholder, you should create a tensor which has the same length as batch_size_placeholder.
mask_0 = tf.one_hot(tf.zeros(batch_size_placeholder, dtype=tf.int32), depth=max_length, on_value=0., off_value=1.)
It will have the same result as mask_1.
A simple example to show the difference.
batch_size_placeholder = tf.placeholder(dtype=tf.int32)
a = [0]*batch_size_placeholder
b = tf.zeros(batch_size_placeholder, dtype=tf.int32)
with tf.Session() as sess:
print(sess.run([a, b], feed_dict={batch_size_placeholder : 3}))
# [array([0], dtype=int32), array([0, 0, 0], dtype=int32)]
I am trying to implement a OCR project by Keras.So I try to learn from Keras OCR example.I have use my own train data to train a new model and get the .H5 modelfile.
Now I want to test a new image to see my model performance，so I code a
test.py like this:
from keras.models import Model
import cv2
from keras.preprocessing.image import img_to_array
import numpy as np
from keras.models import load_model
from keras import backend as K
from allNumList import alphabet
def labels_to_text(labels):
ret = []
for c in labels:
if c == len(alphabet): # CTC Blank
ret.append("")
else:
ret.append(alphabet[c])
return "".join(ret)
def decode_predict_ctc(out, top_paths = 1):
results = []
beam_width = 5
if beam_width < top_paths:
beam_width = top_paths
for i in range(top_paths):
lables = K.get_value(K.ctc_decode(out, input_length=np.ones(out.shape[0])*out.shape[1],
greedy=False, beam_width=beam_width, top_paths=top_paths)[0][i])[0]
text = labels_to_text(lables)
results.append(text)
return results
def test(modelPath,testPicTest):
img=cv2.imread(testPicTest)
img=cv2.resize(img,(128,64))
img=img_to_array(img)
img=np.array(img,dtype='float')/255.0
img=np.expand_dims(img, axis=0)
img=img.swapaxes(1,2)
model=load_model(modelPath,custom_objects = {'<lambda>': lambda y_true, y_pred: y_pred})
net_out_value = model.predict(img)
top_pred_texts = decode_predict_ctc(net_out_value)
return top_pred_texts
result=test(r'D:\code\testAndExperiment\py\KerasOcr\weights.h5',r'D:\code\testAndExperiment\py\KerasOcr\test\avo.jpg')
print(result)
but I get a error like this:
Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 4 array(s), but instead got the following list of 1 arrays: [array([[[[1., 1., 1.], [1., 1., 1.], [1., 1., 1.], ..., [1., 1., 1.], [1., 1., 1.], [1., 1., 1.]], [[1., 1., 1.], [1., 1., 1.],...
I have references some material：
https://stackoverflow.com/a/49537697/10689350
https://www.dlology.com/blog/how-to-train-a-keras-model-to-recognize-variable-length-text/
How to predict the results for OCR using keras image_ocr example?
some answer show that we should use 4 inputs [input_data, labels, input_length, label_length] in training but besides input_data, everything else is information used only for calculating the loss,so in testing maybe use the input_data is enough.So I just use a picture without labels, input_length, label_length.But I get the error above.
I am confused about if the model needs 4 inputs or 1 in testing?
It doesn't seem reasonable to require 4 inputs during the testing process.and now I have model.h5,what should I do next?
Thanks in advance.
My code is Here:https://github.com/hqabcxyxz/KerasOCR/tree/master
maybe I know why.Because in the OCR example,we make a lambda layer to count CTC loss.This Layer need 4 inputs!
The right way to do test is we make a model without this lambda layer during inference.Then load the model weight by name to do inference.After we get inference result,just use CTC decode it!
I will update my code in github later.....
I'm trying to set the LSTM internal state before training each batch.
I'm sharing my test code and findings, hoping to find an answer and help others that are addressing similar problems.
In particular, for each data I have a feature X (which doesn't change over time) and a sequence P = p1, p2, p3,... p30.
The goal is: given X and p1,p2,p3 predict p4, p5, .. p30.
To this aim, I want to initialize the hidden state of an LSTM with X, as done in several works (e.g., neuraltalk), then the LSTM has to be fit with p1,p2,p3 to predict p4,..,p30.
This initialization is needed before each batch (batch_size=1), therefore I need to have the control of the LSTM states initialization.
Considerint this question Initializing LSTM hidden state Tensorflow/Keras I've tested the following code:
First of all I've added some prints in the reset_states() function defined in recurrent.py, in order to understand what exactly happens.
def reset_states(self, states=None):
if not self.stateful:
raise AttributeError('Layer must be stateful.')
batch_size = self.input_spec[0].shape[0]
if not batch_size:
raise ValueError('If a RNN is stateful, it needs to know '
'its batch size. Specify the batch size '
'of your input tensors: \n'
'- If using a Sequential model, '
'specify the batch size by passing '
'a `batch_input_shape` '
'argument to your first layer.\n'
'- If using the functional API, specify '
'the time dimension by passing a '
'`batch_shape` argument to your Input layer.')
# initialize state if None
if self.states[0] is None:
self.states = [K.zeros((batch_size, self.units))
for _ in self.states]
print "reset states A (all zeros)"
elif states is None:
for state in self.states:
K.set_value(state, np.zeros((batch_size, self.units)))
print "reset states B (all zeros)"
else:
if not isinstance(states, (list, tuple)):
states = [states]
print "reset states C (list or tuple copying)"
if len(states) != len(self.states):
raise ValueError('Layer ' + self.name + ' expects ' +
str(len(self.states)) + ' states, '
'but it received ' + str(len(states)) +
' state values. Input received: ' +
str(states))
for index, (value, state) in enumerate(zip(states, self.states)):
if value.shape != (batch_size, self.units):
raise ValueError('State ' + str(index) +
' is incompatible with layer ' +
self.name + ': expected shape=' +
str((batch_size, self.units)) +
', found shape=' + str(value.shape))
K.set_value(state, value)
print "reset states D (set values)"
print value
print "\n"
Here is the test code:
import tensorflow as tf
from keras.layers import LSTM
from keras.layers import Input
from keras.models import Model
import numpy as np
import keras.backend as K
input = Input(batch_shape=(1,3,1))
lstm_layer = LSTM(10,stateful=True)(input)
>>> reset states A (all zeros)
As you can see, the first print is executed when the lstm layer is created
model = Model(input,lstm_layer)
model.compile(optimizer="adam", loss="mse")
with tf.Session() as sess:
tf.global_variables_initializer().run()
h = sess.run(model.layers[1].states[0])
c = sess.run(model.layers[1].states[1])
print h
>>> [[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]
print c
>>> [[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]
The internal states have been set to all zeros.
As an alternative the function reset_states() can be used
model.layers[1].reset_states()
>>> reset states B (all zeros)
The second message has been printed in this case. Everything seem to work correctly.
Now I want to set the states with arbitrary values.
new_h = K.variable(value=np.ones((1, 10)))
new_c = K.variable(value=np.ones((1, 10))+1)
model.layers[1].states[0] = new_h
model.layers[1].states[1] = new_c
with tf.Session() as sess:
tf.global_variables_initializer().run()
h = sess.run(model.layers[1].states[0])
c = sess.run(model.layers[1].states[1])
print h
>>> [[ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]]
print c
>>> [[ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]]
Ok, I've successfully set both hidden states with my vectors of all one and all two.
However, it worth to exploit the class function reset_states() which takes as input the states.
This function exploits the function K.set_values(x,values) which expects 'values' to be a numpy array.
new_h_5 = np.zeros((1,10))+5
new_c_24 = np.zeros((1,10))+24
model.layers[1].reset_states([new_h_5,new_c_24])
It seems to work, indeed the output is:
>>> reset states D (set values)
>>> [[ 5. 5. 5. 5. 5. 5. 5. 5. 5. 5.]]
>>>
>>>
>>>
>>>
>>> reset states D (set values)
>>> [[ 24. 24. 24. 24. 24. 24. 24. 24. 24. 24.]]
However, if i want to check if the states have been initializated I find the previous initialization values (all one, all two).
with tf.Session() as sess:
tf.global_variables_initializer().run()
hh = sess.run(model.layers[1].states[0])
cc = sess.run(model.layers[1].states[1])
print hh
>>> [[ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]]
print cc
>>> [[ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]]
What is exactly happening here? Why the function seems working according to the prints but doesn't change the values of the internal states?
As you may read here, value parameter sets a value by which a variable should be initialized. So when you call tf.global_variables_initializer().run() your states are initialized with values defined here:
new_h = K.variable(value=np.ones((1, 10)))
new_c = K.variable(value=np.ones((1, 10))+1)
Edit:
It seemed obvious for me but once again I will explain why reset_states doesn't work.
Variable definition: When you defined your inner states to be variables initialized by a certain value the n this certain vaklue will be set every time you call variable_initializer.
Reset states: it will update a current value of this variable but it will not change a default value of initializer. In order to do that you need to reassign this states by yet another variable with a given states set as default.