Develop Reference

Visualize output from Mark RCNN

I followed this tutorial: https://github.com/lih0905/PyTorch_Study/blob/master/8)%20TorchVision%200.3%20Object%20Detection%20finetuning%20tutorial.ipynb and how can you see there is output images at the end of page and it's pretty good.
Now I also did this and my images are cloudy, it looks like a trash.
I trained maskrcnn and save models, and from here I'm loading epoch-2.pt to see results.
This is my code:
PATH = '/home/Nezz/Train/ArT/models_vjezba/epoch-2.pt'
#model = torch.load(PATH)
#model.to(device)
model.load_state_dict(torch.load(PATH))
# pick one image from the test set
img, _ = dataset_test[21]
# put the model in evaluation mode
model.eval()
#evaluate(model, data_loader_test, device=device)
with torch.no_grad():
prediction = model([img.to(device)])
#print(prediction)
imaag = Image.fromarray(img.mul(255).permute(1, 2, 0).byte().numpy())
imag = Image.fromarray(prediction[0]['masks'][0, 0].mul(255).byte().cpu().numpy())
imag.show()
imaag.show()
output:

Copy one layer's weights from one Huggingface BERT model to another

I have a pre-trained model which I load like so:
from transformers import BertForSequenceClassification, AdamW, BertConfig, BertModel
model = BertForSequenceClassification.from_pretrained(
"bert-base-uncased", # Use the 12-layer BERT model, with an uncased vocab.
num_labels = 2, # The number of output labels--2 for binary classification.
# You can increase this for multi-class tasks.
output_attentions = False, # Whether the model returns attentions weights.
output_hidden_states = False, # Whether the model returns all hidden-states.
)
I want to create a new model with the same architecture, and random initial weights, except for the embedding layer:
==== Embedding Layer ====
bert.embeddings.word_embeddings.weight (30522, 768)
bert.embeddings.position_embeddings.weight (512, 768)
bert.embeddings.token_type_embeddings.weight (2, 768)
bert.embeddings.LayerNorm.weight (768,)
bert.embeddings.LayerNorm.bias (768,)
It seems I can do this to create a new model with the same architecture, but then all the weights are random:
configuration = model.config
untrained_model = BertForSequenceClassification(configuration)
So how do I copy over model's embedding layer weights to the new untrained_model?

Weights and bias are just tensor and you can simply copy them with copy_:
from transformers import BertForSequenceClassification, BertConfig
jetfire = BertForSequenceClassification.from_pretrained('bert-base-cased')
config = BertConfig.from_pretrained('bert-base-cased')
optimus = BertForSequenceClassification(config)
parts = ['bert.embeddings.word_embeddings.weight'
,'bert.embeddings.position_embeddings.weight'
,'bert.embeddings.token_type_embeddings.weight'
,'bert.embeddings.LayerNorm.weight'
,'bert.embeddings.LayerNorm.bias']
def joltElectrify (jetfire, optimus, parts):
target = dict(optimus.named_parameters())
source = dict(jetfire.named_parameters())
for part in parts:
target[part].data.copy_(source[part].data)
joltElectrify(jetfire, optimus, parts)

Finetuning locally saved Universal Sentence Encoder - 'InvalidArgumentError: Unsuccessful TensorSliceReader constructor'

I'm trying to finetune saved locally universal sentence embedder large v.3.
Plan is to take tf-hub module, wrap with simple classifier, train model and save USE module with updated weights to use for embeddings.
But after loading module from local folder for re-training I'm receiving error that it cant get files from temporary folder "C:\Users\xxx\AppData\Local\Temp\1\tfhub_modules....":
InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on C:\Users\xxx\AppData\Local\Temp\1\tfhub_modules\96e8f1d3d4d90ce86b2db128249eb8143a91db73\variables\variables: Not found: FindFirstFile failed for: C:/Users/xxx/AppData/Local/Temp/1/tfhub_modules/96e8f1d3d4d90ce86b2db128249eb8143a91db73/variables : The system cannot find the path specified.
; No such process
[[{{node checkpoint_initializer_25}}]]
Save model:
with tf.Session(graph=tf.Graph()) as sess:
module = hub.Module("https://tfhub.dev/google/universal-sentence-encoder-large/3", trainable=True)
text_input = tf.placeholder(dtype=tf.string, shape=[None])
sess.run([tf.global_variables_initializer(), tf.tables_initializer()])
embeddings = module(text_input)
simple_save(sess,
export_dir,
inputs={'text': text_input},
outputs={'embeddings': embeddings},
legacy_init_op=tf.tables_initializer())
Finetune:
text = ["cat", "kitten", "dog", "puppy"]
label = [0, 0, 1, 1]
graph=tf.Graph()
with tf.Session(graph=graph) as sess:
sess.run([tf.global_variables_initializer(), tf.tables_initializer()])
model = tf.saved_model.loader.load(export_dir=**_export_dir,_** sess=sess,
tags=[tag_constants.SERVING])
# universal sentence encoder input/output
input_tensor_name = model.signature_def[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY].inputs['text'].name
in_tensor = tf.get_default_graph().get_tensor_by_name(input_tensor_name)
embedd_tensor_name = model.signature_def[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY].outputs[
'embeddings'].name
out_tensor = tf.get_default_graph().get_tensor_by_name(embedd_tensor_name)
# simple classification on top of use
input_y = tf.placeholder(tf.int32, shape=(None))
labels = tf.one_hot(input_y, 4)
logits = tf.layers.dense(out_tensor, 4)
loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels)
optimizer = tf.train.AdamOptimizer(learning_rate=0.001).minimize(loss)
sess.run(tf.global_variables_initializer())
for epoch in range(2):
feed_dict = {
in_tensor: text,
input_y: label
}
sess.run(optimizer, feed_dict)
message_embeddings = sess.run(out_tensor, feed_dict={in_tensor: text})
print("Embeddings after tuning:", message_embeddings)`
So currently I suspect issues with variables initialization. But I am not sure regarding the correct way.

I've managed to fix this issue by initialization only optimizer variables.
sess.run(tf.global_variables_initializer()) to
uninitialized_vars = []
for var in tf.all_variables():
try:
sess.run(var)
except tf.errors.FailedPreconditionError:
uninitialized_vars.append(var)
sess.run(tf.initialize_variables(uninitialized_vars))
But I faced another issue - after training, I'm saving fine-tuned embedding model, but size became 2.3Gb, instead of 800Mb original. As I understand this is due to the variables of optimizer.

How can I visualize the estimator Trained model in tensorflow?

I have created a model with 3 hidden layers and trained it with the specific data-set.
How can I visualize the Model, with the neuron connections and weights at each iteration.
Here is the snippet of the python code :
#<ALL IMPORT STATEMENTS>
MODEL_DIR = <model_name>
def make_estimator(model_dir):
config = run_config.RunConfig(model_dir=model_dir)
feat_cols = [tf.feature_column.numeric_column("x", shape=<number_of_feat_cols>)]
return estimator.DNNClassifier(config=config, hidden_units=[<>,<>,<>],feature_columns=feat_cols,n_classes=2,optimizer=tf.train.GradientDescentOptimizer(learning_rate=0.001))
data = pd.read_csv(<csv_file>)
feat_data = data.drop('Type',axis=1)
feat_data_matrix = feat_data.as_matrix()
labels = data['Type']
labels_matrix = labels.as_matrix()
deep_model = make_estimator(MODEL_DIR)
input_fn = estimator.inputs.numpy_input_fn(x={'x':feat_data_matrix}, y=labels_matrix, shuffle=True, batch_size=10, num_epochs=1000)
tr_steps = <step_size>
deep_model.train(input_fn=input_fn,steps=tr_steps)
print ("Training Done")
In the code above, I have not created any tensorflow session, without it where can I implement the TensorBoard APIs for visualizing the model ?

By using the Python API simply call the method tf.summary.FileWriter
Then if you load the file written by the SummaryWriter into TensorBoard, the graph is shown.
You have to load the graph like this:
# Launch the graph.
current_session = tf.Session()
current_session.run(init)
# Create a summary writer, add the 'graph' to the event file.
writer = tf.summary.FileWriter(<some-directory>, current_session.graph)
See here.

Tensorflow slim train and validate inception model

I'm trying to fine tune inception models, and validate it with test data. But all the examples given at tensorflow slime web page only either fine-tuning or testing, there is not any example that doing both at same graph and session.
Basically I want to this.
with tf.Graph().as_default():
image, image_raw, label,image_name, label_name = dut.distorted_inputs(params,is_training=is_training)
test_image, test_image_raw, test_label,test_image_name, test_label_name = dut.distorted_inputs(params,is_training=False)
# I'm creating as it is suggested at github slim page:
logits, _ =inception.inception_v2(image, num_classes=N, is_training=True)
tf.get_variable_scope().reuse_variables()
logits_tes, _ =inception.inception_v2(test_image, num_classes=N, is_training=Test)
err=tf.sub(logits, label)
losses = tf.reduce_mean(tf.reduce_sum(tf.square(err)))
# total_loss = model_loss+losses
total_loss = losses+slim.losses.get_total_loss()
test_err=tf.sub(test_logits, test_label)
test_loss= tf.reduce_mean(tf.reduce_sum(tf.square(test_err)))
optimizer = tf.train.AdamOptimizer(learning_rate=0.001)
train_op = slim.learning.create_train_op(total_loss, optimizer)
final_loss = slim.learning.train(
train_op,
logdir=params["cp_file"],
init_fn=ut.get_init_fn(slim,params),
number_of_steps=2,
summary_writer=summary_writer
)
this code fails As it can be seen, I don't have loop separately to call my test models, I want to test my model on my test data at each 10th batch.

Does calling train with number_of_steps=10 and then using the evaluation code work?