After training the cnn model, I want to visualize the weight or print out the weights, what can I do?
I cannot even print out the variables after training.
Thank you!
To visualize the weights, you can use a tf.image_summary() op to transform a convolutional filter (or a slice of a filter) into a summary proto, write them to a log using a tf.train.SummaryWriter, and visualize the log using TensorBoard.
Let's say you have the following (simplified) program:
filter = tf.Variable(tf.truncated_normal([8, 8, 3]))
images = tf.placeholder(tf.float32, shape=[None, 28, 28])
conv = tf.nn.conv2d(images, filter, strides=[1, 1, 1, 1], padding="SAME")
# More ops...
loss = ...
optimizer = tf.GradientDescentOptimizer(0.01)
train_op = optimizer.minimize(loss)
filter_summary = tf.image_summary(filter)
sess = tf.Session()
summary_writer = tf.train.SummaryWriter('/tmp/logs', sess.graph_def)
for i in range(10000):
sess.run(train_op)
if i % 10 == 0:
# Log a summary every 10 steps.
summary_writer.add_summary(filter_summary, i)
After doing this, you can start TensorBoard to visualize the logs in /tmp/logs, and you will be able to see a visualization of the filter.
Note that this trick visualizes depth-3 filters as RGB images (to match the channels of the input image). If you have deeper filters, or they don't make sense to interpret as color channels, you can use the tf.split() op to split the filter on the depth dimension, and generate one image summary per depth.
Like #mrry said, you can use tf.image_summary. For example, for cifar10_train.py, you can put this code somewhere under def train(). Note how you access a var under scope 'conv1'
# Visualize conv1 features
with tf.variable_scope('conv1') as scope_conv:
weights = tf.get_variable('weights')
# scale weights to [0 255] and convert to uint8 (maybe change scaling?)
x_min = tf.reduce_min(weights)
x_max = tf.reduce_max(weights)
weights_0_to_1 = (weights - x_min) / (x_max - x_min)
weights_0_to_255_uint8 = tf.image.convert_image_dtype (weights_0_to_1, dtype=tf.uint8)
# to tf.image_summary format [batch_size, height, width, channels]
weights_transposed = tf.transpose (weights_0_to_255_uint8, [3, 0, 1, 2])
# this will display random 3 filters from the 64 in conv1
tf.image_summary('conv1/filters', weights_transposed, max_images=3)
If you want to visualize all your conv1 filters in one nice grid, you would have to organize them into a grid yourself. I did that today, so now I'd like to share a gist for visualizing conv1 as a grid
You can extract the values as numpy arrays the following way:
with tf.variable_scope('conv1', reuse=True) as scope_conv:
W_conv1 = tf.get_variable('weights', shape=[5, 5, 1, 32])
weights = W_conv1.eval()
with open("conv1.weights.npz", "w") as outfile:
np.save(outfile, weights)
Note that you have to adjust the scope ('conv1' in my case) and the variable name ('weights' in my case).
Then it boils down on visualizing numpy arrays. One example how to visualize numpy arrays is
#!/usr/bin/env python
"""Visualize numpy arrays."""
import numpy as np
import scipy.misc
arr = np.load('conv1.weights.npb')
# Get each 5x5 filter from the 5x5x1x32 array
for filter_ in range(arr.shape[3]):
# Get the 5x5x1 filter:
extracted_filter = arr[:, :, :, filter_]
# Get rid of the last dimension (hence get 5x5):
extracted_filter = np.squeeze(extracted_filter)
# display the filter (might be very small - you can resize the window)
scipy.misc.imshow(extracted_filter)
Using the tensorflow 2 API, There are several options:
Weights extracted using the get_weights() function.
weights_n = model.layers[n].get_weights()[0]
Bias extracted using the numpy() convert function.
bias_n = model.layers[n].bias.numpy()
Related
I have an input that is a time series of 5 dimensions:
a = [[8,3],[2] , [4,5],[1], [9,1],[2]...] #total 100 timestamps. For each element, dims 0,1 are numerical data and dim 2 is a numerical encoding of a category. This is per sample, 3200 samples
The category has 3 possible values (0,1,2)
I want to build a NN such that the last dimension (the category) will go through an embedding layer with output size 8, and then will be concatenated back to the first two dims (the numerical data).
So, this will be something like:
input1 = keras.layers.Input(shape=(2,)) #the numerical features
input2 = keras.layers.Input(shape=(1,)) #the encoding of the categories. this part will be embedded to 5 dims
x2 = Embedding(input_dim=1, output_dim = 8)(input2) #apply it to every timestamp and take only dim 3, so [2],[1], [2]
x = concatenate([input1,x2]) #will get 10 dims at each timepoint, still 100 timepoints
x = LSTM(units=24)(x) #the input has 10 dims/features at each timepoint, total 100 timepoints per sample
x = Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x]) #input1 is 1D vec of the width 2 , input2 is 1D vec with the width 1 and it is going through the embedding
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
How can I do it? (preferably in keras)?
My problem is how to apply the embedding to every time point?
Meaning, if I have 1000 timepoints with 3 dims each, I need to convert it to 1000 timepoints with 8 dims each (The emebedding layer should transform input2 from (1000X1) to (1000X8)
There are a couple of issues you are having here.
First let me give you a working example and explain along the way how to solve your issues.
Imports and Data Generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
max_features_values = [100, 100, 3]
num_observations = 2
input_list = [[[np.random.randint(0, v) for _ in range(num_timesteps)]
for v in max_features_values]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 3, 100)
In order to use an embedding we need to the voc_size as input_dimension, as stated in the LSTM documentation.
Embedding and Concatenation
voc_size = len(np.unique(input_arr[:, 2, :])) + 1 # 4
Now we need to create the inputs. Inputs should be of size [None, 2, num_timesteps] and [None, 1, num_timesteps] where the first dimension is the flexible and will be filled with the number of observations we are passing in. Let's use the embedding right after that using the previously calculated voc_size.
inp1 = layers.Input(shape=(2, num_timesteps)) # TensorShape([None, 2, 100])
inp2 = layers.Input(shape=(1, num_timesteps)) # TensorShape([None, 1, 100])
x2 = layers.Embedding(input_dim=voc_size, output_dim=8)(inp2) # TensorShape([None, 1, 100, 8])
x2_reshaped = tf.transpose(tf.squeeze(x2, axis=1), [0, 2, 1]) # TensorShape([None, 8, 100])
This cannot be easily concatenated since all dimensions must match except for the one along the concatenation axis. But the shapes are not matching unfortunately. Therefore we reshape x2. We do so by removing the first dimension and then transposing.
Now we can concatenate without any issue and everything works in a straight forward fashion:
x = layers.concatenate([inp1, x2_reshaped], axis=1)
x = layers.LSTM(32)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[inp1, inp2], outputs=[x])
Check on Dummy Example
inp1_np = input_arr[:, :2, :]
inp2_np = input_arr[:, 2:, :]
model.predict([inp1_np, inp2_np])
# Output
# array([[0.544262 ],
# [0.6157502]], dtype=float32)
#This outputs values between 0 and 1 just as expected.
In case you are not looking for Embeddings the way it's usually used in Keras (positive integers mapping to dense vectors). You might be looking for some sort of unprojection or basis expansion, in which 3 dimensions get mapped (embedded) to 8 and concatenating the result. This can be done using the kernel trick or other methods, but also happens implicitly in neural networks with non-linear applications.
As such, you can do something like this, following a similar format to pythonic833 because it was good (but with timestamps in the middle per the Keras LSTM documentation asking for [batch, timesteps, feature]):
Input generation
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model
num_timesteps = 100
num_features = 5
num_observations = 2
input_list = [[[np.random.randint(1, 100) for _ in range(num_features)]
for _ in range(num_timesteps)]
for _ in range(num_observations)]
input_arr = np.array(input_list) # shape (2, 100, 5)
Model construction
Then you can process the inputs:
input1 = layers.Input(shape=(num_timesteps, 2,))
input2 = layers.Input(shape=(num_timesteps, 3))
x2 = layers.Dense(8, activation='relu')(input2)
x = layers.concatenate([input1,x2], axis=2) # This produces tensors of shape (None, 100, 10)
x = layers.LSTM(units=24)(x)
x = layers.Dense(1, activation='sigmoid')(x)
model = Model(inputs=[input1, input2] , outputs=[x])
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['acc']
)
Results
inp1_np = input_arr[:, :, :2]
inp2_np = input_arr[:, :, 2:]
model.predict([inp1_np, inp2_np])
which produces
array([[0.44117224],
[0.23611131]], dtype=float32)
Other explanations about basis expansion to check out:
https://stats.stackexchange.com/questions/527258/embedding-data-into-a-larger-dimension-space
https://www.reddit.com/r/MachineLearning/comments/2ffejw/why_dont_researchers_use_the_kernel_method_in/
So currently I have this code which will pad a BxNxNxC tensor to a BxNxNx(C+P)tensor, where B is batch size, C is the number of channels, and P is the number of padding channels I want to add:
A = <some BxNxNxC tensor>
P = <some calculation>
padding_tensor = keras.layers.UpSampling3D(size=[1, 1, P])(tf.zeros_like(A)[:, :, :, 0:1])
# This is the BxNxNx(C+P) tensor
concat = keras.layers.Concatenate(axis=3)([A, padding_tensor])
The reason I do this in a round about way is because I cannot directly create a padding_tensor of the correct size, because it seems impossible to get the batch size to specify the shape.
I want clean way to do this because I am looking at the computation graphs of my Models and this adds a lot of bloat. If it is possible to sort of hide all of these operations into a single computation node I would be happy enough with that but would rather not have to use 3 operations for something as simple as padding.
I also suspect this will be kind of slow, but I don't know enough about tensorflow to really know.
this is my suggestion... I initialize a fake conv2d layer with zeros and make it not trainable, this will produce 0 output
batch, H, W, F, C, P = 32, 28, 28, 3, 5, 6
X = np.random.uniform(0,1, (batch,H,W,F))
inp = Input((H,W,F))
x_c = Conv2D(C,3, padding='same')(inp) # BxNxNxC
x_p = Conv2D(P,3, padding='same', kernel_initializer='zeros', name='zeros')(inp) # BxNxNxP
concat = Concatenate()([x_c,x_p]) # BxNxNx(C+P)
model = Model(inp, concat)
model.get_layer('zeros').trainable = False # important
model.summary()
# check if zeros
model.predict(X)[:,:,:,-P:].sum() # 0
I'm using tf 1.15, i'm trying to make a regression task using a signal.
First of all i load my signals into the pipeline, i have several files, here i simulate the loading using a np.zeros to make the code usable by you.
Every file has this shape (?, 75000, 3), where ? is a random number of elements, 75000 is the number of samples in each element and 3 is the number of signals.
Using the tf.data i unpack them and i get a dataset who output signals with this shape (75000,), and i use them in my keras model.
Everything should be fine until i create the keras model, i copied my input pipeline because during my tests i got different errors using a generic tf.data.dataset or using the dataset built in this way.
import numpy as np
import tensorflow as tf
# called in the dataset pipeline
def my_func(x):
p = np.zeros([86, 75000, 3])
x = p[:,:,0]
y = p[:, :, 1]
z = p[:, :, 2]
return x, y, z
# called in the dataset pipeline
def load_sign(path):
func = tf.compat.v1.numpy_function(my_func, [path], [tf.float64, tf.float64, tf.float64])
return func
# Dataset pipeline
s = [1, 2] # here i have the file paths, i simulate it with numbers
AUTOTUNE = tf.data.experimental.AUTOTUNE
ds = tf.data.Dataset.from_tensor_slices(s)
# ds = ds.map(load_sign, num_parallel_calls=AUTOTUNE)
ds = ds.map(load_sign, num_parallel_calls=AUTOTUNE).unbatch()
itera = tf.data.make_one_shot_iterator(ds)
ABP, ECG, PLETH = itera.get_next()
# Until there everything should be fine
# Here i create my convolutional network
signal = tf.keras.layers.Input(shape=(None,75000), dtype='float32')
x = tf.compat.v1.keras.layers.Conv1D(64, (1), strides=1, padding='same')(signal)
x = tf.keras.layers.Dense(75000)(x)
model = tf.keras.Model(inputs=signal, outputs=x, name='resnet18')
# And finally i try to insert my signal into model
logits = model(PLETH)
I get this error:
ValueError: Input 0 of layer conv1d is incompatible with the layer: its rank is undefined, but the layer requires a defined rank.
Why? And how can i make it works?
Also the input size of my net should be this one according the documentation:
3D tensor with shape: (batch_size, steps, input_dim)
What is the steps? In my case i assume it should be (batch_size, 1, 75000), right?
I am trying to modify a model I found online (https://github.com/apache/incubator-mxnet/tree/master/example/multivariate_time_series) as I work to get to know mxnet. I am trying to build a model that takes both a CNN and RNN network in parallel and then uses the outputs of both to forecast a time series. However, I am running into this error
RuntimeError: simple_bind error. Arguments: data: (128, 96, 20)
softmax_label: (128, 20) Error in operator concat1: [15:44:09]
src/operator/nn/concat.cc:66: Check failed:
shape_assign(&(*in_shape)[i], dshape) Incompatible input shape:
expected [128,0], got [128,96,300]
This is the code, as I have tried to modify it:
def rnn_cnn_model(iter_train, q, filter_list, num_filter, dropout, seasonal_period, time_interval):
# Choose cells for recurrent layers: each cell will take the output of the previous cell in the list
rcells = [mx.rnn.GRUCell(num_hidden=args.recurrent_state_size)]
skiprcells = [mx.rnn.LSTMCell(num_hidden=args.recurrent_state_size)]
input_feature_shape = iter_train.provide_data[0][1]
X = mx.symbol.Variable(iter_train.provide_data[0].name)
Y = mx.sym.Variable(iter_train.provide_label[0].name)
# reshape data before applying convolutional layer (takes 4D shape incase you ever work with images)
rnn_input = mx.sym.reshape(data=X, shape=(0, q, -1))
###############
# RNN Component
###############
stacked_rnn_cells = mx.rnn.SequentialRNNCell()
for i, recurrent_cell in enumerate(rcells):
stacked_rnn_cells.add(recurrent_cell)
stacked_rnn_cells.add(mx.rnn.DropoutCell(dropout))
outputs, states = stacked_rnn_cells.unroll(length=q, inputs=rnn_input, merge_outputs=False)
rnn_features = outputs[-1] #only take value from final unrolled cell for use later
input_feature_shape = iter_train.provide_data[0][1]
X = mx.symbol.Variable(iter_train.provide_data[0].name)
Y = mx.sym.Variable(iter_train.provide_label[0].name)
# reshape data before applying convolutional layer (takes 4D shape incase you ever work with images)
conv_input = mx.sym.reshape(data=X, shape=(0, 1, q, -1))
###############
# CNN Component
###############
outputs = []
for i, filter_size in enumerate(filter_list):
# pad input array to ensure number output rows = number input rows after applying kernel
padi = mx.sym.pad(data=conv_input, mode="constant", constant_value=0,
pad_width=(0, 0, 0, 0, filter_size - 1, 0, 0, 0))
convi = mx.sym.Convolution(data=padi, kernel=(filter_size, input_feature_shape[2]), num_filter=num_filter)
acti = mx.sym.Activation(data=convi, act_type='relu')
trans = mx.sym.reshape(mx.sym.transpose(data=acti, axes=(0, 2, 1, 3)), shape=(0, 0, 0))
outputs.append(trans)
cnn_features = mx.sym.Concat(*outputs, dim=2)
cnn_reg_features = mx.sym.Dropout(cnn_features, p=dropout)
c_features = mx.sym.reshape(data = cnn_reg_features, shape = (-1))
print(type(c_features))
######################
# Prediction Component
######################
print(rnn_features.infer_shape())
neural_components = mx.sym.concat(*[rnn_features, c_features], dim=1)
neural_output = mx.sym.FullyConnected(data=neural_components, num_hidden=input_feature_shape[2])
model_output = neural_output
loss_grad = mx.sym.LinearRegressionOutput(data=model_output, label=Y)
return loss_grad, [v.name for v in iter_train.provide_data], [v.name for v in iter_train.provide_label]
and I believe the crash is happening on this line of code
neural_components = mx.sym.concat(*[rnn_features, c_features], dim=1)
Here is what I have tried in an effort to get my dimensions to match up:
c_features = mx.sym.reshape(data = cnn_reg_features, shape = (-1))
c_features = cnn_reg_features[-1]
c_features = cnn_reg_features[:, -1, :]
I also tried to look at the git issues and Google around, but all I see is advice to use infer_shape. I tried applying this to c_features, but the output was not clear to me
data: ()
gru_i2h_weight: ()
gru_i2h_bias: ()
Basically, I would like to know at each stage as this graph is built what the shape of the symbol is. I am used to this capability in Tensorflow, which makes it easier to build and debug graphs when one has gone astray in doing an incorrect reshape, or simply for getting the sense of how a model works by looking at its dimension. Is there no equivalent opportunity in mxnet?
Given that the data_iter is fed in when producing these symbols I would think the inferred shape should be available. Ultimately my questions are (1) how can I see that shape of a symbol when it uses the data in the iterator and should know all shapes? (2) general guidelines on debugging in this sort of situation?
Thank you.
I have just started developing some simple classifier in Tenosrflow and I've started using this example on Tensorflow site: https://www.tensorflow.org/tutorials/keras/basic_classification
Now I want my model to get images like this as features:
These images should have, as corresponding labels, three arrays: [1,0], [3,0] and [1,3].
My problem is: how can I load into the model these kind of labels (i.e. labels that are arrays and not a single scalar)?
When I try as in the example down here, the only thing I got is an error message that I won't report here because they are generated from my lack of knowledge on the thing that I'm trying to do.
Additional question: how should the last neural layer be? How many neurons should it have?
Here is the code:
import tensorflow as tf
from tensorflow import keras
import skimage
from skimage.color import rgb2gray
import csv
import numpy as np
names = ['Cerchio', 'Quadrato', 'Stella']
images = []
labels = [[]]
test_images = []
test_labels = [[]]
final_images = []
for i in range(1, 501):
images.append(skimage.data.imread("{0}.bmp".format(i)))
for i in range(501, 601):
test_images.append(skimage.data.imread("{0}.bmp".format(i)))
for i in range(601, 701):
final_images.append(skimage.data.imread("{0}.bmp".format(i)))
file = open("labels.csv", "rU")
reader = csv.reader(file, delimiter=",")
for row in reader:
for i in range(0, 499):
if int(row[i]) < 10:
labels.append([int(int(row[i])/10), 0])
else:
labels.append([int(int(row[i])/10), int(row[i])%10])
for i in range(500, 600):
if int(row[i]) < 10:
test_labels.append([int(int(row[i])/10), 0])
else:
test_labels.append([int(int(row[i])/10), int(row[i])%10])
file.close()
images28 = np.array(images)
images28 = rgb2gray(images28)
test_images28 = np.array(test_images)
test_images28 = rgb2gray(test_images28)
final_images28 = np.array(final_images)
final_images28 = rgb2gray(final_images28)
labels = np.array(labels)
test_labels = np.array(test_labels)
print(labels)
model = keras.Sequential([
keras.layers.Flatten(input_shape=(28, 56)),
keras.layers.Dense(128, activation=tf.nn.relu),
keras.layers.Dense(4, activation=tf.nn.softmax)
])
model.compile(optimizer=tf.train.AdamOptimizer(),
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(images28, labels, epochs=5)
test_loss, test_acc = model.evaluate(test_images28, test_labels)
print('Test accuracy:', test_acc)
a = input()
img = final_images28[int(a)]
print(img.shape)
img = (np.expand_dims(img, 0))
print(img.shape)
predictions_single = model.predict(img)
print(predictions_single)
print(names[np.argmax(predictions_single)])
One way is just map the array labels into an index, like [[0,0],[0,0],[0,0]]->0, [[1,0],[0,0],[0,0]]->1,... etc. You'll have 3^6=729 possible labels. If these forms on the images are standard you probably can use just simplest classificator with no hidden layers so it's gonna be dim1xdim2x729 trainable weights. If they are not standard you will be better off using convolutional layers.
You can probably also use fully convolutional model for this problem that is returning 3 dimensional tensor as an output. In this case you can use multidimensional labels. But then you'll have to write custom loss function for it.
After Googling around and toying with my program, I found the solution: a multi-hot encoded array.
In this array, if I have a position for a circle, a square, a star and the blank space (hence a 4 position array), I can feed to my model labels that have a '1' in each corresponding space.
E.g. (referring to the example above):
[1, 0, 1, 0]
[1, 0, 0, 1]
[0, 0, 1, 1]
This did work perfectly.