Problem
I am doing two classes image segmentation, and I want to use loss function of dice coefficient. However validation loss is not improved. How to Solve these problem?
what I did
Using the mothod of one-hot encoding, Processed label image and it has not include backgroung label.
Code
Shape of X is (num of data, 256, 256, 1) # graysacle
Shape of y is (num of data, 256, 256, 2) # two class and exclude background label
one_hot_y = np.zeros((len(y), image_height, image_width, 2))
for i in range(len(y)):
one_hot = to_categorical(y[i])
one_hot_y[i] = one_hot[:,:,1:]
one_hot_y.shape #-> (566, 256, 256, 2)
#### <-- Unet Model --> ####
from tensorflow import keras
from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Concatenate, Conv2DTranspose
from keras import Model
def unet(image_height, image_width, num_classes):
# inputs = Input(input_size)
inputs = Input(shape=(image_height, image_width, 1),name='U-net')
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(pool1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(pool3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(pool4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(conv5)
up6 = Concatenate()([Conv2DTranspose(256, (2, 2), strides=(2, 2), padding='same')(conv5), conv4])
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(up6)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv6)
up7 = Concatenate()([Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same')(conv6), conv3])
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(up7)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv7)
up8 = Concatenate()([Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same')(conv7), conv2])
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(up8)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv8)
up9 = Concatenate()([Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same')(conv8), conv1])
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(up9)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv9)
outputs = Conv2D(num_classes, (1, 1), activation='softmax')(conv9)
return Model(inputs=[inputs], outputs=[outputs])```
#### <-- Dice Score --> ####
from tensorflow.keras import backend as K
def dice_coef(y_true, y_pred):
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
intersection = K.sum(y_true_f * y_pred_f)
return (2. * intersection + 0.0001) / (K.sum(y_true_f) + K.sum(y_pred_f) + 0.0001)
def dice_coef_loss(y_true, y_pred):
return 1 - dice_coef(y_true, y_pred)```
#### <-- Fit the Model --> ####
from tensorflow.keras import optimizers
adam = optimizers.Adam(learning_rate=0.0001)
unet_model.compile(optimizer=adam, loss=[dice_coef_loss],metrics=[dice_coef])
hist = unet_model.fit(X_train,y_train, epochs=epochs, batch_size=batch_size,validation_data=(X_val,y_val), callbacks=[checkpoint,earlystopping])
I tried to replicate your experience. I used the Oxford-IIIT Pets database whose label has three classes: 1: Foreground, 2: Background, 3: Not classified. If class 1 ("Foreground") is removed as you did, then the val_loss does not change during the iterations. On the other hand, if the "Not classified" class is removed, the optimization seems to work. The model fails to discriminate between "Background" and "Not classified", which is conceivable.
Besides, there is a small error in the calculation of the dice coefficient: In the denominator, you need to take the sum of the squares. It doesn't change anything for y_true but for y_pred it does.
I can't say why your code doesn't work, but I can tell you the way I do it. Differences are that I exclude the background and encode the target inside the dice coef calculation function.
Then I define my Dice coefficient as follows:
def dice_coef(y_true, y_pred, smooth=1):
# flatten
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
# one-hot encoding y with 3 labels : 0=background, 1=label1, 2=label2
y_true_f = K.one_hot(K.cast(y_true_f, np.uint8), 3)
y_pred_f = K.one_hot(K.cast(y_pred_f, np.uint8), 3)
# calculate intersection and union exluding background using y[:,1:]
intersection = K.sum(y_true_f[:,1:]* y_pred_f[:,1:], axis=[-1])
union = K.sum(y_true_f[:,1:], axis=[-1]) + K.sum(y_pred_f[:,1:], axis=[-1])
# apply dice formula
dice = K.mean((2. * intersection + smooth)/(union + smooth), axis=0)
return dice
def dice_loss(y_true, y_pred):
return 1-dice_coef
I was also confused about this problem until I understood the following code！！！！
import numpy as np
from PIL import Image
from keras import backend as K
def dice_loss(y_true, y_pred):
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
intersection = K.sum(y_true_f* y_pred_f)
val = (2. * intersection + K.epsilon()) / (K.sum(y_true_f * y_true_f) + K.sum(y_pred_f * y_pred_f) + K.epsilon())
return 1. - val
arr1 = np.array([[[9.6,0.6,0.3],
[0.3,0.5,0.5]],
[[0.5,0.5,0.5],
[0.5,0.5,0.5]],
[[0.5,0.5,0.5],
[0.5,0.5,0.5]],
[[0.5,0.5,0.5],
[0.5,0.5,0.5]]])
arr2= np.array([[[9.6,0.6,0.3],
[0.3,0.5,0.5]],
[[0.5,0.5,0.5],
[0.5,0.5,0.5]],
[[0.5,0.5,0.5],
[0.5,0.5,0.5]],
[[0.5,0.5,0.5],
[0.5,0.5,0.5]]])
loss = dice_loss(arr1,arr2)
print(loss)
Related
I am writing my thesis in machine learning and am trying to build a unet to perform it. The code is as follows:
First i create the dataloader to create the datasets for input:
def dataloader(filepath, subset):
# Initiliaze return arrays - input of shape = HYPERPARAMETER
global size
if subset=="train":
size = 129
elif subset=="test":
size = 18
input_data=np.zeros((size,1024,1024,1))
output_data=np.zeros((size,1024,1024,1))
# Open file and create loop
with open(filepath+"annotation_"+subset+".txt", "r") as input_file:
# Count to pass through the file
count=0
for line in input_file:
line=line.split(" ")
data=cv2.imread(filepath+str(line[0])+".jpg",cv2.IMREAD_GRAYSCALE)
input_data[count,:,:,0]=data
# Case of benevolent
if line[3]=="B":
x=int(line[4])
y=1024-int(line[5])
radius=int(line[6])
for i in range(1024):
for j in range(1024):
if ((radius*radius-(i-x)*(i-x)-(j-y)*(j-y))>0):
# Setting 80 as th value of the benevolent mask
output_data[count,i,j,0]=80
# Case of malevolent
elif line[3]=="M":
x=int(line[4])
y=1024-int(line[5])
radius=int(line[6])
for i in range(1024):
for j in range(1024):
if ((radius*radius-(i-x)*(i-x)-(j-y)*(j-y))>0):
# Setting 160 as th value of the benevolent mask
output_data[count,i,j,0]=160
if count==0:
print(type(data))
print(type(input_data))
cv2.imshow('test',data)
cv2.waitKey(0)
cv2.imshow('image',input_data[count,:,:,0])
cv2.waitKey(0)
cv2.imshow('mask',output_data[count,:,:,0])
cv2.waitKey(0)
cv2.destroyAllWindows()
count=count+1
#input_data=K.zeros_like(input_data)
#output_data=K.zeros_like(output_data)
return input_data, output_data
and then the model and the commands to run it:
def unet_model(optimizer, loss_metric, metrics, sample_width, sample_height, lr=1e-3):
inputs = Input((sample_width, sample_height, 1))
print(inputs.shape)
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
drop1 = Dropout(0.5)(pool1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(drop1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
drop2 = Dropout(0.5)(pool2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(drop2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
drop3 = Dropout(0.3)(pool3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(drop3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)
drop4 = Dropout(0.3)(pool4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(drop4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(conv5)
up6 = concatenate([Conv2DTranspose(256, (2, 2), strides=(2, 2), padding='same')(conv5), conv4], axis=3)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(up6)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv6)
up7 = concatenate([Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same')(conv6), conv3], axis=3)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(up7)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv7)
up8 = concatenate([Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same')(conv7), conv2], axis=3)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(up8)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv8)
up9 = concatenate([Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same')(conv8), conv1], axis=3)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(up9)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv9)
conv10 = Conv2D(1, (1, 1), activation='softmax')(conv9)
model = Model(inputs=[inputs], outputs=[conv10])
model.compile(optimizer=optimizer(lr=lr), loss=loss_metric, metrics=metrics)
return model
#Filepath of datasets
filepath = "/home/tzikos/Downloads/train/"
# Load datasets
train_input, train_output = dataloader(filepath, "train")
test_input, test_output = dataloader(filepath, "test")
train_input = normalize(train_input)
test_input = normalize(test_input)
train_output = normalize(train_output)
test_output = normalize(test_output)
print(train_input.shape)
print(train_output.shape)
print(test_input.shape)
print(test_output.shape)
# Load model
model = unet_model(optimizer=Adam, loss_metric=tf.keras.losses.MeanSquaredError(), metrics=["accuracy"], sample_width=train_input.shape[1], sample_height=train_input.shape[2],lr=1e-3)
model.compile(optimizer="Adam", loss=tf.keras.losses.MeanSquaredError(), metrics=["accuracy"])
history = model.fit(x=train_input, y=train_output, batch_size=1, epochs=30)
# Save weights
model_filepath = '/home/tzikos/Desktop/thesis_DENSE-IN-UNET/unet_weights.h5'
model.save(model_filepath)
# Check results
results = model.evaluate(test_input, test_output)
print(results)
So the problem is the following:
When I train my model i get 0 accuracy and no change in the loss function. So I went and dived into the images.
When I imshow the data variable i get the photo as should be. However when I input it into the numpy array it is transcribed into a binary one where there is black and white and idk why.
So I think that is the problem but i cant see why that is since the data variable is allright
I am not sure what this network is supposed to do, so I will list some mistakes that in your code.
In your function you already compile the model:
model.compile(optimizer=optimizer(lr=lr), loss=loss_metric, metrics=metrics)
return model
After outside the function you do it again:
model.compile(optimizer="Adam", loss=tf.keras.losses.MeanSquaredError(), metrics=["accuracy"])
One issue is about metric and loss. tf.keras.losses.MeanSquaredError() is a regression loss and 'accuracy' is a classification metric.
The most obvious one:
conv10 = Conv2D(1, (1, 1), activation='softmax')(conv9)
Softmax activation will be applied to your last axis. If you check your model.summary() your last axis consist of size 1 which means you have a single element. So you are just returning(outputting) a vector of ones everytime.
I'm using this:
Python version: 3.7.7 (default, May 6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
TensorFlow version: 2.1.0
Eager execution: True
With this U-Net model:
inputs = Input(shape=img_shape)
conv1 = Conv2D(64, (5, 5), activation='relu', padding='same', data_format="channels_last", name='conv1_1')(inputs)
conv1 = Conv2D(64, (5, 5), activation='relu', padding='same', data_format="channels_last", name='conv1_2')(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2), data_format="channels_last", name='pool1')(conv1)
conv2 = Conv2D(96, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv2_1')(pool1)
conv2 = Conv2D(96, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv2_2')(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2), data_format="channels_last", name='pool2')(conv2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv3_1')(pool2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv3_2')(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2), data_format="channels_last", name='pool3')(conv3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv4_1')(pool3)
conv4 = Conv2D(256, (4, 4), activation='relu', padding='same', data_format="channels_last", name='conv4_2')(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2), data_format="channels_last", name='pool4')(conv4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv5_1')(pool4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv5_2')(conv5)
up_conv5 = UpSampling2D(size=(2, 2), data_format="channels_last", name='up_conv5')(conv5)
ch, cw = get_crop_shape(conv4, up_conv5)
crop_conv4 = Cropping2D(cropping=(ch, cw), data_format="channels_last", name='crop_conv4')(conv4)
up6 = concatenate([up_conv5, crop_conv4])
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv6_1')(up6)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv6_2')(conv6)
up_conv6 = UpSampling2D(size=(2, 2), data_format="channels_last", name='up_conv6')(conv6)
ch, cw = get_crop_shape(conv3, up_conv6)
crop_conv3 = Cropping2D(cropping=(ch, cw), data_format="channels_last", name='crop_conv3')(conv3)
up7 = concatenate([up_conv6, crop_conv3])
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv7_1')(up7)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv7_2')(conv7)
up_conv7 = UpSampling2D(size=(2, 2), data_format="channels_last", name='up_conv7')(conv7)
ch, cw = get_crop_shape(conv2, up_conv7)
crop_conv2 = Cropping2D(cropping=(ch, cw), data_format="channels_last", name='crop_conv2')(conv2)
up8 = concatenate([up_conv7, crop_conv2])
conv8 = Conv2D(96, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv8_1')(up8)
conv8 = Conv2D(96, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv8_2')(conv8)
up_conv8 = UpSampling2D(size=(2, 2), data_format="channels_last", name='up_conv8')(conv8)
ch, cw = get_crop_shape(conv1, up_conv8)
crop_conv1 = Cropping2D(cropping=(ch, cw), data_format="channels_last", name='crop_conv1')(conv1)
up9 = concatenate([up_conv8, crop_conv1])
conv9 = Conv2D(64, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv9_1')(up9)
conv9 = Conv2D(64, (3, 3), activation='relu', padding='same', data_format="channels_last", name='conv9_2')(conv9)
ch, cw = get_crop_shape(inputs, conv9)
conv9 = ZeroPadding2D(padding=(ch, cw), data_format="channels_last", name='conv9_3')(conv9)
conv10 = Conv2D(1, (1, 1), activation='sigmoid', data_format="channels_last", name='conv10_1')(conv9)
model = Model(inputs=inputs, outputs=conv10)
And with this functions:
def dice_coef(y_true, y_pred):
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
intersection = K.sum(y_true_f * y_pred_f)
return (2.0 * intersection + 1.0) / (K.sum(y_true_f) + K.sum(y_pred_f) + 1.0)
def dice_coef_loss(y_true, y_pred):
return 1-dice_coef(y_true, y_pred)
To compile the model I do:
model.compile(tf.keras.optimizers.Adam(lr=(1e-4) * 2), loss=dice_coef_loss, metrics=[dice_coef])
And I get this output while training:
Epoch 1/2
5/5 [==============================] - 8s 2s/sample - loss: 1.0000 - dice_coef: 4.5962e-05 - val_loss: 0.9929 - val_dice_coef: 0.0071
Epoch 2/2
5/5 [==============================] - 5s 977ms/sample - loss: 0.9703 - dice_coef: 0.0297 - val_loss: 0.9939 - val_dice_coef: 0.0061
Train on 5 samples, validate on 5 samples
I think the idea is to get a loss closely to zero, but I don't understand the 1.000 I get (maybe it is worst loss value that I can get). But I don't understand the dice_coef value.
What does dice_coef value mean?
Dice loss is a loss function that prevents some of the limitations present in the ordinary Cross Entropy loss.
Limitations of Cross Entropy:
When using cross entropy loss, the statistical distributions of labels play a big role in training accuracy. The more unbalanced the label distributions are, the more difficult the training will be. Although weighted cross entropy loss can alleviate the difficulty, the improvement is not significant nor the intrinsic issue of cross entropy loss is solved. In cross entropy loss, the loss is calculated as the average of per-pixel loss, and the per-pixel loss is calculated discretely, without knowing whether its adjacent pixels are boundaries or not. As a result, cross entropy loss only considers loss in a micro sense rather than considering it globally, which is not enough for image level prediction.
Dice Loss
Dice Coef function can be described as:
which is clearly what your funtion dice_coef(y_true, y_pred) is calculating. More about that Sørensen–Dice coefficient
In the equation above p_i and g_i are pairs of corresponding pixel values of prediction and ground truth, respectively. In boundary detection scenario, their values are either 0 or 1, representing whether the pixel is boundary (value of 1) or not (value of 0). The denominator is the sum of total boundary pixels of both prediction and ground truth, and the numerator is the sum of correctly predicted boundary pixels because the sum increments only when pi and gi match (both of value 1).
The denominator considers the total number of boundary pixels at global scale, while the numerator considers the overlap between the two sets at local scale. Therefore, Dice loss considers the loss information both locally and globally, which is critical for high accuracy.
Regarding your training, since your loss value is decreasing through the training, you shouldn't worry that much, try increasing the number of epoch and analyse the network as it goes through the model.
the dice loss is simply 1 - dice coef. which is what your function is calculating.
Lets say I have a model defined like this:
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import (BatchNormalization, concatenate,
Conv2D, Conv2DTranspose, DepthwiseConv2D,
Dropout, Input, MaxPooling2D,
ReLU, ZeroPadding2D)
input_layer = Input((64, 64, 3))
conv1 = Conv2D(16, (3, 3), padding="same")(input_layer)
conv1 = BatchNormalization()(conv1)
conv1 = ReLU()(conv1)
pool1 = MaxPooling2D((2,2))(conv1)
conv2 = Conv2D(32, (3, 3), padding="same")(pool1)
conv2 = BatchNormalization()(conv2)
conv2 = ReLU()(conv2)
pool2 = MaxPooling2D((2,2))(conv2)
conv3 = Conv2D(64, (3, 3), padding="same")(pool2)
conv3 = BatchNormalization()(conv3)
conv3 = ReLU()(conv3)
pool3 = MaxPooling2D((2,2))(conv3)
mid = Conv2D(128, (3, 3), padding="same")(pool3)
mid = BatchNormalization()(mid)
mid = ReLU()(mid)
dconv3 = Conv2DTranspose(64, (3, 3), strides=(2, 2), padding="same")(mid)
cat3 = concatenate([dconv3, conv3])
dconv2 = Conv2DTranspose(32, (3, 3), strides=(2, 2), padding="same")(dconv3)
cat2 = concatenate([dconv2, conv2])
dconv1 = Conv2DTranspose(16, (3, 3), strides=(2, 2), padding="same")(dconv2)
cat1 = concatenate([dconv1, conv1])
output_layer = Conv2D(1, (1,1), padding="same", activation="sigmoid")(dconv1)
model = Model(input_layer, output_layer)
The model is a very simple UNET which requires that the down sample blocks be concatenated with the upsample blocks. Lets now imagine that I want to define this exact model but with some arbitrary depth aka 2, 3, 4, 5 etc.. downsample and upsample blocks. Instead of having to go in and manually modify the parameters, I would like to automate the model building.
I am very close to accomplishing this, but I fail during concatenation. See below.
class configurable_model():
def __init__(self, csize, channels, start_neurons, depth):
self.csize = csize
self.channels = channels
self.start_neurons = start_neurons
self.depth = depth
def _convblock(self, factor, name):
layer = Sequential(name=name)
layer.add(Conv2D(self.start_neurons * factor, (3, 3), padding="same"))
layer.add(BatchNormalization())
layer.add(ReLU())
return layer
def build_model(self):
model = Sequential()
model.add(Input((self.csize, self.csize, self.channels), name='input'))
factor = 1
for idx in range(self.depth):
model.add(self._convblock(factor, f'downblock{idx}'))
model.add(MaxPooling2D((2,2)))
factor *= 2
model.add(self._convblock(factor, name='middle'))
for idx in reversed(range(self.depth)):
factor //= 2
model.add(Conv2DTranspose(self.start_neurons * factor, (3, 3),
strides=(2, 2), padding="same",
name=f'upblock{idx}'))
#how do I do the concatenation??
model.add(concatenate([model.get_layer(f'upblock{idx}'),
model.get_layer(f'downblock{idx}')]))
model.add(Conv2D(1, (1,1), padding="same",
activation="sigmoid", name='output'))
return model
test = configurable_model(64, 3, 16, 3)
model = test.build_model()
I have tried converting to the functional API, but run into the problem of 'naming' the layers and keeping track of them in the for loops. I tried Concatenate instead of concatenate. I tried model.get_layer('layername').output and model.get_layer('layername').output() in the concatenate statement, etc... nothing is working. The code above gives the error: ValueError: A Concatenate layer should be called on a list of at least 2 inputs.
I was able to get the functional version working by storing the downblocks in a dictionary that I reference later during concatenation. See below:
class configurable_model():
def __init__(self, csize, channels, start_neurons, depth):
self.csize = csize
self.channels = channels
self.start_neurons = start_neurons
self.depth = depth
def _convblock(self, factor, name=None):
block = Sequential(name=name)
block.add(Conv2D(self.start_neurons * factor, (3, 3), padding="same"))
block.add(BatchNormalization())
block.add(ReLU())
block.add(Conv2D(self.start_neurons * factor, (3, 3), padding="same"))
block.add(BatchNormalization())
block.add(ReLU())
return block
def build_model(self):
input_layer = Input((self.csize, self.csize, self.channels), name='input')
x = input_layer
factor = 1
downblocks = {}
for idx in range(self.depth):
x = self._convblock(factor, f'downblock{idx}')(x)
downblocks[f'downblock{idx}'] = x
x = MaxPooling2D((2, 2), name=f'maxpool{idx}')(x)
factor *= 2
x = self._convblock(factor, 'Middle')(x)
for idx in reversed(range(self.depth)):
factor //= 2
x = Conv2DTranspose(self.start_neurons * factor, (3, 3),
strides=(2, 2), padding="same",
name=f'upsample{idx}')(x)
cat = concatenate([x, downblocks[f'downblock{idx}']])
x = self._convblock(factor, f'upblock{idx}')(cat)
output_layer = Conv2D(1, (1, 1), padding="same",
activation="sigmoid", name='output')(x)
return Model(input_layer, output_layer)
I am having problem in shape, how could i resolve this?What can I do ?
File "run_demo.py", line 116, in main
model = get_crfrnn_model_def()
File "./src/crfrnn_model.py", line 70, in get_crfrnn_model_def
score_fused = Add()([score2, score_pool4c])
ValueError: Operands could not be broadcast together with shapes (3, 34, 4) (3, 34, 3)
def get_crfrnn_model_def():
channels, height, weight = 3, 500, 500
# Input
input_shape = (height, weight, 3)
img_input = Input(shape=input_shape)
# Add plenty of zero padding
x = ZeroPadding2D(padding=(100, 100))(img_input)
# VGG-16 convolution block 1
x = Conv2D(64, (3, 3), activation='relu', padding='valid', name='conv1_1')(x)
x = Conv2D(64, (3, 3), activation='relu', padding='same', name='conv1_2')(x)
x = MaxPooling2D((2, 2), strides=(2, 2), name='pool1')(x)
# VGG-16 convolution block 2
x = Conv2D(128, (3, 3), activation='relu', padding='same', name='conv2_1')(x)
x = Conv2D(128, (3, 3), activation='relu', padding='same', name='conv2_2')(x)
x = MaxPooling2D((2, 2), strides=(2, 2), name='pool2', padding='same')(x)
# VGG-16 convolution block 3
x = Conv2D(256, (3, 3), activation='relu', padding='same', name='conv3_1')(x)
x = Conv2D(256, (3, 3), activation='relu', padding='same', name='conv3_2')(x)
x = Conv2D(256, (3, 3), activation='relu', padding='same', name='conv3_3')(x)
x = MaxPooling2D((2, 2), strides=(2, 2), name='pool3', padding='same')(x)
pool3 = x
# VGG-16 convolution block 4
x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv4_1')(x)
x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv4_2')(x)
x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv4_3')(x)
x = MaxPooling2D((2, 2), strides=(2, 2), name='pool4', padding='same')(x)
pool4 = x
# VGG-16 convolution block 5
x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv5_1')(x)
x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv5_2')(x)
x = Conv2D(512, (3, 3), activation='relu', padding='same', name='conv5_3')(x)
x = MaxPooling2D((2, 2), strides=(2, 2), name='pool5', padding='same')(x)
# Fully-connected layers converted to convolution layers
x = Conv2D(4096, (7, 7), activation='relu', padding='valid', name='fc6')(x)
x = Dropout(0.5)(x)
x = Conv2D(4096, (1, 1), activation='relu', padding='valid', name='fc7')(x)
x = Dropout(0.5)(x)
x = Conv2D(3, (1, 1), padding='valid', name='score-fr')(x)
print(x)
# Deconvolution
score2 = Conv2DTranspose(3, (4, 4), strides=2, name='score2')(x)
print(score2)
# Skip connections from pool4
score_pool4 = Conv2D(3, (1, 1), name='score-pool4')(pool4)
score_pool4c = Cropping2D((5, 5),name='score-pool4c')(score_pool4)
print('asd')
print(score_pool4c)
score_fused = Add()([score2, score_pool4c])
score4 = Conv2DTranspose(3, (4, 4), strides=2, name='score4', use_bias=False)(score_fused)
# Skip connections from pool3
score_pool3 = Conv2D(3, (1, 1), name='score-pool3')(pool3)
score_pool3c = Cropping2D((9, 9))(score_pool3)
# Fuse things together
score_final = Add()([score4, score_pool3c])
# Final up-sampling and cropping
upsample = Conv2DTranspose(3, (16, 16), strides=8, name='upsample', use_bias=False)(score_final)
upscore = Cropping2D(((31, 37), (31, 37)))(upsample)
output = CrfRnnLayer(image_dims=(height, weight),
num_classes=3,
theta_alpha=160.,
theta_beta=3.,
theta_gamma=3.,
num_iterations=10,
name='crfrnn')([upscore, img_input])
model = Model(img_input, output, name='crfrnn_net')
return model
import numpy as np
import tensorflow as tf
from keras.engine.topology import Layer
import high_dim_filter_loader
custom_module = high_dim_filter_loader.custom_module
def _diagonal_initializer(shape):
return np.eye(shape[0], shape[1], dtype=np.float32)
def _potts_model_initializer(shape):
return -1 * _diagonal_initializer(shape)
class CrfRnnLayer(Layer):
def __init__(self, image_dims, num_classes,
theta_alpha, theta_beta, theta_gamma,
num_iterations, **kwargs):
self.image_dims = image_dims
self.num_classes = num_classes
self.theta_alpha = theta_alpha
self.theta_beta = theta_beta
self.theta_gamma = theta_gamma
self.num_iterations = num_iterations
self.spatial_ker_weights = None
self.bilateral_ker_weights = None
self.compatibility_matrix = None
super(CrfRnnLayer, self).__init__(**kwargs)
def build(self, input_shape):
# Weights of the spatial kernel
self.spatial_ker_weights = self.add_weight(name='spatial_ker_weights',
shape=(self.num_classes, self.num_classes),
initializer=_diagonal_initializer,
trainable=True)
# Weights of the bilateral kernel
self.bilateral_ker_weights = self.add_weight(name='bilateral_ker_weights',
shape=(self.num_classes, self.num_classes),
initializer=_diagonal_initializer,
trainable=True)
# Compatibility matrix
self.compatibility_matrix = self.add_weight(name='compatibility_matrix',
shape=(self.num_classes, self.num_classes),
initializer=_potts_model_initializer,
trainable=True)
super(CrfRnnLayer, self).build(input_shape)
def call(self, inputs):
unaries = tf.transpose(inputs[0][0, :, :, :], perm=(2, 0, 1))
rgb = tf.transpose(inputs[1][0, :, :, :], perm=(2, 0, 1))
c, h, w = self.num_classes, self.image_dims[0], self.image_dims[1]
all_ones = np.ones((c, h, w), dtype=np.float32)
# Prepare filter normalization coefficients
spatial_norm_vals = custom_module.high_dim_filter(all_ones, rgb, bilateral=False,
theta_gamma=self.theta_gamma)
bilateral_norm_vals = custom_module.high_dim_filter(all_ones, rgb, bilateral=True,
theta_alpha=self.theta_alpha,
theta_beta=self.theta_beta)
q_values = unaries
for i in range(self.num_iterations):
softmax_out = tf.nn.softmax(q_values, 0)
# Spatial filtering
spatial_out = custom_module.high_dim_filter(softmax_out, rgb, bilateral=False,
theta_gamma=self.theta_gamma)
spatial_out = spatial_out / spatial_norm_vals
# Bilateral filtering
bilateral_out = custom_module.high_dim_filter(softmax_out, rgb, bilateral=True,
theta_alpha=self.theta_alpha,
theta_beta=self.theta_beta)
bilateral_out = bilateral_out / bilateral_norm_vals
# Weighting filter outputs
message_passing = (tf.matmul(self.spatial_ker_weights,
tf.reshape(spatial_out, (c, -1))) +
tf.matmul(self.bilateral_ker_weights,
tf.reshape(bilateral_out, (c, -1))))
# Compatibility transform
pairwise = tf.matmul(self.compatibility_matrix, message_passing)
# Adding unary potentials
pairwise = tf.reshape(pairwise, (c, h, w))
q_values = unaries - pairwise
return tf.transpose(tf.reshape(q_values, (1, c, h, w)), perm=(0, 2, 3, 1))
def compute_output_shape(self, input_shape):
return input_shape
OK I think I found the problem.
From the Keras documentation:
data_format: A string, one of "channels_last" or "channels_first". The ordering of the dimensions in the inputs. "channels_last" corresponds to inputs with shape (batch, height, width, channels) while "channels_first" corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be "channels_last".
However, that is a dirty lie. Actually, the image_data_format can be configured from the backend. And two lines in your run_demo.py does exactly that:
from keras import backend as K
K.set_image_dim_ordering('th')
The best part? This seems to be some legacy API. When I googled the function, I can only find it in Keras 1.2.2 documentation (The current version is 2.2.4).
Consider if you really need those two lines. Actually, remove the second line anyway. If you really need it, you can add in K.set_image_data_format('channels_first'), and probably you also need to change input shape to match:
# input_shape = (height, weight, channels)
input_shape = (channels, height, weight)
I think you can try using keras' image_data_format function:
from keras import backend as K
img_width, img_height = 500, 500
if K.image_data_format() == 'channels_first':
input_shape = (3, img_width, img_height)
else:
input_shape = (img_width, img_height, 3)
I'm trying to run my code using the python idle. But when ever I'mtrying to run this , after a specific line the shell is getting restarted
from __future__ import print_function
import numpy as np
from keras.models import Model
from keras.layers import Input, Conv2D, MaxPooling2D, UpSampling2D
from keras.layers import concatenate
from keras.optimizers import Adam
from keras.optimizers import SGD
from keras.callbacks import ModelCheckpoint, LearningRateScheduler
from keras import backend as K
K.set_image_dim_ordering('th') # Theano dimension ordering in this code
img_rows = 512
img_cols = 512
smooth = 1.
def dice_coef(y_true, y_pred):
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
intersection = K.sum(y_true_f * y_pred_f)
return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)
def dice_coef_np(y_true,y_pred):
y_true_f = y_true.flatten()
y_pred_f = y_pred.flatten()
intersection = np.sum(y_true_f * y_pred_f)
return (2. * intersection + smooth) / (np.sum(y_true_f) + np.sum(y_pred_f) + smooth)
def dice_coef_loss(y_true, y_pred):
return -dice_coef(y_true, y_pred)
def get_unet():
inputs = Input((1,img_rows, img_cols))
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(pool1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(pool3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(pool4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(conv5)
#up6 = merge([UpSampling2D(size=(2, 2))(conv5), conv4], mode='concat', concat_axis=1)
up6 = concatenate([UpSampling2D(size=(2, 2))(conv5), conv4], axis=1)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(up6)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv6)
#up7 = merge([UpSampling2D(size=(2, 2))(conv6), conv3], mode='concat', concat_axis=1)
up7 = concatenate([UpSampling2D(size=(2, 2))(conv6), conv3], axis=1)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(up7)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv7)
#up8 = merge([UpSampling2D(size=(2, 2))(conv7), conv2], mode='concat', concat_axis=1)
up8 = concatenate([UpSampling2D(size=(2, 2))(conv7), conv2], axis=1)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(up8)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv8)
#up9 = merge([UpSampling2D(size=(2, 2))(conv8), conv1], mode='concat', concat_axis=1)
up9 = concatenate([UpSampling2D(size=(2, 2))(conv8), conv1], axis=1)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(up9)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv9)
conv10 = Conv2D(1, (1, 1), activation='sigmoid')(conv9)
model = Model(inputs=inputs, outputs=conv10)
model.compile(optimizer=Adam(lr=1.0e-5), loss=dice_coef_loss, metrics=[dice_coef])
return model
def train_and_predict(use_existing):
print('-'*30)
print('Loading and preprocessing train data...')
print('-'*30)
imgs_train = np.load("C:/Users/hirplk/Desktop/unet/Luna2016-Lung-Nodule-Detection-master_new/DATA_PROCESS/scratch/cse/dual/cs5130287/Luna2016/output_final/"+"trainImages.npy").astype(np.float32)
imgs_mask_train = np.load("C:/Users/hirplk/Desktop/unet/Luna2016-Lung-Nodule-Detection-master_new/DATA_PROCESS/scratch/cse/dual/cs5130287/Luna2016/output_final/"+"trainMasks.npy").astype(np.float32)
imgs_test = np.load("C:/Users/hirplk/Desktop/unet/Luna2016-Lung-Nodule-Detection-master_new/DATA_PROCESS/scratch/cse/dual/cs5130287/Luna2016/output_final/"+"testImages.npy").astype(np.float32)
imgs_mask_test_true = np.load("C:/Users/hirplk/Desktop/unet/Luna2016-Lung-Nodule-Detection-master_new/DATA_PROCESS/scratch/cse/dual/cs5130287/Luna2016/output_final/"+"testMasks.npy").astype(np.float32)
mean = np.mean(imgs_train) # mean for data centering
std = np.std(imgs_train) # std for data normalization
imgs_train -= mean # images should already be standardized, but just in case
imgs_train /= std
print('-'*30)
print('Creating and compiling model...')
print('-'*30)
model = get_unet()
# Saving weights to unet.hdf5 at checkpoints
model_checkpoint = ModelCheckpoint('unet.hdf5', monitor='loss', save_best_only=True)
#
# Should we load existing weights?
# Set argument for call to train_and_predict to true at end of script
if use_existing:
model.load_weights('./unet.hdf5')
#
# The final results for this tutorial were produced using a multi-GPU
# machine using TitanX's.
# For a home GPU computation benchmark, on my home set up with a GTX970
# I was able to run 20 epochs with a training set size of 320 and
# batch size of 2 in about an hour. I started getting reseasonable masks
# after about 3 hours of training.
#
print('-'*30)
print('Fitting model...')
print('-'*30)
model.fit(imgs_train, imgs_mask_train, batch_size=2, epochs=10, verbose=1, shuffle=True,
callbacks=[model_checkpoint])
print ('bbbbbbbbbbbbbbbbbbbbbbbbbbbbbb')
This is the output of idle.
RESTART: C:\Users\hirplk\Desktop\unet\DSB3Tutorial-master\tutorial_code\LUNA_train_unet.py
Warning (from warnings module):
File "C:\Research\Python_installation\lib\site-packages\h5py\__init__.py", line 36
from ._conv import register_converters as _register_converters
FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
Using TensorFlow backend.
------------------------------
Loading and preprocessing train data...
------------------------------
------------------------------
Creating and compiling model...
------------------------------
------------------------------
Fitting model...
------------------------------
Epoch 1/10
=============================== RESTART: Shell ===============================
Does this meant my python has crashed? Has anyone experienced this? It worked fine earlier but now I do not have any way of implementing my code
Do I need to reinstall everything from the beginning