Develop Reference

How do you gather the elements of y_pred that do not correspond to the true label in a Keras/tf2.0 custom loss function?

Below is a simple example in numpy of what I would like to do:
import numpy as np
y_true = np.array([0,0,1])
y_pred = np.array([0.1,0.2,0.7])
yc = (1-y_true).astype('bool')
desired = y_pred[yc]
>>> desired
>>> array([0.1, 0.2])
So the prediction corresponding to the ground truth is 0.7, I want to operate on an array containing all the elements of y_pred, except for the ground truth element.
I am unsure of how to make this work within Keras. Here is a working example of the problem in the loss function. Right now 'desired' isn't accomplishing anything, but that is what I need to work with:
# using tensorflow 2.0.0 and keras 2.3.1
import tensorflow.keras.backend as K
import tensorflow as tf
from tensorflow.keras.layers import Input,Dense,Flatten
from tensorflow.keras.models import Model
from keras.datasets import mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Normalize data.
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
# Convert class vectors to binary class matrices.
y_train = tf.keras.utils.to_categorical(y_train, 10)
y_test = tf.keras.utils.to_categorical(y_test, 10)
input_shape = x_train.shape[1:]
x_in = Input((input_shape))
x = Flatten()(x_in)
x = Dense(256,'relu')(x)
x = Dense(256,'relu')(x)
x = Dense(256,'relu')(x)
out = Dense(10,'softmax')(x)
def loss(y_true,y_pred):
yc = tf.math.logical_not(kb.cast(y_true, 'bool'))
desired = tf.boolean_mask(y_pred,yc,axis = 1) #Remove and it runs
CE = tf.keras.losses.categorical_crossentropy(
y_true,
y_pred)
L = CE
return L
model = Model(x_in,out)
model.compile('adam',loss = loss,metrics = ['accuracy'])
model.fit(x_train,y_train)
I end up getting an error
ValueError: Shapes (10,) and (None, None) are incompatible
Where 10 is the number of categories. The end purpose is to implement this: ComplementEntropy
in Keras, where my issue seems to be lines 26-28.

You can remove axis=1 from the Boolean_mask and it will run. And frankly, I don't see why you need axis=1 here.
def loss(y_true,y_pred):
yc = tf.math.logical_not(K.cast(y_true, 'bool'))
print(yc.shape)
desired = tf.boolean_mask(y_pred, yc) #Remove axis=1 and it runs
CE = tf.keras.losses.categorical_crossentropy(
y_true,
y_pred)
L = CE
return L
This is probably what happens. You have y_pred which is a 2D tensor (N=2). Then you have a 2D mask (K=2). But there's this condition K + axis <= N. And if you pass axis=1 this fails.

Using thushv89's answer, here is the full code for how I implemented COT on LeNet from the referenced paper. The one trick is I am not actually flipping back and forth between the two objectives, instead there is just a random weight that flips s.
# using tensorflow 2.0.0 and keras 2.3.1
import tensorflow.keras.backend as kb
import tensorflow as tf
from tensorflow.keras.layers import Conv2D, Input, Dense,Flatten,AveragePooling2D,GlobalAveragePooling2D
from tensorflow.keras.models import Model
from keras.datasets import mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Normalize data.
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
#exapnd dims to fit chn format
x_train = np.expand_dims(x_train,axis=3)
x_test = np.expand_dims(x_test,axis=3)
# Convert class vectors to binary class matrices.
y_train = tf.keras.utils.to_categorical(y_train, 10)
y_test = tf.keras.utils.to_categorical(y_test, 10)
input_shape = x_train.shape[1:]
x_in = Input((input_shape))
act = 'tanh'
x = Conv2D(32, (5, 5), activation=act, padding='same',strides = 1)(x_in)
x = AveragePooling2D((2, 2),strides = (2,2))(x)
x = Conv2D(16, (5, 5), activation=act)(x)
x = AveragePooling2D((2, 2),strides = (2,2))(x)
conv_out = Flatten()(x)
z = Dense(120,activation = act)(conv_out)#120
z = Dense(84,activation = act)(z)#84
last = Dense(10,activation = 'softmax')(z)
model = Model(x_in,last)
def loss(y_true,y_pred, axis=-1):
s = kb.round(tf.random.uniform( (1,), minval=0, maxval=1, dtype=tf.dtypes.float32))
s_ = 1 - s
y_pred = y_pred + 1e-8
yg = kb.max(y_pred,axis=1)
yc = tf.math.logical_not(kb.cast(y_true, 'bool'))
yp_c = tf.boolean_mask(y_pred, yc)
ygc_ = 1/(1-yg+1e-8)
ygc_ = kb.expand_dims(ygc_,axis=1)
Px = yp_c*ygc_ +1e-8
COT = kb.mean(Px*kb.log(Px),axis=1)
CE = -kb.mean(y_true*kb.log(y_pred),axis=1)
L = s*CE +s_*(1/(10-1))*COT
return L
model.compile(loss=loss,
optimizer='adam', metrics=['accuracy'])
model.fit(x_train,y_train,epochs=20,batch_size = 128,validation_data= (x_test,y_test))
pred = model.predict(x_test)
pred_label = np.argmax(pred,axis=1)
label = np.argmax(y_test,axis=1)
cor = (pred_label == label).sum()
acc = print('acc:',cor/label.shape[0])

How can i convert mnist data to RGB format?

I am trying to convert MNIST dataset to RGB format, the actual shape of each image is (28, 28), but i need (28, 28, 3).
import numpy as np
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, _), (x_test, _) = mnist.load_data()
X = np.concatenate([x_train, x_test])
X = X / 127.5 - 1
X.reshape((70000, 28, 28, 1))
tf.image.grayscale_to_rgb(
X,
name=None
)
But i get the following error:
ValueError: Dimension 1 in both shapes must be equal, but are 84 and 3. Shapes are [28,84] and [28,3].

You should store the reshaped 3D [28x28x1] images in an array:
X = X.reshape((70000, 28, 28, 1))
When converting, set an other array to the return value of the tf.image.grayscale_to_rgb() function :
X3 = tf.image.grayscale_to_rgb(
X,
name=None
)
Finally, to plot out one example from the resulting tensor images with matplotlib and tf.session():
import matplotlib.pyplot as plt
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
image_to_plot = sess.run(image)
plt.figure()
plt.imshow(image_to_plot)
plt.grid(False)
The complete code:
import numpy as np
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, _), (x_test, _) = mnist.load_data()
X = np.concatenate([x_train, x_test])
X = X / 127.5 - 1
# Set reshaped array to X
X = X.reshape((70000, 28, 28, 1))
# Convert images and store them in X3
X3 = tf.image.grayscale_to_rgb(
X,
name=None
)
# Get one image from the 3D image array to var. image
image = X3[0,:,:,:]
# Plot it out with matplotlib.pyplot
import matplotlib.pyplot as plt
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
image_to_plot = sess.run(image)
plt.figure()
plt.imshow(image_to_plot)
plt.grid(False)

If you print the shape of X before tf.image.grayscale_to_rgb you will see the output dimension is (70000, 28, 28). Inputs to tf.image.grayscale must have size 1 as it's final dimension.
Expand the final dimension of X to make it compatible with the function
tf.image.grayscale_to_rgb(tf.expand_dims(X, axis=3))

In addition to #DMolony and #Aqwis01 answers, another simple solution could be using numpy.repeat method to duplicate the last dimension of your tensor several times:
X = X.reshape((70000, 28, 28, 1))
X = X.repeat(3, -1) # repeat the last (-1) dimension three times
X_t = tf.convert_to_tensor(X)
assert X_t.shape == (70000, 28, 28, 3)

Python/TensorFlow/Keras - Input to reshape is a tensor with 300 values, but the requested shape has 200 [[{{node decoder_1/reshape_1/Reshape}}]]

I want to transform my data from 2d to 3d for it I've created Autoencoder where code(hidden layer) has 3 neurons. When training starts it throws exception.
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers
from sklearn.datasets import make_circles
input_vector = layers.Input(shape=(1,2))
encoded = layers.Dense(3,activation="relu")(input_vector)
input_encoded = layers.Input(shape=(3,))
x = layers.Dense(3,activation="relu")(input_encoded)
decoded = layers.Reshape((1,2))(x)
encoder = tf.keras.Model(input_vector, encoded, name="encoder")
decoder = tf.keras.Model(input_encoded, decoded, name="decoder")
autoencoder = tf.keras.Model(input_vector, decoder(encoder(input_vector)), name="autoencoder")
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
count = 1000
X, y = make_circles(n_samples=count, noise=0.05)
x_test, y = make_circles(n_samples=count, noise=0.05)
X = np.reshape(X,(count,1,2))
x_test = np.reshape(x_test,(count,1,2))
autoencoder.fit(X, X,
epochs=5,
batch_size=100,
shuffle=True,
validation_data=(x_test, x_test))
Actual result it throws exception
---------------------------------------------------------------------------
InvalidArgumentError: Input to reshape is a tensor with 300 values, but the requested shape has 200
[[{{node decoder_1/reshape_1/Reshape}}]]

Replacing x = layers.Dense(3,activation="relu")(input_encoded) to x = layers.Dense(2,activation="relu")(input_encoded) will fix your problem.
The reason is that input to the layers.Reshape((1,2)) should be of shape (100, 2)(100 is batch size in your case) but you are inputting tensor of shape (100, 3) and hence, the error.

Manipulating scalars in a tensor (Tensorflow)

I would like to extract single values from a tensor and manipulate it while retaining backpropagation. My current implementation:
import keras
from keras import backend as K
from keras.models import Model
from keras.layers import Dense, Activation, Input
import tensorflow as tf
input = Input(shape=(100,1), dtype='float32')
x = Dense(100)(input)
x = Activation('relu')(x)
x = Dense(5)(x)
x = Activation('tanh')(x)
start_pad = 40.0 + 5.0 * x[0] # important line
# ...
zs = K.arange(0.0, 1000, step=1.0)
zs = K.relu( zs - start_pad )
# ...
out = zs # + ...
out = Reshape( (trace_length,1) )(out)
model = Model(inputs = input, outputs = out)
However, start_pad seems to be a tensor with dimensions of x. Running code above gives error:
ValueError: Dimensions must be equal, but are 1000 and 5 for 'sub' (op: 'Sub') with input shapes: [1000], [100,5].
where start_pad object is <tf.Tensor 'add_1:0' shape=(100, 5) dtype=float32>.
I would like to have scalar like value for start_pad and subtract from zs with broadcasting. How do I achive this with Tensorflow/Keras?

Ok, the solution i found is
x = tf.unstack(x, axis=1)
which returns a list of tf tensors

Feed data into lstm using tflearn python

I know there were already some questions in this area, but I couldn't find the answer to my problem.
I have an LSTM (with tflearn) for a regression problem.
I get 3 types of errors, no matter what kind of modifications I do.
import pandas
import tflearn
import tensorflow as tf
from sklearn.cross_validation import train_test_split
csv = pandas.read_csv('something.csv', sep = ',')
X_train, X_test = train_test_split(csv.loc[:,['x1', 'x2',
'x3','x4','x5','x6',
'x7','x8','x9',
'x10']].as_matrix())
Y_train, Y_test = train_test_split(csv.loc[:,['y']].as_matrix())
#create LSTM
g = tflearn.input_data(shape=[None, 1, 10])
g = tflearn.lstm(g, 512, return_seq = True)
g = tflearn.dropout(g, 0.5)
g = tflearn.lstm(g, 512)
g = tflearn.dropout(g, 0.5)
g = tflearn.fully_connected(g, 1, activation='softmax')
g = tflearn.regression(g, optimizer='adam', loss = 'mean_square',
learning_rate=0.001)
model = tflearn.DNN(g)
model.fit(X_train, Y_train, validation_set = (Y_train, Y_test))
n_examples = Y_train.size
def mean_squared_error(y,y_):
return tf.reduce_sum(tf.pow(y_ - y, 2))/(2 * n_examples)
print()
print("\nTest prediction")
print(model.predict(X_test))
print(Y_test)
Y_pred = model.predict(X_test)
print('MSE Test: %.3f' % ( mean_squared_error(Y_test,Y_pred)) )
At the first run when starting new kernel i get
ValueError: Cannot feed value of shape (100, 10) for Tensor 'InputData/X:0', which has shape '(?, 1, 10)'
Then, at the second time
AssertionError: Input dim should be at least 3.
and it refers to the second LSTM layer. I tried to remove the second LSTM an Dropout layers, but then I get
feed_dict[net_inputs[i]] = x
IndexError: list index out of range
If you read this, have a nice day. I you answer it, thanks a lot!!!!

Ok, I solved it. I post it so maybe it helps somebody:
X_train = X_train.reshape([-1,1,10])
X_test = X_test.reshape([-1,1,10])