Develop Reference

How to build pairs for siamese network with ESC-50 dataset?

I´m trying to build a Siamese network which differentiates between spectrograms.
Currently I'm using the ESC-50 data set and I´m wondering if I build the positive and negative pairs correctly.
The network wont learn and is stuck at an accuracy of 50%. I´ve already tried different architectures but nothing seems to work. That´s why I´m assuming my approach in building the pairs is wrong.
Maybe you can help me.
Here´s my approach:
The ESC-50 dataset consists of 50 classes with 40 sound files per class.
I split the dataset in 40 classes (-> 1600 soundfiles) for training. 5 classes (200 soundfiles) for testing and the remaining 5 classes for evaluation:
def split_data(audio_data, labels):
unique_labels = np.unique(labels)
idx_training = [np.where(labels==unique_labels[i])[0] for i in range(40)]
idx_training = [x for xs in idx_training for x in xs]
idx_test = [np.where(labels==unique_labels[i])[0] for i in range(40,45)]
idx_test = [x for xs in idx_test for x in xs]
idx_eval = [np.where(labels==unique_labels[i])[0] for i in range(45,50)]
idx_eval = [x for xs in idx_eval for x in xs]
return idx_training, idx_test, idx_eval
audio_data is an array with all the 2000 sound files of shape (2000,80000).
The function returns the indicies of the training, test and evaluation files in that array.
I pass these indicies to a function called generate_pairs:
def generate_pairs(labels, indizes):
pair_idx = []
pair_labels = []
for i in range(len(indizes)):
idxA = indizes[i]
label = labels[idxA]
same_labels_idx = np.where(labels==label)[0]
idxB = np.random.choice(same_labels_idx)
pair_idx.append([idxA,idxB])
pair_labels.append([1])
not_same_labels_idx = np.where(labels!=label)[0]
not_same_labels_idx = list(set(not_same_labels_idx).intersection(indizes))
neg_idx = np.random.choice(not_same_labels_idx)
pair_idx.append([idxA,neg_idx])
pair_labels.append([0])
return np.array(pair_idx), np.array(pair_labels).astype("float32")
Basically, the function takes a index out of the idx_training (or idx_test, or idx_eval) array and determines the corresponding label. It searches an index where the label is the same and an index where the labels are not the same. This way positive and negative pairs are formed.
The pairs consist of indicies of the audio_files array. The labels for the pairs ("0" or "1") are stored in pair_idx.
Build arrays containing the audio files for training, testing and evaluating:
audio_data_train_1 = audio_data[pair_idx_train[:,0]]
audio_data_train_2 = audio_data[pair_idx_train[:,1]]
audio_data_test_1 = audio_data[pair_idx_test[:,0]]
audio_data_test_2 = audio_data[pair_idx_test[:,1]]
audio_data_eval_1 = audio_data[pair_idx_eval[:,0]]
audio_data_eval_2 = audio_data[pair_idx_eval[:,1]]
Convert these arrays into mel-spectrogramms:
def get_librosa_melspecs(audio_array, name):
melspecs = np.zeros((audio_array.shape[0],64,313))
for i,audio in enumerate(audio_array):
mel = librosa.feature.melspectrogram(y=audio, n_mels=64, n_fft = 1024, hop_length=256, sr=16000)
mel[mel!=0] = np.log(mel[mel!=0])
melspecs[i]=mel
np.save(name, melspecs)
return melspecs
feat_test_1 = get_librosa_melspecs(audio_data_test_1, "features_vgg_test1.npy")
feat_test_2 = get_librosa_melspecs(audio_data_test_2, "features_vgg_test2.npy")
feat_train_1 = get_librosa_melspecs(audio_data_train_1, "features_vgg_train1.npy")
feat_train_2 = get_librosa_melspecs(audio_data_train_2, "features_vgg_train2.npy")
Pass these features into my siamese network:
model.fit(
[feat_train_1[:], feat_train_2[:]], pair_labels_train[:],
validation_data=([feat_test_1[:], feat_test_2[:]], pair_labels_test[:]),
batch_size=BATCH_SIZE,
epochs=EPOCHS,
shuffle=True,
callbacks=[early_stopping]
)
Here are two pictures containing a positive and a negative sample:
Negative image:
Positive image:
1positive

How to Standard Scale multiple Images?

I am working on a classification problem in python and would like to scale the dataset in the first step.
I have 3463 images each with a dimension of (40,90,3) respectively (x, y, channel) . Overall, the array has a dimension of (3463, 40, 90,3)
How can I use the standard scale correctly and how can I display the image?
Code:
#------------- Image Preprocessing -----------------------------------
Eingangsbilder2 = np.asarray(Eingangsbilder2)
print("Image-dim: ",Eingangsbilder2.shape)
scalers = {}
for x in range(0, len(Eingangsbilder2)):
for i in range(0,Eingangsbilder2[x].shape[2]):
scalers[i] = StandardScaler()
Eingangsbilder2[x][:, :, i] = scalers[i].fit_transform(Eingangsbilder2[x][:, :, i])
plt.imshow(Eingangsbilder[2010])

You can get rid of the for loop altogether by applying z-scoring, which is equivallent to scikit-learn StandardScaler to the first "image number" axis:
Eingangsbilder2 = scipy.stats.zscore(Eingangsbilder2, axis=0)
Hint: In Python you can simply write range(len(Eingangsbilder2)), since the first index (unlike MATLAB) always starting with 0

Problems with keras preprocessing

HI I'm preprocessing some image data to run in a simple FF network:
I have two options that in my eyes are the same but one performs a lot better than the other:
Option 1
I save the images in a directory with correspondent subdirectories and run this:
xy_training = tf.keras.preprocessing.image_dataset_from_directory("/content/data/train", image_size=(48,48), color_mode='grayscale',label_mode="int")
xy_validation = tf.keras.preprocessing.image_dataset_from_directory("/content/data/valid", image_size=(48,48), color_mode='grayscale',label_mode="int")
xy_testing = tf.keras.preprocessing.image_dataset_from_directory("/content/data/test", image_size=(48,48), color_mode='grayscale',label_mode="int")
Option 2
I have the raw arrays of the grayscale images and do this
def preprocess(data):
X = []
pixels_list = data["pixels"].values
for pixels in pixels_list:
single_image = np.reshape(pixels.split(" "), (WIDTH,HEIGHT)).astype("float")
X.append(single_image)
# Convert list to 4D array:
X = np.expand_dims(np.array(X), -1)
# Normalize pixel values to be between 0 and 1
X = X / 255.0
return X
train_images= preprocess(train_data)
valid_images= preprocess(valid_data)
test_images= preprocess(test_data)
Option 2 performs so much better than Option 1. Is there a parameter in tf.keras.preprocessing.image_dataset_from_directory( i'm not setting?
Thanks!

This is most probably due to
tf.keras.preprocessing.image_dataset_from_directory
not having a built in normalization function. The other custom function you have is applying normalization, so comparison in not an apple-to-apple one.
You will have to do the normalization in a later step after loading the datasets using image_dataset_from_directory.
Here's a sample code for normalizing after loading a batch dataset:
def normalize(image,label):
image = tf.cast(image/255. ,tf.float32)
label = tf.cast(label ,tf.float32)
return image,label
xy_training = xy_training.map(normalize)
xy_validation = xy_validation.map(normalize)
xy_testing = xy_testing.map(normalize)

Tensorflow Datasets: Is there a way to only modify a certain percentage of labels?

I'm using the following example to analyse the performance of Computer Vision system depending on the data quality.
Keras Implementation Retinanet: https://keras.io/examples/vision/retinanet/
My goal is to corrupt(stretch, shift) certain percentages (10%,20%,30%) of the total bounding boxes across all images. This means that images should be randomly picked and them some of the bounding boxes corrupted so that in total the target percentage is affected.
I'm using the tensorflow datasets as my training data (e.g. https://www.tensorflow.org/datasets/catalog/kitti).
My basic idea was to generate an array in the size of the total amout of boxes and fill it with 1 (modify box) and 0 (ignore box) and then iterate through all boxes:
random_array = np.concatenate((np.ones(int(error_rate_size*TOTAL_NUMBER_OF_BOXES)+1,dtype=int),np.zeros(int((1-error_rate_size)*TOTAL_NUMBER_OF_BOXES)+1,dtype=int)))
The problem is that the implementation I'm using is heavily relying on graph implementation and specifially on the map function (https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map). I would like to follow this pattern in order to keep the implemented data pipeline.
What I am hopeing to do is to use map function in combination with a global counter so I can loop through the array and modify whenever a condition is given. It should roughly look something like this:
COUNT = 0
def damage_data(box):
scaling_range = 2.0
global COUNT
COUNT += 1
if random_array[COUNT]== 1:
new_box = tf.stack(
[
box[0]*scaling_range*tf.random.uniform(shape=(),minval=0.0,maxval=1.0,dtype=tf.float32,seed=1), # x center
box[1]*scaling_range*tf.random.uniform(shape=(),minval=0.0,maxval=1.0,dtype=tf.float32,seed=2), # y center
box[2]*scaling_range*tf.random.uniform(shape=(),minval=0.0,maxval=1.0,dtype=tf.float32,seed=3), # width,
box[3]*scaling_range*tf.random.uniform(shape=(),minval=0.0,maxval=1.0,dtype=tf.float32,seed=4), # height,
],
axis=-1,)
else:
tf.print("Not Changed")
new_box = tf.stack(
[
box[0],
box[1], # y center
box[2], # width,
box[3], # height,
],
axis=-1,)
return new_box
def damage_data_cross_sequential(image, bbox, class_id):
# bbox format [x_center, y_center, width, height]
bbox = tf.map_fn(damage_data,bbox)
return image, bbox, class_id
train_dataset = train_dataset.map(damage_data_cross_sequential,num_parallel_calls=1)
But using this code the variable COUNT is not incremented globally but rather every map() call starts from the initial value 0. I assume this somehow is caused through the graph implementation and the parallel processes in map().
The question is now if there is any way to globally increase a counter through the map function or if I could extend the given dataset with a unique identifier (e.g. add box[5] = id).
I hope the problem is clear and thanks already! :)
--------------UPDATE 1-------------------------------
The second approach as described by #Lescurel is what I'm trying to do.
Some clarifications about the dataset structure.
The number of boxes per image is not identical.It changes from image to image.
e.g. sample 1: ((x_dim, y_dim, 3), (4,4)), sample 2: ((x_dim, y_dim, 3), (2,4))
For a better understanding the structure can be reproduced with the following:
import tensorflow as tf
import tensorflow_datasets as tfds
import numpy as np
valid_ds = tfds.load('kitti', split='validation') # validation is a smaller set
def select_relevant_info(sample):
image = sample["image"]
bbox = sample["objects"]["bbox"]
class_id = tf.cast(sample["objects"]["type"], dtype=tf.int32)
return image, bbox, class_id
valid_ds = valid_ds.map(select_relevant_info)
for sample in valid_ds.take(1):
print(sample)

For plenty of reasons, using a global state is not a terribly good idea, but it's probably even worse in a concurrent context like this one.
There is at least two other ways of implementing what you want:
using a random sample with a threshold as condition to modify the label
put your random array in the dataset as the condition to modify the label.
I personally prefer the first option, which is simpler.
An example.
Lets generate some random data, and create a tf.Dataset. In that example, the total number of sample is 1000:
imgs = tf.random.uniform((1000, 4, 4))
boxes = tf.ones((1000, 4))
ds = tf.data.Dataset.from_tensor_slices((imgs, boxes))
First option: Random Sample
This function will draw a number uniformly between 0 and 1. If this number is higher than the threshold prob, then nothing happens. Otherwise, we modify the label. In that example, it gives a 0.05% chance of modifying the label.
def change_label_with_prob(label, prob=0.05, scaling_range=2.):
return tf.cond(
tf.random.uniform(()) > prob,
lambda: label,
lambda: label*scaling_range*tf.random.uniform((4,), 0., 1., dtype=tf.float32),
)
You can simply call it with Dataset.map:
new_ds = ds.map(lambda img, box: (img, change_label_with_prob(box)))
Second Option : Pass the condition array around
First, we generate an array filled with our conditions: 1 if we want to modify the array, 0 if not.
# lets set the number to change to 200
N_TO_CHANGE = 200
# randomly generated array with 200 "1" and "800" 0.
cond_array = tf.random.shuffle(
tf.concat([tf.ones((N_TO_CHANGE,),dtype=tf.bool), tf.zeros((1000 - N_TO_CHANGE,),dtype=tf.bool)], axis=0)
)
Then we can create a dataset from that array of conditions, and zip it with our previous dataset:
# creating a dataset from the conditional array
ds_cond = tf.data.Dataset.from_tensor_slices(cond_array)
# zipping the two datasets together
ds_data_and_cond = tf.data.Dataset.zip((ds, ds_cond))
# each element of that dataset is ((img, box), cond)
We can write our function, roughly the same as before:
def change_label_with_cond(label, cond, scaling_range=2.0):
# if true, modifies, do nothing otherwise
return tf.cond(
cond,
lambda: label
* scaling_range
* tf.random.uniform((4,), 0.0, 1.0, dtype=tf.float32),
lambda: label,
)
And then map the function on our new dataset, paying attention to the nested shape of each element of the dataset:
ds_changed_label = ds_data_and_cond.map(
lambda img_and_box, z: (img_and_box[0], change_label_with_cond(img_and_box[1], z))
)
# New dataset has a shape (img, box), same as before the zipping

How to pair matrices that are approximately the same in another numpy array

Background
I have the following code that works like a charm and is used to make pairs for a Siamese network:
def make_pairs(images, labels):
# initialize two empty lists to hold the (image, image) pairs and
# labels to indicate if a pair is positive or negative
pairImages = []
pairLabels = []
# calculate the total number of classes present in the dataset
# and then build a list of indexes for each class label that
# provides the indexes for all examples with a given label
#np.unique function finds all unique class labels in our labels list.
#Taking the len of the np.unique output yields the total number of unique class labels in the dataset.
#In the case of the MNIST dataset, there are 10 unique class labels, corresponding to the digits 0-9.
numClasses = len(np.unique(labels))
#idxs have a list of indexes that belong to each class
idx = [np.where(labels == i)[0] for i in range(0, numClasses)]
#let’s now start generating our positive and negative pairs
for idxA in range(len(images)):
# grab the current image and label belonging to the current
# iteration
currentImage = images[idxA]
label = labels[idxA]
# randomly pick an image that belongs to the *same* class
# label
idxB = np.random.choice(idx[label])
posImage = images[idxB]
# prepare a positive pair and update the images and labels
# lists, respectively
pairImages.append([currentImage, posImage])
pairLabels.append([1])
#grab the indices for each of the class labels *not* equal to
#the current label and randomly pick an image corresponding
#to a label *not* equal to the current label
negIdx = np.where(labels != label)[0]
negImage = images[np.random.choice(negIdx)]
# prepare a negative pair of images and update our lists
pairImages.append([currentImage, negImage])
pairLabels.append([0])
#return a 2-tuple of our image pairs and labels
return (np.array(pairImages), np.array(pairLabels))
Ok, this code works by selecting pairs for each image in the MNIST dataset. It builds one pair for that image by randomly selecting another image of the same class (label), and another patch of a different class (label) to make another pair. By running the code, the final shapes of the returned two matrices are:
# load MNIST dataset and scale the pixel values to the range of [0, 1]
print("[INFO] loading MNIST dataset...")
(trainX, trainY), (testX, testY) = mnist.load_data()
# build the positive and negative image pairs
print("[INFO] preparing positive and negative pairs...")
(pairTrain, labelTrain) = make_pairs(trainX, trainY)
(pairTest, labelTest) = make_pairs(testX, testY)
>> print(pairTrain.shape)
(120000, 2, 28, 28)
>> print(labelTrain.shape)
(120000, 1)
My Dataset
I want to do something a little different with another dataset. Suppose that I have another dataset of 5600 RGB images with 28x28x3 dimensions, as can be seen below:
>>> images2.shape
(5600, 28, 28, 3)
I have another array, let's call it labels2, it has 8 labels for all the 5600 images, being 700 images per label as can be seen below:
>>> labels2.shape
(5600,)
>>> len(np.unique(labels2))
8
>>> (labels2==0).sum()
700
>>> (labels2==1).sum()
700
>>> (labels2==2).sum()
700
...
What do I want to do
My dataset is not an MNIST dataset, so the images from the same class are not so similar. I would like to build pairs that are approximately the same in the following manner:
For each image in my dataset, I want to do the following:
1.1. Calculate the similarity through MSE between that image and all the others in the dataset.
1.2 For the set of MSEs of images with the same label as that image, select the images with the 7 smallest MSEs and build 7 pairs, containing that image plus the 7 closest MSE images. These pairs represent the images from the same class for my Siamese Network.
1.3 For the set of MSEs of images with different labels from that image select, for each different label, only one image with the smallest MSEs. Therefore, as there are 7 labels different from the label of that image, there are 7 more pairs for that image.
As there are 5600 images 28x28x3 for my dataset and, for each image, I build 14 pairs (7 of the same class and 7 for different classes) I am expecting to have a pairTrain matrix of size (78400, 2, 28, 28, 3)
What did I do
I have the following code that does exactly what I want:
def make_pairs(images, labels):
# initialize two empty lists to hold the (image, image) pairs and
# labels to indicate if a pair is positive or negative
pairImages = []
pairLabels = []
#In my dataset, there are 8 unique class labels, corresponding to the classes 0-7.
numClasses = len(np.unique(labels))
#Initial lists
pairLabels=[]
pairImages=[]
#let’s now start generating our positive and negative pairs for each image in the dataset
for idxA in range(len(images)):
print("Image "+str(k)+ " out of " +str(len(images)))
k=k+1
#For each image, I need to store the MSE between it and all the others
mse_all=[]
#Get each image and its label
currentImage = images[idxA]
label = labels[idxA]
#Now we need to iterate through all the other images
for idxB in range(len(images)):
candidateImage = images[idxB]
#Calculate the mse and store all mses
mse=np.mean(candidateImage - currentImage)**2
mse_all.append(mse)
mse_all=np.array(mse_all)
#When we finished calculating mse between the currentImage ad all the others,
#let's add 7 pairs that have the smallest mse in the case of images from the
#same class and 1 pair for each different class
#For all classes, do
for i in range(0,numClasses):
#get indices of images for that class
idxs=[np.where(labels == i)[0]]
#Get images of that class
imgs=images[np.array(idxs)]
imgs=np.squeeze(imgs, axis=0)
#get MSEs between the currentImage and all the others of that class
mse_that_class=mse_all[np.array(idxs)]
mse_that_class=np.squeeze(mse_that_class, axis=0)
#if the class is the same class of that image
if i==label:
#Get indices of that class that have the 7 smallest MSEs
indices_sorted = np.argpartition(mse_that_class, numClasses-1)
else:
#Otherwise, get only the smallest MSE
indices_sorted = np.argpartition(mse_that_class, 1)
# Now, lets pair them
for j in range(0,indices_sorted.shape[0]):
image_to_pair=imgs[indices_sorted[j], :, :, :]
pairImages.append([currentImage, image_to_pair])
if i==label:
pairLabels.append([1])
else:
pairLabels.append([0])
del image_to_pair, currentImage, label, mse_that_class, imgs, indices_sorted, idxs, mse_all
return (np.array(pairImages), np.array(pairLabels))
My problem
The problem with my code is that it simply freezes my computer when I am running the pairs construction for image number 2200, I tried to clean the variables after each loop as you can see in the above code (del image_to_pair, currentImage, label, mse_that_class, imgs, indices_sorted, idxs, mse_all). The question is, a (120000, 2, 28, 28) pairImages matrix was not difficult to be built, but a (78400,2,28,28,3) is. So:
Is this a possible memory problem?
can I clean more variables in my code in order to make it work?
Should I have to disconsider the last dimension of my pairImages matrix so it will have a smaller dimension than the first example and thus, will work?
Is there an easier way to solve my problem?
You can find the functional code and input matrices HERE

You can try running gc.collect() at the start of each loop to actively run the garbage collector. Memory in Python is not freed until garbage collection runs. It's not clear to me that your current del statement is doing what you want it to. (del decrements the refcount, but it doesn't necessarily free the memory, and your code is actually feeding it a newly created tuple instead of the variables).
78400 * 2 * 28 * 28 * 3 = 368,793,600, which is multiplied by the size of each piece of data in bytes, which indicates to me that it should be a memory issue. My guess is that the freezing is the computer trying to switch from using RAM to using a swap file on the drive, and using a swap file intensively like this will cause any computer to take a dump.
Your images should also be loaded one at a time via a generator instead of packed into an array.
import gc
gc.collect()
filenames = ["a.jpg", "b.jpg"]
labels = ["a", "b"]
def image_loader(filenames): # this is a generator, not a function
# code to load image
for f in filenames:
gc.collect() # make super sure we're freeing the memory from the last image
image = load_or_something(filename)
yield image
make_pairs(image_loader(filenames), labels)
Generators function exactly like lists with concern to for loops and similar stuff, with the difference that each item in the list is generated on the spot instead of loaded into memory. (It's a bit technical but tl;dr it's a list-maker-thing that only loads the images on the fly).

I believe you can make this part of your code easier, which should help with the run time as well.
#Now we need to iterate through all the other images
for idxB in range(len(images)):
candidateImage = images[idxB]
#Calculate the mse and store all mses
mse=np.mean(candidateImage - currentImage)**2
mse_all.append(mse)
Instead of iterating through your data with a for loop, you can just do this and let the NumPy do broadcasting
# assuming images is a numpy array with shape 5600,28,28,3
mse_all = np.mean( ((images - currentImage)**2).reshape(images.shape[0],-1), axis=1 )
# mse_all.shape 5600,

Some possible issues and optimizations
Apart from trying with forcing the garbage collector to free the unused memory (that seems to not resolve your problem by trying it), I think that there are other issues on your code, unless I didn't understand what's happening.
By looking at the following snippet:
#Agora adiciono os 7 mais parecidos com aquele bloco, sendo 7 da mesma e 1 de cada das outras classes. 1 bloco
for j in range(0,indices_sorted.shape[0]):
It seems that you are iterating for j <- (0..indices_sorted.shape[0]), where indices_sorted.shape[0] is always 700 and I'm not sure that this is what you want. I think you need just j <- (0..6). In addition the values in the images are always below 255 if I got it right. If so, there is a second optimization that could be added: you could force the uint8 type. In short, I think you could refactor to something similar to this:
for i in range(0,numClasses):
# ...
# ...
#print(indices_sorted.shape) # -> (700,)
#print(indices_sorted.shape[0]) # --> always 700!so why range(0, 700?)
#Agora adiciono os 7 mais parecidos com aquele bloco, sendo 7 da mesma e 1 de cada das outras classes. 1 bloco
for j in range(0, 7):
image_to_pair=np.array(imgs[indices_sorted[j], :, :, :], dtype=np.uint8)
pairImages.append([currentImage, image_to_pair])
if i==label:
pairLabels.append([1])
else:
pairLabels.append([0])
del imgs, idxs
#When finished to find the pairs for that image, lets clean the trash
del image_to_pair, currentImage, label, mse_that_class, indices_sorted, mse_all
gc.collect()
A simple and intuitive hint
By doing some tests I saw that by commenting:
pairImages.append([currentImage, image_to_pair])
you have an almost 0 memory footprint.
Other notes
As an additional note, I moved the del operation of imgs and idxs inside the i for
and the major improvement here, seems to be obtained by forcing the correct type:
image_to_pair=np.array(imgs[indices_sorted[j], :, :, :], dtype=np.uint8)
The results
According to my test, with the original code the memory usage for k = 100, 200 is respectively 642351104 Bytes and 783355904 Bytes. So the increase in memory usage for 100 iteration is 134.5 MB.
After having applied the above modification we have 345239552 B and 362336256 B with an increase of only 16,5 MB.