I am trying to build a CNN model for regression problem with limited number of input data with 400 sample size. The inputs are images and labels are extracted from a column of csv file. To increase the input data, I need to augment the input images and match them with existing labels. I am using rotation and flipping augmentation methods. I am not sure how existing labels should be linked to the augmented images and how the final tensorflow dataset should be created to fit the model. Can anyone help me to solve this data augmentation?
#load csv file
labelPath = "/content/drive/MyDrive/Notebook/tepm.csv"
cols = ["temperature"]
df = pd.read_csv(labelPath, sep=" ", header=None, names=cols)
inputPath='/content/drive/MyDrive/Notebook/test_png_64'
images = []
# Load in the images
for filepath in os.listdir(inputPath):
images.append(cv2.imread(inputPath+'/{0}'.format(filepath),flags=(cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)))
images_scaled = np.array(images, dtype="float") / 255.0
(trainY, testY, trainX, testX) = train_test_split(df, images_scaled, test_size=0.20, random_state=42)
(trainY, valY, trainX, valX) = train_test_split(trainY, trainX, test_size=0.20, random_state=42)
def rotate(trainX: tf.Tensor) -> tf.Tensor:
# Rotate 90 degrees
return tf.image.rot90(trainX, tf.random_uniform(shape=[], minval=0, maxval=4, dtype=tf.float32))
def flip(trainX: tf.Tensor) -> tf.Tensor:
trainX = tf.image.random_flip_left_right(trainX)
trainX = tf.image.random_flip_up_down(trainX)
return trainX
update with ImageDataGenerator
datagen = ImageDataGenerator(
vertical_flip=True,
horizontal_flip=True,
fill_mode="nearest")
datagen.fit(trainX)
model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.0001), loss='mean_squared_error', metrics='mse')
ImageDataGenerator should do the trick. It generate batches of tensor image data with real-time data augmentation.
Initially you should consider whether augmentations that preserve the labels are useful, or augmentations that require matching label augmentation, or both. If I am following your code correctly, you have temperature for scalar labels. Without knowing the nature of your images, I'd guess it unlikely that rotations and flips would be temperature-dependent, thus the labels are preserved and you are all set to go with ImageDataGenerator as is. Whether or not those augmentations will help the training is hard to know without trying it. Conversely, ImageDataGenerator does have a Brightness augmentation, which is the sort of thing I could imagine being temperature dependent in an image. In that case, the labels aren't preserved and you'd have to augment them manually, because I don't think ImageDataGenerator has methods for scalar labels. In my experience, it is the latter sort of augmentations (labels not-preserved) which are more obviously useful. But to get matching label augmentation you may have to do a little more manual coding than what comes stock with ImageDataGenerator; fortunately it might not be too hard.
Some of the basic elements for matching label augmentation might go like this (this is not complete code, just snippets):
Set up the subset of parameters for ImageDataGenerator augmentation that make sense for your scalar labels in a convenience dict:
regression_aug = dict(fill_mode='nearest',
rotation_range=3,
width_shift_range=0.1,
height_shift_range=0.1,
Use the ImageDataGenerator method get_random_transform:
self.tparams[i] = self.generator.get_random_transform(self.img_dims)
Apply it to the training image, and further manually apply it to the scalar label(s):
batch_X[i] = self.generator.apply_transform(img[i], self.tparams[i]))
batch_y[i,0] = self.lbl[x,0] - self.tparams[i]['tx']
batch_y[i,1] = self.lbl[x,1] - self.tparams[i]['ty']
batch_y[i,2] = self.lbl[x,2] - self.tparams[i]['theta']
where in this example case I had scalar labels that consisted of position and orientation, such that they could be sensibly be translated and rotated during augmentation.
Related
I have a training dataset which is prepared as follows:
FileName
s01_l01/1_1.png
Labels_onehot
[[0.0, 0.0, 0.0,..., 1.0],..., [0.0, 0.0, 1.0,..., 0.0]]
FileName directly leads to the input images. I would like to preprocess the images (resize them to (224, 224) and rescale them (1./255). Labels_onehot are the one hot encoded labels. These do not have to be further preprocessed.
In the end, the preprocessed images are my X_train data.
X_train.shape
(n, 224, 224, 3)
The labels are the y_train data.
y_train.shape
(n, 8, 36)
Until now, I am preprocessing the images manually and then loading the preprocessed images (one dataset) and the labels (other dataset). This is not a very efficient way and fills out my RAM completely.
But in the end I am able to fit my model relatively easy, e.g. with this code:
model.fit(X_train, y_train, epochs=50, batch_size=32)
Now, I am wondering, how to achieve the same with the ImageDataGenerator. Is there a method to do all the steps I did before manually?
ImageDataGenerator has a number of methods
flow: Takes feature data & label arrays, and returns an iterator that produces generates augmented data batches.
flow_from_directory: Takes a path to a directory and generates batches of augmented data.
First, we will create an instance of ImageDataGenerator.
aug = tf.keras.preprocessing.image.ImageDataGenerator( rotation_range=20,
width_shift_range=0.1, shear_range=0.2, zoom_range=0.4, horizontal_flip=True)
rotation_range: degrees (0 to 180).
width_shift_range and height_shift_range are used for horizontal and
vertical shifts,respectively.
zoom_range: the amount of zoom. If scalar z, the zoom will be
randomly picked in the range [1-z, 1+z].
horizontal_flip: whether to randomly flip images horizontally.
vertical_flip: whether to randomly flip images vertically.
rescale: rescaling factor. This multiplies the image by the rescaling factor. This can be a useful way of normalising the data by specifying a rescaling factor of 1.0/255.
flow function returns a generator, which is a Python iterator object that is used to construct our augmented images
flow_train_generator = aug.flow(x_train, y_train, batch_size=32)
flow_from_directory method returns an Iterator that yields tuples of (x, y) where x is a NumPy array containing a batch of images and y is a NumPy array of corresponding labels.
flow_dict_train_generator = aug.flow_from_directory(
'data/train',
target_size=(150, 150),
batch_size=32,
class_mode='binary')
Please be aware that the flow from the directory can slow down your model training process as you read directly from the disk each time you generate a new batch before making the modifications to the data increase and then pass the batch to the network.
For more details you can refer below link:
https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator
We can generate image dataset using ImageDataGenerator with flow_from_directory method.
train_datagen = ImageDataGenerator(
rescale=1./255, #scale images from integers 0-255 to floats 0-1.
shear_range=0.2,
zoom_range=0.2, # zoom in or out in images
horizontal_flip=True) #horizontal flip of images
train_set = train_datagen.flow_from_directory(..)
and that displays:
Found 200 images belonging to 2 classes
I would like to write a loop for to count the number of images on train_set
For image in train_set:
count = count+1
print(count)
but this does'nt display anything!!
To get the no. of images, try with the below code.
train_set.samples
To access the number of samples of your dataset you first must know which type it is:
If you are using the ImageDataGenerator then:
type(train_ds)
will return
tensorflow.python.keras.preprocessing.image.DirectoryIterator.
In this case you can access the number of samples with:
train_ds.samples
However, if you are creating your creating your dataset using:
train_ds = tf.keras.preprocessing.image_dataset_from_directory( rescale = 1/255. , etc...)
then
type(train_ds)
will return tensorflow.python.data.ops.dataset_ops.BatchDataset
which means that you can access the number of samples indirectly with
len(train_ds.file_paths)
__iter__ will be your solution.
first you try to predict type of train_set .if you do not know.
print(type(train_set))
#then you find **keras.preprocessing.image.DirectoryIterator**
Now you want to apply some function or modification on this type data follow this linkhere
I want to use CNN network to segment 2 objects (binary: "0: object not present, 1: object present") into shapes but I have an issue with data. The train data is 150 images and in "jpg" format and the ground truth (label data) is also 150 images of "png" rasters of 0 and 1 (resulting in black white images).
Now the question is how to load this hybrid of train images and label images in Keras/Tensorflow and if there`s a dummy example and/or demonstration on how to do that in Python, I would be grateful.
You can define one generator for reading the input images and another one for reading the labels using the ImageDataGenerator class and its flow_from_directory() method, and then combine these two generators in a single generator. Just make sure the directory structure and (order of) file names of input and label images are the same:
data_image_gen = ImageDataGenerator(...)
data_label_gen = ImageDataGenerator(...)
image_gen = data_image_gen.flow_from_directory(image_directory,
# no need to return labels
class_mode=None,
# don't shuffle to have the same order as labels
shuffle=False)
image_gen = data_image_gen.flow_from_directory(label_directory,
color_mode='grayscale',
# no need to return labels
class_mode=None,
# don't shuffle to have the same order as images
shuffle=False)
def final_gen(image_gen, label_gen):
for data, labels in zip(image_gen, label_gen):
# divide labels by 255 to make them like masks i.e. 0 and 1
labels /= 255.
# remove the last axis, i.e. (batch_size, n_rows, n_cols, 1) --> (batch_size, n_rows, n_cols)
labels = np.squeeze(labels, axis=-1)
yield data, labels
# ... define your model
# fit the model
model.fit_generator(final_gen(image_gen, label_gen), ...)
Following this tutorial: https://www.tensorflow.org/versions/r1.3/get_started/mnist/pros
I wanted to solve a classification problem with labeled images by myself. Since I'm not using the MNIST database, I spent days creating my own dataset inside tensorflow. It looks like this:
#variables
batch_size = 50
dimension = 784
stages = 10
#step 1 read Dataset
filenames = tf.constant(filenamesList)
labels = tf.constant(labelsList)
#step 2 create Dataset
dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))
#step 3: parse every image in the dataset using `map`
def _parse_function(filename, label):
#convert label to one-hot encoding
one_hot = tf.one_hot(label, stages)
#read image file
image_string = tf.read_file(filename)
image_decoded = tf.image.decode_image(image_string, channels=3)
image = tf.cast(image_decoded, tf.float32)
return image, one_hot
#step 4 final input tensor
dataset = dataset.map(_parse_function)
dataset = dataset.batch(batch_size) #batch_size = 100
iterator = dataset.make_one_shot_iterator()
images, labels = iterator.get_next()
images = tf.reshape(images, [batch_size,dimension]).eval()
labels = tf.reshape(labels, [batch_size,stages]).eval()
for _ in range(10):
dataset = dataset.shuffle(buffer_size = 100)
dataset = dataset.batch(batch_size)
iterator = dataset.make_one_shot_iterator()
images, labels = iterator.get_next()
images = tf.reshape(images, [batch_size,dimension]).eval()
labels = tf.reshape(labels, [batch_size,stages]).eval()
train_step.run(feed_dict={x: images, y_:labels})
Somehow using a higher batch_sizes will break python. What I'm trying to do is to train my neural network with new batches on each iteration. That's why Im also using dataset.shuffle(...). Using dataset.shuffle also breaks my Python.
What I wanted to do (because shuffle breaks) is to batch the whole dataset. By evaluating ('.eval()') I will get a numpy array. I will then shuffle the array with numpy.random.shuffle(images) and then pick up some the first elements to train it.
e.g.
for _ in range(1000):
images = tf.reshape(images, [batch_size,dimension]).eval()
labels = tf.reshape(labels, [batch_size,stages]).eval()
#shuffle
np.random.shuffle(images)
np.random.shuffle(labels)
train_step.run(feed_dict={x: images[0:train_size], y_:labels[0:train_size]})
But then here comes the problem that I can't batch the my whole dataset. It looks like that the data is too big for python to work with.
How should I solve this differently?
Since I'm not using the MNIST database there isn't a function like mnist.train.next_batch(100) which comes handy for me.
Notice how you call shuffle and batch inside your for loop? This is wrong. Datasets in TF work in the style of functional programming, so you are actually defining a pipeline for preprocessing the data to feed into your model. In a way, you give a recipe that answers the question "given this raw data, which operations (map, etc.) should I do to get batches that I can feed into my neural network?"
Now you are modifying that pipeline for every batch! What happens is that the first iteration, the batch size is, say [32 3600]. The next iteration, the elements of this shape are batched again, to [32 32 3600], and so on.
There's a great tutorial on the TF website where you can find out more how Datasets work, but here are a few suggestions how you can resolve your problem.
Move the shuffling to right after "Step 2" in your code. Then you are shuffling the whole dataset so your batches will have a good mixture of examples. Also increase the buffer_size argument, this works in a different way than you probably assume. It's usually a good idea to shuffle as early as possible, as it can be a slow operation if you have a large dataset -- the shuffled part of dataset will have to be read into memory. Here it does not really matter whether you shuffle the filenames and labels, or the read images and labels -- but the latter will have more work to do since the dataset is larger by that time.
Move batching and the iterator generator to be the last steps, just before starting your training loop.
Don't use feed_dict with Dataset iterators to input data into your model. Instead, define your model in terms of the outputs of iterator.get_next() and omit the feed_dict argument. See more details from this Q&A: Tensorflow: create minibatch from numpy array > 2 GB
Ive been getting through a lot of problems with creating tensorflow datasets. So I decided to use OpenCV to import images.
import opencv as cv
imgDataset = []
for i in range(len(files)):
imgDataset.append(cv2.imread(files[i]))
imgDataset = np.asarray(imgDataset)
the shape of imgDataset is (num_img, height, width, col_channels). Getting the i-th image should be imgDataset[i].
shuffling the dataset and getting only batches of it can be done like this:
from sklearn.utils import shuffle
X,y = shuffle(X, y)
X_feed = X[batch_size]
y_feed = y[batch_size]
Then you feed X_feed and y_feed into your model
I'm trying to make an input pipeline in tensorflow for image classification, therefore I want to make batches of images and corresponding labels. The Tensorflow document suggests that we can use tf.train.batch to make batches of inputs:
train_batch, train_label_batch = tf.train.batch(
[train_image, train_image_label],
batch_size=batch_size,
num_threads=1,
capacity=10*batch_size,
enqueue_many=False,
shapes=[[224,224,3], [len(labels),]],
allow_smaller_final_batch=True
)
However, I'm thinking would it be a problem if I feed in the graph like this:
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=train_label_batch, logits=Model(train_batch)))
The question is does the operation in the cost function dequeues images and their corresponding labels, or it returns them separately? Therefore causing the training with wrong images and labels.
There are several things you need to consider to preserve the ordering of images and labels.
let's say we need a function that gives us images and labels.
def _get_test_images(_train=False):
"""
Gets the test images and labels as a batch
Inputs:
======
_train : Boolean if images are from training set
random_crop : Boolean if random cropping is allowed
random_flip : Boolean if random horizontal flip is allowed
distortion : Boolean if distortions are allowed
Outputs:
========
images_batch : Batch of images containing BATCH_SIZE images at a time
label_batch : Batch of labels corresponding to the images in images_batch
idx : Batch of indexes of images
"""
#get images and labels
_,_img_names,_img_class,index= _get_list(_train = _train)
#total number of distinct images used for train will be equal to the images
#fed in tf.train.slice_input_producer as _img_names
img_path,label,idx = tf.train.slice_input_producer([_img_names,_img_class,index],shuffle=False)
img_path,label,idx = tf.convert_to_tensor(img_path),tf.convert_to_tensor(label),tf.convert_to_tensor(idx)
img_path = tf.cast(img_path,dtype=tf.string)
#read file
image_file = tf.read_file(img_path)
#decode jpeg/png/bmp
#tf.image.decode_image won't give shape out. So it will give error while resizing
image = tf.image.decode_jpeg(image_file)
#image preprocessing
image = tf.image.resize_images(image, [IMG_DIM,IMG_DIM])
float_image = tf.cast(image,dtype=tf.float32)
#subtracting mean and divide by standard deviation
float_image = tf.image.per_image_standardization(float_image)
#set the shape
float_image.set_shape(IMG_SIZE)
labels_original = tf.cast(label,dtype=tf.int32)
img_index = tf.cast(idx,dtype=tf.int32)
#parameters for shuffle
batch_size = BATCH_SIZE
min_fraction_of_examples_in_queue = 0.3
num_preprocess_threads = 1
num_examples_per_epoch = MAX_TEST_EXAMPLE
min_queue_examples = int(num_examples_per_epoch *
min_fraction_of_examples_in_queue)
images_batch, label_batch,idx = tf.train.batch(
[float_image,label,img_index],
batch_size=batch_size,
num_threads=num_preprocess_threads,
capacity=min_queue_examples + 3 * batch_size)
# Display the training images in the visualizer.
tf.summary.image('images', images_batch)
return images_batch, label_batch,idx
Here,tf.train.slice_input_producer([_img_names,_img_class,index],shuffle=False) is an interesting thing to look at where if you put shuffle=True it will shuffle all three arrays in coordination.
Second thing is, num_preprocess_threads. As long as you are using single threads for dequeue operation, batches will come out in a deterministic way. But more than one threads will shuffle the arrays randomly. for example for image 0001.jpg if True label is 1 you might get 2 or 4. Once its dequeue it is in tensor form. tf.nn.softmax_cross_entropy_with_logits shouldn't have problem with such tensors.