After training a model (image classification) I would like to see how it performs differently when I evaluate a proper image and various noised versions of it.
The type of noise I'm thinking is a random change in pixels value, I tried with this approach:
# --Inside the generator function that I provide to model.predict_generator--
# dataset is a numpy array with denoised images path
dt = tf.data.Dataset.from_generator(lambda: image_generator(dataset), output_types=(tf.float32))
def image_generator_(image_paths):
for path in image_paths:
# im is keras.preprocessing image
img = im.load_img(path,
color_mode='rgb',
target_size=(224,224))
img_to_numpy = np.array(img)
for _ in range (0, 5):
tmp_numpy_image = img_to_numpy.copy()
for i in range(tmp_numpy_image.shape[0]):
for j in range(tmp_numpy_image.shape[1]):
# add noise
tmp_numpy_image.shape[i][j] = ...
yield tmp_numpy_image
This process works fine but it is very slow. I also use dataset.batch and dataset.prefetch on dt and I didn't found a combination for their values that reduces the algorithm time
Is there a smarter way to do it? I tried by yielding not noised images and to add the noise later inside dataset.map. The problem is that inside map I have to manipulate tensors and I didn't found a way to change each pixel value
SOLUTION
I used #Marat approach and it worked like a charm, the whole process went from 20-30 hours to minutes. My noise was a simple +-1 but I didn't want to go in overflow (255+1 = 0 in uint8) and therefore I only had to use numpy masks
...
tmp_numpy_image = img_to_numpy.copy()
noise = np.random.randint(-1, 1, img_to_numpy.shape)
# tmp_numpy_imag will become of type int32
tmp_numpy_image = tmp_numpy_image + noise
np.putmask(tmp_numpy_image, tmp_numpy_image < 0, 0)
np.putmask(tmp_numpy_image, tmp_numpy_image > 255, 255)
tmp_numpy_image = tmp_numpy_image.astype('uint8')
yield tmp_numpy_image
The biggest overhead here is pixel operations (double for loop). Vectorizing it should result in substantial speedup:
noise_magnitude = 10
...
img_max_value = img_to_numpy.max() * np.ones(img_to_numpy.shape)
for _ in range (0, 5):
# depending on range of values, you might want to adjust noise magnitude
noise = np.random.randint(0, noise_magnitude, img_to_numpy.shape)
# after adding noise, clip values exceeding max values
yield np.maximum(img_to_numpy + noise, img_max_value)
Related
I'm using the following example to analyse the performance of Computer Vision system depending on the data quality.
Keras Implementation Retinanet: https://keras.io/examples/vision/retinanet/
My goal is to corrupt(stretch, shift) certain percentages (10%,20%,30%) of the total bounding boxes across all images. This means that images should be randomly picked and them some of the bounding boxes corrupted so that in total the target percentage is affected.
I'm using the tensorflow datasets as my training data (e.g. https://www.tensorflow.org/datasets/catalog/kitti).
My basic idea was to generate an array in the size of the total amout of boxes and fill it with 1 (modify box) and 0 (ignore box) and then iterate through all boxes:
random_array = np.concatenate((np.ones(int(error_rate_size*TOTAL_NUMBER_OF_BOXES)+1,dtype=int),np.zeros(int((1-error_rate_size)*TOTAL_NUMBER_OF_BOXES)+1,dtype=int)))
The problem is that the implementation I'm using is heavily relying on graph implementation and specifially on the map function (https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map). I would like to follow this pattern in order to keep the implemented data pipeline.
What I am hopeing to do is to use map function in combination with a global counter so I can loop through the array and modify whenever a condition is given. It should roughly look something like this:
COUNT = 0
def damage_data(box):
scaling_range = 2.0
global COUNT
COUNT += 1
if random_array[COUNT]== 1:
new_box = tf.stack(
[
box[0]*scaling_range*tf.random.uniform(shape=(),minval=0.0,maxval=1.0,dtype=tf.float32,seed=1), # x center
box[1]*scaling_range*tf.random.uniform(shape=(),minval=0.0,maxval=1.0,dtype=tf.float32,seed=2), # y center
box[2]*scaling_range*tf.random.uniform(shape=(),minval=0.0,maxval=1.0,dtype=tf.float32,seed=3), # width,
box[3]*scaling_range*tf.random.uniform(shape=(),minval=0.0,maxval=1.0,dtype=tf.float32,seed=4), # height,
],
axis=-1,)
else:
tf.print("Not Changed")
new_box = tf.stack(
[
box[0],
box[1], # y center
box[2], # width,
box[3], # height,
],
axis=-1,)
return new_box
def damage_data_cross_sequential(image, bbox, class_id):
# bbox format [x_center, y_center, width, height]
bbox = tf.map_fn(damage_data,bbox)
return image, bbox, class_id
train_dataset = train_dataset.map(damage_data_cross_sequential,num_parallel_calls=1)
But using this code the variable COUNT is not incremented globally but rather every map() call starts from the initial value 0. I assume this somehow is caused through the graph implementation and the parallel processes in map().
The question is now if there is any way to globally increase a counter through the map function or if I could extend the given dataset with a unique identifier (e.g. add box[5] = id).
I hope the problem is clear and thanks already! :)
--------------UPDATE 1-------------------------------
The second approach as described by #Lescurel is what I'm trying to do.
Some clarifications about the dataset structure.
The number of boxes per image is not identical.It changes from image to image.
e.g. sample 1: ((x_dim, y_dim, 3), (4,4)), sample 2: ((x_dim, y_dim, 3), (2,4))
For a better understanding the structure can be reproduced with the following:
import tensorflow as tf
import tensorflow_datasets as tfds
import numpy as np
valid_ds = tfds.load('kitti', split='validation') # validation is a smaller set
def select_relevant_info(sample):
image = sample["image"]
bbox = sample["objects"]["bbox"]
class_id = tf.cast(sample["objects"]["type"], dtype=tf.int32)
return image, bbox, class_id
valid_ds = valid_ds.map(select_relevant_info)
for sample in valid_ds.take(1):
print(sample)
For plenty of reasons, using a global state is not a terribly good idea, but it's probably even worse in a concurrent context like this one.
There is at least two other ways of implementing what you want:
using a random sample with a threshold as condition to modify the label
put your random array in the dataset as the condition to modify the label.
I personally prefer the first option, which is simpler.
An example.
Lets generate some random data, and create a tf.Dataset. In that example, the total number of sample is 1000:
imgs = tf.random.uniform((1000, 4, 4))
boxes = tf.ones((1000, 4))
ds = tf.data.Dataset.from_tensor_slices((imgs, boxes))
First option: Random Sample
This function will draw a number uniformly between 0 and 1. If this number is higher than the threshold prob, then nothing happens. Otherwise, we modify the label. In that example, it gives a 0.05% chance of modifying the label.
def change_label_with_prob(label, prob=0.05, scaling_range=2.):
return tf.cond(
tf.random.uniform(()) > prob,
lambda: label,
lambda: label*scaling_range*tf.random.uniform((4,), 0., 1., dtype=tf.float32),
)
You can simply call it with Dataset.map:
new_ds = ds.map(lambda img, box: (img, change_label_with_prob(box)))
Second Option : Pass the condition array around
First, we generate an array filled with our conditions: 1 if we want to modify the array, 0 if not.
# lets set the number to change to 200
N_TO_CHANGE = 200
# randomly generated array with 200 "1" and "800" 0.
cond_array = tf.random.shuffle(
tf.concat([tf.ones((N_TO_CHANGE,),dtype=tf.bool), tf.zeros((1000 - N_TO_CHANGE,),dtype=tf.bool)], axis=0)
)
Then we can create a dataset from that array of conditions, and zip it with our previous dataset:
# creating a dataset from the conditional array
ds_cond = tf.data.Dataset.from_tensor_slices(cond_array)
# zipping the two datasets together
ds_data_and_cond = tf.data.Dataset.zip((ds, ds_cond))
# each element of that dataset is ((img, box), cond)
We can write our function, roughly the same as before:
def change_label_with_cond(label, cond, scaling_range=2.0):
# if true, modifies, do nothing otherwise
return tf.cond(
cond,
lambda: label
* scaling_range
* tf.random.uniform((4,), 0.0, 1.0, dtype=tf.float32),
lambda: label,
)
And then map the function on our new dataset, paying attention to the nested shape of each element of the dataset:
ds_changed_label = ds_data_and_cond.map(
lambda img_and_box, z: (img_and_box[0], change_label_with_cond(img_and_box[1], z))
)
# New dataset has a shape (img, box), same as before the zipping
I want to create salt and pepper noise function.
The input is noise_density, i.e. the amount of pixels as noise in the output image and it should return value is the noisy image data source
def salt_pepper(noise_density):
noisesource = ColumnDataSource(data={'image': [noiseImage]})
return noisesource
This function returns an image that is [density]x[density] pixels, using numpy to generate a random array and using PIL to generate the image itself from the array.
def salt_pepper(density):
imarray = numpy.random.rand(density,density,3) * 255
return Image.fromarray(imarray.astype('uint8')).convert('L')
Now, for example, you could run
salt_pepper(500)
To generate an image file that is 500x500px.
Of course, make sure to
import numpy
from PIL import Image
I came up with a vectorized solution which I'm sure can be improved/simplified. Although the interface is not exactly as the requested one, the code is pretty straightforward (and fast 😬) and I'm sure it can be easily adapted.
import numpy as np
from PIL import Image
def salt_and_pepper(image, prob=0.05):
# If the specified `prob` is negative or zero, we don't need to do anything.
if prob <= 0:
return image
arr = np.asarray(image)
original_dtype = arr.dtype
# Derive the number of intensity levels from the array datatype.
intensity_levels = 2 ** (arr[0, 0].nbytes * 8)
min_intensity = 0
max_intensity = intensity_levels - 1
# Generate an array with the same shape as the image's:
# Each entry will have:
# 1 with probability: 1 - prob
# 0 or np.nan (50% each) with probability: prob
random_image_arr = np.random.choice(
[min_intensity, 1, np.nan], p=[prob / 2, 1 - prob, prob / 2], size=arr.shape
)
# This results in an image array with the following properties:
# - With probability 1 - prob: the pixel KEEPS ITS VALUE (it was multiplied by 1)
# - With probability prob/2: the pixel has value zero (it was multiplied by 0)
# - With probability prob/2: the pixel has value np.nan (it was multiplied by np.nan)
# We need to to `arr.astype(np.float)` to make sure np.nan is a valid value.
salt_and_peppered_arr = arr.astype(np.float) * random_image_arr
# Since we want SALT instead of NaN, we replace it.
# We cast the array back to its original dtype so we can pass it to PIL.
salt_and_peppered_arr = np.nan_to_num(
salt_and_peppered_arr, nan=max_intensity
).astype(original_dtype)
return Image.fromarray(salt_and_peppered_arr)
You can load a black and white version of Lena like so:
lena = Image.open("lena.ppm")
bwlena = Image.fromarray(np.asarray(lena).mean(axis=2).astype(np.uint8))
Finally, you can save a couple of examples:
salt_and_pepper(bwlena, prob=0.1).save("sp01lena.png", "PNG")
salt_and_pepper(bwlena, prob=0.3).save("sp03lena.png", "PNG")
Results:
https://i.ibb.co/J2y9HXS/sp01lena.png
https://i.ibb.co/VTm5Vy2/sp03lena.png
I am using Python 3.6 to perform basic image manipulation through Pillow. Currently, I am attempting to take 32-bit PNG images (RGBA) of arbitrary color compositions and sizes and quantize them to a known palette of 16 colors. Optimally, this quantization method should be able to leave fully transparent (A = 0) pixels alone, while forcing all semi-transparent pixels to be fully opaque (A = 255). I have already devised working code that performs this, but I wonder if it may be inefficient:
import math
from PIL import Image
# a list of 16 RGBA tuples
palette = [
(0, 0, 0, 255),
# ...
]
with Image.open('some_image.png').convert('RGBA') as img:
for py in range(img.height):
for px in range(img.width):
pix = img.getpixel((px, py))
if pix[3] == 0: # Ignore fully transparent pixels
continue
# Perform exhaustive search for closest Euclidean distance
dist = 450
best_fit = (0, 0, 0, 0)
for c in palette:
if pix[:3] == c: # If pixel matches exactly, break
best_fit = c
break
tmp = sqrt(pow(pix[0]-c[0], 2) + pow(pix[1]-c[1], 2) + pow(pix[2]-c[2], 2))
if tmp < dist:
dist = tmp
best_fit = c
img.putpixel((px, py), best_fit + (255,))
img.save('quantized.png')
I think of two main inefficiencies of this code:
Image.putpixel() is a slow operation
Calculating the distance function multiple times per pixel is computationally wasteful
Is there a faster method to do this?
I've noted that Pillow has a native function Image.quantize() that seems to do exactly what I want. But as it is coded, it forces dithering in the result, which I do not want. This has been brought up in another StackOverflow question. The answer to that question was simply to extract the internal Pillow code and tweak the control variable for dithering, which I tested, but I find that Pillow corrupts the palette I give it and consistently yields an image where the quantized colors are considerably darker than they should be.
Image.point() is a tantalizing method, but it only works on each color channel individually, where color quantization requires working with all channels as a set. It'd be nice to be able to force all of the channels into a single channel of 32-bit integer values, which seems to be what the ill-documented mode "I" would do, but if I run img.convert('I'), I get a completely greyscale result, destroying all color.
An alternative method seems to be using NumPy and altering the image directly. I've attempted to create a lookup table of RGB values, but the three-dimensional indexing of NumPy's syntax is driving me insane. Ideally I'd like some kind of code that works like this:
img_arr = numpy.array(img)
# Find all unique colors
unique_colors = numpy.unique(arr, axis=0)
# Generate lookup table
colormap = numpy.empty(unique_colors.shape)
for i, c in enumerate(unique_colors):
dist = 450
best_fit = None
for pc in palette:
tmp = sqrt(pow(c[0] - pc[0], 2) + pow(c[1] - pc[1], 2) + pow(c[2] - pc[2], 2))
if tmp < dist:
dist = tmp
best_fit = pc
colormap[i] = best_fit
# Hypothetical pseudocode I can't seem to write out
for iy in range(arr.size):
for ix in range(arr[0].size):
if arr[iy, ix, 3] == 0: # Skip transparent
continue
index = # Find index of matching color in unique_colors, somehow
arr[iy, ix] = colormap[index]
I note with this hypothetical example that numpy.unique() is another slow operation, since it sorts the output. Since I cannot seem to finish the code the way I want, I haven't been able to test if this method is faster anyway.
I've also considered attempting to flatten the RGBA axis by converting the values to a 32-bit integer and desiring to create a one-dimensional lookup table with the simpler index:
def shift(a):
return a[0] << 24 | a[1] << 16 | a[2] << 8 | a[3]
img_arr = numpy.apply_along_axis(shift, 1, img_arr)
But this operation seemed noticeably slow on its own.
I would prefer answers that involve only Pillow and/or NumPy, please. Unless using another library demonstrates a dramatic computational speed increase over any PIL- or NumPy-native solution, I don't want to import extraneous libraries to do something these two libraries should be reasonably capable of on their own.
for loops should be avoided for speed.
I think you should make a tensor like:
d2[x,y,color_index,rgb] = distance_squared
where rgb = 0..2 (0 = r, 1 = g, 2 = b).
Then compute the distance:
d[x,y,color_index] =
sqrt(sum(rgb,d2))
Then select the color_index with the minimal distance:
c[x,y] = min_index(color_index, d)
Finally replace alpha as needed:
alpha = ceil(orig_image.alpha)
img = c,alpha
I already achieved the goal described in the title but I was wondering if there was a more efficient (or generally better) way to do it. First of all let me introduce the problem.
I have a set of images of different sizes but with a width/height ratio less than (or equal) 2 (could be anything but let's say 2 for now), I want to normalize each one, meaning I want all of them to have the same size. Specifically I am going to do so like this:
Extract the max height above all images
Zoom the image so that each image reaches the max height keeping its ratio
Add a padding to the right with just white pixels until the image has a width/height ratio of 2
Keep in mind the images are represented as numpy matrices of grey scale values [0,255].
This is how I'm doing it now in Python:
max_height = numpy.max([len(obs) for obs in data if len(obs[0])/len(obs) <= 2])
for obs in data:
if len(obs[0])/len(obs) <= 2:
new_img = ndimage.zoom(obs, round(max_height/len(obs), 2), order=3)
missing_cols = max_height * 2 - len(new_img[0])
norm_img = []
for row in new_img:
norm_img.append(np.pad(row, (0, missing_cols), mode='constant', constant_values=255))
norm_img = np.resize(norm_img, (max_height, max_height*2))
There's a note about this code:
I'm rounding the zoom ratio because it makes the final height equal to max_height, I'm sure this is not the best approach but it's working (any suggestion is appreciated here). What I'd like to do is to expand the image keeping the ratio until it reaches a height equal to max_height. This is the only solution I found so far and it worked right away, the interpolation works pretty good.
So my final questions are:
Is there a better approach to achieve what explained above (image normalization) ? Do you think I could have done this differently ? Is there a common good practice I'm not following ?
Thanks in advance for your time.
Instead of ndimage.zoom you could use
scipy.misc.imresize. This
function allows you to specify the target size as a tuple, instead of by zoom
factor. Thus you won't have to call np.resize later to get the size exactly as
desired.
Note that scipy.misc.imresize calls
PIL.Image.resize
under the hood, so PIL (or Pillow) is a dependency.
Instead of using np.pad in a for-loop, you could allocate space for the desired array, norm_arr, first:
norm_arr = np.full((max_height, max_width), fill_value=255)
and then copy the resized image, new_arr into norm_arr:
nh, nw = new_arr.shape
norm_arr[:nh, :nw] = new_arr
For example,
from __future__ import division
import numpy as np
from scipy import misc
data = [np.linspace(255, 0, i*10).reshape(i,10)
for i in range(5, 100, 11)]
max_height = np.max([len(obs) for obs in data if len(obs[0])/len(obs) <= 2])
max_width = 2*max_height
result = []
for obs in data:
norm_arr = obs
h, w = obs.shape
if float(w)/h <= 2:
scale_factor = max_height/float(h)
target_size = (max_height, int(round(w*scale_factor)))
new_arr = misc.imresize(obs, target_size, interp='bicubic')
norm_arr = np.full((max_height, max_width), fill_value=255)
# check the shapes
# print(obs.shape, new_arr.shape, norm_arr.shape)
nh, nw = new_arr.shape
norm_arr[:nh, :nw] = new_arr
result.append(norm_arr)
# visually check the result
# misc.toimage(norm_arr).show()
I'm loading in a bunch of 16x16 images from a .csv file in with Numpy. Each row is a list of 256 grayscale values stored in CMO (so the shape is (n,256) where n is the number of images). This means that I can display any individual image with pyplot as:
plot.imshow(np.reshape(images[index], (16,16), order='F'), cmap=cm.Greys_r)
I want to tile these images with a certain number of images per row. I do have a working solution:
def TileImage(imgs, picturesPerRow=16):
# Convert to a true list of 16x16 images
tmp = np.reshape(imgs, (-1, 16, 16), order='F')
img = ""
for i in range(0, tmp.shape[0], picturesPerRow):
# On the last iteration, we may not have exactly picturesPerRow
# images left so we need to pad
if tmp.shape[0] - i >= picturesPerRow:
mid = np.concatenate(tmp[i:i+picturesPerRow], axis=1)
else:
padding = np.zeros((picturesPerRow - (tmp.shape[0] -i), 16, 16))
mid = np.concatenate(np.concatenate((tmp[i:tmp.shape[0]], padding), axis=0), axis=1)
if img == "":
img = mid
else:
img = np.concatenate((img, mid), axis=0)
return img
This works perfectly fine, but it feels like there should be a much cleaner way to do this sort of thing. I'm a bit of a novice at Numpy and I was wondering if there was a cleaner way to tile the flattened data in a way without all the manual padding and conditional concatenation.
Usually these sorts of simple array reshaping operations can be done in a couple of lines with Numpy, so I feel like I'm missing something. (Also, using a "" as a flag as if it were a null pointer seems a bit messy)
Here is a simplified version of your implementation.
Could not think about any simpler way of doing it.
def TileImage(imgs, picturesPerRow=16):
""" Convert to a true list of 16x16 images
"""
# Calculate how many columns
picturesPerColumn = imgs.shape[0]/picturesPerRow + 1*((imgs.shape[0]%picturesPerRow)!=0)
# Padding
rowPadding = picturesPerRow - imgs.shape[0]%picturesPerRow
imgs = vstack([imgs,zeros([rowPadding,imgs.shape[1]])])
# Reshaping all images
imgs = imgs.reshape(imgs.shape[0],16,16)
# Tiling Loop (The conditionals are not necessary anymore)
tiled = []
for i in range(0,picturesPerColumn*picturesPerRow,picturesPerRow):
tiled.append(hstack(imgs[i:i+picturesPerRow,:,:]))
return vstack(tiled)
Hope it helps.