Develop Reference

Python Image Processing - How to remove certain contour and blend the value with surrounding pixels?

I'm doing a project with depth image. But I have problems with noise and failed pixel reading with my depth camera. There are some spots and contours (especially edges) that have zero value. How to just ignore this zero value and blend it with surrounding values?
I have tried dilation and erosion (morph image processing), but I still can't get the right combination. It indeed removed some of the noise, but I just need to get rid of zeros at all points
Image Example:
The zero value is the darkest blue (I'm using colormap)
To illustrate what I want to do, please refer to this poor paint drawing:
I want to get rid the black spot (for example black value is 0 or certain value), and blend it with its surround.
Yes, I'm able to localized the spot using np.where or the similar function, but I have no idea how to blend it. Maybe a filter to be applied? I need to do this in a stream, so I need a fairly fast process, maybe 10-20 fps will do. Thank you in advance!
Update :
Is there a way other than inpaint? I've looked for various inpaints, but I don't need as sophisticated as impainting. I just need to blend it with simple line, curve, or shape and 1D. I think inpaint is an overkill. Besides, I need them to be fast enough to be used for video stream 10-20 fps, or even better.

Here is one way to do that in Python/OpenCV.
Use median filtering to fill the holes.
Read the input
Convert to gray
Threshold to make a mask (spots are black)
Invert the mask (spots are white)
Find the largest spot contour perimeter from the inverted mask and use half of that value as a median filter size
Apply median filtering to the image
Apply the mask to the input
Apply the inverse mask to the median filtered image
Add the two together to form the result
Save the results
Input:
import cv2
import numpy as np
import math
# read image
img = cv2.imread('spots.png')
# convert to gray
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# threshold
mask = cv2.threshold(gray,0,255,cv2.THRESH_BINARY)[1]
# erode mask to make black regions slightly larger
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5,5))
mask = cv2.morphologyEx(mask, cv2.MORPH_ERODE, kernel)
# make mask 3 channel
mask = cv2.merge([mask,mask,mask])
# invert mask
mask_inv = 255 - mask
# get area of largest contour
contours = cv2.findContours(mask_inv[:,:,0], cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
contours = contours[0] if len(contours) == 2 else contours[1]
perimeter_max = 0
for c in contours:
perimeter = cv2.arcLength(c, True)
if perimeter > perimeter_max:
perimeter_max = perimeter
# approx radius from largest area
radius = int(perimeter_max/2) + 1
if radius % 2 == 0:
radius = radius + 1
print(radius)
# median filter input image
median = cv2.medianBlur(img, radius)
# apply mask to image
img_masked = cv2.bitwise_and(img, mask)
# apply inverse mask to median
median_masked = cv2.bitwise_and(median, mask_inv)
# add together
result = cv2.add(img_masked,median_masked)
# save results
cv2.imwrite('spots_mask.png', mask)
cv2.imwrite('spots_mask_inv.png', mask_inv)
cv2.imwrite('spots_median.png', median)
cv2.imwrite('spots_masked.png', img_masked)
cv2.imwrite('spots_median_masked.png', median_masked)
cv2.imwrite('spots_removed.png', result)
cv2.imshow('mask', mask)
cv2.imshow('mask_inv', mask_inv )
cv2.imshow('median', median)
cv2.imshow('img_masked', img_masked)
cv2.imshow('median_masked', median_masked)
cv2.imshow('result', result)
cv2.waitKey(0)
cv2.destroyAllWindows()
Threshold image as mask:
Inverted mask:
Median filtered image:
Masked image:
Masked median filtered image:
Result:

Perhaps using a NaN-adjusted Gaussian filter is good and fast enough? When you consider your zeros/black spots as NaNs, this approach also works for larger black areas.
# import modules
import matplotlib.pyplot as plt
import numpy as np
import skimage
import skimage.filters
# set seed
np.random.seed(42)
# create dummy image
# (smooth for more realisitc appearance)
size = 50
img = np.random.rand(size, size)
img = skimage.filters.gaussian(img, sigma=5)
# create dummy missing/NaN spots
mask = np.random.rand(size, size) < 0.02
img[mask] = np.nan
# define and apply NaN-adjusted Gaussian filter
# (https://stackoverflow.com/a/36307291/5350621)
def nangaussian(U, sigma=1, truncate=4.0):
V = U.copy()
V[np.isnan(U)] = 0
VV = skimage.filters.gaussian(V, sigma=sigma, truncate=truncate)
W = 0*U.copy()+1
W[np.isnan(U)] = 0
WW = skimage.filters.gaussian(W, sigma=sigma, truncate=truncate)
return VV/WW
smooth = nangaussian(img, sigma=1, truncate=4.0)
# do not smooth full image but only copy smoothed NaN spots
fill = img.copy()
fill[mask] = smooth[mask]
# plot results
vmin, vmax = np.nanmin(img), np.nanmax(img)
aspect = 'auto'
plt.subplot(121)
plt.title('original image (white = NaN)')
plt.imshow(img, aspect=aspect, vmin=vmin, vmax=vmax)
plt.axis('off')
plt.subplot(122)
plt.title('filled image')
plt.imshow(fill, aspect=aspect, vmin=vmin, vmax=vmax)
plt.axis('off')

Optimize the cropping function

I'm using the following code to crop an image and retrieve a non-rectangular patch.
def crop_image(img,roi):
height = img.shape[0]
width = img.shape[1]
mask = np.zeros((height, width), dtype=np.uint8)
points = np.array([roi])
cv2.fillPoly(mask, points, (255))
res = cv2.bitwise_and(img, img, mask=mask)
rect = cv2.boundingRect(points) # returns (x,y,w,h) of the rect
cropped = res[rect[1]: rect[1] + rect[3], rect[0]: rect[0] + rect[2]]
return cropped, res
The roi is [(1053, 969), (1149, 1071), (883, 1075), (813, 983)].
The above code works however How do I optimize the speed of the code? It is too slow. Is there any other better way of cropping non-rectangular patches?

I see two parts that could be optimized.
Cropping the image to the bounding rectangle bounds could be applied as the first step. Benefit? you dramatically reduce the size of the images you are working with. You only have to translate the points of the roi by the x,y of the rect and you are good to go.
At the bitwise_and operation, you are "anding" the image with itself and checking at each pixel whether it is allowed by the mask to output it. I guess this is where most time is spent. Instead, you can directly "and" with the mask and save your precious time (no extra mask checking step). Again, a minor tweak to be able to do so, the mask image should have exactly the same shape as the input image (including channels).
Edit:
Modify code to support any number of channels in the input image
The code below does these two things:
def crop_image(img, roi):
height = img.shape[0]
width = img.shape[1]
channels = img.shape[2] if len(img.shape) > 2 else 1
points = np.array([roi])
rect = cv2.boundingRect(points)
mask_shape = (rect[3], rect[2]) if channels == 1 else (rect[3], rect[2], img.shape[2])
#Notice how the mask image size is now the size of the bounding rect
mask = np.zeros(mask_shape, dtype=np.uint8)
#tranlsate the points so that their origin is the bounding rect top left point
for p in points[0]:
p[0] -= rect[0]
p[1] -= rect[1]
mask_filling = tuple(255 for _ in range(channels))
cv2.fillPoly(mask, points, mask_filling)
cropped = img[rect[1]: rect[1] + rect[3], rect[0]: rect[0] + rect[2]]
res = cv2.bitwise_and(cropped, mask)
return cropped, res

Here is one way using Python/OpenCV and Numpy.
Input:
import cv2
import numpy as np
# read image
img = cv2.imread("efile.jpg")
points = np.array( [[ [693,67], [23,85], [62,924], [698,918] ]] )
# get bounding rectangle of points
x,y,w,h = cv2.boundingRect(points)
print(x,y,w,h)
# draw white filled polygon from points on black background as mask
mask = np.zeros_like(img)
cv2.fillPoly(mask, points, (255,255,255))
# fill background of image with black according to mask
masked = img.copy()
masked[mask==0] = 0
# crop to bounding rectangle
cropped = masked[y:y+h, x:x+w]
# write results
cv2.imwrite("efile_mask.jpg", mask)
cv2.imwrite("efile_masked.jpg", masked)
cv2.imwrite("efile_cropped.jpg", cropped)
# display it
cv2.imshow("efile_mask", mask)
cv2.imshow("efile_masked", masked)
cv2.imshow("efile_cropped", cropped)
cv2.waitKey(0)
Mask from provided points:
Image with background made black:
Cropped result:

How to use OpenCV to crop an image based on a certain criteria?

I would like to crop the images like the one below using python's OpenCV library. The area of interest is inside the squiggly lines on the top and bottom, and the lines on the side. The problem is that every image is slightly different. This means that I need some automated way of cropping for the area of interest. I guess the top and the sides would be easy since you could just crop it by 10 pixels or so. But how can I crop out the bottom half of the image where the line is not straight? I have included this example image. The image that follows highlights in pink the area of the image that I am interested in keeping.

Here is one way using Python/OpenCV.
Read input
Get center point (assume it is inside the desired region)
Convert image to grayscale
Floodfill the gray image and set background to black
Get the largest contour and its bounding box
Draw the largest contour as filled on black background as mask
Apply the mask to the input image
Crop the masked input image
Input:
import cv2
import numpy as np
# load image and get dimensions
img = cv2.imread("odd_region.png")
hh, ww, cc = img.shape
# compute center of image (as integer)
wc = ww//2
hc = hh//2
# create grayscale copy of input as basis of mask
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
# create zeros mask 2 pixels larger in each dimension
zeros = np.zeros([hh + 2, ww + 2], np.uint8)
# do floodfill at center of image as seed point
ffimg = cv2.floodFill(gray, zeros, (wc,hc), (255), (0), (0), flags=8)[1]
# set rest of ffimg to black
ffimg[ffimg!=255] = 0
# get contours, find largest and its bounding box
contours = cv2.findContours(ffimg, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
contours = contours[0] if len(contours) == 2 else contours[1]
area_thresh = 0
for cntr in contours:
area = cv2.contourArea(cntr)
if area > area_thresh:
area = area_thresh
outer_contour = cntr
x,y,w,h = cv2.boundingRect(outer_contour)
# draw the filled contour on a black image
mask = np.full([hh,ww,cc], (0,0,0), np.uint8)
cv2.drawContours(mask,[outer_contour],0,(255,255,255),thickness=cv2.FILLED)
# mask the input
masked_img = img.copy()
masked_img[mask == 0] = 0
#masked_img[mask != 0] = img[mask != 0]
# crop the bounding box region of the masked img
result = masked_img[y:y+h, x:x+w]
# draw the contour outline on a copy of result
result_outline = result.copy()
cv2.drawContours(result_outline,[outer_contour],0,(0,0,255),thickness=1,offset=(-x,-y))
# display it
cv2.imshow("img", img)
cv2.imshow("ffimg", ffimg)
cv2.imshow("mask", mask)
cv2.imshow("masked_img", masked_img)
cv2.imshow("result", result)
cv2.imshow("result_outline", result_outline)
cv2.waitKey(0)
cv2.destroyAllWindows()
# write result to disk
cv2.imwrite("odd_region_cropped.png", result)
cv2.imwrite("odd_region_cropped_outline.png", result_outline)
Result:
Result With Contour Drawn:

NumPy/OpenCV 2: how do I crop non-rectangular region?

I have a set of points that make a shape (closed polyline). Now I want to copy/crop all pixels from some image inside this shape, leaving the rest black/transparent. How do I do this?
For example, I have this:
and I want to get this:

*edit - updated to work with images that have an alpha channel.
This worked for me:
Make a mask with all black (all masked)
Fill a polygon with white in the shape of your ROI
combine the mask and your image to get the ROI with black everywhere else
You probably just want to keep the image and mask separate for functions that accept masks. However, I believe this does what you specifically asked for:
import cv2
import numpy as np
# original image
# -1 loads as-is so if it will be 3 or 4 channel as the original
image = cv2.imread('image.png', -1)
# mask defaulting to black for 3-channel and transparent for 4-channel
# (of course replace corners with yours)
mask = np.zeros(image.shape, dtype=np.uint8)
roi_corners = np.array([[(10,10), (300,300), (10,300)]], dtype=np.int32)
# fill the ROI so it doesn't get wiped out when the mask is applied
channel_count = image.shape[2] # i.e. 3 or 4 depending on your image
ignore_mask_color = (255,)*channel_count
cv2.fillPoly(mask, roi_corners, ignore_mask_color)
# from Masterfool: use cv2.fillConvexPoly if you know it's convex
# apply the mask
masked_image = cv2.bitwise_and(image, mask)
# save the result
cv2.imwrite('image_masked.png', masked_image)

The following code would be helpful for cropping the images and get them in a white background.
import cv2
import numpy as np
# load the image
image_path = 'input image path'
image = cv2.imread(image_path)
# create a mask with white pixels
mask = np.ones(image.shape, dtype=np.uint8)
mask.fill(255)
# points to be cropped
roi_corners = np.array([[(0, 300), (1880, 300), (1880, 400), (0, 400)]], dtype=np.int32)
# fill the ROI into the mask
cv2.fillPoly(mask, roi_corners, 0)
# The mask image
cv2.imwrite('image_masked.png', mask)
# applying th mask to original image
masked_image = cv2.bitwise_or(image, mask)
# The resultant image
cv2.imwrite('new_masked_image.png', masked_image)
Input Image:
Mask Image:
Resultant output image:

OpenCV - Apply mask to a color image

How can I apply mask to a color image in latest python binding (cv2)? In previous python binding the simplest way was to use cv.Copy e.g.
cv.Copy(dst, src, mask)
But this function is not available in cv2 binding. Is there any workaround without using boilerplate code?

Here, you could use cv2.bitwise_and function if you already have the mask image.
For check the below code:
img = cv2.imread('lena.jpg')
mask = cv2.imread('mask.png',0)
res = cv2.bitwise_and(img,img,mask = mask)
The output will be as follows for a lena image, and for rectangular mask.

Well, here is a solution if you want the background to be other than a solid black color. We only need to invert the mask and apply it in a background image of the same size and then combine both background and foreground. A pro of this solution is that the background could be anything (even other image).
This example is modified from Hough Circle Transform. First image is the OpenCV logo, second the original mask, third the background + foreground combined.
# http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_houghcircles/py_houghcircles.html
import cv2
import numpy as np
# load the image
img = cv2.imread('E:\\FOTOS\\opencv\\opencv_logo.png')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
# detect circles
gray = cv2.medianBlur(cv2.cvtColor(img, cv2.COLOR_RGB2GRAY), 5)
circles = cv2.HoughCircles(gray, cv2.HOUGH_GRADIENT, 1, 20, param1=50, param2=50, minRadius=0, maxRadius=0)
circles = np.uint16(np.around(circles))
# draw mask
mask = np.full((img.shape[0], img.shape[1]), 0, dtype=np.uint8) # mask is only
for i in circles[0, :]:
cv2.circle(mask, (i[0], i[1]), i[2], (255, 255, 255), -1)
# get first masked value (foreground)
fg = cv2.bitwise_or(img, img, mask=mask)
# get second masked value (background) mask must be inverted
mask = cv2.bitwise_not(mask)
background = np.full(img.shape, 255, dtype=np.uint8)
bk = cv2.bitwise_or(background, background, mask=mask)
# combine foreground+background
final = cv2.bitwise_or(fg, bk)
Note: It is better to use the opencv methods because they are optimized.

import cv2 as cv
im_color = cv.imread("lena.png", cv.IMREAD_COLOR)
im_gray = cv.cvtColor(im_color, cv.COLOR_BGR2GRAY)
At this point you have a color and a gray image. We are dealing with 8-bit, uint8 images here. That means the images can have pixel values in the range of [0, 255] and the values have to be integers.
Let's do a binary thresholding operation. It creates a black and white masked image. The black regions have value 0 and the white regions 255
_, mask = cv.threshold(im_gray, thresh=180, maxval=255, type=cv.THRESH_BINARY)
im_thresh_gray = cv.bitwise_and(im_gray, mask)
The mask can be seen below on the left. The image on its right is the result of applying bitwise_and operation between the gray image and the mask. What happened is, the spatial locations where the mask had a pixel value zero (black), became pixel value zero in the result image. The locations where the mask had pixel value 255 (white), the resulting image retained its original gray value.
To apply this mask to our original color image, we need to convert the mask into a 3 channel image as the original color image is a 3 channel image.
mask3 = cv.cvtColor(mask, cv.COLOR_GRAY2BGR) # 3 channel mask
Then, we can apply this 3 channel mask to our color image using the same bitwise_and function.
im_thresh_color = cv.bitwise_and(im_color, mask3)
mask3 from the code is the image below on the left, and im_thresh_color is on its right.
You can plot the results and see for yourself.
cv.imshow("original image", im_color)
cv.imshow("binary mask", mask)
cv.imshow("3 channel mask", mask3)
cv.imshow("im_thresh_gray", im_thresh_gray)
cv.imshow("im_thresh_color", im_thresh_color)
cv.waitKey(0)
The original image is lenacolor.png that I found here.

Answer given by Abid Rahman K is not completely correct. I also tried it and found very helpful but got stuck.
This is how I copy image with a given mask.
x, y = np.where(mask!=0)
pts = zip(x, y)
# Assuming dst and src are of same sizes
for pt in pts:
dst[pt] = src[pt]
This is a bit slow but gives correct results.
EDIT:
Pythonic way.
idx = (mask!=0)
dst[idx] = src[idx]

The other methods described assume a binary mask. If you want to use a real-valued single-channel grayscale image as a mask (e.g. from an alpha channel), you can expand it to three channels and then use it for interpolation:
assert len(mask.shape) == 2 and issubclass(mask.dtype.type, np.floating)
assert len(foreground_rgb.shape) == 3
assert len(background_rgb.shape) == 3
alpha3 = np.stack([mask]*3, axis=2)
blended = alpha3 * foreground_rgb + (1. - alpha3) * background_rgb
Note that mask needs to be in range 0..1 for the operation to succeed. It is also assumed that 1.0 encodes keeping the foreground only, while 0.0 means keeping only the background.
If the mask may have the shape (h, w, 1), this helps:
alpha3 = np.squeeze(np.stack([np.atleast_3d(mask)]*3, axis=2))
Here np.atleast_3d(mask) makes the mask (h, w, 1) if it is (h, w) and np.squeeze(...) reshapes the result from (h, w, 3, 1) to (h, w, 3).