Develop Reference

How to automatically detect a specific feature from one image and map it to another mask image? Then how to smoothen only the corners of the image?

Using the dlib library I was able to mask the mouth feature from one image (masked).
masked
Similarly, I have another cropped image of the mouth that does not have the mask (colorlip).
colorlip
I had scaled and replaced the images (replaced) and using np.where as shown in the code below.
replaced
#Get the values of the lip and the target mask
lip = pred_toblackscreen[bbox_lip[0]:bbox_lip[1], bbox_lip[2]:bbox_lip[3],:]
target = roi[bbox_mask[0]:bbox_mask[1], bbox_mask[2]:bbox_mask[3],:]
cv2.namedWindow('masked', cv2.WINDOW_NORMAL)
cv2.imshow('masked', target)
#Resize the lip to be the same scale/shape as the mask
lip_h, lip_w, _ = lip.shape
target_h, target_w, _ = target.shape
fy = target_h / lip_h
fx = target_w / lip_w
scaled_lip = cv2.resize(lip,(0,0),fx=fx,fy=fy)
cv2.namedWindow('colorlip', cv2.WINDOW_NORMAL)
cv2.imshow('colorlip', scaled_lip)
update = np.where(target==[0,0,0],scaled_lip,target)
cv2.namedWindow('replaced', cv2.WINDOW_NORMAL)
cv2.imshow('replaced', update)
But the feature shape (lip) in 'colorlip' does not match the 'masked' image. So, there is a misalignment and the edges of the mask look sharp as if the image has been overlayed. How to solve this problem? And how to make the final replaced image look more subtle and normal?

**Update #2: OpenCV Image Inpainting to smooth jagged borders.
OpenCV python inpainting should help with rough borders. Using the mouth landmark model, mouth segmentation mask from DL model or anything that was used the border location can be found. From that draw border with a small chosen width around the mouth contour in a new image and use it as a mask for inpainting. The mask I provided need to be inverted to work.
In input masks one of the mask is wider, one has shadow and last one is narrow. The six output images are generated with radius value of 5 and 20 for all three masks.
Code
import numpy as np
# import cv2 as cv2
# import cv2
import cv2.cv2 as cv2
img = cv2.imread('images/lip_img.png')
#mask = cv2.imread('images/lip_img_border_mask.png',0)
mask = cv2.imread('images/lip_img_border_mask2.png',0)
#mask = cv2.imread('images/lip_img_border_mask3.png',0)
mask = np.invert(mask)
# Choose appropriate method and radius.
radius = 20
dst = cv2.inpaint(img, mask, radius, cv2.INPAINT_TELEA)
# dst = cv2.inpaint(img, mask, radius, cv2.INPAINT_NS)
cv2.imwrite('images/inpainted_lip.jpg', dst)
cv2.imshow('dst',dst)
cv2.waitKey(0)
cv2.destroyAllWindows()
Input Image and Masks
Output Images
**Update #1: Added Deep Image harmonization based blending methods.
Try OpenCV seamless cloning for subtle replacement and getting rid of sharp edges. Also deep learning based image inpainting on sharp corners or combining it with seamless clone may provide better results.
Deep learning based Image Harmonization can be another approach to blend together two images such that the cropped part matches the style of background image. Even in this case the pixel intensity will change to match the background but blending will be smoother. Links are added to bottom of the post.
Example
This code example is based on learnopencv seamless cloning example,
# import cv2
from cv2 import cv2
import numpy as np
src = cv2.imread("images/src_img.jpg")
dst = cv2.imread("images/dest_img.jpg")
src_mask = cv2.imread("images/src_img_rough_mask.jpg")
src_mask = np.invert(src_mask)
cv2.namedWindow('src_mask', cv2.WINDOW_NORMAL)
cv2.imshow('src_mask', src_mask)
cv2.waitKey(0)
# Where to place image.
center = (500,500)
# Clone seamlessly.
output = cv2.seamlessClone(src, dst, src_mask, center, cv2.NORMAL_CLONE)
# Write result
cv2.imwrite("images/opencv-seamless-cloning-example.jpg", output)
cv2.namedWindow('output', cv2.WINDOW_NORMAL)
cv2.imshow('output', output)
cv2.waitKey(0)
Source Image
Rough Mask Image
Destination Image
Final Image
Reference
https://docs.opencv.org/4.5.4/df/da0/group__photo__clone.html
https://learnopencv.com/seamless-cloning-using-opencv-python-cpp/
https://learnopencv.com/face-swap-using-opencv-c-python/
https://github.com/JiahuiYu/generative_inpainting
https://docs.opencv.org/4.x/df/d3d/tutorial_py_inpainting.html
Deep Image Harmonization
https://github.com/bcmi/Image-Harmonization-Dataset-iHarmony4
https://github.com/wasidennis/DeepHarmonization
https://github.com/saic-vul/image_harmonization
https://github.com/wuhuikai/GP-GAN
https://github.com/junleen/RainNet
https://github.com/bcmi/BargainNet-Image-Harmonization
https://github.com/vinthony/s2am

Why isn't my contour closed (Python, OpenCV)?

I've been trying to write a script in opencv for some greyscale image processing. However, I keep running into an issue when finding and drawing contours on thresholded images. Finding contours is easy and gives me the kind of result that I'm looking for. But, when I choose the largest contour by area and intend to draw it seperately, I get a much more 'broken up' result. I've been trying to figure out what is wrong with my code that results in this for a while now, but frankly can't figure it out. Anybody else have a similar experience or possible solution?
My (admittedly very messy) current code is:
import numpy as np
import cv2 as cv
from matplotlib import pyplot as plt
import os
import math as m
import imutils as imt
read_directory = r'E:\Other\Ultrasound_Trad_Alg\Input'
write_directory = r'E:\Other\Ultrasound_Trad_Alg\Output'
os.chdir(read_directory)
image_files = os.listdir(read_directory)
for image_file in image_files:
input_image_grey = cv.imread(image_file,0)
input_image_color = cv.imread(image_file)
input_image_color_2 = input_image_color.copy()
#Initial Black Background Masking Process:
blurred_input = cv.GaussianBlur(input_image_grey,(7,7),0)
_,thresholded_image_binary = cv.threshold(blurred_input,0,255,cv.THRESH_BINARY)
input_contours,hierarchy = cv.findContours(thresholded_image_binary,cv.RETR_EXTERNAL,cv.CHAIN_APPROX_SIMPLE)
chosen_input_contour = max(input_contours, key=cv.contourArea)
input_mask = np.zeros_like(input_image_grey)
cv.drawContours(input_mask, chosen_input_contour, -1, 255, -1)
#For Testing and Visualization:
cv.drawContours(input_image_color, chosen_input_contour, -1, (0,255,0), 1)
cv.drawContours(input_image_color_2, input_contours, -1, (0,255,0), 1)
cv.imshow("test1",thresholded_image_binary)
cv.imshow("test2",input_image_color)
cv.imshow("test3",input_image_color_2)
cv.imshow("mask",input_mask)
print(cv.isContourConvex(chosen_input_contour))
print(cv.contourArea(chosen_input_contour))
cv.waitKey(0)
When I run this code, I get the following set of 4 images just to demonstrate what I am talking about:
https://i.stack.imgur.com/IP7Ip.jpg
As can be seen, the initial set of contours are pretty much exactly what I'm looking for. However, by isolating the largest contour in the image and drawing it separately, I get a broken up contour that does not work for me. I've also checked the contour areas and it shows that I should have exactly 5, meaning there shouldn't be any tiny hidden contours messing with results either. Any thoughts on what is happening?

I think your issue in your Python/OpenCV code is that you have:
input_mask = np.zeros_like(input_image_grey)
and it should be
input_mask = np.zeros_like(thresholded_image_binary)
the binary mask needs to be the same bit-depth as your binary image, not your grayscale image. That is it needs to be 1-bit and not 8-bits.
See if that fixes it.

Unskew a rectangle picture that's inclined and a rombus shape

Hello I have some question is related to correct license plate image alignment. I already have searched in google, but I still cannot find the answer code.
Now, for I solve this problem, using python, opencv.
I have a license plate image that is skewed, inclined, or diagonal like below.
So, I want these images to align and "un-skew".
If you have some code, Would you mind share for me?
Thank you

Use geometric_transformations
import cv2
import numpy as np
# load image
img = cv2.imread('4llcP.png')
rows,cols,ch = img.shape
# cornerpoints of plate
pts1 = np.float32([[64,29],[533,120],[34,318]])
# new positions of points
pts2 = np.float32([[0,0],[cols-30,0],[0,rows-30]])
# transform image
M = cv2.getAffineTransform(pts1,pts2)
dst = cv2.warpAffine(img,M,(cols,rows))
# show image
cv2.imshow('Res',dst)
cv2.waitKey(0)
cv2.destroyAllWindows()

How to improve depth map and what are my stereo images lacking?

I've been trying to convert stereo images into a depth map with use of opencv, but not matter what I do it seems to come out unreadable.
I was able to get an accurate depth image of example images that were provided in the opencv tutorial but not on any other image. Even when I attempted to download other premade, calibrated stereo image from online I get terrible results that are neither accurate nor are even close to quality that I get with the example images.
here is my main python script that I use to make the depth map:
import numpy as np
import cv2
from matplotlib import pyplot as plt
imgL = cv2.imread('calimg_L.png',0)
imgR = cv2.imread('calimg_R.png',0)
# imgL = cv2.imread('./images/example_L.png',0)
# imgR = cv2.imread('./images/example_R.png',0)
stereo = cv2.StereoSGBM_create(numDisparities=16, blockSize=15)
disparity = stereo.compute(imgR,imgL)
norm_image = cv2.normalize(disparity, None, alpha = 0, beta = 1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)
cv2.imwrite("disparityImage.jpg", norm_image)
plt.imshow(norm_image)
plt.show()
where calimg_L.png is a calibrated version of the original image.
Here is the code I use to calibrate my images:
import numpy as np
import cv2
import glob
from matplotlib import pyplot as plt
def createCalibratedImage(inputImage, outputName):
# termination criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((3*3,3), np.float32)
objp[:,:2] = np.mgrid[0:3,0:3].T.reshape(-1,2)
# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.
# org = cv2.imread('./chess.jpg')
# orig_cal_img = cv2.resize(org, (384, 288))
# cv2.imwrite("cal_chess.jpg", orig_cal_img)
images = glob.glob('./chess_webcam/*.jpg')
for fname in images:
print('file in use: ' + fname)
img = cv2.imread(fname)
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
# Find the chess board corners
ret, corners = cv2.findChessboardCorners(gray, (3,3),None)
# print("doing the thing");
print('status: ' + str(ret));
# If found, add object points, image points (after refining them)
if ret == True:
# print("found something");
objpoints.append(objp)
cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)
imgpoints.append(corners)
# Draw and display the corners
cv2.drawChessboardCorners(img, (3,3), corners,ret)
cv2.imshow('img',img)
cv2.waitKey(500)
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1],None,None)
img = inputImage
h, w = img.shape[:2]
newcameramtx, roi=cv2.getOptimalNewCameraMatrix(mtx,dist,(w,h),1,(w,h))
# undistort
print('undistorting...')
mapx,mapy = cv2.initUndistortRectifyMap(mtx,dist,None,newcameramtx,(w,h),5)
dst = cv2.remap(inputImage ,mapx,mapy,cv2.INTER_LINEAR)
# crop the image
x,y,w,h = roi
dst = dst[y:y+h, x:x+w]
# cv2.imwrite('calibresult.png',dst)
cv2.imwrite(outputName + '.png',dst)
cv2.destroyAllWindows()
original_L = cv2.imread('capture_L.jpg')
original_R = cv2.imread('capture_R.jpg')
createCalibratedImage(original_R, "calimg_R")
createCalibratedImage(original_L, "calimg_L")
print("images calibrated and outputed")
This code was taken from opencv tutorial on how to calibrate images and was provided at least 16 images of the chess board, but was only able to identify the chessboard in about 4 - 5 of them. The reason I used such a relatively small grid search of 3x3 is because anything higher left me without any images to use for calibration due to its inability to find the chessboard.
Here is what I get from an example image(sorry for weird link, couldn't find how to upload):
https://ibb.co/DYMcdZc
here is the original:
https://ibb.co/gMkqyXD
https://ibb.co/YQZY40C
This acts a it should, but when I use it with any other image it gives me a mess, for example:
output:
https://ibb.co/kXwgDVn
looks like just a mess of pixels, to be fair when you put it into 'gray' on imshow it looks more readable but it is not very representative of the image's depth, here are the originals:
https://ibb.co/vqDKGS0
https://ibb.co/f0X1gMB
Even worse so, when I take images myself and do calibrate them through the chessboard code, it comes out as just a random mess of white and black pixels, and values of some goes into negatives and some pixels are impossibly high value.
tl;dr I can't get any stereo images to be made into a depth map even though the example image works just fine, why is that?

First I want to say that obtaining a good depth map is not such a simple task, and using the basic StereoMatching won't always lead to good results. Nevertheless, something better can be achieved.
In order:
Calibration: you should be able to find the checkerboard in more images, 4/5 is a very low number for calibration, it is very hard to estimate correctly the camera parameters with such low number. How do the images look like? Did you read them as grayscale images? Usually also using a different number for row and column (not 3x3 grid, like 4x3) helps to understand the checkerboard position (otherwise it could be ambiguous which side is up or right, for example, a 90 rotation would result in 0 rotation).
Rectification: this can be easily checked by looking at the images. Open two images on two different layers (using GIMP or similar) and check for similar points. After you rectified the images, they should lie on the same line. Are they really on the same line? If yes, rectification work, otherwise, you need a better calibration. The stereo matching won't work without this step.
Stereo Matching: if all above steps are correct, then you may have a problem on the parameters of the stereo matching. First thing to check is disparity range (since it looks like you have different resolution between example images and your images, you should check and adapt that value). Min disparity can also help (if you reduce the disparity range, you reduce the error possibilities) and also block size (15 is quite big, smaller is also enough).
From what you say, my guess would be the problem is on the calibration. You should try to check the rectified images, and if the problem is there try to acquire a new dataset (or find online a better one) and calibrate your images there. Once you can calibrate and rectify your images correctly, you should get better results.
I see the code is similar to the tutorial here so I guess that's correct and the main problem are the images. Hope this can help,I can help you more if you test and see where the probelm is!

How can i find cycles in a skeleton image with python libraries?

I have many skeletonized images like this:
How can i detect a cycle, a loop in the skeleton?
Are there "special" functions that do this or should I implement it as a graph?
In case there is only the graph option, can the python graph library NetworkX can help me?

You can exploit the topology of the skeleton. A cycle will have no holes, so we can use scipy.ndimage to find any holes and compare. This isn't the fastest method, but it's extremely easy to code.
import scipy.misc, scipy.ndimage
# Read the image
img = scipy.misc.imread("Skel.png")
# Retain only the skeleton
img[img!=255] = 0
img = img.astype(bool)
# Fill the holes
img2 = scipy.ndimage.binary_fill_holes(img)
# Compare the two, an image without cycles will have no holes
print "Cycles in image: ", ~(img == img2).all()
# As a test break the cycles
img3 = img.copy()
img3[0:200, 0:200] = 0
img4 = scipy.ndimage.binary_fill_holes(img3)
# Compare the two, an image without cycles will have no holes
print "Cycles in image: ", ~(img3 == img4).all()
I've used your "B" picture as an example. The first two images are the original and the filled version which detects a cycle. In the second version, I've broken the cycle and nothing gets filled, thus the two images are the same.

First, let's build an image of the letter B with PIL:
import Image, ImageDraw, ImageFont
image = Image.new("RGBA", (600,150), (255,255,255))
draw = ImageDraw.Draw(image)
fontsize = 150
font = ImageFont.truetype("/usr/share/fonts/truetype/liberation/LiberationMono-Regular.ttf", fontsize)
txt = 'B'
draw.text((30, 5), txt, (0,0,0), font=font)
img = image.resize((188,45), Image.ANTIALIAS)
print type(img)
plt.imshow(img)
you may find a better way to do that, particularly with path to the fonts. Ii would be better to load an image instead of generating it. Anyway, we have now something to work on:
Now, the real part:
import mahotas as mh
img = np.array(img)
im = img[:,0:50,0]
im = im < 128
skel = mh.thin(im)
noholes = mh.morph.close_holes(skel)
plt.subplot(311)
plt.imshow(im)
plt.subplot(312)
plt.imshow(skel)
plt.subplot(313)
cskel = np.logical_not(skel)
choles = np.logical_not(noholes)
holes = np.logical_and(cskel,noholes)
lab, n = mh.label(holes)
print 'B has %s holes'% str(n)
plt.imshow(lab)
And we have in the console (ipython):
B has 2 holes

Converting your skeleton image to a graph representation is not trivial, and I don't know of any tools to do that for you.
One way to do it in the bitmap would be to use a flood fill, like the paint bucket in photoshop. If you start a flood fill of the image, the entire background will get filled if there are no cycles. If the fill doesn't get the entire image then you've found a cycle. Robustly finding all the cycles could require filling multiple times.
This is likely to be very slow to execute, but probably much faster to code than a technique where you trace the skeleton into graph data structure.