Related
I applied the floodfill function in opencv to extract the foreground from the background but some of the objects in the image were not recognized by the algorithm so I would like to know how I can improve my detections and what modifications are necessary.
image = cv2.imread(args["image"])
image = cv2.resize(image, (800, 800))
h,w,chn = image.shape
ratio = image.shape[0] / 800.0
orig = image.copy()
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
gray = cv2.GaussianBlur(gray, (5, 5), 0)
edged = cv2.Canny(gray, 75, 200)
# show the original image and the edge detected image
print("STEP 1: Edge Detection")
cv2.imshow("Image", image)
cv2.imshow("Edged", edged)
warped1 = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
T = threshold_local(warped1, 11, offset = 10, method = "gaussian")
warped1 = (warped1 > T).astype("uint8") * 255
print("STEP 3: Apply perspective transform")
seed = (10, 10)
foreground, birdEye = floodFillCustom(image, seed)
cv2.circle(birdEye, seed, 50, (0, 255, 0), -1)
cv2.imshow("originalImg", birdEye)
cv2.circle(birdEye, seed, 100, (0, 255, 0), -1)
cv2.imshow("foreground", foreground)
cv2.imshow("birdEye", birdEye)
gray = cv2.cvtColor(foreground, cv2.COLOR_BGR2GRAY)
cv2.imshow("gray", gray)
cv2.imwrite("gray.jpg", gray)
threshImg = cv2.threshold(gray, 1, 255, cv2.THRESH_BINARY)[1]
h_threshold,w_threshold = threshImg.shape
area = h_threshold*w_threshold
cv2.imshow("threshImg", threshImg)[![enter image description here][1]][1]
The floodFillCustom function is as follows -
def floodFillCustom(originalImage, seed):
originalImage = np.maximum(originalImage, 10)
foreground = originalImage.copy()
cv2.floodFill(foreground, None, seed, (0, 0, 0),
loDiff=(10, 10, 10), upDiff=(10, 10, 10))
return [foreground, originalImage]
A little bit late, but here's an alternative solution for segmenting the tools. It involves converting the image to the CMYK color space and extracting the K (Key) component. This component can be thresholded to get a nice binary mask of the tools, the procedure is very straightforward:
Convert the image to the CMYK color space
Extract the K (Key) component
Threshold the image via Otsu's thresholding
Apply some morphology (a closing) to clean up the mask
(Optional) Get bounding rectangles of all the tools
Let's see the code:
# Imports
import cv2
import numpy as np
# Read image
imagePath = "C://opencvImages//"
inputImage = cv2.imread(imagePath+"DAxhk.jpg")
# Create deep copy for results:
inputImageCopy = inputImage.copy()
# Convert to float and divide by 255:
imgFloat = inputImage.astype(np.float) / 255.
# Calculate channel K:
kChannel = 1 - np.max(imgFloat, axis=2)
# Convert back to uint 8:
kChannel = (255*kChannel).astype(np.uint8)
The first step is to convert the BGR image to CMYK. There's no direct conversion in OpenCV for this, so I applied directly the conversion formula. We can get every color space component from that formula, but we are only interested on the K channel. The conversion is easy, but we need to be careful with the data types. We need to operate on float arrays. After getting the K channel, we convert back the image to an unsigned 8-bit array, this is the resulting image:
Let's threshold this image using Otsu's thresholding method:
# Threshold via Otsu:
_, binaryImage = cv2.threshold(kChannel, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
This yields the following binary image:
Looks very nice! Additionally, we can clean it up a little bit (joining the little gaps) using a morphological closing. Let's apply a rectangular structuring element of size 5 x 5 and use 2 iterations:
# Use a little bit of morphology to clean the mask:
# Set kernel (structuring element) size:
kernelSize = 5
# Set morph operation iterations:
opIterations = 2
# Get the structuring element:
morphKernel = cv2.getStructuringElement(cv2.MORPH_RECT, (kernelSize, kernelSize))
# Perform closing:
binaryImage = cv2.morphologyEx(binaryImage, cv2.MORPH_CLOSE, morphKernel, None, None, opIterations, cv2.BORDER_REFLECT101)
Which results in this:
Very cool. What follows is optional. We can get the bounding rectangles for every tool by looking for the outer (external) contours:
# Find the contours on the binary image:
contours, hierarchy = cv2.findContours(binaryImage, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# Look for the outer bounding boxes (no children):
for _, c in enumerate(contours):
# Get the contours bounding rectangle:
boundRect = cv2.boundingRect(c)
# Get the dimensions of the bounding rectangle:
rectX = boundRect[0]
rectY = boundRect[1]
rectWidth = boundRect[2]
rectHeight = boundRect[3]
# Set bounding rectangle:
color = (0, 0, 255)
cv2.rectangle( inputImageCopy, (int(rectX), int(rectY)),
(int(rectX + rectWidth), int(rectY + rectHeight)), color, 5 )
cv2.imshow("Bounding Rectangles", inputImageCopy)
cv2.waitKey(0)
Which produces the final image:
I am using cv2 findChessBoardCorners for camera calibration in a vision application. My call to the function looks like this:
def auto_detect_checkerboard(self, image):
retval, corners = cv2.findChessboardCorners(image, (7, 7), flags=cv2.CALIB_CB_ADAPTIVE_THRESH
+ cv2.CALIB_CB_EXHAUSTIVE)
if(retval):
return corners[0][0], corners[0][1]
else:
print("No Checkerboard Found")
assert False
But it seems to fail to find any corners on all images I have tried with it so far. The most trivial example I have used is
Is there an issue with my use of the the function? Or is there an issue with the image that I need to deal with in preprocessing?
So far I have tried converting to grayscale, and applying a Gaussian filter, neither of which seem to have made a difference.
My approach for the problem is to perform color-segmentation to get a binary mask. Next, using binary mask to remove the background to make the board visible, removed from artifacts. Finally output the chess border features in an accurate way.
Performing color-segmentation: We convert the loaded image to the HSV format define lower/upper ranges and perform color segmentation using cv2.inRange to obtain a binary mask.
Extracting chess-board: After obtaining binary mask we will use it to remove the background and separate chess part from the rest of the image using cv2.bitwise_and. Arithmetic operation and is highly useful for defining roi in hsv colored images.
Displaying chess-board features. After extracting the chessboard from the image, we will set the patternSizeto (7, 7) and flags to adaptive_thresh + fast_check + normalize image inspired from the source.
Steps:
Color-segmentation to get the binary mask.
lwr = np.array([0, 0, 143])
upr = np.array([179, 61, 252])
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
msk = cv2.inRange(hsv, lwr, upr)
Removing background using mask
krn = cv2.getStructuringElement(cv2.MORPH_RECT, (50, 30))
dlt = cv2.dilate(msk, krn, iterations=5)
res = 255 - cv2.bitwise_and(dlt, msk)
Displaying Chess-board features
res = np.uint8(res)
ret, corners = cv2.findChessboardCorners(res, (7, 7),
flags=cv2.CALIB_CB_ADAPTIVE_THRESH +
cv2.CALIB_CB_FAST_CHECK +
cv2.CALIB_CB_NORMALIZE_IMAGE)
if ret:
print(corners)
fnl = cv2.drawChessboardCorners(img, (7, 7), corners, ret)
cv2.imshow("fnl", fnl)
cv2.waitKey(0)
else:
print("No Checkerboard Found")
Code:
import cv2
import numpy as np
# Load the image
img = cv2.imread("kFM1C.jpg")
# Color-segmentation to get binary mask
lwr = np.array([0, 0, 143])
upr = np.array([179, 61, 252])
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
msk = cv2.inRange(hsv, lwr, upr)
# Extract chess-board
krn = cv2.getStructuringElement(cv2.MORPH_RECT, (50, 30))
dlt = cv2.dilate(msk, krn, iterations=5)
res = 255 - cv2.bitwise_and(dlt, msk)
# Displaying chess-board features
res = np.uint8(res)
ret, corners = cv2.findChessboardCorners(res, (7, 7),
flags=cv2.CALIB_CB_ADAPTIVE_THRESH +
cv2.CALIB_CB_FAST_CHECK +
cv2.CALIB_CB_NORMALIZE_IMAGE)
if ret:
print(corners)
fnl = cv2.drawChessboardCorners(img, (7, 7), corners, ret)
cv2.imshow("fnl", fnl)
cv2.waitKey(0)
else:
print("No Checkerboard Found")
To find lower and upper boundaries of the mask, you may find useful: HSV-Threshold-script
In my environment (opencv-python 4.7.0.68, opencv 4.5.4), just converting it to grey scale can make it work without additional adjustment (at least detected all but the lower left corners). After downsample by resize(), all corners are detected.
img_captured = cv2.imread('example.jpg', cv2.IMREAD_GRAYSCALE)
# img_captured = cv2.resize(img_captured, (350, 350))
GRID = (7, 7)
found, corners = cv2.findChessboardCorners(
img_captured, GRID, cv2.CALIB_CB_ADAPTIVE_THRESH)
cv2.drawChessboardCorners(img_captured_corners, GRID, corners, found)
cv2.imshow('img_captured_corners', img_captured_corners)
findChessboardCorners no resize
There is also findChessboardCornersSB. From my experience, it works generally better than the plain version. However, I don't know benchmark difference between the two methods.
findChessboardCornersSB
Let's say that we have the following black and white image (image.png):
We load the image using OpenCV2 with the following code:
import cv2
img = cv2.imread('image.png')
How can we detect the corners of all white rectangles in an image? We can assume that all rectangles are parallel to the corners of the image.
Result in this case should be in the following form (res[0] is left rectangle and res[1] is right rectangle):
res = [
[
(20, 30), # upper left
(40, 30), # upper right
(20, 80), # bottom left
(40, 80) # bottom right
],
[
(100, 20), # upper left
(140, 20), # upper right
(100, 70), # bottom left
(140, 70) # bottom right
]
]
There are a few possibilities:
the "Harris Corner Detector" is good at finding corners - see here
you can use OpenCV's findContours()
you could use "Hit-or-Miss" morphology to look for corners
you could convolve the image with a kernel and look for specific outputs
So, looking at the last option, if we slide a 2x2 pixel kernel over the image and multiply each of the elements underneath the kernel by 1 and add them all together, and then find all the pixels where that total comes to 255, that will be a 2x2 square where exactly one pixel is white - and that is a corner:
import cv2
import numpy as np
# Load image as greyscale
im = cv2.imread('tZHHE.png',cv2.IMREAD_GRAYSCALE)
# Make a 2x2 kernel of ones
kernel = np.ones((2,2), dtype=np.uint8)
# Convolve the image with the kernel
res = cv2.filter2D(im.astype(np.int16), -1, kernel)
# Get coordinates of points where the sum of the 2x2 window comes to 255
corners = np.where(res==255)
Sample Output
(array([101, 101, 118, 118, 166, 166, 174, 174]),
array([274, 307, 117, 134, 274, 307, 117, 134]))
Looking at the "Hit-or-Miss" morphology method, I will do it with ImageMagick straight in the Terminal, but you can equally do it with other Python libraries:
magick tZHHE.png -alpha off -morphology HMT Corners result.png
As always, I am indebted to Anthony Thyssen for his ImageMagick examples as linked above. We are looking for these specific shapes with the "Hit-or-Miss" morphology:
Keywords: Python, OpenCV, image processing, convolution, corner detect, corner detector, corner detection, ImageMagick Hit-or-Miss morphology.
This works for any number of rectangles:
import cv2 as cv
import pprint as pprint
img = cv.imread("tZHHE.png") # read image
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # make grayscale image
cv.imshow("Our initial image",img) # show our original image
corners = cv.goodFeaturesToTrack(gray,2000,0.01,5) # find our corners, 2000 is the number of corners we can detect, 5 is the distance between corners
xylist = [] #put all of our xy coords in here
for corn in corners: # extract our corners and put them in xylist
x,y = corn[0]
xylist.append((x,y))
x = int(x)
y = int(y)
cv.rectangle(img, (x-2,y-2), (x+2,y+2), (100,100,0),-1) # now mark where our corners are on our original image
res = [[] for i in range(int(len(xylist)/4))] # generate i nested lists for our rectangles
for index, item in enumerate(xylist): # format coordinates as you want them
res[index % int(len(xylist)/4)].append(item)
print("\n"+"found ",int(len(xylist)/4) ,"rectangles\n") # how many rectangles did we have?
print(res)
cv.imshow("Corners?", img) # show our new image with rectangle corners marked
Try findContours()
I suggest you to try findContours() with its companion boundingRect().
Here is how you can make it work.
Load the image in grayscale, then pass it to the function findContours().
img = cv2.imread('tZHHe.png', cv2.IMREAD_GRAYSCALE)
contours, hierarchy = cv2.findContours(img, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
Getting the bounding box from contours, it returns x, y coordinates of the top left corner and w, h the width and height of the box:
[cv2.boundingRect(contour) for contour in contours]
#=> [(117, 118, 17, 56), (274, 101, 33, 65)]
View in action
Try this maybe in a Jupyter Notebook to see a kind of animation:
def corners_from_bounding_rect(bounding_rect):
x, y, w, h = bounding_rect
points = {'top_left': (x, y), 'top_right':(x+w, y), 'bottom-left': (x, y+h), 'bottom-rigth':(x+w, y+h)}
return points
Retrieve the points from contours using the method defined:
corner_groups = [corners_from_bounding_rect(cv2.boundingRect(cnt)) for cnt in contours]
# [{'top_left': (117, 118),
# 'top_right': (134, 118),
# 'bottom-left': (117, 174),
# 'bottom-rigth': (134, 174)},
# {'top_left': (274, 101),
# 'top_right': (307, 101),
# 'bottom-left': (274, 166),
# 'bottom-rigth': (307, 166)}]
Then plot the sequence:
pinned_img = img.copy()
for n, corners in enumerate(corner_groups):
for name, point in corners.items():
cv2.circle(pinned_img, point, 10, 255)
plt.title(f'{n}-{name}')
plt.imshow(pinned_img)
plt.show()
The first image from the squence:
Try this code
import cv2
img=cv2.imread('tZHHE.png') # Read my Image
imgContours=img.copy() # Copy my Image for Contours
imgCanny=cv2.Canny(imgContours,10,50) # Image to Edges
contours,hierarchy =cv2.findContours(imgCanny,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)
rectangles=[]
for cont in reversed(contours):
area=cv2.contourArea(cont)
x, y, w, h = cv2.boundingRect(cont)
rectangles.append([[x,y],[x+w,y],[x+w,y+h],[x,y+h]])
cv2.imshow("con", imgContours)
cv2.waitKey(0)
print(rectangles)
cv2.destroyAllWindows()
left to right rectangles
Output:
[[[273, 100], [307, 100], [307, 166], [273, 166]], [[116, 117], [134, 117], [134, 174], [116, 174]]]
I am using OpenCV to estimate a webcam's intrinsic matrix from a series of chessboard images - as detailed in this tutorial, and reverse project a pixel to a direction (in term of azimuth/elevation angles).
The final goal is to let the user select a point on the image, estimate the direction of this point in relation to the center of the webcam, and use this as DOA for a beam-forming algorithm.
So once I have estimated the intrinsic matrix, I reverse project the user-selected pixel (see code below) and display it as azimuth/elevation angles.
result = [0, 0, 0] # reverse projected point, in homogeneous coord.
while 1:
_, img = cap.read()
if flag: # If the user has clicked somewhere
result = np.dot(np.linalg.inv(mtx), [mouse_x, mouse_y, 1])
result = np.arctan(result) # convert to angle
flag = False
cv2.putText(img, '({},{})'.format(mouse_x, mouse_y), (20, 440), cv2.FONT_HERSHEY_SIMPLEX,
0.5, (0, 255, 0), 2, cv2.LINE_AA)
cv2.putText(img, '({:.2f},{:.2f})'.format(180/np.pi*result[0], 180/np.pi*result[1]), (20, 460),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2, cv2.LINE_AA)
cv2.imshow('image', img)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
My problem is that I'm not sure whether my results are coherent. The major incoherence is that, the point of the image corresponding to the {0,0} angle is noticeably off the image center, as seen below (camera image has been replaced by a black background for privacy reasons) :
I don't really see a simple yet efficient way of measuring the accuracy (the only method I could think of was to use a servo motor with a laser on it, just under the camera and point it to the computed direction).
Here is the intrinsic matrix after calibration with 15 images :
I get an error of around 0.44 RMS which seems satisfying.
My calibration code :
nCalFrames = 12 # number of frames for calibration
nFrames = 0
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001) # termination criteria
objp = np.zeros((9*7, 3), np.float32)
objp[:, :2] = np.mgrid[0:9, 0:7].T.reshape(-1, 2)
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.
cap = cv2.VideoCapture(0)
previousTime = 0
gray = 0
while 1:
# Capture frame-by-frame
_, img = cap.read()
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# Find the chess board corners
ret, corners = cv2.findChessboardCorners(gray, (9, 7), None)
# If found, add object points, image points (after refining them)
if ret:
corners2 = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
if time.time() - previousTime > 2:
previousTime = time.time()
imgpoints.append(corners2)
objpoints.append(objp)
img = cv2.bitwise_not(img)
nFrames = nFrames + 1
# Draw and display the corners
img = cv2.drawChessboardCorners(img, (9, 7), corners, ret)
cv2.putText(img, '{}/{}'.format(nFrames, nCalFrames), (20, 460), cv2.FONT_HERSHEY_SIMPLEX,
2, (0, 255, 0), 2, cv2.LINE_AA)
cv2.putText(img, 'press \'q\' to exit...', (255, 15), cv2.FONT_HERSHEY_SIMPLEX,
0.5, (0, 0, 255), 1, cv2.LINE_AA)
# Display the resulting frame
cv2.imshow('Webcam Calibration', img)
if nFrames == nCalFrames:
break
if cv2.waitKey(1) & 0xFF == ord('q'):
break
RMS_error, mtx, disto_coef, _, _ = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None)
EDIT: another test method would be to use a whiteboard with known angles points and estimate the error by comparing with experimental results, but I don't know how to set up such a system
Regarding your first concern, it is normal to have the principal point off the image center. The estimated point, which is the point of zero elevation and azimuth, is the one that minimizes the radial distortion coefficients, and for a low value wide angle lens (e.g., that of a typical webcam) it can be easily off by noticeable amount.
Your calibration should be ok up to the call to calibrateCamera. However, in your code snippet it seems your ignoring the distortion coefficients. What is missing is initUndistortRectifyMap, which lets you also re-center the principal point if that matters.
h, w = img.shape[:2]
# compute new camera matrix with central principal point
new_mtx,roi = cv2.getOptimalNewCameraMatrix(mtx,disto_coef,(w,h),1,(w,h))
print(new_mtx)
# compute undistort maps
mapx,mapy = cv2.initUndistortRectifyMap(mtx,disto_coef,None,new_mtx,(w,h),5)
It essentially makes focal length equal in both dimensions and centers the principal point (see OpenCV python documentation for parameters).
Then, at each
_, img = cap.read()
you must undistort the image before rendering
# apply the remap
img = cv2.remap(img,mapx,mapy,cv2.INTER_LINEAR)
# crop the image
x,y,w,h = roi
img = img[y:y+h, x:x+w]
here, I put background to green to emphasize the barrel distortion. The output could be something like this (camera image replaced by checkerboard for privacy reasons):
If you do all these, your calibration target is accurate and your calibration samples fill the entire image area you should be quite confident of the computation. However, to validate the measured azimuth and elevation with respect to the undistorted image's pixel readings, I'd maybe suggest tape measure from the lenses first principal point and a calibration plate placed in normal angle right in front of the camera. There you can compute the expected angles and compare.
Hope this helps.
I'm doing finger detection i.e. the fingertips using VGA camera. I have used HSV and image thresholding and I am able to detect the fingertips.
The Problem: I can now find the centroid of the white spot in a black and white image if there is only one white spot (if I place just one finger); but if I place multiple fingers more white spots will come in the final image. So I want to find each centroid separately.
I want to find all centroids of each of the white spots i.e. if I place more than one finger in front of the camera. Look at the code below
thresholded_img = cv.CreateImage(cv.GetSize(hsv_img), 8, 1)
cv.InRangeS(hsv_img, (0,0,200), (0,0,255), thresholded_img)
moments = cv.Moments(cv.GetMat(thresholded_img,1), 0)
area = cv.GetCentralMoment(moments, 0, 0)
x = cv.GetSpatialMoment(moments, 1, 0)/area
y = cv.GetSpatialMoment(moments, 0, 1)/area
posY=y
posX=x
Here thresholded_img is a black and white image where the fingertips alone is represented as white and all other in black.
In this code if the thresholded_img contains a single white spot then I can get the x and y coordinates of the centroid of that dot correctly!
But how to find centroid of each white dots in this image?
But if there is multiple white dots in the thresholded image then it's finding wrong centroid!
How can I change the above code to find (x,y coordinates) separate centroid of each white dots in a single frame (image)?
Refer this image please.
http://www.csksoft.net/data/pic/laserkbd/036.jpg
I tested the following code with the image you uploaded.
I got the following textual output:
cv2 version: 2.4.4
centroids: [(580, 547), (437, 546), (276, 545), (115, 545), (495, 425), (334, 424), (174, 424), (24, 423), (581, 304), (437, 303), (277, 303), (117, 302), (495, 182), (334, 181), (174, 181), (25, 181), (581, 60), (438, 59), (277, 59), (117, 59)]
and this image:
#!/usr/bin/env python
import cv2
img = cv2.imread('HFOUG.jpg',cv2.CV_LOAD_IMAGE_GRAYSCALE)
_,img = cv2.threshold(img,0,255,cv2.THRESH_OTSU)
h, w = img.shape[:2]
contours0, hierarchy = cv2.findContours( img.copy(), cv2.RETR_LIST, cv2.CHAIN_APPROX_NONE)
moments = [cv2.moments(cnt) for cnt in contours0]
# Nota Bene: I rounded the centroids to integer.
centroids = [( int(round(m['m10']/m['m00'])),int(round(m['m01']/m['m00'])) ) for m in moments]
print 'cv2 version:', cv2.__version__
print 'centroids:', centroids
for c in centroids:
# I draw a black little empty circle in the centroid position
cv2.circle(img,c,5,(0,0,0))
cv2.imshow('image', img)
0xFF & cv2.waitKey()
cv2.destroyAllWindows()
See also this answer https://stackoverflow.com/a/9059648/15485 to the question Python OpenCV - Find black areas in a binary image