Develop Reference

OpenCV: Can't find large rectangle contour

Opencv vers: 4.5
I'm trying to re-create the dimensions of an object by setting it up on a grid and taking as close to a top-down photo I can which I will then get the contours of the largest bounding rectangle and then perspective warp.
I'm currently unable to get the contour for a large bounding square however, it continually only finds smaller rectangles/squares which I'm assuming would not be large enough to properly fix the perspective.
First image: Original
Second image: What I get with my code using openCV
Third image: Close to what I'd ideally get
My code:
import imutils
import numpy as np
import cv2 as cv
# load the query image
image = cv.imread("path/to/image")
# make image greyscale, blur, find edges
grayscale_image = cv.cvtColor(image, cv.COLOR_BGR2GRAY)
thresh = cv.adaptiveThreshold(grayscale_image, 255, cv.ADAPTIVE_THRESH_GAUSSIAN_C,
cv.THRESH_BINARY, 11, 2)
# find contours in the threshed image, keep only the largest
# ones
cnts = cv.findContours(
thresh.copy(), cv.RETR_LIST, cv.CHAIN_APPROX_SIMPLE)
cnts = imutils.grab_contours(cnts)
cnts = sorted(cnts, key=cv.contourArea, reverse=True)[:5]
# draw contours for reference
cv.drawContours(image, cnts, -1, (0, 255, 0), 3)
Instead of adaptive thresholding for pre-processing I've tried using bilateral filter or gaussian blur into canny edge detection but the outcome still doesn't find large rectangles.
Any help would be greatly appreciated as I'm at a loss on why it can't detect larger squares. Also, if people think there's a better method for fixing the perspective so that I can accurately recreate the board dimensions please let me know.

You may apply the following stages:
Apply threshold using cv2.threshold (instead of cv2.adaptiveThreshold).
Apply opening with long column vector for keeping only the vertical lines.
Find contours in vert_lines.
Sort contours left to right.
Draw most left and most right contours on a sketch (black) image.
Apply opening with long row vector for keeping only the horizontal lines, find contours, sort top to bottom, and draw top and bottom contours.
Find inner contours in the sketch image (with the left, right, top and bottom lines).
The inner contour is the smallest one.
Here is a code sample:
import imutils
import numpy as np
import cv2
# load the query image
image = cv2.imread("image.png")
# make image greyscale, blur, find edges
grayscale_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
#thresh = cv2.adaptiveThreshold(grayscale_image, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
thresh = cv2.threshold(grayscale_image, 0, 255, cv2.THRESH_OTSU)[1] # Apply automatic threshold (use THRESH_OTSU).
rect_im = np.zeros_like(thresh) # Sketch image
# Apply opening with long column vector for keeping only the vertical lines.
vert_lines = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, np.ones(50))
# Apply opening with long row vector for keeping only the horizontal lines.
horz_lines = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, np.ones((1,50)))
# Find contours in vert_lines
cnts = imutils.grab_contours(cv2.findContours(vert_lines, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE))
# Sort contours left to right.
cnts = sorted(cnts, key=lambda c: cv2.boundingRect(c)[0]) # cv2.boundingRect(c)[0] is the left side x coordinate.
cv2.drawContours(rect_im, [cnts[0], cnts[-1]], -1, 255, -1) # Draw left and right contours
# Find contours in horz_lines
cnts = imutils.grab_contours(cv2.findContours(horz_lines, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE))
# Sort contours top to bottom.
cnts = sorted(cnts, key=lambda c: cv2.boundingRect(c)[1]) # cv2.boundingRect(c)[1] is the top y coordinate.
cv2.drawContours(rect_im, [cnts[0], cnts[-1]], -1, 255, -1) # Draw top and bottom contours
# Find contours in rect_im
cnts = imutils.grab_contours(cv2.findContours(rect_im, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)) # Note: use RETR_TREE for getting inner contour.
c = min(cnts, key=cv2.contourArea) # Get the smallest contour
# Draw contour for reference
cv2.drawContours(image, [c], -1, (0, 255, 0), 3)
Results:
thresh:
vert_lines:
horz_lines:
Left and right lines:
rect_im:
image (output):

Counting the number of shapes in an image

I am trying to count the number of shapes in an image of a board game. For some reason, the count is completely off.
I have been using code from the web (for example, https://www.pyimagesearch.com/2016/02/01/opencv-center-of-contour/)
I only need opencv for this, has anyone else had this problem, and how do I limit this to only the squares that I am interested in?
This is the same image as below but the background is now black. Now there are more center-dots than necessary all around the edge of the image.
I want the center dot to appear in every square on the board

I used paint to remove the edges of the numbers that were interrupting the white lines of each box. I assume that those numbers and dots were things that you added and not part of the original image.
Here is the image after my modifications
I made a mask by looking for white in the image. I dilated the results to get clearer separation and inverted so that each box was white.
I used findContours to get the contour of each white object in the mask. I filtered the contours by size to get rid of the arrows, letter bits, and the background of the image. From there I drew the centroid of each remaining contour.
import cv2
import numpy as np
# load image
img = cv2.imread("chutes_n_ladders.png");
# mask
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY);
mask = cv2.inRange(gray, 240, 255);
# dilate and invert
kernel = np.ones((3,3), np.uint8);
mask = cv2.dilate(mask, kernel, iterations = 1);
mask = cv2.bitwise_not(mask);
# contours
_, contours, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE);
# remove very large and very small contours
filtered = [];
low = 1000;
high = 100000;
for con in contours:
area = cv2.contourArea(con);
if low < area and area < high:
filtered.append(con);
# draw centers of each
print("Shapes: " + str(len(filtered)));
for con in filtered:
M = cv2.moments(con);
cx = int(M['m10']/M['m00']);
cy = int(M['m01']/M['m00']);
cv2.circle(img, (cx, cy), 10, (0, 200, 100), -1);
# show
cv2.imshow("Image", img);
cv2.imshow("Mask", mask);
cv2.waitKey(0);

How to detect intersecting shapes with OpenCV findContours?

I have two intersecting ellipses in a black and white image. I am trying to use OpenCV findContours to identify the separate shapes as separate contours using this code (and attached image below).
import numpy as np
import matplotlib.pyplot as plt
import cv2
import skimage.morphology
img_3d = cv2.imread("C:/temp/test_annotation_overlap.png")
img_grey = cv2.cvtColor(img_3d, cv2.COLOR_BGR2GRAY)
contours = cv2.findContours(img_grey, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)[-2]
fig, ax = plt.subplots(len(contours)+1,1, figsize=(5, 20))
thicker_img_grey = skimage.morphology.dilation(img_grey, skimage.morphology.disk(radius=3))
ax[0].set_title("ORIGINAL IMAGE")
ax[0].imshow(thicker_img_grey, cmap="Greys")
for i, contour in enumerate(contours):
new_img = np.zeros_like(img_grey)
cv2.drawContours(new_img, contour, -1, (255,255,255), 10)
ax[i+1].set_title(f"Contour {i}")
ax[i+1].imshow(new_img, cmap="Greys")
plt.show()
However four contours are found, none of which are the original contour:
How can I configure OpenCV.findContours to identify the two separate shapes? (Note I have already played around with Hough circles and found it unreliable for the images I am analysing)

Maybe I overkilled with this approach but it could be used as a working approach. You could find all the contours on the image - you will get the two contours that are like a "semicircle", the contour of the intersection and the contour that is the outer shape of the two addjointed circles. Smallest three contours should be the two semicircles and the intersection. If you draw combinations of two out of these three contours, you will get three mask out of which two will have the combination of one semicircle and the intersection. If you perform closing on the mask you will get your circle. Then you should simply make an algorithm to detect which two masks represent a full circle and you will get your result. Here is the sample solution:
import numpy as np
import cv2
# Function for returning solidity of contour - ratio of contour area to its
# convex hull area.
def checkSolidity(cnt):
area = cv2.contourArea(cnt)
hull = cv2.convexHull(cnt)
hull_area = cv2.contourArea(hull)
solidity = float(area)/hull_area
return solidity
img_orig = cv2.imread("circles.png")
# Had to dilate the image so the contour was completly connected.
img = cv2.dilate(img_orig, np.ones((3, 3), np.uint8))
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) # Grayscale transformation.
# Otsu threshold.
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)[1]
# Search for contours.
contours = cv2.findContours(thresh, cv2.CHAIN_APPROX_NONE, cv2.RETR_TREE)[0]
# Sorting contours from smallest to biggest.
contours.sort(key=lambda cnt: cv2.contourArea(cnt))
# Three contours - two semi circles and the intersection of the circles.
cnt1 = contours[0]
cnt2 = contours[1]
cnt3 = contours[2]
# Create three empty images
h, w = img.shape[:2]
mask1 = np.zeros((h, w), np.uint8)
mask2 = np.zeros((h, w), np.uint8)
mask3 = np.zeros((h, w), np.uint8)
# Draw all combinations of two out of three contours on the masks.
# The goal here is to draw one semicircle and the intersection together.
cv2.drawContours(mask1, [cnt1], 0, (255, 255, 255), -1)
cv2.drawContours(mask1, [cnt3], 0, (255, 255, 255), -1)
cv2.drawContours(mask2, [cnt2], 0, (255, 255, 255), -1)
cv2.drawContours(mask2, [cnt3], 0, (255, 255, 255), -1)
cv2.drawContours(mask3, [cnt1], 0, (255, 255, 255), -1)
cv2.drawContours(mask3, [cnt2], 0, (255, 255, 255), -1)
# Perform closing operation on the masks so that you get uniform contours.
kernel_size = 25
kernel = np.ones((kernel_size, kernel_size), np.uint8)
mask1 = cv2.morphologyEx(mask1, cv2.MORPH_CLOSE, kernel)
mask2 = cv2.morphologyEx(mask2, cv2.MORPH_CLOSE, kernel)
mask3 = cv2.morphologyEx(mask3, cv2.MORPH_CLOSE, kernel)
masks = [] # List for storing all the masks.
masks.append(mask1)
masks.append(mask2)
masks.append(mask3)
# List where you will append solidity of the found biggest contour of every mask.
solidity = []
for mask in masks:
cnts = cv2.findContours(mask, cv2.CHAIN_APPROX_NONE, cv2.RETR_TREE)[0]
cnt = max(cnts, key=lambda c: cv2.contourArea(c))
s = checkSolidity(cnt)
solidity.append(s)
# Index of the mask with smallest solidity.
min_solidity = solidity.index(min(solidity))
# The mask with the contour that has smallest solidity should be the one that
# has two semicirles drawn instead of one semicircle and the intersection.
#You could build a better function to check which mask is the one with
# two semicircles... like maybe the contour with the largest
# height and width of the bounding box etc.
# I chose solidity because it is enough for this example.
# Selection of colors.
colors = {
0: (0, 0, 255),
1: (0, 255, 0),
2: (255, 0, 0),
}
# Draw contours of the mask other two masks - those two that have the
# semicircle and the intersection.
for i, s in enumerate(solidity):
if s != solidity[min_solidity]:
cnts = cv2.findContours(
masks[i], cv2.CHAIN_APPROX_NONE, cv2.RETR_TREE)[0]
cnt = max(cnts, key=lambda c: cv2.contourArea(c))
cv2.drawContours(img_orig, [cnt], 0, colors[i], 1)
# Display result
cv2.imshow("img", img_orig)
cv2.waitKey(0)
cv2.destroyAllWindows()
Result:

Philosophically, you want to find two circles, because you search for them, you expect centers and radii. Graphically the figures are connected, we can see them separated because we know what a "circle" is and extrapolate the coordinates, which match the parts which overlap.
So what about finding the minimum enclosing circle for each contour (or in some cases fitEllipse and use the their parameters): https://docs.opencv.org/master/dd/d49/tutorial_py_contour_features.html
Then say draw that circle in a clear image and take the pixel coordinates which are not zero - by a mask or compute them by drawing a circle step by step.
Then compare these coordinates with the coordinates in the other contours with some precision and append the matching coordinates to the current contour.
Finally: draw the extended contour on a clear canvas and apply HoughCircles for a single non-overlapping circle. (Or compute the center and radius, the coordinates of a circle and comparing to the contour with a precision.)

For reference here I will post the solution I came up with based on some ideas here and a few more. This solution was 99.9% effective and recovering ellipses from images often including many overlapping, contained, and with other image noise such as lines, text and so on.
The code is too length and distributed to post here but the pseudocode is as follows.
Segment the image
Run cv2 findContours with RETR_EXTERNAL to get separate regions in the image
For each image, fill in the interior, apply mask, and extract the region to be processed independently of other regions.
Remaining steps are executed for each region independently
Run cv2 findContours with RETR_LIST to get all internal and external contours
For each contour found, apply polygon smoothing to reduce effect of pixellation
For each smoothed contour, identify the continuous segments within that contour which have the same curvature sign i.e. segments which are entirely curving right, or curving left (just compute angles and sign changes)
Within each segment, fit an ellipse model with least squares (scikit-learn EllipseModel)
Perform the Lee algorithm on the original image to compute for each pixel its minimum distance to a white pixel
For each model, perform a greedy local neighbourhood search to improve the fit against the original - the fit being the fitted ellipse maximum distance to white pixel, from the output of the lee algorithm
Not simple or elegant but is highly accurate for the content I am dealing with confirmed from manual review of a large number of images.

Remove top section of image above border line to detect text document

Using OpenCV (python) I am trying to remove the section of image which is above the border line (white area in this sample image where ORIGINAL is writtn) in the image shown below
Using horizontal and vertical kernels I am able to draw the wireframe, however that does not work many times because many times due to scanning quality few horizontal or vertical lines appear outside the wireframe which causes wrong contour detection. In this image also you can see on top right there is noise which I am detecting as topmost horizontal line.
What I want is, once I get the actual box then I can simply use x, y coordinates for OCR scanning of needed fields (like reference number, Issued In etc).
Following is what I have been able to extract using the code below. However not able to clip the outer extra section of image due to noisy horizontal or vertical lines outside this wireframe. Also tried filling outside section with black and then detecting the contours.
Suggestions please...
kernel_length = np.array(image).shape[1]//40
# A verticle kernel of (1 X kernel_length), which will detect all the verticle lines from the image.
verticle_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1, kernel_length))
# A horizontal kernel of (kernel_length X 1), which will help to detect all the horizontal line from the image.
hori_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (kernel_length, 1))
# A kernel of (3 X 3) ones.
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3))
# Morphological operation to detect verticle lines from an image
img_temp1 = cv2.erode(gray, verticle_kernel, iterations=3)
verticle_lines_img = cv2.dilate(img_temp1, verticle_kernel, iterations=3)

Instead of trying to find horizontal/vertical lines to detect the text document, a simple contour filtering approach should work here. The idea is to threshold the image to obtain a binary image then find contours and sort using contour area. The largest contour should be the text document. We can then apply a four point perspective transform to obtain a birds eye view of the image. Here's the results:
Input image:
Output:
Notice how the output image only has the desired text document and is aligned without a skewed angle.
Code
from imutils.perspective import four_point_transform
import cv2
import numpy
# Load image, grayscale, Gaussian blur, Otsu's threshold
image = cv2.imread("1.jpg")
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (3,3), 0)
thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]
# Find contours and sort for largest contour
cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)
displayCnt = None
for c in cnts:
# Perform contour approximation
peri = cv2.arcLength(c, True)
approx = cv2.approxPolyDP(c, 0.02 * peri, True)
if len(approx) == 4:
displayCnt = approx
break
# Obtain birds' eye view of image
warped = four_point_transform(image, displayCnt.reshape(4, 2))
cv2.imshow("thresh", thresh)
cv2.imshow("warped", warped)
cv2.imshow("image", image)
cv2.waitKey()

How to extract rectangles of varying edge intensity from images?

I am trying to extract the account number from an image of a cheque. The logic that I have is that, I am trying to find the rectangle that contains the account number, slice the bounding rectangle and then feed the slice into an OCR to get the text out of it.
The problem I am facing is when the rectangle is not very prominent and light colour, I am not able to get the rectangle contour since the edges are not connected totally.
How to overcome this?
Things I tried, but did not work are
I cannot increase the erosion iteration, to erode it more, because then the edges connect with the surrounding black pixels and form a different shape.
Reducing the threshold offset might help, but, it seems inefficient. Since the code has to work with several types of images. I can start with offset 10 and keep incrementing the offset and checking if I found the rectangle or not. This will increase the time a lot for cheques with prominent rectangles that work well at offset 20 or more. And since I don't have a condition to check if the edges of the rectangle are prominent or not, the loop has to be applied in all the cheques.
Keeping the above points in mind. Can someone help me out with a solution to this problem?
Libraries used and versions
scikit-image==0.13.1
opencv-python==3.3.0.10
Code
from skimage.filters import threshold_adaptive, threshold_local
import cv2
Step 1:
image = cv2.imread('cropped.png')
Step 2:
Using adaptive threshold from skimage to remove the background, so that I can get the account number rectangle box. This works fine for the cheques where the rectangle is more pronounced, but when the rectangle edges are thin, or are lighter in colour, the threshold results in
unconnected edges, because of which I am not able to find the contours. I have attached examples of this further down in the question.
account_number_block = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
account_number_block = threshold_adaptive(account_number_block, 251, offset=20)
account_number_block = account_number_block.astype("uint8") * 255
Step 3:
Erode the image a bit to try to connect small disconnections in the edges
kernel = np.ones((3,3), np.uint8)
account_number_block = cv2.erode(account_number_block, kernel, iterations=5)
Find the contours
(_, cnts, _) = cv2.findContours(account_number_block.copy(), cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
# cnts = sorted(cnts, key=cv2.contourArea)[:3]
rect_cnts = [] # Rectangular contours
for cnt in cnts:
approx = cv2.approxPolyDP(cnt,0.01*cv2.arcLength(cnt,True),True)
if len(approx) == 4:
rect_cnts.append(cnt)
rect_cnts = sorted(rect_cnts, key=cv2.contourArea, reverse=True)[:1]
Working Example
Step 1: Original Image
Step 2: After thresholding to remove the background.
Step 3: Finding contours to find rectangle box of the account number.
Failure Working example - Light rectangular boundary.
Step 1: Read original image
Step 2: After thresholding to remove the background. Notice that the edges of the rectangle are not connected, because of which I am not able to get the contour out of it.
Step 3: Finding contours to find rectangle box of the account number.

import numpy as np
import cv2
import pytesseract as pt
from PIL import Image
#Run Main
if __name__ == "__main__" :
image = cv2.imread("image.jpg", -1)
# resize image to speed up computation
rows,cols,_ = image.shape
image = cv2.resize(image, (np.int32(cols/2),np.int32(rows/2)))
# convert to gray and binarize
gray_img = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
binary_img = cv2.adaptiveThreshold(gray_img, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 9, 9)
# note: erosion and dilation works on white forground
binary_img = cv2.bitwise_not(binary_img)
# dilate the image to fill the gaps
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3,3))
dilated_img = cv2.morphologyEx(binary_img, cv2.MORPH_DILATE, kernel,iterations=2)
# find contours, discard contours which do not belong to a rectangle
(_, cnts, _) = cv2.findContours(dilated_img, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
rect_cnts = [] # Rectangular contours
for cnt in cnts:
approx = cv2.approxPolyDP(cnt,0.01*cv2.arcLength(cnt,True),True)
if len(approx) == 4:
rect_cnts.append(cnt)
# sort contours based on area
rect_cnts = sorted(rect_cnts, key=cv2.contourArea, reverse=True)[:1]
# find bounding rectangle of biggest contour
box = cv2.boundingRect(rect_cnts[0])
x,y,w,h = box[:]
# extract rectangle from the original image
newimg = image[y:y+h,x:x+w]
# use 'pytesseract' to get the text in the new image
text = pt.image_to_string(Image.fromarray(newimg))
print(text)
cv2.namedWindow('Image', cv2.WINDOW_NORMAL)
cv2.imshow('Image', newimg)
cv2.waitKey(0)
cv2.destroyAllWindows()
result: 03541140011724
result: 34785736216