Develop Reference

Finding text between two lines using Python OpenCV

I want to identify and highlight / crop the text between two lines using Python (cv2).
One line is a wavy line at the top, and the second line somewhere in the page. This line can appear at any height on the page, ranging from just after 1 line to just before the last line.
An example,
I believe I need to use HoughLinesP() somehow with proper parameters for this.
I've tried some examples involving a combination of erode + dilate + HoughLinesP.
e.g.
img = cv2.imread(image)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
kernel_size = 5
blur_gray = cv2.GaussianBlur(gray, (kernel_size, kernel_size), 0)
# erode / dilate
erode_kernel_param = (5, 200) # (5, 50)
dilate_kernel_param = (5, 5) # (5, 75)
img_erode = cv2.erode(blur_gray, np.ones(erode_kernel_param))
img_dilate = cv2.dilate(img_erode, np.ones(dilate_kernel_param))
# %% Second, process edge detection use Canny.
low_threshold = 50
high_threshold = 150
edges = cv2.Canny(img_dilate, low_threshold, high_threshold)
# %% Then, use HoughLinesP to get the lines.
# Adjust the parameters for better performance.
rho = 1 # distance resolution in pixels of the Hough grid
theta = np.pi / 180 # angular resolution in radians of the Hough grid
threshold = 15 # min number of votes (intersections in Hough grid cell)
min_line_length = 600 # min number of pixels making up a line
max_line_gap = 20 # max gap in pixels between connectable line segments
line_image = np.copy(img) * 0 # creating a blank to draw lines on
# %% Run Hough on edge detected image
# Output "lines" is an array containing endpoints of detected line segments
lines = cv2.HoughLinesP(edges, rho, theta, threshold, np.array([]),
min_line_length, max_line_gap)
if lines is not None:
for line in lines:
for x1, y1, x2, y2 in line:
cv2.line(line_image, (x1, y1), (x2, y2), (255, 0, 0), 5)
# %% Draw the lines on the image
lines_edges = cv2.addWeighted(img, 0.8, line_image, 1, 0)
However, in many cases the lines dont get identified propery.
Some examples of errors being,
Too many lines being identified (ones in the text as well)
Lines not being identified completely
Lines not being identified at all
Am I on the right track? Do I just need to hit the correct combination of parameters for this purpose? or is there a simpler way / trick which will let me reliably crop the text between these two lines?
In case it's relevant, I need to do this for ~450 pages.
Here's the link to the book, in case someone wants to examine more examples of pages.
https://archive.org/details/in.ernet.dli.2015.553713/page/n13/mode/2up
Thank you.
Solution
I've made minor modifications to the answer by Ari (Thank you), and made the code a bit more comprehensible for my own sake, here's my code.
The core idea is,
Find contours and their bounding rectangles.
Two "widest" contours would represent the two lines.
Thereafter, take the lower side of the top rectangle and upper side of the bottom rectangle to bound the area (text) we are interested in.
for image in images:
base_img = cv2.imread(image)
height, width, channels = base_img.shape
img = cv2.cvtColor(base_img, cv2.COLOR_BGR2GRAY)
ret, img = cv2.threshold(img, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
img = cv2.bitwise_not(img)
contours, hierarchy = cv2.findContours(
img, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE
)
# Get rectangle bounding contour
rects = [cv2.boundingRect(contour) for contour in contours]
# Rectangle is (x, y, w, h)
# Top-Left point of the image is (0, 0), rightwards X, downwards Y
# Sort the contours bigger width first
rects.sort(key=lambda r: r[2], reverse=True)
# Get the 2 "widest" rectangles
line_rects = rects[:2]
line_rects.sort(key=lambda r: r[1])
# If at least two rectangles (contours) were found
if len(line_rects) >= 2:
top_x, top_y, top_w, top_h = line_rects[0]
bot_x, bot_y, bot_w, bot_h = line_rects[1]
# Cropping the img
# Crop between bottom y of the upper rectangle (i.e. top_y + top_h)
# and the top y of lower rectangle (i.e. bot_y)
crop_img = base_img[top_y+top_h:bot_y]
# Highlight the area by drawing the rectangle
# For full width, 0 and width can be used, while
# For exact width (erroneous) top_x and bot_x + bot_w can be used
rect_img = cv2.rectangle(
base_img,
pt1=(0, top_y + top_h),
pt2=(width, bot_y),
color=(0, 255, 0),
thickness=2
)
cv2.imwrite(image.replace('.jpg', '.rect.jpg'), rect_img)
cv2.imwrite(image.replace('.jpg', '.crop.jpg'), crop_img)
else:
print(f"Insufficient contours in {image}")

You can find the Contours, and then take the two with the biggest width.
base_img = cv2.imread('a.png')
img = cv2.cvtColor(base_img, cv2.COLOR_BGR2GRAY)
ret, img = cv2.threshold(img, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
img = cv2.bitwise_not(img)
cnts, hierarchy = cv2.findContours(img, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
# sort the cnts bigger width first
cnts.sort(key=lambda c: cv2.boundingRect(c)[2], reverse=True)
# get the 2 big lines
lines = [cv2.boundingRect(cnts[0]), cv2.boundingRect(cnts[1])]
# higher line first
lines.sort(key=lambda c: c[1])
# croping the img
crop_img = base_img[lines[0][1]:lines[1][1]]

How can I find the center the hole with hough circles

For this image, I tried to use hough cirlce to find the center of the "black hole".
After playing with the parameters of cv2.HoughCircles for a long time, the following is the best I can get.
raw image:
# reproducible code for stackoverflow
import cv2
import os
import sys
from matplotlib import pyplot as plt
import numpy as np
# read image can turn it gray
img = cv2.imread(FILE)
cimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img_gray = dst = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
plt.figure(figsize = (18,18))
plt.imshow(cimg, cmap = "gray")
# removing noises
element = cv2.getStructuringElement(cv2.MORPH_RECT, (5, 5))
closing = cv2.morphologyEx(y, cv2.MORPH_CLOSE, element, iterations = 7)
plt.figure(figsize = (12,12))
plt.imshow(closing, cmap = "gray")
# try to find the circles
circles = cv2.HoughCircles(closing,cv2.HOUGH_GRADIENT,3,50,
param1=50,param2=30,minRadius=20,maxRadius=50)
circles = np.uint16(np.around(circles))
for i in circles[0,:]:
# draw the outer circle
cv2.circle(cimg,(i[0],i[1]),i[2],(0,255,0),2)
# draw the center of the circle
cv2.circle(cimg,(i[0],i[1]),2,(0,0,255),3)
plt.figure(figsize = (12,12))
plt.imshow(cimg)
Update::
The one with Canny:
edges = cv2.Canny(closing, 100, 300)
plt.figure(figsize = (12,12))
plt.imshow(edges, cmap = "gray")
circles = cv2.HoughCircles(edges,cv2.HOUGH_GRADIENT,2,50,
param1=50,param2=30,minRadius=20,maxRadius=60)
circles = np.uint16(np.around(circles))
cimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
for i in circles[0,:]:
# draw the outer circle
cv2.circle(cimg,(i[0],i[1]),i[2],(0,255,0),2)
# draw the center of the circle
cv2.circle(cimg,(i[0],i[1]),2,(0,0,255),3)
plt.figure(figsize = (12,12))
plt.imshow(cimg)
Still not the right circle that is wanted.
Update:
#crackanddie
Sometimes there is 6 or 9 in the identity number.
The circle in 6 or 9 is not very round.
Is there any way to filter that out?

This is an alternative method if you do not want to implement or fiddle with Hough's parameters. You must be sure there's at least one circle visible in your picture. The idea is to create a segmentation mask based on the CMYK color space and filter the blobs of interest by circularity and area. These are the steps:
Convert the image from BGR to CMYK
Threshold the K channel to get a binary mask
Filter blobs by circularity and area
Approximate the filtered blobs as circles
I'm choosing the CMYK color space because the circle is mostly black. The K (key) channel (in this case - black) should do a good job of representing the blob of interest, albeit, with some noise - as usual. Let's see the code:
# Imports:
import cv2
import numpy as np
# image path
path = "D://opencvImages//"
fileName = "dyj3O.jpg"
# load image
bgr = cv2.imread(path + fileName)
Alright, we need to convert the image from BGR to CMYK. OpenCV does not offer the conversion, so we need to do it manually. The formula is very straightforward. I'm just interested on the K channel, so I just calculate it like this:
# Make float and divide by 255:
bgrFloat = bgr.astype(np.float) / 255.
# Calculate K as (1 - whatever is biggest out of bgrFloat)
kChannel = 1 - np.max(bgrFloat, axis=2)
# Convert back to uint 8:
kChannel = 255 * kChannel
kChannel = kChannel.astype(np.uint8)
Gotta keep en eye on the data types, because there are float operations going on. This is the result:
As you see, the hole is almost 100% white, that's cool, we can threshold this image via Otsu like this:
# Compute binary mask of the hole via Otsu:
_, binaryImage = cv2.threshold(kChannel, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
Which gives you this nice binary mask:
Now, here comes the laborious part. Let's find contours on this image. For every contour/blob compute circularity and area. Use this info to filter noise and get the contour of interest, keep in mind that a perfect circle should have circularity close to 1.0. Once you get a contour of interest, approximate a circle to it. This is the process:
# Find the big contours/blobs on the filtered image:
contours, hierarchy = cv2.findContours(binaryImage, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
# Store the detected circles here:
detectedCircles = []
# Look for the potential contours of interest:
for _, c in enumerate(contours):
# Get the blob's area and perimeter:
contourArea = cv2.contourArea(c)
contourPerimeter = cv2.arcLength(c, True)
# Compute circularity:
if contourPerimeter > 0:
circularity = (4 * 3.1416 * contourArea) / (pow(contourPerimeter, 2))
else:
circularity = 0.0
# Set the min threshold values to identify the
# blob of interest:
minCircularity = 0.7
minArea = 2000
if circularity >= minCircularity and contourArea >= minArea:
# Approximate the contour to a circle:
(x, y), radius = cv2.minEnclosingCircle(c)
# Compute the center and radius:
center = (int(x), int(y))
# Cast radius to in:
radius = int(radius)
# Store the center and radius:
detectedCircles.append([center, radius])
# Draw the circles:
cv2.circle(bgr, center, radius, (0, 255, 0), 2)
cv2.imshow("Detected Circles", bgr)
print("Circles Found: " + str(len(detectedCircles)))
Additionally, I have stored the circle (center and radius) in the detectedCircles list. This is the final result:
Circles Found: 1

Here it is:
import numpy as np
import cv2
def threshold_gray_const(image_, rang: tuple):
return cv2.inRange(image_, rang[0], rang[1])
def binary_or(image_1, image_2):
return cv2.bitwise_or(image_1, image_2)
def negate_image(image_):
return cv2.bitwise_not(image_)
def particle_filter(image_, power):
# Abdrakov's particle filter
nb_components, output, stats, centroids = cv2.connectedComponentsWithStats(image_, connectivity=8)
sizes = stats[1:, -1]
nb_components = nb_components - 1
min_size = power
img2 = np.zeros(output.shape, dtype=np.uint8)
for i in range(0, nb_components):
if sizes[i] >= min_size:
img_to_compare = threshold_gray_const(output, (i + 1, i + 1))
img2 = binary_or(img2, img_to_compare)
img2 = img2.astype(np.uint8)
return img2
def reject_borders(image_):
# Abdrakov's border rejecter
out_image = image_.copy()
h, w = image_.shape[:2]
for row in range(h):
if out_image[row, 0] == 255:
cv2.floodFill(out_image, None, (0, row), 0)
if out_image[row, w - 1] == 255:
cv2.floodFill(out_image, None, (w - 1, row), 0)
for col in range(w):
if out_image[0, col] == 255:
cv2.floodFill(out_image, None, (col, 0), 0)
if out_image[h - 1, col] == 255:
cv2.floodFill(out_image, None, (col, h - 1), 0)
return out_image
src = cv2.imread("your_image")
img_gray = dst = cv2.cvtColor(src, cv2.COLOR_BGR2GRAY)
element = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3))
closing = cv2.morphologyEx(img_gray, cv2.MORPH_CLOSE, element, iterations=2)
tv, thresh = cv2.threshold(closing, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
neg = negate_image(thresh)
rej = reject_borders(neg)
filtered = particle_filter(rej, 300)
edges = cv2.Canny(filtered, 100, 200)
circles = cv2.HoughCircles(edges, cv2.HOUGH_GRADIENT, 3, 50, param1=50, param2=30, minRadius=20, maxRadius=50)
circles = np.uint16(np.around(circles))
for i in circles[0, :]:
# draw the outer circle
cv2.circle(src, (i[0], i[1]), i[2], (0, 255, 0), 2)
# draw the center of the circle
cv2.circle(src, (i[0], i[1]), 2, (0, 0, 255), 3)
cv2.imshow("closing", closing)
cv2.imshow("edges", edges)
cv2.imshow("out", src)
cv2.waitKey(0)
I changed cv2.morphologyEx parameters a bit, because they were too strong. And after this noise removing I made a binary image using cv2.THRESH_OTSU parameter, negated it, rejected borders and filtered a bit. Then I used cv2.Canny to find edges and this 'cannied' image I passed into cv2.HoughCircles. If any questions - ask me :)

If you want to use a "thinking out of the box" solution then check this solution out. Remember this might have a few false positives in some cases and would only work in cases where circle contour is complete or joined.
import numpy as np
import cv2
import matplotlib.pyplot as plt
from math import pi
pi_eps = 0.1
rgb = cv2.imread('/path/to/your/image/find_circle.jpg')
gray = cv2.cvtColor(rgb, cv2.COLOR_BGR2GRAY)
th = cv2.adaptiveThreshold(gray,255, cv2.ADAPTIVE_THRESH_MEAN_C,cv2.THRESH_BINARY_INV,21,5)
contours, hier = cv2.findContours(th.copy(),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
out_img = rgb.copy()
for i in range(len(contours)):
x,y,w,h = cv2.boundingRect(contours[i])
ar = min(w,h)/max(w,h)
# For a circle aspect ratio is close to 1.0
# In your use case circle diameter is between 40px-100px
if ar < 0.9 or \
w < 40 or w > 100:
continue
# P = 2 * PI * r
perimeter = cv2.arcLength(contours[i], True)
if perimeter == 0:
continue
# Second level confirmation could be done using PI = P * P / (4 * A)
# A = PI * r * r
area = cv2.contourArea(contours[i])
if area == 0:
continue
# d = (w+h) / 2 average diameter
# A contour is a circle if (P / d) = PI
ctr_pi = perimeter / ((w+h) / 2)
if abs(ctr_pi - pi) < pi_eps * pi:
cv2.circle(out_img, (int(x+w/2), int(y+h/2)), int(max(w,h)/2), (0, 255, 0), 1)
print("Center of the circle: ", x + w/2, y+h/2)
plt.imshow(out_img)

Finding length of each line segment passing through centroid , and how to constrict the line till outer contour

Input Image
Processed Image
import numpy as np
import cv2
img = cv2.imread('Image(i).png', 0)
ret, img =cv2.threshold(img, 128, 255, cv2.THRESH_BINARY)
img_bw = img<=120
img_bw =img_bw.astype('uint8')
#Fit the ellipses
contours0, hierarchy = cv2.findContours( img.copy(), cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
outer_ellipse = [cv2.approxPolyDP(contours0[0], 0.1, True)]
inner_ellipse = [cv2.approxPolyDP(contours0[0], 0.1, True)]
ref = np.zeros_like(img_bw)
out=img.copy()
h, w = img.shape[:2]
vis = np.zeros((h, w, 3), np.uint8)
cv2.drawContours( vis, outer_ellipse, -1, (255,0,0), 1)
cv2.drawContours( vis, inner_ellipse, -1, (0,0,255), 1)
##Extract contour of ellipses
cnt_outer = np.vstack(outer_ellipse).squeeze()
cnt_inner = np.vstack(inner_ellipse).squeeze()
#Determine centroid
M = cv2.moments(cnt_inner)
cx = int(M['m10']/M['m00'])
cy = int(M['m01']/M['m00'])
print (cx, cy)
#Draw full segment lines
#cv2.line(vis,(cx,0),(cx,w),(150,0,0),1)
width = img.shape[1]
height = img.shape[0]
N = 20
for i in range(N):
tmp = np.zeros_like(img_bw)
theta = i*(360/N)
theta *= np.pi/180.0
cv2.line(tmp, (cx, cy),
(int(cx-np.cos(theta)*w),
int(cy+np.sin(theta)*h)), (150,0,0), 1)
(row,col) = np.nonzero(np.logical_and(tmp, ref))
#cv2.line(out, (cx, cy), (col,row),(255,0,0), 1)
# Show the image
cv2.imshow('Output', out)
cv2.waitKey(0)
cv2.destroyAllWindows()
As seen in processed image the lines passing through centroid are not constricted till outer contour and are passing trough it.
I want the lines to be stopped at the outer contour so as that I can measure distance from centroid to the outer contour.
First image is the input image and second image is of line segments passing through centroid.

Here's a possible approach:
draw your outer contour filled with black on a white background
You now have a black ellipse. Then, without actually drawing anything:
use skimage.draw.line to get the list of points along all your radii
use Numpy argmax() to get first white pixel along radii
Here is the code:
#!/usr/bin/env python3
import cv2
import math
from skimage.draw import line
import numpy as np
# Load image as greyscale
img = cv2.imread('ellipses.png', cv2.IMREAD_GRAYSCALE)
_, img = cv2.threshold(img, 128, 255, cv2.THRESH_BINARY)
h, w = img.shape
#Fit the ellipses
contours, hierarchy = cv2.findContours( img, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
outer_ellipse = [cv2.approxPolyDP(contours[0], 0.1, True)]
# Draw outer contour filled with black on white background
vis = np.zeros_like(img) + 255
cnt = cv2.drawContours(vis, outer_ellipse, -1, 0, -1)
# Centroid by existing method
cx, cy = 365, 335
maxThickness = 0
# Take 10 points along top
for x in range(0,w,int(w/10)):
# ... and bottom
for y in 0, h-1:
# Get y and x of all pixels between centroid and top and bottom edge
yy, xx = line(cy, cx, 0, x)
firstWhiteIndex = np.argmax(vis[yy,xx])
fx, fy = xx[firstWhiteIndex], yy[firstWhiteIndex]
# Get length of this radial line
length = np.sqrt((cx-fx)**2 + (cy-fy)**2)
# Remember if longer than all others so far seen
if length > maxThickness:
maxThickness = length
fxMax, fyMax = fx, fy
# Take 10 points down left side
for y in range(0,h,int(h/10)):
# ... and right
for x in 0, w-1:
# Get y and x of all pixels between centroid and left and right edge
yy, xx = line(cy, cx, 0, x)
firstWhiteIndex = np.argmax(vis[yy,xx])
fx, fy = xx[firstWhiteIndex], yy[firstWhiteIndex]
# Get length of this radial line
length = np.sqrt((cx-fx)**2 + (cy-fy)**2)
# Remember if longer than all others so far seen
if length > maxThickness:
maxThickness = length
fxMax, fyMax = fx, fy
print(f'Max thickness: {maxThickness}')
# Draw thickest radius in mid-grey
cv2.line(img, (cx,cy), (fxMax, fyMax), 128, 5)
cv2.imwrite('result.png', img)

I have an approach that is not the best but this is what I can think of now.
While drawing lines in the above image, modify the code and do the following:
Before the for loop, draw a binary image of the same size containing only the outer contour circle. Save this image for later use.
Now in the for loop, draw each line in a separate binary blank image. Thus, now you will have two images, first the image having only the outer circle, and second image will only contain the line.
Now perform a bitwise_and operation on these 2 images.
Now you will get a white pixel only that is the point of intersection of the line and the outer circle.
Now find the coordinates of the white pixel in the image found and hence you will have the coordinate of point of intersection.
Obviously this is not the most efficient way but it is real time. Also, keep this in mind that the outer circle width should be atleast 2 in the image and the lines should be of width 1. You may get more than one point of intersection in some cases, take any 1 of them. The difference in them will be only of 1-2 pixels that can be neglected.

Calculating center of an object in an image

I was reading this post to calculate the center of an image using OpenCV which uses Moments. But I am trying to calculate the center of an object I detected using HoughLinesP. Is there a way with OpenCV I could do this?
Here is the image for which I am trying to calculate the centers.
The line segments were found and the output image looks like:
import cv2
import numpy as np
import math
img = cv2.imread("./images/octa.jpg")
b,g,r = cv2.split(img)
smoothed = cv2.GaussianBlur(g, (3,3), 0)
edges = cv2.Canny(smoothed, 15, 60, apertureSize = 3)
lines = cv2.HoughLinesP(edges,1,np.pi/180,35, 30, 20)
print("length of lines detected ", lines.shape)
for line in lines:
for x1,y1,x2,y2 in line:
cv2.line(img,(x1,y1),(x2,y2),(255,0,0),2)
print("x1,y1", x1,",",y1, " --- ", "x2,y2", x2,",",y2)
cv2.imshow('detected',img)
cv2.waitKey(0)
cv2.destroyAllWindows()
Using the coordinates how could I calculate the center of this image? How could I use Moments here?
One constraint I have is that I cannot use Contour methods included with OpenCV.

The following code was used with cv2 version of 3.3.1.
I closely followed the opencv docs and it worked fine.
import cv2
img = cv2.imread("octa.jpg", 0)
ret,thresh = cv2.threshold(img,100,255,0)
im2, contours, hierachy = cv2.findContours(thresh, 1, 2)
cnt = contours[0]
M = cv2.moments(cnt)
cx = int(M['m10']/M['m00'])
cy = int(M['m01']/M['m00'])
im2 = cv2.cvtColor(im2, cv2.COLOR_GRAY2RGB)
cv2.polylines(im2, cnt, True, (0, 0, 255), 2)
cv2.circle(im2, (cx, cy), 5, (0, 0, 255), 1)
cv2.imshow("res", im2)
Two notes:
you need to add the argument 0 to imread otherwise the contour finding would not work
I set the threshold just a little bit lower, so only the contours of the octagon were found
Result:
If you use a different version of cv2, you can just change the docs to your version; the documentation is really good.
You also may want to blur your image a bit or do some other preprocessing, but in this case, there was no need for it.
EDIT Without contour:
I took the helpful comments from this post and tinkered around a bit. This does not use contours. It finds lines and uses them to find the center
import cv2
import numpy as np
mg = cv2.imread('octa.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
kernel_size = 5
blur_gray = cv2.GaussianBlur(gray,(kernel_size, kernel_size),0)
ret,thresh = cv2.threshold(blur_gray,100,255,0)
low_threshold = 50
high_threshold = 150
edges = cv2.Canny(thresh, low_threshold, high_threshold)
rho = 1 # distance resolution in pixels of the Hough grid
theta = np.pi / 180 # angular resolution in radians of the Hough grid
threshold = 15 # minimum number of votes (intersections in Hough grid cell)
min_line_length = 50 # minimum number of pixels making up a line
max_line_gap = 50 # maximum gap in pixels between connectable line segments
line_image = np.copy(img) * 0 # creating a blank to draw lines on
# Run Hough on edge detected image
# Output "lines" is an array containing endpoints of detected line segments
lines = cv2.HoughLinesP(edges, rho, theta, threshold, np.array([]),
min_line_length, max_line_gap)
for line in lines:
for x1,y1,x2,y2 in line:
cv2.line(line_image,(x1,y1),(x2,y2),(255,0,0),2)
lines_edges = cv2.addWeighted(img, 0.5, line_image, 1, 0)
line_image_gray = cv2.cvtColor(line_image, cv2.COLOR_RGB2GRAY)
M = cv2.moments(line_image_gray)
cx = int(M['m10']/M['m00'])
cy = int(M['m01']/M['m00'])
cv2.circle(lines_edges, (cx, cy), 5, (0, 0, 255), 1)
cv2.imshow("res", lines_edges)
Result:
Found lines are drawn in blue; the center in red

Advanced square detection (with connected region)

if the squares has connected region in image, how can I detect them.
I have tested the method mentioned in
OpenCV C++/Obj-C: Advanced square detection
It did not work well.
Any good ideas ?
import cv2
import numpy as np
def angle_cos(p0, p1, p2):
d1, d2 = (p0-p1).astype('float'), (p2-p1).astype('float')
return abs( np.dot(d1, d2) / np.sqrt( np.dot(d1, d1)*np.dot(d2, d2) ) )
def find_squares(img):
squares = []
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# cv2.imshow("gray", gray)
gaussian = cv2.GaussianBlur(gray, (5, 5), 0)
temp,bin = cv2.threshold(gaussian, 80, 255, cv2.THRESH_BINARY)
# cv2.imshow("bin", bin)
contours, hierarchy = cv2.findContours(bin, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
cv2.drawContours( gray, contours, -1, (0, 255, 0), 3 )
#cv2.imshow('contours', gray)
for cnt in contours:
cnt_len = cv2.arcLength(cnt, True)
cnt = cv2.approxPolyDP(cnt, 0.02*cnt_len, True)
if len(cnt) == 4 and cv2.contourArea(cnt) > 1000 and cv2.isContourConvex(cnt):
cnt = cnt.reshape(-1, 2)
max_cos = np.max([angle_cos( cnt[i], cnt[(i+1) % 4], cnt[(i+2) % 4] ) for i in xrange(4)])
if max_cos < 0.1:
squares.append(cnt)
return squares
if __name__ == '__main__':
img = cv2.imread('123.bmp')
#cv2.imshow("origin", img)
squares = find_squares(img)
print "Find %d squres" % len(squares)
cv2.drawContours( img, squares, -1, (0, 255, 0), 3 )
cv2.imshow('squares', img)
cv2.waitKey()
I use some method in the opencv example, but the result is not good.

Applying a Watershed Transform based on the Distance Transform will separate the objects:
Handling objects at the border is always problematic, and often discarded, so that pink rectangle at top left not separated is not a problem at all.
Given a binary image, we can apply the Distance Transform (DT) and from it obtain markers for the Watershed. Ideally there would be a ready function for finding regional minima/maxima, but since it isn't there, we can make a decent guess on how we can threshold DT. Based on the markers we can segment using Watershed, and the problem is solved. Now you can worry about distinguishing components that are rectangles from those that are not.
import sys
import cv2
import numpy
import random
from scipy.ndimage import label
def segment_on_dt(img):
dt = cv2.distanceTransform(img, 2, 3) # L2 norm, 3x3 mask
dt = ((dt - dt.min()) / (dt.max() - dt.min()) * 255).astype(numpy.uint8)
dt = cv2.threshold(dt, 100, 255, cv2.THRESH_BINARY)[1]
lbl, ncc = label(dt)
lbl[img == 0] = lbl.max() + 1
lbl = lbl.astype(numpy.int32)
cv2.watershed(cv2.cvtColor(img, cv2.COLOR_GRAY2BGR), lbl)
lbl[lbl == -1] = 0
return lbl
img = cv2.cvtColor(cv2.imread(sys.argv[1]), cv2.COLOR_BGR2GRAY)
img = cv2.threshold(img, 0, 255, cv2.THRESH_OTSU)[1]
img = 255 - img # White: objects; Black: background
ws_result = segment_on_dt(img)
# Colorize
height, width = ws_result.shape
ws_color = numpy.zeros((height, width, 3), dtype=numpy.uint8)
lbl, ncc = label(ws_result)
for l in xrange(1, ncc + 1):
a, b = numpy.nonzero(lbl == l)
if img[a[0], b[0]] == 0: # Do not color background.
continue
rgb = [random.randint(0, 255) for _ in xrange(3)]
ws_color[lbl == l] = tuple(rgb)
cv2.imwrite(sys.argv[2], ws_color)
From the above image you can consider fitting ellipses in each component to determine rectangles. Then you can use some measurement to define whether the component is a rectangle or not. This approach has a greater chance to work for rectangles that are fully visible, and will likely produce bad results for partially visible ones. The following image shows the result of such approach considering that a component is a rectangle if the rectangle from the fitted ellipse is within 10% of component's area.
# Fit ellipse to determine the rectangles.
wsbin = numpy.zeros((height, width), dtype=numpy.uint8)
wsbin[cv2.cvtColor(ws_color, cv2.COLOR_BGR2GRAY) != 0] = 255
ws_bincolor = cv2.cvtColor(255 - wsbin, cv2.COLOR_GRAY2BGR)
lbl, ncc = label(wsbin)
for l in xrange(1, ncc + 1):
yx = numpy.dstack(numpy.nonzero(lbl == l)).astype(numpy.int64)
xy = numpy.roll(numpy.swapaxes(yx, 0, 1), 1, 2)
if len(xy) < 100: # Too small.
continue
ellipse = cv2.fitEllipse(xy)
center, axes, angle = ellipse
rect_area = axes[0] * axes[1]
if 0.9 < rect_area / float(len(xy)) < 1.1:
rect = numpy.round(numpy.float64(
cv2.cv.BoxPoints(ellipse))).astype(numpy.int64)
color = [random.randint(60, 255) for _ in xrange(3)]
cv2.drawContours(ws_bincolor, [rect], 0, color, 2)
cv2.imwrite(sys.argv[3], ws_bincolor)

Solution 1:
Dilate your image to delete connected components.
Find contours of detected components. Eliminate contours which are not rectangles by introducing some measure (ex. ratio perimeter / area).
This solution will not detect rectangles connected to borders.
Solution 2:
Dilate to delete connected components.
Find contours.
Approximate contours to reduce their points (for rectangle contour should be 4 points).
Check if angle between contour lines is 90 degrees.
Eliminate contours which have no 90 degrees.
This should solve problem with rectangles connected to borders.

You have three problems:
The rectangles are not very strict rectangles (the edges are often somewhat curved)
There are a lot of them.
They are often connected.
It seems that all your rects are essentially the same size(?), and do not greatly overlap, but the pre-processing has connected them.
For this situation the approach I would try is:
dilate your image a few times (as also suggested by #krzych) - this will remove the connections, but result in slightly smaller rects.
Use scipy to label and find_objects - You now know the position and slice for every remaining blob in the image.
Use minAreaRect to find the center, orientation, width and height of each rectangle.
You can use step 3. to test whether the blob is a valid rectangle or not, by its area, dimension ratio or proximity to the edge..
This is quite a nice approach, as we assume each blob is a rectangle, so minAreaRect will find the parameters for our minimum enclosing rectangle. Further we could test each blob using something like humoments if absolutely neccessary.
Here is what I was suggesting in action, boundary collision matches shown in red.
Code:
import numpy as np
import cv2
from cv2 import cv
import scipy
from scipy import ndimage
im_col = cv2.imread('jdjAf.jpg')
im = cv2.imread('jdjAf.jpg',cv2.CV_LOAD_IMAGE_GRAYSCALE)
im = np.where(im>100,0,255).astype(np.uint8)
im = cv2.erode(im, None,iterations=8)
im_label, num = ndimage.label(im)
for label in xrange(1, num+1):
points = np.array(np.where(im_label==label)[::-1]).T.reshape(-1,1,2).copy()
rect = cv2.minAreaRect(points)
lines = np.array(cv2.cv.BoxPoints(rect)).astype(np.int)
if any([np.any(lines[:,0]<=0), np.any(lines[:,0]>=im.shape[1]-1), np.any(lines[:,1]<=0), np.any(lines[:,1]>=im.shape[0]-1)]):
cv2.drawContours(im_col,[lines],0,(0,0,255),1)
else:
cv2.drawContours(im_col,[lines],0,(255,0,0),1)
cv2.imshow('im',im_col)
cv2.imwrite('rects.png',im_col)
cv2.waitKey()
I think the Watershed and distanceTransform approach demonstrated by #mmgp is clearly superior for segmenting the image, but this simple approach can be effective depending upon your needs.