Develop Reference

Detecting Grid blocks using Computer vision in realtime (Open CV, Python)

I want to detect the centroid of individual blocks in the following grids for path planning. The idea is that a central navigation system like overhead camera will detect the blocks of grids along with the bot and help in navigation. Till now I have tried Hough lines Probabilistic and Harris corner detection but both of them either detect extra points or fail in real world scenario. I want to detect the blocks in real time and number them. Those numbering should not change or the whole path planning will be messed up.
Is there any solution to this problem that I missed.
thanks in advance

You need to learn how to eliminate noise. This is not a complete answer. The more time you spend and learn, the better your results will be.
import cv2
import numpy as np
import sys
# Load source as grayscale
im = cv2.imread(sys.path[0]+'/im.jpg', cv2.IMREAD_GRAYSCALE)
H, W = im.shape[:2]
# Convert im to black and white
im = cv2.adaptiveThreshold(
im, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 21, 2)
# Remove noise
im = cv2.medianBlur(im, 11)
im = cv2.erode(im, np.ones((15, 15)))
# Fill the area around the shape
im = ~im
mask = np.zeros((H+2, W+2), np.uint8)
cv2.floodFill(im, mask, (0, 0), 255)
cv2.floodFill(im, mask, (W-1, 0), 255)
cv2.floodFill(im, mask, (0, H-1), 255)
cv2.floodFill(im, mask, (W-1, H-1), 255)
# Remove noise again
im = cv2.dilate(im, np.ones((15, 15)))
# Find the final blocks
cnts, _ = cv2.findContours(~im, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
for c in cnts:
x, y, w, h = cv2.boundingRect(c)
cv2.circle(im, (x+w//2, y+h//2), max(w, h)//2, 127, 5)
print("Found any: ", len(cnts) > 0)
# Save the output
cv2.imwrite(sys.path[0]+'/im_.jpg', im)

Get corners of rectangle from black and white image

Let's say that we have the following black and white image (image.png):
We load the image using OpenCV2 with the following code:
import cv2
img = cv2.imread('image.png')
How can we detect the corners of all white rectangles in an image? We can assume that all rectangles are parallel to the corners of the image.
Result in this case should be in the following form (res[0] is left rectangle and res[1] is right rectangle):
res = [
[
(20, 30), # upper left
(40, 30), # upper right
(20, 80), # bottom left
(40, 80) # bottom right
],
[
(100, 20), # upper left
(140, 20), # upper right
(100, 70), # bottom left
(140, 70) # bottom right
]
]

There are a few possibilities:
the "Harris Corner Detector" is good at finding corners - see here
you can use OpenCV's findContours()
you could use "Hit-or-Miss" morphology to look for corners
you could convolve the image with a kernel and look for specific outputs
So, looking at the last option, if we slide a 2x2 pixel kernel over the image and multiply each of the elements underneath the kernel by 1 and add them all together, and then find all the pixels where that total comes to 255, that will be a 2x2 square where exactly one pixel is white - and that is a corner:
import cv2
import numpy as np
# Load image as greyscale
im = cv2.imread('tZHHE.png',cv2.IMREAD_GRAYSCALE)
# Make a 2x2 kernel of ones
kernel = np.ones((2,2), dtype=np.uint8)
# Convolve the image with the kernel
res = cv2.filter2D(im.astype(np.int16), -1, kernel)
# Get coordinates of points where the sum of the 2x2 window comes to 255
corners = np.where(res==255)
Sample Output
(array([101, 101, 118, 118, 166, 166, 174, 174]),
array([274, 307, 117, 134, 274, 307, 117, 134]))
Looking at the "Hit-or-Miss" morphology method, I will do it with ImageMagick straight in the Terminal, but you can equally do it with other Python libraries:
magick tZHHE.png -alpha off -morphology HMT Corners result.png
As always, I am indebted to Anthony Thyssen for his ImageMagick examples as linked above. We are looking for these specific shapes with the "Hit-or-Miss" morphology:
Keywords: Python, OpenCV, image processing, convolution, corner detect, corner detector, corner detection, ImageMagick Hit-or-Miss morphology.

This works for any number of rectangles:
import cv2 as cv
import pprint as pprint
img = cv.imread("tZHHE.png") # read image
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # make grayscale image
cv.imshow("Our initial image",img) # show our original image
corners = cv.goodFeaturesToTrack(gray,2000,0.01,5) # find our corners, 2000 is the number of corners we can detect, 5 is the distance between corners
xylist = [] #put all of our xy coords in here
for corn in corners: # extract our corners and put them in xylist
x,y = corn[0]
xylist.append((x,y))
x = int(x)
y = int(y)
cv.rectangle(img, (x-2,y-2), (x+2,y+2), (100,100,0),-1) # now mark where our corners are on our original image
res = [[] for i in range(int(len(xylist)/4))] # generate i nested lists for our rectangles
for index, item in enumerate(xylist): # format coordinates as you want them
res[index % int(len(xylist)/4)].append(item)
print("\n"+"found ",int(len(xylist)/4) ,"rectangles\n") # how many rectangles did we have?
print(res)
cv.imshow("Corners?", img) # show our new image with rectangle corners marked

Try findContours()
I suggest you to try findContours() with its companion boundingRect().
Here is how you can make it work.
Load the image in grayscale, then pass it to the function findContours().
img = cv2.imread('tZHHe.png', cv2.IMREAD_GRAYSCALE)
contours, hierarchy = cv2.findContours(img, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
Getting the bounding box from contours, it returns x, y coordinates of the top left corner and w, h the width and height of the box:
[cv2.boundingRect(contour) for contour in contours]
#=> [(117, 118, 17, 56), (274, 101, 33, 65)]
View in action
Try this maybe in a Jupyter Notebook to see a kind of animation:
def corners_from_bounding_rect(bounding_rect):
x, y, w, h = bounding_rect
points = {'top_left': (x, y), 'top_right':(x+w, y), 'bottom-left': (x, y+h), 'bottom-rigth':(x+w, y+h)}
return points
Retrieve the points from contours using the method defined:
corner_groups = [corners_from_bounding_rect(cv2.boundingRect(cnt)) for cnt in contours]
# [{'top_left': (117, 118),
# 'top_right': (134, 118),
# 'bottom-left': (117, 174),
# 'bottom-rigth': (134, 174)},
# {'top_left': (274, 101),
# 'top_right': (307, 101),
# 'bottom-left': (274, 166),
# 'bottom-rigth': (307, 166)}]
Then plot the sequence:
pinned_img = img.copy()
for n, corners in enumerate(corner_groups):
for name, point in corners.items():
cv2.circle(pinned_img, point, 10, 255)
plt.title(f'{n}-{name}')
plt.imshow(pinned_img)
plt.show()
The first image from the squence:

Try this code
import cv2
img=cv2.imread('tZHHE.png') # Read my Image
imgContours=img.copy() # Copy my Image for Contours
imgCanny=cv2.Canny(imgContours,10,50) # Image to Edges
contours,hierarchy =cv2.findContours(imgCanny,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)
rectangles=[]
for cont in reversed(contours):
area=cv2.contourArea(cont)
x, y, w, h = cv2.boundingRect(cont)
rectangles.append([[x,y],[x+w,y],[x+w,y+h],[x,y+h]])
cv2.imshow("con", imgContours)
cv2.waitKey(0)
print(rectangles)
cv2.destroyAllWindows()
left to right rectangles
Output:
[[[273, 100], [307, 100], [307, 166], [273, 166]], [[116, 117], [134, 117], [134, 174], [116, 174]]]

Automatic point detection

I have a depth image (in blue) and of it, I generate an outline image, where I want to take some points automatically (highlighted in red).
I already have the points (x,y) of the outline, but I would like to pick up the points marked in red automatically.
For this task, I am using python 3.6 (OpenCV ...) and I have a 435i realsense camera that gives me the depth image.
Would it be possible to get an approximate value of these coordinates automatically?
Image: https://i.stack.imgur.com/qiTvh.png
a small example of the code:
# function tha captures the edges
def get_edges(frame):
image = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
image = cv2.GaussianBlur(image, (5, 5), 0)
_, image = cv2.threshold(image, 100, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
image = cv2.bitwise_not(image)
element = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3, 3))
image = cv2.erode(image, element)
image = cv2.dilate(image, element)
image = cv2.Canny(image, 100, 200, L2gradient=True)
return image
# ... read the frames, process the image ...
image_process = get_edges(depth_frame)
# taking the contour indices
indices = np.where(image_process == 255)
# get coordiniates
coordinates = zip(indices[1], indices[0])
My variable "coordinates" has all the points (x,y) of the outline, but I would like to automatically set the points in red highlighted in the image.

I managed to solve using Pose Estimation
Final image
Where I have the return of the points and coordinates:
points = {'A': (67, 121), 'B': (66, 21), 'C': (165, 38), 'D': (133, 143)}

Can I search for black pixels in a photo and find out where they are in the photo using python?

I want to search for black pixels in a screenshot I took using pyautogui and I want to find the x and y location of those pixels using python so that I can move the mouse to the black pixels locations using pynput. I tried using imageio but I could not find a command that would do what I want. I asked this a few hours Ago but it was closed so I made the necessary edits to it.

Here is one way in Python/OpenCV/Numpy using np.argwhere on a thresholded image that isolates the black spots.
Read the input
Threshold using inRange on black and invert
Use np.argwhere to located the coordinates of the black pixels in the mask
Print the results
Input (4 black clusters near the 4 corners):
import cv2
import numpy as np
# read input
img = cv2.imread("lena_black_spots.png")
low = (0,0,0)
high = (0,0,0)
mask = cv2.inRange(img, low, high)
mask = 255 - mask
# find black coordinates
coords = np.argwhere(mask==0)
for p in coords:
pt = (p[0],p[1])
print (pt)
# save output
cv2.imwrite('lena_black_spots_mask.png', mask)
cv2.imshow('img', img)
cv2.imshow('mask', mask)
cv2.waitKey(0)
cv2.destroyAllWindows()
Mask:
Coordinates:
(18, 218)
(18, 219)
(19, 218)
(19, 219)
(20, 218)
(20, 219)
(38, 21)
(38, 22)
(39, 21)
(39, 22)
(173, 244)
(173, 245)
(174, 244)
(174, 245)
(194, 23)
(194, 24)
(195, 23)
(195, 24)

I'd recommend looking into the ImageMagick library. That's the go-to library for working with images in Python.

How to identify contours associated with my objects and find their geometric centroid

Problem Statement & Background Info:
EDIT: Constraints: The red coloring on the flange changes over time, so I'm not trying to use color recognition to identify my object at this moment unless it can be robust. Addition,ally external illumination my be a factor since this will be in an outdoor area in the future.
I have RGB-Depth camera and with it, I'm able to capture this scene. Where each pixel (x,y) has a depth value.
Applying a gradient magnitude filter to the depth map associated with my image I'm able to get the following edge map.
The gradient magnitudes are given the value of 0 if they had a magnitude that wasn't zero. Black (255) is for magnitude values associated with 0 (homogenous depth or a flat surface).
From this edge map I dialated the edges so picking up the contours would be easier.
Then I found the contours in the image and tried to plot just the 5 biggest contours.
PROBLEM
Is there a way to reliably find the contours associated with my objects (the red box and metal fixture) and then find their geometric centroid? I keep running into the issue that I can find contours in the image, but I have no way of selectively screening for the contours that are my objects and not noise.
I have provided the image I used for the image processing, but for some reason, OpenCV saves the image as a black image, and when you read it in using...
gray = cv2.imread('GRAYTEST.jpeg', cv2.IMREAD_GRAYSCALE)
it appears blue-ish and not a binary white/black image as I show. So sorry about that.
Here is the image:
Sorry, I don't know why it saved just as a black image, but if you read it in OpenCV it should show up with the same lines as "magnitude of gradients" plot.
MY CODE
gray = cv2.imread('GRAYTEST.jpeg', cv2.IMREAD_GRAYSCALE)
plt.imshow(gray)
plt.title('gray start image')
plt.show()
blurred = cv2.bilateralFilter(gray, 8, 25, 25) # blurr image while preserving edges
kernel = np.ones((3, 3), np.uint8) # define a kernel (block) to apply filters to
dialated = cv2.dilate(blurred, kernel, iterations=1)
plt.title('dialated')
plt.imshow(dialated)
plt.show()
#Just performs a canny edge dectection on an image
edges_empty = self.Commons.CannyE_Auto(dialated) # Canny edge image for some sigma
#makes an empty image using the same diemensions of the given image
empty2 = self.Commons.make_empty(gray)
_, contours, _ = cv2.findContours(edges_empty, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
cnts = sorted(contours, key=cv2.contourArea, reverse=True)[:5] # get largest five contour area
cv2.drawContours(empty2, cnts, -1, (255, 0, 0), thickness=1)
plt.title('contours')
plt.imshow(empty2)
plt.show()

Instead of performing the blurring operation, dialation, canny edge detection on my already thresholded image, I just performed the contour detection on my original image.
Then I was able to find a decent contour for the outline of my image by modifying findContour command.
_, contours, _ = cv2.findContours(gray, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
replacing cv2.RETR_TREE with cv2.RETR_EXTERNAL I was able to get only the contours that were associated with an object's outline, rather than trying to get contours within the object. Switching to cv2.CHAIN_APPROX_NONE didn't show any noticeable improvements, but it may provide better contours for more complex geometries.
for c in cnts:
# compute the center of the contour
M = cv2.moments(c)
cX = int(M["m10"] / M["m00"])
cY = int(M["m01"] / M["m00"])
# draw the contour and center of the shape on the image
cv2.drawContours(empty2, [c], -1, (255, 0, 0), thickness=1)
perimeter = np.around(cv2.arcLength(c, True), decimals=3)
area = np.around(cv2.contourArea(c), decimals=3)
cv2.circle(empty2, (cX, cY), 7, (255, 255, 255), -1)
cv2.putText(empty2, "center", (cX - 20, cY - 20),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2)
cv2.putText(empty2, "P:{}".format(perimeter), (cX - 50, cY - 50),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2)
cv2.putText(empty2, "A:{}".format(area), (cX - 100, cY - 100),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2)
Using the above code I was able to label centroid of each contour as well as information about each contours perimeter and area.
However, I was unable to perform a test that would select which contour was my desired contour. I have an idea to capture my object in a more ideal setting and find its centroid, perimeter, and associated area. This way when I find a new contour I can compare it with how close it is to my known values.
I think this method could work to remove contours that are too large or too small.
If anyone knows of a better solution that would be fantastic!