Develop Reference

Using image colors to draw boundaries on image

I am trying to use OpenCV to take an RGB image and identify a boundary or draw a line where the sidewalk meets the grass. Typical methods like canny edge detection and then using hough lines is not extremely helpful since it is easily influenced by other potential lines in the environment.
Let's say I have the RGB image below RGB sidewalk image, in the image, there is a clear boundary where the sidewalk meets the grass. This boundary becomes even more prominent when you convert into the HSV space and blur the image as shown in HSV sidewalk image. I believe color segmentation is the best bet I am just not sure how to approach it. Using the code hsv_img = cv2.cvtColor(blur, cv2.COLOR_BGR2HSV) green_low = np.array([25 , 0, 50] ) green_high = np.array([75, 255, 255]) curr_mask = cv2.inRange(hsv_img, green_low, green_high)
I was able to generate a mask that almost gets me to where I want as shown in this figure grass mask. I just need to use this mask to draw my line without getting mixed up with the other greens detected in the picture.

Improving background extraction with shiny objects (OpenCV/Python)

I've been working on a 'fun' solution that combines multiple videos from a security camera into one video, the idea being to overlay foreground motion over many hours into a few seconds for a 'quick preview'. Thanks to Stephen and Bahramdum for getting me on the right path.
The project is open source, for anyone to see. So far, I've played with the following background extractions:
OpenCV BackgroundSubtraction using a variety of algorithms
Mask R-CNN
Yolov3/TinyYolo
Optical flow
(I haven't yet tried detection+centroid tracking, will do so next)
Based on my experiments so far, OpenCV's background extraction generally works the best due to the fact that it extracts foreground purely based on motion. Plus its very fast. True, that it also extracts things like moving leaves etc, but we can work on removing those.
Here is an example of 3 hours of video blended into one short video.
https://youtu.be/C-mJfzvFTdg
My current challenge is that it is doing a bad job with shiny objects, like cars.
Here is an example:
Background subtraction consistently does a bad job with extracting polygons for shiny objects and findContours does no better either.
I've tried several options, but my current approach is documented here, the gist of which is:
Convert frame to HSV
remove intensity (I read this in another SO thread for shiny objects)
Apply background subtraction
Clean up outside noise with MORPH_OPEN
Blur mask to hopefully connect near while blobs
find contours on new masks
only keep contours of certain min area
create a new mask, where we draw only these contours with fill
Do a final dilation to connect close filled contours of new masks
10.Use this new mask to extract foreground from the frame and overlay it with current blended video
Would anyone have suggestions on how to improve extraction for reflective objects?
self.fgbg = cv2.createBackgroundSubtractorMOG2(detectShadows=False,
history=self.history)
frame_hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
frame_hsv[:,:,0] = 0 # see if removing intensity helps
# gray = cv2.cvtColor(frame_hsv, cv2.COLOR_BGR2GRAY)
# create initial background subtraction
frame_mask = self.fgbg.apply(frame_hsv)
# remove noise
frame_mask = cv2.morphologyEx(frame_mask, cv2.MORPH_OPEN, self.kernel_clean)
# blur to merge nearby masks, hopefully
frame_mask = cv2.medianBlur(frame_mask,15)
#frame_mask = cv2.GaussianBlur(frame_mask,(5,5),cv2.BORDER_DEFAULT)
#frame_mask = cv2.blur(frame_mask,(20,20))
h,w,_ = frame.shape
new_frame_mask = np.zeros((h,w),dtype=np.uint8)
copy_frame_mask = frame_mask.copy()
# find contours of mask
relevant = False
ctrs,_ = cv2.findContours(copy_frame_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
rects = []
# only select relevant contours
for contour in ctrs:
area = cv2.contourArea(contour)
if area >= self.min_blend_area:
x,y,w,h = cv2.boundingRect(contour)
pts = Polygon([[x,y], [x+w,y], [x+w, y+h], [x,y+h]])
if g.poly_mask is None or g.poly_mask.intersects(pts):
relevant = True
cv2.drawContours(new_frame_mask, [contour], -1, (255, 255, 255), -1)
rects.append([x,y,w,h])
# do a dilation to again, combine the contours
frame_mask = cv2.dilate(new_frame_mask,self.kernel_fill,iterations = 5)
frame_mask = new_frame_mask

I found way too many variations on different conditions to have something predictable using openCV's background extraction
So I switched to yolov3 for object identification (added it as a new option, actually). While TinyYOLO was pretty terrible, YoloV3 seems to be adequate, albeit much slower.
Unfortunately, given YoloV3 only does rectangle and not subject masks, I had to switch to addWeighted method of cv2 to blend a new object on top of the blended frame.
Example: https://github.com/pliablepixels/zmMagik/blob/master/sample/1.gif

Image Processing: Algorithm Improvement for Real-Time FedEx Logo Detector

I've been working on a project involving image processing for logo detection. Specifically, the goal is to develop an automated system for a real-time FedEx truck/logo detector that reads frames from a IP camera stream and sends a notification on detection. Here's a sample of the system in action with the recognized logo surrounded in the green rectangle.
Some constraints on the project:
Uses raw OpenCV (no deep learning, AI, or trained neural networks)
Image background can be noisy
The brightness of the image can vary greatly (morning, afternoon, night)
The FedEx truck/logo can have any scale, rotation, or orientation since it could be parked anywhere on the sidewalk
The logo could potentially be fuzzy or blurry with different shades depending on the time of day
There may be many other vehicles with similar sizes or colors in the same frame
Real-time detection (~25 FPS from IP camera)
The IP camera is in a fixed position and the FedEx truck will always be in the same orientation (never backwards or upside down)
The Fedex Truck will always be the "red" variation instead of the "green" variation
Current Implementation/Algorithm
I have two threads:
Thread #1 - Captures frames from the IP camera using cv2.VideoCapture() and resizes frame for further processing. Decided to handle grabbing frames in a separate thread to improve FPS by reducing I/O latency since cv2.VideoCapture() is blocking. By dedicating an independent thread just for capturing frames, this would allow the main processing thread to always have a frame available to perform detection on.
Thread #2 - Main processing/detection thread to detect FedEx logo using color thresholding and contour detection.
Overall Pseudo-algorithm
For each frame:
Find bounding box for purple color of logo
Find bounding box for red/orange color of logo
If both bounding boxes are valid/adjacent and contours pass checks:
Combine bounding boxes
Draw combined bounding boxes on original frame
Play sound notification for detected logo
Color thresholding for logo detection
For color thresholding, I have defined HSV (low, high) thresholds for purple and red to detect the logo.
colors = {
'purple': ([120,45,45], [150,255,255]),
'red': ([0,130,0], [15,255,255])
}
To find the bounding box coordinates for each color, I follow this algorithm:
Blur the frame
Erode and dilate the frame with a kernel to remove background noise
Convert frame from BGR to HSV color format
Perform a mask on the frame using the lower and upper HSV color bounds with set color thresholds
Find largest contour in the mask and obtain bounding coordinates
After performing a mask, I obtain these isolated purple (left) and red (right) sections of the logo.
False positive checks
Now that I have the two masks, I perform checks to ensure that the found bounding boxes actually form a logo. To do this, I use cv2.matchShapes() which compares the two contours and returns a metric showing the similarity. The lower the result, the higher the match. In addition, I use cv2.pointPolygonTest() which finds the shortest distance between a point in the image and a contour for additional verification. My false positive process involves:
Checking if the bounding boxes are valid
Ensuring the two bounding boxes are adjacent based on their relative proximity
If the bounding boxes pass the adjacency and similarity metric test, the bounding boxes are combined and a FedEx notification is triggered.
Results
This check algorithm is not really robust as there are many false positives and failed detections. For instance, these false positives were triggered.
While this color thresholding and contour detection approach worked in basic cases where the logo was clear, it was severely lacking in some areas:
There is latency problems from having to compute bounding boxes on each frame
It occasionally false detects when the logo is not present
Brightness and time of day had a great impact on detection accuracy
When the logo was on a skewed angle, color threshold detection worked but was unable to detect the logo due to the check algorithm.
Would anyone be able to help me improve my algorithm or suggest alternative detection strategies? Is there any other way to perform this detection since color thresholding is highly dependent on exact calibration? If possible, I would like to move away from color thresholding and the multiple layers of filters since it's not very robust. Any insight or advice is greatly appreciated!

You might want to take a look at feature matching. The goal is to find features in two images, a template image, and a noisy image and match them. This would allow you to find the template (the logo) in the noisy image (the camera image).
A feature is, in essence, things that humans would find interesting in an image, such as corners or open spaces. I would recommend using a scale-invariant feature transform (SIFT) as a feature detection algorithm. The reason I suggest using SIFT is that it is invariant to image translation, scaling, and rotation, partially invariant to illumination changes and robust to local geometric distortion. This matches your specification.
I generated the above image using code modified from the OpenCV docs docs on SIFT feature detection:
import numpy as np
import cv2
from matplotlib import pyplot as plt
img = cv2.imread('main.jpg',0) # target Image
# Create the sift object
sift = cv2.xfeatures2d.SIFT_create(700)
# Find keypoints and descriptors directly
kp, des = sift.detectAndCompute(img, None)
# Add the keypoints to the final image
img2 = cv2.drawKeypoints(img, kp, None, (255, 0, 0), 4)
# Show the image
plt.imshow(img2)
plt.show()
You will notice when doing this that a large number of the features do land on the FedEx logo (Above).
The next thing I did was try matching the features from the video feed to the features in the FedEx logo. I did this using the FLANN feature matcher. You could have gone with many approaches (including brute force) but because you are working on a video feed this is probably your best option. The code below is inspired from the OpenCV docs on feature matching:
import numpy as np
import cv2
from matplotlib import pyplot as plt
logo = cv2.imread('logo.jpg', 0) # query Image
img = cv2.imread('main2.jpg',0) # target Image
# Create the sift object
sift = cv2.xfeatures2d.SIFT_create(700)
# Find keypoints and descriptors directly
kp1, des1 = sift.detectAndCompute(img, None)
kp2, des2 = sift.detectAndCompute(logo,None)
# FLANN parameters
FLANN_INDEX_KDTREE = 1
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks=50) # or pass empty dictionary
flann = cv2.FlannBasedMatcher(index_params,search_params)
matches = flann.knnMatch(des1,des2,k=2)
# Need to draw only good matches, so create a mask
matchesMask = [[0,0] for i in range(len(matches))]
# ratio test as per Lowe's paper
for i,(m,n) in enumerate(matches):
if m.distance < 0.7*n.distance:
matchesMask[i]=[1,0]
# Draw lines
draw_params = dict(matchColor = (0,255,0),
singlePointColor = (255,0,0),
matchesMask = matchesMask,
flags = 0)
# Display the matches
img3 = cv2.drawMatchesKnn(img,kp1,logo,kp2,matches,None,**draw_params)
plt.imshow(img3, )
plt.show()
Using this I was able to get the following features matched as seen below. You will notice that there are outliers. However the majority of features match:
The final step would then to be to simply draw a bounding box around this image. I will link you to another stack overflow question which does something similar but with the orb detector. Here is another way to get a bounding box using the OpenCV docs.
I hope this helps!

You can help the detector with preprocessing the image, then you don't need as many training images.
First we reduce the barrel distortion.
import cv2
img = cv2.imread('fedex.jpg')
margin = 150
# add border as the undistorted image is going to be larger
img = cv2.copyMakeBorder(
img,
margin,
margin,
margin,
margin,
cv2.BORDER_CONSTANT,
0)
import numpy as np
width = img.shape[1]
height = img.shape[0]
distCoeff = np.zeros((4,1), np.float64)
k1 = -4.5e-5;
k2 = 0.0;
p1 = 0.0;
p2 = 0.0;
distCoeff[0,0] = k1;
distCoeff[1,0] = k2;
distCoeff[2,0] = p1;
distCoeff[3,0] = p2;
cam = np.eye(3, dtype=np.float32)
cam[0,2] = width/2.0 # define center x
cam[1,2] = height/2.0 # define center y
cam[0,0] = 12. # define focal length x
cam[1,1] = 12. # define focal length y
dst = cv2.undistort(img, cam, distCoeff)
Then we transform the image in a way as if the camera is facing the FedEx truck right on. That is wherever along the curb the truck is parked, the FedEx logo will have almost the same size and orientation.
# use four points for homography estimation, coordinated taken from undistorted image
# 1. top-left corner of F
# 2. bottom-left corner of F
# 3. top-right of E
# 4. bottom-right of E
pts_src = np.array([[1083, 235], [1069, 343], [1238, 301],[1201, 454]])
pts_dst = np.array([[1069, 235],[1069, 320],[1201, 235],[1201, 320]])
h, status = cv2.findHomography(pts_src, pts_dst)
im_out = cv2.warpPerspective(dst, h, (dst.shape[1], dst.shape[0]))

OpenCV Detect scratches on fruits

For a little experiment in Python I'm doing I want to find small scratches on fruits. The scratches are very small and hard to detect by human eye.
I'm using a high resolution camera for that experiment.
Here is the defect I want to detect:
Original Image:
This is my result with very few lines of code:
So I found the contours of my fruit. How can I proceed to finding the scratch? The RGB Value is similar to other parts of the fruit. So how can I differentiate between A scratch, and a part of the fruit?
My code:
# Imports
import numpy as np
import cv2
import time
# Read Image & Convert
img = cv2.imread('IMG_0441.jpg')
result = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
# Filtering
lower = np.array([1,60,50])
upper = np.array([255,255,255])
result = cv2.inRange(result, lower, upper)
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(9,9))
result = cv2.dilate(result,kernel)
# Contours
im2, contours, hierarchy = cv2.findContours(result.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
result = cv2.cvtColor(result, cv2.COLOR_GRAY2BGR)
if len(contours) != 0:
for (i, c) in enumerate(contours):
area = cv2.contourArea(c)
if area > 100000:
print(area)
cv2.drawContours(img, c, -1, (255,255,0), 12)
x,y,w,h = cv2.boundingRect(c)
cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,0),12)
# Stack results
result = np.vstack((result, img))
resultOrig = result.copy()
# Save image to file before resizing
cv2.imwrite(str(time.time())+'_0_result.jpg',resultOrig)
# Resize
max_dimension = float(max(result.shape))
scale = 900/max_dimension
result = cv2.resize(result, None, fx=scale, fy=scale)
# Show results
cv2.imshow('res',result)
cv2.waitKey(0)
cv2.destroyAllWindows()

I changed your image to HSL colour space.
I can't see the scratch in the L channel, so the greyscale approach suggested earlier is going to be difficult.
But the scratch is quite noticeable in the hue plane.
You could use an edge detector to find the blemish in the hue channel. Here I use a difference of gaussians detector (with sizes 20 and 4).

personal guess is to use some algorithm to detect the grayscale change. The grayscale variation around the scratch should be bigger than the variation in other area. Sobel and Scharr Derivatives could be an option. This is a link to python-openCV about image gradient. You can first crop out the fruit with coutour application

If you really want to use conventional computer vision techniques, you should start with edges that can be detected on the fruit. Some of the edges are caused by the bumps on the fruit, so you have to look at various features of the area around the edges to find the difference between scratches and bumps. After you look at about a hundred scratches, you should be able to come up with some rules.
But this process is going to be very tiring, and my guess is you will not have much luck. A better way to approach this problem is to train a deep neural network by manually annotating scratches on about 100 images, and letting the network find out by itself how to distinguish scratches from the rest of the fruit.
If you are a beginner to these stuff, search for PyImageSearch and LearnOpenCV. Both are very resourceful sites where you can learn quickly.

OpenCV (Python): Construct Rectangle from thresholded image

The image below shows an aerial photo of a house block (re-oriented with the longest side vertical), and the same image subjected to Adaptive Thresholding and Difference of Gaussians.
Images: Base; Adaptive Thresholding; Difference of Gaussians
The roof-print of the house is obvious (to the human eye) on the AdThresh image: it's a matter of connecting some obvious dots. In the sample image, finding the blue-bounded box below -
Image with desired rectangle marked in blue
I've had a crack at implementing HoughLinesP() and findContours(), but get nothing sensible (probably because there's some nuance that I'm missing). The python script-chunk that fails to find anything remotely like the blue box, is as follows:
import cv2
import numpy as np
from matplotlib import pyplot as plt
# read in full (RGBA) image - to get alpha layer to use as mask
img = cv2.imread('rotated_12.png', cv2.IMREAD_UNCHANGED)
grey = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
# Otsu's thresholding after Gaussian filtering
blur_base = cv2.GaussianBlur(grey,(9,9),0)
blur_diff = cv2.GaussianBlur(grey,(15,15),0)
_,thresh1 = cv2.threshold(grey,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
thresh = cv2.adaptiveThreshold(grey,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY,11,2)
DoG_01 = blur_base - blur_diff
edges_blur = cv2.Canny(blur_base,70,210)
# Find Contours
(ed, cnts,h) = cv2.findContours(grey, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
cnts = sorted(cnts, key = cv2.contourArea, reverse = True)[:4]
for c in cnts:
approx = cv2.approxPolyDP(c, 0.1*cv2.arcLength(c, True), True)
cv2.drawContours(grey, [approx], -1, (0, 255, 0), 1)
# Hough Lines
minLineLength = 30
maxLineGap = 5
lines = cv2.HoughLinesP(edges_blur,1,np.pi/180,20,minLineLength,maxLineGap)
print "lines found:", len(lines)
for line in lines:
cv2.line(grey,(line[0][0], line[0][1]),(line[0][2],line[0][3]),(255,0,0),2)
# plot all the images
images = [img, thresh, DoG_01]
titles = ['Base','AdThresh','DoG01']
for i in xrange(len(images)):
plt.subplot(1,len(images),i+1),plt.imshow(images[i],'gray')
plt.title(titles[i]), plt.xticks([]), plt.yticks([])
plt.savefig('a_edgedetect_12.png')
cv2.destroyAllWindows()
I am trying to set things up without excessive parameterisation. I'm wary of 'tailoring' an algorithm for just this one image since this process will be run on hundreds of thousands of images (with roofs/rooves of different colours which may be less distinguishable from background). That said, I would love to see a solution that 'hit' the blue-box target - that way I could at the very least work out what I've done wrong.
If anyone has a quick-and-dirty way to do this sort of thing, it would be awesome to get a Python code snippet to work with.
The 'base' image ->
Base Image

You should apply the following:
1. Contrast Limited Adaptive Histogram Equalization-CLAHE and convert to gray-scale.
2. Gaussian Blur & Morphological transforms (dialation, erosion, etc) as mentioned by #bad_keypoints. This will help you get rid of the background noise. This is the most tricky step as the results will depend on the order in which you apply (first Gaussian Blur and then Morphological transforms or vice versa) and the window sizes you choose for this purpose.
3. Apply Adaptive thresholding
4. Apply Canny's Edge detection
5. Find contour having four corner points
As said earlier you need to tweak with input parameters of these functions and also need to validate these parameters with other images. As it might be possible that it will work for this case but not for other cases. Based on trial and error you need to fix the parameter values.