I am working on using video from a football (soccer) match and try to map the frames to a top view of the pitch using homography. I have started to get find all the white lines from the frames using both Hough lines as well as using the line segment detector, where the line segment detector seems to work slightly better. See my code and examples below:
import cv2
import numpy as np
cv2.imread("frame-27.jpg")
hsv = cv2.cvtColor(frame, cv2.COLOR_RGB2HSV)
mask_green = cv2.inRange(hsv, (36, 25, 25), (86, 255, 255)) # green mask to select only the field
frame_masked = cv2.bitwise_and(frame, frame, mask=mask_green)
gray = cv2.cvtColor(frame_masked, cv2.COLOR_RGB2GRAY)
_, thresh = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY)
canny = cv2.Canny(gray, 50, 150, apertureSize=3)
# Hough line detection
lines = cv2.HoughLinesP(canny, 1, np.pi / 180, 50, None, 50, 20)
# Line segment detection
lines_lsd = lsd.detect(canny)[0]
This uses this input frame
and returns this image for the Hough lines
and this image for the line segment detection.
My question is twofold: (1) any ideas on how I can further refine the line detection (i.e. decrease false positives such as lines around players and outside of the field) and (2) a good way to use the detection lines to create a homography so I can map the frame to a higher overview of the field (like this). Any help is greatly appreciated!
Pencils of lines
Cluster line segments in two pencils of lines using RANSAC. A pencil of lines is a set of lines that intersect at a common point. With homogeneous coordinates, the intersection point may potentially be at infinity (e.g. if all the lines are parallel).
You can find two random line segments, compute their intersection, and then consider all the lines that go somewhat close to that intersection point (within some threshold). Repeat this until you find the two pencils with the greatest amount of line segments. We can then assume that these pencils correspond to the vanishing points.
Here, the blue and red segments correspond to two pencils of lines. The green segments are outliers. As you can see the RANSAC algorithm is extremely good at rejecting outliers.
Rectifying
I am not aware of built-in OpenCV rectification for line segments specifically; the existing functions are designed to work with point correspondences.
Run an optimization to recover the homography to deal with orientation. Generally the homography H is of the form H = KRK^-1 where K is the intrinsic matrix and R is the rotation matrix.
For example, you can run a nonlinear least squares optimization on the manifold of the Lie group SO(3) to recover the R matrix. For example you can use LocalParameterization in Ceres solver. But it is pretty simple to implement this yourself in Python too. If the focal length is unknown, you'll have to add that as an optimization parameter as well.
Instead of nonlinear least squares there may be other methods. Some methods estimate the homography directly, but may not preserve the correct aspect ratio.
You can preview the homography by calling opencv's WarpPerspective function.
Estimate translation and scale
This will involve knowing something about the geometry of the football field. You can detect some salient features unique to the football field and estimate the scale. For example you can detect the circular arcs using the circle Hough transform.
Related
I am working on a problem where i need to find the path the robot can take without hitting any crop rows.Raw Image
My initial approach was to convert this into birds eye view and then use canny and skeletonize techniques.Then I applied Hough transform to come up with the crop rows.This works well when the rows are straight but if i rotate the image by 45 degrees I couldn't find any rows with Hough transform.So I decided to use another approach.
First I only selected the green region and applied morphological filters to remove small branches which come out
img = cv2.imread('''')
min_green2 = np.array([45, 50, 50])
max_green2 = np.array([75, 250, 250])
image_blur = cv2.GaussianBlur(img, (5, 5), 0)
image_blur_hsv = cv2.cvtColor(image_blur, cv2.COLOR_BGR2HSV)
image_green = cv2.inRange(image_blur_hsv, min_red2, max_red2)
se1 = cv2.getStructuringElement(cv2.MORPH_RECT, (7,7))
se2 = cv2.getStructuringElement(cv2.MORPH_RECT, (3,3))
mask = cv2.morphologyEx(image_green, cv2.MORPH_OPEN, se1)
I ended up with thisFinal_output
Now I want to detect the path the robot can take which is the black region.So only the first row is my region of interest and I tried different methods to draw a line in center of the row but couldn't find any help in opencv.I did manage to get a work around by splitting the image into two vertically and used cv2.fitline function to get a line joining one side of row and did the same with other side of row and finally I plotted the center line.But this is not an ideal approach and I feel like there might be some opencv functions to do it in much better way.Can some one help me with this or guide me in right way.
This is the final output I am looking for
Final expected result with green color showing the center of path
So, here is my approach using numpy and scipy, which yielded this result:
.
Without doing any bluring or morphological operations, use the Canny edge detector:
edges = cv2.Canny(image, 100, 200, None, 3, cv2.DIST_L2)
Notice that most of the edges surround the track your robot wants to follow. Since each edge is a collection of white pixels, we could calculate a column's total intensity:
normalized = cv2.normalize(edges, None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX)
column_intensity = normalized.sum(axis=0)
Plotting the results, we get
If we were to find the minimum of the graph, then we would find the x direction, where most of the edges are avoided. But first, let's smooth the function so as to avoid some noise.
# smooth function through moving average
window_size = 30
window = np.ones((window_size,)) / window_size
smoothed = np.convolve(column_intensity, window, mode="valid")
Since there are a lot of local minima, our additional constraint is that the x-direction the robot should take is the closest to the center of the image.
# find indices of local minima and select the one closest to the center
indices = scipy.signal.argrelmin(smoothed)[0]
distances = np.abs(indices - int(width / 2))
x = indices[np.argmin(distances)]
Now that we have the x-direction, we need to determine a y coordinate so as to estimate the angle the robot should rotate (tan(angle)=y/x). There are as many choices as there are rows in the image, which means the y coordinate needs to be manually set. If we choose a y closer to the robot, the angle will be more volatile as the robot advances. Conversely, if we choose a y that is far from the robot, then it will be less volatile but less accurate as well. That is up to you; the final image was created with a y = 400.
I hope this fits your needs :)
Summary of Question
I am detecting object silhouettes in front of a light source. To simplify the background and remove noise, I require masking everything that isn't the light source. How can I tell when the object would be on the edge of the masked area?
Assumptions
Assume featureless (monochrome black and white for edge detection) and ambiguous (a square in image 1 may be a circle in image 2) in shape.
Detailed Explanation of the Problem with "High Quality" Figures
Consider a silhouette in front of a light source. It is distinct and we can tell it is nested within the outer contour. Figure 1 depicts a simplified case.
We can treat our outer circle as a mask in this case, and easily ignore everything NOT within the contour. Figure 2 depicts the simplified case with some edge detection.
Everything works lovely until the silhouette moves to the edge of the light source. Suddenly we run into problems. Figure 3 is an example of a shape on the edge.
The silhouette is indistinguishable from the black of the background/masked area. OpenCV either assumes that suddenly the contour of our light source is funny shaped and there is no other object to be detected.
The Question Restated
What tools can I use to detect that there has been some sort of interruption of the edge shape? Is there a good/computational cheap way of determining if our silhouette is intersecting with another?
Graveyard of What I Know Does NOT Work
Assuming a static or simple silhouette shape. The figures are cartoons representing a more complicated real problem.
Assuming a perfectly round light source. HoughCircles does not work.
You can use the cv2.log_polar function to unwrap the circle/oval shape.
After that, np.argmax can be used to find the curve. Try smoothing out the curve using Scipy's signal.savgol_filter(). When the object blocks the light source, there will be a big difference between the smoothed line and the argmax data:
This is the code that I used:
import numpy as np
import cv2
# Read the image
img = cv2.imread('/home/stephen/Desktop/JkgJw.png', 0)
# Find the log_polar image
log_polar = cv2.logPolar(img, (img.shape[0]/2, img.shape[1]/2), 40, cv2.WARP_FILL_OUTLIERS)
# Create a background to draw on
bg = np.zeros_like(log_polar)
# Iterate through each row in the image and get the points on the edge
h,w = img.shape
points = []
for col in range(h-1):
col_slice = log_polar[col:col+1, :]
curve = np.argmax(255-col_slice)
cv2.circle(bg, (curve, col), 0, 255, 1)
points.append((curve, col))
cv2.imshow('log_polar', log_polar)
cv2.waitKey(0)
cv2.destroyAllWindows()
import scipy
from scipy import signal
x,y = zip(*points)
x_smooth = signal.savgol_filter(x,123,2)
import matplotlib.pyplot as plt
plt.plot(x)
plt.plot(x_smooth)
plt.show()
Problem:
I want to detect lines in a given image using OpenCV in Python. Although there are multiple obvious vertical lines, neither normal HoughLines nor probabilistic HoughLines does find them. As I spent plenty of time playing around with the Parameters, I guess I am doing something fundamental wrong here. I am Aware of the fact, that hough-lines is usually applied on edges, e.g. after using canny. Due to canny´s non-maximum supression, canny does not give good results here.
Image, where detecting the vertical lines Fails :
Why:
Given this (image of a water meter) :
I want to detect the rectangle around each digit. To detect the rectangles, I used sobel filters in x and y direction and calculated Magnitude and angle/Phase of the Gradient. As I assume the image to be rotated correctly in this step, I extract vertical and horizontal edges as shown in the image. My hope was to make use of houghLines to find the bounding boxes. Finding the horizontal lines works perfectly, as seen in the
Debug plot containing further insights on the Problem, where as I does not work on the vertical components (second row) :
Detecting the rectangles around each digit would help me to
locate the Region of Interest
cut out the region inside the rectangle, in other words the digit. Several other approches to detect the digits directly by using contours, all had the problem of the outer rectangles interfering with the digit.
Update: the Code for detecting the vertical lines:
#img is initialized with the binarized, vertical component image, as shown above
minLength = 30
maxGap = 7
angle_res = np.pi / 180
rad_res = 2
threshold_val = 100
linesP = cv2.HoughLinesP(img, rad_res, angle_res, threshold_val, minLineLength=minLength, maxLineGap=maxGap)
cdst = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
cdstP = np.copy(cdst)
if linesP is None:
print("Error when finding lines (probabilistic hough transformation). No lines detected")
else:
# Copy edges to the images that will display the results in BGR
for i in range(0, len(linesP)):
l = linesP[i][0]
cv2.line(cdstP, (l[0], l[1]), (l[2], l[3]), (255,0,0), 3, cv2.LINE_AA)
plt.imshow(cdstP); plt.show()
First apply canny edge with proper settings of threshold. Then apply probabilistic hough line transform. After applying hough transform filter the lines with slope. You want to filter the box so you need to filter horizontal and vertical lines. After filtering lines apply morphological dilation and erosion operation back to back to resultant image to get neat box around each digit. While applying hough transform select parameters minimum line length, maximum line length and maximum line gap appropriately.
You can use trackbar function while selecting appropriate parameters. The sample code is given below for selection of threshold for canny edge.
import cv2
import numpy as np
cv2.namedWindow('Result')
img = cv2.imread('qkEuE.png')
v1 = 0
v2 = 0
def doEdges():
edges = cv2.Canny(img,v1,v2)
edges = cv2.cvtColor(edges,cv2.COLOR_GRAY2BGR)
res = np.concatenate((img,edges),axis = 0)
cv2.imshow('Result',res)
def setVal1(val):
global v1
v1 = val
doEdges()
def setVal2(val):
global v2
v2 = val
doEdges()
cv2.createTrackbar('Val1','Result',0,500,setVal1)
cv2.createTrackbar('Val2','Result',0,500,setVal2)
cv2.imshow('Result',img)
cv2.waitKey(0)
cv2.destroyAllWindows
Hope it helps you.
I have a binary image, and I am trying to represent a graph, such that the white parts of the image are the vertices and edges, where the big area white spots are the vertices, and the edges are the white parts that connect between the big white parts that I detected as vertices.
I managed to find the center of the big white parts by using OpenCV functions such as erosion, findContours and moments, using moments centroids.
So I have the vertices of the graph.
My next goal is to get the edges, meaning finding lines that are IN WHITE AREAS only, represented by 2 points, (x1,y1) and (x2,y2).
I tried using all kinds of function such as:
cv2.Canny()
cv2.findLine
cv2.findContour with different parameters on the binary image
For understanding my goal one can think about it as a maze, where the beginning of it is the biggest white spot in the image, and the end of the maze is the second biggest white spot, and the places you can walk through are all white areas of the image.
Some code segments I used in my project:
First finds the edges, given a binary image (finalImage) and return the centroids
def findCentroids(finalImage):
_, contours0, hierarchy = cv2.findContours(finalImage.copy(), cv2.RETR_LIST, cv2.CHAIN_APPROX_NONE)
moments = [cv2.moments(cnt) for cnt in contours0]
centroids = []
for M in moments:
if M["m00"] != 0:
cX = int(M["m10"] / M["m00"])
cY = int(M["m01"] / M["m00"])
centroids.append((cX, cY))
return centroids
So like I found centroids, I want to find more centroids (to make the image less erosed) and then perhaps find all edges that connect between these centroids. This doesn't seem like a good method so I hope to get better approaches in answers.
EDIT
So I thought about another idea, which is to use the connected components method. I try to use the connected components supplied by cv2, likewise:
output = cv2.connectedComponentsWithStats((imageForEdges), 8, cv2.CV_32S)
But the outcome is that only black spots are recognized as components, which is the opposite as what I need. I tried to use the inverted image and it gave the same results, as I assume the algorithm prefers the spots that are completely bounded, and not the background (which is the white color in my case, and the whole purpose of me using it, that it finds areas that are not bounded
)
Did you check out Iwanowski's algorithm ?
https://pdfs.semanticscholar.org/cd14/22f1e33022b0bede3f4a03844bc7dcc979ed.pdf
"The paper describes a method for the analysis of the content of a binary image in order to find its structure. The class of images it deals with consists of images showing at its foreground groups of objects connected one to another forming a graph-like structure. Described method extract automatically this structure from image bitmap and produces a matrix containing connections between all the objects shown on the input image"
I've been laboring on a pet project for a bit on how to find a simple basketball in an image. I've tried a bunch of permutations of using hough.circles and transform , etc for the last few weeks but I cant seem to come anywhere close to isolating the basketball with the code examples and my own tinkering.
Here is an example photo:
And here is the result after a simple version of circle finding code I've been tinkering with:
Anyone have any idea where I have gone wrong and how I can get it right?
Here is the the code I am fiddling with:
import cv2
import cv2.cv as cv # here
import numpy as np
def draw_circles(storage, output):
circles = np.asarray(storage)
for circle in circles:
Radius, x, y = int(circle[0][3]), int(circle[0][0]), int(circle[0][4])
cv.Circle(output, (x, y), 1, cv.CV_RGB(0, 255, 0), -1, 8, 0)
cv.Circle(output, (x, y), Radius, cv.CV_RGB(255, 0, 0), 3, 8, 0)
orig = cv.LoadImage('basket.jpg')
processed = cv.LoadImage('basket.jpg',cv.CV_LOAD_IMAGE_GRAYSCALE)
storage = cv.CreateMat(orig.width, 1, cv.CV_32FC3)
#use canny, as HoughCircles seems to prefer ring like circles to filled ones.
cv.Canny(processed, processed, 5, 70, 3)
#smooth to reduce noise a bit more
cv.Smooth(processed, processed, cv.CV_GAUSSIAN, 7, 7)
cv.HoughCircles(processed, storage, cv.CV_HOUGH_GRADIENT, 2, 32.0, 30, 550)
draw_circles(storage, orig)
cv.imwrite('found_basketball.jpg',orig)
I agree with the other posters, that using the colour of the basketball is a good approach. Here is some simple code that does that:
import cv2
import numpy as np
im = cv2.imread('../media/basketball.jpg')
# convert to HSV space
im_hsv = cv2.cvtColor(im, cv2.COLOR_BGR2HSV)
# take only the orange, highly saturated, and bright parts
im_hsv = cv2.inRange(im_hsv, (7,180,180), (11,255,255))
# To show the detected orange parts:
im_orange = im.copy()
im_orange[im_hsv==0] = 0
# cv2.imshow('im_orange',im_orange)
# Perform opening to remove smaller elements
element = np.ones((5,5)).astype(np.uint8)
im_hsv = cv2.erode(im_hsv, element)
im_hsv = cv2.dilate(im_hsv, element)
points = np.dstack(np.where(im_hsv>0)).astype(np.float32)
# fit a bounding circle to the orange points
center, radius = cv2.minEnclosingCircle(points)
# draw this circle
cv2.circle(im, (int(center[1]), int(center[0])), int(radius), (255,0,0), thickness=3)
out = np.vstack([im_orange,im])
cv2.imwrite('out.png',out)
result:
I assume that:
Always one and only one basketball is present
The basketball is the principal orange item in the scene
With these assumptions, if we find anything the correct colour, we can assume its the ball and fit a circle to it. This way we don't do any circle detection at all.
As you can see in the upper image, there are some smaller orangey elements (from the shorts) which would mess up our ball radius estimate. The code uses an opening operation (erosion followed by dilation), to remove these. This works nicely for your example image. But for other images a different method might be better: using circle detection too, or contour shape, size, or if we are dealing with a video, we could track the ball position.
I ran this code (only modified for video) on a random short basketball video, and it worked surprisingly ok (not great.. but ok).
A few thoughts:
Filter by color first to simplify the image. If you're looking specifically for an orange basketball, you could eliminate a lot of other colors. I'd recommend using HSI color space instead of RGB, but in any case you should be able to exclude colors that are some distance in color 3-space from your trained basketball color.
Try substituting Sobel or some other kernel-based edge detector that doesn't rely on manual parameters. Display the edge image to see if it looks "right" to you.
Allow for weaker edges. In the grayscale image, the contrast between the basketball and the player's dark jersey is not as great as the difference between the white undershirt and the black jersey.
Hough may yield unexpected results if the object is only nominally circular in cross section, but is actually elongated or has noisy edges in the actual image. I usually write my own Hough algorithm and haven't touched the OpenCV implementation, so I'm not sure what parameter to change, but see if you can allow for fuzzier edges.
Maybe eliminate the smooth operation. In any case, try smooth before finding edges rather than the other way around.
Try to write your own rough Hough algorithm. Although a quickie implementation may not be as flexible as the OpenCV implementation, by getting your hands dirty you may stumble onto the source of the problem.