Related
Links to all images at the bottom
I have drawn a line over an arrow which captures the angle of that arrow. I would like to then remove the arrow, keep only the line, and use cv2.minAreaRect to determine the angle. So far I've got everything to work except removing the original arrow, which results in an incorrect angle generated by the cv2.minAreaRect bounding box.
Really, I just want the bold black line running through the arrow to use to measure the angle, not the arrow itself. if anyone has an idea to make this work, or a simpler way, please let me know. Thanks
Code:
import numpy as np
import cv2
image = cv2.imread("templates/a_15.png")
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
ret, thresh = cv2.threshold(image, 127, 255, 0)
contours, hierarchy = cv2.findContours(thresh, 1, 2)
cont = contours[0]
rows,cols = image.shape[:2]
[vx,vy,x,y] = cv2.fitLine(cont, cv2.DIST_L2,0,0.01,0.01)
leftish = int((-x*vy/vx) + y)
rightish = int(((cols-x)*vy/vx)+y)
line = cv2.line(image,(cols-1,rightish),(0,leftish),(0,255,0),10)
# thresholding
thresh = cv2.threshold(line, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1]
# compute rotated bounding box based on all pixel values > 0 and
# use coordinates to compute a rotated bounding box of those coordinates
coordinates = np.column_stack(np.where(thresh > 0))
w = coordinates[0]
h = coordinates[1]
# Compute minimum rotated angle that contains entire image.
# Return angle values in the range [-90, 0).
# As the rectangle rotates clockwise, angle values increase towards 0.
# Once 0 is reached, angle is set back to -90 degrees.
angle = cv2.minAreaRect(coordinates)[-1]
# for angles less than -45 degrees, add 90 degrees to angle to take the inverse.
if angle < - 45:
angle = -(90 + angle)
else:
angle = -angle
# rotate image
(h, w) = image.shape[:2]
center = (w // 2, h // 2) # image center
RM = cv2.getRotationMatrix2D(center, angle, 1.0)
rotated = cv2.warpAffine(image, RM, (w, h),
flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_REPLICATE)
# correction angle for validation
cv2.putText(rotated, "Angle {:.2f} degrees".format(angle),
(10, 30), cv2.FONT_HERSHEY_DUPLEX, 0.9, (0, 255, 0), 2)
# output
print("[INFO] angle: {:.3f}".format(angle))
cv2.imshow("Line", line)
cv2.imshow("Input", image)
cv2.imshow("Rotated", rotated)
cv2.waitKey(0)
Images
original
current results
goal
Here's a possible solution. The main idea is to identify de "tip" and the "tail" of the arrow approximating some key points. After you have identified both ends, you can draw a line joining both points. It is also an advantage to know which of the endpoints is the tip, because that way you can measure the angle from a constant point.
There's more than one way to achieve this. I choose something that I have applied in the past: I will use this approach to identify the endpoints of the overall shape. My assumption is that the tip will yield more points than the tail. After that, I'll cluster all the endpoints in two groups: tip and tail. I can use K-Means for that, as it will return the mean centers for both clusters. After that, we have our tip and tail points that can be joined easily with a line. These are the steps:
Convert the image to grayscale
Get the skeleton of the image, to normalize the shape to a width of 1 pixel
Apply the method described in the link to get the arrow's endpoints
Divide the endpoints in two clusters and use K-Means to get their centers
Join both endpoints with a line
Let's see the code:
# imports:
import cv2
import numpy as np
# image path
path = "D://opencvImages//"
fileName = "CoXeb.png"
# Reading an image in default mode:
inputImage = cv2.imread(path + fileName)
# Grayscale conversion:
grayscaleImage = cv2.cvtColor(inputImage, cv2.COLOR_BGR2GRAY)
grayscaleImage = 255 - grayscaleImage
# Extend the borders for the skeleton:
extendedImg = cv2.copyMakeBorder(grayscaleImage, 5, 5, 5, 5, cv2.BORDER_CONSTANT)
# Store a deep copy of the crop for results:
grayscaleImageCopy = cv2.cvtColor(extendedImg, cv2.COLOR_GRAY2BGR)
# Compute the skeleton:
skeleton = cv2.ximgproc.thinning(extendedImg, None, 1)
The first step is to get the skeleton of the arrow. As I said, this step is needed prior to the convolution-based method that identifies the endpoints of a shape. Computing the skeleton normalizes the shape to a one pixel width. However, sometimes, if the shape is too close to the "canvas" borders, the skeleton could show some artifacts. This is avoided with a border extension. The skeleton of the arrow is this:
Check that image out. If we identify the endpoints, the tip will exhibit at least 3 points, while the tail at least 1. That's handy - the tip will always have more points than the tail. If only we could detect those points... Luckily, we can:
# Threshold the image so that white pixels get a value of 0 and
# black pixels a value of 10:
_, binaryImage = cv2.threshold(skeleton, 128, 10, cv2.THRESH_BINARY)
# Set the end-points kernel:
h = np.array([[1, 1, 1],
[1, 10, 1],
[1, 1, 1]])
# Convolve the image with the kernel:
imgFiltered = cv2.filter2D(binaryImage, -1, h)
# Extract only the end-points pixels, those with
# an intensity value of 110:
binaryImage = np.where(imgFiltered == 110, 255, 0)
# The above operation converted the image to 32-bit float,
# convert back to 8-bit uint
binaryImage = binaryImage.astype(np.uint8)
This endpoint detecting method convolves the skeleton with a special kernel that identifies endpoints. It returns a binary image where all the endpoints have the value 110. After thresholding this mid-result, we get this image, which represents the arrow endpoints:
Nice, as you see, we can group the points in two clusters and get their cluster centers. Sounds like a job for K-Means, because that's exactly what it does. We first need to treat our data, though, because K-Means operates on defined-shaped arrays of float data:
# Find the X, Y location of all the end-points
# pixels:
Y, X = binaryImage.nonzero()
# Reshape the arrays for K-means
Y = Y.reshape(-1,1)
X = X.reshape(-1,1)
Z = np.hstack((X, Y))
# K-means operates on 32-bit float data:
floatPoints = np.float32(Z)
# Set the convergence criteria and call K-means:
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 10, 1.0)
ret, label, center = cv2.kmeans(floatPoints, 2, None, criteria, 10, cv2.KMEANS_RANDOM_CENTERS)
# Set the cluster count, find the points belonging
# to cluster 0 and cluster 1:
cluster1Count = np.count_nonzero(label)
cluster0Count = np.shape(label)[0] - cluster1Count
print("Elements of Cluster 0: "+str(cluster0Count))
print("Elements of Cluster 1: " + str(cluster1Count))
The last two lines prints the endpoints that are assigned to Cluster 0 Cluster 1, respectively. That outputs this:
Elements of Cluster 0: 3
Elements of Cluster 1: 2
Just as expected - well, kinda. Seems that Cluster 0 is the tip and cluster 2 the tail! But the tail actually got 2 points. If you look the image of the skeleton closely, you'll see there's a small bifurcation at the tail. That's why we, in reality, got two points instead of just one. Alright, let's get the center points and draw them on the original input:
# Look for the cluster of max number of points
# That cluster will be the tip of the arrow:
maxCluster = 0
if cluster1Count > cluster0Count:
maxCluster = 1
# Check out the centers of each cluster:
matRows, matCols = center.shape
# Store the ordered end-points here:
orderedPoints = [None] * 2
# Let's identify and draw the two end-points
# of the arrow:
for b in range(matRows):
# Get cluster center:
pointX = int(center[b][0])
pointY = int(center[b][1])
# Get the "tip"
if b == maxCluster:
color = (0, 0, 255)
orderedPoints[0] = (pointX, pointY)
# Get the "tail"
else:
color = (255, 0, 0)
orderedPoints[1] = (pointX, pointY)
# Draw it:
cv2.circle(grayscaleImageCopy, (pointX, pointY), 3, color, -1)
cv2.imshow("End-Points", grayscaleImageCopy)
cv2.waitKey(0)
This is the resulting image:
The tip always gets drawn in red while the tail is drawn in blue. Very cool, let's store these points in the orderedPoints list and draw the final line in a new "canvas", with dimension same as the original image:
# Store the tip and tail points:
p0x = orderedPoints[1][0]
p0y = orderedPoints[1][1]
p1x = orderedPoints[0][0]
p1y = orderedPoints[0][1]
# Create a new "canvas" (image) using the input dimensions:
imageHeight, imageWidth = binaryImage.shape[:2]
newImage = np.zeros((imageHeight, imageWidth), np.uint8)
newImage = 255 - newImage
# Draw a line using the detected points:
(x1, y1) = orderedPoints[0]
(x2, y2) = orderedPoints[1]
lineColor = (0, 0, 0)
cv2.line(newImage , (x1, y1), (x2, y2), lineColor, thickness=2)
cv2.imshow("Detected Line", newImage)
cv2.waitKey(0)
The line overlaid on the original image and the new image containing only the line:
It sounds like you want to measure the angle of the line but because you are measuring a line you drew in the original image, you must now filter out the original image to get an accurate measure of the line...which you drew with coordinates you know the endpoints of?
I guess:
make a better filter?
draw the line in a blank image and detect angle there?
determine the angle from the known coordinates?
Since you were asking for just a line, I tried that...just made a blank image, drew your detected line on it and then used that downstream...
blankIm = np.ones((height, width, channels), dtype=np.uint8)
blankIm.fill(255)
line = cv2.line(blankIm,(cols-1,rightish),(0,leftish),(0,255,0),10)
I was using this code to detect the rectangle on the photo, at first it was working well, untill i realized that i would have an object that is also a square in the middle :
Question:
How can i properly detect the 4 corners like on the first result picture without detecting the corner of the thing in the middle of the square. Thanks a lot.
Code:
import numpy as np
import cv2
img = cv2.imread('Photos/lastBoard.png')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
canny = cv2.Canny(gray, 100, 200)
corners = cv2.goodFeaturesToTrack(gray, 25, 0.01, 50)
corner_list = []
for corner in corners:
x, y = corner.ravel()
if(y < 700 and (50 < x < 800 )):
corner_list.append([int(x), int(y)])
cv2.circle(img, (x, y), 5, (36, 255, 12), -1)
cv2.imshow("yo", img)
cv2.waitKey(0)
My man, it breaks my heart you aren't using the techniques and processing we covered in your last question. You have already plenty of functions you could re-use. The rectangle you are trying to segment has a unique color (kind of green) and has a defined area and aspect ratio! Look all the things you have on the table, they are smaller than the rectangle! Plus, the rectangle is almost a square! That means that its aspect ratio is close to 1.0. If you somehow segment the rectangle, approximating its corners should be relativity easy.
This is valuable info, because it allows you to trace your action plan. I see you are using cv2.goodFeaturesToTrack to detect the corners of everything. That's OK, but it could be simplified. I propose a plan of action very similar to last time:
Try to segment the rectangle using its color, let's compute an
HSV-based mask
Let's clean the mask from noise using an area filter and some morphology
Find contours - we are looking for the biggest green contour, the rectangle.
The contour of interest has defined features. Use the area and aspect ratio to filter garbage contours.
Once you have the contour/blob of interest, approximate its corners.
Let's see the code:
# imports:
import numpy as np
import cv2
# image path
path = "D://opencvImages//"
fileName = "table1.jpg"
# Reading an image in default mode:
inputImage = cv2.imread(path + fileName)
inputCopy = inputImage.copy()
# The HSV mask values:
lowerValues = np.array([58, 151, 25])
upperValues = np.array([86, 255, 75])
# Convert the image to HSV:
hsvImage = cv2.cvtColor(inputImage, cv2.COLOR_BGR2HSV)
# Create the HSV mask
mask = cv2.inRange(hsvImage, lowerValues, upperValues)
The first steps aim to create the HSV mask. Very similar to last time, I've defined the HSV range of interest already and applied exactly the same stuff as before. You could (and should) explore more exotic techniques latter, but let's stick with what we know works for the time being, as the project surely is due soon. This is the result:
You see how the mask is pretty nice already? Only the green puck and the rectangle survived the thresholding. It doesn't matter that the rectangle is not complete, because we're gonna approximate its contour with a bounding rectangle! Alright, let's clean this bad boy a little bit better. Use a filterArea (this is exactly the same function we saw last time) and then a closing (dilate followed by erode) just to get a nice mask:
# Run a minimum area filter:
minArea = 50
mask = areaFilter(minArea, mask)
# Pre-process mask:
kernelSize = 3
structuringElement = cv2.getStructuringElement(cv2.MORPH_RECT, (kernelSize, kernelSize))
iterations = 2
mask = cv2.morphologyEx(mask, cv2.MORPH_DILATE, structuringElement, None, None, iterations, cv2.BORDER_REFLECT101)
mask = cv2.morphologyEx(mask, cv2.MORPH_ERODE, structuringElement, None, None, iterations, cv2.BORDER_REFLECT101)
This is the filtered mask, the noise is mostly gone:
Now, let's find contours and filtered based on area and aspect ratio, just like last time. The parameters, however, are different, because our target is not the plucks, but the rectangle:
# Find the big contours/blobs on the filtered image:
contours, hierarchy = cv2.findContours(mask, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
# Store the poly approximation and bound
contoursPoly = [None] * len(contours)
# Store the corners of the square here:
detectedCorners = []
# Look for the outer bounding boxes:
for _, c in enumerate(contours):
# Approximate the contour to a polygon:
contoursPoly = cv2.approxPolyDP(c, 3, True)
# Convert the polygon to a bounding rectangle:
boundRect = cv2.boundingRect(contoursPoly)
# Get the bounding rect's data:
rectX = boundRect[0]
rectY = boundRect[1]
rectWidth = boundRect[2]
rectHeight = boundRect[3]
# Calculate the rect's area:
rectArea = rectWidth * rectHeight
# Calculate the aspect ratio:
aspectRatio = rectWidth / rectHeight
delta = abs(1.0 - aspectRatio)
# Set the min threshold values to identify the
# blob of interest:
minArea = 2500
epsilon = 0.2
Alright, so far so good, I hope. As you see I approximated the contour to a 4-vertex polygon and then computed its bounding rectangle. This approximation should fit very nicely to our blob of interest. Now, apply the contour filter and use the bounding rectangle data to approximate the corners. I approximate each corner, one by one, and store them in the
detectedCorners array. Then, we can draw 'em. Here, still inside the for loop:
# Is this bounding rectangle we
# are looking for?
if rectArea > minArea and delta < epsilon:
# Compute the corners/vertices:
# Corner 1 (top left)
corner1 = (rectX, rectY)
detectedCorners.append(corner1)
# Corner 2 (top right)
corner2 = (rectX + rectWidth, rectY)
detectedCorners.append(corner2)
# Corner 3 (bottom left)
corner3 = (rectX, rectY + rectHeight)
detectedCorners.append(corner3)
# Corner 4 (bottom right)
corner4 = (rectX + rectWidth, rectY + rectHeight)
detectedCorners.append(corner4)
# Draw the corner points:
for p in detectedCorners:
color = (0, 0, 255)
cv2.circle(inputCopy, (p[0], p[1]), 5, color, -1)
cv2.imshow("Square Corners", inputCopy)
cv2.waitKey(0)
Here are the results for both images. The approximated corners are the red dots:
Here's the definition and implementation of the areaFilter function:
def areaFilter(minArea, inputImage):
# Perform an area filter on the binary blobs:
componentsNumber, labeledImage, componentStats, componentCentroids = \
cv2.connectedComponentsWithStats(inputImage, connectivity=4)
# Get the indices/labels of the remaining components based on the area stat
# (skip the background component at index 0)
remainingComponentLabels = [i for i in range(1, componentsNumber) if componentStats[i][4] >= minArea]
# Filter the labeled pixels based on the remaining labels,
# assign pixel intensity to 255 (uint8) for the remaining pixels
filteredImage = np.where(np.isin(labeledImage, remainingComponentLabels) == True, 255, 0).astype('uint8')
return filteredImage
I have detected a lane boundary line which is not straight using hough transform and then extracted that line separately. Then blended with another image that has a straight line. Now I need to calculate the angle between those two lines, but I do not know the coordinates of those lines. So I tried with code that gives the coordinates of vertical lines, but it can not specifically identify those coordinates. Is there a way to measure the angle between those lines? Here is my coordinate calculation code and blended image with two lines
import cv2 as cv
import numpy as np
src = cv.imread("blended2.png", cv.IMREAD_COLOR)
if len(src.shape) != 2:
gray = cv.cvtColor(src, cv.COLOR_BGR2GRAY)
else:
gray = src
gray = cv.bitwise_not(gray)
bw = cv.adaptiveThreshold(gray, 255, cv.ADAPTIVE_THRESH_MEAN_C, cv.THRESH_BINARY, 15, -2)
horizontal = np.copy(bw)
vertical = np.copy(bw)
cols = horizontal.shape[1]
horizontal_size = int(cols / 30)
horizontalStructure = cv.getStructuringElement(cv.MORPH_RECT, (horizontal_size, 1))
horizontal = cv.erode(horizontal, horizontalStructure)
horizontal = cv.dilate(horizontal, horizontalStructure)
cv.imwrite("img_horizontal8.png", horizontal)
h_transpose = np.transpose(np.nonzero(horizontal))
print("h_transpose")
print(h_transpose[:100])
rows = vertical.shape[0]
verticalsize = int(rows / 30)
verticalStructure = cv.getStructuringElement(cv.MORPH_RECT, (1, verticalsize))
vertical = cv.erode(vertical, verticalStructure)
vertical = cv.dilate(vertical, verticalStructure)
cv.imwrite("img_vertical8.png", vertical)
v_transpose = np.transpose(np.nonzero(vertical))
print("v_transpose")
print(v_transpose[:100])
img = src.copy()
# edges = cv.Canny(vertical,50,150,apertureSize = 3)
minLineLength = 100
maxLineGap = 200
lines = cv.HoughLinesP(vertical,1,np.pi/180,100,minLineLength,maxLineGap)
for line in lines:
for x1,y1,x2,y2 in line:
cv.line(img,(x1,y1),(x2,y2),(0,255,0),2)
cv.imshow('houghlinesP_vert', img)
cv.waitKey(0)
One approach is to use the Hough Transform to detect the lines and obtain the angle of each line. The angle between the two lines can then be found by subtracting the difference between the two lines.
We begin by performing an arithmetic average using np.mean to essentially threshold the image which results in this.
image = cv2.imread('2.png')
# Compute arithmetic mean
image = np.mean(image, axis=2)
Now we perform skimage.transform.hough_line to detect lines
# Perform Hough Transformation to detect lines
hspace, angles, distances = hough_line(image)
# Find angle
angle=[]
for _, a , distances in zip(*hough_line_peaks(hspace, angles, distances)):
angle.append(a)
Next we obtain the angle for each line and find the difference to obtain our result
# Obtain angle for each line
angles = [a*180/np.pi for a in angle]
# Compute difference between the two lines
angle_difference = np.max(angles) - np.min(angles)
print(angle_difference)
16.08938547486033
Full code
from skimage.transform import (hough_line, hough_line_peaks)
import numpy as np
import cv2
image = cv2.imread('2.png')
# Compute arithmetic mean
image = np.mean(image, axis=2)
# Perform Hough Transformation to detect lines
hspace, angles, distances = hough_line(image)
# Find angle
angle=[]
for _, a , distances in zip(*hough_line_peaks(hspace, angles, distances)):
angle.append(a)
# Obtain angle for each line
angles = [a*180/np.pi for a in angle]
# Compute difference between the two lines
angle_difference = np.max(angles) - np.min(angles)
print(angle_difference)
I have 20 small images (that I want to put in a target area of a background image (13x12). I have already marked my target area with a circle, I have the coordinates of the circle in two arrays of pixels. Now I want to know how I can randomly add my 20 small images in random area in my arrays of pixels which are basically the target area (the drawn circle).
In my code, I was trying for just one image, if it works, I'll pass the folder of my 20 small images.
# Depencies importation
import cv2
import numpy as np
# Saving directory
saving_dir = "../Saved_Images/"
# Read the background image
bgimg = cv2.imread("../Images/background.jpg")
# Resizing the bacground image
bgimg_resized = cv2.resize(bgimg, (2050,2050))
# Read the image that will be put in the background image (exemple of 1)
# I'm just trying with one, if it works, I'll pass the folder of the 20
small_img = cv2.imread("../Images/small.jpg")
# Convert the resized background image to gray
bgimg_gray = cv2.cvtColor(bgimg, cv2.COLOR_BGR2GRAY)
# Convert the grayscale image to a binary image
ret, thresh = cv2.threshold(bgimg_gray,127,255,0)
# Determine the moments of the binary image
M = cv2.moments(thresh)
# calculate x,y coordinate of center
cX = int(M["m10"] / M["m00"])
cY = int(M["m01"] / M["m00"])
# Drawing the circle in the background image
circle = cv2.circle(bgimg, (cX, cY), 930, (0,0,255), 9)
print(circle) # This returns None
# Getting the coordinates of the circle
combined = bgimg[:,:,0] + bgimg[:,:,1] + bgimg[:,:,2]
rows, cols = np.where(combined >= 0)
# I have those pixels in rows and cols, but I don't know
# How to randomly put my small image in those pixel
# Saving the new image
cv2.imwrite(saving_dir+"bgimg"+".jpg", bgimg)
cv2.namedWindow('image', cv2.WINDOW_NORMAL)
cv2.resizeWindow("Test", 1000, 1200)
# Showing the images
cv2.imshow("image", bgimg)
# Waiting for any key to stop the program execution
cv2.waitKey(0)
In the expected results, the small images must be placed in the background image randomly.
If you have the center and the radius of your circle, you can easily generate random coordinates by randomly choosing an angle theta from [0, 2*pi], calculating corresponding x and y values by cos(theta) and sin(theta) and scaling these by some random chosen scaling factors from [0, radius]. I prepared some code for you, see below.
I omitted a lot of code from yours (reading, preprocessing, saving) to focus on the relevant parts (see how to create a minimal, complete, and verifiable example). Hopefully, you can integrate the main idea of my solution into your code on your own. If not, I will provide further explanations.
import cv2
import numpy as np
# (Artificial) Background image (instead of reading an actual image...)
bgimg = 128 * np.ones((401, 401, 3), np.uint8)
# Circle parameters (obtained somehow...)
center = (200, 200)
radius = 100
# Draw circle in background image
cv2.circle(bgimg, center, radius, (0, 0, 255), 3)
# Shape of small image (known before-hand...?)
(w, h) = (13, 12)
for k in range(200):
# (Artificial) Small image (instead of reading an actual image...)
smallimg = np.uint8(np.add(128 * np.random.rand(w, h, 3), (127, 127, 127)))
# Select random angle theta from [0, 2*pi]
theta = 2 * np.pi * np.random.rand()
# Select random distance factors from center
factX = (radius - w/2) * np.random.rand()
factY = (radius - h/2) * np.random.rand()
# Calculate random coordinates for small image from angle and distance factors
(x, y) = np.uint16(np.add((np.cos(theta) * factX - w/2, np.sin(theta) * factY - h/2), center))
# Replace (rather than "add") determined area in background image with small image
bgimg[x:x+smallimg.shape[0], y:y+smallimg.shape[1]] = smallimg
cv2.imshow("bgimg", bgimg)
cv2.waitKey(0)
The exemplary output:
Caveat: I haven't paid attention, if the small images might violate the circle boundary. Therefore, some additional checks or limitations to the scaling factors must be added.
EDIT: I edited my above code. To take the below comment into account, I shift the small image by (width/2, height/2), and limit the radius scale factor accordingly, so that the circle boundary isn't violated, neither top/left nor bottom/right.
Before, it was possible, that the boundary is violated in the bottom/right part (n = 200):
After the edit, this should be prevented (n = 20000):
The touching of the red line in the image is due to the line's thickness. For "safety reasons", one could add another 1 pixel distance.
How can I take two images of an object from different angles and draw epipolar lines on one based on points from the other?
For example, I would like to be able to select a point on the left picture using a mouse, mark the point with a circle, and then draw an epipolar line on the right image corresponding to the marked point.
I have 2 XML files which contain a 3x3 camera matrix and a list of 3x4 projection matrices for each picture. The camera matrix is K. The projection matrix for the left picture is P_left. The projection matrix for the right picture is P_right.
I have tried this approach:
Choose a pixel coordinate (x,y) in the left image (via mouse click)
Calculate a point p in the left image with K^-1 * (x,y,1)
Calulate the pseudo inverse matrix P+ of P_left (using np.linalg.pinv)
Calculate the epipole e' of the right image: P_right * (0,0,0,1)
Calculate the skew symmetric matrix e'_skew of e'
Calculate the Fundamental matrix F: e'_skew * P_right * P+
Calculate the epipolar line l' on the right image: F * p
Calculate a point p' in the right image: P_right * P+ * p
Transform p' and l back to pixel coordinates
Draw a line using cv2.line through p' and l
I just did this a few days ago and it works just fine. Here's the method I used:
Calibrate camera(s) to obtain camera matricies and distortion matricies (Using openCV getCorners and calibrateCamera, you can find lots of tutorials on this, but it sounds like you already have this info)
Perform stereo calibration with openCV stereoCalibrate(). It takes as parameters all of the camera and distortion matricies. You need this to determine the correlation between the two visual fields. You will get back several matricies, the rotation matrix R, translation vector T, essential matrix E and fundamental matrix F.
You then want to do undistortion using openCV getOptimalNewCameraMatrix and undistort(). This will get rid of a lot of camera aberrations (it will give you better results)
Finally, use openCV's computeCorrespondEpilines to calculate the lines and plot them. I will include some code below you can try out in Python. When I run it, I can get images like this (The colored points have their corresponding epilines drawn in the other image)
Heres some code (Python 3.0). It uses two static images and static points, but you could easily select the points with the cursor. You can also refer to the OpenCV docs on calibration and stereo calibration here.
import cv2
import numpy as np
# find object corners from chessboard pattern and create a correlation with image corners
def getCorners(images, chessboard_size, show=True):
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((chessboard_size[1] * chessboard_size[0], 3), np.float32)
objp[:, :2] = np.mgrid[0:chessboard_size[0], 0:chessboard_size[1]].T.reshape(-1, 2)*3.88 # multiply by 3.88 for large chessboard squares
# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.
for image in images:
frame = cv2.imread(image)
# height, width, channels = frame.shape # get image parameters
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
ret, corners = cv2.findChessboardCorners(gray, chessboard_size, None) # Find the chess board corners
if ret: # if corners were found
objpoints.append(objp)
corners2 = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria) # refine corners
imgpoints.append(corners2) # add to corner array
if show:
# Draw and display the corners
frame = cv2.drawChessboardCorners(frame, chessboard_size, corners2, ret)
cv2.imshow('frame', frame)
cv2.waitKey(100)
cv2.destroyAllWindows() # close open windows
return objpoints, imgpoints, gray.shape[::-1]
# perform undistortion on provided image
def undistort(image, mtx, dist):
img = cv2.imread(image, cv2.IMREAD_GRAYSCALE)
image = os.path.splitext(image)[0]
h, w = img.shape[:2]
newcameramtx, _ = cv2.getOptimalNewCameraMatrix(mtx, dist, (w, h), 1, (w, h))
dst = cv2.undistort(img, mtx, dist, None, newcameramtx)
return dst
# draw the provided points on the image
def drawPoints(img, pts, colors):
for pt, color in zip(pts, colors):
cv2.circle(img, tuple(pt[0]), 5, color, -1)
# draw the provided lines on the image
def drawLines(img, lines, colors):
_, c, _ = img.shape
for r, color in zip(lines, colors):
x0, y0 = map(int, [0, -r[2]/r[1]])
x1, y1 = map(int, [c, -(r[2]+r[0]*c)/r[1]])
cv2.line(img, (x0, y0), (x1, y1), color, 1)
if __name__ == '__main__':
# undistort our chosen images using the left and right camera and distortion matricies
imgL = undistort("2L/2L34.bmp", mtxL, distL)
imgR = undistort("2R/2R34.bmp", mtxR, distR)
imgL = cv2.cvtColor(imgL, cv2.COLOR_GRAY2BGR)
imgR = cv2.cvtColor(imgR, cv2.COLOR_GRAY2BGR)
# use get corners to get the new image locations of the checcboard corners (undistort will have moved them a little)
_, imgpointsL, _ = getCorners(["2L34_undistorted.bmp"], chessboard_size, show=False)
_, imgpointsR, _ = getCorners(["2R34_undistorted.bmp"], chessboard_size, show=False)
# get 3 image points of interest from each image and draw them
ptsL = np.asarray([imgpointsL[0][0], imgpointsL[0][10], imgpointsL[0][20]])
ptsR = np.asarray([imgpointsR[0][5], imgpointsR[0][15], imgpointsR[0][25]])
drawPoints(imgL, ptsL, colors[3:6])
drawPoints(imgR, ptsR, colors[0:3])
# find epilines corresponding to points in right image and draw them on the left image
epilinesR = cv2.computeCorrespondEpilines(ptsR.reshape(-1, 1, 2), 2, F)
epilinesR = epilinesR.reshape(-1, 3)
drawLines(imgL, epilinesR, colors[0:3])
# find epilines corresponding to points in left image and draw them on the right image
epilinesL = cv2.computeCorrespondEpilines(ptsL.reshape(-1, 1, 2), 1, F)
epilinesL = epilinesL.reshape(-1, 3)
drawLines(imgR, epilinesL, colors[3:6])
# combine the corresponding images into one and display them
combineSideBySide(imgL, imgR, "epipolar_lines", save=True)
Hopefully this helps!