I have several fish images in my database , My Goal is to find similarity score between user input fish image and images in database. For that I am using opencv Feature matching + Homograpy from this link.
http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.html#feature-homography
My current code is as followed.
query_image = '/home/zealous/Pictures/train_images/AbudefdufWhitleyiJER.jpg'
trained_image_folder = '/home/zealous/Pictures/train_images'
My current code is as followed.
def feature_matcher(query_image, image_folder):
min_match_count = 10
img1 = cv2.imread(query_image, 0)
surf = cv2.xfeatures2d.SURF_create(800)
kp1, des1 = surf.detectAndCompute(img1, None)
bf = cv2.BFMatcher(cv2.NORM_L2)
all_files = next(os.walk(image_folder))[2]
for file_name_temp in all_files:
try:
train_image = image_folder + '/' + file_name_temp
img2 = cv2.imread(train_image, 0)
surf = cv2.xfeatures2d.SURF_create(800)
kp2, des2 = surf.detectAndCompute(img2, None)
matches = bf.knnMatch(des1, des2, k=2)
good = []
for m, n in matches:
if m.distance < 0.7*n.distance:
good.append(m)
if len(good) > min_match_count:
src_pts = np.float32([kp1[m.queryIdx].pt for m in good]).reshape(-1,1,2)
dst_pts = np.float32([kp2[m.trainIdx].pt for m in good]).reshape(-1,1,2)
M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0)
matchesMask = mask.ravel().tolist()
h, w = img1.shape
pts = np.float32([[0, 0], [0, h-1], [w-1, h-1], [w-1, 0]]).reshape(-1,1,2)
dst = cv2.perspectiveTransform(pts, M)
if not M==None:
print "\n"
print "-"*2, file_name_temp
print "number of good matches", len(good)
print "*"*10, matchesMask
I am getting pretty good output which I am assuming by seeing number of good matches and matchesMask variable (which contains some 0's and 1's). If database contains same image as input image then there will be many good matches and all matchesMask elements will be 1.
My question is how to calculate similarity score based on this? should I assume that the more number of 1's (Inliers) are there in matchesMask, more both images are similar or should I take ratio between number of 1's(inliers) and 0's(outliers) and calculate similarity based on that.
I know this has been discussed in many questions , but all the suggestions and answers are in C++ language , so I cant figure out solution..
In a similarity score you don't want to include outliers - they are outliers because they don't help with your data. Just take the number of 1s (inliers) as the similarity score - you should get decent results.
Related
I have master/golden picture that I want to align another picture that capture to it.
the diff can be in the angle(not that much) and in the scale.
of course also in the brightness and point of view of the camera(again not that much).
when I take the master/golden picture I set known position for the object that interest me, in this case it led. I must do it becuase if one led not turn off I need to know reporting which one is missing, so the known position is mainly for knowing which led are missing. so I do the alignment to search in the known position the led I want to analyze.
for example, this is the master/golden picture,
be noted that the black rectangle is to demonstrate what I am searching for, its not blended on the picture.(only the rectangle around the led).
image I capture to align:
what have I try:
perspective transform using 2/4 anchor from the golden picture.
using algoritem like surf/sift.
surf/sift gave me better result then the first option, but still I have some issues(the led get out of the rectangle).
see below after the alignment, the led are out of the rectangle(no all of them)
increase the rectangle size its not an option because the led are too close to each other.
I think the issue with the sift it that it take wrong matches. I try to avoid them without success.
How I try to avoid wrong matches:
I know for a fact that the angle won't change significantly, so I tried to ignore twin points that have a large slope between them Using this code: good = [m for m in good if abs((kpts1[m.queryIdx].pt[1] - kpts2[m.trainIdx].pt[1])/(kpts1[m.queryIdx].pt[0] - kpts2[m.trainIdx].pt[0] ))<0.5] but it didn't worked as I can still saw matches with big slope.
I think maybe the solution for me is to eliminate matches with slope or matches that are not in the same are by giving threshold to the match point, but I am not sure how to do it.
anyone have a solution for me? maybe try something diff then sift/perspective transform?
all the code I am using:
imgname1 = options.source_path
imgname2 = options.output_path
img1 = cv2.imread(imgname1)
img2 = cv2.imread(imgname2)
gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)
surf = cv2.xfeatures2d.SURF_create()
# orb = cv2.ORB_create(nfeatures=1500)
matcher = cv2.FlannBasedMatcher(dict(algorithm = 1, trees = 5), {})
# bf = cv2.BFMatcher(cv2.NORM_HAMMING,crossCheck=True)
kpts1, descs1 = surf.detectAndCompute(gray1,None)
kpts2, descs2 = surf.detectAndCompute(gray2,None)
# kpts1, descs1 = orb.detectAndCompute(gray1,None)
# kpts2, descs2 = orb.detectAndCompute(gray2,None)
matches = matcher.knnMatch(descs1, descs2,2)
# matches = bf.match(descs1,descs2)
matches = sorted(matches, key = lambda x:x[0].distance)
good = [m1 for (m1, m2) in matches if m1.distance < 0.7 * m2.distance]
# good = matches[:50]
good = [m for m in good if abs((kpts1[m.queryIdx].pt[1] - kpts2[m.trainIdx].pt[1])/(kpts1[m.queryIdx].pt[0] - kpts2[m.trainIdx].pt[0] ))>1]
canvas = img2.copy()
if len(good)>MIN_MATCH_COUNT:
src_pts = np.float32([ kpts1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
dst_pts = np.float32([ kpts2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)
# for pt_src in src_pts:
# print(pt_src[0][0])
M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RHO ,5.0) #try to use RANCA, but RHO gave me the best results.
h,w = img1.shape[:2]
pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
dst = cv2.perspectiveTransform(pts,M)
cv2.polylines(canvas,[np.int32(dst)],True,(0,255,0),3, cv2.LINE_AA)
else:
print( "Not enough matches are found - {}/{}".format(len(good),MIN_MATCH_COUNT))
matched = cv2.drawMatches(img1,kpts1,canvas,kpts2,good,None,flags=2)
h,w = img1.shape[:2]
pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
dst = cv2.perspectiveTransform(pts,M)
perspectiveM = cv2.getPerspectiveTransform(np.float32(dst),pts)
found = cv2.warpPerspective(img2,perspectiveM,(w,h))
orig_path = options.output_path
path = re.split("/",orig_path)
orig_path = orig_path.replace(path[-1],"")
with open(orig_path + '/matrix_transform.pkl', 'wb') as outp:
pickle.dump(perspectiveM, outp, pickle.HIGHEST_PROTOCOL)
pickle.dump(w, outp, pickle.HIGHEST_PROTOCOL)
pickle.dump(h, outp, pickle.HIGHEST_PROTOCOL)
cv2.imwrite(orig_path + "/matched.jpg", matched)
my_str = imgname2
substr = ".jpg"
inserttxt = "_transform"
idx = my_str.index(substr)
my_str = my_str[:idx] + inserttxt + my_str[idx:]
print(my_str)
cv2.imwrite(my_str, found)
I've did quite a search about image stitching on python and most are for panoramic images, warping and rotating the images to combine them into one.
What I'm trying to do is using computer images, so they are digital and can be template matched without a problem, it will always be 2D without need of warping.
Basically here I have pieces of a map that is zoomed in and I want to make a massive image of this small pictures, here we have all the images used: https://imgur.com/a/HZIeT3z
import os
import numpy as np
import cv2
def stitchImagesWithoutWarp(img1, img2):
orb = cv2.ORB_create()
kp1, des1 = orb.detectAndCompute(img1,None)
kp2, des2 = orb.detectAndCompute(img2,None)
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(des1,des2)
matches = sorted(matches, key = lambda x:x.distance)
good_matches = matches[:10]
src_pts = np.float32([kp1[m.queryIdx].pt for m in good_matches]).reshape(-1,1,2)
dst_pts = np.float32([kp2[m.trainIdx].pt for m in good_matches]).reshape(-1,1,2)
start = (abs(int(dst_pts[0][0][0]-src_pts[0][0][0])), abs(int(dst_pts[0][0][1]-src_pts[0][0][1])))
h1, w1 = img1.shape[:2]
h2, w2 = img2.shape[:2]
vis = np.zeros((start[1]+h1,start[0]+w1,3), np.uint8)
vis[start[1]:start[1]+h1, start[0]:start[0]+w1, :3] = img1
vis[:h2, :w2, :3] = img2
return vis
imgList = []
for it in os.scandir("images"):
imgList.append(cv2.imread(it.path))
vis = stitchImagesWithoutWarp(imgList[0],imgList[1])
for index in range(2,len(imgList)):
cv2.imshow("result", vis)
cv2.waitKey()
vis = stitchImagesWithoutWarp(vis,imgList[index])
By running this code I can successfully stitch the first four images together, such as this:
But once I stitch the fifth image it seems to have wrong match and incorrectly, but I always get the best match by distance on NORM_HAMMING, this is the result:
The thing is, this is the first image, in this order, that the best match point (var start) is negative in the x axis, here is the matching points in the imgur order:
(7, 422)
(786, 54)
(394, 462)
(-350, 383)
I attempted switching the top image, doing specific code for negative match but I've believe I was deviating the performance.
Also noting from the docs the first image should be the query and the second supposed to be the target, but I couldn't get it to work by inverting the vis variable in function param.
The main issue here was when recognized points weren't on the screen (negative values), it needs offsets to adjust, I also incremented a little bit to the code and verified if the matches were legit, as if all the calculated displacement were in average the around the matched first pick in brute force.
with the average of 2MB for each image, without preprocessing the images/downscaling/compressing, after stitching 9 images together I got the average of 1050ms in my PC, as for other algorithms tested (that warped the image) took around 2-3seconds for stitching 2 of those images.
here is the final result:
import os
import numpy as np
import cv2
def averageTuple(tupleList):
avgX, avgY = 0,0
for tuple in tupleList:
avgX += tuple[0]
avgY += tuple[1]
return (int(avgX/len(tupleList)),int(avgY/len(tupleList)))
def tupleInRange(t1, t2, dif=3):
if t1[0] + dif > t2[0] and t1[0] - dif < t2[0]:
if t1[1] + dif > t2[1] and t1[1] - dif < t2[1]:
return True
return False
def rgbToRGBA(img):
b_channel, g_channel, r_channel = cv2.split(img)
alpha_channel = np.ones(b_channel.shape, dtype=b_channel.dtype) * 255
return cv2.merge((b_channel, g_channel, r_channel, alpha_channel))
def cropAlpha(img,extraRange=0.05):
y, x = img[:, :, 3].nonzero() # get the nonzero alpha coordinates
minx = int(np.min(x)*(1-extraRange))
miny = int(np.min(y)*(1-extraRange))
maxx = int(np.max(x)*(1+extraRange))
maxy = int(np.max(y)*(1+extraRange))
return img[miny:maxy, minx:maxx]
def stitchImagesWithoutWarp(img1, img2):
if len(cv2.split(img1)) != 4:
img1 = rgbToRGBA(img1)
if len(cv2.split(img2)) != 4:
img2 = rgbToRGBA(img2)
orb = cv2.ORB_create()
kp1, des1 = orb.detectAndCompute(img1,None)
kp2, des2 = orb.detectAndCompute(img2,None)
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(des1,des2)
matches = sorted(matches, key = lambda x:x.distance)
good_matches = matches[:10]
src_pts = np.float32([kp1[m.queryIdx].pt for m in good_matches]).reshape(-1,1,2)
dst_pts = np.float32([kp2[m.trainIdx].pt for m in good_matches]).reshape(-1,1,2)
pointsList = []
for index in range(0,len(src_pts)):
curPoint = (int(dst_pts[index][0][0]-src_pts[index][0][0])), (int(dst_pts[index][0][1]-src_pts[index][0][1]))
pointsList.append(curPoint)
start = pointsList[0]
avgTuple = averageTuple(pointsList)
if not tupleInRange(start, avgTuple): return img1
h1, w1 = img1.shape[:2]
h2, w2 = img2.shape[:2]
ax = abs(start[0])
ay = abs(start[1])
vis = np.zeros((ay+h1,ax+w1,4), np.uint8)
ofst2 = (ax if start[0]<0 else 0, ay if start[1]<0 else 0)
ofst1 = (0 if start[0]<0 else ax, 0 if start[1]<0 else ay)
vis[ofst1[1]:ofst1[1]+h1, ofst1[0]:ofst1[0]+w1, :4] = img1
vis[ofst2[1]:ofst2[1]+h2, ofst2[0]:ofst2[0]+w2, :4] = img2
return cropAlpha(vis)
imgList = []
for it in os.scandir("images"):
imgList.append(cv2.imread(it.path))
vis = stitchImagesWithoutWarp(imgList[0],imgList[1])
for index in range(2,len(imgList)):
vis = stitchImagesWithoutWarp(vis,imgList[index])
cv2.imwrite("output.png", cropAlpha(vis,0))
here is the output image (compressed in JPEG for stackoverflow):
Even though the ORB Feature Matching seems quite solid and i only take the 20 best matches for cv.findHomography, the resulting polyline is terrible. Note that in the results shown in the attached image, the top right image is a video stream. Therefor the variation in results matched. Is there a library that could be used to receive better results? Or am I doing any major mistakes in my code?
# des1 & des2 are created with cv.ORB_create(10000, 1.2, nlevels=8, edgeThreshold=5)
kp2, des2 = orb.detectAndCompute(gray, None)
matches = bf.knnMatch(des1, des2, k=2)
good = []
for m, n in matches:
if m.distance < 0.75 * n.distance:
good.append(m)
matches = sorted(good, key=lambda x: x.distance)
src_pts = np.float32([kp1[m.queryIdx].pt for m in matches[:20]]).reshape(-1, 1, 2)
dst_pts = np.float32([kp2[m.trainIdx].pt for m in matches[:20]]).reshape(-1, 1, 2)
M, mask = cv.findHomography(dst_pts, src_pts, cv.RANSAC, 5.0)
matchesMask = mask.ravel().tolist()
h = src_pts.max(0)[0][1] - src_pts.min(0)[0][1]
w = src_pts.max(0)[0][0] - src_pts.min(0)[0][0]
pts = np.float32([[0, 0], [0, h - 1], [w - 1, h - 1], [w - 1, 0]]).reshape(-1, 1, 2)
dst = cv.perspectiveTransform(pts, M)
img3 = None
img3 = cv.drawMatchesKnn(img1, kp1, gray, kp2, good, img3, flags=cv.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)
img3 = cv.polylines(img3, [np.int32(dst)], True, (0, 0, 255), 3, cv.LINE_AA)
# Code for showing img3 would follow
There could be several problems with this setup:
Pattern itself. It has repeated squares, so there could be matches that connect different squares on the first and on the second image. This can produce a lot of outliers so that homography can't fit in a reasonable way
Low image quality. The smaller image has low resolution and a bit blurry, which makes matching more difficult, so more outliers can happen. The image has higher resolution and is just displayed in small scale, so this point is not valid.
Feature points are located in a small region of an image and you try to project corners of the image, which are far from the points. This makes homography estimation very unstable so that uncertainties in coordinates of feature points become magnified several times. Even jitter of less than 1 pixel can result in projection errors of up to about 8 pixels. And this can be even worse, because RANSAC threshold of 5.0 can result in coordinates with lower precision.
I'm a dental student and currently trying to write a script for analyzing and extracting handwritten digits from dental records. I already have a rough version of the script finished but my recognition rate is pretty low. A big problem with analyzing the data is a grid that proves difficult to remove.
Scanned form that I want to analyse (white fields are for anonymity):
Empty form:
I've tried different solutions for this problem (Erosion/Dilation, HoughLineTransform and susbtraction of the Lines).
Using featurematching and substracting with an empty template currently give me the best results.
Results:
Eroding and dilating this image gives even better results
Results:
![][4]
But this needs a new calibration nearly every time i try it.
Do you know of a more elegant solution to my problem.
Could SURF matching give better results?
Thank you very much!
Here's my code so far:
GOOD_MATCH_PERCENT = 0.15
def match_img_to_template(input_img, template_img, MAX_FEATURES, GOOD_MATCH_PERCENT):
# blurring of the input image
template_img = cv2.GaussianBlur(template_img, (3, 3), cv2.BORDER_DEFAULT)
# equalizing the histogramm of the input image
img_preprocessed = cv2.equalizeHist(input_img)
# ORB Detector
orb = cv2.ORB_create(MAX_FEATURES)
kp1, des1 = orb.detectAndCompute(img_preprocessed, None)
kp2, des2 = orb.detectAndCompute(template_img, None)
# Brute Force Matching
matcher= cv2.DescriptorMatcher_create(cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING)
matches = matcher.match(des1, des2, None)
matches.sort(key=lambda x:x.distance, reverse=False)
numGoodMatches = int(len(matches) * GOOD_MATCH_PERCENT)
matches = matches[:numGoodMatches]
# Remove not so good matches
numGoodMatches = int(len(matches) * GOOD_MATCH_PERCENT)
matches = matches[:numGoodMatches]
# Extract location of good matches
points1 = np.zeros((len(matches), 2), dtype=np.float32)
points2 = np.zeros((len(matches), 2), dtype=np.float32)
for i, match in enumerate(matches):
points1[i, :] = kp1[match.queryIdx].pt
points2[i, :] = kp2[match.trainIdx].pt
# Find homography
h, mask = cv2.findHomography(points1, points2, cv2.RANSAC)
# Use homography
height, width = template_img.shape
input_warped = cv2.warpPerspective(input_img, h, (width, height))
ret1, input_warped_thresh = cv2.threshold(input_warped,0,255,cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)
diff = cv2.absdiff(template_img, input_warped_thresh)
ret, diff = cv2.threshold(diff, 20, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C + cv2.THRESH_BINARY)
diff = cv2.equalizeHist(diff)
# Create kernels
kernel1 = np.ones((3,3),np.uint8)
kernel2 = np.ones((6,6), np.uint8)
# erode dilate to remove the grid
diff_erode = cv2.erode(diff,kernel1)
diff_dilated = cv2.dilate(diff_erode,kernel2)
# invert diff_dilate
diff_dilated_inv = cv2.bitwise_not(diff_dilated)
return diff_dilated_inv
So I'm trying to overlay a thermal image with an rgb image using SIFT to match features and homography so that I can overlay them later on. The code I have works with about 50% of the thermal/rgb sets I have but many sets, such as this one, give horrible results. I think the homography is fine but doesn't work because the matches are way off. I'll attach some code, any advice on how to tune this would be great as I've spent a long time already trying to get this working on my own. Thanks!
MIN_MATCH_COUNT = 10
sift = cv2.xfeatures2d.SIFT_create(sigma=1.6, contrastThreshold=0.04,edgeThreshold = 15)
kp1, des1 = sift.detectAndCompute(rgb, None)
kp2, des2 = sift.detectAndCompute(thermal, None)
bf = cv2.BFMatcher()
matches = bf.knnMatch(des1, des2, k=2)
good = []
for m, n in matches:
if m.distance < 0.8 * n.distance:
good.append(m)
good = sorted(good, key=lambda x: x.distance)
img3 = cv2.drawMatches(rgb, kp1, thermal, kp2, good, None, flags=2)
which gives the following
Then I do homography with RANSAC on the found matches
if len(good) > MIN_MATCH_COUNT:
src_pts = np.float32([kp1[m.queryIdx].pt for m in good]).reshape(-1, 1, 2)
dst_pts = np.float32([kp2[m.trainIdx].pt for m in good]).reshape(-1, 1, 2)
M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0, maxIters=1000)
matchesMask = mask.ravel().tolist()
h, w, c = rgb.shape
pts = np.float32([[0, 0], [0, h - 1], [w - 1, h - 1], [w - 1, 0]]).reshape(-1, 1, 2)
dst = cv2.perspectiveTransform(pts, M)
thermal = cv2.polylines(thermal, [np.int32(dst)], True, 255, 3, cv2.LINE_AA) # draw lines around as a box
draw_params = dict(matchColor=(0, 255, 0), # draw matches in green color
singlePointColor=None,
matchesMask=matchesMask, # draw only inliers
flags=2)
img3 = cv2.drawMatches(rgb, kp1, thermal, kp2, good, None, **draw_params)
resulting in this
Like I said, I think that this is failing because the BFMatcher is not finding correct matches but I am not sure why. Again any and all help is very appreciated! I've tried using an orb detector, converting the rgb image to grayscale, and pre-sclaing images to similar sizes and still get bad results.
Here is an example of a working rgb-thermal pair to demonstrate what I am trying to do.
The problem with your image is that it's so simple compared to natural images (no color, no major differences in texture, etc.), you cannot reliably use SIFT and other techniques made with normal photos in mind. Most of your wrong matches are actually good matches, since the matches look locally similar to each other (after obtaining the descriptor).
My suggestion is to look at alternatives that match images using structural information, or add information to the image (e.g. using a height rainbow colormap since your images can be seen as bumpmaps; using the distance transform + colormap might work too, or using both mentioned + edge detection as the 3 channels for a very weird but heterogeneous color image) and see if SIFT behaves differently.