Develop Reference

Keypoints detection and matching between binary masks

I am trying to match keypoints using opencv (tutorial) between images shown below.
The thing is that I am not sure if I need to adjust some parameters or I am entirely using wrong method. Taking only right side of map.png did not help either.
Here is my code and also result.
import numpy as np
import cv2
import matplotlib.pyplot as plt
img1 = cv2.imread('../map.png',0)
img2 = cv2.imread('../mask.png',0)
orb = cv2.ORB_create()
kp1, des1 = orb.detectAndCompute(img1,None)
kp2, des2 = orb.detectAndCompute(img2,None)
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(des1,des2)
matches = sorted(matches, key = lambda x:x.distance)
img3 = cv2.drawMatches(img1,kp1,img2,kp2,matches[:20], None, flags=2)
cv2.imwrite('test.png', img3)

Feature detectors such as ORB used by you are designed to match feature points between images that differ in translation, rotation and scale. They are not intended to be used when images differ significantly in perspective (that is your case) and therefore your approach doesn't work. Moreover, such algorithms are designed for images that are rich in texture such as photos. In your case the features are repetative (multiple feature points extracted from first image, such as line endings, can be matched to a single point in the other).
In your case you should consider another features such as those based on lines intersections, see this tutorial for more information. This is only a hint, not the solution for your problem as it is really challenging.

Extracting SIFT features of image dataset to be matched

I have image dataset ant want to extract its features in order to be compared with the query image to select the best features inside threshold. I'm able to extract images features and select the best ones in two corresponding images as the following code:
img1 = cv2.imread("path\of\training\image")
img2 = cv2.imread("path\of\query\image")
# Initiate SIFT detector
sift = cv2.xfeatures2d.SIFT_create()
# find the key-points and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
# FLANN parameters
FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks=100) # or pass empty dictionary
flann = cv2.FlannBasedMatcher(index_params,search_params)
matches = flann.knnMatch(des1,des2,k=2)
# Need to draw only good matches, so create a mask
matchesMask = [[0,0] for i in range(len(matches))]
# ratio test as per Lowe's paper
for i,(m,n) in enumerate(matches):
if m.distance < 0.8*n.distance:
matchesMask[i]=[1,0]
draw_params = dict(matchColor = (0,255,0),
singlePointColor = (255,0,0),
matchesMask = matchesMask,
flags = 0)
img3 = cv2.drawMatchesKnn(img1,kp1,img2,kp2,matches,None,**draw_params)
plt.imshow(img3,),plt.show()
I want to compare the query image features with features of all images inside dataset, to select the best ones in order to recognize the specific object. How can I combine all dataset features and compare them with the query image features? can anyone please help me with thanks.

The first and naivest idea to solve your problem is: with your query image, which has some features represented by vectors, you find the nearest neighbours to those vectors in your features/vectors dataset, then the result should be the image which has most nearest features to the query features.
Basically, you have to calculate the distances between your vectors and FAISS give you some effort ways to do that.
I found this site that may help you:
https://waltyou.github.io/Faiss-In-Project-English/. The author faced the same situation as you did. And he used above way to get through it.
"To solve this problem, you can assign multiple ids to multiple vectors of an image when building a Faiss index. In this way, after searching with multiple vectors of a picture, in the returned result, only the number of times the associated id appears can be counted, and the similarity level can be obtained."

using SIFT to locate correspondence between images

I have a 512x512 image which gave 6109 SIFT keypoints.
Now, I rotated it then I got 6070 SIFT keypoints.
I tried following
import cv2
import numpy as np
np.set_printoptions(threshold=np.nan)
img = cv2.imread('home.jpg')
gray= cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
sift = cv2.xfeatures2d.SIFT_create()
kp,des = sift.detectAndCompute(gray,None)
print('kp',len(kp))
print('des',des.shape)
for i in range(len(kp)):
print('x ',kp[i].pt[0])
print('y ',kp[i].pt[1])
print('size',kp[i].size)
print('response',kp[i].response)
print('descriptor',np.sort(des[i]))
print('\n')
Send the output to a text file.
Now I did the same with rotated image.
You must have noticed that I am sorting the 128 value descriptor before writing it in file so that on rotation if order of values changes in descriptor then still there will be match.
My conclusion is that among those 12179 descriptors(6070+6109), only two were identical. Rest differed atleast by one number,usually by many I saw.
Can you suggest a python code that will show the x,y coordinates of correspondences between the two images ?
Also, there is something fundamentally wrong i must be doing here in comparing descriptors.Please point that out .
Thanks

Why are you sorting the descriptor? 128-dimensional descriptor is something that describes the feature keypoint(as the name suggests). By sorting you are essentially losing the description of your key points/features.
Searching for an identical descriptor is not a good way to find correspondence. You have to find descriptors with minimum Euclidean distance(or some other distance measure), to get correspondence. There are many algorithms for that in opencv like Brute force matches or knn based matcher or flann based matcher. YOu have to give the descriptor as input to these matcher(do not modify your decsriptor values by sorting or some other opertation)
For example
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
# BFMatcher with default params
bf = cv2.BFMatcher()
matches = bf.knnMatch(des1,des2, k=2)
matches will have index for correspondences and distance between those correspondences
You can get sort this matches based on matches.distance
and get x and y coordinate using matches.queryIdx and matches.trainIdx
you can get x and y location using kp1[matches[idx].trainIdx] and kp2[matches[idx].queryIdx]

this is implementation of details provided by Garvita Tiwari.
I am using python 2.7.12 with opencv 3.4.1
import cv2
import numpy as np
src_img = cv2.imread('src.jpg')
test_img = cv2.imread('test.jpg')
src_gray= cv2.cvtColor(src_img,cv2.COLOR_BGR2GRAY)
test_gray= cv2.cvtColor(test_img,cv2.COLOR_BGR2GRAY)
sift = cv2.xfeatures2d.SIFT_create()
src_kp,src_desc = sift.detectAndCompute(src_gray,None)
test_kp,test_desc = sift.detectAndCompute(test_gray,None)
bf = cv2.BFMatcher()
matches = bf.knnMatch(src_desc,test_desc, k=2)
good = []
for m,n in matches:
if m.distance < 0.75*n.distance:
good.append([m])
print 'so the correspondences are'
for i in range(len(good)):
print int(src_kp[good[i][0].queryIdx].pt[0]),int(src_kp[good[i][0].queryIdx].pt[1]) ,'->',int(test_kp[good[i][0].trainIdx].pt[0]) ,int(test_kp[good[i][0].trainIdx].pt[1])

Detect more than 1 object on picture

I wrote a little script which let to find an object in a global picture by the SIFT descriptors method. But I have a question about multiple detections in the same picture.
I have this global picture :
I have this template :
My script looks like :
import numpy as np
import cv2
#########################
# SIFT descriptors part #
#########################
img1 = cv2.imread('/Users/valentinjungbluth/Desktop/SIFT:SURF Algo/lampe.jpg',0)
img2 = cv2.imread('/Users/valentinjungbluth/Desktop/SIFT:SURF Algo/ville.jpg',0)
# Initiate SIFT detector
sift = cv2.xfeatures2d.SIFT_create()
print (img1.dtype)
print (img2.dtype)
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
bf = cv2.BFMatcher()
matches = bf.knnMatch(des1,des2,k=2)
good = []
for m,n in matches :
if m.distance < 0.2*n.distance :
good.append([m])
img3 = cv2.drawMatchesKnn(img1,kp1,img2,kp2,good,None,flags=2)
cv2.imwrite('matches.jpg',img3)
And the result is :
My question is :
How I can detect this others lamps ? Because all lamps are very similar and I want to match with all lamps which are present in the picture.
Thank you so much !
EDIT With Micka's answer :
Nothing appears at 0.2 scale distance, but if I put 0.75 :

This is a good question. There are couple of ways I can think of doing this:
1.Sliding Windowing technique - You can search for the "template" in the global image by making a window, the size of the template, and sliding it in the entire image. You can do this for a pyramid so the scale and translational changes are taken care of.
SIFT - Try matching the global image with the template and find all matches. Then you should filter the matches with relative pose. May be you'd require another filtering but I think this method is more general as it caters for more constraints than the previous.
Hope it helps!

Try to allow more good matches by being more permissive in the condition.
good = []
for m,n in matches :
if m.distance < 0.2*n.distance :
good.append([m])
A more robust approach would be to describe the lamp using the sift features extracted from the template image(s), and then try to find those features using a sliding window over the image. For each window, compute the sift features, and compute a "distance" to your template's features. If the distance is smaller than a given threshold, then the window contains a lamp!

OpenCV feature matching for multiple images

How can I optimise the SIFT feature matching for many pictures using FLANN?
I have a working example taken from the Python OpenCV docs. However this is comparing one image with another and it's slow. I need it to search for features matching in a series of images (a few thousands) and I need it to be faster.
My current idea:
Run through all the images and save the features. How?
Compare an image from a camera with this above base, and find the correct one. How?
Give me the result, matching image or something.
http://docs.opencv.org/trunk/doc/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.html
import sys # For debugging only
import numpy as np
import cv2
from matplotlib import pyplot as plt
MIN_MATCH_COUNT = 10
img1 = cv2.imread('image.jpg',0) # queryImage
img2 = cv2.imread('target.jpg',0) # trainImage
# Initiate SIFT detector
sift = cv2.SIFT()
# find the keypoints and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)
flann = cv2.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(des1,des2,k=2)
# store all the good matches as per Lowe's ratio test.
good = []
for m,n in matches:
if m.distance MIN_MATCH_COUNT:
src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)
M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC,5.0)
matchesMask = mask.ravel().tolist()
h,w = img1.shape
pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
dst = cv2.perspectiveTransform(pts,M)
img2 = cv2.polylines(img2,[np.int32(dst)],True,255,3, cv2.LINE_AA)
else:
print "Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT)
matchesMask = None
draw_params = dict(matchColor = (0,255,0), # draw matches in green color
singlePointColor = None,
matchesMask = matchesMask, # draw only inliers
flags = 2)
img3 = cv2.drawMatches(img1,kp1,img2,kp2,good,None,**draw_params)
plt.imshow(img3, 'gray'),plt.show()
UPDATE
After trying out many things I might have come closer to the solution now. I hope it's possible to build the index and then search in it like this:
flann_params = dict(algorithm=1, trees=4)
flann = cv2.flann_Index(npArray, flann_params)
idx, dist = flann.knnSearch(queryDes, 1, params={})
However I still haven't managed to build an accepted npArray to the flann_Index parameter.
loop through all images as image:
npArray.append(sift.detectAndCompute(image, None))
npArray = np.array(npArray)

I never solved this in Python, however I switched environment to C++ where you get more OpenCV examples and don't have to use a wrapper with less documentation.
An example on the issue I had with matching in multiple files can be found here: https://github.com/Itseez/opencv/blob/2.4/samples/cpp/matching_to_many_images.cpp

Along with the reply of #stanleyxu2005 I'd like to add some tips as to how to do the whole matching itself since I'm currently working of such a thing.
I strongly recommend to create some custom class that wraps around the cv::Mat but also stores various other essential pieces of data. In my case I have an ImageContainer store the original image (that I will use for the final stitching), the processed one (grayscaled, undistorted etc.), its keypoints and the descriptors for those. By doing so you can access all the matching-relevant information in a pretty well organized well. You can either implement the keypoint extraction and descriptor generation in it or do that outside the class and just store the results in that container.
Store all image containers in some kind of a structure (vector is usually a good choice) for easy access.
I also created a class called ImageMultiMatchContainer, which stores a pointer to a given query image (all images are query images), a vector with pointers to all train images (for a single query image of the image set all others are train images) that were matched to it and also a vector of the match vectors for each of those matches. Here I stumbled across a storage issue namely - first you have to skip matching of an image with itself because it is pointless and second you have the problem of comparing two images two times and thus generating a considerable overhead if you have a lot of images. The second problem is due to the fact that we iterate through all images (query images) and compare them to the rest in the set (train images). At some point we have image X (query) matched with image Y (train), but later we also have image Y (now query) matched with image X (now train). As you can see this is also pointless since it's basically matching the same pair of images twice. This can be solved (currently working on this) by creating a class (MatchContainer) that stores a pointer to each of the two images in a matched pair and also the match vector. You store this in a central location (in my case this is my matcher class) and for each image as query image you check the list of matched images of the train image. If it's empty then you create a new MatchContainer and add it to the rest of the MatchContainers. If it's not then you look in it and see if the current query image is not present there (comparing pointers is a fast operation). If it is then you just pass the pointer to that MatchContainer's vector item that stores the matches for those two images. If that is not the case, you do as if it's empty and create a new MatchContainer etc. MatchingContainers should be stored in a data structure with a small access times since you will be looking at them a lot and iterating from start to end costs too much time. I'm considering using a map but maybe a tree of some sort can offer some advantages as well.
The homography estimation is a very tricky part. Here I recommend you look at bundle block adjustment. I saw that the stitcher class in OpenCV has a BundleBase-class but haven't tested it yet to see what's in it.
A general recommendation is to look at the stitching process in OpenCV and read the source code. The stitching pipeline is a straight forward set of processes and you just have to see how exactly you can implement the single steps.

Here are several pieces of my advice:
You should reduce the amount of point data by using proper techniques.
Calculate the reference image repeatedly is a waste. You should persistent all calculated reference.
Do not put the calculate on a mobile device. You'd better upload the calculated reference of a captured image to a powerful server and do the searching there.
This is a very interesting topic. My ears are opening too.