Develop Reference

Keypoints detection and matching between binary masks

I am trying to match keypoints using opencv (tutorial) between images shown below.
The thing is that I am not sure if I need to adjust some parameters or I am entirely using wrong method. Taking only right side of map.png did not help either.
Here is my code and also result.
import numpy as np
import cv2
import matplotlib.pyplot as plt
img1 = cv2.imread('../map.png',0)
img2 = cv2.imread('../mask.png',0)
orb = cv2.ORB_create()
kp1, des1 = orb.detectAndCompute(img1,None)
kp2, des2 = orb.detectAndCompute(img2,None)
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(des1,des2)
matches = sorted(matches, key = lambda x:x.distance)
img3 = cv2.drawMatches(img1,kp1,img2,kp2,matches[:20], None, flags=2)
cv2.imwrite('test.png', img3)

Feature detectors such as ORB used by you are designed to match feature points between images that differ in translation, rotation and scale. They are not intended to be used when images differ significantly in perspective (that is your case) and therefore your approach doesn't work. Moreover, such algorithms are designed for images that are rich in texture such as photos. In your case the features are repetative (multiple feature points extracted from first image, such as line endings, can be matched to a single point in the other).
In your case you should consider another features such as those based on lines intersections, see this tutorial for more information. This is only a hint, not the solution for your problem as it is really challenging.

using SIFT to locate correspondence between images

I have a 512x512 image which gave 6109 SIFT keypoints.
Now, I rotated it then I got 6070 SIFT keypoints.
I tried following
import cv2
import numpy as np
np.set_printoptions(threshold=np.nan)
img = cv2.imread('home.jpg')
gray= cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
sift = cv2.xfeatures2d.SIFT_create()
kp,des = sift.detectAndCompute(gray,None)
print('kp',len(kp))
print('des',des.shape)
for i in range(len(kp)):
print('x ',kp[i].pt[0])
print('y ',kp[i].pt[1])
print('size',kp[i].size)
print('response',kp[i].response)
print('descriptor',np.sort(des[i]))
print('\n')
Send the output to a text file.
Now I did the same with rotated image.
You must have noticed that I am sorting the 128 value descriptor before writing it in file so that on rotation if order of values changes in descriptor then still there will be match.
My conclusion is that among those 12179 descriptors(6070+6109), only two were identical. Rest differed atleast by one number,usually by many I saw.
Can you suggest a python code that will show the x,y coordinates of correspondences between the two images ?
Also, there is something fundamentally wrong i must be doing here in comparing descriptors.Please point that out .
Thanks

Why are you sorting the descriptor? 128-dimensional descriptor is something that describes the feature keypoint(as the name suggests). By sorting you are essentially losing the description of your key points/features.
Searching for an identical descriptor is not a good way to find correspondence. You have to find descriptors with minimum Euclidean distance(or some other distance measure), to get correspondence. There are many algorithms for that in opencv like Brute force matches or knn based matcher or flann based matcher. YOu have to give the descriptor as input to these matcher(do not modify your decsriptor values by sorting or some other opertation)
For example
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
# BFMatcher with default params
bf = cv2.BFMatcher()
matches = bf.knnMatch(des1,des2, k=2)
matches will have index for correspondences and distance between those correspondences
You can get sort this matches based on matches.distance
and get x and y coordinate using matches.queryIdx and matches.trainIdx
you can get x and y location using kp1[matches[idx].trainIdx] and kp2[matches[idx].queryIdx]

this is implementation of details provided by Garvita Tiwari.
I am using python 2.7.12 with opencv 3.4.1
import cv2
import numpy as np
src_img = cv2.imread('src.jpg')
test_img = cv2.imread('test.jpg')
src_gray= cv2.cvtColor(src_img,cv2.COLOR_BGR2GRAY)
test_gray= cv2.cvtColor(test_img,cv2.COLOR_BGR2GRAY)
sift = cv2.xfeatures2d.SIFT_create()
src_kp,src_desc = sift.detectAndCompute(src_gray,None)
test_kp,test_desc = sift.detectAndCompute(test_gray,None)
bf = cv2.BFMatcher()
matches = bf.knnMatch(src_desc,test_desc, k=2)
good = []
for m,n in matches:
if m.distance < 0.75*n.distance:
good.append([m])
print 'so the correspondences are'
for i in range(len(good)):
print int(src_kp[good[i][0].queryIdx].pt[0]),int(src_kp[good[i][0].queryIdx].pt[1]) ,'->',int(test_kp[good[i][0].trainIdx].pt[0]) ,int(test_kp[good[i][0].trainIdx].pt[1])

How do you use Akaze in Open CV on python

I found example in c++:
http://docs.opencv.org/3.0-beta/doc/tutorials/features2d/akaze_matching/akaze_matching.html
But there isn't any example in python showing how to use this feature detector (also couldn't find anything more in documentation about AKAZE there is ORB SIFT, SURF, etc but not what I'm looking for)
http://docs.opencv.org/3.1.0/db/d27/tutorial_py_table_of_contents_feature2d.html#gsc.tab=0
Can someone could share or show me where I can find information how to match images in python with akaze?

I am not sure on where to find it, the way I made it work was through this function which used the Brute Force matcher:
def kaze_match(im1_path, im2_path):
# load the image and convert it to grayscale
im1 = cv2.imread(im1_path)
im2 = cv2.imread(im2_path)
gray1 = cv2.cvtColor(im1, cv2.COLOR_BGR2GRAY)
gray2 = cv2.cvtColor(im2, cv2.COLOR_BGR2GRAY)
# initialize the AKAZE descriptor, then detect keypoints and extract
# local invariant descriptors from the image
detector = cv2.AKAZE_create()
(kps1, descs1) = detector.detectAndCompute(gray1, None)
(kps2, descs2) = detector.detectAndCompute(gray2, None)
print("keypoints: {}, descriptors: {}".format(len(kps1), descs1.shape))
print("keypoints: {}, descriptors: {}".format(len(kps2), descs2.shape))
# Match the features
bf = cv2.BFMatcher(cv2.NORM_HAMMING)
matches = bf.knnMatch(descs1,descs2, k=2) # typo fixed
# Apply ratio test
good = []
for m,n in matches:
if m.distance < 0.9*n.distance:
good.append([m])
# cv2.drawMatchesKnn expects list of lists as matches.
im3 = cv2.drawMatchesKnn(im1, kps1, im2, kps2, good[1:20], None, flags=2)
cv2.imshow("AKAZE matching", im3)
cv2.waitKey(0)
Remember that the feature vectors are binary vectors. Therefore, the similarity is based on the Hamming distance, rather than the commonly used L2 norm or Euclidean distance if you will.

I searched for the same tutorial and found out the tutorial is given in 3 alternate languages C++, Python & Java. There are 3 hyperlinks for them before the start of code area.
Try this [ https://docs.opencv.org/3.4/db/d70/tutorial_akaze_matching.html ]

OpenCV feature matching for multiple images

How can I optimise the SIFT feature matching for many pictures using FLANN?
I have a working example taken from the Python OpenCV docs. However this is comparing one image with another and it's slow. I need it to search for features matching in a series of images (a few thousands) and I need it to be faster.
My current idea:
Run through all the images and save the features. How?
Compare an image from a camera with this above base, and find the correct one. How?
Give me the result, matching image or something.
http://docs.opencv.org/trunk/doc/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.html
import sys # For debugging only
import numpy as np
import cv2
from matplotlib import pyplot as plt
MIN_MATCH_COUNT = 10
img1 = cv2.imread('image.jpg',0) # queryImage
img2 = cv2.imread('target.jpg',0) # trainImage
# Initiate SIFT detector
sift = cv2.SIFT()
# find the keypoints and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)
flann = cv2.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(des1,des2,k=2)
# store all the good matches as per Lowe's ratio test.
good = []
for m,n in matches:
if m.distance MIN_MATCH_COUNT:
src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)
M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC,5.0)
matchesMask = mask.ravel().tolist()
h,w = img1.shape
pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
dst = cv2.perspectiveTransform(pts,M)
img2 = cv2.polylines(img2,[np.int32(dst)],True,255,3, cv2.LINE_AA)
else:
print "Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT)
matchesMask = None
draw_params = dict(matchColor = (0,255,0), # draw matches in green color
singlePointColor = None,
matchesMask = matchesMask, # draw only inliers
flags = 2)
img3 = cv2.drawMatches(img1,kp1,img2,kp2,good,None,**draw_params)
plt.imshow(img3, 'gray'),plt.show()
UPDATE
After trying out many things I might have come closer to the solution now. I hope it's possible to build the index and then search in it like this:
flann_params = dict(algorithm=1, trees=4)
flann = cv2.flann_Index(npArray, flann_params)
idx, dist = flann.knnSearch(queryDes, 1, params={})
However I still haven't managed to build an accepted npArray to the flann_Index parameter.
loop through all images as image:
npArray.append(sift.detectAndCompute(image, None))
npArray = np.array(npArray)

I never solved this in Python, however I switched environment to C++ where you get more OpenCV examples and don't have to use a wrapper with less documentation.
An example on the issue I had with matching in multiple files can be found here: https://github.com/Itseez/opencv/blob/2.4/samples/cpp/matching_to_many_images.cpp

Along with the reply of #stanleyxu2005 I'd like to add some tips as to how to do the whole matching itself since I'm currently working of such a thing.
I strongly recommend to create some custom class that wraps around the cv::Mat but also stores various other essential pieces of data. In my case I have an ImageContainer store the original image (that I will use for the final stitching), the processed one (grayscaled, undistorted etc.), its keypoints and the descriptors for those. By doing so you can access all the matching-relevant information in a pretty well organized well. You can either implement the keypoint extraction and descriptor generation in it or do that outside the class and just store the results in that container.
Store all image containers in some kind of a structure (vector is usually a good choice) for easy access.
I also created a class called ImageMultiMatchContainer, which stores a pointer to a given query image (all images are query images), a vector with pointers to all train images (for a single query image of the image set all others are train images) that were matched to it and also a vector of the match vectors for each of those matches. Here I stumbled across a storage issue namely - first you have to skip matching of an image with itself because it is pointless and second you have the problem of comparing two images two times and thus generating a considerable overhead if you have a lot of images. The second problem is due to the fact that we iterate through all images (query images) and compare them to the rest in the set (train images). At some point we have image X (query) matched with image Y (train), but later we also have image Y (now query) matched with image X (now train). As you can see this is also pointless since it's basically matching the same pair of images twice. This can be solved (currently working on this) by creating a class (MatchContainer) that stores a pointer to each of the two images in a matched pair and also the match vector. You store this in a central location (in my case this is my matcher class) and for each image as query image you check the list of matched images of the train image. If it's empty then you create a new MatchContainer and add it to the rest of the MatchContainers. If it's not then you look in it and see if the current query image is not present there (comparing pointers is a fast operation). If it is then you just pass the pointer to that MatchContainer's vector item that stores the matches for those two images. If that is not the case, you do as if it's empty and create a new MatchContainer etc. MatchingContainers should be stored in a data structure with a small access times since you will be looking at them a lot and iterating from start to end costs too much time. I'm considering using a map but maybe a tree of some sort can offer some advantages as well.
The homography estimation is a very tricky part. Here I recommend you look at bundle block adjustment. I saw that the stitcher class in OpenCV has a BundleBase-class but haven't tested it yet to see what's in it.
A general recommendation is to look at the stitching process in OpenCV and read the source code. The stitching pipeline is a straight forward set of processes and you just have to see how exactly you can implement the single steps.

Here are several pieces of my advice:
You should reduce the amount of point data by using proper techniques.
Calculate the reference image repeatedly is a waste. You should persistent all calculated reference.
Do not put the calculate on a mobile device. You'd better upload the calculated reference of a captured image to a powerful server and do the searching there.
This is a very interesting topic. My ears are opening too.

Image stitching Python

I have to stitch two or more images together using python and openCV.
I found this code for finding keypoints and matches, but I don't know how to continue.
Help me please!
import numpy as np
import cv2
MIN_MATCH_COUNT = 10
img1 = cv2.imread('a.jpg',0) # queryImage
img2 = cv2.imread('b.jpg',0) # trainImage
# Initiate SIFT detector
sift = cv2.SIFT()
# find the keypoints and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)
flann = cv2.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(des1,des2,k=2)
# store all the good matches as per Lowe's ratio test.
good = []
for m,n in matches:
if m.distance < 0.7*n.distance:
good.append(m)

Your question is not very clear, but I assume what you mean is that you have a bunch of images and you want to have opencv find the corresponding landmarks and then warp/scale each picture so that they can form one big image.
A method without using the stitcher class, basically looping over pictures and determining the best fitting one each iteration, is documented in this github code

One approach to image stitching consists of the following steps.
Firstly, as you've already figured out, you need a feature point detector and the some way to find correspondences between feature points on both images. It's typically a good idea to eliminate a lot of correspondences because they will likely contain a lot of noise. A super simple way to eliminate a lot of noise is to look for symmetry in the matches.
This is roughly what your code does up to this point.
Next, to stitch images together, you need to warp one of the images to match the perspective of the other image. This is done by estimating the homography using the correspondences. Because your correspondences will still likely contain a lot of noise, we typically use RANSAC to robustly estimate the homography.
A quick google search provides many examples of this being implemented.