I'm working on a image analysis code for detecting motion in subsequent image snapshots. To do this, I decided to use the optical flow function in opencv that finds good points to track in an initial image and predict the points in a subsequent image.
# feature extraction of points to track
pt = cv2.goodFeaturesToTrack(img1,**features_param)
# convert points to floating-point
p0 =np.float32(pt).reshape(-1,1,2)
# get predicted points using lucas-kanade optical flow
p1,st,err =cv2.calcOpticalFlowPyrLK(img1, img2,p0,
None,**lk_params)
In order to find points that were predicted correctly the optical flow function is run in reverse (second image first). Then an absolute difference is calculated between initial points (tracked points) and the backward predicted (p0r), if the value is below one then that points was predicted correctly if not it is a "bad" point.
# forward-backward error detection
p0r,st,err =cv2.calcOpticalFlowPyrLK(img2,img1,p1,
None,**lk_params)
# get correctly predicted points via absolute difference
d = abs(p0-p0r).reshape(-1, 2).max(-1)
good = d < 1
Go through the predicted points p1 and find values that fit the "good" condition.
# cycle through all current and new keypoints and only keep
# those that satisfy the "good" condition above
# Initialize a list to hold new keypoints
new_keypoints = list()
# get good points
for (x, y), good_flag,ind in zip(p1.reshape(-1, 2), good,enumerate(good)):
if not good_flag:
continue
new_keypoints.append((x,y))
I need to check which original points in p0 ended up predicted in new_keypoints.
After a torturing my brain I managed to get a solution to my problem. I created a for loop that goes through each point in the numpy array (p0) and predicts the point, if meets the "good" criterion it is appended to a list otherwise it is omitted. I then proceed to calculate the euclidean distance between the point and its newly predicted position. Here's the code:
Solution
# feature extraction of points to track
pt = cv2.goodFeaturesToTrack(img1,**features_param)
p0 =np.float32(pt).reshape(-1,1,2)
pr,st,err =cv2.calcOpticalFlowPyrLK(img1, img2,p0,
None,**lk_params)
# append correctly predicted points
dist = list()
for loop in p0:
p1,st,err =cv2.calcOpticalFlowPyrLK(img1, img2,loop,
None,**lk_params)
p0r,st,err =cv2.calcOpticalFlowPyrLK(img2,img1,p1,
None,**lk_params)
# calculate euclidean distance of predicted points
if abs(loop-p0r).reshape(-1, 2).max(-1) < 1:
dst = distance.euclidean(loop,p0r)
dist.append(dst)
Related
I am working in a discrete 2D grid of points in which there are "shapes" that I would like to create points outside of. I have been able to identify the vertices of these points and take convex hulls. So far, this leads to this and all is good and well. The purple here is the shape in question and the red line is the convex contour I have computed.
What I would like to do now is create two neighborhoods of points outside this shape. The first one is a set of points directly outside (as close as the grid size will allow), the second is another set of points but offset some distance away (the distance is not fixed, but rather an input).
I have attempted to write this in Python and get okay results. Here is an example of my current output. The problem is I notice the offsets are not perfect, for example look at the bottom most point in the image I attached. It kinks downwards whereas the original shape does not. It's not too bad in this example, but in other cases where the shape is smaller or if I take a smaller offset it gets worse. I also have an issue where the offsets sometimes overlap, even if they are supposed to be some distance away. I would also like there to be one line in each section of the contour, not two lines (for example in the top left).
My current attempt uses the Shapely package to handle most of the computational geometry. An outline of what I do once I have found the vertices of the convex contour is to offset these vertices by some amount, and interpolate along each pair of vertices to obtain many points alone these lines. Afterwards I use a coordinate transform to identify all points to the nearest grid point. This is how I obtain my final set of points. Below is the actual code I have written.
How can I improve this so I don't run into the issues I described?
Function #1 - Computes the offset points
def OutsidePoints(vertices, dist):
poly_line = LinearRing(vertices)
poly_line_offset = poly_line.buffer(dist, resolution=1, join_style=2, mitre_limit=1).exterior
new_vertices = list(poly_line_offset.coords)
new_vertices = np.asarray(new_vertices)
shape = sg.Polygon(new_vertices)
points = []
for t in np.arange(0, shape.length, step_size):
temp_points = np.transpose(shape.exterior.interpolate(t).xy)
points.append(temp_points[0])
points = np.array(points)
points = np.unique(points, axis=0)
return points
Function #2 - Transforming these points into points that are on my grid
def IndexFinder(points):
index_points = invCoordinateTransform(points)
for i in range(len(index_points)):
for j in range(2):
index_points[i][j] = math.floor(index_points[i][j])
index_points = np.unique(index_points, axis=0)
return index_points
Many thanks!
I'm using lucas-kanade opencv implementation to track objects between frames. I want to be able to do the following two things:
Calculate the distance moved by each point between frames
Track bounding boxes for each object across frames
I have obtained the features to track using cv2.goodFeaturesToTrack(). I also add the bounding boxes of objects to the features to be tracked. Right now I am using the following to calculate distance between the points
np.sqrt(np.square(new_pts - old_pts).sum(axis=1).sum(axis=1)). I am not quite sure if this is the correct way to do this because the indices of the points might be different in the new_pts.
Is the assumption that every index in old_pts corresponds to the same feature in new_pts array correct?
Secondly, is there a way to track bounding boxes across frames using lucas kanade?
In new_pts points have the same index. But they can be not founded - see to the status array: if status[i] == 1 then new_pts[i] contains a new coordinates of the old_pts[i].
For the more robustness it can to search direct flow (goodFeaturesToTrack(frame1) -> LK flow), backward flow (goodFeaturesToTrack(frame2) -> LK flow) and leave the points whose coordinates are equal in both directions.
I have an image mask (differenceM) like this:
For every single white pixel (pixel value != 0), I want to calculate the minimum distance from that pixel to a set of points. The set of points are points on the external contour which will be stored as a numpy array of [x_val y_val]. I was thinking of doing this:
...
def calcMinDist(dilPoints):
...
#returns 2d array (same shape as image)
def allMinDistDil(dilMask):
dilPoints = getPoints(dilMask)
...
return arrayOfMinValues
#more code here
blkImg = np.zeros(maskImage.shape,dtype=np.uint8)
blkImg.fill(0)
img_out = np.where(differenceM,allMinDistDil(dilatedMask),blkImg)
....
But the problem with this is, in order to calculate minimum distance from a pixel point to a set of points (obtained from getPoints function), I'll need to pass in the pixel point (index?) as well. But (if my understanding is correct,) with this where function, it only checks for true and false values in the first parameter...So the way I wrote the np.where() function won't work.
I've considered using nested for loops for this problem but I'm trying to avoid using for loops because I have a lot of images to process.
May I ask for suggestions to solve this? Any help would be greatly appreciated!
(not enough rep to comment) As for distance you probably want scipy.spatial.distance.cdist( X, Y ) . You can calculate a minimum distance with as simple as:
from scipy.spatial import distance
def min_distance(points, set_of_points):
return distance.cdist(np.atleast_1d(point), set_of_points).min()
As for np.where can you provide a bit more on your data structure? Most of the times a simple boolean mask will do the job...
Instead of using the np.where() function to find the specific pixels that are not zero, I applied:
diffMaskNewArray = np.transpose(np.nonzero(binaryThreshMask))
to get the points where the values are not zeros. With this array of points, I iterated through each point in this array and compared it with the array of boundary points of the masks and used:
shortestDistDil = np.amin(distance.cdist(a, b, 'euclidean'))
to find the min distance between the point and the set of boundary points.
I've interpolated a spline to fit pixel data from an image with a curve that I would like to straighten. I'm not sure what tools are appropriate to solve this problem. Can someone recommend an approach?
Here's how I'm getting my spline:
import numpy as np
from skimage import io
from scipy import interpolate
import matplotlib.pyplot as plt
from sklearn.neighbors import NearestNeighbors
import networkx as nx
# Read a skeletonized image, return an array of points on the skeleton, and divide them into x and y coordinates
skeleton = io.imread('skeleton.png')
curvepoints = np.where(skeleton==False)
xpoints = curvepoints[1]
ypoints = -curvepoints[0]
# reformats x and y coordinates into a 2-dimensional array
inputarray = np.c_[xpoints, ypoints]
# runs a nearest neighbors algorithm on the coordinate array
clf = NearestNeighbors(2).fit(inputarray)
G = clf.kneighbors_graph()
T = nx.from_scipy_sparse_matrix(G)
# sorts coordinates according to their nearest neighbors order
order = list(nx.dfs_preorder_nodes(T, 0))
xx = xpoints[order]
yy = ypoints[order]
# Loops over all points in the coordinate array as origin, determining which results in the shortest path
paths = [list(nx.dfs_preorder_nodes(T, i)) for i in range(len(inputarray))]
mindist = np.inf
minidx = 0
for i in range(len(inputarray)):
p = paths[i] # order of nodes
ordered = inputarray[p] # ordered nodes
# find cost of that order by the sum of euclidean distances between points (i) and (i+1)
cost = (((ordered[:-1] - ordered[1:])**2).sum(1)).sum()
if cost < mindist:
mindist = cost
minidx = i
opt_order = paths[minidx]
xxx = xpoints[opt_order]
yyy = ypoints[opt_order]
# fits a spline to the ordered coordinates
tckp, u = interpolate.splprep([xxx, yyy], s=3, k=2, nest=-1)
xpointsnew, ypointsnew = interpolate.splev(np.linspace(0,1,270), tckp)
# prints spline variables
print(tckp)
# plots the spline
plt.plot(xpointsnew, ypointsnew, 'r-')
plt.show()
My broader project is to follow the approach outlined in A novel method for straightening curved text-lines in stylistic documents. That article is reasonably detailed in finding the line that describes curved text, but much less so where straightening the curve is concerned. I have trouble visualizing the only reference to straightening that I see is in the abstract:
find the angle between the normal at a point on the curve and the vertical line, and finally visit each point on the text and rotate by their corresponding angles.
I also found Geometric warp of image in python, which seems promising. If I could rectify the spline, I think that would allow me to set a range of target points for the affine transform to map to. Unfortunately, I haven't found an approach to rectify my spline and test it.
Finally, this program implements an algorithm to straighten splines, but the paper on the algorithm is behind a pay wall and I can't make sense of the javascript.
Basically, I'm lost and in need of pointers.
Update
The affine transformation was the only approach I had any idea how to start exploring, so I've been working on that since I posted. I generated a set of destination coordinates by performing an approximate rectification of the curve based on the euclidean distance between points on my b-spline.
From where the last code block left off:
# calculate euclidian distances between adjacent points on the curve
newcoordinates = np.c_[xpointsnew, ypointsnew]
l = len(newcoordinates) - 1
pointsteps = []
for index, obj in enumerate(newcoordinates):
if index < l:
ord1 = np.c_[newcoordinates[index][0], newcoordinates[index][1]]
ord2 = np.c_[newcoordinates[index + 1][0], newcoordinates[index + 1][1]]
length = spatial.distance.cdist(ord1, ord2)
pointsteps.append(length)
# calculate euclidian distance between first point and each consecutive point
xpositions = np.asarray(pointsteps).cumsum()
# compose target coordinates for the line after the transform
targetcoordinates = [(0,0),]
for element in xpositions:
targetcoordinates.append((element, 0))
# perform affine transformation with newcoordinates as control points and targetcoordinates as target coordinates
tform = PiecewiseAffineTransform()
tform.estimate(newcoordinates, targetcoordinates)
I'm presently hung up on errors with the affine transform (scipy.spatial.qhull.QhullError: QH6154 Qhull precision error: Initial simplex is flat (facet 1 is coplanar with the interior point)
), but I'm not sure whether it's because of a problem with how I'm feeding the data in, or because I'm abusing the transform to do my projection.
I got the same error with you when using scipy.spatial.ConvexHull.
First, let me explain my project: what i wanted to do is to segment the people from its background(image matting). In my code, first I read an image and a trimap, then according to the trimap, I segment the original image to foreground, bakground and unknown pixels. Here is part of the coed:
img = scipy.misc.imread('sweater_black.png') #color_image
trimap = scipy.misc.imread('sw_trimap.png', flatten='True') #trimap
bg = trimap == 0 #background
fg = trimap == 255 #foreground
unknown = True ^ np.logical_or(fg,bg) #unknown pixels
fg_px = img[fg] #here i got the rgb value of the foreground pixels,then send them to the ConvexHull
fg_hull = scipy.spatial.ConvexHull(fg_px)
But i got an error here.So I check the Array of fg_px and then I found this array is n*4. which means every scalar i send to ConvexHull has four values. Howerver, the input of ConvexHUll should be 3 dimension.
I source my error and found that the input color image is 32bits(rgb channel and alpha channel) which means it has an alpha channel. After transferring the image to 24 bit (which means only rgb channels), the code works.
In one sentence, the input of ConvexHull should be b*4, so check your input data! Hope this works for you~
I am doing my best to replicate the algorithm described here in this paper for making an inpainting algorithm. The idea is to get the contour or edge points of the part of the image that needs to be inpainted. In order to find the most linear point in the region, the orthogonal normal vector is found. On page 6, a short description of the implementation is given.
In our implementation the contour
δΩ of the target region is modelled as a dense list of image point
locations. Given a point p ∈ δΩ, the normal direction np
is computed as follows: i) the positions of the
“control” points of δΩ are filtered via a bi-dimensional Gaussian
kernel and, ii) np is estimated as the unit vector orthogonal to
the line through the preceding and the successive points in the
list.
So it appears that I need to put all these points in a gaussian filter. How do I set up a bi-dimensional Gaussian filter when we have a single dimension or a list of points?
Lets say our contour is a box shape at points, then I create a 1 dimensional list of points: [1,1],[1,2],[1,3],[2,1],[2,3],[3,1],[3,2],[3,3]. Do I need to simply make a new 2d matrix table and put the points in and leave the middle point at [2,2] empty, then run a Gaussian filter on it? This doesn't seem very dense though.
I am trying to run this through python libraries.
a dense list of image points
is simply a line.
You are basically applying a gaussian filter to a black and white image where the line is black and background is white, from what I understand. I think by doing that, they approximate the curve model fitting.
Convolve all of the points in the 2D region surrounding the point and then overwrite the point with the result.
This will make any curve on the edge of the target region less sharp, lowering the noise in the calculation of the normal, which would be the vector orthogonal to the two points that surround the current one.