Related
I'm writing for Android with OpenCV. I'm segmenting an image similar to below using marker-controlled watershed, without the user manually marking the image. I'm planning to use the regional maxima as markers.
minMaxLoc() would give me the value, but how can I restrict it to the blobs which is what I'm interested in? Can I utilize the results from findContours() or cvBlob blobs to restrict the ROI and apply maxima to each blob?
First of all: the function minMaxLoc finds only the global minimum and global maximum for a given input, so it is mostly useless for determining regional minima and/or regional maxima. But your idea is right, extracting markers based on regional minima/maxima for performing a Watershed Transform based on markers is totally fine. Let me try to clarify what is the Watershed Transform and how you should correctly use the implementation present in OpenCV.
Some decent amount of papers that deal with watershed describe it similarly to what follows (I might miss some detail, if you are unsure: ask). Consider the surface of some region you know, it contains valleys and peaks (among other details that are irrelevant for us here). Suppose below this surface all you have is water, colored water. Now, make holes in each valley of your surface and then the water starts to fill all the area. At some point, differently colored waters will meet, and when this happen, you construct a dam such that they don't touch each other. In the end you have a collection of dams, which is the watershed separating all the different colored water.
Now, if you make too many holes in that surface, you end up with too many regions: over-segmentation. If you make too few you get an under-segmentation. So, virtually any paper that suggests using watershed actually presents techniques to avoid these problems for the application the paper is dealing with.
I wrote all this (which is possibly too naïve for anyone that knows what the Watershed Transform is) because it reflects directly on how you should use watershed implementations (which the current accepted answer is doing in a completely wrong manner). Let us start on the OpenCV example now, using the Python bindings.
The image presented in the question is composed of many objects that are mostly too close and in some instances overlapping. The usefulness of watershed here is to separate correctly these objects, not to group them into a single component. So you need at least one marker for each object and good markers for the background. As an example, first binarize the input image by Otsu and perform a morphological opening for removing small objects. The result of this step is shown below in the left image. Now with the binary image consider applying the distance transform to it, result at right.
With the distance transform result, we can consider some threshold such that we consider only the regions most distant to the background (left image below). Doing this, we can obtain a marker for each object by labeling the different regions after the earlier threshold. Now, we can also consider the border of a dilated version of the left image above to compose our marker. The complete marker is shown below at right (some markers are too dark to be seen, but each white region in the left image is represented at the right image).
This marker we have here makes a lot of sense. Each colored water == one marker will start to fill the region, and the watershed transformation will construct dams to impede that the different "colors" merge. If we do the transform, we get the image at left. Considering only the dams by composing them with the original image, we get the result at right.
import sys
import cv2
import numpy
from scipy.ndimage import label
def segment_on_dt(a, img):
border = cv2.dilate(img, None, iterations=5)
border = border - cv2.erode(border, None)
dt = cv2.distanceTransform(img, 2, 3)
dt = ((dt - dt.min()) / (dt.max() - dt.min()) * 255).astype(numpy.uint8)
_, dt = cv2.threshold(dt, 180, 255, cv2.THRESH_BINARY)
lbl, ncc = label(dt)
lbl = lbl * (255 / (ncc + 1))
# Completing the markers now.
lbl[border == 255] = 255
lbl = lbl.astype(numpy.int32)
cv2.watershed(a, lbl)
lbl[lbl == -1] = 0
lbl = lbl.astype(numpy.uint8)
return 255 - lbl
img = cv2.imread(sys.argv[1])
# Pre-processing.
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
_, img_bin = cv2.threshold(img_gray, 0, 255,
cv2.THRESH_OTSU)
img_bin = cv2.morphologyEx(img_bin, cv2.MORPH_OPEN,
numpy.ones((3, 3), dtype=int))
result = segment_on_dt(img, img_bin)
cv2.imwrite(sys.argv[2], result)
result[result != 255] = 0
result = cv2.dilate(result, None)
img[result == 255] = (0, 0, 255)
cv2.imwrite(sys.argv[3], img)
I would like to explain a simple code on how to use watershed here. I am using OpenCV-Python, but i hope you won't have any difficulty to understand.
In this code, I will be using watershed as a tool for foreground-background extraction. (This example is the python counterpart of the C++ code in OpenCV cookbook). This is a simple case to understand watershed. Apart from that, you can use watershed to count the number of objects in this image. That will be a slightly advanced version of this code.
1 - First we load our image, convert it to grayscale, and threshold it with a suitable value. I took Otsu's binarization, so it would find the best threshold value.
import cv2
import numpy as np
img = cv2.imread('sofwatershed.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
ret,thresh = cv2.threshold(gray,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
Below is the result I got:
( even that result is good, because great contrast between foreground and background images)
2 - Now we have to create the marker. Marker is the image with same size as that of original image which is 32SC1 (32 bit signed single channel).
Now there will be some regions in the original image where you are simply sure, that part belong to foreground. Mark such region with 255 in marker image. Now the region where you are sure to be the background are marked with 128. The region you are not sure are marked with 0. That is we are going to do next.
A - Foreground region:- We have already got a threshold image where pills are white color. We erode them a little, so that we are sure remaining region belongs to foreground.
fg = cv2.erode(thresh,None,iterations = 2)
fg :
B - Background region :- Here we dilate the thresholded image so that background region is reduced. But we are sure remaining black region is 100% background. We set it to 128.
bgt = cv2.dilate(thresh,None,iterations = 3)
ret,bg = cv2.threshold(bgt,1,128,1)
Now we get bg as follows :
C - Now we add both fg and bg :
marker = cv2.add(fg,bg)
Below is what we get :
Now we can clearly understand from above image, that white region is 100% foreground, gray region is 100% background, and black region we are not sure.
Then we convert it into 32SC1 :
marker32 = np.int32(marker)
3 - Finally we apply watershed and convert result back into uint8 image:
cv2.watershed(img,marker32)
m = cv2.convertScaleAbs(marker32)
m :
4 - We threshold it properly to get the mask and perform bitwise_and with the input image:
ret,thresh = cv2.threshold(m,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
res = cv2.bitwise_and(img,img,mask = thresh)
res :
Hope it helps!!!
ARK
Foreword
I'm chiming in mostly because I found both the watershed tutorial in the OpenCV documentation (and C++ example) as well as mmgp's answer above to be quite confusing. I revisited a watershed approach multiple times to ultimately give up out of frustration. I finally realized I needed to at least give this approach a try and see it in action. This is what I've come up with after sorting out all of the tutorials I've come across.
Aside from being a computer vision novice, most of my trouble probably had to do with my requirement to use the OpenCVSharp library rather than Python. C# doesn't have baked-in high-power array operators like those found in NumPy (though I realize this has been ported via IronPython), so I struggled quite a bit in both understanding and implementing these operations in C#. Also, for the record, I really despise the nuances of, and inconsistencies in most of these function calls. OpenCVSharp is one of the most fragile libraries I've ever worked with. But hey, it's a port, so what was I expecting? Best of all, though -- it's free.
Without further ado, let's talk about my OpenCVSharp implementation of the watershed, and hopefully clarify some of the stickier points of watershed implementation in general.
Application
First of all, make sure watershed is what you want and understand its use. I am using stained cell plates, like this one:
It took me a good while to figure out I couldn't just make one watershed call to differentiate every cell in the field. On the contrary, I first had to isolate a portion of the field, then call watershed on that small portion. I isolated my region of interest (ROI) via a number of filters, which I will explain briefly here:
Start with source image (left, cropped for demonstration purposes)
Isolate the red channel (left middle)
Apply adaptive threshold (right middle)
Find contours then eliminate those with small areas (right)
Once we have cleaned the contours resulting from the above thresholding operations, it is time to find candidates for watershed. In my case, I simply iterated through all contours greater than a certain area.
Code
Say we've isolated this contour from the above field as our ROI:
Let's take a look at how we'll code up a watershed.
We'll start with a blank mat and draw only the contour defining our ROI:
var isolatedContour = new Mat(source.Size(), MatType.CV_8UC1, new Scalar(0, 0, 0));
Cv2.DrawContours(isolatedContour, new List<List<Point>> { contour }, -1, new Scalar(255, 255, 255), -1);
In order for the watershed call to work, it will need a couple of "hints" about the ROI. If you're a complete beginner like me, I recommend checking out the CMM watershed page for a quick primer. Suffice to say we're going to create hints about the ROI on the left by creating the shape on the right:
To create the white part (or "background") of this "hint" shape, we'll just Dilate the isolated shape like so:
var kernel = Cv2.GetStructuringElement(MorphShapes.Ellipse, new Size(2, 2));
var background = new Mat();
Cv2.Dilate(isolatedContour, background, kernel, iterations: 8);
To create the black part in the middle (or "foreground"), we'll use a distance transform followed by threshold, which takes us from the shape on the left to the shape on the right:
This takes a few steps, and you may need to play around with the lower bound of your threshold to get results that work for you:
var foreground = new Mat(source.Size(), MatType.CV_8UC1);
Cv2.DistanceTransform(isolatedContour, foreground, DistanceTypes.L2, DistanceMaskSize.Mask5);
Cv2.Normalize(foreground, foreground, 0, 1, NormTypes.MinMax); //Remember to normalize!
foreground.ConvertTo(foreground, MatType.CV_8UC1, 255, 0);
Cv2.Threshold(foreground, foreground, 150, 255, ThresholdTypes.Binary);
Then we'll subtract these two mats to get the final result of our "hint" shape:
var unknown = new Mat(); //this variable is also named "border" in some examples
Cv2.Subtract(background, foreground, unknown);
Again, if we Cv2.ImShow unknown, it would look like this:
Nice! This was easy for me to wrap my head around. The next part, however, got me quite puzzled. Let's look at turning our "hint" into something the Watershed function can use. For this we need to use ConnectedComponents, which is basically a big matrix of pixels grouped by the virtue of their index. For example, if we had a mat with the letters "HI", ConnectedComponents might return this matrix:
0 0 0 0 0 0 0 0 0
0 1 0 1 0 2 2 2 0
0 1 0 1 0 0 2 0 0
0 1 1 1 0 0 2 0 0
0 1 0 1 0 0 2 0 0
0 1 0 1 0 2 2 2 0
0 0 0 0 0 0 0 0 0
So, 0 is the background, 1 is the letter "H", and 2 is the letter "I". (If you get to this point and want to visualize your matrix, I recommend checking out this instructive answer.) Now, here's how we'll utilize ConnectedComponents to create the markers (or labels) for watershed:
var labels = new Mat(); //also called "markers" in some examples
Cv2.ConnectedComponents(foreground, labels);
labels = labels + 1;
//this is a much more verbose port of numpy's: labels[unknown==255] = 0
for (int x = 0; x < labels.Width; x++)
{
for (int y = 0; y < labels.Height; y++)
{
//You may be able to just send "int" in rather than "char" here:
var labelPixel = (int)labels.At<char>(y, x); //note: x and y are inexplicably
var borderPixel = (int)unknown.At<char>(y, x); //and infuriatingly reversed
if (borderPixel == 255)
labels.Set(y, x, 0);
}
}
Note that the Watershed function requires the border area to be marked by 0. So, we've set any border pixels to 0 in the label/marker array.
At this point, we should be all set to call Watershed. However, in my particular application, it is useful just to visualize a small portion of the entire source image during this call. This may be optional for you, but I first just mask off a small bit of the source by dilating it:
var mask = new Mat();
Cv2.Dilate(isolatedContour, mask, new Mat(), iterations: 20);
var sourceCrop = new Mat(source.Size(), source.Type(), new Scalar(0, 0, 0));
source.CopyTo(sourceCrop, mask);
And then make the magic call:
Cv2.Watershed(sourceCrop, labels);
Results
The above Watershed call will modify labels in place. You'll have to go back to remembering about the matrix resulting from ConnectedComponents. The difference here is, if watershed found any dams between watersheds, they will be marked as "-1" in that matrix. Like the ConnectedComponents result, different watersheds will be marked in a similar fashion of incrementing numbers. For my purposes, I wanted to store these into separate contours, so I created this loop to split them up:
var watershedContours = new List<Tuple<int, List<Point>>>();
for (int x = 0; x < labels.Width; x++)
{
for (int y = 0; y < labels.Height; y++)
{
var labelPixel = labels.At<Int32>(y, x); //note: x, y switched
var connected = watershedContours.Where(t => t.Item1 == labelPixel).FirstOrDefault();
if (connected == null)
{
connected = new Tuple<int, List<Point>>(labelPixel, new List<Point>());
watershedContours.Add(connected);
}
connected.Item2.Add(new Point(x, y));
if (labelPixel == -1)
sourceCrop.Set(y, x, new Vec3b(0, 255, 255));
}
}
Then, I wanted to print these contours with random colors, so I created the following mat:
var watershed = new Mat(source.Size(), MatType.CV_8UC3, new Scalar(0, 0, 0));
foreach (var component in watershedContours)
{
if (component.Item2.Count < (labels.Width * labels.Height) / 4 && component.Item1 >= 0)
{
var color = GetRandomColor();
foreach (var point in component.Item2)
watershed.Set(point.Y, point.X, color);
}
}
Which yields the following when shown:
If we draw on the source image the dams that were marked by a -1 earlier, we get this:
Edits:
I forgot to note: make sure you're cleaning up your mats after you're done with them. They WILL stay in memory and OpenCVSharp may present with some unintelligible error message. I should really be using using above, but mat.Release() is an option as well.
Also, mmgp's answer above includes this line: dt = ((dt - dt.min()) / (dt.max() - dt.min()) * 255).astype(numpy.uint8), which is a histogram stretching step applied to the results of the distance transform. I omitted this step for a number of reasons (mostly because I didn't think the histograms I saw were too narrow to begin with), but your mileage may vary.
I have two images:
I want to measure how straight/smooth the text borders are rendered.
First image is rendered perfectly straight, so it deserves a quality measure 1. On the other hand, the second image is rendered with a lot of variant curves (rough in a way) that is why it deserves a quality measure less than 1. How will I measure it using image processing or any Python function or any function written in other languages?
Clarification :
There are font styles that are rendered originally with straight strokes but there are also font styles that are rendered smoothly just like the cursive font styles. What I'm really after is to differentiate the text border surface roughness of the characters by giving it a quality measure.
I want to measure how straight/smooth the text borders are rendered in an image.
Inversely, it can also be said that I want to measure how rough the text borders are rendered in an image.
I don't know any python function, but I would:
1) Use potrace to trace the edges and convert them to bezier curves. Here's a vizualisation:
2) Then let's zoom to the top part of the P for example:
You draw lines perpendicular to the curve for a finite length (let's say 100 pixels). You plot the color intensity (you can convert to HSI or HSV and use one of those channels, or just convert to grayscale and take the pixel value directly) over that line:
3) Then you calculate the standard deviation of the derivative. Small standard deviation means sharp edges, large standard deviation means blurry edges. For a perfect edge, the standard deviation would be zero.
4) For every edge were you drew a perpendicular line, you now have a "smoothness" value. You can then average all the smoothness values per edge, per letter, per word or per image, as you see fit. Also, the more perpendicular lines you draw, the more accurate your smoothness value, but the more computationally intensive.
I would try something simple like creating a 'roughness' metric using a few functions from the opencv library, since it's easy to work with in Python (and C++, as well as other wrappers).
For example (without actual source, since I'm typing on my phone):
Preprocess to create binary images (many standard ways).
Use cv2.findContours to get outlines of the letters.
Use cv2.arcLength on each contour as denominators.
Use cv2.approxPolyDP to simplify each contour.
Use cv2.arcLength on each simplified contour as numerators.
Calculate ratios of simplified over full arc lengths.
In step 5, ratios closer to 1.0 require less simplification, so they're presumably less rough. Ratios closer to 0.0 require a lot of simplification, and are therefore probably very rough. Of course, you'll have to tweak the contour finding code to get appropriate outlines to work with, and you'll need to manage numerical precision to keep the math calculations meaningful, but hopefully the idea is clear enough.
OpenCV also has the useful functions cv2.convexHull and cv2.convexityDefects that you might find interesting in related work. However, they didn't seem appropriate for the letters here, since internal features on letters like M for example would be more challenging to address.
Speaking of rough things, I admit this algorithmic outline is incredibly rough! However, I hope it gives you a useful idea to try that seems straightforward to implement quickly to start getting quantitative feedback.
One idea might be simply to get the average of the number of vertices per character in Python/OpenCV using cv2.CHAIN_APPROX_SIMPLE.
Since you have the same characters and you want to know how straight they are, the CHAIN_APPROX_SIMPLE measures only horizontal and vertical corner vertices. For your first image, there should be much fewer vertices than for your second image.
CHAIN_APPROX_SIMPLE compresses horizontal, vertical, and diagonal
segments and leaves only their end points. For example, an up-right
rectangular contour is encoded with 4 points.
import cv2
import numpy as np
# read image
img = cv2.imread('lemper1.png')
#img = cv2.imread('lemper2.png')
# convert to grayscale
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
# threshold
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)[1]
# invert
thresh = 255 - thresh
# get contours and compute average number of vertices per character (contour)
result = img.copy()
contours = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
contours = contours[0] if len(contours) == 2 else contours[1]
num_contours = 0
sum = 0
for cntr in contours:
cv2.drawContours(result, [cntr], 0, (0,0,255), 1)
num_vertices = len(cntr)
sum = sum + num_vertices
num_contours = num_contours + 1
smoothness = (sum / num_contours)
print(smoothness)
# save resulting images
cv2.imwrite('lemper1_contours.png',result)
#cv2.imwrite('lemper2_contours.png',result)
# show thresh and result
cv2.imshow("thresh", thresh)
cv2.imshow("contours", result)
cv2.waitKey(0)
cv2.destroyAllWindows()
First image average number of vertices: 49.666666666666664
Second image average number of vertices: 179.14285714285714
So smaller number of vertices means straighter characters.
Preface
There's a few nice ideas presented here based around the properties of the character contours as polylines. Whilst there is some inherent flaws in this approach due to them being a function of resolution and scale, I would like to offer one further interruption of the same. My algorithm is still susceptible but it may offer a different perspective.
Theory
The method I propose is to compare common characters by the number of inflections in their contours. In this context, what I mean by inflection, is a sign change between the cross products of successive polyline segments as vector. For example; consider a polyline contour of a circle, starting at the mid y coordinate and the x+ most coordinate. If we were to trace the polyline contour CW (clockwise) around the perimeter, each line segment would be incrementally a CW transform of the prior. If at any time a segment turned "away" or "outwards", this transform would be CCW (counter-clockwise) and the cross product will invert. A "rough" circle will therefore have inflections, a "perfect" or "smooth" circle will have none.
Algorithm
The algorithm follows the steps below using the Emgu.CV. C# code below that:
The images are loaded and converted to binary by means of thresholding
The binary images then undergo contour detection and these contours are sort by their bounding box, left to right, so and their indices match the occurrence order of the character they contour.
Each contour is then re-pointed to an equal number of segments in order to normalize for scale and resolution differences between images/characters.
Each contour is "walked" and its number of inflections counted.
// [Some basic extensions are omitted for clarity]
// Load the images
Image<Rgb, byte> baseLineImage = new Image<Rgb, byte>("BaseLine.png");
Image<Rgb, byte> testCaseImage = new Image<Rgb, byte>("TestCase.png");
// Convert them to Gray Scale
Image<Gray, byte> baseLineGray = baseLineImage.Convert<Gray, byte>();
Image<Gray, byte> testCaseGray = testCaseImage.Convert<Gray, byte>();
// Threshold the images to binary
Image<Gray, byte> baseLineBinary = baseLineGray.ThresholdBinaryInv(new Gray(100), new Gray(255));
Image<Gray, byte> testCaseBinary = testCaseGray.ThresholdBinaryInv(new Gray(100), new Gray(255));
// Some dilation required on the test image so that the characters are continuous
testCaseBinary = testCaseBinary.Dilate(3);
// Extract the the contours from the images to isolate the character profiles
// and sort them left to right so as the indicies match the character order
VectorOfVectorOfPoint baseLineContours = new VectorOfVectorOfPoint();
Mat baseHierarchy = new Mat();
CvInvoke.FindContours(
baseLineBinary,
baseLineContours,
baseHierarchy,
RetrType.External,
ChainApproxMethod.ChainApproxSimple);
var baseLineContoursList = baseLineContours.ToList();
baseLineContoursList.Sort(new ContourComparer());
VectorOfVectorOfPoint testCaseContours = new VectorOfVectorOfPoint();
Mat testHierarchy = new Mat();
CvInvoke.FindContours(
testCaseBinary,
testCaseContours,
testHierarchy,
RetrType.External,
ChainApproxMethod.ChainApproxSimple);
var testCaseContoursList = testCaseContours.ToList();
testCaseContoursList.Sort(new ContourComparer());
var baseLineRepointedContours = RepointContours(baseLineContoursList, 50);
var testCaseRepointedContours = RepointContours(testCaseContoursList, 50);
var baseLineInflectionCounts = GetContourInflections(baseLineRepointedContours);
var testCaseInflectionCounts = GetContourInflections(testCaseRepointedContours);
Inflection Detection/Counting
static List<List<Point>> GetContourInflections(List<VectorOfPoint> contours)
{
// A resultant list to return the inflection points
List<List<Point>> result = new List<List<Point>>();
// Calculate the forward to reverse cross product at each vertex
List<double> crossProducts;
// Points used to store 2D Vectors as X,Y (I,J)
Point priorVector, forwardVector;
foreach (VectorOfPoint contour in contours)
{
crossProducts = new List<double>();
for (int p = 0; p < contour.Size; p++)
{
// Determine the vector to the prior to this vertex
priorVector = p == 0 ?
priorVector = new Point()
{
X = contour[p].X - contour[contour.Size - 1].X,
Y = contour[p].Y - contour[contour.Size - 1].Y
} :
priorVector = new Point()
{
X = contour[p].X - contour[p - 1].X,
Y = contour[p].Y - contour[p - 1].Y
};
// Determine the vector to the next vector
// If this is the lst vertex, loop back to vertex 0
forwardVector = p == contour.Size - 1 ?
new Point()
{
X = contour[0].X - contour[p].X,
Y = contour[0].Y - contour[p].Y,
} :
new Point()
{
X = contour[p + 1].X - contour[p].X,
Y = contour[p + 1].Y - contour[p].Y,
};
// Calculate the cross product of the prior and forward vectors
crossProducts.Add(forwardVector.X * priorVector.Y - forwardVector.Y * priorVector.X);
}
// Given the calculated cross products, detect the inflection points
List<Point> inflectionPoints = new List<Point>();
for (int p = 1; p < contour.Size; p++)
{
// If there is a sign change between this and the prior cross product, an inflection,
// or change from CW to CCW bearing increments has occurred. To and from zero products
// are ignored
if ((crossProducts[p] > 0 && crossProducts[p-1] < 0) ||
(crossProducts[p] < 0 && crossProducts[p-1] > 0))
{
inflectionPoints.Add(contour[p]);
}
}
result.Add(inflectionPoints);
}
return result;
}
Output
L: Baseline Inflections:0 Testcase Inflections:22
E: Baseline Inflections:1 Testcase Inflections:16
M: Baseline Inflections:4 Testcase Inflections:15
P: Baseline Inflections:11 Testcase Inflections:17
E: Baseline Inflections:1 Testcase Inflections:10
R: Baseline Inflections:9 Testcase Inflections:16
Contours (Blue) and Inflections (Red)
I've performed some basic image operations and managed to isolate the object I am interested in.
As a result, I ended up with a container; get<0>(results) holding the corresponding (x, y) of the object. For visual purposes, I drew these points on a different frame named contourFrame
My question is, how do I map these points back to the original image?
I have looked into the combination of findHomography() and warpPerspective but in order to find the homography matrix, I need to provide respective destination points which is what I am looking for in the first place.
remap() seems to do what I am looking for but I think its a misunderstanding of the API on my part but after trying it out, it returns a blank white frame.
Is remap() even the way to go about this? If yes, how? If no, what other method can I use to map my contour points to another image?
A Python solution/suggestion works with me as well.
Edit I
For the current scenario, its a 1-1 mapping back to the original image. But what if I rotate the original image? Or move my camera back and forth i.e. move it closer and further away? How can I still map back the coordinates to the newly oriented original image?
Original Image
Results
Top Left: Canny Edge
Top Right: Dilated
Bottom Left: Eroded
Bottom Right: Frame with desired object/contour
tuple<vector<vector<Point>>, Mat, Mat> removeStupidIcons(Mat& edges)
{
Mat dilated, eroded;
vector<vector<Point>> contours, filteredContours;
dilate(edges, dilated, Mat(), Point(-1, -1), 5);
erode(dilated, eroded, Mat(), Point(-1, -1), 5);
findContours(eroded, contours, CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE);
for(vector<Point>contour: contours)
if(contourArea(contour) < 200)
filteredContours.push_back(contour);
return make_tuple(filteredContours, dilated, eroded);
}
Mat mapPoints(Mat& origin, Mat& destination)
{
Mat remapWindow, mapX, mapY;
mapX.create(origin.size(), CV_32FC1);
mapY.create(origin.size(), CV_32FC1);
remap(origin, destination, mapX, mapY, CV_INTER_LINEAR);
return destination;
}
int main(int argc, const char * argv[])
{
string image_path = "ipad.jpg";
original = imread(image_path);
blur(original, originalBlur, Size(15, 15));
cvtColor(originalBlur, originalGrey, CV_RGB2GRAY);
Canny(originalGrey, cannyEdges, 50, 130, 3);
cannyEdges.convertTo(cannyFrame, CV_8U);
tuple<vector<vector<Point>>, Mat, Mat> results = removeStupidIcons(cannyEdges);
//
// get<0>(results) -> contours
// get<1>(results) -> matrix after dilation
// get<2>(results) -> matrix after erosion
Mat contourFrame = Mat::zeros(original.size(), CV_8UC3);
Scalar colour = Scalar(rand()&255, rand()&255, rand()&255);
drawContours(contourFrame, get<0>(results), -1, colour, 3);
Mat contourCopy, originalCopy;
original.copyTo(originalCopy); contourFrame.copyTo(contourCopy);
// a white background is returned.
Mat mappedFrame = mapPoints(originalCopy, contourCopy);
imshow("Canny", cannyFrame);
imshow("Original", original);
imshow("Contour", contourFrame);
imshow("Eroded", get<2>(results));
imshow("Dilated", get<1>(results));
imshow("Original Grey", originalGrey);
imshow("Mapping Result", contourCopy);
waitKey(0);
return 0;
}
I've been trying to blend two images. The current approach I'm taking is, I obtain the coordinates of the overlapping region of the two images, and only for the overlapping regions, I blend with a hardcoded alpha of 0.5, before adding it. SO basically I'm just taking half the value of each pixel from overlapping regions of both the images, and adding them. That doesn't give me a perfect blend because the alpha value is hardcoded to 0.5. Here's the result of blending of 3 images:
As you can see, the transition from one image to another is still visible. How do I obtain the perfect alpha value that would eliminate this visible transition? Or is there no such thing, and I'm taking a wrong approach?
Here's how I'm currently doing the blending:
for i in range(3):
base_img_warp[overlap_coords[0], overlap_coords[1], i] = base_img_warp[overlap_coords[0], overlap_coords[1],i]*0.5
next_img_warp[overlap_coords[0], overlap_coords[1], i] = next_img_warp[overlap_coords[0], overlap_coords[1],i]*0.5
final_img = cv2.add(base_img_warp, next_img_warp)
If anyone would like to give it a shot, here are two warped images, and the mask of their overlapping region: http://imgur.com/a/9pOsQ
Here is the way I would do it in general:
int main(int argc, char* argv[])
{
cv::Mat input1 = cv::imread("C:/StackOverflow/Input/pano1.jpg");
cv::Mat input2 = cv::imread("C:/StackOverflow/Input/pano2.jpg");
// compute the vignetting masks. This is much easier before warping, but I will try...
// it can be precomputed, if the size and position of your ROI in the image doesnt change and can be precomputed and aligned, if you can determine the ROI for every image
// the compression artifacts make it a little bit worse here, I try to extract all the non-black regions in the images.
cv::Mat mask1;
cv::inRange(input1, cv::Vec3b(10, 10, 10), cv::Vec3b(255, 255, 255), mask1);
cv::Mat mask2;
cv::inRange(input2, cv::Vec3b(10, 10, 10), cv::Vec3b(255, 255, 255), mask2);
// now compute the distance from the ROI border:
cv::Mat dt1;
cv::distanceTransform(mask1, dt1, CV_DIST_L1, 3);
cv::Mat dt2;
cv::distanceTransform(mask2, dt2, CV_DIST_L1, 3);
// now you can use the distance values for blending directly. If the distance value is smaller this means that the value is worse (your vignetting becomes worse at the image border)
cv::Mat mosaic = cv::Mat(input1.size(), input1.type(), cv::Scalar(0, 0, 0));
for (int j = 0; j < mosaic.rows; ++j)
for (int i = 0; i < mosaic.cols; ++i)
{
float a = dt1.at<float>(j, i);
float b = dt2.at<float>(j, i);
float alpha = a / (a + b); // distances are not between 0 and 1 but this value is. The "better" a is, compared to b, the higher is alpha.
// actual blending: alpha*A + beta*B
mosaic.at<cv::Vec3b>(j, i) = alpha*input1.at<cv::Vec3b>(j, i) + (1 - alpha)* input2.at<cv::Vec3b>(j, i);
}
cv::imshow("mosaic", mosaic);
cv::waitKey(0);
return 0;
}
Basically you compute the distance from your ROI border to the center of your objects and compute the alpha from both blending mask values. So if one image has a high distance from the border and other one a low distance from border, you prefer the pixel that is closer to the image center. It would be better to normalize those values for cases where the warped images aren't of similar size.
But even better and more efficient is to precompute the blending masks and warp them. Best would be to know the vignetting of your optical system and choose and identical blending mask (typically lower values of the border).
From the previous code you'll get these results:
ROI masks:
Blending masks (just as an impression, must be float matrices instead):
image mosaic:
There are 2 obvious problems with your images:
Border area has distorted lighting conditions
That is most likely caused by the optics used to acquire images. So to remedy that you should use only inside part of the images (cut off few pixels from border.
So when cut off 20 pixels from the border and blending to common illumination I got this:
As you can see the ugly border seam is away now only the illumination problems persists (see bullet #2).
Images are taken at different lighting conditions
Here the subsurface scattering effects hits in making the images "not-compatible". You should normalize them to some uniform illumination or post process the blended result line by line and when coherent bump detected multiply the rest of line so the bump will be diminished.
So the rest of the line should be multiplied by constant i0/i1. These kind if bumps can occur only on the edges between overlap values so you can either scan for them or use those positions directly ... To recognize valid bump it should have neighbors nearby in previous and next lines along the whole image height.
You can do this also in y axis direction in the same way ...
On a recent set of images, my OpenCV code stopped finding the correct area of a contour. This appears to happen when the contour is not closed. I have tried to ensure the contour is closed to no avail.
Edit: The problem is that there are gaps in the contour.
Background:
I have a series of images of a capsule in a channel and I want to measure the area of the shape as well as the centroid from the moments.
Problem:
When the contour is not closed, the moments are wrong.
Edit: When I have gaps, the contour is not of the whole shape and hence the incorrect area.
What I do:
Read image -> img =cv2.imread(fileName,0)
apply Canny filter -> edges = cv2.Canny(img,lowerThreshold,lowerThreshold*2)
find contours -> contours, hierarchy = cv2.findContours(edges,cv2.cv.CV_RETR_LIST,cv2.cv.CV_CHAIN_APPROX_NONE)
find longest contour
ensure contour is closed
find moments -> cv2.moments(cnt)
A working example with test images can be found here.
There is a question regarding closing a contour but neither of the suggestions worked. Using cv2.approxPolyDP does not change the results, although it should return a closed contour. Adding the first point of the contour as the last, in order to make it closed, also does not resolve the issue.
An example of an image with the contour draw on it is below. Here, the area is determined as 85 while in an almost identical image it is 8660, which is what it should be.
Any advice would be appriciated.
Code:
img =cv2.imread(fileName,0)
edges = cv2.Canny(img,lowerThreshold,lowerThreshold*2)
contours, hierarchy = cv2.findContours(edges,cv2.cv.CV_RETR_LIST,cv2.cv.CV_CHAIN_APPROX_NONE) #cv2.cv.CV_CHAIN_APPROX_NONE or cv2.cv.CV_CHAIN_APPROX_SIMPLE
#Select longest contour as this should be the capsule
lengthC=0
ID=-1
idCounter=-1
for x in contours:
idCounter=idCounter+1
if len(x) > lengthC:
lengthC=len(x)
ID=idCounter
if ID != -1:
cnt = contours[ID]
cntFull=cnt.copy()
#approximate the contour, where epsilon is the distance to
#the original contour
cnt = cv2.approxPolyDP(cnt, epsilon=1, closed=True)
#add the first point as the last point, to ensure it is closed
lenCnt=len(cnt)
cnt= np.append(cnt, [[cnt[0][0][0], cnt[0][0][1]]])
cnt=np.reshape(cnt, (lenCnt+1,1, 2))
lenCntFull=len(cntFull)
cntFull= np.append(cntFull, [[cntFull[0][0][0], cntFull[0][0][1]]])
cntFull=np.reshape(cntFull, (lenCntFull+1,1, 2))
#find the moments
M = cv2.moments(cnt)
MFull = cv2.moments(cntFull)
print('Area = %.2f \t Area of full contour= %.2f' %(M['m00'], MFull['m00']))
My problem was, as #HugoRune pointed out, that there are gaps in the countour. The solution is to close the gaps.
I found it difficult to find a general method to close the gaps, so I iterativly change the threshold of the Canny filter and performing morphological closing until a closed contour is found.
For those struggeling with the same problem, there are several good answers how to close contours, such as this or this
Having dealt with a similar problem, an alternative solution (and arguably simpler with less overhead) is to use the morphology opening functionality, which performs an erosion followed by a dilation. If you turn this into a binary image first, perform the opening operation, and the do the Canny detection, that should do the same thing, but without having to iterate with the filter. The only thing you will have to do is play with the kernel size a couple times to identify an appropriate size without losing too much detail. I have found this to be a fairly robust way of making sure the contours are closed.
Morphological operations documentation
An alternate approach is to use the contour points to find the area. Here nContours has previously been found thru cvFindContours(). I have used MFC CArray here. You can use std::vector alternatively.
////////////////////////////////////////////
CvSeq* MasterContour = NULL;
if (nContours > 1)
{
// Find the biggest contour
for (int i = 0; i < nContours; i++)
{
CvRect rect = cvBoundingRect(m_contour, 1);
if (rect.width > rectMax.width)
MasterContour = m_contour;
if (m_contour->h_next != 0)
m_contour = m_contour->h_next;
else
break;
}
}
else
MasterContour = m_contour;
arOuterContourPoints.RemoveAll();
CArray<CPoint, CPoint> arOuterTrackerPoints;
for (int i = 0; i < MasterContour->total; i++)
{
CvPoint *pPt;
pPt = (CvPoint *)cvGetSeqElem(MasterContour, i);
arOuterContourPoints.Add(CPoint(pPt->x, pPt->y));
}
int nOuterArea = 0;
for (int i = 0; i < arOuterContourPoints.GetSize(); i++)
{
if (i == (arOuterContourPoints.GetSize() - 1))
nOuterArea += (arOuterContourPoints[i].x * arOuterContourPoints[0].y - arOuterContourPoints[0].x * arOuterContourPoints[i].y);
else
nOuterArea += (arOuterContourPoints[i].x * arOuterContourPoints[i+1].y - m_arOuterContourPoints[i+1].x * m_arOuterContourPoints[i].y);
}
nOuterAreaPix = abs(nOuterArea / 2.0);
/////////////////////////////////////////////////////////////