Develop Reference

Apply a brightness mask using Lab color space and Opencv to remove Vignetting

Good day everyone,
I'm trying to compensate the vignetting effect of a Basler camera using a brightness mask. First thing first, I have taken a picture of a white screen projected by a projector, to have an almost perfect white image with the vignetting effect visible on the angles.
The idea was to then invert the vignetting (so that white=255 become 0, so no change, and vice-versa). And everything works pretty well using Lab color space, apart from the fact that even if I use clipping (0-255) the luminosity channel seems to overflow and the brightest zone becoming dark.
Below an example and the code to reproduce, as well as the image and vignette I'm using right now. Apart from the overflow zone, the method seems to work well and the sides / angles get brighter as I want.
Problem demonstration
Download Vignette image
Download Sample image
import cv2
import numpy as np
if __name__ == "__main__":
image = cv2.imread("img/4mm_raw2.jpeg")
mask = cv2.imread("img/vignetting.png")
h, w, c = image.shape
# Convert to LAB color space
lab_img = cv2.cvtColor(image, cv2.COLOR_BGR2Lab)
lab_mask = cv2.cvtColor(mask, cv2.COLOR_BGR2Lab)
# Invert the vignetting (white no change, black increase brightness)
inv_mask = (255 - lab_mask[:,:,0])
# Add the vignetting contribution, clipping to the channel limit (0-255)
lab_img[:,:,0] = np.clip(lab_img[:,:,0] + inv_mask, 0, 255)
# Back to RGB
result = cv2.cvtColor(lab_img, cv2.COLOR_Lab2BGR)
# Resize for visualization
res_img = cv2.resize(image, (w//2, h//2))
res_dst = cv2.resize(result, (w//2, h//2))
stack = np.hstack((res_img, res_dst))
cv2.imshow('Difference', stack)
cv2.waitKey(0)
Using a double for loop and changing pixel per pixel, works fine, but it's too too slow, for the type of application I'm developing.
inv_mask = (255 - lab_mask[:,:,0])
for y in range(0, h):
for x in range(0, w):
new = lab_img.item(y, x, 0) + inv_mask.item(y, x)
lab_img[y, x, 0] = np.clip(new, 0, 255)
The question is, how can I solve this and also if there is a better way to achieve the same result (color spaces...)
Thanks,
Best
Solution:
My bad, I found the solution myself on the last try before give up.
The brightness channel needs to be converted to 16 bit otherwise it overflows before even been evaluated by Numpy clip.
a = lab_img[:,:,0].astype(np.int16) + inv_mask
lab_img[:,:,0] = np.clip(a,0,255)
However, I leave the question open for any advice or better methodologies. Thanks

Detect the path robot can take between crop rows

I am working on a problem where i need to find the path the robot can take without hitting any crop rows.Raw Image
My initial approach was to convert this into birds eye view and then use canny and skeletonize techniques.Then I applied Hough transform to come up with the crop rows.This works well when the rows are straight but if i rotate the image by 45 degrees I couldn't find any rows with Hough transform.So I decided to use another approach.
First I only selected the green region and applied morphological filters to remove small branches which come out
img = cv2.imread('''')
min_green2 = np.array([45, 50, 50])
max_green2 = np.array([75, 250, 250])
image_blur = cv2.GaussianBlur(img, (5, 5), 0)
image_blur_hsv = cv2.cvtColor(image_blur, cv2.COLOR_BGR2HSV)
image_green = cv2.inRange(image_blur_hsv, min_red2, max_red2)
se1 = cv2.getStructuringElement(cv2.MORPH_RECT, (7,7))
se2 = cv2.getStructuringElement(cv2.MORPH_RECT, (3,3))
mask = cv2.morphologyEx(image_green, cv2.MORPH_OPEN, se1)
I ended up with thisFinal_output
Now I want to detect the path the robot can take which is the black region.So only the first row is my region of interest and I tried different methods to draw a line in center of the row but couldn't find any help in opencv.I did manage to get a work around by splitting the image into two vertically and used cv2.fitline function to get a line joining one side of row and did the same with other side of row and finally I plotted the center line.But this is not an ideal approach and I feel like there might be some opencv functions to do it in much better way.Can some one help me with this or guide me in right way.
This is the final output I am looking for
Final expected result with green color showing the center of path

So, here is my approach using numpy and scipy, which yielded this result:
.
Without doing any bluring or morphological operations, use the Canny edge detector:
edges = cv2.Canny(image, 100, 200, None, 3, cv2.DIST_L2)
Notice that most of the edges surround the track your robot wants to follow. Since each edge is a collection of white pixels, we could calculate a column's total intensity:
normalized = cv2.normalize(edges, None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX)
column_intensity = normalized.sum(axis=0)
Plotting the results, we get
If we were to find the minimum of the graph, then we would find the x direction, where most of the edges are avoided. But first, let's smooth the function so as to avoid some noise.
# smooth function through moving average
window_size = 30
window = np.ones((window_size,)) / window_size
smoothed = np.convolve(column_intensity, window, mode="valid")
Since there are a lot of local minima, our additional constraint is that the x-direction the robot should take is the closest to the center of the image.
# find indices of local minima and select the one closest to the center
indices = scipy.signal.argrelmin(smoothed)[0]
distances = np.abs(indices - int(width / 2))
x = indices[np.argmin(distances)]
Now that we have the x-direction, we need to determine a y coordinate so as to estimate the angle the robot should rotate (tan(angle)=y/x). There are as many choices as there are rows in the image, which means the y coordinate needs to be manually set. If we choose a y closer to the robot, the angle will be more volatile as the robot advances. Conversely, if we choose a y that is far from the robot, then it will be less volatile but less accurate as well. That is up to you; the final image was created with a y = 400.
I hope this fits your needs :)

Methods for detecting a known shape/object in an image using OpenCV

My task is to detect an object in a given image using OpenCV (I do not care whether it is the Python or C++ implementation). The object, shown below in three examples, is a black rectangle with five white rectagles within. All dimensions are known.
However, the rotation, scale, distance, perspective, lighting conditions, camera focus/lens, and background of the image are not known. The edge of the black rectangle is not guaranteed to be fully visible, however there will not be anything in front of the five white rectangles ever - they will always be fully visible. The end goal is to be able to detect the presence of this object within an image, and rotate, scale, and crop to show the object with the perspective removed. I am fairly confident that I can adjust the image to crop to just the object, given its four corners. However I am not so confident that I can reliably find those four corners. In ambiguous cases, not finding the object is preferred to misidentifying some other feature of the image as the object.
Using OpenCV I have come up with the following methods, however I feel I might be missing something obvious. Are there any more methods available, or is one of these the optimal solution?
Edge based outline
First idea was to look for the outside edge of the object.
Using Canny edge detection (after scaling to known size, grayscaling and gaussian blurring), finding a contour which best matches the outer shape of the object.
This deals with perspective, colour, size issues, but fails when there is a complicated background for example, or if there is something of similar shape to the object elsewhere in the image. Maybe this could be improved by a better set of rules for finding the correct contour - perhaps involving the five white rectangles as well as the outer edge.
Feature detection
The next idea was to match to a known template using feature detecting.
Using ORB feature detecting, descriptor matching and homography (from this tutorial) fails, I believe because the features it is detecting are very similar to other features within the object (lots of coreners which are precisely one-quarter white and three-quarters black). However, I do like the idea of matching to a known template - this idea makes sense to me. I suppose though that because the object is quite basic geometrically, it's likely to find a lot of false positives in the feature matching step.
Parallel Lines
Using Houghlines or HoughLinesP, looking for evenly spaced parallel lines. Have just started down this road so need to investigate the best methods for thresholding etc. While it looks messy for images with complex backgrounds, I think it may work well as I can rely on the fact that the white rectangles within the black object should always be high contrast, giving a good indication of where the lines are.
'Barcode Scan'
My final idea is to scan the image by line, looking for the white to black pattern.
I have not started this method, but the idea is to take a strip of the image (at some angle), convert to HSV colour space, and look for the regular black-to-white pattern appearing five times sequentially in the Value column. This idea sounds promising to me, as I believe it should ignore many of the unknown variables.
Thoughts
I have looked at a number of OpenCV tutorials, as well as SO questions such as this one, however because my object is quite geometrically simple I am having issues implementing the ideas given.
I feel like this is an achievable task, however my struggle is knowing which method to pursue further. I have experimented with the first two ideas quite a bit, and while I haven't achieved anything very reliable, maybe there is something I am missing. Is there a standard way of achieving this task which I have not thought of, or is one of my suggested methods the most sensible?
EDIT: Once the corners are found using one of the above methods (or some other method), I am thinking of using Hu Moments or OpenCV's matchShapes() function to remove any false positives.
EDIT2: Added some more input image examples as requested by #Timo
Orig1
Orig2
Orig3
Extra image 1
Extra image 2
Extra image 3
Extra image 4

I had some time looking into the problem and made a little python script. I'm detecting the white rectangles inside your shape. Paste the code into a .py file and copy all input images in an input subfolder. The final result of the image is just a dummy atm and the script isn't complete yet. I'll try to continue it in the next couple of days. The script will create a debug subfolder where it'll save some images that show the current detection state.
import numpy as np
import cv2
import os
INPUT_DIR = 'input'
DEBUG_DIR = 'debug'
OUTPUT_DIR = 'output'
IMG_TARGET_SIZE = 1000
# each algorithm must return a rotated rect and a confidence value [0..1]: (((x, y), (w, h), angle), confidence)
def main():
# a list of all used algorithms
algorithms = [rectangle_detection]
# load and prepare images
files = list(os.listdir(INPUT_DIR))
images = [cv2.imread(os.path.join(INPUT_DIR, f), cv2.IMREAD_GRAYSCALE) for f in files]
images = [scale_image(img) for img in images]
for img, filename in zip(images, files):
results = [alg(img, filename) for alg in algorithms]
roi, confidence = merge_results(results)
display = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
display = cv2.drawContours(display, [cv2.boxPoints(roi).astype('int32')], -1, (0, 230, 0))
cv2.imshow('img', display)
cv2.waitKey()
def merge_results(results):
'''Merges all results into a single result.'''
return max(results, key=lambda x: x[1])
def scale_image(img):
'''Scales the image so that the biggest side is IMG_TARGET_SIZE.'''
scale = IMG_TARGET_SIZE / np.max(img.shape)
return cv2.resize(img, (0,0), fx=scale, fy=scale)
def rectangle_detection(img, filename):
debug_img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
_, binarized = cv2.threshold(img, 50, 255, cv2.THRESH_BINARY)
contours, _ = cv2.findContours(binarized, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
# detect all rectangles
rois = []
for contour in contours:
if len(contour) < 4:
continue
cont_area = cv2.contourArea(contour)
if not 1000 < cont_area < 15000: # roughly filter by the volume of the detected rectangles
continue
cont_perimeter = cv2.arcLength(contour, True)
(x, y), (w, h), angle = rect = cv2.minAreaRect(contour)
rect_area = w * h
if cont_area / rect_area < 0.8: # check the 'rectangularity'
continue
rois.append(rect)
# save intermediate results in the debug folder
rois_img = cv2.drawContours(debug_img, contours, -1, (0, 0, 230))
rois_img = cv2.drawContours(rois_img, [cv2.boxPoints(rect).astype('int32') for rect in rois], -1, (0, 230, 0))
save_dbg_img(rois_img, 'rectangle_detection', filename, 1)
# todo: detect pattern
return rois[0], 1.0 # dummy values
def save_dbg_img(img, folder, filename, index=0):
'''Writes the given image to DEBUG_DIR/folder/filename_index.png.'''
folder = os.path.join(DEBUG_DIR, folder)
if not os.path.exists(folder):
os.makedirs(folder)
cv2.imwrite(os.path.join(folder, '{}_{:02}.png'.format(os.path.splitext(filename)[0], index)), img)
if __name__ == "__main__":
main()
Here is an example image of the current WIP
The next step is to detect the pattern / relation between mutliple rectangles. I'll update this answer when I make progress.

Detecting circular regions with OpenCV (with lot of small objects)

In the original picture, I would like to detect circular regions. (glands) I managed to get to know the outlines of the regions, but because of the many smaller objects (nuclei), I can not go any further.
My original idea was to remove small objects using the cv2.connectedComponentsWithStats function. But unfortunately, as shown in the picture, the glandy regions also contain small objects, they are not connected properly. The function also throws out the small regions that outline the glands, leaving some parts out of the contours.
Can someone help me to find a solution to this problem?
Thank you very much in advance
Original picture
The approximate contour of the glands (with a lot of small objects in it)
After cv2.connectedComponentsWithStats

OpenCV
I think you can solve your task by using the Hough transform. Something like this could work for you (you have to adjust the parameters according to your needs):
import sys
import cv2 as cv
import numpy as np
def main(argv):
filename = argv[0]
src = cv.imread(filename, cv.IMREAD_COLOR)
if src is None:
print ('Error opening image!')
print ('Usage: hough_circle.py [image_name -- default ' + default_file + '] \n')
return -1
gray = cv.cvtColor(src, cv.COLOR_BGR2GRAY)
gray = cv.medianBlur(gray, 5)
rows = gray.shape[0]
circles = cv.HoughCircles(gray, cv.HOUGH_GRADIENT, 1, rows / 32,
param1=100, param2=30,
minRadius=20, maxRadius=200)
if circles is not None:
circles = np.uint16(np.around(circles))
for i in circles[0, :]:
center = (i[0], i[1])
# circle center
cv.circle(src, center, 1, (0, 100, 100), 3)
# circle outline
radius = i[2]
cv.circle(src, center, radius, (255, 0, 255), 2)
cv.imshow("detected circles", src)
cv.waitKey(0)
return 0
if __name__ == "__main__":
main(sys.argv[1:])
Some additional preprocessing might be required, to get rid of the noise, e.g. Morphological Transformations and performing edge detection right before the transformation might be helpful as well.
Neural Networks
Another option would be to use a neural network for image segmentation. A quite successful one is Mask RCNN. There is already a working python implementation on GitHub: Mask RCNN - Nucleus.

How can i pythonically us opencv to find a a basketball in an image?

I've been laboring on a pet project for a bit on how to find a simple basketball in an image. I've tried a bunch of permutations of using hough.circles and transform , etc for the last few weeks but I cant seem to come anywhere close to isolating the basketball with the code examples and my own tinkering.
Here is an example photo:
And here is the result after a simple version of circle finding code I've been tinkering with:
Anyone have any idea where I have gone wrong and how I can get it right?
Here is the the code I am fiddling with:
import cv2
import cv2.cv as cv # here
import numpy as np
def draw_circles(storage, output):
circles = np.asarray(storage)
for circle in circles:
Radius, x, y = int(circle[0][3]), int(circle[0][0]), int(circle[0][4])
cv.Circle(output, (x, y), 1, cv.CV_RGB(0, 255, 0), -1, 8, 0)
cv.Circle(output, (x, y), Radius, cv.CV_RGB(255, 0, 0), 3, 8, 0)
orig = cv.LoadImage('basket.jpg')
processed = cv.LoadImage('basket.jpg',cv.CV_LOAD_IMAGE_GRAYSCALE)
storage = cv.CreateMat(orig.width, 1, cv.CV_32FC3)
#use canny, as HoughCircles seems to prefer ring like circles to filled ones.
cv.Canny(processed, processed, 5, 70, 3)
#smooth to reduce noise a bit more
cv.Smooth(processed, processed, cv.CV_GAUSSIAN, 7, 7)
cv.HoughCircles(processed, storage, cv.CV_HOUGH_GRADIENT, 2, 32.0, 30, 550)
draw_circles(storage, orig)
cv.imwrite('found_basketball.jpg',orig)

I agree with the other posters, that using the colour of the basketball is a good approach. Here is some simple code that does that:
import cv2
import numpy as np
im = cv2.imread('../media/basketball.jpg')
# convert to HSV space
im_hsv = cv2.cvtColor(im, cv2.COLOR_BGR2HSV)
# take only the orange, highly saturated, and bright parts
im_hsv = cv2.inRange(im_hsv, (7,180,180), (11,255,255))
# To show the detected orange parts:
im_orange = im.copy()
im_orange[im_hsv==0] = 0
# cv2.imshow('im_orange',im_orange)
# Perform opening to remove smaller elements
element = np.ones((5,5)).astype(np.uint8)
im_hsv = cv2.erode(im_hsv, element)
im_hsv = cv2.dilate(im_hsv, element)
points = np.dstack(np.where(im_hsv>0)).astype(np.float32)
# fit a bounding circle to the orange points
center, radius = cv2.minEnclosingCircle(points)
# draw this circle
cv2.circle(im, (int(center[1]), int(center[0])), int(radius), (255,0,0), thickness=3)
out = np.vstack([im_orange,im])
cv2.imwrite('out.png',out)
result:
I assume that:
Always one and only one basketball is present
The basketball is the principal orange item in the scene
With these assumptions, if we find anything the correct colour, we can assume its the ball and fit a circle to it. This way we don't do any circle detection at all.
As you can see in the upper image, there are some smaller orangey elements (from the shorts) which would mess up our ball radius estimate. The code uses an opening operation (erosion followed by dilation), to remove these. This works nicely for your example image. But for other images a different method might be better: using circle detection too, or contour shape, size, or if we are dealing with a video, we could track the ball position.
I ran this code (only modified for video) on a random short basketball video, and it worked surprisingly ok (not great.. but ok).

A few thoughts:
Filter by color first to simplify the image. If you're looking specifically for an orange basketball, you could eliminate a lot of other colors. I'd recommend using HSI color space instead of RGB, but in any case you should be able to exclude colors that are some distance in color 3-space from your trained basketball color.
Try substituting Sobel or some other kernel-based edge detector that doesn't rely on manual parameters. Display the edge image to see if it looks "right" to you.
Allow for weaker edges. In the grayscale image, the contrast between the basketball and the player's dark jersey is not as great as the difference between the white undershirt and the black jersey.
Hough may yield unexpected results if the object is only nominally circular in cross section, but is actually elongated or has noisy edges in the actual image. I usually write my own Hough algorithm and haven't touched the OpenCV implementation, so I'm not sure what parameter to change, but see if you can allow for fuzzier edges.
Maybe eliminate the smooth operation. In any case, try smooth before finding edges rather than the other way around.
Try to write your own rough Hough algorithm. Although a quickie implementation may not be as flexible as the OpenCV implementation, by getting your hands dirty you may stumble onto the source of the problem.