I am using landmark points from dlib library to select the forehead, nose and eye area from my face based on this question: Is there a way to get the area of the forehead (bounding box) by using opencv/dlib and for a live stream video. It works like a charm and i have the points exactly where i want them, what i would like to do is crop the image where the landmarks are set using convexhull polygons.
What i am trying to do is go from this:
to this:
And save it afterwards
Is there a any way to do it? even if it doesn't look pretty. Here's my current code for facial tracking:
import cv2
import dlib
from imutils import face_utils
import imutils
import numpy as np
cap = cv2.VideoCapture(0)
detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor("shape_predictor_81_face_landmarks.dat")
def face_remap(shape):
remapped_image = cv2.convexHull(shape)
return remapped_image
while True:
_, frame = cap.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
out_face = np.zeros_like(frame)
faces = detector(gray, 1)
for face in faces:
x1 = face.left()
y1 = face.top()
x2 = face.right()
y2 = face.bottom()
aam = [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 77, 75, 75, 55, 69, 70, 71, 80, 72, 73, 79, 74, 78]
landmarks = predictor(gray, face)
shape = face_utils.shape_to_np(landmarks)
remapped_shape = np.zeros_like(shape)
feature_mask = np.zeros((frame.shape[0], frame.shape[1]))
x_pts = []
y_pts = []
for n in (aam):
x = landmarks.part(n).x
y = landmarks.part(n).y
x_pts.append(x)
y_pts.append(y)
cv2.circle(frame, (x, y), 1, (255, 0, 0), -1)
x1 = min(x_pts)
x2 = max(x_pts)
y1 = min(y_pts)
y2 = max(y_pts)
cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 0, 255), 3)
cv2.imshow("out", frame)
k = cv2.waitKey(1)
if k == 27:
remapped_shape = face_remap(shape)
cv2.fillConvexPoly(feature_mask, remapped_shape[0:27], 1)
feature_mask = feature_mask.astype(np.bool)
out_face[feature_mask] = frame[feature_mask]
cv2.imwrite("out_face.png", out_face)
break
cap.release()
cv2.destroyAllWindows()
this code will crop the face image from the background and set the landmark as showed in the example. however i would like to crop around the landmarks
You selected only a subset of the 81 landmarks dlib identifies on a face, and discarded the landmarks associated with the mouth, chin and the outer contour of the face.
You should do an additional selection leaving only the points at the boundary of the region you are interested in.
Furthermore, you should order the selected points such that connecting them, in the right order, will form a polygon marking exactly the region you want to crop.
Once you have the polygon you can use this method to crop it:
Use cv2.fillPoly() to draw mask from the polygon.
Get only the cropped polygon portion of the mask from the image using cv2.bitwsie_and().
Get the bounding rect of the cropped region using cv2.boundingRect().
As #Shai mentioned in comments, your shape is not convex at all.
To crop image out of convexhull polygons, you can first get mask out of the convexhull and then apply bitwise and operation on original image using the mask.
Perhaps a function like this would help.
def mask_from_contours(ref_img, contours):
mask = numpy.zeros(ref_img.shape, numpy.uint8)
mask = cv2.drawContours(mask, contours, -1, (255,255,255), -1)
return cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
If you just want crop the image out of the polygon, you can programmatically connect the landmarks, hard-code by landmark index, and get a mask out of the polygon, and then apply bitwise-and operation.
Here's code sample for filling polygon.
import numpy as np
import cv2
import matplotlib.pyplot as plt
a3 = np.array( [[[10,10],[100,10],[100,100],[10,100]]], dtype=np.int32 )
im = np.zeros([240,320],dtype=np.uint8)
cv2.fillPoly( im, a3, 255 )
Related
I am trying to analyze the red wells with different intensities from the attached image.
I want to analyze how the intensity of each red colored well differs from the others.
Does anyone have a solution without using the following code:
import numpy as np
import argparse
import cv2
# Construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", help="path to the image")
args = vars(ap.parse_args())
# Load the image
image = cv2.imread(args["image"])
# Define the list of boundaries
boundaries = [
([17, 15, 100], [50, 56, 200]),
([86, 31, 4], [220, 88, 50]),
([25, 146, 190], [62, 174, 250]),
([103, 86, 65], [145, 133, 128])
]
# Loop over the boundaries
for (lower, upper) in boundaries:
# Create NumPy arrays from the boundaries
lower = np.array(lower, dtype="uint8")
upper = np.array(upper, dtype="uint8")
# Find the colors within the specified boundaries and apply the mask
mask = cv2.inRange(image, lower, upper)
output = cv2.bitwise_and(image, image, mask=mask)
Image I'm trying to process:
Assume we want to automatically find the red wells.
Assume we know that the wells are circular, and that the color is red (reddish).
We also know that the wells are not tiny, but we don't want to overfit the solution too much...
We may use the following stages:
Use cv2.bilateralFilter for reducing noise (while keeping edges fairly sharp).
Find red pixels as described in the following post.
Convert from BGR to HSV, and find the pixels in range that is considered red.
According to the post there are "lower_red" and "upper_red" ranges (we are keeping the ranges without tuning).
Find contours.
Iterate the contours.
Skip small contours (assumed to be noise).
Use cv2.minEnclosingCircle as described here.
Draw a filled circle for each contour, to form a mask, and use cv2.bitwise_and as used in your question.
Code sample:
import numpy as np
import argparse
import cv2
# Construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", help="path to the image")
args = vars(ap.parse_args())
# Load the image
image = cv2.imread(args["image"])
img = cv2.bilateralFilter(image, 11, 75, 75)
# https://stackoverflow.com/questions/30331944/finding-red-color-in-image-using-python-opencv
# Convert from BGR to HSV
img_hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
# Lower mask (0-10)
lower_red = np.array([0, 50, 50])
upper_red = np.array([10, 255, 255])
mask0 = cv2.inRange(img_hsv, lower_red, upper_red)
# Upper mask (170-180)
lower_red = np.array([170, 50, 50])
upper_red = np.array([180, 255, 255])
mask1 = cv2.inRange(img_hsv, lower_red, upper_red)
# Join the masks
raw_mask = mask0 | mask1
ctns = cv2.findContours(raw_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[-2] # Find contours
mask = np.zeros_like(raw_mask) # Fill mask with zeros
idx = 0
# Iterate contours
for c in ctns:
area = cv2.contourArea(c) # Find the area of each contours
if (area > 50): # Ignore small contours (assume noise).
cv2.drawContours(mask, [c], 0, 255, -1)
# https://docs.opencv.org/3.4/dd/d49/tutorial_py_contour_features.html
(x, y), radius = cv2.minEnclosingCircle(c)
center = (int(x), int(y))
radius = int(radius)
cv2.circle(mask, center, radius, 255, -1)
tmp_mask = np.zeros_like(mask)
cv2.circle(tmp_mask, center, radius, 255, -1)
output = cv2.bitwise_and(image, image, mask=tmp_mask)
cv2.imshow(f'output{idx}', output) # Show output images for testing
cv2.imwrite(f'output{idx}.png', output) # Save output images for testing
idx += 1
cv2.imshow('image', image)
cv2.imshow('img', img)
cv2.imshow('raw_mask', raw_mask)
cv2.imshow('mask', mask)
cv2.waitKey()
cv2.destroyAllWindows()
Results:
Image after bilateral filter:
raw_mask:
mask:
output0.png:
output1.png:
output2.png:
output3.png:
output4.png:
In case the circles are too large, we can reduce the radius, and get only the center of each well.
Let's say that we have the following black and white image (image.png):
We load the image using OpenCV2 with the following code:
import cv2
img = cv2.imread('image.png')
How can we detect the corners of all white rectangles in an image? We can assume that all rectangles are parallel to the corners of the image.
Result in this case should be in the following form (res[0] is left rectangle and res[1] is right rectangle):
res = [
[
(20, 30), # upper left
(40, 30), # upper right
(20, 80), # bottom left
(40, 80) # bottom right
],
[
(100, 20), # upper left
(140, 20), # upper right
(100, 70), # bottom left
(140, 70) # bottom right
]
]
There are a few possibilities:
the "Harris Corner Detector" is good at finding corners - see here
you can use OpenCV's findContours()
you could use "Hit-or-Miss" morphology to look for corners
you could convolve the image with a kernel and look for specific outputs
So, looking at the last option, if we slide a 2x2 pixel kernel over the image and multiply each of the elements underneath the kernel by 1 and add them all together, and then find all the pixels where that total comes to 255, that will be a 2x2 square where exactly one pixel is white - and that is a corner:
import cv2
import numpy as np
# Load image as greyscale
im = cv2.imread('tZHHE.png',cv2.IMREAD_GRAYSCALE)
# Make a 2x2 kernel of ones
kernel = np.ones((2,2), dtype=np.uint8)
# Convolve the image with the kernel
res = cv2.filter2D(im.astype(np.int16), -1, kernel)
# Get coordinates of points where the sum of the 2x2 window comes to 255
corners = np.where(res==255)
Sample Output
(array([101, 101, 118, 118, 166, 166, 174, 174]),
array([274, 307, 117, 134, 274, 307, 117, 134]))
Looking at the "Hit-or-Miss" morphology method, I will do it with ImageMagick straight in the Terminal, but you can equally do it with other Python libraries:
magick tZHHE.png -alpha off -morphology HMT Corners result.png
As always, I am indebted to Anthony Thyssen for his ImageMagick examples as linked above. We are looking for these specific shapes with the "Hit-or-Miss" morphology:
Keywords: Python, OpenCV, image processing, convolution, corner detect, corner detector, corner detection, ImageMagick Hit-or-Miss morphology.
This works for any number of rectangles:
import cv2 as cv
import pprint as pprint
img = cv.imread("tZHHE.png") # read image
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # make grayscale image
cv.imshow("Our initial image",img) # show our original image
corners = cv.goodFeaturesToTrack(gray,2000,0.01,5) # find our corners, 2000 is the number of corners we can detect, 5 is the distance between corners
xylist = [] #put all of our xy coords in here
for corn in corners: # extract our corners and put them in xylist
x,y = corn[0]
xylist.append((x,y))
x = int(x)
y = int(y)
cv.rectangle(img, (x-2,y-2), (x+2,y+2), (100,100,0),-1) # now mark where our corners are on our original image
res = [[] for i in range(int(len(xylist)/4))] # generate i nested lists for our rectangles
for index, item in enumerate(xylist): # format coordinates as you want them
res[index % int(len(xylist)/4)].append(item)
print("\n"+"found ",int(len(xylist)/4) ,"rectangles\n") # how many rectangles did we have?
print(res)
cv.imshow("Corners?", img) # show our new image with rectangle corners marked
Try findContours()
I suggest you to try findContours() with its companion boundingRect().
Here is how you can make it work.
Load the image in grayscale, then pass it to the function findContours().
img = cv2.imread('tZHHe.png', cv2.IMREAD_GRAYSCALE)
contours, hierarchy = cv2.findContours(img, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
Getting the bounding box from contours, it returns x, y coordinates of the top left corner and w, h the width and height of the box:
[cv2.boundingRect(contour) for contour in contours]
#=> [(117, 118, 17, 56), (274, 101, 33, 65)]
View in action
Try this maybe in a Jupyter Notebook to see a kind of animation:
def corners_from_bounding_rect(bounding_rect):
x, y, w, h = bounding_rect
points = {'top_left': (x, y), 'top_right':(x+w, y), 'bottom-left': (x, y+h), 'bottom-rigth':(x+w, y+h)}
return points
Retrieve the points from contours using the method defined:
corner_groups = [corners_from_bounding_rect(cv2.boundingRect(cnt)) for cnt in contours]
# [{'top_left': (117, 118),
# 'top_right': (134, 118),
# 'bottom-left': (117, 174),
# 'bottom-rigth': (134, 174)},
# {'top_left': (274, 101),
# 'top_right': (307, 101),
# 'bottom-left': (274, 166),
# 'bottom-rigth': (307, 166)}]
Then plot the sequence:
pinned_img = img.copy()
for n, corners in enumerate(corner_groups):
for name, point in corners.items():
cv2.circle(pinned_img, point, 10, 255)
plt.title(f'{n}-{name}')
plt.imshow(pinned_img)
plt.show()
The first image from the squence:
Try this code
import cv2
img=cv2.imread('tZHHE.png') # Read my Image
imgContours=img.copy() # Copy my Image for Contours
imgCanny=cv2.Canny(imgContours,10,50) # Image to Edges
contours,hierarchy =cv2.findContours(imgCanny,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)
rectangles=[]
for cont in reversed(contours):
area=cv2.contourArea(cont)
x, y, w, h = cv2.boundingRect(cont)
rectangles.append([[x,y],[x+w,y],[x+w,y+h],[x,y+h]])
cv2.imshow("con", imgContours)
cv2.waitKey(0)
print(rectangles)
cv2.destroyAllWindows()
left to right rectangles
Output:
[[[273, 100], [307, 100], [307, 166], [273, 166]], [[116, 117], [134, 117], [134, 174], [116, 174]]]
I try to detect the yellow line in the following picture but shadows obscured the yellow roads. Do we have any method to deal with that?
We can detect the yellow in this question:About Line detection by using OpenCV and How to delete the other object from figure by using opencv?.
The coding is as follows:
import cv2
import numpy as np
image = cv2.imread('Road.jpg')
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
low_yellow = np.array([18, 94, 140])
up_yellow = np.array([48, 255, 255])
mask = cv2.inRange(hsv, low_yellow, up_yellow)
edges = cv2.Canny(mask, 75, 150)
lines = cv2.HoughLinesP(edges, 1, np.pi / 180, 50, maxLineGap=250)
for line in lines:
x1, y1, x2, y2 = line[0]
cv2.line(image, (x1, y1), (x2, y2), (0, 255, 0), 5)
# cv2.imshow('image', img)
cv2.imwrite("result.jpg", edges)
Here is the code to convert to lab and auto-threshold
You'll have to detect the lines using a proper method.
An advanced solution would be training a dataset to do segmentation (neural network Ex : Unet )
import cv2
import numpy as np
img = cv2.imread('YourImagePath.jpg')
cv2.imshow("Original", img)
k = cv2.waitKey(0)
# Convert To lab
lab = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)
# display b channel
cv2.imshow("Lab", lab[:, :, 2])
k = cv2.waitKey(0)
# auto threshold using Otsu
ret , mask = cv2.threshold(lab[:, :, 2] , 0 , 255 , cv2.THRESH_BINARY+
cv2.THRESH_OTSU)
#display Binary
cv2.imshow("Binary", mask)
k = cv2.waitKey(0)
cv2.destroyAllWindows()
Output:
Using Amazon's Rekognition, I have extracted the bounding boxes of interest from the JSON response using the following:
def __init__(self, image):
self.shape = image.shape
def bounding_box_convert(self, bounding_box):
xmin = int(bounding_box['Left'] * self.shape[1])
xmax = xmin + int(bounding_box['Width'] * self.shape[1])
ymin = int(bounding_box['Top'] * self.shape[0])
ymax = ymin + int(bounding_box['Height'] * self.shape[0])
return (xmin,ymin,xmax,ymax)
def polygon_convert(self, polygon):
pts = []
for p in polygon:
x = int(p['X'] * self.shape[1])
y = int(p['Y'] * self.shape[0])
pts.append( [x,y] )
return pts
def get_bounding_boxes(jsondata):
objectnames = ('Helmet','Hardhat')
bboxes = []
a = jsondata
if('Labels' in a):
for label in a['Labels']:
#-- skip over anything that isn't hardhat,helmet
if(label['Name'] in objectnames):
print('extracting {}'.format(label['Name']))
lbl = "{}: {:0.1f}%".format(label['Name'], label['Confidence'])
print(lbl)
for instance in label['Instances']:
coords = tmp.bounding_box_convert(instance['BoundingBox'])
bboxes.append(coords)
return bboxes
if __name__=='__main__':
imagefile = 'image011.jpg'
bgr_image = cv2.imread(imagefile)
tmp = Tmp(bgr_image)
jsonname = 'json_000'
fin = open(jsonname, 'r')
jsondata = json.load(fin)
bb = get_bounding_boxes(jsondata)
print(bb)
The output is a list of bounding boxes:
[(865, 731, 1077, 906), (1874, 646, 2117, 824)]
I am able to easily extract one position from the list and save as a new image using:
from PIL import Image
img = Image.open("image011.jpg")
area = (865, 731, 1077, 906)
cropped_img = img.crop(area)
cropped_img.save("cropped.jpg")
However, I haven't found a good solution to crop and save multiple bounding boxes from the image using the 'bb' list output.
I did find a solution that extracts the information from a csv here: Most efficient/quickest way to crop multiple bounding boxes in 1 image, over thousands of images?.
But, I believe there is a more efficient way than saving the bounding box data to a csv and reading it back in.
I'm not very strong at writing my own functions - all suggestions are much appreciated!
Assuming your bounding box coordinates are in the form of x,y,w,h you can do ROI = image[y:y+h,x:x+w] to crop. With this input image:
Using the script from how to get ROI Bounding Box Coordinates without Guess & Check to obtain the x,y,w,h bounding box coordinates to crop out these ROIs:
We simply iterate through the bounding box list and crop it using Numpy slicing. The extracted ROIs:
Here's a minimum example:
import cv2
import numpy as np
image = cv2.imread('1.png')
bounding_boxes = [(17, 24, 47, 47),
(74, 28, 47, 50),
(125, 15, 51, 61),
(184, 18, 53, 53),
(247, 25, 44, 46),
(296, 6, 65, 66)
]
num = 0
for box in bounding_boxes:
x,y,w,h = box
ROI = image[y:y+h, x:x+w]
cv2.imwrite('ROI_{}.png'.format(num), ROI)
num += 1
cv2.imshow('ROI', ROI)
cv2.waitKey()
I've been trying to do a 4 point perspective transform in order to start doing some OCR.
Starting with the following image I can detect the number plate
and crop it out with the green box being the bounding box and the red dots being the corners of the rectangle I want to square up.
This is the output of the transform.
At a first look it seams to have done the transform inside out (taking the parts either side rather than between the points).
I'm using the imutils package to do the transform and working from this and this as a guide. I'm sure it's something relatively simple I'm missing.
#!/usr/bin/python
import numpy as np
import cv2
import imutils
from imutils import contours
from imutils.perspective import four_point_transform
img = cv2.imread("sample7-smaller.jpg")
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
blurred = cv2.bilateralFilter(gray,15,75,75)
v = np.median(blurred)
lower = int(max(0, (1.0 - 0.33) * v))
upper = int(min(255, (1.0 + 0.33) * v))
edged = cv2.Canny(blurred, lower, upper, 255)
conts = cv2.findContours(edged.copy(), cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE)
conts = conts[0] if imutils.is_cv2() else conts[1]
conts = sorted(conts, key=cv2.contourArea, reverse=True)
for cnt in conts:
approx = cv2.approxPolyDP(cnt,0.01*cv2.arcLength(cnt,True),True)
if len(approx) == 4:
x,y,w,h = cv2.boundingRect(cnt)
cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,0),2)
for i in approx:
cv2.circle(img,(i[0][0], i[0][1]),2,(0,0,255), thickness=4)
warped = four_point_transform(img, approx.reshape(4,2))
cv2.imshow("crop",img[y:y+h,x:x+w])
cv2.imshow("warped", warped)
cv2.waitKey(0)
I would recommend you to use the OpenCV Perspective Transform method, to get the desired results, as per the given image:
First mark the position of src points:
src_pts = np.array([[8, 136], [415, 52], [420, 152], [14, 244]], dtype=np.float32)
And suppose you want to fit this number plate in a matrix of shape 50x200, so destination points would be:
dst_pts = np.array([[0, 0], [200, 0], [200, 50], [0, 50]], dtype=np.float32)
Find the perspective Transform Matrix as :
M = cv2.getPerspectiveTransform(src_pts, dst_pts)
warp = cv2.warpPerspective(img, M, (200, 50))
EDIT: As you didn't wanted to hard code the final width, height of plate, So in order to make the calculations more flexible you can calculate the width and height of the plate from the 4 marker points as:
def get_euler_distance(pt1, pt2):
return ((pt1[0] - pt2[0])**2 + (pt1[1] - pt2[1])**2)**0.5
src_pts = np.array([[8, 136], [415, 52], [420, 152], [14, 244]], dtype=np.float32)
width = get_euler_distance(src_pts[0][0], src_pts[0][1])
height = get_euler_distance(src_pts[0][0], src_pts[0][3])
dst_pts = np.array([[0, 0], [width, 0], [width, height], [0, height]], dtype=np.float32)
M = cv2.getPerspectiveTransform(src_pts, dst_pts)
warp = cv2.warpPerspective(img, M, (width, height))