Related
has anyone used template matching on OpenCV to detect images using transparent templates and it worked?
I'm trying to use template matching to detect multiple boxes that make up a table. The template used is a box of the same size but with a transparent background because the contents of the cell can be in the form of text with any font size.
Here the Image
Here the Template 1
Here the Template 2
img = cv.imread('image.png',-1)
template1 = cv.imread('template 1.png',-1)
template2 = cv.imread('template 2.png',-1)
h,w,_ = template.shape
method = cv.TM_CCOEFF_NORMED
res = cv.matchTemplate(img,template1,method)
# res = cv.matchTemplate(img,template2,method)
loc = np.where(res >= 0.8)
boxes = list()
for pt in zip(*loc[::-1]):
boxes.append((pt[0],pt[1],pt[0]+w,pt[1]+h))
boxes = non_max_suppression(np.array(boxes))
for (x1, y1, x2, y2) in boxes:
cv.rectangle(tem, (x1, y1), (x2, y2),(23,255,255), 1)
When running the match template module using template1, the results of the process fail to detect the existing boxes, whereas when using template2, the results of the process are able to detect boxes but only 1 out of 4 boxes can be detected.
The 4 boxes can be detected if the threshold used is lowered to 0.5, but when tested using other images it fails to detect the boxes correctly (sometimes the detection results are more than they should be, and sometimes the coordinates don't match).
From the result, I think the result appears because the match template process for transparent image failed to run and the frequency of appearance of black color in each box still affects the method.
If we look at the tutorial it shows that using a mask can solve this problem, but I am confused about which mask to use. In this example, the mask used is defined and there is no mention of how to create the mask. Anyone can help?
Following #fmw42 example in the link, I have managed to detect the box as needed.
However, in its implementation, adjustments need to be made because the use of black color as a template (border) followed by an transparent background which is also black causes the detection process to fail.
Here I solve it by changing the black color in template (border) to red color so that the template is composed of a red color in the border and a transparent black color. The colors used may be adjusted to your liking.
img = cv.imread('image.png')
img[:,:,:2] = 255 # change colors to red
template_all = cv.imread('template 1.png',cv.IMREAD_UNCHANGED)
template = template_all[:,:,0:3]
template[:,:,2] = 255 # change colors to red
alpha = template_all[:,:,3]
alpha = cv.merge([alpha,alpha,alpha])
h,w = template.shape[:2]
method = cv.TM_CCORR_NORMED
res = cv.matchTemplate(img,template,method,mask=alpha)
loc = np.where(res >= 0.9)
result = img.copy()
boxes = list()
for pt in zip(*loc[::-1]):
boxes.append((pt[0],pt[1],pt[0]+w,pt[1]+h))
boxes = non_max_suppression(np.array(boxes))
for (x1, y1, x2, y2) in boxes:
cv.rectangle(result, (x1, y1), (x2, y2),(23,255,255), 1)
cv.imshow('result',result)
cv.waitKey(0)
cv.destroyAllWindows()
The goal is to find the line that represents the distance between the "hole" and the outer edge (transition from black to white). I was able to successfully binarize this photo and get a very clean black and white image. The next step would be to find the (almost) vertical line on it and calculate the perpendicular distance to the midpoint of this vertical line and the hole.
original picture
hole - zoomed in
ps: what I call "hole" is a shadow. I am shooting a laser into a hole. So the lines we can see is a steel and the black part without a line is a hole. The 2 white lines serve as a reference to measure the distance.
Is Canny edge detection the best approach? If so, what are good values for the A, B and C parameters? I can't tune it. I'm getting too much noise.
This is not complete; You have to take the time to reach the final result. But this idea might help you.
Preprocessors:
import os
import cv2
import numpy as np
Main code:
# Read original image
dir = os.path.abspath(os.path.dirname(__file__))
im = cv2.imread(dir+'/'+'im.jpg')
h, w = im.shape[:2]
print(w, h)
# Convert image to Grayscale
imGray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
cv2.imwrite(dir+'/im_1_grayscale.jpg', imGray)
# Eliminate noise and display laser light better
imHLine = imGray.copy()
imHLine = cv2.GaussianBlur(imHLine, (0, 9), 21) # 5, 51
cv2.imwrite(dir+'/im_2_h_line.jpg', imHLine)
# Make a BW mask to find the ROI of laser array
imHLineBW = cv2.threshold(imHLine, 22, 255, cv2.THRESH_BINARY)[1]
cv2.imwrite(dir+'/im_3_h_line_bw.jpg', imHLineBW)
# Remove noise with mask and extract just needed area
imHLineROI = imGray.copy()
imHLineROI[np.where(imHLineBW == 0)] = 0
imHLineROI = cv2.GaussianBlur(imHLineROI, (0, 3), 6)
imHLineROI = cv2.threshold(imHLineROI, 25, 255, cv2.THRESH_BINARY)[1] # 22
cv2.imwrite(dir+'/im_4_ROI.jpg', imHLineROI)
# Found laser array and draw box around it
cnts, _ = cv2.findContours(
imHLineROI, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts.sort(key=lambda x: cv2.boundingRect(x)[0])
pts = []
for cnt in cnts:
x2, y2, w2, h2 = cv2.boundingRect(cnt)
if h2 < h/10:
cv2.rectangle(im, (x2, y2), (x2+w2, y2+h2), (0, 255, 0), 1)
pts.append({'x': x2, 'y': y2, 'w': w2, 'h': h2})
circle = {
'left': (pts[0]['x']+pts[0]['w'], pts[0]['y']+pts[0]['h']/2),
'right': (pts[1]['x'], pts[1]['y']+pts[1]['h']/2),
}
circle['center'] = calculateMiddlePint(circle['left'], circle['right'])
circle['radius'] = (circle['right'][0]-circle['left'][0])//2
# Draw line and Circle inside it
im = drawLine(im, circle['left'], circle['right'], color=(27, 50, 120))
im = cv2.circle(im, circle['center'], circle['radius'], (255, 25, 25), 3)
# Remove pepper/salt noise to find metal edge
imVLine = imGray.copy()
imVLine = cv2.medianBlur(imVLine, 17)
cv2.imwrite(dir+'/im_6_v_line.jpg', imVLine)
# Remove remove the shadows to find metal edge
imVLineBW = cv2.threshold(imVLine, 50, 255, cv2.THRESH_BINARY)[1]
cv2.imwrite(dir+'/im_7_v_bw.jpg', imVLineBW)
# Finding the right vertical edge of metal
y1, y2 = h/5, h-h/5
x1 = horizantalDistance(imVLineBW, y1)
x2 = horizantalDistance(imVLineBW, y2)
pt1, pt2 = (x1, y1), (x2, y2)
imVLineBW = drawLine(imVLineBW, pt1, pt2)
cv2.imwrite(dir+'/im_8_v_bw.jpg', imVLineBW)
# Draw lines
im = drawLine(im, pt1, pt2)
im = drawLine(im, calculateMiddlePint(pt1, pt2), circle['center'])
# Draw final image
cv2.imwrite(dir+'/im_8_output.jpg', im)
Extra functions:
Find the first white pixel in one line of picture:
# This function only processes on a horizontal line of the image
# Its job is to examine the pixels one by one from the right and
# report the distance of the first white pixel from the right of
# the image.
def horizantalDistance(im, y):
y = int(y)
h, w = im.shape[:2]
for i in range(0, w):
x = w-i-1
if im[y][x] == 255:
return x
return -1
To draw a line in opencv:
def drawLine(im, pt1, pt2, color=(128, 0, 200), thickness=2):
return cv2.line(
im,
pt1=(int(pt1[0]), int(pt1[1])),
pt2=(int(pt2[0]), int(pt2[1])),
color=color,
thickness=thickness,
lineType=cv2.LINE_AA # Anti-Aliased
)
To calculate middle point of two 2d points:
def calculateMiddlePint(p1, p2):
return (int((p1[0]+p2[0])/2), int((p1[1]+p2[1])/2))
Output:
Original image:
Eliminate noise and process to see the laser array better:
Find the laser area to extract the hole:
Work on another copy of image to find the right side of metal object:
Remove the shadows to better see the right edge:
The final output:
I first defined an ROI area. I changed the code later but did not change the names of the variables. If you were asked.
You can try one of binarized Canny, or straight binarized image (Otsu). In both cases, you will obtain a quasi-straight line from which you can extract the right-most white pixel on every row or so.
Then use line fitting to these points (depending on image quality the fitting needs to be robust or not).
It would be wise to calibrate the method against ground truth, because the edge your are looking after borders a gradient area, and the location of the edge might be offset by several pixels.
I have different type of invoice files, I want to find table in each invoice file. In this table position is not constant. So I go for image processing. First I tried to convert my invoice into image, then I found contour based on table borders, Finally I can catch table position.
For the task I used below code.
with Image(page) as page_image:
page_image.alpha_channel = False #eliminates transperancy
img_buffer=np.asarray(bytearray(page_image.make_blob()), dtype=np.uint8)
img = cv2.imdecode(img_buffer, cv2.IMREAD_UNCHANGED)
ret, thresh = cv2.threshold(img, 127, 255, 0)
im2, contours, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
margin=[]
for contour in contours:
# get rectangle bounding contour
[x, y, w, h] = cv2.boundingRect(contour)
# Don't plot small false positives that aren't text
if (w >thresh1 and h> thresh2):
margin.append([x, y, x + w, y + h])
#data cleanup on margin to extract required position values.
In this code thresh1, thresh2 i'll update based on the file.
So using this code I can successfully read positions of tables in images, using this position i'll work on my invoice pdf file. For example
Sample 1:
Sample 2:
Sample 3:
Output:
Sample 1:
Sample 2:
Sample 3:
But, now I have a new format which doesn't have any borders but it's a table. How to solve this? Because my entire operation depends only on borders of the tables. But now I don't have a table borders. How can I achieve this? I don't have any idea to move out from this problem. My question is, Is there any way to find position based on table structure?.
For example My problem input looks like below:
I would like to find its position like below:
How can I solve this?
It is really appreciable to give me an idea to solve the problem.
Thanks in advance.
Vaibhav is right. You can experiment with the different morphological transforms to extract or group pixels into different shapes, lines, etc. For example, the approach can be the following:
Start from the Dilation to convert the text into the solid spots.
Then apply the findContours function as a next step to find text
bounding boxes.
After having the text bounding boxes it is possible to apply some
heuristics algorithm to cluster the text boxes into groups by their
coordinates. This way you can find a groups of text areas aligned
into rows and columns.
Then you can apply sorting by x and y coordinates and/or some
analysis to the groups to try to find if the grouped text boxes can
form a table.
I wrote a small sample illustrating the idea. I hope the code is self explanatory. I've put some comments there too.
import os
import cv2
import imutils
# This only works if there's only one table on a page
# Important parameters:
# - morph_size
# - min_text_height_limit
# - max_text_height_limit
# - cell_threshold
# - min_columns
def pre_process_image(img, save_in_file, morph_size=(8, 8)):
# get rid of the color
pre = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# Otsu threshold
pre = cv2.threshold(pre, 250, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1]
# dilate the text to make it solid spot
cpy = pre.copy()
struct = cv2.getStructuringElement(cv2.MORPH_RECT, morph_size)
cpy = cv2.dilate(~cpy, struct, anchor=(-1, -1), iterations=1)
pre = ~cpy
if save_in_file is not None:
cv2.imwrite(save_in_file, pre)
return pre
def find_text_boxes(pre, min_text_height_limit=6, max_text_height_limit=40):
# Looking for the text spots contours
# OpenCV 3
# img, contours, hierarchy = cv2.findContours(pre, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
# OpenCV 4
contours, hierarchy = cv2.findContours(pre, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
# Getting the texts bounding boxes based on the text size assumptions
boxes = []
for contour in contours:
box = cv2.boundingRect(contour)
h = box[3]
if min_text_height_limit < h < max_text_height_limit:
boxes.append(box)
return boxes
def find_table_in_boxes(boxes, cell_threshold=10, min_columns=2):
rows = {}
cols = {}
# Clustering the bounding boxes by their positions
for box in boxes:
(x, y, w, h) = box
col_key = x // cell_threshold
row_key = y // cell_threshold
cols[row_key] = [box] if col_key not in cols else cols[col_key] + [box]
rows[row_key] = [box] if row_key not in rows else rows[row_key] + [box]
# Filtering out the clusters having less than 2 cols
table_cells = list(filter(lambda r: len(r) >= min_columns, rows.values()))
# Sorting the row cells by x coord
table_cells = [list(sorted(tb)) for tb in table_cells]
# Sorting rows by the y coord
table_cells = list(sorted(table_cells, key=lambda r: r[0][1]))
return table_cells
def build_lines(table_cells):
if table_cells is None or len(table_cells) <= 0:
return [], []
max_last_col_width_row = max(table_cells, key=lambda b: b[-1][2])
max_x = max_last_col_width_row[-1][0] + max_last_col_width_row[-1][2]
max_last_row_height_box = max(table_cells[-1], key=lambda b: b[3])
max_y = max_last_row_height_box[1] + max_last_row_height_box[3]
hor_lines = []
ver_lines = []
for box in table_cells:
x = box[0][0]
y = box[0][1]
hor_lines.append((x, y, max_x, y))
for box in table_cells[0]:
x = box[0]
y = box[1]
ver_lines.append((x, y, x, max_y))
(x, y, w, h) = table_cells[0][-1]
ver_lines.append((max_x, y, max_x, max_y))
(x, y, w, h) = table_cells[0][0]
hor_lines.append((x, max_y, max_x, max_y))
return hor_lines, ver_lines
if __name__ == "__main__":
in_file = os.path.join("data", "page.jpg")
pre_file = os.path.join("data", "pre.png")
out_file = os.path.join("data", "out.png")
img = cv2.imread(os.path.join(in_file))
pre_processed = pre_process_image(img, pre_file)
text_boxes = find_text_boxes(pre_processed)
cells = find_table_in_boxes(text_boxes)
hor_lines, ver_lines = build_lines(cells)
# Visualize the result
vis = img.copy()
# for box in text_boxes:
# (x, y, w, h) = box
# cv2.rectangle(vis, (x, y), (x + w - 2, y + h - 2), (0, 255, 0), 1)
for line in hor_lines:
[x1, y1, x2, y2] = line
cv2.line(vis, (x1, y1), (x2, y2), (0, 0, 255), 1)
for line in ver_lines:
[x1, y1, x2, y2] = line
cv2.line(vis, (x1, y1), (x2, y2), (0, 0, 255), 1)
cv2.imwrite(out_file, vis)
I've got the following output:
Of course to make the algorithm more robust and applicable to a variety of different input images it has to be adjusted correspondingly.
Update: Updated the code with respect to the OpenCV API changes for findContours. If you have older version of OpenCV installed - use the corresponding call. Related post.
You can try applying some morphological transforms (such as Dilation, Erosion or Gaussian Blur) as a pre-processing step before your findContours function
For example
blur = cv2.GaussianBlur(g, (3, 3), 0)
ret, thresh1 = cv2.threshold(blur, 150, 255, cv2.THRESH_BINARY)
bitwise = cv2.bitwise_not(thresh1)
erosion = cv2.erode(bitwise, np.ones((1, 1) ,np.uint8), iterations=5)
dilation = cv2.dilate(erosion, np.ones((3, 3) ,np.uint8), iterations=5)
The last argument, iterations shows the degree of dilation/erosion that will take place (in your case, on the text). Having a small value will results in small independent contours even within an alphabet and large values will club many nearby elements. You need to find the ideal value so that only that block of your image gets.
Please note that I've taken 150 as the threshold parameter because I've been working on extracting text from images with varying backgrounds and this worked out better. You can choose to continue with the value you've taken since it's a black & white image.
There are many types of tables in the document images with too much variations and layouts. No matter how many rules you write, there will always appear a table for which your rules will fail. These types of problems are genrally solved using ML(Machine Learning) based solutions. You can find many pre-implemented codes on github for solving the problem of detecting tables in the images using ML or DL (Deep Learning).
Here is my code along with the deep learning models, the model can detect various types of tables as well as the structure cells from the tables: https://github.com/DevashishPrasad/CascadeTabNet
The approach achieves state of the art on various public datasets right now (10th May 2020) as far as the accuracy is concerned
More details : https://arxiv.org/abs/2004.12629
this would be helpful for you.
I've drawn a bounding box for each word in my invoice, then I will chose only fields that I want. You can use for that ROI (Region Of Interest)
import pytesseract
import cv2
img = cv2.imread(r'path\Invoice2.png')
d = pytesseract.image_to_data(img, output_type=pytesseract.Output.DICT)
n_boxes = len(d['level'])
for i in range(n_boxes):
(x, y, w, h) = (d['left'][i], d['top'][i], d['width'][i], d['height'][i])
img = cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 1)
cv2.imshow('img', img)
cv2.waitKey(0)
You will get this output:
I am trying to perform perspective transformation with homography using the opencv-python package.
I have a background and foreground image and would like to perform perspective transform and stitch the foreground image on the background image given four (x, y) coordinates, as follow:
bgImg = cv2.imread(BACK_PATH, cv2.IMREAD_COLOR)
fgImg = cv2.imread(FORE_PATH, cv2.IMREAD_COLOR)
bgHeight, bgWidth, dpt = bgImg.shape
origImageCoords = np.array([(0, 0),
(0, bgHeight),
(bgWidth, bgHeight),
(bgWidth, 0)])
stitchingCoords = []
def transformPerspective():
y0 = 285
y1 = 400
x0 = 447
x1 = 600
stitchingCoords.append((x0, y0))
stitchingCoords.append((x0, y1))
stitchingCoords.append((x1, y1))
stitchingCoords.append((x1, y0))
homography = cv2.findHomography(origImageCoords, np.array(stitchingCoords))
dst_corners = cv2.warpPerspective(src=fgImg, M=homography[0], dsize=(bgWidth, bgHeight))
showFinal(bgImg, dst_corners)
After the perspective transformation is done using cv2.findhomography(), I mask the foreground and background images using appropriate masks and add them together, as follow:
def showFinal(src1, src2):
grayed = cv2.cvtColor(src2, cv2.COLOR_BGR2GRAY)
_, grayed = cv2.threshold(grayed, 0, 255, cv2.THRESH_BINARY)
grayedInv = cv2.bitwise_not(grayed)
src1final = cv2.bitwise_and(src1, src1, mask=grayedInv)
src2final = cv2.bitwise_and(src2, src2, mask=grayed)
finalImage = cv2.add(src1final, src2final)
cv2.namedWindow("output", cv2.WINDOW_AUTOSIZE)
cv2.imshow("output", finalImage)
Problem
The problem is the that the find result is wrong because the transformed-foreground image is not stitched inside the four coordinates that I used for finding the homography.
Could anyone guide me as to why this error is occurring?
Expected Output
Actual Output
I have a transparent png image foo.png and I've opened another image with:
im = Image.open("foo2.png")
Now what I need is to merge foo.png with foo2.png.
(foo.png contains some text and I want to print that text on foo2.png)
from PIL import Image
background = Image.open("test1.png")
foreground = Image.open("test2.png")
background.paste(foreground, (0, 0), foreground)
background.show()
First parameter to .paste() is the image to paste. Second are coordinates, and the secret sauce is the third parameter. It indicates a mask that will be used to paste the image. If you pass a image with transparency, then the alpha channel is used as mask.
Check the docs.
Image.paste does not work as expected when the background image also contains transparency. You need to use real Alpha Compositing.
Pillow 2.0 contains an alpha_composite function that does this.
background = Image.open("test1.png")
foreground = Image.open("test2.png")
Image.alpha_composite(background, foreground).save("test3.png")
EDIT: Both images need to be of the type RGBA. So you need to call convert('RGBA') if they are paletted, etc.. If the background does not have an alpha channel, then you can use the regular paste method (which should be faster).
As olt already pointed out, Image.paste doesn't work properly, when source and destination both contain alpha.
Consider the following scenario:
Two test images, both contain alpha:
layer1 = Image.open("layer1.png")
layer2 = Image.open("layer2.png")
Compositing image using Image.paste like so:
final1 = Image.new("RGBA", layer1.size)
final1.paste(layer1, (0,0), layer1)
final1.paste(layer2, (0,0), layer2)
produces the following image (the alpha part of the overlayed red pixels is completely taken from the 2nd layer. The pixels are not blended correctly):
Compositing image using Image.alpha_composite like so:
final2 = Image.new("RGBA", layer1.size)
final2 = Image.alpha_composite(final2, layer1)
final2 = Image.alpha_composite(final2, layer2)
produces the following (correct) image:
One can also use blending:
im1 = Image.open("im1.png")
im2 = Image.open("im2.png")
blended = Image.blend(im1, im2, alpha=0.5)
blended.save("blended.png")
Had a similar question and had difficulty finding an answer. The following function allows you to paste an image with a transparency parameter over another image at a specific offset.
import Image
def trans_paste(fg_img,bg_img,alpha=1.0,box=(0,0)):
fg_img_trans = Image.new("RGBA",fg_img.size)
fg_img_trans = Image.blend(fg_img_trans,fg_img,alpha)
bg_img.paste(fg_img_trans,box,fg_img_trans)
return bg_img
bg_img = Image.open("bg.png")
fg_img = Image.open("fg.png")
p = trans_paste(fg_img,bg_img,.7,(250,100))
p.show()
def trans_paste(bg_img,fg_img,box=(0,0)):
fg_img_trans = Image.new("RGBA",bg_img.size)
fg_img_trans.paste(fg_img,box,mask=fg_img)
new_img = Image.alpha_composite(bg_img,fg_img_trans)
return new_img
Here is my code to merge 2 images of different sizes, each with transparency and with offset:
from PIL import Image
background = Image.open('image1.png')
foreground = Image.open("image2.png")
x = background.size[0]//2
y = background.size[1]//2
background = Image.alpha_composite(
Image.new("RGBA", background.size),
background.convert('RGBA')
)
background.paste(
foreground,
(x, y),
foreground
)
background.show()
This snippet is a mix of the previous answers, blending elements with offset while handling images with different sizes, each with transparency.
the key code is:
_, _, _, alpha = image_element_copy.split()
image_bg_copy.paste(image_element_copy, box=(x0, y0, x1, y1), mask=alpha)
the full function is:
def paste_image(image_bg, image_element, cx, cy, w, h, rotate=0, h_flip=False):
image_bg_copy = image_bg.copy()
image_element_copy = image_element.copy()
image_element_copy = image_element_copy.resize(size=(w, h))
if h_flip:
image_element_copy = image_element_copy.transpose(Image.FLIP_LEFT_RIGHT)
image_element_copy = image_element_copy.rotate(rotate, expand=True)
_, _, _, alpha = image_element_copy.split()
# image_element_copy's width and height will change after rotation
w = image_element_copy.width
h = image_element_copy.height
x0 = cx - w // 2
y0 = cy - h // 2
x1 = x0 + w
y1 = y0 + h
image_bg_copy.paste(image_element_copy, box=(x0, y0, x1, y1), mask=alpha)
return image_bg_copy
the above function supports:
position(cx, cy)
auto resize image_element to (w, h)
rotate image_element without cropping it
horizontal flip