Develop Reference

How to find the largest blank(white) square area in the doc and return its coordinates and area?

I need to find the largest empty area in the document and display its coordinates, center point and area, using python to put a QR Code there.
I think OpenCV and Numpy should be enough for this task.
What kinda THRESH to use? Because there are a lot of types of scans:
gray, BW, with color, and how to find the contour properly?
How this can be implemented in the fastest way? An example using the
first scan from google is attached, where you can see that the code
should find the largest empty square area.
#Mark Setchell Thanks! This code works perfectly for all docs with a white background, but when I use smth with a color in the background it finds a completely different area. Also, to keep thin lines in the docs I used Erode after thresholding. Tried to change thresholding and erode parameters, still not working properly.
Edited post, added color pictures.

Here's a possible approach:
#!/usr/bin/env python3
import cv2
import numpy as np
def largestSquare(im):
# Make image square of 100x100 to simplify and speed up
s = 100
work = cv2.resize(im, (s,s), interpolation=cv2.INTER_NEAREST)
# Make output accumulator - uint16 is ok because...
# ... max value is 100x100, i.e. 10,000 which is less than 65,535
# ... and you can make a PNG of it too
p = np.zeros((s,s), np.uint16)
# Find largest square
for i in range(1, s):
for j in range(1, s):
if (work[i][j] > 0 ):
p[i][j] = min(p[i][j-1], p[i-1][j], p[i-1][j-1]) + 1
else:
p[i][j] = 0
# Save result - just for illustration purposes
cv2.imwrite("result.png",p)
# Work out what the actual answer is
ind = np.unravel_index(np.argmax(p, axis=None), p.shape)
print(f'Location: {ind}')
print(f'Length of side: {p[ind]}')
# Load image and threshold
im = cv2.imread('page.png', cv2.IMREAD_GRAYSCALE)
_, thr = cv2.threshold(im,127,255,cv2.THRESH_BINARY | cv2.THRESH_OTSU)
# Get largest white square
largestSquare(thr)
Output
Location: (21, 77)
Length of side: 18
Notes:
I edited out your red annotation so it didn't interfere with my algorithm.
I did Otsu thresholding to get pure black and white - that may or may not be appropriate to your use case. It will depend on your scans and paper background etc.
I scaled the image down to 100x100 so it doesn't take all day to run. You will need to scale the results back up to the size of your original image but I assume you can do that easily enough.
Keywords: Image processing, image, Python, OpenCV, largest white square, largest empty space.

How to skew an image by moving its vertex?

I'm trying to find a way to transform an image by translating one of its vertexes.
I have already found various methods for transforming an image like rotation and scaling, but none of the methods involved skewing like so:
There is shearing, but it's not the same since it can move two or more of the image's vertex while I only want to move one.
What can I use that can perform such an operation?

I took your "cat-thing" and resized it to a nice size, added some perfectly vertical and horizontal white gridlines and added some extra canvas in red at the bottom to give myself room to transform it. That gave me this which is 400 pixels wide and 450 pixels tall:
I then used ImageMagick to do a "Bilinear Forward Transform" in Terminal. Basically you give it 4 pairs of points, the first pair is where the top-left corner is before the transform and then where it must move to. The next pair is where the top-right corner is originally followed by where it ends up. Then the bottom-right. Then the bottom-left. As you can see, 3 of the 4 pairs are unmoved - only the bottom-right corner moves. I also made the virtual pixel black so you can see where pixels were invented by the transform in black:
convert cat.png -matte -virtual-pixel black -interpolate Spline -distort BilinearForward '0,0 0,0 399,0 399,0 399,349 330,430 0,349 0,349' bilinear.png
I also did a "Perspective Transform" using the same transform coordinates:
convert cat.png -matte -virtual-pixel black -distort Perspective '0,0 0,0 399,0 399,0 399,349 330,430 0,349 0,349' perspective.png
Finally, to illustrate the difference, I made a flickering comparison between the 2 images so you can see the difference:
I am indebted to Anthony Thyssen for his excellent work here which I commend to you.
I understand you were looking for a Python solution and would point out that there is a Python binding to ImageMagick called Wand which you may like to use - here.
Note that I only used red and black to illustrate what is going on (atop the Stack Overflow white background) and where aspects of the result come from, you would obviously use white for both!

The perspective transformation is likely what you want, since it preserves straight lines at any angle. (The inverse bilinear only preserves horizontal and vertical straight lines).
Here is how to do it in ImageMagick, Python Wand (based upon ImageMagick) and Python OpenCV.
Input:
ImageMagick
(Note the +distort makes the output the needed size to hold the full result and is not restricted to the size of the input. Also the -virtual-pixel white sets color of the area outside the image pixels to white. The points are ordered clockwise from the top left in pairs as inx,iny outx,outy)
convert cat.png -virtual-pixel white +distort perspective \
"0,0 0,0 359,0 359,0 379,333 306,376 0,333 0,333" \
cat_perspective_im.png
Python Wand
(Note the best_fit=true makes the output the needed size to hold the full result and is not restricted to the size of the input.)
#!/bin/python3.7
from wand.image import Image
from wand.display import display
with Image(filename='cat.png') as img:
img.virtual_pixel = 'white'
img.distort('perspective', (0,0, 0,0, 359,0, 359,0, 379,333, 306,376, 0,333, 0,333), best_fit=True)
img.save(filename='cat_perspective_wand.png')
display(img)
Python OpenCV
#!/bin/python3.7
import cv2
import numpy as np
# Read source image.
img_src = cv2.imread('cat.png')
# Four corners of source image
# Coordinates are in x,y system with x horizontal to the right and y vertical downward
pts_src = np.float32([[0,0], [359,0], [379,333], [0,333]])
# Four corners of destination image.
pts_dst = np.float32([[0, 0], [359,0], [306,376], [0,333]])
# Get perspecive matrix if only 4 points
m = cv2.getPerspectiveTransform(pts_src,pts_dst)
# Warp source image to destination based on matrix
# size argument is width x height
# compute from max output coordinates
img_out = cv2.warpPerspective(img_src, m, (359+1,376+1), cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT, borderValue=(255, 255, 255))
# Save output
cv2.imwrite('cat_perspective_opencv.png', img_out)
# Display result
cv2.imshow("Warped Source Image", img_out)
cv2.waitKey(0)
cv2.destroyAllWindows()

Detect if an OCR text image is upside down

I have some hundreds of images (scanned documents), most of them are skewed. I wanted to de-skew them using Python.
Here is the code I used:
import numpy as np
import cv2
from skimage.transform import radon
filename = 'path_to_filename'
# Load file, converting to grayscale
img = cv2.imread(filename)
I = cv2.cvtColor(img, COLOR_BGR2GRAY)
h, w = I.shape
# If the resolution is high, resize the image to reduce processing time.
if (w > 640):
I = cv2.resize(I, (640, int((h / w) * 640)))
I = I - np.mean(I) # Demean; make the brightness extend above and below zero
# Do the radon transform
sinogram = radon(I)
# Find the RMS value of each row and find "busiest" rotation,
# where the transform is lined up perfectly with the alternating dark
# text and white lines
r = np.array([np.sqrt(np.mean(np.abs(line) ** 2)) for line in sinogram.transpose()])
rotation = np.argmax(r)
print('Rotation: {:.2f} degrees'.format(90 - rotation))
# Rotate and save with the original resolution
M = cv2.getRotationMatrix2D((w/2,h/2),90 - rotation,1)
dst = cv2.warpAffine(img,M,(w,h))
cv2.imwrite('rotated.jpg', dst)
This code works well with most of the documents, except with some angles: (180 and 0) and (90 and 270) are often detected as the same angle (i.e it does not make difference between (180 and 0) and (90 and 270)). So I get a lot of upside-down documents.
Here is an example:
The resulted image that I get is the same as the input image.
Is there any suggestion to detect if an image is upside down using Opencv and Python?
PS: I tried to check the orientation using EXIF data, but it didn't lead to any solution.
EDIT:
It is possible to detect the orientation using Tesseract (pytesseract for Python), but it is only possible when the image contains a lot of characters.
For anyone who may need this:
import cv2
import pytesseract
print(pytesseract.image_to_osd(cv2.imread(file_name)))
If the document contains enough characters, it is possible for Tesseract to detect the orientation. However, when the image has few lines, the orientation angle suggested by Tesseract is usually wrong. So this can not be a 100% solution.

Python3/OpenCV4 script to align scanned documents.
Rotate the document and sum the rows. When the document has 0 and 180 degrees of rotation, there will be a lot of black pixels in the image:
Use a score keeping method. Score each image for it's likeness to a zebra pattern. The image with the best score has the correct rotation. The image you linked to was off by 0.5 degrees. I omitted some functions for readability, the full code can be found here.
# Rotate the image around in a circle
angle = 0
while angle <= 360:
# Rotate the source image
img = rotate(src, angle)
# Crop the center 1/3rd of the image (roi is filled with text)
h,w = img.shape
buffer = min(h, w) - int(min(h,w)/1.15)
roi = img[int(h/2-buffer):int(h/2+buffer), int(w/2-buffer):int(w/2+buffer)]
# Create background to draw transform on
bg = np.zeros((buffer*2, buffer*2), np.uint8)
# Compute the sums of the rows
row_sums = sum_rows(roi)
# High score --> Zebra stripes
score = np.count_nonzero(row_sums)
scores.append(score)
# Image has best rotation
if score <= min(scores):
# Save the rotatied image
print('found optimal rotation')
best_rotation = img.copy()
k = display_data(roi, row_sums, buffer)
if k == 27: break
# Increment angle and try again
angle += .75
cv2.destroyAllWindows()
How to tell if the document is upside down? Fill in the area from the top of the document to the first non-black pixel in the image. Measure the area in yellow. The image that has the smallest area will be the one that is right-side-up:
# Find the area from the top of page to top of image
_, bg = area_to_top_of_text(best_rotation.copy())
right_side_up = sum(sum(bg))
# Flip image and try again
best_rotation_flipped = rotate(best_rotation, 180)
_, bg = area_to_top_of_text(best_rotation_flipped.copy())
upside_down = sum(sum(bg))
# Check which area is larger
if right_side_up < upside_down: aligned_image = best_rotation
else: aligned_image = best_rotation_flipped
# Save aligned image
cv2.imwrite('/home/stephen/Desktop/best_rotation.png', 255-aligned_image)
cv2.destroyAllWindows()

Assuming you did run the angle-correction already on the image, you can try the following to find out if it is flipped:
Project the corrected image to the y-axis, so that you get a 'peak' for each line. Important: There are actually almost always two sub-peaks!
Smooth this projection by convolving with a gaussian in order to get rid of fine structure, noise, etc.
For each peak, check if the stronger sub-peak is on top or at the bottom.
Calculate the fraction of peaks that have sub-peaks on the bottom side. This is your scalar value that gives you the confidence that the image is oriented correctly.
The peak finding in step 3 is done by finding sections with above average values. The sub-peaks are then found via argmax.
Here's a figure to illustrate the approach; A few lines of you example image
Blue: Original projection
Orange: smoothed projection
Horizontal line: average of the smoothed projection for the whole image.
here's some code that does this:
import cv2
import numpy as np
# load image, convert to grayscale, threshold it at 127 and invert.
page = cv2.imread('Page.jpg')
page = cv2.cvtColor(page, cv2.COLOR_BGR2GRAY)
page = cv2.threshold(page, 127, 255, cv2.THRESH_BINARY_INV)[1]
# project the page to the side and smooth it with a gaussian
projection = np.sum(page, 1)
gaussian_filter = np.exp(-(np.arange(-3, 3, 0.1)**2))
gaussian_filter /= np.sum(gaussian_filter)
smooth = np.convolve(projection, gaussian_filter)
# find the pixel values where we expect lines to start and end
mask = smooth > np.average(smooth)
edges = np.convolve(mask, [1, -1])
line_starts = np.where(edges == 1)[0]
line_endings = np.where(edges == -1)[0]
# count lines with peaks on the lower side
lower_peaks = 0
for start, end in zip(line_starts, line_endings):
line = smooth[start:end]
if np.argmax(line) < len(line)/2:
lower_peaks += 1
print(lower_peaks / len(line_starts))
this prints 0.125 for the given image, so this is not oriented correctly and must be flipped.
Note that this approach might break badly if there are images or anything not organized in lines in the image (maybe math or pictures). Another problem would be too few lines, resulting in bad statistics.
Also different fonts might result in different distributions. You can try this on a few images and see if the approach works. I don't have enough data.

You can use the Alyn module. To install it:
pip install alyn
Then to use it to deskew images(Taken from the homepage):
from alyn import Deskew
d = Deskew(
input_file='path_to_file',
display_image='preview the image on screen',
output_file='path_for_deskewed image',
r_angle='offest_angle_in_degrees_to_control_orientation')`
d.run()
Note that Alyn is only for deskewing text.

Lines connecting corners over hand drawn image in python with cv2

I am looking to detect lines connecting corners over a hand drawn image like this. I am using Harris Corner Detection to find the corners of the image. Next I am connecting all of the corners with lines and iterating though the points to see if they match the pixels from original image and setting a threshold for each line pixel cover to say what is acceptable to say it is a correct line connecting corners.Image of connected lines. It works... but it is very slow. Is there a better way to do this or different method I should use? (Hough lines will not work because of the possibility of curved lines, I only want the lines connecting corners.
for i in c_corners: #corners thru harris and coorected with subpix
x1,y1 = i.ravel()
for k in c_corners:
x2,y2 = k.ravel()
if x1 != x2 and y1 != y2: #ignore vertical lines
linePoints = line_points(x1,y1, x2,y2) # function to get line pnts
totalLinePoints = len(linePoints)
coverPoints = 0
########## This is where I think the slow down is happening and could be optimized
for m in originalImage: #image is dialated to help detection
for n in linePoints:
match = np.all(m == n)
if match == True:
coverPoints += 1
print("Line Cover = ", (coverPoints/totalLinePoints))
if (coverPoints/totalLinePoints) > .65:
good_lines.append([x1,y1,x2,y2])
Any help at all is appreciated, thank you!

My original approach was to create a blank image and draw each line on it, and then use cv2.bitwise_and() with the binary (dilated) image to count how many pixels were in agreement, and if they met a threshold, then draw those lines over the original image. However setting a threshold for the number of pixels penalizes small lines. A better indicator would be the ratio of the number of correct matches to incorrect matches (I realize now that's what you were actually doing). Furthermore this is a little more robust towards dilation and the line thickness you choose to draw your lines.
However the general method you're using is not very robust to issues in the drawing where, like this one, synthetic lines may be able to fit easily to lines they don't belong to, because many drawn curves may hit a line segment. You can see this issue in the output of my code:
I simply hardcoded some corner estimates and went from there. Note the use of itertools to help create all possible pairs of points to define line segments.
import cv2
import numpy as np
import itertools
img = cv2.imread('drawing.png')
bin_inv = cv2.bitwise_not(img) # flip image colors
bin_inv = cv2.cvtColor(bin_inv, cv2.COLOR_BGR2GRAY) # make one channel
bin_inv = cv2.dilate(bin_inv, np.ones((5,5)))
corners = ((517, 170),
(438, 316),
(574, 315),
(444, 436),
(586, 436))
lines = itertools.combinations(corners,2) # create all possible lines
line_img = np.ones_like(img)*255 # white image to draw line markings on
for line in lines: # loop through each line
bin_line = np.zeros_like(bin_inv) # create a matrix to draw the line in
start, end = line # grab endpoints
cv2.line(bin_line, start, end, color=255, thickness=5) # draw line
conj = (bin_inv/255 + bin_line/255) # create agreement image
n_agree = np.sum(conj==2)
n_wrong = np.sum(conj==1)
if n_agree/n_wrong > .05: # high agreements vs disagreements
cv2.line(line_img, start, end, color=[0,200,0], thickness=5) # draw onto original img
# combine the identified lines with the image
marked_img = cv2.addWeighted(img, .5, line_img, .5, 1)
cv2.imwrite('marked.png', marked_img)
I tried a lot of different settings (playing with thickness, dilation, different ratios, etc) and couldn't get that spurious longer line from appearing. It fits the original black pixels super well though, so I'm not sure how you would be able to get rid of it if you used this method. It's got the curve from the top-right line going for it, as well as the middle line it crosses, and the curve at the bottom right which trends that direction for a bit. Regardless, this only takes two seconds to run, so at least it's faster than your current code.

How to detect a laser line in an image using Python

What's the quickest most reliable method of detecting a roughly horizontal red laser line in an image using Python? I'm working on a small project related to 3d laser scanning, and I need to be able to detect the laser in an image in order to calculate distance from its distortion.
To start, I have two images, a reference image A known to contain no laser line, and an image B that definitely contains a laser line, possibly distorted. e.g.
Sample image A:
Sample image B:
Since these are RGB, but the laser is red, I remove some noise by stripping out the blue and green channels using this function:
from PIL import Image
import numpy as np
def only_red(im):
"""
Strips out everything except red.
"""
data = np.array(im)
red, green, blue, alpha = data.T
im2 = Image.fromarray(red.T)
return im2
That gets me these images:
Next, I try and eliminate more noise by taking the difference of these two images using PIL.ImageChops.difference(). Ideally, the exposure between the two images would be identical, causing the difference to contain nothing except the laser line. Unfortunately, because the laser adds light, the exposure and overall brightness of each image is significantly different, resulting in a difference that still has considerable noise. e.g.
My final step is to make a "best guess" as to where the line is. Since I know the line will be roughly horizontal and the laser line should be the brightest thing in the image, I scan each column and find the row with the brightest pixel, which I assume to be the laser line. The code for this is:
import os
from PIL import Image, ImageOps
import numpy as np
x = Image.open('laser-diff.png', 'r')
x = x.convert('L')
out = Image.new("L", x.size, "black")
pix = out.load()
y = np.asarray(x.getdata(), dtype=np.float64).reshape((x.size[1], x.size[0]))
print y.shape
for col_i in xrange(y.shape[1]):
col_max = max([(y[row_i][col_i], row_i) for row_i in xrange(y.shape[0])])
col_max_brightness, col_max_row = col_max
print col_i, col_max
pix[col_i, col_max_row] = 255
out.save('laser-line.png')
All I really need to perform my distance calculation is the array of col_max values, but the laser-line.png helps me visualize the success, and looks like:
As you can see, the estimate is pretty close, but it still has some noise, mostly on the left-hand side of the image where the laser line is absorbed by a matte black finish.
What can I do to improve my accuracy and/or speed? I'm trying to run this on an ARM platform like the Raspberry Pi, so I'm worried my code might to be too inefficient to run well.
I'm not fully familiar with Numpy's matrix functions, so I had to settle for a slow for loop to scan each column instead of something more efficient. Is there a fast way to find the row with the brightest pixel per column in Numpy?
Also, is there a reliable way to equalize the images prior to performing the difference without dimming the laser line?

First enter the color that is the laser and leaves only the red color (in this case). Then apply the same effects and check the result.
In this case, you will have a much less polluted result.
Result
A problem is encountered in analyzing the red on the door, that has been lost.

I tried to do something. I don't think it's totally robust. But on your example it works relatively well.
I used canny edge detection to detect edge in your "difference" image. And then applied the Hough line transform as in this tutorial.
So I started with your processed image (that I call lineDetection.jpg in the code).
Here is the final script
import cv2
import numpy as np
img = cv2.imread('lineDetection.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
edges = cv2.Canny(gray,10,100)
minLineLength = 50
maxLineGap = 20
lines = cv2.HoughLinesP(edges,0.05,np.pi/5000,10,minLineLength,maxLineGap)
print(len(lines))
for i in range(len(lines)):
x1,y1,x2,y2 = lines[i][0]
cv2.line(img,(x1,y1),(x2,y2),(0,255,0),2)
cv2.imwrite('houghlines5.jpg',img)
In green line detected on the processed image. (You could add it to the original image for nicer effect)
Hope it helps.

First you can probably rescale the intensity of your negative image before subtracting it from your positive, to remove more noise. For example maybe rescaling by the ratios of the average intesity might be a good first try?
You can also try to put a threshold: if your max in below whatever good value, then it is probably not your laser but a noisy point...
Then yes numpy can find the best row / col with the argmax function.