Develop Reference

fitting ellipse in images with poor contrast

I am working on image processing in Python, on the topic of underwater photogrammetry. My goal is to fit an ellipse to fidicual markers, and retrieve its a) center, b) axis and c) orientation.
My markers are
radial,
white on black background, and some have a
binary code:
A ML-model delivers a small image snippets for each marker in each image, containting only the center of the marker.
So far, I've implemented these approaches:
Using openCV:
a) Thresholding, which results in a binary image (cv2.threshold)
b) Find Contours (cv2.findContours)
c) fit Ellipse (v2.fitEllipse)
Using Scikit:
a) Detect edge (using Canny)
b) Apply hough transform
Star operator (work in progress)
a) Estimate ellipse center
b) Send 360rays in all directions
c) Build an array, comprising coordinates of the largest gradient on each ray
d) Calculate best-fit ellipse using least-square method
e) Use the new center to repeat process (possibly several iterations required)
I perform these methods for each color-channel seperately. So far, the results between channels differ within several pixels for the ellipse center.
Do you have any suggestions on what pre-processing methods I should use, prior detecting/fitting the ellipse?
Any thoughts on which of the above methods will lead to the most accurate results?

This is amazing! Thank you. I just started to read about moments (e.g. https://www.pythonpool.com/opencv-moments/) and inertia.
However, there is a challange applying your code to this example:
As you can see, the image was poorly cropped, and the inertia of the image is more in the image center than in the center of the expected ellipse.
My first attempt to fix this is to binarize the image first:
import cv2 as cv2
T = int(cv2.mean(image)[0])
ret,image = cv2.threshold(image,T,255,0)
Is that a reasonable approach? I fear, that the binarization will have an unwanted impact on the moments of inertia. Thank you for claryfying.

This code finds the center of mass of the image, and the main axis of symmetry by calculating the moments of inertia.
I tried many libraries that calculate moments of inertia of images, but they give strange results (like 4x4 matrix for what should be a 2x2 matrix of inertia.
Also, ndimage.measurements.center_of_mass() appears to return (Cy,Cx) (row, column)
So, I resorted to manually calculating the moments of inertia
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image as Pim
from io import BytesIO
import requests
photoURL = "https://i.stack.imgur.com/EcLYk.png"
response = requests.get(photoURL)
image = np.array(Pim.open(BytesIO(response.content)).convert('L')) # Convert to greyscale
plt.imshow(image)
if True: # calculate eigen vectors = main axis of inertia
# xCoord, yCoord are the column and row numbers in image
xCoord, yCoord = np.meshgrid(np.arange(image.shape[1]), np.arange(image.shape[0]))
# mass M is the total sum of Image
M = np.sum(image)
# Cx, Cy are the coordinates of the center of mass
#Cx = sum(xCoord * image) / sum(image)
Cx = np.einsum('ij,ij', xCoord, image)/M
Cy = np.einsum('ij,ij', yCoord, image)/M
# Ixx is the second order moment of image respect to the horizontal axis passing through the center of mass
# Ixx=sum(Image*y^2)
Ixx = np.einsum('ij,ij,ij', yCoord-Cy, yCoord-Cy, image)
# Iyy is the second order moment of image respect to the vertical axis passing through the center of mass
# Iyy=sum(Image*x^2)
Iyy = np.einsum('ij,ij,ij', xCoord-Cx, xCoord-Cx, image)
# Ixy is the second order moment of image respect to both axis passing through the center of mass
# Ixy=sum(Image*x*y)
Ixy = np.einsum('ij,ij,ij', xCoord-Cx, yCoord-Cy, image)
inertiaMatrix = np.array([[Ixx, Ixy],
[Ixy, Iyy]])
eigValues, eigVectors = np.linalg.eig(inertiaMatrix)
# Plot center of mass
plt.scatter(Cx, Cy, c='r')
# Plot eigenvectors from center to direction of eigenvectors
plt.quiver(Cx, Cy, eigVectors[0, 0], eigVectors[1, 0], color='r', scale=2)
plt.quiver(Cx, Cy, eigVectors[0, 1], eigVectors[1, 1], color='r', scale=2)
plt.show()
nothing = 0

How to resize an image with "varying scaling"?

I need to resize an image, but with a "varying scaling" in the y axis, after warping:
Plotted Image
Original input image
Warped output image
The image (left one) was taken at an angle, so I've used the getPerspectiveTransform and warpPerspective OpenCV functions to get the top/plan view of the image (right one).
But, now the top half of the warped image is stretched and the bottom half is squashed, and this amount of stretch/squash is varying continuously as you go down the image. So, I need to do the opposite.
For example: The zebra crossing lines in the warped image are thicker at the top of the image and thinner at the bottom. I want them to all be the same thickness and same vertical distance from each other essentially.
Badly drawn but something like this: (if we ignore the 2 people, I think this is what the final output image should be like.)
predicted output image
My end goal is to measure distance between people's feet in an image (shown by green dots), but I've got that section sorted already.
By vertically scaling the warped image to make it linear, it will allow me to accurately measure the real distance in the x & y direction from a top/plan view, (i.e each pixel in the x or y direction is say 1cm in real distance)
I was thinking of multiplying each row of the image by a factor (e.g. top rows multiply by smaller number like 0.8 or 0.9, and bottom rows multiply by bigger number like 1.1 or 1.2), but I really don't know how to do that.
Code:
import cv2 as cv
from matplotlib import pyplot as plt
import numpy as np
# READ IMAGE
imgOrig = cv.imread('.jpg')
# RESIZE IMAGE
width = int(1000)
ratio = imgOrig.shape[1]/width
height = int(imgOrig.shape[0]/ratio)
dsize = (width, height)
img = cv.resize(imgOrig, dsize)
feetLocation = [[280, 500], [740, 496]]
cv.circle(img,(280, 500),5,(0,255,0),thickness= 10)
cv.circle(img,(740, 496),5,(0,255,0),thickness= 10)
# WARPING
pts1 = np.float32([[0, -0], [width, 0], [-1800, height], [width + 1800, height]])
pts2 = np.float32([[0, 0], [width, 0], [0, height], [width, height]])
M = cv.getPerspectiveTransform(pts1, pts2)
dst = cv.warpPerspective(img, M, (width, height))
#DISPLAY IMAGES
plt.subplot(121),plt.imshow(img),plt.title('Original Image')
plt.subplot(122),plt.imshow(dst),plt.title('Warped Image')
plt.show()

I was working on a solution, before the several edits were applied. I focussed on the actual boxes only. If, instead, you actually need the surrounding, too, the following approach won't help you much, I'm afraid. Also, I assumed the bottom box to be fully included. So, if that one's somehow cut like presented in your new desired final output, additional work would be needed to handle that case.
From the given image, you could mask the gray-ish part around and between the single boxes using the saturation and value channels from the HSV color space:
Following, row-wise sum all pixels, apply some moving average to clean the signal, and detect the peaks in that signal:
The bottom image border must be manually added, since there is no gray-ish border (most likely because the box is somehow cut).
Now, for each of these "peak rows", determine the first and last masked pixels, and build boxes from each two neighbouring "peak rows". Finally, for each of these boxes, apply a distinct perspective transform to a given size. If needed, stack those boxes vertically for example:
That'd be the whole code:
import cv2
import matplotlib.pyplot as plt
import numpy as np
from scipy.signal import find_peaks
# Read original image
imgOrig = cv2.cvtColor(cv2.imread('DInAq.jpg'), cv2.COLOR_BGR2RGB)
# Resize image
width = int(1000)
ratio = imgOrig.shape[1] / width
height = int(imgOrig.shape[0] / ratio)
dsize = (width, height)
img = cv2.resize(imgOrig, dsize)
# Mask low saturation and medium to high value (i.e. gray-ish/white-ish colors)
img_gauss = cv2.GaussianBlur(img, (5, 5), -1)
h, s, v = cv2.split(cv2.cvtColor(img_gauss, cv2.COLOR_BGR2HSV))
mask = (s < 24) & (v > 64)
# Row-wise sum mask pixels, apply moving average filter, and find peaks
row_sum = np.sum(mask, axis=1)
row_sum = np.convolve(row_sum, np.ones(5)/5, 'same')
peaks = find_peaks(row_sum, prominence=50)[0]
peaks = np.insert(peaks, 4, img.shape[0]-1)
# Find first and last pixels per "peak row"
x1 = [np.argwhere(mask[p, :]).min() for p in peaks]
x2 = [np.argwhere(mask[p, :]).max() for p in peaks]
# Collect single boxes
boxes = []
for i in np.arange(len(peaks)-1, 0, -1):
boxes.append([[x1[i], peaks[i]],
[x1[i-1], peaks[i-1]],
[x2[i-1], peaks[i-1]],
[x2[i], peaks[i]]])
# Warp each box individually to a given size
warped = []
bw, bh = [400, 400]
for box in reversed(boxes):
pts1 = np.float32(box)
pts2 = np.float32([[0, bh-1], [0, 0], [bw-1, 0], [bw-1, bh-1]])
M = cv2.getPerspectiveTransform(pts1, pts2)
warped.append(cv2.warpPerspective(img, M, (bw, bh)))
# Output
plt.figure(1)
plt.subplot(121), plt.imshow(img), plt.title('Original image')
for box in boxes:
pts = np.array(box)
plt.plot(pts[:, 0], pts[:, 1], 'rx')
plt.subplot(122), plt.imshow(np.vstack(warped)), plt.title('Warped image')
plt.tight_layout(), plt.show()
That's kind of an automated way to detect and extract the single boxes. For better results, you could set up a simple GUI (solely using OpenCV, for example), and let the user click on the exact corners, and build the boxes to be transformed from there.
----------------------------------------
System information
----------------------------------------
Platform: Windows-10-10.0.16299-SP0
Python: 3.9.1
PyCharm: 2021.1
Matplotlib: 3.4.1
NumPy: 1.20.2
OpenCV: 4.5.1
SciPy: 1.6.2
----------------------------------------

determining the average colour of a given circular sample of an image?

What I am trying to achieve is similar to photoshop/gimp's eyedropper tool: take a round sample of a given area in an image and return the average colour of that circular sample.
The simplest method I have found is to take a 'regular' square sample, mask it as a circle, then reduce it to 1 pixel, but this is very CPU-demanding (especially when repeated millions of times).
A more mathematically complex method is to take a square area and average only the pixels that fall within a circular area within that sample, but determining what pixel is or isn't within that circle, repeated, is CPU-demanding as well.
Is there a more succinct, less-CPU-demanding means to achieve this?

Here's a little example of skimage.draw.circle() which doesn't actually draw a circle but gives you the coordinates of points within a circle which you can use to index Numpy arrays with.
#!/usr/bin/env python3
import numpy as np
from skimage.io import imsave
from skimage.draw import circle
# Make rectangular canvas of mid-grey
w, h = 200, 100
img = np.full((h, w), 128, dtype=np.uint8)
# Get coordinates of points within a central circle
Ycoords, Xcoords = circle(h//2, w//2, 45)
# Make all points in circle=200, i.e. fill circle with 200
img[Ycoords, Xcoords] = 200
# Get mean of points in circle
print(img[Ycoords, Xcoords].mean()) # prints 200.0
# DEBUG: Save image for checking
imsave('result.png',img)

I'm sure that there's a more succinct way to go about it, but:
import math
import numpy as np
import imageio as ioimg # as scipy's i/o function is now depreciated
from skimage.draw import circle
import matplotlib.pyplot as plt
# base sample dimensions (rest below calculated on this).
# Must be an odd number.
wh = 49
# tmp - this placement will be programmed later
dp = 500
#load work image (from same work directory)
img = ioimg.imread('830.jpg')
# convert to numpy array (droppying the alpha while we're at it)
np_img = np.array(img)[:,:,:3]
# take sample of resulting array
sample = np_img[dp:wh+dp, dp:wh+dp]
#==============
# set up numpy circle mask
## this mask will be multiplied against each RGB layer in extracted sample area
# set up basic square array
sample_mask = np.zeros((wh, wh), dtype=np.uint8)
# set up circle centre coords and radius values
xy, r = math.floor(wh/2), math.ceil(wh/2)
# use these values to populate circle area with ones
rr, cc = circle(xy, xy, r)
sample_mask[rr, cc] = 1
# add axis to make array multiplication possible (do I have to do this)
sample_mask = sample_mask[:, :, np.newaxis]
result = sample * sample_mask
# count number of nonzero values (this will be our median divisor)
nz = np.count_nonzero(sample_mask)
sample_color = []
for c in range(result.shape[2]):
sample_color.append(int(round(np.sum(result[:,:,c])/nz)))
print(sample_color) # will return array like [225, 205, 170]
plt.imshow(result, interpolation='nearest')
plt.show()
Perhaps asking this question here wasn't necessary (it has been a while since I've python-ed, and was hoping that some new library had been developed for this since), but I hope this can be a reference for others who have the same goal.
This operation will be performed for every pixel in the image (sometimes millions of times) for thousands of images (scanned pages), so therein are my performance issue worries, but thanks to numpy, this code is pretty quick.

matplotlib markers / mask on image pixels

So I have an image and I have a pixel mask for that image, where the mask is the same size as the image and contains values of 0 and 1, where if it is 0 I don't want to modify the image, and if it is 1 I want to add a transparent color over that pixel of the image.
Basically I want to highlight certain segments of the image but still see what is underneath.
Now I have searched high and low but haven't found a simple way to do this. I used np.where with the mask to get the pixel locations of the 1's to use with the plot functions. I first tried scatter plots with a small marker size and no edge color (small scatter plot markers in matplotlib are always black), but the markers are not one image pixel in size, they seem to be an absolute size and so depending on the size of the figure the transparency is affected and weird patterns are created from the overlapping markers.
Just the regular pyplot plot function created the exact look I desired (where the coloring was smooth and invariant to figure size) but it also colored horizontal connections between disjoint segments in the mask (since it is drawing lines I guess), so I couldn't use that.
What worked the best was patches, which I came across in this question: (How to set a fixed/static size of circle marker on a scatter plot?). I found that rectangular patches with width and height of 1 gave me the exact desired effect, where I could put a transparent color over certain pixels of the image. However this proved to produce a ton (tens of thousands) of rectangles for certain images, and so it was quite slow. Even when using a PatchCollection instead of calling addPatch every time it was still slow.
Now I can probably just join adjacent rectangles to reduce the number of things needing to be drawn, but I was just wondering if there was an easier way to do this?
Thanks.

You can do a semitransparent overlay either using masked arrays or by setting the alpha values in an RGBA image. Here are both worked through (using the example of three semitransparent red squares placed over a circular pattern), and they give similar images (so I'll only show one):
from pylab import *
from numpy import ma
x = y = linspace(-6, 6, 100)
X, Y = meshgrid(x, y)
z3 = X*X + Y*Y # circular pattern
# first, do this with a masked array
figure()
# z4 = 3 diagonal square
# zm = a uniform image (ones), with a mask of squares (~z4)
z4 = np.repeat(np.repeat(eye(3, dtype=bool), 40, axis=0), 40, axis=1)
zm = ma.masked_where(~z4, ones((120,120)))
imshow(z3, cmap=cm.jet)
imshow(zm, cmap=cm.bwr, alpha=.3, vmin=0, vmax=1) #cm.bwr is an easy way to get red
# do this by changing alpha for each pixel
figure()
z5 = zeros((120, 120, 4), dtype=float)
z5[..., 0] = 1
z5[..., 3] = .4*z4.astype(float)
imshow(z3, cmap=cm.jet)
imshow(z5)
show()
I think both approaches can produce the same results for all cases, but:
1. the masked arrays can be a more direct approach if the mask or composition becomes complicated, and masking gives you more flexibility in drawing your overlay image since, for example, you can use colormaps rather than specifying the full RGBA for every pixel, but,
2. the masked array approach doesn't give full pixel-by-pixel control over the alpha value like RGBA does.
z1 = sin(X*Y)
z1 = cos(2*X)
z2 = cos(5*(X+Y))
zm = ma.masked_where( (z2<.5) & (Y>0), z1)
figure()
imshow(z3)
imshow(zm, cmap=cm.gray, alpha=.4, vmin=-2, vmax=2)
show()
It's a bit crazy, but here's what's going on: The primary image is a circular pattern that goes from blue to red (z3). Then there are vertical bars that faintly shade this (z1) but only in half of the figure and in narrow alternate diagonal bands on the other half (due to the mask). Here's a more complicated image using masked arrays:

Just to add on to what tom10 has posted, the masked arrays do work great with colormaps, but I also wrote a small function in the meantime that should work with any RGB color tuple.
def overlayImage(im, mask, col, alpha):
maskRGB = np.tile(mask[..., np.newaxis], 3)
untocuhed = (maskRGB == False) * im
overlayComponent = alpha * np.array(col) * maskRGB
origImageComponent = (1 - alpha) * maskRGB * im
return untocuhed + overlayComponent + origImageComponent
im is the rgb image
mask is a boolean mask of the image, such that mask.shape + (3,) = im.shape
col is just the 3-tuple rgb value you want to mask the image with
alpha is just the alpha value / transparency for the mask

I also needed a clear contour on my areas. Thus, you can easily add a contour plot on top: e.g., create a dummy numpy array and set a different value in each area of interest.
Here's an example build on top of tom10's answer with a different condition:
x = y = linspace(-6, 6, 100)
X, Y = meshgrid(x, y)
z3 = X*X + Y*Y # circular pattern
# first, do this with a masked array
figure()
imshow(z3, cmap=cm.jet, extent = (-6,6,-6,6));
zm = ma.masked_where((z3>=0.7) & (z3<=1.5), ones(np.shape(z3)));
imshow(zm, cmap=cm.bwr, alpha=.4, vmin=0, vmax=1, extent = (-6,6,-6,6)) #cm.bwr is an easy way to get red
# Build dummy array of 1s and 0s (you can play with different values to obtain different contours for different regions):
temp_vector = ones(np.shape(z3));
temp_vector[(z3>=0.7) & (z3<=1.5)] = 0.0;
temp_vector[(z3>8.2)] = 2.0; # etc.
# Create contour. I found only one contour necessary:
contour(X, Y, temp_vector, 1, colors=['r','g']);
show()
Which yields:

Python OpenCV - Find black areas in a binary image

There is any method/function in the python wrapper of Opencv that finds black areas in a binary image? (like regionprops in Matlab)
Up to now I load my source image, transform it into a binary image via threshold and then invert it to highlight the black areas (that now are white).
I can't use third party libraries such as cvblobslob or cvblob

Basically, you use the findContours function, in combination with many other functions OpenCV provides for especially this purpose.
Useful functions used (surprise, surprise, they all appear on the Structural Analysis and Shape Descriptors page in the OpenCV Docs):
findContours
drawContours
moments
contourArea
arcLength
boundingRect
convexHull
fitEllipse
example code (I have all the properties from Matlab's regionprops except WeightedCentroid and EulerNumber - you could work out EulerNumber by using cv2.RETR_TREE in findContours and looking at the resulting hierarchy, and I'm sure WeightedCentroid wouldn't be that hard either.
# grab contours
cs,_ = cv2.findContours( BW.astype('uint8'), mode=cv2.RETR_LIST,
method=cv2.CHAIN_APPROX_SIMPLE )
# set up the 'FilledImage' bit of regionprops.
filledI = np.zeros(BW.shape[0:2]).astype('uint8')
# set up the 'ConvexImage' bit of regionprops.
convexI = np.zeros(BW.shape[0:2]).astype('uint8')
# for each contour c in cs:
# will demonstrate with cs[0] but you could use a loop.
i=0
c = cs[i]
# calculate some things useful later:
m = cv2.moments(c)
# ** regionprops **
Area = m['m00']
Perimeter = cv2.arcLength(c,True)
# bounding box: x,y,width,height
BoundingBox = cv2.boundingRect(c)
# centroid = m10/m00, m01/m00 (x,y)
Centroid = ( m['m10']/m['m00'],m['m01']/m['m00'] )
# EquivDiameter: diameter of circle with same area as region
EquivDiameter = np.sqrt(4*Area/np.pi)
# Extent: ratio of area of region to area of bounding box
Extent = Area/(BoundingBox[2]*BoundingBox[3])
# FilledImage: draw the region on in white
cv2.drawContours( filledI, cs, i, color=255, thickness=-1 )
# calculate indices of that region..
regionMask = (filledI==255)
# FilledArea: number of pixels filled in FilledImage
FilledArea = np.sum(regionMask)
# PixelIdxList : indices of region.
# (np.array of xvals, np.array of yvals)
PixelIdxList = regionMask.nonzero()
# CONVEX HULL stuff
# convex hull vertices
ConvexHull = cv2.convexHull(c)
ConvexArea = cv2.contourArea(ConvexHull)
# Solidity := Area/ConvexArea
Solidity = Area/ConvexArea
# convexImage -- draw on convexI
cv2.drawContours( convexI, [ConvexHull], -1,
color=255, thickness=-1 )
# ELLIPSE - determine best-fitting ellipse.
centre,axes,angle = cv2.fitEllipse(c)
MAJ = np.argmax(axes) # this is MAJor axis, 1 or 0
MIN = 1-MAJ # 0 or 1, minor axis
# Note: axes length is 2*radius in that dimension
MajorAxisLength = axes[MAJ]
MinorAxisLength = axes[MIN]
Eccentricity = np.sqrt(1-(axes[MIN]/axes[MAJ])**2)
Orientation = angle
EllipseCentre = centre # x,y
# ** if an image is supplied with the BW:
# Max/Min Intensity (only meaningful for a one-channel img..)
MaxIntensity = np.max(img[regionMask])
MinIntensity = np.min(img[regionMask])
# Mean Intensity
MeanIntensity = np.mean(img[regionMask],axis=0)
# pixel values
PixelValues = img[regionMask]

After inverting binary image to turn black to white areas, apply cv.FindContours function. It will give you boundaries of the region you need.
Later you can use cv.BoundingRect to get minimum bounding rectangle around region. Once you got the rectangle vertices, you can find its center etc.
Or to find centroid of region, use cv.Moment function after finding contours. Then use cv.GetSpatialMoments in x and y direction. It is explained in opencv manual.
To find area, use cv.ContourArea function.

Transform it to binary image using threshold with the CV_THRESH_BINARY_INV flag, you get threshold + inversion in one step.

If you can consider using another free library, you could use SciPy. It has a very convenient way of counting areas:
from scipy import ndimage
def count_labels(self, mask_image):
"""This function returns the count of labels in a mask image."""
label_im, nb_labels = ndimage.label(mask_image)
return nb_labels
If necessary you can use:
import cv2 as opencv
image = opencv.inRange(image, lower_threshold upper_threshold)
before to get a mask image, which contains only black and white, where white are the objects in the given range.

I know this is an old question, but for completeness I wanted to point out that cv2.moments() will not always work for small contours. In this case, you can use cv2.minEnclosingCircle() which will always return the center coordinates (and radius), even if you have only a single point. Slightly more resource-hungry though, I think...