I have trained a face segmentation model on the CelebA-Mask-HQ dataset (https://github.com/switchablenorms/CelebAMask-HQ ) that is able to create a color segmentation mapping of an image with different colours for the background, eyes, face, hair, etc. The model produces a numpy array of shape (1024,1024,3). The outputted segmentation maps are a bit noisy, with some random pixels in the face labelled as eyes for example, or cloth labels popping up when it is actually background, please see the image below:
As you can see in the image, in the top left corner you see green pixels and in the face around the mustache you see green pixels (above the yellow upper lip map).
I would like to remove this 'noise' from the segmentation map, by changing these wrongly labeled small segments in the image, which are surrounded by larger correctly labeled areas, automatically to the most dominant color in that area (with adaptable window size). I could not find built-in opencv functionality for this. Do you know any efficient way to do this (I need to 'denoise' a large set of images, so ideally in a vectorized numpy-only way)?
It is very important that the image after denoising only contains the set of predefined label colors (19 different colors in total), so the noise needs to be recolored in an absolute manner without averaging (which would introduce new colors to the color palette of the image).
Thank you!
I can point you away from openCV and towards scikit-image which I am more familiar with. I would tackle this using an approach borrowed from this tutorial.
Specifically, I would do something like this:
label_image = label(image)
for region in regionprops(label_image):
# only recolor areas that are under a certain threshold size
if region.area <= 100:
#get creative with which color to recolor with...
minr, minc, maxr, maxc = region.bbox
colors = np.bincount(label_image[minr : maxr, minc:maxc])
max_color = -1
for i in range(len(colors)):
if (colors[i] > max_color) and (i != region.label):
max_color = colors[i]
crop_image = label_image[minr : maxr, minc:maxc]
label_image[minr : maxr, minc:maxc][crop_image == region.label] = max_color
I haven't tried this code out...but I think something like this may work. Let me know if it is helpful or not.
Related
I am doing instance semantic segmentation and my main goal is to be able to change the color of the detected objects in the image. I obtained the mask and now when it comes to coloring I can't figure out how to maintain the intensity of the color from the original object so the colored one doesnt look so artificial.
Let me explain it on example. The image below represent the walls masks. Now I want to have the masks reflect original shading/intensity/gradient ( I am not sure what exactly plays a major part here), so the walls after coloring look normal.
images comparison:
Image using cv2.min
raw mask
I tried doing cv2.min and just multiplying the mask by the original image in the location of coloring just with /255. normalization, It is better but I still feel like there is some much cleverer way to do it.
color = (195, 123, 150)
rgb[markers == clicked_pixel] = color
rgb = cv2.min(rgb, rgb_copy)
# alternatively
rgb[markers == clicked_pixel] = color * (rgb_copy[markers == clicked_pixel]/255)
painted = pygame.surfarray.make_surface(rgb)
painted = pygame.transform.scale(painted, size)
display.blit(painted, (0, 0))
I have let's say the following image:
I've figured out how to get each line using EASYOCR. However, I want to know what color is of the text. I've tried to apply a threshold and use bitmasking, but what would I do if the background color is of anything other color than white?
As of the below comment, I have changed my code to this:
def dominant_colors(image,n): # PIL image input
image = image.resize((150, 150)) # optional, to reduce time
ar = np.asarray(image)
shape = ar.shape
ar = ar.reshape(np.product(shape[:2]), shape[2]).astype(float)
kmeans = sklearn.cluster.MiniBatchKMeans(
n_clusters=n,
init="k-means++",
max_iter=20,
random_state=1000
).fit(ar)
codes = kmeans.cluster_centers_
vecs, _dist = scipy.cluster.vq.vq(ar, codes) # assign codes
counts, _bins = np.histogram(vecs, len(codes)) # count occurrences
colors = []
for index in np.argsort(counts)[::-1]:
# if index!=3:
colors.append(tuple([int(code) for code in codes[index]]))
return colors
dc = dominant_colors(Image.open('./mix.png'),2)
Now, it is working, however it is highly dependent on the image provided. When the FONTS AND WORDS are different, the results are quite different.
On getting the result, and on drawing back on image, it can clearly be seen that for some part, the detected is incorrect
I came across this problem almost 2 years ago and here's what I did to solve the issue.
I used kmeans on image to cluster it into k = 2 clusters. The output would be 2 most prominent clusters namely the background and the foreground. You can convert this into a binary image by using open-cv binary thresholding. Now at this point you don't know which one is foreground and which one is background, so you use pixel count on the binary image. In my case the background always had more pixels so that was really easy to distinguish the text (foreground) from the background.
With this method, it doesn't matter which color is the background and which color is the foreground text, it also doesn't matter if you have minor noise cause it would cope up with it.
This technique solved it entirely for me, I hope it does the same for you or at least give you some leads.
A chem student asked me for help with plotting image segmenetation:
A stationary camera takes a picture of the experimental setup every second over a period of a few minutes, so like 300 images yield.
The relevant parts in the setup are two adjacent layers of differently-colored foams observed from the side, a 2-color sandwich shrinking from both sides, basically, except one of the foams evaporates a bit faster.
I'd like to segment each of the images in the way that would let me plot both foam regions' "width" against time.
Here is a "diagram" :)
I want to go from here --> To here
Ideally, given a few hundred of such shots, in which only the widths change, I get an array of scalars back that I can plot. (Going to look like a harmonic series on either side of the x-axis)
I have a bit of python and matlab experience, but have never used OpenCV or Image Processing toolbox in matlab, or actually never dealt with any computer vision in general. Could you guys throw like a roadmap of what packages/functions to use or steps one should take and i'll take it from there?
I'm not sure how to address these things:
-selecting at which slice along the length of the slice the algorithm measures the width(i.e. if the foams are a bit uneven), although this can be ignored.
-which library to use to segment regions of the image based on their color, (some k-means shenanigans probably), and selectively store the spatial parameters of the resulting segments?
-how to iterate that above over a number of files.
Thank you kindly in advance!
Assume your Intensity will be different after converting into gray scale ( if not, just convert to other color space like HSV or LAB, then just use one of the components)
img = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
First, Threshold your grayscaled input into a few bands
ret,thresh1 = cv.threshold(img,128,255,cv.THRESH_BINARY)
ret,thresh2 = cv.threshold(img,27,255,cv.THRESH_BINARY_INV)
ret,thresh3 = cv.threshold(img,77,255,cv.THRESH_TRUNC)
ret,thresh4 = cv.threshold(img,97,255,cv.THRESH_TOZERO)
ret,thresh5 = cv.threshold(img,227,255,cv.THRESH_TOZERO_INV)
The value should be tested out by your actual data. Here Im just give a example
Clean up the segmented image using median filter with a radius larger than 9. I do expect some noise. You can also use ROI here to help remove part of noise. But personally I`m lazy, I just wrote program to handle all cases and angle
threshholed_images_aftersmoothing = cv2.medianBlur(threshholed_images,9)
Each band will be corresponding to one color (layer). Now you should have N segmented image from one source. where N is the number of layers you wish to track
Second use opencv function bounding rect to find location and width/height of each Layer AKA each threshholed_images_aftersmoothing. Eg. boundingrect on each sub-segmented images.
C++: Rect boundingRect(InputArray points)
Python: cv2.boundingRect(points) → retval¶
Last, the rect have x,y, height and width property. You can use a simple sorting order to sort from top to bottom layer based on rect attribute x. Run though all vieo to obtain the x(layer id) , height vs time graph.
Rect API
Public Attributes
_Tp **height** // this is what you are looking for
_Tp width
_Tp **x** // this tells you the position of the band
_Tp y
By plot the corresponding heights (|AB| or |CD|) over time, you can obtain the graph you needed.
The more correct way is to use Kalman filter to track the position and height graph as I would expect some sort of bubble will occur and will interfere with the height of the layers.
To be honest, i didnt expect a chem student to be good at this. Haha good luck
Anything wrong you can find me here or Email me if i`m not watching stackoverflow
You can select a region of interest straight down the middle of the foams, a few pixels wide. If you stack these regions for each image it will show the shrink over time.
If for example you use 3 pixel width for the roi, the result of 300 images will be a 900 pixel wide image, where the left is the start of the experiment and the right is the end. The following image can help you understand:
Though I have not fully tested it, this code should work. Note that there must only be images in the folder you reference.
import cv2
import numpy as np
import os
# path to folder that holds the images
path = '.'
# dimensions of roi
x = 0
y = 0
w = 3
h = 100
# store references to all images
all_images = os.listdir(path)
# sort images
all_images.sort()
# create empty result array
result = np.empty([h,0,3],dtype=np.uint8)
for image in all_images:
# load image
img = cv2.imread(path+'/'+image)
# get the region of interest
roi = img[y:y+h,x:x+w]
# add the roi to previous results
result = np.hstack((result,roi))
# optinal: save result as image
# cv2.imwrite('result.png',result)
# display result - can also plot with matplotlib
cv2.imshow('Result', result)
cv2.waitKey(0)
cv2.destroyAllWindows()
Update after question edit:
If the foams have different colors, your can use easily separate them by color by converting the image you hsv and using inrange (example). This creates a mask (=2D array with values from 0-255, one for each pixel) that you can use to calculate average height and extract the parameters and area of the image.
You can find a script to help you find the HSV colors for separation on this GitHub
I am trying to count cells in a microscopy image, and i need to differentiate between the membrane signal and organelles within.
There is only one color, as we are visualizing a protein within the cells using GFP
Right now i am using skimage package (measure, labels). This method kinda works, as it can find connected black regions, and by using the convex of these in together with the bounding box, i can achieve the following (inside: red, membrane: blue):
I am however having problems with organelles (bright spots inside) that touch the membrane and hence I lose signal from the inside (which then is added to the membrane signal - which is a problem).
Any suggestions for a better method?
from skimage import measure
from skimage.segmentation import clear_border
image= ndimage.gaussian_filter(raw_image, sigma=(0.5,0.5), order=0)
median = np.median(image)
mask_inv =np.ma.masked_where(image>median*1.5,image) # was 5
array = np.zeros(image.shape)
img_contour_inv =np.array(array+mask_inv,dtype=np.float)
mask_inverse_bool = img_contour_inv>0
labels = measure.label(mask_inverse_bool,connectivity=1)
df=measure.regionprops(lables, intensity_image=intensity_image)
Followed by some plotting sorting by size and plotting yields image 2
You can try this method:
Find the black spots as you have done in Inside image. Make Inside
image a black and white image.
Now inverse the Inside image and multiply it with the raw image. Bright spots will be
left behind.
To get cell contours, you can subtract the bright spots from the raw image.
Another method could be using thresholding to find bright spots.
I am analyzing an image for finding brown objects in an image. I am thresholding an image and taking darkest parts as brown cells. However depending on the quality of an image objects cannot be identified sometimes. Is there any solution for that in OpenCV Python, such as pre-processing the gray scale image and defining what brown means for that particular image?
The code that I am using to find brown dots is as follows:
def countBrownDots(imageFile):
im = cv2.imread(imageFile)
#changing color space
gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
gray = increaseBrighntness(gray)
l1,thresh = cv2.threshold(gray,10,255,cv2.THRESH_BINARY_INV)
thresh = ndimage.gaussian_filter(thresh, 16)
l2,thresh = cv2.threshold(thresh,70,255,cv2.THRESH_BINARY)
thresh = ndimage.gaussian_filter(thresh, 16)
cv2.imshow("thresh22",thresh)
rmax = pymorph.regmax(thresh)
nim = pymorph.overlay(thresh, rmax)
seeds,nr_nuclei = ndimage.label(rmax)
cv2.imshow("original",im)
cv2.imshow("browns",nim)
Here is an input image example:
Have a look at the image in HSV color space, here are the 3 planes stacked side by side
Although people have suggested segmenting on the basis of hue, there is actually more discriminative information in the saturation and value planes. For this particular image you would probably get a better result with the gray scale (i.e. value plane) than with the hue. However that is no reason to discard the color information.
As proof of concept (using Gimp) for color segmentation, I just randomly picked a brown spot and changed all colors with a color distance of less than 60 from that spot to green to get this:
If you play with the parameters a bit you will probably get what you want. Then write the code.
I tried pre-processing mean shift filtering to posterize the image, but that didn't really help.