Develop Reference

Why is my 2D FFT convolution displacing the result? [duplicate]

I'm using zero padding around my image and convolution kernel, converting them to the Fourier domain, and inverting them back to get the convolved image, see code below. The result, however, is wrong. I was expecting a blurred image, but the output is four shifted quarters. Why is the output wrong, and how can I fix the code?
Input image:
Result of convolution:
from PIL import Image,ImageDraw,ImageOps,ImageFilter
import numpy as np
from scipy import fftpack
from copy import deepcopy
import imageio
## STEP 1 ##
im1=Image.open("pika.jpeg")
im1=ImageOps.grayscale(im1)
im1.show()
print("s",im1.size)
## working on this image array
im_W=np.array(im1).T
print("before",im_W.shape)
if(im_W.shape[0]%2==0):
im_W=np.pad(im_W, ((1,0),(0,0)), 'constant')
if(im_W.shape[1]%2==0):
im_W=np.pad(im_W, ((0,0),(1,0)), 'constant')
print("after",im_W.shape)
Boxblur=np.array([[1/9,1/9,1/9],[1/9,1/9,1/9],[1/9,1/9,1/9]])
dim=Boxblur.shape[0]
##padding before frequency domain multipication
pad_size=(Boxblur.shape[0]-1)/2
pad_size=int(pad_size)
##padded the image(starts here)
p_im=np.pad(im_W, ((pad_size,pad_size),(pad_size,pad_size)), 'constant')
t_b=(p_im.shape[0]-dim)/2
l_r=(p_im.shape[1]-dim)/2
t_b=int(t_b)
l_r=int(l_r)
##padded the image(ends here)
## padded the kernel(starts here)
k_im=np.pad(Boxblur, ((t_b,t_b),(l_r,l_r)), 'constant')
print("hjhj",k_im)
print("kernel",k_im.shape)
##fourier transforms image and kernel
fft_im = fftpack.fftshift(fftpack.fft2(p_im))
fft_k = fftpack.fftshift(fftpack.fft2(k_im))
con_in_f=fft_im*fft_k
ifft2 = abs(fftpack.ifft2(fftpack.ifftshift(con_in_f)))
convolved=(np.log(abs(ifft2))* 255 / np.amax(np.log(abs(ifft2)))).astype(np.uint8)
final=Image.fromarray(convolved.T)
final.show()
u=im1.filter(ImageFilter.Kernel((3,3), [1/9,1/9,1/9,1/9,1/9,1/9,1/9,1/9,1/9], scale=None, offset=0))
u.show()

The Discrete Fourier transform (DFT) and, by extension, the FFT (which computes the DFT) have the origin in the first element (for an image, the top-left pixel) for both the input and the output. This is the reason we often use the fftshift function on the output, so as to shift the origin to a location more familiar to us (the middle of the image).
This means that we need to transform a 3x3 uniform weighted blurring kernel to look like this before passing it to the FFT function:
1/9 1/9 0 0 ... 0 1/9
1/9 1/9 0 0 ... 0 1/9
0 0 0 0 ... 0 0
... ... ...
0 0 0 0 ... 0 0
1/9 1/9 0 0 ... 0 1/9
That is, the middle of the kernel is at the top-left corner of the image, with the pixels above and to the left of the middle wrapping around and appearing at the right and bottom ends of the image.
We can do this using the ifftshift function, applied to the kernel after padding. When padding the kernel, we need to take care that the origin (middle of the kernel) is at location k_im.shape // 2 (integer division), within the kernel image k_im. Initially the origin is at [3,3]//2 == [1,1]. Usually, the image whose size we're matching is even in size, for example [256,256]. The origin there will be at [256,256]//2 == [128,128]. This means that we need to pad a different amount to the left and to the right (and bottom and top). We need to be careful computing this padding:
sz = img.shape # the sizes we're matching
kernel = np.ones((3,3)) / 9
sz = (sz[0] - kernel.shape[0], sz[1] - kernel.shape[1]) # total amount of padding
kernel = np.pad(kernel, (((sz[0]+1)//2, sz[0]//2), ((sz[1]+1)//2, sz[1]//2)), 'constant')
kernel = fftpack.ifftshift(kernel)
Note that the input image, img, does not need to be padded (though you can do this if you want to enforce a size for which the FFT is cheaper). There is also no need to apply fftshift to the result of the FFT before multiplication, and then reverse this shift right after, these shifts are redundant. You should use fftshift only if you want to display the Fourier domain image. Finally, applying logarithmic scaling to the filtered image is wrong.
The resulting code is (I'm using pyplot for display, not using PIL at all):
import numpy as np
from scipy import misc
from scipy import fftpack
import matplotlib.pyplot as plt
img = misc.face()[:,:,0]
kernel = np.ones((3,3)) / 9
sz = (img.shape[0] - kernel.shape[0], img.shape[1] - kernel.shape[1]) # total amount of padding
kernel = np.pad(kernel, (((sz[0]+1)//2, sz[0]//2), ((sz[1]+1)//2, sz[1]//2)), 'constant')
kernel = fftpack.ifftshift(kernel)
filtered = np.real(fftpack.ifft2(fftpack.fft2(img) * fftpack.fft2(kernel)))
plt.imshow(filtered, vmin=0, vmax=255)
plt.show()
Note that I am taking the real part of the inverse FFT. The imaginary part should contain only values very close to zero, which are the result of rounding errors in the computations. Taking the absolute value, though common, is incorrect. For example, you might want to apply a filter to an image that contains negative values, or apply a filter that produces negative values. Taking the absolute value here would create artefacts. If the output of the inverse FFT contains imaginary values significantly different from zero, then there is an error in the way that the filtering kernel was padded.
Also note that the kernel here is tiny, and consequently the blurring effect is tiny too. To better see the effect of the blurring, make a larger kernel, for example np.ones((7,7)) / 49.

Not able to reproduce the statement that spectral power of natural images falls with frequency as 1/f**2

As the title says, I am trying to reproduce the following statement from (1)
Empirically, many authors have found that the spectral power of
natural images falls with frequency, f, according to a power law,
1/f**p, with estimated values for p typically near 2.
For this purpose, I used the code from (2) with some minor modifications
import matplotlib.image as mpimg
import numpy as np
import scipy.stats as stats
import pylab as pl
from scipy.stats import linregress
from skimage import color
# Load image in greyscale
image = color.rgb2gray(mpimg.imread("clouds.png"))
image_X,image_Y = image.shape[0], image.shape[1]
# Do FFT
fourier_image = np.fft.fftn(image)
fourier_amplitudes = np.abs(fourier_image)**2
# Get wave vector
kfreq_X = np.fft.fftfreq(image_X) * image_X
kfreq_Y = np.fft.fftfreq(image_Y) * image_Y
kfreq2D = np.meshgrid(kfreq_X, kfreq_Y)
knrm = np.sqrt(kfreq2D[0]**2 + kfreq2D[1]**2)
knrm = knrm.flatten()
fourier_amplitudes = fourier_amplitudes.flatten()
kbins = np.arange(0.5, max(image_X, image_Y)/2 + 1., 1.)
kvals = 0.5 * (kbins[1:] + kbins[:-1])
Abins, _, _ = stats.binned_statistic(knrm, fourier_amplitudes,
statistic = "mean",
bins = kbins)
Abins *= 4. * np.pi / 3. * (kbins[1:]**3 - kbins[:-1]**3)
# print(linregress(np.log(kvals[20:-300]), np.log(Abins[20:-300])))
# pl.plot(kvals, Abins)
pl.loglog(kvals, Abins)
pl.xlabel("$k$")
pl.ylabel("$P(k)$")
pl.tight_layout()
pl.savefig("cloud_power_spectrum.png", dpi = 300, bbox_inches = "tight")
When I look at power spectrum of the image used in tutorial (2) my p estimate is around -1.3.
When I tried with my own RGB image. I would not get nice power spectrum distribution as described and shown in (2). Instead, there are few peaks.
Questions
1. Since I am not able to get even close to p~-2 for any of the images, I was wondering if my code is correct?
2. If the code is okay, is there any other reason why I am not able to get p~-2?
3. Are those peaks in RGB image some artefact of my conversion to grayscale maybe or this is expected behaviour?
(1) https://www.cns.nyu.edu/pub/eero/simoncelli01-reprint.pdf
(2) https://bertvandenbroucke.netlify.app/2019/05/24/computing-a-power-spectrum-in-python/

Bert Vandenbroucke updated the blog post from which OP obtained the code on March 10, 2021 to address a number of issues, among them:
Bin area formula 4. * np.pi / 3. * (kbins[1:]**3 - kbins[:-1]**3) (3D spherical shell bin volume) was corrected to np.pi * (kbins[1:]**2 - kbins[:-1]**2) (2D ring bin area).
There is now a warning that frequencies above 1D Nyquist (0.5 cycle per pixel), that is, frequencies between 1D and 2D Nyquist (0.5*sqrt(2) cycle per pixel), are not correctly sampled. Original code completely broke down with a Nyquist checkerboard (alternating black and white pixels) with witdh=height=(power of 2).

Plotting FFT frequencies in Hz in Python

I am implementing the method from this paper:
https://dspace.mit.edu/bitstream/handle/1721.1/66243/Picard_Noncontact%20Automated.pdf?sequence=1&isAllowed=y
The main idea is cardiac pulse measurement using a set of frames (N=300) from a 10s video so the frame rate equals 30 fps.
red = [item[:,:,0] for item in imgs]
green = [item[:,:,1] for item in imgs]
blue = [item[:,:,2] for item in imgs]
red_avg = [item.mean() for item in red]
green_avg = [item.mean() for item in green]
blue_avg = [item.mean() for item in blue]
red_mean, red_std = np.array(red_avg).mean(), np.array(red_avg).std()
green_mean, green_std = np.array(green_avg).mean(), np.array(green_avg).std()
blue_mean, blue_std = np.array(blue_avg).mean(), np.array(blue_avg).std()
red_avg = [(item - red_mean)/red_std for item in red_avg]
green_avg = [(item - green_mean)/green_std for item in green_avg]
blue_avg = [(item - blue_mean)/blue_std for item in blue_avg]
data = np.vstack([signal.detrend(red_avg), signal.detrend(green_avg), signal.detrend(blue_avg)]).reshape(300,3)
from sklearn.decomposition import FastICA
transformer = FastICA(n_components=3)
X_transformed = transformer.fit_transform(data)
from scipy.fftpack import fft
first = X_transformed.T[0]
second = X_transformed.T[1]
third = X_transformed.T[2]
ff = np.fft.fft(first)
fs = np.fft.fft(second)
ft = np.fft.fft(third)
imgs - is the initial list of arrays with 300 image pixel values.
As you can see, I split all frames into RGB channels and thus have traces x_i(t), where i = 1,2,3
After standardization, we detrend all the traces and stack them to further apply ICA and then FFT all the three components.
The method then claims that we need to plot power vs frequency (Hz) and select the component that is most likely to be heart pulse.
Finally, we applied the fast Fourier transform (FFT) on the selected source signal to
obtain the power spectrum. The pulse frequency was designated as the frequency that
corresponded to the highest power of the spectrum within an operational frequency band. For
our experiments, we set the operational range to [0.75, 4] Hz (corresponding to [45, 240]
bpm) to provide a wide range of heart rate measurements.
Here's how I try to visualize the frequencies:
import numpy as np
import matplotlib.pyplot as plt
from scipy import signal
data = ft
print(fs.size)
ps = np.abs(np.fft.fft(data))**2
sampling_rate = 30
freqs = np.fft.fftfreq(data.size, 1/sampling_rate)
idx = np.argsort(freqs)
#print(idx)
plt.plot(freqs[idx], ps[idx])
What I get is totally different since the range of frequencies is from $-15$ to $15$ and I have no idea whether this is in Hz or not.
The three images above are what I get when I execute the code to visualize frequencies and signal power.
I would appreciate any help or suggestions.

You should really learn how to work with images/videos as nD-tensors. Doing that you can replace all your data wrangling with much more concise code:
import numpy as np
import matplotlib.pyplot as plt
from scipy.fftpack import fft
from sklearn.decomposition import FastICA
images = [np.random.rand(640, 480, 3) for _ in range(30)]
# Create tensor with all images
images = np.array(images)
images.shape
# Take average of all pixels, for each image and each channel individually
avgs = np.mean(images, axis=(1, 2))
mean, std = np.mean(avgs), np.std(avgs)
# Normalize all average channels
avgs = (avgs - mean) / std
# Detrend across images
avgs = scipy.signal.detrend(avgs, axis=0)
transformer = FastICA(n_components=3)
X_transformed = transformer.fit_transform(avgs)
X_ff = np.fft.fft(X_transformed, axis=0)
plt.plot(np.abs(X_ff) ** 2)
To answer your question a little bit: I think you are mistakenly taking the fourier spectrum of the fourier spectrum of the third PCA component.
FFT(FFT(PCA[:, 2]))
while you meant to take one FFT only:
FFT(PCA[:, 2]))
Regarding the -15...15 axis: You have set your sampling frequency as 30Hz (or 30fps in video-terms). That means you can detect anything up to 15Hz in your video.
In fourier theory, there exists a thing as called "negative frequency". Now since we're mostly analyzing real signals (as opposed to complex signals), that negative frequency is always the same as the positive frequency. This means your spectrum is always symmetric and you can disregard the left half.
However since you've taken the FFT twice, you're looking at the FFT of a complex signal, which does have negative frequencies. That's the reason why your spectra are asymmetric and confusing.
Additionally, I believe you're confusing reshaping and transposing. Before PCA, you're assembling your data like
np.vstack([red, green, blue]) # shape = (3, 300)
which you want to transpose to get (300, 3). If you reshape instead, you're not swapping rows and columns but are interpreting the same data in a different shape.

apply gaussian blur to an image ussing python

I am trying to replicate the following smoothing of an image using a Gaussian filter (images from a journal):
In the paper says that in order to get from the left image to the right image I have to apply a gaussian fiter with values x,y = 1,...,100 and sigma = 14 to obtain the "best resuts"
I have developed the following program in python to try to achieve this smoothing:
import scipy.ndimage as ndimage
import matplotlib.pyplot as plt
img = ndimage.imread('left2.png')
img = ndimage.gaussian_filter(img, sigma=(14), order=0)
plt.imshow(img)
plt.show()
for some reason the result obtained is not similar to the picture in the right.
Can someone please point out what do I have to modify in the program to get from the left image to the right image?
Thank you.

I'm going to take a guess here:
Because they mention that their x and y values range from 0-100, they're probably applying a "sigma = 14 unit blur" instead of a "sigma = 14 pixel blur".
The sigma parameter in scipy.ndimage.gaussian_filter is in pixel units. If I'm correct about the author's intent, you'll need to scale the sigma parameter you pass in.
If the authors specified that both x and y ranged from 0-100, the sigma in the x and y directions will be different, as your input data appears have a different number of rows than columns (i.e. it isn't a perfectly square image).
Perhaps try something similar to this?
nrows, ncols = img.shape
sigma = (14 * nrows / 100.0, 14 * ncols / 100.0)
img = ndimage.gaussian_filter(img, sigma=sigma)

Peak detection in a noisy 2d array

I'm trying to get python to return, as close as possible, the center of the most obvious clustering in an image like the one below:
In my previous question I asked how to get the global maximum and the local maximums of a 2d array, and the answers given worked perfectly. The issue is that the center estimation I can get by averaging the global maximum obtained with different bin sizes is always slightly off than the one I would set by eye, because I'm only accounting for the biggest bin instead of a group of biggest bins (like one does by eye).
I tried adapting the answer to this question to my problem, but it turns out my image is too noisy for that algorithm to work. Here's my code implementing that answer:
import numpy as np
from scipy.ndimage.filters import maximum_filter
from scipy.ndimage.morphology import generate_binary_structure, binary_erosion
import matplotlib.pyplot as pp
from os import getcwd
from os.path import join, realpath, dirname
# Save path to dir where this code exists.
mypath = realpath(join(getcwd(), dirname(__file__)))
myfile = 'data_file.dat'
x, y = np.loadtxt(join(mypath,myfile), usecols=(1, 2), unpack=True)
xmin, xmax = min(x), max(x)
ymin, ymax = min(y), max(y)
rang = [[xmin, xmax], [ymin, ymax]]
paws = []
for d_b in range(25, 110, 25):
# Number of bins in x,y given the bin width 'd_b'
binsxy = [int((xmax - xmin) / d_b), int((ymax - ymin) / d_b)]
H, xedges, yedges = np.histogram2d(x, y, range=rang, bins=binsxy)
paws.append(H)
def detect_peaks(image):
"""
Takes an image and detect the peaks usingthe local maximum filter.
Returns a boolean mask of the peaks (i.e. 1 when
the pixel's value is the neighborhood maximum, 0 otherwise)
"""
# define an 8-connected neighborhood
neighborhood = generate_binary_structure(2,2)
#apply the local maximum filter; all pixel of maximal value
#in their neighborhood are set to 1
local_max = maximum_filter(image, footprint=neighborhood)==image
#local_max is a mask that contains the peaks we are
#looking for, but also the background.
#In order to isolate the peaks we must remove the background from the mask.
#we create the mask of the background
background = (image==0)
#a little technicality: we must erode the background in order to
#successfully subtract it form local_max, otherwise a line will
#appear along the background border (artifact of the local maximum filter)
eroded_background = binary_erosion(background, structure=neighborhood, border_value=1)
#we obtain the final mask, containing only peaks,
#by removing the background from the local_max mask
detected_peaks = local_max - eroded_background
return detected_peaks
#applying the detection and plotting results
for i, paw in enumerate(paws):
detected_peaks = detect_peaks(paw)
pp.subplot(4,2,(2*i+1))
pp.imshow(paw)
pp.subplot(4,2,(2*i+2) )
pp.imshow(detected_peaks)
pp.show()
and here's the result of that (varying the bin size):
Clearly my background is too noisy for that algorithm to work, so the question is: how can I make that algorithm less sensitive? If an alternative solution exists then please let me know.
EDIT
Following Bi Rico advise I attempted smoothing my 2d array before passing it on to the local maximum finder, like so:
H, xedges, yedges = np.histogram2d(x, y, range=rang, bins=binsxy)
H1 = gaussian_filter(H, 2, mode='nearest')
paws.append(H1)
These were the results with a sigma of 2, 4 and 8:
EDIT 2
A mode ='constant' seems to work much better than nearest. It converges to the right center with a sigma=2 for the largest bin size:
So, how do I get the coordinates of the maximum that shows in the last image?

Answering the last part of your question, always you have points in an image, you can find their coordinates by searching, in some order, the local maximums of the image. In case your data is not a point source, you can apply a mask to each peak in order to avoid the peak neighborhood from being a maximum while performing a future search. I propose the following code:
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
import copy
def get_std(image):
return np.std(image)
def get_max(image,sigma,alpha=20,size=10):
i_out = []
j_out = []
image_temp = copy.deepcopy(image)
while True:
k = np.argmax(image_temp)
j,i = np.unravel_index(k, image_temp.shape)
if(image_temp[j,i] >= alpha*sigma):
i_out.append(i)
j_out.append(j)
x = np.arange(i-size, i+size)
y = np.arange(j-size, j+size)
xv,yv = np.meshgrid(x,y)
image_temp[yv.clip(0,image_temp.shape[0]-1),
xv.clip(0,image_temp.shape[1]-1) ] = 0
print xv
else:
break
return i_out,j_out
#reading the image
image = mpimg.imread('ggd4.jpg')
#computing the standard deviation of the image
sigma = get_std(image)
#getting the peaks
i,j = get_max(image[:,:,0],sigma, alpha=10, size=10)
#let's see the results
plt.imshow(image, origin='lower')
plt.plot(i,j,'ro', markersize=10, alpha=0.5)
plt.show()
The image ggd4 for the test can be downloaded from:
http://www.ipac.caltech.edu/2mass/gallery/spr99/ggd4.jpg
The first part is to get some information about the noise in the image. I did it by computing the standard deviation of the full image (actually is better to select an small rectangle without signal). This is telling us how much noise is present in the image.
The idea to get the peaks is to ask for successive maximums, which are above of certain threshold (let's say, 3, 4, 5, 10, or 20 times the noise). This is what the function get_max is actually doing. It performs the search of maximums until one of them is below the threshold imposed by the noise. In order to avoid finding the same maximum many times it is necessary to remove the peaks from the image. In the general way, the shape of the mask to do so depends strongly on the problem that one want to solve. for the case of stars, it should be good to remove the star by using a Gaussian function, or something similar. I have chosen for simplicity a square function, and the size of the function (in pixels) is the variable "size".
I think that from this example, anybody can improve the code by adding more general things.
EDIT:
The original image looks like:
While the image after identifying the luminous points looks like this:

Too much of a n00b on Stack Overflow to comment on Alejandro's answer elsewhere here. I would refine his code a bit to use a preallocated numpy array for output:
def get_max(image,sigma,alpha=3,size=10):
from copy import deepcopy
import numpy as np
# preallocate a lot of peak storage
k_arr = np.zeros((10000,2))
image_temp = deepcopy(image)
peak_ct=0
while True:
k = np.argmax(image_temp)
j,i = np.unravel_index(k, image_temp.shape)
if(image_temp[j,i] >= alpha*sigma):
k_arr[peak_ct]=[j,i]
# this is the part that masks already-found peaks.
x = np.arange(i-size, i+size)
y = np.arange(j-size, j+size)
xv,yv = np.meshgrid(x,y)
# the clip here handles edge cases where the peak is near the
# image edge
image_temp[yv.clip(0,image_temp.shape[0]-1),
xv.clip(0,image_temp.shape[1]-1) ] = 0
peak_ct+=1
else:
break
# trim the output for only what we've actually found
return k_arr[:peak_ct]
In profiling this and Alejandro's code using his example image, this code about 33% faster (0.03 sec for Alejandro's code, 0.02 sec for mine.) I expect on images with larger numbers of peaks, it would be even faster - appending the output to a list will get slower and slower for more peaks.

I think the first step needed here is to express the values in H in terms of the standard deviation of the field:
import numpy as np
H = H / np.std(H)
Now you can put a threshold on the values of this H. If the noise is assumed to be Gaussian, picking a threshold of 3 you can be quite sure (99.7%) that this pixel can be associated with a real peak and not noise. See here.
Now the further selection can start. It is not exactly clear to me what exactly you want to find. Do you want the exact location of peak values? Or do you want one location for a cluster of peaks which is in the middle of this cluster?
Anyway, starting from this point with all pixel values expressed in standard deviations of the field, you should be able to get what you want. If you want to find clusters you could perform a nearest neighbour search on the >3-sigma gridpoints and put a threshold on the distance. I.e. only connect them when they are close enough to each other. If several gridpoints are connected you can define this as a group/cluster and calculate some (sigma-weighted?) center of the cluster.
Hope my first contribution on Stackoverflow is useful for you!

The way I would do it:
1) normalize H between 0 and 1.
2) pick a threshold value, as tcaswell suggests. It could be between .9 and .99 for example
3) use masked arrays to keep only the x,y coordinates with H above threshold:
import numpy.ma as ma
x_masked=ma.masked_array(x, mask= H < thresold)
y_masked=ma.masked_array(y, mask= H < thresold)
4) now you can weight-average on the masked coordinates, with weight something like (H-threshold)^2, or any other power greater or equal to one, depending on your taste/tests.
Comment:
1) This is not robust with respect to the type of peaks you have, since you may have to adapt the thresold. This is the minor problem;
2) This DOES NOT work with two peaks as it is, and will give wrong results if the 2nd peak is above threshold.
Nonetheless, it will always give you an answer without crashing (with pros and cons of the thing..)

I'm adding this answer because it's the solution I ended up using. It's a combination of Bi Rico's comment here (May 30 at 18:54) and the answer given in this question: Find peak of 2d histogram.
As it turns out using the peak detection algorithm from this question Peak detection in a 2D array only complicates matters. After applying the Gaussian filter to the image all that needs to be done is to ask for the maximum bin (as Bi Rico pointed out) and then obtain the maximum in coordinates.
So instead of using the detect-peaks function as I did above, I simply add the following code after the Gaussian 2D histogram is obtained:
# Get 2D histogram.
H, xedges, yedges = np.histogram2d(x, y, range=rang, bins=binsxy)
# Get Gaussian filtered 2D histogram.
H1 = gaussian_filter(H, 2, mode='nearest')
# Get center of maximum in bin coordinates.
x_cent_bin, y_cent_bin = np.unravel_index(H1.argmax(), H1.shape)
# Get center in x,y coordinates.
x_cent_coor , y_cent_coord = np.average(xedges[x_cent_bin:x_cent_bin + 2]), np.average(yedges[y_cent_g:y_cent_g + 2])