Sample program that upscales 2x2 matrix to 5x5 using bilinear interpolation.
Result that OpenCV produces has artifacts at the borders for such simple case.
gy, gx = np.mgrid[0:2, 0:2]
gx = np.float32(gx)
print(gx)
res = cv2.resize(gx,(5,5), fx=0, fy=0, interpolation=cv2.INTER_LINEAR)
print(res)
Output:
[[ 0. 1.]
[ 0. 1.]]
[[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]]
Expected output:
[[0 0.25 0.5 0.75 1
0 0.25 0.5 0.75 1
0 0.25 0.5 0.75 1
0 0.25 0.5 0.75 1
0 0.25 0.5 0.75 1]]
What is the problem?
TL;DR
I tested with other image processing libraries (scikit-image, Pillow and Matlab) and none of them return the expected result.
Odds are this behavior is due to the method to perform the bi-linear interpolation to get efficient results or somehow a convention rather than a bug in my opinion.
I have posted a sample code to perform image resizing with a bi-linear interpolation (check if everything is ok of course, I am not sure how to properly handle the image indexes...) that outputs the expected result.
Partial answer to the question.
What is the output of some other image processing libraries?
scikit-image
The Python module scikit-image contains lot of image processing algorithms. Here the outputs of the skimage.transform.resize method (skimage.__version__: 0.12.3):
mode='constant' (default)
Code:
import numpy as np
from skimage.transform import resize
image = np.array( [
[0., 1.],
[0., 1.]
] )
print 'image:\n', image
image_resized = resize(image, (5,5), order=1, mode='constant')
print 'image_resized:\n', image_resized
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0. 0.07 0.35 0.63 0.49]
[ 0. 0.1 0.5 0.9 0.7 ]
[ 0. 0.1 0.5 0.9 0.7 ]
[ 0. 0.1 0.5 0.9 0.7 ]
[ 0. 0.07 0.35 0.63 0.49]]
mode='edge'
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]]
mode='symmetric'
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]]
mode='reflect'
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]]
mode='wrap'
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]]
As you can see, the default resize mode (constant) produces a different output but the edge mode returns the same result than OpenCV. None of the resize mode produces the expected result thought.
More information about Interpolation: Edge Modes.
This picture sums up all the results in our case:
Pillow
Pillow
is the friendly PIL fork by Alex Clark and Contributors. PIL is the
Python Imaging Library by Fredrik Lundh and Contributors.
What about PIL.Image.Image.resize (PIL.__version__: 4.0.0)?
Code:
import numpy as np
from PIL import Image
image = np.array( [
[0., 1.],
[0., 1.]
] )
print 'image:\n', image
image_pil = Image.fromarray(image)
image_resized_pil = image_pil.resize((5,5), resample=Image.BILINEAR)
print 'image_resized_pil:\n', np.asarray(image_resized_pil, dtype=np.float)
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized_pil:
[[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]]
Pillow image resizing matches the output of the OpenCV library.
Matlab
Matlab proposes a toolbox named Image Processing Toolbox. The function imresize in this toolbox allows to resize image.
Code:
image = zeros(2,1,'double');
image(1,2) = 1;
image(2,2) = 1;
image
image_resize = imresize(image, [5 5], 'bilinear')
Result:
image =
0 1
0 1
image_resize =
0 0.1000 0.5000 0.9000 1.0000
0 0.1000 0.5000 0.9000 1.0000
0 0.1000 0.5000 0.9000 1.0000
0 0.1000 0.5000 0.9000 1.0000
0 0.1000 0.5000 0.9000 1.0000
Again, it is not the expected output with Matlab but the same result with the two previous examples.
Custom bi-linear image resize method
Basic principle
See this Wikipedia article on Bilinear interpolation for more complete information.
This figure should basically illustrates what happens when up-scaling from a 2x2 image to a 4x4 image:
With a nearest neighbor interpolation, the destination pixel at (0,0) will get the value of the source pixel at (0,0) as well as the pixels at (0,1), (1,0) and (1,1).
With a bi-linear interpolation, the destination pixel at (0,0) will get a value which is a linear combination of the 4 neighbors in the source image:
The four red dots show the data points and the green dot is the point
at which we want to interpolate.
R1 is calculated as: R1 = ((x2 – x)/(x2 – x1))*Q11 + ((x – x1)/(x2 – x1))*Q21.
R2 is calculated as: R2 = ((x2 – x)/(x2 – x1))*Q12 + ((x – x1)/(x2 – x1))*Q22.
Finally, P is calculated as a weighted average of R1 and R2: P = ((y2 – y)/(y2 – y1))*R1 + ((y – y1)/(y2 – y1))*R2.
Using coordinates normalized between [0, 1] simplifies the formula.
C++ implementation
This blog post (Resizing Images With Bicubic Interpolation) contains C++ code to perform image resizing with a bi-linear interpolation.
This is my own adaptation (some modifications about the indexes compared to the original code, not sure if it is correct) of the code to work with cv::Mat:
#include <iostream>
#include <opencv2/core.hpp>
float lerp(const float A, const float B, const float t) {
return A * (1.0f - t) + B * t;
}
template <typename Type>
Type resizeBilinear(const cv::Mat &src, const float u, const float v, const float xFrac, const float yFrac) {
int u0 = (int) u;
int v0 = (int) v;
int u1 = (std::min)(src.cols-1, (int) u+1);
int v1 = v0;
int u2 = u0;
int v2 = (std::min)(src.rows-1, (int) v+1);
int u3 = (std::min)(src.cols-1, (int) u+1);
int v3 = (std::min)(src.rows-1, (int) v+1);
float col0 = lerp(src.at<Type>(v0, u0), src.at<Type>(v1, u1), xFrac);
float col1 = lerp(src.at<Type>(v2, u2), src.at<Type>(v3, u3), xFrac);
float value = lerp(col0, col1, yFrac);
return cv::saturate_cast<Type>(value);
}
template <typename Type>
void resize(const cv::Mat &src, cv::Mat &dst) {
float scaleY = (src.rows - 1) / (float) (dst.rows - 1);
float scaleX = (src.cols - 1) / (float) (dst.cols - 1);
for (int i = 0; i < dst.rows; i++) {
float v = i * scaleY;
float yFrac = v - (int) v;
for (int j = 0; j < dst.cols; j++) {
float u = j * scaleX;
float xFrac = u - (int) u;
dst.at<Type>(i, j) = resizeBilinear<Type>(src, u, v, xFrac, yFrac);
}
}
}
void resize(const cv::Mat &src, cv::Mat &dst, const int width, const int height) {
if (width < 2 || height < 2 || src.cols < 2 || src.rows < 2) {
std::cerr << "Too small!" << std::endl;
return;
}
dst = cv::Mat::zeros(height, width, src.type());
switch (src.type()) {
case CV_8U:
resize<uchar>(src, dst);
break;
case CV_64F:
resize<double>(src, dst);
break;
default:
std::cerr << "Src type is not supported!" << std::endl;
break;
}
}
int main() {
cv::Mat img = (cv::Mat_<double>(2,2) << 0, 1, 0, 1);
std::cout << "img:\n" << img << std::endl;
cv::Mat img_resize;
resize(img, img_resize, 5, 5);
std::cout << "img_resize=\n" << img_resize << std::endl;
return EXIT_SUCCESS;
}
It produces:
img:
[0, 1;
0, 1]
img_resize=
[0, 0.25, 0.5, 0.75, 1;
0, 0.25, 0.5, 0.75, 1;
0, 0.25, 0.5, 0.75, 1;
0, 0.25, 0.5, 0.75, 1;
0, 0.25, 0.5, 0.75, 1]
Conclusion
In my opinion, it is unlikely that the OpenCV resize() function is wrong as none of the others image processing libraries I can test produce the expected output and moreover can produce the same OpenCV output with the good parameter.
I tested against two Python modules (scikit-image and Pillow) as they are easy to use and oriented about image processing. I was also able to test with Matlab and its image processing toolbox.
A rough custom implementation of the bi-linear interpolation for image resizing produces the expected result. Two possibilities for me could explain this behavior:
the difference is inherent to the method these image processing libraries use rather than a bug (maybe they use a method to resize images efficiently with some loss compared to a strict bi-linear implementation?)?
it is a somehow a convention to interpolate properly excluding the border?
These libraries are open-source and one can explore into their source code to understand where the discrepancy comes from.
The linked answer shows that the interpolation works only between the two original blue dots but I cannot explain why this behavior.
Why this answer?
This answer, even if it partially answers the OP question, is a good way for me to summarize the few things I found about this topic. I believe also it could help in some way other people who may found this.
As I will explain below, the output:
[[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]]
would be the correct solution. So while opencv has small rounding errors, it is mostly correct.
The reason: your input image doesn't assume an image with the values "0" and "1" in the corners of the image, but in the center of the pixels.
So this the wrong model of what your 2x2 image looks like:
Instead, your image looks like this, with the "colors" defined in the red points. Everything to the left of the center of the left two pixels is just white, and everything to the right of the center of the right two pixels is just black, and values between the pixel centers are interpolated:
Converting the image to 5x5 pixels:
and looking at the centers of the pixels, you see how you would get "0.1" and "0.9" instead of "0.25" and "0.75"
Related
Can anyone explain why this code gives different results in Python vs Matlab?
Matlab
z = [0.97, 0.85, 0];
p = [0.97, 0.94, 0];
k = 0.2;
disp(zp2sos(z,p,k))
Output:
0.2000 -0.1700 0 1.0000 -0.9400 0
1.0000 -0.9700 0 1.0000 -0.9700 0
Python
from scipy.signal import zpk2sos
z = [0.97, 0.85, 0]
p = [0.97, 0.94, 0]
k = 0.2
print(zpk2sos(z,p,k))
Output:
[[ 0.2 0. 0. 1. 0. 0. ]
[ 1. -1.82 0.8245 1. -1.91 0.9118]]
I solved this by using this code I found on github which defines a version of zpk2sos which gave me the same output as the MATLAB zp2sos. You might need to remove some old outdated packages from the code, but otherwise it worked fine for me.
https://gist.github.com/endolith/4525003/8ac88d203f874fd2b97498d50da818bbf5fac0f8
I'm working on a project where I need to calibrate to cameras. As you know one needs to define a plane grid points in the 3D-world and find their correspondences on the image plane. Therefore, the first camera has the following 3D_grid points:
mport cv2 as cv
import numpy as np
WPoints_cam1 = np.zeros((9*3,3), np.float64)
WPoints_cam1[:,:2] = np.mgrid[0:9,0:3].T.reshape(-1,2)*0.4
print(WPoints_cam1)
[[0. 0. 0. ]# world coordinate center
[0.4 0. 0. ]
[0.8 0. 0. ]
[1.2 0. 0. ]
[1.6 0. 0. ]
[2. 0. 0. ]
[2.4 0. 0. ]
[2.8 0. 0. ]
[3.2 0. 0. ]
[0. 0.4 0. ]
[0.4 0.4 0. ]
[0.8 0.4 0. ]
[1.2 0.4 0. ]
[1.6 0.4 0. ]
[2. 0.4 0. ]
[2.4 0.4 0. ]
[2.8 0.4 0. ]
[3.2 0.4 0. ]
[0. 0.8 0. ]
[0.4 0.8 0. ]
[0.8 0.8 0. ]
[1.2 0.8 0. ]
[1.6 0.8 0. ]
[2. 0.8 0. ]
[2.4 0.8 0. ]
[2.8 0.8 0. ]
[3.2 0.8 0. ]]
As seen above the first grid (for the first camera) starts from the defined reference 3D_point (0,0,0) and ends by the point (3.2,0.8 0) with a constant offset of 0.4 and 9x3 dimension
Note that all Z coordinates were put to Z=0 (Zhengyou Zhang calibration)
Now my question is, as I need to define a second grid(for the second camera) that also refers to the defined 3D_coordinate center (0,0,0), I need to define a grid that starts from (3.6,0,0) and ends with (6.8,0.8,0) with the same offset 0.4 and has a dimension 9x3
I believe this is easy to do. However I can't think out of the box due to my beginner level of experience.
Would appreciate for some help and thanks in advance.
You can scale each column like this:
np.mgrid[0:8, 0:3].T.reshape(-1,2) * np.array([(7.8 - 3.6) / 7, 0.4]) + np.array([3.6, 0])
or combine it into scaling matrix like this (and then add on a vector for the translation)
np.mgrid[0:8, 0:3].T.reshape(-1,2) # np.array([[(7.8 - 3.6) / 7, 0], [0, 0.4]]).T + np.array([3.6, 0])
regarding where (7.8 - 3.6) / 7 comes from, the numerator should be self evident. The denominator is the same but for your original dimensions. With 0:8 the max is 7 and the min is 0 so the denominator becomes 7 - 0.
I have a matrix :
matrix = np.array([[[0,0.5,0.6],[0.9,1.2,0]],[[0,0.5,0.6],[0.9,1.2,0]]])
I want to replace all the values 0.55 < x < 0.95 by 0.55.
PS : My question is similar to this question. But the answer does not work in my case.
You can use np.where:
matrix = np.array([[[0,0.5,0.6],[0.9,1.2,0]],[[0,0.5,0.6],[0.9,1.2,0]]])
matrix[np.where((matrix > 0.55) & (matrix < 0.95))] = 0.55
# Or
# matrix[(matrix > 0.55) & (matrix < 0.95)] = 0.55
Output:
>>> matrix
array([[[0. , 0.5 , 0.55],
[0.55, 1.2 , 0. ]],
[[0. , 0.5 , 0.55],
[0.55, 1.2 , 0. ]]])
I do have a pandas DataFrame (size = 34,19) which I want to use as a lookup table.
But the values I want to look up are "between" the values in the dataframe
For example:
0.1 0.2 0.3 0.4 0.5
0.1 4.01 31.86 68.01 103.93 139.2
0.2 24.07 57.49 91.37 125.21 158.57
0.3 44.35 76.4 108.97 141.57 173.78
0.4 59.66 91.02 122.8 154.62 186.13
0.5 87.15 117.9 148.86 179.83 210.48
0.6 106.92 137.41 168.26 198.99 229.06
0.7 121.73 152.48 183.4 213.88 243.33
I know want to look up the value for x = 5.5 y = 1.004, so the answer should be around 114.
I tried it with different methods from scipy but the values I get are always way off.
Last method I used was :inter = interpolate.interpn([np.array(np.arange(34)), np.array(np.arange(19))], np_matrix, [x_value, y_value],)
I even get wrong values for points in the grid which do exist.
Can someone tell me what I'm doing wrong or recommend an easy solution for the task?
EDIT:
An additional problem is:
My raw data, from an .xlsx file, look like:
0.1 0.2 0.3 0.4 0.5
0.1 4.01 31.86 68.01 103.93 139.2
0.2 24.07 57.49 91.37 125.21 158.57
0.3 44.35 76.4 108.97 141.57 173.78
0.4 59.66 91.02 122.8 154.62 186.13
0.5 87.15 117.9 148.86 179.83 210.48
0.6 106.92 137.41 168.26 198.99 229.06
0.7 121.73 152.48 183.4 213.88 243.33
But pandas adds an Index column:
0.1 0.2 0.3 0.4 0.5
0 0.1 4.01 31.86 68.01 103.93 139.2
1 0.2 24.07 57.49 91.37 125.21 158.57
2 0.3 44.35 76.4 108.97 141.57 173.78
3 0.4 59.66 91.02 122.8 154.62 186.13
4 0.8 87.15 117.9 148.86 179.83 210.48
5 1.0 106.92 137.41 168.26 198.99 229.06
6 1.7 121.73 152.48 183.4 213.88 243.33
So if I want to access x = 0.4 y = 0.15 I have to input x = 3, y = 0.15.
Data are read with:
model_references = pd.ExcelFile(model_references_path)
Matrix = model_references.parse('Model_References')
n = Matrix.stack().reset_index().values
out = interpolate.griddata(n[:,0:2], n[:,2], (Stroke, Current), method='cubic')
You can reshape data to 3 columns with stack - first column for index, second for columns and last for values, last get values by scipy.interpolate.griddata
from scipy.interpolate import griddata
a = 5.5
b = 1.004
n = df.stack().reset_index().values
#https://stackoverflow.com/a/8662243
out = griddata(n[:,0:2], n[:,2], [(a, b)], method='linear')
print (out)
[104.563]
Detail:
n = df.stack().reset_index().values
print (n)
[[ 1. 1. 4.01]
[ 1. 2. 31.86]
[ 1. 3. 68.01]
[ 1. 4. 103.93]
[ 1. 5. 139.2 ]
[ 2. 1. 24.07]
[ 2. 2. 57.49]
[ 2. 3. 91.37]
[ 2. 4. 125.21]
[ 2. 5. 158.57]
[ 3. 1. 44.35]
[ 3. 2. 76.4 ]
[ 3. 3. 108.97]
[ 3. 4. 141.57]
[ 3. 5. 173.78]
[ 4. 1. 59.66]
[ 4. 2. 91.02]
[ 4. 3. 122.8 ]
[ 4. 4. 154.62]
[ 4. 5. 186.13]
[ 5. 1. 87.15]
[ 5. 2. 117.9 ]
[ 5. 3. 148.86]
[ 5. 4. 179.83]
[ 5. 5. 210.48]
[ 5. 1. 106.92]
[ 5. 2. 137.41]
[ 5. 3. 168.26]
[ 5. 4. 198.99]
[ 5. 5. 229.06]
[ 6. 1. 121.73]
[ 6. 2. 152.48]
[ 6. 3. 183.4 ]
[ 6. 4. 213.88]
[ 6. 5. 243.33]]
Try interp2d from scipy.
import numpy as np
from scipy.interpolate import interp2d
x = [1, 2, 3, 4, 5, 6, 7]
y = [1, 2, 3, 4, 5]
z = [[4.01, 31.86, 68.01, 103.93, 139.2],
[24.07, 57.49, 91.37, 125.21, 158.57],
[44.35, 76.4, 108.97, 141.57, 173.78],
[59.66, 91.02, 122.8, 154.62, 186.13],
[87.15, 117.9, 148.86, 179.83, 210.48],
[106.92, 137.41, 168.26, 198.99, 229.06],
[121.73, 152.48, 183.4, 213.88, 243.33]]
z = np.array(z).T
f = interp2d(x, y, z)
f(x = 5.5, y = 1.004) # returns 97.15748
Try to change method's kind argument in order to experiment with return value.
Here's a simple code in python.
end = np.zeros((11,2))
alpha=0
while(alpha<=1):
end[int(10*alpha)] = alpha
print(end[int(10*alpha)])
alpha+=0.1
print('')
print(end)
and output:
[ 0. 0.]
[ 0.1 0.1]
[ 0.2 0.2]
[ 0.3 0.3]
[ 0.4 0.4]
[ 0.5 0.5]
[ 0.6 0.6]
[ 0.7 0.7]
[ 0.8 0.8]
[ 0.9 0.9]
[ 1. 1.]
[[ 0. 0. ]
[ 0.1 0.1]
[ 0.2 0.2]
[ 0.3 0.3]
[ 0.4 0.4]
[ 0.5 0.5]
[ 0.6 0.6]
[ 0.8 0.8]
[ 0. 0. ]
[ 1. 1. ]
[ 0. 0. ]]
It is easy to notice that 0.7 is missing and after 0.8 goes 0 instead of 0.9 etc... Why are these outputs differ?
It's because of floating point errors. Run this:
import numpy as np
end = np.zeros((11, 2))
alpha=0
while(alpha<=1):
print("alpha is ", alpha)
end[int(10*alpha)] = alpha
print(end[int(10*alpha)])
alpha+=0.1
print('')
print(end)
and you will see that alpha is, successively:
alpha is 0
alpha is 0.1
alpha is 0.2
alpha is 0.30000000000000004
alpha is 0.4
alpha is 0.5
alpha is 0.6
alpha is 0.7
alpha is 0.7999999999999999
alpha is 0.8999999999999999
alpha is 0.9999999999999999
Basically floating point numbers like 0.1 are stored inexactly on your computer. If you add 0.1 together say 8 times, you won't necessarily get 0.8 -- the small errors can accumulate and give you a different number, in this case 0.7999999999999999. Numpy arrays must take integers as indexes however, so it uses the int function to force this to round down to the nearest integer -- 7 -- which causes that row to be overwritten.
To solve this, you must rewrite your code so that you only ever use integers to index into an array. One slightly crude way would be to round the float to the nearest integer using the round function. But really you should rewrite your code so that it iterates over integers and coverts them into floats, rather than iterating over floats and converting them into integers.
You can read more about floating point numbers here:
https://docs.python.org/3/tutorial/floatingpoint.html
As #Denziloe pointed, this is due to floating point errors.
If you look at the definition of int():
If x is floating point, the conversion truncates towards zero
To solve your problem use round() instead of int()