Related
I want to create the same result in Python and don't use any loop method.
Matlab code of what I want to do:
x = [1 2 3 4];
y = [1 2 3];
z = 2.^(x'-y)
The result of this Matlab code:
z =
1.0000 0.5000 0.2500
2.0000 1.0000 0.5000
4.0000 2.0000 1.0000
8.0000 4.0000 2.0000
Is there any good Python method for doing so?
I'm confused about this for a long time...
You will want to use numpy for this. It will do array based numerical operations similar to what you are used to with Matlab.
Note the use of np.newaxis to increase the dimensions of x so that array broadcasting will work correctly. Broadcasting is similar to implicit expansion in Matlab.
Also, the dot is this 2. does not serve the same purpose as in Matlab (indicating element-wise operation). This is just to force the 2 to be a float. Otherwise, you will get errors about raising integers to negative integers.
import numpy as np
x = np.array([1, 2, 3, 4])
y = np.array([1, 2, 3])
z = 2.**(x[:,np.newaxis]-y)
array([[1. , 0.5 , 0.25],
[2. , 1. , 0.5 ],
[4. , 2. , 1. ],
[8. , 4. , 2. ]])
I will try to explain my problem. So I have two DataFrames , Df1 and Df2.
Each of them has 3 columns and 4 rows.
I will solve a quadratic functions with np.polyfit.
M=3
for t in range(M-1,0,-1):
regs = np.polyfit(Df1[:,t],Df2[:,t+1],2)
C = np.polyval(regs,Df1[:,t])
But I want to use only the values which are smaller than 1.1
Df1[Df1 < 1.1]
Now I have something like that
[1. , 1.09, 1.08, NaN]
[1. , 1., 1.07, 1.04]
[1. , NaN, 1.01, NaN]
[1. , 0.78, NaN,0.95]
And my Df2 looks like
[0.1 , 0., 0.08, 0.]
[0.1 , 0.11, 0., 0.09]
[0.1 , 0.33, 0.22, 0.]
[0.1 , 0.09, 0.108, 0.]
So what I want to do is for each column from Df1, if Df1 has a NaN
Then I don't want to calculate it.
Here is what I tried to explain:
X =[1.08,1.07,1.01]
Y =[0.,0.09,0]
I tried this one
S = [[1.,1.09,1.08,1.34],[1.,1.16,1.26,1.54],[1.,1.22,1.07,1.03],[1.,0.93,0.97,0.92],[1.,1.11,1.56,1.52],
[1.,0.76,0.77,0.9],[1.,0.92,0.84,1.01],[1.,0.88,1.22,1.34]]
K= 1.1
Sn = np.asarray(S)
r = 0.06
T=1
M=3
dt = T/M
h= np.maximum(K-Sn,0)
V = np.copy(h)
disk = np.exp(-r*dt)
for i in range(M-1,0,-1):
reg = np.polyfit(Sn[:,i],V[:,i+1]*disk,2)
C = np.polyval(reg,Sn[:,i])
V[:,i] = np.where(C > h[:,i],V[:,i+1]*disk,h[:,i])
C0 = disk* 1/8 * np.sum(V[:,1])
And my result for C0 is 0.11973..
This is the Longstaff Schwartz Monte Carlo Algorithm for pricing American Options.
But in the paper from Longstaff Schwartz ,they get a little different result
https://people.math.ethz.ch/~hjfurrer/teaching/LongstaffSchwartzAmericanOptionsLeastSquareMonteCarlo.pdf
(Page120)
They get 0.114. But I don't see my mistake
I am trying to convert a multi-index pandas DataFrame into a numpy.ndarray. The DataFrame is below:
s1 s2 s3 s4
Action State
1 s1 0.0 0 0.8 0.2
s2 0.1 0 0.9 0.0
2 s1 0.0 0 0.9 0.1
s2 0.0 0 1.0 0.0
I would like the resulting numpy.ndarray to be the following with np.shape() = (2,2,4):
[[[ 0.0 0.0 0.8 0.2 ]
[ 0.1 0.0 0.9 0.0 ]]
[[ 0.0 0.0 0.9 0.1 ]
[ 0.0 0.0 1.0 0.0]]]
I have tried df.as_matrix() but this returns:
[[ 0. 0. 0.8 0.2]
[ 0.1 0. 0.9 0. ]
[ 0. 0. 0.9 0.1]
[ 0. 0. 1. 0. ]]
How do I return a list of lists for the first level with each list representing an Action records.
You could use the following:
dim = len(df.index.get_level_values(0).unique())
result = df.values.reshape((dim1, dim1, df.shape[1]))
print(result)
[[[ 0. 0. 0.8 0.2]
[ 0.1 0. 0.9 0. ]]
[[ 0. 0. 0.9 0.1]
[ 0. 0. 1. 0. ]]]
The first line just finds the number of groups that you want to groupby.
Why this (or groupby) is needed: as soon as you use .values, you lose the dimensionality of the MultiIndex from pandas. So you need to re-pass that dimensionality to NumPy in some way.
One way
In [151]: df.groupby(level=0).apply(lambda x: x.values.tolist()).values
Out[151]:
array([[[0.0, 0.0, 0.8, 0.2],
[0.1, 0.0, 0.9, 0.0]],
[[0.0, 0.0, 0.9, 0.1],
[0.0, 0.0, 1.0, 0.0]]], dtype=object)
Using Divakar's suggestion, np.reshape() worked:
>>> print(P)
s1 s2 s3 s4
Action State
1 s1 0.0 0 0.8 0.2
s2 0.1 0 0.9 0.0
2 s1 0.0 0 0.9 0.1
s2 0.0 0 1.0 0.0
>>> np.reshape(P,(2,2,-1))
[[[ 0. 0. 0.8 0.2]
[ 0.1 0. 0.9 0. ]]
[[ 0. 0. 0.9 0.1]
[ 0. 0. 1. 0. ]]]
>>> np.shape(P)
(2, 2, 4)
Elaborating on Brad Solomon's answer, to get a sligthly more generic solution - indexes of different sizes and an unfixed number of indexes - one could do something like this:
def df_to_numpy(df):
try:
shape = [len(level) for level in df.index.levels]
except AttributeError:
shape = [len(df.index)]
ncol = df.shape[-1]
if ncol > 1:
shape.append(ncol)
return df.to_numpy().reshape(shape)
If df has missing sub-indexes reshape will not work. One way to add them would be (maybe there are better solutions):
def enforce_df_shape(df):
try:
ind = pd.MultiIndex.from_product([level.values for level in df.index.levels])
except AttributeError:
return df
fulldf = pd.DataFrame(-1, columns=df.columns, index=ind) # remove -1 to fill fulldf with nan
fulldf.update(df)
return fulldf
If you are just trying to pull out one column, say s1, and get an array with shape (2,2) you can use the .index.levshape like this:
x = df.s1.to_numpy().reshape(df.index.levshape)
This will give you a (2,2) containing the value of s1.
Sample program that upscales 2x2 matrix to 5x5 using bilinear interpolation.
Result that OpenCV produces has artifacts at the borders for such simple case.
gy, gx = np.mgrid[0:2, 0:2]
gx = np.float32(gx)
print(gx)
res = cv2.resize(gx,(5,5), fx=0, fy=0, interpolation=cv2.INTER_LINEAR)
print(res)
Output:
[[ 0. 1.]
[ 0. 1.]]
[[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]]
Expected output:
[[0 0.25 0.5 0.75 1
0 0.25 0.5 0.75 1
0 0.25 0.5 0.75 1
0 0.25 0.5 0.75 1
0 0.25 0.5 0.75 1]]
What is the problem?
TL;DR
I tested with other image processing libraries (scikit-image, Pillow and Matlab) and none of them return the expected result.
Odds are this behavior is due to the method to perform the bi-linear interpolation to get efficient results or somehow a convention rather than a bug in my opinion.
I have posted a sample code to perform image resizing with a bi-linear interpolation (check if everything is ok of course, I am not sure how to properly handle the image indexes...) that outputs the expected result.
Partial answer to the question.
What is the output of some other image processing libraries?
scikit-image
The Python module scikit-image contains lot of image processing algorithms. Here the outputs of the skimage.transform.resize method (skimage.__version__: 0.12.3):
mode='constant' (default)
Code:
import numpy as np
from skimage.transform import resize
image = np.array( [
[0., 1.],
[0., 1.]
] )
print 'image:\n', image
image_resized = resize(image, (5,5), order=1, mode='constant')
print 'image_resized:\n', image_resized
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0. 0.07 0.35 0.63 0.49]
[ 0. 0.1 0.5 0.9 0.7 ]
[ 0. 0.1 0.5 0.9 0.7 ]
[ 0. 0.1 0.5 0.9 0.7 ]
[ 0. 0.07 0.35 0.63 0.49]]
mode='edge'
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]]
mode='symmetric'
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]]
mode='reflect'
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]]
mode='wrap'
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized:
[[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]
[ 0.3 0.1 0.5 0.9 0.7]]
As you can see, the default resize mode (constant) produces a different output but the edge mode returns the same result than OpenCV. None of the resize mode produces the expected result thought.
More information about Interpolation: Edge Modes.
This picture sums up all the results in our case:
Pillow
Pillow
is the friendly PIL fork by Alex Clark and Contributors. PIL is the
Python Imaging Library by Fredrik Lundh and Contributors.
What about PIL.Image.Image.resize (PIL.__version__: 4.0.0)?
Code:
import numpy as np
from PIL import Image
image = np.array( [
[0., 1.],
[0., 1.]
] )
print 'image:\n', image
image_pil = Image.fromarray(image)
image_resized_pil = image_pil.resize((5,5), resample=Image.BILINEAR)
print 'image_resized_pil:\n', np.asarray(image_resized_pil, dtype=np.float)
Result:
image:
[[ 0. 1.]
[ 0. 1.]]
image_resized_pil:
[[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]
[ 0. 0.1 0.5 0.89999998 1. ]]
Pillow image resizing matches the output of the OpenCV library.
Matlab
Matlab proposes a toolbox named Image Processing Toolbox. The function imresize in this toolbox allows to resize image.
Code:
image = zeros(2,1,'double');
image(1,2) = 1;
image(2,2) = 1;
image
image_resize = imresize(image, [5 5], 'bilinear')
Result:
image =
0 1
0 1
image_resize =
0 0.1000 0.5000 0.9000 1.0000
0 0.1000 0.5000 0.9000 1.0000
0 0.1000 0.5000 0.9000 1.0000
0 0.1000 0.5000 0.9000 1.0000
0 0.1000 0.5000 0.9000 1.0000
Again, it is not the expected output with Matlab but the same result with the two previous examples.
Custom bi-linear image resize method
Basic principle
See this Wikipedia article on Bilinear interpolation for more complete information.
This figure should basically illustrates what happens when up-scaling from a 2x2 image to a 4x4 image:
With a nearest neighbor interpolation, the destination pixel at (0,0) will get the value of the source pixel at (0,0) as well as the pixels at (0,1), (1,0) and (1,1).
With a bi-linear interpolation, the destination pixel at (0,0) will get a value which is a linear combination of the 4 neighbors in the source image:
The four red dots show the data points and the green dot is the point
at which we want to interpolate.
R1 is calculated as: R1 = ((x2 – x)/(x2 – x1))*Q11 + ((x – x1)/(x2 – x1))*Q21.
R2 is calculated as: R2 = ((x2 – x)/(x2 – x1))*Q12 + ((x – x1)/(x2 – x1))*Q22.
Finally, P is calculated as a weighted average of R1 and R2: P = ((y2 – y)/(y2 – y1))*R1 + ((y – y1)/(y2 – y1))*R2.
Using coordinates normalized between [0, 1] simplifies the formula.
C++ implementation
This blog post (Resizing Images With Bicubic Interpolation) contains C++ code to perform image resizing with a bi-linear interpolation.
This is my own adaptation (some modifications about the indexes compared to the original code, not sure if it is correct) of the code to work with cv::Mat:
#include <iostream>
#include <opencv2/core.hpp>
float lerp(const float A, const float B, const float t) {
return A * (1.0f - t) + B * t;
}
template <typename Type>
Type resizeBilinear(const cv::Mat &src, const float u, const float v, const float xFrac, const float yFrac) {
int u0 = (int) u;
int v0 = (int) v;
int u1 = (std::min)(src.cols-1, (int) u+1);
int v1 = v0;
int u2 = u0;
int v2 = (std::min)(src.rows-1, (int) v+1);
int u3 = (std::min)(src.cols-1, (int) u+1);
int v3 = (std::min)(src.rows-1, (int) v+1);
float col0 = lerp(src.at<Type>(v0, u0), src.at<Type>(v1, u1), xFrac);
float col1 = lerp(src.at<Type>(v2, u2), src.at<Type>(v3, u3), xFrac);
float value = lerp(col0, col1, yFrac);
return cv::saturate_cast<Type>(value);
}
template <typename Type>
void resize(const cv::Mat &src, cv::Mat &dst) {
float scaleY = (src.rows - 1) / (float) (dst.rows - 1);
float scaleX = (src.cols - 1) / (float) (dst.cols - 1);
for (int i = 0; i < dst.rows; i++) {
float v = i * scaleY;
float yFrac = v - (int) v;
for (int j = 0; j < dst.cols; j++) {
float u = j * scaleX;
float xFrac = u - (int) u;
dst.at<Type>(i, j) = resizeBilinear<Type>(src, u, v, xFrac, yFrac);
}
}
}
void resize(const cv::Mat &src, cv::Mat &dst, const int width, const int height) {
if (width < 2 || height < 2 || src.cols < 2 || src.rows < 2) {
std::cerr << "Too small!" << std::endl;
return;
}
dst = cv::Mat::zeros(height, width, src.type());
switch (src.type()) {
case CV_8U:
resize<uchar>(src, dst);
break;
case CV_64F:
resize<double>(src, dst);
break;
default:
std::cerr << "Src type is not supported!" << std::endl;
break;
}
}
int main() {
cv::Mat img = (cv::Mat_<double>(2,2) << 0, 1, 0, 1);
std::cout << "img:\n" << img << std::endl;
cv::Mat img_resize;
resize(img, img_resize, 5, 5);
std::cout << "img_resize=\n" << img_resize << std::endl;
return EXIT_SUCCESS;
}
It produces:
img:
[0, 1;
0, 1]
img_resize=
[0, 0.25, 0.5, 0.75, 1;
0, 0.25, 0.5, 0.75, 1;
0, 0.25, 0.5, 0.75, 1;
0, 0.25, 0.5, 0.75, 1;
0, 0.25, 0.5, 0.75, 1]
Conclusion
In my opinion, it is unlikely that the OpenCV resize() function is wrong as none of the others image processing libraries I can test produce the expected output and moreover can produce the same OpenCV output with the good parameter.
I tested against two Python modules (scikit-image and Pillow) as they are easy to use and oriented about image processing. I was also able to test with Matlab and its image processing toolbox.
A rough custom implementation of the bi-linear interpolation for image resizing produces the expected result. Two possibilities for me could explain this behavior:
the difference is inherent to the method these image processing libraries use rather than a bug (maybe they use a method to resize images efficiently with some loss compared to a strict bi-linear implementation?)?
it is a somehow a convention to interpolate properly excluding the border?
These libraries are open-source and one can explore into their source code to understand where the discrepancy comes from.
The linked answer shows that the interpolation works only between the two original blue dots but I cannot explain why this behavior.
Why this answer?
This answer, even if it partially answers the OP question, is a good way for me to summarize the few things I found about this topic. I believe also it could help in some way other people who may found this.
As I will explain below, the output:
[[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]
[ 0. 0.1 0.5 0.9 1. ]]
would be the correct solution. So while opencv has small rounding errors, it is mostly correct.
The reason: your input image doesn't assume an image with the values "0" and "1" in the corners of the image, but in the center of the pixels.
So this the wrong model of what your 2x2 image looks like:
Instead, your image looks like this, with the "colors" defined in the red points. Everything to the left of the center of the left two pixels is just white, and everything to the right of the center of the right two pixels is just black, and values between the pixel centers are interpolated:
Converting the image to 5x5 pixels:
and looking at the centers of the pixels, you see how you would get "0.1" and "0.9" instead of "0.25" and "0.75"
Here's a simple code in python.
end = np.zeros((11,2))
alpha=0
while(alpha<=1):
end[int(10*alpha)] = alpha
print(end[int(10*alpha)])
alpha+=0.1
print('')
print(end)
and output:
[ 0. 0.]
[ 0.1 0.1]
[ 0.2 0.2]
[ 0.3 0.3]
[ 0.4 0.4]
[ 0.5 0.5]
[ 0.6 0.6]
[ 0.7 0.7]
[ 0.8 0.8]
[ 0.9 0.9]
[ 1. 1.]
[[ 0. 0. ]
[ 0.1 0.1]
[ 0.2 0.2]
[ 0.3 0.3]
[ 0.4 0.4]
[ 0.5 0.5]
[ 0.6 0.6]
[ 0.8 0.8]
[ 0. 0. ]
[ 1. 1. ]
[ 0. 0. ]]
It is easy to notice that 0.7 is missing and after 0.8 goes 0 instead of 0.9 etc... Why are these outputs differ?
It's because of floating point errors. Run this:
import numpy as np
end = np.zeros((11, 2))
alpha=0
while(alpha<=1):
print("alpha is ", alpha)
end[int(10*alpha)] = alpha
print(end[int(10*alpha)])
alpha+=0.1
print('')
print(end)
and you will see that alpha is, successively:
alpha is 0
alpha is 0.1
alpha is 0.2
alpha is 0.30000000000000004
alpha is 0.4
alpha is 0.5
alpha is 0.6
alpha is 0.7
alpha is 0.7999999999999999
alpha is 0.8999999999999999
alpha is 0.9999999999999999
Basically floating point numbers like 0.1 are stored inexactly on your computer. If you add 0.1 together say 8 times, you won't necessarily get 0.8 -- the small errors can accumulate and give you a different number, in this case 0.7999999999999999. Numpy arrays must take integers as indexes however, so it uses the int function to force this to round down to the nearest integer -- 7 -- which causes that row to be overwritten.
To solve this, you must rewrite your code so that you only ever use integers to index into an array. One slightly crude way would be to round the float to the nearest integer using the round function. But really you should rewrite your code so that it iterates over integers and coverts them into floats, rather than iterating over floats and converting them into integers.
You can read more about floating point numbers here:
https://docs.python.org/3/tutorial/floatingpoint.html
As #Denziloe pointed, this is due to floating point errors.
If you look at the definition of int():
If x is floating point, the conversion truncates towards zero
To solve your problem use round() instead of int()