Related
I am having trouble understanding why my code below is not producing the optimal result. I am attempting to create a step function but this clearly isn't working as the solution value for model.Z isn't one of the range points specified.
Any help in understanding/correcting this is greatly appreciated.
What I am trying to do
Maximize Z * X, subject to:
/ 20.5 , X <= 5
Z(X) = | 10 , 5 <= X <= 10
\ 9 , 10 <= X <= 11
Even better, I'd like to solve under the following conditions (disjoint at breakpoints):
/ 20.5 , X <= 5
Z(X) = | 10 , 5 < X <= 10
\ 9 , 10 < X <= 11
where X and Z are floating point numbers.
I would expect X to be 5 and Z to be 20.5, however model results are 7.37 and 15.53.
Code
from pyomo.core import *
# Break points for step-function
DOMAIN_PTS = [5., 10., 11.]
RANGE_PTS = [20.5, 10., 9.]
# Define model and variables
model = ConcreteModel()
model.X = Var(bounds=(5,11))
model.Z = Var()
# Set piecewise constraint
model.con = Piecewise(model.Z,model.X,
pw_pts=DOMAIN_PTS ,
pw_constr_type='EQ',
f_rule=RANGE_PTS,
force_pw=True,
pw_repn='SOS2')
model.obj = Objective(expr= model.Z * model.X, sense=maximize)
opt = SolverFactory('gurobi')
opt.options['NonConvex'] = 2
obj_val = opt.solve(model)
print(value(model.X))
print(value(model.Z))
print(model.obj())
I would never piecewice linearize z, but always z*x. If you have a piecewise linear expression for z only, then z*x is nonlinear (and in a nasty way). If you however write down a piecewise linear expression for z*x then the whole thing becomes linear. Note that discontinuities in the piecewise functions require attention.
It is important to understand mathematically what you are writing down before you pass it on to a solver.
Piecewise in Pyomo is intended to interpolate linearly between some bounds given by another variable. This means that if you give the bounds given as you're trying, your interpolating as this (sorry for such a bad graph) which basically means that you're placing a line between x=5 and x=10 a line given by Z= 31 - 2.1X, and another line between 10 and 11. In fact, Gurobi is computing the optimal result, since x its a NonNegativeReal and in such a line Z= 31 - 2.1X, x=7.37 give as result Z=15.53.
Now, I understand that you want rather a step function than a interpolation, something similar to this (sorry for such a bad graph, again), then you need to change your DOMAIN_PTS and RANGE_PTS in order to correctly model what you want
# Break points for step-function
DOMAIN_PTS = [5.,5., 10., 10.,11.]
RANGE_PTS = [20.5,10., 10., 9., 9.]
In this way, you're interpolating between f(x)=20.5: 5<=x<=10 and so on.
When running the ODR algorithm on some experiment data, I've been asked to run it with the following model:
It is clear that this fitting function is containing a redundant degree of freedom.
When I run the fitting on my experiment data I get enormous relative errors of beta, starting from 8000% relative error.
When I try to run the fitting again but with a fitting function that doesn't have a redundant degree of freedom, such as:
I don't get this kind of problem.
Why is this happening? Why the ODR algorithm is so sensitive for redundant degrees of freedom? I wasn't able to answer these questions to my supervisors. An answer will be much appreciated.
Reproducing code example:
from scipy.odr import RealData, Model, ODR
def func1(a, x):
return a[0] * (x + a[1]) / (a[3] * (x + a[1]) + a[1] * x) + a[2]
def func2(a, x):
return a[0] / (x + a[1]) + a[2]
# fmt: off
zx = [
1911.125, 2216.95, 2707.71, 3010.225, 3410.612, 3906.015, 4575.105, 5517.548,
6918.481,
]
dx = [
0.291112577, 0.321695254, 0.370771197, 0.401026507, 0.441068641, 0.490601621,
0.557573268, 0.651755155, 0.79184836,
]
zy = [
0.000998056, 0.000905647, 0.000800098, 0.000751041, 0.000699982, 0.000650532,
0.000600444, 0.000550005, 0.000500201,
]
dy = [
5.49029e-07, 5.02824e-07, 4.5005e-07, 4.25532e-07, 3.99991e-07, 3.75266e-07,
3.50222e-07, 3.25003e-07, 3.00101e-07,
]
# fmt: on
data = RealData(x=zx, y=zy, sx=dx, sy=dy)
print("Func 1")
print("======")
beta01 = [
1.46,
4775.4,
0.01,
1000,
]
model1 = Model(func1)
odr1 = ODR(data, model1, beta0=beta01)
result1 = odr1.run()
print("beta", result1.beta)
print("sd beta", result1.sd_beta)
print("relative", result1.sd_beta / result1.beta * 100)
print()
print()
print("Func 2")
print("======")
beta02 = [
1,
1,
1,
]
model2 = Model(func2)
odr2 = ODR(data, model2, beta0=beta02)
result2 = odr2.run()
print("beta", result2.beta)
print("sd beta", result2.sd_beta)
print("relative", result2.sd_beta / result2.beta * 100)
This prints out:
Func 1
======
beta [ 1.30884537e+00 -2.82585952e+03 7.79755196e-04 9.47943376e+01]
sd beta [1.16144608e+02 3.73765816e+06 6.12613738e-01 4.20775596e+03]
relative [ 8873.82193523 -132266.24068473 78564.88054498 4438.82627453]
Func 2
======
beta [1.40128121e+00 9.80844274e+01 3.00511669e-04]
sd beta [2.73990552e-03 3.22344713e+00 3.74538794e-07]
relative [0.1955286 3.28640051 0.12463369]
Scipy/Numpy/Python version information:
Versions are:
Scipy - 1.4.1
Numpy - 1.18.2
Python - 3.7.2
The problem is not with the degrees of freedom.
The degrees of freedom is the difference between the number of data points and the number of fitting parameters.
The problem has the same number of degrees of freedom for the two formulae, as they have the same number of parameters.
It also looks like that you do not have free degrees of freedom, which is good news, it means that it can potentially be fitted.
However, you are right that first expression has some problem: the parameters you are trying to fit are not independent.
This is probably better understood with some simpler example.
Consider the following expression:
y = x + b + c
which you try to fit, given n data for x and y with n >> 2.
The question is: what are the optimal value for b and c? This cannot be answered. All you can say from x and y data is about the combination. Therefore, if b + c is 0, the fit cannot tell us if b = 1000, c = -1000 or b = 1, c= -1, but at least we can say that given b we can determine c.
What is the error on a given b? Potentially infinite. That is the reason for the fitting to give you that large relative error.
I am looking to curve fit the following data, such that I get it to fit a trend with the condition of zero slope at the edges. The output of polyfit fits that data, but not with zero slopes at the edges.
Here is what I'm looking to output - pardon my Paint job. I need to it to fit like this so I can properly remove this sine/cosine bias of the data that isn't real towards the center.
Here is the data:
[0.23353535 0.25586247 0.26661164 0.26410896 0.24963951 0.22670266
0.19955422 0.17190263 0.1598439 0.17351905 0.18212444 0.18438673
0.17952432 0.18314894 0.19265689 0.19432385 0.19605163 0.20326011
0.20890851 0.20590997 0.21856518 0.23771665 0.24530019 0.23940831
0.22078396 0.23075128 0.2346082 0.22466281 0.24384843 0.26339594
0.26414153 0.24664183 0.24278978 0.31023648 0.3614195 0.37773436
0.3505998 0.28893167 0.23965877 0.24063917 0.27922502 0.32716477
0.36553767 0.42293146 0.50968856 0.5458872 0.52192533 0.45243764
0.36313155 0.3683921 0.40942553 0.4420537 0.46145585 0.4648034
0.4523771 0.4272876 0.39404616 0.3570107 0.35060245 0.3860975
0.3996996 0.44551122 0.46611032 0.45998383 0.4309985 0.38563925
0.37105605 0.4074444 0.48815584 0.5704579 0.6448988 0.7018853
0.73397845 0.73739105 0.7122451 0.6618154 0.591451 0.5076601
0.48578677 0.47347385 0.4791471 0.48306277 0.47025493 0.43479836
0.44380915 0.45868078 0.5341566 0.57549906 0.55790776 0.56244135
0.57668275 0.561856 0.67564166 0.7512851 0.76957643 0.7266262
0.734133 0.7231936 0.6776926 0.60511285 0.51599765 0.5579323
0.56723005 0.5440337 0.5775593 0.5950776 0.5722321 0.57858473
0.5652703 0.54723704 0.59561515 0.7071321 0.8169259 0.91443264
0.9883759 1.0275097 1.0235045 0.9737119 1.029139 1.1354861
1.1910824 1.1826864 1.1092159 0.9832138 0.9643041 0.92324203
0.9093703 0.88915515 1.0007693 1.0542978 1.0857164 1.0211861
0.88474303 0.8458009 0.76522666 0.7478076 0.90081936 1.0690157
1.1569089 1.1493248 1.0622779 1.0327609 0.9805119 0.9583969
0.8973544 0.9543319 0.9777171 0.94951093 0.97323567 1.0244237
1.0569099 1.0951824 1.0771195 1.3078191 1.7212077 2.09409
2.320331 2.3279085 2.125451 1.7908521 1.4180487 1.0744424
1.0218129 1.0916439 1.1255138 1.125803 1.1139745 1.2187989
1.300092 1.3025533 1.2312403 1.221301 1.2535597 1.2298189
1.1458241 1.1012102 1.0889369 1.1558667 1.3051153 1.4143198
1.6345526 1.8093723 1.9037704 1.8961821 1.7866236 1.5958548
1.3865516 1.5308585 1.6140417 1.627337 1.5733193 1.4981418
1.5048542 1.4935548 1.4798748 1.4131776 1.3792214 1.3728334
1.3683671 1.3593615 1.2995907 1.2965002 1.366058 1.4795257
1.5462885 1.61591 1.5968509 1.5222199 1.6210756 1.7074443
1.8351102 2.3187535 2.6568012 2.7676315 2.6480794 2.3636303
2.0673316 1.9607923 1.8074365 1.713272 1.5893831 1.4734347
1.507817 1.5213271 1.6091452 1.7162323 1.7608733 1.7497622
1.9187828 2.0197518 2.0487514 2.01107 1.9193696 1.7904462
1.8558109 2.1955926 2.4700975 2.6562278 2.675197 2.6645825
2.6295316 2.4182043 2.2114453 2.2506614 2.2086055 2.0497518
1.9557768 1.901191 2.067513 2.1077373 2.0159333 1.8138607
1.5413624 1.600069 1.7631899 1.9541935 1.9340311 1.805134
2.0671906 2.2247658 2.2641945 2.3594956 2.2504601 1.9749025
1.8905054 2.0679731 2.1193469 2.0307171 2.0717037 2.0340347
1.925536 1.7820351 1.9467943 2.315468 2.4834185 2.3751369
2.0240622 1.9363666 2.1732547 2.3113241 2.3264208 2.22015
2.0187428 1.7619076 1.796859 1.8757095 2.0501778 2.44711
2.6179967 2.508112 2.1694388 1.7242104 1.7671669 1.862043
1.8392721 1.7120028 1.6650634 1.6319505 1.482931 1.5240219
1.5815579 1.5691646 1.4766116 1.3731087 1.4666644 1.4061015
1.3652745 1.425564 1.4006845 1.5000012 1.581379 1.6329607
1.6444355 1.6098644 1.5300899 1.6876912 1.8968476 2.048039
2.1006014 2.0271482 1.8300935 1.6986666 1.9628603 2.0521066
1.9337255 1.6407858 1.2583638 1.2110122 1.2476432 1.2360718
1.2886397 1.2862154 1.2343681 1.1458222 1.209224 1.2475786
1.2353342 1.1797879 1.0963987 1.0928186 1.1553882 1.1569618
1.1932304 1.3002363 1.3386917 1.2973225 1.1816871 1.0557054
0.9350373 0.896656 0.8565816 0.90168726 0.9897751 1.02342
1.0232298 1.1199353 1.1466643 1.1081418 1.0377598 1.0348651
1.0223045 1.0607077 1.0089502 0.885213 1.023178 1.1131796
1.1331098 1.0779471 0.9626393 0.81472665 0.85455835 0.87542623
0.87286425 0.89130884 0.9545931 1.0355722 1.0201533 0.93568784
0.9180018 0.8202782 0.7450139 0.72550577 0.68578506 0.6431666
0.66193295 0.6386373 0.7060119 0.7650972 0.80093855 0.803342
0.76590335 0.7151591 0.6946282 0.7136788 0.7714012 0.8022328
0.79840165 0.8543819 0.8586749 0.8028453 0.7383879 0.73423904
0.65107304 0.61139977 0.5940311 0.6151931 0.59349155 0.54995483
0.5837645 0.5891752 0.56406695 0.5638191 0.5762535 0.58305734
0.5830114 0.57470953 0.5568098 0.52852243 0.49031836 0.45275375
0.47168964 0.46634504 0.4600581 0.45332378 0.41508177 0.3834329
0.4137769 0.41392407 0.3824464 0.36310086 0.434278 0.48041886
0.49433306 0.475708 0.43060693 0.36886734 0.34740242 0.34108457
0.36160505 0.40907663 0.43613982 0.4394311 0.42070773 0.38575593
0.3827834 0.4338096 0.46581286 0.45669746 0.40830874 0.3505502
0.32584783 0.3381971 0.33949164 0.36409503 0.3759155 0.3610108
0.37174097 0.39990777 0.38925973 0.34376588 0.32478797 0.32705626
0.3228174 0.30941254 0.28542265 0.2687348 0.25517422 0.26127565
0.27331188 0.3028561 0.31277937 0.29953563 0.2660389 0.27051866
0.2913383 0.30363902 0.30684754 0.3011791 0.28737035 0.26648855
0.26413882 0.25501928 0.23947525 0.21937743 0.19659272 0.18965112
0.21511254 0.23329383 0.24157354 0.2391297 0.22697571 0.20739041
0.1855308 0.18856761 0.19565174 0.20542233 0.21473111 0.22244582
0.22726117 0.22789808 0.22336568 0.21322969 0.20314343 0.2031754
0.19738965 0.1959791 0.20284075 0.20859875 0.21363212 0.21804498
0.22160804 0.22381367]
This came close, but not exactly it as the edges aren't zero slope: How do I fit a sine curve to my data with pylab and numpy?
Is there anything available that will let me do this without having to write up a custom algorithm to handle this? Thanks.
Here is a Lorentzian type of peak equation fitted to your data, and for the "x" values I used an index similar to what I see on the Example Output plot in your post. I have also zoomed in on the peak center to better display the sinusoidal shapes you mention. You may be able to subtract the predicted value from this peak equation to condition or pre-process the raw data as you discuss.
a = 1.7056067124682076E+02
b = 7.2900803359572393E+01
c = 2.5047064423525464E+02
d = 1.4184767800540945E+01
Offset = -2.4940994412221318E-01
y = a/ (b + pow((x-c)/d, 2.0)) + Offset
Starting from you own example based on a sine fit, I added the constraints such that the derivatives of the model have to be zero at the end points. I did this using symfit, a package I wrote to make this kind of thing easier. If you prefer to do this using scipy you can adapt the example to that syntax if you want, symfit is just a wrapper around their minimizers that adds symbolical manipulations using sympy.
# Make variables and parameters
x, y = variables('x, y')
a, b, c, d = parameters('a, b, c, d')
# Initial guesses
b.value = 1e-2
c.value = 100
# Create a model object
model = Model({y: a * sin(b * x + c) + d})
# Take the derivative and constrain the end-points to be equal to zero.
dydx = D(model[y], x).doit()
constraints = [Eq(dydx.subs(x, xdata[0]), 0),
Eq(dydx.subs(x, xdata[-1]), 0)]
# Do the fit!
fit = Fit(model, x=xdata, y=ydata, constraints=constraints)
fit_result = fit.execute()
print(fit_result)
plt.plot(xdata, ydata)
plt.plot(xdata, model(x=xdata, **fit_result.params).y)
plt.show()
This prints: (from current symfit PR#221, which has better reporting of the results.)
Parameter Value Standard Deviation
a 8.790393e-01 1.879788e-02
b 1.229586e-02 3.824249e-04
c 9.896017e+01 1.011472e-01
d 1.001717e+00 2.928506e-02
Status message Optimization terminated successfully.
Number of iterations 10
Objective <symfit.core.objectives.LeastSquares object at 0x0000016F670DF080>
Minimizer <symfit.core.minimizers.SLSQP object at 0x0000016F78057A58>
Goodness of fit qualifiers:
chi_squared 29.72125657199736
objective_value 14.86062828599868
r_squared 0.8695978050586373
Constraints:
--------------------
Question: a*b*cos(c) == 0?
Answer: 1.5904051811454707e-17
Question: a*b*cos(511*b + c) == 0?
Answer: -6.354261416082215e-17
To fit a hyperbolic function I am trying to use the following code:
import numpy as np
from scipy.optimize import curve_fit
def hyperbola(x, s_1, s_2, o_x, o_y, c):
# x > Input x values
# s_1 > slope of line 1
# s_2 > slope of line 2
# o_x > x offset of crossing of asymptotes
# o_y > y offset of crossing of asymptotes
# c > curvature of hyperbola
b_2 = (s_1 + s_2) / 2
b_1 = (s_2 - s_1) / 2
return o_y + b_1 * (x - o_x) + b_2 * np.sqrt((x - o_x) ** 2 + c ** 2 / 4)
min_fit = np.array([-3.0, 0.0, -2.0, -10.0, 0.0])
max_fit = np.array([0.0, 3.0, 3.0, 0.0, 10.0])
guess = np.array([-2.5/3.0, 4/3.0, 1.0, -4.0, 0.5])
vars, covariance = curve_fit(f=hyperbola, xdata=n_step, ydata=n_mean, p0=guess, bounds=(min_fit, max_fit))
Where n_step and n_mean are measurement values generated earlier on. The code runs fine and gives no error message, but it only returns the initial guess with a very small change. Also, the covariance matrix contains only zeros. I tried to do the same fit with a better initial guess, but that does not have any influence.
Further, I plotted the exact same function with the initial guess as input and that gives me indeed a function which is close to the real values. Does anyone know where I make a mistake here? Or do I use the wrong function to make my fit?
The issue must lie with n_step and n_mean (which are not given in the question as currently stated); when trying to reproduce the issue with some arbitrarily chosen set of input parameters, the optimization works as expected. Let's try it out.
First, let's define some arbitrarily chosen input parameters in the given parameter space by
params = [-0.1, 2.95, -1, -5, 5]
Let's see what that looks like:
import matplotlib.pyplot as plt
xs = np.linspace(-30, 30, 100)
plt.plot(xs, hyperbola(xs, *params))
Based on this, let us define some rather crude inputs for xdata and ydata by
xdata = np.linspace(-30, 30, 10)
ydata = hyperbola(xs, *params)
With these, let us run the optimization and see if we match our given parameters:
vars, covariance = curve_fit(f=hyperbola, xdata=xdata, ydata=ydata, p0=guess, bounds=(min_fit, max_fit))
print(vars) # [-0.1 2.95 -1. -5. 5. ]
That is, the fit is perfect even though our params are rather different from our guess. In other words, if we are free to choose n_step and n_mean, then the method works as expected.
In order to try to challenge the optimization slightly, we could also try to add a bit of noise:
np.random.seed(42)
xdata = np.linspace(-30, 30, 10)
ydata = hyperbola(xdata, *params) + np.random.normal(0, 10, size=len(xdata))
vars, covariance = curve_fit(f=hyperbola, xdata=xdata, ydata=ydata, p0=guess, bounds=(min_fit, max_fit))
print(vars) # [ -1.18173287e-01 2.84522636e+00 -1.57023215e+00 -6.90851334e-12 6.14480856e-08]
plt.plot(xdata, ydata, '.')
plt.plot(xs, hyperbola(xs, *vars))
Here we note that the optimum ends up being different from both our provided params and the guess, still within the bounds provided by min_fit and max_fit, and still provided a good fit.
I have lognormal distributed data in x0 and y0 arrays:
x0.ravel() = array([19.8815 , 19.0141 , 18.1857 , 17.3943 , 16.6382 , 15.9158 ,
15.2254 , 14.5657 , 13.9352 , 13.3325 , 12.7564 , 12.2056 ,
11.679 , 11.1755 , 10.6941 , 10.2338 , 9.79353, 9.37249,
8.96979, 8.58462, 8.21619, 7.86376, 7.52662, 7.20409,
6.89552, 6.6003 , 6.31784, 6.04757, 5.78897, 5.54151,
5.30472, 5.07812, 4.86127, 4.65375, 4.45514, 4.26506,
4.08314, 3.90903, 3.74238, 3.58288, 3.4302 , 3.28407,
3.14419, 3.01029, 2.88212, 2.75943, 2.64198, 2.52955,
2.42192, 2.31889, 2.22026, 2.12583, 2.03543, 1.94889,
1.86604, 1.78671, 1.71077, 1.63807, 1.56845, 1.50181,
1.43801, 1.37691, 1.31842, 1.26242, 1.2088 , 1.15746,
1.10832, 1.06126, 1.01619])
y0.ravel() =array([1.01567e+03, 8.18397e+02, 7.31992e+02, 1.11397e+03, 2.39987e+03,
2.73762e+03, 4.65722e+03, 7.06308e+03, 9.67945e+03, 1.38983e+04,
1.98178e+04, 1.97461e+04, 3.28070e+04, 4.48814e+04, 5.80853e+04,
7.35511e+04, 8.94090e+04, 1.08274e+05, 1.28276e+05, 1.50281e+05,
1.69258e+05, 1.91944e+05, 2.16416e+05, 2.37259e+05, 2.57426e+05,
2.74818e+05, 2.90343e+05, 3.01369e+05, 3.09232e+05, 3.13713e+05,
3.17225e+05, 3.19177e+05, 3.17471e+05, 3.14415e+05, 3.08396e+05,
2.95692e+05, 2.76097e+05, 2.52075e+05, 2.29330e+05, 1.97843e+05,
1.74262e+05, 1.46360e+05, 1.20599e+05, 9.82223e+04, 7.80995e+04,
6.34618e+04, 4.77460e+04, 3.88737e+04, 3.23715e+04, 2.58129e+04,
2.15724e+04, 1.58737e+04, 1.13006e+04, 7.64983e+03, 4.64590e+03,
3.31463e+03, 2.40929e+03, 3.02183e+03, 1.47422e+03, 1.06046e+03,
1.34875e+03, 8.26674e+02, 9.53167e+02, 6.47428e+02, 9.83651e+02,
8.93673e+02, 1.23637e+03, 0.00000e+00, 8.36573e+01])
I want to use curve_fit to get an function, that fits my data points, to gain the mu (and then the exp(mu) for the median) and the sigma of this distribution.
import numpy as np
from scipy.optimize import *
def f(x, mu, sigma) :
return 1/(np.sqrt(2*np.pi)*sigma*x)*np.exp(-((np.log(x)-
mu)**2)/(2*sigma**2))
params, extras = curve_fit(f, x0.ravel(), y0.ravel())
print "mu=%g, sigma=%g" % (params[0], params[1])
plt.plot(x0, y0, "o")
plt.plot(x0, f(x0 ,params[0], params[1]))
plt.legend(["data", "fit"], loc="best")
plt.show()
The result is the following:
mu=1.47897, sigma=0.0315236
Curve_fit
Obviously the function does not fit the data by any means.
When i multiply the fitting function by, lets say 1.3*10^(5) in the code:
plt.plot(x0, 1.3*10**5*f(x0 ,params[0], params[1]))
This is the result:
manually changed fitting curve
The calculated mu value, which is the mean value of the related normal distribution seems right, because when im using:
np.mean(np.log(x))
i get 1.4968838412183132, which is quite similar to the mu i obtain from curve_fit.
Calculating the median with
np.exp(np.mean(np.log(x))
gives 4.4677451525990675, which seems to be ok.
But unless i see the fitting function going threw my datapoints, i do not really trust these numbers. My problem is obviously, that the fitting function has no information of the (big) y0 values. How can i change that?
Any help apreciated!
The problem is, that your data are not(!) showing a lognormal pdf, since they are not normalized properly. Note, that the integral over a pdf has to be 1. If you numerically integrate your data and nomalize by that, e.g.
y1 = y0/np.trapz(x0, y0)
your approach works fine.
params, extras = curve_fit(f, x0, y1)
plt.plot(x0, y1, "o")
plt.plot(x0, f(x0 ,params[0], params[1]))
plt.legend(["data", "fit"], loc="best")
plt.show()
and
print("mu=%g, sigma=%g" % (params[0], params[1]))
resulting in
mu=1.80045, sigma=0.372185