I have a function of two variables k and T.
If have the value of the function for a number of (k,T) couple. However I do not have the same amount for each. For example I know the values f of the function at 2 T and 3 k:
F(k1,T1) = f1
F(k1,T2) = f2
F(k2,T1) = f3
F(k2,T2) = f4
F(k3,T1) = f5
F(k3,T2) = f6
I also know the form of the function F:
def func(X, a, b, c, omega):
T,k = X # The two variables
n = 1.0 / ( np.exp(omega / T ) - 1.0 )
return a * k * n + b * k**2 * (n + 1.0)
I would like to find the value of a,b,c and omega that minimize the error.
I tried with curve_fit:
k = [k1,k2,k3]
T = [T1,T2]
F[k1,T1] = f1
F[k1,T2] = f2
F[k2,T1] = f3
F[k2,T2] = f4
F[k3,T1] = f5
F[k3,T2] = f6
popt, pcov = curve_fit(func, (T,k), F )
However I get the following error (in my practical case I have 19 k values and 4 T values):
return a * k * n + b * k**2 * (n + 1.0)
ValueError: operands could not be broadcast together with shapes (19,) (4,)
Now if I create an array of higher dimension:
X = np.zeros((4,19,2))
for ii in np.arange(19):
X[0,ii,:] = np.array([T[0],k[ii]])
X[1,ii,:] = np.array([T[1],k[ii]])
X[2,ii,:] = np.array([T[2],k[ii]])
X[3,ii,:] = np.array([T[3],k[ii]])
and pass that:
def func(X, a, b, c, omega):
T = X[:,:,0]
k = X[:,:,1]
n = 1.0 / ( np.exp(omega / T ) - 1.0 )
return a * k * n + b * k**2 * (n + 1.0)
popt, pcov = curve_fit(func, X, F )
then I get the following issue:
minpack.error: Result from function call is not a proper array of floats.
Thank you in advance.
You need an array of pairs of data with the input X (probably your original dataset already looks like that) and the corresponding output array F:
X = np.array([k1,T1],[k1,T2],[k2,T1],[k2,T2],[k3,T1],[k3,T2])
F = [f1,f2,f3,f4,f5,f6]
Then calling the curve_fit function is directly:
popt, pcov = curve_fit(func, (X[:,0],X[:,1]),F)
Alternatively you can use single arrays for the k and T and use them in place of X[:,0] and X[:,1], but note that they should have the same dimensions since each element corresponds with the individual value of k and T of each observation/experiment. In other words, the index in the k or T array tells you the label of the corresponding observation.
Related
I have 2 formulas that describes the behaviour in 2 perpendicular axes. Also I have data from FEM simulation. The goal is to use least mean square method to get parameters Rr, Lr and cm.
I wanted to use scipy.curve_fit unfortunately it accepts only single function as an input. In this case i would need it to accept 2 functions as an input.
I did something in excel where arguments are inserted by hand to prove that it can/can not be perfectly fitted. They cant be but i would like to get "best" fit.
Any idea how it can be solved besides hard coding the last mean square method by hand to calculate deviances and find min?
Thank you so much for help.
If not using packages like lmfit or similar, fitting curves with shared parameters will always require to write some sort of wrapper. Personally I'd write a residual function and use scipy.optimize.least_squares, but if one insists to use curve_fit, this would be a possible wrapper:
import numpy as np
from scipy.optimize import curve_fit
def f1( x, c, L, R):
a = c**2 * x / ( R**2 + (x * L )**2 )
return a * x * L
def f2( x, c, L, R):
a = c**2 * x / ( R**2 + (x * L )**2 )
return a * R
def falt( x, c, L, R, n=-1):
"""
by construction x is the doubled x-list, 0 <= nn / l < 1
and >= 1/2 is the second part
"""
if isinstance( x, ( list, tuple, np.ndarray ) ):
### curve_fit sends array
l = len( x )
out = [ falt( xx, c, L, R, n=( nn / l ) ) for nn, xx in enumerate( x ) ]
else:
if n < 0.5:
out = f1( x, c, L, R)
else:
out = f2( x, c, L, R)
return out
## some data
c0=1.2
L0=0.3
R0= 0.45
size = 99
xl = np.linspace( 0, 10, size )
y1l = f1( xl , c0, L0, R0 ) + ( 2 * np.random.random( size=size ) - 1 ) * 0.1
y2l = f2( xl , c0, L0, R0 ) + ( 2 * np.random.random( size=size ) - 1 ) * 0.1
sol, err = curve_fit(
falt,
np.append( xl, xl ),
np.append( y1l, y2l )
)
print( sol )
You can put the relative importance of the functions in a hyperparamter lambda, then use func1 + lambda * func2.
With code:
importance_of_func1_relative_to_func2 = 1
def objective(args1, args2):
return func1(args1) * importance_of_func1_relative_to_func2 + func2(args2)
If I understand the goal (not sure of that!), I think that what you might want to do is have a single function that evaluates your 2 values for Fperp and Fpara and then concatenates them. You wrote those as both being multiplied by |z| (maybe abs(zhat)?) -- I cannot tell if that should be a common scaling factor, a fitting variable, or some other array of values...
Anyway, I might suggest a function like
def f_model(omega, cm, rr, lr, zhat):
ll = lr * omega
scale = abs(zhat) * cm**2 / (rr**2 + ll**2)
fpara = scale * ll * omega
fperp = scale * rr * omega
return np.concatenate((fpara, fperp))
Then you would want to arrange the data that you model with this function to also be the concatenation of the data corresponding to fpara and fperp.
That concatenation would effectively fit Fpara and Fperp together, weighting them evenly in the fit.
I have a problem. I want to calculate everything in the quadratic function. equations for reference:
ax^2 + bx + c
a(x-p)^2 + q
I made 8 possible inputs in tkinter and I want my program to try and calculate everything if possible. Otherwise to return sth like not enough data.
Equations:
delta = b^2-4ac
p = -b/(2a)
p = (x1+x2)/2
q = -delta/(4a)
#if delta>0
x2 = (-b-sqrt(delta))/(2a)
x1 = (-b+sqrt(delta))/(2a)
#if delta=0
x0 = -b/(2a)
#if delta<0 no solutions
#a, b, c are the coefficients.
b = -2ap
c = p^2*a+q
Example:
Input:
p = 3
q = -9
x1 = 0.877868
a = 2
Output:
b = -12
c = 9
x2 = 5.12132
delta = 72
So, for example, I give it [x1, x2, a] and it will calculate [q, p, b, c, and delta] if possible.
Is there a function that I can give all different formulas to and it will try to calculate everything?
For now, my only idea is to brute force it in 'try' or with 'ifs', but I feel like it would take thousands of lines of code, so I won't do that.
I found out that you can use sympy's solve. This is my solution. I used Eq for defining equations and then solved them.
def missingVariables(dictOfVariables):
symbols = dict(a=sym('a'), b=sym('b'), c=sym('c'), p=sym('p'), q=sym('q'), x1=sym('x1'), x2=sym('x2'),
delta=sym('delta'))
var = mergeDicts(symbols, dictOfVariables)
deltaEQ = Eq((var['b'] ** 2 - 4 * var['a'] * var['c']), var['delta'])
x1EQ = Eq(((-var['b'] - sqrt(var['delta'])) / (2 * var['a'])), var['x1'])
x2EQ = Eq(((-var['b'] + sqrt(var['delta'])) / (2 * var['a'])), var['x2'])
pEQ = Eq((-var['b']) / (2 * var['a']), var['p'])
pEQ2 = Eq(((var['x1'] + var['x2']) / 2), var['p'])
qEQ = Eq(((-var['delta']) / (4 * var['a'])), var['q'])
bEQ = Eq((-2 * var['a'] * var['p']), var['b'])
cEQ = Eq((var['a'] * var['p'] ** 2 + var['q']), var['c'])
solution = solve((deltaEQ, x1EQ, x2EQ, pEQ, pEQ2, qEQ, bEQ, cEQ))
solution = solution[0]
new_dict = {}
for k, v in solution.items():
try:
new_dict[str(k)] = round(float(v), 4)
except TypeError:
new_dict[str(k)] = v
return new_dict
As mentioned above, the function below works, however its very slow. I am very interested in using faster/optimised numpy (or other) vectorized alternatives. I have not posted the entire script here due to it being too large.
My specific question is - are there suitable numpy (or other) functions that I can use to 1) reduce run time and 2) reduce code volume of this function, specifically the for loop?
Edit: mass, temp, U and dpdh are functions that carry out simple algebraic calculations and return constants
def my_system(t, y, n, hIn, min, mAlumina, cpAlumina, sa, V):
dydt = np.zeros(3 * n) #setting up zeros array for solution (solving for [H0,Ts0,m0,H1,Ts1,m1,H2,Ts2,m2,..Hn,Tsn,mn])
# y = [h_0, Ts_0, m_0, ... h_n, Ts_n, m_n]
# y[0] = hin
# y[1] = Ts0
# y[2] = minL
i=0
## Using thermo
T = temp(y[i],P) #initial T
m = mass(y[i],P) #initial m
#initial values
dydt[i] = (min * (hIn - y[i]) + (U(hIn,P,min) * sa * (y[i + 1] - T))) / m # dH/dt (eq. 2)
dydt[i + 1] = -(U(hIn,P,min) * sa * (y[i + 1] - T)) / (mAlumina * cpAlumina) # dTs/dt from eq.3
dmdt = dydt[i] * dpdh(y[i], P) * V # dm/dt (holdup variation) eq. 4b
dydt[i + 2] = min - dmdt # mass flow out (eq.4a)
for i in range(3, 3 * n, 3): #starting at index 3, and incrementing by 3 because we are solving for 'triplets' [h,Ts,m] in each loop
## Using thermo
T = temp(y[i],P)
m = mass(y[i],P)
# [h, TS, mdot]
dydt[i] = (dydt[i-1] * (y[i - 3] - y[i]) + (U(y[i-3], P, dydt[i-1]) * sa * (y[i + 1] - T))) / m # dH/dt (eq.2), dydt[i-1] is the mass of the previous tank
dydt[i + 1] = -(U(y[i-3], P, dydt[i-1]) * sa * (y[i + 1] - T)) / (mAlumina * cpAlumina) # dTs/dt eq. (3)
dmdt = dydt[i] * dpdh(y[i], P) * V # Equation 4b
dydt[i + 2] = dydt[i-1] - dmdt # Equation 4a
return dydt
The functions mass, temp, U, and dpdh used inside the my_system function all take numbers as input, perform some simple algebraic operation and return a number (no need to optimise these I am just providing them for further context)
def temp(H,P):
# returns temperature given enthalpy (after processing function)
T = flasher.flash(H=H, P=P, zs=zs, retry=True).T
return T
def mass(H, P):
# returns mass holdup in mol
m = flasher.flash(H=H, P=P, zs=zs, retry=True).rho()*V
return m
def dpdh(H, P):
res = flasher.flash(H=H, P=P, zs=zs, retry=True)
if res.phase_count == 1:
if res.phase == 'L':
drho_dTf = res.liquid0.drho_dT()
else:
drho_dTf = res.gas.drho_dT()
else:
drho_dTf = res.bulk._equilibrium_derivative(of='rho', wrt='T', const='P')
dpdh = drho_dTf/res.dH_dT_P()
return dpdh
def U(H,P,m):
# Given T, P, m
air = Mixture(['nitrogen', 'oxygen'], Vfgs=[0.79, 0.21], H=H, P=P)
mu = air.mu*1000/mWAir #mol/m.s
cp = air.Cpm #J/mol.K
kg = air.k #W/m.K
g0 = m/areaBed #mol/m2.s
a = sa*n/vTotal #m^2/m^3 #QUESTIONABLE
psi = 1
beta = 10
pr = (mu*cp)/kg
re = (6*g0)/(a*mu*psi)
hfs = ((2.19*(re**1/3)) + (0.78*(re**0.619)))*(pr**1/3)*(kg)/diameterParticle
h = 1/((1/hfs) + ((diameterParticle/beta)/kAlumina))
return h
Reference Image:
enter image description here
For improving the speed, you can see Numba, which is useable if you use NumPy a lot but not every code can be used with Numba. Apart from that, the formulation of the equation system is confusing. You are solving 3 equations and adding the result to a single dydt list by 3 elements each. You can simply create three lists, solve each equation and add them to their respective list. For this, you need to re-write my_system as:
import numpy as np
def my_system(t, RHS, hIn, Ts0, minL, mAlumina, cpAlumina, sa, V):
# get initial boundary condition values
y1 = RHS[0]
y2 = RHS[1]
y3 = RHS[2]
## Using thermo
T = # calculate T
m = # calculate m
# [h, TS, mdot] solve dy1dt for h, dy2dt for TS and dy3dt for mdot
dy1dt = # dH/dt (eq.2), y1 corresponds to initial or previous value of dy1dt
dy2dt = # dTs/dt eq. (3), y2 corresponds to initial or previous value of dy2dt
dmdt = # Equation 4b
dy3dt = # Equation 4a, y3 corresponds to initial or previous value of dy3dt
# Left-hand side of ODE
LHS = np.zeros([3,])
LHS[0] = dy1dt
LHS[1] = dy2dt
LHS[2] = dy3dt
return LHS
In this function, you can pass RHS as a list with initial values ([dy1dt, dy2dt, dy3dt]) which will be unpacked as y1, y2, and y3 respectively and use them for respective differential equations. The solved equations (next values) will be saved to dy1dt, dy2dt, and dy3dt which will be returned as a list LHS.
Now you can solve this using scipy.integrate.odeint. Therefore, you can leave the for loop structure and solve the equations by using this method as follows:
hIn = #some val
Ts0 = #some val
minL = #some val
mAlumina = #some vaL
cpAlumina = #some val
sa = #some val
V = #some val
P = #some val
## Using thermo
T = temp(hIn,P) #initial T
m = mass(hIn,P) #initial m
#initial values
y01 = # calculate dH/dt (eq. 2)
y02 = # calculate dTs/dt from eq.3
dmdt = # calculate dm/dt (holdup variation) eq. 4b
y03 = # calculatemass flow out (eq.4a)
n = # time till where you want to solve the equation system
y0 = [y01, y02, y03]
step_size = 1
t = np.linspace(0, n, int(n/step_size)) # use that start time to which initial values corresponds
res = odeint(my_sytem, y0, t, args=(hIn, Ts0, minL, mAlumina, cpAlumina, sa, V,), tfirst=True)
print(res[:,0]) # print results for dH/dt
print(res[:,1]) # print results for dTs/dt
print(res[:,2]) # print results for Equation 4a
Here, I have passed all the initial values as y0 and chosen a step size of 1 which you can change as per your need.
I'm trying to get the coefficients of a function p(T,x). I provided the data for p, T and x from excel sheets via panda. The following Code works quite nice for me:
import pandas as pd
import os
from scipy.optimize import curve_fit
import numpy as np
df = pd.read_excel(os.path.join(os.path.dirname(__file__), "./Data.xlsx"))
T = np.array(df['T'], dtype=float)
x = np.array(df['x'], dtype=float)
p = np.array(df['p'], dtype=float)
p_s = 67.17
def func(X, a, b, c, d, e, f):
T, x = X
return x * p_s + x * (1 - x) * (a + b * T + c * T ** 2 + d * x + e * x * T + f * x * T ** 2) * p_s
popt, pcov = curve_fit(func, (T, x), p)
print("a = %s , b = %s, c = %s, d = %s, e = %s, f = %s" % (popt[0], popt[1], popt[2], popt[3], popt[4], popt[5]))
My acutal problem is, that the function is swinging a little bit in the end.
Because of this behaviour i get two x values for one p value, which i dont want.
So to avoid this little swing i want to accomplish a boundary condition for the fitting that say something like dp/dx (for constant T) > 0. With dp/dx I mean the derivation of the function after x.
Is this possible with the normal bound paramter of curve_fit? How can I do this?
EDIT:
As suggested I've messed a little bit around with least_square function but I guess that I came to a point where I don't realy understand what I'm doing or have to do.
T = np.array(df['T'], dtype=float)
x = np.array(df['x'], dtype=float)
p = np.array(df['p'], dtype=float)
p_s = 67
def f(X, z):
T, x = X
return x * p_s + x * (1 - x) * (z[0] + z[1] * T + z[2] * T ** 2 + z[3] * x + z[4] * x * T + z[5] * x * T ** 2) * p_s
def g(X,p,z):
return p - f(X ,z)
z0 = np.array([0,0,0,0,0,0], dtype=float)
res, flag = least_squares(g,z0, args=(T,x,p))
print(res)
With this code I get the following error:
TypeError: g() takes 3 positional arguments but 4 were given
Starting with:
a,b=np.ogrid[0:n+1:1,0:n+1:1]
B=np.exp(1j*(np.pi/3)*np.abs(a-b))
B[z,b] = np.exp(1j * (np.pi/3) * np.abs(z - b +x))
B[a,z] = np.exp(1j * (np.pi/3) * np.abs(a - z +x))
B[diag,diag]=1-1j/np.sqrt(3)
this produces an n*n grid that acts as a matrix.
n is just a number chosen to represent the indices, i.e. an a*b matrix where a and b both go up to n.
Where z is a constant I choose to replace a row and column with the B[z,b] and B[a,z] formulas. (Essentially the same formula but with a small number added to the np.abs(a-b))
The diagonal of the matrix is given by the bottom line:
B[diag,diag]=1-1j/np.sqrt(3)
where,
diag=np.arange(n+1)
I would like to repeat this code 50 times where the only thing that changes is x so I will end up with 50 versions of the B np.ogrid. x is a randomly generated number between -0.8 and 0.8 each time.
x=np.random.uniform(-0.8,0.8)
I want to generate 50 versions of B with random values of x each time and take a geometric average of the 50 versions of B using the definition:
def geo_mean(y):
y = np.asarray(y)
return np.prod(y ** (1.0 / y.shape[0]), axis=-1)
I have tried to set B as a function of some index and then use a for _ in range(): loop, this doesn't work. Aside from copy and pasting the block 50 times and denoting each one as B1, B2, B3 etc; I can't think of another way of working this out.
EDIT:
I'm now using part of a given solution in order to show clearly what I am looking for:
#A matrix with 50 random values between -0.8 and 0.8 to be used in the loop
X=np.random.uniform(-0.8,0.8, (50,1))
#constructing the base array before modification by random x values in position z
a,b = np.ogrid[0:n+1:1,0:n+1:1]
B = np.exp(1j * ( np.pi / 3) * np.abs( a - b ))
B[diag,diag] = 1 - 1j / np.sqrt(3)
#list to store all modified arrays
randomarrays = []
for i in range( 0,50 ):
#copy array and modify it
Bnew = np.copy( B )
Bnew[z, b] = np.exp( 1j * ( np.pi / 3 ) * np.abs(z - b + X[i]))
Bnew[a, z] = np.exp( 1j * ( np.pi / 3 ) * np.abs(a - z + X[i]))
randomarrays.append(Bnew)
Bstack = np.dstack(randomarrays)
#calculate the geometric mean value along the axis that was the row in 2D arrays
B0 = geo_mean(Bstack)
From this example, every iteration of i uses the same value of X, I can't seem to get a way to get each new loop of i to use the next value in the matrix X. I am unsure of the ++ action in python, I know it does not work in python, I just don't know how to use the python equivalent. I want a loop to use a value of X, then the next loop to use the next value and so on and so forth so I can dstack all the matrices at the end and find a geo_mean for each element in the stacked matrices.
One pedestrian way would be to use a list comprehension or generator expression:
>>> def f(n, z, x):
... diag = np.arange(n+1)
... a,b=np.ogrid[0:n+1:1,0:n+1:1]
... B=np.exp(1j*(np.pi/3)*np.abs(a-b))
... B[z,b] = np.exp(1j * (np.pi/3) * np.abs(z - b +x))
... B[a,z] = np.exp(1j * (np.pi/3) * np.abs(a - z +x))
... B[diag,diag]=1-1j/np.sqrt(3)
... return B
...
>>> X = np.random.uniform(-0.8, 0.8, (10,))
>>> np.prod((*map(np.power, map(f, 10*(4,), 10*(2,), X), 10 * (1/10,)),), axis=0)
But in your concrete example we can do much better than that;
using the identity exp(a) x exp(b) = exp(a + b) we can convert the geometric mean after exponentiation to an arithmetic mean before exponentition. A bit of care is required because of the multivaluedness of the complex n-th root which occurs in the geometric mean. In the code below we normalize the angles occurring to range -pi, pi so as to always hit the same branch as the n-th root.
Please also note that the geo_mean function you provide is definitely wrong. It fails the basic sanity check that taking the average of copies of the same thing should return the same thing. I've provided a better version. It is still not perfect, but I think there actually is no perfect solution, because of the nonuniqueness of the complex root.
Because of this I recommend taking the average before exponentiating. As long as your random spread is less than pi this allows a well-defined averaging procedure with an average that is actually close to the samples
import numpy as np
def f(n, z, X, do_it_pps_way=True):
X = np.asanyarray(X)
diag = np.arange(n+1)
a,b=np.ogrid[0:n+1:1,0:n+1:1]
B=np.exp(1j*(np.pi/3)*np.abs(a-b))
X = X.reshape(-1,1,1)
if do_it_pps_way:
zbx = np.mean(np.abs(z-b+X), axis=0)
azx = np.mean(np.abs(a-z+X), axis=0)
else:
zbx = np.mean((np.abs(z-b+X)+3) % 6 - 3, axis=0)
azx = np.mean((np.abs(a-z+X)+3) % 6 - 3, axis=0)
B[z,b] = np.exp(1j * (np.pi/3) * zbx)
B[a,z] = np.exp(1j * (np.pi/3) * azx)
B[diag,diag]=1-1j/np.sqrt(3)
return B
def geo_mean(y):
y = np.asarray(y)
dim = len(y.shape)
y = np.atleast_2d(y)
v = np.prod(y, axis=0) ** (1.0 / y.shape[0])
return v[0] if dim == 1 else v
def geo_mean_correct(y):
y = np.asarray(y)
return np.prod(y ** (1.0 / y.shape[0]), axis=0)
# demo that orig geo_mean is wrong
B = np.exp(1j * np.random.random((5, 5)))
# the mean of four times the same thing should be the same thing:
if not np.allclose(B, geo_mean([B, B, B, B])):
print('geo_mean failed')
if np.allclose(B, geo_mean_correct([B, B, B, B])):
print('but geo_mean_correct works')
n, z, m = 10, 3, 50
X = np.random.uniform(-0.8, 0.8, (m,))
B0 = f(n, z, X, do_it_pps_way=False)
B1 = np.prod((*map(np.power, map(f, m*(n,), m*(z,), X), m * (1/m,)),), axis=0)
B2 = geo_mean_correct([f(n, z, x) for x in X])
# This is the recommended way:
B_recommended = f(n, z, X, do_it_pps_way=True)
print()
print(np.allclose(B1, B0))
print(np.allclose(B2, B1))
I think you should rely more on numpy functionality, when approaching your problem. Not a numpy expert myself, so there is surely room for improvement:
from scipy.stats import gmean
n = 2
z = 1
a = np.arange(n + 1).reshape(1, n + 1)
#constructing the base array before modification by random x values in position z
B = np.exp(1j * (np.pi / 3) * np.abs(a - a.T))
B[a, a] = 1 - 1j / np.sqrt(3)
#list to store all modified arrays
random_arrays = []
for _ in range(50):
#generate random x value
x=np.random.uniform(-0.8, 0.8)
#copy array and modify it
B_new = np.copy(B)
B_new[z, a] = np.exp(1j * (np.pi / 3) * np.abs(z - a + x))
B_new[a, z] = np.exp(1j * (np.pi / 3) * np.abs(a - z + x))
random_arrays.append(B_new)
#store all B arrays as a 3D array
B_stack = np.stack(random_arrays)
#calculate the geometric mean value along the axis that was the row in 2D arrays
geom_mean_for_rows = gmean(B_stack, axis = 2)
It uses the geometric mean function from scipy.stats module to have a vectorised approach for this calculation.