I´m trying to construct a function that return the derivate of f, a function of one variable.
The return value should be a function approximating the derivative of f'
using the symmetric difference quotient, so that the returned function will compute (f(x+h) -f(x-h))/2h.
The function should start like this:
def derivative(f, x):
which should approximate the derivative of function f around the point x.
Does anyone have a clue what type of code I can use to construct this type of function?
/Alex
For a general function f(x), you can straightforwardly obtain a numerical approximation to its first derivative by the standard (second-order) approximation (f(x+h) - f(x-h)) /2h. The main challenge is to choose h to be small compared to the lenghscale over which f(x) shows non-quadratic variation, but sufficiently large to avoid round-off errors when subtracting nearby values of f(x).
However, if you want an algebraic method of differentiating your function, then things are more challenging. The easy cases are where f(x) is known to be a polynomial, so can be represented by a vector of coefficients of powers of x. In that case, numpy.polyder() can be used to compute the coefficients of the n'th derivative.
For more complicated functions, you may want to look at SymPy.
Both the numpy.polyder() and SymPy options require you to represent your function in a way that is specialized to these particular tools. I'm not aware of any method that can take an ordinary Python function and construct another function that implements the exact derivative.
what do you want the function to return?
If you want the value of the derivative in a certain x, you probably need three arguments:
def derivative(f, h, x):
return (f(x+h) - f(x-h))/2h
if you want to get a function which calculates the above for any x you can use:
def derivative(f, h):
return lambda x: (f(x+h) - f(x-h))/2h
Your best best bet would probably be to use SymPy which can do symbolic integration and differentation among other things:
>>> from sympy import *
>>> x, y, z = symbols('x y z')
>>> diff(x**2, x)
2*x
First you can define a function f (Example: f(x) = x ^ 2):
def f(x): return x ** 2
Next using the definition of derivative:
def derivative(function, x, accuracy = 20): # The 'Default' of accuracy is 20 and is an optional argument.
step = 1 / accuracy
return (function(x + step) - function(x - step)) / (step * 2)
~~~~~~~~~~~~~~~~~~~~~
By the way, I believe this is a typo:
def derivative(f, h):
Since you are approximating the derivative of function f around the point x, it should be:
def derivative(f, x):
As shown in my code
Related
Let us suppose that my function is
(z : C -> C)
z = x - i*y
now here the real part is,
u(x, y) = x
the imaginary part is,
v(x, y) = -y
so, when we get the derivatives, we find
d_u_x(x,y) = 1 # derivative of u wrt x
d_u_y(x,y) = 0
d_v_x(x, y) = 0
d_v_y(x, y) = -1
so, here,
d_u_x != d_v_y
thus, it does not follow Cauchy Reimann equation.
but, then comes the Wirtinger calculus, that says, I could write my function as,
u(x, y) = ((x + iy) + (x - iy))/2
= (z + z.conj())/2
v(x, y) = (((x + iy) - (x - iy))/2i
= (z - z.conj())/2i
but what after this, how do I find the gradient.
plus, in PyTorch, what is the correct way to specify such a function,
if I do,
import torch
a = torch.randn(1, dtype=torch.cfloat, requires_grad=True)
f = a.conj()
f.backward()
print(a.grad)
is this a correct way?
You may find the following page of interest:
When you use PyTorch to differentiate any function f(z) with complex domain and/or codomain, the gradients are computed under the assumption that the function is a part of a larger real-valued loss function g(input)=L. The gradient computed is ∂L/∂z* (note the conjugation of z), the negative of which is precisely the direction of steepest descent used in Gradient Descent algorithm. Thus, all the existing optimizers work out of the box with complex parameters.
This convention matches TensorFlow’s convention for complex differentiation, but is different from JAX (which computes ∂L/∂z).
If you have a real-to-real function which internally uses complex operations, the convention here doesn’t matter: you will always get the same result that you would have gotten if it had been implemented with only real operations.
...
For optimization problems, only real valued objective functions are used in the research community since complex numbers are not part of any ordered field and so having complex valued loss does not make much sense.
It also turns out that no interesting real-valued objective fulfill the Cauchy-Riemann equations. So the theory with homomorphic function cannot be used for optimization and most people therefore use the Wirtinger calculus.
https://pytorch.org/docs/stable/notes/autograd.html
I'm currently working through some exercises on multivariable function calculus and thought I would have a go at making my own function to determine gradient and hessian at a defined point for any function. I'm currently having issues when attempting to substitute the resulting matrices with coordinate values for an arbitrary function. I've already managed to solve specific examples, but my attempt to make a function to solve a user defined function isn't working correctly.
def multivariable_function(function, variables, substitute=(0,0)):
"""Determines Gradient and Hessian vectors for multivariable function.
Args:
function: Enter the multivariable function
variables: Enter list of variable names
substitute: Default = (0,0)
Returns:
gradient/hessian matrices for given coordinate
To do:
Include sympy symbol() generation within function
"""
#derive_by_array returns a gradient matrix for multivariable function
Gradient = simplify(derive_by_array(function, variables))
#derive_by_array returns a Hessian matrix for multivariable function
Hessian = simplify(derive_by_array(derive_by_array(function, variables), variables))
#Line currently isn't doing anything
Gradient.subs(zip(variables, substitute))
return Gradient, Hessian
This is the basic function so far in operation.
multivariable_function((x**2)*(y**3) + exp(2*x + x*y - 1) - (x**3 + 3*y**2)**2, (x,y))`
which yields the following result, I am however aiming to substitute the desired values into the gradient and hessian matrices to achieve the following desired result. I managed to achieved the desired result using the following.
from sympy import *
x, y, z, K, T, r, σ, h, a, f, μ, c, t, m, x1, x2, x3 = symbols('x, y, z, K, T, r, σ, h, a, f, μ, c, t, m, x1, x2, x3') # Variables used must be defined in sympy.
init_printing(use_unicode=False) #Print the answers in unicode characters
function = (x**2)*(y**3) + exp(2*x + x*y - 1) - (x**3 + 3*y**2)**2
Gradient_1 = simplify(derive_by_array(function, (x, y)))
Hessian_1 = simplify(derive_by_array(derive_by_array(function, (x, y)), (x, y)))
Gradient_1.subs(x, 0).subs(y,0), Hessian_1.subs(x,0).subs(y,0)
After viewing the issue raised here, it seems zipping the two lists should enable the subs() function to work, but it currently isn't for me. I attempted to loop through 'variables and 'substitute' to sequentially apply .subs(), however I'm finding the function only works if the method is chained for all replacement variables, as in the example above.
Does anyone know how I can apply the .subs() n times for a given coordinate to yield the relevant gradient/hessian matrices?
The variable Gradient is of type
sympy.tensor.array.dense_ndim_array.ImmutableDenseNDimArray
Like almost all SymPy objects, with exception of mutable matrices, it is immutable. The method subs does not modify it in place; it returns a new object, which needs to be assigned.
Gradient = Gradient.subs(zip(variables, substitute))
Hessian = Hessian.subs(zip(variables, substitute))
Then the function works as expected, returning
([2*exp(-1), 0], [[4*exp(-1), exp(-1)], [exp(-1), 0]])
But I suggest not passing generators to subs; there are outstanding issues involving that. Convert to a list or a dict first, to be safe. (There is also a difference there: should substitutions be consecutive or simultaneous, although this does not matter when substituting numbers for symbols.)
subs_dict = dict(zip(variables, substitute))
Gradient = Gradient.subs(subs_dict)
Hessian = Hessian.subs(subs_dict)
I'm solving the integral numerically using python:
where a(x) can take on any value; positive, negative, inside or outside the the [-1;1] and eta is an infinitesimal positive quantity. There is a second outer integral of which changes the value of a(x)
I'm trying to solve this using the Sokhotski–Plemelj theorem:
However this involves determining the principle value, which I can't find any method to in python. I know it's implemented in Matlab, but does anyone know of either a library or some other way of the determining the principal value in python (if a principle value exists)?
You can use sympy to evaluate the integral directly. Its real part with eta->0 is the principal value:
from sympy import *
x, y, eta = symbols('x y eta', real=True)
re(integrate(1/(x - y + I*eta), (x, -1, 1))).simplify().subs({eta: 0})
# -> log(Abs(-y + 1)/Abs(y + 1))
Matlab's symbolic toolbox int gives you the same result, of course (I'm not aware of other relevant tools in Matlab for this --- please specify if you know a specific one).
You asked about numerical computation of a principal value. The answer there is that if you only have a function f(y) whose analytical form or behavior you don't know, it's in general impossible to compute them numerically. You need to know things such as where the poles of the integrand are and what order they are.
If you on the other hand know your integral is of the form f(y) / (y - y_0), scipy.integrate.quad can compute the principal value for you, for example:
import numpy as np
from scipy import integrate, special
# P \int_{-1}^1 dx 1/(x - wvar) * (1 + sin(x))
print(integrate.quad(lambda x: 1 + np.sin(x), -1, 1, weight='cauchy', wvar=0))
# -> (1.8921661407343657, 2.426947531830592e-13)
# Check against known result
print(2*special.sici(1)[0])
# -> 1.89216614073
See here for details.
I try to generate a generic fit polynom using SciPy's curve_fitmethod. My current simplified code looks like the following:
import functools
import scipy.optimize
def __fit_polynom_order_6(self, data):
def func(x, c1=None, c2=None, c3=None, c4=None, c5=None, c6=None):
return c1*x + c2*x**2 + c3*x**3 + c4*x**4 + c5*x**5 + c6*x**6
x, y = data[:,0], data[:,1]
popt, pcov = scipy.optimize.curve_fit(func, x, y)
func_fit = functools.partial(func, c1=popt[0],c2=popt[1],c3=popt[2],c4=popt[3],c5=popt[4],c6=popt[5])
return func_fit
Now I want also to do fits with polynoms of order n and thus generate a generic function __fit_polynom_order_n(self, n, data) that generates the polynom automatically and does essentially the same thing as my function above but with arbitrary polynoms.
My attempts doing this all came to nothing. Can you help? Thanks in advance!
There is already a function for that, np.polyfit:
fit = np.polyfit(x, y, n)
On the other hand, your func does not have a constant term. Is that on purpose?
If you wish to write your own polyfit-type method, you might want to study the source code for np.polyfit. You'll see that the problem is set up as a linear matrix equation and solved with np.linalg.lstsq, rather than the more general-purpose scipy.optimize.curve_fit.
# set up least squares equation for powers of x
lhs = vander(x, order)
rhs = y
c, resids, rank, s = lstsq(lhs, rhs, rcond)
Useful reference:
np.vander -- aha, this can be used to evaluate the polynomial at x. If you want to eliminate the constant term, you'd have to chop off the right-most column returned by np.vander.
I have a timeseries that I want to fit to function using Scipy.optimize.leastsq.
fitfunc= lambda a, x: a[0]+a[1]*exp(-x/a[4])+a[2]*exp(-x/a[5])+a[3]*exp(-x /a[6])
errfunc lambda a,x,y: fitfunc(a,x) - y
Next I would pass errfunc to leastsq to minimze it. The fit-function I use is a sum of exponentials decaying with different timescales a(4:6) and different weights (a(0:4)). (as a sideuqestion: can I use leastsq with more than 1 parameter arrays? I didn't suceed to do so....)
The question: How can I put additional side conditions on the parameters entering the fit-function. I want for example that sum(a(0:4))=1.0
Just use
import numpy as np
def fitfunc(p, x):
a = np.zeros(7)
a[1:7] = p[:6]
a[0] = 1 - a[1:4].sum()
return a[0] + a[1]*exp(-x/a[4]) + a[2]*exp(-x/a[5]) + a[3]*exp(-x/a[6])
def errfunc(p, x, y1, y2):
return np.concatenate((
fitfunc(p[:6], x) - y1,
fitfunc(p[6:12], x) - y2
))
Generally, lambda-functions are considered bad style (and they don't add anything in your code). To have several functions in the least square fit you may just append the functions as I indicated using np.concatenate. It doesn't make much sense, if none of the parameters are correlated, though. It will only slow down convergence of the algorithm. The side condition you asked for, is implemented by just calculating one weight based on the constraint you gave (see 1 - a[1:4].sum()).
If you can't solve the equations for you constraints, and you can live with the constraint being satisfied with some tolerance, another possibility is to add a term to the chi-square with the large weight which guarantees that the constraint is almost satisfied.
E.g if you need that \sum(sin(p[i])==1 ,you can do the following:
constraint_func = lambda a: sin(a).sum()-1
def fitfunc (a,x):
np.concatenate((a[0]+a[1]*exp(-x/a[4])+a[2]*exp(-x/a[5])+a[3]*exp(-x /a[6]),
[constraint_func(a)]))
def errfunc(a,x,y):
tolerance = 1e-10
return np.concatenate((fitfunc(a,x) - y, [tolerance]))
Obviously the convergence will be slower, but will be still guaranteed.