Related
Overall, I want to calculate a fourier transform of a given data set and filter out some of the frequencies with the biggest absolute values. So:
1) Given a data array D with accompanying times t, 2) find the k biggest fourier coefficients and 3) remove those coefficients from the data, in order to filter out certain signals from the original data.
Something goes wrong in the end when plotting the filtered data set over the given times. I'm not exactly sure, where the error is. The final 'filtered data' plot doesn't look even slightly smoothened and somehow changes its position compared with the original data. Is my code completely bad?
Part 1):
n = 1000
limit_low = 0
limit_high = 0.48
D = np.random.normal(0, 0.5, n) + np.abs(np.random.normal(0, 2, n) * np.sin(np.linspace(0, 3*np.pi, n))) + np.sin(np.linspace(0, 5*np.pi, n))**2 + np.sin(np.linspace(1, 6*np.pi, n))**2
scaling = (limit_high - limit_low) / (max(D) - min(D))
D = D * scaling
D = D + (limit_low - min(D)) # given data
t = linspace(0,D.size-1,D.size) # times
Part 2):
from numpy import linspace
import numpy as np
from scipy import fft, ifft
D_f = fft.fft(D) # fft of D-dataset
#---extract the k biggest coefficients out of D_f ---
k = 15
I, bigvals = [], []
for i in np.argsort(-D_f):
if D_f[i] not in bigvals:
bigvals.append(D_f[i])
I.append(i)
if len(I) == k:
break
bigcofs = np.zeros(len(D_f))
bigcofs[I] = D_f[I] # array with only zeros in in except for the k maximal coefficients
Part 3):
D_filter = fft.ifft(bigcofs)
D_new = D - D_filter
p1=plt.plot(t,D,'r')
p2=plt.plot(t,D_new,'b');
plt.legend((p1[0], p2[0]), ('original data', 'filtered data'))
I appreciate your help, thanks in advance.
There were two issues I noticed:
you likely want the components with largest absolute value, so np.argsort(-np.abs(D_f)) instead of np.argsort(-D_f).
More subtly: bigcofs = np.zeros(len(D_f)) is of type float64 and was discarding the imaginary part at the line bigcofs[I] = D_f[I]. You can fix this with bigcofs = np.zeros(len(D_f), dtype=complex)
I've improved your code slightly below to get desired results:
import numpy as np
from scipy import fft, ifft
import matplotlib.pyplot as plt
n = 1000
limit_low = 0
limit_high = 0.48
N_THRESH = 10
D = 0.5*np.random.normal(0, 0.5, n) + 0.5*np.abs(np.random.normal(0, 2, n) * np.sin(np.linspace(0, 3*np.pi, n))) + np.sin(np.linspace(0, 5*np.pi, n))**2 + np.sin(np.linspace(1, 6*np.pi, n))**2
scaling = (limit_high - limit_low) / (max(D) - min(D))
D = D * scaling
D = D + (limit_low - min(D)) # given data
t = np.linspace(0,D.size-1,D.size) # times
# transformed data
D_fft = fft.fft(D)
# Create boolean mask for N largest indices
idx_sorted = np.argsort(-np.abs(D_fft))
idx = idx_sorted[0:N_THRESH]
mask = np.zeros(D_fft.shape).astype(bool)
mask[idx] = True
# Split fft above, below N_THRESH points:
D_below = D_fft.copy()
D_below[mask] = 0
D_above = D_fft.copy()
D_above[~mask] = 0
#inverse separated functions
D_above = fft.ifft(D_above)
D_below = fft.ifft(D_below)
# plot
plt.ion()
f, (ax1, ax2, ax3) = plt.subplots(3,1)
l1, = ax1.plot(t, D, c="r", label="original")
l2, = ax2.plot(t, D_above, c="g", label="top {} coeff. signal".format(N_THRESH))
l3, = ax3.plot(t, D_below, c="b", label="remaining signal")
f.legend(handles=[l1,l2,l3])
plt.show()
It is the first time I am trying to write a Poincare section code at Python.
I borrowed the piece of code from here:
https://github.com/williamgilpin/rk4/blob/master/rk4_demo.py
and I have tried to run it for my system of second order coupled odes. The problem is that I do not see what I was expecting to. Actually, I need the Poincare section when x=0 and px>0.
I believe that my implementation is not the best out there. I would like to:
Improve the way that the initial conditions are chosen.
Apply the correct conditions (x=0 and px>0) in order to acquire the correct Poincare section.
Create one plot with all the collected poincare section data, not four separate ones.
I would appreciate any help.
This is the code:
from matplotlib.pyplot import *
from scipy import *
from numpy import *
# a simple Runge-Kutta integrator for multiple dependent variables and one independent variable
def rungekutta4(yprime, time, y0):
# yprime is a list of functions, y0 is a list of initial values of y
# time is a list of t-values at which solutions are computed
#
# Dependency: numpy
N = len(time)
y = array([thing*ones(N) for thing in y0]).T
for ii in xrange(N-1):
dt = time[ii+1] - time[ii]
k1 = dt*yprime(y[ii], time[ii])
k2 = dt*yprime(y[ii] + 0.5*k1, time[ii] + 0.5*dt)
k3 = dt*yprime(y[ii] + 0.5*k2, time[ii] + 0.5*dt)
k4 = dt*yprime(y[ii] + k3, time[ii+1])
y[ii+1] = y[ii] + (k1 + 2.0*(k2 + k3) + k4)/6.0
return y
# Miscellaneous functions
n= 1.0/3.0
kappa1 = 0.1
kappa2 = 0.1
kappa3 = 0.1
def total_energy(valpair):
(x, y, px, py) = tuple(valpair)
return .5*(px**2 + py**2) + (1.0/(1.0*(n+1)))*(kappa1*np.absolute(x)**(n+1)+kappa2*np.absolute(y-x)**(n+1)+kappa3*np.absolute(y)**(n+1))
def pqdot(valpair, tval):
# input: [x, y, px, py], t
# takes a pair of x and y values and returns \dot{p} according to the Hamiltonian
(x, y, px, py) = tuple(valpair)
return np.array([px, py, -kappa1*np.sign(x)*np.absolute(x)**n+kappa2*np.sign(y-x)*np.absolute(y-x)**n, kappa2*np.sign(y-x)*np.absolute(y-x)**n-kappa3*np.sign(y)*np.absolute(y)**n]).T
def findcrossings(data, data1):
# returns indices in 1D data set where the data crossed zero. Useful for generating Poincare map at 0
prb = list()
for ii in xrange(len(data)-1):
if (((data[ii] > 0) and (data[ii+1] < 0)) or ((data[ii] < 0) and (data[ii+1] > 0))) and data1[ii] > 0:
prb.append(ii)
return array(prb)
t = linspace(0, 1000.0, 100000)
print ("step size is " + str(t[1]-t[0]))
# Representative initial conditions for E=1
E = 1
x0=0
y0=0
init_cons = [[x0, y0, np.sqrt(2*E-(1.0*i/10.0)*(1.0*i/10.0)-2.0/(n+1)*(kappa1*np.absolute(x0)**(n+1)+kappa2*np.absolute(y0-x0)**(n+1)+kappa3*np.absolute(y0)**(n+1))), 1.0*i/10.0] for i in range(-10,11)]
outs = list()
for con in init_cons:
outs.append( rungekutta4(pqdot, t, con) )
# plot the results
fig1 = figure(1)
for ii in xrange(4):
subplot(2, 2, ii+1)
plot(outs[ii][:,1],outs[ii][:,3])
ylabel("py")
xlabel("y")
title("Full trajectory projected onto the plane")
fig1.suptitle('Full trajectories E = 1', fontsize=10)
# Plot Poincare sections at x=0 and px>0
fig2 = figure(2)
for ii in xrange(4):
subplot(2, 2, ii+1)
xcrossings = findcrossings(outs[ii][:,0], outs[ii][:,3])
yints = [.5*(outs[ii][cross, 1] + outs[ii][cross+1, 1]) for cross in xcrossings]
pyints = [.5*(outs[ii][cross, 3] + outs[ii][cross+1, 3]) for cross in xcrossings]
plot(yints, pyints,'.')
ylabel("py")
xlabel("y")
title("Poincare section x = 0")
fig2.suptitle('Poincare Sections E = 1', fontsize=10)
show()
You need to compute the derivatives of the Hamiltonian correctly. The derivative of |y-x|^n for x is
n*(x-y)*|x-y|^(n-2)=n*sign(x-y)*|x-y|^(n-1)
and the derivative for y is almost, but not exactly (as in your code), the same,
n*(y-x)*|x-y|^(n-2)=n*sign(y-x)*|x-y|^(n-1),
note the sign difference. With this correction you can take larger time steps, with correct linear interpolation probably even larger ones, to obtain the images
I changed the integration of the ODE to
t = linspace(0, 1000.0, 2000+1)
...
E_kin = E-total_energy([x0,y0,0,0])
init_cons = [[x0, y0, (2*E_kin-py**2)**0.5, py] for py in np.linspace(-10,10,8)]
outs = [ odeint(pqdot, con, t, atol=1e-9, rtol=1e-8) ) for con in init_cons[:8] ]
Obviously the number and parametrization of initial conditions may change.
The computation and display of the zero-crossings was changed to
def refine_crossing(a,b):
tf = -a[0]/a[2]
while abs(b[0])>1e-6:
b = odeint(pqdot, a, [0,tf], atol=1e-8, rtol=1e-6)[-1];
# Newton step using that b[0]=x(tf) and b[2]=x'(tf)
tf -= b[0]/b[2]
return [ b[1], b[3] ]
# Plot Poincare sections at x=0 and px>0
fig2 = figure(2)
for ii in xrange(8):
#subplot(4, 2, ii+1)
xcrossings = findcrossings(outs[ii][:,0], outs[ii][:,3])
ycrossings = [ refine_crossing(outs[ii][cross], outs[ii][cross+1]) for cross in xcrossings]
yints, pyints = array(ycrossings).T
plot(yints, pyints,'.')
ylabel("py")
xlabel("y")
title("Poincare section x = 0")
and evaluating the result of a longer integration interval
I have 4-dimensional data, say for the temperature, in an numpy.ndarray.
The shape of the array is (ntime, nheight_in, nlat, nlon).
I have corresponding 1D arrays for each of the dimensions that tell me which time, height, latitude, and longitude a certain value corresponds to, for this example I need height_in giving the height in metres.
Now I need to bring it onto a different height dimension, height_out, with a different length.
The following seems to do what I want:
ntime, nheight_in, nlat, nlon = t_in.shape
nheight_out = len(height_out)
t_out = np.empty((ntime, nheight_out, nlat, nlon))
for time in range(ntime):
for lat in range(nlat):
for lon in range(nlon):
t_out[time, :, lat, lon] = np.interp(
height_out, height_in, t[time, :, lat, lon]
)
But with 3 nested loops, and lots of switching between python and numpy, I don't think this is the best way to do it.
Any suggestions on how to improve this? Thanks
scipy's interp1d can help:
import numpy as np
from scipy.interpolate import interp1d
ntime, nheight_in, nlat, nlon = (10, 20, 30, 40)
heights = np.linspace(0, 1, nheight_in)
t_in = np.random.normal(size=(ntime, nheight_in, nlat, nlon))
f_out = interp1d(heights, t_in, axis=1)
nheight_out = 50
new_heights = np.linspace(0, 1, nheight_out)
t_out = f_out(new_heights)
I was looking for a similar function that works with irregularly spaced coordinates, and ended up writing my own function. As far as I see, the interpolation is handled nicely and the performance in terms of memory and speed is also quite good. I thought I'd share it here in case anyone else comes across this question looking for a similar function:
import numpy as np
import warnings
def interp_along_axis(y, x, newx, axis, inverse=False, method='linear'):
""" Interpolate vertical profiles, e.g. of atmospheric variables
using vectorized numpy operations
This function assumes that the x-xoordinate increases monotonically
ps:
* Updated to work with irregularly spaced x-coordinate.
* Updated to work with irregularly spaced newx-coordinate
* Updated to easily inverse the direction of the x-coordinate
* Updated to fill with nans outside extrapolation range
* Updated to include a linear interpolation method as well
(it was initially written for a cubic function)
Peter Kalverla
March 2018
--------------------
More info:
Algorithm from: http://www.paulinternet.nl/?page=bicubic
It approximates y = f(x) = ax^3 + bx^2 + cx + d
where y may be an ndarray input vector
Returns f(newx)
The algorithm uses the derivative f'(x) = 3ax^2 + 2bx + c
and uses the fact that:
f(0) = d
f(1) = a + b + c + d
f'(0) = c
f'(1) = 3a + 2b + c
Rewriting this yields expressions for a, b, c, d:
a = 2f(0) - 2f(1) + f'(0) + f'(1)
b = -3f(0) + 3f(1) - 2f'(0) - f'(1)
c = f'(0)
d = f(0)
These can be evaluated at two neighbouring points in x and
as such constitute the piecewise cubic interpolator.
"""
# View of x and y with axis as first dimension
if inverse:
_x = np.moveaxis(x, axis, 0)[::-1, ...]
_y = np.moveaxis(y, axis, 0)[::-1, ...]
_newx = np.moveaxis(newx, axis, 0)[::-1, ...]
else:
_y = np.moveaxis(y, axis, 0)
_x = np.moveaxis(x, axis, 0)
_newx = np.moveaxis(newx, axis, 0)
# Sanity checks
if np.any(_newx[0] < _x[0]) or np.any(_newx[-1] > _x[-1]):
# raise ValueError('This function cannot extrapolate')
warnings.warn("Some values are outside the interpolation range. "
"These will be filled with NaN")
if np.any(np.diff(_x, axis=0) < 0):
raise ValueError('x should increase monotonically')
if np.any(np.diff(_newx, axis=0) < 0):
raise ValueError('newx should increase monotonically')
# Cubic interpolation needs the gradient of y in addition to its values
if method == 'cubic':
# For now, simply use a numpy function to get the derivatives
# This produces the largest memory overhead of the function and
# could alternatively be done in passing.
ydx = np.gradient(_y, axis=0, edge_order=2)
# This will later be concatenated with a dynamic '0th' index
ind = [i for i in np.indices(_y.shape[1:])]
# Allocate the output array
original_dims = _y.shape
newdims = list(original_dims)
newdims[0] = len(_newx)
newy = np.zeros(newdims)
# set initial bounds
i_lower = np.zeros(_x.shape[1:], dtype=int)
i_upper = np.ones(_x.shape[1:], dtype=int)
x_lower = _x[0, ...]
x_upper = _x[1, ...]
for i, xi in enumerate(_newx):
# Start at the 'bottom' of the array and work upwards
# This only works if x and newx increase monotonically
# Update bounds where necessary and possible
needs_update = (xi > x_upper) & (i_upper+1<len(_x))
# print x_upper.max(), np.any(needs_update)
while np.any(needs_update):
i_lower = np.where(needs_update, i_lower+1, i_lower)
i_upper = i_lower + 1
x_lower = _x[[i_lower]+ind]
x_upper = _x[[i_upper]+ind]
# Check again
needs_update = (xi > x_upper) & (i_upper+1<len(_x))
# Express the position of xi relative to its neighbours
xj = (xi-x_lower)/(x_upper - x_lower)
# Determine where there is a valid interpolation range
within_bounds = (_x[0, ...] < xi) & (xi < _x[-1, ...])
if method == 'linear':
f0, f1 = _y[[i_lower]+ind], _y[[i_upper]+ind]
a = f1 - f0
b = f0
newy[i, ...] = np.where(within_bounds, a*xj+b, np.nan)
elif method=='cubic':
f0, f1 = _y[[i_lower]+ind], _y[[i_upper]+ind]
df0, df1 = ydx[[i_lower]+ind], ydx[[i_upper]+ind]
a = 2*f0 - 2*f1 + df0 + df1
b = -3*f0 + 3*f1 - 2*df0 - df1
c = df0
d = f0
newy[i, ...] = np.where(within_bounds, a*xj**3 + b*xj**2 + c*xj + d, np.nan)
else:
raise ValueError("invalid interpolation method"
"(choose 'linear' or 'cubic')")
if inverse:
newy = newy[::-1, ...]
return np.moveaxis(newy, 0, axis)
And this is a small example to test it:
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import interp1d as scipy1d
# toy coordinates and data
nx, ny, nz = 25, 30, 10
x = np.arange(nx)
y = np.arange(ny)
z = np.tile(np.arange(nz), (nx,ny,1)) + np.random.randn(nx, ny, nz)*.1
testdata = np.random.randn(nx,ny,nz) # x,y,z
# Desired z-coordinates (must be between bounds of z)
znew = np.tile(np.linspace(2,nz-2,50), (nx,ny,1)) + np.random.randn(nx, ny, 50)*0.01
# Inverse the coordinates for testing
z = z[..., ::-1]
znew = znew[..., ::-1]
# Now use own routine
ynew = interp_along_axis(testdata, z, znew, axis=2, inverse=True)
# Check some random profiles
for i in range(5):
randx = np.random.randint(nx)
randy = np.random.randint(ny)
checkfunc = scipy1d(z[randx, randy], testdata[randx,randy], kind='cubic')
checkdata = checkfunc(znew)
fig, ax = plt.subplots()
ax.plot(testdata[randx, randy], z[randx, randy], 'x', label='original data')
ax.plot(checkdata[randx, randy], znew[randx, randy], label='scipy')
ax.plot(ynew[randx, randy], znew[randx, randy], '--', label='Peter')
ax.legend()
plt.show()
Following the criteria of numpy.interp, one can assign the left/right bounds to the points outside the range adding this lines after within_bounds = ...
out_lbound = (xi <= _x[0,...])
out_rbound = (_x[-1,...] <= xi)
and
newy[i, out_lbound] = _y[0, out_lbound]
newy[i, out_rbound] = _y[-1, out_rbound]
after newy[i, ...] = ....
If I understood well the strategy used by #Peter9192, I think the changes are in the same line. I've checked a little bit, but maybe some strange case could not work properly.
Let 0 <= x <= 1. I have two columns f and g of length 5000 respectively. Now I plot:
plt.plot(x, f, '-')
plt.plot(x, g, '*')
I want to find the point 'x' where the curve intersects. I don't want to find the intersection of f and g.
I can do it simply with:
set(f) & set(g)
You can use np.sign in combination with np.diff and np.argwhere to obtain the indices of points where the lines cross (in this case, the points are [ 0, 149, 331, 448, 664, 743]):
import numpy as np
import matplotlib.pyplot as plt
x = np.arange(0, 1000)
f = np.arange(0, 1000)
g = np.sin(np.arange(0, 10, 0.01) * 2) * 1000
plt.plot(x, f, '-')
plt.plot(x, g, '-')
idx = np.argwhere(np.diff(np.sign(f - g))).flatten()
plt.plot(x[idx], f[idx], 'ro')
plt.show()
First it calculates f - g and the corresponding signs using np.sign. Applying np.diff reveals all the positions, where the sign changes (e.g. the lines cross). Using np.argwhere gives us the exact indices.
For those who are using or open to use the Shapely library for geometry-related computations, getting the intersection will be much easier. You just have to construct LineString from each line and get their intersection as follows:
import numpy as np
import matplotlib.pyplot as plt
from shapely.geometry import LineString
x = np.arange(0, 1000)
f = np.arange(0, 1000)
g = np.sin(np.arange(0, 10, 0.01) * 2) * 1000
plt.plot(x, f)
plt.plot(x, g)
first_line = LineString(np.column_stack((x, f)))
second_line = LineString(np.column_stack((x, g)))
intersection = first_line.intersection(second_line)
if intersection.geom_type == 'MultiPoint':
plt.plot(*LineString(intersection).xy, 'o')
elif intersection.geom_type == 'Point':
plt.plot(*intersection.xy, 'o')
And to get the x and y values as NumPy arrays you would just write:
x, y = LineString(intersection).xy
# x: array('d', [0.0, 149.5724669847373, 331.02906176584617, 448.01182730277833, 664.6733061190541, 743.4822641140581])
# y: array('d', [0.0, 149.5724669847373, 331.02906176584617, 448.01182730277833, 664.6733061190541, 743.4822641140581])
or if an intersection is only one point:
x, y = intersection.xy
Here's a solution which:
Works with N-dimensional data
Uses Euclidean distance rather than merely finding cross-overs in the y-axis
Is more efficient with lots of data (it queries a KD-tree, which should query in logarathmic time instead of linear time).
You can change the distance_upper_bound in the KD-tree query to define how close is close enough.
You can query the KD-tree with many points at the same time, if needed. Note: if you need to query thousands of points at once, you can get dramatic performance increases by querying the KD-tree with another KD-tree.
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy.spatial import cKDTree
from scipy import interpolate
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1], projection='3d')
ax.axis('off')
def upsample_coords(coord_list):
# s is smoothness, set to zero
# k is degree of the spline. setting to 1 for linear spline
tck, u = interpolate.splprep(coord_list, k=1, s=0.0)
upsampled_coords = interpolate.splev(np.linspace(0, 1, 100), tck)
return upsampled_coords
# target line
x_targ = [1, 2, 3, 4, 5, 6, 7, 8]
y_targ = [20, 100, 50, 120, 55, 240, 50, 25]
z_targ = [20, 100, 50, 120, 55, 240, 50, 25]
targ_upsampled = upsample_coords([x_targ, y_targ, z_targ])
targ_coords = np.column_stack(targ_upsampled)
# KD-tree for nearest neighbor search
targ_kdtree = cKDTree(targ_coords)
# line two
x2 = [3,4,5,6,7,8,9]
y2 = [25,35,14,67,88,44,120]
z2 = [25,35,14,67,88,44,120]
l2_upsampled = upsample_coords([x2, y2, z2])
l2_coords = np.column_stack(l2_upsampled)
# plot both lines
ax.plot(x_targ, y_targ, z_targ, color='black', linewidth=0.5)
ax.plot(x2, y2, z2, color='darkgreen', linewidth=0.5)
# find intersections
for i in range(len(l2_coords)):
if i == 0: # skip first, there is no previous point
continue
distance, close_index = targ_kdtree.query(l2_coords[i], distance_upper_bound=.5)
# strangely, points infinitely far away are somehow within the upper bound
if np.isinf(distance):
continue
# plot ground truth that was activated
_x, _y, _z = targ_kdtree.data[close_index]
ax.scatter(_x, _y, _z, 'gx')
_x2, _y2, _z2 = l2_coords[i]
ax.scatter(_x2, _y2, _z2, 'rx') # Plot the cross point
plt.show()
Well, I was looking for a matplotlib for two curves which were different in size and had not the same x values. Here is what I come up with:
import numpy as np
import matplotlib.pyplot as plt
import sys
fig = plt.figure()
ax = fig.add_subplot(111)
# x1 = [1,2,3,4,5,6,7,8]
# y1 = [20,100,50,120,55,240,50,25]
# x2 = [3,4,5,6,7,8,9]
# y2 = [25,200,14,67,88,44,120]
x1=[1.4,2.1,3,5.9,8,9,12,15]
y1=[2.3,3.1,1,3.9,8,9,11,9]
x2=[1,2,3,4,6,8,9,12,14]
y2=[4,12,7,1,6.3,7,5,6,11]
ax.plot(x1, y1, color='lightblue',linewidth=3, marker='s')
ax.plot(x2, y2, color='darkgreen', marker='^')
y_lists = y1[:]
y_lists.extend(y2)
y_dist = max(y_lists)/200.0
x_lists = x1[:]
x_lists.extend(x2)
x_dist = max(x_lists)/900.0
division = 1000
x_begin = min(x1[0], x2[0]) # 3
x_end = max(x1[-1], x2[-1]) # 8
points1 = [t for t in zip(x1, y1) if x_begin<=t[0]<=x_end] # [(3, 50), (4, 120), (5, 55), (6, 240), (7, 50), (8, 25)]
points2 = [t for t in zip(x2, y2) if x_begin<=t[0]<=x_end] # [(3, 25), (4, 35), (5, 14), (6, 67), (7, 88), (8, 44)]
# print points1
# print points2
x_axis = np.linspace(x_begin, x_end, division)
idx = 0
id_px1 = 0
id_px2 = 0
x1_line = []
y1_line = []
x2_line = []
y2_line = []
xpoints = len(x_axis)
intersection = []
while idx < xpoints:
# Iterate over two line segments
x = x_axis[idx]
if id_px1>-1:
if x >= points1[id_px1][0] and id_px1<len(points1)-1:
y1_line = np.linspace(points1[id_px1][1], points1[id_px1+1][1], 1000) # 1.4 1.401 1.402 etc. bis 2.1
x1_line = np.linspace(points1[id_px1][0], points1[id_px1+1][0], 1000)
id_px1 = id_px1 + 1
if id_px1 == len(points1):
x1_line = []
y1_line = []
id_px1 = -1
if id_px2>-1:
if x >= points2[id_px2][0] and id_px2<len(points2)-1:
y2_line = np.linspace(points2[id_px2][1], points2[id_px2+1][1], 1000)
x2_line = np.linspace(points2[id_px2][0], points2[id_px2+1][0], 1000)
id_px2 = id_px2 + 1
if id_px2 == len(points2):
x2_line = []
y2_line = []
id_px2 = -1
if x1_line!=[] and y1_line!=[] and x2_line!=[] and y2_line!=[]:
i = 0
while abs(x-x1_line[i])>x_dist and i < len(x1_line)-1:
i = i + 1
y1_current = y1_line[i]
j = 0
while abs(x-x2_line[j])>x_dist and j < len(x2_line)-1:
j = j + 1
y2_current = y2_line[j]
if abs(y2_current-y1_current)<y_dist and i != len(x1_line) and j != len(x2_line):
ymax = max(y1_current, y2_current)
ymin = min(y1_current, y2_current)
xmax = max(x1_line[i], x2_line[j])
xmin = min(x1_line[i], x2_line[j])
intersection.append((x, ymin+(ymax-ymin)/2))
ax.plot(x, y1_current, 'ro') # Plot the cross point
idx += 1
print "intersection points", intersection
plt.show()
Intersection probably occurs between points. Let's explore the example bellow.
import numpy as np
import matplotlib.pyplot as plt
xs=np.arange(0, 20)
y1=np.arange(0, 20)*2
y2=np.array([1, 1.5, 3, 8, 9, 20, 23, 21, 13, 23, 18, 20, 23, 24, 31, 28, 30, 33, 37, 36])
plotting the 2 curves above, along with their intersections, using as intersection the average coordinates before and after proposed from idx intersection, all points are closer to the first curve.
idx=np.argwhere(np.diff(np.sign(y1 - y2 )) != 0).reshape(-1) + 0
plt.plot(xs, y1)
plt.plot(xs, y2)
for i in range(len(idx)):
plt.plot((xs[idx[i]]+xs[idx[i]+1])/2.,(y1[idx[i]]+y1[idx[i]+1])/2., 'ro')
plt.legend(['Y1', 'Y2'])
plt.show()
using as intersection the average coordinates before and after but for both y1 and y2 curves usually are closer to true intersection
plt.plot(xs, y1)
plt.plot(xs, y2)
for i in range(len(idx)):
plt.plot((xs[idx[i]]+xs[idx[i]+1])/2.,(y1[idx[i]]+y1[idx[i]+1]+y2[idx[i]]+y2[idx[i]+1])/4., 'ro')
plt.legend(['Y1', 'Y2'])
plt.show()
For an even more accurate intersection estimation we could use interpolation.
For arrays f and g, we could simply do the following:
np.pad(np.diff(np.array(f > g).astype(int)), (1,0), 'constant', constant_values = (0,))
This will give the array of all the crossover points. Every 1 is a crossover from below to above and every -1 a crossover from above to below.
Even if f and g intersect, you cannot be sure that f[i]== g[i] for integer i (the intersection probably occurs between points).
You should instead test like
# detect intersection by change in sign of difference
d = f - g
for i in range(len(d) - 1):
if d[i] == 0. or d[i] * d[i + 1] < 0.:
# crossover at i
x_ = x[i]
I had a similar problem, but with one discontinue function, like the tangent function. To avoid get points on the discontinuity, witch i didn't want to consider a intersection, i added a tolerance parameter on the previous solutions that use np.diff and np.sign. I set the tolerance parameter as the mean of the differences between the two data points, witch suffices in my case.
import numpy as np
import matplotlib.pyplot as plt
fig,ax = plt.subplots(nrows = 1,ncols = 2)
x = np.arange(0, 1000)
f = 2*np.arange(0, 1000)
g = np.tan(np.arange(0, 10, 0.01) * 2) * 1000
#here we set a threshold to decide if we will consider that point as a intersection
tolerance = np.abs(np.diff(f-g)).mean()
idx = np.argwhere((np.diff(np.sign(f - g)) != 0) & (np.abs(np.diff(f-g)) <= tolerance)).flatten()
#general case (tolerance = infinity)
tolerance = np.inf
idx2 = np.argwhere((np.diff(np.sign(f - g)) != 0) & (np.abs(np.diff(f-g)) <= tolerance)).flatten()
ax1,ax2 = ax
ax1.plot(x,f); ax1.plot(x,g)
ax2.plot(x,f); ax2.plot(x,g)
ax1.plot(x[idx], f[idx], 'o'); ax1.set_ylim(-3000,3000)
ax2.plot(x[idx2],f[idx2], 'o'); ax2.set_ylim(-3000,3000)
plt.show()
As a documented and tested function (credit for the algorithm goes to #Matt, I only changed the example to something simpler and used linspace instead of arange to handle non-integers better):
from typing import Iterable, Tuple
import numpy as np
import doctest
def intersect(x: np.array, f: np.array, g: np.array) -> Iterable[Tuple[(int, int)]]:
"""
Finds the intersection points between `f` and `g` on the domain `x`.
Given:
- `x`: The discretized domain.
- `f`: The discretized values of the first function calculated on the
discretized domain.
- `g`: The discretized values of the second function calculated on the
discretized domain.
Returns:
An iterable containing the (x,y) points of intersection.
Test case, line-parabola intersection:
>>> x = np.linspace(0, 10, num=10000)
>>> f = 3 * x
>>> g = np.square(x)
>>> list(intersect(x, f, g))
[(0.0, 0.0), (2.999299929992999, 8.997899789978998)]
"""
idx = np.argwhere(np.diff(np.sign(f - g))).flatten()
return zip(x[idx], f[idx])
if __name__ == "__main__":
doctest.testmod()
In Python 2, just remove the type hints.
There may be multiple intersections, you can find the (x,y) point at every intersection by the following list comprehension
intersections = [(x[i], f[i]) for i,_ in enumerate(zip(f,g)) if f[i] == g[i]]
As a simple example
>>> x = [1,2,3,4,5]
>>> f = [2,4,6,8,10]
>>> g = [10,8,6,4,2]
>>> [(x[i], f[i]) for i,_ in enumerate(zip(f,g)) if f[i] == g[i]]
[(3, 6)]
So this found one intersection point at x = 3, y = 6. Note that if you are using float the two values may not be exactly equal, so you could use some tolerance instead of ==.
I'm trying to solve an nonlinear optimal control problem subject to dynamic ( h(x, x', u) = 0 ) constraint.
given:
f(x) = (u(t) - u(0)(t))^2 # u0(t) is the initial input provided to the system
h(x) = y'(t) - integral(sqrt(u(t))*y(t) + y(t)) = 0 # a nonlinear differential equation
-2 < y(t) < 10 # system state is bounded to this range
-2 < u(t) < 10 # system state is bounded to this range
u0(t) # will be defined as an arbitrary piecewise-linear function
I've tried to translate the problem into python code using openopt and scipy:
import numpy as np
from scipy.integrate import *
from openopt import NLP
import matplotlib.pyplot as plt
from operator import and_
N = 15*4
y0 = 10
t0 = 0
tf = 10
lb, ub = np.ones(2)*-2, np.ones(2)*10
t = np.linspace(t0, tf, N)
u0 = np.piecewise(t, [t < 3, and_(3 <= t, t < 6), 6 <= t], [2, lambda t: t - 3, lambda t: -t + 9])
p = np.empty(N, dtype=np.object)
r = np.empty(N, dtype=np.object)
y = np.empty(N, dtype=np.object)
u = np.empty(N, dtype=np.object)
ff = np.empty(N, dtype=np.object)
for i in range(N):
t = np.linspace(t0, tf, N)
b, a = t[i], t[i - 1]
integrand = lambda t, u1, y1 : np.sqrt(u1)*y1 + y1
integral = lambda u1, y1 : fixed_quad(integrand, a, b, args=(u1, y1))[0]
f = lambda x1: ((x1[1] - u0[i])**2).sum()
h = lambda x1: x1[0] - y0 - integral(x1[0], x1[1])
p[i] = NLP(f, (y0, u0[i]), h=h, lb=lb, ub=ub)
r[i] = p[i].solve('scipy_slsqp')
y0 = r[i].xf[0]
y[i] = r[i].xf[0]
u[i] = r[i].xf[1]
ff[i] = r[i].ff
figure1 = plt.figure()
axis1 = figure1.add_subplot(311)
plt.plot(u0)
axis2 = figure1.add_subplot(312)
plt.plot(u)
axis2 = figure1.add_subplot(313)
plt.plot(y)
plt.show()
Now the problem is, running the code with a positive initial y0 like y0 = 10 , the code will result satisfying results.
But giving y0 = 0 or a negative one y0 = -1, nlp problem will be deficient, saying:
"NO FEASIBLE SOLUTION has been obtained (1 constraint is equal to NaN, MaxResidual = 0, objFunc = nan)"
Also, considering the piecewise-linear initial u0, if you put any number other than 0 at the first range of the function at t < 3, meaning:
u0 = np.piecewise(t, [t < 3, and_(3 <= t, t < 6), 6 <= t], [2, lambda t: t - 3, lambda t: -t + 9])
instead of:
u0 = np.piecewise(t, [t < 3, and_(3 <= t, t < 6), 6 <= t], [0, lambda t: t - 3, lambda t: -t + 9])
This will result in the same error again.
Any ideas ?
Thanks in advance.
My first thought is that you seem to be solving a 2-dimensional Optimal Control problem as if it were a 1-Dimensional problem.
The constraint dynamics $h(x, x', t)$ are really a second order ODE.
y''(t) - sqrt(u(t))*y(t) + y(t)) = 0
Starting from this I would reword my system as a 2-dimensional, 1st order system in the standard way.
My second thought is that you seem to be optimizing independently, for $u(t)$, at each time step, whereas the problem is to optimize globally for $u(.)$, the entire function. So if anything, the call to NLP should be outside the for loop...
There are dedicated Optimal Control Open Source toolboxes:
Pythonically, there is JModellica: http://www.jmodelica.org/.
Alternatively, I have also successfully used: ACADO, http://sourceforge.net/p/acado/wiki/Home/ (in C++)
There are some very capable modeling tools now such as CasADi, Pyomo, and Gekko that didn't exist when the question was asked. Here is a solution with Gekko. One issue is that sqrt(u) needs to have a positive u value to avoid imaginary numbers.
from gekko import GEKKO
import numpy as np
m = GEKKO(remote=False)
u0=1; N=61
m.time = np.linspace(0,10,N)
# lb of -2 leads to imaginary number with sqrt(-)
u = m.MV(u0,lb=1e-2,ub=10); u.STATUS=1
m.options.MV_STEP_HOR = 18 # allow move at 3, 6 time units
m.options.MV_TYPE = 1 # piecewise linear
y = m.Var(10,lb=-2,ub=10)
m.Minimize((u-u0)**2)
m.Minimize(y**2) # otherwise solution is u=u0
m.Equation(y.dt()==m.integral(m.sqrt(u)*y-y))
m.options.IMODE=6
m.options.SOLVER=1
m.solve()
import matplotlib.pyplot as plt
plt.figure(figsize=(7,4))
plt.plot(m.time,u,label='u')
plt.plot(m.time,y,label='y')
plt.legend(); plt.grid()
plt.savefig('solution.png',dpi=300)
plt.show()