Develop Reference

Optimize a set of coordinates based on a cost function

I have a set of (x_i,y_i) coordinates defining a path between an initial point and a final position, that I do not want to move. Right now I am optimizing the path based on distance between points (to reduce the length of the path) and curvature of the points which are included into the cost function and jacobian. The problem is that to accomplish that I have to create a state vector wit all the stacked and find the minimum. The state space gets length 2*N where N is the number of samples that I have and it runs extremely slow even for small number of points. The layout is the following one (just including distance between points):
def jacobian_function(self, state_vector):
"""
The jacobian of the cost function to be used by the optimizer
:param state_vector : The extended state vector
"""
gradient_array = np.zeros(len(state_vector))
for i in range(1, int(len(state_vector) / self.num_states) - 1): # -1 to not modify last point
state_before = state_vector[2 * (i - 1):2 * i]
state_curr = state_vector[2 * i:2 * (i + 1)]
state_next = state_vector[2 * (i + 1):2 * (i + 2)]
factor = 0.1 * 2 * (
state_before - 2 * state_curr + state_next)
gradient_array[2 * i:2 * (i + 1)] += -2 * factor
gradient_array[2 * (i - 1):2 * i] += factor
gradient_array[2 * (i + 1):2 * (i + 2)] += factor
return gradient_array
def cost_function(self, state_vector):
"""
The cost function to be used by the optimizer
:param state_vector : The extended state vector
"""
cost = 0
for i in range(1, int(len(state_vector) / 2) - 1): # -1 to not modify last point
smooth_vector = state_vector[2 * (i + 1):2 * (i + 2)] + state_vector[
2 * (i - 1):2 * i] - 2 * state_vector[
2 * i:2 * (i + 1)]
cost += 0.1 * np.dot(smooth_vector, smooth_vector)
return cost
states = np.array([[1,2,3,4,5],[1,2,3,4,5]])
states_extended = np.reshape(states, states.shape[0] * states.shape[1], order='F')
result = optimize.minimize(fun=self.cost_function, states_extended, method="BFGS",jac=self.jacobian_function, options={'disp': True})
Do you have any idea to speed up the process while considering all the data points (normally N is around 800)?

Filling points in a grid - Forward Euler algorithm - wrong output

I will very briefly try to explain what I'm doing to those who are less experienced with mathematics, it's really quite simple.
We are trying to fill a grid, as follows:
We find the orange point, U(j,n+1), using three points in a row below it, U(j-1,n), U(j,n), U(j,n+1)
Where the value of U in the entire bottom row is given, and is periodic. So theoretically we can fill this entire grid.
The formula for calculating the orange point is:
U(j,n+1) = U(j,n) + (delta_t / (2 * delta_x)) * (U(j+1,n) - U(j-1,n))
We can write it easily as a system of linear equations as follows:
And now we just repeat this process of multiplying by this matrix (iterating through the time variable) as much as we want. That's a simple way to numerically approximate a solution to a partial differential equation.
I wrote a code that does this, and then I compare my final row, to the known solution of the differential equation.
This is the code
import math
import numpy
def f(x):
return math.cos(2 * math.pi * x)
def solution(x, t):
return math.cos(2 * math.pi * (x + t))
# setting everything up
N = 16
Lambda = 10 ** (-20)
Delta_x = 1/(N+1)
Delta_t = Lambda * Delta_x * Delta_x
t_f = 5
v_0 = numpy.zeros((N, 1))
# Filling first row, initial condition was given
for i in range(N):
v_0[i, 0] = f(i * Delta_x)
# Create coefficient matrix
M = numpy.zeros((N, N))
for i in range(N):
M[i, i - 1] = -Delta_t / (2 * Delta_x)
M[i, i] = 1
M[i, (i + 1) % N] = Delta_t / (2 * Delta_x)
# start iterating through time
v_i = v_0
for i in range(math.floor(t_f / Delta_t) - 1):
v_i = numpy.dot(M, v_i)
v_final = v_i
if (Delta_t * math.ceil(t_f / Delta_t) != t_f): #we don't reach t_f exactly using Delta_t
v_final = (1/2) * (v_i + numpy.dot(M, v_i))
u = numpy.zeros(v_final.shape)
for i in range(N):
u[i, 0] = solution(i * Delta_x, t_f)
for x in range(v_final.shape[0]):
print (v_final[x], u[x])
theoretically speaking, I should be able to find lambda small enough such that v_final and the known solution, u, will be very similar.
But I can't. No matter how small I make lambda, how finde I make the grid, I seem to converge to something incorrect. They aren't close.
I can't for the life of me figure out the problem.
Does anyone have an idea what might be wrong?

You should have Delta_x = 1.0/N, as you divide the interval into N cells.
You get N+1 points on the grid from u[0] to u[N], but as per boundary condition u[N]=u[0], there you also only use an array of length N to hold all the node values.
Per your given formulas you have gamma = dt/(2*dx), thus the reverse computation should be dt = gamma*2*dx or in your variable names
Delta_t = Lambda * 2 * Delta_x
Or you are aiming at the error of the method which is O(dt, dx²) so that it would make sense to have dt = c*dx^2, but not with a ridiculous factor like of c=1e-20, if you want the time discretization error small against the space discretization error, c=0.1 or c=0.01 should be sufficient.
import numpy as np
def f(x):
return np.cos(2 * np.pi * x)
def solution(x, t):
return f(x + t)
# setting everything up
N_x = 16
Lambda = 1e-2
Delta_x = 1./N_x
Delta_t = Lambda * Delta_x * Delta_x
t_f = 5
N_t = int(t_f/Delta_t+0.5); t_f = N_t*Delta_t
# Filling first row, initial condition was given
x = np.arange(0,N_x,1) * Delta_x
v_0 = f(x)
# Create coefficient matrix
M = np.zeros((N_x, N_x))
for i in range(N_x):
M[i, i - 1] = -Delta_t / (2 * Delta_x)
M[i, i] = 1
M[i, (i + 1) % N_x] = Delta_t / (2 * Delta_x)
# start iterating through time
v_i = v_0[:]
for i in range(N_t):
v_i = np.dot(M, v_i)
v_final = v_i
u = solution(x, t_f)
for vx, ux in zip(v_final, u):
print (vx, ux)
The Euler method is also not the most precise method, the expected error is in the range exp(L*t_f)*dx^2 = e^5/N_x^2=0.58 for N_x=16 where L=1 was taken as approximate Lipschitz constant. Now if you increase to N_x=50 this error estimate reduces to 0.06 which is also visible in the results.
The t exact solution of the x discretized problem is cos(2*pi*(x+c*t)) where c=sin(2*pi*dx)/(2*pi*dx). If you compare against that formula, the errors should be really small of size O(dt).

Generalized Distance Transform in Python

I'm currently trying to implement the GDT described by Felzenszwalb and Huttenlocher (http://www.cs.cornell.edu/~dph/papers/dt.pdf) inside of Python for an image processing algorithm. However I used the algorithm described in the paper they published a few years back but got faulty results. I found a C# implementation here: https://dsp.stackexchange.com/questions/227/fastest-available-algorithm-for-distance-transform/29727?noredirect=1#comment55866_29727
And converted it to Python (which is pretty much the same I had before).
This is my code:
def of_column(dataInput):
output = zeros(dataInput.shape)
n = len(dataInput)
k = 0
v = zeros((n,))
z = zeros((n + 1,))
v[0] = 0
z[0] = -inf
z[1] = +inf
s = 0
for q in range(1, n):
while True:
s = (((dataInput[q] + q * q) - (dataInput[v[k]] + v[k] * v[k])) / (2.0 * q - 2.0 * v[k]))
if s <= z[k]:
k -= 1
else:
break
k += 1
v[k] = q
z[k] = s
z[k + 1] = +inf
k = 0
for q in range(n):
while z[k + 1] < q:
k += 1
output[q] = ((q - v[k]) * (q - v[k]) + dataInput[v[k]])
return output
I still can't find my error. When giving the algorithm a binary (boolean) numpy array it just returns the array itself not the Distance Transform. Why is this not working in Python?

I got it working after hours and hours. The answer given in the link above implementing the code in C# suggests putting up the "white" areas to a very large number. My dataInput array was a boolean array (0, 1). I replaced all 1s with 2^32 and it works just fine. The higher the number the more blurry it gets. The lower the more similar to the source it gets.

I would like to add the function for 2D that works with the 1D function described previously:
###############################################################################
# distance transform of 1d function using squared distance
###############################################################################
def dt_1d(dataInput, n):
output = np.zeros(dataInput.shape)
k = 0
v = np.zeros((n,))
z = np.zeros((n + 1,))
v[0] = 0
z[0] = -np.inf
z[1] = +np.inf
for q in range(1, n):
s = (((dataInput[q] + q * q) - (dataInput[v[k]] + v[k] * v[k])) / (2.0 * q - 2.0 * v[k]))
while s <= z[k]:
k -= 1
s = (((dataInput[q] + q * q) - (dataInput[v[k]] + v[k] * v[k])) / (2.0 * q - 2.0 * v[k]))
k += 1
v[k] = q
z[k] = s
z[k + 1] = +np.inf
k = 0
for q in range(n):
while z[k + 1] < q:
k += 1
value = ((q - v[k]) * (q - v[k]) + dataInput[v[k]])
if value > 255: value = 255
if value < 0: value = 0
output[q] = value
print output
return output
###############################################################################
# distance transform of 2d function using squared distance
###############################################################################
def dt_2d(dataInput):
height, width = dataInput.shape
f = np.zeros(max(height, width))
# transform along columns
for x in range(width):
f = dataInput[:,x]
dataInput[:,x] = dt_1d(f, height)
# transform along rows
for y in range(height):
f = dataInput[y,:]
dataInput[y,:] = dt_1d(f, width)
return dataInput
I hope it helps.

Hermite interpolation in Python

I have this program for calculating Hermite interpolation.
Problem is, that its behave really bad.
This is chart for 35 Chebyshev nodes. If I put more points, peak on the beginning will be higher(its about 10^7 with this amount of nodes).
I interpolated this same function using Lagrange method (green, its shifted so it can be seen) and as you can see it looks fine.
Here is the code:
def hermit_interpolate(input): #input is list of tuples [(x1,y1),(x2,y2)...] xi are Chebyshev nodes
points = [(input[0][0], input[0][1] - 0), (input[0][0], calculate_f_p_x(input[0][0]))] #"input[0][1] - 0" this is just to change type of second element
#calculate_f_p_x returns value of derivative
for k in range(1, len(input)): #Divided differences and derivatives in one list alternately
points.append((input[k][0], (input[k][1] - input[k - 1][1]) / (
input[k][0] - input[k - 1][0])))
points.append((input[k][0], calculate_f_p_x(input[k][0])))
x, c = zip(*points)
x = list(x)
c = list(c)
n = len(points)
for i in range(2, n): #calculating factors
for j in range(n - 1, i - 1, -1):
c[j] = (c[j] - c[j - 1]) / (x[j] - x[j - i])
def result_polynomial(xpoint): #here is function to calculate value for given x
val = c[0]
factor = 1.0
for l in range(1, n):
factor *= (xpoint - x[l - 1])
val += (c[l] * factor)
return val
return result_polynomial
I can't seen what's wrong here.
Thanks!

This code actually works:
def hermit_interpolate(input): #input is list of tuples [(x1,y1),(x2,y2),...,(xn,yn)] xi are Chebyshev nodes
n = len(input)
points = numpy.zeros(shape=(2 * n + 1, 2 * n + 1))
X, Y = zip(*input)
X = list(X)
Y = list(Y)
for i in range(0, 2 * n, 2):
points[i][0] = X[i / 2]
points[i + 1][0] = X[i / 2]
points[i][1] = Y[i / 2]
points[i + 1][1] = Y[i / 2]
for i in range(2, 2 * n + 1):
for j in range(1 + (i - 2), 2 * n):
if i == 2 and j % 2 == 1:
points[j][i] = calculate_f_p_x(X[j / 2]);
else:
points[j][i] = (points[j][i - 1] - points[j - 1][i - 1]) / (
points[j][0] - points[(j - 1) - (i - 2)][0])
def result_polynomial(xpoint): #here is function to calculate value for given x
val = 0
for i in range(0, 2 * n):
factor = 1.
j = 0
while j < i:
factor *= (xpoint - X[j / 2])
if j + 1 != i:
factor *= (xpoint - X[j / 2])
j += 1
j += 1
val += factor * points[i][i + 1]
return val
return result_polynomia

How to set up and solve simultaneous equations in python

For a fixed integer n, I have a set of 2(n-1) simultaneous equations as follows.
M(p) = 1+((n-p-1)/n)*M(n-1) + (2/n)*N(p-1) + ((p-1)/n)*M(p-1)
N(p) = 1+((n-p-1)/n)*M(n-1) + (p/n)*N(p-1)
M(1) = 1+((n-2)/n)*M(n-1) + (2/n)*N(0)
N(0) = 1+((n-1)/n)*M(n-1)
M(p) is defined for 1 <= p <= n-1. N(p) is defined for 0 <= p <= n-2. Notice also that p is just a constant integer in every equation so the whole system is linear.
I have been using Maple but I would like to set these up and solve them in python now, maybe using numpy.linalg.solve (or any other better method). I actually only want the value of M(n-1). For example, when n=2 the answer should be M(1) = 4, I believe. This is because the equations become
M(1) = 1+(2/2)*N(0)
N(0) = 1 + (1/2)*M(1)
Therefore
M(1)/2 = 1+1
and so
M(1) = 4.
If I want to plug in n=50, say, how can you set up this system of simultaneous equations in python so that numpy.linalg.solve can solve them?
Update The answers are great but they use dense solvers where the system of equations is sparse. Posted follow up to Using scipy sparse matrices to solve system of equations .

Updated: added implementation using scipy.sparse
This gives the solution in the order N_max,...,N_0,M_max,...,M_1.
The linear system to solve is of the shape A dot x == const 1-vector.
x is the sought after solution vector.
Here I ordered the equations such that x is N_max,...,N_0,M_max,...,M_1.
Then I build up the A-coefficient matrix from 4 block matrices.
Here's a snapshot for the example case n=50 showing how you can derive the coefficient matrix and understand the block structure. The coefficient matrix A is light blue, the constant right side is orange. The sought after solution vector x is here light green and used to label the columns. The first column show from which of the above given eqs. the row (= eq.) has been derived:
As Jaime suggested, multiplying by n improves the code. This is not reflected in the spreadsheet above but has been implemented in the code below:
Implementation using numpy:
import numpy as np
import numpy.linalg as linalg
def solve(n):
# upper left block
n_to_M = -2. * np.eye(n-1)
# lower left block
n_to_N = (n * np.eye(n-1)) - np.diag(np.arange(n-2, 0, -1), 1)
# upper right block
m_to_M = n_to_N.copy()
m_to_M[1:, 0] = -np.arange(1, n-1)
# lower right block
m_to_N = np.zeros((n-1, n-1))
m_to_N[:,0] = -np.arange(1,n)
# build A, combine all blocks
coeff_mat = np.hstack(
(np.vstack((n_to_M, n_to_N)),
np.vstack((m_to_M, m_to_N))))
# const vector, right side of eq.
const = n * np.ones((2 * (n-1),1))
return linalg.solve(coeff_mat, const)
Solution using scipy.sparse:
from scipy.sparse import spdiags, lil_matrix, vstack, hstack
from scipy.sparse.linalg import spsolve
import numpy as np
def solve(n):
nrange = np.arange(n)
diag = np.ones(n-1)
# upper left block
n_to_M = spdiags(-2. * diag, 0, n-1, n-1)
# lower left block
n_to_N = spdiags([n * diag, -nrange[-1:0:-1]], [0, 1], n-1, n-1)
# upper right block
m_to_M = lil_matrix(n_to_N)
m_to_M[1:, 0] = -nrange[1:-1].reshape((n-2, 1))
# lower right block
m_to_N = lil_matrix((n-1, n-1))
m_to_N[:, 0] = -nrange[1:].reshape((n-1, 1))
# build A, combine all blocks
coeff_mat = hstack(
(vstack((n_to_M, n_to_N)),
vstack((m_to_M, m_to_N))))
# const vector, right side of eq.
const = n * np.ones((2 * (n-1),1))
return spsolve(coeff_mat.tocsr(), const).reshape((-1,1))
Example for n=4:
[[ 7.25 ]
[ 7.76315789]
[ 8.10526316]
[ 9.47368421] # <<< your result
[ 9.69736842]
[ 9.78947368]]
Example for n=10:
[[ 24.778976 ]
[ 25.85117842]
[ 26.65015984]
[ 27.26010007]
[ 27.73593401]
[ 28.11441922]
[ 28.42073207]
[ 28.67249606]
[ 28.88229939]
[ 30.98033266] # <<< your result
[ 31.28067182]
[ 31.44628982]
[ 31.53365219]
[ 31.57506477]
[ 31.58936225]
[ 31.58770694]
[ 31.57680467]
[ 31.560726 ]]

Here's an entirely different approach, using sympy. It's not fast, but it allows me to copy the RHS of your equations exactly, limiting the thinking I need to do (always a plus), and gives fractional answers.
from sympy import Integer, Symbol, Eq, solve
def build_equations(n):
ni = n
n = Integer(n)
Ms = {p: Symbol("M{}".format(p)) for p in range(ni)}
Ns = {p: Symbol("N{}".format(p)) for p in range(ni-1)}
M = lambda i: Ms[int(i)] if i >= 1 else 0
N = lambda i: Ns[int(i)]
M_eqs = {}
M_eqs[1] = Eq(M(1), 1+((n-2)/n)*M(n-1) + (2/n)*N(0))
for p in range(2, ni):
M_eqs[p] = Eq(M(p), 1+((n-p-1)/n)*M(n-1) + (2/n)*N(p-1) + ((p-1)/n)*M(p-1))
N_eqs = {}
N_eqs[0] = Eq(N(0), 1+((n-1)/n)*M(n-1))
for p in range(1, ni-1):
N_eqs[p] = Eq(N(p), 1+((n-p-1)/n)*M(n-1) + (p/n)*N(p-1))
return M_eqs.values() + N_eqs.values()
def solve_system(n, show=False):
eqs = build_equations(n)
sol = solve(eqs)
if show:
print 'equations:'
for eq in sorted(eqs):
print eq
print 'solution:'
for var, val in sorted(sol.items()):
print var, val, float(val)
return sol
which gives
>>> solve_system(2, True)
equations:
M1 == N0 + 1
N0 == M1/2 + 1
solution:
M1 4 4.0
N0 3 3.0
{M1: 4, N0: 3}
>>> solve_system(3, True)
equations:
M1 == M2/3 + 2*N0/3 + 1
M2 == M1/3 + 2*N1/3 + 1
N0 == 2*M2/3 + 1
N1 == M2/3 + N0/3 + 1
solution:
M1 34/5 6.8
M2 33/5 6.6
N0 27/5 5.4
N1 5 5.0
{M2: 33/5, M1: 34/5, N1: 5, N0: 27/5}
and
>>> solve_system(4, True)
equations:
M1 == M3/2 + N0/2 + 1
M2 == M1/4 + M3/4 + N1/2 + 1
M3 == M2/2 + N2/2 + 1
N0 == 3*M3/4 + 1
N1 == M3/2 + N0/4 + 1
N2 == M3/4 + N1/2 + 1
solution:
M1 186/19 9.78947368421
M2 737/76 9.69736842105
M3 180/19 9.47368421053
N0 154/19 8.10526315789
N1 295/38 7.76315789474
N2 29/4 7.25
{N2: 29/4, N1: 295/38, M1: 186/19, M3: 180/19, N0: 154/19, M2: 737/76}
which seems to match the other answers.

This is messy, but solves your problem, barring a very probable mistake transcribing the coefficients:
from __future__ import division
import numpy as np
n = 2
# Solution vector is [N[0], N[1], ..., N[n - 2], M[1], M[2], ..., M[n - 1]]
n_pos = lambda p : p
m_pos = lambda p : p + n - 2
A = np.zeros((2 * (n - 1), 2 * (n - 1)))
# p = 0
# N[0] + (1 - n) / n * M[n-1] = 1
A[n_pos(0), n_pos(0)] = 1 # N[0]
A[n_pos(0), m_pos(n - 1)] = (1 - n) / n #M[n - 1]
for p in xrange(1, n - 1) :
# M[p] + (1 + p - n) /n * M[n - 1] - 2 / n * N[p - 1] +
# (1 - p) / n * M[p - 1] = 1
A[m_pos(p), m_pos(p)] = 1 # M[p]
A[m_pos(p), m_pos(n - 1)] = (1 + p - n) / n # M[n - 1]
A[m_pos(p), n_pos(p - 1)] = -2 / n # N[p - 1]
if p > 1 :
A[m_pos(p), m_pos(p - 1)] = (1 - p) / n # M[p - 1]
# N[p] + (1 + p -n) / n * M[n - 1] - p / n * N[p - 1] = 1
A[n_pos(p), n_pos(p)] = 1 # N[p]
A[n_pos(p), m_pos(n - 1)] = (1 + p - n) / n # M[n - 1]
A[n_pos(p), n_pos(p - 1)] = -p / n # N[p - 1]
if n > 2 :
# p = n - 1
# M[n - 1] - 2 / n * N[n - 2] + (2 - n) / n * M[n - 2] = 1
A[m_pos(n - 1), m_pos(n - 1)] = 1 # M[n - 1]
A[m_pos(n - 1), n_pos(n - 2)] = -2 / n # N[n - 2]
A[m_pos(n - 1), m_pos(n - 2)] = (2 - n) / n # M[n - 2]
else :
# p = 1
#M[1] - 2 / n * N[0] = 1
A[m_pos(n - 1), m_pos(n - 1)] = 1
A[m_pos(n - 1), n_pos(n - 2)] = -2 / n
X = np.linalg.solve(A, np.ones((2 * (n - 1),)))
But it gives a solution of
>>> X[-1]
6.5999999999999979
for M(2) when n=3, which is not what you came up with.