I have a convex optimization problem with separable, convex, piecewise linear functions f_i(var_i) each defined by a list of points [(values, costs)] and a couple other terms that are also convex. I'm trying to figure out how two build those piecewise functions in CVXPY.
How do I take the below two lists of points and add them to a CVXPY objective as piecewise functions?
import cvxpy as cp
w = cp.Variable(n)
f1_points = [(-5, 10), (-2, -1), (0, 0)] # -5 <= var1 <= 0 (Convex)
f2_points = [(-4, 5), (0, 0)] # -4 <= var2 <= 0 (Linear)
f1_cost_function = ...
f2_cost_function = ...
constraints = [cp.sum(w) = 0] + ...
problem = cp.Problem(cp.Minimize(cp.sum([f1_cost_function, f2_cost_function] + ...)), constraints)
So this does not appear directly possible in CVXPY from the list of points. However if the piecewise functions are rewritten as point-slope functions instead of a collection of points, the cvxpy maximum function can be used for to make the piecewise linear function.
f1_functions = [f1_line1, f1_line2, ...]
f1 = cp.maximum(f1_functions)
This is described with an example in the user guide.
If your curve is simple, like in this picture, and you're objective function is to minimize y, then you can do this simply by putting constraints like this:
contraints = [y <= f1, y <= f2, y <= f3, y <= f4 ]
objective = cp.minimize(y)
Picture of simple Piece-wise functions for a non-linear curve
Related
Setup for the problem:
I have a canvas which represents a city, each second I add a new resident to the city. The each resident has a job with a location that is randomly sampled from a distribution. Each resident also has a custom cost function that helps them decide where they want to live which they do by minimizing this cost function with respect to two variables x and y. So the function for example looks something like:
cost(x,y) = distance_to_job(x,y) + distance_to_center_of_city(x,y) + population_density(x,y)
where population_density(x,y) is just the population density at point (x,y). Naturally population_density(x,y) (without any transformations) is a piecewise non-differentiable function as one has to define a grid of blocks in the city and keep track of how many people per grid unit there is (think of a population density map of the world, each country has a distinct value that isn't necessarily the same as its neighbor, so if you were to map this on a 3-D plot, the function that you map would not be smooth).
Let me know if this setup is confusing, I'll try to make it a bit more clear.
The Question:
One could define a transformation where between each grid cell you designate a steep but smooth transition between the values of the piecewise function but as of now my population density function is not smooth and not differentiable at each boundary between grid cells. At first I did not think that SGD optimization in tensorflow would not work as I don't have a differentiable cost function but it seems to run fine. I am confused about what exactly is going on here and would love any clarification about how SGD optimization works and if my code is doing what I want it to.
Relevant Code:
def concentrationLookup(self, x, y):
r_index = int(x // (self.city.total_w / self.city.rows))
c_index = int(y // (self.city.total_h / self.city.cols))
return self.city.grid[r_index, c_index]
tf_jobCost = lambda x,y: (0.1/travelCost) * (tf.pow(x - self.jobx, 2) + tf.pow(y - self.joby, 2))
tf_cityCost = lambda x,y: 0.01 * (tf.pow(x - self.city.centerX, 2)) + 0.01*(tf.pow(y - self.city.centerY, 2))
xVar = tf.Variable(locX)
yVar = tf.Variable(locY)
self.costfn = lambda: tf_jobCost(xVar, yVar) + tf_cityCost(xVar, yVar) + self.concentrationLookup(xVar, yVar)
opt = tf.keras.optimizers.SGD(learning_rate = 3.0)
for _ in range(100):
opt.minimize(self.costfn, var_list = [xVar, yVar])
self.x = xVar.numpy()
self.y = yVar.numpy()
I believe it's treating population_density(x,y) as a constant function of x,y. In other words, it doesn't contribute to the gradient, and doesn't contribute to the solution.
You can also verify this by zeroing out other components of the loss, and verifying that opt.minimize() fails with something like ValueError: No gradients provided for any variable....
I think the solution should be to forget that the function is piecewise constant and non-differentiable, and instead to treat it as piecewise linear instead. In that case, concentrationLookup(x,y) can be written as returning a bilinearly-weighted sum of points at the 4 neighboring pixels.
Something like this:
def concentrationLookup(x, y):
r = x / (total_w / rows) # no quantizing
c = y / (total_h / cols) # no quantizing
r1, c1 = int(r), int(c) # lower bounds
r2, c2 = r1 + 1, c1 + 1 # upper bounds
w_r2, w_c2 = r - r1, c - c1
w_r1, w_c1 = 1.0 - w_r2, 1.0 - w_c2
# Assume constant boundary conditions.
c2 = tf.clip_by_value(c2, 0, grid.shape[1]-1)
c1 = tf.clip_by_value(c1, 0, grid.shape[1]-1)
r2 = tf.clip_by_value(r2, 0, grid.shape[0]-1)
r1 = tf.clip_by_value(r1, 0, grid.shape[0]-1)
return w_r1*w_c1*grid[r1, c1] + w_r2*w_c2*grid[r2,c2] + w_r1*w_c2*grid[r1,c2] + w_r2*w_c1*grid[r2, c1]
In this case, the gradient seems to be well defined.
I have a combinatorics problem that I can't solve.
Given a set of vectors and a target vector, return a scalar for each vector, so that the average of the scaled vectors in the set is closest to the target.
Edit: Weights w_i are in range [0, 1]. This is a constrained optimisation problem:
minimise d(avg(w_i * x_i), target)
subject to sum(w_i) - 1 = 0
If i had to name this problem it would be unbounded subset average.
I have looked at the unbounded knapsack and similar problems, but a dynamic programming implementation seems to be impossible due to the interdependence of the numbers.
I also inplemented a genetic algorithm that is able to approximate the weights moderately well, but it takes too long and I was initially hoping to solve the problem using dynamic programming.
Is there any hope?
Visualization
In a 2D space the solution to the problem can be represented like this
Problem class identification
As recognized by others this is a an optimization problem. You have linear constraints and a convex objective function, it can be cast to quadratic programming, (read Least squares session)
Casting to standard form
If you want to minimize the average of w[i] * x[i], this is sum(w[i] * x[i]) / N, if you arrange w[i] as the elements of a (1 x N_vectors) matrix, and each vector x[i] as the i-th row of a (N_vectors x DIM) matrix, it becomes w # X / N_vectors (with # being the matrix product operator).
To cast to that form you would have to construct a matrix so that each rows of A*x < b expressing -w[i] < 0, the equality is sum(w) = 1 becomes sum(w) < 1 and -sum(w) < -1. But there there are amazing tools to automate this part.
Implementation
This can be readily implemented using cvxpy, and you don't have to care about expanding all the constraints.
The following function solves the problem and if the vectors have dimension 2 plot the result.
import cvxpy;
import numpy as np
import matplotlib.pyplot as plt
def place_there(X, target):
# some linear algebra arrangements
target = target.reshape((1, -1))
ncols = target.shape[1]
X = np.array(X).reshape((-1, ncols))
N_vectors = X.shape[0]
# variable of the problem
w = cvxpy.Variable((1, X.shape[0]))
# solve the problem with the objective of minimize the norm of w * X - T (# is the matrix product)
P = cvxpy.Problem(cvxpy.Minimize(cvxpy.norm((w # X) / N_vectors - target)), [w >= 0, cvxpy.sum(w) == 1])
# here it is solved
print('Distance from target is: ', P.solve())
# show the solution in a nice plot
# w.value is the w that gave the optimal solution
Y = w.value.transpose() * X / N_vectors
path = np.zeros((X.shape[0] + 1, 2))
path[1:, :] = np.cumsum(Y, axis=0)
randColors=np.random.rand( 3* X.shape[0], 3).reshape((-1, 3)) * 0.7
plt.quiver(path[:-1,0], path[:-1, 1], Y[:, 0], Y[:, 1], color=randColors, angles='xy', scale_units='xy', scale=1)
plt.plot(target[:, 0], target[:, 1], 'or')
And you can run it like this
target = np.array([[1.234, 0.456]]);
plt.figure(figsize=(12, 4))
for i in [1,2,3]:
X = np.random.randn(20) * 100
plt.subplot(1,3,i)
place_there(X, target)
plt.xlim([-3, 3])
plt.ylim([-3, 3])
plt.grid()
plt.show();
I'm trying to integrate a piecewise function using Sagemath, and finding it to be impossible. My original code is below, but it's wrong due to accidental evaluation described here.
def f(x):
if(x < 0):
return 3 * x + 3
else:
return -3 * x + 3
g(x) = integrate(f(t), t, 0, x)
The fix for plotting mentioned on the website is to use f instead of f(t), but this is apparently unsupported for the integrate() function since a TypeError is raised.
Is there a fix for this that I'm unaware of?
Instead of defining a piecewise function via def, use the built-in piecewise class:
f = Piecewise([[(-infinity, 0), 3*x+3],[(0, infinity), -3*x+3]])
f.integral()
Output:
Piecewise defined function with 2 parts, [[(-Infinity, 0), x |--> 3/2*x^2 + 3*x], [(0, +Infinity), x |--> -3/2*x^2 + 3*x]]
The piecewise functions have their own methods, such as .plot(). Plotting does not support infinite intervals, though. A plot can be obtained with finite intervals
f = Piecewise([[(-5, 0), 3*x+3],[(0, 5), -3*x+3]])
g = f.integral()
g.plot()
But you also want to subtract g(0) from g. This is not as straightforward as g-g(0), but not too bad, either: get the list of pieces with g.list(), subtract g(0) from each function, then recombine.
g0 = Piecewise([(piece[0], piece[1] - g(0)) for piece in g.list()])
g0.plot()
And there you have it:
By extending this approach, we don't even need to put finite intervals in f from the beginning. The following plots g - g(0) on a given interval [a,b], by modifying the domain:
a = -2
b = 3
g0 = Piecewise([((max(piece[0][0], a), min(piece[0][1], b)), piece[1] - g(0)) for piece in g.list()])
g.plot()
In addition to using the Piecewise class, this can easily be fixed by defining g(x) as a Python function as well:
def f(x):
if(x < 0):
return 3 * x + 3
else:
return -3 * x + 3
def g(x):
(y, e) = integral_numerical(f, 0, x)
return y
Then plot(g) works just fine.
I have a generic question on how to solve optimization problems of the Min-Max type, using the PICOS package in Python. I found little information in this context while searching the PICOS documentation and on the web as well.
I can imagine a simple example of the below form.
Given a matrix M, find x* = argmin_x [ max_y x^T M y ], where x > 0, y > 0, sum(x) = 1 and sum(y) = 1.
I have tried a few methods, starting with the most straightforward idea of having minimax, minmax keywords in the objective function of PICOS Problem class. It turns out that none of these keywords are valid, see the package documentation for objective functions. Furthermore, having nested objective functions also turns out to be invalid.
In the last of my naive attempts, I have two functions, Max() and Min() which are both solving a linear optimization problem. The outer function, Min(), should minimize the inner function Max(). So, I have used Max() in the objective function of the outer optimization problem.
import numpy as np
import picos as pic
import cvxopt as cvx
def MinMax(mat):
## Perform a simple min-max SDP formulated as:
## Given a matrix M, find x* = argmin_x [ max_y x^T M y ], where x > 0, y > 0, sum(x) = sum(y) = 1.
prob = pic.Problem()
## Constant parameters
M = pic.new_param('M', cvx.matrix(mat))
v1 = pic.new_param('v1', cvx.matrix(np.ones((mat.shape[0], 1))))
## Variables
x = prob.add_variable('x', (mat.shape[0], 1), 'nonnegative')
## Setting the objective function
prob.set_objective('min', Max(x, M))
## Constraints
prob.add_constraint(x > 0)
prob.add_constraint((v1 | x) == 1)
## Print the problem
print("The optimization problem is formulated as follows.")
print prob
## Solve the problem
prob.solve(verbose = 0)
objVal = prob.obj_value()
solution = np.array(x.value)
return (objVal, solution)
def Max(xVar, M):
## Given a vector l, find y* such that l y* = max_y l y, where y > 0, sum(y) = 1.
prob = pic.Problem()
# Variables
y = prob.add_variable('y', (M.size[1], 1), 'nonnegative')
v2 = pic.new_param('v1', cvx.matrix(np.ones((M.size[1], 1))))
# Setting the objective function
prob.set_objective('max', ((xVar.H * M) * y))
# Constraints
prob.add_constraint(y > 0)
prob.add_constraint((v2 | y) == 1)
# Solve the problem
prob.solve(verbose = 0)
sol = prob.obj_value()
return sol
def print2Darray(arr):
# print a 2D array in a readable (matrix like) format on the standard output
for ridx in range(arr.shape[0]):
for cidx in range(arr.shape[1]):
print("%.2e \t" % arr[ridx,cidx]),
print("")
print("========")
return None
if __name__ == '__main__':
## Testing the Simple min-max SDP
mat = np.random.rand(4,4)
print("## Given a matrix M, find x* = argmin_x [ max_y x^T M y ], where x > 0, y > 0, sum(x) = sum(y) = 1.")
print("M = ")
print2Darray(mat)
(optval, solution) = MinMax(mat)
print("Optimal value of the function is %.2e and it is attained by x = %s and that of y = %.2e." % (optval, np.array_str(solution)))
When I run the above code, it gives me the following error message.
10:stackoverflow pavithran$ python minmaxSDP.py
## Given a matrix M, find x* = argmin_x [ max_y x^T M y ], where x > 0, y > 0, sum(x) = sum(y) = 1.
M =
1.46e-01 9.23e-01 6.50e-01 7.30e-01
6.13e-01 6.80e-01 8.35e-01 4.32e-02
5.19e-01 5.99e-01 1.45e-01 6.91e-01
6.68e-01 8.46e-01 3.67e-01 3.43e-01
========
Traceback (most recent call last):
File "minmaxSDP.py", line 80, in <module>
(optval, solution) = MinMax(mat)
File "minmaxSDP.py", line 19, in MinMax
prob.set_objective('min', Max(x, M))
File "minmaxSDP.py", line 54, in Max
prob.solve(verbose = 0)
File "/Library/Python/2.7/site-packages/picos/problem.py", line 4135, in solve
self.solver_selection()
File "/Library/Python/2.7/site-packages/picos/problem.py", line 6102, in solver_selection
raise NotAppropriateSolverError('no solver available for problem of type {0}'.format(tp))
picos.tools.NotAppropriateSolverError: no solver available for problem of type MIQP
10:stackoverflow pavithran$
At this point, I am stuck and unable to fix this problem.
Is it just that PICOS does not natively support min-max problem or is my way of encoding the problem, incorrect?
Please note: The reason I am insisting on using PICOS is that ideally, I would like to know the answer to my question in the context of solving a min-max semidefinite program (SDP). But I think the addition of semidefinite constraints is not hard, once I can figure out how to do a simple min-max problem using PICOS.
The first answer is that min-max problems are not natively supported in PICOS. However, whenever the inner maximization problem is a convex optimization problem, you can reformulate it as a minimization problem (by taking the Lagrangian dual), and so you get a min-min problem.
Your particular problem is a standard zero-sum game, and can be reformulated as: (assuming M is of dimension n x m):
min_x max_{i=1...m} [M^T x]_i = min_x,t t s.t. [M^T x]_i <= t (for i=1...m)
In Picos:
import picos as pic
import cvxopt as cvx
n=3
m=4
M = cvx.normal(n,m) #generate a random matrix
P = pic.Problem()
x = P.add_variable('x',n,lower=0)
t = P.add_variable('t',1)
P.add_constraint(M.T*x <= t)
P.add_constraint( (1|x) == 1)
P.minimize(t)
print 'the solution is x='
print x
If you also need the optimal y, then you can show that it corresponds to the optimal value of the constraint M'x <= t:
print 'the solution of the inner max-problem is y='
print P.constraints[0].dual
Best,
Guillaume.
The thing is, im trying to design of fitting procedure for my purposes and want to use scipy`s differential evolution algorithm as a general estimator of initial values which then will be used in LM algorithm for better fitting. The function i want to minimize with DE is the least squares between analytically defined non-linear function and some experimental values. Point at which i stuck is the function design. As its stated in scipy reference: "function must be in the form f(x, *args) , where x is the argument in the form of a 1-D array and args is a tuple of any additional fixed parameters needed to completely specify the function"
There is an ugly example of code which i wrote just for illustrative purposes:
def func(x, *args):
"""args[0] = x
args[1] = y"""
result = 0
for i in range(len(args[0][0])):
result += (x[0]*(args[0][0][i]**2) + x[1]*(args[0][0][i]) + x[2] - args[0][1][i])**2
return result**0.5
if __name__ == '__main__':
bounds = [(1.5, 0.5), (-0.3, 0.3), (0.1, -0.1)]
x = [0,1,2,3,4]
y = [i**2 for i in x]
args = (x, y)
result = differential_evolution(func, bounds, args=args)
print(func(bounds, args))
I wanted to supply raw data as a tuple into the function but it seems that its not how its suppose to be since interpreter isn't happy with the function. The problem should be easy solvable, but i really frustrated, so advice will be much appreciated.
This is kinda straightforward solution which shows the idea, also code isn`t very pythonic but for simplicity i think its good enough. Ok as example we want to fit equation of a kind y = ax^2 + bx + c to a data obtained from equation y = x^2. It obvious that parameter a = 1 and b,c should equal to 0. Since differential evolution algorithm finds minimum of a function we want to find a minimum of a root mean square deviation (again, for simplicity) of analytic solution of general equation (y = ax^2 + bx + c) with given parameters (providing some initial guess) vs "experimental" data. So, to the code:
from scipy.optimize import differential_evolution
def func(parameters, *data):
#we have 3 parameters which will be passed as parameters and
#"experimental" x,y which will be passed as data
a,b,c = parameters
x,y = data
result = 0
for i in range(len(x)):
result += (a*x[i]**2 + b*x[i]+ c - y[i])**2
return result**0.5
if __name__ == '__main__':
#initial guess for variation of parameters
# a b c
bounds = [(1.5, 0.5), (-0.3, 0.3), (0.1, -0.1)]
#producing "experimental" data
x = [i for i in range(6)]
y = [x**2 for x in x]
#packing "experimental" data into args
args = (x,y)
result = differential_evolution(func, bounds, args=args)
print(result.x)