Develop Reference

Optimization Problem with fast matrix-vector multiplication in Python / cvxpy

I want to solve the following (convex) minimization problem:
min ||x||_1 under the constraints sgn(A[x,R]=y) and ||x||_2 = 1
where A is a mx(N+1) matrix, x in R^N a vector, and \[x,R\] a vector that is created by appending a given number R. The objective is to find the optimal value for x.
A is a Fourier matrix and there are fast matrix-vector, inversion, etc. algorithms available. Since this matrix is really big, I need to use an optimization algorithm that utilizes this.
Currently, I use the following implementation in cvxpy, which is way too slow:
import cvxpy as cvx
# rewrite the problem in the form x = x^- + x^+
n = A.shape[1]-1
vx = cvx.Variable(2*n)
objective = cvx.Minimize(cvx.pnorm(vx, 1)) # min ||x||_1
constraints = [vx >= 0, cvx.multiply(A[:,:n] # vx[:n] - A[:,:n] # vx[n:] + A[:,n]*R, y) >= 0,
cvx.norm(vx, 2) <= R] # sgn(A[x,1]) = y, ||x||_2 <= R
x, solve_time = solve(vx, objective, constraints)
solution = x[:n] - x[n:]
Is there a way to use fast matrix computations in cvxpy? Or is there a better library? I found a few implementations that can do this for one special algorithm but not in the general case, so I was not able to implement my problem.

No. The solver will not call your matrix multiplication code. They do their own linear algebra, which is very different in many ways. In a sense your matrix multiplication is just notation for the problem statement.
Regarding performance, it depends heavily on where the bottleneck is. Is it in generating the model (in cvxpy itself) or in the solver? What solver are you using? Consider using a different solver. Obviously, we don't have enough information (and no reproducible example) to answer this question.

Assistance in solving a linear system of equations with least_squares

I'm hoping to just get some assistance conceptually about how to solve a linear system of equations with penalty functions. Example code is at the bottom.
Let's say I'm trying trying to do a fit of this equation:
Ie=IaX+IbY+IcZ
where Ie, Ia, Ib, and Ic are constants, and X,Y,Z are variables
I could easily solve this system of equations using scipy.least_squares, but I want to constrain the system with 2 contraints.
1. X+Y+Z=1
2. X,Y,Z > 0 and X,Y,Z < 1
To do this, I then modified the above function.
Ie-Ic=X(Ia-Ic)+Y(Ib-Ic) where X+Y+Z=1 I solved for Z
therefore
Ax=B where A=[Ia-Ic,Ib-Ic] and B=[Ie-Ic] given bounds (0,1)
This solves the 2nd criteria of X,Y,Z > 0 and X,Y,Z < 1, but it does not solve the 1st.
To resolve the first issue, an additional constraint needs to be made, where X+Y<1, and this I don't quite know how to do.
So I presume least_squares has some built in penalty function for it's bounds. I.E.
chi^2=||A-Bx||^2+P
where P is the conditions, if X,Y,Z>0 then P = 10000
thus, giving high chi squared values and thus setting the constraints
I don't know how I can add a further condition. So if X+Y<1 then P=10000 or something similar along those lines.
In short, least_squares enables you to set bounds on the individual values, but I'd like to set some further constraints, and I don't quite know how to do this with least_squares. I've seen additional inequality constraint options in scipy.minimize, but I don't quite know how to apply that to a linear system of equations with the format Ax=B.
So as an example code, lets say I've already done the calculations and obtained my A matrix of constants and my B vector. I use least squares, get my values for X and Y, and can calculate Z since X+Y+Z=1. The issue here is in my minimization, I did not set a constraint that X+Y<1, so in some cases you can actually get values where X+Y>1. So I'd like to find a method where I can set that additional constraint, in addition to the bounds constraint for the individual variables:
Ax=np.array([[1,2],[2,4],[3,4]])
B=np.array([0,1,2])
solution=lsq_linear(Ax,B,lsq_solver='lsmr',bounds=(0,1))
X=solution.x[0]
Y=solution.x[1]
Z=1-sum(solution.x)
If minimize is the solution here, can you please show me how to set it up given the above matrix of A and array of B?
Any advice, tips, or help to point me in the right direction is greatly appreciated!
Edit:
So I found something similar on here: Minimizing Least Squares with Algebraic Constraints and Bounds
So I thought I'd apply it to my case, but I don't think I've been able to apply it properly.
Ax=np.array([[1,2],[2,4],[3,4]])
B=np.array([0,1,2])
def fun(x,a1,a2,y):
fun_output=x[0]*a1+x[1]*a2
return np.sum((fun_output-y)**2)
cons = [{"type": "eq", "fun": lambda x: x[0] + x[1] - 1}]
bnds = [(0, 1), (0, 1)]
xinit = np.array([1, 1])
solution=minimize(fun,args=(Ax[:,0],Ax[:,1], B), x0=xinit, bounds=bnds, constraints=cons)
solution_2=lsq_linear(Ax,B,bounds=(0,1))
print(solution.x)
print(solution_2.x)
Issue is, the output of this differs from lsq_linear, and I almost always get a very close to zero value for Z regardless of what the input arrays are. I don't know if I'm setting this up/understanding this correctly.

Your initial guess xinit is not feasible and doesn't satisfy your constraint.
IMO, solving the initial problem directly as a constrained nonlinear optimization problem (NLP) instead of rewriting it is the easier approach. Assuming you have all the data points Ia, Ib, Ic and Ie (you didn't provide all of them), you can use the following code snippet which is in the same vein as the linked answer of mine in your question.
from scipy.optimize import minimize
import numpy as np
def fun_to_fit(coeffs, *args):
x, y, z = coeffs
Ia, Ib, Ic = args
return Ia*x + Ib*y + Ic*z
def objective(coeffs, *args):
Ia, Ib, Ic, Ie = args
residual = Ie - fun_to_fit(coeffs, Ia, Ib, Ic)
return np.sum(residual**2)
# Constraint: x + y + z == 1
con = [{'type': 'eq', 'fun': lambda coeffs: np.sum(coeffs) - 1}]
# bounds
bounds = [(0, 1), (0, 1), (0, 1)]
# initial guess (fulfils the constraint and lies within the bounds)
x0 = np.array([0.25, 0.5, 0.25])
# your given data points
#Ia = np.array(...)
#Ib = np.array(...)
#Ic = np.array(...)
#Ie = np.array(...)
# solve the NLP
res = minimize(lambda coeffs: objective(coeffs, Ia, Ib, Ic, Ie), x0=x0, bounds=bounds, constraint=con)

Imposing monotonicity with scipy.optimize.minimize

I am trying to minimize a function of a vector of length 20, but I want to constrain the solution to be monotonic, i.e.
x[1] <= x[2]... <= x[20]
I have tried to implement this in the following way using "constraints" for this routine:
cons = tuple([{'type':'ineq', 'fun': lambda x: x[i]- x[i-1]} for i in range(1, len(node_vals))])
res = sp.optimize.minimize(localisation, b, args=(d), constraints = cons) #optimize
However, the results I get are not monotonic, even when the initial guess b is, it seems that the optimizer is completely ignoring the constraints. What could be going wrong? I have also tried changing the constraint to x[i]**3 - x[i+1]**3 to make it "smoother", but it didn't help at all. My objective function, localisation is the integral of solution to an eigenvalue problem whose parameters are defined beforehand:
def localisation(node_vals, domain): #calculate localisation for solutions with piecewise linear grading
f = piecewise(node_vals, domain) #create piecewise linear function using given values at nodes
#plt.plot(domain, f(domain))
M = diff_matrix(f(domain)) #differentiation matrix created from piecewise linear function
m = np.concatenate(([0], get_solutions(M)[1][:, 0], [0]))
integral = num_int(domain, m)
return integral

You didn’t post a minimum reproducible example that we can run. However, did you try to specify which optimization algorithm to use in SciPy? Something like this:
res = sp.optimize.minimize(localisation, b, args=(d), constraints = cons, method=‘SLSQP’)

I'm having a very similar problem but with additional upper and lower bounds on the monotonicity property. I'm tackling the problem like this (maybe it helps you):
Using the Trust-Region Constrained Algorithm given by scipy. This provides us a way of dealing with linear constraints in a matrix-manner:
lb <= A.dot(x) <= ub
where lb & and ub are the lower (upper) bounds of this constraint problem and A is the matrix, representing the linear constraint problem.
every row of matrix A is a linear term which defines a constraint
If, for example, x[0] <= x[1], then this can be transformed into x[0] - x[1] <= 0 which in terms of the linear constraint matrix A looks like this [1, -1,...], provided that the upper bound vector has a 0 value on this level of course (vice versa is also possible but either way, having at least one of both, lower or upper bound, makes this easy)
Setting up enough of these inequalities and at the same time merging a couple of those into a single inequality may create a sufficient matrix to solve this.
Hope this helps a bit, It did the job for my problem.

TypeError: must be real number, not LpAffineExpression, Optimization with PuLP

I have the following code:
First I define the minkowski distance. It is the distance between two vectors. x represents vector 1, y represent vector 2, w represents weights of all criteria, p_value represents the p of the minkowski distance.
def p_root(value, root):
root_value = 1 / float(root)
return float(round(Decimal(value) **
Decimal(root_value), 3))
def minkowski_distance(x, y, w, p_value):
# pass the p_root function to calculate
# all the value of vector parallelly
return (p_root(sum(pow(c*(abs(a - b)), p_value)
for a, b, c in zip(x, y, w)), p_value))
Then I use PuLP for optimization.
prob = LpProblem("Sorting Problem", LpMinimize)
w1 = LpVariable("Weight_1", 0, cat="Continuous")
w2 = LpVariable("Weight_2", 0, cat="Continuous")
w3 = LpVariable("Weight_3", 0, cat="Continuous")
w4 = LpVariable("Weight_4", 0, cat="Continuous")
epsilon1=LpVariable("Classification_Error_1",0, cat="Continuous")
epsilon2=LpVariable("Classification_Error_2",0, cat="Continuous")
epsilon3=LpVariable("Classification_Error_3",0, cat="Continuous")
epsilon4=LpVariable("Classification_Error_4",0, cat="Continuous")
.
.
.
Here, epsilon values represent the classification error, and goal is to minimize them.
prob += (
epsilon1+epsilon2+epsilon3+epsilon4+epsilon5+epsilon6+epsilon7+epsilon8+epsilon9+epsilon10+epsilon11+epsilon12,
"Sum_of_Classification_Errors",
)
This makes the sum of the criterion weights equal to one.
prob+= w1+w2+w3+w4 == 1
Here, I put the weight values in a list to utilize in the minkowski function.
w=[w1,w2,w3,w4]
Classification error occurs when an alternative is assigned to a further centroid than the one it actually is closer to. Here, we represent it.
prob+= minkowski_distance(train_vector[0],centroid_vector[0],w,1) - epsilon1<=minkowski_distance(train_vector[0],centroid_vector[1],w,1)-r
prob+= minkowski_distance(train_vector[0],centroid_vector[0],w,1) - epsilon1<=minkowski_distance(train_vector[0],centroid_vector[2],w,1)-r
.
.
.
However, at this point I get the following error. TypeError: must be real number, not LpAffineExpression. I have checked other similar questions before posting this one and the problem in those questions were trying to solve nonlinear problem with PuLP(It is a linear programming solver), or confusing the meaning of a variable in linear programs. I believe here there is a different case as in minkowski distances when the distance norm is 1 or infinite, the problem is linear. I used the distance norm as 1 here, but still I got this error.
P.S: I know the code does not seem very efficient, but I got a couple of errors trying to make it more efficient and now I am trying to ensure that the code itself works before simplifying it.

PuLP is for linear models only (see the documentation). You try to pass on a non-linear model. Obviously, that will never work. No good reason to even try.
The code presented is incomplete, so it is difficult to decipher what is exactly going on, but you can use a distance function as long as it depends on exogenous data only. I.e. the weights (decision variables) should not be part of the distance function.

scipy.integrate.solve_bvp for Neumann boundary conditions?

The question is what it says on the tin. I'm trying to numerically solve a boundary value problem and my friend is asking me whether the solver would work for these conditions. The page for the solver doesn't give the conditions we have i.e. bc(y(a),y(b), p) = 0 but the form of our question is y(0) = some constant value and y'(b) = 0, giving our Neumann conditions, Would you need to rewrite the function to have a first order reduction like in the shooting method?

I suppose that you are solving some second order ODE
y''(t) = F(t,y(t), y'(t))
For the first order system use the state vector u = [y, v] = [y, y']. Then the minimum to call the BVP solver is
def ode(t,u): y,v = u; return [v, F(t,y,v)];
def bc(u0, ub): return [u0[0]-y0, ub[1]];
y0 = someconstantvalue;
t_init = [0,b];
u_init = [[y0, y0], [0, 0]];
res = solve_bvp(ode, bc, t_init, u_init)
Always check res.success or at least print out res.message. You might need a better initial guess, for non-linear problems different initial guesses can give different solutions. res.sol contains the solution as an interpolating function, for generating plots it is better to use this function than the sparse collection of internal steps in res.x and res.y