Develop Reference

Optimization Problem with fast matrix-vector multiplication in Python / cvxpy

I want to solve the following (convex) minimization problem:
min ||x||_1 under the constraints sgn(A[x,R]=y) and ||x||_2 = 1
where A is a mx(N+1) matrix, x in R^N a vector, and \[x,R\] a vector that is created by appending a given number R. The objective is to find the optimal value for x.
A is a Fourier matrix and there are fast matrix-vector, inversion, etc. algorithms available. Since this matrix is really big, I need to use an optimization algorithm that utilizes this.
Currently, I use the following implementation in cvxpy, which is way too slow:
import cvxpy as cvx
# rewrite the problem in the form x = x^- + x^+
n = A.shape[1]-1
vx = cvx.Variable(2*n)
objective = cvx.Minimize(cvx.pnorm(vx, 1)) # min ||x||_1
constraints = [vx >= 0, cvx.multiply(A[:,:n] # vx[:n] - A[:,:n] # vx[n:] + A[:,n]*R, y) >= 0,
cvx.norm(vx, 2) <= R] # sgn(A[x,1]) = y, ||x||_2 <= R
x, solve_time = solve(vx, objective, constraints)
solution = x[:n] - x[n:]
Is there a way to use fast matrix computations in cvxpy? Or is there a better library? I found a few implementations that can do this for one special algorithm but not in the general case, so I was not able to implement my problem.

No. The solver will not call your matrix multiplication code. They do their own linear algebra, which is very different in many ways. In a sense your matrix multiplication is just notation for the problem statement.
Regarding performance, it depends heavily on where the bottleneck is. Is it in generating the model (in cvxpy itself) or in the solver? What solver are you using? Consider using a different solver. Obviously, we don't have enough information (and no reproducible example) to answer this question.

Piecewise Linear Function as Constraint

I'm trying to use pyscipopt to solve a linear programming problem, but am unable to fit the piecewise linear function as a constraint.
The constraint is expressed as follows:
I've tried to write it as the following:
cfm = quicksum( max(quicksum(cf[i][t] * q[i] - L[t] for i in range(I)), 0) for t in range(T) / quicksum(L[t] for t in range(T)) <= cfm_max
Where cfm_max = 0.15, in this case.
But it is probably very wrong since it returns a NotImplementedError. I've seen examples in piecewise.py found together with the package, but their usage seems different enough to not work in my case.
Would appreciate any help, thanks.

I think this can be written as:
sum(t,y[t]) <= 0.15*sum(t,L[t])
y[t] >= sum(i,CF[i,t]*q[i])-L[t]
where y[t] are non-negative variables. This is now completely linear (no division, no max()).

Finding A Solution To A System of Inequalities in Python Without A Optimization Function

I have a large system of linear inequalities I am trying to solve for in python, ideally with non-negative integers.
Eg:
I have a matrix A of size (1000*700) and matrix b of size (700) where each line in the system is set up such that A*C <= b. I am trying to solve for C. Multiple solutions online involve Scipy optimization module, however this requires an optimization function. Trying to put all 0's for the coefficients in the optimization function and inputting the constraints as the system of inequalities gives me a solution which is wrong. I have put down a part of my code below.
from scipy.optimize import linprog
coefficients_min_y = [0] * len(A[0])
res = linprog(coefficients_min_y, A_ub=A, b_ub=b, bounds=(0, None))
if res.success:
coefficients = res.x
else:
print("Solution is infeasible")
matrixMult = np.dot(A, coefficients)
Given that linprog outputs a feasible solution, I use matrixMult to check if the solution satisfies the conditions. However it is not solving correctly. Is this because of the optimization function or am I doing something else wrong.

Improper integral of the second kind

I am trying to calculate exact value of an improper integral of 2nd kind with sympy:
from sympy import integrate, log
from sympy.abc import x
print (integrate(log(x) * log(x) /(1+x*x), (x,0,1)))
This code return a lot of mistakes. May be I need to use another approach? I have try with Integral and got nothing.
I 'd like to calculate these integrals from Dwight tables (863.61 and 863.10):
I may calculate them with numerical methods but rather I'd like to get exact solutions with sympy. Is it possible to get exact solution of an improper integral of the 2nd kind with sympy? Or these integrals are too complicated for sympy?

Floating point numbers are poison for symbolic computations, especially as complicated as symbolic integration. Don't put them in symbolic integrals.
Also, declaring positive variables as such can be a big help.
x = symbols('x', positive=True)
int1 = integrate(log(x)**2 / (1 + x**2), (x, 0, 1))
int2 = integrate(log(1/x) / (1 - x), (x, 0, 1))
No errors now, but int1 is just the original integral un-evaluated; SymPy did not succeed in finding its value. It seems to be beyond its ability.
For the second one it returns polylog(2, -exp_polar(I*pi)). The presence of complex number I*pi and exp_polar means SymPy was doing some complex plane work where the amount of winding around the origin might matter. The function exp_polar is different from exp in that exp_polar(2*I*pi) does not simplify to 1 like exp(2*I*pi) does: it keeps the distinction between turning by 360 degrees and not turning at all.
But if we ignore all that and put exp in the result,
polylog(2, -exp(I*pi))
evaluates to pi**2 / 6, the correct value of the second integral.

equation system with fsolve

I try to find a solution for a system of equations by using scipy.optimize.fsolve in python 2.7. The goal is to calculate equilibrium concentrations for a chemical system. Due to the nature of the problem, some of the constants are very small. Now for some combinations i do get a proper solution. For some parameters i don't find a solution. Either the solutions are negative, which is not reasonable from a physical point of view or fsolve produces:
ier = 3, 'xtol=0.000000 is too small, no further improvement in the approximate\n solution is possible.')
ier = 4, 'The iteration is not making good progress, as measured by the \n improvement from the last five Jacobian evaluations.')
ier = 5, 'The iteration is not making good progress, as measured by the \n improvement from the last ten iterations.')
It seems to me, based on my research, that the failure to find proper solutions of the equation system is connected to the datatype float.64 not being precise enough. As a friend pointed out, the system is not well conditioned with parameters differing in several magnitudes.
So i tried to use fsolve with the mpfr type provided by the gmpy2 module but that resulted in the following error:
TypeError: Cannot cast array data from dtype('O') to dtype('float64') according to the rule 'safe'
Now here is a small example with parameter which lead to a solution if the randomized starting parameters fit happen to be good. However if the constant C_HCL is chosen to be something like 1e-4 or bigger then i never find a proper solution.
from numpy import *
from scipy.optimize import *
K_1 = 1e-8
K_2 = 1e-8
K_W = 1e-30
C_HCL = 1e-11
C_NAOH = K_W/C_HCL
C_HL = 1e-6
if C_HCL-C_NAOH > 0:
Saeure_Base = C_HCL-C_NAOH+sqrt(K_W)
OH_init = K_W/(Saeure_Base)
elif C_HCL-C_NAOH < 0:
OH_init = C_NAOH-C_HCL+sqrt(K_W)
Saeure_Base = K_W/OH_init
# some randomized start parameters
G1 = random.uniform(0, 2)*Saeure_Base
G2 = random.uniform(0, 2)*OH_init
G3 = random.uniform(1, 2)*C_HL*(sqrt(K_W))/(Saeure_Base+OH_init)
G4 = random.uniform(0.1, 1)*(C_HL - G3)/2
G5 = C_HL - G3 - G4
zGuess = array([G1,G2,G3,G4,G5])
#equation system / 5 variables --> H3O, OH, HL, H2L, L
def myFunction(z):
H3O = z[0]
OH = z[1]
HL = z[2]
H2L = z[3]
L = z[4]
F = empty((5))
F[0] = H3O*L/HL - K_1
F[1] = OH*H2L/HL - K_2
F[2] = K_W - OH*H3O
F[3] = C_HL - HL - H2L - L
F[4] = OH+L+C_HCL-H2L-H3O-C_NAOH
return F
z = fsolve(myFunction,zGuess, maxfev=10000, xtol=1e-15, full_output=1,factor=0.1)
print z
So the questions are. Is this problem based on the precision of float.64 and
if yes , (how) can it be solved with python? Is fsolve the way to go? Would i need to change the fsolve function so it accepts a different data type?

The root of your problem is either theoretical or numerical.
The scipy.optimize.fsolvefunction is based on the MINPACK Fortran solver (http://www.netlib.org/minpack/). This solver use a Newton-Raphson optimisation algorithm to provide the solution.
There are underlying assumptions about the smoothness of the function when you use this algorithm. For example, the jacobian matrix at the solution point x is supposed to be invertible. The one you are more concerned about is the basins of attraction.
In order to converge, the starting point of the algorithm needs to be near the actual solution, i.e. in the basins of attraction. This condition is always met for convex functions, however it is easy to find some functions for which this algorithm behaves badly. Your function is one of this as you have a fraction of your inputs parameters.
To address this issue you should just change the starting point. This starting point becomes also very important for functions with multiple solutions: this picture from the wikipedia article shows you the solution found depending of the starting point (five colours for five solutions); so you should be careful with your solution and actually check the "physical" aspects of your solution.
For the numerical aspects, the Newton-Raphson algorithm needs to have the value of the jacobian matrix (the derivatives matrix). If it is not provided to the MINPACK solver, the jacobian is estimated with a finite-difference formula. The perturbation step for the finite difference formula need to be provided epsfcn=None, the None being here as default value only in the case where fprimeis provided (there is no need for the jacobian estimation in this case). So first you should incorporate that. You could also specify directly the jacobian by derivating your function by hand.
However, the minimum value for the step size will be the machine precision, also called machine epsilon. For your problem, you have very small inputs values which can be a problem. I would suggest multiply everyone of them by the same value (like 10^6), it is equivalent to a change of the units but will avoid rounding up errors and problems with machine precision.
This problem is also important when you look at the parameter xtol=1e-15 you provided. In your error message, it gives xtol=0.000000, as it is below machine precision and cannot be taken into account. Also, if you look at your line F[2] = K_W - OH*H3O, given the machine precision, it does not matter if K_W is 1e-15or 1e-30. 0 is a solution for both of this case compare to the machine precision. To avoid this problem, just multiply everything by a bigger value.
So to sum up:
For the Newton-Raphson algorithm, the initialisation point matters !
For this algorithm, you should specify how you compute the jacobian !
In numerical computation, never work with small values. You can easily change the dimension to something different: it is basic units conversion, like working in gram instead of kilogram.