Develop Reference

GEKKO returned non-optimal solution

I want to use GEKKO to solve the following optimization problem:
Minimize x'Qx + 1e-10 * sum_{i=1}^n x_i^0.1
subject to 1' x = 1 and x >= 0
However, the following code returns sol = [0., 0., 0. ,0. ,1.] and Objective: 1.99419 as a solution. Which is far from optimal, I'll explain why below.
import numpy as np
from gekko import GEKKO
n = 5
m = GEKKO(remote=False)
m.options.SOLVER = 1
m.options.IMODE = 3
x = [m.Var(lb=0, ub=1) for _ in range(n)]
m.Equation(m.sum(x) == 1)
np.random.seed(0)
Q = np.random.uniform(-1, 1, size=(n, n))
Q = np.dot(Q.T, Q)
## Add h_i^p
c, p = 1e-10, 0.1
for i in range(n):
m.Obj(c * x[i] ** p)
for j in range(n):
m.Obj(x[i] * Q[i, j] * x[j])
m.solve(disp=True)
sol = np.array(x).flatten()
This is clearly wrong since if we only optimize the quadratic part (x'Qx) using below code, and put the solution to the initial objective, we get a much smaller objective value (Objective: 0.02489503). The 1e-10 * sum_{i=1}^n x_i^p is esentially ignored since it is very small.
m1 = GEKKO(remote=False)
m1.options.SOLVER = 1
m1.options.OTOL = 1e-10
x1 = [m1.Var(lb=0, ub=1) for _ in range(n)]
m1.Equation(m1.sum(x1) == 1)
m1.qobj(b=np.zeros(n), A=2 * Q, x=x1, otype='min')
m1.solve(disp=True)
sol = np.array(x1).flatten()
Is there any way to resolve this? Thank you!

Gekko solves nonlinear programming optimization problems with gradient-based methods: interior point and active set SQP. It looks like there is a problem with the objective function. Use matrix operations in Numpy to simplify the objective definition.
## Create Objective
c, p = 1e-10, 0.1
obj = np.dot(np.dot(x,Q),x) + c*m.sum([xi**p for xi in x])
m.Minimize(obj)
Here is the modified script that solves with Gekko. Increase MAX_ITER if the default limit of 250 is reached.
import numpy as np
from gekko import GEKKO
n = 5
m = GEKKO(remote=False)
m.options.SOLVER = 3
m.options.IMODE = 3
x = m.Array(m.Var,n,value=0.1, lb=1e-6, ub=1)
m.Equation(m.sum(x) == 1)
np.random.seed(0)
Q = np.random.uniform(-1, 1, size=(n, n))
Q = np.dot(Q.T, Q)
print(Q)
## Create Objective
c, p = 1e-10, 0.1
obj = np.dot(np.dot(x,Q),x) + c*m.sum([xi**p for xi in x])
m.Minimize(obj)
# adjust solver tolerance
m.options.RTOL=1e-10
m.options.OTOL=1e-10
m.options.MAX_ITER = 1000
m.solve(disp=True)
sol = np.array(x).flatten()
print('x: ', sol)
print('obj: ', m.options.OBJFCNVAL)
This gives an optimal solution that is also global because it is a Quadratic Programming (QP) problem (convex optimization). Using a nonlinear programming (SQP) solver for QP problems gives a solution with the IPOPT solver:
x: [[0.36315827507] [0.081993130341] [1e-06] [0.086231281612] [0.46861632269]]
obj: 0.024895918696

As far as I could see, gekko looks like it's built for machine learning, which focuses on local optimization opposed to global optimization, and typically most libraries will not be able to guarantee you optimal solutions.
If you really want optimal solutions, than for this case I would suggest looking into interval arithmetic. There are packages such as mpmath which can offer this, though I have yet to see optimizers using it in my brief time searching.
The TL;DR on how interval arithmetic works is you feed in a range of inputs and get back a range of outputs. For example, you can test if 1 is in the range of possible outputs for x1 + x2 + x3 + x4, and you can see the minimum/maximum potential values for your objective function. In this way, you can progressively split your intervals in half, keeping only intervals for which your constraints are potentially satisfied and for which your objective function's maximum potential is at least the largest minimum potential. This allows you to achieve guaranteed convergence to global optimums at the cost of a lot more computation.

CVXPY numerical instability

I noticed that if you add to your target something multiplied by zero it will affect the solution significantly. I found this bug when using cvxpy on my dataset which I cant upload of course. But here is an example from their official resource: https://www.cvxpy.org/examples/basic/sdp.html
import cvxpy as cp
import numpy as np
# Generate a random SDP.
n = 3
p = 3
np.random.seed(1)
C = np.random.randn(n, n)
A = []
b = []
for i in range(p):
A.append(np.random.randn(n, n))
b.append(np.random.randn())
# Define and solve the CVXPY problem.
# Create a symmetric matrix variable.
X = cp.Variable((n,n), symmetric=True)
# The operator >> denotes matrix inequality.
constraints = [X >> 0]
constraints += [
cp.trace(A[i] # X) == b[i] for i in range(p)]
prob = cp.Problem(cp.Minimize(cp.trace(C # X)),
constraints)
prob.solve()
# Print result.
print("The optimal value is", prob.value)
print("A solution X is")
print(X.value)
Now do the same but change
prob = cp.Problem(cp.Minimize(cp.trace(C # X) + 0*cp.trace(C # X)**9),
constraints)
You will see that value of X will change.

I am newbie in python and doing coding for my physics project which requires to generate a matrix with a variable E

I am newbie in python and doing coding for my physics project which requires to generate a matrix with a variable E for which first element of the matrix has to be solved. Please help me. Thanks in advance.
Here is the part of code
import numpy as np
import pylab as pl
import math
import cmath
import sympy as sy
from scipy.optimize import fsolve
#Constants(Values at temp 10K)
hbar = 1.055E-34
m0=9.1095E-31 #free mass of electron
q= 1.602E-19
v = [0.510,0,0.510] # conduction band offset in eV
m1= 0.043 #effective mass in In_0.53Ga_0.47As
m2 = 0.072 #effective mass in Al_0.48In_0.52As
d = [-math.inf,100,math.inf] # dimension of structure in nanometers
'''scaling factor to with units of E in eV, mass in terms of free mass of electron, length in terms
of nanometers '''
s = (2*q*m0*1E-18)/(hbar)**2
#print('scaling factor is ',s)
E = sy.symbols('E') #Suppose energy of incoming particle is 0.3eV
m = [0.043,0.072,0.043] #effective mass of electrons in layers
for i in range(3):
print ('Effective mass of e in layer', i ,'is', m[i])
k=[ ] #Defining an array for wavevectors in different layers
for i in range(3):
k.append(sy.sqrt(s*m[i]*(E-v[i])))
print('Wave vector in layer',i,'is',k[i])
x = []
for i in range(2):
x.append((k[i+1]*m[i])/(k[i]*m[i+1]))
# print(x[i])
#Define Boundary condition matrix for two interfaces.
D0 = (1/2)*sy.Matrix([[1+x[0],1-x[0]], [1-x[0], 1+x[0]]], dtype = complex)
#print(D0)
#A = sy.matrix2numpy(D0,dtype=complex)
D1 = (1/2)*sy.Matrix([[1+x[1],1-x[1]], [1-x[1], 1+x[1]]], dtype = complex)
#print(D1)
#a=eye(3,3)
#print(a)
#Define Propagation matrix for 2nd layer or quantum well
#print(d[1])
#print(k[1])
P1 = 1*sy.Matrix([[sy.exp(-1j*k[1]*d[1]), 0],[0, sy.exp(1j*k[1]*d[1])]], dtype = complex)
#print(P1)
print("abs")
T= D0*P1*D1
#print('Transfer Matrix is given by:',T)
#print('Dimension of tranfer matrix T is' ,T.shape)
#print(T[0,0]
# I want to solve T{0,0} = 0 equation for E
def f(x):
return T[0,0]
x0= 0.5 #intial guess
x = fsolve(f, x0)
print("E is",x)
'''
y=sy.Eq(T[0,0],0)
z=sy.solve(y,E)
print('z',z)
'''
**The main part i guess is the part of the code where i am trying to solve the equation.***Steps I am following:
Defining a symbol E by using sympy
Generating three matrices which involves sum formulae and with variable E
Generating a matrix T my multiplying those 3 matrices,note that elements are complex and involves square roots of negative number.
I need to solve first element of this matrix T[0,0]=0,for variable E and find out value of E. I used fsolve for soving T[0,0]=0.*

Just a note for future questions, please leave out unused imports such as numpy and leave out zombie code like # a = eye(3,3). This helps keep the code as clean and short as possible. Also, the sample code would not run because of indentation problems, so when you copy and paste code, make sure it works before you do so. Always try to make your questions as short and modular as possible.
The expression of T[0,0] is too complex to solve analytically by SymPy so numerical approximation is needed. This leaves 2 options:
using SciPy's solvers which are advanced but require type casting to float values since SciPy does not deal with SymPy objects in any way.
using SymPy's root solvers which are less advanced but are probably simpler to use.
Both of these will only ever produce a single number as output since you can't expect numeric solvers to find every root. If you wanted to find more than one, then I advise that you use a list of points that you want to use as initial values, input each of them into the solvers and keep track of the distinct outputs. This will however never guarantee that you have obtained every root.
Only mix SciPy and SymPy if you are comfortable using both with no problems. SciPy doesn't play at all with SymPy and you should only have list, float, and complex instances when working with SciPy.
import math
import sympy as sy
from scipy.optimize import newton
# Constants(Values at temp 10K)
hbar = 1.055E-34
m0 = 9.1095E-31 # free mass of electron
q = 1.602E-19
v = [0.510, 0, 0.510] # conduction band offset in eV
m1 = 0.043 # effective mass in In_0.53Ga_0.47As
m2 = 0.072 # effective mass in Al_0.48In_0.52As
d = [-math.inf, 100, math.inf] # dimension of structure in nanometers
'''scaling factor to with units of E in eV, mass in terms of free mass of electron, length in terms
of nanometers '''
s = (2 * q * m0 * 1E-18) / hbar ** 2
E = sy.symbols('E') # Suppose energy of incoming particle is 0.3eV
m = [0.043, 0.072, 0.043] # effective mass of electrons in layers
for i in range(3):
print('Effective mass of e in layer', i, 'is', m[i])
k = [] # Defining an array for wavevectors in different layers
for i in range(3):
k.append(sy.sqrt(s * m[i] * (E - v[i])))
print('Wave vector in layer', i, 'is', k[i])
x = []
for i in range(2):
x.append((k[i + 1] * m[i]) / (k[i] * m[i + 1]))
# Define Boundary condition matrix for two interfaces.
D0 = (1 / 2) * sy.Matrix([[1 + x[0], 1 - x[0]], [1 - x[0], 1 + x[0]]], dtype=complex)
D1 = (1 / 2) * sy.Matrix([[1 + x[1], 1 - x[1]], [1 - x[1], 1 + x[1]]], dtype=complex)
# Define Propagation matrix for 2nd layer or quantum well
P1 = 1 * sy.Matrix([[sy.exp(-1j * k[1] * d[1]), 0], [0, sy.exp(1j * k[1] * d[1])]], dtype=complex)
print("abs")
T = D0 * P1 * D1
# did not converge for 0.5
x0 = 0.75
# method 1:
def f(e):
# evaluate T[0,0] at e and remove all sympy related things.
result = complex(T[0, 0].replace(E, e))
return result
solution1 = newton(f, x0)
print(solution1)
# method 2:
solution2 = sy.nsolve(T[0,0], E, x0)
print(solution2)
This prints:
(0.7533104353644469-0.023775286117722193j)
1.00808496181754 - 0.0444042144405285*I
Note that the first line is a native Python complex instance while the second is an instance of SymPy's complex number. One can convert the second simply with print(complex(solution2)).
Now, you'll notice that they produce different numbers but both are correct. This function seems to have a lot of zeros as can be shown from the Geogebra plot:
The red axis is Re(E), green is Im(E) and blue is |T[0,0]|. Each of those "spikes" are probably zeros.

Test for conditional independence in python as part of the PC algorithm

I'm implementing the PC algorithm in python. Such algorithm constructs the graphical model of a n-variate gaussian distribution. This graphical model is basically the skeleton of a directed acyclic graph, which means that if a structure like:
(x1)---(x2)---(x3)
Is in the graph, then x1 is independent by x3 given x2. More generally if A is the adjacency matrix of the graph and A(i,j)=A(j,i) = 0 (there is a missing edge between i and j) then i and j are conditionally independent, by all the variables that appear in any path from i to j. For statistical and machine learning purposes, it is be possible to "learn" the underlying graphical model.
If we have enough observations of a jointly gaussian n-variate random variable we could use the PC algorithm that works as follows:
given n as the number of variables observed, initialize the graph as G=K(n)
for each pair i,j of nodes:
if exists an edge e from i to j:
look for the neighbours of i
if j is in neighbours of i then remove j from the set of neighbours
call the set of neighbours k
TEST if i and j are independent given the set k, if TRUE:
remove the edge e from i to j
This algorithm computes also the separating set of the graph, that are used by another algorithm that constructs the dag starting from the skeleton and the separation set returned by the pc algorithm. This is what i've done so far:
def _core_pc_algorithm(a,sigma_inverse):
l = 0
N = len(sigma_inverse[0])
n = range(N)
sep_set = [ [set() for i in n] for j in n]
act_g = complete(N)
z = lambda m,i,j : -m[i][j]/((m[i][i]*m[j][j])**0.5)
while l<N:
for (i,j) in itertools.permutations(n,2):
adjacents_of_i = adj(i,act_g)
if j not in adjacents_of_i:
continue
else:
adjacents_of_i.remove(j)
if len(adjacents_of_i) >=l:
for k in itertools.combinations(adjacents_of_i,l):
if N-len(k)-3 < 0:
return (act_g,sep_set)
if test(sigma_inverse,z,i,j,l,a,k):
act_g[i][j] = 0
act_g[j][i] = 0
sep_set[i][j] |= set(k)
sep_set[j][i] |= set(k)
l = l + 1
return (act_g,sep_set)
a is the tuning-parameter alpha with which i will test for conditional independence, and sigma_inverse is the inverse of the covariance matrix of the sampled observations. Moreover, my test is:
def test(sigma_inverse,z,i,j,l,a,k):
def erfinv(x): #used to approximate the inverse of a gaussian cumulative density function
sgn = 1
a = 0.147
PI = numpy.pi
if x<0:
sgn = -1
temp = 2/(PI*a) + numpy.log(1-x**2)/2
add_1 = temp**2
add_2 = numpy.log(1-x**2)/a
add_3 = temp
rt1 = (add_1-add_2)**0.5
rtarg = rt1 - add_3
return sgn*(rtarg**0.5)
def indep_test_ijK(K): #compute partial correlation of i and j given ONE conditioning variable K
part_corr_coeff_ij = z(sigma_inverse,i,j) #this gives the partial correlation coefficient of i and j
part_corr_coeff_iK = z(sigma_inverse,i,K) #this gives the partial correlation coefficient of i and k
part_corr_coeff_jK = z(sigma_inverse,j,K) #this gives the partial correlation coefficient of j and k
part_corr_coeff_ijK = (part_corr_coeff_ij - part_corr_coeff_iK*part_corr_coeff_jK)/((((1-part_corr_coeff_iK**2))**0.5) * (((1-part_corr_coeff_jK**2))**0.5)) #this gives the partial correlation coefficient of i and j given K
return part_corr_coeff_ijK == 0 #i independent from j given K if partial_correlation(i,k)|K == 0 (under jointly gaussian assumption) [could check if abs is < alpha?]
def indep_test():
n = len(sigma_inverse[0])
phi = lambda p : (2**0.5)*erfinv(2*p-1)
root = (n-len(k)-3)**0.5
return root*abs(z(sigma_inverse,i,j)) <= phi(1-a/2)
if l == 0:
return z(sigma_inverse,i,j) == 0 #i independent from j <=> partial_correlation(i,j) == 0 (under jointly gaussian assumption) [could check if abs is < alpha?]
elif l == 1:
return indep_test_ijK(k[0])
elif l == 2:
return indep_test_ijK(k[0]) and indep_test_ijK(k[1]) #ASSUMING THAT IJ ARE INDEPENDENT GIVEN Y,Z <=> IJ INDEPENDENT GIVEN Y AND IJ INDEPENDENT GIVEN Z
else: #i have to use the independent test with the z-fisher function
return indep_test()
Where z is a lambda that receives a matrix (the inverse of the covariance matrix), an integer i, an integer j and it computes the partial correlation of i and j given all the rest of variables with the following rule (which I read in my teacher's slides):
corr(i,j)|REST = -var^-1(i,j)/sqrt(var^-1(i,i)*var^-1(j,j))
The main core of this application is the indep_test() function:
def indep_test():
n = len(sigma_inverse[0])
phi = lambda p : (2**0.5)*erfinv(2*p-1)
root = (n-len(k)-3)**0.5
return root*abs(z(sigma_inverse,i,j)) <= phi(1-a/2)
This function implements a statistical test which uses the fisher's z-transform of estimated partial correlations. I am using this algorithm in two ways:
Generate data from a linear regression model and compare the learned DAG with the expected one
Read a dataset and learn the underlying DAG
In both cases i do not always get correct results, either because I know the DAG underlying a certain dataset, or because i know the generative model but it does not coincide with the one my algorithm learns. I perfectly know that this is a non-trivial task and I may have misunderstand theoretical concept as well as committed error even in parts of the code i have omitted here; but first i'd like to know (from someone who is more experienced than me), if the test i wrote is right, and also if there are library functions that perform this kind of tests, i tried searching but i couldn't find any suitable function.

I get to the point. The most critical issue in the above code, regards the following error:
sqrt(n-len(k)-3)*abs(z(sigma_inverse[i][j])) <= phi(1-alpha/2)
I was mistaking the mean of n, it is not the size of the precision matrix but the number of total multi-variate observations (in my case, 10000 instead of 5). Another wrong assumption is that z(sigma_inverse[i][j]) has to provide the partial correlation of i and j given all the rest. That's not correct, z is the Fisher's transform on a proper subset of the precision matrix which estimates the partial correlation of i and j given the K. The correct test is the following:
if len(K) == 0: #CM is the correlation matrix, we have no variables conditioning (K has 0 length)
r = CM[i, j] #r is the partial correlation of i and j
elif len(K) == 1: #we have one variable conditioning, not very different from the previous version except for the fact that i have not to compute the correlations matrix since i start from it, and pandas provide such a feature on a DataFrame
r = (CM[i, j] - CM[i, K] * CM[j, K]) / math.sqrt((1 - math.pow(CM[j, K], 2)) * (1 - math.pow(CM[i, K], 2))) #r is the partial correlation of i and j given K
else: #more than one conditioning variable
CM_SUBSET = CM[np.ix_([i]+[j]+K, [i]+[j]+K)] #subset of the correlation matrix i'm looking for
PM_SUBSET = np.linalg.pinv(CM_SUBSET) #constructing the precision matrix of the given subset
r = -1 * PM_SUBSET[0, 1] / math.sqrt(abs(PM_SUBSET[0, 0] * PM_SUBSET[1, 1]))
r = min(0.999999, max(-0.999999,r))
res = math.sqrt(n - len(K) - 3) * 0.5 * math.log1p((2*r)/(1-r)) #estimating partial correlation with fisher's transofrmation
return 2 * (1 - norm.cdf(abs(res))) #obtaining p-value
I hope someone could find this helpful

scipy.optimize solution using python for the following equation

I am very new to scipy and doing data analysis in python. I am trying to solve the following regularized optimization problem and unfortunately I haven't been able to make too much sense from the scipy documentation. I am looking to solve the following constrained optimization problem using scipy.optimize
Here is the function I am looking to minimize:
here A is an m X n matrix , the first term in the minimization is the residual sum of squares, the second is the matrix frobenius (L2 norm) of a sparse n X n matrix W, and the third one is an L1 norm of the same matrix W.
In the function A is an m X n matrix , the first term in the minimization is the residual sum of squares, the second term is the matrix frobenius (L2 norm) of a sparse n X n matrix W, and the third one is an L1 norm of the same matrix W.
I would like to know how to minimize this function subject to the constraints that:
wj >= 0
wj,j = 0
I would like to use coordinate descent (or any other method that scipy.optimize provides) to solve the above problem. I would like so direction on how to achieve this as I have no idea how to take the frobenius norm or how to tune the parameters beta and lambda or whether the scipy.optimize will tune and return the parameters for me. Any help regarding these questions would be much appreciated.
Thanks in advance!

How large is m and n?
Here is a basic example for how to use fmin:
from scipy import optimize
import numpy as np
m = 5
n = 3
a = np.random.rand(m, n)
idx = np.arange(n)
def func(w, beta, lam):
w = w.reshape(n, n)
w2 = np.abs(w)
w2[idx, idx] = 0
return 0.5*((a - np.dot(a, w2))**2).sum() + lam*w2.sum() + 0.5*beta*(w2**2).sum()
w = optimize.fmin(func, np.random.rand(n*n), args=(0.1, 0.2))
w = w.reshape(n, n)
w[idx, idx] = 0
w = np.abs(w)
print w
If you want to use coordinate descent, you can implement it by theano.
http://deeplearning.net/software/theano/

Your problem seems tailor-made for cvxopt - http://cvxopt.org/
and in particular
http://cvxopt.org/userguide/solvers.html#problems-with-nonlinear-objectives
using fmin would likely be slower, since it does not take advantage of gradient / Hessian information.
The code in HYRY's answer also has the drawback that as far as fmin is concerned the diagonal W is a variable and fmin would try to move the W-diagonal values around until it realizes that they don't do anything (since the objective function resets them to zero). Here is the implementation in cvxopt of HYRY's code that explicitly enforces the zero-constraints and uses gradient info, WARNING: I couldn't derive the Hessian for your objective... and you might double-check the gradient as well:
'''CVXOPT version:'''
from numpy import *
from cvxopt import matrix, mul
''' warning: CVXOPT uses column-major order (Fortran) '''
m = 5
n = 3
n_active = (n)*(n-1)
A = matrix(random.rand(m*n),(m,n))
ids = arange(n)
beta = 0.1;
lam = 0.2;
W = matrix(zeros(n*n), (n,n));
def cvx_objective_func(w=None, z=None):
if w is None:
num_nonlinear_constraints = 0;
w_0 = matrix(1, (n_active,1), 'd');
return num_nonlinear_constraints, w_0
#main call:
'calculate objective:'
'form W matrix, warning _w is column-major order (Fortran)'
'''column-major order!'''
_w = matrix(w, (n, n-1))
for k in xrange(n):
W[k, 0:k] = _w[k, 0:k]
W[k, k+1:n] = _w[k, k:n-1]
squared_error = A - A*W
objective_value = .5 * sum( mul(squared_error,squared_error)) +\
.5* beta*sum(mul(W,W)) +\
lam * sum(abs(W));
'not sure if i calculated this right...'
_Df = -A.T*(squared_error) + beta*W + lam;
'''column-major order!'''
Df = matrix(0., (1, n*(n-1)))
for jdx in arange(n):
for idx in list(arange(0,jdx)) + list(arange(jdx+1,n)):
idx = int(idx);
jdx = int(jdx)
Df[0, jdx*(n-1) + idx] = _Df[idx, jdx]
if z is None:
return objective_value, Df
'''Also form hessian of objective+non-linear constraints
(but there are no nonlinear constraints) :
This is the trickiest part...
WARNING: H is for sure coded wrong'''
H = matrix(1., (n_active, n_active))
return objective_value, Df, H
m, w_0 = cvx_objective_func()
print cvx_objective_func(w_0)
G = -matrix(diag(ones(n_active),), (n_active,n_active))
h = matrix(0., (n_active,1), 'd')
from cvxopt import solvers
print solvers.cp(cvx_objective_func, G=G, h=h)
having said that, the tricks to eliminate the equality/inequality constraints in HYRY's code are quite cute