I have this kind of data :
import random
data=random.sample(range(1, 100), 5)
x= [-1,1,1,-1,1]
def f(x,data):
prod=[a * b for a, b in zip(data, x)]
result=abs(sum(prod))
return result
I Would like to find the best x composed of -1 or 1 to minimize the value of f(x)
Maybe we can use scipy.minimise() but how can we add the -1 or 1 as a constrain on the value inside of x ?
Does somebody have an idea ?
You want to solve a mixed-integer linear programming problem (MILP), which aren't supported yet by scipy.optimize.
However, you can use a modelling package like PuLP to formulate your MILP and pass it to a MILP solver. Note that your MIP can be formulated as
(P)
min |f(x)| = |d_0 * x_0 + ... + d_n * x_n|
s.t. x_i ∈ {-1, 1} ∀ i = 0,...,n
which is the same as
(P')
min |f(x)| = |d_0 * (2*x_0 - 1) + ... + d_n * (2*x_n - 1)|
s.t. x_i ∈ {0, 1} ∀ i = 0,...,n
and can be implemented like this
min abs_obj
s.t. f(x) <= abs_obj
f(x) >= -1.0*abs_obj
x_i ∈ {0, 1} ∀ i = 0,...,n
In code:
import pulp
import random
data = random.sample(range(1, 100), 5)
# pulp model
mdl = pulp.LpProblem("our_model", sense=pulp.LpMinimize)
# the binary variables x
x = pulp.LpVariable.dicts("x", range(5), cat="Binary")
# the variable that stores the absolute value of the objective
abs_obj = pulp.LpVariable("abs_obj")
# set the MIP objective
mdl += abs_obj
# Define the objective: |f(x)| = abs_obj
mdl += pulp.lpSum((2 * x[i] - 1) * data[i] for i in range(5)) <= abs_obj
mdl += pulp.lpSum((2 * x[i] - 1) * data[i] for i in range(5)) >= -1.0*abs_obj
# solve the problem
mdl.solve()
# your solution
signs = [1 if var.varValue > 0 else -1 for var in x.values()]
Alternatively, if you don't want to use another package, you can use scipy.optimize.minimize and implement a simple penalty method. Thereby you solve the problem (P') by solving the penalty problem
min |f(x)| + Ɛ * (x_0 * (1 - x_0) + ... + x_n * (1 - x_n))
with 0 <= x_i <= 1
where Ɛ is a given penalty parameter. Here, the idea is that the right penalty term equals zero for an integer solution.
Note that as the case may be that you need to solve a sequence of penalty problems to achieve convergence to an integer solution. Thus, I'd highly recommend sticking to a MILP solver instead of implementing a penalty method on your own.
Yes, you can do it using scipy.optimize.minimize:
from scipy.optimize import minimize
minimize(f, [0] * len(data), args=data, bounds=[(-1, 1)] * len(data))
This call minimizes f which you defined in the original post.
It passes a zero array as an initial guess for the minimization problem.
The argument f requires is 'data' which is specified by the argument 'args'.
The constraints you want are specified by the argument 'bounds' as a list of min/max tuples with the length of the input data.
Related
I have a constrained optimization problem where I am trying to minimize an objective function of 100+ variables which is of the form
Min F(x) = f(x1) + f(x2) + ... + f(xn)
Subject to functional constraint
(g(x1) + g(x2) + ... + g(xn))/(f(x1) + f(x2) + ... + f(xn)) - constant >= 0
I also have individual bounds for each variable x1, x2, x3...xn
a <= x1 <= b
c <= x2 <= d
...
For this, I wrote a python script, using the scipy.optimize.minimize implementation with constraints and bounds, but I am unable to fulfill my bounds and constraints in the solutions. These are all cases where optimization could converge to a solution (message: success)
Here is a sample of my code:
df is my pandas dataset
B(x) is LogNorm transform based on x and other constants
Values U, c, lb, ub are pre-calculated constant dictionaries for each index in df
import scipy
df = pd.DataFrame(..)
k = set(df.index.values) ## list of indexes to iterate on
val = 0.25 ## Arbitrary
def obj(x):
fn = 0
for n,i in enumerate(k):
x0 = x[n]
fn1 = (U[i]) * B(x0) * (x0)
fn += fn1
return fn
def cons(x):
cn = 1
c1 = 0
c2 = 0
for n,i in enumerate(k):
x0 = x[n]
c1 += (U[i]) * (B(x0) * (x0 - c[i])
c2 += (U[i]) * (B(x0) * (x0)
cn = c1/(c2)
return cn - val
const = [{'type':'ineq', 'fun':cons}]
bnds = tuple((lb[i], ub[i]) for i in k) ## Lower, Upper for each element ((lb1, ub1), (lb2, ub2)...)
x_init = [lb[i] for i in k] ## for eg. starting from lower bound
## Solution
sol = scipy.optimize.minimize(obj, x_init, method = 'COBYLA', bounds = bnds, constraints = const)
I have more pointed questions if that helps:
Is there a way to construct the same equation concisely/ without the use of loops (given the number of variables could depend on input data and I have no control over it)?
Is there any noticeable issue in my application of bounds? I can't seem to get the final values of all variables follow individual bounds.
Similarly, is there a visible flaw in the construction on constraint equation? My results often DO NOT follow the constraints is repeated runs with different inputs.
Any help with either of the questions can help me progress further at work.
I have also looked into a Lagrangian solution of the same but so far I am unable to solve it for undefined number of (n) variables.
Thanks!
Please, consider the following optimisation problem. Specifically, x and b are (1,n) vectors, C is (n,n) symmetric matrix, k is an arbitrary constant and i is a (1,n) vector of ones.
Please, also consider the following equivalent optimisation problem. In such case, k is determined during the optimisation process so there is no need to scale the values in x to obtain the solution y.
Please, also consider the following code for solving both the problems with cvxpy.
import cvxpy as cp
import numpy as np
def problem_1(C):
n, t = np.shape(C)
x = cp.Variable(n)
b = np.array([1 / n] * n)
obj = cp.quad_form(x, C)
constraints = [b.T # cp.log(x)>=0.5, x >= 0]
cp.Problem(cp.Minimize(obj), constraints).solve()
return (x.value / (np.ones(n).T # x.value))
def problem_2(C):
n, t = np.shape(C)
y = cp.Variable(n)
k = cp.Variable()
b = np.array([1 / n] * n)
obj = cp.quad_form(y, C)
constraints = [b.T # cp.log(y)>=k, np.ones(n)#y.T==1, y >= 0]
cp.Problem(cp.Minimize(obj), constraints).solve()
return y.value
While the first function do provide me with the correct solution for a sample set of data I am using, the second does not. Specifically, values in y differ heavily while employing the second function with some of them being equal to zero (which cannot be since all values in b are positive and greater than zero). I am wondering wether or not the second function minimise also k. Its value should not be minimised on the contrary it should just be determined during the optimisation problem as the one that leads to the solution that minimise the objective function.
UPDATE_1
I just found that the solution that I obtain with the second formulation of the problem is equal to the one derived with the following equations and function. It appears that the constraint with the logarithmic barrier and the k variable is ignored.
def problem_3(C):
n, t = np.shape(C)
y = cp.Variable(n)
k = cp.Variable()
b = np.array([1 / n] * n)
obj = cp.quad_form(y, C)
constraints = [np.ones(n)#y.T==1, y >= 0]
cp.Problem(cp.Minimize(obj), constraints).solve()
return y.value
UPDATE_2
Here is the link to a sample input C - https://www.dropbox.com/s/kaa7voufzk5k9qt/matrix_.csv?dl=0. In such case the correct output for both problem_1 and problem_2 is approximately equal to [0.0659 0.068 0.0371 0.1188 0.1647 0.3387 0.1315 0.0311 0.0441] since they are equivalent by definition. I am able to obtain the the correct output by solving only problem_1. Solving problem_2 leads to [0.0227 0. 0. 0.3095 0.3392 0.3286 0. 0. 0. ] which is wrong since it happens to be the correct output for problem_3.
UPDATE_3
To be clear, by definition problem_2 exhibits solution equal to the solution of problem_3 when the parameter k goes to minus infinity.
UPDATE_4
Please consider the following code that is for solving problem_1 using SciPy Optimize instead CVXPY. By imposing k=9 the correct optimal solution can still be achieved which is consistent with problem_1 being independent of the parameter.
import scipy.optimize as opt
def obj(x, C):
return x.T # C # x
def problem_1_1(C):
n, t = np.shape(C)
b = np.array([1 / n] * n)
constraints = [{"type": "eq", "fun": lambda x: (b * np.log(x)).sum() - 9}]
res = opt.minimize(
obj,
x0 = np.array([1 / n] * n),
args = (C),
bounds = ((0, None),) * n,
constraints = constraints
)
return (res['x'] / (np.ones(n).T # res['x']))
UPDATE_5
By considering the code in UPDATE_4, whenever k is set equal to 10 the correct solution is still achieved however appears the following warning. I suppose that is due to rounding error that might occur during the optimisation process.
Untitled.py:56: RuntimeWarning: divide by zero encountered in
log {"type": "eq", "fun": lambda x: (b * np.log(x)).sum() - 10}
I am wondering if there is a way to impose strict inequality constraint with CVXPY or apply a condition on the logarithm argument. Please consider the following modified code for problem_1_1.
import scipy.optimize as opt
def obj(x, C):
return x.T # C # x
def problem_1_1(C):
n, t = np.shape(C)
b = np.array([1 / n] * n)
constraints = [{"type": "eq", "fun": lambda x: (b * np.log(x if x.all() > 0 else 1e-100)).sum() - 10}]
res = opt.minimize(
obj,
x0 = np.array([1 / n] * n),
args = (C),
bounds = ((0, None),) * n,
constraints = constraints
)
return (res['x'] / (np.ones(n).T # res['x']))
UPDATE_6
To be thorough, the correct value of optimal k is approximatively -2.4827186402337564.
If you let be arbitrary then you are basically saying that is greater or equal to some arbitrary number, which is trivially true, so the constraint becomes irrelevant.
I believe you should either fix the value of or turn this problem into a minimax problem by determining a tadeoff betweenmaximizing and minimizing .
My problem: Iterating over Gurobi "Var" decision variable creates TypeError: '<' not supported between instances of 'Var' and 'int' and issue with exponentation (i.e. **/ pow())
Key information on the Gurobi optimization:
Objective function: for three items maximize the sum of (price * expected value)
Expected value is calculated through two defined formulas:
1) PDF = probability density function
2) EV = Expected value which is the integration of the PDF over a
specific integral
The decision variable "upperBound" is supposed to maximize the upper bound of this integral, the lower bound is 0
Below the model:
from gurobipy import *
import pandas as pd, numpy as np, scipy.integrate as integrate
import math
mu = pd.DataFrame(np.array([10, 15]), index = ["product1", "product2"])
sigma = pd.DataFrame(np.array([1, 1.5]), index = mu.index.values)
price = pd.Series(np.array([10, 10]), index = mu.index.values)
m = Model("maxUpperBound")
ub = m.addVars(mu.index.values, vtype=GRB.INTEGER, name="upperBound")
def PDF (y, mu, sigma):
return y * (1.0 / (sigma * (2.0 * math.pi)**(1/2)) * math.exp(-1.0 * (y - mu)**2 / (2.0 * (sigma**2))))
def EV(ub, mu, sigma):
lb = 0
value, error = integrate.quad(PDF, lb, ub, args=(mu, sigma))
return value
m.setObjective(
quicksum(
price.loc[i] * EV(ub[i], mu.loc[i], sigma.loc[i])
for i in mu.index.values
),
GRB.MAXIMIZE)
m.addConstr(
(quicksum(ub[i]
for i in mu.index.values)
<= 100),
"Limit100"
)
m.optimize()
for v in m.getVars():
print(v.varName, v.x)
Your code is a bit hard to read but these lines likely cause the error:
ub = m.addVars(mu.index.values, vtype=GRB.INTEGER, name="upperBound")
...
for ub in ub[i]:
ub is a tupledict and ub[i] would refer to one Var object. You cannot iterate over a single Var.
I guess you want to write something like this:
for ub_i in ub:
EDIT
The problem has been solved over in the Gurobi forum:
tldr: The objective function is too complex and cannot be handled by Gurobi in this form.
Gurobi is not able to optimize with respect to the PDF function as this is not a mixed-integer problems comprised of linear, (convex or nonconvex) quadratic, or second-order constraints/objective.
Instead solved this by calculating the expected value upfront for each product_ub combination. ub can takes values between 0 and 100 (see constraint). Used the expected value thereafter in the objective function.
I was trying to resolve a LP problem with a constraint that is calculated by dividing variable A by variable B.
The simple version of the problem is as below:
The product is made by two materials (A and B)
% of A should be greater than 50%
% of B should be less than 40%
Total amount of A and B are 100
Objective: What's the minimum amount of A?
The code is like:
from pulp import *
prob = LpProblem('Simple problem', LpMinimize)
x = LpVariable('x', 0, None, 'Integer')
y = LpVariable('y', 0, None, 'Integer')
prob += x
prob += x / (x + y) > 0.5 # <== Where the error happens
prob += y / (x + y) < 0.4
prob += x + y == 100
prob.solve()
print 'Result: %s' % LpStatus[prob.status]
print 'Amount of A: %s' % value(prob.objective)
However I'm getting an error message saying:
TypeError: Expressions cannot be divided by a non-constant expression
It looks like PuLP does not support variable as divisor.
https://github.com/coin-or/pulp/blob/master/src/pulp/pulp.py#L800
Any idea? If PuLP is not the right library to use, I'm happy to switch to any library that fits in.
UPDATE 27 Nov 2015
For some reason, the sample above does not make sense (not working as expected). I am very newbie to this library. Maybe it's not the right one to solve my problem at all. So if any one has suggestions of other libraries, it'd be appreciated.
BTW, Koen Peters's advice below is great. The error is gone after taking his advice. Thank you.
Linear Programming doesn't understand divisions, hence the error :)
You have to reformulate it so that the division is formulated linearly.
In this case:
prob += x / (x + y) > 0.5
prob += y / (x + y) < 0.4
is equivalent to:
prob += x > 0.5 * (x + y)
prob += y < 0.4 * (x + y)
Which are linear constraints.
Good luck!
I felt like zero shouldn't be allowed in my solution — and I included a variable that was the sum of x and y (think you're referring to it as A).
from pulp import LpProblem, LpStatus, LpVariable
from pulp import LpMinimize, LpMaximize, lpSum, value
# I feel like zero shouldn't be allowed for lower bound...
x = LpVariable("x", lowBound=1, cat="Integer")
y = LpVariable("y", lowBound=1, cat="Integer")
z = LpVariable("z", lowBound=1, cat="Integer")
model = LpProblem("Divison problem", LpMinimize)
model += x
model += z == x + y
model += z == 100
# Rather than division, we can use multiplication
model += x >= z * 0.5
model += y <= z * 0.4
model.solve()
print(LpStatus[model.status])
print(f"""
x = {int(x.varValue):>3} # 60
y = {int(y.varValue):>3} # 40
z = {int(z.varValue):>3} # 100
""")
I'm implementing Bayesian Changepoint Detection in Python/NumPy (if you are interested have a look at the paper). I need to compute likelihoods for data in ranges [a, b], where a and b can have all values from 1 to n. However I can prune the computation at some points, so that I don't have to compute every likelihood. On the other hand some likelihoods are used more than once, so that I can save time by saving the values in a matrix P[a, b]. Right now I check whether the value is already computed, whenever I use it, but I find that a bit of a hassle. It looks like this:
# ...
P = np.ones((n, n)) * np.inf # a likelihood can't get inf, so I use it
# as pseudo value
for a in range(n):
for b in range(a, n):
# The following two lines get annoying and error prone if you
# use P more than once
if P[a, b] == np.inf:
P[a, b] = likelihood(data, a, b)
Q[a] += P[a, b] * g[a] * Q[a - 1] # some computation using P[a, b]
# ...
I wonder, whether there is a more intuitive and pythonic way to achieve this, without having the if ... statement before every use of a P[a, b]. Something like an automagical function call if some condition is not met. I could of course make the likelihood function aware of the fact that it could save values, but then it needs some kind of state (e.g. becomes an object). I want to avoid that.
The likelihood function
Since it was asked for in a comment, I add the likelihood function. It actually computes the conjugate prior and then the likelihood. And all in log representation... So it is quite complicated.
from scipy.special import gammaln
def gaussian_obs_log_likelihood(data, t, s):
n = s - t
mean = data[t:s].sum() / n
muT = (n * mean) / (1 + n)
nuT = 1 + n
alphaT = 1 + n / 2
betaT = 1 + 0.5 * ((data[t:s] - mean) ** 2).sum() + ((n)/(1 + n)) * (mean**2 / 2)
scale = (betaT*(nuT + 1))/(alphaT * nuT)
# splitting the PDF of the student distribution up is /much/ faster. (~ factor 20)
prob = 1
for yi in data[t:s]:
prob += np.log(1 + (yi - muT)**2/(nuT * scale))
lgA = gammaln((nuT + 1) / 2) - np.log(np.sqrt(np.pi * nuT * scale)) - gammaln(nuT/2)
return n * lgA - (nuT + 1)/2 * prob
Although I work with Python 2.7, both answers for 2.7 and 3.x are appreciated.
I would use a sibling of defaultdict for this (you can't use defaultdict directly since it won't tell you the key that is missing):
class Cache(object):
def __init__(self):
self.cache = {}
def get(self, a, b):
key = (a,b)
result = self.cache.get(key, None)
if result is None:
result = likelihood(data, a, b)
self.cache[key] = result
return result
Another approach would be using a cache decorator on likelihood as described here.