Develop Reference

Pulp Integer Programming Constraint ignored

I am trying to solve a LpProblem with only boolean variables and Pulp seems to be ignoring some constraints. To give some context about the problem:
I want to find an optimal solution to the problem schools face when trying to create classroom groups. In this case, students are given a paper to write at most 5 other students and the school guarantees them that they will be together with at least one of those students. To see how I modeled this problem into an integer programming problem please refer to this question.
In that link you will see that my variables are defined as x_ij = 1 if student i will be together with student j, and x_i_j = 0 otherwise. Also, in that link I ask about the constraint that I am having trouble implementing with Pulp: if x_i_j = 1 and x_j_k = 1, then by transitive property, x_i_k = 1. In other words, if student i is with student j, and student j is with student k, then, student i will inherently be together with student k.
My objective is to maximize the sum of all the elements of the matrix obtained when performing a Hadamard product between the input matrix and the variables matrix. In other words, I want to contemplate as many of the student's requests as possible.
I will now provide some code snippets and screen captures that should help visualize the problem:
Inputs (just a sample: the real matrix is 37x37)
Output
As you can see in this last image, x_27 = 1 and x_37 = 1 but x_23 = 0 which doesn't make sense.
Here is how I define my variables
def define_variables():
variables = []
for i in range(AMOUNT_OF_STUDENTS):
row = []
for j in range(AMOUNT_OF_STUDENTS):
row.append(LpVariable(f"x_{i}_{j}", lowBound=0, upBound=1, cat='Integer'))
variables.append(row)
return variables
Here is how I define the transitive constraints
for i in range(len(variables)):
for j in range(i, len(variables)):
if i != j:
problem += variables[i][j] == variables[j][i] # Symmetry
for k in range(j, len(variables)):
if i < j < k < len(variables):
problem += variables[i][j] + variables[j][k] - variables[i][k] <= 1 # Transitive
problem += variables[i][j] + variables[i][k] - variables[j][k] <= 1
problem += variables[j][k] + variables[i][k] - variables[i][j] <= 1
When printing the LpProblem I see the constraint that is apparently not working:
As you can see in the output: x_2_7 = 1 and x_3_7 = 1. Therefore, to satisfy this constraint, x_2_3 should also be 1, but as you can also see in the output, it is 0.
Any ideas about what could be happening? I've been stuck for days and the problem seems to be modeled fine and it worked when I only had 8 students (64 variables). Now that I have 37 students (1369 variables) it seems to be behaving oddly. The solver arrives to a solution but it seems to be ignoring some constraints.
Any help is very much appreciated! Thank you in advance.

The constraint is working correctly. Find below the analysis:
(crossposted from github: https://github.com/coin-or/pulp/issues/377)
import pulp as pl
import pytups as pt
path = 'debugSolution.txt'
# import model
_vars, prob = pl.LpProblem.from_json(path)
# get all variables with non-zero value
vars_value = pt.SuperDict(_vars).vfilter(pl.value)
# export the lp
prob.writeLP('debugSolution.lp')
# the constraint you show in the SO problem is:
# _C3833: - x_2_3 + x_2_7 + x_3_7 <= 1
'x_2_7' in vars_value
# True, so x_2_7 has value 1
'x_3_7' in vars_value
# False, so x_3_7 has value 0
'x_2_3' in vars_value
# False, so x_2_3 has value 0
So -0 + 1 + 0 <= 1 means the constraint is respected. There must be a problem with bringing back the value of x_3_7 somewhere because you think is 1 when in pulp it's 0.

This is called a set partitioning problem and PuLP has an example in their documentation here.
In essence, instead of modeling your variables as indicators of whether student A is in the same class as student B, you'll define a mapping between a set of students and a set of classrooms. You can then apply your student preferences as either constraints or part of a maximization objective.

Constraining Pulp Binary Variable Selection Based on Another Variable

I'm creating an optimisation script for Fantasy Football. It starts off quite easily- loading in players & their relevant details.
The key in this game is that 15 players can be selected in your squad but only 11 can be fielded per week.
What I would like to do is have 2 variables- one defining that the player is in your squad and a sub-variable that determines whether you put the player in your starting 11.
I have tried a few things- one broad solution is that have 2 unrelated variables. 1 that selects 11 starters and a second that selects 4 subs. This works well for 1 week, but for example one week Player A from your squad might be best starting and the next he's better on the bench. Therefore I would get a more optimal solution if I can make the starting 11 variable a subset of the squad variable.
I've attached the code defining the variables and my attempt at creating a constraint that would link them together. (there are other constraints that all successfully work. For example I can pick a starting 11 or a squad of 15 to maximize expected results without issue, but I cannot pick a starting 11 within a squad of 15.
#VECTORS OF BINARY DECISIONS VARIABLES
squad_variables = []
for rownum in ID:
variable = str('x' + str(rownum))
variable = pulp.LpVariable(str(variable), lowBound = 0, upBound = 1, cat= 'Integer')
squad_variables.append(variable)
xi_variables = []
for rownum in ID:
bariable = str('y' + str(rownum))
bariable = pulp.LpVariable(str(bariable), lowBound = 0, upBound = 1, cat= 'Integer')
xi_variables.append(bariable)
The code below is not working for this task and is the root of the problem..
#ID CONSTRAINTS (ie. only 15 unique id selection across both systems)
id_usage = ""
for rownum in ID:
for i, player in enumerate(squad_variables):
if rownum == i:
formula = max(1*xi_variables[rownum],(1*player))
id_usage += formula
prob += (id_usage ==15)
Any help would be greatly appreciated- perhaps this is simply a non-linear problem. Thank you :)

You want a constraint that says "if x[i] = 0 then y[i] = 0". The typical way to do this is through the constraint y[i] <= x[i]. Note that this only works if both variables are binary; otherwise a modified approach is necessary. I can't quite follow your PuLP code so I won't try to give you the code for this constraint, but I assume you'll be able to implement it once you understand the logic.

Optimising a Python script

I've been trying to complete the below task in Python:
http://codeforces.com/problemset/problem/4/C
I created a simple script for it as can be seen below, but it returns a runtime error for the 7th test. I believe this is due to perhaps the code is taking too long, so I require assistance optimising it. I have looked at map and filter commands and tried implementing them, without success.
a=int(input())
entered_usernames=[]
n=0
while n<a:
y=input()
entered_usernames.append(y)
n+=1
valid_usernames=[]
for i in entered_usernames:
if i not in valid_usernames:
valid_usernames.append(i)
print('OK')
else:
count=1
while i+str(count) in valid_usernames:
count+=1
valid_usernames.append(i+str(count))
print(i+str(count))

You can try changing valid_usernames to a set instead of a list.
For a list list_a operation x in list_a takes (on average) linear time.
For a set set_a operation x in set_a takes (on average) constant time.
(source: https://wiki.python.org/moin/TimeComplexity)
This simple change could improve runtime a bit.
What also strikes me as potentially very slow is this fragment:
while i+str(count) in valid_usernames:
count+=1
However, if you want to improve this, you need to think about using a completely different data structure.

Why don't you use a lookup dict with a counter and solve this in O(N) time?
total = int(input()) # get the first input (total usernames)
database = {} # our 'database' / lookup dict
candidates = [input() for _ in range(total)] # pick usernames from the input
for candidate in candidates: # loop through each candidate
if candidate in database: # already used, print with a counter
print(candidate + str(database[candidate]))
database[candidate] += 1 # increase the counter
else: # the candidate doesn't exist in the 'database'...
print("OK")
database[candidate] = 1 # initialize counter for the next time

Why don't you try
valid_usernames.append(i+str(valid_usernames.count(i)))
print(i+str(valid_usernames.count(i))

How to can I assign a variable from a mini-function, limiting the scope of variables used?

Is there any way I can assign a variable from a complex calculation?
I know I can create a function and use that, but sometimes that's overkill. I would like to assign a variable from a complex calculation without declaring a bunch of local variables.
Take these statements for example, I'm only interested in using the 'total_days' variable from now on, but putting the calculation all on a long line is messy:
days_until = (self.last_date - self.first_date).days
opening_time = datetime.combine(self.last_time, EXCHANGE_OPEN_TIME)
time_open = self.last_time - opening_time
ratio = time_open.total_seconds()
/EXCHANGE_OPENING_HOURS.total_seconds()
total_days = days_until + ratio if ratio < 1 else days_until + 1
In my mind, I would like to use something like:
total_days = (
days_until = (self.last_date - self.first_date).days
time_open = self.last_time - opening_time
ratio = time_open.total_seconds()
/EXCHANGE_OPENING_HOURS.total_seconds()
return days_until + ratio if ratio < 1 else days_until + 1
)

Here is your code sample (I stripped out self for a bit of extra clarity):
days_until = (last_date - first_date).days
opening_time = datetime.combine(last_time, EXCHANGE_OPEN_TIME)
time_open = last_time - opening_time
ratio = time_open.total_seconds() / EXCHANGE_OPENING_HOURS.total_seconds()
total_days = days_until + ratio if ratio < 1 else days_until + 1
Everything is broken into short, readable lines. The result of each calculation is stored into a neat name that indicates what it is. Will you ever use the intermediate steps again? Perhaps not. But suppose a bug is found with this calculation. Can you see how by having it in discrete steps like this makes it simple to find where things go wrong?
Moreover, it's interesting that you say you're only interested in total_days from this point on. It's not that your locals (days_until, opening_time, time_open, ratio) are not useful: to rewrite your calculation in a single line without them would not only be unwieldy in the extreme, but would require the calculation of some parts twice (like where you write ratio if ratio).
I'm not sure what the objection is here, except that this somehow "pollutes" the local scope. (What else is local scope for?) A common solution to this, which your question precludes, would be to turn this calculation into a small function. Not only would it then be more reusable, but all your locals would be confined to the small function's scope, and wouldn't bother the larger scope you're working in (which I guess by your code sample is some class's method).
Without doing this, the only way left to "clean up" the scope is del. So after your last statement (setting total_days), you would write:
del days_until, opening_time, time_open, ratio
Hopefully this makes as little sense to you as it does to me. The cost of deleting these references (CPU time) vs leaving them in memory is, to me, unjustifiable. Since this is the last available solution and seems completely pointless, I leave you with my initial assessment: you were right in the first place, your code is fine, and this is a non-issue.

How to get something random in datastore (AppEngine)?

Currently i'm using something like this:
images = Image.all()
count = images.count()
random_numb = random.randrange(1, count)
image = Image.get_by_id(random_numb)
But it turns out that the ids in the datastore on AppEngine don't start from 1.
I have two images in datastore and their ids are 6001 and 7001.
Is there a better way to retrieve random images?

The datastore is distributed, so IDs are non-sequential: two datastore nodes need to be able to generate an ID at the same time without causing a conflict.
To get a random entity, you can attach a random float between 0 and 1 to each entity on create. Then to query, do something like this:
rand_num = random.random()
entity = MyModel.all().order('rand_num').filter('rand_num >=', rand_num).get()
if entity is None:
entity = MyModel.all().order('rand_num').get()
Edit: Updated fall-through case per Nick's suggestion.

Another solution (if you don't want to add an additional property). Keep a set of keys in memory.
import random
# Get all the keys, not the Entities
q = ItemUser.all(keys_only=True).filter('is_active =', True)
item_keys = q.fetch(2000)
# Get a random set of those keys, in this case 20
random_keys = random.sample(item_keys, 20)
# Get those 20 Entities
items = db.get(random_keys)
The above code illustrates the basic method for getting only keys and then creating a random set with which to do a batch get. You could keep that set of keys in memory, add to it as you create new ItemUser Entities, and then have a method that returns a n random Entities. You'll have to implement some overhead to manage the memcached keys. I like this solution better if you're performing the query for random elements often (I assume using a batch get for n Entities is more efficient than a query for n Entities).

I think Drew Sears's answer above (attach a random float to each entity on create) has a potential problem: every item doesn't have an equal chance of getting picked. For example, if there are only 2 entities, and one gets a rand_num of 0.2499, and the other gets 0.25, the 0.25 one will get picked almost all the time. This might or might not matter to your application. You could fix this by changing the rand_num of an entity every time it is selected, but that means each read also requires a write.
And pix's answer will always select the first key.
Here's the best general-purpose solution I could come up with:
num_images = Image.all().count()
offset = random.randrange(0, num_images)
image = Image.all().fetch(1, offset)[0]
No additional properties needed, but the downside is that count() and fetch() both have performance implications if the number of Images is large.

Another (less efficient) method, which requires no setup:
query = MyModel.all(keys_only=True)
# query.filter("...")
selected_key = None
n = 0
for key in query:
if random.randint(0,n)==0:
selected_key = key
n += 1
# just in case the query is empty
if selected_key is None:
entry = None
else:
entry = MyModel.get(selected_key)