I am running a optimization problem trying to solve how to best meet a power demand. When i run the code for all 8760 hours of the year, i get the error in the picture. The amount of nonzeros leads me to believe my code is written in an un-optimal way.
My current code looks like this (with some variables defined earlier). I know this sort of problem can be solved with more memory but as I have 32 GB I think it should be sufficient. The code runs fine for for smaller amounts of hours, for example 1000. Is there any way to better optimize my code?
model.plants = pyo.Set(initialize=['Nuclear','Gas','Wind','Solar'])
model.periods = pyo.Set(initialize=list_hours)
def production_bounds(model,i,j):
return(P_min[i],P_max[i][j])
model.pBounds = pyo.Var(model.plants,model.periods,bounds=production_bounds)
def load_balance(model,i,j):
return (model.pBounds['Nuclear',j] + model.pBounds['Gas',j] + model.pBounds['Wind',j] + model.pBounds['Solar',j] == Load_demand[j])
model.load_constraint = pyo.Constraint(model.plants,model.periods,rule=load_balance)
def renewable_con(model,i,j):
x = 0
for j in model.periods:
x += model.pBounds['Wind',j] + model.pBounds['Solar',j]
return (x>=renewable_limit)
model.renew_constraint = pyo.Constraint(model.plants,model.periods,rule=renewable_con)
def obj_func(model):
final_costs = 0
for i in model.plants:
for j in model.periods:
final_costs = final_costs + get_costs_func(i, j)
return final_costs
model.obj = pyo.Objective(rule = obj_func, sense = pyo.minimize)
model.dual = pyo.Suffix(direction=pyo.Suffix.IMPORT)
opt = SolverFactory('gurobi', solver_io="python")
results = opt.solve(model,tee=True)
Related
I'd like to know why python gives me two different times when I re-order the two nested for loops.
The difference is that significant that causes inaccurate results.
This one almost gives me the result I expect to see:
for i in range(20000):
for j in possibleChars:
entered_pwd = passStr + j + possibleChars[0] * leftPassLen
st = time.perf_counter_ns()
verify_password(stored_pwd, entered_pwd)
endTime = time.perf_counter_ns() - st
tmr[j] += endTime
But this code generate inaccurate results from my view:
for i in possibleChars:
for j in range(20000):
entered_pwd = passStr + i + possibleChars[0] * leftPassLen
st = time.perf_counter_ns()
verify_password(stored_pwd, entered_pwd)
endTime = time.perf_counter_ns() - st
tmr[i] += endTime
This is the function I'm attempting to run timing attack on it:
def verify_password(stored_pwd, entered_pwd):
if len(stored_pwd) != len(entered_pwd):
return False
for i in range(len(stored_pwd)):
if stored_pwd[i] != entered_pwd[i]:
return False
return True
I also observed a problem with character 'U' (capital case), so to have successful runs I had to delete it from my possibleChars list.
The problem is when I measure the time for 'U', it is always near double as other chars.
Let me know if you have any question.
Summing up the timings may not be a good idea here:
One interruption due to e.g., scheduling will have a huge effect on the total and may completely invalidate your measurements.
Iterating like in the first loop is probably more likely to spread noise more evenly across the measurements (this is just an educated guess though).
However, it would be better to use the median or minimum time instead of the sum.
This way, you eliminate all noisy measurements.
That being said, I don't expect the timing difference to be huge and python being a high-level language will generate more noisy measurements compared to more low-level languages (because of garbage collection and so on).
But it still works :)
I've implemented an example relying on the minimum time (instead of the sum).
On my local machine, it works reliably except for the last character, where the timing difference is way smaller:
import time
import string
# We want to leak this
stored_pwd = "S3cret"
def verify_password(entered_pwd):
if len(stored_pwd) != len(entered_pwd):
return False
for i in range(len(stored_pwd)):
if stored_pwd[i] != entered_pwd[i]:
return False
return True
possibleChars = string.printable
MEASUREMENTS = 2000 # works even with numbers as small as 20 (for me)
def find_char(prefix, len_pwd):
tmr = {i: 9999999999 for i in possibleChars}
for i in possibleChars:
for j in range(MEASUREMENTS):
entered_pwd = prefix + i + i * (len_pwd - len(prefix) - 1)
st = time.perf_counter_ns()
verify_password(entered_pwd)
endTime = time.perf_counter_ns() - st
tmr[i] = min(endTime, tmr[i])
return max(tmr.items(), key = lambda x: x[1])[0]
def find_length(max_length = 100):
tmr = [99999999 for i in range(max_length + 1)]
for i in range(max_length + 1):
for j in range(MEASUREMENTS):
st = time.perf_counter_ns()
verify_password("X" * i)
endTime = time.perf_counter_ns() - st
tmr[i] = min(endTime, tmr[i])
return max(enumerate(tmr), key = lambda x: x[1])[0]
length = find_length()
print(f"password length: {length}")
recovered_password = ""
for i in range(length):
recovered_password += find_char(recovered_password, length)
print(f"{recovered_password}{'?' * (length - len(recovered_password))}")
print(f"Password: {recovered_password}")
My goal is to write the following model using docplex.mp.model in python. which ptj takes binary variable {0,1}.
[summation from of Ptj from j=1 to t][t = 1,.....,8]
here is the code I wrote:
N = 8
(period_list = [t for t in range(1, no_of_period+1)])
(j = period_list)
p = Mode.binary_var_dict(period_list, name = 'p')
for t in period_list:
for j in range(1,t+1):
Model.add_constraints(Model.sum(p[t,j]) == 1)
but I got an error. Could anyone help me with this problem please?
Your code has numerous issues.
First, you need to create one instance of docplex.mp.model.Model to add constraints to: all your calls to Model.<fn> should be rewritten as mdl.<fn> as they are instance methods.
Second, the variable dict you create has periods as keys, that is, 1,2,..P
so querying p[t,j] is sure to crash with KeyError. If you need a square matrix of variables for each couple of periods, use Model.binary_var_matrix.
Third: Model.add_constraints (with a final S) expects an iterable, but you are passing one constraint, this is also sure to crash.
Lastly, using ranges starting at 1 is not the simplest nor the safest choice with Docplex.
Here is a code, freely derived from your sample, which I guess is close to what you need:
pr = range(1, no_of_period+1)
from docplex.mp.model import Model
m = Model()
p = m.binary_var_matrix(pr, pr, name = 'p')
m.add_constraints( (m.sum(p[t,j] for j in pr) == 1) for t in pr)
print(m.lp_string)
and the output is:
Minimize
obj:
Subject To
c1: p_1_1 + p_1_2 + p_1_3 = 1
c2: p_2_1 + p_2_2 + p_2_3 = 1
c3: p_3_1 + p_3_2 + p_3_3 = 1
[..snip..]
I wrote a code that works great for solving equations numerically, but there is I a specific equation that when I get in there and try running the code - it will run and no output will ever come out!
Equation I got an output for: x^3−3*x+2−a*(np.sin(x))
Equation I didn't get an output for: (x-1)(x-2)(x-3)-a*(np.cos(x))
I also tried writing the second equation without brackets, like this: x^3-6*x^2+11*x-6-a*np.cos(x)
and it didn't help. where is the problem?!
this is my code:
import math
import numpy as np
h =1e-5
eps =1e-8
#function of the equation
def nf(x,a,c):
c=math.cos(x)
solu=(x-1)*(x-2)*(x-3)-a*c
return(solu)
#numerical method
def sl(a,x):
c=math.cos(x)
f = nf(x,a,c)
while abs(f)>eps:
x = x - h*f/(nf(x+h,a,c)-f)
f = nf(x,a,c)
return(x)
N = 101
mya = np.linspace(0.0,1.0,N)
myb = np.zeros(mya.shape)
myc = np.zeros(mya.shape)
myd = np.zeros(mya.shape)
for i in range(0,N):
myb[i] = sl(mya[i],1.0)
myc[i] = sl(mya[i],2.0)
myd[i] = sl(mya[i],3.0)
print(myb[i])
print(myc[i])
print(myd[i])
The problem is that for some input to sl, abs(f)>eps might not ever become False, creating an infinite loop. I havn't investigated the mathematical problem of yours, so I can't solve this problem "for real". What I can provide is automatic detection of when this happens, so that the code returns without a result rather than looping forever.
def sl(a,x):
c=math.cos(x)
f = nf(x,a,c)
count, maxcount = 0, 1000
while abs(f)>eps:
x = x - h*f/(nf(x+h,a,c)-f)
f = nf(x,a,c)
count += 1
if count > maxcount:
return
return(x)
Here, a maximum of 1000 iterations is allowed before a solution is deemed unreachable. In such a case, sl returns None, which when inserted into your NumPy float arrays becomes np.nan.
Upon investigating the output, only myc[60] fails in this way.
Your nf function was a little weird. You were passing c = math.cos(x) into nf() but whithin nf() you tried to assign c to math.cos(x) again. Just use the value c what you passed. Commenting it out fixes your code. As for the mathematical correctness, I cannot determine that unless you provide a better explanation what youre trying to do.
import math
import numpy as np
h =1e-5
eps =1e-8
#function of the equation
def nf(x,a,c):
# this line is not needed. Commenting allows your code to run
# c=math.cos(x)
solu=(x-1)*(x-2)*(x-3)-a*c
return(solu)
#numerical method
def sl(a,x):
c = math.cos(x)
f = nf(x,a,c)
while abs(f)>eps:
x = x - h*f/(nf(x+h,a,c)-f)
f = nf(x,a,c)
return(x)
N = 101
mya = np.linspace(0.0,1.0,N)
myb = np.zeros(mya.shape)
myc = np.zeros(mya.shape)
myd = np.zeros(mya.shape)
for i in range(0,N):
myb[i] = sl(mya[i],1.0)
myc[i] = sl(mya[i],2.0)
myd[i] = sl(mya[i],3.0)
print(myb[i])
print(myc[i])
print(myd[i])
Output:
3.2036907284
0.835006605064
0.677633820877
I have this python code that should run just fine. I'm running it on Anaconda's Spyder Ipython console, or on the Anaconda terminal itself, because that is the only way I can use the "numba" library and its "jit" decorator.
However, either one always "freezes" or "hangs" just about whenever I run it. There is nothing wrong with the code itself, or else I'd get an error.
Sometimes, the code runs all the way through perfectly fine, sometimes it just prints the first line from the first function, and sometimes the code stops anywhere in the middle.
I've tried seeing under which conditions the same problems reproduce, but I haven't been able to get any insights.
My code is:
import time
import numpy as np
import random
from numba import vectorize, cuda, jit, njit, prange, float64, float32, int64
from numba.numpy_support import from_dtype
import numba
#jit(nopython = True)
def make_array(number_of_rows, row_size, starting_size):
q = np.zeros((number_of_rows,row_size))
q[:,0]=starting_size
return(q)
q = make_array(5,5,5)
#jit(nopython = True)
def row_size(array):
return(array.shape[1])
#jit(nopython = True)
def number_of_rows(array):
return(array.shape[0])
#jit(nopython = True)
def foo(array):
result = np.zeros(array.size).reshape(1,array.shape[1])
result[:] = array[:]
shedding_row = np.zeros(array.size).reshape(1,array.shape[1])
birth_row = np.zeros(array.size).reshape(1,array.shape[1])
for i in range((array.shape[0])):
for j in range((array.shape[1])-1):
if result[i,j] !=0:
shedding = (np.random.poisson( (result[i,j])**.2, 1))[0]
birth = (np.random.poisson( (3), 1))[0]
birth = 0
result[i,j+1] = result[i,j] - shedding + birth
shedding_row[i,j+1] = shedding
birth_row[i,j+1] = birth
if result[i,j] == 0:
result[i,j] = result[i,j]
return(result, shedding_row)
#jit(nopython = True)
def foo_two(array):
result = np.zeros(array.size).reshape(array.shape[0],array.shape[1])
result_two = np.zeros(array.size).reshape(array.shape[0],array.shape[1])
i = 0
while i != (result.shape[0]):
fill_in_row= 0*np.arange(1 * result.shape[1]).reshape(1, result.shape[1])
fill_in_row[0] = array[i]
result[i], shedding_row = foo(fill_in_row)
result_two[i] = shedding_row
i+=1
return(result, result_two)
#jit(nopython = True)
def foo_three(array):
array_sum = np.sum(array, axis = 0)
array_sum = array_sum.reshape(1,array_sum.size)
result = np.zeros(array_sum.size).reshape(1,array_sum.size)
for i in range((result.shape[0])):
for j in range((result.shape[1])):
shed_death_param = .2
shed_metastasis_param = .3
combined_number = (int(array_sum[i,j])) * (shed_death_param+shed_metastasis_param)
for q in range(int(combined_number)):
random_number = random.randint(1, 7)
if random_number == 5:
result[i,j]+=1
number_to_add = (int(array_sum[i,j])) - (int(combined_number))
if j < row_size(array_sum) - 1:
(array_sum[i,j+1]) += number_to_add
return(result)
#jit(nopython = True)
def foo_four(array):
result = np.zeros(array.size).reshape(1,array.size)
for i in range((result.shape[0])):
for j in range((result.shape[1])):
if int(array[i,j])!= 0:
for q in range(int(array[i,j])):
addition = np.zeros((1,result.shape[1]))
addition[0][j] = 1
result = np.concatenate((result, addition), axis=0)
if result.shape[0]!=1:
result = result[1:]
return(result)
def the_process(array):
array, master_shedding_array = (foo_two(array))
master_metastasis_array = foo_three(master_shedding_array)
new_array = (foo_four(master_metastasis_array))
print("new_array is\n", new_array)
return(array,new_array)
def the_bigger_process(array):
big_array = make_array(1,row_size(array),0)
big_metastasis_array = make_array(1,row_size(array),0)
counter =0
i = 0
while counter < row_size(array)-1:
print("We begin, before the_process is called")
updated_array,metastasis_array = the_process(array)
big_array = np.concatenate((big_array, updated_array), axis=0)
if sum( metastasis_array[0] ) != 0:
big_metastasis_array = np.concatenate((big_metastasis_array, metastasis_array), axis=0)
i+=1
third_big_metastasis_array = big_metastasis_array[np.where(big_metastasis_array[:,i] == 1)]
array = third_big_metastasis_array
counter+=1
big_array = big_array[1:]
big_metastasis_array = big_metastasis_array[1:]
return(big_array,big_metastasis_array)
something, big_metastasis_array = the_bigger_process(q)
print("something is\n",something)
print("big_metastasis_array is\n",big_metastasis_array)
I know it's best to just post the part of your code that's relevant, but this such an unusual situation where the code is actually fine, that I thought I should post all of it.
This is a screenshot of when I ran the code two consecutive times, clearly the first time it printed the outputs I wanted just fine, and then the next time it froze. And sometimes it freezes in between.
Of course I put many print functions all over when I was testing if I could see some pattern, but I couldn't, and I took out all those print functions in the code above. But the truth is, this code would freeze in the middle, and there was no consistency or "replicability" to it.
I've googled around but couldn't find anyone else with a similar issue.
You are passing a bad value to np.random.poisson. In your code result[i, j] can sometimes be negative, which is causing an NaN in numba, whereas in python it return an actual (negative) value. In python you might get a ValueError, but numba is failing in a different way that causes the process to hang.
You have to decide whether it makes sense for your particular problem, but if I add, the check between the # ****** comments:
#jit(nopython=True)
def foo(array):
result = np.zeros(array.size).reshape(1, array.shape[1])
result[:] = array[:]
shedding_row = np.zeros(array.size).reshape(1, array.shape[1])
birth_row = np.zeros(array.size).reshape(1, array.shape[1])
for i in range((array.shape[0])):
for j in range((array.shape[1]) - 1):
if result[i, j] != 0:
# ******
if result[i, j] < 0:
continue
# ******
shedding = (np.random.poisson( (result[i, j])**.2, 1))[0]
birth = (np.random.poisson((3), 1))[0]
....
in foo, then the code stops hanging.
As a general debugging tip, it's good to run your code with the jit decorators commented out to see if anything strange is happening.
I am trying to make a program that outputs all possibilities to put + or - or nothing between the numbers 1,2,…,9 such that the result is 100
Understandably, there are a few people who have uploaded solutions already on the internet, but I want to come up with my own. Here's the non working code:
"""
This program outputs all possibilities to put + or - or nothing between the numbers 1,2,…,9 (in this order) such that the result is 100
"""
class solver:
def __init__(self):
"""
self.possibilities stores Arrays of type : [0]- always the sum of all operations
[1:9] all operations in int, where 0 equals plus, 1 equals minus, 2 equals nothing
"""
self.possibilities = []
self.possibilities.append([100])
for i in range(7):
self.possibilities.extend(self.makeNewIteration(i+1))
print(self.possibilities)
for obj in self.possibilities:
if 100 is obj[0]:
print(obj)
def makeNewIteration(self, i):
for obj in self.possibilities:
if(len(obj)<9):
if(obj[-1] is 3):#if case 3
#recalculate the result
currentResult = int(obj[0] + self.conceal(i-1, i))
else: currentResult = int(obj[0])
#print(obj)
possibilitiesNew = []
possibilitiesNew.append([currentResult + i] + obj[1:] + [1])#case 1
possibilitiesNew.append([currentResult - i] + obj[1:] + [2])#case 2
possibilitiesNew.append([currentResult] + obj[1:] + [3])#case 3
print("Iteration: "+str(i)+" : "+str(possibilitiesNew))
self.possibilities.remove(obj)#remove the old object
else:
print("finished.")
return possibilitiesNew
def conceal(self, x, y):
# makes 12 out of x=1 and y=2
return int(f'{x}{y}')
solve = solver()
The more I think about it the more problems I have.
I used to learn programming with an OOP mindset, and both the fact that it was a while ago and that this question is much easier with a procedural work flow makes me stuck. For example, what happens if there is two times "nothing" in a row ? does 1, 2, 3, 4 become 12,23 and ? ... I hope someone could fix the code a bit and I would figure out what I did wrong
Although eval should be avoided, here's another solution that uses it:
from itertools import product
[''.join(i) for i in product(*[[str(j)+'+', str(j) + '-', str(j)] for j in range(1, 9)] + ['9']) if eval(''.join(i))==100]
#['1+2+3-4+5+6+78+9', '1+2+34-5+67-8+9', '1+23-4+5+6+78-9', '1+23-4+56+7+8+9', '12+3+4+5-6-7+89', '12+3-4+5+67+8+9', '12-3-4+5-6+7+89', '123+4-5+67-89', '123+45-67+8-9', '123-4-5-6-7+8-9', '123-45-67+89']
My first approach would be
import itertools as it
import re
all_those_terms = (''.join([(sgn + str(i+1)) for i, sgn in enumerate(tpl)]) for tpl in it.product(['', '-', '+'], repeat=9) if tpl[0]!='')
for s in all_those_terms:
r = re.findall('[+-]\d+', '+'+s)
calc = sum(map(int, r))
if calc == 100:
print(s, '=', 100)
-1+2-3+4+5+6+78+9 = 100
+123-45-67+89 = 100
+123-4-5-6-7+8-9 = 100
+123+45-67+8-9 = 100
+123+4-5+67-89 = 100
+12-3-4+5-6+7+89 = 100
+12+3-4+5+67+8+9 = 100
+12+3+4+5-6-7+89 = 100
+1+23-4+56+7+8+9 = 100
+1+23-4+5+6+78-9 = 100
+1+2+34-5+67-8+9 = 100
+1+2+3-4+5+6+78+9 = 100
However, eval should be replaced, as it's dangerous...
I'll edit as soon as I have some time again...
EDIT: eval replaced...