Develop Reference

Grabbing a value from a dataframe based on random variables created

I have a dataframe with a specific value called rate_multiplier that I need to grab and compare it to the rate_multiplier I get from AWS s3 bucket.
In order to grab the rate_multiplier in the dataframe, I need to take the random variables that I created for a "person" and match them to the dataframe which gives a specific rate_mulitplier based on these certain characteristics
For example:
Random variables created:
Life_term = 25
Gender = F
Rate_class = Best-AB
Age = 27
Coverage = 2310
Dataframe:
Life_term Benefit_Length_months Gender Rate_class Age State Coverage band (low end) Coverage band (high end) Coverage Band Rate_multiplier
0 15 180 M Best-AA 18 Default 500 1199 500-1199 2.31
1 15 180 M Best-AA 19 Default 500 1199 500-1199 2.21
2 15 180 M Best-AA 20 Default 500 1199 500-1199 2.11
3 15 180 M Best-AA 21 Default 500 1199 500-1199 2.03
4 15 180 M Best-AA 22 Default 500 1199 500-1199 1.95
... ... ... ... ... ... ... ... ... ... ...
34987 10 120 F Nicotine Average-CD 61 Default 3600 10000 3600+ 19.10
34988 10 120 F Nicotine Average-CD 62 Default 3600 10000 3600+ 21.27
34989 10 120 F Nicotine Average-CD 63 Default 3600 10000 3600+ 23.44
34990 10 120 F Nicotine Average-CD 64 Default 3600 10000 3600+ 25.61
34991 10 120 F Nicotine Average-CD 65 Default 3600 10000 3600+ 27.78
So for this example, my randomly generated person would get a rate_multiplier of:
0.93
My code is as follows:
rate_mult_df.loc[(rate_mult_df['Life_term'] == 15) & (rate_mult_df['Gender'] == 'F') & (rate_mult_df['Rate_class'] == 'Best-AB') & (rate_mult_df['Age'] == 27) & (rate_mult_df['Coverage band (low end)'] <= 2310) & (rate_mult_df['Coverage band (high end)'] >= 2310)]
Is the right way to grab the rate_muliplier for the randomly genreated person or is there any easier way? Any and all help is appreciated. Please let me know if my question is clear enough. Working on that everyday.

For perfomance reasons I'd use .query()
rate_multiplier = df.query(
"Life_term == 15 &"
" Gender == 'F' &"
" Rate_class == 'Best-AB' &"
" Age == 27 &"
" `Coverage band (low end)` == 2310 &"
" `Coverage band (high end)` == 2310"
)["Rate_multiplier"].squeeze()
"Easier" depends on your workflow. For example if you want to query from a dictionary you could use:
def get_rate_multiplier(search_params: dict) -> str:
return " and ".join(
[f"({k} == '{v}')" if type(v) == str else f"({k} == {v})" for k, v in search_params.items()]
)
random_person = {
"Life_term": 15, "Gender": "F", "Rate_class": "Best-AB",
"Age": 27, "Coverage band (low end)": 2310, "Coverage band (high end)": 2310
}
rate_multiplier = float(df.query(get_rate_multiplier(random_person))["Rate_multiplier"].squeeze())

Pyomo:For loop in Pyomo constraints

I am struggling on making constraints using for loop in Python Pyomo. My code concept is the following:
model.Q = Set()
model.sbase=Param(model.Q, within = PositiveReals)
My code can run properly with one Sbase. But if I get 3 different numbers of Sbase. And what I want to do is to output 3 different image by using these 3 numbers.
set Q := 1 2 3;
param sbase:=
1 800
2 1000
3 1200;
What i have done so far is that:(really don't know how to do this..)
from pyomo.environ import *
import matplotlib.pyplot as plt
import numpy as np
import matplotlib.pyplot as plt
import random
# create a model
model = AbstractModel()
# declare decision variables
model.Q = Set()
model.J = Set()
model.A = Param(model.J, within = PositiveReals)
model.B = Param(model.J, within = PositiveReals)
model.C = Param(model.J, within = PositiveReals)
model.P_min = Param(model.J, within = PositiveReals)
model.P_max= Param(model.J, within = PositiveReals)
model.P = Var(model.J, within = NonNegativeReals)
model.sbase=Param(model.Q, within = PositiveReals)
# declare objective
def obj_expression(model):
return sum(model.A[j] * model.P[j]**2 + model.B[j] * model.P[j] + model.C[j] for j in model.J)
model.profit = Objective(rule = obj_expression, sense=minimize)
# declare constraints
def lower_bound(model,j):
return model.P_min[j] <= model.P[j]
model.laborA = Constraint(model.J, rule = lower_bound)
def upper_bound(model, j):
return model.P[j] <= model.P_max[j]
model.laborB = Constraint(model.J, rule = upper_bound)
for q in range(2): #from this line ，really confused about this
def sum_labor(model,i,q):???
if q==1:??
return sum(model.P[j] for j in model.J) >= (model.sbase[Q] for q in model.Q)??
elif q==2:??
reture sum(model.P[j] for j in model.J) > = (model.sbase[Q] for q in model.Q)
model.laborC = Constraint(model.sbase,rule = sum_labor)
instance = model.create_instance("E:\pycharm_project\PD\pd.dat")
opt = SolverFactory('Ipopt')
results = opt.solve(instance)
plt.figure(figsize=(8,6))
plt.subplot(121)
for i in instance.J:
plt.bar(i,value(instance.P[i]))
plt.xlabel('Generators')
plt.ylabel('Power')
plt.title('Power distribution')
plt.subplot(122)
x=[0.1]
plt.bar(x,value(instance.profit),alpha=0.7,width=0.015)
plt.xlim(0,0.2)
plt.xlabel('')
plt.ylabel('Cost')
plt.title('minimum cost')
plt.show()
print('\n\n---------------------------')
print('Cost: ',value(instance.profit))
#DATA
set Q := 1 2 3;
set J := g1 g2 g3 g4 g5 g6 g7 g8 g9 g10 ;
param : A B C P_min P_max:=
g1 0.0148 12.1 82 80 200
g2 0.0289 12.6 49 120 320
g3 0.0135 13.2 100 50 150
g4 0.0127 13.9 105 250 520
g5 0.0261 13.5 72 80 280
g6 0.0212 15.4 29 50 150
g7 0.0382 14.0 32 30 120
g8 0.0393 13.5 40 30 110
g9 0.0396 15.0 25 20 80
g10 0.0510 14.3 15 20 60
;
param sbase:=
1 800
2 1000
3 1200;
Do you have any suggestion on how to do it?
Thanks!

Here is a toy example that I think answers your question. You can use something like this or still read your sbase variable in from data and then just index it to set the value in a similar way within the loop.
from pyomo.environ import *
m = ConcreteModel()
m.I = Set(initialize=[1,]) # mixed type set for demo only.
m.sbase = Param(mutable=True) # for the purposes of each instantiation, sbase is FIXED, so no index reqd.
m.x = Var(m.I, domain=NonNegativeReals )
## constraint
m.C1 = Constraint(expr=sum(m.sbase*m.x[i] for i in m.I) <= 10 )
## objective
m.OBJ = Objective(expr=sum(m.sbase*m.x[i] for i in m.I), sense=maximize)
solver = SolverFactory('cbc')
sbase_values = {1, 2, 5}
for sbase in sbase_values:
m.sbase = sbase
results = solver.solve(m)
print(f'\n ***** solved for sbase = {sbase} *****')
m.display()

Perform operations on numbers in dataframe through a function with an if statement

I have been working with pandas to analyze and perform some lengthy operations on a dataset through defined functions (for convenience and also since I use the same functions in operations not involving pandas). I am trying to perform some operations based on which number is larger, using if and else statements.
I have not been able to find a workaround in other answers. Here is a short simplified example of what sort of logical operations I am trying to perform:
import pandas as pd
df = pd.DataFrame({"A": [177,166,155,125,146,149,192,160,111,85],
"B": [26.2,27,26.8,23.4,23.3,17.5,26.4,25.7,18.9,15.8],
"C": [9.2,99.1,29.3,8.6,8,7.2,10,39.4,47.25,4.5,]})
x = 'A'
y = 'B'
z = 'C'
def test(a,b,c):
h = a*b/c
return h
df['D'] = test(df[x],df[y],df[z])
Functions have been working out for me so far like that:
print(df['D'])
0 504.065217
1 45.227043
2 141.774744
3 340.116279
4 425.225000
5 362.152778
6 506.880000
7 104.365482
8 44.400000
9 298.444444
Name: D, dtype: float64
I am looking to get such operations working:
def test2(a,b,c):
if a > b:
return a*c
else:
return b*c
df['E'] = test2(df[x],df[y],df[z])
print(df['E'])
I am getting the obvious error:
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

You need:
df['E'] = df.apply(lambda x: test2(x['A'], x['B'], x['C']) ,1)
Output:
A B C E
0 177 26.2 9.20 1628.40
1 166 27.0 99.10 16450.60
2 155 26.8 29.30 4541.50
3 125 23.4 8.60 1075.00
4 146 23.3 8.00 1168.00
5 149 17.5 7.20 1072.80
6 192 26.4 10.00 1920.00
7 160 25.7 39.40 6304.00
8 111 18.9 47.25 5244.75
9 85 15.8 4.50 382.50

#try this (still using a function):
def test2(row):
if row['A'] > row['B']:
return row['A'] * row['C']
else:
return row['B']*row['C']
df['E'] = df.apply(test2, axis=1)
A B C D E
0 177 26.2 9.20 504.065217 1628.40
1 166 27.0 99.10 45.227043 16450.60
2 155 26.8 29.30 141.774744 4541.50
3 125 23.4 8.60 340.116279 1075.00
4 146 23.3 8.00 425.225000 1168.00
5 149 17.5 7.20 362.152778 1072.80
6 192 26.4 10.00 506.880000 1920.00
7 160 25.7 39.40 104.365482 6304.00
8 111 18.9 47.25 44.400000 5244.75
9 85 15.8 4.50 298.444444 382.50
But using np.where is the best solution.

Another solution is to use np.where:
df["E"] = np.where(df.A > df.B, df.A*df.C, df.B*df.C)
print(df)
# A B C E
# 0 177 26.2 9.20 1628.40
# 1 166 27.0 99.10 16450.60
# 2 155 26.8 29.30 4541.50
# 3 125 23.4 8.60 1075.00
# 4 146 23.3 8.00 1168.00
# 5 149 17.5 7.20 1072.80
# 6 192 26.4 10.00 1920.00
# 7 160 25.7 39.40 6304.00
# 8 111 18.9 47.25 5244.75
# 9 85 15.8 4.50 382.50
In this situation, np.where seems to be more efficient than apply (40 times faster). Here the efficiency results:
import timeit
def numpy_where(df):
def func():
df["E"] = np.where(df.A > df.B, df.A*df.C, df.B*df.C)
return func
def pandas_apply(df):
def test2(a, b, c):
if a > b:
return a*c
return b*c
def func():
df["E"] = df.apply(lambda x: test2(x['A'], x['B'], x['C']), 1)
return func
t = timeit.Timer(numpy_where(df))
print(t.timeit(100))
# 0.0711541
t = timeit.Timer(pandas_apply(df))
print(t.timeit(100))
# 2.8589093

Python and Number Theory: How can we create the generating function for q(n) (the number of partitions of n into distinct parts)?

From https://en.wikipedia.org/wiki/Partition_%28number_theory%29#Restricted_partitions, we know that the number of partitions of an integer p(n) is given by
Can be written in python as:
def partitions(n, I=1):
yield(n,)
for i in range(I, n//2 + 1):
for p in partitions(n-i, i):
yield (i,) + p
My question is: How can I modify this to return q(n), the number of partitions containing distinct parts?
i.e.;
p(3)=2, because
3=2+1
3=1+1+1
(1,1,1) are not distinct.
but q(3)=1, because only
3=2+1 contains distinct elements.
The generating function of q(n) is given by
I can't find a good product function in python that returns a product from n to infinity.

Well no computer can calculate products from n to infinity using actual integers.
I think the most efficient solution is as follows
import functools
#functools.lru_cache(maxsize=None) # save previous results
def unique_partitions(n, I=1):
yield (n,)
for i in range(I, n//2 + 1):
for p in unique_partitions(n - i, i):
if i not in p: # eliminate non-unique results
yield (i,) + p
You can then count them as
def q(n):
count = 0
for _ in unique_partitions(n):
count += 1
return count

Рус
Добрый день, я не могу оставить комментарий под ответом FHTMitchell, поэтому оставляю его отдельно.
В моём случае строка
#functools.lru_cache(maxsize=None) # save previous results
приводит к ошибкам в расчётах. Также прилагаю свой вариант функции для расчёта всех значений от 1 до N.
En (google translate)
Good afternoon, I can not leave a comment under the answer FHTMitchell, so I leave my comment separately. In my case, the string
# functools.lru_cache (maxsize = None) # save previous results
leads to errors in calculations. I also attach my version of the function to calculate all values ​​from 1 to N.
Число Эталон Расчёт Ошибка
1 1 1
2 1 1
3 2 2
4 2 2
5 3 3
6 4 3 Ошибка
7 5 4 Ошибка
8 6 4 Ошибка
9 8 5 Ошибка
10 10 5 Ошибка
11 12 6 Ошибка
12 15 6 Ошибка
13 18 7 Ошибка
14 22 7 Ошибка
15 27 8 Ошибка
16 32 8 Ошибка
17 38 9 Ошибка
18 46 9 Ошибка
19 54 10 Ошибка
20 64 10 Ошибка
21 76 11 Ошибка
22 89 11 Ошибка
23 104 12 Ошибка
24 122 12 Ошибка
25 142 13 Ошибка
Мой вариант функции:
My variant of the function:
def partit(N=10):
part = {}
for i in range(1, N + 1):
part[i] = [(j, i - j) for j in range(1, i + 1) if j < i - j]
for i in range(1, N + 1):
temp = []
for up, down in part[i]:
p2 = [(up, (u, d)) for u, d in part[down] if u > up]
temp.extend(p2)
part[i].extend(temp)
return {i:len(part[i])+1 for i in part}
Может кому-то пригодится ряд из 100 первых чисел:
Someone can use a series of 100 first numbers:
1 1 2 2 3 4 5 6 8 10 12 15 18 22 27 32 38 46 54 64 76 89 104 122 142 165 192 222 256 296 340 390 448 512 585 668 760 864 982 1113 1260 1426 1610 1816 2048 2304 2590 2910 3264 3658 4097 4582 5120 5718 6378 7108 7917 8808 9792 10880 12076 13394 14848 16444 18200 20132 22250 24576 27130 29927 32992 36352 40026 44046 48446 53250 58499 64234 70488 77312 84756 92864 101698 111322 121792 133184 145578 159046 173682 189586 206848 225585 245920 267968 291874 317788 345856 376256 409174 444793

Pyomo's parameter estimation in an ODE system with missing values in time series

I have an ODE system of 7 equations for explaining a particular set of microorganisms dynamics of the form:
Where the are the different chemical and microorganisms species involved (even sub-indexes for chemical compounds), the are the yield coefficients and the are the pseudo-reactions:
I am using Pyomo for the estimation of all my unknown parameters, which are basically all the yield coefficients and kinetic constants (15 in total).
The following code works perfectly when is used with complete experimental time series for each of the dynamical variables:
from pyomo.environ import *
from pyomo.dae import *
m = AbstractModel()
m.t = ContinuousSet()
m.MEAS_t = Set(within=m.t) # Measurement times, must be subset of t
m.x1_meas = Param(m.MEAS_t)
m.x2_meas = Param(m.MEAS_t)
m.x3_meas = Param(m.MEAS_t)
m.x4_meas = Param(m.MEAS_t)
m.x5_meas = Param(m.MEAS_t)
m.x6_meas = Param(m.MEAS_t)
m.x7_meas = Param(m.MEAS_t)
m.x1 = Var(m.t,within=PositiveReals)
m.x2 = Var(m.t,within=PositiveReals)
m.x3 = Var(m.t,within=PositiveReals)
m.x4 = Var(m.t,within=PositiveReals)
m.x5 = Var(m.t,within=PositiveReals)
m.x6 = Var(m.t,within=PositiveReals)
m.x7 = Var(m.t,within=PositiveReals)
m.k1 = Var(within=PositiveReals)
m.k2 = Var(within=PositiveReals)
m.k3 = Var(within=PositiveReals)
m.k4 = Var(within=PositiveReals)
m.k5 = Var(within=PositiveReals)
m.k6 = Var(within=PositiveReals)
m.k7 = Var(within=PositiveReals)
m.k8 = Var(within=PositiveReals)
m.k9 = Var(within=PositiveReals)
m.y1 = Var(within=PositiveReals)
m.y2 = Var(within=PositiveReals)
m.y3 = Var(within=PositiveReals)
m.y4 = Var(within=PositiveReals)
m.y5 = Var(within=PositiveReals)
m.y6 = Var(within=PositiveReals)
m.x1dot = DerivativeVar(m.x1,wrt=m.t)
m.x2dot = DerivativeVar(m.x2,wrt=m.t)
m.x3dot = DerivativeVar(m.x3,wrt=m.t)
m.x4dot = DerivativeVar(m.x4,wrt=m.t)
m.x5dot = DerivativeVar(m.x5,wrt=m.t)
m.x6dot = DerivativeVar(m.x6,wrt=m.t)
m.x7dot = DerivativeVar(m.x7,wrt=m.t)
def _init_conditions(m):
yield m.x1[0] == 51.963
yield m.x2[0] == 6.289
yield m.x3[0] == 0
yield m.x4[0] == 6.799
yield m.x5[0] == 0
yield m.x6[0] == 4.08
yield m.x7[0] == 0
m.init_conditions=ConstraintList(rule=_init_conditions)
def _x1dot(m,i):
if i==0:
return Constraint.Skip
return m.x1dot[i] == - m.y1*m.k1*m.x1[i]*m.x2[i]/(m.k2+m.x1[i]) - m.y2*m.k3*m.x1[i]*m.x4[i]/(m.k4+m.x1[i])
m.x1dotcon = Constraint(m.t, rule=_x1dot)
def _x2dot(m,i):
if i==0:
return Constraint.Skip
return m.x2dot[i] == m.k1*m.x1[i]*m.x2[i]/(m.k2+m.x1[i]) - m.k7*m.x2[i]*m.x3[i]
m.x2dotcon = Constraint(m.t, rule=_x2dot)
def _x3dot(m,i):
if i==0:
return Constraint.Skip
return m.x3dot[i] == m.y3*m.k1*m.x1[i]*m.x2[i]/(m.k2+m.x1[i]) - m.y4*m.k5*m.x3[i]*m.x6[i]/(m.k6+m.x3[i])
m.x3dotcon = Constraint(m.t, rule=_x3dot)
def _x4dot(m,i):
if i==0:
return Constraint.Skip
return m.x4dot[i] == m.k3*m.x1[i]*m.x4[i]/(m.k4+m.x1[i]) - m.k8*m.x4[i]*m.x3[i]
m.x4dotcon = Constraint(m.t, rule=_x4dot)
def _x5dot(m,i):
if i==0:
return Constraint.Skip
return m.x5dot[i] == m.y5*m.k3*m.x1[i]*m.x4[i]/(m.k4+m.x1[i])
m.x5dotcon = Constraint(m.t, rule=_x5dot)
def _x6dot(m,i):
if i==0:
return Constraint.Skip
return m.x6dot[i] == m.k5*m.x3[i]*m.x6[i]/(m.k6+m.x3[i]) - m.k9*m.x6[i]*m.x7[i]
m.x6dotcon = Constraint(m.t, rule=_x6dot)
def _x7dot(m,i):
if i==0:
return Constraint.Skip
return m.x7dot[i] == m.y6*m.k5*m.x3[i]*m.x6[i]/(m.k6+m.x3[i])
m.x7dotcon = Constraint(m.t, rule=_x7dot)
def _obj(m):
return sum((m.x1[i]-m.x1_meas[i])**2+(m.x2[i]-m.x2_meas[i])**2+(m.x3[i]-m.x3_meas[i])**2+(m.x4[i]-m.x4_meas[i])**2+(m.x5[i]-m.x5_meas[i])**2+(m.x6[i]-m.x6_meas[i])**2+(m.x7[i]-m.x7_meas[i])**2 for i in m.MEAS_t)
m.obj = Objective(rule=_obj)
m.pprint()
instance = m.create_instance('exp.dat')
instance.t.pprint()
discretizer = TransformationFactory('dae.collocation')
discretizer.apply_to(instance,nfe=30)#,ncp=3)
solver=SolverFactory('ipopt')
results = solver.solve(instance,tee=True)
However, I am trying to run the same estimation routine in another experimental data that have missing values at the end of one or maximum two time series of some of the dynamical variables.
In other words, these complete experimental data looks like (in the .dat file):
set t := 0 6 12 18 24 30 36 42 48 54 60 66 72 84 96 120 144;
set MEAS_t := 0 6 12 18 24 30 36 42 48 54 60 66 72 84 96 120 144;
param x1_meas :=
0 51.963
6 43.884
12 24.25
18 26.098
24 11.871
30 4.607
36 1.714
42 4.821
48 5.409
54 3.701
60 3.696
66 1.544
72 4.428
84 1.086
96 2.337
120 2.837
144 3.486
;
param x2_meas :=
0 6.289
6 6.242
12 7.804
18 7.202
24 6.48
30 5.833
36 6.644
42 5.741
48 4.568
54 4.252
60 5.603
66 5.167
72 4.399
84 4.773
96 4.801
120 3.866
144 3.847
;
param x3_meas :=
0 0
6 2.97
12 9.081
18 9.62
24 6.067
30 11.211
36 16.213
42 10.215
48 20.106
54 22.492
60 5.637
66 5.636
72 13.85
84 4.782
96 9.3
120 4.267
144 7.448
;
param x4_meas :=
0 6.799
6 7.73
12 7.804
18 8.299
24 8.208
30 8.523
36 8.507
42 8.656
48 8.49
54 8.474
60 8.203
66 8.127
72 8.111
84 8.064
96 6.845
120 6.721
144 6.162
;
param x5_meas :=
0 0
6 0.267
12 0.801
18 1.256
24 1.745
30 5.944
36 3.246
42 7.787
48 7.991
54 6.943
60 8.593
66 8.296
72 6.85
84 8.021
96 7.667
120 7.209
144 8.117
;
param x6_meas :=
0 4.08
6 4.545
12 4.784
18 4.888
24 5.293
30 5.577
36 5.802
42 5.967
48 6.386
54 6.115
60 6.625
66 6.835
72 6.383
84 6.605
96 5.928
120 5.354
144 4.975
;
param x7_meas :=
0 0
6 0.152
12 1.616
18 0.979
24 4.033
30 5.121
36 2.759
42 3.541
48 4.278
54 4.141
60 6.139
66 3.219
72 5.319
84 4.328
96 3.621
120 4.208
144 5.93
;
While one of my incomplete data sets could have all time series complete, but one like this:
param x6_meas :=
0 4.08
6 4.545
12 4.784
18 4.888
24 5.293
30 5.577
36 5.802
42 5.967
48 6.386
54 6.115
60 6.625
66 6.835
72 6.383
84 6.605
96 5.928
120 5.354
144 .
;
I have knowledge that one can specify to Pyomo to take the derivative of certain variables with respect to a different time serie. However, after tried it, it hadn't worked, and I guess that is because that these are coupled ODE. So basically my question is if there is a way to overcome this issue in Pyomo.
Thanks in advance.

I think all you need to do is slightly modify your objective function like this:
def _obj(m):
sum1 = sum((m.x1[i]-m.x1_meas[i])**2 for i in m.MEAS_t if i in m.x1_meas.keys())
sum2 = sum((m.x2[i]-m.x2_meas[i])**2 for i in m.MEAS_t if i in m.x2_meas.keys())
sum3 = sum((m.x3[i]-m.x3_meas[i])**2 for i in m.MEAS_t if i in m.x3_meas.keys())
sum4 = sum((m.x4[i]-m.x4_meas[i])**2 for i in m.MEAS_t if i in m.x4_meas.keys())
sum5 = sum((m.x5[i]-m.x5_meas[i])**2 for i in m.MEAS_t if i in m.x5_meas.keys())
sum6 = sum((m.x6[i]-m.x6_meas[i])**2 for i in m.MEAS_t if i in m.x6_meas.keys())
sum7 = sum((m.x7[i]-m.x7_meas[i])**2 for i in m.MEAS_t if i in m.x7_meas.keys())
return sum1+sum2+sum3+sum4+sum5+sum6+sum7
m.obj = Objective(rule=_obj)
This double checks that i is a valid index for each set of measurements before adding that index to the sum. If you knew apriori which measurement sets were missing data then you could simplify this function by only doing this check on those sets and summing over the others like you were before.