Develop Reference

Optimizing python function for Numba

The following is the python function which I am trying to rewrite in python:
def freeoh_count_nojit(coord=np.array([[]]),\
molInterfaceIndex=np.array([]),\
hNeighbourList=np.array([]),\
topol=np.array([[]]),\
cos_HAngle=0.0,\
cellsize=np.array([]),\
is_orig_def=False,is_new_def=True):
labelArray=[]
freeOHcosDA=[]; freeOHcosDAA=[]
mol1Coord=np.zeros((3,3),dtype=float)
labelArray=np.empty(molInterfaceIndex.shape[0], dtype="U10")
for i in range(molInterfaceIndex.shape[0]): # loop over selected molecules
mol2CoordList=[]; timesave=[]
mol1Coord=np.array([coord[k] for k in topol[molInterfaceIndex[i]]]) # extract center molecule
gen = np.array([index for index in hNeighbourList[i] if index!=-1]) # remove padding
for j in range(gen.shape[0]):
mol2CoordList.append([coord[k] for k in topol[gen[j]]]) # extract neighbors
mol2Coord=np.array(mol2CoordList).reshape(-1,3)
if is_orig_def:
acceptor,donor,cosAngle=interface_hbonding_orig(mol1Coord,mol2Coord,cos_HAngle,cellsize)
labelArray[i]="D"*np.abs(2-np.sum(donor))+"A"*np.clip(np.array([np.sum(acceptor)]),1,2)[0]
elif is_new_def:
acceptor,donor,cosAngle=interface_hbonding_new(mol1Coord,mol2Coord,cos_HAngle,cellsize)
labelArray[i]="D"*np.abs(2-np.sum(donor))+"A"*np.sum(acceptor)
if labelArray[i] in "DA":
freeOHcosDA.append(cosAngle)
elif labelArray[i] in "DAA":
freeOHcosDAA.append(cosAngle)
freeOHcos=freeOHcosDA+freeOHcosDAA
return labelArray, freeOHcos
The function takes in a coordinates frame of the simulation molecules. The code selects a central molecule from an index list molInterfaceIndex and extracts its neighbouring molecules coordinates from a pre-generated neighbour (generated from scipy.spatial.KDTree hence cannot be called from a jitted function). The central molecule and its neighbour are send to a jitted function which then returns two [1,1] and a scaler which are then used to label the central molecule.
My attempt at rewriting the above python function is below:
#njit(cache=True,parallel=True)
def freeoh_count_jit(coord=np.array([[]]),\
molInterfaceIndex=np.array([]),\
hNeighbourList=np.array([]),\
topol=np.array([[]]),\
cos_HAngle=0.0,\
cellsize=np.array([]),\
is_orig_def=False,is_new_def=True):
NAtomsMol=3 #No. of atoms in a molecule
_M=molInterfaceIndex.shape[0]
_N=hNeighbourList.shape[1]
mol1Coord=np.zeros((NAtomsMol,3),dtype=np.float64)
mol2Coord=np.zeros((_N*NAtomsMol,3),dtype=np.float64)
acceptor=np.zeros((_M,2),dtype=int)
donor=np.zeros((_M,2),dtype=int)
cosAngle=np.zeros(_M,dtype=np.float64)
gen=np.zeros(_M,dtype=int)
freeOHMask = np.zeros(_M, dtype=int) == 0
labelArray=np.empty(_M, dtype="U10")
for i in range(_M): # loop over selected molecules
for index,j in enumerate(topol[molInterfaceIndex[i]]):
mol1Coord[index]=coord[j] # extract center molecule
for indexJ,j in enumerate(hNeighbourList[i]):
for indexK,k in enumerate(topol[j]):
mol2Coord[indexK+topol[j].shape[0]*indexJ]=coord[k] # extract neighbors
gen[i] = len(np.array([index for index in hNeighbourList[i] if index!=-1]))*NAtomsMol # get actual number of neighbor atoms
if is_orig_def:
acceptor[i],donor[i],cosAngle[i]=interface_hbonding_orig(mol1Coord,mol2Coord[:gen[i]],cos_HAngle,cellsize)
labelArray[i]="D"*np.abs(2-np.sum(donor[i]))+"A"*np.clip(np.array([np.sum(acceptor[i])]),1,2)[0]
elif is_new_def:
acceptor[i],donor[i],cosAngle[i]=interface_hbonding_new(mol1Coord,mol2Coord[:gen[i]],cos_HAngle,cellsize)
labelArray[i]="D"*np.abs(2-np.sum(donor[i]))+"A"*np.sum(acceptor[i])
freeOHMask[np.where(cosAngle > 1.0)] = False
return acceptor, donor, labelArray, freeOHMask
The main issue is that #jit function seem to be providing incorrect results while using numba.prange for the outer loop. Also, the execution time for the function increases per call which is a bit confusing. The functions interface_hbonding_orig() and interface_hbonding_new() are already jitted so I think they are out of scope of discussion here. One of the bigger questions is that whether I even to jit this function at all as the most time consuming part is supposing to be the array selection in the initial few initial lines in the outer loop. If anyone has any suggestions for rewriting this function or even for algorithm design, it would be really helpful.

2SLS with fixed effects Python

I am trying to rebuild a paper (Dreher et al. 2020. Aid, China, and Growth: Evidence from a New Global Development Finance Dataset) with Python. The paper's calculations were performed with Stata. I managed to rebuild everything until now, but I am stuck.
The data of this project is PanelData and I have to include year specific effects and country specific effects,
time_specific_effects = True, entity_effects = True / other_effects = data4.code
(because the variable data4.code is here the same as the entity).
The plan is to use 2SLS:
Regression
IV Strategy
The Stata code is given by the author:
*xtivreg2 growth_pc (l2.OFn_all= l3.IV_reserves_OFn_all_1_ln l3.IV_factor1_OFn_all_1_ln) l.population_ln time* if code!="CHN", fe first savefprefix(first) cluster(code) endog(l2.OFn_all)*
I rebuilt all the lagged variables in Python using shift() and it worked:
data4["l3IV_reserves_OFn_all_1_ln"] = data4["IV_reserves_OFn_all_1_ln"].shift(3)
data4["l3IV_factor1_OFn_all_1_ln"] = data4["IV_factor1_OFn_all_1_ln"].shift(3)
So the setup is the same as it is for the author:
As far as I know, there is no library in Python that can perform 2SLS with fixed effects. So I thought that I will just use linear model PanelOLS (which is suited for panel data with fixed effects) to perform the First Stage and Second Stage separately:
dependendFS = data4.l2OFn_all
exog2 = sm.tools.add_constant(data4[["l1population_ln", "l3IV_reserves_OFn_all_1_ln","l3IV_factor1_OFn_all_1_ln"]])
mod = lm.panel.PanelOLS(dependendFS, exog2, time_effects = True, entity_effects=True, drop_absorbed=True)
mod_new21c = mod.fit(cov_type='clustered', clusters = data4.code)
# Safe the fitted values
fitted_c = mod_new21c.fitted_values
data4["fitted_values_c"] = fitted_c
dependentSS = data4.growth_pc
exog = sm.tools.add_constant(data4[["fitted_values_c", "l1population_ln"]])
mod = lm.panel.PanelOLS(dependentSS, exog, time_effects=True, entity_effects= True)
mod_new211c = mod.fit(cov_type='clustered', clusters = data4.code)
I tried several combinations of the fixed effects and for the covariance, but it did not so far deliver the results I need. Here is my output for the Second Stage:
Results after Second Stage
and this is what they should look like:
dependent variable is growth p.c, SE in brackets
Where is my mistake? Do I have to adjust my data or the output of the First Stage since I am separately performing 2SLS? Is there a mistake or a better method of estimating 2SLS in Python?

Hardy Cross in Python

Im trying to write a code in python to solve pipe network problem using Hardy Cross. I'm almost done but the results shown does not satisfy the conservation of water in junction with outflow.
while True:
a1=-(kAB*GIVEN[0][3]**2+kBE*GIVEN[1][3]**2+kEI*GIVEN[2][3]**2-kIH*GIVEN[3][3]**2+kHA*GIVEN[4][3]**2)\
/(2*(kAB*GIVEN[0][3]+kBE*GIVEN[1][3]+kEI*GIVEN[2][3]+kIH*GIVEN[3][3]+kHA*GIVEN[4][3]))
a2=-(kBC*GIVEN[5][3]**2+kCF*GIVEN[6][3]**2+kFE*GIVEN[7][3]**2-kBE*GIVEN[1][3]**2)/(2*(kBC*GIVEN[5][3]+kCF*GIVEN[6][3]+kFE*GIVEN[7][3]+kBE*GIVEN[1][3]))
a3=-(kCD*GIVEN[8][3]**2+kDG*GIVEN[9][3]**2+kGF*GIVEN[10][3]**2-kCF*GIVEN[6][3]**2)/(2*(kCD*GIVEN[8][3]+kDG*GIVEN[9][3]+kGF*GIVEN[10][3]+kCF*GIVEN[6][3]))
a4=-(-kFE*GIVEN[7][3]**2-kGF*GIVEN[10][3]**2-kGJ*GIVEN[11][3]**2-kJI*GIVEN[12][3]**2-kEI*GIVEN[2][3]**2)\
/(2*(kFE*GIVEN[7][3]+kGF*GIVEN[10][3]+kGJ*GIVEN[11][3]+kJI*GIVEN[12][3]+kEI*GIVEN[2][3]))
if abs(a1)<0.001 and abs(a2)<0.001 and abs(a3)<0.001 and abs(a4)<0.001:
print("P_AB = ",GIVEN[0][3]*1000,"L/s"
break
else:
#compute corrected flow
#abs function is used to remove negative when Qcorrected is used for another iteration
GIVEN[0][3]=abs(GIVEN[0][3]+a1 )#AB
GIVEN[1][3]=abs(GIVEN[1][3]+a1-a2 )#BE
GIVEN[2][3]=abs(GIVEN[2][3]+a1-a4 )#EI
GIVEN[3][3]=abs(GIVEN[3][3]-a1 )#IH
GIVEN[4][3]=abs(GIVEN[4][3]+a1 )#HA
GIVEN[5][3]=abs(GIVEN[5][3]+a2 )#BC
GIVEN[6][3]=abs(GIVEN[6][3]+a2-a3 )#CF
GIVEN[7][3]=abs(GIVEN[7][3]+a2-a4 )#FE
GIVEN[8][3]=abs(GIVEN[8][3]+a3 )#CD
GIVEN[9][3]=abs(GIVEN[9][3]+a3 )#DG
GIVEN[10][3]=abs(GIVEN[10][3]+a3-a4 )#GF
GIVEN[11][3]=abs(GIVEN[11][3]-a4 )#GJ
GIVEN[12][3]=abs(GIVEN[12][3]-a4 )#JI

Inverse Dynamics Controller Drake

I currently have an array of desired position vs. time of an object in my plant. I am using an inverse dynamics controller in order to drive the object to this desired position but I'm experiencing some difficulties. Here is how I am doing this:
I created the controller system
ID_cont = InverseDynamicsController(robot=controller_plant, kp=np.array([0.5]), ki=np.array([0.3]), kd=np.array([0.4]), has_reference_acceleration=False)
ID_controller = builder.AddSystem(ID_cont)
I got the controller input and output ports
control_estimated_state_input_port = ID_controller.get_input_port(0)
control_desired_state_input_port = ID_controller.get_input_port(1)
control_output_port = ID_controller.get_output_port(0)
I added a constant state source (likely wrong to do) and a state interpolator
constant_state_source = ConstantVectorSource(np.array([0.0]))
builder.AddSystem(constant_state_source)
position_to_state = StateInterpolatorWithDiscreteDerivative(controller_plant.num_positions(),
controller_plant.time_step())
builder.AddSystem(position_to_state)
I wired the controller to the plant
builder.Connect(constant_state_source.get_output_port(), position_to_state.get_input_port())
builder.Connect(position_to_state.get_output_port(), control_desired_state_input_port)
builder.Connect(plant.get_state_output_port(model_instance_1), control_estimated_state_input_port)
builder.Connect(control_output_port, plant.get_actuation_input_port(model_instance_1))
Next, I am trying to create a while loop that advances the simulation and changes the 'constant vector source' so I can feed in my position vs. time values but I'm unsure if the reason this isn't working out is because this is the complete wrong approach or if this is the right approach but I just have a few things wrong
diagram_context = diagram.CreateDefaultContext()
sim_time_temp = diagram_context.get_time()
time_step = 0.1
while sim_time_temp < duration:
ID_controller_context = diagram.GetMutableSubsystemContext(ID_controller, diagram_context)
simulator.AdvanceTo(sim_time_temp)
sim_time_temp = sim_time_temp + time_step

I added a constant state source (likely wrong to do) and a state interpolator
As you suspected, this is not the best way to go if you already have a desired sequence of positions and times that you want the system to track. Instead, you should use a TrajectorySource. Since you have a set of positions samples, positions (num_times x num_positions array), that you'd like the system to hit at specified times (num_times x 1 array), PiecewisePolynomial.CubicShapePreserving is a reasonable choice for building the trajectory.
desired_position_trajectory = PiecewisePolynomial.CubicShapePreserving(times, positions)
desired_state_source = TrajectorySource(desired_position_trajectory,
output_derivative_order=1)
builder.AddSystem(desired_state_source)
The output_derivative_order=1 argument makes desired_state_source output a [position, velocity] vector rather than just a position vector. You can connect desired_state_source directly to the controller, without an interpolator.
With this setup, you can advance the simulation all the way to duration without the need for a while loop.

Acces Cell Data from Paraview Programmable Filter

I need to create a programmable filter using Paraview.
The idea is to create a vector called Speed equal to the speed in the non-rotating part equal to the speed+rotational speed in the rotational one.
The problem is that I can't accept the value of the speed in each single cell.
input0 = inputs[0]
radius=3
Speed1=input0.PointData["U"]
K=vtk.vtkDoubleArray()
X=input0.PointData["X"]
Y=input0.PointData["Y"]
Z=input0.PointData["Z"]
pdi = self.GetInput()
numPts = pdi.GetNumberOfPoints()
for i in range(0, numPts):
if X.getvalue(i)^2+Y.getvalue(i)^2<radius:
temp=U.getvalue(i)
else:
temp=U.getvalue(i)+rot
Speed.InsertNextValue(1)
output.PointData.append(Speed, "Speed")
The problem is that X.getvalue(i) is not working.

The correct syntax is
X[i]
The documentation can be found here