Develop Reference

How to add a total column that sums up values from dynamic columns

I have the following DataFrame
party 2022 - 45 2022 - 46 2022 - 48 2022 - 49 2022 - 50 2022 - 51 2022 - 52 2023 - 01 2023 - 02 2023 - 06 2023 - 10 scheduled_total ledger_balance
0 V00011 0.00 0.0 1917.50 6894.00 5743.50 3826.00 3826.00 0.00 0.00 0.00 0.00 NaN -37145.00
1 V00020 7327.22 0.0 0.00 5652.00 5652.00 5652.00 0.00 0.00 0.00 0.00 0.00 NaN -45863.01
2 V00117 0.00 0.0 2265.50 3776.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NaN -8589.50
3 V00144 23986.55 0.0 0.00 11629.63 11629.63 11629.63 11629.63 11629.63 0.00 0.00 0.00 NaN -276629.91
4 V00153 0.00 0.0 1794.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NaN -8769.00
5 V00198 0.00 2655.0 2655.00 2655.00 2655.00 2655.00 0.00 0.00 0.00 0.00 0.00 NaN -10620.00
6 V00229 11868.53 0.0 7327.75 14837.50 14837.50 14837.50 0.00 0.00 0.00 0.00 0.00 NaN -103789.08
7 V00235 0.00 0.0 9600.00 9600.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NaN -43200.00
8 V00241 0.00 0.0 5575.50 5575.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NaN -11151.00
9 V00261 0.00 0.0 6208.50 17201.75 11502.25 5630.50 763.50 0.00 0.00 0.00 0.00 NaN -34131.22
10 V00319 0.00 0.0 0.00 13865.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 NaN -27731.00
11 V00325 0.00 0.0 0.00 0.00 850.00 0.00 0.00 0.00 0.00 0.00 0.00 NaN -3568.00
12 V00345 0.00 0.0 0.00 5000.25 0.00 0.00 0.00 0.00 5000.25 5000.25 5000.25 NaN -20001.00
total NaN 43182.30 2655.0 37343.75 96686.63 52869.88 44230.63 16219.13 11629.63 5000.25 5000.25 5000.25 NaN -631187.72
The columns after party, till scheduled_total are dynamic columns created using pivot. It represents "year - week". I want to get total of each row in "scheduled_total". Any help?
Here is the code used to generate the above output:
pr = frappe.db.get_list("Payment Request",filters={"docstatus":1,"status":"Initiated","party_type":"Supplier"},fields=["name", "transaction_date", "payment_request_type", "party", "supplier_name", "reference_name", "grand_total", "status"])
df = pd.DataFrame.from_records(pr)
df['transaction_date'] = pd.to_datetime(df['transaction_date'])
df["week"] = df['transaction_date'].dt.year.astype(str) + " - " + df['transaction_date'].dt.week.map("{:02}".format).astype(str)
df = df.groupby(["party", "week", "supplier_name"],as_index=False)["grand_total"].sum()
df = df.sort_values(by=['week'],ascending=True)
df = df.pivot(index=["party","supplier_name"], columns='week', values='grand_total').reset_index().rename_axis(None, axis=1).fillna(0)
df["scheduled_total"] = df.agg("sum", axis=0)
df["ledger_balance"] = df.apply(lambda x: get_balance_on(party_type="Supplier",party=x['party']), axis=1)
df.loc['total'] = df.sum(numeric_only=True)
print(df.to_string())
df.to_excel("/home/frappe/frappe-bench/apps/zarnik/zarnik/ap_automation/report/payment_request_summary/payment_request_summary.xlsx")
column_names = df.columns.values
print(column_names)
I have tried the following,
df["scheduled_total"] = df.agg("sum", axis=0)
This returns NaN.

create pandas column with conditionals

The value of column A can be -1, 0 and 1. How to create a new column with conditions: column A; value different from 0 in the line below (-1,1), add the value of column X, if value of column X equals 0 then add the next value different from 0. In the other lines of this new column, assign the value 0.
import pandas as pd
d = {'a': [-1,-1,-1,0,0,-1,-1,-1,-1,-1,-1,-1,-1,0,0,0,0,0,1,1,1,0,0,0,0,1,1,1,1,1],
'x': [0.00,-2.13,0.00,0.00,0.00,0.00,0.00,0.21,-0.63,0.00,0.29,-0.11,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.62,0.00,
-0.36,0.00,-0.03,0.00,0.00,0.00,0.22,0.05,0.00]}
df = pd.DataFrame(data=d)
df
desired result
a x r
0 -1 0.00 0.00
1 -1 -2.13 0.00
2 -1 0.00 0.00
3 0 0.00 0.00
4 0 0.00 0.00
5 -1 0.00 0.00
6 -1 0.00 0.00
7 -1 0.21 0.21
8 -1 -0.63 0.00
9 -1 0.00 0.00
10 -1 0.29 0.00
11 -1 -0.11 0.00
12 -1 0.00 0.00
13 0 0.00 0.00
14 0 0.00 0.00
15 0 0.00 0.00
16 0 0.00 0.00
17 0 0.00 0.00
18 1 0.00 0.00
19 1 0.62 0.62
20 1 0.00 0.00
21 0 -0.36 0.00
22 0 0.00 0.00
23 0 -0.03 0.00
24 0 0.00 0.00
25 1 0.00 0.00
26 1 0.00 0.00
27 1 0.22 0.22
28 1 0.05 0.00
29 1 0.00 0.00

CPUs not fully used when using Python Ray for parallel processing

first post here, so first of all thanks a lot for this insanely helpful community!
I am running a parallel bioinformatics calculation using Ray with Python 3.6.
I've set it up to run on an AWS EC2 instance with 64 cores on Ubuntu 18. I have to add I am fairly new to Python and especially Ray.
The code is:
#ray.remote
def calculate_stuff(sequence): # computationally intensive function calling on external SWIG wrapped C-Code
fc = external_function(sequence)
return fc
ray.init()
result_ids = []
for i in permutlist: # permutlist is large list containing large strings (300 - 1000 chars)
result_ids.append(calculate_stuff.remote(i))
allmfe = ray.get(result_ids)
This code runs for quite some hours, one loop of calculate_stuff takes ~1s.
I run the script using tmux, when I leave the session and check for CPU usage with mpstat -P ALL
the output is:
09:37:22 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
09:37:22 all 62.16 0.00 0.26 0.00 0.00 0.01 0.00 0.00 0.00 37.56
09:37:22 0 62.28 0.00 0.26 0.00 0.00 0.05 0.00 0.00 0.00 37.41
09:37:22 1 62.22 0.00 0.25 0.00 0.00 0.02 0.00 0.00 0.00 37.51
09:37:22 2 62.24 0.00 0.26 0.00 0.00 0.01 0.00 0.00 0.00 37.48
09:37:22 3 62.27 0.00 0.26 0.00 0.00 0.01 0.00 0.00 0.00 37.46
09:37:22 4 62.37 0.00 0.29 0.00 0.00 0.01 0.00 0.00 0.00 37.33
09:37:22 5 62.23 0.00 0.28 0.00 0.00 0.01 0.00 0.00 0.00 37.48
09:37:22 6 62.24 0.00 0.27 0.00 0.00 0.01 0.00 0.00 0.00 37.48
This looks the same for all 64 cores.
My question now: Why do I get only 2/3 CPU usage and wasting 1/3 on idling?
Anyone who can help me save time and therefore money more than welcome. Thanks!

Normalising data for plotting

I am trying to plot the data shown below in a normalised way, in order to have the maximum value on the y-axis equal to 1.
Dataset:
%_F %_M %_C %_D Label
0 0.00 0.00 0.08 0.05 0.0
1 0.00 0.00 0.00 0.14 0.0
2 0.00 0.00 0.10 0.01 1.0
3 0.01 0.01 0.07 0.05 1.0
4 0.00 0.00 0.07 0.14 0.0
6 0.00 0.00 0.07 0.05 0.0
7 0.00 0.00 0.05 0.68 0.0
8 0.00 0.00 0.03 0.09 0.0
9 0.00 0.00 0.04 0.02 0.0
10 0.00 0.00 0.06 0.02 0.0
I tried as follows:
cols_to_norm = ["%_F", "%_M", "%_C", "%_D"]
df[cols_to_norm] = df[cols_to_norm].apply(lambda x: (x - x.min()) / (x.max() - x.min()))
but I am not completely sure about the output.
In fact, if a plot as follows
df.pivot_table(index='Label').plot.bar()
I get a different result. I think it is because I am not considering in the first code the index on Label.

there are multiple techniques normalize
this shows technique which uses native pandas
import io
df = pd.read_csv(io.StringIO(""" %_F %_M %_C %_D Label
0 0.00 0.00 0.08 0.05 0.0
1 0.00 0.00 0.00 0.14 0.0
2 0.00 0.00 0.10 0.01 1.0
3 0.01 0.01 0.07 0.05 1.0
4 0.00 0.00 0.07 0.14 0.0
6 0.00 0.00 0.07 0.05 0.0
7 0.00 0.00 0.05 0.68 0.0
8 0.00 0.00 0.03 0.09 0.0
9 0.00 0.00 0.04 0.02 0.0
10 0.00 0.00 0.06 0.02 0.0"""), sep="\s+")
fig, ax = plt.subplots(2, figsize=[10,6])
df2 = (df-df.min())/(df.max()-df.min())
df.plot(ax=ax[0], kind="line")
df2.plot(ax=ax[1], kind="line")

Why is there a performance difference between the order of a nested loop?

I have a process that loops through two lists, one being relatively large while the other being significantly smaller.
Example:
larger_list = list(range(15000))
smaller_list = list(range(2500))
for ll in larger_list:
for sl in smaller_list:
pass
I scaled the sized down of the lists to test performance, and I noticed there is a decent difference between which list is looped through first.
import timeit
larger_list = list(range(150))
smaller_list = list(range(25))
def large_then_small():
for ll in larger_list:
for sl in smaller_list:
pass
def small_then_large():
for sl in smaller_list:
for ll in larger_list:
pass
print('Larger -> Smaller: {}'.format(timeit.timeit(large_then_small)))
print('Smaller -> Larger: {}'.format(timeit.timeit(small_then_large)))
>>> Larger -> Smaller: 114.884992572
>>> Smaller -> Larger: 98.7751009799
At first glance, they look identical - however there is 16 second difference between the two functions.
Why is that?

When you disassemble one of your functions you get:
>>> dis.dis(small_then_large)
2 0 SETUP_LOOP 31 (to 34)
3 LOAD_GLOBAL 0 (smaller_list)
6 GET_ITER
>> 7 FOR_ITER 23 (to 33)
10 STORE_FAST 0 (sl)
3 13 SETUP_LOOP 14 (to 30)
16 LOAD_GLOBAL 1 (larger_list)
19 GET_ITER
>> 20 FOR_ITER 6 (to 29)
23 STORE_FAST 1 (ll)
4 26 JUMP_ABSOLUTE 20
>> 29 POP_BLOCK
>> 30 JUMP_ABSOLUTE 7
>> 33 POP_BLOCK
>> 34 LOAD_CONST 0 (None)
37 RETURN_VALUE
>>>
Looking at address 29 & 30, it looks like these will execute every time the inner loop ends. The two loops look basically the same, but these two instructions are executed each time the inner loop exits. Having the smaller number on the inside would cause these to be executed more often, hence increasing the time (vs the larger number on the inner loop).

This same phenomenon was under discussion in this duplicate and got me interested in what goes on in the C land of CPython. Built python with:
% ./configure --enable-profiling
% make coverage
Tests
% ./python -c "larger_list = list(range(15000))
smaller_list = list(range(2500))
for sl in smaller_list:
for ll in larger_list:
pass"
% mv gmon.out soflgmon.out
% ./python -c "larger_list = list(range(15000))
smaller_list = list(range(2500))
for ll in larger_list:
for sl in smaller_list:
pass"
% mv gmon.out lofsgmon.out
Results
Short list of long lists (total time for a single run 1.60):
% gprof python soflgmon.out|head -n40
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
46.25 0.74 0.74 3346 0.00 0.00 PyEval_EvalFrameEx
25.62 1.15 0.41 37518735 0.00 0.00 insertdict
14.38 1.38 0.23 37555121 0.00 0.00 lookdict_unicode_nodummy
7.81 1.50 0.12 37506675 0.00 0.00 listiter_next
4.06 1.57 0.07 37516233 0.00 0.00 PyDict_SetItem
0.62 1.58 0.01 2095 0.00 0.00 _PyEval_EvalCodeWithName
0.62 1.59 0.01 3 0.00 0.00 untrack_dicts
0.31 1.59 0.01 _PyDict_SetItem_KnownHash
0.31 1.60 0.01 listiter_len
0.00 1.60 0.00 87268 0.00 0.00 visit_decref
0.00 1.60 0.00 73592 0.00 0.00 visit_reachable
0.00 1.60 0.00 71261 0.00 0.00 _PyThreadState_UncheckedGet
0.00 1.60 0.00 49742 0.00 0.00 _PyObject_Alloc
0.00 1.60 0.00 48922 0.00 0.00 PyObject_Malloc
0.00 1.60 0.00 48922 0.00 0.00 _PyObject_Malloc
0.00 1.60 0.00 47487 0.00 0.00 PyDict_GetItem
0.00 1.60 0.00 44246 0.00 0.00 _PyObject_Free
0.00 1.60 0.00 43637 0.00 0.00 PyObject_Free
0.00 1.60 0.00 30034 0.00 0.00 slotptr
0.00 1.60 0.00 24892 0.00 0.00 type_is_gc
0.00 1.60 0.00 24170 0.00 0.00 r_byte
0.00 1.60 0.00 23774 0.00 0.00 PyErr_Occurred
0.00 1.60 0.00 20371 0.00 0.00 _PyType_Lookup
0.00 1.60 0.00 19930 0.00 0.00 PyLong_FromLong
0.00 1.60 0.00 19758 0.00 0.00 r_string
0.00 1.60 0.00 19080 0.00 0.00 _PyLong_New
0.00 1.60 0.00 18887 0.00 0.00 lookdict_unicode
0.00 1.60 0.00 18878 0.00 0.00 long_dealloc
0.00 1.60 0.00 17639 0.00 0.00 PyUnicode_InternInPlace
0.00 1.60 0.00 17502 0.00 0.00 rangeiter_next
0.00 1.60 0.00 14776 0.00 0.00 PyObject_GC_UnTrack
0.00 1.60 0.00 14578 0.00 0.00 descr_traverse
0.00 1.60 0.00 13520 0.00 0.00 r_long
0.00 1.60 0.00 13058 0.00 0.00 PyUnicode_New
0.00 1.60 0.00 12298 0.00 0.00 _Py_CheckFunctionResult
...
Long list of short lists (total time for a single run 1.64):
gprof python lofsgmon.out|head -n40
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
48.78 0.80 0.80 3346 0.00 0.00 PyEval_EvalFrameEx
17.99 1.09 0.29 37531168 0.00 0.00 insertdict
11.59 1.28 0.19 37531675 0.00 0.00 listiter_next
11.28 1.47 0.18 37580156 0.00 0.00 lookdict_unicode_nodummy
6.71 1.58 0.11 37528666 0.00 0.00 PyDict_SetItem
1.22 1.60 0.02 _PyDict_SetItem_KnownHash
0.61 1.61 0.01 5525 0.00 0.00 update_one_slot
0.61 1.62 0.01 120 0.00 0.00 PyDict_Merge
0.30 1.62 0.01 18178 0.00 0.00 lookdict_unicode
0.30 1.63 0.01 11988 0.00 0.00 insertdict_clean
0.30 1.64 0.01 listiter_len
0.30 1.64 0.01 listiter_traverse
0.00 1.64 0.00 96089 0.00 0.00 _PyThreadState_UncheckedGet
0.00 1.64 0.00 87245 0.00 0.00 visit_decref
0.00 1.64 0.00 74743 0.00 0.00 visit_reachable
0.00 1.64 0.00 62232 0.00 0.00 _PyObject_Alloc
0.00 1.64 0.00 61412 0.00 0.00 PyObject_Malloc
0.00 1.64 0.00 61412 0.00 0.00 _PyObject_Malloc
0.00 1.64 0.00 59815 0.00 0.00 PyDict_GetItem
0.00 1.64 0.00 55231 0.00 0.00 _PyObject_Free
0.00 1.64 0.00 54622 0.00 0.00 PyObject_Free
0.00 1.64 0.00 36274 0.00 0.00 PyErr_Occurred
0.00 1.64 0.00 30034 0.00 0.00 slotptr
0.00 1.64 0.00 24929 0.00 0.00 type_is_gc
0.00 1.64 0.00 24617 0.00 0.00 _PyObject_GC_Alloc
0.00 1.64 0.00 24617 0.00 0.00 _PyObject_GC_Malloc
0.00 1.64 0.00 24170 0.00 0.00 r_byte
0.00 1.64 0.00 20958 0.00 0.00 PyObject_GC_Del
0.00 1.64 0.00 20371 0.00 0.00 _PyType_Lookup
0.00 1.64 0.00 19918 0.00 0.00 PyLong_FromLong
0.00 1.64 0.00 19758 0.00 0.00 r_string
0.00 1.64 0.00 19068 0.00 0.00 _PyLong_New
0.00 1.64 0.00 18845 0.00 0.00 long_dealloc
0.00 1.64 0.00 18507 0.00 0.00 _PyObject_GC_New
0.00 1.64 0.00 17639 0.00 0.00 PyUnicode_InternInPlace
...
The difference is marginal (2.4%), and profiling adds to run time, so it is difficult to say how much it actually would've been. The total time also includes the creation of the test lists, so that hides the true difference further.
The reason for the 16s difference in the original test is that timeit.timeit runs the given statement or function number=1000000 times by default, so that would add up to a whopping 40,000s in this case. Don't quote that value though, as it is an artifact of profiling. With your original test code and non profiling python3 on this machine I get:
Larger -> Smaller: 40.29234626500056
Smaller -> Larger: 33.09413992699956
which would mean a difference of
In [1]: (40.29234626500056-33.09413992699956)/1000000
Out[1]: 7.198206338001e-06
per single run (7.2µs), 18% in total.
So as stated in the former answer, POP_BLOCK gets executed more, but it's not just that, but the whole inner loop setup:
0.00 1.64 0.00 16521 0.00 0.00 PyFrame_BlockSetup
0.00 1.64 0.00 16154 0.00 0.00 PyFrame_BlockPop
Compared to the short list of long lists:
0.00 1.60 0.00 4021 0.00 0.00 PyFrame_BlockSetup
0.00 1.60 0.00 3748 0.00 0.00 set_next
0.00 1.60 0.00 3654 0.00 0.00 PyFrame_BlockPop
That has negligible impact though.