Develop Reference

How to create files from a groupby object, based on the length of the dataframe

I have a dataframe (df) that looks like this (highly simplified):
ID A B C VALUE
1 10 462 2241 217
2 11 498 6953 217
3 67 120 6926 654
4 68 898 7153 654
5 87 557 4996 654
6 88 227 6475 911
7 47 875 5097 911
8 48 143 8953 111
9 65 157 4470 111
10 66 525 9328 111
The 'VALUE' column contains a variable number of rows with identical values. I am trying to output a series of csv files that contain all of the rows that contain a 'VALUE' length == 2, ==3 etc. For example:
to_csv('/Path/to/VALUE_len_2.csv')
ID A B C VALUE
1 10 462 2241 217
2 11 498 6953 217
6 88 227 6475 911
7 47 875 5097 911
to_csv('/Path/to/VALUE_len_3.csv')
ID A B C VALUE
3 67 120 6926 654
4 68 898 7153 654
5 87 557 4996 654
to_csv('/Path/to/VALUE_len_4.csv')
ID A B C VALUE
7 47 875 5097 111
8 48 143 8953 111
9 65 157 4470 111
10 66 525 9328 111
I can get the desired output of one length value at a time, e.g., using:
df = pd.concat(v for _, v in df.groupby("VALUE") if len(v) == 2)
df.to_csv("/Path/to/VALUE_len_2.csv")
However, I have dozens of values to test. I would like to put this in a for loop on the order of:
mylist = [2,3,4,5,6,7,8,9] or len([2,3,4,5,6,7,8,9])
grouped = df.groupby(['VALUE'])
output = '/Path/to/VALUE_len_{}.csv'
for loop here:
if item in my list found in grouped:
output rows to csv
else:
pass
I've tried various constructions to iterate the groupby object using the items in the list, and I haven't been able to make anything work.
It might be an issue with trying to use a groupby object this way, but it is more than likely it is my inability to get the syntax right to complete the iteration.

It doesn't make sense to use a predetermined list to create the filenames.
df_len will be used to generate a filename using an f-string.
Path.exists() is used to determine if the file exists or not
import pandas as pd
from pathlib import Path
# test data
data = {'ID': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 'A': [10, 11, 67, 68, 87, 88, 47, 48, 65, 66], 'B': [462, 498, 120, 898, 557, 227, 875, 143, 157, 525], 'C': [2241, 6953, 6926, 7153, 4996, 6475, 5097, 8953, 4470, 9328], 'VALUE': [217, 217, 654, 654, 654, 911, 911, 111, 111, 111]}
df = pd.DataFrame(data)
# groupby value
for group, data in df.groupby('VALUE'):
# get the length of the dataframe
df_len = len(data)
# create a filename with df_len
file = Path(f'/path/to/VALUE_len_{df_len}.csv')
# if the file exists, append without the header
if file.exists():
data.to_csv(file, index=False, mode='a', header=False)
# create a new file
else:
data.to_csv(file, index=False)
If you must only create a file for dataframes of a specific length
desired_length = [2, 3, 4, 5, 6, 7, 8, 9]
# groupby value
for group, data in df.groupby('VALUE'):
# get the length of the dataframe
df_len = len(data)
# create a filename with df_len
file = Path(f'/path/to/VALUE_len_{df_len}.csv')
# check if the length of the dataframe is in the desired length
if df_len in desired_length:
# if the file exists, append without the header
if file.exists():
data.to_csv(file, index=False, mode='a', header=False)
# create a new file
else:
data.to_csv(file, index=False)

# find the VALUE_len of every VALUE first
df['VALUE_len'] = df.groupby('VALUE')['ID'].transform('count')
cols = df.columns[:-1]
# ['ID', 'A', 'B', 'C', 'VALUE']
# save group by VALUE_len
for VALUE_len, group in df.groupby('VALUE_len'):
file = Path(f'/path/to/VALUE_len_{VALUE_len}.csv')
group[cols].to_csv(file, index=False)

Create new column with various conditional logic between other columns

I have the following dataset
test = pd.DataFrame({'date':['2018-08-01','2018-08-02','2018-08-03','2019-09-01','2019-09-02','2019-09-03','2020-01-02','2020-01-03','2020-01-04','2020-10-04','2020-10-05'],
'account':['a','a','a','b','b','b','c','c','c','d','e'],
'tot_chg':[2072,2072,2072,322,322,322,483,483,483,140,570],
'denied':[1878,1036,1036,322,161,161,150,322,322,105,570],
'denied_sum':[1878,2914,3950,322,483,644,150,472,794,105,570]})
in which I would like to append a new column called denied_true based on the following parameters:
while denied_sum is less than tot_chgs, return denied
until the denied_sum exceeds tot_chgs, then compute the remaining difference between the sum of all prior denied_true less the tot_chgs
and if denied ever equals tot_chgs at the first instance, just return denied and make remaining rows for the account 0
The output should effectively look like this:
The dataframe for the output is:
output = pd.DataFrame({'date':['2018-08-01','2018-08-02','2018-08-03','2019-09-01','2019-09-02','2019-09-03','2020-01-02','2020-01-03','2020-01-04','2020-10-04','2020-10-05'],
'account':['a','a','a','b','b','b','c','c','c','d','e'],
'tot_chg':[2072,2072,2072,322,322,322,483,483,483,140,570],
'denied':[1878,1036,1036,322,161,161,150,322,322,105,570],
'denied_sum':[1878,2914,3950,322,483,644,150,472,794,105,570],
'denied_true':[1878,194,0,322,0,0,150,322,11,105,570]})
So far, I have tried the following code using where, but it's missing the condition of subtract the previous denied_true value from the tot_chgs
test['denied_true'] = test.denied_sum.to_numpy()
test.denied_true.where(test.denied_sum.le(test.tot_chg),other=0,inplace=True)
test
However, I'm not really sure how to append multiple conditions to this where function. Maybe I need if/elif loops, or a boolean mask. Any help would be greatly appreciated!

You can convert DataFrame into OrderedDict and to handle it this straightforward way:
import pandas as pd
from collections import OrderedDict
test = pd.DataFrame({'date': ['2018-08-01', '2018-08-02', '2018-08-03', '2019-09-01', '2019-09-02', '2019-09-03', '2020-01-02', '2020-01-03', '2020-01-04', '2020-10-04', '2020-10-05'],
'account': ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c', 'd', 'e'],
'tot_chg': [2072, 2072, 2072, 322, 322, 322, 483, 483, 483, 140, 570],
'denied': [1878, 1036, 1036, 322, 161, 161, 150, 322, 322, 105, 570],
'denied_sum': [1878, 2914, 3950, 322, 483, 644, 150, 472, 794, 105, 570]})
# convert DataFrame into OrderedDict
od = test.to_dict(into=OrderedDict)
# functions (samples)
def zero(dict, row):
# if denied == denied_sum
# change the dict...
return dict['denied'][row]
def ex(dict, row):
# if exceeds
# change the dict...
return 'exceed()'
def eq(dict, row):
# if equals
# change the dict...
return 'equal()'
def get_value(dict, row):
# conditions
if dict['denied'][row] == dict['denied_sum'][row]: return zero(dict, row)
if dict['denied_sum'][row] < dict['tot_chg'][row]: return dict['denied'][row]
if dict['denied_sum'][row] > dict['tot_chg'][row]: return ex(dict, row)
if dict['denied_sum'][row] == dict['tot_chg'][row]: return eq(dict, row)
# MAIN
# make a list (column) of 'denied_true' values
denied_true_list = [(row, get_value(od, row)) for row in range(len(od["date"]))]
# convert the list into a dict
denied_true_dict = {'denied_true': OrderedDict(denied_true_list)}
# add the dict to the OrderedDict
od.update(OrderedDict(denied_true_dict))
# convert the OrderedDict back into DataFrame
test = pd.DataFrame(od)
Input:
date account tot_chg denied denied_sum
0 2018-08-01 a 2072 1878 1878
1 2018-08-02 a 2072 1036 2914
2 2018-08-03 a 2072 1036 3950
3 2019-09-01 b 322 322 322
4 2019-09-02 b 322 161 483
5 2019-09-03 b 322 161 644
6 2020-01-02 c 483 150 150
7 2020-01-03 c 483 322 472
8 2020-01-04 c 483 322 794
9 2020-10-04 d 140 105 105
10 2020-10-05 e 570 570 570
Output:
date account tot_chg denied denied_sum denied_true
0 2018-08-01 a 2072 1878 1878 1878
1 2018-08-02 a 2072 1036 2914 exceed()
2 2018-08-03 a 2072 1036 3950 exceed()
3 2019-09-01 b 322 322 322 322
4 2019-09-02 b 322 161 483 exceed()
5 2019-09-03 b 322 161 644 exceed()
6 2020-01-02 c 483 150 150 150
7 2020-01-03 c 483 322 472 322
8 2020-01-04 c 483 322 794 exceed()
9 2020-10-04 d 140 105 105 105
10 2020-10-05 e 570 570 570 570
I didn't make a full implementation of your logic in the functions since it's just a sample.
About the same (probably it would be a bit easer) can be done via DataFrame > JSON > DataFrame.
Update. I've tried to implement the function ex(). Here is how it might look like.
def ex(dict, row):
# if exceeds
denied_true_slice = denied_true_list[0:row] # <-- global list
tot_chg_slice = [dict['tot_chg'][r] for r in range(row)]
denied_true_sum = sum ([v for r, v in enumerate(denied_true_slice) if tot_chg_slice[r] > v])
value = tot_chg_slice[-1] - denied_true_sum
return value if value > 0 else 0
I'm not quite sure if it works as supposed. Since I'm not fully understand the quirky conditions. But I'm sure it looks rather ugly and cryptic and probably isn't in line with best Stackoverflow's examples.
Now there is the global list, so, MAIN section now looks like this:
# MAIN
# make a list (column) of 'denied_true' values
denied_true_list = [] # <-- the global list
for row, _ in enumerate(od['date']):
denied_true_list.append(get_value(od,row))
denied_true_list = [(row, value) for row, value in enumerate(denied_true_list)]
# convert the list into a dict
denied_true_dict = {'denied_true': OrderedDict(denied_true_list)}
# add the dict to the OrderedDict
od.update(OrderedDict(denied_true_dict))
# convert the OrderedDict back into DataFrame
test = pd.DataFrame(od)
Output:
date account tot_chg denied denied_sum denied_true
0 2018-08-01 a 2072 1878 1878 1878
1 2018-08-02 a 2072 1036 2914 194
2 2018-08-03 a 2072 1036 3950 0
3 2019-09-01 b 322 322 322 322
4 2019-09-02 b 322 161 483 0
5 2019-09-03 b 322 161 644 0
6 2020-01-02 c 483 150 150 150
7 2020-01-03 c 483 322 472 322
8 2020-01-04 c 483 322 794 0
9 2020-10-04 d 140 105 105 105
10 2020-10-05 e 570 570 570 570
I believe it could be done much more pretty via native Pandas tools.

Plotting select rows and columns of DataFrame (python)

I have searched around but have not found an exact way to do this. I have a data frame of several baseball teams and their statistics. Like the following:
RK TEAM GP AB R H 2B 3B HR TB RBI AVG OBP SLG OPS
1 Milwaukee 163 5542 754 1398 252 24 218 2352 711 .252.323 .424 .747
2 Chicago Cubs 163 5624 761 1453 286 34 167 2308 722 .258.333 .410 .744
3 LA Dodgers 163 5572 804 1394 296 33 235 2461 756 .250.333 .442 .774
4 Colorado 163 5541 780 1418 280 42 210 2412 748 .256.322 .435 .757
5 Baltimore 162 5507 622 1317 242 15 188 2153 593 .239.298 .391 .689
I want to be able to plot two teams on the X-axis and then perhaps 3 metrics (ex: R, H, TB) on the Y-axis with the two teams side by side in bar chart format. I haven't been able to figure out how to do this. Any ideas?
Thank you.

My approach was to create a new dataframe which only contains the columns you are interested in plotting:
import pandas as pd
import matplotlib.pyplot as plt
data = [[1, 'Milwaukee', 163, 5542, 754, 1398, 252, 24, 218, 2352, 711, .252, .323, .424, .747],
[2, 'Chicago Cubs', 163, 5624, 761, 1453, 286, 34, 167, 2308, 722, .258, .333, .410, .744],
[3, 'LA Dodgers', 163, 5572, 804, 1394, 296, 33, 235, 2461, 756, .250, .333, .442, .774],
[4, 'Colorado', 163, 5541, 780, 1418, 280, 42, 210, 2412, 748, .256, .322, .435, .757],
[5, 'Baltimore', 162, 5507, 622, 1317, 242, 15, 188, 2153, 593, .239, .298, .391, .689]]
df = pd.DataFrame(data, columns=['RK', 'TEAM', 'GP', 'AB', 'R', 'H', '2B', '3B', 'HR', 'TB', 'RBI', 'AVG', 'OBP', 'SLG', 'OPS'])
dfplot = df[['TEAM', 'R', 'H', 'TB']].copy()
fig = plt.figure()
ax = fig.add_subplot(111)
width = 0.4
dfplot.plot(kind='bar', x='TEAM', ax=ax, width=width, position=1)
plt.show()
This creates the following output:

How to print the n-th elements of arrays with matching first items?

I am working in Python and I have the following data:
['DDX58_HUMAN', 'gnl|CDD|256537', '819', '923']
['DDX58_HUMAN', 'gnl|CDD|260076', '111', '189']
['DDX58_HUMAN', 'gnl|CDD|260076', '4', '93']
['DDX58_HUMAN', 'gnl|CDD|238005', '258', '410']
['DDX58_HUMAN', 'gnl|CDD|238034', '606', '741']
['DICER_HUMAN', 'gnl|CDD|239209', '886', '1008']
['DICER_HUMAN', 'gnl|CDD|238333', '1681', '1846']
['DICER_HUMAN', 'gnl|CDD|238333', '1296', '1376']
['DICER_HUMAN', 'gnl|CDD|238333', '1547', '1583']
['DICER_HUMAN', 'gnl|CDD|251903', '630', '722']
['DICER_HUMAN', 'gnl|CDD|238005', '58', '209']
['DICER_HUMAN', 'gnl|CDD|238034', '444', '553']
I need to print the 2nd, 3rd, and 4th items after matching first items like this:
DDX58_HUMAN gnl|CDD|256537 819 923 gnl|CDD|260076 111 189 gnl|CDD|260076 4
93 gnl|CDD|238005 258 410 gnl|CDD|238034 606 741
DICER_HUMAN gnl|CDD|239209 886 1008 gnl|CDD|238333 1681 1846 gnl|CDD|238333
1296 1376 gnl|CDD|238333 1547 1583 gnl|CDD|251903 630 722 gnl|CDD|238005 58
209 gnl|CDD|238034 444 553
How can I achieve this?

Here is a sample code for what you want to do :
I assume you have this data in python lists
you can just traverse each list , and store values in a dictionary based on first element of the list and you will be able to get the unique entries.
mylist = [['DDX58_HUMAN', 'gnl|CDD|256537', '819', '923']
,['DDX58_HUMAN', 'gnl|CDD|260076', '111', '189']
,['DDX58_HUMAN', 'gnl|CDD|260076', '4', '93']
,['DDX58_HUMAN', 'gnl|CDD|238005', '258', '410']
,['DDX58_HUMAN', 'gnl|CDD|238034', '606', '741']
,['DICER_HUMAN', 'gnl|CDD|239209', '886', '1008']
,['DICER_HUMAN', 'gnl|CDD|238333', '1681', '1846']
,['DICER_HUMAN', 'gnl|CDD|238333', '1296', '1376']
,['DICER_HUMAN', 'gnl|CDD|238333', '1547', '1583']
,['DICER_HUMAN', 'gnl|CDD|251903', '630', '722']
,['DICER_HUMAN', 'gnl|CDD|238005', '58', '209']
,['DICER_HUMAN', 'gnl|CDD|238034', '444', '553']]
myDict = {}
for items in mylist :
myDict.setdefault(items[0],[]).append(" ".join(x for x in items[1:]))
for k,v in myDict.items():
print(k," : "," ".join(x for x in v))
Output
DDX58_HUMAN : gnl|CDD|256537 819 923 gnl|CDD|260076 111 189 gnl|CDD|260076 4 93 gnl|CDD|238005 258 410 gnl|CDD|238034 606 741
DICER_HUMAN : gnl|CDD|239209 886 1008 gnl|CDD|238333 1681 1846 gnl|CDD|238333 1296 1376 gnl|CDD|238333 1547 1583 gnl|CDD|251903 630 722 gnl|CDD|238005 58 209 gnl|CDD|238034 444 553
If your data is in .txt file :
Just read the text file and remove unwanted braces using re module and then same above mentioned logic will work.
import re
with open("data.txt") as mylist :
myDict = {}
mainList = []
for items in mylist.readlines() :
dataString = re.sub(r"[\[[\]]","",items.rstrip()).split(",")
mainList.append(dataString)
myDict = {}
for items in mainList :
myDict.setdefault(items[0],[]).append("".join(x for x in items[1:]))
for k,v in myDict.items():
print(k," : ","".join(x for x in v))
Output :
'DICER_HUMAN' : 'gnl|CDD|239209' '886' '1008' 'gnl|CDD|238333' '1681' '1846' 'gnl|CDD|238333' '1296' '1376' 'gnl|CDD|238333' '1547' '1583' 'gnl|CDD|251903' '630' '722' 'gnl|CDD|238005' '58' '209' 'gnl|CDD|238034' '444' '553'
'DDX58_HUMAN' : 'gnl|CDD|256537' '819' '923' 'gnl|CDD|260076' '111' '189' 'gnl|CDD|260076' '4' '93' 'gnl|CDD|238005' '258' '410' 'gnl|CDD|238034' '606' '741'

How to convert from String to List/Array .?

I have 3 strings
a=38 186 298 345 0.93345
27 198 277 389 0.86006
33 127 293 354 0.89782
Type(a)
len(a) shows it as 22 including splace between 2 numbers
Want to convert them to list
Need as below
b=[[38 186 298 345 0.93345][27 198 277 389 0.86006][33 127 293 354 0.89782]]

Is this what you aim for:
a = '''38 186 298 345 0.93345
27 198 277 389 0.86006
33 127 293 354 0.89782'''
b = [line.split() for line in a.split('\n')]
b
#[['38', '186', '298', '345', '0.93345'],
# ['27', '198', '277', '389', '0.86006'],
# ['33', '127', '293', '354', '0.89782']]

Split them by newline and spaces. Use this python function.
string_name.split(str="")
For more info: https://www.tutorialspoint.com/python/string_split.htm

This is one solution specific to your data.
Note your inputs are not valid Python, I have resolved that below.
a1 = '38 186 298 345 0.93345'
a2 = '27 198 277 389 0.86006'
a3 = '33 127 293 354 0.89782'
res = [[float(j) if float(j) < 1 else int(j) for j in i.split()] \
for i in [a1, a2, a3]]
# [[38, 186, 298, 345, 0.93345],
# [27, 198, 277, 389, 0.86006],
# [33, 127, 293, 354, 0.89782]]