Develop Reference

Add a column matching other two columns, one contains representative values

I have three data frames as follows:
df1
col1 CAND_SNP
1 a1
1 a2
1 a3
1 a4
2 b1
3 c1
3 c2
3 c3
df2
col1 LEAD_SNP
1 a1
2 b1
3 c1
df3
snp col2
a3 x1
a21 x2
a31 x3
a41 x4
b11 x5
c11 x6
c21 x7
c31 x8
I need to match CAND_SNP of df1 with snp of df3 to populate a new column in df2 with values "yes" or "no". The match needs to be groupwise for col1 of df1. In the above example, there are 3 groups in col1 of df1. If any of these group's corresponding value in CAND_SNP matches with snp of df3 then the new column of df2 would be "yes" as below: Any help?
df2
col1 LEAD_SNP col3
1 a1 Yes
2 b1 No
3 c1 No

If I understand correctly, you can group df1 by col1 and look up whether a value of col2 exists in col1 of df3. Then merge with df2:
df1['col3'] = df1.groupby('col1')['CAND_SNP'].apply(lambda s: s.isin(df3['snp']))
df2 = df2.merge(df1.groupby('col1')['col3'].any(), left_on='col1', right_index=True, how='left')
And if you need 'Yes'/'No' as values, use
df2.col3.map({True: 'Yes', False: 'No'})

Split column based on input string into multiple columns in pandas python

I have below pandas data frame and I am trying to split col1 into multiple columns based on split_format string.
Inputs:
split_format = 'id-id1_id2|id3'
data = {'col1':['a-a1_a2|a3', 'b-b1_b2|b3', 'c-c1_c2|c3', 'd-d1_d2|d3'],
'col2':[20, 21, 19, 18]}
df = pd.DataFrame(data).style.hide_index()
df
col1 col2
a-a1_a2|a3 20
b-b1_b2|b3 21
c-c1_c2|c3 19
d-d1_d2|d3 18
Expected Output:
id id1 id2 id3 col2
a a1 a2 a3 20
b b1 b2 b3 21
c c1 c2 c3 19
d d1 d2 d3 18
**Note: The special characters and column name in split_string can be changed.

I think I am able to figure it out.
col_name = re.split('[^0-9a-zA-Z]+',split_format)
df[col_name] = df['col1'].str.split('[^0-9a-zA-Z]+',expand=True)
del df['col1']
df
col2 id id1 id2 id3
0 20 a a1 a2 a3
1 21 b b1 b2 b3
2 19 c c1 c2 c3
3 18 d d1 d2 d3

I parse the symbols and then recursively evaluate the resulting strings from the token split on the string. I flatten the resulting list and their recursive evaluate the resulting list until all the symbols have been evaluated.
split_format = 'id-id1_id2|id3'
data = {'col1':['a-a1_a2|a3', 'b-b1_b2|b3', 'c-c1_c2|c3', 'd-d1_d2|d3'],
'col2':[20, 21, 19, 18]}
df = pd.DataFrame(data)
symbols=[]
for x in split_format:
if x.isalnum()==False:
symbols.append(x)
result=[]
def parseTree(stringlist,symbols,result):
#print("String list",stringlist)
if len(symbols)==0:
[result.append(x) for x in stringlist]
return
token=symbols.pop(0)
elements=[]
for item in stringlist:
elements.append(item.split(token))
flat_list = [item for sublist in elements for item in sublist]
parseTree(flat_list,symbols,result)
df2=pd.DataFrame(columns=["id","id1","id2","id3"])
for key, item in df.iterrows():
symbols2=symbols.copy()
value=item['col1']
parseTree([value],symbols2,result)
a_series = pd. Series(result, index = df2.columns)
df2=df2.append(a_series, ignore_index=True)
result.clear()
df2['col2']=df['col2']
print(df2)
output:
id id1 id2 id3 col2
0 a a1 a2 a3 20
1 b b1 b2 b3 21
2 c c1 c2 c3 19
3 d d1 d2 d3 18

How to compute each cell as a function of index and column?

I have a use-case where it naturally fits to compute each cell of a pd.DataFrame as a function of the corresponding index and column i.e.
import pandas as pd
import numpy as np
data = np.empty((3, 3))
data[:] = np.nan
df = pd.DataFrame(data=data, index=[1, 2, 3], columns=['a', 'b', 'c'])
print(df)
> a b c
>1 NaN NaN NaN
>2 NaN NaN NaN
>3 NaN NaN NaN
and I'd like (this is only a mock example) to get a result that is a function f(index, column):
> a b c
>1 a1 b1 c1
>2 a2 b2 c2
>3 a3 b3 c3
In order to accomplish this I need a way different to apply or applymap where the lambda gets the coordinates in terms of the index and col i.e.
def my_cell_map(ix, col):
return col + str(ix)

Here is possible use numpy - add index values to columns with broadcasting and pass to DataFrame constructor:
a = df.columns.to_numpy() + df.index.astype(str).to_numpy()[:, None]
df = pd.DataFrame(a, index=df.index, columns=df.columns)
print (df)
a b c
1 a1 b1 c1
2 a2 b2 c2
3 a3 b3 c3
EDIT: For processing by columns names is possible use x.name with index values:
def f(x):
return x.name + x.index.astype(str)
df = df.apply(f)
print (df)
a b c
1 a1 b1 c1
2 a2 b2 c2
3 a3 b3 c3
EDIT1: For your function is necessary use another lambda function for loop by index values:
def my_cell_map(ix, col):
return col + str(ix)
def f(x):
return x.index.map(lambda y: my_cell_map(y, x.name))
df = df.apply(f)
print (df)
a b c
1 a1 b1 c1
2 a2 b2 c2
3 a3 b3 c3
EDIT2: Also is possible loop by index and columns values and set by loc, if large DataFrame performance should be slow:
for c in df.columns:
for i in df.index:
df.loc[i, c] = my_cell_map(i, c)
print (df)
a b c
1 a1 b1 c1
2 a2 b2 c2
3 a3 b3 c3

How to merge duplicates as new columns

I am attempting to merge to dataframes on 1 column for which I would like the output of duplicates to be an extra column instead of a new row.
What happens now:
df1 = pd.DataFrame({'A': ['A0'],
'B': ['B0']})
df2 = pd.DataFrame({'A': ['A0', 'A0'],
'C': ['C4', 'C5']})
df1.merge(df2, on = 'A', how = 'left')
Gives the output:
A B C
0 A0 B0 C4
1 A0 B0 C5
What I would like the output to be:
A B C_1 C_2
0 A0 B0 C4 C5
Thanks!

Create unique values of column A in df2 by MultiIndex by DataFrame.set_index with counter column by GroupBy.cumcount, reshape by Series.unstack and flatten Multiindex by map with join::
df2 = df2.set_index(['A', df2.groupby('A').cumcount().add(1).astype(str)]).unstack()
df2.columns = df2.columns.map('_'.join)
df2 = df2.reset_index()
print (df2)
A C_1 C_2
0 A0 C4 C5
df = df1.merge(df2, on = 'A', how = 'left')
print (df)
A B C_1 C_2
0 A0 B0 C4 C5

In one line of code:
df1.merge(df2.assign(Cs=range(0,len(df2))).pivot(index='A',columns='Cs'),on='A')
A B (C, 0) (C, 1)
0 A0 B0 C4 C5

groupby and remove pair records in pandas

I have a dataframe like this,
col1 col2 col3 col4
a1 b1 c1 +
a1 b1 c1 +
a1 b2 c2 +
a1 b2 c2 -
a1 b2 c2 +
If there two records with identical values in col1,col2 and col3 and opposite sign in col4, they should be removed from dataframe.
Output:
col1 col2 col3 col4
a1 b1 c1 +
a1 b1 c1 +
a1 b2 c2 +
So far I tried pandas duplicated and groupby but didn't succeeded with finding pairs. How to do this ?

I think need cumcount for count groups define all 4 columns and then groupby again with helper Series define +- groups and compare with set:
s = df.groupby(['col1','col2','col3', 'col4']).cumcount()
df = df[~df.groupby(['col1','col2','col3', s])['col4']
.transform(lambda x: set(x) == set(['+','-']))]
print (df)
col1 col2 col3 col4
0 a1 b1 c1 +
1 a1 b1 c1 +
6 a1 b2 c2 +
For better understanding create new column:
df['help'] = df.groupby(['col1','col2','col3', 'col4']).cumcount()
print (df)
col1 col2 col3 col4 help
0 a1 b1 c1 + 0
1 a1 b1 c1 + 1
2 a1 b2 c2 + 0
3 a1 b2 c2 - 0
4 a1 b2 c2 + 1
df = df[~df.groupby(['col1','col2','col3', 'help'])['col4']
.transform(lambda x: set(x) == set(['+','-']))]
print (df)
col1 col2 col3 col4 help
0 a1 b1 c1 + 0
1 a1 b1 c1 + 1
4 a1 b2 c2 + 1

Here's my attempt:
df[df.assign(ident=df.assign(count=df.col4.eq('+').astype(int))\
.groupby(['col1','col2','col3','count']).cumcount())\
.groupby(['col1','col2','col3','ident']).transform(lambda x: len(x) < 2)['col4']]
Output:
col1 col2 col3 col4
0 a1 b1 c1 +
1 a1 b1 c1 +
4 a1 b2 c2 +
On a more robust test set:
df = pd.DataFrame(
[['a1', 'b1', 'c1', '+'], ['a1', 'b1', 'c1', '+'], ['a1', 'b2', 'c2', '+'], ['a1', 'b2', 'c2', '-'], ['a1', 'b2', 'c2', '+'],
['a1','b3','c3','+'],['a1','b3','c3','-'],['a1','b3','c3','-'],['a1','b3','c3','-'],['a1','b3','c3','+'],['a1','b3','c3','+'],['a1','b3','c3','+'],['a1','b3','c3','+']],
columns=['col1', 'col2', 'col3', 'col4']
)
Input dataframe:
col1 col2 col3 col4
0 a1 b1 c1 +
1 a1 b1 c1 +
2 a1 b2 c2 +
3 a1 b2 c2 -
4 a1 b2 c2 +
5 a1 b3 c3 +
6 a1 b3 c3 -
7 a1 b3 c3 -
8 a1 b3 c3 -
9 a1 b3 c3 +
10 a1 b3 c3 +
11 a1 b3 c3 +
12 a1 b3 c3 +
df[df.assign(ident=df.assign(count=df.col4.eq('+').astype(int))\
.groupby(['col1','col2','col3','count']).cumcount())\
.groupby(['col1','col2','col3','ident']).transform(lambda x: len(x) < 2)['col4']]
Output:
col1 col2 col3 col4
0 a1 b1 c1 +
1 a1 b1 c1 +
4 a1 b2 c2 +
11 a1 b3 c3 +
12 a1 b3 c3 +

Considering the comment saying that " If there two records with identical values in col1,col2 and col3 and opposite sign in col4, they should be removed from dataframe", then:
1) Identify and drop duplicates: df.drop_duplicates()
2) Group them by the three first columns: df.groupby(['col1', 'col2', 'col3'])
3) Only keep groups that are of size 1 (otherwise, it means we have both "+" and "-"): .filter(lambda group: len(group) == 1)
All in one:
df.drop_duplicates().groupby(['col1', 'col2', 'col3']).filter(lambda g: len(g) == 1)

First, group dataframe by col1, col2 and col3. Then, apply method, that will subtract group's rows with different signs in col4.
In this method, replace values of col4, + to 1 and - to -1. Then sum values in col4(let's call variable, that keeps that sum signed_row_count). There only 2 results that are possible, either + rows will dominate(positive sum value) or - rows will(negative sum value). So, you can return new dataframe, with, either signed_row_count number of rows with + sign in col4 or signed_row_count number of rows with - sign in col4, depending on sign of the sum.
Here is code:
df = pd.DataFrame(
[['a1', 'b1', 'c1', '+'], ['a1', 'b1', 'c1', '+'], ['a1', 'b2', 'c2', '+'], ['a1', 'b2', 'c2', '-'], ['a1', 'b2', 'c2', '+']],
columns=['col1', 'col2', 'col3', 'col4']
)
print(df)
# col1 col2 col3 col4
# 0 a1 b1 c1 +
# 1 a1 b1 c1 +
# 2 a1 b2 c2 +
# 3 a1 b2 c2 -
# 4 a1 b2 c2 +
def subtract_rows(df):
signed_row_count = df['col4'].replace({'+': 1, '-': -1}).sum()
if signed_row_count >= 0:
result = pd.DataFrame([df.iloc[0][['col1', 'col2', 'col3']].tolist() + ['+']] * signed_row_count, columns=df.columns)
else:
result = pd.DataFrame([df.iloc[0][['col1', 'col2', 'col3']].tolist() + ['-']] * abs(signed_row_count), columns=df.columns)
return result
reduced_df = (df.groupby(['col1', 'col2', 'col3'])
.apply(subtract_rows)
.reset_index(drop=True))
print(reduced_df)
# col1 col2 col3 col4
# 0 a1 b1 c1 +
# 1 a1 b1 c1 +
# 2 a1 b2 c2 +