Develop Reference

Pandas: Comparing 2 dataframes without iterating

Considering I have 2 dataframes as shown below (DF1 and DF2), I need to compare DF2 with DF1 such that I can identify all the Matching, Different, Missing values for all the columns in DF2 that match columns in DF1 (Col1, Col2 & Col3 in this case) for rows with same EID value (A, B, C & D). I do not wish to iterate on each row of a dataframe as it can be time-consuming.
Note: There can around 70 - 100 columns. This is just a sample dataframe I am using.
DF1
EID Col1 Col2 Col3 Col4
0 A a1 b1 c1 d1
1 B a2 b2 c2 d2
2 C None b3 c3 d3
3 D a4 b4 c4 d4
4 G a5 b5 c5 d5
DF2
EID Col1 Col2 Col3
0 A a1 b1 c1
1 B a2 b2 c9
2 C a3 b3 c3
3 D a4 b4 None
Expected output dataframe
EID Col1 Col2 Col3 New_Col
0 A a1 b1 c1 Match
1 B a2 b2 c2 Different
2 C None b3 c3 Missing in DF1
3 D a4 b4 c4 Missing in DF2

Firstly, you will need to filter df1 based on df2.
new_df = df1.loc[df1['EID'].isin(df2['EID']), df2.columns]
EID Col1 Col2 Col3
0 A a1 b1 c1
1 B a2 b2 c2
2 C None b3 c3
3 D a4 b4 c4
Next, since you have a big dataframe to compare, you can change both the new_df and df2 to numpy arrays.
array1 = new_df.to_numpy()
array2 = df2.to_numpy()
Now you can compare it row-wise using np.where
new_df['New Col'] = np.where((array1 == array2).all(axis=1),'Match', 'Different')
EID Col1 Col2 Col3 New Col
0 A a1 b1 c1 Match
1 B a2 b2 c2 Different
2 C None b3 c3 Different
3 D a4 b4 c4 Different
Finally, to convert the row with None value, you can use df.loc and df.isnull
new_df.loc[new_df.isnull().any(axis=1), ['New Col']] = 'Missing in DF1'
new_df.loc[df2.isnull().any(axis=1), ['New Col']] = 'Missing in DF2'
EID Col1 Col2 Col3 New Col
0 A a1 b1 c1 Match
1 B a2 b2 c2 Different
2 C None b3 c3 Missing in DF1
3 D a4 b4 c4 Missing in DF2

One thing to note is that "Match", "Different", "Missing in DF1", and "Missing in DF1" are not mutually exclusive.
You can have some values missing in DF1, but also missing in DF2.
However, based on your post, the priority seems to be:
"Match" > "Missing in DF1" > "Missing in DF2" > "Different".
Also, it seems like you're using EID as an index, so it makes more sense to use it as the dataframe index. You can call .reset_index() if you want it as a column.
The approach is to use the equality operator / null check element-wise, then call .all and .any across columns.
import numpy as np
import pandas as pd
def compare_dfs(df1, df2):
# output dataframe has df2 dimensions, but df1 values
result = df1.reindex(index=df2.index, columns=df2.columns)
# check if values match; note that None == None, but np.nan != np.nan
eq_check = (result == df2).all(axis=1)
# null values are understood to be "missing"
# change the condition otherwise
null_check1 = result.isnull().any(axis=1)
null_check2 = df2.isnull().any(axis=1)
# create New_Col based on inferred priority
result.loc[:, "New_Col"] = None
result.loc[result["New_Col"].isnull() & eq_check, "New_Col"] = "Match"
result.loc[
result["New_Col"].isnull() & null_check1, "New_Col"
] = "Missing in DF1"
result.loc[
result["New_Col"].isnull() & null_check2, "New_Col"
] = "Missing in DF2"
result["New_Col"].fillna("Different", inplace=True)
return result
You can test your inputs in a jupyter notebook:
import itertools as it
df1 = pd.DataFrame(
np.array(["".join(i) for i in it.product(list("abcd"), list("12345"))])
.reshape((4, 5))
.T,
index=pd.Index(list("ABCDG"), name="EID"),
columns=[f"Col{i + 1}" for i in range(4)],
)
df1.loc["C", "Col1"] = None
df2 = df1.iloc[:4, :3].copy()
df2.loc["B", "Col3"] = "c9"
df2.loc["D", "Col3"] = None
display(df1)
display(df2)
display(compare_dfs(df1, df2))
Which should give these results:
Col1 Col2 Col3 Col4
EID
A a1 b1 c1 d1
B a2 b2 c2 d2
C None b3 c3 d3
D a4 b4 c4 d4
G a5 b5 c5 d5
Col1 Col2 Col3
EID
A a1 b1 c1
B a2 b2 c9
C None b3 c3
D a4 b4 None
Col1 Col2 Col3 New_Col
EID
A a1 b1 c1 Match
B a2 b2 c2 Different
C None b3 c3 Missing in DF1
D a4 b4 c4 Missing in DF2
On my i7 6600U local machine, the function takes ~1 sec for a dataset with 1 million rows, 80 columns.
rng = np.random.default_rng(seed=0)
test_size = (1_000_000, 100)
df1 = (
pd.DataFrame(rng.random(test_size))
.rename_axis(index="EID")
.rename(columns=lambda x: f"Col{x + 1}")
)
df2 = df1.sample(frac=0.8, axis=1)
# add difference
df2 += rng.random(df2.shape) > 0.9
# add nulls
df1[rng.random(df1.shape) > 0.99] = np.nan
df2[rng.random(df2.shape) > 0.99] = np.nan
%timeit compare_dfs(df1, df2)
953 ms ± 199 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Underneath it all, you're still going to be doing iterations. However, what you can do is merge the two columns on the EID, perform and outer join, and then use an apply function to generate your new_col.
df3 = pd.merge(df1, df2, on='EID', how='outer', lsuffix='df1_', rsuffix='df2_')
df3['comparison'] = df3.apply(lambda x: comparison_function(x), axis=1)
# your comparison_function will have your checks that result in missing in df1, df2, etc
You can then use

try:
#1.
DF1 = DF1.drop('Col4', axis=1)
df= pd.merge(DF2, DF1.loc[df['EID'].ne('G')], on=['Col1','Col2', 'Col3', 'EID'], how='left', indicator='New Col')
df['New Col'] = np.where(df['New Col'] == 'left_only', "Missing in DF1", df['New Col'])
df = df.merge(pd.merge(DF2.loc[:, ['EID','Col1','Col2']], DF1.loc[DF1['EID'].ne('G'), [ 'EID', 'Col1','Col2',]], on=['EID', 'Col1','Col2', ], how='left', indicator='col1_col2'), on=['EID','Col1','Col2'], how='left')
df = df.merge(pd.merge(DF2.loc[:, ['EID','Col2','Col3']], DF1.loc[DF1['EID'].ne('G'), [ 'EID', 'Col2','Col3',]], on=['EID', 'Col2','Col3', ], how='left', indicator='col2_col3'), on=['EID','Col2','Col3'], how='left')
df = df.merge(pd.merge(DF2.loc[:, ['EID','Col1','Col3']], DF1.loc[DF1['EID'].ne('G'), [ 'EID', 'Col1','Col3',]], on=['EID', 'Col1','Col3', ], how='left', indicator='col1_col3'), on=['EID','Col1','Col3'], how='left')
a1 = df['New Col'].eq('both') #match
a2 = df['col1_col2'].eq('both') & df['New Col'].eq('Missing in DF1') #same by Col1 & Col2 --> Different
a3 = df['col2_col3'].eq('both') & df['New Col'].eq('Missing in DF1') #same by Col2 & Col3 --> Different
a4 = df['col1_col3'].eq('both') & df['New Col'].eq('Missing in DF1') #same by Col1 & Col3 --> Different
df['New Col'] = np.select([a1, a2, a3, a4], ['match', 'Different/ same Col1 & Col2', 'Different/ same Col2 & Col3', 'Different/ same Col1 & Col3'], df['New Col'])
df = df.drop(columns=['col1_col2', 'col2_col3', 'col1_col3'])
EID Col1 Col2 Col3 New Col
0 A a1 b1 c1 match
1 B a2 b2 c9 Different/ same Col1 & Col2
2 C a3 b3 c3 Different/ same Col2 & Col3
3 D a4 b4 None Different/ same Col1 & Col2
or
#2.
DF1 = DF1.drop('Col4', axis=1)
df= pd.merge(DF2, DF1.loc[df['EID'].ne('G')], on=['Col1','Col2', 'Col3', 'EID'], how='left', indicator='New Col')
df['New Col'] = np.where(df['New Col'] == 'left_only', "Missing in DF1", df['New Col'])
df = df.merge(pd.merge(DF2.loc[:, ['EID','Col1','Col2']], DF1.loc[DF1['EID'].ne('G'), [ 'EID', 'Col1','Col2',]], on=['EID', 'Col1','Col2', ], how='left', indicator='col1_col2'), on=['EID','Col1','Col2'], how='left')
a1 = df['New Col'].eq('both') #match
a2 = df['col1_col2'].eq('both') & df['New Col'].eq('Missing in DF1') #Different
df['New Col'] = np.select([a1, a2], ['match', 'Different'], df['New Col'])
df = df.drop(columns=['col1_col2'])
EID Col1 Col2 Col3 New Col
0 A a1 b1 c1 match
1 B a2 b2 c9 Different
2 C a3 b3 c3 Missing in DF1
3 D a4 b4 None Different
Note1: no iteration
Note2: goal of this solution: compare DF2 with DF1 such that you can identify all the Matching, Different, Missing values for all the columns in DF2 that match columns in DF1 (Col1, Col2 & Col3 in this case) for rows with same EID value (A, B, C & D)

temp_df1 = df1[df2.columns] # to compare the only available columns in df2
joined_df = df2.merge(temp_df1, on='EID') # default indicator is '_x' for left table (df2) and '_y' for right table (df1)
# getting the columns that need to be compared
cols = list(df2.columns)
cols.remove('EID')
cols_left = [i+'_x' for i in cols]
cols_right = [i+'_y' for i in cols]
# getting back the table
temp_df2 = joined_df[cols_left]
temp_df2.columns=cols
temp_df1 = joined_df[cols_right]
temp_df1.columns=cols
output_df = joined_df[['EID']].copy()
output_df[cols] = temp_df1
filt = (temp_df2 == temp_df1).all(axis=1)
output_df.loc[filt, 'New_Col'] = 'Match'
output_df.loc[~filt, 'New_Col'] = 'Different'
output_df.loc[temp_df2.isna().any(axis=1), 'New_Col'] = 'Missing in df2' # getting missing values in df2
output_df.loc[temp_df1.isna().any(axis=1), 'New_Col'] = 'Missing in df1' # getting missing values in df1
output_df
EID Col1 Col2 Col3 New_Col
0 A a1 b1 c1 Match
1 B a2 b2 c2 Different
2 C NaN b3 c3 Missing in df1
3 D a4 b4 c4 Missing in df2

Add a list to a dataframe in Python

I have a df
col1 col2
0 1 ONE AAKLD
1 2 TWO ERBB
2 3 THE COCCNUT
3 4 WOW AACE
and I have the following lists
list1 = ['a1', 'a2', 'a3']
list2 = ['b1', 'b2', 'b3']
list3 = ['c1', 'c2', 'c3']
I want to add the list values to different columns of particular rows in the df based on condition i.e., if col2 in df has AA ,the i need the list1 to be appended to that.
Expected output:
col1 col2 1 2 3
0 1 ONE AAKLD a1 a2 a3
1 2 TWO ERBB b1 b2 b3
2 3 THE COCCNUT c1 c2 c3
3 4 WOW AACE a1 a2 a3
Thanks

Check below provides required output.
import pandas as pd
import numpy as np
df_1 = pd.DataFrame( {'col1':[1,2,3,4,], 'col2':['AA','BB','CC','AA']})
list1 = ['a1', 'a2', 'a3']
list2 = ['b1', 'b2', 'b3']
list3 = ['c1', 'c2', 'c3']
df = pd.DataFrame([list1, list2, list3], columns=['1','2','3'])
df['col2'] = df['1'].str.split('', expand=True)[1].str.upper()*2
pd.merge(df_1, df, left_on='col2',right_on='col2')
Update as per OP's comments below
Building upon earlier logic of MERGE. You can easily change MATCHING criteria if need arise.
import pandas as pd
import numpy as np
## Data Prep
df_1 = pd.DataFrame( {'col1':[1,2,3,4,], 'col2':['ONE AAKLD','TWO ERBB','THE COCCNUT','WOW AACE']})
df_1['join_col'] = 1
list1 = ['a1', 'a2', 'a3']
list2 = ['b1', 'b2', 'b3']
list3 = ['c1', 'c2', 'c3']
## Logic
df = pd.DataFrame([list1, list2, list3], columns=['1','2','3'])
df['match_col'] = df['1'].str.split('', expand=True)[1].str.upper()*2
df['join_col'] = 1
df_2 = pd.merge(df_1, df, left_on='join_col', right_on = 'join_col')
df_2['Is_Match'] = df_2[['col2', 'match_col']].apply(lambda x: x[1] in x[0] , axis = 1)
df_2[df_2['Is_Match'] == True][['col1','col2','1','2','3']]
Output:

another option is using map:
d = {'AA':list1,'BB':list2,'CC':list3}
df_1[[1,2,3]] = df_1['col2'].map(d).agg(pd.Series)
print(df_1)
'''
col1 col2 1 2 3
0 1 AA a1 a2 a3
1 2 BB b1 b2 b3
2 3 CC c1 c2 c3
3 4 AA a1 a2 a3
UPD
so it's not absolutly clear what your data is, anyway you can try this (it won't work properly if you have more than one double caracters per string):
df_1[[1,2,3]] = df_1['col2'].str.extract(r'(([A-Z])\2)')[0].map(d).agg(pd.Series)
>>> df_1
'''
col1 col2 1 2 3
0 1 ONE AAKLD a1 a2 a3
1 2 TWO ERBB b1 b2 b3
2 3 THE COCCNUT c1 c2 c3
3 4 WOW AACE a1 a2 a3

You can play with pd.concat() and Pandas' Series.
df = pd.DataFrame({"col1":[1,2,3],"col2":[4,5,6]})
lst = pd.Series([7,8,9])
pd.concat((df,lst),axis=1)

Merging two DataFrames horizontally without reindexing the first

I want to stack two DataFrames horizontally without re-indexing the first DataFrame (df1) as these indices contain some important information. However, indices on the second DataFrame (df2) has no significance and can be modified.
I could not find any way without converting the df2 to numpy and passing the indices of df1 at creation. For better understanding please find the below example.
df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3'],
'D': ['D0', 'D1', 'D2', 'D3']},
index=[0, 2, 3,4])
df2 = pd.DataFrame({'A1': ['A4', 'A5', 'A6', 'A7'],
'C': ['C4', 'C5', 'C6', 'C7'],
'D2': ['D4', 'D5', 'D6', 'D7']},
index=[ 4, 5, 6 ,7])
print(df1)
print(df2)
A B D
-------------
0 A0 B0 D0
2 A1 B1 D1
3 A2 B2 D2
4 A3 B3 D3
A1 C D2
-------------
4 A4 C4 D4
5 A5 C5 D5
6 A6 C6 D6
7 A7 C7 D7
Result I want:
A B D A1 C D2
--------------------------
0 A0 B0 D0 A4 C4 D4
2 A1 B1 D1 A5 C5 D5
3 A2 B2 D2 A6 C6 D6
4 A3 B3 D3 A7 C7 D7
PS: I would prefer a "one-shot" command to achieve this instead of using loops and adding each value.

Change the index of df2 to the index of df1 and them concatenate the dataframes:
df2.index = df1.index
pd.concat([df1, df2], axis=1)

Understanding the FutureWarning on using join_axes when concatenating with Pandas

I have two DataFrames:
df1:
A B C
1 A1 B1 C1
2 A2 B2 C2
df2:
B C D
3 B3 C3 D3
4 B4 C4 D4
Columns B and C are identical for both.
I'd like to concatenate them vertically and keep the columns of the first DataFrame:
pd.concat([df1, df2], join_axes=[df1.columns]):
A B C
1 A1 B1 C1
2 A2 B2 C2
3 NaN B3 C3
4 NaN B4 C4
This works, but raises a
FutureWarning: The join_axes-keyword is deprecated. Use .reindex or .reindex_like on the result to achieve the same functionality.
I couldn't find (either in the documentation or through Google) how to "Use .reindex or .reindex_like on the result to achieve the same functionality".
Colab notebook illustrating issue: https://colab.research.google.com/drive/13EBq2z0Nh05JY7ovrdnLGtfeqdKVvZq0

Just like what the error mentioned add reindex
pd.concat([df1,df2.reindex(columns=df1.columns)])
Out[286]:
A B C
1 A1 B1 C1
2 A2 B2 C2
3 NaN B3 C3
4 NaN B4 C4

df1 = pd.DataFrame({'A': ['A1', 'A2'], 'B': ['B1', 'B2'], 'C': ['C1', 'C2']})
df2 = pd.DataFrame({'B': ['B3', 'B4'], 'C': ['C3', 'C4'], 'D': ['D1', 'D2']})
pd.concat([df1, df2], sort=False)[df1.columns]
yields the desired result.

OR...
pd.concat([df1, df2], sort=False).reindex(df1.columns, axis=1)
Output:
A B C
1 A1 B1 C1
2 A2 B2 C2
3 NaN B3 C3
4 NaN B4 C4

groupby and remove pair records in pandas

I have a dataframe like this,
col1 col2 col3 col4
a1 b1 c1 +
a1 b1 c1 +
a1 b2 c2 +
a1 b2 c2 -
a1 b2 c2 +
If there two records with identical values in col1,col2 and col3 and opposite sign in col4, they should be removed from dataframe.
Output:
col1 col2 col3 col4
a1 b1 c1 +
a1 b1 c1 +
a1 b2 c2 +
So far I tried pandas duplicated and groupby but didn't succeeded with finding pairs. How to do this ?

I think need cumcount for count groups define all 4 columns and then groupby again with helper Series define +- groups and compare with set:
s = df.groupby(['col1','col2','col3', 'col4']).cumcount()
df = df[~df.groupby(['col1','col2','col3', s])['col4']
.transform(lambda x: set(x) == set(['+','-']))]
print (df)
col1 col2 col3 col4
0 a1 b1 c1 +
1 a1 b1 c1 +
6 a1 b2 c2 +
For better understanding create new column:
df['help'] = df.groupby(['col1','col2','col3', 'col4']).cumcount()
print (df)
col1 col2 col3 col4 help
0 a1 b1 c1 + 0
1 a1 b1 c1 + 1
2 a1 b2 c2 + 0
3 a1 b2 c2 - 0
4 a1 b2 c2 + 1
df = df[~df.groupby(['col1','col2','col3', 'help'])['col4']
.transform(lambda x: set(x) == set(['+','-']))]
print (df)
col1 col2 col3 col4 help
0 a1 b1 c1 + 0
1 a1 b1 c1 + 1
4 a1 b2 c2 + 1

Here's my attempt:
df[df.assign(ident=df.assign(count=df.col4.eq('+').astype(int))\
.groupby(['col1','col2','col3','count']).cumcount())\
.groupby(['col1','col2','col3','ident']).transform(lambda x: len(x) < 2)['col4']]
Output:
col1 col2 col3 col4
0 a1 b1 c1 +
1 a1 b1 c1 +
4 a1 b2 c2 +
On a more robust test set:
df = pd.DataFrame(
[['a1', 'b1', 'c1', '+'], ['a1', 'b1', 'c1', '+'], ['a1', 'b2', 'c2', '+'], ['a1', 'b2', 'c2', '-'], ['a1', 'b2', 'c2', '+'],
['a1','b3','c3','+'],['a1','b3','c3','-'],['a1','b3','c3','-'],['a1','b3','c3','-'],['a1','b3','c3','+'],['a1','b3','c3','+'],['a1','b3','c3','+'],['a1','b3','c3','+']],
columns=['col1', 'col2', 'col3', 'col4']
)
Input dataframe:
col1 col2 col3 col4
0 a1 b1 c1 +
1 a1 b1 c1 +
2 a1 b2 c2 +
3 a1 b2 c2 -
4 a1 b2 c2 +
5 a1 b3 c3 +
6 a1 b3 c3 -
7 a1 b3 c3 -
8 a1 b3 c3 -
9 a1 b3 c3 +
10 a1 b3 c3 +
11 a1 b3 c3 +
12 a1 b3 c3 +
df[df.assign(ident=df.assign(count=df.col4.eq('+').astype(int))\
.groupby(['col1','col2','col3','count']).cumcount())\
.groupby(['col1','col2','col3','ident']).transform(lambda x: len(x) < 2)['col4']]
Output:
col1 col2 col3 col4
0 a1 b1 c1 +
1 a1 b1 c1 +
4 a1 b2 c2 +
11 a1 b3 c3 +
12 a1 b3 c3 +

Considering the comment saying that " If there two records with identical values in col1,col2 and col3 and opposite sign in col4, they should be removed from dataframe", then:
1) Identify and drop duplicates: df.drop_duplicates()
2) Group them by the three first columns: df.groupby(['col1', 'col2', 'col3'])
3) Only keep groups that are of size 1 (otherwise, it means we have both "+" and "-"): .filter(lambda group: len(group) == 1)
All in one:
df.drop_duplicates().groupby(['col1', 'col2', 'col3']).filter(lambda g: len(g) == 1)

First, group dataframe by col1, col2 and col3. Then, apply method, that will subtract group's rows with different signs in col4.
In this method, replace values of col4, + to 1 and - to -1. Then sum values in col4(let's call variable, that keeps that sum signed_row_count). There only 2 results that are possible, either + rows will dominate(positive sum value) or - rows will(negative sum value). So, you can return new dataframe, with, either signed_row_count number of rows with + sign in col4 or signed_row_count number of rows with - sign in col4, depending on sign of the sum.
Here is code:
df = pd.DataFrame(
[['a1', 'b1', 'c1', '+'], ['a1', 'b1', 'c1', '+'], ['a1', 'b2', 'c2', '+'], ['a1', 'b2', 'c2', '-'], ['a1', 'b2', 'c2', '+']],
columns=['col1', 'col2', 'col3', 'col4']
)
print(df)
# col1 col2 col3 col4
# 0 a1 b1 c1 +
# 1 a1 b1 c1 +
# 2 a1 b2 c2 +
# 3 a1 b2 c2 -
# 4 a1 b2 c2 +
def subtract_rows(df):
signed_row_count = df['col4'].replace({'+': 1, '-': -1}).sum()
if signed_row_count >= 0:
result = pd.DataFrame([df.iloc[0][['col1', 'col2', 'col3']].tolist() + ['+']] * signed_row_count, columns=df.columns)
else:
result = pd.DataFrame([df.iloc[0][['col1', 'col2', 'col3']].tolist() + ['-']] * abs(signed_row_count), columns=df.columns)
return result
reduced_df = (df.groupby(['col1', 'col2', 'col3'])
.apply(subtract_rows)
.reset_index(drop=True))
print(reduced_df)
# col1 col2 col3 col4
# 0 a1 b1 c1 +
# 1 a1 b1 c1 +
# 2 a1 b2 c2 +