Develop Reference

Compare values across multiple columns in pandas and count the instances in which values in the last column is higher than the others

I have a DataFrame that looks like this:
Image of DataFrame
What I would like to do is to compare the values in all four columns (A, B, C, and D) for every row and count the number of times in which D has the smaller value than A, B, or C for each row and add it into the 'Count' column. So for instance, 'Count' should be 1 for the second row, the third row, and 2 for the last row.
Thank you in advance!

You can use vectorize the operation using gt and sum methods along an axis:
df['Count'] = df[['A', 'B', 'C']].gt(df['D'], axis=0).sum(axis=1)
print(df)
# Output
A B C D Count
0 1 2 3 4 0
1 4 3 2 1 3
2 2 1 4 3 1

In the future, please do not post data as an image.
Use a lambda function and compare across all columns, then sum across the columns.
data = {'A': [1,47,4316,8511],
'B': [4,1,3,4],
'C': [2,7,9,1],
'D': [32,17,1,0]
}
df = pd.DataFrame(data)
df['Count'] = df.apply(lambda x: x['D'] < x, axis=1).sum(axis=1)
Output:
A B C D Count
0 1 4 2 32 0
1 47 1 7 17 1
2 4316 3 9 1 3
3 8511 4 1 0 3

How to compare two Dataframe and update particular column in one of the Dataframe? [duplicate]

I'd like to concatenate two dataframes A, B to a new one without duplicate rows (if rows in B already exist in A, don't add):
Dataframe A:
I II
0 1 2
1 3 1
Dataframe B:
I II
0 5 6
1 3 1
New Dataframe:
I II
0 1 2
1 3 1
2 5 6
How can I do this?

The simplest way is to just do the concatenation, and then drop duplicates.
>>> df1
A B
0 1 2
1 3 1
>>> df2
A B
0 5 6
1 3 1
>>> pandas.concat([df1,df2]).drop_duplicates().reset_index(drop=True)
A B
0 1 2
1 3 1
2 5 6
The reset_index(drop=True) is to fix up the index after the concat() and drop_duplicates(). Without it you will have an index of [0,1,0] instead of [0,1,2]. This could cause problems for further operations on this dataframe down the road if it isn't reset right away.

In case you have a duplicate row already in DataFrame A, then concatenating and then dropping duplicate rows, will remove rows from DataFrame A that you might want to keep.
In this case, you will need to create a new column with a cumulative count, and then drop duplicates, it all depends on your use case, but this is common in time-series data
Here is an example:
df_1 = pd.DataFrame([
{'date':'11/20/2015', 'id':4, 'value':24},
{'date':'11/20/2015', 'id':4, 'value':24},
{'date':'11/20/2015', 'id':6, 'value':34},])
df_2 = pd.DataFrame([
{'date':'11/20/2015', 'id':4, 'value':24},
{'date':'11/20/2015', 'id':6, 'value':14},
])
df_1['count'] = df_1.groupby(['date','id','value']).cumcount()
df_2['count'] = df_2.groupby(['date','id','value']).cumcount()
df_tot = pd.concat([df_1,df_2], ignore_index=False)
df_tot = df_tot.drop_duplicates()
df_tot = df_tot.drop(['count'], axis=1)
>>> df_tot
date id value
0 11/20/2015 4 24
1 11/20/2015 4 24
2 11/20/2015 6 34
1 11/20/2015 6 14

I'm surprised that pandas doesn't offer a native solution for this task.
I don't think that it's efficient to just drop the duplicates if you work with large datasets (as Rian G suggested).
It is probably most efficient to use sets to find the non-overlapping indices. Then use list comprehension to translate from index to 'row location' (boolean), which you need to access rows using iloc[,]. Below you find a function that performs the task. If you don't choose a specific column (col) to check for duplicates, then indexes will be used, as you requested. If you chose a specific column, be aware that existing duplicate entries in 'a' will remain in the result.
import pandas as pd
def append_non_duplicates(a, b, col=None):
if ((a is not None and type(a) is not pd.core.frame.DataFrame) or (b is not None and type(b) is not pd.core.frame.DataFrame)):
raise ValueError('a and b must be of type pandas.core.frame.DataFrame.')
if (a is None):
return(b)
if (b is None):
return(a)
if(col is not None):
aind = a.iloc[:,col].values
bind = b.iloc[:,col].values
else:
aind = a.index.values
bind = b.index.values
take_rows = list(set(bind)-set(aind))
take_rows = [i in take_rows for i in bind]
return(a.append( b.iloc[take_rows,:] ))
# Usage
a = pd.DataFrame([[1,2,3],[1,5,6],[1,12,13]], index=[1000,2000,5000])
b = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]], index=[1000,2000,3000])
append_non_duplicates(a,b)
# 0 1 2
# 1000 1 2 3 <- from a
# 2000 1 5 6 <- from a
# 5000 1 12 13 <- from a
# 3000 7 8 9 <- from b
append_non_duplicates(a,b,0)
# 0 1 2
# 1000 1 2 3 <- from a
# 2000 1 5 6 <- from a
# 5000 1 12 13 <- from a
# 2000 4 5 6 <- from b
# 3000 7 8 9 <- from b

Another option:
concatenation = pd.concat([
dfA,
dfB[dfB['I'].isin(dfA['I']) == False], # <-- get all the data in dfB that doesn't show up in dfB (based on values in column 'I')
])
The object concatenation will be:
I II
0 1 2
1 3 1
2 5 6

Matching on basis of a pair of columns in pandas

I have a data frame df1 with multiple columns. I have df2 with same set of columns. I want to get the records of df1 which aren't present in df2. I am able to perform this task as below:
df1[~df1['ID'].isin(df2['ID'])]
Now I want to the same operation, but on the combination of NAME and ID. This means if the NAME and ID together as a pair from df1 also exists as the same pair in df2, then that whole record should not be part of my result.
How do I accomplish this task using pandas?

I don't think that the currently accepted answer is actually correct. It was my impression that you would like to drop a value pair in df1 if that pair also exists in the other dataframe, independent of the row position that they take in the respective dataframes.
Consider the following dataframes
df1 = pd.DataFrame({'a': list('ABC'), 'b': list('CDF')})
df2 = pd.DataFrame({'a': list('ABAC'), 'b': list('CFFF')})
df1
a b
0 A C
1 B D
2 C F
df2
a b
0 A C
1 B F
2 A F
3 C F
So you would like to drop row 0 and 2 in df1. However, with the above suggestion you get
df1.isin(df2)
a b
0 True True
1 True False
2 False True
What you can do instead is
compare_cols = ['a','b']
mask = pd.Series(list(zip(*[df1[c] for c in compare_cols]]))).isin(list(zip(*[df2[c] for c in compare_cols])))
mask
0 True
1 False
2 True
dtype: bool
That is, you construct a Series of tuples from the columns you would like to compare coming from the first dataframe, and then check whether these tuples exist in the list of tuples obtained in the same way from the respective columns in the second dataframe.
Final step: df1 = df1.loc[~mask.values]
As pointed out by #rvrvrv in the comments, it is best to use mask.values instead of just mask in case df1 and mask do not have the same index (or one uses the df1 index in the construction of mask.)

It's actually pretty easy.
df1[(~df1[['ID', 'Name']].isin(df2[['ID', 'Name']])).any(axis=1)]
You pass the column names that you want to compare as a list. The interesting part is what it outputs.
Let's say df1 equals:
ID Name
0 0 0
1 1 1
2 2 2
3 3 3
4 4 4
5 5 5
6 6 6
7 7 7
8 8 8
9 1 1
And df2 equals:
ID Name
0 0 0
1 1 1
2 2 2
3 3 3
4 4 4
5 5 5
6 6 6
7 7 7
8 8 8
9 1 9
Every (ID, Name) pair between df1 and df2 matches except for row 9. The result of my answer will return:
ID Name
9 1 1
Which is exactly what you want.
In more detail, when you do the mask:
~df[['ID', 'Name']].isin(df2[['ID', 'Name']]
You get this:
ID Name
0 False False
1 False False
2 False False
3 False False
4 False False
5 False False
6 False False
7 False False
8 False False
9 False True
And we want to select the row where one of those columns is true. For this, we can add the any(axis=1) onto the end which creates:
0 False
1 False
2 False
3 False
4 False
5 False
6 False
7 False
8 False
9 True
And then when you index using this series, it will only select row 9.

Isin() would not work here as it is also comparing the index.
Let's have a look at a super powerful tool of pandas : merge()
If we consider the nice example given by user3820991, we have :
df1 = pd.DataFrame({'a': list('ABC'), 'b': list('CDF')})
df2 = pd.DataFrame({'a': list('ABAC'), 'b': list('CFFF')})
df1
a b
0 A C
1 B D
2 C F
df2
a b
0 A C
1 B F
2 A F
3 C F
The basic merge method of pandas is the 'inner' join. This will give you the equivalent of isin() method for two columns:
df1.merge(df2[['a','b']], how='inner')
a b
0 A C
1 C F
If you would like te equivalent of the not(isin()), then just change the merge method by 'outer' join (left join would work, but for the beauty of the example, you have more possibilities with the outer join).
This will give you all the rows in both dataframe, we only have to add the indicator=True to be able to select the one we want:
df1.merge(df2[['a','b']], how='outer', indicator=True)
a b _merge
0 A C both
1 B D left_only
2 C F both
3 B F right_only
4 A F right_only
We want the rows that are in df1 but not in df2, so 'left_only'. In a one liner code, you have :
pd.merge(df1, df2, on=['a','b'], how="outer", indicator=True
).query('_merge=="left_only"').drop(columns='_merge')
a b
1 B D

You can create a new column by concatenating NAME and ID and use this new column the same way you used ID in your question:
df1['temp'] = df1['NAME'].astype(str)+df1['ID'].astype(str)
df2['temp'] = df2['NAME'].astype(str)+df2['ID'].astype(str)
df1[~df1['temp'].isin(df2['temp'])].drop('temp',1)

Consolidating dataframes

I have 3 pandas data frames with matching indices. Some operations have trimmed data frames in different ways (removed rows), so that some indices in one data frame may not exist in the other.
I'd like to consolidate all 3 data frames, so they all contain rows with indices that are present in all 3 of them. How is this achievable?
import pandas as pd
data = pd.DataFrame.from_dict({'a': [1,2,3,4], 'b': [3,4,5,6], 'c': [6,7,8,9]})
a = pd.DataFrame(data['a'])
b = pd.DataFrame(data['b'])
c = pd.DataFrame(data['c'])
a = a[a['a'] <= 3]
b = b[b['b'] >= 4]
# some operation here that removes rows that aren't present in all (intersection of all dataframe's indices)
print a
a
1 2
2 3
print b
b
1 4
2 5
print c
c
1 7
2 8
Update
Sorry, I got carried away and forgot what I wanted to achieve when I wrote the examples. The actual intent was to keep the 3 dataframes separate. Apologies for the misleading example (I corrected it now).

Use merge and pass param left_index=True and right_index=True, the default type of merge is inner, so only values that exist on both left and right will be merged.
In [6]:
a.merge(b, left_index=True, right_index=True).merge(c, left_index=True, right_index=True)
Out[6]:
a b c
1 2 4 7
2 3 5 8
[2 rows x 3 columns]
To modify the original dataframes so that now only contain the rows that exist in all you can do this:
In [12]:
merged = a.merge(b, left_index=True, right_index=True).merge(c, left_index=True, right_index=True)
merged
Out[12]:
a b c
1 2 4 7
2 3 5 8
In [14]:
a = a.loc[merged.index]
b = b.loc[merged.index]
c = c.loc[merged.index]
In [15]:
print(a)
print(b)
print(c)
a
1 2
2 3
b
1 4
2 5
c
1 7
2 8
So we merge all of them on index values that are present in all of them and then use the index to filter the original dataframes.

Take a look at concat, which can be used for a variety of combination operations. Here you want to have the join type set to inner (because the want the intersection), and axis set to 1 (combining columns).
In [123]: pd.concat([a,b,c], join='inner', axis=1)
Out[123]:
a b c
1 2 4 7
2 3 5 8

How to concatenate two dataframes without duplicates?

I'd like to concatenate two dataframes A, B to a new one without duplicate rows (if rows in B already exist in A, don't add):
Dataframe A:
I II
0 1 2
1 3 1
Dataframe B:
I II
0 5 6
1 3 1
New Dataframe:
I II
0 1 2
1 3 1
2 5 6
How can I do this?

The simplest way is to just do the concatenation, and then drop duplicates.
>>> df1
A B
0 1 2
1 3 1
>>> df2
A B
0 5 6
1 3 1
>>> pandas.concat([df1,df2]).drop_duplicates().reset_index(drop=True)
A B
0 1 2
1 3 1
2 5 6
The reset_index(drop=True) is to fix up the index after the concat() and drop_duplicates(). Without it you will have an index of [0,1,0] instead of [0,1,2]. This could cause problems for further operations on this dataframe down the road if it isn't reset right away.

In case you have a duplicate row already in DataFrame A, then concatenating and then dropping duplicate rows, will remove rows from DataFrame A that you might want to keep.
In this case, you will need to create a new column with a cumulative count, and then drop duplicates, it all depends on your use case, but this is common in time-series data
Here is an example:
df_1 = pd.DataFrame([
{'date':'11/20/2015', 'id':4, 'value':24},
{'date':'11/20/2015', 'id':4, 'value':24},
{'date':'11/20/2015', 'id':6, 'value':34},])
df_2 = pd.DataFrame([
{'date':'11/20/2015', 'id':4, 'value':24},
{'date':'11/20/2015', 'id':6, 'value':14},
])
df_1['count'] = df_1.groupby(['date','id','value']).cumcount()
df_2['count'] = df_2.groupby(['date','id','value']).cumcount()
df_tot = pd.concat([df_1,df_2], ignore_index=False)
df_tot = df_tot.drop_duplicates()
df_tot = df_tot.drop(['count'], axis=1)
>>> df_tot
date id value
0 11/20/2015 4 24
1 11/20/2015 4 24
2 11/20/2015 6 34
1 11/20/2015 6 14

I'm surprised that pandas doesn't offer a native solution for this task.
I don't think that it's efficient to just drop the duplicates if you work with large datasets (as Rian G suggested).
It is probably most efficient to use sets to find the non-overlapping indices. Then use list comprehension to translate from index to 'row location' (boolean), which you need to access rows using iloc[,]. Below you find a function that performs the task. If you don't choose a specific column (col) to check for duplicates, then indexes will be used, as you requested. If you chose a specific column, be aware that existing duplicate entries in 'a' will remain in the result.
import pandas as pd
def append_non_duplicates(a, b, col=None):
if ((a is not None and type(a) is not pd.core.frame.DataFrame) or (b is not None and type(b) is not pd.core.frame.DataFrame)):
raise ValueError('a and b must be of type pandas.core.frame.DataFrame.')
if (a is None):
return(b)
if (b is None):
return(a)
if(col is not None):
aind = a.iloc[:,col].values
bind = b.iloc[:,col].values
else:
aind = a.index.values
bind = b.index.values
take_rows = list(set(bind)-set(aind))
take_rows = [i in take_rows for i in bind]
return(a.append( b.iloc[take_rows,:] ))
# Usage
a = pd.DataFrame([[1,2,3],[1,5,6],[1,12,13]], index=[1000,2000,5000])
b = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]], index=[1000,2000,3000])
append_non_duplicates(a,b)
# 0 1 2
# 1000 1 2 3 <- from a
# 2000 1 5 6 <- from a
# 5000 1 12 13 <- from a
# 3000 7 8 9 <- from b
append_non_duplicates(a,b,0)
# 0 1 2
# 1000 1 2 3 <- from a
# 2000 1 5 6 <- from a
# 5000 1 12 13 <- from a
# 2000 4 5 6 <- from b
# 3000 7 8 9 <- from b

Another option:
concatenation = pd.concat([
dfA,
dfB[dfB['I'].isin(dfA['I']) == False], # <-- get all the data in dfB that doesn't show up in dfB (based on values in column 'I')
])
The object concatenation will be:
I II
0 1 2
1 3 1
2 5 6