Develop Reference

Get pandas column where two column values are equal

I want to subset a DataFrame by two columns in different dataframes if the values in the columns are the same. Here is an example of df1 and df2:
df1
A
0 apple
1 pear
2 orange
3 apple
df2
B
0 apple
1 orange
2 orange
3 pear
I would like the output to be a subsetted df1 based upon the df2 column:
A
0 apple
2 orange
I tried
df1 = df1[df1.A == df2.B] but get the following error:
ValueError: Can only compare identically-labeled Series objects
I do not want to rename the column in either.
What is the best way to do this? Thanks

If need compare index values with both columns create Multiindex and use Index.isin:
df = df1[df1.set_index('A', append=True).index.isin(df2.set_index('B', append=True).index)]
print (df)
A
0 apple
2 orange

Pandas: Calculate true positive rate for each row

I have a dataframe like this, with one column being the label and the other columns being predictions
label pred1 pred2 pred3
0 Apple Apple Orange Apple
1 Orange Orange Orange Orange
I would like to extend this dataframe with the true positive rate (TP/TP+FN) for each row. This column should look like this:
Score
0 0.66
1 1.00
I am unsure on how to go on about this. Are there pandas functions that would help with this task?
Executable code: https://www.online-python.com/WP7wbgcqMS

Here is one approach where we convert the data to long format and check if the label equals the prediction. The average of the True/False values will be your Score.
import pandas as pd
d = {'Label': ['Apple','Orange'], 'pred1': ['Apple','Orange'], 'pred2': ['Orange','Orange'], 'pred3': ['Apple','Orange']}
df = pd.DataFrame(data=d)
df = df.melt(id_vars='Label', value_name='pred')
df['match'] = df['Label'].eq(df['pred'])
df.groupby('Label')['match'].mean().reset_index(name='Score')
Output
Label Score
0 Apple 0.666667
1 Orange 1.000000

maybe like this
temp = df.T.apply(lambda x: x[0]==x).astype(int)
(temp.sum()-1)/(temp.count()-1)
Out:
0 0.666667
1 1.000000

Faster way to identify and compare rows based on matching conditions within a dataframe having millions of rows

I have a dataframe as below.
Date Fruit level_0 Num Color
0 2013-11-25 Apple DF2 22.1 Red
1 2013-11-24 Banana DF1 22.1 Yellow
2 2013-11-24 Banana DF2 122.1 Yellow
3 2013-11-23 Celery DF1 10.2 Green
4 2013-11-24 Orange DF1 8.6 Orange
5 2013-11-24 Orange DF2 8.6 Orange1
6 2013-11-25 Orange DF1 8.6 Orange
I need to find and compare the rows within the dataframe and see which columns have data mismatch. The rows that are selected for comparison should be only those which have the same "Date" and "Fruit" values but different "level_0" values. So in the dataframe i need to compare rows having index 1 and 2 since they have same value for "Date" & "Fruit", but different "level_0" values. When comparing these since they differ in the "Num" column, we need to suffix a label(say "NM" ) beside the value in both rows. Rows which have only one occurrence of "Date" & "Fruit" combination will need to have a label (say "Miss") suffixed to the value in "Fruit" column.
Example of expected output below:
1.)Is it possible to get such an output?
2.)Is there a fast way get it, as my actual dataset contains millions of rows and 20-25 columns?

This is pretty complex, since there are lot different filters you want to do. If I get you right, you want
for rows that have the same "Date" and "Fruit" values, and
of those rows, those that have different "level_0" values, and
of those rows, those that have different "Num" values to get -NM. From your example you want to do the same with the "Color"-column.
Rows that are the only occurence of a "Date" and "Fruit" value get -Miss.
First, you'll need to make Num a string column, since we are adding suffixes. Then we groupby Date and Fruit (1). Then, since you wanted the groups to have different level_0 values, we make filter on that called diff_frames (2). Then we add the suffixes using transform on both columns if they have two unique elements (3).
df['Num'] = df['Num'].astype(str)
g = df.groupby(['Date', 'Fruit'])
diff_frames = g['level_0'].transform(lambda s: s.nunique() == 2)
df[['Num', 'Color']] = df[diff_frames].groupby(['Date', 'Fruit'])[['Num', 'Color']].transform(
lambda s: s+'-NM' if s.nunique() == 2 else s)
Then, for the second part, we get the non-duplicated rows in Date and Fruit, and add -Miss to the Fruit column. (4)
df.loc[~df.duplicated(subset=['Date', 'Fruit'], keep=False), 'Fruit'] += '-Miss'
print(df)
Date Fruit level_0 Num Color
0 0 Apple-Miss DF2 22.1 Red
1 1 Banana DF1 22.1-NM Yellow
2 1 Banana DF2 122.1-NM Yellow
3 2 Celery-Miss DF1 10.2 Green
4 3 Orange DF1 8.6 Orange-NM
5 3 Orange DF2 8.6 Orange1-NM
6 4 Orange-Miss DF2 8.6 Orange

Convert column into rows in Pandas dataframe

I have Pandas dataframe with two columns. One is unique identifier and second is the name of product attached to this unique identifier. I have duplicate values for identifier and product names. I want to convert one column of product names into several columns without duplicating identifier. Maybe I need to aggregate product names through identifier.
My dataframe looks like:
ID Product_Name
100 Apple
100 Banana
200 Cherries
200 Apricots
200 Apple
300 Avocados
I want to have dataframe like this:
ID
100 Apple Banana
200 Cherries Apricots Apple
300 Avocados
Each product along each identifier has to be in separate column
I tried pd.melt, pd.pivot, pd.pivot_table but only errors and this errors says No numeric types to aggregate
Any idea how to do this?

Use cumcount for new columns names to MultiIndex by set_index and reshape by unstack:
df = df.set_index(['ID',df.groupby('ID').cumcount()])['Product_Name'].unstack()
Or create Series of lists and new DataFrame by contructor:
s = df.groupby('ID')['Product_Name'].apply(list)
df = pd.DataFrame(s.values.tolist(), index=s.index)
print (df)
0 1 2
ID
100 Apple Banana NaN
200 Cherries Apricots Apple
300 Avocados NaN NaN
But if want 2 column DataFrame:
df1 = df.groupby('ID')['Product_Name'].apply(' '.join).reset_index(name='new')
print (df1)
ID new
0 100 Apple Banana
1 200 Cherries Apricots Apple
2 300 Avocados

use pivot funtion pivoting it can do the required thing!!

How to remove the extra row (or column) after transpose() in Pandas

After using transpose on a dataframe there is always an extra row as a remainder from the initial dataframe's index for example:
import pandas as pd
df = pd.DataFrame({'fruit':['apple','banana'],'number':[3,5]})
df
fruit number
0 apple 3
1 banana 5
df.transpose()
0 1
fruit apple banana
number 3 5
Even when i have no index:
df.reset_index(drop = True, inplace = True)
df
fruit number
0 apple 3
1 banana 5
df.transpose()
0 1
fruit apple banana
number 3 5
The problem is that when I save the dataframe to a csv file by:
df.to_csv(f)
this extra row stays at the top and I have to remove it manually every time.
Also this doesn't work:
df.to_csv(f, index = None)
because the old index is no longer considered an index (just another row...).
It also happened when I transposed the other way around and I got an extra column which i could not remove.
Any tips?

I had the same problem, I solved it by reseting index before doing the transpose. I mean df.set_index('fruit').transpose():
import pandas as pd
df = pd.DataFrame({'fruit':['apple','banana'],'number':[3,5]})
df
fruit number
0 apple 3
1 banana 5
And df.set_index('fruit').transpose() gives:
fruit apple banana
number 3 5

Instead of removing the extra index, why don't try setting the new index that you want and then use slicing ?
step 1: Set the new index you want:
df.columns = df.iloc[0]
step 2: Create a new dataframe removing extra row.
df_new = df[1:]