I have the following DataFrame:
>>> df = pd.DataFrame({"a": [1, 1, 1, 1, 2, 2, 3, 3, 3], "b": [1, 5, 7, 9, 2, 4, 6, 14, 5], "c": [1, 0, 0, 1, 1, 1, 1, 0, 1]})
>>> df
a b c
0 1 1 1
1 1 5 0
2 1 7 0
3 1 9 1
4 2 2 1
5 2 4 1
6 3 6 1
7 3 14 0
8 3 5 1
I want to calculate the mode of column c for every unique value in a and then select the rows where c has this value.
This is my own solution:
>>> major_types = df.groupby(['a'])['c'].apply(lambda x: pd.Series.mode(x)[0])
>>> df = df.merge(major_types, how="left", right_index=True, left_on="a", suffixes=("", "_major"))
>>> df = df[df['c'] == df['c_major']].drop(columns="c_major", axis=1)
Which would output the following:
>>> df
a b c
1 1 5 0
2 1 7 0
4 2 2 1
5 2 4 1
6 3 6 1
8 3 5 1
It is very insufficient for large DataFrames. Any idea on what to do?
IIUC, GroupBy.transform instead apply + merge
df.loc[df['c'].eq(df.groupby('a')['c'].transform(lambda x: x.mode()[0]))]
a b c
1 1 5 0
2 1 7 0
4 2 2 1
5 2 4 1
6 3 6 1
8 3 5 1
Or
s = df.groupby(['a','c'])['c'].transform('size')
df.loc[s.eq(s.groupby(df['c']).transform('max'))]
Related
I have a pandas DataFrame that looks similar to the one below:
df = pd.DataFrame({
'label': [0, 0, 2, 3, 8, 8, 9],
'value1': [2, 1, 9, 8, 7, 4, 2],
'value2': [0, 1, 9, 4, 2, 3, 1],
})
>>> df
label value1 value2
0 0 2 0
1 0 1 1
2 2 9 9
3 3 8 4
4 8 7 2
5 8 4 3
6 9 2 1
Values in the label column are not complete (not range(0, n, 1)) due to previously slicing. I would like to reorder this label and assign a sequential range of ascending values so that it becomes:
>>> df
label value1 value2
0 1 2 0
1 1 1 1
2 2 9 9
3 3 8 4
4 4 7 2
5 4 4 3
6 5 2 1
I currently use the code below. Because my real DataFrame has thousands of unique values any suggestions to do this a bit more efficiently (not including looping over every unique value) would be appreciated.
for new_idx, idx in enumerate(df.label.unique()):
df.loc[df['label'] == idx, ['label']] = new_idx
Thanks in advance
Use factorize for improve performance:
df['label'] = pd.factorize(df['label'])[0] + 1
print (df)
label value1 value2
0 1 2 0
1 1 1 1
2 2 9 9
3 3 8 4
4 4 7 2
5 4 4 3
6 5 2 1
Another idea with Series.rank:
df['label'] = df['label'].rank(method='dense').astype(int)
print (df)
label value1 value2
0 1 2 0
1 1 1 1
2 2 9 9
3 3 8 4
4 4 7 2
5 4 4 3
6 5 2 1
Working same only of same ordering:
#dta changed for see difference
df = pd.DataFrame({
'label': [0, 10, 10, 3, 8, 8, 9],
'value1': [2, 1, 9, 8, 7, 4, 2],
'value2': [0, 1, 9, 4, 2, 3, 1],
})
df['label1'] = pd.factorize(df['label'])[0] + 1
df['label2'] = df['label'].rank(method='dense').astype(int)
print (df)
label value1 value2 label1 label2
0 0 2 0 1 1
1 10 1 1 2 5
2 10 9 9 2 5
3 3 8 4 3 2
4 8 7 2 4 3
5 8 4 3 4 3
6 9 2 1 5 4
I have an carid and I would like to see all buyers who had something to do with this carid. So I would like to have all buyers who have bought carid 3.
How do I do that?
import pandas as pd
d = {'Buyerid': [1,1,2,2,3,3,3,4,5,5,5],
'Carid': [1,2,3,4,4,1,2,4,1,3,5]}
df = pd.DataFrame(data=d)
print(df)
Buyerid Carid
0 1 1
1 1 2
2 2 3
3 2 4
4 3 4
5 3 1
6 3 2
7 4 4
8 5 1
9 5 3
10 5 5
# What I want
Buyerid Carid
2 2 3
3 2 4
8 5 1
9 5 3
10 5 5
I have already tested this df = df.loc[df['Carid']==3,'Buyerid'], but this only gives me the line with CardID 3 but not the complete buyer.
How to select rows from a DataFrame based on column values
I looked at that, but I only get this here
Buyerid Carid
2 2 3
9 5 3
Do the following:
import pandas as pd
d = {'Buyerid': [1, 1, 2, 2, 3, 3, 3, 4, 5, 5, 5],
'Carid': [1, 2, 3, 4, 4, 1, 2, 4, 1, 3, 5]}
df = pd.DataFrame(data=d)
# get all buyers
buyers = set(df.loc[df['Carid'] == 3, 'Buyerid'])
# boolean mask for filtering
mask = df['Buyerid'].isin(buyers)
print(df[mask])
Output
Buyerid Carid
2 2 3
3 2 4
8 5 1
9 5 3
10 5 5
You can use df.loc:
df.loc[df['Carid']==3,'Buyerid']
I have a pandas dataframe as follows:
df = pd.DataFrame({'A':[4, 4, 1, 5, 1, 1],
'B':[2, 2, 2, 5, 2, 2],
'C':[1, 1, 3, 5, 3, 3],
'D':['q', 'e', 'r', 'y', 'u',' w']})
which looks like
A B C D
0 4 2 1 q
1 4 2 1 e
2 1 2 3 r
3 5 5 5 y
4 1 2 3 u
5 1 2 3 w
I would like to add a new column that is the count of duplicate rows, with respect to only the columns A, B, and C. This would look like
A B C D Count
0 4 2 1 q 2
1 4 2 1 e 2
2 1 2 3 r 3
3 5 5 5 y 1
4 1 2 3 u 3
5 1 2 3 w 3
I'm guessing this will be something like df.groupby(['A','B','C']).size() but I am unsure how to map the values back to the new 'Count' column. Thanks!
We can do transform
df['Count'] = df.groupby(['A','B','C']).D.transform('count')
df['Count']
0 2
1 2
2 3
3 1
4 3
5 3
Name: Count, dtype: int64
Say you have a multiindex DataFrame
x y z
a 1 0 1 2
2 3 4 5
b 1 0 1 2
2 3 4 5
3 6 7 8
c 1 0 1 2
2 0 4 6
Now you have another DataFrame which is
col1 col2
0 a 1
1 b 1
2 b 3
3 c 1
4 c 2
How do you split the multiindex DataFrame based on the one above?
Use loc by tuples:
df = df1.loc[df2.set_index(['col1','col2']).index.tolist()]
print (df)
x y z
a 1 0 1 2
b 1 0 1 2
3 6 7 8
c 1 0 1 2
2 0 4 6
df = df1.loc[[tuple(x) for x in df2.values.tolist()]]
print (df)
x y z
a 1 0 1 2
b 1 0 1 2
3 6 7 8
c 1 0 1 2
2 0 4 6
Or join:
df = df2.join(df1, on=['col1','col2']).set_index(['col1','col2'])
print (df)
x y z
col1 col2
a 1 0 1 2
b 1 0 1 2
3 6 7 8
c 1 0 1 2
2 0 4 6
Simply using isin
df[df.index.isin(list(zip(df2['col1'],df2['col2'])))]
Out[342]:
0 1 2 3
index1 index2
a 1 1 0 1 2
b 1 1 0 1 2
3 3 6 7 8
c 1 1 0 1 2
2 2 0 4 6
You can also do this using the MultiIndex reindex method https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reindex.html
## Recreate your dataframes
tuples = [('a', 1), ('a', 2),
('b', 1), ('b', 2),
('b', 3), ('c', 1),
('c', 2)]
data = [[1, 0, 1, 2],
[2, 3, 4, 5],
[1, 0, 1, 2],
[2, 3, 4, 5],
[3, 6, 7, 8],
[1, 0, 1, 2],
[2, 0, 4, 6]]
idx = pd.MultiIndex.from_tuples(tuples, names=['index1','index2'])
df= pd.DataFrame(data=data, index=idx)
df2 = pd.DataFrame([['a', 1],
['b', 1],
['b', 3],
['c', 1],
['c', 2]])
# Answer Question
idx_subset = pd.MultiIndex.from_tuples([(a, b) for a, b in df2.values], names=['index1', 'index2'])
out = df.reindex(idx_subset)
print(out)
0 1 2 3
index1 index2
a 1 1 0 1 2
b 1 1 0 1 2
3 3 6 7 8
c 1 1 0 1 2
2 2 0 4 6
I have a dataframe with two columns:
x y
0 1
1 1
2 2
0 5
1 6
2 8
0 1
1 8
2 4
0 1
1 7
2 3
What I want is:
x val1 val2 val3 val4
0 1 5 1 1
1 1 6 8 7
2 2 8 4 3
I know that the values in column x are repeated all N times.
You could use groupby/cumcount to assign column numbers and then call pivot:
import pandas as pd
df = pd.DataFrame({'x': [0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2],
'y': [1, 1, 2, 5, 6, 8, 1, 8, 4, 1, 7, 3]})
df['columns'] = df.groupby('x')['y'].cumcount()
# x y columns
# 0 0 1 0
# 1 1 1 0
# 2 2 2 0
# 3 0 5 1
# 4 1 6 1
# 5 2 8 1
# 6 0 1 2
# 7 1 8 2
# 8 2 4 2
# 9 0 1 3
# 10 1 7 3
# 11 2 3 3
result = df.pivot(index='x', columns='columns')
print(result)
yields
y
columns 0 1 2 3
x
0 1 5 1 1
1 1 6 8 7
2 2 8 4 3
Or, if you can really rely on the values in x being repeated in order N times,
N = 3
result = pd.DataFrame(df['y'].values.reshape(-1, N).T)
yields
0 1 2 3
0 1 5 1 1
1 1 6 8 7
2 2 8 4 3
Using reshape is quicker than calling groupby/cumcount and pivot, but it
is less robust since it relies on the values in y appearing in the right order.