I have a dataframe with individuals and their household IDs and I would like to create a variable that contains the household size.
I am using Python 3.7. I tried to use the groupby function combined with the size (I tried count as well count) function. The idea is for each observation about an individual, I want to count in the dataframe the number of observations with the same household ID and store it in a new variable.
Consider that each observation has a household ID (hh_id) and that I would like to store the household size in the hh_size variable.
I tried the following:
df['hh_size'] = df.groupby('hh_id').size
I expect hh_size variable to contain for each observation the household size. However, I get a column with only nan.
When I usedf.groupby('hh_id').size alone, I get the expected result but I cannot manage to store it in the hh_size variable.
For example:
individual hh_id hh_size
1 1 2
2 1 2
3 2 1
4 3 1
Thanks,
Julien
If I understand it you have to convert it to new DataFrame - .to_frame(name='hh_size') - and you may have to reset index.
import pandas as pd
df = pd.DataFrame({
'individual': [1,1,2,2,3,4],
'hh_id': [1,1,1,1,2,3],
})
sizes = df.groupby(['individual', 'hh_id']).size()
new_df = sizes.to_frame(name='hh_size').reset_index()
print(new_df)
Result:
individual hh_id hh_size
0 1 1 2
1 2 1 2
2 3 2 1
3 4 3 1
Related
I have a pandas dataframe
import pandas as pd
df =pd.DataFrame({'name':['john','joe','bill','richard','sam'],
'cluster':['1','2','3','1','2']})
df['cluster'].value_counts() will give the number of occurrences of items based on the column cluster.
Is it possible to retain only the rows which have the maximum number of occurrences in the column cluster?
The expected output is
The cluster 1 and 2 have the same number of occurrences, so all the rows for cluster 1 and 2 need to be retained.
Group by 'cluster' and use transform('count') to get a Series of occurrences by clusters with the appropriate shape. Then use it to mask only the rows corresponding to the max occurrences.
cluster_counts = df.groupby('cluster')['name'].transform('count')
res = df[cluster_counts == cluster_counts.max()]
Output:
>>> res
name cluster
0 john 1
1 joe 2
3 richard 1
4 sam 2
Setup:
import pandas as pd
df = pd.DataFrame({'name':['john','joe','bill','richard','sam'],
'cluster':['1','2','3','1','2']})
Use this
# find the most common clusters then filter those clusters
df[df.cluster.isin(df.cluster.mode())]
You can get the max count of cluster value through df['cluster'].value_counts() then use isin to filter cluster column
c = df['cluster'].value_counts()
out = df[df['cluster'].isin(c[c.eq(c.max())].index)]
print(out)
name cluster
0 john 1
1 joe 2
3 richard 1
4 sam 2
I have a large dataset (df) with lots of columns and I am trying to get the total number of each day.
|datetime|id|col3|col4|col...
1 |11-11-2020|7|col3|col4|col...
2 |10-11-2020|5|col3|col4|col...
3 |09-11-2020|5|col3|col4|col...
4 |10-11-2020|4|col3|col4|col...
5 |10-11-2020|4|col3|col4|col...
6 |07-11-2020|4|col3|col4|col...
I want my result to be something like this
|datetime|id|col3|col4|col...|Count
6 |07-11-2020|4|col3|col4|col...| 1
3 |5|col3|col4|col...| 1
2 |10-11-2020|5|col3|col4|col...| 1
4 |4|col3|col4|col...| 2
1 |11-11-2020|7|col3|col4|col...| 1
I tried to use resample like this df = df.groupby(['id','col3', pd.Grouper(key='datetime', freq='D')]).sum().reset_index() and this is my result. I am still new to programming and Pandas but I have read up on pandas docs and am still unable to do it.
|datetime|id|col3|col4|col...
6 |07-11-2020|4|col3|1|0.0
3 |07-11-2020|5|col3|1|0.0
2 |10-11-2020|5|col3|1|0.0
4 |10-11-2020|4|col3|2|0.0
1 |11-11-2020|7|col3|1|0.0
try this:
df = df.groupby(['datetime','id','col3']).count()
If you want the count values for all columns based only on the date, then:
df.groupby('datetime').count()
And you'll get a DataFrame who has the date time as the index and the column cells representing the number of entries for that given index.
I have a tall pandas dataframe called use with columns ID, Date, .... Each row is unique, but each ID has many rows, with one row ID per date.
ID Date Other_data
1 1-1-01 10
2 1-1-01 23
3 1-1-01 0
1 1-2-01 11
3 1-2-01 1
1 1-3-01 9
2 1-3-01 20
3 1-3-01 2
I also have a list of unique ids, ids=use['ID'].drop_duplicates
I want to find the intersection of all of the dates, that is, only the dates for which each ID has data. The end result in this toy problem should be [1-1-01, 1-3-01]
Currently, I loop through, subsetting by ID and taking the intersection. Roughly speaking, it looks like this:
dates = use['Date'].drop_duplicates()
for i in ids:
id_dates = use[(use['ID'] == i)]['Date'].values
dates = set(dates).intersection(id_dates)
This strikes me as horrifically inefficient. What is a more efficient way to identify dates where each ID has data?
Thanks very much!
Using crosstab, when the value is 0 should be the target row . using df.eq(0).any(1). to find it
df=pd.crosstab(use.ID,use.Date)
df
Out[856]:
Date 1-1-01 1-2-01 1-3-01
ID
1 1 1 1
2 1 0 1
3 1 1 1
Find the unique IDs per date, then check if that's all of them.
gp = df.groupby('Date').ID.nunique()
gp[gp == df.ID.nunique()].index.tolist()
#['1-1-01', '1-3-01']
I have seen a variant of this question asked that keeps the top n rows of each group in a pandas dataframe and the solutions use n as an absolute number rather than a percentage here Pandas get topmost n records within each group. However, in my dataframe, each group has different numbers of rows in it and I want to keep the top n% rows of each group. How would I approach this problem?
You can construct a Boolean series of flags and filter before you groupby. First let's create an example dataframe and look at the number of row for each unique value in the first series:
np.random.seed(0)
df = pd.DataFrame(np.random.randint(0, 2, (10, 3)))
print(df[0].value_counts())
0 6
1 4
Name: 0, dtype: int64
Then define a fraction, e.g. 50% below, and construct a Boolean series for filtering:
n = 0.5
g = df.groupby(0)
flags = (g.cumcount() + 1) <= g[1].transform('size') * n
Then apply the condition, set the index as the first series and (if required) sort the index:
df = df.loc[flags].set_index(0).sort_index()
print(df)
1 2
0
0 1 1
0 1 1
0 1 0
1 1 1
1 1 0
As you can see, the resultant dataframe only has 3 0 indices and 2 1 indices, in each case half the number in the original dataframe.
Here is another option which builds on some of the answers in the post you mentioned
First of all here is a quick function to either round up or round down. If we want the top 30% of rows of a dataframe 8 rows long then we would try to take 2.4 rows. So we will need to either round up or down.
My preferred option is to round up. This is because, for eaxample, if we were to take 50% of the rows, but had one group which only had one row, we would still keep that one row. I kept this separate so that you can change the rounding as you wish
def round_func(x, up=True):
'''Function to round up or round down a float'''
if up:
return int(x+1)
else:
return int(x)
Next I make a dataframe to work with and set a parameter p to be the fraction of the rows from each group that we should keep. Everything follows and I have commented it so that hopefully you can follow.
import pandas as pd
df = pd.DataFrame({'id':[1,1,1,2,2,2,2,3,4],'value':[1,2,3,1,2,3,4,1,1]})
p = 0.30 # top fraction to keep. Currently set to 80%
df_top = df.groupby('id').apply( # group by the ids
lambda x: x.reset_index()['value'].nlargest( # in each group take the top rows by column 'value'
round_func(x.count().max()*p))) # calculate how many to keep from each group
df_top = df_top.reset_index().drop('level_1', axis=1) # make the dataframe nice again
df looked like this
id value
0 1 1
1 1 2
2 1 3
3 2 1
4 2 2
5 2 3
6 2 4
7 3 1
8 4 1
df_top looks like this
id value
0 1 3
1 2 4
2 2 3
3 3 1
4 4 1
I am new to pandas. I'm trying to sort a column within each group. So far, I was able to group first and second column values together and calculate the mean value in third column. But I am still struggling to sort 3rd column.
This is my input dataframe
This is my dataframe after applying groupby and mean function
I used the following line of code to group input dataframe,
df_o=df.groupby(by=['Organization Group','Department']).agg({'Total Compensation':np.mean})
Please let me know how to sort the last column for each group in 1st column using pandas.
It seems you need sort_values:
#for return df add parameter as_index=False
df_o=df.groupby(['Organization Group','Department'],
as_index=False)['Total Compensation'].mean()
df_o = df_o.sort_values(['Total Compensation','Organization Group'])
Sample:
df = pd.DataFrame({'Organization Group':['a','b','a','a'],
'Department':['d','f','a','a'],
'Total Compensation':[1,8,9,1]})
print (df)
Department Organization Group Total Compensation
0 d a 1
1 f b 8
2 a a 9
3 a a 1
df_o=df.groupby(['Organization Group','Department'],
as_index=False)['Total Compensation'].mean()
print (df_o)
Organization Group Department Total Compensation
0 a a 5
1 a d 1
2 b f 8
df_o = df_o.sort_values(['Total Compensation','Organization Group'])
print (df_o)
Organization Group Department Total Compensation
1 a d 1
0 a a 5
2 b f 8