I have a dataframe like this:
Bool Hour
0 False 12
1 False 24
2 False 12
3 False 24
4 True 12
5 False 24
6 False 12
7 False 24
8 False 12
9 False 24
10 False 12
11 True 24
and I would like to backfill the True value in 'Bool' column to the point when 'Hour' first reaches '12'. The result would be something like this:
Bool Hour Result
0 False 12 False
1 False 24 False
2 False 12 True <- desired backfill
3 False 24 True <- desired backfill
4 True 12 True
5 False 24 False
6 False 12 False
7 False 24 False
8 False 12 False
9 False 24 False
10 False 12 True <- desired backfill
11 True 24 True
Any help is greatly appreciated! Thank you very much!
This is a little bit hard to achieve , here we can use groupby with idxmax
s=(~df.Bool&df.Hour.eq(12)).iloc[::-1].groupby(df.Bool.iloc[::-1].cumsum()).transform('idxmax')
df['result']=df.index>=s.iloc[::-1]
df
Out[375]:
Bool Hour result
0 False 12 False
1 False 24 False
2 False 12 True
3 False 24 True
4 True 12 True
5 False 24 False
6 False 12 False
7 False 24 False
8 False 12 False
9 False 24 False
10 False 12 True
11 True 24 True
IIUC, you can do:
s = df['Bool'].shift(-1)
df['Result'] = df['Bool'] | s.where(s).groupby(df['Hour'].eq(12).cumsum()).bfill()
Output:
Bool Hour Result
0 False 12 False
1 False 24 False
2 False 12 True
3 False 24 True
4 True 12 True
5 False 24 False
6 False 12 False
7 False 24 False
8 False 12 False
9 False 24 False
10 False 12 True
11 True 24 True
create a groupID s on consecutive False and separate True from them. Groupby on Hour equals 12 by using s. Use transform sum and cumsum to get the count of True on 12 from bottom-up on each group and return True on 0 and or with values of Bool
s = df.Bool.ne(df.Bool.shift()).cumsum()
s1 = df.where(df.Bool).Bool.bfill()
g = df.Hour.eq(12).groupby(s)
df['bfill_Bool'] = (g.transform('sum') - g.cumsum()).eq(0) & s1 | df.Bool
Out[905]:
Bool Hour bfill_Bool
0 False 12 False
1 False 24 False
2 False 12 True
3 False 24 True
4 True 12 True
5 False 24 False
6 False 12 False
7 False 24 False
8 False 12 False
9 False 24 False
10 False 12 True
11 True 24 True
Related
I am using Python and have the following Pandas Dataframe:
idx
result
grouping
1
False
2
True
3
True
4
False
5
True
6
True
7
True
8
False
9
True
10
True
11
True
12
True
What I would like is to do the following logic...
if the result is False then I want grouping to be the idx value.
if the result is True then I want the grouping to be the previous grouping value
So the end result will be:
idx
result
grouping
1
False
1
2
True
1
3
True
1
4
False
4
5
True
4
6
True
4
7
True
4
8
False
8
9
True
8
10
True
8
11
True
8
12
True
8
I have tried all sorts to get this working from using the Pandas shift() command to using lambda, but I am just not getting it.
I know I could iterate through the dataframe and perform the calculation but there has to be a better method.
examples of what I have tried and failed with are:
df['grouping'] = df['idx'] if not df['result'] else df['grouping'].shift(1)
df['grouping'] = df.apply(lambda x: x['idx'] if not x['result'] else x['grouping'].shift(1), axis=1)
Many Thanks for any assistance you can provide.
mask true values then forward fill
df['grouping'] = df['idx'].mask(df['result']).ffill(downcast='infer')
idx result grouping
0 1 False 1
1 2 True 1
2 3 True 1
3 4 False 4
4 5 True 4
5 6 True 4
6 7 True 4
7 8 False 8
8 9 True 8
9 10 True 8
10 11 True 8
11 12 True 8
Hello i have a series of bool values where i have performed logical operations on them since they are series i have used & instead of and here are the series
z = ~(df['HOME ZIP'].isin(zip_series['zipcd_ZIP_CD']))
here is type of z
type(z)
pandas.core.series.Series
0 False
1 False
2 False
3 False
4 False
5 False
6 False
7 False
8 False
9 False
10 False
11 False
12 False
13 False
14 False
15 False
16 False
17 False
18 False
19 False
20 False
21 False
22 False
23 False
24 False
25 False
26 False
27 False
28 False
29 False
30 False
Name: HOME ZIP, dtype: bool
similarly
y = df['HOME ZIP'].astype(str).str.len() != 0
x = (df['HOME ZIP'].isnull() == False)
values of x and y are both True
0 True
1 True
2 True
3 True
4 True
5 True
6 True
7 True
8 True
9 True
10 True
11 True
12 True
13 True
14 True
15 True
16 True
17 True
18 True
19 True
20 True
21 True
22 True
23 True
24 True
25 True
26 True
27 True
28 True
29 True
30 True
Name: HOME ZIP, dtype: bool
x & y & z values are here
0 False
1 False
2 False
3 False
4 False
5 False
6 False
7 False
8 False
9 False
10 False
11 False
12 False
13 False
14 False
15 False
16 False
17 False
18 False
19 False
20 False
21 False
22 False
23 False
24 False
25 False
26 False
27 False
28 False
29 False
30 False
Name: HOME ZIP, dtype: bool
but when i keep all x y,z as single statements i am getting different output
(df['HOME ZIP'].isnull() == False) & df['HOME ZIP'].astype(str).str.len() != 0 & ~(df['HOME ZIP'].isin(zip_series['zipcd_ZIP_CD']))
0 True
1 True
2 True
3 True
4 True
5 True
6 True
7 True
8 True
9 True
10 True
11 True
12 True
13 True
14 True
15 True
16 True
17 True
18 True
19 True
20 True
21 True
22 True
23 True
24 True
25 True
26 True
27 True
28 True
29 True
30 True
Name: HOME ZIP, dtype: bool
You should add () for each condition
(df['HOME ZIP'].isnull() == False) &
(df['HOME ZIP'].astype(str).str.len() != 0) &
(~(df['HOME ZIP'].isin(zip_series['zipcd_ZIP_CD'])))
I want to return a boolean index using separate columns. Where End is in Item, I want to return False.
I'm meeting those conditions but I want to account for all unique values in Seq. For each unique group in Seq, if any row matches the previous condition, then return False for all those unique groups.
df = pd.DataFrame({
'Item' : ['Start','A','B','B','G','Start','A','B','B','A','X','Start','A','H'],
})
End = ['X','Y','Z']
df['Seq'] = df['Item'].eq('Start').groupby(df['Item'].eq('Start').cumsum()).transform('idxmax')
m2 = df.Item.isin(End)
out:
0 True
1 True
2 True
3 True
4 True
5 True
6 True
7 True
8 True
9 True
10 False
11 True
12 True
13 True
intended out:
0 True
1 True
2 True
3 True
4 True
5 True
6 False
7 False
8 False
9 False
10 False
11 True
12 True
13 True
Instead of idxmax, use max and then negate the result:
~df.Item.isin(End).groupby(df.Item.eq('Start').cumsum()).transform('max')
0 True
1 True
2 True
3 True
4 True
5 False
6 False
7 False
8 False
9 False
10 False
11 True
12 True
13 True
Name: Item, dtype: bool
To exclude row with Start:
~(df.Item.isin(End).groupby(df.Item.eq('Start').cumsum()).transform('max') & df.Item.ne('Start'))
Group the boolean mask m2 by Seq and transform with any then negate the output
~(m2.groupby(df['Seq']).transform('any'))
0 True
1 True
2 True
3 True
4 True
5 False
6 False
7 False
8 False
9 False
10 False
11 True
12 True
13 True
Name: Item, dtype: bool
My dataframe df:
SCHOOL CLASS GRADE
A Spanish nan
A Spanish nan
A Math 4000
A Math 7830
A Math 3893
B . nan
B . nan
B Biology 1929
B Biology 4839
B Biology 8195
C Spanish nan
C English 2003
C English 1000
C Biology 4839
C Biology 8191
If I do:
school_has_only_two_classes = df.groupby('SCHOOL').CLASS
.transform(lambda series: series.nunique()) == 2
I get
0 True
1 True
2 True
3 True
4 True
5 True
6 True
7 True
8 True
9 True
10 False
11 False
12 False
13 False
14 False
15 False
The transform works fine for the school C. BUT, if I do:
school_has_spanish = df.groupby('SCHOOL').CLASS.transform(lambda series: series.str.contains('^Spanish$',regex=True))
or
school_has_spanish = df.groupby('SCHOOL').CLASS.transform(lambda series: series=='Spanish')
I get the following result which is not what I was expecting:
0 True
1 True
2 False
3 False
4 False
5 False
6 False
7 False
8 False
9 False
10 True
11 False
12 False
13 False
14 False
15 False
The transform just does not spread all True's to the other rows of the group. Result I was expecting:
0 True
1 True
2 True
3 True
4 False
5 False
6 False
7 False
8 False
9 False
10 True
11 True
12 True
13 True
14 True
15 True
Any help is appreciated.
Check any with contains
df.CLASS.str.contains('Spanish').groupby(df.SCHOOL).transform('any')
Out[230]:
0 True
1 True
2 True
3 True
4 True
5 False
6 False
7 False
8 False
9 False
10 True
11 True
12 True
13 True
14 True
Name: CLASS, dtype: bool
Best described by an example. Input is
ts val
0 10 False
1 20 True
2 20 False
3 30 True
4 40 False
5 40 False
6 40 False
7 60 True
8 60 False
desired output is
ts val
0 10 False
1 20 True
2 20 True
3 30 True
4 40 False
5 40 False
6 40 False
7 60 True
8 60 True
The idea is as follows: if we see at least one True value inside the same ts cluster(i.e. same ts value), make all other values True that have the exact same timestamp.
You can use groupby on column 'ts', and then apply using .any() to determine whether any of val is True in the cluster/group.
import pandas as pd
# your data
# =====================
print(df)
Out[58]:
ts val data
0 10 False 0.3332
1 20 True -0.6877
2 20 False -0.6004
3 30 True 0.1922
4 40 False 0.2472
5 40 False -0.0117
6 40 False 0.8607
7 60 True -1.1464
8 60 False 0.0698
# processing
# =====================
# as suggested by #DSM, transform is best way to do it
df['val'] = df.groupby('ts')['val'].transform(any)
Out[61]:
ts val data
0 10 False 0.3332
1 20 True -0.6877
2 20 True -0.6004
3 30 True 0.1922
4 40 False 0.2472
5 40 False -0.0117
6 40 False 0.8607
7 60 True -1.1464
8 60 True 0.0698