Develop Reference

Sliding Window and comparing elements of DataFrame to a threshold

Assume I have the following dataframe:
Time Flag1
0 0
10 0
30 0
50 1
70 1
90 0
110 0
My goal is to identify if within any window that time is less than lets the number in the row plus 35, then if any element of flag is 1 then that row would be 1. For example consider the above example:
The first element of time is 0 then 0 + 35 = 35 then in the window of values less than 35 (which is Time =0, 10, 30) all the flag1 values are 0 therefore the first row will be assigned to 0 and so on. Then the next window will be 10 + 35 = 45 and still will include (0,10,30) and the flag is still 0. So the complete output is:
Time Flag1 Output
0 0 0
10 0 0
30 0 1
50 1 1
70 1 1
90 1 1
110 1 1
To implement this type of problem, I thought I can use two for loops like this:
Output = []
for ii in range(Data.shape[0]):
count =0
th = Data.loc[ii,'Time'] + 35
for jj in range(ii,Data.shape[0]):
if (Data.loc[jj,'Time'] < th and Data.loc[jj,'Flag1'] == 1):
count = 1
break
output.append(count)
However this looks tedious. since the inner for loop should go for continue for the entire length of data. Also I am not sure if this method checks the boundary cases for out of bound index when we are reaching to end of the dataframe. I appreciate if someone can comment on something easier than this. This is like a sliding window operation only comparing number to a threshold.
Edit: I do not want to compare two consecutive rows only. I want if for example 30 + 35 = 65 then as long as time is less than 65 then if flag1 is 1 then output is 1.
The second example:
Time Flag1 Output
0 0 0
30 0 1
40 0 1
60 1 1
90 1 1
140 1 1
200 1 1
350 1 1

Assuming a window k rows before and k rows after as mentioned in my comment:
import pandas as pd
Data = pd.DataFrame([[0,0], [10,0], [30,0], [50,1], [70,1], [90,1], [110,1]],
columns=['Time', 'Flag1'])
k = 1 # size of window: up to k rows before and up to k rows after
n = len(Data)
output = [0]*n
for i in range(n):
th = Data['Time'][i] + 35
j0 = max(0, i - k)
j1 = min(i + k + 1, n) # the +1 is because range is non-inclusive of end
output[i] = int(any((Data['Time'][j0 : j1] < th) & (Data['Flag1'][j0 : j1] > 0)))
Data['output'] = output
print(Data)
gives the same output as the original example. And you can change the size of the window my modifying k.
Of course, if the idea is to check any row afterward, then just use j1 = n in my example.

import pandas as pd
Data = pd.DataFrame([[0,0],[10,0],[30,0],[50,1],[70,1],[90,1],[110,1]],columns=['Time','Flag1'])
output = Data.index.map(lambda x: 1 if any((Data.Time[x+1:]<Data.Time[x]+35)*(Data.Flag1[x+1:]==1)) else 0).values
output[-1] = Data.Flag1.values[-1]
Data['output'] = output
print(Data)
# show
Time Flag1 output
0 0 0
30 0 1
40 0 1
50 1 1
70 1 1
90 1 1
110 1 1

while loop: number less than number in Python

This is my first time posting here. Sorry if it's not the correct way. In the following code, there are three loops iterations. I can understand why. But I need it to be only 2. Is there a way to do this?
weeks_x = 0
weeks_year = 52
count = 0
while weeks_x < weeks_year:
weeks_x = weeks_x + 25
count = count + 1

Adding a print() statement can be useful in debugging:
weeks_x = 0
weeks_year = 52
count = 0
while weeks_x < weeks_year:
print('weeks_x:', weeks_x, '- weeks_year', weeks_year)
weeks_x = weeks_x + 25
count =+ 1
output:
weeks_x: 0 - weeks_year 52
weeks_x: 25 - weeks_year 52
weeks_x: 50 - weeks_year 52
If you specifically needed two loop iterations:
weeks_x = 0
weeks_year = 52
count = 0
while count < 2:
print('weeks_x:', weeks_x, '- weeks_year', weeks_year)
weeks_x = weeks_x + 25
count =+ 1
count variable is counting the number of loops.
Alternatively, weeks_x * 2 will give the same result as the loop - otherwise if there is functionality that requires this I suggest looking at for loops with range().

Try debugging your code with simple print statements
weeks_x = 0
weeks_year = 52
count = 0
while weeks_x < weeks_year:
weeks_x = weeks_x + 25
count = count + 1
print(f'Iteration Number {count} and weeks_x is {weeks_x}')
The output makes it crystal why the loop runs for 3 times
Iteration Number 1 and weeks_x is 25
Iteration Number 2 and weeks_x is 50
Iteration Number 3 and weeks_x is 75
Since only after the 3rd iteration the value of weeks_x is enough to break this condition weeks_x < 52 it goes for 3 turns
You can limit the number of times the loop runs
weeks_x = 0
weeks_year = 52
count = 0
# while weeks_x < weeks_year: # REMOVE THIS
while count < 2: # ADD THIS
weeks_x = weeks_x + 25
count = count + 1
print(f'Iteration Number {count} and weeks_x is {weeks_x}')

Initializing before while
weeks_x = 0
weeks_year = 52
count = 0
weeks_x += 25
while weeks_x < weeks_year:
count += 1
print('Count {}'.format(count), 'Week Value {}'.format(weeks_x), sep='-->', end='\n')
weeks_x += 25
Output:-
Count 1-->Week Value 25
Count 2-->Week Value 50

sample test case is running well, but when i submit, it keeps on processing(HACKERRANK) (PROBLEM NAME-Grading Students)

Sam is a professor at the university and likes to round each student's
according to these rules:
1) If the difference between the grades and the next multiple of 5 is less than 3, round up to the next multiple of 5
2)If the value of grade is less than 38, no rounding occurs as the result will still be a failing grade.
def gradingStudents(grades):
# Write your code here
gr = [None]*len(grades)
j = 0
for i in grades:
e = 5- (i% 5)
if e < 3 and i+e >= 40 and i+e <= 100:
gr[j] = i + e
else:
gr[j] = i
j += 1
return gr;
Sample Input 0
4
73
67
38
33
Sample Output 0
75
67
40
33

Complex Groupby Pandas Operation to Replace For Loops and If Statements

I have a complex group of a group problem I need help with.
I have names of drivers, each of who have driven several cars over time. Each time they turn on the car and drive, I capture cycles and hours, which are transmitted remotely.
What I am trying to do is use grouping to see when the driver gets a new car.
I'm using Car_Cycles and Car_Hours to monitor for a reset (new car). The hours and cycles are tabulated for each driver in ascending sequence until there's a new car and reset. I want to make each car a sequence, but logically can only recognize the car by a cycle/hour reset.
I used a for loop with if statements to do this on the dataframe, and the process time takes several hours. I have several hundred thousand rows with each containing about 20 columns.
My data comes from sensors over a moderately reliable connection, so I want to filter by using the following criteria: A new group is only valid when both Car_Hours and Car_Cycles are less the previous group's last row for 2 consecutive rows. Using both outputs and checking for two rows of change sufficiently filters all erroneous data.
If someone could show me how to quickly solve for Car_Group without using my cumbersome for loops and if statements, I would greatly appreciate it.
Also, for those how are very venturous, I added my original for loop below with if statements. Note I did some other data analysis/tracking within each group to look at other behavior of the car. If you dare look at that code and show me an efficient Pandas replacement, all the more kudos.
name Car_Hours Car_Cycles Car_Group DeltaH
jan 101 404 1 55
jan 102 405 1 55
jan 103 406 1 56
jan 104 410 1 55
jan 105 411 1 56
jan 0 10 2 55
jan 1 12 2 58
jan 2 14 2 57
jan 3 20 2 59
jan 4 26 2 55
jan 10 36 2 56
jan 15 42 2 57
jan 27 56 2 57
jan 100 61 2 58
jan 500 68 2 58
jan 2 4 3 56
jan 3 15 3 57
pete 190 21 1 54
pete 211 29 1 58
pete 212 38 1 55
pete 304 43 1 56
pete 14 20 2 57
pete 15 27 2 57
pete 36 38 2 58
pete 103 47 2 55
mike 1500 2001 1 55
mike 1512 2006 1 59
mike 1513 2012 1 58
mike 1515 2016 1 57
mike 1516 2020 1 55
mike 1517 2024 1 57
..............
for i in range(len(file)):
if i == 0:
DeltaH_limit = 57
car_thresholds = 0
car_threshold_counts = 0
car_threshold_counts = 0
car_change_true = 0
car_change_index_loc = i
total_person_thresholds = 0
person_alert_count = 0
person_car_count = 1
person_car_change_count = 0
total_fleet_thresholds = 0
fleet_alert_count = 0
fleet_car_count = 1
fleet_car_change_count = 0
if float(file['Delta_H'][i]) >= DeltaH_limit:
car_threshold_counts += 1
car_thresholds += 1
total_person_thresholds += 1
total_fleet_thresholds += 1
elif i == 1:
if float(file['Delta_H'][i]) >= DeltaH_limit:
car_threshold_counts += 1
car_thresholds += 1
total_person_thresholds += 1
total_fleet_thresholds += 1
elif i > 1:
if file['name'][i] == file['name'][i-1]: #is same person?
if float(file['Delta_H'][i]) >= DeltaH_limit:
car_threshold_counts += 1
car_thresholds += 1
total_person_thresholds += 1
total_fleet_thresholds += 1
else:
car_threshold_counts = 0
if car_threshold_counts == 3:
car_threshold_counts += 1
person_alert_count += 1
fleet_alert_count += 1
#Car Change?? Compare cycles and hours to look for reset
if i+1 < len(file):
if file['name'][i] == file['name'][i+1] == file['name'][i-1]:
if int(file['Car_Cycles'][i]) < int(file['Car_Cycles'][i-1]) and int(file['Car_Hours'][i]) < int(file['Car_Hours'][i-1]):
if int(file['Car_Cycles'][i+1]) < int(file['Car_Cycles'][i-1]) and int(file['Car_Hours'][i]) < int(file['Car_Hours'][i-1]):
car_thresholds = 0
car_change_true = 1
car_threshold_counts = 0
car_threshold_counts = 0
old_pump_first_flight = car_change_index_loc
car_change_index_loc = i
old_pump_last_flight = i-1
person_car_count += 1
person_car_change_count += 1
fleet_car_count += 1
fleet_car_change_count += 1
print(i, ' working hard!')
else:
car_change_true = 0
else:
car_change_true = 0
else:
car_change_true = 0
else:
car_change_true = 0
else: #new car
car_thresholds = 0
car_threshold_counts = 0
car_threshold_counts = 0
car_change_index_loc = i
car_change_true = 0
total_person_thresholds = 0
person_alert_count = 0
person_car_count = 1
person_car_change_count = 0
if float(file['Delta_H'][i]) >= DeltaH_limit:
car_threshold_counts += 1
car_thresholds += 1
total_person_thresholds += 1
total_fleet_thresholds += 1
file.loc[i, 'car_thresholds'] = car_thresholds
file.loc[i, 'car_threshold_counts'] = car_threshold_counts
file.loc[i, 'car_threshold_counts'] = car_threshold_counts
file.loc[i, 'car_change_true'] = car_change_true
file.loc[i, 'car_change_index_loc'] = car_change_index_loc
file.loc[i, 'total_person_thresholds'] = total_person_thresholds
file.loc[i, 'person_alert_count'] = person_alert_count
file.loc[i, 'person_car_count'] = person_car_count
file.loc[i, 'person_car_change_count'] = person_car_change_count
file.loc[i, 'Total_Fleet_Thresholds'] = total_fleet_thresholds
file.loc[i, 'Fleet_Alert_Count'] = fleet_alert_count
file.loc[i, 'fleet_car_count'] = fleet_car_count
file.loc[i, 'fleet_car_change_count'] = fleet_car_change_count

IIUC, and all we need to do is reproduce Car_Group, we can take advantage of a few tricks:
def twolow(s):
return (s < s.shift()) & (s.shift(-1) < s.shift())
new_hour = twolow(df["Car_Hours"])
new_cycle = twolow(df["Car_Cycles"])
new_name = df["name"] != df["name"].shift()
name_group = new_name.cumsum()
new_cargroup = new_name | (new_hour & new_cycle)
cargroup_without_reset = new_cargroup.cumsum()
cargroup = (cargroup_without_reset -
cargroup_without_reset.groupby(name_group).transform(min) + 1)
Trick #1: if you want to find out where a transition occurs, compare something to a shifted version of itself.
Trick #2: if you have a True where every new group begins, when you take the cumulative sum of that, you get a series where every group has an integer associated with it.
The above gives me
>>> cargroup.head(10)
0 1
1 1
2 1
3 1
4 1
5 2
6 2
7 2
8 2
9 2
dtype: int32
>>> (cargroup == df.Car_Group).all()
True

Is it efficient to perform individual, nested if statements?

I'm working on the following problem:
You are driving a little too fast, and a police officer stops you. Write code to compute the result, encoded as an int value: 0=no ticket, 1=small ticket, 2=big ticket. If speed is 60 or less, the result is 0. If speed is between 61 and 80 inclusive, the result is 1. If speed is 81 or more, the result is 2. Unless it is your birthday -- on that day, your speed can be 5 higher in all cases.
I came up with the following code:
def caught_speeding(speed, is_birthday):
if is_birthday == True:
if speed <= 65:
return 0
elif speed <= 85:
return 1
else:
return 2
else:
if speed <= 60:
return 0
elif speed <= 80:
return 1
else:
return 2
I feel like checking each one individually is a bit inefficient, or is it ok?

You gotta love the bisect module.
def caught_speeding(speed, is_birthday):
l=[60,80]
if is_birthday:
speed-=5
return bisect.bisect_left(l,speed)

I have no problems with your code. It is readable and clear.
If you want to go for less lines then you can do something like this:
def caught_speeding(speed, is_birthday):
adjustment = 5 if is_birthday else 0
if speed <= 60 + adjustment:
return 0
elif speed <= 80 + adjustment:
return 1
else:
return 2

You can do this:
def caught_speeding(speed, is_birthday):
if is_birthday:
speed = speed - 5
if speed <= 60:
return 0
elif speed <= 80:
return 1
else:
return 2
Doing is_birthday == True means you didn't quite get booleans yet ;-)

Check this one. It is optimised:
def caught_speeding(speed, is_birthday):
if speed in range(0,66 if is_birthday else 61):
return 0
elif speed in range(0,86 if is_birthday else 81):
return 1
return 2

Assuming that speed is an integer, and that efficiency means speed of running, not speed of understanding:
>>> def t(speed, is_birthday):
... speed -= 5 * is_birthday
... return speed // 61 + speed // 81
...
>>> for s in xrange(58, 87):
... print s, t(s, False), t(s, True)
...
58 0 0
59 0 0
60 0 0
61 1 0
62 1 0
63 1 0
64 1 0
65 1 0
66 1 1
67 1 1
68 1 1
69 1 1
70 1 1
71 1 1
72 1 1
73 1 1
74 1 1
75 1 1
76 1 1
77 1 1
78 1 1
79 1 1
80 1 1
81 2 1
82 2 1
83 2 1
84 2 1
85 2 1
86 2 2
>>>

def caught_speeding(speed, is_birthday):
speed -= 5 * is_birthday
return 0 if speed < 61 else 2 if speed > 80 else 1