I Have a df having 2 columns *Total Idle Time and Month as below:
Total Idle Time Month
0 0:00:00 December
1 0:02:24 December
2 26:00:00 December
3 0:53:05 December
4 28:03:39 December
Here the Total Idle Time column is of string format, but I want to convert it into time format as I want to add the total idle time in the month of December.
I tried converting the column to datetime as below:
data['Total Idle Time '] = pd.to_datetime(data['Total Idle Time '], format='%H:%M:%S')
However, I got an error as follow:
time data '28:03:39' does not match format '%H:%M:%S' (match)
I thought of converting the column to int and adding them up based on the hours and minutes, but I am not successful in doing so. Is there any way to do this thing?
You could try using pd.to_timedelta() instead here:
>>> df['Idle Time'] = pd.to_timedelta(df["Idle Time"])
>>> df
Total Idle_Time Month
0 0 0 days 00:00:00 December
1 1 0 days 00:02:24 December
2 2 1 days 02:00:00 December
3 3 0 days 00:53:05 December
4 4 1 days 04:03:39 December
You can use this to convert to numeric if you want, by scaling the results of .total_seconds():
# in hours
>>> df['Idle Time'] = df['Idle Time'].dt.total_seconds() / 3600
>>> df
Total Idle_Time Month
0 0 0.000000 December
1 1 0.040000 December
2 2 26.000000 December
3 3 0.884722 December
4 4 28.060833 December
Related
Can anyone help me get the first weekday (as a date) of a given year in Python?
This gives me the first Monday of the current week:
test = datetime.today() - timedelta(days=datetime.today().isoweekday() % 7)
print(test)
This is easy once you break it down:
Let's define a function that takes an input yr and returns the first weekday of that year.
def first_weekday(yr: int) -> datetime.date:
What is the first day of the year yr?
first_day = datetime.date(yr, 1, 1)
Is that day a weekday? If so, it is the first weekday of the year.
day_of_week = first_day.isoweekday()
if day_of_week < 6: return first_day
datetime.date.isoweekday() returns 1 for Monday and 7 for Sunday.
Why did I use isoweekday instead of weekday? No reason
If not, find how many days until Monday. Subtracting the day_of_week from 8 should do the trick!
We would subtract from 7 for weekday().
days_to_monday = 8 - day_of_week
first_weekday = first_day + datetime.timedelta(days=days_to_monday)
return first_weekday
Put this all together, and test:
for yr in range(2019, 2025):
print(yr, first_weekday(yr))
prints:
2019 2019-01-01
2020 2020-01-01
2021 2021-01-01
2022 2022-01-03
2023 2023-01-02
2024 2024-01-01
I was trying to calculate the week number starting from first Monday of October. Is there any functions in pandas or datetime to do the calculation efficiently?
MWE
import pandas as pd
from datetime import date,datetime
df = pd.DataFrame({'date': pd.date_range('2021-10-01','2022-11-01',freq='1D')})
df['day_name'] = df['date'].dt.day_name()
df2 = df.head()
df2['Fiscal_Week'] = [52,52,52,1,1] # 2021-10-4 is monday, so for oct4, week is 1
# How to do it programatically for any year?
df2
date day_name Fiscal_Week
0 2021-10-01 Friday 52
1 2021-10-02 Saturday 52
2 2021-10-03 Sunday 52
3 2021-10-04 Monday 1
4 2021-10-05 Tuesday 1
Shift dates by the number of days to new year, they use standard Monday week number formatting (%W):
df['Fiscal_Week'] = (
df['date'] + pd.DateOffset(days=91)
).dt.strftime('%W').astype(int).replace({0:None}).fillna(method='ffill')
The offset in days can be calculated manually (I assume the fiscal year start is fixed)
The replace part is needed because leftovers from the previous year are considered week 0. The previous week might be 52 or 53, so replacing with NA and then fill forward
I am trying to create 2 columns based of a column that contains numerical values.
Value
0
4
10
24
null
49
Expected Output:
Value Day Hour
0 Sunday 12:00am
4 Sunday 4:00am
10 Sunday 10:00am
24 Monday 12:00am
null No Day No Time
49 Tuesday 1:00am
Continued.....
Code I am trying out:
value = df.value.unique()
Sunday_Starting_Point = pd.to_datetime('Sunday 2015')
(Sunday_Starting_Point + pd.to_timedelta(Value, 'h')).dt.strftime('%A %I:%M%P')
Thanks for looking!
I think unique values are not necessary, you can use 2 times dt.strftime for 2 columns with replace with NaT values:
Sunday_Starting_Point = pd.to_datetime('Sunday 2015')
x = pd.to_numeric(df.Value, errors='coerce')
s = Sunday_Starting_Point + pd.to_timedelta(x, unit='h')
df['Day'] = s.dt.strftime('%A').replace('NaT','No Day')
df['Hour'] = s.dt.strftime('%I:%M%p').replace('NaT','No Time')
print (df)
Value Day Hour
0 0.0 Sunday 12:00AM
1 4.0 Sunday 04:00AM
2 10.0 Sunday 10:00AM
3 24.0 Monday 12:00AM
4 NaN No Day No Time
5 49.0 Tuesday 01:00AM
My idea is about forecasting data with different time period :
For example :
A day month year quarter week
Date
2016-01-04 36.81 4 1 2016 1 1
2016-01-05 35.97 5 1 2016 1 1
2016-01-06 33.97 6 1 2016 1 1
2016-01-07 33.29 7 1 2016 1 1
2016-01-08 33.20 8 1 2016 1 2
2016-01-11 31.42 11 1 2016 1 2
2016-01-12 30.42 12 1 2016 1 2
I have daily data and i wanted to forecast data in month and again convert month back to daily.
The method I used is getting percentage of each day and month from it's sum
Here are some code i used:
converted_data = data.groupby( [data['month'], data['day']] )['A'].sum()
average = converted_data/converted_data.sum()
average
which give the following result:
month day A
1 3 0.002218
4 0.003815
5 0.003801
...
12 26 0.002522
27 0.004764
28 0.004822
29 0.004839
30 0.002277
By using this when i want to convert back from yearly data to daily i just multipling the average by result of the forecast
But this does not work when i wanted to convert daily data to quarterly data.
Can anyone suggested any idea on how to do it.
Thank you for your consideration.
Edit
the data i want is the percentage of the data in day restective to quarter
something like:
A = total when day is equal to 1 in data and also for day 2 and 3...
#for example my data is
Date value
1/1/2000 50
1/2/2000 50
1/3/2000 40
than A of day 1 is 140
B = total when quarter is equal to 1 in data and and also for quarter 2 and 3 4
#for example my data is
Date value
1/1/2000 4000 #-->quarter 1
1/4/2000 5000 #-->quarter 2
1/7/2000 2000 #-->quarter 3
1/10/2000 1000 #-->quarter 4
1/1/20001 2000 #-->quarter 1
than average of day 1 respective quarter is 140/6000 as a is in quarter one
The data above is what I converted.
First, I am receiving and input data as daily and I converted from pandas series to dataframe shown above the column, day, month, year, quarter, week was extracted in order to group the data my method work well for converting to year and month.
The reason why I do these because when my input is given in daily and I wanted to convert to year in order to do forecasting.
After the forecasting i will be getting the forecast value in yearly form so i wanted to convert it back to daily and the method i did is to find the portion of previous data.
I am so sorry, for the unclear question.
Again thank for your help.
I have a dataframe with one column in datetime format and the other columns in integers and floats. I would like to group the dataframe by the weekday of the first column. The other columns would be added.
print (df)
Day Butter Bread Coffee
2019-07-01 00:00:00 2 2 4
2019-07-01 00:00:00 1 2 1
2019-07-02 00:00:00 5 4 8
Basically the outcome would be sometime alike:
print (df)
Day Butter Bread Coffee
Monday 3 4 5
Tuesday 5 4 8
I am flexible if it says exactly Monday, or MO or 01 for the first day of the week, as long it is visible which consumption was done on Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, and Sunday.
You should convert your "Day" to datetime type and then you can extract the day of the week and aggregate over the rest of the columns:
import pandas as pd
df['Day'] = pd.to_datetime(df['Day'])
df.groupby(df['Day'].dt.day_name()).sum()
try using .dt.day_name() and groupby(),sum()
df = pd.DataFrame(data={'day':['2019-07-01 00:00:00','2019-07-01 00:00:00','2019-07-02 00:00:00'],
'butter':[2,1,5],
'bread':[2,2,4],
'coffee':[4,1,8]})
df['day'] = pd.to_datetime(df['day']).dt.day_name()
df.groupby(['day'],as_index=False).sum()
day butter bread coffee
0 Monday 3 4 5
1 Tuesday 5 4 8