I have tried many suggestions from here but none of them solved.
I have two columns with observations like this: 15:08:19
If I write
df.time_entry.describe()
it appears:
count 814262
unique 56765
top 15:03:00
freq 103
Name: time_entry, dtype: object
I've already run this code:
df['time_entry'] = pd.to_datetime(df['time_entry'],format= '%H:%M:%S', errors='ignore' ).dt.time
But rerunning the describe code still returns dtype: object.
What is the purpose of dt.time?
Just remove dt.time and your conversion from object to datetime will work perfectly fine.
df['time_entry'] = pd.to_datetime(df['time_entry'],format= '%H:%M:%S')
The problem is that you are using the datetime accessor (.dt) with the property time and then you are not able to subtract the two columns from eachother. So, just leave out .dt.time and it should work.
Here is some data with 2 columns of strings
df = pd.DataFrame()
df['time_entry'] = ['12:01:00', '15:03:00', '16:43:00', '14:11:00']
df['time_entry2'] = ['13:03:00', '14:04:00', '19:23:00', '18:12:00']
print(df)
time_entry time_entry2
0 12:01:00 13:03:00
1 15:03:00 14:04:00
2 16:43:00 19:23:00
3 14:11:00 18:12:00
Convert both columns to datetime dtype
df['time_entry'] = pd.to_datetime(df['time_entry'], format= '%H:%M:%S', errors='ignore')
df['time_entry2'] = pd.to_datetime(df['time_entry2'], format= '%H:%M:%S', errors='ignore')
print(df)
time_entry time_entry2
0 1900-01-01 12:01:00 1900-01-01 13:03:00
1 1900-01-01 15:03:00 1900-01-01 14:04:00
2 1900-01-01 16:43:00 1900-01-01 19:23:00
3 1900-01-01 14:11:00 1900-01-01 18:12:00
print(df.dtypes)
time_entry datetime64[ns]
time_entry2 datetime64[ns]
dtype: object
(Optional) Specify timezone
df['time_entry'] = df['time_entry'].dt.tz_localize('US/Central')
df['time_entry2'] = df['time_entry2'].dt.tz_localize('US/Central')
Now perform the time difference (subtraction) between the 2 columns and get the time difference in number of days (as a float)
Method 1 gives Diff_days1
Method 2 gives Diff_days2
Method 3 gives Diff_days3
df['Diff_days1'] = (df['time_entry'] - df['time_entry2']).dt.total_seconds()/60/60/24
df['Diff_days2'] = (df['time_entry'] - df['time_entry2']) / np.timedelta64(1, 'D')
df['Diff_days3'] = (df['time_entry'].sub(df['time_entry2'])).dt.total_seconds()/60/60/24
print(df)
time_entry time_entry2 Diff_days1 Diff_days2 Diff_days3
0 1900-01-01 12:01:00 1900-01-01 13:03:00 -0.043056 -0.043056 -0.043056
1 1900-01-01 15:03:00 1900-01-01 14:04:00 0.040972 0.040972 0.040972
2 1900-01-01 16:43:00 1900-01-01 19:23:00 -0.111111 -0.111111 -0.111111
3 1900-01-01 14:11:00 1900-01-01 18:12:00 -0.167361 -0.167361 -0.167361
EDIT
If you're trying to access datetime attributes, then you can do so by using the time_entry column directly (not the time difference column). Here's an example
df['day1'] = df['time_entry'].dt.day
df['time1'] = df['time_entry'].dt.time
df['minute1'] = df['time_entry'].dt.minute
df['dayofweek1'] = df['time_entry'].dt.weekday
df['day2'] = df['time_entry2'].dt.day
df['time2'] = df['time_entry2'].dt.time
df['minute2'] = df['time_entry2'].dt.minute
df['dayofweek2'] = df['time_entry2'].dt.weekday
print(df[['day1', 'time1', 'minute1', 'dayofweek1',
'day2', 'time2', 'minute2', 'dayofweek2']])
day1 time1 minute1 dayofweek1 day2 time2 minute2 dayofweek2
0 1 12:01:00 1 0 1 13:03:00 3 0
1 1 15:03:00 3 0 1 14:04:00 4 0
2 1 16:43:00 43 0 1 19:23:00 23 0
3 1 14:11:00 11 0 1 18:12:00 12 0
Related
I want to create a new column which contains the values of column diff(s) but in percentage.
Finish Time diff (s)
0 1900-01-01 00:42:43.500 0 days 00:00:00
1 1900-01-01 00:44:01.200 0 days 00:01:17
2 1900-01-01 00:44:06.500 0 days 00:01:23
3 1900-01-01 00:44:29.500 0 days 00:01:46
4 1900-01-01 00:44:47.500 0 days 00:02:04
to further understand the data:
df["diff(s)"] = df["Finish Time"] - min(df["Finish Time"])
Finish Time datetime64[ns]
diff (s) timedelta64[ns]
dtype: object
df["diff(%)"] = ((df["Finish Time"]/min(df["Finish
Time"]))*100)
-> results in this error
TypeError: cannot perform __truediv__ with this index type:
DatetimeArray
It depends how are defined percentages - if need divide by summed timedeltas:
df["diff(s)"] = df["Finish Time"] - df["Finish Time"].min()
df["diff(%)"] = (df["diff(s)"] / df["diff(s)"].sum()) * 100
print (df)
Finish Time diff(s) diff(%)
0 1900-01-01 00:42:43.500 0 days 00:00:00 0.000000
1 1900-01-01 00:44:01.200 0 days 00:01:17.700000 19.887382
2 1900-01-01 00:44:06.500 0 days 00:01:23 21.243921
3 1900-01-01 00:44:29.500 0 days 00:01:46 27.130791
4 1900-01-01 00:44:47.500 0 days 00:02:04 31.737906
Or using Series.pct_change:
df["diff(%)"] = df["diff(s)"].pct_change() * 100
I've been trying to convert a milliseconds (0, 5000, 10000) column into a new column with the format: 00:00:00 (00:00:05, 00:00:10 etc)
I tried datetime.datetime.fromtimestamp(5000/1000.0) but it didn't give me the format I wanted.
Any help appreciated!
The best is probably to convert to TimeDelta (using pandas.to_timedelta).
Thus you'll benefit from the timedelta object properties
s = pd.Series([0, 5000, 10000])
s2 = pd.to_timedelta(s, unit='ms')
output:
0 0 days 00:00:00
1 0 days 00:00:05
2 0 days 00:00:10
dtype: timedelta64[ns]
If you really want the '00:00:00' format, use instead pandas.to_datetime:
s2 = pd.to_datetime(s, unit='ms').dt.time
output:
0 00:00:00
1 00:00:05
2 00:00:10
dtype: object
optionally with .astype(str) to have strings
Convert values to timedeltas by to_timedelta:
df['col'] = pd.to_timedelta(df['col'], unit='ms')
print (df)
col
0 0 days 00:00:00
1 0 days 00:00:05
2 0 days 00:00:10
I have a column in my dataframe which I want to convert to a Timestamp. However, it is in a bit of a strange format that I am struggling to manipulate. The column is in the format HHMMSS, but does not include the leading zeros.
For example for a time that should be '00:03:15' the dataframe has '315'. I want to convert the latter to a Timestamp similar to the former. Here is an illustration of the column:
message_time
25
35
114
1421
...
235347
235959
Thanks
Use Series.str.zfill for add leading zero and then to_datetime:
s = df['message_time'].astype(str).str.zfill(6)
df['message_time'] = pd.to_datetime(s, format='%H%M%S')
print (df)
message_time
0 1900-01-01 00:00:25
1 1900-01-01 00:00:35
2 1900-01-01 00:01:14
3 1900-01-01 00:14:21
4 1900-01-01 23:53:47
5 1900-01-01 23:59:59
In my opinion here is better create timedeltas by to_timedelta:
s = df['message_time'].astype(str).str.zfill(6)
df['message_time'] = pd.to_timedelta(s.str[:2] + ':' + s.str[2:4] + ':' + s.str[4:])
print (df)
message_time
0 00:00:25
1 00:00:35
2 00:01:14
3 00:14:21
4 23:53:47
5 23:59:59
I have an Excel file with a column named StartTime having hh:mm:ss XX data and the cells are in `h:mm:ss AM/FM' custom format. For example,
ID StartTime
1 12:00:00 PM
2 1:00:00 PM
3 2:00:00 PM
I used the following code to read the file
df = pd.read_excel('./mydata.xls',
sheet_name='Sheet1',
converters={'StartTime' : str},
)
df shows
ID StartTime
1 12:00:00
2 1:00:00
3 2:00:00
Is it a bug or how do you overcome this? Thanks.
[Update: 7-Dec-2018]
I guess I may have made changes to the Excel file that made it weird. I created another Excel file and present here (I could not attach an Excel file here, and it is not safe too):
I created the following code to test:
import pandas as pd
df = pd.read_excel('./Book1.xlsx',
sheet_name='Sheet1',
converters={'StartTime': str,
'EndTime': str
}
)
df['Hours1'] = pd.NaT
df['Hours2'] = pd.NaT
print(df,'\n')
df.loc[~df.StartTime.isnull() & ~df.EndTime.isnull(),
'Hours1'] = pd.to_datetime(df.EndTime) - pd.to_datetime(df.StartTime)
df['Hours2'] = pd.to_datetime(df.EndTime) - pd.to_datetime(df.StartTime)
print(df)
The outputs are
ID StartTime EndTime Hours1 Hours2
0 0 11:00:00 12:00:00 NaT NaT
1 1 12:00:00 13:00:00 NaT NaT
2 2 13:00:00 14:00:00 NaT NaT
3 3 NaN NaN NaT NaT
4 4 14:00:00 NaN NaT NaT
ID StartTime EndTime Hours1 Hours2
0 0 11:00:00 12:00:00 3600000000000 01:00:00
1 1 12:00:00 13:00:00 3600000000000 01:00:00
2 2 13:00:00 14:00:00 3600000000000 01:00:00
3 3 NaN NaN NaT NaT
4 4 14:00:00 NaN NaT NaT
Now the question has become: "Using pandas to perform time delta from 2 "hh:mm:ss XX" columns in Microsoft Excel". I have changed the title of the question too. Thank you for those who replied and tried it out.
The question is
How to represent the time value to hour instead of microseconds?
It seems that the StartTime column is formated as text in your file.
Have you tried reading it with parse_dates along with a parser function specified via the date_parser parameter? Should work similar to read_csv() although the docs don't list the above options explicitly despite them being available.
Like so:
pd.read_excel(r'./mydata.xls',
parse_dates=['StartTime'],
date_parser=lambda x: pd.datetime.strptime(x, '%I:%M:%S %p').time())
Given the update:
pd.read_excel(r'./mydata.xls', parse_dates=['StartTime', 'EndTime'])
(df['EndTime'] - df['StartTime']).dt.seconds//3600
alternatively
# '//' is available since pandas v0.23.4, otherwise use '/' and round
(df['EndTime'] - df['StartTime'])//pd.Timedelta(1, 'h')
both resulting in the same
0 1
1 1
2 1
dtype: int64
I have a data file that contains the evendata such as event starting date (Date), starting time (KOTime) and event time (EveTime).
The following is the sample of data.
df = pd.DataFrame()
df['Date'] = ['2018/08/12','2018/08/12','2018/08/12','2018/08/12','2018/08/12','2018/08/12']
df['KOTime'] = ['12:30:00','12:30:00','12:30:00','12:30:00','12:30:00','12:30:00']
df['EveTime'] = ['04:50:00','01:03:00','1900-01-03 05:22:00','1900-01-02 16:04:00','1900-01-01 10:28:00','1900-01-01 16:23:00']
Evetime is not formatted in the raw data file as can be seen in the data.
if the Evetime is greater than 24 hours, it is shown as 1900-01-xx .
If we look at the 3rd value of EveTime, it is shown as 1900-01-03 05:22:00.
It is supposed to be 2018/08/12 13:47:22.
I want to create a new column that contains Date and EveTime and the expected output is as follow:
2018/08/12 12:34:50
2018/08/12 12:31:03
2018/08/12 13:47:22
2018/08/12 13:34:04
2018/08/12 12:40:28
2018/08/12 12:46:23
Can anyone suggest me how to do to get the format mentioned above?
I think need convert values to timedeltas and add to datetimes column:
#convert to numeric
num = pd.to_numeric(df['EveTime'].str[-11:-8], errors='coerce')
#convert to timedeltas with seconds
td1 = pd.to_timedelta(np.where(num > 1, num, 0) * 24 * 60, unit='s')
td2 = pd.to_timedelta('00:' + df['EveTime'].str[-8:-3])
df['date'] = pd.to_datetime(df['Date'] + ' ' + df['KOTime']) + td1 + td2
print (df)
Date KOTime EveTime date
0 2018/08/12 12:30:00 04:50:00 2018-08-12 12:34:50
1 2018/08/12 12:30:00 01:03:00 2018-08-12 12:31:03
2 2018/08/12 12:30:00 1900-01-03 05:22:00 2018-08-12 13:47:22
3 2018/08/12 12:30:00 1900-01-02 16:04:00 2018-08-12 13:34:04
4 2018/08/12 12:30:00 1900-01-01 10:28:00 2018-08-12 12:40:28
5 2018/08/12 12:30:00 1900-01-01 16:23:00 2018-08-12 12:46:23
print (td1)
TimedeltaIndex(['00:00:00', '00:00:00', '01:12:00', '00:48:00', '00:00:00',
'00:00:00'],
dtype='timedelta64[ns]', freq=None)
print (td2)
0 00:04:50
1 00:01:03
2 00:05:22
3 00:16:04
4 00:10:28
5 00:16:23
Name: EveTime, dtype: timedelta64[ns]