Develop Reference

Convert object to datetime, then split into 2 columns for removing duplicates [duplicate]

This question already has answers here:
Splitting timestamp column into separate date and time columns
(9 answers)
Closed 1 year ago.
I am having difficulty with converting date object into datetime.
My date & time object within the df looks like this:
Date
2/28/20 18:00
and I am doing this to convert it to datetime format;
from datetime import datetime
df['Date'] = pd.to_datetime(df['Date'], format='%m/%d/%y %H:%M)
I thought this worked as when I do dtypes I am getting the Date column in datetime format. This is how the date column became reflected as after running the above:
Date
2020-02-28 18:00:00
However after converting to datetime, I would like to split the datetime column into 'Date' and 'Time', with the data displayed like this:
Date Time
2020-02-28 18:00
I tried to run this:
df['Date'] = df['Date'].dt.strftime('%Y-%m-%d')
df['Time'] = df['Date'].dt.strftime('%H:%M')
but am getting 'AttributeError: Can only use .dt accessor with datetimelike values'
The main goal I want to achieve after splitting the into 2 columns, date and time, is so that I can then remove duplicates and keep only the last timing for each date in the dataframe.
Any advice appreciated, thanks in advance.

The problem is that in the next df['Date'].dt, df['Date'] has been converted to string. You need to swap the assign sequence.
df['Time'] = df['Date'].dt.strftime('%H:%M')
df['Date'] = df['Date'].dt.strftime('%Y-%m-%d')

Convert multiple columns to Datetime at once keeping just the time

Trying to change multiple columns to the same datatype at once,
columns contain time data like hours minute and seconds, like
And the data
and I'm not able to change multiple columns at once to using pd.to_datetime to only the time format, I don't want the date because, if I do pd.to_datetime the date also gets added to the column which is not required, just want the time
how to convert the column to DateTime and only keep time in the column

First You can't have a datetime with only time in it in pandas/python.
So
Because python time is object in pandas convert all columns to datetimes (but there are also dates):
cols = ['Total Break Time','col1','col2']
df[cols] = df[cols].apply(pd.to_datetime)
Or convert columns to timedeltas, it looks like similar times, but possible working by datetimelike methods in pandas:
df[cols] = df[cols].apply(pd.to_timedelta)

You can pick only time as below:
import time
df['Total Break Time'] = pd.to_datetime(df['Total Break Time'],format= '%H:%M:%S' ).dt.time
Then you can repeat this for all your columns, as I suppose you already are.
The catch is, to convert to datetime and then only picking out what you need.

Convert Python object column in dataframe to time without date using Pandas

I have a column in my dataframe that lists time in HH:MM:SS. When I run dtype on the column, it comes up with dtype('o') and I want to be able to use it as the x-axis for plotting some of my other signals. I saw previous documentation on using to_datetime and tried to use that to convert it to a usable time format for matplotlib.
Used pandas version is 0.18.1
I used:
time=pd.to_datetime(df.Time,format='%H:%M:%S')
where the output then becomes:
time
0 1900-01-01 00:00:01
and is carried out for the rest of the data points in the column.
Even though I specified just hour,minutes,and seconds I am still getting date. Why is that? I also tried
time.hour()
just to extract the hour portion but then I get an error that it doesn't have an 'hour' attribute.
Any help is much appreciated! Thanks!

Now in 2019, using pandas 0.25.0 and Python 3.7.3.
(Note : Edited answer to take plotting in account)
Even though I specified just hour,minutes,and seconds I am still getting date. Why is that?
According to pandas documentation I think it's because in a pandas Timestamp (equivalent of Datetime) object, the arguments year, month and day are mandatory, while hour, minutes and seconds are optional.
Therefore if you convert your object-type object in a Datetime, it must have a year-month-day part - if you don't indicate one, it will be the default 1900-01-01.
Since you also have a Date column in your sample, you can use it to have a datetime column with the right dates that you can use to plot :
import pandas as pd
df['Time'] = df.Date + " " + df.Time
df['Time'] = pd.to_datetime(df['Time'], format='%m/%d/%Y %H:%M:%S')
df.plot('Time', subplots=True)
With this your 'Time' column will display values like : 2016-07-25 01:12:07 and its dtype is datetime64[ns].
That being said, IF you plot day by day and you only want to compare times within a day (and not dates+times), having a default date does not seem bothering as long as it's the same date for all times - the times will be correctly compared on a same day, be it a wrong one.
And in the least likely case you would still want a time-only column, this is the reverse operation :
import pandas as pd
df['Time-only'] = pd.to_datetime(df['Time'], format='%H:%M:%S').dt.time
As explained before, it doesn't have a date (year-month-day) so it cannot be a datetime object, therefore this column will be in Object format.

You can extract a time object like:
import pandas as pd
df = pd.DataFrame([['12:10:20']], columns={"time": "item"})
time = pd.to_datetime(df.time, format='%H:%M:%S').dt.time[0]
After which you can extract desired properties as:
hour = time.hour
(Source)

Dropping rows from a Dataframe based on Date

How can I drop rows from Dataframe df if the dates associated with df['maturity_dt'] are less that today's date?
I am currently doing the following:
todays_date = datetime.date.today()
datenow = datetime.datetime.combine(todays_date, datetime.datetime.min.time()) #Converting to datetime
for (i,row) in df.iterrows():
if datetime.datetime.strptime(row['maturity_dt'], '%Y-%m-%d %H:%M:%S.%f') < datenow):
df.drop(df.index[i])
However, its taking too long and I was hoping to do something like: df = df[datetime.datetime.strptime(df['maturity_dt'], '%Y-%m-%d %H:%M:%S.%f') < datenow, but this results in the error TypeError: must be str, not Series
Thank You

Haven't tried it but maybe the pandas native functions will iterate faster. Something like:
df['dt']=pandas.Datetimeindex(df['maturity_dt'])
newdf=df.loc[df['dt']<=todays_date].copy()

Instead of parsing the date in each row, you could format your comparison date in the same format as these dates are stored and then you could just do a string comparison.
Also, if there is a way to drop multiple rows in a single call, you could use your loop just to gather the indices of those rows to be dropped, then use that call to drop them in bunches.

Issue with pandas to_datetime function

I have a column that is unix timestamps. I want to convert this column to just dates in a %y-%m-%d format. Just to test the to_datetime() function I did the below, which works as expected and gives me the column in a format like this 2015-05-12 00:11:30 :
df['time'] = pd.to_datetime(df['time'], unit='s')
When I add in the format argument Like below, I get an error:
df['time'] = pd.to_datetime(df['time'], unit='s', format='%d/%m/%Y')
The error is ValueError: time data 1431389490 does not match format '%d/%m/%Y'
How can I strip off the hours, minutes and seconds so I am only left with 2014-05-12?

If you want to extract just the date, you can do that in a second step after converting to datetime:
x = pd.to_datetime(pd.Series([1431389490]), unit='s')
# Datetime columns have a `.dt` attribute, with useful properties
# and methods for working with dates
x.dt.date
Out[7]:
0 2015-05-12
dtype: object
This will discard the information about hours and minutes, but you will be able to work with the resulting column/series easily because the result is a datetime.date object, e.g. subtracting to find the number of days between your column and a certain date.
If you want to keep the information about hours and minutes, but only display it differently, I'm not sure that's possible.