Develop Reference

How to remove hours, minutes, seconds and UTC offset from pandas date column? I'm running with streamlit and pandas

How to remove T00:00:00+05:30 after year, month and date values in pandas? I tried converting the column into datetime but also it's showing the same results, I'm using pandas in streamlit. I tried the below code
df['Date'] = pd.to_datetime(df['Date'])
The output is same as below :
Date
2019-07-01T00:00:00+05:30
2019-07-01T00:00:00+05:30
2019-07-02T00:00:00+05:30
2019-07-02T00:00:00+05:30
2019-07-02T00:00:00+05:30
2019-07-03T00:00:00+05:30
2019-07-03T00:00:00+05:30
2019-07-04T00:00:00+05:30
2019-07-04T00:00:00+05:30
2019-07-05T00:00:00+05:30
Can anyone help me how to remove T00:00:00+05:30 from the above rows?

If I understand correctly, you want to keep only the date part.
Convert date strings to datetime
df = pd.DataFrame(
columns={'date'},
data=["2019-07-01T02:00:00+05:30", "2019-07-02T01:00:00+05:30"]
)
date
0 2019-07-01T02:00:00+05:30
1 2019-07-02T01:00:00+05:30
2 2019-07-03T03:00:00+05:30
df['date'] = pd.to_datetime(df['date'])
date
0 2019-07-01 02:00:00+05:30
1 2019-07-02 01:00:00+05:30
Remove the timezone
df['datetime'] = df['datetime'].dt.tz_localize(None)
date
0 2019-07-01 02:00:00
1 2019-07-02 01:00:00
Keep the date only
df['date'] = df['date'].dt.date
0 2019-07-01
1 2019-07-02

Don't bother with apply to Python dates or string changes. The former will leave you with an object type column and the latter is slow. Just round to the day frequency using the library function.
>>> pd.Series([pd.Timestamp('2000-01-05 12:01')]).dt.round('D')
0 2000-01-06
dtype: datetime64[ns]
If you have a timezone aware timestamp, convert to UTC with no time zone then round:
>>> pd.Series([pd.Timestamp('2019-07-01T00:00:00+05:30')]).dt.tz_convert(None) \
.dt.round('D')
0 2019-07-01
dtype: datetime64[ns]

Pandas doesn't have a builtin conversion to datetime.date, but you could use .apply to achieve this if you want to have date objects instead of string:
import pandas as pd
import datetime
df = pd.DataFrame(
{"date": [
"2019-07-01T00:00:00+05:30",
"2019-07-01T00:00:00+05:30",
"2019-07-02T00:00:00+05:30",
"2019-07-02T00:00:00+05:30",
"2019-07-02T00:00:00+05:30",
"2019-07-03T00:00:00+05:30",
"2019-07-03T00:00:00+05:30",
"2019-07-04T00:00:00+05:30",
"2019-07-04T00:00:00+05:30",
"2019-07-05T00:00:00+05:30"]})
df["date"] = df["date"].apply(lambda x: datetime.datetime.fromisoformat(x).date())
print(df)

Convert multiple format date into only one

I have a date column in df which needs to align the format.
The column has multiple formats at the moment such as 20201203, 05/06/20, 2019-09-15 00:00:1568480400.
My expected results need to be in YYYY-MM-DD format, I tried pd.to_datetime(df, format='%Y-%m-%d')before, but then received 2 errors 05/06/20 doesn't match format specified and second must be in 0..59
I assume that there would be some initial codes to pre-process the data, is it true. Or is there any proper function? Please help.
Thank you so much, everyone.

Use to_datetime function without format parameter. Let Pandas infer the datetime format:
df = pd.DataFrame({"date": ["20201203", "05/06/20", "2019-09-15 00:00:00.1568480400"]})
df["date"] = pd.to_datetime(df["date"]).dt.strftime("%Y-%m-%d")
>>> df
date
0 2020-12-03
1 2020-05-06
2 2019-09-15
Check your input data:
2019-09-15 00:00:1568480400 is not valid:
ParserError: second must be in 0..59: 2019-09-15 00:00:1568480400

Input
from dateutil.parser import parse
df=pd.DataFrame({
'Date':['20201203', '05/06/20', '2019-09-15 00:00:1568480400']
})
Two options
First
df.Date = df.Date.str.split(' ',1).str[0] ## If `'2019-09-15 00:00:1568480400'` is a valid date in your df.
df["Date"] = pd.to_datetime(df["Date"]).dt.strftime("%Y-%m-%d")
Second
for i in range(len(df['Date'])):
df['Date'][i] = parse(df['Date'][i])
df['Date'] = pd.to_datetime(df['Date']).dt.strftime("%Y-%m-%d")
df
Output
Date
0 2020-12-03
1 2020-05-06
2 2019-09-15

How to extract date AND hour from date time in python?

What's the best way to do this? I thought about extracting the two separately then combining them? This doesn't seem like it should be the most efficient way?
df['date'] = df['datetime'].dt.date
df['hour'] = df['datetime'].hour
df['dateAndHour'] = df['datetime'].dt.date.astype(str) + ' ' + df['datetime'].dt.hour.astype(str)

You can use strftime and it depends on the format your date is in and how you want to combine them
from datetime import datetime
import pandas as pd
df = pd.DataFrame({'date':[datetime.now()]})
df['date-hour'] = df.date.dt.strftime('%Y-%m-%d %H')
df
date date-hour
0 2020-11-18 11:03:38.390393 2020-11-18 11

Depends what you want to do with it, but one way to do this would be to use strftime to format the datetime column to %Y-%m-%d %H or similar:
>>> df
datetime
0 2020-01-01 12:15:00
1 2020-10-22 11:11:11
>>> df.datetime.dt.strftime("%Y-%m-%d %H")
0 2020-01-01 12
1 2020-10-22 11
Name: datetime, dtype: object

Remove the days in the timedelta object

I have a column in a pandas dataframe that is created after subtracting two times. I now have a timedelta object like this -1 days +02:45:00. I just need to remove the -1 days and want it to be 02:45:00. Is there a way to do this?

I think you can subtract days converted to timedeltas:
td = pd.to_timedelta(['-1 days +02:45:00','1 days +02:45:00','0 days +02:45:00'])
df = pd.DataFrame({'td': td})
df['td'] = df['td'] - pd.to_timedelta(df['td'].dt.days, unit='d')
print (df.head())
td
0 02:45:00
1 02:45:00
2 02:45:00
print (type(df.loc[0, 'td']))
<class 'pandas._libs.tslibs.timedeltas.Timedelta'>
Or convert timedeltas to strings and extract strings between days and .:
df['td'] = df['td'].astype(str).str.extract('days (.*?)\.')
print (df.head())
td
0 +02:45:00
1 02:45:00
2 02:45:00
print (type(df.loc[0, 'td']))
<class 'str'>

I found this method easy, others didnt work for me
df['column'] = df['column'].astype(str).map(lambda x: x[7:])
It slices of the days part and you only get time part

If your column is named time1, you can do it like this:
import pandas as pd
import datetime as dt
df['time1'] = pd.to_datetime(str(df.time1)[11:19]) #this slice can be adjusted
df['time1'] = df.time1.dt.time
this is going to convert the timedelta to str, slice the time part from it, convert it to datetime and extract the time from that.

I found a very easy solution for other people who may encounter this problem:
if timedelta_obj.days < 0:
timedelta_obj.days = datetime.timedelta(
seconds=timedelta_obj.total_seconds() + 3600*24)

Pandas excel import changes the Date format

Im learning python (3.6 with anaconda) for my studies.
Im using pandas to import a xls file with 2 columns : Date (dd-mm-yyyy) and price.
But pandas changes the date format :
xls_file = pd.read_excel('myfile.xls')
print(xls_file.iloc[0, 0])
Im getting :
2010-01-04 00:00:00
instead of :
04-01-2010 or at least : 2010-01-04
I dont know why hh:mm:ss is added, I get the same result for each row from the Date column. I tried also different things using to_datetime but it didnt fix it.
Any idea ?
Thanks

What you need is to define the format that the datetime values get printed. There might be a more elegant way to do it but something like that will work:
In [11]: df
Out[11]:
id date
0 1 2017-09-12
1 2 2017-10-20
# Specifying the format
In [16]: print(pd.datetime.strftime(df.iloc[0,1], "%Y-%m-%d"))
2017-09-12
If you want to store the date as string in your specific format then you can also do something like:
In [17]: df["datestr"] = pd.datetime.strftime(df.iloc[0,1], "%Y-%m-%d")
In [18]: df
Out[18]:
id date datestr
0 1 2017-09-12 2017-09-12
1 2 2017-10-20 2017-09-12
In [19]: df.dtypes
Out[19]:
id int64
date datetime64[ns]
datestr object
dtype: object