Develop Reference

python pandas converting UTC integer to datetime

I am calling some financial data from an API which is storing the time values as (I think) UTC (example below):
enter image description here
I cannot seem to convert the entire column into a useable date, I can do it for a single value using the following code so I know this works, but I have 1000's of rows with this problem and thought pandas would offer an easier way to update all the values.
from datetime import datetime
tx = int('1645804609719')/1000
print(datetime.utcfromtimestamp(tx).strftime('%Y-%m-%d %H:%M:%S'))
Any help would be greatly appreciated.

Simply use pandas.DataFrame.apply:
df['date'] = df.date.apply(lambda x: datetime.utcfromtimestamp(int(x)/1000).strftime('%Y-%m-%d %H:%M:%S'))
Another way to do it is by using pd.to_datetime as recommended by Panagiotos in the comments:
df['date'] = pd.to_datetime(df['date'],unit='ms')

You can use "to_numeric" to convert the column in integers, "div" to divide it by 1000 and finally a loop to iterate the dataframe column with datetime to get the format you want.
import pandas as pd
import datetime
df = pd.DataFrame({'date': ['1584199972000', '1645804609719'], 'values': [30,40]})
df['date'] = pd.to_numeric(df['date']).div(1000)
for i in range(len(df)):
df.iloc[i,0] = datetime.utcfromtimestamp(df.iloc[i,0]).strftime('%Y-%m-%d %H:%M:%S')
print(df)
Output:
date values
0 2020-03-14 15:32:52 30
1 2022-02-25 15:56:49 40

Python change time format

I'm trying to change the time format of my data that's now in form of 15:41:28:4330 or hh:mm:ss:msmsmsms to seconds.
I browsed through some of the pandas documentation but can't seem to find this format anywhere.
Would it be possible to simply calculate the seconds from that time format row by row?

You'll want to obtain a timedelta and take the total_seconds method to get seconds after midnight. So you can parse to datetime first, and subtract the default date (that will be added automatically). Ex:
#1 - via datetime
import pandas as pd
df = pd.DataFrame({'time': ["15:41:28:4330"]})
df['time'] = pd.to_datetime(df['time'], format='%H:%M:%S:%f')
df['sec_after_mdnt'] = (df['time']-df['time'].dt.floor('d')).dt.total_seconds()
df
time sec_after_mdnt
0 1900-01-01 15:41:28.433 56488.433
Alternatively, you can clean your time format and parse directly to timedelta:
#2 - str cleaning & to timedelta
df = pd.DataFrame({'time': ["15:41:28:4330"]})
# last separator must be a dot...
df['time'] = df['time'].str[::-1].str.replace(':', '.', n=1, regex=False).str[::-1]
df['sec_after_mdnt'] = pd.to_timedelta(df['time']).dt.total_seconds()
df
time sec_after_mdnt
0 15:41:28.4330 56488.433

Problem converting time into pandas datetime

I am trying to convert a date column containing only hours, minutes and seconds ito a datetime form using pandas.to_datetime(). However, it adds year and date automatically. I also tried using
pandas.to_datetime(df["time"], format = %H:%M:%S").dt.time, again the data type remains object.
Is there any method that can change into datetime format without year and date?

Something like this?
df['Time'] = pd.to_datetime(df['Time'], format='%H:%M:%S', errors='ignore')

put .dt.time on the end
df['Time'] = pd.to_datetime(df['Time'], format='%H:%M:%S', errors='ignore').dt.time

pandas.to_datetime with different length date strings

I have a column of timestamps that I would like to convert to datetime in my pandas dataframe. The format of the dates is %Y-%m-%d-%H-%M-%S which pd.to_datetime does not recognize. I have manually entered the format as below:
df['TIME'] = pd.to_datetime(df['TIME'], format = '%Y-%m-%d-%H-%M-%S')
My problem is some of the times do not have seconds so they are shorter
(format = %Y-%m-%d-%H-%M).
How can I get all of these strings to datetimes?
I was thinking I could add zero seconds (-0) to the end of my shorter dates but I don't know how to do that.

try strftime and if you want the right format and if Pandas can't recognize your custom datetime format, you should provide it explicetly
from functools import partial
df1 = pd.DataFrame({'Date': ['2018-07-02-06-05-23','2018-07-02-06-05']})
newdatetime_fmt = partial(pd.to_datetime, format='%Y-%m-%d-%H-%M-%S')
df1['Clean_Date'] = (df1.Date.str.replace('-','').apply(lambda x: pd.to_datetime(x).strftime('%Y-%m-%d-%H-%M-%S'))
.apply(newdatetime_fmt))
print(df1,df1.dtypes)
output:
Date Clean_Date
0 2018-07-02-06-05-23 2018-07-02 06:05:23
1 2018-07-02-06-05 2018-07-02 06:05:00
Date object
Clean_Date datetime64[ns]

How to convert unusual 24 hour date time format in python?

I have a dataframe column full datetime type that are in the format
2016Oct03:14:38:33
Right now, the data type of this column of the dataframe is String. I would like to convert it into datetime in order to be able perform some numerical operations like subtractions on them. I have tried specifying the format while using pd.to_datetime but as the time is in a 24 hr format, it is throwing up an error. What is the best way to do this? Thanks in advance!

There doesn't seem to be anything unusual about the time format at all; 24 hour is absolutely standard.
Just the normal strptime is fine:
datetime.strptime(my_date, '%Y%b%d:%H:%M:%S')

You need to_datetime with parameter format:
df = pd.DataFrame({'dates':['2016Oct03:14:38:33',
'2016Oct03:14:38:33',
'2016Oct03:14:38:33']})
print (df)
dates
0 2016Oct03:14:38:33
1 2016Oct03:14:38:33
2 2016Oct03:14:38:33
df.dates = pd.to_datetime(df.dates, format='%Y%b%d:%H:%M:%S')
print (df)
dates
0 2016-10-03 14:38:33
1 2016-10-03 14:38:33
2 2016-10-03 14:38:33

Duplicated question
Use datetime.strptime
Ex:
from datetime import datetime
date_object = datetime.strptime('2016Oct03:14:38:33', '%Y%b%d:%H:%M:%S')
Doc : https://docs.python.org/2/library/datetime.html