Related
How do I convert a numpy.datetime64 object to a datetime.datetime (or Timestamp)?
In the following code, I create a datetime, timestamp and datetime64 objects.
import datetime
import numpy as np
import pandas as pd
dt = datetime.datetime(2012, 5, 1)
# A strange way to extract a Timestamp object, there's surely a better way?
ts = pd.DatetimeIndex([dt])[0]
dt64 = np.datetime64(dt)
In [7]: dt
Out[7]: datetime.datetime(2012, 5, 1, 0, 0)
In [8]: ts
Out[8]: <Timestamp: 2012-05-01 00:00:00>
In [9]: dt64
Out[9]: numpy.datetime64('2012-05-01T01:00:00.000000+0100')
Note: it's easy to get the datetime from the Timestamp:
In [10]: ts.to_datetime()
Out[10]: datetime.datetime(2012, 5, 1, 0, 0)
But how do we extract the datetime or Timestamp from a numpy.datetime64 (dt64)?
.
Update: a somewhat nasty example in my dataset (perhaps the motivating example) seems to be:
dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100')
which should be datetime.datetime(2002, 6, 28, 1, 0), and not a long (!) (1025222400000000000L)...
You can just use the pd.Timestamp constructor. The following diagram may be useful for this and related questions.
Welcome to hell.
You can just pass a datetime64 object to pandas.Timestamp:
In [16]: Timestamp(numpy.datetime64('2012-05-01T01:00:00.000000'))
Out[16]: <Timestamp: 2012-05-01 01:00:00>
I noticed that this doesn't work right though in NumPy 1.6.1:
numpy.datetime64('2012-05-01T01:00:00.000000+0100')
Also, pandas.to_datetime can be used (this is off of the dev version, haven't checked v0.9.1):
In [24]: pandas.to_datetime('2012-05-01T01:00:00.000000+0100')
Out[24]: datetime.datetime(2012, 5, 1, 1, 0, tzinfo=tzoffset(None, 3600))
To convert numpy.datetime64 to datetime object that represents time in UTC on numpy-1.8:
>>> from datetime import datetime
>>> import numpy as np
>>> dt = datetime.utcnow()
>>> dt
datetime.datetime(2012, 12, 4, 19, 51, 25, 362455)
>>> dt64 = np.datetime64(dt)
>>> ts = (dt64 - np.datetime64('1970-01-01T00:00:00Z')) / np.timedelta64(1, 's')
>>> ts
1354650685.3624549
>>> datetime.utcfromtimestamp(ts)
datetime.datetime(2012, 12, 4, 19, 51, 25, 362455)
>>> np.__version__
'1.8.0.dev-7b75899'
The above example assumes that a naive datetime object is interpreted by np.datetime64 as time in UTC.
To convert datetime to np.datetime64 and back (numpy-1.6):
>>> np.datetime64(datetime.utcnow()).astype(datetime)
datetime.datetime(2012, 12, 4, 13, 34, 52, 827542)
It works both on a single np.datetime64 object and a numpy array of np.datetime64.
Think of np.datetime64 the same way you would about np.int8, np.int16, etc and apply the same methods to convert between Python objects such as int, datetime and corresponding numpy objects.
Your "nasty example" works correctly:
>>> from datetime import datetime
>>> import numpy
>>> numpy.datetime64('2002-06-28T01:00:00.000000000+0100').astype(datetime)
datetime.datetime(2002, 6, 28, 0, 0)
>>> numpy.__version__
'1.6.2' # current version available via pip install numpy
I can reproduce the long value on numpy-1.8.0 installed as:
pip install git+https://github.com/numpy/numpy.git#egg=numpy-dev
The same example:
>>> from datetime import datetime
>>> import numpy
>>> numpy.datetime64('2002-06-28T01:00:00.000000000+0100').astype(datetime)
1025222400000000000L
>>> numpy.__version__
'1.8.0.dev-7b75899'
It returns long because for numpy.datetime64 type .astype(datetime) is equivalent to .astype(object) that returns Python integer (long) on numpy-1.8.
To get datetime object you could:
>>> dt64.dtype
dtype('<M8[ns]')
>>> ns = 1e-9 # number of seconds in a nanosecond
>>> datetime.utcfromtimestamp(dt64.astype(int) * ns)
datetime.datetime(2002, 6, 28, 0, 0)
To get datetime64 that uses seconds directly:
>>> dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100', 's')
>>> dt64.dtype
dtype('<M8[s]')
>>> datetime.utcfromtimestamp(dt64.astype(int))
datetime.datetime(2002, 6, 28, 0, 0)
The numpy docs say that the datetime API is experimental and may change in future numpy versions.
I think there could be a more consolidated effort in an answer to better explain the relationship between Python's datetime module, numpy's datetime64/timedelta64 and pandas' Timestamp/Timedelta objects.
The datetime standard library of Python
The datetime standard library has four main objects
time - only time, measured in hours, minutes, seconds and microseconds
date - only year, month and day
datetime - All components of time and date
timedelta - An amount of time with maximum unit of days
Create these four objects
>>> import datetime
>>> datetime.time(hour=4, minute=3, second=10, microsecond=7199)
datetime.time(4, 3, 10, 7199)
>>> datetime.date(year=2017, month=10, day=24)
datetime.date(2017, 10, 24)
>>> datetime.datetime(year=2017, month=10, day=24, hour=4, minute=3, second=10, microsecond=7199)
datetime.datetime(2017, 10, 24, 4, 3, 10, 7199)
>>> datetime.timedelta(days=3, minutes = 55)
datetime.timedelta(3, 3300)
>>> # add timedelta to datetime
>>> datetime.timedelta(days=3, minutes = 55) + \
datetime.datetime(year=2017, month=10, day=24, hour=4, minute=3, second=10, microsecond=7199)
datetime.datetime(2017, 10, 27, 4, 58, 10, 7199)
NumPy's datetime64 and timedelta64 objects
NumPy has no separate date and time objects, just a single datetime64 object to represent a single moment in time. The datetime module's datetime object has microsecond precision (one-millionth of a second). NumPy's datetime64 object allows you to set its precision from hours all the way to attoseconds (10 ^ -18). It's constructor is more flexible and can take a variety of inputs.
Construct NumPy's datetime64 and timedelta64 objects
Pass an integer with a string for the units. See all units here. It gets converted to that many units after the UNIX epoch: Jan 1, 1970
>>> np.datetime64(5, 'ns')
numpy.datetime64('1970-01-01T00:00:00.000000005')
>>> np.datetime64(1508887504, 's')
numpy.datetime64('2017-10-24T23:25:04')
You can also use strings as long as they are in ISO 8601 format.
>>> np.datetime64('2017-10-24')
numpy.datetime64('2017-10-24')
Timedeltas have a single unit
>>> np.timedelta64(5, 'D') # 5 days
>>> np.timedelta64(10, 'h') 10 hours
Can also create them by subtracting two datetime64 objects
>>> np.datetime64('2017-10-24T05:30:45.67') - np.datetime64('2017-10-22T12:35:40.123')
numpy.timedelta64(147305547,'ms')
Pandas Timestamp and Timedelta build much more functionality on top of NumPy
A pandas Timestamp is a moment in time very similar to a datetime but with much more functionality. You can construct them with either pd.Timestamp or pd.to_datetime.
>>> pd.Timestamp(1239.1238934) #defaults to nanoseconds
Timestamp('1970-01-01 00:00:00.000001239')
>>> pd.Timestamp(1239.1238934, unit='D') # change units
Timestamp('1973-05-24 02:58:24.355200')
>>> pd.Timestamp('2017-10-24 05') # partial strings work
Timestamp('2017-10-24 05:00:00')
pd.to_datetime works very similarly (with a few more options) and can convert a list of strings into Timestamps.
>>> pd.to_datetime('2017-10-24 05')
Timestamp('2017-10-24 05:00:00')
>>> pd.to_datetime(['2017-1-1', '2017-1-2'])
DatetimeIndex(['2017-01-01', '2017-01-02'], dtype='datetime64[ns]', freq=None)
Converting Python datetime to datetime64 and Timestamp
>>> dt = datetime.datetime(year=2017, month=10, day=24, hour=4,
minute=3, second=10, microsecond=7199)
>>> np.datetime64(dt)
numpy.datetime64('2017-10-24T04:03:10.007199')
>>> pd.Timestamp(dt) # or pd.to_datetime(dt)
Timestamp('2017-10-24 04:03:10.007199')
Converting numpy datetime64 to datetime and Timestamp
>>> dt64 = np.datetime64('2017-10-24 05:34:20.123456')
>>> unix_epoch = np.datetime64(0, 's')
>>> one_second = np.timedelta64(1, 's')
>>> seconds_since_epoch = (dt64 - unix_epoch) / one_second
>>> seconds_since_epoch
1508823260.123456
>>> datetime.datetime.utcfromtimestamp(seconds_since_epoch)
>>> datetime.datetime(2017, 10, 24, 5, 34, 20, 123456)
Convert to Timestamp
>>> pd.Timestamp(dt64)
Timestamp('2017-10-24 05:34:20.123456')
Convert from Timestamp to datetime and datetime64
This is quite easy as pandas timestamps are very powerful
>>> ts = pd.Timestamp('2017-10-24 04:24:33.654321')
>>> ts.to_pydatetime() # Python's datetime
datetime.datetime(2017, 10, 24, 4, 24, 33, 654321)
>>> ts.to_datetime64()
numpy.datetime64('2017-10-24T04:24:33.654321000')
>>> dt64.tolist()
datetime.datetime(2012, 5, 1, 0, 0)
For DatetimeIndex, the tolist returns a list of datetime objects. For a single datetime64 object it returns a single datetime object.
One option is to use str, and then to_datetime (or similar):
In [11]: str(dt64)
Out[11]: '2012-05-01T01:00:00.000000+0100'
In [12]: pd.to_datetime(str(dt64))
Out[12]: datetime.datetime(2012, 5, 1, 1, 0, tzinfo=tzoffset(None, 3600))
Note: it is not equal to dt because it's become "offset-aware":
In [13]: pd.to_datetime(str(dt64)).replace(tzinfo=None)
Out[13]: datetime.datetime(2012, 5, 1, 1, 0)
This seems inelegant.
.
Update: this can deal with the "nasty example":
In [21]: dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100')
In [22]: pd.to_datetime(str(dt64)).replace(tzinfo=None)
Out[22]: datetime.datetime(2002, 6, 28, 1, 0)
If you want to convert an entire pandas series of datetimes to regular python datetimes, you can also use .to_pydatetime().
pd.date_range('20110101','20110102',freq='H').to_pydatetime()
> [datetime.datetime(2011, 1, 1, 0, 0) datetime.datetime(2011, 1, 1, 1, 0)
datetime.datetime(2011, 1, 1, 2, 0) datetime.datetime(2011, 1, 1, 3, 0)
....
It also supports timezones:
pd.date_range('20110101','20110102',freq='H').tz_localize('UTC').tz_convert('Australia/Sydney').to_pydatetime()
[ datetime.datetime(2011, 1, 1, 11, 0, tzinfo=<DstTzInfo 'Australia/Sydney' EST+11:00:00 DST>)
datetime.datetime(2011, 1, 1, 12, 0, tzinfo=<DstTzInfo 'Australia/Sydney' EST+11:00:00 DST>)
....
NOTE: If you are operating on a Pandas Series you cannot call to_pydatetime() on the entire series. You will need to call .to_pydatetime() on each individual datetime64 using a list comprehension or something similar:
datetimes = [val.to_pydatetime() for val in df.problem_datetime_column]
This post has been up for 4 years and I still struggled with this conversion problem - so the issue is still active in 2017 in some sense. I was somewhat shocked that the numpy documentation does not readily offer a simple conversion algorithm but that's another story.
I have come across another way to do the conversion that only involves modules numpy and datetime, it does not require pandas to be imported which seems to me to be a lot of code to import for such a simple conversion. I noticed that datetime64.astype(datetime.datetime) will return a datetime.datetime object if the original datetime64 is in micro-second units while other units return an integer timestamp. I use module xarray for data I/O from Netcdf files which uses the datetime64 in nanosecond units making the conversion fail unless you first convert to micro-second units. Here is the example conversion code,
import numpy as np
import datetime
def convert_datetime64_to_datetime( usert: np.datetime64 )->datetime.datetime:
t = np.datetime64( usert, 'us').astype(datetime.datetime)
return t
Its only tested on my machine, which is Python 3.6 with a recent 2017 Anaconda distribution. I have only looked at scalar conversion and have not checked array based conversions although I'm guessing it will be good. Nor have I looked at the numpy datetime64 source code to see if the operation makes sense or not.
import numpy as np
import pandas as pd
def np64toDate(np64):
return pd.to_datetime(str(np64)).replace(tzinfo=None).to_datetime()
use this function to get pythons native datetime object
I've come back to this answer more times than I can count, so I decided to throw together a quick little class, which converts a Numpy datetime64 value to Python datetime value. I hope it helps others out there.
from datetime import datetime
import pandas as pd
class NumpyConverter(object):
#classmethod
def to_datetime(cls, dt64, tzinfo=None):
"""
Converts a Numpy datetime64 to a Python datetime.
:param dt64: A Numpy datetime64 variable
:type dt64: numpy.datetime64
:param tzinfo: The timezone the date / time value is in
:type tzinfo: pytz.timezone
:return: A Python datetime variable
:rtype: datetime
"""
ts = pd.to_datetime(dt64)
if tzinfo is not None:
return datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second, tzinfo=tzinfo)
return datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second)
I'm gonna keep this in my tool bag, something tells me I'll need it again.
I did like this
import pandas as pd
# Custom function to convert Pandas Datetime to Timestamp
def toTimestamp(data):
return data.timestamp()
# Read a csv file
df = pd.read_csv("friends.csv")
# Replace the "birthdate" column by:
# 1. Transform to datetime
# 2. Apply the custom function to the column just converted
df["birthdate"] = pd.to_datetime(df["birthdate"]).apply(toTimestamp)
Some solutions work well for me but numpy will deprecate some parameters.
The solution that work better for me is to read the date as a pandas datetime and excract explicitly the year, month and day of a pandas object.
The following code works for the most common situation.
def format_dates(dates):
dt = pd.to_datetime(dates)
try: return [datetime.date(x.year, x.month, x.day) for x in dt]
except TypeError: return datetime.date(dt.year, dt.month, dt.day)
Only way I managed to convert a column 'date' in pandas dataframe containing time info to numpy array was as following: (dataframe is read from csv file "csvIn.csv")
import pandas as pd
import numpy as np
df = pd.read_csv("csvIn.csv")
df["date"] = pd.to_datetime(df["date"])
timestamps = np.array([np.datetime64(value) for dummy, value in df["date"].items()])
indeed, all of these datetime types can be difficult, and potentially problematic (must keep careful track of timezone information). here's what i have done, though i admit that i am concerned that at least part of it is "not by design". also, this can be made a bit more compact as needed.
starting with a numpy.datetime64 dt_a:
dt_a
numpy.datetime64('2015-04-24T23:11:26.270000-0700')
dt_a1 = dt_a.tolist() # yields a datetime object in UTC, but without tzinfo
dt_a1
datetime.datetime(2015, 4, 25, 6, 11, 26, 270000)
# now, make your "aware" datetime:
dt_a2=datetime.datetime(*list(dt_a1.timetuple()[:6]) + [dt_a1.microsecond], tzinfo=pytz.timezone('UTC'))
... and of course, that can be compressed into one line as needed.
I have a library which returns the timezone of a location in the format: America/New_York. Based on that timezone name, I want to calculate the hours between that time zone and UTC, taking into account daylight savings time and all. I'm using Python.
My first idea was to use the Google python library and search for 'America/New_York time' but that only gave me back a list of urls which I could visit to get the info myself. It would be awesome if I could get the current time seen if I were to manually search 'America/New_York time' into google, right into my program.
I'm sure this question has been asked before, but I am new to stack overflow and python so help is appreciated.
Thanks!
The offset from UTC depends on the date (since daylight saving time may or may not be in effect). So you need to provide a datetime for the comparison.
ZoneInfo.utcoffset will return a timedelta object directly.
>>> from zoneinfo import ZoneInfo
>>> from datetime import datetime
>>> ZoneInfo("America/New_York").utcoffset(datetime(2021, 10, 23)) #EDT
datetime.timedelta(days=-1, seconds=72000)
>>> ZoneInfo("America/New_York").utcoffset(datetime(2021, 11, 15)) #EST
datetime.timedelta(days=-1, seconds=68400)
>>> ZoneInfo("Asia/Tokyo").utcoffset(datetime(2021, 10, 23))
datetime.timedelta(seconds=32400)
Not a complete answer, but maybe you could implement a dictionary that connects these format to the normal format with three letters. With this you can then use datetime and pytz to make the rest. If you don't have too many possible outputs in the current format this would be feasible, otherwise of course not.
>>> from datetime import datetime, timedelta
>>> from datetime import timezone
>>> from zoneinfo import ZoneInfo
>>> dt1 = datetime(2020, 11, 1, 8, tzinfo=timezone.utc)
>>> dt2 = datetime(2020, 11, 1, 8, tzinfo=ZoneInfo("America/New_York"))
>>> dt2 - dt1
datetime.timedelta(seconds=18000)
>>>
Note that the difference will be four for five hours depending on whether daylight saving time is in effect or not.
How do I convert a numpy.datetime64 object to a datetime.datetime (or Timestamp)?
In the following code, I create a datetime, timestamp and datetime64 objects.
import datetime
import numpy as np
import pandas as pd
dt = datetime.datetime(2012, 5, 1)
# A strange way to extract a Timestamp object, there's surely a better way?
ts = pd.DatetimeIndex([dt])[0]
dt64 = np.datetime64(dt)
In [7]: dt
Out[7]: datetime.datetime(2012, 5, 1, 0, 0)
In [8]: ts
Out[8]: <Timestamp: 2012-05-01 00:00:00>
In [9]: dt64
Out[9]: numpy.datetime64('2012-05-01T01:00:00.000000+0100')
Note: it's easy to get the datetime from the Timestamp:
In [10]: ts.to_datetime()
Out[10]: datetime.datetime(2012, 5, 1, 0, 0)
But how do we extract the datetime or Timestamp from a numpy.datetime64 (dt64)?
.
Update: a somewhat nasty example in my dataset (perhaps the motivating example) seems to be:
dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100')
which should be datetime.datetime(2002, 6, 28, 1, 0), and not a long (!) (1025222400000000000L)...
You can just use the pd.Timestamp constructor. The following diagram may be useful for this and related questions.
Welcome to hell.
You can just pass a datetime64 object to pandas.Timestamp:
In [16]: Timestamp(numpy.datetime64('2012-05-01T01:00:00.000000'))
Out[16]: <Timestamp: 2012-05-01 01:00:00>
I noticed that this doesn't work right though in NumPy 1.6.1:
numpy.datetime64('2012-05-01T01:00:00.000000+0100')
Also, pandas.to_datetime can be used (this is off of the dev version, haven't checked v0.9.1):
In [24]: pandas.to_datetime('2012-05-01T01:00:00.000000+0100')
Out[24]: datetime.datetime(2012, 5, 1, 1, 0, tzinfo=tzoffset(None, 3600))
To convert numpy.datetime64 to datetime object that represents time in UTC on numpy-1.8:
>>> from datetime import datetime
>>> import numpy as np
>>> dt = datetime.utcnow()
>>> dt
datetime.datetime(2012, 12, 4, 19, 51, 25, 362455)
>>> dt64 = np.datetime64(dt)
>>> ts = (dt64 - np.datetime64('1970-01-01T00:00:00Z')) / np.timedelta64(1, 's')
>>> ts
1354650685.3624549
>>> datetime.utcfromtimestamp(ts)
datetime.datetime(2012, 12, 4, 19, 51, 25, 362455)
>>> np.__version__
'1.8.0.dev-7b75899'
The above example assumes that a naive datetime object is interpreted by np.datetime64 as time in UTC.
To convert datetime to np.datetime64 and back (numpy-1.6):
>>> np.datetime64(datetime.utcnow()).astype(datetime)
datetime.datetime(2012, 12, 4, 13, 34, 52, 827542)
It works both on a single np.datetime64 object and a numpy array of np.datetime64.
Think of np.datetime64 the same way you would about np.int8, np.int16, etc and apply the same methods to convert between Python objects such as int, datetime and corresponding numpy objects.
Your "nasty example" works correctly:
>>> from datetime import datetime
>>> import numpy
>>> numpy.datetime64('2002-06-28T01:00:00.000000000+0100').astype(datetime)
datetime.datetime(2002, 6, 28, 0, 0)
>>> numpy.__version__
'1.6.2' # current version available via pip install numpy
I can reproduce the long value on numpy-1.8.0 installed as:
pip install git+https://github.com/numpy/numpy.git#egg=numpy-dev
The same example:
>>> from datetime import datetime
>>> import numpy
>>> numpy.datetime64('2002-06-28T01:00:00.000000000+0100').astype(datetime)
1025222400000000000L
>>> numpy.__version__
'1.8.0.dev-7b75899'
It returns long because for numpy.datetime64 type .astype(datetime) is equivalent to .astype(object) that returns Python integer (long) on numpy-1.8.
To get datetime object you could:
>>> dt64.dtype
dtype('<M8[ns]')
>>> ns = 1e-9 # number of seconds in a nanosecond
>>> datetime.utcfromtimestamp(dt64.astype(int) * ns)
datetime.datetime(2002, 6, 28, 0, 0)
To get datetime64 that uses seconds directly:
>>> dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100', 's')
>>> dt64.dtype
dtype('<M8[s]')
>>> datetime.utcfromtimestamp(dt64.astype(int))
datetime.datetime(2002, 6, 28, 0, 0)
The numpy docs say that the datetime API is experimental and may change in future numpy versions.
I think there could be a more consolidated effort in an answer to better explain the relationship between Python's datetime module, numpy's datetime64/timedelta64 and pandas' Timestamp/Timedelta objects.
The datetime standard library of Python
The datetime standard library has four main objects
time - only time, measured in hours, minutes, seconds and microseconds
date - only year, month and day
datetime - All components of time and date
timedelta - An amount of time with maximum unit of days
Create these four objects
>>> import datetime
>>> datetime.time(hour=4, minute=3, second=10, microsecond=7199)
datetime.time(4, 3, 10, 7199)
>>> datetime.date(year=2017, month=10, day=24)
datetime.date(2017, 10, 24)
>>> datetime.datetime(year=2017, month=10, day=24, hour=4, minute=3, second=10, microsecond=7199)
datetime.datetime(2017, 10, 24, 4, 3, 10, 7199)
>>> datetime.timedelta(days=3, minutes = 55)
datetime.timedelta(3, 3300)
>>> # add timedelta to datetime
>>> datetime.timedelta(days=3, minutes = 55) + \
datetime.datetime(year=2017, month=10, day=24, hour=4, minute=3, second=10, microsecond=7199)
datetime.datetime(2017, 10, 27, 4, 58, 10, 7199)
NumPy's datetime64 and timedelta64 objects
NumPy has no separate date and time objects, just a single datetime64 object to represent a single moment in time. The datetime module's datetime object has microsecond precision (one-millionth of a second). NumPy's datetime64 object allows you to set its precision from hours all the way to attoseconds (10 ^ -18). It's constructor is more flexible and can take a variety of inputs.
Construct NumPy's datetime64 and timedelta64 objects
Pass an integer with a string for the units. See all units here. It gets converted to that many units after the UNIX epoch: Jan 1, 1970
>>> np.datetime64(5, 'ns')
numpy.datetime64('1970-01-01T00:00:00.000000005')
>>> np.datetime64(1508887504, 's')
numpy.datetime64('2017-10-24T23:25:04')
You can also use strings as long as they are in ISO 8601 format.
>>> np.datetime64('2017-10-24')
numpy.datetime64('2017-10-24')
Timedeltas have a single unit
>>> np.timedelta64(5, 'D') # 5 days
>>> np.timedelta64(10, 'h') 10 hours
Can also create them by subtracting two datetime64 objects
>>> np.datetime64('2017-10-24T05:30:45.67') - np.datetime64('2017-10-22T12:35:40.123')
numpy.timedelta64(147305547,'ms')
Pandas Timestamp and Timedelta build much more functionality on top of NumPy
A pandas Timestamp is a moment in time very similar to a datetime but with much more functionality. You can construct them with either pd.Timestamp or pd.to_datetime.
>>> pd.Timestamp(1239.1238934) #defaults to nanoseconds
Timestamp('1970-01-01 00:00:00.000001239')
>>> pd.Timestamp(1239.1238934, unit='D') # change units
Timestamp('1973-05-24 02:58:24.355200')
>>> pd.Timestamp('2017-10-24 05') # partial strings work
Timestamp('2017-10-24 05:00:00')
pd.to_datetime works very similarly (with a few more options) and can convert a list of strings into Timestamps.
>>> pd.to_datetime('2017-10-24 05')
Timestamp('2017-10-24 05:00:00')
>>> pd.to_datetime(['2017-1-1', '2017-1-2'])
DatetimeIndex(['2017-01-01', '2017-01-02'], dtype='datetime64[ns]', freq=None)
Converting Python datetime to datetime64 and Timestamp
>>> dt = datetime.datetime(year=2017, month=10, day=24, hour=4,
minute=3, second=10, microsecond=7199)
>>> np.datetime64(dt)
numpy.datetime64('2017-10-24T04:03:10.007199')
>>> pd.Timestamp(dt) # or pd.to_datetime(dt)
Timestamp('2017-10-24 04:03:10.007199')
Converting numpy datetime64 to datetime and Timestamp
>>> dt64 = np.datetime64('2017-10-24 05:34:20.123456')
>>> unix_epoch = np.datetime64(0, 's')
>>> one_second = np.timedelta64(1, 's')
>>> seconds_since_epoch = (dt64 - unix_epoch) / one_second
>>> seconds_since_epoch
1508823260.123456
>>> datetime.datetime.utcfromtimestamp(seconds_since_epoch)
>>> datetime.datetime(2017, 10, 24, 5, 34, 20, 123456)
Convert to Timestamp
>>> pd.Timestamp(dt64)
Timestamp('2017-10-24 05:34:20.123456')
Convert from Timestamp to datetime and datetime64
This is quite easy as pandas timestamps are very powerful
>>> ts = pd.Timestamp('2017-10-24 04:24:33.654321')
>>> ts.to_pydatetime() # Python's datetime
datetime.datetime(2017, 10, 24, 4, 24, 33, 654321)
>>> ts.to_datetime64()
numpy.datetime64('2017-10-24T04:24:33.654321000')
>>> dt64.tolist()
datetime.datetime(2012, 5, 1, 0, 0)
For DatetimeIndex, the tolist returns a list of datetime objects. For a single datetime64 object it returns a single datetime object.
One option is to use str, and then to_datetime (or similar):
In [11]: str(dt64)
Out[11]: '2012-05-01T01:00:00.000000+0100'
In [12]: pd.to_datetime(str(dt64))
Out[12]: datetime.datetime(2012, 5, 1, 1, 0, tzinfo=tzoffset(None, 3600))
Note: it is not equal to dt because it's become "offset-aware":
In [13]: pd.to_datetime(str(dt64)).replace(tzinfo=None)
Out[13]: datetime.datetime(2012, 5, 1, 1, 0)
This seems inelegant.
.
Update: this can deal with the "nasty example":
In [21]: dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100')
In [22]: pd.to_datetime(str(dt64)).replace(tzinfo=None)
Out[22]: datetime.datetime(2002, 6, 28, 1, 0)
If you want to convert an entire pandas series of datetimes to regular python datetimes, you can also use .to_pydatetime().
pd.date_range('20110101','20110102',freq='H').to_pydatetime()
> [datetime.datetime(2011, 1, 1, 0, 0) datetime.datetime(2011, 1, 1, 1, 0)
datetime.datetime(2011, 1, 1, 2, 0) datetime.datetime(2011, 1, 1, 3, 0)
....
It also supports timezones:
pd.date_range('20110101','20110102',freq='H').tz_localize('UTC').tz_convert('Australia/Sydney').to_pydatetime()
[ datetime.datetime(2011, 1, 1, 11, 0, tzinfo=<DstTzInfo 'Australia/Sydney' EST+11:00:00 DST>)
datetime.datetime(2011, 1, 1, 12, 0, tzinfo=<DstTzInfo 'Australia/Sydney' EST+11:00:00 DST>)
....
NOTE: If you are operating on a Pandas Series you cannot call to_pydatetime() on the entire series. You will need to call .to_pydatetime() on each individual datetime64 using a list comprehension or something similar:
datetimes = [val.to_pydatetime() for val in df.problem_datetime_column]
This post has been up for 4 years and I still struggled with this conversion problem - so the issue is still active in 2017 in some sense. I was somewhat shocked that the numpy documentation does not readily offer a simple conversion algorithm but that's another story.
I have come across another way to do the conversion that only involves modules numpy and datetime, it does not require pandas to be imported which seems to me to be a lot of code to import for such a simple conversion. I noticed that datetime64.astype(datetime.datetime) will return a datetime.datetime object if the original datetime64 is in micro-second units while other units return an integer timestamp. I use module xarray for data I/O from Netcdf files which uses the datetime64 in nanosecond units making the conversion fail unless you first convert to micro-second units. Here is the example conversion code,
import numpy as np
import datetime
def convert_datetime64_to_datetime( usert: np.datetime64 )->datetime.datetime:
t = np.datetime64( usert, 'us').astype(datetime.datetime)
return t
Its only tested on my machine, which is Python 3.6 with a recent 2017 Anaconda distribution. I have only looked at scalar conversion and have not checked array based conversions although I'm guessing it will be good. Nor have I looked at the numpy datetime64 source code to see if the operation makes sense or not.
import numpy as np
import pandas as pd
def np64toDate(np64):
return pd.to_datetime(str(np64)).replace(tzinfo=None).to_datetime()
use this function to get pythons native datetime object
I've come back to this answer more times than I can count, so I decided to throw together a quick little class, which converts a Numpy datetime64 value to Python datetime value. I hope it helps others out there.
from datetime import datetime
import pandas as pd
class NumpyConverter(object):
#classmethod
def to_datetime(cls, dt64, tzinfo=None):
"""
Converts a Numpy datetime64 to a Python datetime.
:param dt64: A Numpy datetime64 variable
:type dt64: numpy.datetime64
:param tzinfo: The timezone the date / time value is in
:type tzinfo: pytz.timezone
:return: A Python datetime variable
:rtype: datetime
"""
ts = pd.to_datetime(dt64)
if tzinfo is not None:
return datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second, tzinfo=tzinfo)
return datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second)
I'm gonna keep this in my tool bag, something tells me I'll need it again.
I did like this
import pandas as pd
# Custom function to convert Pandas Datetime to Timestamp
def toTimestamp(data):
return data.timestamp()
# Read a csv file
df = pd.read_csv("friends.csv")
# Replace the "birthdate" column by:
# 1. Transform to datetime
# 2. Apply the custom function to the column just converted
df["birthdate"] = pd.to_datetime(df["birthdate"]).apply(toTimestamp)
Some solutions work well for me but numpy will deprecate some parameters.
The solution that work better for me is to read the date as a pandas datetime and excract explicitly the year, month and day of a pandas object.
The following code works for the most common situation.
def format_dates(dates):
dt = pd.to_datetime(dates)
try: return [datetime.date(x.year, x.month, x.day) for x in dt]
except TypeError: return datetime.date(dt.year, dt.month, dt.day)
Only way I managed to convert a column 'date' in pandas dataframe containing time info to numpy array was as following: (dataframe is read from csv file "csvIn.csv")
import pandas as pd
import numpy as np
df = pd.read_csv("csvIn.csv")
df["date"] = pd.to_datetime(df["date"])
timestamps = np.array([np.datetime64(value) for dummy, value in df["date"].items()])
indeed, all of these datetime types can be difficult, and potentially problematic (must keep careful track of timezone information). here's what i have done, though i admit that i am concerned that at least part of it is "not by design". also, this can be made a bit more compact as needed.
starting with a numpy.datetime64 dt_a:
dt_a
numpy.datetime64('2015-04-24T23:11:26.270000-0700')
dt_a1 = dt_a.tolist() # yields a datetime object in UTC, but without tzinfo
dt_a1
datetime.datetime(2015, 4, 25, 6, 11, 26, 270000)
# now, make your "aware" datetime:
dt_a2=datetime.datetime(*list(dt_a1.timetuple()[:6]) + [dt_a1.microsecond], tzinfo=pytz.timezone('UTC'))
... and of course, that can be compressed into one line as needed.
Is there a nicer way than the following to return today's date in the YYYY-MM-DD format?
str(datetime.datetime.today()).split()[0]
Use strftime:
>>> from datetime import datetime
>>> datetime.today().strftime('%Y-%m-%d')
'2021-01-26'
To also include a zero-padded Hour:Minute:Second at the end:
>>> datetime.today().strftime('%Y-%m-%d %H:%M:%S')
'2021-01-26 16:50:03'
To get the UTC date and time:
>>> datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')
'2021-01-27 00:50:03'
You can use datetime.date.today() and convert the resulting datetime.date object to a string:
from datetime import date
today = str(date.today())
print(today) # '2017-12-26'
I always use the isoformat() method for this.
from datetime import date
today = date.today().isoformat()
print(today) # '2018-12-05'
Note that this also works on datetime objects if you need the time in the standard ISO 8601 format as well.
from datetime import datetime
now = datetime.today().isoformat()
print(now) # '2018-12-05T11:15:55.126382'
Very late answer, but you can simply use:
import time
today = time.strftime("%Y-%m-%d")
# 2023-02-08
Datetime is just lovely if you like remembering funny codes. Wouldn't you prefer simplicity?
>>> import arrow
>>> arrow.now().format('YYYY-MM-DD')
'2017-02-17'
This module is clever enough to understand what you mean.
Just do pip install arrow.
Addendum: In answer to those who become exercised over this answer let me just say that arrow represents one of the alternative approaches to dealing with dates in Python. That's mostly what I meant to suggest.
Are you working with Pandas?
You can use pd.to_datetime from the pandas library. Here are various options, depending on what you want returned.
import pandas as pd
pd.to_datetime('today') # pd.to_datetime('now')
# Timestamp('2019-03-27 00:00:10.958567')
As a python datetime object,
pd.to_datetime('today').to_pydatetime()
# datetime.datetime(2019, 4, 18, 3, 50, 42, 587629)
As a formatted date string,
pd.to_datetime('today').isoformat()
# '2019-04-18T04:03:32.493337'
# Or, `strftime` for custom formats.
pd.to_datetime('today').strftime('%Y-%m-%d')
# '2019-03-27'
To get just the date from the timestamp, call Timestamp.date.
pd.to_datetime('today').date()
# datetime.date(2019, 3, 27)
Aside from to_datetime, you can directly instantiate a Timestamp object using,
pd.Timestamp('today') # pd.Timestamp('now')
# Timestamp('2019-04-18 03:43:33.233093')
pd.Timestamp('today').to_pydatetime()
# datetime.datetime(2019, 4, 18, 3, 53, 46, 220068)
If you want to make your Timestamp timezone aware, pass a timezone to the tz argument.
pd.Timestamp('now', tz='America/Los_Angeles')
# Timestamp('2019-04-18 03:59:02.647819-0700', tz='America/Los_Angeles')
Yet another date parser library: Pendulum
This one's good, I promise.
If you're working with pendulum, there are some interesting choices. You can get the current timestamp using now() or today's date using today().
import pendulum
pendulum.now()
# DateTime(2019, 3, 27, 0, 2, 41, 452264, tzinfo=Timezone('America/Los_Angeles'))
pendulum.today()
# DateTime(2019, 3, 27, 0, 0, 0, tzinfo=Timezone('America/Los_Angeles'))
Additionally, you can also get tomorrow() or yesterday()'s date directly without having to do any additional timedelta arithmetic.
pendulum.yesterday()
# DateTime(2019, 3, 26, 0, 0, 0, tzinfo=Timezone('America/Los_Angeles'))
pendulum.tomorrow()
# DateTime(2019, 3, 28, 0, 0, 0, tzinfo=Timezone('America/Los_Angeles'))
There are various formatting options available.
pendulum.now().to_date_string()
# '2019-03-27'
pendulum.now().to_formatted_date_string()
# 'Mar 27, 2019'
pendulum.now().to_day_datetime_string()
# 'Wed, Mar 27, 2019 12:04 AM'
Rationale for this answer
A lot of pandas users stumble upon this question because they believe it is a python question more than a pandas one. This answer aims to be useful to folks who are already using these libraries and would be interested to know that there are ways to achieve these results within the scope of the library itself.
If you are not working with pandas or pendulum already, I definitely do not recommend installing them just for the sake of running this code! These libraries are heavy and come with a lot of plumbing under the hood. It is not worth the trouble when you can use the standard library instead.
from datetime import datetime
date = datetime.today().date()
print(date)
Use f-strings, they are usually the best choice for any text-variable mix:
from datetime import date
print(f'{date.today():%Y-%m-%d}')
Taken from Python f-string formatting not working with strftime inline which has the official links as well.
If you need e.g. pacific standard time (PST) you can do
from datetime import datetime
import pytz
tz = pytz.timezone('US/Pacific')
datetime.now(tz).strftime('%Y-%m-%d %H:%M:%S')
# '2021-09-02 10:21:41'
my code is a little complicated but I use it a lot
strftime("%y_%m_%d", localtime(time.time()))
reference:'https://strftime.org/
you can look at the reference to make anything you want
for you what YYYY-MM-DD just change my code to:
strftime("%Y-%m-%d", localtime(time.time()))
This works:
from datetime import date
today =date.today()
Output in this time: 2020-08-29
Additional:
this_year = date.today().year
this_month = date.today().month
this_day = date.today().day
print(today)
print(this_year)
print(this_month)
print(this_day)
To get day number from date is in python
for example:19-12-2020(dd-mm-yyy)order_date
we need 19 as output
order['day'] = order['Order_Date'].apply(lambda x: x.day)
The time module can be initialized using seconds since epoch:
>>> import time
>>> t1=time.gmtime(1284286794)
>>> t1
time.struct_time(tm_year=2010, tm_mon=9, tm_mday=12, tm_hour=10, tm_min=19,
tm_sec=54, tm_wday=6, tm_yday=255, tm_isdst=0)
Is there an elegant way to initialize a datetime.datetime object in the same way?
datetime.datetime.fromtimestamp will do, if you know the time zone, you could produce the same output as with time.gmtime
>>> datetime.datetime.fromtimestamp(1284286794)
datetime.datetime(2010, 9, 12, 11, 19, 54)
or
>>> datetime.datetime.utcfromtimestamp(1284286794)
datetime.datetime(2010, 9, 12, 10, 19, 54)
Seconds since epoch to datetime to strftime:
>>> ts_epoch = 1362301382
>>> ts = datetime.datetime.fromtimestamp(ts_epoch).strftime('%Y-%m-%d %H:%M:%S')
>>> ts
'2013-03-03 01:03:02'
From the docs, the recommended way of getting a timezone aware datetime object from seconds since epoch is:
Python 3:
from datetime import datetime, timezone
datetime.fromtimestamp(timestamp, timezone.utc)
Python 2, using pytz:
from datetime import datetime
import pytz
datetime.fromtimestamp(timestamp, pytz.utc)
Note that datetime.datetime.fromtimestamp(timestamp) and .utcfromtimestamp(timestamp) fail on windows for dates before Jan. 1, 1970 while negative unix timestamps seem to work on unix-based platforms. The docs say this:
"This may raise ValueError, if the timestamp is out of the range of
values supported by the platform C gmtime() function. It’s common for
this to be restricted to years in 1970 through 2038"
See also Issue1646728
For those that want it ISO 8601 compliant, since the other solutions do not have the T separator nor the time offset (except Meistro's answer):
from datetime import datetime, timezone
result = datetime.fromtimestamp(1463288494, timezone.utc).isoformat('T', 'microseconds')
print(result) # 2016-05-15T05:01:34.000000+00:00
Note, I use fromtimestamp because if I used utcfromtimestamp I would need to chain on .astimezone(...) anyway to get the offset.
If you don't want to go all the way to microseconds you can choose a different unit with the
isoformat() method.