Develop Reference

python datetime .timestamp() doesnt give accurate timestamp. It gets behind by 7 minutes when I convert it back

I am using python 3.6
I converted midnight of "US/Pacific" timezone to unix timestamp.
When I convert the timestamp back to "US/Pacific" local time, I see its 7 minutes before the midnight.
Can anyone shade light on what is happening?
>>> (
datetime(2021, 5, 5, 0, 0)
.replace(tzinfo=pytz.timezone("US/Pacific"))
.timestamp()
)
>>> 1620201180.0
>>> datetime.utcfromtimestamp(1620201180.0)
>>> datetime.datetime(2021, 5, 5, 7, 53)
>>> datetime.fromtimestamp(1620201180.0, tz=pytz.timezone("US/Pacific"))
>>> datetime.datetime(2021, 5, 5, 0, 53, tzinfo=<DstTzInfo 'US/Pacific' PDT-1 day, 17:00:00 DST>)
FYI I am located at "Asia/Kathmandu" timezone.

datetime.now(tz) and datetime(year, month, day, tzinfo=tz) don't have same UTC offset

Why are the UTC offsets in the 2 datetimes created in the code below different? I'm using pytz==2019.1
from datetime import datetime
import pytz
EASTERN = pytz.timezone('US/Eastern')
dt1 = datetime.now(EASTERN)
dt2 = datetime(2020, 8, 11, 15, 30, tzinfo=EASTERN)
print(f'dt1: {dt1}')
print(f'dt2: {dt2}')
results:
dt1: 2020-08-11 18:35:47.984745-04:00
dt2: 2020-08-11 15:30:00-04:56
The first one shows an UTC offset of -04:00 which is correct (for this time of year), but the 2nd one is giving an UTC offset of -04:56.
How can I declare a datetime with the 2nd method and get the UTC offset from the first method.

The issue with the shown code is that datetime.now(tz) can handle a pytz.timezone as tz argument while the default constructor datetime(y,m,d,...) cannot. In this case, you have to use the localize method of the timezone,
from datetime import datetime
import pytz
EASTERN = pytz.timezone('US/Eastern')
dt1 = datetime.now(EASTERN)
dt2 = EASTERN.localize(datetime(2020, 8, 11, 15, 30))
print(f'dt1: {dt1}')
print(f'dt2: {dt2}')
# prints
# dt1: 2020-08-12 10:07:09.738763-04:00
# dt2: 2020-08-11 15:30:00-04:00
dateutil avoids this issue, more info can be found here. That would make the code work "as it is":
from dateutil.tz import gettz
EASTERN = gettz('US/Eastern')
dt1 = datetime.now(EASTERN)
dt2 = datetime(2020, 8, 11, 15, 30, tzinfo=EASTERN)
print(f'dt1: {dt1}')
print(f'dt2: {dt2}')
# prints e.g.
# dt1: 2020-08-12 10:20:07.796811-04:00
# dt2: 2020-08-11 15:30:00-04:00
In addition pytz is likely to be deprecated with Python 3.9 as you'll have zoneinfo as part of the standard library for these kinds of jobs.

with regards to dt1:
datetime.now() will return the tzinfo from utc. You can see this if you do the following:
>>> repr(EASTERN.fromutc(datetime.now()))
"datetime.datetime(2020, 8, 11, 12, 33, 28, 849873, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>)"
the tz.fromutc() is what datetime.now() does behind the scenes.
with regards to dt2:
if you check the repr of EASTERN you will see the following:
>>> repr(EASTERN)
"<DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>"
>>> repr(dt2)
"datetime.datetime(2020, 8, 11, 15, 30, tzinfo=<DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>)"
I'm not very comfortable with timezone differences so I don't want to steer you in the wrong direction, but from what I can tell both of these two options yield the same result:
dt2 = datetime(2020, 8, 11, 15, 30, tzinfo=EASTERN)
dt2_eastern_corrected = EASTERN.fromutc(dt2)
dt3 = datetime(2020, 8, 11, 15, 30)
dt3_eastern_corrected = EASTERN.fromutc(dt3
>>> repr(dt2_eastern_corrected)
"datetime.datetime(2020, 8, 11, 11, 30, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>)"
>>> repr(dt3_eastern_corrected)
"datetime.datetime(2020, 8, 11, 11, 30, tzinfo=<DstTzInfo 'US/Eastern' EDT-1 day, 20:00:00 DST>)"

Saving a local datetime offset the time by 4 minutes

I'm trying to modify a datetime based on a timezone on save and on load the following way:
An input datetime, along with a input timezone are sent to the server and the server should update the datetime to reflect the timezone. So when it saves in the database (PostregSQL), the UTC time is saved (after the offset caused by the timezone, of course).
To reflect this here's a simpler example that fails in the same way:
Some imports:
>>> import datetime
>>> import pytz
>>> from apps.myapp.models import Project
Creating the two inputs:
>>> input_date = timezone.now()
>>> input_date
datetime.datetime(2017, 2, 7, 16, 7, 14, 377429, tzinfo=<UTC>)
>>> current_tz = pytz.timezone('America/New_York')
>>> current_tz
<DstTzInfo 'America/New_York' LMT-1 day, 19:04:00 STD>
As you can see, the timezone is not 5h (24 - 19 = 5), but 4h56. At this stage I'm thinking that's OK, it may be related to the Daylight Saving Time.
Now I'm replacing the timezone on the input date:
>>> input_date = input_date.replace(tzinfo=current_tz)
>>> input_date
datetime.datetime(2017, 2, 7, 16, 7, 14, 377429, tzinfo=<DstTzInfo 'America/New_York' LMT-1 day, 19:04:00 STD>)
As expected, the time hasn't changed, but the timezone has, which is fine.
I'll assign this value to a project (the launch_date is a DateTimeField without any specific option):
>>> project = Project.objects.get(pk=1)
>>> project.launch_date
datetime.datetime(2017, 1, 14, 8, 53, 57, 241718, tzinfo=<UTC>)
>>> project.launch_date = input_date
>>> project.launch_date
datetime.datetime(2017, 2, 7, 16, 7, 14, 377429, tzinfo=<DstTzInfo 'America/New_York' LMT-1 day, 19:04:00 STD>)
Now I'll save this into (and refresh from) the database, leaving Django/PostgreSQL do the maths:
>>> project.save()
>>> project.refresh_from_db()
>>> project.launch_date
datetime.datetime(2017, 2, 7, 21, 3, 14, 377429, tzinfo=<UTC>)
As expected the date is now 4h56 ahead of the previous date. I'm trying now to get back the local time:
>>> project.launch_date.astimezone(current_tz)
datetime.datetime(2017, 2, 7, 16, 3, 14, 377429, tzinfo=<DstTzInfo 'America/New_York' EST-1 day, 19:00:00 STD>)
>>> input_date
datetime.datetime(2017, 2, 7, 16, 7, 14, 377429, tzinfo=<DstTzInfo 'America/New_York' LMT-1 day, 19:04:00 STD>)
This time, the offset is perfectly 5h. And I'm missing 4 minutes.
3 questions here:
Where is this 4 minutes coming from?
Why is astimezone not using the 4 minutes as well?
How can a datetime be converted to UTC, saved, loaded and converted back to local?

pytz time zones are a little weird, as you can see by multiple questions on StackOverflow. They often won't show the correct offset or timezone name unless they are allowed to adjust themselves to the datetime they are being paired with. Here is what the documentation has to say:
This library only supports two ways of building a localized time. The first is to use the localize() method provided by the pytz library. This is used to localize a naive datetime (datetime with no timezone information):
The second way of building a localized time is by converting an existing localized time using the standard astimezone() method:
Unfortunately using the tzinfo argument of the standard datetime constructors “does not work” with pytz for many timezones.
It does not say so explicitly, but using replace has the same problem as using the datetime constructor.
To accomplish what your code was doing without the 4-minute discrepancy, you can use localize():
>>> input_date
datetime.datetime(2017, 2, 7, 16, 7, 14, 377429, tzinfo=<UTC>)
>>> current_tz.localize(input_date.replace(tzinfo=None))
datetime.datetime(2017, 2, 7, 16, 7, 14, 377429, tzinfo=<DstTzInfo 'America/New_York' EST-1 day, 19:00:00 STD>)
I suspect that's a bug though, and you really want to do a timezone conversion from UTC:
>>> input_date.astimezone(current_tz)
datetime.datetime(2017, 2, 7, 11, 7, 14, 377429, tzinfo=<DstTzInfo 'America/New_York' EST-1 day, 19:00:00 STD>)

Get UNIX timestamp (ie UTC timestamp) from a Python timezone aware date

My computer is running in Pacific time (hence datetime.datetime.fromtimestamp(0) gives me 1969-12-31 16:00:00). My problem is that given a timezone aware datetime object in Python, I want to get the UNIX timestamp (ie the UTC timestamp). What is the best way to do so?

import calendar
import datetime
import pytz
d = datetime.datetime.now(pytz.timezone('America/Los_Angeles'))
# d == datetime.datetime(2012, 7, 10, 1, 6, 36, 37542, tzinfo=<DstTzInfo 'America/Los_Angeles' PDT-1 day, 17:00:00 DST>)
ts = calendar.timegm(d.utctimetuple())
# ts == 1341907596
# test with UTC epoch:
d = datetime.datetime(1969, 12, 31, 16, 0, 0, 0, pytz.timezone('America/Los_Angeles'))
# d == datetime.datetime(1969, 12, 31, 16, 0, tzinfo=<DstTzInfo 'America/Los_Angeles' PST-1 day, 16:00:00 STD>)
ts = calendar.timegm(d.utctimetuple())
# ts == 0

inconsistency when switching between timezones in python

I have a datetime object created like this:
tm = datetime.datetime.strptime('2010 Aug 04 14:15:16', '%Y %b %d %H:%M:%S')
>>> tm
datetime.datetime(2010, 8, 4, 14, 15, 16)
I then set the timezone like this:
tm.replace(tzinfo=pytz.timezone('UTC'))
>>> tm
datetime.datetime(2010, 8, 4, 14, 15, 16, tzinfo=<UTC>)
Eventually, I change the timezone to US/Pacific like this:
>>> tm = tm.astimezone(pytz.timezone('US/Pacific'))
>>> tm
datetime.datetime(2010, 8, 4, 7, 15, 16, tzinfo=<DstTzInfo 'US/Pacific' PDT-1 day, 17:00:00 DST>)
This is correct. But if I were to go into the other direction:
>>> tm = tm.replace(tzinfo =pytz.timezone('US/Pacific'))
>>> tm
datetime.datetime(2010, 8, 4, 14, 15, 16, tzinfo=<DstTzInfo 'US/Pacific' PST-1 day, 16:00:00 STD>)
As you can see, using replace() with the PST timezone incorrectly assumes it is not daylight savings time. Therefore, when I convert to UTC, it adds 8 hours instead of 7.
Why is it doing this and how do I change it so it does what I want (correctly realize DST or not)?

Yes, this is why you use .astimezone and not .replace when you have a datetime with a timezone. Using .astimezone gives the timezone a chance to adjust for things like daylight savings. Only use .replace to give a naïve datetime a tzinfo object.