Develop Reference

Compute midnight timestamp in UTC +0 using DateTime python 3.5

I need to scrap an online database which contain +/- 24h of data at fixed interval using an API query which contain a timestamp. Because i don't know where the server is choose something simple like midnigth UTC.
I found lot of documentation on SO to compute UTC aware of local zone. I'm actually using this protocole to get actual UTC Date :
import datetime
myDate = datetime.datetime.now(datetime.timezone.utc)
print("TZ INFO = ", myDate.tzinfo) # return UTC+00:00
print("DATE ", myDate) # return 2017-07-08 14:14:24.137003+00:00
print("ISO DATE = ", myDate.timestamp()) # return 1499523264.137003
First question, why the timestamp() returned take in account the local timezone : 1499523264.137003 is equal to ~16h15, so UTC +2 corresponding to France Zone. Why timestamp() doesn't return only the UTC + 0 timestamp ? How can i get an UTC + 0 timestamp ?
Second question, i try to generate a midnight date to query the API, so like i saw on many post on SO, i try to use the replace() function :
myDate = myDate.replace(hour=0, minute=0, second=0,microsecond=0).astimezone(pytz.utc)
print (myDate) # return 2017-07-08 00:00:00+00:00
But when i try to print (myDate.timestamp()) return another time a UTC + 2 timestamp, so 2AM of 2017-07-08. How can i get midnight UTC + 0 timestamp easily ?

I would suggest using the pendulum module since it makes timezone and date calculations easy to perform.
pendulum is aware of daylight savings time schemes, as indicated here for London and Paris. It can also provide the UTC time shorn of an adjustment for daylight savings time. When you need to provide an adjustment to UTC you can simply using the replace method in conjunction with UTC.
>>> import pendulum
>>> pendulum.create(2017,7,9,0,0,0,0,'Europe/London')
<Pendulum [2017-07-09T00:00:00+01:00]>
>>> pendulum.create(2017,7,9,0,0,0,0,'Europe/Paris')
<Pendulum [2017-07-09T00:00:00+02:00]>
>>> pendulum.create(2017,7,9,0,0,0,0,'UTC')
<Pendulum [2017-07-09T00:00:00+00:00]>
>>> t = pendulum.create(2017,7,9,0,0,0,0,'UTC')
>>> t.replace(hour=+2)
<Pendulum [2017-07-09T02:00:00+00:00]>

Given a UTC time, get a specified timezone's midnight

Note this is not quite the same as this question. That question assumes the time you want is "now", which is not the same as for an arbitrary point in time.
I have a UTC, aware, datetime object, call it point_in_time (e.g. datetime(2017, 3, 12, 16, tzinfo=tz.tzutc())).
I have a timezone, call it location (e.g. 'US/Pacific'), because I care about where it is, but its hours offset from UTC may change throughout the year with daylight savings and whatnot.
I want to
1) get the date of point_in_time if I'm standing in location,
2) get midnight of that date if I'm standing in location.
===
I tried to simply use .astimezone(timezone('US/Pacific')) and then .replace(hours=0, ...) to move to midnight, but as you might notice about my example point_in_time, the midnight for that date is on the other side of a daylight savings switch!
The result was that I got a time representing UTC datetime(2017, 3, 12, 7), instead of a time representing UTC datetime(2017, 3, 12, 8), which is the true midnight.
EDIT:
I'm actually thinking the difference between mine and the linked question is that I'm looking for the most recent midnight in the past. That question's answer seems to be able to give a midnight that could be in the past or future, perhaps?

Your example highlights the perils of doing datetime arithmetic in a local time zone.
You can probably achieve this using pytz's normalize() function, but here's the method that occurs to me:
point_in_time = datetime(2017, 3, 12, 16, tzinfo=pytz.utc)
pacific = pytz.timezone("US/Pacific")
pacific_time = point_in_time.astimezone(pacific)
pacific_midnight_naive = pacific_time.replace(hour=0, tzinfo=None)
pacific_midnight_aware = pacific.localize(pacific_midnight_naive)
pacific_midnight_aware.astimezone(pytz.utc) # datetime(2017, 3, 12, 8)
In other words, you first convert to Pacific time to figure out the right date; then you convert again from midnight on that date to get the correct local time.

Named timezones such as "US/Pacific" are by definition daylight-savings aware. If you wish to use a fixed non-daylight-savings-aware offset from GMT you can use the timezones "Etc/GMT+*", where * is the desired offset. For example for US Pacific Standard Time you would use "Etc/GMT+8":
import pandas as pd
point_in_time = pd.to_datetime('2017-03-12 16:00:00').tz_localize('UTC')
# not what you want
local_time = point_in_time.tz_convert("US/Pacific")
(local_time - pd.Timedelta(hours=local_time.hour)).tz_convert('UTC')
# Timestamp('2017-03-12 07:00:00+0000', tz='UTC')
# what you want
local_time = point_in_time.tz_convert("Etc/GMT+8")
(local_time - pd.Timedelta(hours=local_time.hour)).tz_convert('UTC')
# Timestamp('2017-03-12 08:00:00+0000', tz='UTC')
See the docs at http://pvlib-python.readthedocs.io/en/latest/timetimezones.html for more info.
EDIT Now that I think about it, Midnight PST will always be 8am UTC, so you could simplify this as
if point_in_time.hour >=8:
local_midnight = point_in_time - point_in_time.hour + 8
else:
local_midnight = point_in_time - point_in_time.hour - 16

Convert localtime to unix timestamp [duplicate]

I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
I am trying to convert them into epoch timestamps for UTC time.
I wrote the following function:
def ymdhms_timezone_dst_to_epoch(input_str, tz="US/Eastern"):
print(input_str)
dt = datetime.datetime.fromtimestamp(time.mktime(time.strptime(input_str,'%Y-%m-%d %H:%M:%S')))
local_dt = pytz.timezone(tz).localize(dt)
print(local_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
utc_dt = local_dt.astimezone(pytz.utc)
print(utc_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
e = int(utc_dt.strftime("%s"))
print(e)
return e
Given string `2015-04-20 21:12:07` this prints:
2015-04-20 21:12:07
2015-04-20 21:12:07 EDT-0400 #<- so far so good?
2015-04-21 01:12:07 UTC+0000 #<- so far so good?
1429596727
which looks ok up to the epoch timestamp. But http://www.epochconverter.com/epoch/timezones.php?epoch=1429596727 says it should mao to
Greenwich Mean Time Apr 21 2015 06:12:07 UTC.
What is wrong?

I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
A portable way is to use pytz:
#!/usr/bin/env python
from datetime import datetime
import pytz # $ pip install pytz
naive_dt = datetime.strptime('2015-04-20 21:12:07', '%Y-%m-%d %H:%M:%S')
tz = pytz.timezone('US/Eastern')
eastern_dt = tz.normalize(tz.localize(naive_dt))
print(eastern_dt)
# -> 2015-04-20 21:12:07-04:00
I am trying to convert them into epoch timestamps for UTC time.
timestamp = (eastern_dt - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
# -> 1429578727.0
See Converting datetime.date to UTC timestamp in Python.
There are multiple issues in your code:
time.mktime() may return a wrong result for ambiguous input time (50% chance) e.g., during "fall back" DST transition in the Fall
time.mktime() and datetime.fromtimestamp() may fail for past/future dates if they have no access to a historical timezone database on a system (notably, Windows)
localize(dt) may return a wrong result for ambiguous or non-existent time i.e., during DST transitions. If you know that the time corresponds to the summer time then use is_dst=True. tz.normalize() is necessary here, to adjust possible non-existing times in the input
utc_dt.strftime("%s") is not portable and it does not respect tzinfo object. It interprets input as a local time i.e., it returns a wrong result unless your local timezone is UTC.
Can I just always set is_dst=True?
You can, if you don't mind getting imprecise results for ambiguous or non-existent times e.g., there is DST transition in the Fall in America/New_York time zone:
>>> from datetime import datetime
>>> import pytz # $ pip install pytz
>>> tz = pytz.timezone('America/New_York')
>>> ambiguous_time = datetime(2015, 11, 1, 1, 30)
>>> time_fmt = '%Y-%m-%d %H:%M:%S%z (%Z)'
>>> tz.localize(ambiguous_time).strftime(time_fmt)
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=False).strftime(time_fmt) # same
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=True).strftime(time_fmt) # different
'2015-11-01 01:30:00-0400 (EDT)'
>>> tz.localize(ambiguous_time, is_dst=None).strftime(time_fmt)
Traceback (most recent call last):
...
pytz.exceptions.AmbiguousTimeError: 2015-11-01 01:30:00
The clocks are turned back at 2a.m. on the first Sunday in November:
is_dst disambiguation flag may have three values:
False -- default, assume the winter time
True -- assume the summer time
None -- raise an exception for ambiguous/non-existent times.
is_dst value is ignored for existing unique local times.
Here's a plot from PEP 0495 -- Local Time Disambiguation that illustrates the DST transition:
The local time repeats itself twice in the fold (summer time -- before the fold, winter time -- after).
To be able to disambiguate the local time automatically, you need some additional info e.g., if you read a series of local times then it may help if you know that they are sorted: Parsing of Ordered Timestamps in Local Time (to UTC) While Observing Daylight Saving Time.

First of all '%s' is not supported on all platforms , its actually working for you because your platform C library’s strftime() function (that is called by Python) supports it. This function is what is causing the issue most probably, I am guessing its not timezone aware , hence when taking difference from epoch time it is using your local timezone, which is most probably EST(?)
Instead of relying on '%s' , which only works in few platforms (linux, I believe) , you should manually subtract the datetime you got from epoch (1970/1/1 00:00:00) to get the actual seconds since epoch . Example -
e = (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
Demo -
>>> (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
1429578727.0
This correctly corresponds to the date-time you get.

I don't exactly know why but you have to remove the timezone info from your utc_dt before using %s to print it.
e = int(utc_dt.replace(tzinfo=None).strftime("%s"))
print(e)
return e

Python datetime.datetime from time.structtime difference

I use feedparser to grab the entries from some RSS feeds.
The entries have a published_parsed field which are parsed by feedparser into time.structtime.
I use this function to convert the time.structtime into a datetime.datetime:
def publishParsedToDatetime(structTime):
return datetime.datetime.fromtimestamp(time.mktime(structTime))
Input (structtime):
time.struct_time(tm_year=2015, tm_mon=8, tm_mday=1, tm_hour=20, tm_min=28, tm_sec=33, tm_wday=5, tm_yday=213, tm_isdst=0)
Output (datetime):
2015-08-01 21:28:33
I see a problem which could be timezone related, there is 1 hour difference between the structtime and the datetime values.
The structtime value is UTC.
But the datetime.datetime value is neither UTC, nor my current timezone (CET, Central European Time, we observe Summertime, so we have UTC + 2hrs at the moment).
How can this be explained?

Actually, as explained in the documentation for datetime.fromtimestamp, it converts to local time by default:
Return the local date and time corresponding to the POSIX timestamp, such as is returned by time.time(). If optional argument tz is None or not specified, the timestamp is converted to the platform’s local date and time, and the returned datetime object is naive
The 1 hour difference can then be explained by the field tm_isdst=0 tells it to not use daylight savings (despite your local time zone using it).
To see this more clearly, we construct two test cases
import time, datetime
# this is how your time object was constructed before
tm_isdst = 0
t = time.mktime((2015, 8, 1, 20, 28, 33, 5, 213, tm_isdst))
print("Old conversion: {0}".format(datetime.datetime.fromtimestamp(t)))
# this is what happens if you let mktime "divine" a time zone
tm_isdst = -1
t = time.mktime((2015, 8, 1, 20, 28, 33, 5, 213, tm_isdst))
print("New conversion: {0}".format(datetime.datetime.fromtimestamp(t)))
The output of this is as follows:
Old conversion: 2015-08-01 21:28:33
New conversion: 2015-08-01 20:28:33
The problem then, you see, is that the structTime object being passed to your publishParsedToDatetime has tm_isdst=0 but the time stamp you wanted to parse was for a DST time zone.
As you have already noted in another comment, the proper solution to this is probably to always use UTC in your back-end code, and only do time zone handling when showing the time to the user, or when reading user input.

calendar.timegm takes a UTC timetuple as input and returns its timestamp.
In contrast, time.mktime takes a local timetuple as input and returns its (UTC) timestamp. All timestamps represent seconds since the Epoch, 1970-1-1 00:00:00 UTC.
utcfromtimestamp takes a timestamp as input and converts it to a naive
(i.e. timezone-unaware) UTC datetime.
fromtimestamp takes the same timestamp and converts it to the corresponding
naive local datetime.
Since your timetuples (e.g. structTime) are UTC timetuples, you should use calendar.timegm, not time.mktime, to find the correct timestamp.
Once you have the correct timestamp, fromtimestamp will return the corresponding naive local datetime.
import time
import calendar
import datetime as DT
timetuple = (2015, 8, 1, 20, 28, 33, 5, 213, 0)
timestamp = calendar.timegm(timetuple)
naive_local_date = DT.datetime.fromtimestamp(timestamp)
print('Naive local: {}'.format(naive_local_date))
yields
Naive local: 2015-08-01 22:28:33

Convert to UTC Timestamp

# parses some string into that format.
datetime1 = datetime.strptime(somestring, "%Y-%m-%dT%H:%M:%S")
# gets the seconds from the above date.
timestamp1 = time.mktime(datetime1.timetuple())
# adds milliseconds to the above seconds.
timeInMillis = int(timestamp1) * 1000
How do I (at any point in that code) turn the date into UTC format? I've been ploughing through the API for what seems like a century and cannot find anything that I can get working. Can anyone help? It's currently turning it into Eastern time i believe (however I'm in GMT but want UTC).
EDIT: I gave the answer to the guy with the closest to what I finally found out.
datetime1 = datetime.strptime(somestring, someformat)
timeInSeconds = calendar.timegm(datetime1.utctimetuple())
timeInMillis = timeInSeconds * 1000
:)

datetime.utcfromtimestamp is probably what you're looking for:
>>> timestamp1 = time.mktime(datetime.now().timetuple())
>>> timestamp1
1256049553.0
>>> datetime.utcfromtimestamp(timestamp1)
datetime.datetime(2009, 10, 20, 14, 39, 13)

I think you can use the utcoffset() method:
utc_time = datetime1 - datetime1.utcoffset()
The docs give an example of this using the astimezone() method here.
Additionally, if you're going to be dealing with timezones, you might want to look into the PyTZ library which has lots of helpful tools for converting datetime's into various timezones (including between EST and UTC)
With PyTZ:
from datetime import datetime
import pytz
utc = pytz.utc
eastern = pytz.timezone('US/Eastern')
# Using datetime1 from the question
datetime1 = datetime.strptime(somestring, "%Y-%m-%dT%H:%M:%S")
# First, tell Python what timezone that string was in (you said Eastern)
eastern_time = eastern.localize(datetime1)
# Then convert it from Eastern to UTC
utc_time = eastern_time.astimezone(utc)

def getDateAndTime(seconds=None):
"""
Converts seconds since the Epoch to a time tuple expressing UTC.
When 'seconds' is not passed in, convert the current time instead.
:Parameters:
- `seconds`: time in seconds from the epoch.
:Return:
Time in UTC format.
"""
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(seconds))`
This converts local time to UTC
time.mktime(time.localtime(calendar.timegm(utc_time)))
http://feihonghsu.blogspot.com/2008/02/converting-from-local-time-to-utc.html
If converting a struct_time to seconds-since-the-epoch is done using mktime, this
conversion is in local timezone. There's no way to tell it to use any specific timezone, not even just UTC. The standard 'time' package always assumes that a time is in your local timezone.

You probably want one of these two:
import time
import datetime
from email.Utils import formatdate
rightnow = time.time()
utc = datetime.datetime.utcfromtimestamp(rightnow)
print utc
print formatdate(rightnow)
The two outputs look like this
2009-10-20 14:46:52.725000
Tue, 20 Oct 2009 14:46:52 -0000