I need to scrap an online database which contain +/- 24h of data at fixed interval using an API query which contain a timestamp. Because i don't know where the server is choose something simple like midnigth UTC.
I found lot of documentation on SO to compute UTC aware of local zone. I'm actually using this protocole to get actual UTC Date :
import datetime
myDate = datetime.datetime.now(datetime.timezone.utc)
print("TZ INFO = ", myDate.tzinfo) # return UTC+00:00
print("DATE ", myDate) # return 2017-07-08 14:14:24.137003+00:00
print("ISO DATE = ", myDate.timestamp()) # return 1499523264.137003
First question, why the timestamp() returned take in account the local timezone : 1499523264.137003 is equal to ~16h15, so UTC +2 corresponding to France Zone. Why timestamp() doesn't return only the UTC + 0 timestamp ? How can i get an UTC + 0 timestamp ?
Second question, i try to generate a midnight date to query the API, so like i saw on many post on SO, i try to use the replace() function :
myDate = myDate.replace(hour=0, minute=0, second=0,microsecond=0).astimezone(pytz.utc)
print (myDate) # return 2017-07-08 00:00:00+00:00
But when i try to print (myDate.timestamp()) return another time a UTC + 2 timestamp, so 2AM of 2017-07-08. How can i get midnight UTC + 0 timestamp easily ?
I would suggest using the pendulum module since it makes timezone and date calculations easy to perform.
pendulum is aware of daylight savings time schemes, as indicated here for London and Paris. It can also provide the UTC time shorn of an adjustment for daylight savings time. When you need to provide an adjustment to UTC you can simply using the replace method in conjunction with UTC.
>>> import pendulum
>>> pendulum.create(2017,7,9,0,0,0,0,'Europe/London')
<Pendulum [2017-07-09T00:00:00+01:00]>
>>> pendulum.create(2017,7,9,0,0,0,0,'Europe/Paris')
<Pendulum [2017-07-09T00:00:00+02:00]>
>>> pendulum.create(2017,7,9,0,0,0,0,'UTC')
<Pendulum [2017-07-09T00:00:00+00:00]>
>>> t = pendulum.create(2017,7,9,0,0,0,0,'UTC')
>>> t.replace(hour=+2)
<Pendulum [2017-07-09T02:00:00+00:00]>
Related
How can I compute the time differential between two time zones in Python? That is, I don't want to compare TZ-aware datetime objects and get a timedelta; I want to compare two TimeZone objects and get an offset_hours. Nothing in the datetime library handles this, and neither does pytz.
Here is a solution using the Python library Pytz which solves the issue of ambiguous times at the end of daylight saving time.
from pytz import timezone
import pandas as pd
def tz_diff(date, tz1, tz2):
'''
Returns the difference in hours between timezone1 and timezone2
for a given date.
'''
date = pd.to_datetime(date)
return (tz1.localize(date) -
tz2.localize(date).astimezone(tz1))\
.seconds/3600
The examples below calculate the difference in hours between UTC and Australia time for the first of January and first of June respectively. Notice how daylight savings are taken into consideration.
utc = timezone('UTC')
aus = timezone('Australia/Sydney')
tz_diff('2017-01-01', utc, aus)
# 11.0
tz_diff('2017-06-01', utc, aus)
# 10.0
Thanks
The first thing you have to know is that the offset between two time zones depends not only on the time zones in question, but on the date you're asking about. For example, the dates on which Daylight Savings Time began and ended changed in the US in 2007. While fundamental time zone logistics change only infrequently in any single location, the rate of change globally is impossible to ignore. Therefore, you have to incorporate the date in question into your function.
Having completed the necessary preface, the actual function isn't too hard to write if you take advantage of the pendulum library. It should look something like this:
import pendulum
def tz_diff(home, away, on=None):
"""
Return the difference in hours between the away time zone and home.
`home` and `away` may be any values which pendulum parses as timezones.
However, recommended use is to specify the full formal name.
See https://gist.github.com/pamelafox/986163
As not all time zones are separated by an integer number of hours, this
function returns a float.
As time zones are political entities, their definitions can change over time.
This is complicated by the fact that daylight savings time does not start
and end on the same days uniformly across the globe. This means that there are
certain days of the year when the returned value between `Europe/Berlin` and
`America/New_York` is _not_ `6.0`.
By default, this function always assumes that you want the current
definition. If you prefer to specify, set `on` to the date of your choice.
It should be a `Pendulum` object.
This function returns the number of hours which must be added to the home time
in order to get the away time. For example,
```python
>>> tz_diff('Europe/Berlin', 'America/New_York')
-6.0
>>> tz_diff('Europe/Berlin', 'Asia/Kabul')
2.5
```
"""
if on is None:
on = pendulum.today()
diff = (on.set(tz=home) - on.set(tz=away)).total_hours()
# what about the diff from Tokyo to Honolulu? Right now the result is -19.0
# it should be 5.0; Honolulu is naturally east of Tokyo, just not so around
# the date line
if abs(diff) > 12.0:
if diff < 0.0:
diff += 24.0
else:
diff -= 24.0
return diff
As stated in the documentation, you may not get a stable result for this between any two given locations as you sweep across the days of the year. However, implementing a variant which chooses the median result over the days of the current year is an exercise left for the reader.
Here's another solution:
from datetime import datetime
from pytz import timezone
from dateutil.relativedelta import relativedelta
utcnow = timezone('utc').localize(datetime.utcnow()) # generic time
here = utcnow.astimezone(timezone('US/Eastern')).replace(tzinfo=None)
there = utcnow.astimezone(timezone('Asia/Ho_Chi_Minh')).replace(tzinfo=None)
offset = relativedelta(here, there)
offset.hours
Here what we're doing is converting a time to two different time zones. Then, we remove the time zone information so that when you calculate the difference between the two using relativedelta, we trick it into thinking that these are two different moments in time instead of the same moment in different time zones.
The above result will return -11, however this amount can change throughout the year since US/Eastern observes DST and Asia/Ho_Chi_Minh does not.
I created two functions to deal with timezone.
import datetime
import pytz
def diff_hours_tz(from_tz_name, to_tz_name, negative=False):
"""
Returns difference hours between timezones
res = diff_hours_tz("UTC", "Europe/Paris") : 2
"""
from_tz = pytz.timezone(from_tz_name)
to_tz = pytz.timezone(to_tz_name)
utc_dt = datetime.datetime.now(datetime.timezone.utc)
dt_from = dt_to = datetime.datetime.utcnow()
dt_from = from_tz.localize(dt_from)
dt_to = to_tz.localize(dt_to)
from_d = dt_from - utc_dt
if from_d.days < 0:
return diff_hours_tz(to_tz_name, from_tz_name, True)
dt_delta = dt_from - dt_to
negative_int = -1 if negative else 1
return int(dt_delta.seconds/3600)*negative_int
def dt_tz_to_tz(dt, from_tz_name, to_tz_name):
"""
Apply difference hours between timezones to a datetime object
dt_new = dt_tz_to_tz(datetime.datetime.now(), "UTC", "Europe/Paris")
"""
hours = diff_hours_tz(from_tz_name, to_tz_name)
return dt+datetime.timedelta(hours=hours)
# Usage example
res = diff_hours_tz("Europe/Paris", "America/New_York")
# Result : -6
res = diff_hours_tz("UTC", "Europe/Paris")
# Result : 2
now = datetime.datetime.now()
# Result : 2019-06-18 15:10:31.720105
dt_new = dt_tz_to_tz(now, "UTC", "Europe/Paris")
# Result : 2019-06-18 17:10:31.720105
dt_new = dt_tz_to_tz(now, "Europe/Paris", "America/New_York")
# Result : 2019-06-18 09:10:31.720105
dt_new = dt_tz_to_tz(now, "America/New_York", "Europe/Paris")
# Result : 2019-06-18 21:10:31.720105
I hope it will help !
Here is a code snippet to get the difference between UTC and US/Eastern, but it should work for any two timezones.
# The following algorithm will work no matter what is the local timezone of the server,
# but for the purposes of this discussion, let's assume that the local timezone is UTC.
local_timestamp = datetime.now()
# Assume that utc_timestamp == 2019-01-01 12:00.
utc_timestamp = pytz.utc.localize(local_timestamp)
# If it was 12:00 in New York, it would be 20:00 in UTC. So us_eastern_timestamp is a UTC
# timestamp with the value of 2019-01-01 20:00.
us_eastern_timestamp = timezone("US/Eastern").localize(local_timestamp).astimezone(pytz.utc)
# delta is a Python timedelta object representing the interval between the two timestamps,
# which, in our example, is -8 hours.
delta = utc_timestamp - us_eastern_timestamp
# In the last line, we convert the timedelta into an integer representing the number of
# hours.
print round(delta.total_seconds() / 60.0 / 60.0)
(tz_from.localize(date) - tz_to.localize(date)).seconds/3600.0
Where tz_from and tz_to are the starting and ending timezones. You must specify a particular date.
from datetime import datetime
from zoneinfo import ZoneInfo
dt = datetime.now() # 2020-09-13
tz0, tz1 = "Europe/Berlin", "US/Eastern" # +2 vs. -4 hours rel. to UTC
utcoff0, utcoff1 = dt.astimezone(ZoneInfo(tz0)).utcoffset(), dt.astimezone(ZoneInfo(tz1)).utcoffset()
print(f"hours offset between {tz0} -> {tz1} timezones: {(utcoff1-utcoff0).total_seconds()/3600}")
>>> hours offset between Europe/Berlin -> US/Eastern timezones: -6.0
a way to do this with Python 3.9's standard library.
I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
I am trying to convert them into epoch timestamps for UTC time.
I wrote the following function:
def ymdhms_timezone_dst_to_epoch(input_str, tz="US/Eastern"):
print(input_str)
dt = datetime.datetime.fromtimestamp(time.mktime(time.strptime(input_str,'%Y-%m-%d %H:%M:%S')))
local_dt = pytz.timezone(tz).localize(dt)
print(local_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
utc_dt = local_dt.astimezone(pytz.utc)
print(utc_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
e = int(utc_dt.strftime("%s"))
print(e)
return e
Given string `2015-04-20 21:12:07` this prints:
2015-04-20 21:12:07
2015-04-20 21:12:07 EDT-0400 #<- so far so good?
2015-04-21 01:12:07 UTC+0000 #<- so far so good?
1429596727
which looks ok up to the epoch timestamp. But http://www.epochconverter.com/epoch/timezones.php?epoch=1429596727 says it should mao to
Greenwich Mean Time Apr 21 2015 06:12:07 UTC.
What is wrong?
I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
A portable way is to use pytz:
#!/usr/bin/env python
from datetime import datetime
import pytz # $ pip install pytz
naive_dt = datetime.strptime('2015-04-20 21:12:07', '%Y-%m-%d %H:%M:%S')
tz = pytz.timezone('US/Eastern')
eastern_dt = tz.normalize(tz.localize(naive_dt))
print(eastern_dt)
# -> 2015-04-20 21:12:07-04:00
I am trying to convert them into epoch timestamps for UTC time.
timestamp = (eastern_dt - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
# -> 1429578727.0
See Converting datetime.date to UTC timestamp in Python.
There are multiple issues in your code:
time.mktime() may return a wrong result for ambiguous input time (50% chance) e.g., during "fall back" DST transition in the Fall
time.mktime() and datetime.fromtimestamp() may fail for past/future dates if they have no access to a historical timezone database on a system (notably, Windows)
localize(dt) may return a wrong result for ambiguous or non-existent time i.e., during DST transitions. If you know that the time corresponds to the summer time then use is_dst=True. tz.normalize() is necessary here, to adjust possible non-existing times in the input
utc_dt.strftime("%s") is not portable and it does not respect tzinfo object. It interprets input as a local time i.e., it returns a wrong result unless your local timezone is UTC.
Can I just always set is_dst=True?
You can, if you don't mind getting imprecise results for ambiguous or non-existent times e.g., there is DST transition in the Fall in America/New_York time zone:
>>> from datetime import datetime
>>> import pytz # $ pip install pytz
>>> tz = pytz.timezone('America/New_York')
>>> ambiguous_time = datetime(2015, 11, 1, 1, 30)
>>> time_fmt = '%Y-%m-%d %H:%M:%S%z (%Z)'
>>> tz.localize(ambiguous_time).strftime(time_fmt)
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=False).strftime(time_fmt) # same
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=True).strftime(time_fmt) # different
'2015-11-01 01:30:00-0400 (EDT)'
>>> tz.localize(ambiguous_time, is_dst=None).strftime(time_fmt)
Traceback (most recent call last):
...
pytz.exceptions.AmbiguousTimeError: 2015-11-01 01:30:00
The clocks are turned back at 2a.m. on the first Sunday in November:
is_dst disambiguation flag may have three values:
False -- default, assume the winter time
True -- assume the summer time
None -- raise an exception for ambiguous/non-existent times.
is_dst value is ignored for existing unique local times.
Here's a plot from PEP 0495 -- Local Time Disambiguation that illustrates the DST transition:
The local time repeats itself twice in the fold (summer time -- before the fold, winter time -- after).
To be able to disambiguate the local time automatically, you need some additional info e.g., if you read a series of local times then it may help if you know that they are sorted: Parsing of Ordered Timestamps in Local Time (to UTC) While Observing Daylight Saving Time.
First of all '%s' is not supported on all platforms , its actually working for you because your platform C library’s strftime() function (that is called by Python) supports it. This function is what is causing the issue most probably, I am guessing its not timezone aware , hence when taking difference from epoch time it is using your local timezone, which is most probably EST(?)
Instead of relying on '%s' , which only works in few platforms (linux, I believe) , you should manually subtract the datetime you got from epoch (1970/1/1 00:00:00) to get the actual seconds since epoch . Example -
e = (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
Demo -
>>> (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
1429578727.0
This correctly corresponds to the date-time you get.
I don't exactly know why but you have to remove the timezone info from your utc_dt before using %s to print it.
e = int(utc_dt.replace(tzinfo=None).strftime("%s"))
print(e)
return e
I receive time in the format 12:00 and I also have a timezone in the format <DstTzInfo 'Europe/Paris' PMT+0:09:00 STD>. I want to see, if this date is already passed current time in that timezone.
My current timezone may be different from the timezone I receive, so I use the following:
tz = pytz.timezone('Europe/Paris')
d = parse("14:39").replace(tzinfo=tz)
now = datetime.datetime.now(tz)
print (d - now).total_seconds()
Even though the current time in Paris is 14:39, I still get 6600 seconds. I also tried now = datetime.datetime.replace(tzinfo=tz), but I get -3600 seconds.
What I want is having time in a specified timezone, check whether it passed or not.
What about this:
from datetime import datetime
from dateutil import tz
tz = tz.gettz('Europe/Paris')
now = datetime.now().replace(tzinfo=tz)
d = datetime.strptime("{0}/{1}/{2} 14:39".format(now.year,now.month,now.day), "%Y/%m/%d %H:%M").replace(tzinfo=tz)
print (d-now).total_seconds()
I keep the current time in now, in the Europe/Paris timezone, and also create d in the same timezone. If their difference is negative the date is already passed current time in that timezone.
There are a ton of questions about UTC datetime conversions and there doesn't seems to be a consensus of a "best way".
According to this: http://lucumr.pocoo.org/2011/7/15/eppur-si-muove/ , pytz is the best way to go. he shows converting to timezone like this datetime.datetime.utcnow().replace(tzinfo=pytz.utc) but he doesn't say how to get the user's timezone...
This guy https://stackoverflow.com/a/7465359/523051 says "localize adjusts for Daylight Savings Time, replace does not"
Everyone I see using pytz is supplying their own timezone (users_timezone = timezone("US/Pacific")), which I don't understand because you can't know if that's where your viewer is...
This guy https://stackoverflow.com/a/4771733/523051 has a way to auto-detect the timezones, but this is using the dateutil library, and not pytz, as is recommended by both Armin Ronacher and the official python docs ( http://docs.python.org/library/datetime.html#strftime-and-strptime-behavior , just above that anchor in yellow box)
All I need is the most simplest, future-proof, all daylight savings time/etc considered way to take my datetime.utcnow() stamp (2012-08-25 10:59:56.511479), convert it the user's timezone. And show it like this:
Aug 25 - 10:59AM
and if the year is not the current year, I'd like to say
Aug 25 '11 - 10:59AM
alright, here it is (also, my first contribution to SO :))
it does require 2 external libraries which may throw some off...
from datetime import datetime
from dateutil import tz
import pytz
def standard_date(dt):
"""Takes a naive datetime stamp, tests if time ago is > than 1 year,
determines user's local timezone, outputs stamp formatted and at local time."""
# determine difference between now and stamp
now = datetime.utcnow()
diff = now - dt
# show year in formatting if date is not this year
if (diff.days / 365) >= 1:
fmt = "%b %d '%y # %I:%M%p"
else:
fmt = '%b %d # %I:%M%p'
# get users local timezone from the dateutils library
# http://stackoverflow.com/a/4771733/523051
users_tz = tz.tzlocal()
# give the naive stamp timezone info
utc_dt = dt.replace(tzinfo=pytz.utc)
# convert from utc to local time
loc_dt = utc_dt.astimezone(users_tz)
# apply formatting
f = loc_dt.strftime(fmt)
return f
# parses some string into that format.
datetime1 = datetime.strptime(somestring, "%Y-%m-%dT%H:%M:%S")
# gets the seconds from the above date.
timestamp1 = time.mktime(datetime1.timetuple())
# adds milliseconds to the above seconds.
timeInMillis = int(timestamp1) * 1000
How do I (at any point in that code) turn the date into UTC format? I've been ploughing through the API for what seems like a century and cannot find anything that I can get working. Can anyone help? It's currently turning it into Eastern time i believe (however I'm in GMT but want UTC).
EDIT: I gave the answer to the guy with the closest to what I finally found out.
datetime1 = datetime.strptime(somestring, someformat)
timeInSeconds = calendar.timegm(datetime1.utctimetuple())
timeInMillis = timeInSeconds * 1000
:)
datetime.utcfromtimestamp is probably what you're looking for:
>>> timestamp1 = time.mktime(datetime.now().timetuple())
>>> timestamp1
1256049553.0
>>> datetime.utcfromtimestamp(timestamp1)
datetime.datetime(2009, 10, 20, 14, 39, 13)
I think you can use the utcoffset() method:
utc_time = datetime1 - datetime1.utcoffset()
The docs give an example of this using the astimezone() method here.
Additionally, if you're going to be dealing with timezones, you might want to look into the PyTZ library which has lots of helpful tools for converting datetime's into various timezones (including between EST and UTC)
With PyTZ:
from datetime import datetime
import pytz
utc = pytz.utc
eastern = pytz.timezone('US/Eastern')
# Using datetime1 from the question
datetime1 = datetime.strptime(somestring, "%Y-%m-%dT%H:%M:%S")
# First, tell Python what timezone that string was in (you said Eastern)
eastern_time = eastern.localize(datetime1)
# Then convert it from Eastern to UTC
utc_time = eastern_time.astimezone(utc)
def getDateAndTime(seconds=None):
"""
Converts seconds since the Epoch to a time tuple expressing UTC.
When 'seconds' is not passed in, convert the current time instead.
:Parameters:
- `seconds`: time in seconds from the epoch.
:Return:
Time in UTC format.
"""
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(seconds))`
This converts local time to UTC
time.mktime(time.localtime(calendar.timegm(utc_time)))
http://feihonghsu.blogspot.com/2008/02/converting-from-local-time-to-utc.html
If converting a struct_time to seconds-since-the-epoch is done using mktime, this
conversion is in local timezone. There's no way to tell it to use any specific timezone, not even just UTC. The standard 'time' package always assumes that a time is in your local timezone.
You probably want one of these two:
import time
import datetime
from email.Utils import formatdate
rightnow = time.time()
utc = datetime.datetime.utcfromtimestamp(rightnow)
print utc
print formatdate(rightnow)
The two outputs look like this
2009-10-20 14:46:52.725000
Tue, 20 Oct 2009 14:46:52 -0000