I am trying to parse an RSS feed. Entries in the feed have date elements like:
<dc:date>2016-09-21T16:00:00+02:00</dc:date>
Using feedparser, I try to do:
published_time = datetime.fromtimestamp(mktime(entry.published_parsed))
But the problem is that I seem to be getting the wrong time stored in the database. In this particular case, the datetime is stored as:
2016-09-21 13:00:00
... when I would expect 14:00 - the correct UTC time.
I assume the problem is in our django settings, where we have:
TIME_ZONE = 'Europe/Berlin'
Because when I switch to:
TIME_ZONE = 'UTC'
... the datatime is stored as correct UTC time:
2016-09-21 14:00:00
Is there any way to keep the django settings as they are, but to parse and store this datetime correctly, without the django timezone setting affecting it?
EDIT:
Maybe it's more clear like this...
print entry.published_parsed
published_time = datetime.fromtimestamp(mktime(entry.published_parsed))
print published_time
localized_time = pytz.timezone(settings.TIME_ZONE).localize(published_time, is_dst=None)
print localized_time
time.struct_time(tm_year=2016, tm_mon=9, tm_mday=21, tm_hour=14, tm_min=0, tm_sec=0, tm_wday=2, tm_yday=265, tm_isdst=0)
2016-09-21 15:00:00
2016-09-21 15:00:00+02:00
feedparser's entry.published_parsed is always a utc time tuple whatever the input time string is. To get timezone-aware datetime object:
from datetime import datetime
utc_time = datetime(*entry.published_parsed[:6], tzinfo=utc)
where utc is a tzinfo object such as datetime.timezone.utc, pytz.utc, or just your custom tzinfo (for older python versions).
You shouldn't pass utc time to mktime() that expects a local time. Same error: Have a correct datetime with correct timezone.
Make sure USE_TZ=True so that django uses aware datetime objects everywhere. Given a timezone-aware datetime object, django should save it to db correctly whatever your TIME_ZONE or timezone.get_current_timezone() are.
Have you tried using datetime.utcfromtimestamp() instead of datetime.fromtimestamp()?
As a secondary solution, you can get the unparsed data (I believe it's available as entry.published?) and just use python-dateutil to parse the string, then convert it to pytz.utc timezone like this.
>>> import pytz
>>> from dateutil import parser
>>> dt = parser.parse('2016-09-21T16:00:00+02:00')
>>> dt
datetime.datetime(2016, 9, 21, 16, 0, tzinfo=tzoffset(None, 7200))
>>> dt.astimezone(pytz.utc)
datetime.datetime(2016, 9, 21, 14, 0, tzinfo=<UTC>)
Use
published_time = pytz.utc.localize(datetime.utcfromtimestamp(calendar.timegm(parsed_entry.published_parsed)))
Feedparser can parse a large range of date formats, you can find them here.
As you can see in feedparser/feedparser/datetimes/__init__.py, the built-in function from Feedparser _parse_date does the following:
Parses a variety of date formats into a 9-tuple in GMT
This means that in parsed_entry.published_parsed you have a time.struct_time object in GMT timezone.
When you convert it to a datetime object using
published_time = datetime.fromtimestamp(mktime(parsed_entry.published_parsed))
the problem is that mktime assumes that the passed tuple is in local time, which is not, it's GMT/UTC! Other than that you don't properly localize the datetime object at the end of the conversion.
You need to replace that conversion with the following, remembering that Feedparser returns a GMT struct_time, and localize that with the timezone you like (UTC for the sake of simplicity).
You use calendar.timegm, which gives the number of seconds between epoch and the date passed as a parameter, assuming that the passed object is in UTC/GMT (we know from Feedparser it is)
You use utcfromtimestamp to obtain a naive datetime object (which we know represents a datetime in UTC, but Python does not at this moment)
With pytz.utc.localize you properly localize in UTC the datetime object.
Example:
import calendar
from datetime import datetime
import pytz
localized_dt = pytz.utc.localize(datetime.utcfromtimestamp(calendar.timegm(parsed_entry.published_parsed)))
As long as you are consistent, it doesn't matter if you use fromtimestamp or utcfromtimestamp. If you use fromtimestamp you need to tell Python that the datetime object you created has the local timezone. Supposing you are in Europe/Berlin, this is also fine:
pytz.timezone('Europe/Berlin').localize(datetime.fromtimestamp(calendar.timegm(parsed_entry.published_parsed)))
Were parsed_entry.published_parsed also in local timezone, mktime must be used in place of calendar.timegm.
As an alternative you can parse yourself the data string you get from Feedparser parsed_entry['published']
from dateutil import parser
localized_dt = parser.parse(parsed_entry['published'])
You can check that the following returns True:
parser.parse(parsed_entry['published']) == pytz.utc.localize(datetime.utcfromtimestamp(calendar.timegm(parsed_entry.published_parsed)))
The Django TIME_ZONE setting doesn't actually matter, because it's used only for visualization purposes or to automatically convert naive datetimes.
When USE_TZ is True, this is the default time zone that Django will use to display datetimes in templates and to interpret datetimes entered in forms.
What is important is to always use properly localized datetimes, no matter which time zone is used. As long as they are not in naive format, they will be properly handled by Django.
Related
I have the following timestamp 1550588656 which translates to 2019-02-19 15:04:16+00:00 in UTC time convention.
I want to convert it to my country's time convention (UTC or GMT -3 in this time of the year) so it should translate to 2019-02-19 12:04:16+00:00
I have read on other SO questions that first I have to convert the timestamp to an UTC aware Datetime object and then localize it, I'm doing it like this
# string format time
naive_datetime = datetime.fromtimestamp(timestamp).strftime('%Y-%m-%d %H:%M:%S')
# string parse time
naive_datetime = datetime.strptime(naive_datetime, "%Y-%m-%d %H:%M:%S")
# make naive Datetime object UTC aware
utc_datetime = naive_datetime.replace(tzinfo=pytz.UTC)
So now it's not a naive Datetime object, from here I should be able to localize it to my country's timezone. In Python that is pytz.timezone('America/Santiago')
So it should go something like this
cltime = pytz.timezone('America/Santiago')
local_datetime = utc_datetime.astimezone(cltime)
But I'm getting 2019-02-19 09:04:16-03:00 (UTC or GTM -6 ) as a result and I don't know why.
Can someone explain? My intuition tells me it's probably a simple thing I'm not looking at, but I've spent some minutes in it and I haven't been able to tell yet.
If you look at the documentation for fromtimestamp:
Return the local date and time corresponding to the POSIX timestamp
So the problem is that it is already doing a conversion from UTC to the local time, and you're doing it a second time.
First of all you have epoc time (UTC timestamp). You need to convert it into datetime object (native) which is followed by converting native time to aware time and than finally convert it to your local time.
Convert your timestamp to native datetime object
native_datetime = datetime.fromtimestamp(1550588656)
convert native datetime object to aware time (add timezone info, will add timezone info to native timezone UTC for current)
import pytz
utc_time = native_datetime.replace(tzinfo=pytz.timezone("UTC"))
localising aware datetime to your local datetime
local_time = utc_time.astimezone(pytz.timezone("America/Santiago"))
You can replace "America/Santiago" with your local time zone
I think this would help you to solve your problem. Thanks!
While I read just about every post related to timezone conversions I'm still having some issues and my converted time is incorrect
settings.py
TIME_ZONE = 'UTC'
USE_TZ = True
views.py
utc = datetime.utcnow()
instance_time_zone = pytz.timezone(instance.timezone) # 'US/Pacific'
start_date = instance_time_zone.localize(datetime.utcnow(), is_dst=None)
template.html
utc: Oct. 2, 2015, 5:32 p.m. #correct time
start_date: Oct. 3, 2015, 1:32 a.m. #incorrect time
For some reason, the converted time is wrong and 15 hours ahead of the Pacific Time and 8 hours ahead of the UTC time.
timezone.localize() should be used for naive datetime objects (objects with no timezone of their own). The timezone is attached to that datetime as if the date and time are correct for that timezone. So in your case you 'localised' UTC as if it is your local time without DST, shifting it 8 hours in the wrong direction.
You used a UTC timestamp however, so you need to attach the UTC timezone to that, then move the timestamp to the desired timezone:
utc = pytz.utc.localize(datetime.utcnow())
instance_time_zone = pytz.timezone(instance.timezone) # 'US/Pacific'
start_date = utc.astimezone(instance_time_zone)
Note that the utc value is now a datetime object with timezone, so you can then use the datetime.astimezone() method to produce a value in the desired target timezone from it.
Demo:
>>> from datetime import datetime
>>> utc = pytz.utc.localize(datetime.utcnow())
>>> utc
datetime.datetime(2015, 10, 2, 17, 58, 10, 168575, tzinfo=<UTC>)
>>> instance_time_zone = pytz.timezone('US/Pacific')
>>> utc.astimezone(instance_time_zone)
datetime.datetime(2015, 10, 2, 10, 58, 10, 168575, tzinfo=<DstTzInfo 'US/Pacific' PDT-1 day, 17:00:00 DST>)
Now the produced datetime is properly 5 hours removed from UTC.
If you are outputting these values into a Django template, however, note that Django will also transform the timezone. See the Django timezone documentation, specifically the section on using aware datetime objects in templates:
When you enable time zone support, Django converts aware datetime objects to the current time zone when they’re rendered in templates. This behaves very much like format localization.
and from the current time zone section:
You should set the current time zone to the end user’s actual time zone with activate(). Otherwise, the default time zone is used.
It then doesn't matter what timezone you moved the datetime object to; it'll use whatever is the current timezone to display the value. You generally want to use aware datetime objects in the UTC timezone, then use activate() to switch what timezone everything is displayed in.
So in Django, just use timezone.now() everywhere, and let the templating system worry about converting that to a given timezone.
To get the current time in django, use timezone.now():
from django.utils import timezone
start_date = timezone.now()
If instance.timezone refers to the same timezone as timezone.get_current_timezone() (default is TIME_ZONE) then it is all you need (timezone.now() returns an aware datetime object in UTC (if USE_TZ=True) that is converted during rendering to the current time zone).
Otherwise, you could call timezone.activate(instance.timezone) to set the current time zone.
If you want (you don't need to) you can convert the timezones explicitly:
import pytz
from django.utils import timezone
now = timezone.localtime(timezone.now(), pytz.timezone(instance.timezone))
Outside django code, you could get the current time in a given timezone by passing the tzinfo explicitly:
from datetime import datetime
import pytz
start_date = datetime.now(pytz.timezone('America/Los_Angeles'))
It works even during ambiguous local times.
To convert an existing naive datetime object that represents time in a given pytz timezone:
start_date = instance_time_zone.localize(datetime_in_instance_time_zone,
is_dst=None)
This code raises an exception for ambiguous/non-existing times in instance time zone (e.g. during DST transitions). If it is ok to return an imprecise result in some cases instead of an exception then don't pass is_dst=None:
tz = instance_time_zone
start_date = tz.normalize(tz.localize(datetime_in_instance_time_zone))
For more details about is_dst, see "Can I just always set is_dst=True?" section.
I'd like to use the timestamp from a database result and convert it to my locale time format. The timestamp itself is saved in UTC format: 2015-03-30 07:19:06.746037+02. After calling print value.strftime(format) with the format %d.%m.%Y %H:%M %z the output will be 30.03.2015 07:19 +0200. This might be the correct way to display timestamps with timezone information but unfortunately users here are not accustomed to that. What I want to achieve is the following for the given timestamp: 30.03.2015 09:19. Right now I'm adding two hours via
is_dst = time.daylight and time.localtime().tm_isdst > 0
utc_offset = - (tine.altzone if is_dst else time.timezone)
value = value + timedelta(seconds=utc_offset)
I was wondering if there is a more intelligent solution to my problem. (timestamp.tzinfo has a offset value, can/should this be used instead? The solution needs to be DST aware too.)
In your question the timestamp is already in desired timezone, so you don't need to do anything.
If you want to convert it to some other timezone you should be able to use;
YourModel.datetime_column.op('AT TIME ZONE')('your timezone name')
or,
func.timezone('your timezone name', YourModel.datetime_column)
in SQLAlchemy level.
On python level, consider using pytz
You don't need to do conversions manually when you use time zone aware database timestamps unless the timezone you want to display is different from the system timezone.
When you read and write datetime objects to the database the timezone info of the datetime object is taken into account, this means what you get back is the time in the local time zone, in your case +0200.
This SO post answers how to get local time from a timezoned timestamp.
Basically, use tzlocal.
import time
from datetime import datetime
import pytz # $ pip install pytz
from tzlocal import get_localzone # $ pip install tzlocal
# get local timezone
local_tz = get_localzone()
# test it
# utc_now, now = datetime.utcnow(), datetime.now()
ts = time.time()
utc_now, now = datetime.utcfromtimestamp(ts), datetime.fromtimestamp(ts)
local_now = utc_now.replace(tzinfo=pytz.utc).astimezone(local_tz) # utc -> local
assert local_now.replace(tzinfo=None) == now
Using '%z' pattern of datetime.strptime()
I have a string text that represent a date and I'm perfectly able to parse it and transform it into a clean datetime object:
date = "[24/Aug/2014:17:57:26"
dt = datetime.strptime(date, "[%d/%b/%Y:%H:%M:%S")
Except that I can't catch the entire date string with the timezone using the %z pattern as specified here
date_tz = 24/Aug/2014:17:57:26 +0200
dt = datetime.strptime(date, "[%d/%b/%Y:%H:%M:%S %z]")
>>> ValueError: 'z' is a bad directive in format '[%d/%b/%Y:%H:%M:%S %z]'
Because as this bug report says
strftime() is implemented per platform
I precise that there is no such a problem with the naive tzinfo directive '%Z'
Workaround : Casting tzinfo string into tzinfo object
I can perfectly make the following workaround by transforming the GST time format string into a tzinfo object [as suggested here][4] using dateutil module
and then insert tzinfo into datetime object
Question: Make %z available for my plateform?
But as I will obviously need %z pattern for further project I would like to find a solution to avoid this workaround and using external module for this simple task.
Can you suggest me some reading on it? I supposed that newer version of python (I'm on 2.7) can handle it but I'd rather not changing my version now for this little but crucial detail.
[EDIT]
Well, seeing comments make me reformulated my question how to parse Email time zone indicator using strptime() without being aware of locale time?
strptime() is implemented in pure Python. Unlike strftime(); it [which directives are supported] doesn't depend on platform. %z is supported since Python 3.2:
>>> from datetime import datetime
>>> datetime.strptime('24/Aug/2014:17:57:26 +0200', '%d/%b/%Y:%H:%M:%S %z')
datetime.datetime(2014, 8, 24, 17, 57, 26, tzinfo=datetime.timezone(datetime.timedelta(0, 7200)))
how to parse Email time zone indicator using strptime() without being aware of locale time?
There is no concrete timezone implementation in Python 2.7. You could easily implement the UTC offset parsing, see How to parse dates with -0400 timezone string in python?
In continue to #j-f-sebastians 's answer, here is a fix for python 2.7
Instead of using:
datetime.strptime(t,'%Y-%m-%dT%H:%M %z')
use the timedelta to account for the timezone, like this:
from datetime import datetime,timedelta
def dt_parse(t):
ret = datetime.strptime(t[0:16],'%Y-%m-%dT%H:%M')
if t[17]=='+':
ret-=timedelta(hours=int(t[18:20]),minutes=int(t[20:]))
elif t[17]=='-':
ret+=timedelta(hours=int(t[18:20]),minutes=int(t[20:]))
return ret
print(dt_parse('2017-01-12T14:12 -0530'))
The Answer of Uri is great, saved my life, but when you have
USE_TZ = True you need to be careful with the time, for avoid the warning "RuntimeWarning: DateTimeField" is better if you add the utc to the return.
import pytz
from datetime import datetime, timedelta
def dt_parse(t):
ret = datetime.strptime(t[0:19],'%Y-%m-%dT%H:%M:%S')
if t[23]=='+':
ret-=timedelta(hours=int(t[24:26]), minutes=int(t[27:]))
elif t[23]=='-':
ret+=timedelta(hours=int(t[24:26]), minutes=int(t[27:]))
return ret.replace(tzinfo=pytz.UTC)
I have a case where I get an CET datetime (via my API) and I want to save it in my django database.
When I do this I always get a runtime warning:
datetime.fromtimestamp(last_content_update)
When I add the replace part I don't get the error but my date is not correct (1h shift):
datetime.fromtimestamp(last_content_update).replace(tzinfo=utc)
Any idea how I can load my datetime timezone aware?
Ok, finally I found out that the fromtimestamp() takes an timezone parameter!
Just do it like this:
datetime.fromtimestamp(last_content_update, tz=pytz.timezone("Europe/Paris"))
Here is a link to the documentation.
You need to do this in two steps:
First, make the datetime timezone-aware - in your case, using the CET (Central European Time) timezone.
Second, convert the datetime to UTC before saving it. This is because Django stores datetimes as UTC:
When support for time zones is enabled, Django stores datetime
information in UTC in the database...
This can be done like this:
import pytz
from django.utils import timezone
# Convert the naive datetime into a CET datetime
local_datetime = timezone.make_aware(naive_datetime, pytz.timezone('CET'))
# Convert the CET datetime to UTC
utc_datetime = local_datetime.astimezone(timezone.utc)
# Now it can be saved in the database on a model field
MyDateModel.objects.create(title='Another date model',
created=utc_datetime)