Develop Reference

How to convert UTC to EST with Python and take care of daylight saving automatically?

If I have a bunch of data with date & time in UTC format, how can I convert them to EST.
It can determine when they will be -4(in summer) and -5(in winter) automatically every year?
Thanks

You'll need to use the pytz module (available from PyPI):
import pytz
from datetime import datetime
est = pytz.timezone('US/Eastern')
utc = pytz.utc
fmt = '%Y-%m-%d %H:%M:%S %Z%z'
winter = datetime(2016, 1, 24, 18, 0, 0, tzinfo=utc)
summer = datetime(2016, 7, 24, 18, 0, 0, tzinfo=utc)
print(winter.strftime(fmt))
print(summer.strftime(fmt))
print(winter.astimezone(est).strftime(fmt))
print(summer.astimezone(est).strftime(fmt))
which will print:
2016-01-24 18:00:00 UTC+0000
2016-07-24 18:00:00 UTC+0000
2016-01-24 13:00:00 EST-0500
2016-07-24 14:00:00 EDT-0400
The reason why you'll need to use 'US/Eastern' and not 'EST' is exemplified in the last two lines of output.

If you have a pandas series with object datatype, you can first convert it into a DateTime series using pd.to_datetime()
df[col] = pd.to_datetime(your_series, format = '%Y-%m-%d %H:%M:%S', errors ='coerce')
Check if it is timezone aware or not by using series.dt.tz
df[col].dt.tz
If it's not timezone aware, we should make it timezone aware by using series.dt.tz_localize(). Also, do read about the ambiguous and nonexistent parameters of this function
df[col] = your_series[col].dt.tz_localize('UTC')
Now convert this series into the required timezone by series.dt.tz_convert()
df[col] = your_series[col].dt.tz_convert('US/Eastern')
The above method will take care of daylight savings time. If you want to check more timezones you can pip install pytz and
import pytz
pytz.common_timezones

In case you just want the normalized hour offset for your existing timedelta shifting:
from datetime import datetime
import pytz
def curr_est_offset():
tz_est = pytz.timezone('US/Eastern')
offset = tz_est.utcoffset(datetime.utcnow())
offset_seconds = (offset.days * 86400) + offset.seconds
offset_hours = offset_seconds // 3600
return offset_hours # -4 or -5

As mentioned above, you can use pandas.DataFrame.tz_convert() like this:
import pandas as pd
from datetime import datetime
df = pd.read_csv("your_data_file_path.csv", index_col=False, engine='python')
df['Date'] = pd.to_datetime(df['Date'])
df['Date'] = df['Date'].dt.tz_localize('US/Eastern').dt.tz_convert('UTC')
df['Date'] = df['Date'].apply(lambda x: datetime.replace(x, tzinfo=None))
What the last row does is removing the timezone info from the datetime object, so you can operate with the date and time only (don't worry, that doesn't change the timezone again, it just strips it from the timestamp string).

This is thebjorn's answer converted from Python 2 to Python 3 with a few additional comments. Thanks to thebjorn.
For convention, I use these terms:
EST: Eastern Standard Time (winter)
EDT: Eastern Daylight Time
(summer)
EPT: Eastern Prevailing Time (standard or daylight saving
time as appropriate)
Code:
# Convert EPT / UTC
import pytz
import datetime from datetime as dtdt
# Timezones
ept = pytz.timezone('US/Eastern')
utc = pytz.utc
# str format
fmt = '%Y-%m-%d %H:%M:%S %Z%z'
print("\nEPT/UTC examples:")
print("\nWinter (EST) example:")
# Create a UTC time in the winter
winter_utc = dtdt(2016, 1, 24, 18, 0, 0, tzinfo=utc)
print(" UTC: ", winter_utc.strftime(fmt))
# Convert from UTC to eastern prevailing time. Since, the timestamp is in the
# winter, prevailing time is standard time.
winter_ept = winter_utc.astimezone(ept)
print(" EPT: ", winter_ept.strftime(fmt))
# Let's convert back to UTC to show we get back to the original value.
winter_utc2 = winter_ept.astimezone(utc)
print(" UTC: ", winter_utc2.strftime(fmt))
# Let's do that again for a summer datetime.
print("\nSummer (EDT) example:")
summer_utc = dtdt(2016, 7, 24, 18, 0, 0, tzinfo=utc)
print(" UTC: ", summer_utc.strftime(fmt))
# Convert from UTC to eastern prevailing time. Since, the timestamp is in the
# winter, prevailing time is daylight saving time.
summer_ept = summer_utc.astimezone(ept)
print(" EPT: ", summer_ept.strftime(fmt))
# Let's convert back to UTC to show we get back to the original value.
summer_utc2 = summer_ept.astimezone(utc)
print(" UTC: ", summer_utc2.strftime(fmt))
Console:
EPT/UTC examples:
Winter (EST) example:
UTC: 2016-01-24 18:00:00 UTC+0000
EPT: 2016-01-24 13:00:00 EST-0500
UTC: 2016-01-24 18:00:00 UTC+0000
Summer (EDT) example:
UTC: 2016-07-24 18:00:00 UTC+0000
EPT: 2016-07-24 14:00:00 EDT-0400
UTC: 2016-07-24 18:00:00 UTC+0000

Python: converting date string to UTC

I am parsing a 3rd party website HTML with dates and times which are always in UK time format, however they don't have any timezone info in the source. Converting the string to an object is easy enough using datetime.strptime(), but how do I add timezone info?
Ultimately, I need to convert these strings to a datetime object in UTC format. The code will always run on a PC which is timezone aware, i.e. datetime.now() will return UK time.
temp = '07/12/2017 13:30'
dt = datetime.strptime(temp, '%d/%m/%Y %H:%M')
Is there a nicer way to do this?
offset = datetime.now() - datetime.utcnow()
dt -= offset

Use pytz
import datetime
import pytz
temp = '07/12/2017 13:30'
dt = datetime.strptime(temp, '%d/%m/%Y %H:%M')
timezone = pytz.timezone("Etc/Greenwich")
d_aware = timezone.localize(dt)
d_aware.tzinfo
> <DstTzInfo 'Etc/Greenwich' PST-1 day, 16:00:00 STD>
d_aware
datetime.datetime(2017, 12, 7, 13, 30, tzinfo=<StaticTzInfo 'Etc/Greenwich'>)

There are some good libraries that make working with dates so much easier. I like dateparser, parsedatetime, and arrow;
import dateparser as dp
dt = dp.parse('07-12-2017 13:30 PST')
print (dt)
dt = dp.parse("Yesterday at 3:00am EST")
print(dt)
2017-07-12 13:30:00-08:00
2017-12-06 17:07:07.557109-05:00

Python construct datetime having weekday with other time parameters

I need to construct datetime object from different parameters. For example, I get these parameters:
integer number from 0 to 6. This one indicates weekday.
hour and minutes in float format (from 0.0 to 24.0).
And then I know which week of the current year it is going to be. So, for example, let say that datetime should be from 2014-07-18 00:00:00 to 2014-07-24 23:59:00 (seconds can be ignored and left at 00). So to get exact datetime I need to use above defined parameters.
Let say I would get these parameters 4 (meaning Friday), and 9.5 (meaning 09:30).
So by doing such construction, I should get a date that would be: 2014-07-22 09:30:00. How could accomplish such thing?
Do I need to somehow get for example day of the month by knowing which is the week of the year and which weekday it is?
P.S. A bit more detailed example of what I'm trying to accomplish
from datetime import datetime
today = datetime.today() #using this to get the week I'll be working with.
today = today.replace(day=?) #how to get which day I
#need to enter by having weekday and knowing that week is the present one?

You could do something like that, if your parameters are weekday and t (time):
from datetime import timedelta
monday = today - timedelta(days=today.weekday())
result = (monday + timedelta(days=weekday)).replace(hour=int(t), minutes=int((t - int(t)) * 60))

If you have a starting date, use the relative value of the datetime.datetime.weekday() value to construct a timedelta() object that'll put you onto the right weekday, then replace the hour and minutes:
from datetime import timedelta
def relative_date(reference, weekday, timevalue):
hour, minute = divmod(timevalue, 1)
minute *= 60
days = reference.weekday() - weekday
return (reference - timedelta(days=days)).replace(
hour=int(hour), minute=int(minute), second=0, microsecond=0)
Demo:
>>> from datetime import timedelta, datetime
>>> def relative_date(reference, weekday, timevalue):
... hour, minute = divmod(timevalue, 1)
... minute *= 60
... days = reference.weekday() - weekday
... return (reference - timedelta(days=days)).replace(
... hour=int(hour), minute=int(minute), second=0, microsecond=0)
...
>>> relative_date(datetime.now(), 4, 9.5)
datetime.datetime(2014, 8, 22, 9, 30)
>>> relative_date(datetime.now() - timedelta(days=30), 6, 11.75)
datetime.datetime(2014, 7, 27, 11, 45)

I would use timedelta to add the difference between weekdays to the datetime
from datetime import datetime, timedelta
friday = 4
today = datetime.now()
friday_this_week = today + timedelta(friday - today.weekday())
In your case just replace today with a date that is in the week you want.

parsing month year pairs into datetime

Let's say i have 2 strings 'Jan-2010' and 'Mar-2010' and i want to parse it such that it returns 2 datetime objects: 1-Jan-2010 and 31-Mar-2010 (i.e. the last day).
What would be the best strategy in python? Should i just split the string into tokens or use regular expressions and then use the calendar functions to get say the last day of the month for 'Mar-2010' (getting the first day is trivial, its always 1 in this case unless i wanted the first working day of the month).
Any suggestions? Thanks in advance.

strptime does the string parsing into dates on your behalf:
def firstofmonth(MmmYyyy):
return datetime.datetime.strptime(MmmYyyy, '%b-%Y').date()
much better than messing around with tokenization, regexp, &c!-).
To get the date of the last day of the month, you can indeed use the calendar module:
def lastofmonth(MmmYyyy):
first = firstofmonth(MmmYyyy)
_, lastday = calendar.monthrange(first.year, first.month)
return datetime.date(first.year, first.month, lastday)
You could ALMOST do it neatly with datetime alone, e.g., an ALMOST working approach:
def lastofmonth(MmmYyyy):
first = firstofmonth(MmmYyyy)
return first.replace(month=first.month+1, day=1
) - datetime.timedelta(days=1)
but, alas!, this breaks for December, and the code needed to specialcase December makes the overall approach goofier than calendar affords;-).

I highly recommend using the python timeseries module, which you can download and read about here:
http://pytseries.sourceforge.net/
You should also use the dateutil package for parsing the date string, which you can find here:
http://labix.org/python-dateutil
Then you can do something like this
import datetime
import dateutil.parser
import scikits.timeseries as TS
m1 = TS.Date('M', datetime=dateutil.parser.parse('Jan-2010'))
m2 = TS.Date('M', datetime=dateutil.parser.parse('Mar-2010'))
d1 = m1.asfreq('D', relation='START') # returns a TS.Date object
d2 = m2.asfreq('D', relation='END')
firstDay = d1.datetime
lastDay = d2.datetime
This solution is dependent out outside modules, but they're very powerful and well written.

from datetime import datetime, timedelta
def first_day(some_date):
return some_date.replace(day=1, hour=0, minute=0, second=0, microsecond=0)
def next_month(some_date):
return first_day(first_day(some_date) + timedelta(days=31))
def last_day(some_date):
return next_month(some_date) - timedelta(days=1)
# testing:
months = [('Jan-2010', 'Mar-2010'), # your example
('Apr-2009', 'Apr-2009'), # same month, 30 days
('Jan-2008', 'Dec-2008'), # whole year
('Jan-2007', 'Feb-2007')] # february involved
for date1, date2 in months:
print first_day(datetime.strptime(date1, '%b-%Y')),
print '-',
print last_day(datetime.strptime(date2, '%b-%Y'))
That prints:
2010-01-01 00:00:00 - 2010-03-31 00:00:00
2009-04-01 00:00:00 - 2009-04-30 00:00:00
2008-01-01 00:00:00 - 2008-12-31 00:00:00
2007-01-01 00:00:00 - 2007-02-28 00:00:00

i know it's long time gone, but if someone needs:
from dateutil import rrule
from dateutil import parser
from datetime import datetime
first_day = parser.parse('Jan-2010',default=datetime(1,1,1))
last_day = rrule.rrule(rrule.MONTHLY,count=1,bymonthday=-1, bysetpos=1,dtstart=parser.parse('Mar-2010'))

Riffing on Alex Martelli's:
import datetime
def lastofmonthHelper(MmmYyyy): # Takes a date
return MmmYyyy.replace(year=MmmYyyy.year+(MmmYyyy.month==12), month=MmmYyyy.month%12 + 1, day=1) - datetime.timedelta(days=1)
>>> for month in range(1,13):
... t = datetime.date(2009,month,1)
... print t, lastofmonthHelper(t)
...
2009-01-01 2009-01-31
2009-02-01 2009-02-28
2009-03-01 2009-03-31
2009-04-01 2009-04-30
2009-05-01 2009-05-31
2009-06-01 2009-06-30
2009-07-01 2009-07-31
2009-08-01 2009-08-31
2009-09-01 2009-09-30
2009-10-01 2009-10-31
2009-11-01 2009-11-30
2009-12-01 2009-12-31
You don't have to use the first day of the month, BTW. I would have put this in a comment but we all know how the formatting would have turned out. Feel free to upvote Alex.
If you call with the result of a firstofmonth() call, you get the desired result:
>>> lastofmonthHelper(firstofmonth('Apr-2009'))
datetime.date(2009, 4, 30)

How do I get the UTC time of "midnight" for a given timezone?

The best I can come up with for now is this monstrosity:
>>> datetime.utcnow() \
... .replace(tzinfo=pytz.UTC) \
... .astimezone(pytz.timezone("Australia/Melbourne")) \
... .replace(hour=0,minute=0,second=0,microsecond=0) \
... .astimezone(pytz.UTC) \
... .replace(tzinfo=None)
datetime.datetime(2008, 12, 16, 13, 0)
I.e., in English, get the current time (in UTC), convert it to some other timezone, set the time to midnight, then convert back to UTC.
I'm not just using now() or localtime() as that would use the server's timezone, not the user's timezone.
I can't help feeling I'm missing something, any ideas?

I think you can shave off a few method calls if you do it like this:
>>> from datetime import datetime
>>> datetime.now(pytz.timezone("Australia/Melbourne")) \
.replace(hour=0, minute=0, second=0, microsecond=0) \
.astimezone(pytz.utc)
BUT… there is a bigger problem than aesthetics in your code: it will give the wrong result on the day of the switch to or from Daylight Saving Time.
The reason for this is that neither the datetime constructors nor replace() take DST changes into account.
For example:
>>> now = datetime(2012, 4, 1, 5, 0, 0, 0, tzinfo=pytz.timezone("Australia/Melbourne"))
>>> print now
2012-04-01 05:00:00+10:00
>>> print now.replace(hour=0)
2012-04-01 00:00:00+10:00 # wrong! midnight was at 2012-04-01 00:00:00+11:00
>>> print datetime(2012, 3, 1, 0, 0, 0, 0, tzinfo=tz)
2012-03-01 00:00:00+10:00 # wrong again!
However, the documentation for tz.localize() states:
This method should be used to construct localtimes, rather
than passing a tzinfo argument to a datetime constructor.
Thus, your problem is solved like so:
>>> import pytz
>>> from datetime import datetime, date, time
>>> tz = pytz.timezone("Australia/Melbourne")
>>> the_date = date(2012, 4, 1) # use date.today() here
>>> midnight_without_tzinfo = datetime.combine(the_date, time())
>>> print midnight_without_tzinfo
2012-04-01 00:00:00
>>> midnight_with_tzinfo = tz.localize(midnight_without_tzinfo)
>>> print midnight_with_tzinfo
2012-04-01 00:00:00+11:00
>>> print midnight_with_tzinfo.astimezone(pytz.utc)
2012-03-31 13:00:00+00:00
No guarantees for dates before 1582, though.

#hop's answer is wrong on the day of transition from Daylight Saving Time (DST) e.g., Apr 1, 2012. To fix it tz.localize() could be used:
tz = pytz.timezone("Australia/Melbourne")
today = datetime.now(tz).date()
midnight = tz.localize(datetime.combine(today, time(0, 0)), is_dst=None)
utc_dt = midnight.astimezone(pytz.utc)
The same with comments:
#!/usr/bin/env python
from datetime import datetime, time
import pytz # pip instal pytz
tz = pytz.timezone("Australia/Melbourne") # choose timezone
# 1. get correct date for the midnight using given timezone.
today = datetime.now(tz).date()
# 2. get midnight in the correct timezone (taking into account DST)
#NOTE: tzinfo=None and tz.localize()
# assert that there is no dst transition at midnight (`is_dst=None`)
midnight = tz.localize(datetime.combine(today, time(0, 0)), is_dst=None)
# 3. convert to UTC (no need to call `utc.normalize()` due to UTC has no
# DST transitions)
fmt = '%Y-%m-%d %H:%M:%S %Z%z'
print midnight.astimezone(pytz.utc).strftime(fmt)

This is more straightforward with dateutil.tz than pytz:
>>>import datetime
>>>import dateutil.tz
>>>midnight=(datetime.datetime
.now(dateutil.tz.gettz('Australia/Melbourne'))
.replace(hour=0, minute=0, second=0, microsecond=0)
.astimezone(dateutil.tz.tzutc()))
>>>print(midnight)
2019-04-26 14:00:00+00:00
The tzinfo documentation recommends dateutil.tz since Python 3.6. The tzinfo objects from dateutil.tz have no problems with anomalies like DST without requiring the localize functionality of pytz. Using the example from user3850:
>>> now = (datetime.datetime(2012, 4, 1, 5,
... tzinfo = dateutil.tz.gettz('Australia/Melbourne')))
>>> print(now.replace(hour = 0).astimezone(dateutil.tz.tzutc()))
2012-03-31 13:00:00+00:00

Setting the TZ environment variable modifies what timezone Python's date and time functions work with.
>>> time.gmtime()
(2008, 12, 17, 1, 16, 46, 2, 352, 0)
>>> time.localtime()
(2008, 12, 16, 20, 16, 47, 1, 351, 0)
>>> os.environ['TZ']='Australia/Melbourne'
>>> time.localtime()
(2008, 12, 17, 12, 16, 53, 2, 352, 1)

Each time zone has a number, eg US/Central = -6. This is defined as the offset in hours from UTC. Since 0000 is midnight, you can simply use this offset to find the time in any time zone when it is midnight UTC. To access that, I believe you can use time.timezone
According to The Python Docs, time.timezone actually gives the negative value of this number:
time.timezone
The offset of the local (non-DST) timezone, in seconds west of UTC (negative in most of Western Europe, positive in the US, zero in the UK).
So you would simply use that number for the time in hours if it's positive (i.e., if it's midnight in Chicago (which has a +6 timezone value), then it's 6000 = 6am UTC).
If the number is negative, subtract from 24. For example, Berlin would give -1, so 24 - 1 => 2300 = 11pm.

It's worth remarking that we can adapt the answer given by #jfs to find tomorrow's midnight or yesterday's midnight, etc. The trick is to add a certain number of days to the aware timezone. This works because although this usually adds 24 hours, sometimes it might add 23 or 25 based on DST issues.
from datetime import datetime, time, timedelta
import pytz
def midnight_UTC(offset):
# Construct a timezone object
tz = pytz.timezone('Australia/Melbourne')
# Work out today/now as a timezone-aware datetime
today = datetime.now(tz)
# Adjust by the offset. Note that that adding 1 day might actually move us 23 or 25
# hours into the future, depending on daylight savings. This works because the {today}
# variable is timezone aware
target_day = today + timedelta(days=1) * offset
# Discard hours, minutes, seconds and microseconds
midnight_aware = tz.localize(
datetime.combine(target_day, time(0, 0, 0, 0)), is_dst=None)
# Convert to UTC
midnight_UTC = midnight_aware.astimezone(pytz.utc)
return midnight_UTC
print("The UTC time of the previous midnight is:", midnight_UTC(0))
print("The UTC time of the upcoming midnight is:", midnight_UTC(1))