I am trying to convert a column that contains Zulu formatted timestamps to a typical datetime format. This is an example of the format the dates are in: 1533953335000.
So far, I have been using this:
from pyspark.sql import functions as f
from pyspark.sql import types as t
df=df.withColumn('DTMZ',f.date_format(df.DTMZ.cast(dataType=t.TimestampType()), "yyyy-MM-dd"))
df=df.withColumn('DTMZ', f.to_date(df.DTMZ.cast(dataType=t.TimestampType())))
My output when I use the above code is: 50579-01-17
I am hoping to be able to view these dates in a typical readable format.
Could anyone help me out with this?
datetime in Python and most languages and databases is a binary type, it has no format. Python has no timestamp type. The value 1533953335000 looks to be a UNIX timestamp, an offset since 1970-01-01.Normally that's in seconds but in this case it seems it's in milliseconds.
You can convert a UNIX timestamp to a datetime using datetime.fromtimestamp. 1533953335000 must be divided by 1000 first because fromtimestamp expects seconds :
>>> from datetime import datetime
>>> datetime.fromtimestamp(1533953335000/1000)
datetime.datetime(2018, 8, 11, 5, 8, 55)
This returns a local time. Zulu is sometimes used to refer to UTC/+00:00 for historical reasons :
The time zone using UTC is sometimes denoted UTC±00:00 or by the letter Z—a reference to the equivalent nautical time zone (GMT), which has been denoted by a Z since about 1950. Time zones were identified by successive letters of the alphabet and the Greenwich time zone was marked by a Z as it was the point of origin. The letter also refers to the "zone description" of zero hours, which has been used since 1920 (see time zone history). Since the NATO phonetic alphabet word for Z is "Zulu", UTC is sometimes known as "Zulu time". This is especially true in aviation, where "Zulu" is the universal standard
To get UTC time, the datetime.utcfromtimestamp function is used :
>>> datetime.utcfromtimestamp(1533953335000/1000)
datetime.datetime(2018, 8, 11, 2, 8, 55)
Notice that the results are identical. Both methods return naive objects, ie objects with no timezone information. There's no way to tell if this is local or UTC, summer or winter time. The timezone must be passed explicitly to get an "aware" object :
>>> from datetime import timezone
>>> datetime.fromtimestamp(1533953335000/1000,timezone.utc)
datetime.datetime(2018, 8, 11, 2, 8, 55, tzinfo=datetime.timezone.utc)
Other languages and databases have explicit timezone-aware types, eg datetimeoffset
CODE:
import pytz
from django.utils import timezone
KST = pytz.timezone('Asia/Seoul')
UTC = pytz.timezone('UTC')
default_time = timezone.datetime(2021, 11, 29, 16, 44)
current_manual_kst = KST.localize(default_time)
current_manual_utc = default_time
print(current_manual_kst.timestamp())
print(current_manual_utc.timestamp())
RESULT:
>>> 1638171840.0
>>> 1638204240.0
So, I can see that results are different.
I thought timestamps should be the same but results are not.
Why this happened?
And How to get the same timestamps (by default: UTC) from KST.localized datetime?
A timestamp is expressed in UNIX time, which is the number of seconds since midnight January 1st 1970 UTC. In order to convert a datetime to such a UNIX timestamp, that datetime needs to be interpreted as some timezone. Because you can't express it relative to 1970 UTC without defining what timezone it's in. So if you have a naïve datetime object (without timezone) and take its timestamp(), it is interpreted as being in your local timezone and is converted to UTC from there.
And 16:44 in Seoul is apparently a different time than 16:44 in your computer's "local" timezone.
I want to know about the difference between time.time() and datetime.datetime.utcnow(). Does both returns the current UTC time?
The time.time() function returns the number of seconds since the epoch, as seconds. Note that the "epoch" is defined as the start of January 1st, 1970 in UTC. So the epoch is defined in terms of UTC and establishes a global moment in time. No matter where you are "seconds past epoch" (time.time()) returns the same value at the same moment.
You can read more about time here.
Regarding datetime, datetime.datetime.utcnow() returns the current UTC date and time, with tzinfo None. This is like now(), but returns the current UTC date and time, as a naive datetime object. An aware current UTC datetime can be obtained by calling datetime.now(timezone.utc)
Here is some sample output I ran on my computer, converting it to a string as well:
>>> import time
>>> ts = time.time()
>>> print(ts)
1583748843.6486485
>>> import datetime
>>> st = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
>>> print(st)
2020-03-09 12:14:03
Does both returns the current UTC time?
Yes and no.
The thing is how they return it.
Module datetime and classes in it (including datetime as class, which you use here) are for operating on date and time - thus they store the date/time in a structured form (object) and allow operations on it (methods).
>>> import datetime
>>> datetime.datetime.utcnow()
datetime.datetime(2020, 3, 9, 10, 4, 4, 284553)
While time module is more about straight C-like time operations. In this case: just returning timestamp (seconds from epoch) - as float (not a special class object).
>>> import time
>>> time.time()
1583748064.798787
I have dt = datetime(2013,9,1,11), and I would like to get a Unix timestamp of this datetime object.
When I do (dt - datetime(1970,1,1)).total_seconds() I got the timestamp 1378033200.
When converting it back using datetime.fromtimestamp I got datetime.datetime(2013, 9, 1, 6, 0).
The hour doesn't match. What did I miss here?
solution is
import time
import datetime
d = datetime.date(2015,1,5)
unixtime = time.mktime(d.timetuple())
If you want to convert a python datetime to seconds since epoch you should do it explicitly:
>>> import datetime
>>> datetime.datetime(2012, 04, 01, 0, 0).strftime('%s')
'1333234800'
>>> (datetime.datetime(2012, 04, 01, 0, 0) - datetime.datetime(1970, 1, 1)).total_seconds()
1333238400.0
In Python 3.3+ you can use timestamp() instead:
>>> import datetime
>>> datetime.datetime(2012, 4, 1, 0, 0).timestamp()
1333234800.0
What you missed here is timezones.
Presumably you've five hours off UTC, so 2013-09-01T11:00:00 local and 2013-09-01T06:00:00Z are the same time.
You need to read the top of the datetime docs, which explain about timezones and "naive" and "aware" objects.
If your original naive datetime was UTC, the way to recover it is to use utcfromtimestamp instead of fromtimestamp.
On the other hand, if your original naive datetime was local, you shouldn't have subtracted a UTC timestamp from it in the first place; use datetime.fromtimestamp(0) instead.
Or, if you had an aware datetime object, you need to either use a local (aware) epoch on both sides, or explicitly convert to and from UTC.
If you have, or can upgrade to, Python 3.3 or later, you can avoid all of these problems by just using the timestamp method instead of trying to figure out how to do it yourself. And even if you don't, you may want to consider borrowing its source code.
(And if you can wait for Python 3.4, it looks like PEP 341 is likely to make it into the final release, which means all of the stuff J.F. Sebastian and I were talking about in the comments should be doable with just the stdlib, and working the same way on both Unix and Windows.)
Rather than this expression to create a POSIX timestamp from dt,
(dt - datetime(1970,1,1)).total_seconds()
Use this:
int(dt.strftime("%s"))
I get the right answer in your example using the second method.
EDIT: Some followup... After some comments (see below), I was curious about the lack of support or documentation for %s in strftime. Here's what I found:
In the Python source for datetime and time, the string STRFTIME_FORMAT_CODES tells us:
"Other codes may be available on your platform.
See documentation for the C library strftime function."
So now if we man strftime (on BSD systems such as Mac OS X), you'll find support for %s:
"%s is replaced by the number of seconds since the Epoch, UTC (see mktime(3))."
Anyways, that's why %s works on the systems it does. But there are better solutions to OP's problem (that take timezones into account). See #abarnert's accepted answer here.
For working with UTC timezones:
time_stamp = calendar.timegm(dt.timetuple())
datetime.utcfromtimestamp(time_stamp)
You've missed the time zone info (already answered, agreed)
arrow package allows to avoid this torture with datetimes; It is already written, tested, pypi-published, cross-python (2.6 — 3.xx).
All you need: pip install arrow (or add to dependencies)
Solution for your case
dt = datetime(2013,9,1,11)
arrow.get(dt).timestamp
# >>> 1378033200
bc = arrow.get(1378033200).datetime
print(bc)
# >>> datetime.datetime(2013, 9, 1, 11, 0, tzinfo=tzutc())
print(bc.isoformat())
# >>> '2013-09-01T11:00:00+00:00'
If your datetime object represents UTC time, don't use time.mktime, as it assumes the tuple is in your local timezone. Instead, use calendar.timegm:
>>> import datetime, calendar
>>> d = datetime.datetime(1970, 1, 1, 0, 1, 0)
>>> calendar.timegm(d.timetuple())
60
def dt2ts(dt, utc=False):
if utc:
return calendar.timegm(dt.timetuple())
if dt.tzinfo is None:
return int(time.mktime(dt.timetuple()))
utc_dt = dt.astimezone(tz.tzutc()).timetuple()
return calendar.timegm(utc_dt)
If you want UTC timestamp :time.mktime just for local dt .Use calendar.timegm is safe but dt must the utc zone so change the zone to utc. If dt in UTC just use calendar.timegm.
def datetime_to_epoch(d1):
"""
January 1st, 1970 at 00:00:00 UTC is referred to as the Unix epoch
:param d1: input date
:return: seconds since unix epoch
"""
if not d1.tzinfo:
raise ValueError("date is missing timezone information")
d2 = datetime(1970, 1, 1, tzinfo=timezone.utc)
time_delta = d1 - d2
ts = int(time_delta.total_seconds())
return ts
def epoch_to_datetime_string(timestamp, tz_name="UTC", **kwargs):
"""
method to convert unix timestamp to date time string
:param ts: 10 digit unix timestamp in seconds
:param tz_name: timezone name
:param kwargs: formatter=<formatter-string>
:return: date time string in timezone
"""
naive_date = datetime.fromtimestamp(timestamp)
aware_date = naive_date.astimezone(pytz.timezone(tz_name))
formatter = kwargs.pop("formatter", "%d %b %Y %H:%M:%S")
return aware_date.strftime(formatter)
Well, when converting TO unix timestamp, python is basically assuming UTC, but while converting back it will give you a date converted to your local timezone.
See this question/answer;
Get timezone used by datetime.datetime.fromtimestamp()
This class will cover your needs, you can pass the variable into ConvertUnixToDatetime & call which function you want it to operate based off.
from datetime import datetime
import time
class ConvertUnixToDatetime:
def __init__(self, date):
self.date = date
# Convert unix to date object
def convert_unix(self):
unix = self.date
# Check if unix is a string or int & proceeds with correct conversion
if type(unix).__name__ == 'str':
unix = int(unix[0:10])
else:
unix = int(str(unix)[0:10])
date = datetime.utcfromtimestamp(unix).strftime('%Y-%m-%d %H:%M:%S')
return date
# Convert date to unix object
def convert_date(self):
date = self.date
# Check if datetime object or raise ValueError
if type(date).__name__ == 'datetime':
unixtime = int(time.mktime(date.timetuple()))
else:
raise ValueError('You are trying to pass a None Datetime object')
return type(unixtime).__name__, unixtime
if __name__ == '__main__':
# Test Date
date_test = ConvertUnixToDatetime(datetime.today())
date_test = date_test.convert_date()
print(date_test)
# Test Unix
unix_test = ConvertUnixToDatetime(date_test[1])
print(unix_test.convert_unix())
import time
from datetime import datetime
time.mktime(datetime.now().timetuple())
This question already has an answer here:
pytz.astimezone not accounting for daylight savings?
(1 answer)
Closed 7 years ago.
I'm writing a python script which contains two lines of code converting the date that got passed into the method to UTC time:
print "Timezone: %s" % get_localzone()
date = datetime.now(tz=get_localzone())
print "Local time: %s" % date
utc = pytz.utc
utc_date = date.astimezone(utc)
print "UTC date: %s" % utc_date
and the result is:
Timezone: America/Chicago
Local time: 2015-06-17 14:58:45.224827-05:00
UTC date: 2015-06-17 19:58:45.224827+00:00
As you can see the offset in local time is "-05:00", nothing wrong with it, but when I create a customized datetime object with the same timezone:
date = datetime(2015, 6, 17, 14, 58, 45, tzinfo=get_localzone())
The result becomes:
Timezone: America/Chicago
Local time: 2015-06-17 14:58:45-05:51
The offset changed from "-05:00" to "-05:51". I even used the same time that the first "datetime.now()" generated, and the timezone did not change, would someone please explain to me why is this happening? Thanks!
Instead of assigning the tzinfo parameter, use the localize method from pytz.
tz = get_localzone()
date = tz.localize(datetime(2015, 6, 17, 14, 58, 45))
This is discussed prominently in the pytz documentation, starting with the the first "Note" box, and in the very first code sample.
It's also shown in the tzlocal documentation, which is where (I assume) your get_localzone() method is coming from.
FYI, the -05:51 offset comes from the original LMT value of the America/Chicago time zone, which is -05:50:36 and is assumed to have been in use way back in 1883 as shown here. It's rounded to the nearest minute, giving the -05:51 LMT value in Python. You are seeing that offset because the localize method wasn't called, so pytz is just using the first offset known to that time zone's entry.