python: Calculating seconds from a datetime string with extra characters - python

I want to create a new column which contains seconds since 1970 for each row for the following input file:
timestamp, air_temp, rh, pressure, dir, spd
2016-11-30T00:00:00Z,-36.50,56.00,624.60,269.00,5.80
2016-11-30T01:00:00Z,-35.70,55.80,624.70,265.00,5.90
2016-11-30T02:00:00Z,-34.80,56.00,625.00,266.00,6.30
The first column represents the timestamp but it contains extra characters 'T' and 'Z'. My current code looks like this:
i = 0
ip_file.readline()
for line in ip_file:
line = line.strip()
year[i] = int(line[0:4])
month[i] = int(line[5:7])
day[i] = int(line[8:10])
hour[i] = int(line[11:13])
time[i] = (datetime(year[i],month[i],day[i],hour[i])-datetime(1970, 1, 1)).total_seconds()
i += 1
This returns me what I want but it takes long time if input file is big. If the timestamp didn't had those extra characters, I would have directly used it instead of calculating year, month, day and hour. Is there a better way? Any thoughts would be appreciated.

Instead of using string slice. Why not split the string by comma? And use strptime method in datetime module to convert string datetime to datetime object.
Example:
import datetime
with open(path, "r") as infile:
for i in infile.readlines()[1:]:
dVal = i.strip().split(",")[0]
print (datetime.datetime.strptime(dVal, '%Y-%m-%dT%H:%M:%SZ')-datetime.datetime(1970, 1, 1)).total_seconds()
Output:
1480464000.0
1480467600.0
1480471200.0

Input:
import datetime as dt
line = '2016-11-30T00:00:00Z,-36.50,56.00,624.60,269.00,5.80'
# We know the datetime data is always 20 characters long
line_dt_str = line[:20]
line_secs_since_epoch = dt.datetime.strptime(line_dt_str, '%Y-%m-%dT%H:%M:%SZ').timestamp()
print(line_secs_since_epoch)
Output:
1480482000.0
Note that there is a difference between calling .timestamp() and subtracting your datetime from the 1970 epoch. This comes from how these two methods handle (or don't handle) daylight savings time. Read more here

You can achieve this by first splitting your line in file on , and casting it to datetime object
>>> import datetime
>>> line = '2016-11-30T00:00:00Z,-36.50,56.00,624.60,269.00,5.80'
>>> t = datetime.strptime(line.split(',')[0], '%Y-%m-%dT%H:%M:%SZ')
To convert to seconds you can simply use:
>>> int(t.strftime("%s"))
>>> 1480435200

Related

How to change Day/Month/Year format to Month/Day/Year in Python

I have a column that the date is in Day/Month/Year format and it is stored in object format. How can I change it to Month/Day/Year format in python?
Here is an example: How can I change 13/3/2021 to 3/13/2021?
Simple solution is using split:
def convert_format1(s: str):
d, m, y = s.split("/")
return "/".join((m, d, y))
Or you can use datetime module to convert string to datetime object(.strptime) and vice versa(.strftime):
from datetime import datetime
def convert_format2(s: str):
source_format = "%d/%m/%Y"
destination_format = "%m/%d/%Y"
d = datetime.strptime(s, source_format)
return d.strftime(destination_format)
You can then apply these functions to your dataframe's column.
note: AFAIK .strftime() method adds zero padding to the string representation of day and month. If you don't want this behavior you have to strip it manually after that.
note: second function is safer since it checks the dates to be valid as well.
import datetime
t = datetime.date.today()
print(t.month,t.day,t.year,sep="/")
change the value inside the print should let you set yourself

Python - Time delta from string and now()

I have spent some time trying to figure out how to get a time delta between time values. The only issue is that one of the times was stored in a file. So I have one string which is in essence str(datetime.datetime.now()) and datetime.datetime.now().
Specifically, I am having issues getting a delta because one of the objects is a datetime object and the other is a string.
I think the answer is that I need to get the string back in a datetime object for the delta to work.
I have looked at some of the other Stack Overflow questions relating to this including the following:
Python - Date & Time Comparison using timestamps, timedelta
Comparing a time delta in python
Convert string into datetime.time object
Converting string into datetime
Example code is as follows:
f = open('date.txt', 'r+')
line = f.readline()
date = line[:26]
now = datetime.datetime.now()
then = time.strptime(date)
delta = now - then # This does not work
Can anyone tell me where I am going wrong?
For reference, the first 26 characters are acquired from the first line of the file because this is how I am storing time e.g.
f.write(str(datetime.datetime.now())
Which would write the following:
2014-01-05 13:09:42.348000
time.strptime returns a struct_time.
datetime.datetime.now() returns a datetime object.
The two can not be subtracted directly.
Instead of time.strptime you could use datetime.datetime.strptime, which returns a datetime object. Then you could subtract now and then.
For example,
import datetime as DT
now = DT.datetime.now()
then = DT.datetime.strptime('2014-1-2', '%Y-%m-%d')
delta = now - then
print(delta)
# 3 days, 8:17:14.428035
By the way, you need to supply a date format string to time.strptime or DT.datetime.strptime.
time.strptime(date)
should have raised a ValueError.
It looks like your date string is 26 characters long. That might mean you have a date string like 'Fri, 10 Jun 2011 11:04:17 '.
If that is true, you may want to parse it like this:
then = DT.datetime.strptime('Fri, 10 Jun 2011 11:04:17 '.strip(), "%a, %d %b %Y %H:%M:%S")
print(then)
# 2011-06-10 11:04:17
There is a table describing the available directives (like %Y, %m, etc.) here.
Try this:
import time
import datetime
d = datetime.datetime.now()
now = time.mktime(d.timetuple())
And then apply the delta
if you have the year,month,day of 'then' you may use:
year = 2013
month = 1
day = 1
now_date = datetime.datetime.now()
then_date = now_date.replace(year = year, month = month, day = day)
delta = now_date - then_date

Two-Digit dates in Python

I am trying to print the date and month as 2 digit numbers.
timestamp = date.today()
difference = timestamp - datetime.timedelta(localkeydays)
localexpiry = '%s%s%s' % (difference.year, difference.month, difference.day)
print localexpiry
This gives the output as 201387. This there anyway to get the output as 20130807. This is because I am comparing this against a string of a similar format.
Use date formatting with date.strftime():
difference.strftime('%Y%m%d')
Demo:
>>> from datetime import date
>>> difference = date.today()
>>> difference.strftime('%Y%m%d')
'20130807'
You can do the same with the separate integer components of the date object, but you need to use the right string formatting parameters; to format an integer to two digits with leading zeros, use %02d, for example:
localexpiry = '%04d%02d%02d' % (difference.year, difference.month, difference.day)
but using date.strftime() is more efficient.
You can also use format (datetime, date have __format__ method):
>>> import datetime
>>> dt = datetime.date.today()
>>> '{:%Y%m%d}'.format(dt)
'20130807'
>>> format(dt, '%Y%m%d')
'20130807'
two digit month, day and four digit year in django like this 04/14/2019: {{text1.pub_date|date:"m/d/Y"}}

Slicing and Replacing Unicode String Characters

I have the following loop I'm trying to use to replace characters in a unicode string. The data I'm getting for this loop is in the following format: YYYY-MM-DD HH:MM:SS
This data is apparently stored in UTC, so when I grab it and append these times & dates to my list appts_list its 4 hours ahead.
I've gotten as far as slicing the unicode string and doing the math on these characters and getting what would be the correct hour I need, but I'm having a problem getting that back into a string so I can write it to my list appts_list.
I'm getting TypeError when I try to write the integer for the correct hour time_slice_int back into the original string. I decided to try to put the entire string into a list and change them there, but that isn't working either.
Ideally I want an appointment for '2013-06-28 15:30:00' to be entered into my appts_list as '2013-06-28 11:30:00'.
The print statements are there for me to debug as I ran it. They are not necessary for the final version.
Anyone have any suggestions or solutions?
for appt in todays_appts:
time = appt['apptdateourtime_c']
time_slice = time[11:13]
time_slice_int = int(time_slice)
time_slice_int -= 4
print(time_slice_int)
appt_time = list(time)
print(appt_time)
print(appt_time[11:13])
#appt_time[11:13] = time_slice_int
#appts_list.append()
print('AppointmentScheduled')
#print(appt['apptdateourtime_c'])
#print(time)
print('')
You should use the datetime module here:
>>> from datetime import datetime, timedelta
>>> strs = '2013-06-28 15:30:00'
>>> d = datetime.strptime(strs, "%Y-%m-%d %H:%M:%S")
datetime.strptime returns a datetime object:
>>> d
datetime.datetime(2013, 6, 28, 15, 30)
>>> d.hour
15
>>> d.month
6
Now decrease 4 hours from the above datetime object(d) using timedelta and assign the new object to a variable:
>>> d1 = d - timedelta(hours = 4)
Now use datetime.strftime to get a string of required format:
>>> datetime.strftime(d1,"%Y-%m-%d %H:%M:%S")
'2013-06-28 11:30:00'

Read in Dates from file to be used in plots

I am trying to read date stamps in from a data logger and use these dates in plots. I have been playing with matplotlib dates date2num, datestr2num, and datetime but I keep getting formatting errors and am having trouble finding what the correct syntax and keywords are to do this (and also what they mean). I have been reading through the matplotlib help with not much luck. If you have any help or a better way to read in this information I would love the feedback.
import numpy as n
import matplotlib.pyplot as p
import matplotlib.dates as d
import datetime as dt
fileobj=open("filename",'r')
data=fileobj.readlines()
fileobj.close()
time=n.empty(len(data))
for i in range(len(data)):
strings=data[i].split(',')
if i >5:
some_time_dt = dt.datetime.strptime(str(strings[0]), '%Y-%m-%d %H:%M:%S')
time = d.date2num(some_time_dt)
Example data:
"2013-02-28 16:53:30",1588,11.85,24.35,22.93,24.1,25.05,22.06,22.2,30.94,21.99,22.7,21.91,22‌​.02,21.79 ,21.72
"2013-02-28 16:53:31",1589,11.85,24.35,23,24.12,25.05,22.09,22.25,31.19,21.97,22.71,21.91,22‌​.02,21.78 ,21.72
"2013-02-28 16:53:32",1590,11.85,24.35,22.98,24.12,25.05,22.12,22.3,31.35,21.98,22.68,21.9,2‌​2.01,21.7 4,21.69
"2013-02-28 16:53:33",1591,11.85,24.35,22.95,24.14,25.06,22.15,22.33,31.49,21.96,22.67,21.87‌​,22,21.73 ,21.66
March 20,2013
I was able to get this to plot but I need to know how to get rid of the UTC label that prints as the time is not in UTC but in PST. I would prefer to just not show a timezone at all.
A simple solution would be to parse the file twice, once for the dates and once for the data:
import numpy as np
import datetime as dt
D = np.loadtxt("filename",delimiter=",",usecols=[0],dtype="str")
Z = np.loadtxt("filename",delimiter=",",usecols=range(1,10))
DATES = [dt.datetime.strptime(d,'"%Y-%m-%d %H:%M:%S"') for d in D]
You could also use the converters argument to pass a lambda function to loadtxt() so that it does the string to datetime object conversion for you. It doesn't save you any lines of code, I'm just noting it for a bit of variety:
datey = lambda x: dt.datetime.strptime(x,'"%Y-%m-%d %H:%M:%S"')
D = np.loadtxt("filename",delimiter=",",usecols=[0],
dtype=dt.datetime,converters={0:datey})
Z = np.loadtxt("filename",delimiter=",",usecols=range(1,10))
It sounds like you are having errors with parsing the dates. There may be a problem with the date format string you are using.
There is a table with all the options available for datetime parsing here:
http://docs.python.org/2/library/datetime.html#strftime-strptime-behavior
I normally do this kind of thing with pandas (see my comment above) but here's a rough solution using Python's built-in CSV module.
import csv
import datetime
data = []
for row in csv.reader(open('file.txt')):
row[0] = datetime.datetime.strptime(row[0], '%Y-%m-%d %H:%M:%S') # Parse date.
row[1:] = map(float, row[1:]) # Convert data from strings to floats.
data.append(row)
There are fancier ways, but is a straightforward approach.
Running this on the data above, I get
[[datetime.datetime(2013, 2, 28, 16, 53, 30), 1588.0, 11.85...], ...]
You need to strip the " from you time string!
If there's only one time per line , i would change the for loop for something like this:
import time as time_module # there's is a var named time
lineNumber = 0
for line in data:
lineNumber += 1
if line <= 5:
continue # skip the first 5 lines
line = line.split(',')
timeString = line[0].strip('"')
print timeString
print time_module.strptime(timeString, '%Y-%m-%d %H:%M:%S')
time = d.date2num(time_module.strptime(timeString, '%Y-%m-%d %H:%M:%S'))

Categories

Resources