I've been trying to figure out how to generate the same Unix epoch time that I see within InfluxDB next to measurement entries.
Let me start by saying I am trying to use the same date and time in all tests:
April 01, 2017 at 2:00AM CDT
If I view a measurement in InfluxDB, I see time stamps such as:
1491030000000000000
If I view that measurement in InfluxDB using the -precision rfc3339 it appears as:
2017-04-01T07:00:00Z
So I can see that InfluxDB used UTC
I cannot seem to generate that same timestamp through Python, however.
For instance, I've tried a few different ways:
>>> calendar.timegm(time.strptime('04/01/2017 02:00:00', '%m/%d/%Y %H:%M:%S'))
1491012000
>>> calendar.timegm(time.strptime('04/01/2017 07:00:00', '%m/%d/%Y %H:%M:%S'))
1491030000
>>> t = datetime.datetime(2017,04,01,02,00,00)
>>> print "Epoch Seconds:", time.mktime(t.timetuple())
Epoch Seconds: 1491030000.0
The last two samples above at least appear to give me the same number, but it's much shorter than what InfluxDB has. I am assuming that is related to the precision, InfluxDB does things down to nanosecond I think?
Python Result: 1491030000
Influx Result: 1491030000000000000
If I try to enter a measurement into InfluxDB using the result Python gives me it ends up showing as:
1491030000 = 1970-01-01T00:00:01.49103Z
So I have to add on the extra nine 0's.
I suppose there are a few ways to do this programmatically within Python if it's as simple as adding on nine 0's to the result. But I would like to know why I can't seem to generate the same precision level in just one conversion.
I have a CSV file with tons of old timestamps that are simply, "4/1/17 2:00". Every day at 2 am there is a measurement.
I need to be able to convert that to the proper format that InfluxDB needs "1491030000000000000" to insert all these old measurements.
A better understanding of what is going on and why is more important than how to programmatically solve this in Python. Although I would be grateful to responses that can do both; explain the issue and what I am seeing and why as well as ideas on how to take a CSV with one column that contains time stamps that appear as "4/1/17 2:00" and convert them to timestamps that appear as "1491030000000000000" either in a separate file or in a second column.
InfluxDB can be told to return epoch timestamps in second precision in order to work more easily with tools/libraries that do not support nanosecond precision out of the box, like Python.
Set epoch=s in query parameters to enable this.
See influx HTTP API timestamp format documentation.
Something like this should work to solve your current problem. I didn't have a test csv to try this on, but it will likely work for you. It will take whatever csv file you put where "old.csv" is and create a second csv with the timestamp in nanoseconds.
import time
import datetime
import csv
def convertToNano(date):
s = date
secondsTimestamp = time.mktime(datetime.datetime.strptime(s, "%d/%m/%y %H:%M").timetuple())
nanoTimestamp = str(secondsTimestamp).replace(".0", "000000000")
return nanoTimestamp
with open('old.csv', 'rb') as old_csv:
csv_reader = csv.reader(old_csv)
with open('new.csv', 'wb') as new_csv:
csv_writer = csv.writer(new_csv)
for i, row in enumerate(csv_reader):
if i != 0:
# Put whatever rows the data appears in and the row you want the data to go in here
row.append(convertToNano(row[<location of date in the row>]))
csv_writer.writerow(row)
As to why this is happening, after reading this it seems like you aren't the only one getting frustrated by this issue. It seems as though influxdb just happens to be using a different type of precision then most python modules. I didn't really see any way to get around it than doing the string manipulation of the date conversion unfortunately.
Related
I'm relatively uninitiated when it comes to Python, and I'm trying to figure out how to take an output I'm getting from a sensor into proper day, month, year and hour, minute, second format.
An example of the output, which also includes a basic counter (the first output), and a timestamp (the third output) is shown below:
(305, struct_time(tm_year=2022, tm_mon=11, tm_mday=9, tm_hour=16, tm_min=42, tm_sec=8, tm_wday=2, tm_yday=313, tm_isdst=-1), 7.036)
I've seen a lot of questions and answers for this, but I'm left feeling kind of stumped on all of them because I'm not sure how to take the output I have (real_time, which gives a struct_time output) and turn it into this format. Any help (and understanding about my lack of fluency in this field) would be really appreciated!
time.strftime exists for exactly this purpose:
import time
now_local = time.localtime()
fmt = "%d/%m/%Y %H:%M:%S"
out = time.strftime(fmt, now_local)
print(out)
However, two words of warning:
time.struct_time is not "timezone aware". This will turn out to matter when you least expect it. Unless you are very sure that you know the timezone of the incoming data, and have the correct safeguards in your application and database for managing time zone iformation, use the datetime.datetime class instead.
D/M/Y date format can be ambiguous. Y-M-D format is substantially safer. It is not ambiguous in any widely-used locale, and it has the extra benefit that lexical ordering of Y-M-D strings is also a correct ordering of the dates that they represent. This format is laid out by RFC 3339 and has become widely accepted as the standard, correct formatting for datetime strings.
So as it turns out, I was able to find a solution after all. Essentially I just used this function:
def _format_datetime(datetime):
return "{:02}/{:02}/{} {:02}:{:02}:{:02}".format(
datetime.tm_mon,
datetime.tm_mday,
datetime.tm_year,
datetime.tm_hour,
datetime.tm_min,
datetime.tm_sec,
)
And then applied it to the struct_time output as such (with real_time being said output):
real_time = time.localtime()
current_time = time.monotonic()
formatted_time = _format_datetime(real_time)
Hopefully this helps other people using CircuitPython for similar purposes!
I am trying to achieve a script, which will create an Orange data table with just a single column containing a custom time stamp.
Usecase: I need a complete time stamp so I can merge some other csv files later on. I'm working in the Orange GUI BTW and am not working in the actual python shell or any other IDE (in case this information makes any difference).
Here's what I have come up with so far:
From Orange.data import Domain, Table, TimeVariable
import numpy as np
domain = Domain([TimeVariable("Timestamp")])
# Timestamp from 22-03-08 to 2022-03-08 in minute steps
arr = np.arange("2022-03-08", "2022-03-15", dtype="datetime64[m]")
# Obviously necessary to achieve a correct format for the matrix
arr = arr.reshape(-1,1)
out_data = Table.from_numpy(domain, arr)
However the results do not match:
>>> print(arr)
[['2022-03-08T00:00']
['2022-03-08T00:01']
['2022-03-08T00:02']
...
['2022-03-14T23:57']
['2022-03-14T23:58']
['2022-03-14T23:59']]
>>> print(out_data)
[[27444960.0],
[27444961.0],
[27444962.0],
...
[27455037.0],
[27455038.0],
[27455039.0]]
Obviously I'm missing something when handing over the data from numpy but I'm having a real hard time trying to understand the documentation.
I've also found this post which seems to tackle a similar issue, but I haven't figured out how to apply the solution on my problem.
I would be really glad if anyone could help me out here. Please try to use simple terms and concepts.
Thank you for the question, and apologies for the weak documentation of the TimeVariable.
In your code, you must change two things to work.
First, it is necessary to set whether the TimeVariable includes time and/or date data:
TimeVariable("Timestamp", have_date=True) stores only date information -- it is analogous to datetime.date
TimeVariable("Timestamp", have_time=True) stores only time information (without date) -- it is analogous to datetime.time
TimeVariable("Timestamp", have_time=True, have_date=True) stores date and time -- it is analogous to datetime.datetime
You didn't set that information in your example, so both were False by default. For your case, you must set both to True since your attribute will hold the date-time values.
The other issue is that Orange's Table stores date-time values as UNIX epoch (seconds from 1970-01-01), and so also Table.from_numpy expect values in this format. Values in your current arr array are in minutes instead. I just transformed the dtype in the code below to seconds.
Here is the working code:
from Orange.data import Domain, Table, TimeVariable
import numpy as np
# Important: set whether TimeVariable contains time and/or date
domain = Domain([TimeVariable("Timestamp", have_time=True, have_date=True)])
# Timestamp from 22-03-08 to 2022-03-08 in minute steps
arr = np.arange("2022-03-08", "2022-03-15", dtype="datetime64[m]").astype("datetime64[s]")
# necessary to achieve a correct format for the matrix
arr = arr.reshape(-1,1)
out_data = Table.from_numpy(domain, arr)
I have a dataframe with two columns with different times in string format, I want to find the difference between the two columns so I use the following code
operational_data_clean['Pick/pack start-time'] = pd.to_datetime(operational_data_clean['Pick/pack start-time'])
operational_data_clean['Flight launched-time'] = pd.to_datetime(operational_data_clean['Flight launched-time'])
operational_data_clean['time_to_launch'] = 0
operational_data_clean['time_to_launch'] = operational_data_clean['Flight launched-time'] - operational_data_clean['Pick/pack start-time']
but this code when I run the first time I get good results but when I run the second time it add todays date on the 'Pick/pack start-time' and 'Flight launched-time' value.
How can I convert this time only to hours without getting the dates that are messing my data.
I am assuming you are running your code with jupyter notebook.
When you execute your code, your variable operational_data_clean['Pick/pack start-time'] becomes pd.to_datetime(operational_data_clean['Pick/pack start-time']).
So when you execute the block one more time, jupyter remembers your variables and therefore will perform the same function again, essentially doing this:
pd.to_datetime(pd.to_datetime(operational_data_clean['Pick/pack start-time'])).
As for pd.to_datetime() itself, I would advise to look through the pandas docs.
This question already has answers here:
Convert weird Python date format to readable date
(2 answers)
Closed 7 years ago.
I'm importing data from an Excel spreadsheet into python. My dates are coming through in a bizarre format of which I am not familiar and cannot parse.
in excel: (7/31/2015)
42216
after I import it:
u'/Date(1438318800000-0500)/'
Two questions:
what format is this and how might I parse it into something more intuitive and easier to read?
is there a robust, swiss-army-knife-esque way to convert dates without specifying input format?
Timezones necessarily make this more complex, so let's ignore them...
As #SteJ remarked, what you get is (close to) the time in seconds since 1st January 1970. Here's a Wikipedia article how that's normally used. Oddly, the string you get seems to have a timezone (-0500, EST in North America) attached. Makes no sense if it's properly UNIX time (which is always in UTC), but we'll pass on that...
Assuming you can get it reduced to a number (sans timezone) the conversion into something sensible in Python is really straight-forward (note the reduction in precision; your original number is the number of milliseconds since the epoch, rather than the standard number of seconds from the epoch):
from datetime import datetime
time_stamp = 1438318800
time_stamp_dt = datetime.fromtimestamp(time_stamp)
You can then get time_stamp_dt into any format you think best using strftime, e.g., time_stamp_dt.strftime('%m/%d/%Y'), which pretty much gives you what you started with.
Now, assuming that the format of the string you provided is fairly regular, we can extract the relevant time quite simply like this:
s = '/Date(1438318800000-0500)/'
time_stamp = int(s[6:16])
Suppose there’s a sensor which records the date and time at every activation. I have this data stored as a list in a .json file in the format (e.g.) "2000-01-01T00:30:15+00:00".
Now, what I want to do is import this file in python and use NumPy/ Mathplotlib to plot how many times this sensor is activated per day.
My problem is, using this data, I don’t know how to write an algorithm which counts how many times the sensor is activated daily. (This should be simple, but due to limited Python knowledge, I’m stuck). Supposedly there is a way to split this list wrt T, bin each recording by date (e.g. “2000-01-01”) and then count the recordings on this date.
How would you count how many times the sensor is activated? (to then make a plot showing the number of activations each day?)
First of all you need to load your JSON file:
import json
with open("logfile.json", "r") as logfile:
records = json.load(logfile)
Records will be a list or dictionary containing your records.
Assuming that your logfile looks like:
[u"2000-01-01T00:30:15+00:00",
u"2000-01-01T00:30:16+00:00",
...
]
Records will be a list of strings. So parsing the dates is just:
import datetime
for record in records:
datepart, _ = record.split("T")
date = datetime.datetime.strptime(datepart, "%Y-%m-%d")
Hopefully that's clear enough. Using "string".split and datetime.strptime should do the trick, although you don't have to parse this into a date object just to bin it but it may make things easier later on.
Finally, binning should be pretty straightforward using a dictionary of lists. Starting
from what we've got above let's add binning:
import collections
import datetime
date_bins = collections.defaultdict(list)
for record in records:
datepart, _ = record.split("T")
date = datetime.datetime.strptime(datepart, "%Y-%m-%d")
date_bins[date].append(record)
This should give you a dictionary where each key is a date and each value is the list of records that were logged on that day.
You'll probably want to sort this by date (although you may be able to use collections.OrderedDict if the data is already in order).
Counting activations per day could be something like:
for date in date_bins:
print "activations on %s: %s"%(date, len(date_bins[date]))
Of course it's a little bit more work to take that information and massage it into a format that matplotlib needs but it shouldn't be too bad from here.
if your json file load a list like:
j_list = [('2000-01-01T00:30:15+00:00', 'xx'),
('2000-01-01T00:30:15+00:00', 'yyy'),
('2000-01-02T00:30:15+00:00', 'zzz')]
Note: this assumes the json file returns a list of lists with the timestamp as the first element. Adjust accordingly.
There are parsers in dateutil and datetime to parse the timestamp.
If counting is really all you are doing, even that might be overkill. You could:
>>> from itertools import groupby
>>> [(k,len(list(l))) for k,l in groupby(j_list,lambda x: x[0][:10])]
[('2000-01-01', 2), ('2000-01-02', 1)]