I want to store a list of datetimes in a binary file in Python.
EDIT: by "binary" I mean the best digital representation for each datatype. The application for this is to save GPS trackpoints composed by (unix-timestamp, latitude, longitude, elevation), so the whole structure is little-endian "Long, float, float, float", with four bytes to each value.
NOTE: I don't use "unix-timestamp" due to any affection to the Unix platform, but only as an unequivocal way to represent the value of a datetime.
Currently, I am doing like the code below, but besides some timezone confusion that I'm still working out (my timezone is -3), I believe converting to int and back might not be the right way, since datetime and datetime64 are native types in python/numpy, if I'm not mistaken. Thus, a datetime64 would need eight bytes instead of the four I am using for the (long)unix-timestamp.
import datetime
import calendar
import struct
now = datetime.datetime.now()
print now
stamp = calendar.timegm(now.utctimetuple())
print stamp
binarydatetime = struct.pack('<L', stamp)
recoverstamp = struct.unpack('<L', binarydatetime)[0]
print recoverstamp
recovernow = datetime.datetime.fromtimestamp(recoverstamp)
print recovernow
So the main question is: "is this the pythonic way to converting naive datetime to binary and back?"
And the aditional question is: "if everything in this code is supposed to be naive, why do I have a timezone offset?"
Thanks for reading!
I have found a way using the Unix timestamp and storing it as an integer. This works for me because I don't need a subsecond resolution, but I think long integers would allow for microsecond resolution with some modifications of the code.
The changes from my original consist in replacing calendar.timegm by time.mktime and also utctimetuple by timetuple, to keep everything naive.
This:
import datetime
import struct
import time
now = datetime.datetime.now()
print now
stamp = time.mktime(now.timetuple())
print stamp
recoverstamp = datetime.datetime.fromtimestamp(stamp)
print recoverstamp
binarydatetime = struct.pack('<L', stamp)
recoverbinstamp = struct.unpack('<L', binarydatetime)[0]
print recoverbinstamp
recovernow = datetime.datetime.fromtimestamp(recoverbinstamp)
print recovernow
Returns this:
2013-09-02 11:06:28.064000
1378130788.0
2013-09-02 11:06:28
1378130788
2013-09-02 11:06:28
From this, I can easily write the packed binarydatetime to file, and read it back later.
By far the most simple solution is to use Temporenc: http://temporenc.readthedocs.org/
It takes care of all the encoding/decoding and allows you to write a Python datetime object to a file:
now = datetime.datetime.now()
temporenc.pack(fp, now)
To read it back, this suffices:
dt = temporenc.unpack(fp)
print(dt)
If you want to save an object to a binary file, you may think about pickle
As far as I know, it writes a byte stream, so it's binary enough, but the result is
Python-specific
Also, I think it should work with datetime.
And you'll have this for saving:
with open('your_file', 'wb') as file:
pickle.Pickler(file).dump(your_var)
And this for recovering
with open('your_file', 'rb') as file:
recovered=pickle.Unpickler(file).load()
Related
This is my data :
dates = np.arange("2018-01-01", "2021-12-31", dtype="datetime64[D]")
I now want to convert from :
"2018-01-01" -> "Jan-01-2018" ["Monthname-day-year"] format
How to i do this ?
Is it possible to initialize this in the way we want to convert ?
Can i use something like:
for i in dates:
i = i.replace(i.month,i.strftime("%b"))
You can try this:
from datetime import datetime
dates = np.arange("2018-01-01", "2021-12-31", dtype="datetime64[D]")
result_dates = []
for date in dates.astype(datetime):
result_dates.append(date.strftime("%b-%d-%Y"))
But you will need to convert result dates as shown in the code
I feel compelled to elaborate on Silvio Mayolo's very relevant but ostensibly ignored comment above. Python stores a timestamp as structure (see How does Python store datetime internally? for more information) Hence, the DateTime does not as such have a 'format'. A format only becomes necessary when you want to print the date because you must first convert the timestamp to a string. Thus, you do NOT need to initialise any format. You only need to declare a format when the time comes to print the timestamp.
While you CAN store the date as a string in your dataframe index in a specific format, you CANNOT perform time related functions on it without first converting the string back to a time variable. ie current_time.hour will return an integer with the current hour if current_time is a datetime variable but will crash if it is a string formatted as a timestamp (such as "2023-01-15 17:23").
This is important to understand, because eventually you will need to manipulate the variables and need to understand whether you are working with a time or a string.
Taking data from a CSV file, trying to change the dates to the right format (ie change from 2016-12-25 to 2016-12)
This is the code right now:
import csv
csvfile = open('XML_project.csv')
linesreader = csv.reader(csvfile, delimiter=',')
from re import sub
from decimal import Decimal
import statistics as s
import datetime
date = []
for l in linesreader:
#date manipulation
temp = l[4]
temp_two = datetime.datetime.strptime(temp, "%Y-%m")
date.append(twmp_two)
csvfile.close
It says the file has unconverted data and I don't know how to fix it
Please edit your post to put triple backticks around your code - it will preserve the formatting and indentation:
```
your code
goes here
```
That said, I think your problem is the call to strptime(). If you also include the full stack trace that came with the error this should be evident (with the strptime() call at the top of the stack). It might also help to see a few lines of example CSV data.
Anyway, you have this:
temp_two = datetime.datetime.strptime(temp, "%Y-%m")
Suppose you have the date 2021-12-25: strptime will only match the 2021-12 part. That is no doubt your aim, but strptime likes to parse the entire string - that way you have more confidence that you have a correct "%Y-%m" format string.
So you want:
temp_two = datetime.datetime.strptime(temp, "%Y-%m-%d")
That should match the whole date field (therefore no error). Then you want to produce a new YYYY-mm style string for your dates, from that datetime object, like this:
yyyy_mm = temp_two.strftime("%Y-%m")
which you can then store:
date.append(yyyy_mm)
Minor other remarks:
it is normal to put all the imports at the top of your file (makes them easy to see)
it is typical to import specific names from the datetime module because of the unfortunately same-named datetime class
Eg:
from datetime import datetime
which lets you use datetime.strptime() instead of the more cumbersome datetime.datetime.strptime().
So this question is more of best way to handle this sort of input in python. Here is an example of input date 2018-12-31 23:59:59.999999. The millisecond part may or may not be part of input.
I am currently using this code to convert this to datetime
input_ts = datetime.datetime.strptime(input_str, '%Y-%m-%dT%H:%M:%S.%f')
But the problem in this case is that it will throw an exception if input string doesn't contain milliseconds part i.e., 2018-12-31 23:59:59
In Java, I could have approached this problem in two ways. (its a pseudo explanation, without taking into account of small boundary checks)
(preferred approach). Check the input string length. if its less than 19 then it is missing milliseconds. Append .000000 to it.
(not preferred). Let the main code parse the string, if it throws an exception, then parse it with new time format i.e., %Y-%m-%dT%H:%M:%S
The third approach could be just strip off milliseconds.
I am not sure if python has anything built-in to handle these kind of situations. Any suggestions?
You could use python-dateutil library, it is smart enough to parse most of the basic date formats.
import dateutil.parser
dateutil.parser.parse('2018-12-31 23:59:59.999999')
dateutil.parser.parse('2018-12-31 23:59:59')
In case you don't want to install any external libraries, you could iterate over list of different formats as proposed in this answer.
from datetime import datetime # import datetime class from datetime package
dt = datetime.now() # get current time
dt1 = dt1.strftime('%Y-%m-%d %H:%M:%S') # converting time to string
dt3 = dt2.strptime('2018/5/20','%Y/%m/%d') # converting a string to specified time
I have a python script that generates a datetime string using this line of code:
data['timestamp'] = datetime.isoformat(datetime.utcnow())
That generates something like the following:
2017-05-24T04:08:09.530033
How do I convert that to "MYSQL insertable" datetime format in a clean way?
Thanks!
Try to use MySQL's STR_TO_DATE() function to parse the string that you're attempting to insert.
I hope this may help you
You can specify any type of format like this depending on the one you `ve set in mysql
data['timestamp'] =pd.to_datetime(data['timestamp'] , format='%d%b%Y:%H:%M:%S.%f')
First off, it looks like you ran from datetime import * rather than import datetime. That's tempting because it lets you type less when you want to refer to parts of the module, but it can get you into name collision issues later. An alternative with less typing is something like import datetime as dt, that way later you can just use dt.datetime. This will make your code cleaner.
MySQL accepts several date formats, which can be read about in detail here. In particular:
The DATETIME type is used for values that contain both date and time
parts. MySQL retrieves and displays DATETIME values in YYYY-MM-DD HH:MM:SS format.
ISO8601 numbers look just like that! 2017-05-24T04:19:32
So if the only difference is the "T" in the middle instead of a space, just run something like this, assuming you don't change your import statements.
timestamp = str(datetime.isoformat(datetime.utcnow()))
timestamp = timestamp.replace("T", " ")
data['timestamp'] = timestamp
I would like a simple way to find and reformat text of the format 'DD/MM/YYYY' into 'YYYY/MM/DD' to be compatible with MySQL TIMESTAMPs, in a list of text items that may or may not contain a date atall, under python. (I'm thinking RegEx?)
Basically i am looking for a way to inspect a list of items and correct any timestamp formats found.
Great thing about standards is that there are so many to choose from....
You can read the string into a datetime object and then output it back as a string using a different format. For e.g.
>>> from datetime import datetime
>>> datetime.strptime("31/12/2009", "%d/%m/%Y").strftime("%Y/%m/%d")
'2009/12/31'
Basically i am looking for a way to inspect a list of items and correct any timestamp formats found.
If the input format is inconsistent, can vary, then you are better off with dateutil.
>>> from dateutil.parser import parse
>>> parse("31/12/2009").strftime("%Y/%m/%d")
'2009/12/31'
Dateutil can handle a lot of input formats automatically. To operate on a list you can map the a wrapper over the parse function over the list and convert the values appropriately.
If you're using the MySQLdb (also known as "mysql-python") module, for any datetime or timestamp field you can provide a datetime type instead of a string. This is the type that is returned, also and is the preferred way to provide the value.
For Python 2.5 and above, you can do:
from datetime import datetime
value = datetime.strptime(somestring, "%d/%m/%Y")
For older versions of python, it's a bit more verbose, but not really a big issue.
import time
from datetime import datetime
timetuple = time.strptime(somestring, "%d/%m/%Y")
value = datetime(*timetuple[:6])
The various format-strings are taken directly from what's accepted by your C library. Look up man strptime on unix to find other acceptable format values. Not all of the time formats are portable, but most of the basic ones are.
Note datetime values can contain timezones. I do not believe MySQL knows exactly what to do with these, though. The datetimes I make above are usually considered as "naive" datetimes. If timezones are important, consider something like the pytz library.