Taking data from a CSV file, trying to change the dates to the right format (ie change from 2016-12-25 to 2016-12)
This is the code right now:
import csv
csvfile = open('XML_project.csv')
linesreader = csv.reader(csvfile, delimiter=',')
from re import sub
from decimal import Decimal
import statistics as s
import datetime
date = []
for l in linesreader:
#date manipulation
temp = l[4]
temp_two = datetime.datetime.strptime(temp, "%Y-%m")
date.append(twmp_two)
csvfile.close
It says the file has unconverted data and I don't know how to fix it
Please edit your post to put triple backticks around your code - it will preserve the formatting and indentation:
```
your code
goes here
```
That said, I think your problem is the call to strptime(). If you also include the full stack trace that came with the error this should be evident (with the strptime() call at the top of the stack). It might also help to see a few lines of example CSV data.
Anyway, you have this:
temp_two = datetime.datetime.strptime(temp, "%Y-%m")
Suppose you have the date 2021-12-25: strptime will only match the 2021-12 part. That is no doubt your aim, but strptime likes to parse the entire string - that way you have more confidence that you have a correct "%Y-%m" format string.
So you want:
temp_two = datetime.datetime.strptime(temp, "%Y-%m-%d")
That should match the whole date field (therefore no error). Then you want to produce a new YYYY-mm style string for your dates, from that datetime object, like this:
yyyy_mm = temp_two.strftime("%Y-%m")
which you can then store:
date.append(yyyy_mm)
Minor other remarks:
it is normal to put all the imports at the top of your file (makes them easy to see)
it is typical to import specific names from the datetime module because of the unfortunately same-named datetime class
Eg:
from datetime import datetime
which lets you use datetime.strptime() instead of the more cumbersome datetime.datetime.strptime().
Related
I am trying to use timestamp information to graph.
However, when I convert the number into hh:mm:ss. It does not work.
I have tried this:
timestamp = [dt.strptime(str(datetime.timedelta(seconds=round(t/1000))),'%H:%M:%S') for t in timestamp1]
Also I tried this
timestamp = [dt.strftime(dt.strptime(str(datetime.timedelta(seconds=round(t / 1000))), '%H:%M:%S'), '%H:%M:%S') for t in timestamp]
However, it is possible to see the list with the new values. However, I have problems with the graphs and these new values.
Can anybody help me?
Try using this and make sure to import the time module!
timestamp = [time.strftime('%H:%M:%S', time.gmtime(t/1000)) for t in timestamp1]
In your code you're using datetime.strptime(date_string, format) function, it's main use is to parse time strings in order to create datetime object.
In order to get a parsed string of a desired format you should use date.strftime(format) function instead.
So basically in your code you can only add it and should get the wanted result:
import datetime
from datetime import datetime as dt
timestamp = [dt.strftime(dt.strptime(str(datetime.timedelta(seconds=round(t / 1000))), '%H:%M:%S'), '%H:%M:%S') for t in timestamp1]
I'm not sure what was your input and desired output, but you could also use date.fromtimestamp(timestamp) for the parsing of timestamps
So this question is more of best way to handle this sort of input in python. Here is an example of input date 2018-12-31 23:59:59.999999. The millisecond part may or may not be part of input.
I am currently using this code to convert this to datetime
input_ts = datetime.datetime.strptime(input_str, '%Y-%m-%dT%H:%M:%S.%f')
But the problem in this case is that it will throw an exception if input string doesn't contain milliseconds part i.e., 2018-12-31 23:59:59
In Java, I could have approached this problem in two ways. (its a pseudo explanation, without taking into account of small boundary checks)
(preferred approach). Check the input string length. if its less than 19 then it is missing milliseconds. Append .000000 to it.
(not preferred). Let the main code parse the string, if it throws an exception, then parse it with new time format i.e., %Y-%m-%dT%H:%M:%S
The third approach could be just strip off milliseconds.
I am not sure if python has anything built-in to handle these kind of situations. Any suggestions?
You could use python-dateutil library, it is smart enough to parse most of the basic date formats.
import dateutil.parser
dateutil.parser.parse('2018-12-31 23:59:59.999999')
dateutil.parser.parse('2018-12-31 23:59:59')
In case you don't want to install any external libraries, you could iterate over list of different formats as proposed in this answer.
from datetime import datetime # import datetime class from datetime package
dt = datetime.now() # get current time
dt1 = dt1.strftime('%Y-%m-%d %H:%M:%S') # converting time to string
dt3 = dt2.strptime('2018/5/20','%Y/%m/%d') # converting a string to specified time
I have a python script that generates a datetime string using this line of code:
data['timestamp'] = datetime.isoformat(datetime.utcnow())
That generates something like the following:
2017-05-24T04:08:09.530033
How do I convert that to "MYSQL insertable" datetime format in a clean way?
Thanks!
Try to use MySQL's STR_TO_DATE() function to parse the string that you're attempting to insert.
I hope this may help you
You can specify any type of format like this depending on the one you `ve set in mysql
data['timestamp'] =pd.to_datetime(data['timestamp'] , format='%d%b%Y:%H:%M:%S.%f')
First off, it looks like you ran from datetime import * rather than import datetime. That's tempting because it lets you type less when you want to refer to parts of the module, but it can get you into name collision issues later. An alternative with less typing is something like import datetime as dt, that way later you can just use dt.datetime. This will make your code cleaner.
MySQL accepts several date formats, which can be read about in detail here. In particular:
The DATETIME type is used for values that contain both date and time
parts. MySQL retrieves and displays DATETIME values in YYYY-MM-DD HH:MM:SS format.
ISO8601 numbers look just like that! 2017-05-24T04:19:32
So if the only difference is the "T" in the middle instead of a space, just run something like this, assuming you don't change your import statements.
timestamp = str(datetime.isoformat(datetime.utcnow()))
timestamp = timestamp.replace("T", " ")
data['timestamp'] = timestamp
I have a csv file containing the pointing personal. of this form:
3,23/02/2015,08:27,08:27,12:29,13:52,19:48
3,24/02/2015,08:17,12:36,13:59,19:28
5,23/02/2015,10:53,13:44
5,25/02/2015,09:05,12:34,12:35,13:30,19:08
5,26/02/2015,08:51,12:20,13:46,18:47,18:58
and I want it cleaning. in this way:
ID, DATE, IN,BREAK_OUT, BREAK_IN, OUT, WORK_TIME
3,Monday 23/02/2015,08:27,12:29,13:52,19:48,08:00hours
3,Tuesday 24/02/2015,08:17,12:36,13:59,19:28,08:00hours
5,Monday 23/02/2015,10:53,NAN,13:44,NAN,2houres
5,Wednesday 25/02/2015,09:05,12:34,13:30,19:08,08hours
can you help me please
think you
I'd suggest you use pandas to import the data from the file
import pandas as pd
pd.read_csv(filepath, sep = ',')
should do the trick, assuming filepath leads to your csv. I'd then suggest that you use the datetime functions to convert your strings to dates you can calculate with (I think you could also use numpys datetime64 types, I'm just not used to them).
import datetime as dt
day = dt.datetime.strptime('23/02/2015', '%d/%m/%Y')
in = dt.datetime.combine(day, dt.datetime.strptime('08:27', '%H:%M').time())
should do the trick. It is necessary, that your in is also a datetime object, not only a time object, otherwise you cannot substract them (which would be the necessary next step to calculate the Worktime.
Is think this should be a bit to get you started, You'll find the pandas documentation here and the datetime documentation here.
If you have further questions, try to ask your question more specific.
This question might help you out: How to split string into column
First, read the whole file and split the columns. Check if there's data or not and write it back into a new file.
If you need additional help, tell us what you tried, what worked for you and what didn't and so on. We won't write a complete program/script for you.
I want to store a list of datetimes in a binary file in Python.
EDIT: by "binary" I mean the best digital representation for each datatype. The application for this is to save GPS trackpoints composed by (unix-timestamp, latitude, longitude, elevation), so the whole structure is little-endian "Long, float, float, float", with four bytes to each value.
NOTE: I don't use "unix-timestamp" due to any affection to the Unix platform, but only as an unequivocal way to represent the value of a datetime.
Currently, I am doing like the code below, but besides some timezone confusion that I'm still working out (my timezone is -3), I believe converting to int and back might not be the right way, since datetime and datetime64 are native types in python/numpy, if I'm not mistaken. Thus, a datetime64 would need eight bytes instead of the four I am using for the (long)unix-timestamp.
import datetime
import calendar
import struct
now = datetime.datetime.now()
print now
stamp = calendar.timegm(now.utctimetuple())
print stamp
binarydatetime = struct.pack('<L', stamp)
recoverstamp = struct.unpack('<L', binarydatetime)[0]
print recoverstamp
recovernow = datetime.datetime.fromtimestamp(recoverstamp)
print recovernow
So the main question is: "is this the pythonic way to converting naive datetime to binary and back?"
And the aditional question is: "if everything in this code is supposed to be naive, why do I have a timezone offset?"
Thanks for reading!
I have found a way using the Unix timestamp and storing it as an integer. This works for me because I don't need a subsecond resolution, but I think long integers would allow for microsecond resolution with some modifications of the code.
The changes from my original consist in replacing calendar.timegm by time.mktime and also utctimetuple by timetuple, to keep everything naive.
This:
import datetime
import struct
import time
now = datetime.datetime.now()
print now
stamp = time.mktime(now.timetuple())
print stamp
recoverstamp = datetime.datetime.fromtimestamp(stamp)
print recoverstamp
binarydatetime = struct.pack('<L', stamp)
recoverbinstamp = struct.unpack('<L', binarydatetime)[0]
print recoverbinstamp
recovernow = datetime.datetime.fromtimestamp(recoverbinstamp)
print recovernow
Returns this:
2013-09-02 11:06:28.064000
1378130788.0
2013-09-02 11:06:28
1378130788
2013-09-02 11:06:28
From this, I can easily write the packed binarydatetime to file, and read it back later.
By far the most simple solution is to use Temporenc: http://temporenc.readthedocs.org/
It takes care of all the encoding/decoding and allows you to write a Python datetime object to a file:
now = datetime.datetime.now()
temporenc.pack(fp, now)
To read it back, this suffices:
dt = temporenc.unpack(fp)
print(dt)
If you want to save an object to a binary file, you may think about pickle
As far as I know, it writes a byte stream, so it's binary enough, but the result is
Python-specific
Also, I think it should work with datetime.
And you'll have this for saving:
with open('your_file', 'wb') as file:
pickle.Pickler(file).dump(your_var)
And this for recovering
with open('your_file', 'rb') as file:
recovered=pickle.Unpickler(file).load()