I'm using Peewee to access a SQLite DB. How can I save a timestamp in the following format?
'%Y-%m-%d %H:%M:%S' # year-month-day hour-minute-second
(basically: just crop out the microseconds which are present by default)
EDIT: I was able to make this work, using the following in the class definition:
created = DateTimeField(default=datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S))
However, I'm still intersted to know if there's a "built in" way of doing this with Peewee, using the 'formats' parameter.
That actually won't work since the timestamp is evaluated at import time (or the time it's declared). Peewee actually doesn't care about the format going into the database, since it'll just take the python datetime object and let pysqlite convert it.
If you wanted, the best way would be to subclass DateTimeField and override the db_value method.
Related
I have a DB with time entries formatted as follow:
2018-11-05T08:58:00Z
I'm trying to generate SQL queries to compare "now()" with the time in the DB to determine which row(s) to return.
I'm battling to "convert" my local time (now()) to an equivalent time format so that I can use < or > operations against the DB values.
Additionally, I am not sure if the problem has two parts. The example fo the time above is not in a "Datetime" field in MySQL but stored simply as TEXT, leaving me to suspect that I would need to "convert" the DB entries into another format first?
The following code, using the datetime module, works for me (tested in Python 3.6):
import datetime
value = "2018-11-05T08:58:00Z"
dt = datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ")
# Result is: datetime.datetime(2018, 11, 5, 8, 58)
This will convert your string values to datetime instances, which you can then compare to now(). The values that get created should be naive (meaning they have no associated timezone information).
However, if you are sure that now() for you is not UTC (aka Zulu time), you may need to do a conversion. This could be possible if, for example, you are using Django's timezone.now() and your configured timezone is something other than UTC. In this case, I might convert the result of now() to UTC, so you only have to convert one value. The pytz module can easily handle this kind of thing.
Check this :
import time
time = time.strftime('%Y-%m-%dT%H:%M:%SZ')
print(time)
(using django 1.11.2, python 2.7.10, mysql 5.7.18)
If we imagine a simple model:
class Event(models.Model):
happened_datetime = DateTimeField()
value = IntegerField()
What would be the most elegant (and quickest) way to run something similar to:
res = Event.objects.all().aggregate(
Avg('happened_datetime')
)
But that would be able to extract the average time of day for all members of the queryset. Something like:
res = Event.objects.all().aggregate(
AvgTimeOfDay('happened_datetime')
)
Would it be possible to do this on the db directly?, i.e., without running a long loop client-side for each queryset member?
EDIT:
There may be a solution, along those lines, using raw SQL:
select sec_to_time(avg(time_to_sec(extract(HOUR_SECOND from happened_datetime)))) from event_event;
Performance-wise, this runs in 0.015 second for ~23k rows on a laptop, not optimised, etc. Assuming that could yield accurate/correct results and since time is only a secondary factor, could I be using that?
Add another integer field to your model that contains only the hour of the day extracted from the happened_datetime.
When creating/updating a model instance you need to update this new field accordingly whenever the happened_datetime is set/updated. You can extract the hours of the day for example by reading datetime.datetime.hour. Or use strftime to create a value to your liking.
Aggregation should then work as proposed by yourself.
EDIT:
Django's ORM has Extract() as a function. Example from the docs adapted to your use case:
>>> # How many experiments completed in the same year in which they started?
>>> Event.objects.aggregate(
... happenend_datetime__hour=Extract('happenend_datetime', 'hour'))
(Not tested!)
https://docs.djangoproject.com/en/1.11/ref/models/database-functions/#extract
So after a little search and tries.. the below seems to work. Any comments on how to improve (or hinting as to why it is completely wrong), are welcome! :-)
res = Event.objects.raw('''
SELECT id, sec_to_time(avg(time_to_sec(extract(HOUR_SECOND from happened_datetime)))) AS average_time_of_day
FROM event_event
WHERE happened_datetime BETWEEN %s AND %s;''', [start_datetime, end_datetime])
print res[0].__dict__
# {'average_time_of_day': datetime.time(18, 48, 10, 247700), '_state': <django.db.models.base.ModelState object at 0x0445B370>, 'id': 9397L}
Now the ID returned is that of the last object falling in the datetime range for the WHERE clause. I believe Django just inserts that because of "InvalidQuery: Raw query must include the primary key".
Quick explanation of the SQL series of function calls:
Extract HH:MM:SS from all datetime fields
Convert the time values to seconds via time_to_sec.
average all seconds values
convert averaged seconds value back into time format (HH:MM:SS)
Don't know why Django insists on returning microseconds but that is not really relevant. (maybe the local ms at which the time object was instantiated?)
Performance note: this seems to be extremely fast but then again I haven't tested that bit. Any insight would be kindly appreciated :)
I have a python script that generates a datetime string using this line of code:
data['timestamp'] = datetime.isoformat(datetime.utcnow())
That generates something like the following:
2017-05-24T04:08:09.530033
How do I convert that to "MYSQL insertable" datetime format in a clean way?
Thanks!
Try to use MySQL's STR_TO_DATE() function to parse the string that you're attempting to insert.
I hope this may help you
You can specify any type of format like this depending on the one you `ve set in mysql
data['timestamp'] =pd.to_datetime(data['timestamp'] , format='%d%b%Y:%H:%M:%S.%f')
First off, it looks like you ran from datetime import * rather than import datetime. That's tempting because it lets you type less when you want to refer to parts of the module, but it can get you into name collision issues later. An alternative with less typing is something like import datetime as dt, that way later you can just use dt.datetime. This will make your code cleaner.
MySQL accepts several date formats, which can be read about in detail here. In particular:
The DATETIME type is used for values that contain both date and time
parts. MySQL retrieves and displays DATETIME values in YYYY-MM-DD HH:MM:SS format.
ISO8601 numbers look just like that! 2017-05-24T04:19:32
So if the only difference is the "T" in the middle instead of a space, just run something like this, assuming you don't change your import statements.
timestamp = str(datetime.isoformat(datetime.utcnow()))
timestamp = timestamp.replace("T", " ")
data['timestamp'] = timestamp
I need to insert rows into PG one of the fields is date and time with time stamp, this is the time of incident, so I can not use --> current_timestamp function of Postgres at the time of insertion, so how can I then insert the time and date which I collected before into pg row in the same format as it would have been created by current_timestamp at that point in time.
If you use psycopg2 (and possibly some other client library), you can simply pass a Python datetime object as a parameter to a SQL-query:
from datetime import datetime, timezone
dt = datetime.now(timezone.utc)
cur.execute('INSERT INTO mytable (mycol) VALUES (%s)', (dt,))
(This assumes that the timestamp with time zone type is used on the database side.)
More Python types that can be adapted into SQL (and returned as Python objects when a query is executed) are listed here.
A timestamp does not have "a format".
The recommended way to deal with timestamps is to use a PreparedStatement where you just pass a placeholder in the SQL and pass a "real" object through the API of your programming language. As I don't know Python, I don't know if it supports PreparedStatements and how the syntax for that would be.
If you want to put a timestamp literal into your generated SQL, you will need to follow some formatting rules when specifying the value (a literal does have a format).
Ivan's method will work, although I'm not 100% sure if it depends on the configuration of the PostgreSQL server.
A configuration (and language) independent solution to specify a timestamp literal is the ANSI SQL standard:
INSERT INTO some_table
(ts_column)
VALUES
(TIMESTAMP '2011-05-16 15:36:38');
Yes, that's the keyword TIMESTAMP followed by a timestamp formatted in ISO style (the TIMESTAMP keyword defines that format)
The other solution would be to use the to_timestamp() function where you can specify the format of the input literal.
INSERT INTO some_table
(ts_column)
VALUES
(to_timestamp('16-05-2011 15:36:38', 'dd-mm-yyyy hh24:mi:ss'));
Just use 'now'
http://www.postgresql.org/docs/8.0/static/datatype-datetime.html
Date and time input is accepted in almost any reasonable format,
including ISO 8601, SQL-compatible, traditional POSTGRES, and others.
For some formats, ordering of month, day, and year in date input is
ambiguous and there is support for specifying the expected ordering of
these fields.
In other words: just write anything and it will work.
Or check this table with all the unambiguous formats.
Sure, just pass a string value for that timestamp column in the format: '2011-05-16 15:36:38' (you can also append a timezone there, like 'PST'). PostgreSQL will automatically convert the string to a timestamp. See http://www.postgresql.org/docs/9.0/static/datatype-datetime.html#DATATYPE-DATETIME-INPUT
from datetime import datetime as dt
then use this in your code:
cur.execute('INSERT INTO my_table (dt_col) VALUES (%s)', (dt.now(),))
Just use
now()
or
CURRENT_TIMESTAMP
I prefer the latter as I like not having additional parenthesis but thats just personal preference.
How to set default datetime format in python because i have multiple tuples to send via template on client side. This is not good approach to set each object's value to specified format. I want to set a datetime format on server side and these converted values will be shown to client. I tried
datetime.strftime("%Y-%m-%d %X")
but it is giving error.
strftime is a method of datetime objects - it doesn't set a default representation, which seems to be what you suggest. For example, you might call it like this:
>>> import datetime
>>> now = datetime.datetime.now()
>>> now.strftime("%Y-%m-%d %X")
'2011-03-17 10:14:12'
If you need to do this a lot, it would be worth creating a method that wraps this conversion of a datetime to a string. The documentation for the datetime module can be found here.
I'm not sure I understand your issue, but this might help
http://docs.djangoproject.com/en/dev/ref/settings/
there is a datetime format section, this sets datetime format globally.