I'm working on a web app that will allow the user to do a lookup by date, so that, for example:
results = Something.objects.filter(end = date)
I'm planning on passing in the date information via the URL, like this:
example.com/invoicer?2/9/1984
I'll then grab the date via request.GET, break off the first part and store it as the month, delete the slash, break off the second part as the date, etc. etc.
I'm not too worried about error/input checking, since only trusted administrators will have access to this, but it seems like a crappy way of going about generating the datetime.
Any better ideas?
[...] break off the first part and store it as the month, delete the slash, break off the second part as the date, etc. etc.
That's indeed is not an ideal way to generate the datetime. You're better off with string parsing:
>>> datetime.datetime.strptime('2/9/1984', '%m/%d/%Y')
datetime.datetime(1984, 2, 9, 0, 0)
You'll have an easier time if you use a parameter for the value:
invoicer?date=2/9/1984
and you might prefer to use an ISO8601 date:
invoicer?date=19840209
I'm not sure what your user interface to this is, or are you expecting people to type these URLs by hand? If not, IS08601 is the way to go.
Seems pretty normal. One other way we do it is not to use the query string but the path:
example.com/invoicer/end_date/1984-02-09
you can convert to a unix/epoch date and pass as a long integer.
super simple and no worrying about string parsing. just use python's time.gmtime to get your time from the number of seconds.
example.com/invoicer?date=1280139973
time.gmtime(1280139973)
Related
I made a huge misstake building up a database, but it works perfectly except for 1 feature. Changing the program in all the places where it needs to be changed for that feature to work would be a titanic job of weeks, so let's hope this workaround is possible.
The issue: I've stored data in a SQLite database as "dd/mm/yyyy" TextField format instead of DateField.
The need: I need to sort by dates on a union query, to get the last number of records in that union following my custom date format. They are from different tables, so I can't just use rowid or stuff like that to get the last ones, I need to do it by date and I can't change the already stored data in the database because there are already invoices created with that format ("dd/mm/yyyy" is the default date format in my country).
This is the query that captures data:
records = []
limited_to = 25
union = (facturas | contado | albaranes | presupuestos)
for record in (union
.select_from(union.c.idunica, union.c.fecha, union.c.codigo,
union.c.tipo, union.c.clienterazonsocial,
union.c.totalimporte, union.c.pagada,
union.c.contabilizar, union.c.concepto1,
union.c.cantidad1, union.c.precio1,
union.c.observaciones)
.order_by(union.c.fecha.desc()) # TODO this is what I need to change.
.limit(limited_to)
.tuples()):
records.append(record)
Now to complicate things even more, the union is already created by a really complex where clause for each database before it's transformed into an union query.
So my only hope is: Is there a way to make order_by follow a custom date format instead?
To clarify, this is the simple transformation that I'd need the order_by clause to follow, because I assume SQLite wouldn't have issues sorting if this would be the date format:
def reverse_date(date: str) -> str:
"""Reverse the date order from dd/mm/yyyy dates into yyyy-mm-dd"""
yyyy, mm, dd = date.split("/")
return f"{yyyy}-{mm}-{dd}"
Note: I've left lot of code out because I think it's unnecesary. This is the minimum amount of code needed to understand the problem. Let me know if you need more data.
Update: Trying this workaround, it seems to work fine. Need more testing but it's promising. If someone ever faces the same issue, here you go:
.order_by(sqlfunc.Substr(union.c.fecha, 7)
.concat('/')
.concat(sqlfunc.Substr(union.c.fecha, 4, 2))
.concat('/')
.concat(sqlfunc.Substr(union.c.fecha, 1, 2))
.desc())
Happy end of 2020 year!
As you pointed out, if you want the dates to sort properly, they need to be in yyyy-mm-dd format, which is the text format you should always use in SQLite (or something with the same year, month, day, order).
You might be able to do a rearrangement here using re.sub:
.order_by(re.sub(r'(\d{2})/(\d{2})/(\d{4})', r'\3-\2-\1',
union.c.fecha))
Here we are using regex to capture the year, month, and day components in separate capture groups. Then, we replace with these components in the correct order for sorting.
I have a function that consumes a datetime string that is returned from a DB query. Right now the query returns a datetime object.
What I am looking for is what would be the preferred way to create my datetime string. I have not done any performance profiling yet, just looking for previous experiences from people.
It depends.
Normally, the database is just a repository for data; it is not a formatting engine. This implies that you should expect to get strings like "2019-06-24 13:47:24" or numbers like 1561409293 and you deal with them from there.
However, it is often more straightforward to simply call DATE_FORMAT() in your SELECT statement. This is especially handy when the SELECT can generate the entire 'report' without further manipulation.
Another way to decide... Which approach requires fewer keystrokes on your part? Or has the least chance of programming errors? Or...
You say "consumes a datetime string that is returned from a DB query" -- but what will it do with it? If it will be manipulating it in more than one way, then a client "object" sounds like the better approach. If you will simply display the datetime, then DATE_FORMAT() may be better.
There is no noticeable performance difference.
If you have a datetime object, could could just keep it around in your code as a datetime object, extracting whatever information you need from it. Then when you really need the actual string, use strftime to format it in the way you want.
>>> from datetime import datetime
>>> t = datetime.now()
>>> t
datetime.datetime(2019, 6, 24, 14, 23, 45, 835379)
>>> print(t.month)
6
>>> print(t.second)
45
>>> as_string = t.strftime("%B %d, %Y")
>>> print(as_string)
June 24, 2019
>>> as_another_string = t.strftime("%Y-%h-%d %H:%m")
>>> print(as_another_string)
2019-Jun-24 14:06
This page shows you the sorts of format codes you can call upon, in order to extract whichever date/time information you want to display in your string:
I have a form that takes a user's birthdate. By default, Django accepts three input formats for dates, including MM/DD/YY. However if I enter something like 02/13/45, it saves as 02/13/2045. I've looked through the places I expected to find some threads in the docs but still nothing. Can someone push me in the right direction?
DateField just uses the datetime.strptime method, which in turn uses the underlying C strftime implementation; there's no way to change the way it parses a two-digit year. As mentioned in the comment, probably the best way to do this is to check if it's in the future, then subtract 100.
def clean_birthdate(self):
birthdate = self.cleaned_data('birthdate')
if birthdate > datetime.datetime.today():
birthdate = birthdate.replace(year=(birthdate.year-100))
return birthdate
(using django 1.11.2, python 2.7.10, mysql 5.7.18)
If we imagine a simple model:
class Event(models.Model):
happened_datetime = DateTimeField()
value = IntegerField()
What would be the most elegant (and quickest) way to run something similar to:
res = Event.objects.all().aggregate(
Avg('happened_datetime')
)
But that would be able to extract the average time of day for all members of the queryset. Something like:
res = Event.objects.all().aggregate(
AvgTimeOfDay('happened_datetime')
)
Would it be possible to do this on the db directly?, i.e., without running a long loop client-side for each queryset member?
EDIT:
There may be a solution, along those lines, using raw SQL:
select sec_to_time(avg(time_to_sec(extract(HOUR_SECOND from happened_datetime)))) from event_event;
Performance-wise, this runs in 0.015 second for ~23k rows on a laptop, not optimised, etc. Assuming that could yield accurate/correct results and since time is only a secondary factor, could I be using that?
Add another integer field to your model that contains only the hour of the day extracted from the happened_datetime.
When creating/updating a model instance you need to update this new field accordingly whenever the happened_datetime is set/updated. You can extract the hours of the day for example by reading datetime.datetime.hour. Or use strftime to create a value to your liking.
Aggregation should then work as proposed by yourself.
EDIT:
Django's ORM has Extract() as a function. Example from the docs adapted to your use case:
>>> # How many experiments completed in the same year in which they started?
>>> Event.objects.aggregate(
... happenend_datetime__hour=Extract('happenend_datetime', 'hour'))
(Not tested!)
https://docs.djangoproject.com/en/1.11/ref/models/database-functions/#extract
So after a little search and tries.. the below seems to work. Any comments on how to improve (or hinting as to why it is completely wrong), are welcome! :-)
res = Event.objects.raw('''
SELECT id, sec_to_time(avg(time_to_sec(extract(HOUR_SECOND from happened_datetime)))) AS average_time_of_day
FROM event_event
WHERE happened_datetime BETWEEN %s AND %s;''', [start_datetime, end_datetime])
print res[0].__dict__
# {'average_time_of_day': datetime.time(18, 48, 10, 247700), '_state': <django.db.models.base.ModelState object at 0x0445B370>, 'id': 9397L}
Now the ID returned is that of the last object falling in the datetime range for the WHERE clause. I believe Django just inserts that because of "InvalidQuery: Raw query must include the primary key".
Quick explanation of the SQL series of function calls:
Extract HH:MM:SS from all datetime fields
Convert the time values to seconds via time_to_sec.
average all seconds values
convert averaged seconds value back into time format (HH:MM:SS)
Don't know why Django insists on returning microseconds but that is not really relevant. (maybe the local ms at which the time object was instantiated?)
Performance note: this seems to be extremely fast but then again I haven't tested that bit. Any insight would be kindly appreciated :)
I tried to submit a time var with value of 2016-03-12T01:47:57+00:00 in a timestamp field, it gives me error saying to check the syntax for errors, however when I use a function to normalize the date
t = datetime.datetime.strptime(data['time'], '%Y-%m-%dT%H:%M:%SZ').strftime('%Y-%m-%d:%H:%M:%S')
I get an error like this.
time data '2016-03-12T01:47:57+00:00' does not match format '%Y-%m-%dT%H:%M:%SZ'
What's causing your problem has already been clarified by others, but please allow me to suggest my favorite solution for cases such as yours:
from dateutil import parser
parser.parse(data['time'])
More about the dateutil module here.
There are a few problems here:
The %Z (note the captial Z!) is for time zone, for example GMT. I
think you want the lower case option: %z, which is for UTC offset.
You can read here in the docs what all the options do :)
https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior
You need the % symbol before each option. You cannot write %Sz, you must write %S%z. Otherwise Python is trying to match something like 2016-03-12T01:47:57z, rather than 2016-03-12T01:47:57+00:00
Unfortunately, you can't use the %z option with strptime, see this answer:
ISO to datetime object: 'z' is a bad directive
My solution:
It sounds like you don't even want to use the UTC offset. That's fine! If you can't change the way your date string is generated, perhaps this is the best option (though it's maybe a little dirty):
t = datetime.datetime.strptime(data['time'][:-6], '%Y-%m-%dT%H:%M:%S')
This will remove the UTC offset from the string.
If you can change the way your datetime string is being generated, that would be a better solution, but I realise you might not be able to do so.
I hope this helps!