Develop Reference

Letting SQL timestamp data when inserting it through python?

I have a python script that reads logs and inserts the logs into a SQL DB(through python). Is there a way that SQL can automatically timestamp the logs when they are inserted so I don't have to read the date and time from the logs in my script?
Thanks!

you could put a column in your database , this
date_registered TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
or you could add it in python like another value to the insert
import datetime
datetime.datetime.now()

Django/Python and Raw SQL Querying with PostgreSQL

I'm practicing my raw SQL querying in a Django project using cursor.execute.
Here's my Django models.py database schema:
class Client(models.Model):
date_incorporated = models.DateTimeField(default=timezone.now)
And here's the psql description of the table:
# \d+ client
Column | Type | Modifiers | Storage |
-------------------+--------------------------+--------------------+----------+
date_incorporated | timestamp with time zone | not null | plain |
Here's where I get confused:
If I use psql to query the data from the table, I get:
# SELECT date_incorporated FROM client;
date_incorporated
------------------------
2017-06-14 19:42:15-04
2017-11-02 19:42:33-04
(2 rows)
This makes sense to me. In the PostgreSQL docs, it shows that this is (I believe) just a string that is correctly formatted and stored as a UTC timestamp.
When I go through Django using this query:
cursor.execute('SELECT date_incorporated FROM client;')
data = [dict(zip(columns,row)) for row in cursor.fetchall()]
(using the dictfetchall method from the Django docs)
...my date_incorporated field gets turned into a python datetime object.
{'date_incorporated': datetime.datetime(2017, 11, 2, 23, 42, 33, tzinfo=<UTC>)}
In this app I'm building, I wanted a user to be able to input raw SQL, and put that inputted string into the cursor.execute(rawSQL) function to be executed. I expected the output to be the same as the psql version.
If I was using the Django ORM, I might've expected the timestamp with time zone to be converted to a time-zone aware datetime object, but since I'm doing a raw SQL call, I expected to get back 2017-06-14 19:42:15-04, not a python datetime object.
Is the fetchall method still acting as the Django ORM and converting certain fields?

I believe this is standard conversion from using any interface driver.
You would get the same result even if you use py-postgressql, i.e. the cursor is doing the conversion according to the field type defined in the database.
Long story short, the dictfetchall is not doing any conversion, but rather parsing the converted result from the cursor.

Query to compare between date with time and date without time - python using access db

I need help to create query to compare between date with time and date without time. I am using python with access db (pypyodbc).
In the database I have a column that contains date/time (includes time), and in python I have a datetime object (without time).
I want to write a sql query that compares just the dates of the two.
For Example:
cur.execute("SELECT * FROM MDSSDB WHERE [ValidStartTime] = #2016-05-17#")
The ValidStartTime includes time so it doesn't work. I want just the date from the ValidStartTime.

Consider using MS Access' DateValue function that extracts only the date component (TimeValue being the time component counterpart).
Also, consider passing your date value as parameter to better integrate with your Python environment with no need to concatenate into Access' # form. Below passes a parameter as tuple of one item:
from datetime import datetime
...
cur.execute("SELECT * FROM MDSSDB WHERE DateValue([ValidStartTime]) = ?", (datetime(2016, 5, 17),))

python sql where using dates

I am trying to get all records that are within the current week(monday to sunday) and then I was planning to insert them back into the database but with the date of lesson being increased by 7. However I get the following error:
sqlite3.OperationalError: near ">=": syntax error
I may be wrong but I think that this is due to how python stores dates is there a way around this, if not I can always get all records in table into array and filter that array in python. The code for the sql is underneath:
with sqlite3.connect("GuitarLessons.db") as db:
cursor = db.cursor()
cursor.row_factory = sqlite3.Row
sql = "select *"\
"from tblBookings"\
"where DateOfLesson >= ?"\
"and DateOfLesson <= ?"
cursor.execute(sql,(startweekd,endweekd))
BookingList = cursor.fetchall()
print(BookingList)
The rest of my code is just calculating the start and end date for that week.
import datetime
from datetime import date, timedelta
import sqlite3
tdate = datetime.datetime.today()
tday = datetime.datetime.today().weekday()
tdadd = 7 - (tday+1)
endweekd = date.today() + timedelta(days=tdadd)
startweekd = endweekd - timedelta(days=7)
endweekd = endweekd.strftime("%d/%m/%y")
startweekd = startweekd.strftime("%d/%m/%y")
print(startweekd)
print(endweekd)

SQLite (nowhere, not just in Python) does not support dates.
So you have to convert the dates, on query, but also on storage, to some format it will understand. There are two options:
Number of seconds since some epoch, e.g. unix time, possibly fractional.
Strings.
To make comparison of strings work, the dates must be stored in the ISO 8601 format (or at least that order). ISO 8601 timestamp has format specification "%FT%T" (or on systems that don't understand %F or %T "%Y-%m-%dT%H:%M:%S"). Or just dates as "%F"/"%Y-%m-%d". You can use different separators, but the only thing that will gain you is some confusion. Also SQLite has some built-in functions to work with date in ISO 8601 format.
I believe you can define the conversion somewhere so it will then be used automatically when binding query parameters, but I don't remember where. Manually is guaranteed to work.

sqllite requires date to be in YYYY:MM:DD format. You probably should use strftime with the following parameters:
endweekd = endweekd.strftime("%Y:%m:%d")
startweekd = startweekd.strftime("%Y:%m:%d")

How do I GROUP BY on every given increment of a field value?

I have a Python application. It has an SQLite database, full of data about things that happen, retrieved by a Web scraper from the Web. This data includes time-date groups, as Unix timestamps, in a column reserved for them. I want to retrieve the names of organisations that did things and count how often they did them, but to do this for each week (i.e. 604,800 seconds) I have data for.
Pseudocode:
for each 604800-second increment in time:
select count(time), org from table group by org
Essentially what I'm trying to do is iterate through the database like a list sorted on the time column, with a step value of 604800. The aim is to analyse how the distribution of different organisations in the total changed over time.
If at all possible, I'd like to avoid pulling all the rows from the db and processing them in Python as this seems a) inefficient and b) probably pointless given that the data is in a database.

Not being familiar with SQLite I think this approach should work for most databases, as it finds the weeknumber and subtracts the offset
SELECT org, ROUND(time/604800) - week_offset, COUNT(*)
FROM table
GROUP BY org, ROUND(time/604800) - week_offset
In Oracle I would use the following if time was a date column:
SELECT org, TO_CHAR(time, 'YYYY-IW'), COUNT(*)
FROM table
GROUP BY org, TO_CHAR(time, 'YYYY-IW')
SQLite probably has similar functionality that allows this kind of SELECT which is easier on the eye.

Create a table listing all weeks since the epoch, and JOIN it to your table of events.
CREATE TABLE Weeks (
week INTEGER PRIMARY KEY
);
INSERT INTO Weeks (week) VALUES (200919); -- e.g. this week
SELECT w.week, e.org, COUNT(*)
FROM Events e JOIN Weeks w ON (w.week = strftime('%Y%W', e.time))
GROUP BY w.week, e.org;
There are only 52-53 weeks per year. Even if you populate the Weeks table for 100 years, that's still a small table.

To do this in a set-based manner (which is what SQL is good at) you will need a set-based representation of your time increments. That can be a temporary table, a permanent table, or a derived table (i.e. subquery). I'm not too familiar with SQLite and it's been awhile since I've worked with UNIX. Timestamps in UNIX are just # seconds since some set date/time? Using a standard Calendar table (which is useful to have in a database)...
SELECT
C1.start_time,
C2.end_time,
T.org,
COUNT(time)
FROM
Calendar C1
INNER JOIN Calendar C2 ON
C2.start_time = DATEADD(dy, 6, C1.start_time)
INNER JOIN My_Table T ON
T.time BETWEEN C1.start_time AND C2.end_time -- You'll need to convert to timestamp here
WHERE
DATEPART(dw, C1.start_time) = 1 AND -- Basically, only get dates that are a Sunday or whatever other day starts your intervals
C1.start_time BETWEEN #start_range_date AND #end_range_date -- Period for which you're running the report
GROUP BY
C1.start_time,
C2.end_time,
T.org
The Calendar table can take whatever form you want, so you could use UNIX timestamps in it for the start_time and end_time. You just pre-populate it with all of the dates in any conceivable range that you might want to use. Even going from 1900-01-01 to 9999-12-31 won't be a terribly large table. It can come in handy for a lot of reporting type queries.
Finally, this code is T-SQL, so you'll probably need to convert the DATEPART and DATEADD to whatever the equivalent is in SQLite.