I have a python script that reads logs and inserts the logs into a SQL DB(through python). Is there a way that SQL can automatically timestamp the logs when they are inserted so I don't have to read the date and time from the logs in my script?
Thanks!
you could put a column in your database , this
date_registered TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
or you could add it in python like another value to the insert
import datetime
datetime.datetime.now()
Related
I am working with a Teradata table that has a timestamp column: TIMESTAMP(6) with data that looks like this:
2/14/2019 13:09:51.210000
Currently I have a Python time variable that I want to send into the Teradata table via SQL, that looks like below:
from datetime import datetime
time = datetime.now().strftime("%m/%d/%Y %H:%M:%S")
02/14/2019 13:23:24
How can I reformat that to insert correctly? It is error'ing out with:
teradata.api.DatabaseError: (6760, '[22008] [Teradata][ODBC Teradata Driver][Teradata Database](-6760)Invalid timestamp.')
I tried using the same format the Teradata timestamp column uses:
time = datetime.now().strftime("%mm/%dd/%YYYY %HH24:%MI:%SS")
Same error message
Thanks
Figured it out. Turned out to be unrelated to the timestamp, and I had to reformat the DataFrame column it was being read from. Changing the Data Type fixed it:
final_result_set['RECORD_INSERTED'] = pd.to_datetime(final_result_set['RECORD_INSERTED'])
Now when looping through and inserting via SQL, the following worked fine for populating 'RECORD_INSERTED':
time = datetime.now().strftime("%m/%d/%Y %H:%M:%S")
Sorry for the confusion
I am trying to build a SQL query that will filter based on system date (Query for all sales done in the last 7 days):
import datetime
import pandas as pd
import psycopg2
con = p.connect(db_details)
cur = con.cursor()
df = pd.read_sql("""select store_name,count(*) from sales
where created_at between datetime.datetime.now() - (datetime.today() - timedelta(7))""",con=con)
I get an error
psycopg2.NotSupportedError: cross-database references are not implemented: datetime.datetime.now
You are mixing Python syntax into your SQL query. SQL is parsed and executed by the database, not by Python, and the database knows nothing about datetime.datetime.now() or datetime.date() or timedelta()! The specific error you see is caused by your Python code being interpreted as SQL instead and as SQL, datetime.datetime.now references the now column of the datetime table in the datetime database, which is a cross-database reference, and psycopg2 doesn't support queries that involve multiple databases.
Instead, use SQL parameters to pass in values from Python to the database. Use placeholders in the SQL to show the database driver where the values should go:
params = {
# all rows after this timestamp, 7 days ago relative to 'now'
'earliest': datetime.datetime.now() - datetime.timedelta(days=7),
# if you must have a date *only* (no time component), use
# 'earliest': datetime.date.today() - datetime.timedelta(days=7),
}
df = pd.read_sql("""
select store_name,count(*) from sales
where created_at >= %(latest)s""", params=params, con=con)
This uses placeholders as defined by the psycopg2 parameters documentation, where %(latest)s refers to the latest key in the params dictionary. datetime.datetime() instances are directly supported by the driver.
Note that I also fixed your 7 days ago expression, and replaced your BETWEEN syntax with >=; without a second date you are not querying for values between two dates, so use >= to limit the column to dates at or after the given date.
datetime.datetime.now() is not a proper SQL syntax, and thus cannot be executed by read_sql(). I suggest either using the correct SQL syntax that computes current time, or creating variables for each datetime.datetime.now() and datetime.today() - timedelta(7) and replacing them in your string.
edit: Do not follow the second suggestion. See comments below by Martijn Pieters.
Maybe you should remove that Python code inside your SQL, compute your dates in python and then use the strftime function to convert them to strings.
Then you'll be able to use them in your SQL query.
Actually, you do not necessarily need any params or computations in Python. Just use the corresponding SQL statement which should look like this:
select store_name,count(*)
from sales
where created_at >= now()::date - 7
group by store_name
Edit: I also added a group by which I think is missing.
I have built a web app in Python and Flask and am having trouble pulling the date and time from my SQLite database.
I enter the date into the DB with the following line-
order.order_placed = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
Which with my current example enters the following into the DB -
2018-05-01 12:08:49
But when I call order.order_placed I get datetime.date(2018, 5, 1)
Even if I call str(order.order_placed) I get '2018-05-01'
Can someone help me get the full date and time out of the database? Thanks!
It's possible that you're using DateField when in actuality you want to use DateTimeField.
Furthermore, you don't need to call strftime before storing the data. Peewee works nicely with Python datetime objects.
I'm trying to understand the raw manner in which PostgreSQL saves timestamp data types. I get 2 different results depending on the client I use:
1. psql
# SELECT date_incorporated FROM client;
date_incorporated
------------------------
2017-06-14 19:42:15-04
2. records python module
rows = db.query('SELECT date_incorporated FROM client')
print(rows[0])
# {"date_incorporated": "2017-06-14T19:42:15-04:00"}
Since the psql interface and records module are both supposed to be giving me back the raw data, I can't understand why both are giving me back different formats of the timestamp they have stored.
The two differences I see so far are the T's in the middle between the date and time in the records version, and also the differing ways in which it shows the time zone at the end of the string
Is one of them altering it? Which one is showing the real data?
https://www.postgresql.org/docs/current/static/datatype-datetime.html
All timezone-aware dates and times are stored internally in UTC. They
are converted to local time in the zone specified by the TimeZone
configuration parameter before being displayed to the client.
https://www.postgresql.org/docs/current/static/datatype-datetime.html#DATATYPE-DATETIME-OUTPUT
The output format of the date/time types can be set to one of the four
styles ISO 8601, SQL (Ingres), traditional POSTGRES (Unix date
format), or German. The default is the ISO format.
EG:
t=# select now();
now
-------------------------------
2017-11-29 09:07:31.716276+00
(1 row)
t=# set datestyle to SQL;
SET
t=# select now();
now
--------------------------------
11/29/2017 09:07:52.341416 UTC
(1 row)
so the time is saved not the way it is returned. at least not neseserely. You can control up to some level how it it returned to your client. psql does not process time. but python does. not records I believe but python itself
https://en.wikipedia.org/wiki/ISO_8601
T is the time designator that precedes the time components of the
representation.
And that T is definetely not added by postgres itself (unless you deliberately format the date with to_char)
Is it possible to insert a timestamp value into a Cassandra keyspace using prepared statements of Python Cassandra driver? When I tried to do that, I got the following error message:
Expected: <class 'cassandra.cqltypes.DateType'>, Got: <type 'str'>
I see that this problem had been discussed before. But not sure whether it has been resolved. How to do this? Doing the same using simple statements would be inefficient.
Yes, you can insert a timestamp value via prepared statements by binding a datetime object. I have tried it with success.
Like Aaron said, you need to use a datetime object. Given a simple table definition:
CREATE TABLE stackoverflow2.timestamps (
bucket text,
value timestamp,
PRIMARY KEY (bucket, value)
) WITH CLUSTERING ORDER BY (value DESC)
This code will INSERT ten timestamps into the timestamps table, given a valid (connected) session:
preparedInsert = session.prepare(
"""
INSERT INTO stackoverflow2.timestamps (bucket,value) VALUES (?,?);
"""
)
#end prepare statements
for counter in range(1,10):
currentTime = datetime.datetime.today()
bucket = currentTime.strftime("%Y%m")
session.execute(preparedInsert,[bucket,currentTime])
Essentially, the datetime.datetime.today() line creates a datetime object with the current time. The strftime creates a string time bucket from it, and then the preparedInsert puts them both into Cassandra.