Issue:
I stored some dates as text in my Postgres table, and I want to convert it over to actual dates, again in Postgres.
Im not sure if there is a better way to do this or what im doing wrong. I have pulled a bunch of data into a PostgreSQL database in just text format. As a result I need to go back through and clean it up. I am running into issues with the data. I need to convert it into a format that PostgreSQL can use. I went about pulling it back into python and trying to convert and kick it back. Is this the best way to do this? Also I am having issue with datetime.strptime.. I believe i've got the directive correct but no go. :/
import psycopg2
from datetime import datetime
# connect to the PostgreSQL database
conn = psycopg2.connect(
"dbname='postgres' user='postgres' host=10.0.75.1 password='mysecretpassword'")
# create a new cursor
cur = conn.cursor()
cur.execute("""SELECT "Hash","Date" FROM nas """)
# commit the changes to the database
myDate = cur.fetchall()
for rows in myDate:
target = rows[1]
datetime.strptime(target, '%B %d, %Y, %H:%M:%S %p %Z')
Here is a Postgres query which can convert your strings into actual timestamps:
select
ts_col,
to_timestamp(ts_col, 'Month DD, YYYY HH:MI:SS PM')::timestamp with time zone
from your_table;
For a full solution, you might take the following steps:
create a new timestamp column ts_col_new in your table
update that column using the logic from the above query
then delete the old column containing text
The update might look something like this:
update your_table
set ts_col_new = to_timestamp(ts_col, 'Month DD, YYYY HH:MI:SS PM')::timestamp with time zone;
Related
I have saved date entries in a sqlite database table. The date fields were converted to a format taken from qdatewidget like this:
due_date = self.dateEdit.date().toString(self.dateEdit_3.displayFormat())
which makes it a string like 10/14/2021, but saves it as a date, which is fine.
Now i would like to read this entries back, but sqlite wont give anything back when i try this:
day = (self.dateEdit_2.date().toString(self.dateEdit_3.displayFormat()))
because day then is also a string 10/14/2021, but the entries in sqlite are date. How could i solve this?
I'm using psycopg2 to insert some data into a PostgreSQL database.
What I'm doing:
import psycopg2
conn = psycopg2.connect(host="127.0.0.1", port='5432', database="mydatabase", user="root", password="root")
cursor = conn.cursor()
# MY-SQL-FILE has a bunch of INSERT statements, such as:
# INSERT INTO mytable (mydate) values ('1991-18-03');
cursor.execute(open("/MY-SQL-FILE.sql").read())
conn.commit()
conn.close()
What I end up with
Executing the above script will create a row in my database where the date is: 1991-03-17 21:00:00
Obs: The field type of mydate is timestamptz
What I expected
I expected to get a row where the date were 1991-03-18 00:00:00
Funny thing is if I try to execute the same insert statement into DBeaver I get 1991-03-18 00:00:00
What I want to know
Why is this happening? (it most likely will work if I convert the date from string to datetime)
But I want to know why this happens, where does this 21:00:00 comes from, if it were a DB TZ problem the time would not be the same everytime I think...
I am working with a Teradata table that has a timestamp column: TIMESTAMP(6) with data that looks like this:
2/14/2019 13:09:51.210000
Currently I have a Python time variable that I want to send into the Teradata table via SQL, that looks like below:
from datetime import datetime
time = datetime.now().strftime("%m/%d/%Y %H:%M:%S")
02/14/2019 13:23:24
How can I reformat that to insert correctly? It is error'ing out with:
teradata.api.DatabaseError: (6760, '[22008] [Teradata][ODBC Teradata Driver][Teradata Database](-6760)Invalid timestamp.')
I tried using the same format the Teradata timestamp column uses:
time = datetime.now().strftime("%mm/%dd/%YYYY %HH24:%MI:%SS")
Same error message
Thanks
Figured it out. Turned out to be unrelated to the timestamp, and I had to reformat the DataFrame column it was being read from. Changing the Data Type fixed it:
final_result_set['RECORD_INSERTED'] = pd.to_datetime(final_result_set['RECORD_INSERTED'])
Now when looping through and inserting via SQL, the following worked fine for populating 'RECORD_INSERTED':
time = datetime.now().strftime("%m/%d/%Y %H:%M:%S")
Sorry for the confusion
I am trying to build a SQL query that will filter based on system date (Query for all sales done in the last 7 days):
import datetime
import pandas as pd
import psycopg2
con = p.connect(db_details)
cur = con.cursor()
df = pd.read_sql("""select store_name,count(*) from sales
where created_at between datetime.datetime.now() - (datetime.today() - timedelta(7))""",con=con)
I get an error
psycopg2.NotSupportedError: cross-database references are not implemented: datetime.datetime.now
You are mixing Python syntax into your SQL query. SQL is parsed and executed by the database, not by Python, and the database knows nothing about datetime.datetime.now() or datetime.date() or timedelta()! The specific error you see is caused by your Python code being interpreted as SQL instead and as SQL, datetime.datetime.now references the now column of the datetime table in the datetime database, which is a cross-database reference, and psycopg2 doesn't support queries that involve multiple databases.
Instead, use SQL parameters to pass in values from Python to the database. Use placeholders in the SQL to show the database driver where the values should go:
params = {
# all rows after this timestamp, 7 days ago relative to 'now'
'earliest': datetime.datetime.now() - datetime.timedelta(days=7),
# if you must have a date *only* (no time component), use
# 'earliest': datetime.date.today() - datetime.timedelta(days=7),
}
df = pd.read_sql("""
select store_name,count(*) from sales
where created_at >= %(latest)s""", params=params, con=con)
This uses placeholders as defined by the psycopg2 parameters documentation, where %(latest)s refers to the latest key in the params dictionary. datetime.datetime() instances are directly supported by the driver.
Note that I also fixed your 7 days ago expression, and replaced your BETWEEN syntax with >=; without a second date you are not querying for values between two dates, so use >= to limit the column to dates at or after the given date.
datetime.datetime.now() is not a proper SQL syntax, and thus cannot be executed by read_sql(). I suggest either using the correct SQL syntax that computes current time, or creating variables for each datetime.datetime.now() and datetime.today() - timedelta(7) and replacing them in your string.
edit: Do not follow the second suggestion. See comments below by Martijn Pieters.
Maybe you should remove that Python code inside your SQL, compute your dates in python and then use the strftime function to convert them to strings.
Then you'll be able to use them in your SQL query.
Actually, you do not necessarily need any params or computations in Python. Just use the corresponding SQL statement which should look like this:
select store_name,count(*)
from sales
where created_at >= now()::date - 7
group by store_name
Edit: I also added a group by which I think is missing.
I am trying to get all records that are within the current week(monday to sunday) and then I was planning to insert them back into the database but with the date of lesson being increased by 7. However I get the following error:
sqlite3.OperationalError: near ">=": syntax error
I may be wrong but I think that this is due to how python stores dates is there a way around this, if not I can always get all records in table into array and filter that array in python. The code for the sql is underneath:
with sqlite3.connect("GuitarLessons.db") as db:
cursor = db.cursor()
cursor.row_factory = sqlite3.Row
sql = "select *"\
"from tblBookings"\
"where DateOfLesson >= ?"\
"and DateOfLesson <= ?"
cursor.execute(sql,(startweekd,endweekd))
BookingList = cursor.fetchall()
print(BookingList)
The rest of my code is just calculating the start and end date for that week.
import datetime
from datetime import date, timedelta
import sqlite3
tdate = datetime.datetime.today()
tday = datetime.datetime.today().weekday()
tdadd = 7 - (tday+1)
endweekd = date.today() + timedelta(days=tdadd)
startweekd = endweekd - timedelta(days=7)
endweekd = endweekd.strftime("%d/%m/%y")
startweekd = startweekd.strftime("%d/%m/%y")
print(startweekd)
print(endweekd)
SQLite (nowhere, not just in Python) does not support dates.
So you have to convert the dates, on query, but also on storage, to some format it will understand. There are two options:
Number of seconds since some epoch, e.g. unix time, possibly fractional.
Strings.
To make comparison of strings work, the dates must be stored in the ISO 8601 format (or at least that order). ISO 8601 timestamp has format specification "%FT%T" (or on systems that don't understand %F or %T "%Y-%m-%dT%H:%M:%S"). Or just dates as "%F"/"%Y-%m-%d". You can use different separators, but the only thing that will gain you is some confusion. Also SQLite has some built-in functions to work with date in ISO 8601 format.
I believe you can define the conversion somewhere so it will then be used automatically when binding query parameters, but I don't remember where. Manually is guaranteed to work.
sqllite requires date to be in YYYY:MM:DD format. You probably should use strftime with the following parameters:
endweekd = endweekd.strftime("%Y:%m:%d")
startweekd = startweekd.strftime("%Y:%m:%d")