I am trying to convert a date-time string value into an integer when I store it in a sqlite table. I am using Python (Flask). I am getting an 'Error in insert operation' with the following query.
Any ideas?
with sql.connect("flaskjournal.db") as con:
cur = con.cursor()
t_i = strftime('%s','time_in')
cur.execute("INSERT INTO entries (beach, board, swell, wind, score, notes, time_in)
VALUES (?,?,?,?,?,?,?)",(beach, board, swell, wind, score, notes, t_i))
con.commit()
msg = "Record successfully added"
You can use the timestamp instead.
import time
from datetime import datetime
timestamp = int(time.mktime(datetime_object.timetuple())
Make t_i insert a unix timestamp via your create table schema. so you don't need to muck with it from python.
sqlite> create table t1 (
...> id integer primary key,
...> time t_i default (strftime('%s', 'now')),
...> txt text);
Related
I am using psycopg2 to create a table partition and insert some rows into this newly created partition. The table is RANGE partitioned on a date type column.
Psycopg2 code:
conn = connect_db()
cursor = conn.cursor()
sysdate = datetime.now().date()
sysdate_str = sysdate.strftime('%Y%m%d')
schema_name = "schema_name"
table_name = "transaction_log"
# Add partition if not exists for current day
sql_add_partition = sql.SQL("""
CREATE TABLE IF NOT EXISTS {table_partition}
PARTITION of {table}
FOR VALUES FROM (%(sysdate)s) TO (maxvalue);
""").format(table = sql.Identifier(schema_name, table_name), table_partition = sql.Identifier(schema_name, f'{table_name}_{sysdate_str}'))
print(cursor.mogrify(sql_add_partition, {'sysdate': dt.date(2015,6,30)}))
cursor.execute(sql_add_partition, {'sysdate': sysdate})
Formatted output of cursor.mogrify():
CREATE TABLE IF NOT EXISTS "schema_name"."transaction_log_20211001"
PARTITION of "schema_name"."transaction_log"
FOR VALUES FROM ('2021-10-01'::date) TO (maxvalue);
Error received:
ERROR: syntax error at or near "::"
LINE 3: for values FROM ('2021-10-01'::date) TO (maxvalue);
Interestingly enough, psycopg2 appears to be attempting to cast the string '2021-10-01' to a date object with the "::date" syntax, and according to the postgreSQL documentation, this appears to be valid (although there are no explicit examples given in the docs), however executing the statement with both pyscopg2 and in a postgreSQL query editor yields this syntax error. However, executing the following statement in a postgreSQL SQL editor is successful:
CREATE TABLE IF NOT EXISTS "schema_name"."transaction_log_20211001"
PARTITION of "schema_name"."transaction_log"
FOR VALUES FROM ('2021-10-01') TO (maxvalue);
Any ideas on how to get psycopg2 to format the query correctly?
To follow up on #LaurenzAlbe comment:
sql_add_partition = sql.SQL("""
CREATE TABLE IF NOT EXISTS {table_partition}
PARTITION of {table}
FOR VALUES FROM (%(sysdate)s) TO (maxvalue);
""").format(table = sql.Identifier(schema_name, table_name), table_partition = sql.Identifier(schema_name, f'{table_name}_{sysdate_str}'))
print(cursor.mogrify(sql_add_partition, {'sysdate': '2021-10-01'}))
#OR
sql_add_partition = sql.SQL("""
CREATE TABLE IF NOT EXISTS {table_partition}
PARTITION of {table}
FOR VALUES FROM ({sysdate}) TO (maxvalue);
""").format(table = sql.Identifier(schema_name, table_name),
table_partition = sql.Identifier(schema_name, f'{table_name}_{sysdate_str}'),
sysdate=sql.Literal('2021-10-01'))
print(cursor.mogrify(sql_add_partition))
#Formatted as
CREATE TABLE IF NOT EXISTS "schema_name"."transaction_log_20211001"
PARTITION of "schema_name"."transaction_log"
FOR VALUES FROM ('2021-10-01') TO (maxvalue);
Pass the date in as a literal value instead of a date object. psycopg2 does automatic adaptation of date(time) objects to Postgres date/timestamp types(Datetime adaptation) which is what is biting you.
UPDATE
Per my comment, the reason why it needs to be a literal is explained here Create Table:
Each of the values specified in the partition_bound_spec is a literal, NULL, MINVALUE, or MAXVALUE. Each literal value must be either a numeric constant that is coercible to the corresponding partition key column's type, or a string literal that is valid input for that type.
Using pandas and sqlite3.
I have about 1000 csv files with data that needs a little bit of prunning and then I have to insert them into a database.
The code I am using:
def insert_underlying(underlying_df, conn):
# convert timestamp to timestamp dtype
underlying_df['timestamp'] = pd.to_datetime(underlying_df['timestamp'])
# underlying_df = underlying_df.tail(len(underlying_df)-1)
# sort it
underlying_df.sort_values('timestamp', inplace=True)
# remove the contractname column
underlying_df = underlying_df[['timestamp', 'close', 'bid', 'ask']]
# insert underlying into tickdata
# check if table exists # if it doesn't create
c.execute("CREATE TABLE IF NOT EXISTS {}tick(timestamp timestamp, close REAL, bid REAL, ask REAL)" .format(symbol))
# Insert to external db
query = 'INSERT OR REPLACE INTO ' + symbol + 'tick (timestamp, close, bid, ask) VALUES (?,?,?,?)'
conn.executemany(query, underlying_df.to_records(index=False))
conn.commit()
From what I have read on similar questions, executemany is iterating over the rows and inserting the records. This is very slow. My experience with sql is beginner at best.
What would be the best/fastest way to do this?
Thanks
I am trying to insert dates into a MySql database using pymysql and parameters. Some of the rows have a date field but for some others that particular date field is missing. The empty rows give an error of the type "pymysql.err.InternalError: (1292, "Incorrect date value:". Below is a piece of code that reproduces the error:
import pymysql
db=pymysql.connect("localhost","testuser","test1234","TESTDB")
cursor = db.cursor()
cursor.execute("SELECT VERSION()")
data = cursor.fetchone()
print ("Database version : %s " % data)
query = "DROP TABLE IF EXISTS birthdays"
cursor.execute(query)
query="CREATE TABLE birthdays(id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,\
name VARCHAR(20),\
birthday DATE NULL DEFAULT NULL)"
cursor.execute(query)
db.commit()
birthdays={'David':'2014-05-22','Larry':'014-05-22', 'Barry':''}
for k,v in birthdays.items():
print(k,v)
query="INSERT INTO birthdays (name,birthday) VALUES ('%s','%s')"%(k,v)
cursor.execute(query)
db.commit()
db.close()
The problem is with Barry and its empty date. I have tried setting Barry's date to None but it is not working. If I set it up to "NULL" and remove the quotes from the date parameter (('%s',%s) instead of ('%s','%s')) it works for Barry but note for the others.
thank you very much in advance,
Gabriel Vidal
You can use below code i have just change in your array and set 'Barry':None because When using mysqldb and cursor.execute(), pass the value None:
birthdays={'David':'2014-05-22','Larry':'014-05-22', 'Barry':None}
for k,v in birthdays.items():
print(k,v)
query="INSERT INTO birthdays (name,birthday) VALUES ('%s','%s')"%(k,v)
cursor.execute(query)
db.commit()
db.close()
For further detail here
In case someone drops by:
The following code snippet solves the problem you had. Note that you should keep your execute statements outside of the for loop to minimize the number of connections you do with the database.
birthdays = {'David':'2014-05-22','Larry':'2014-05-22', 'Barry':'NULL'}
values = ""
for k,v in birthdays.items():
values += "('%s', '%s')," % (k,v)
values = values[:-1].replace("'NULL'", "NULL") # Drop the final comma and the quotes of NULL values
query = f"INSERT INTO birthdays (name,birthday) VALUES {values}" # >= 3.6
cursor.execute(query) # Interaction with the database occurs only once
db.commit()
This generates the following statement:
INSERT INTO birthdays (name,birthday) VALUES ('David', '2014-05-22'),('Larry', '2014-05-22'),('Barry', NULL)
which is a valid sql statement
I have created the following table in MySQL
create table DT (Date varchar(20), Time varchar(20))
I am using the following python code to insert into mysql. I am not getting any error nor the date and time is getting written into mysql.
import mysql.connector
from datetime import datetime
conn = mysql.connector.connect(host = 'localhost', database = 'mydatabase', user = 'root', password = '')
cursor = conn.cursor()
i = datetime.now()
xdate = i.strftime('%Y-%m-%d')
xtime = i.strftime('%H-%M-%S')
sqql = 'insert into DT (Date, Time) values (%s,%s);'
cursor.execute(sqql, xdate, xtime)
conn.commit()
conn.close()
I am not getting any error nor getting the date and time inserted into mysql. However when I directly do the insert in mysql it works..
insert into DT (Date, Time) values ('2017-02-06', '19-54-36');
Not sure how to resolve? Can someone help?
You are not passing the query parameters correctly, put them into a tuple, replace:
cursor.execute(sqql, xdate, xtime)
with:
cursor.execute(sqql, (xdate, xtime))
According to this example and the documentation, the method cursor.execute() takes a string query as first parameter, and the second parameter is a tuple or a dictionary that contains the values of that query. So you should call the method this way:
cursor.execute(sqql, (xdate, xtime))
In case you are using xdate and xtime just in this invocation, you can even simplify this call by:
cursor.execute(sqql, (i.strftime('%Y-%m-%d'), i.strftime('%H-%M-%S')))
In addition, you can also use a dictionary in order to improve the readability of the code:
sqql = 'insert into DT (Date, Time) values (%(xdate)s, %(xtime)s);'
cursor.execute(sqql, {'xdate': `xdate`, 'xtime`: xtime})
I'm having some trouble modifying a sql table that contains a date.
This is the code I'm running:
import pymysql
cnx=pymysql.connect(dbstuff)
cursor=cnx.cursor()
cursor.execute("SELECT * FROM devices ORDER BY ip_addr,Port ASC")
results=cursor.fetchall()
lst = [(i,)+line[1:] for i, line in enumerate(results, 1)]
values = ', '.join(map(str,lst))
query="DELETE FROM devices; INSERT INTO devices VALUES {}".format(values)
query = query.replace("None","NULL")
cursor.execute(query)
cnx.commit()
cnx.close()
If the date column in sql is NULL, this runs without a problem. If there is a date inserted into the field, I get the following error:
"FUNCTION datetime.datetime does not exist."
When I look at the results of the sql select query, the date column value is converted to the following:
datetime.datetime(2016, 8, 16, 11, 24, 4)
I assume this is a python thing and not a sql thing. I haven't been to find anyway to convert this to a format sql would understand.
For datetime values you may use this:
your_datetime = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
This variable will hold a string value like this:
2018-05-06 21:14:37
And then use the following SQL command:
"INSERT INTO your_table (your_datetime_column) VALUES (str_to_date(\'%s\', '%%Y-%%m-%%d %%H:%%i:%%s'))" % (your_datetime)
DO NOT USE FOLLOWING - it won't work with the .format() method:
"INSERT INTO your_table (your_datetime_column) VALUES ({0})".format (your_datetime)
Calling the str function of the datetime.datetime object will convert it to the string format required by SQL. It is currently trying to insert a python object.
Simply wrap your object like this:
dt = str(my_datetime_object)
This error is raised when you pass invalid format date string in SQL query. Even a space in '2018-05-01 11:02:02 ' can cause this error. The best way to get rid of it is to avoid string formatting while making sql queries.
c=db.cursor()
c.execute("INSERT INTO devices VALUES(%s,%s,%s)",(value1,value2,value3))
c.close()
If you are unknown that how many values will have then, you have to generate (%s,%s...) dynamically.
import pymysql
cnx=py
mysql.connect(dbstuff)
cursor=cnx.cursor()
cursor.execute("SELECT * FROM devices ORDER BY ip_addr,Port ASC")
results=cursor.fetchall()
lst = [(i,)+line[1:] for i, line in enumerate(results, 1)]
formatt=','.join(['%s']*len(lst))
query="INSERT INTO devices VALUES (%s)"%formatt
cursor.execute(query,lst)
cnx.commit()
cnx.close()