I am trying to insert dates into a MySql database using pymysql and parameters. Some of the rows have a date field but for some others that particular date field is missing. The empty rows give an error of the type "pymysql.err.InternalError: (1292, "Incorrect date value:". Below is a piece of code that reproduces the error:
import pymysql
db=pymysql.connect("localhost","testuser","test1234","TESTDB")
cursor = db.cursor()
cursor.execute("SELECT VERSION()")
data = cursor.fetchone()
print ("Database version : %s " % data)
query = "DROP TABLE IF EXISTS birthdays"
cursor.execute(query)
query="CREATE TABLE birthdays(id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,\
name VARCHAR(20),\
birthday DATE NULL DEFAULT NULL)"
cursor.execute(query)
db.commit()
birthdays={'David':'2014-05-22','Larry':'014-05-22', 'Barry':''}
for k,v in birthdays.items():
print(k,v)
query="INSERT INTO birthdays (name,birthday) VALUES ('%s','%s')"%(k,v)
cursor.execute(query)
db.commit()
db.close()
The problem is with Barry and its empty date. I have tried setting Barry's date to None but it is not working. If I set it up to "NULL" and remove the quotes from the date parameter (('%s',%s) instead of ('%s','%s')) it works for Barry but note for the others.
thank you very much in advance,
Gabriel Vidal
You can use below code i have just change in your array and set 'Barry':None because When using mysqldb and cursor.execute(), pass the value None:
birthdays={'David':'2014-05-22','Larry':'014-05-22', 'Barry':None}
for k,v in birthdays.items():
print(k,v)
query="INSERT INTO birthdays (name,birthday) VALUES ('%s','%s')"%(k,v)
cursor.execute(query)
db.commit()
db.close()
For further detail here
In case someone drops by:
The following code snippet solves the problem you had. Note that you should keep your execute statements outside of the for loop to minimize the number of connections you do with the database.
birthdays = {'David':'2014-05-22','Larry':'2014-05-22', 'Barry':'NULL'}
values = ""
for k,v in birthdays.items():
values += "('%s', '%s')," % (k,v)
values = values[:-1].replace("'NULL'", "NULL") # Drop the final comma and the quotes of NULL values
query = f"INSERT INTO birthdays (name,birthday) VALUES {values}" # >= 3.6
cursor.execute(query) # Interaction with the database occurs only once
db.commit()
This generates the following statement:
INSERT INTO birthdays (name,birthday) VALUES ('David', '2014-05-22'),('Larry', '2014-05-22'),('Barry', NULL)
which is a valid sql statement
Related
cHandler = myDB.cursor()
cHandler.execute('select UserId,C1,LogDate from DeviceLogs_12_2019') // data from remote sql server database
curs = connection.cursor()
curs.execute("""select * from biometric""") //data from my database table
lst = []
result= cHandler.fetchall()
for row in result:
lst.append(row)
lst2 = []
result2= curs.fetchall()
for row in result2:
lst2.append(row)
t = []
r = [elem for elem in lst if not elem in lst2]
for i in r:
print(i)
t.append(i)
for i in t:
frappe.db.sql("""Insert into biometric(UserId,C1,LogDate) select '%s','%s','%s' where not exists(select * from biometric where UserID='%s' and LogDate='%s')""",(i[0],i[1],i[2],i[0],i[2]),as_dict=1)
I am trying above code to insert data into my table if record not exists but getting error :
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '1111'',''in'',''2019-12-03 06:37:15'' where not exists(select * from biometric ' at line 1")
Is there anything I am doing wrong or any other way to achieve this?
It appears you have potentially four problems:
There is a from clause missing between select and where not exists.
When using a prepared statement you do not enclose your placeholder arguments, %s, within quotes. Your SQL should be:
Your loop:
Loop:
t = []
r = [elem for elem in lst if not elem in lst2]
for i in r:
print(i)
t.append(i)
If you are trying to only include rows from the remote site that will not be duplicates, then you should explicitly check the two fields that matter, i.e. UserId and LogDate. But what is the point since your SQL is taking care of making sure that you are excluding these duplicate rows? Also, what is the point of copying everything form r to t?
SQL:
Insert into biometric(UserId,C1,LogDate) select %s,%s,%s from DUAL where not exists(select * from biometric where UserID=%s and LogDate=%s
But here is the problem even with the above SQL:
If the not exists clause is false, then the select %s,%s,%s from DUAL ... returns no columns and the column count will not match the number of columns you are trying to insert, namely three.
If your concern is getting an error due to duplicate keys because (UserId, LogDate) is either a UNIQUE or PRIMARY KEY, then add the IGNORE keyword on the INSERT statement and then if a row with the key already exists, the insertion will be ignored. But there is no way of knowing since you have not provided this information:
for i in t:
frappe.db.sql("Insert IGNORE into biometric(UserId,C1,LogDate) values(%s,%s,%s)",(i[0],i[1],i[2]))
If you do not want multiple rows with the same (UserId, LogDate) combination, then you should define a UNIQUE KEY on these two columns and then the above SQL should be sufficient. There is also an ON DUPLICATE KEY SET ... variation of the INSERT statement where if the key exists you can do an update instead (look this up).
If you don't have a UNIQUE KEY defined on these two columns or you need to print out those rows which are being updated, then you do need to test for the presence of the existing keys. But this would be the way to do it:
cHandler = myDB.cursor()
cHandler.execute('select UserId,C1,LogDate from DeviceLogs_12_2019') // data from remote sql server database
rows = cHandler.fetchall()
curs = connection.cursor()
for row in rows:
curs.execute("select UserId from biometric where UserId=%s and LogDate=%s", (ros[0], row[2])) # row already in biometric table?
biometric_row = curs.fetchone()
if biometric_row is None: # no, it is not
print(row)
frappe.db.sql("Insert into biometric(UserId,C1,LogDate) values(%s, %s, %s)", (row[0],row[1],row[2]))
How do I insert a python dictionary into a Postgresql2 table? I keep getting the following error, so my query is not formatted correctly:
Error syntax error at or near "To" LINE 1: INSERT INTO bill_summary VALUES(To designate the facility of...
import psycopg2
import json
import psycopg2.extras
import sys
with open('data.json', 'r') as f:
data = json.load(f)
con = None
try:
con = psycopg2.connect(database='sanctionsdb', user='dbuser')
cur = con.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute("CREATE TABLE bill_summary(title VARCHAR PRIMARY KEY, summary_text VARCHAR, action_date VARCHAR, action_desc VARCHAR)")
for d in data:
action_date = d['action-date']
title = d['title']
summary_text = d['summary-text']
action_date = d['action-date']
action_desc = d['action-desc']
q = "INSERT INTO bill_summary VALUES(" +str(title)+str(summary_text)+str(action_date)+str(action_desc)+")"
cur.execute(q)
con.commit()
except psycopg2.DatabaseError, e:
if con:
con.rollback()
print 'Error %s' % e
sys.exit(1)
finally:
if con:
con.close()
You should use the dictionary as the second parameter to cursor.execute(). See the example code after this statement in the documentation:
Named arguments are supported too using %(name)s placeholders in the query and specifying the values into a mapping.
So your code may be as simple as this:
with open('data.json', 'r') as f:
data = json.load(f)
print(data)
""" above prints something like this:
{'title': 'the first action', 'summary-text': 'some summary', 'action-date': '2018-08-08', 'action-desc': 'action description'}
use the json keys as named parameters:
"""
cur = con.cursor()
q = "INSERT INTO bill_summary VALUES(%(title)s, %(summary-text)s, %(action-date)s, %(action-desc)s)"
cur.execute(q, data)
con.commit()
Note also this warning (from the same page of the documentation):
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
q = "INSERT INTO bill_summary VALUES(" +str(title)+str(summary_text)+str(action_date)+str(action_desc)+")"
You're writing your query in a wrong way, by concatenating the values, they should rather be the comma-separated elements, like this:
q = "INSERT INTO bill_summary VALUES({0},{1},{2},{3})".format(str(title), str(summery_text), str(action_date),str(action_desc))
Since you're not specifying the columns names, I already suppose they are in the same orders as you have written the value in your insert query. There are basically two way of writing insert query in postgresql. One is by specifying the columns names and their corresponding values like this:
INSERT INTO TABLE_NAME (column1, column2, column3,...columnN)
VALUES (value1, value2, value3,...valueN);
Another way is, You may not need to specify the column(s) name in the SQL query if you are adding values for all the columns of the table. However, make sure the order of the values is in the same order as the columns in the table. Which you have used in your query, like this:
INSERT INTO TABLE_NAME VALUES (value1,value2,value3,...valueN);
I have a FLOAT column in a MySQL table which I am trying to populate using a Python script. Most of the time for the data I'm using the values are valid floats but sometimes they are "NaN". So when inserting into the table I check if the value is "NaN" and if so I set it to None which, to my understanding, be insert as NULL in the table. This is not happening and I'm unsure as to why.
Here is the Python code:
for row in zip(plasma_potential_set, floating_potential_set):
row = [scan_result_id] + list(row)
for i in range(len(row)):
if str(row[i]).lower() == "nan":
row[i] = None
sql_insert = ('INSERT INTO langmuir_result(scan_data_id,
plasma_potential, floating_potential)'
'VALUES(%s, "%s", "%s")')
cursor.execute(sql_insert, row)
mydb.commit()
cursor.close()
Here is the CREATE statement for the table:
CREATE TABLE result (result_id INT auto_increment, scan_data_id INT,
plasma_potential FLOAT, floating_potential FLOAT, PRIMARY KEY (result_id));
Here is the error I get:
_mysql_exceptions.DataError: (1265, "Data truncated for column 'plasma_potential' at row 1")
Here is what the row looks like after I set any invalid values to None:
[1, None, 17.4651]
Why is this happening and how can I fix it? Thank you in advance.
Have looked at this question but mine is slightly different and this one doesn't have an answer: Unable to use None (NULL) values in python mysql.connector in prepared INSERT statement
You just have to remove the " in the VALUES argument list.
Change your insert statement to:
sql_insert = ('INSERT INTO langmuir_result(scan_data_id, plasma_potential, floating_potential)'
'VALUES(%s, %s, %s)')
and it will work as expected.
try:
for i in (0, nCount):
query = "INSERT INTO students (Name, ID, Birth) VALUES (%s %d %d);"
data = ('Y', 2, 9)
cur.execute(query, data)
conn.commit()
except:
print("I can't INSERT into students")
Connect to database and select operation are work but insert operation is not.
I use PostgreSQL and pgAdmin4
And create database use pgAdmin4
The DB structure is Name(text), Id(integer), Birth(integer)
-------------------------------------------------------------------------
Solve
* Change type of all columns to text (not text[]).
* Change column name in DB to small letter.
From my best knowledge you missed , in values clause. I think you need write smthng like values (%s,%d,%d)
I'm having some trouble modifying a sql table that contains a date.
This is the code I'm running:
import pymysql
cnx=pymysql.connect(dbstuff)
cursor=cnx.cursor()
cursor.execute("SELECT * FROM devices ORDER BY ip_addr,Port ASC")
results=cursor.fetchall()
lst = [(i,)+line[1:] for i, line in enumerate(results, 1)]
values = ', '.join(map(str,lst))
query="DELETE FROM devices; INSERT INTO devices VALUES {}".format(values)
query = query.replace("None","NULL")
cursor.execute(query)
cnx.commit()
cnx.close()
If the date column in sql is NULL, this runs without a problem. If there is a date inserted into the field, I get the following error:
"FUNCTION datetime.datetime does not exist."
When I look at the results of the sql select query, the date column value is converted to the following:
datetime.datetime(2016, 8, 16, 11, 24, 4)
I assume this is a python thing and not a sql thing. I haven't been to find anyway to convert this to a format sql would understand.
For datetime values you may use this:
your_datetime = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
This variable will hold a string value like this:
2018-05-06 21:14:37
And then use the following SQL command:
"INSERT INTO your_table (your_datetime_column) VALUES (str_to_date(\'%s\', '%%Y-%%m-%%d %%H:%%i:%%s'))" % (your_datetime)
DO NOT USE FOLLOWING - it won't work with the .format() method:
"INSERT INTO your_table (your_datetime_column) VALUES ({0})".format (your_datetime)
Calling the str function of the datetime.datetime object will convert it to the string format required by SQL. It is currently trying to insert a python object.
Simply wrap your object like this:
dt = str(my_datetime_object)
This error is raised when you pass invalid format date string in SQL query. Even a space in '2018-05-01 11:02:02 ' can cause this error. The best way to get rid of it is to avoid string formatting while making sql queries.
c=db.cursor()
c.execute("INSERT INTO devices VALUES(%s,%s,%s)",(value1,value2,value3))
c.close()
If you are unknown that how many values will have then, you have to generate (%s,%s...) dynamically.
import pymysql
cnx=py
mysql.connect(dbstuff)
cursor=cnx.cursor()
cursor.execute("SELECT * FROM devices ORDER BY ip_addr,Port ASC")
results=cursor.fetchall()
lst = [(i,)+line[1:] for i, line in enumerate(results, 1)]
formatt=','.join(['%s']*len(lst))
query="INSERT INTO devices VALUES (%s)"%formatt
cursor.execute(query,lst)
cnx.commit()
cnx.close()