Develop Reference

python mutiple sql execution by using insert or update

I got the following error occurs when executing sql in Python.
TypeError: not enough arguments for format string
I used executemany function to execute multiple sql statements at once without running a for statement.
And I want to execute the sql statement by receiving the list in "idx".
fruit_idx_list=['123','456','789']
fruit_list = [('123','apple',70,7),('456','strawberry',60,6),('789','banana',100,10)]
sql = "MERGE INTO fruit_test " \
+ "USING DUAL " \
+ "ON idx = {idx}".format(idx = fruit_idx_list) \ #the suspicious part1
+ "WHEN NOT MATCHED THEN " \
+ "INSERT (idx, name, price, vat) VALUES (%s, %s, %s, %s) " \
+ "WHEN MATCHED THEN " \
+ "UPDATE SET idx = %s name = %s, price = %s vat = %s; " #the suspicious part2
cur.executemany(sql, fruit_list)
conn.commit()
I think I have 2 suspicious part to occur sql error.
One is the part which pass fruit_idx_list then execute sql and another is the update statement part.
Please help me to figure out. Thank you in advance

Python variables in mysql request

I'm quite new in Python (Python 3.4.6) :)
I'm trying to insert into a mysql db some lines but with variables.
At the beginning, I've a dictionary list_hosts.
Here is my code :
import mysql.connector
import time
db = mysql.connector.connect(host='localhost', user='xxxxx', passwd='xxxxx', database='xxxxx')
cursor = db.cursor()
now_db = time.strftime('%Y-%m-%d %H:%M:%S')
for key, value in list_hosts
key_db += key+", "
value_ex += "%s, "
value_db += "\""+value+"\", "
key_db = key_db.strip(" ")
key_db = key_db.strip(",")
value_ex = value_ex.strip(" ")
value_ex = value_ex.strip(",")
value_db = value_db.strip(" ")
value_db = value_db.strip(",")
add_host = ("INSERT INTO nagios_hosts (date_created, date_modified, "+key_db+") VALUES ("+value_ex+")")
data_host = ("\""+now_db+"\", \""+now_db+"\", "+value_db)
cursor.execute(add_host, data_host)
db.commit()
db.close()
Example of list_hosts:
OrderedDict([('xxxx1', 'data1'), ('xxxx2', 'data2'), ('xxxx3', 'data3'), ('xxxx4', 'data4'), ('xxxx5', 'data5'), ('xxxx6', 'data6')])
I've simplified the code of course.
I did it like this as I've never have the same amount of items in the dictionnary.
I'm trying to create something like this :
add_host - INSERT INTO TABLE (date_created, date_modified, xxxx1, xxxx2, xxxx3, xxxx4, xxxx5, xxxx6) VALUES (%s, %s, %s, %s, %s, %s)
data_host - now, now, data1, data2, data3, data4, data5, data6
Where there are never the same number of xxxx...
They all exist in the DB, but I don't need to fill each column for each item in the dictionnary.
When I execute I get this error :
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'xxxxxxxxxxxxxxxxxxxxx' at line 1
As I'm beginning with Python, I think there are a lot of things we can clean too... don't hesitate :)

Here's a canonical python3 (python2 compatible) solution:
import time
from collections import OrderedDict
list_hosts = OrderedDict([("field1", "value1"), ("field2", "value2"), ("fieldN", "valueN")])
# builds a comma-separated string of db placeholders for the values:
placeholders = ", ".join(["%s"] * (len(list_hosts) + 2))
# builds a comma-separated string of field names
fields = ", ".join(("date_created","date_modified") + tuple(list_hosts.keys()))
# builds a tuple of values including the dates
now_db = time.strftime('%Y-%m-%d %H:%M:%S')
values = (now_db, now_db) + tuple(list_hosts.values())
# build the SQL query:
sql = "INSERT INTO nagio_hosts({}) VALUES({})".format(fields, placeholders)
# and safely execute it
cursor.execute(sql, values)
db.commit()

As #khelwood mentioned in the comments, you should use parameterized queries.
If the number of columns you're inserting varies, you might prefer to generate a tuple and use it in a parameterized query then.
cursor.execute() accepts two parameters:
a query as a string;
parameters as a tuple.
The idea is to generate the string and the tuple and pass those to cursor.execute().
You'll need something like this:
list_hosts = {'xxxx1': 'data1', 'xxxx2': 'data2', 'xxxx3': 'data3', 'xxxx4': 'data4'}
keys = [] # creating a list for keys
values = () # creating a tuple for values
for key, value in list_hosts.items():
keys.append(key)
values = values + (value,)
keys_str = ', '.join(keys)
ps = ', '.join(['%s'] * len(list_hosts))
query = "INSERT INTO tbl (%s) VALUES (%s)" % (keys_str, ps)
print(query)
# INSERT INTO tbl (data_created, data_modified, xxxx1, xxxx2, xxxx3, xxxx4) VALUES (%s, %s, %s, %s)
cursor.execute(query, values)
Just tried it on a sample data, works fine!

Use a python dictionary to insert into mysql

I am trying to take the data from a dictionary (the example is simplified for readability) and insert it into a mysql database.
I have the following piece of code.
import pymysql
conn = pymysql.connect(server, user , password, "db")
cur = conn.cursor()
ORFs={'E7': '562', 'E6': '83', 'E1': '865', 'E2': '2756 '}
table="genome"
cols = ORFs.keys()
vals = ORFs.values()
sql = "INSERT INTO %s (%s) VALUES(%s)" % (
table, ",".join(cols), ",".join(vals))
print sql
print ORFs.values()
cur.execute(sql, ORFs.values())
cur.close()
conn.close()
the print sql statement returns
INSERT INTO genome (E7,E6,E1,E2) VALUES(562,83,865,2756 )
when I type this directly into the mysql command line, the mysql command works. But when I run the python script I get an error:
<type 'exceptions.TypeError'>: not all arguments converted during string formatting
args = ('not all arguments converted during string formatting',)
message = 'not all arguments converted during string formatting'
As always, any suggestions would be highly appreciated.

The previous answer doesn't work for non string dictionary value. This one is a revised version.
format_string = ','.join(['%s'] * len(dict))
self.db.set("""INSERT IGNORE INTO listings ({0}) VALUES ({1})""".format(", ".join(dict.keys()),format_string),
(dict.values()))

sql = "INSERT INTO %s (%s) VALUES(%s)" % (
table, ",".join(cols), ",".join(vals))
This SQL includes values and cur.execute(sql, ORFs.values()) has values, too.
So, it should be cur.execute(sql).

In my case, I will skip null columns.
data = {'k': 'v'}
fs = ','.join(list(map(lambda x: '`' + x + '`', [*data.keys()])))
vs = ','.join(list(map(lambda x: '%(' + x + ')s', [*data.keys()])))
sql = "INSERT INTO `%s` (%s) VALUES (%s)" % (table, fs, vs)
count = cursor.execute(sql, data)

sqlite3.Warning: You can only execute one statement at a time

I get the error when running this code:
import sqlite3
user_name = raw_input("Please enter the name: ")
user_email = raw_input("Please enter the email: ")
db = sqlite3.connect("customer")
cursor=db.cursor()
sql = """INSERT INTO customer
(name, email) VALUES (?,?);,
(user_name, user_email)"""
cursor.execute(sql)
Why is this happening?

While the other posters are correct about your statement formatting you are receiving this particular error because you are attempting to perform multiple statements in one query (notice the ; in your query which separates statements).
From Python sqlite3 docs:
"execute() will only execute a single SQL statement. If you try to execute more than one
statement with it, it will raise a Warning. Use executescript() if you want to execute
multiple SQL statements with one call."
https://docs.python.org/2/library/sqlite3.html
Now your statement will not execute properly even if you use executescript() because there are other issues with the way it is formatted (see other posted answers). But the error you are receiving is specifically because of your multiple statements. I am posting this answer for others that may have wandered here after searching for that error.

Use executescript instead of execute
execute() will only execute a single SQL statement. If you try to execute more than one statement with it, it will raise a Warning. Use executescript() if you want to execute multiple SQL statements with one call.
https://docs.python.org/2/library/sqlite3.html#sqlite3.Cursor.execute

You have a ;, in the middle of the query string - that is an invalid syntax. Pass a dictionary as a second argument to execute if you want to use a named parameter binding.
sql = "INSERT INTO customer (name, email) VALUES (:name, :email)"
cursor.execute(sql, {'name':user_name, 'email':user_email})

Try this:
sql = """INSERT INTO customer
(name, email) VALUES (?,?)"""
cursor.execute(sql, (user_name, user_email))

import sqlite3
def DB():
List = {"Name":"Omar", "Age":"33"}
columns = ', '.join("" + str(x).replace('/', '_') + "" for x in List.keys())
values = ', '.join("'" + str(x).replace('/', '_') + "'" for x in List.values())
sql_qry = "INSERT INTO %s ( %s ) values (?,?) ; ( %s )" % ('Table Name', columns, values)
conn = sqlite3.connect("DBname.db")
curr = conn.cursor()
# curr.execute("""create table if not exists TestTable(
# Name text,
# Age text
# )""")
# print columns
# print values
# print sql
# sql = 'INSERT INTO yell (Name , Age) values (%s, %s)'
curr.execute(sql_qry)
DB()

Python/postgres/psycopg2: getting ID of row just inserted

I'm using Python and psycopg2 to interface to postgres.
When I insert a row...
sql_string = "INSERT INTO hundred (name,name_slug,status) VALUES ("
sql_string += hundred_name + ", '" + hundred_slug + "', " + status + ");"
cursor.execute(sql_string)
... how do I get the ID of the row I've just inserted? Trying:
hundred = cursor.fetchall()
returns an error, while using RETURNING id:
sql_string = "INSERT INTO domes_hundred (name,name_slug,status) VALUES ("
sql_string += hundred_name + ", '" + hundred_slug + "', " + status + ") RETURNING id;"
hundred = cursor.execute(sql_string)
simply returns None.
UPDATE: So does currval (even though using this command directly into postgres works):
sql_string = "SELECT currval(pg_get_serial_sequence('hundred', 'id'));"
hundred_id = cursor.execute(sql_string)
Can anyone advise?
thanks!

cursor.execute("INSERT INTO .... RETURNING id")
id_of_new_row = cursor.fetchone()[0]
And please do not build SQL strings containing values manually. You can (and should!) pass values separately, making it unnecessary to escape and SQL injection impossible:
sql_string = "INSERT INTO domes_hundred (name,name_slug,status) VALUES (%s,%s,%s) RETURNING id;"
cursor.execute(sql_string, (hundred_name, hundred_slug, status))
hundred = cursor.fetchone()[0]
See the psycopg docs for more details: http://initd.org/psycopg/docs/usage.html#passing-parameters-to-sql-queries

I ended up here because I had a similar problem, but we're using Postgres-XC, which doesn't yet support the RETURNING ID clause. In that case you can use:
cursor.execute('INSERT INTO ........')
cursor.execute('SELECT LASTVAL()')
lastid = cursor.fetchone()['lastval']
Just in case it was useful for anyone!

Consider a RETURNING clause http://www.postgresql.org/docs/8.3/static/sql-insert.html

For me, neither ThiefMaster's answer worked nor Jamie Brown's. What worked for me was a mix of both, and I'd like to answer here so it can help someone else.
What I needed to do was:
cursor.execute('SELECT LASTVAL()')
id_of_new_row = cursor.fetchone()[0]
The statement lastid = cursor.fetchone()['lastval'] didn't work for me, even after cursor.execute('SELECT LASTVAL()'). The statement id_of_new_row = cursor.fetchone()[0] alone didn't work either.
Maybe I'm missing something.

ThiefMaster's approach worked for me, for both INSERT and UPDATE commands.
If cursor.fetchone() is called on a cursor after having executed an INSERT/UPDATE command but lacked a return value (RETURNING clause) an exception will be raised: ProgrammingError('no results to fetch'))
insert_query = """
INSERT INTO hundred (id, name, name_slug, status)
VALUES (DEFAULT, %(name)s, %(name_slug)s, %(status)s)
RETURNING id;
"""
insert_query_values = {
"name": "",
"name_slug": "",
"status": ""
}
connection = psycopg2.connect(host="", port="", dbname="", user="", password="")
try:
with connection:
with connection.cursor() as cursor:
cursor.execute(insert_query, insert_query_values)
num_of_rows_affected = cursor.rowcount
new_row_id = cursor.fetchone()
except psycopg2.ProgrammingError as ex:
print("...", ex)
raise ex
finally:
connection.commit()
connection.close()