I am struggling with generating the delete query where parameters for the query is actually a set of values.
So I need to delete rows where parameters are a pair values for example:
delete from table where col1 = %s and col2 = %s
which can be executed in Python like:
cur = conn.cursor()
cur.execute(query, (col1_value, col2_value))
Now I would like to run a query:
delete from table where (col1, col2) in ( (col1_value1, col2_value1), (col1_value2, col2_value2) );
I can generate the queries and values and execute the exact SQL but I can't quite generate prepared statement.
I tried:
delete from table where (col1, col2) in %s
and
delete from table where (col1, col2) in (%s)
But when I try to execute:
cur.execute(query, list_of_col_value_tuples)
or
cur.execute(query, tuple_of_col_value_tuples)
I get an exception that indicates that psycopg2 cannot convert arguments to strings.
Is there any way to use psycopg2 to execute a query like this?
You could dynamically add %s placeholders to your query:
cur = con.cursor()
query = "delete from table where (role, username) in (%s)"
options = [('admin', 'foo'), ('user', 'bar')]
placeholders = '%s,' * len(options)
query = query % placeholders[:-1] # remove last comma
print(query)
print(cur.mogrify(query, options).decode('utf-8'))
Out:
delete from table where (role, user) in (%s,%s)
delete from table where (role, user) in (('admin', 'foo'),('user', 'bar'))
Alternatively, build the query using psycopg2.sql as answered there.
Actually the resolution is quite easy if carefully constructed.
In the miscellaneous goodies of psycopg2 there is a function execute_values.
While all the examples that are given by psycopg2 deal with inserts as the function basically converts the list of arguments into a VALUES list if the call to delete is formatted like so:
qry = "delete from table where (col1, col2) in (%s)"
The call:
execute_values(cur=cur, qry=qry, argslist=<list of value tuples>)
will make the delete perform exactly as required.
Related
I'm trying to pass the same parameters to an oracle query in two separate places in the SQL code.
My code works if I hard code the criteria for table2 like this:
# define parameters
years = ['2018','2019']
placeholder= ':d'
placeholders= ', '.join(placeholder for unused in years)
placeholders
# create cursor
cursor = connection.cursor()
# query
qry = """
select * from table1
INNER
JOIN table2
ON table1_id = table2_id
where table1_year in (%s)
and table2_year in ['2018','2019'] --here's where I say I'm hard coding criteria
""" % placeholders
data = cursor.execute(qry, years)
df = pd.DataFrame(data.fetchall(), columns = [column[0] for column in cursor.description])
# close database connection
connection.close()
If I try to use the parameter for table2 like this:
qry = """
select * from table1
INNER
JOIN table2
ON table1_id = table2_id
where table1_year in (%s)
and table2_year in (%s) --part of code I'm having issues with
""" % placeholders
I get the following error:
TypeError: not enough arguments for format string
I can't simply rewrite the SQL because I frequently have to use someone else's code and it wouldn't be feasible to rewrite all of it.
If you want to fill multiple placeholders, you have to supply the same number of parameters.
"one meal: %s" % "sandwich" # works
"two meals: %s, %s" % "sandwich" # not working
"two meals: %s, %s" % ("sandwich", "sandwich") # works
NOTE: It is a bad/dangerous thing to use string formatting for the assembly of SQL queries (lookup "SQL Injection"). In your case it is fine, but in general you should use parameterized queries, especially when dealing with input from untrusted sources like user input. You don't want a user to input "2018; DROP TABLE table1;".
I am trying to iterate through a dataframe and fetching values from indivdidual column to use as my parameters in sql query.
for index,frame in df1.iterrows():
sql = "select * from issuers where column_1 = %s;"
cur.execute(sql, frame['column_1'])
row = cur.fetchone()
id = row[0]
print id
But I am getting the following error
"TypeError: not all arguments converted during string formatting"
How can I solve this? In case I need to add multiple parameters, how can I do that?
Instead of this:
cur.execute(sql, frame['column_1'])
Try this:
cur.execute(sql, [frame['column_1']])
The second parameter of execute is a list containing all values to be inseted into sql.
In order to insert multiple values use something as follows:
sql = "select * from issuers where column_1 = %s and column_2 = %s;"
cur.execute(sql, ["val1", "val2"])
For more information please refere to the documentation
EDIT
Here an example for INSERT INTO in SQL.
sql = "INSERT INTO user (firstname, lastname) VALUES (%s, %s)"
cur.execute(sql, ["John", "Doe"])
try:
for i in (0, nCount):
query = "INSERT INTO students (Name, ID, Birth) VALUES (%s %d %d);"
data = ('Y', 2, 9)
cur.execute(query, data)
conn.commit()
except:
print("I can't INSERT into students")
Connect to database and select operation are work but insert operation is not.
I use PostgreSQL and pgAdmin4
And create database use pgAdmin4
The DB structure is Name(text), Id(integer), Birth(integer)
-------------------------------------------------------------------------
Solve
* Change type of all columns to text (not text[]).
* Change column name in DB to small letter.
From my best knowledge you missed , in values clause. I think you need write smthng like values (%s,%d,%d)
I'm trying to insert rows on a MySQL table using pymysql (Python 3), the relevant code is the following.
def saveLogs(DbConnection, tableName, results):
for row in results:
formatStrings = ",".join(["?"]*len(row))
sql = "INSERT INTO %s VALUES (%s);"%(tableName,formatStrings)
DbConnection.cursor().execute(sql, tuple(row))
DbConnection.commit()
I'm using "?" for the types, but I get the error not all arguments converted during string formatting. row is a list composed of strings, ints and datetime.datetime. I guess the issue is the "?" but I have checked the PEP 249 and it's still not clear to me how should I do it. Any suggestions?
Use string formatting for the table name only (though make sure you trust the source or have a proper validation in place). For everything else, use query parameters:
def saveLogs(DbConnection, tableName, results):
cursor = DbConnection.cursor()
sql = "INSERT INTO {0} VALUES (%s, %s, %s)".format(tableName)
for row in results:
cursor.execute(sql, row)
DbConnection.commit()
There is also that executemany() method:
def saveLogs(DbConnection, tableName, results):
cursor = DbConnection.cursor()
cursor.executemany("INSERT INTO {0} VALUES (%s, %s, %s)".format(tableName), results)
DbConnection.commit()
I need to write an INSERT statement that first checks to see if the data already exists. The current code is inside python using psycopg2 to connect to a postgresql db:
sql = """IF NOT EXISTS (SELECT * FROM table \
WHERE col_1 = (%s) AND col_2 = (%s) ) \
INSERT INTO table (col1, col2) \
VALUES (%s, %s);"""
data = ( col1_data, col2_data, col1_data, col2_data)
try:
CURSOR.execute(sql, data)
DB.commit()
except:
print "Cursor failed INSERT INTO table.\n"
which does not work (and I haven't done quality error handling so I don't get any good information).
So, I went into psql and tried just:
IF NOT EXISTS (SELECT * FROM t WHERE c1=d1 AND c2=d2)
INSERT INTO t (c1, c2) VALUES (d1,d2);
and I got the following error:
ERROR: syntax error at or near "IF"
LINE 1: IF NOT EXISTS (SELECT * FROM table WHERE c1 = d1...
^
So I BELIEVE my error is in the sql not the python (though I could be wrong) since this works:
sql = """INSERT INTO t2 (col_0, col_1, col_2) \
VALUES (%s, %s, %s);"""
data = (d1, d2, time.time())
try:
CURSOR.execute(sql, data)
DB.commit()
except:
print "Cursor failed to INSERT INTO t2.\n"
For table 1, my CREATE was:
db=> CREATE TABLE table (
col_0 SERIAL PRIMARY KEY,
col_1 varchar(16),
col_2 smallint
);
NOTICE: CREATE TABLE will create implicit sequence "pm_table_ip_id_seq" for serial column "pm_table.ip_id"
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "pm_table_pkey" for table "pm_table"
CREATE TABLE
I am grateful for any help and guidance.
I used plpgsql for such a requirement in my project
insert_function = """
CREATE LANGUAGE plpgsql;
CREATE FUNCTION insert_if_unique (sql_insert TEXT)
RETURNS VOID
LANGUAGE plpgsql
AS $$
BEGIN
EXECUTE sql_insert;
RETURN;
EXCEPTION WHEN unique_violation THEN
RETURN;
-- do nothing
END;
$$;
"""
cursor.execute(insert_function);
You can use something like below to use it:
cursor.execute("insert_if_unique("+sql+")"%data)
The above query is not parameterized. So please be wary of SQL injection if you are getting the input from an external source.
Note: You can use cursor.mogrify() to evade SQL injection attacks.
sql = cursor.mogrify(sql,data)
cursor.execute("insert_if_unique("+sql+")")
Try reversing those. NOT EXISTS condition with a subquery:
INSERT INTO t (c1, c2) VALUES (d1,d2)
WHERE NOT EXISTS (SELECT * FROM t WHERE c1=d1 AND c2=d2)