I am trying to iterate through a dataframe and fetching values from indivdidual column to use as my parameters in sql query.
for index,frame in df1.iterrows():
sql = "select * from issuers where column_1 = %s;"
cur.execute(sql, frame['column_1'])
row = cur.fetchone()
id = row[0]
print id
But I am getting the following error
"TypeError: not all arguments converted during string formatting"
How can I solve this? In case I need to add multiple parameters, how can I do that?
Instead of this:
cur.execute(sql, frame['column_1'])
Try this:
cur.execute(sql, [frame['column_1']])
The second parameter of execute is a list containing all values to be inseted into sql.
In order to insert multiple values use something as follows:
sql = "select * from issuers where column_1 = %s and column_2 = %s;"
cur.execute(sql, ["val1", "val2"])
For more information please refere to the documentation
EDIT
Here an example for INSERT INTO in SQL.
sql = "INSERT INTO user (firstname, lastname) VALUES (%s, %s)"
cur.execute(sql, ["John", "Doe"])
Related
I am struggling with generating the delete query where parameters for the query is actually a set of values.
So I need to delete rows where parameters are a pair values for example:
delete from table where col1 = %s and col2 = %s
which can be executed in Python like:
cur = conn.cursor()
cur.execute(query, (col1_value, col2_value))
Now I would like to run a query:
delete from table where (col1, col2) in ( (col1_value1, col2_value1), (col1_value2, col2_value2) );
I can generate the queries and values and execute the exact SQL but I can't quite generate prepared statement.
I tried:
delete from table where (col1, col2) in %s
and
delete from table where (col1, col2) in (%s)
But when I try to execute:
cur.execute(query, list_of_col_value_tuples)
or
cur.execute(query, tuple_of_col_value_tuples)
I get an exception that indicates that psycopg2 cannot convert arguments to strings.
Is there any way to use psycopg2 to execute a query like this?
You could dynamically add %s placeholders to your query:
cur = con.cursor()
query = "delete from table where (role, username) in (%s)"
options = [('admin', 'foo'), ('user', 'bar')]
placeholders = '%s,' * len(options)
query = query % placeholders[:-1] # remove last comma
print(query)
print(cur.mogrify(query, options).decode('utf-8'))
Out:
delete from table where (role, user) in (%s,%s)
delete from table where (role, user) in (('admin', 'foo'),('user', 'bar'))
Alternatively, build the query using psycopg2.sql as answered there.
Actually the resolution is quite easy if carefully constructed.
In the miscellaneous goodies of psycopg2 there is a function execute_values.
While all the examples that are given by psycopg2 deal with inserts as the function basically converts the list of arguments into a VALUES list if the call to delete is formatted like so:
qry = "delete from table where (col1, col2) in (%s)"
The call:
execute_values(cur=cur, qry=qry, argslist=<list of value tuples>)
will make the delete perform exactly as required.
Need to pass 2 variables to sql query body
The official documentation says that parameters are passed through a variable as follows, but this only works with one parameter:
SQL = "INSERT INTO authors (name) VALUES (%s);" # Note: no quotes
data = ("O'Reilly", )
cur.execute(SQL, data)
how it would be possible to pass two variables, that is, something like this:
SQL = "INSERT INTO authors (%s) VALUES (%s);"
data_1 = ("name",)
data_2 = ("O'Reilly", )
cur.execute(SQL, data_1,data_2)
I'm trying to insert rows on a MySQL table using pymysql (Python 3), the relevant code is the following.
def saveLogs(DbConnection, tableName, results):
for row in results:
formatStrings = ",".join(["?"]*len(row))
sql = "INSERT INTO %s VALUES (%s);"%(tableName,formatStrings)
DbConnection.cursor().execute(sql, tuple(row))
DbConnection.commit()
I'm using "?" for the types, but I get the error not all arguments converted during string formatting. row is a list composed of strings, ints and datetime.datetime. I guess the issue is the "?" but I have checked the PEP 249 and it's still not clear to me how should I do it. Any suggestions?
Use string formatting for the table name only (though make sure you trust the source or have a proper validation in place). For everything else, use query parameters:
def saveLogs(DbConnection, tableName, results):
cursor = DbConnection.cursor()
sql = "INSERT INTO {0} VALUES (%s, %s, %s)".format(tableName)
for row in results:
cursor.execute(sql, row)
DbConnection.commit()
There is also that executemany() method:
def saveLogs(DbConnection, tableName, results):
cursor = DbConnection.cursor()
cursor.executemany("INSERT INTO {0} VALUES (%s, %s, %s)".format(tableName), results)
DbConnection.commit()
I know how to use MySQL pattern matching, for example:
SELECT * FROM table WHERE col LIKE '%mid%'
I also know how to bind parameters into a sql query in python, for example:
import MySQLdb
s = 'something'
db = MySQLdb.connect(host=blablabla...)
cur = db.cursor()
sql = "SELECT * FROM table WHERE col = %s"
cur.execute(sql, s)
data = cur.fetchall()
db.close()
But I can't find a method to combine these together in one query, like
sql = "SELECT * FROM table WHERE col LIKE '%%s%'"
cur.execute(sql, s)
where the first and the third '%' are pattern character and the middle '%s' is used to bind parameter s.
Anyone have an idea?
Alright, I will answer myself..
#dsgdfg inspired me and here is my code:
sql = "SELECT * FROM table WHERE col LIKE %s"
cur.execute(sql, "%"+s+"%")
sql = "SELECT * FROM table WHERE col LIKE CONCAT('%', %s, '%')"
cur.execute(sql, s)
(I am assuming that execute deals with escaping, thereby preventing SQL injection.)
for the people using py charm
you can do this
qq=input("enter the author name:")
print(pd.read_sql_query("select Book_Id,Book_Name,book_author from bookd where book_author like '%s'" %("%"+qq+"%",), conn2))
I am trying to access PostgreSQL using psycopg2:
sql = """
SELECT
%s
FROM
table;
"""
cur = con.cursor()
input = (['id', 'name'], )
cur.execute(sql, input)
data = pd.DataFrame.from_records(cur.fetchall())
However, the returned result is:
0
0 [id, name]
1 [id, name]
2 [id, name]
3 [id, name]
4 [id, name]
If I try to access single column, it looks like:
0
0 id
1 id
2 id
3 id
4 id
It looks like something is wrong with the quoting around column name (single quote which should not be there):
In [49]: print cur.mogrify(sql, input)
SELECT
'id'
FROM
table;
but I am following doc: http://initd.org/psycopg/docs/usage.html#
Anyone can tell me what is going on here? Thanks a lot!!!
Use the AsIs extension
import psycopg2
from psycopg2.extensions import AsIs
column_list = ['id','name']
columns = ', '.join(column_list)
cursor.execute("SELECT %s FROM table", (AsIs(columns),))
And mogrify will show that it is not quoting the column names and passing them in as is.
Nowadays, you can use sql.Identifier to do this in a clean and secure way :
from psycopg2 import sql
statement = """
SELECT
{id}, {name}
FROM
table;
"""
with con.cursor() as cur:
cur.execute(sql.SQL(statement).format(
id=sql.SQL.Identifier("id"),
name=sql.SQL.Identifier("name")
))
data = pd.DataFrame.from_records(cur.fetchall())
More information on query composition here : https://www.psycopg.org/docs/sql.html
The reason was that you were passing the string representation of the array ['id', 'name'] as SQL query parameter but not as the column names. So the resulting query was similar to
SELECT 'id, name' FROM table
Looks your table had 5 rows so the returned result was just this literal for each row.
Column names cannot be the SQL query parameters but can be just the usual string parameters which you can prepare before executing the query-
sql = """
SELECT
%s
FROM
table;
"""
input = 'id, name'
sql = sql % input
print(sql)
cur = con.cursor()
cur.execute(sql)
data = pd.DataFrame.from_records(cur.fetchall())
In this case the resulting query is
SELECT
id, name
FROM
table;