I'm having this function that communicates via pymysql to an SQL database stored to my localhost. I know there are similar posts about formatting an SQL section especially this one but could anyone suggest a solution?
Always getting TypeError: can't concat tuple to bytes. I suppose it's sth with the WHERE clause.
def likeMovement(pID):
print("Give a rating for the movement with #id:%s" %pID)
rate=input("Give from 0-5: ")
userID=str(1)
print(rate,type(rate))
print(pID,type(pID))
print(userID,type(userID))
cursor=con.cursor()
sqlquery='''UDPATE likesartmovement SET likesartmovement.rating=%s WHERE
likesartmovement.artisticID=? AND likesartmovement.userID=?''' % (rate,),
(pID,userID)
cursor.execute(sqlquery)
TypeError: not all arguments converted during string formatting
Thanks in advance!
The problem is that you're storing (pID,userID) as part of a tuple stored in sqlquery, instead of passing them as the arguments to execute:
sqlquery='''UDPATE likesartmovement SET likesartmovement.rating=%s WHERE
likesartmovement.artisticID=? AND likesartmovement.userID=?''' % (rate,)
cursor.execute(sqlquery, (pID,userID))
It may be clearer to see why these are different if you take a simpler example:
s = 'abc'
spam(s, 2)
s = 'abc', 2
spam(s)
Obviously those two don't do the same thing.
While we're at it:
You have to spell UPDATE right.
You usually want to use query parameters for SET clauses for exactly the same reasons you want to for WHERE clauses.
You don't need to include the table name in single-table operations, and you're not allowed to include the table name in SET clauses in single-table updates.
So:
sqlquery='''UDPATE likesartmovement SET rating=? WHERE
artisticID=? AND userID=?'''
cursor.execute(sqlquery, (rating, pID, userID))
I am trying to extract the names from the db that have A in the second position.
In sql it's simple but python sees the '_A%' as end of query.
Has anyone faced this problem before and came out with a solution?
I saw a similar question and the accept result was to use '% %' instead of ' %', but this didn't worked.
This is my query:
def queryDelivery(start_date):
query_basictable = """
SELECT Code,Quantity, Datetime
FROM Mytable
WHERE Datetime>= '%s 12:00:00' AND Name LIKE '_A%'
""" %(start_date)
delivery_data= pd.read_sql(sql=query_basictable, con=engine)
return delivery_data
I was thinking about passing the symbol '_A%' to a variable and the do something like a substitute but when try to assign the symbol hits syntax error
variable = ''_A%' '
Name LIKE variable
How can I do this in a clean way?
Don't do it this way. As soon as you do this, if someone inserts a start_date like "'; drop table students; --" you have a problem.
I tested placeholders in Python 2.7 and it looks like you don't run into the problem until you use the % operator.
A much better way is to write your SQL statements in a way that every value passed in can be used in a placeholder. Then use placeholder syntax and the syntax becomes LIKE ? || '%'
What I want is execute the sql
select * from articles where author like "%steven%".
For the sake of safety, i used like this way :
cursor.execute('select * from articles where %s like %s', ('author', '%steven%')
Then the result is just empty, not get a syntax error, but just empty set.
But I am pretty sure there is some thing inside, I can get result use the first sql. Is there anything run with my code ?
You can't set a column name like a parameter where you're doing where %s like %s. To dynamically set the column name you need to do actual string manipulation like:
sql = 'select * from articles where '+ sql_identifier('author') +' like %s'
cursor.execute(sql, ('%steven%',))
Where sql_identifier is your lib's function for making an identifier safe for SQL injection. Something like:
# don't actually use this!
def sql_identifier(s):
return '"%s"' % s.replace('"','')
But with actual testing and knowledge of the DB engine you're using.
The problem here is fact a minor mistake. Thanks to #Asad Saeeduddin, when I try to use print cursor._last_executed to check what has happened. I found that what is in fact executed is
SELECT * FROM articles WHERE 'title' LIKE '%steven%', look the quotation mark around the title, that's the reason why I got empty set.
So always remember the string after formatting will have a quotation around
Check out the following lines of Python 2.6 code I found:
key = 'hire_date'
update_dict['key'] = update_dict[key] #added e.g. {..., 'key': '12/31/1999'}
if key == 'hire_date':
query_string = "UPDATE employee SET " + key + "= CAST(%(key)s AS DATE) WHERE emp_id = '" + emp.employee + "'"
I've tested this code, and it works. It successfully updates the employee's hire_date field in the database to whatever date 'key''s value in the dictionary is.
I was in the middle of parameterizing it when I noticed the %(key)s somehow manages to get the value of the dictionary at 'key'. How does it do that? I always thought you had to add % dictionaryOrTupleOrWhatever after the string for this to work.
In the code that you pasted, no interpolation takes place. However, the variable could later be interpolated:
>>> x = "%(var)s" # no interpolation yet
>>> d = {'var': 88}
>>> x % d # interpolate into the stored string
'88'
Given that the code you posted looks like SQL, it could also be interpolated later via an SQL library call. Many SQL interface libraries provide a similar sort of string-substitution using % signs and encourage users to use these rather than the built-in string substitution, since the SQL library versions have various safeguards to prevent malicious injection attacks.
I bet you'll find later in the code that there is a DB API execute statement that takes update_dict as a parameter. The DB API then does the substitution instead of Python string formatting and thus properly handles binding.
Have a look at this: http://furius.ca/pubcode/pub/antiorm/lib/python/dbapiext.html#escaping-in-the-dbapi-2-0
For some reasons, I would like to do an explicit quoting of a string value (becoming a part of constructed SQL query) instead of waiting for implicit quotation performed by cursor.execute method on contents of its second parameter.
By "implicit quotation" I mean:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;"
cursor.execute( query, (value,) ) # value will be correctly quoted
I would prefer something like that:
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
READY_TO_USE_QUOTING_FUNCTION(value)
cursor.execute( query ) # value will be correctly quoted, too
Is such low level READY_TO_USE_QUOTING_FUNCTION expected by Python DB API specification (I couldn't find such functionality in PEP 249 document). If not, maybe Psycopg2 provides such function? If not, maybe Django provides such function? I would prefer not to write such function myself...
Ok, so I was curious and went and looked at the source of psycopg2. Turns out I didn't have to go further than the examples folder :)
And yes, this is psycopg2-specific. Basically, if you just want to quote a string you'd do this:
from psycopg2.extensions import adapt
print adapt("Hello World'; DROP DATABASE World;")
But what you probably want to do is to write and register your own adapter;
In the examples folder of psycopg2 you find the file 'myfirstrecipe.py' there is an example of how to cast and quote a specific type in a special way.
If you have objects for the stuff you want to do, you can just create an adapter that conforms to the 'IPsycopgSQLQuote' protocol (see pydocs for the myfirstrecipe.py-example...actually that's the only reference I can find to that name) that quotes your object and then registering it like so:
from psycopg2.extensions import register_adapter
register_adapter(mytype, myadapter)
Also, the other examples are interesting; esp. 'dialtone.py' and 'simple.py'.
I guess you're looking for the mogrify function.
Example:
>>> cur.mogrify("INSERT INTO test (num, data) VALUES (%s, %s)", (42, 'bar'))
"INSERT INTO test (num, data) VALUES (42, E'bar')"
You should try to avoid doing your own quoting. Not only will it be DB-specific as people have pointed out, but flaws in quoting are the source of SQL injection bugs.
If you don't want to pass around queries and values separately, then pass around a list of the parameters:
def make_my_query():
# ...
return sql, (value1, value2)
def do_it():
query = make_my_query()
cursor.execute(*query)
(I probably have the syntax of cursor.execute wrong) The point here is that just because cursor.execute takes a number of arguments, that doesn't mean you have to handle them all separately. You can deal with them as one list.
This'll be database dependent (iirc, mysql allows \ as an escape character, while something like oracle expects quotes to be doubled: 'my '' quoted string').
Someone correct me if i'm wrong, but the double-quoting method is the standard method.
It may be worth looking at what other db abstraction libraries do (sqlalchemy, cx_Oracle, sqlite, etc).
I've got to ask - why do you want to inline the values instead of bind them?
This is going to be DB dependent. In the case of MySQLdb, for example, the connection class has a literal method that will convert the value to the correct escaped representation for passing to MySQL (that's what cursor.execute uses).
I imagine Postgres has something similar, but I don't think there is a function to escape values as part of the DB API 2.0 spec.
I don't think you give any sufficient reasoning behind your avoidance to do this The Right Way. Please, use the APi as it is designed and don't try so hard to make your code less readable for the next guy and more fragile.
Your code snippet would get just like this, according to psycopg extension docs
from psycopg2.extensions import adapt
value = "Unsafe string"
query = "SELECT * FROM some_table WHERE some_char_field = %s;" % \
adapt(value).getquoted()
cursor.execute( query ) # value will be correctly quoted, too
The getquoted function returns the value as a quoted and escaped string, so you could also go: "SELECT * FROM some_table WHERE some_char_field = " + adapt(value).getquoted() .
PyPika in another good option for building SQL statements. Usage example (based on an example on the project's homepage):
>>> from pypika import Order, Query
>>> Query.from_('customers').select('id', 'fname', 'lname', 'phone').orderby('id', order=Order.desc)
SELECT "id","fname","lname","phone" FROM "customers" ORDER BY "id" DESC
If you use django you might want to use the quoting function which is automatically adapted to the currently configured DBMS :
from django.db import backend
my_quoted_variable = backend.DatabaseOperations().quote_name(myvar)
import re
def db_quote(s):
return "\"" + re.escape(s) + "\""
can do the job of simple quoting that works at least with MySQL. What we really need, though is cursor.format() function that would work like cursor.execute() except it would return the resulting query instead of executing it. There are times when you do not want the query to be executed quite yet - e.g you may want to log it first, or print it out for debugging before you go ahead with it.