I have a little script that creates a certain INSERT SQL statement for me.
For postgresql I need to wrap the values to be inserted within two single quotes.
Unfortunately some of the value strings to be inserted also contain a single quote, and I need to escape them automatically.
for line in f:
out.write('(\'' + line[:2] + '\', \'' + line[3:-1] + '\'),\n')
How can I make sure that any single quote (e.g. ' ) inside line[3:-1] is automatically escaped?
Thanks,
UPDATE:
e.g. the line
CI|Cote D'ivoire
fails due '
Update 2:
I can't use double quotes in values, e.g.
INSERT INTO "App_country" (country_code, country_name) VALUES ("AF", "Afghanistan")
I get the error message: ERROR: column "AF" does not exist
This however works fine:
INSERT INTO "App_country" (country_code, country_name) VALUES ('AF', 'Afghanistan')
As described in the PEP-249, the DBPI is a generic interface to various databases. Different implementations exist for different databases. For postgres there is psycopg. from the docs:
cur.execute(
... """INSERT INTO some_table (an_int, a_date, a_string)
... VALUES (%s, %s, %s);""",
... (10, datetime.date(2005, 11, 18), "O'Reilly"))
You simple pass your parameters in a tuple. The underlying library escapes it for you. This is much safer and easier than trying to roll your own.
The SQL standard way to escape a quote is to double it:
'This won''t be a problem.'
So replace every quote with two quotes (and use double quotes in Python to stay sane):
out.write("('" + line[:2] + "', '" + line[3:-1].replace("'", "''") + "'),\n")
Never use a generated, rolled-your-own escaping for DML. Use the appropriate DBAPI as Keith has mentioned. Work would have gone into that to make sure escapes from various sources and type conversion can occur almost transparently. If you're using DDL such as a CREATE TABLE whatever (...) - you can be more slight slack-handed if you trust your own datasource.
using data shown in example:
import sqlite3
text = "CI|Cote D'ivoire" # had to been escaped as it's a string literal, but from another data source - possibly not...
code, name = text.split('|', 1)
db = sqlite3.connect(':memory:')
db.execute('create table something(code, name)')
db.execute('insert into something(code, name) values(?, ?)', (code, name))
for row in db.execute('select * from something'):
print row
# (u'CI', u"Cote D'ivoire")
For a complete solution toadd escape characters to a string, use:
re.escape(string)
>>> re.escape('\ a.*$')
'\\\\\\ a\\.\\*\\$'
for more, see: http://docs.python.org/library/re.html
Not sure if there are some SQL related limitations, but you could always use double quotes to surround your string that contains the single quote.
Eg.
print "That's all Folks!"
or single quotes to surround double quotes:
print 'The name of the file is "rosebud".'
Related
I'm looking to escape special characters in string for Python 2.7.
For example, if I have :
str = "You're the best "dog" on earth."
I would have :
str = "You\'re the best \"dog\" on earth."
I want it because I'm inserting strings in SQL database using pymySQL and I can't find a way to do this.
I guess escaping characters must be like this ? (not really sure)
I also would find a way to do the reverse action remove escpaing characters.
You are approaching this entirely the wrong way. You should never need to escape special characters when inserting a string into a SQL database: always use parametrised SQL queries and any needed escaping will be done for you. If you start trying to escape the strings yourself you are opening your code up to all manner of security problems.
with connection.cursor() as cursor:
# Create a new record
sql = "INSERT INTO `mytable` (`thestring`) VALUES (%s)"
cursor.execute(sql, (str,))
If you ever find yourself building a query string out of data that has come from any outside source stop and reconsider: you should never need to do that.
You don't need to escape values for the purpose of SQL by hand! Let the database API take care of that.
Form a valid string literal in Python source code:
str = "You're the best \"dog\" on earth."
str = 'You\'re the best "dog" on earth.'
str = """You're the best "dog" on earth."""
These are all equivalent, you just need to escape the appropriate quotes that you're using as string literal terminators.
Use the database API correctly and don't worry about escaping. From the manual:
sql = "INSERT INTO `users` (`email`, `password`) VALUES (%s, %s)"
cursor.execute(sql, ('webmaster#python.org', 'very-secret'))
Escaping is handled by separating the query and values, not by adding backslashes.
I would like to store windows path in MySQL without escaping the backslashes. How can I do this in Python? I am using MySQLdb to insert records into the database. When I use MySQLdb.escape_string(), I notice that the backslashes are removed.
Have a look at os.path.normpath(thePath)
I can't remember if it's that one, but there IS a standard os.path formating function that gives double backslashes, that can be stored in a db "as is" and reused later "as is". I have no more windows machine and cannot test it anymore.
Just use a dictionary to add slashes wherever required to make the query valid :
http://codepad.org/7mjbwKBf
def addslashes(s):
dict = {"\0":"\\\0", "\\":"\\\\"} #add more here
return ''.join(dict.get(x,x) for x in s)
query = "INSERT INTO MY_TABLE id,path values(23,'c:\windows\system\')";
print(addslashes(query));
I have a python script that reads raw movie text files into an sqlite database.
I use re.escape(title) to add escape chars into the strings to make them db safe before executing the inserts.
Why does this not work:
In [16]: c.execute("UPDATE movies SET rating = '8.7' WHERE name='\'Allo\ \'Allo\!\"\ \(1982\)'")
--------------------------------------------------------------------------- OperationalError Traceback (most recent call last)
/home/rajat/Dropbox/amdb/<ipython console> in <module>()
OperationalError: near "Allo": syntax error
Yet this works (removed \' in two places) :
In [17]: c.execute("UPDATE movies SET rating = '8.7' WHERE name='Allo\ Allo\!\"\ \(1982\)'") Out[17]: <sqlite3.Cursor object at 0x9666e90>
I can't figure it out. I also can't ditch those leading quotes because they're actually part of the movie title.
Thank you.
You're doing it wrong. Literally. You should be using parameters, like this:
c.execute("UPDATE movies SET rating = ? WHERE name = ?", (8.7, "'Allo 'Allo! (1982)"))
Like that, you won't need to do any quoting at all and (if those values are coming from anyone untrusted) you'll be 100% safe (here) from SQL injection attacks too.
I use re.escape(title) to add escape
chars into the strings to make them db
safe
Note that re.escape makes a string re-safe -- nothing to do with making it db safe. Rather, as #Donal says, what you need is the parameter substitution concept of the Python DB API -- that makes things "db safe" as you need.
SQLite doesn't support backslash escape sequences. Apostrophes in string literals are indicated by doubling them: '''Allo ''Allo! (1982)'.
But, like Donal said, you should be using parameters.
I've one simple tip you could use to handle this problem:
When your SQL statement string has single quote:', then you could use double quote to enclose your statement string. And when your SQL statement string has double quotes:", then you could use single quote:" to enclose your statement string.
E.g.
sqlString="UPDATE movies SET rating = '8.7' WHERE name='Allo Allo !' (1982 )"
c.execute(sqlString)
Or,
sqlString='UPDATE movies SET rating = "8.7" WHERE name="Allo Allo !" (1982 )'
c.execute(sqlString)
This solution works for me in Python environment.
I'm parsing a xml file and inserting it into database.
However since some text containes double or single quotation I'm having problem with insertion. Currently I'm using the code shown below. But it seems it's inefficient.
s = s.replace('"', ' ')
s = s.replace("'", ' ')
Is there any way I can insert text without replacing these quotations?
OR
Is there any efficient way to substitute them efficiently ?
Thanks !
Why can't you insert strings containing quote marks into your database? Is there some weird data type that permits any character except a quote mark? Or are you building an insert statement with literal strings, rather than binding your strings to query parameters as you should be doing?
If you're doing
cursor.execute('insert into mytable (somefield) values ("%s")' % (mystring))
then that's unsafe and wrong. Instead, you should be doing
cursor.execute('insert into mytable (somefield) values (%(myparam)s)',
dict(myparam=mystring))
you should use str.translate instead of doing two replace() calls
>>> import string
>>> quotes_to_spaces=string.maketrans('"\''," ")
>>> s=s.translate(quotes_to_spaces)
You could try something like _mysql.escape_string():
>>> import _mysql
>>> a = '''I said, "Don't do that"'''
>>> a
'I said, "Don\'t do that"'
>>> _mysql.escape_string(a)
'I said, \\"Don\\\'t do that\\"'
However, the manual recommends using connection.escape_string(), but I think you need a database connection first.
Is there a Python equivalent of PHP's mysql_real_escape_string?
I'm trying to insert some strings into a MySQL db direct from Python, and keep getting tripped up by quotes in the strings.
mysql_string = "INSERT INTO candidate (name, address) VALUES "
for k, v in v_dict.iteritems():
mysql_string += " ('" + v_dict['name'] + "', '" + v_dict['address'] + "'), "
mysql_string += ";"
cursor.execute(mysql_string)
I've tried re.escape() but that escapes every non-alphanumeric character in the strings, which isn't what I need - I just need to escape single quotes in this instance (plus more generally anything else that might trip up MySQL).
Could do this manually I guess, but is there a smarter way to do it in Python?
If you are using mysql-python, just try
MySQLdb.escape_string(SQL)
Example
>>> import MySQLdb
>>> MySQLdb.escape_string("'")
"\\'"
cursor.executemany('INSERT INTO candidate (name, address) VALUES (%s, %s)',
[(v_dict['name'], v_dict['address'])] * len(v_dict))
should do what your code appears to attempt -- inserting the same identical values N times (you're looping on v_dict.iteritems() but completely ignoring the loop variables, and instad just reusing those specific two values from v_dict each and every time). Of course if you mean something completely different the second line will need to be changed accordingly, but the key idea anyway is to use placeholders, not string formatting, for such tasks.
Is there a Python equivalent of PHP's mysql_real_escape_string?
Yes, it's escape_string on the database connection object.
Not escape_string on the MySQLdb module, which is equivalent to mysql_escape_string in PHP, and has the same potential vulnerability with multibyte character sets. You shouldn't use MySQLdb.escape_string.
In any case, you are much better off using parameterised queries as in Alex's answer. This makes it easier to port to other databases (it's part of the DB-API standard, which escape_string is not), and it's generally more convenient. (And much more convenient than parameterisation is in PHP.)
In this particular case you just need to use executemany method of the cursor.
mysql_string = "INSERT INTO candidate (name, address) VALUES (%s,%s);"
cursor.executemany(mysql_string, v_dict.iteritems())
beside MySQLdb.escape_string(SQL), pymysql python library also provide escape_string() function.
>>> import pymysql
>>> pymysql.escape_string("'")
"\\'"
>>>
MySQLdb.escape_string is equivalent. All of that is described in documentation.