I'm parsing a xml file and inserting it into database.
However since some text containes double or single quotation I'm having problem with insertion. Currently I'm using the code shown below. But it seems it's inefficient.
s = s.replace('"', ' ')
s = s.replace("'", ' ')
Is there any way I can insert text without replacing these quotations?
OR
Is there any efficient way to substitute them efficiently ?
Thanks !
Why can't you insert strings containing quote marks into your database? Is there some weird data type that permits any character except a quote mark? Or are you building an insert statement with literal strings, rather than binding your strings to query parameters as you should be doing?
If you're doing
cursor.execute('insert into mytable (somefield) values ("%s")' % (mystring))
then that's unsafe and wrong. Instead, you should be doing
cursor.execute('insert into mytable (somefield) values (%(myparam)s)',
dict(myparam=mystring))
you should use str.translate instead of doing two replace() calls
>>> import string
>>> quotes_to_spaces=string.maketrans('"\''," ")
>>> s=s.translate(quotes_to_spaces)
You could try something like _mysql.escape_string():
>>> import _mysql
>>> a = '''I said, "Don't do that"'''
>>> a
'I said, "Don\'t do that"'
>>> _mysql.escape_string(a)
'I said, \\"Don\\\'t do that\\"'
However, the manual recommends using connection.escape_string(), but I think you need a database connection first.
Related
I have a script that migrates data from one database to another written in python and sql using the psycopg2 library.
I retrieve a string from the first database and store it for later in a list so I can put it into the second database when I finish gathering all the data I need.
If the string has an apostrophe in it then python will represent the string using " ". The problem with this is that sql interprets " " as specifying a column name and ' ' for strings whereas python interprets both as strings. I wish to force python to use apostrophes to represent the string (or another suitable workaround)
Google has not turned up anything. Can't even find a mention of the fact that python will use " " when you have apostrophes in your string. I have considered replacing apostrophes in my string with a different character and converting it back later but this seems like a clumsy solution.
For example
MyString = 'it\'s'
MyList = [MyString]
print(MyList) # returns "it's"
print(MyList[0]) # returns it's
When I insert the new values into the database I am in inserting the whole list as the values.
INSERT INTO table VALUES MyList
This is where the error crops up because the string is using " " instead of ' '.
A solution on either the python or sql side would work.
Found a fix. It's a bit janky but it works. Convert the list into a string. Use the replace function like so:
MyString = MyString.replace('"',"'")
And then use that string instead.
I am having trouble with the replace() method. I want to replace some part of a string, and the part which I want to replace consist of multiple escape characters. It looks like something like this;
['<div class=\"content\">rn
To remove it, I have a block of code;
garbage_text = "[\'<div class=\\\"content\\\">rn "
entry = entry.replace(garbage_text,"")
However, it does not work. Anything is removed from my complete string. Can anybody point out where exactly I am thinking wrong about it? Thanks in advance.
Addition:
The complete string looks like this;
"['<div class=\"content\">rn gitar calmak icin kullanilan minik plastik garip nesne.rn </div>']"
You could use the triple quote format for your replacement string so that you don't have to bother with escaping at all:
garbage_text = """['<div class="content">rn """
Perhaps your 'entry' is not formatted correctly?
With an extra variable 'text', the following worked in Python 3.6.7:
>>> garbage_text
'[\'<div class=\\\'content\'\\">rn '
>>> text
'[\'<div class=\\\'content\'\\">rn And then there were none'
>>> entry = text.replace(garbage_text, "")
>>> entry
'And then there were none'
I know that variants of this topic have been discussed elsewhere, but none of the other threads were helpful.
I want to hand over a string from python to sql. It might however happen that apostrophes (') occur in the string. I want to escape them with a backslash.
sql = "update tf_data set authors=\'"+(', '.join(authors).replace("\'","\\\'"))+"\' where tf_data_id="+str(tf_data_id)+";"
However, this will always give \\' in my string. Therefore, the backslash itself is escaped and the sql statement doesn't work.
Can someone help me or give me an alternative to the way I am doing this?
Thanks
Simply don't.
Also don't concatenate sql queries as these are prone to sql injections.
Instead, use a parameterized query:
sql = "update tf_data set authors=%(authors)s where tf_data_id=%(data_id)s"
# or :authors and :data_id, I get confused with all those sql dialects out there
authors = ', '.join(authors)
data_id = str(tf_data_id)
# db or whatever your db instance is called
db.execute(sql, {'authors': authors, 'data_id': data_id})
You're using double-quoted strings, but still escaping the single quotes within them. That's not required, all you need to do is escape the backslash that you want to use in the replace operation.
>>> my_string = "'Hello there,' I said."
>>> print(my_string)
'Hello there,' I said.
>>> print(my_string.replace("'", "\\'"))
\'Hello there,\' I said.
Note that I'm using print. If you just ask Python to show you its representation of the string after the replace operation, you'll see double backslashes because they need to be escaped.
>>> my_string.replace("'", "\\'")
"\\'Hello there,\\' I said."
As others have alluded to, if you are using a python package to execute your SQL use the provided methods with parameter placeholders(if available).
My answer addresses the escaping issues mentioned.
Use a String literal with prefix r
print(r"""the\quick\fox\\\jumped\'""")
Output:
the\quick\fox\\\jumped\'
I have a little script that creates a certain INSERT SQL statement for me.
For postgresql I need to wrap the values to be inserted within two single quotes.
Unfortunately some of the value strings to be inserted also contain a single quote, and I need to escape them automatically.
for line in f:
out.write('(\'' + line[:2] + '\', \'' + line[3:-1] + '\'),\n')
How can I make sure that any single quote (e.g. ' ) inside line[3:-1] is automatically escaped?
Thanks,
UPDATE:
e.g. the line
CI|Cote D'ivoire
fails due '
Update 2:
I can't use double quotes in values, e.g.
INSERT INTO "App_country" (country_code, country_name) VALUES ("AF", "Afghanistan")
I get the error message: ERROR: column "AF" does not exist
This however works fine:
INSERT INTO "App_country" (country_code, country_name) VALUES ('AF', 'Afghanistan')
As described in the PEP-249, the DBPI is a generic interface to various databases. Different implementations exist for different databases. For postgres there is psycopg. from the docs:
cur.execute(
... """INSERT INTO some_table (an_int, a_date, a_string)
... VALUES (%s, %s, %s);""",
... (10, datetime.date(2005, 11, 18), "O'Reilly"))
You simple pass your parameters in a tuple. The underlying library escapes it for you. This is much safer and easier than trying to roll your own.
The SQL standard way to escape a quote is to double it:
'This won''t be a problem.'
So replace every quote with two quotes (and use double quotes in Python to stay sane):
out.write("('" + line[:2] + "', '" + line[3:-1].replace("'", "''") + "'),\n")
Never use a generated, rolled-your-own escaping for DML. Use the appropriate DBAPI as Keith has mentioned. Work would have gone into that to make sure escapes from various sources and type conversion can occur almost transparently. If you're using DDL such as a CREATE TABLE whatever (...) - you can be more slight slack-handed if you trust your own datasource.
using data shown in example:
import sqlite3
text = "CI|Cote D'ivoire" # had to been escaped as it's a string literal, but from another data source - possibly not...
code, name = text.split('|', 1)
db = sqlite3.connect(':memory:')
db.execute('create table something(code, name)')
db.execute('insert into something(code, name) values(?, ?)', (code, name))
for row in db.execute('select * from something'):
print row
# (u'CI', u"Cote D'ivoire")
For a complete solution toadd escape characters to a string, use:
re.escape(string)
>>> re.escape('\ a.*$')
'\\\\\\ a\\.\\*\\$'
for more, see: http://docs.python.org/library/re.html
Not sure if there are some SQL related limitations, but you could always use double quotes to surround your string that contains the single quote.
Eg.
print "That's all Folks!"
or single quotes to surround double quotes:
print 'The name of the file is "rosebud".'
I have a python script that reads raw movie text files into an sqlite database.
I use re.escape(title) to add escape chars into the strings to make them db safe before executing the inserts.
Why does this not work:
In [16]: c.execute("UPDATE movies SET rating = '8.7' WHERE name='\'Allo\ \'Allo\!\"\ \(1982\)'")
--------------------------------------------------------------------------- OperationalError Traceback (most recent call last)
/home/rajat/Dropbox/amdb/<ipython console> in <module>()
OperationalError: near "Allo": syntax error
Yet this works (removed \' in two places) :
In [17]: c.execute("UPDATE movies SET rating = '8.7' WHERE name='Allo\ Allo\!\"\ \(1982\)'") Out[17]: <sqlite3.Cursor object at 0x9666e90>
I can't figure it out. I also can't ditch those leading quotes because they're actually part of the movie title.
Thank you.
You're doing it wrong. Literally. You should be using parameters, like this:
c.execute("UPDATE movies SET rating = ? WHERE name = ?", (8.7, "'Allo 'Allo! (1982)"))
Like that, you won't need to do any quoting at all and (if those values are coming from anyone untrusted) you'll be 100% safe (here) from SQL injection attacks too.
I use re.escape(title) to add escape
chars into the strings to make them db
safe
Note that re.escape makes a string re-safe -- nothing to do with making it db safe. Rather, as #Donal says, what you need is the parameter substitution concept of the Python DB API -- that makes things "db safe" as you need.
SQLite doesn't support backslash escape sequences. Apostrophes in string literals are indicated by doubling them: '''Allo ''Allo! (1982)'.
But, like Donal said, you should be using parameters.
I've one simple tip you could use to handle this problem:
When your SQL statement string has single quote:', then you could use double quote to enclose your statement string. And when your SQL statement string has double quotes:", then you could use single quote:" to enclose your statement string.
E.g.
sqlString="UPDATE movies SET rating = '8.7' WHERE name='Allo Allo !' (1982 )"
c.execute(sqlString)
Or,
sqlString='UPDATE movies SET rating = "8.7" WHERE name="Allo Allo !" (1982 )'
c.execute(sqlString)
This solution works for me in Python environment.