I have an ArcPy script that is meant to select the desired attributes (both E-dom and E-subdom – see example below) from an attribute table based on a query.
myQuery = '"VM_POLY" = \'E-dom\'' or '"VM_POLY" = \'E-subdom\''
myInputShapeFile = r"D:\FieldStudy.shp"
myOutputShapefile = r"D:\Endangered.shp"
arcpy.Select_analysis(myInputShapeFile, myOutputShapefile, myQuery)
When the script is finished, only one type of attribute from the query is selected. In the attribute table for myInputShapeFile, there is both E-dom and E-subdom attributes, but the script will only return E-dom in the myOutputShapefile. I know the problem is possibly in the query (myQuery) but I am not sure how to resolve it.
If someone could provide some guidance, it would greatly be appreciated.
Could it be that you got your apostrophes wrong?
myQuery = '"VM_POLY" = \'E-dom\'' or '"VM_POLY" = \'E-subdom\''
# ^ ¨ ¨^ ^ ¨ ¨^
(Apostrophes marked by a ^ delimit a Python string; those marked with ¨ are escaped and therefore part of the string.)
I suppose your query ought to be:
myQuery = '"VM_POLY" = \'E-dom\' or "VM_POLY" = \'E-subdom\''
# ^ ¨ ¨ ¨ ¨^
because the or should not be interpreted by Python (logical operator applied to two strings), but should be part of the query text.
Python string formatting (and the SQL "IN" operator) make it (slightly) easier to handle the complex quote syntax:
myQuery = "\"{}\" IN ('{}', '{}')".format("VM_POLY", "E-dom", "E-subdom")
I would simplify your query with specifiers.
myQuery = ' "%s" = %s or "%s" = %s ' %('VM_POLY','E-dom','VM_POLY','E-subdom')
I know it seems quirky at first but its a good way to ensure your quotes and double quotes are positioned correctly for such things that require specific formatting. They have become my best friend as of late.
NOTE: if you have numbers you would like to inject into the string you can do so by using %d instead of %s.
Try this:
myQuery = '"VM_POLY" = "E-dom" or "VM_POLY"= "E-subdom"'
As was stated the or operator was being mishandled because it was not within the confines of your quotation. You also had the escape character \ in there which probably was not needed but you were attempting to work around the fact that your or operator was exposed and not part of the SQL query and thus picked up by the Python interpreter instead.
Related
I'm using neo4j python lib to manipulate the neo4j graph database.
I need to format the cypher query like
query = 'create (n:Person {{nickname: "{0}"}}) return n;'.format(nickname)
If the nickname contains a " itself like A"B, the generated query will be
create (n:Person {nickname: "A"B"}) return n; which will raise a syntax error in cql running.
In fact the cypher query language support '' and "" to indicate a string.
So my problem is, if there is a smart way, when the variable nickname contains ' or ", the formatted string could automatically use the right quotation marks?
You can use the repr function to format the string with proper quotation marks:
query = 'create (n:Person {{nickname: {0}}}) return n;'.format(repr(nickname))
since its behavior is exactly what you want, enclosing a given string with double quotes when the string contains a single quote, and single quotes when the string contains a double quote:
>>> print(repr("A'B"))
"A'B"
>>> print(repr('A"B'))
'A"B'
You can use """Lorem ipsum""" format string which allows you to use symbols and ascii chars. i.e
query = """create (n:Person {{nickname: "{0}"}}) return n;""".format(nickname)
You could also use single quotes and double qoutes in the string.
What you need is called "quotes escaping". The most simple way to do that is:
nickname='A"B'
query = 'create (n:Person {{nickname: "{0}"}}) return n;'.format(nickname.replace('"','\\"'))
print(query)
>>>
create (n:Person {nickname: "A\"B"}) return n;
If you would like a more "formal" way, you can do following:
import json
person = {'nickname': 'A"B'}
query = 'create (n:Person {0}) return n;'.format(json.dumps(person))
print(query)
I know that variants of this topic have been discussed elsewhere, but none of the other threads were helpful.
I want to hand over a string from python to sql. It might however happen that apostrophes (') occur in the string. I want to escape them with a backslash.
sql = "update tf_data set authors=\'"+(', '.join(authors).replace("\'","\\\'"))+"\' where tf_data_id="+str(tf_data_id)+";"
However, this will always give \\' in my string. Therefore, the backslash itself is escaped and the sql statement doesn't work.
Can someone help me or give me an alternative to the way I am doing this?
Thanks
Simply don't.
Also don't concatenate sql queries as these are prone to sql injections.
Instead, use a parameterized query:
sql = "update tf_data set authors=%(authors)s where tf_data_id=%(data_id)s"
# or :authors and :data_id, I get confused with all those sql dialects out there
authors = ', '.join(authors)
data_id = str(tf_data_id)
# db or whatever your db instance is called
db.execute(sql, {'authors': authors, 'data_id': data_id})
You're using double-quoted strings, but still escaping the single quotes within them. That's not required, all you need to do is escape the backslash that you want to use in the replace operation.
>>> my_string = "'Hello there,' I said."
>>> print(my_string)
'Hello there,' I said.
>>> print(my_string.replace("'", "\\'"))
\'Hello there,\' I said.
Note that I'm using print. If you just ask Python to show you its representation of the string after the replace operation, you'll see double backslashes because they need to be escaped.
>>> my_string.replace("'", "\\'")
"\\'Hello there,\\' I said."
As others have alluded to, if you are using a python package to execute your SQL use the provided methods with parameter placeholders(if available).
My answer addresses the escaping issues mentioned.
Use a String literal with prefix r
print(r"""the\quick\fox\\\jumped\'""")
Output:
the\quick\fox\\\jumped\'
What is the preferred way to write Python doc string?
""" or "
In the book Dive Into Python, the author provides the following example:
def buildConnectionString(params):
"""Build a connection string from a dictionary of parameters.
Returns string."""
In another chapter, the author provides another example:
def stripnulls(data):
"strip whitespace and nulls"
return data.replace("\00", "").strip()
Both syntax work. The only difference to me is that """ allows us to write multi-line doc.
Are there any differences other than that?
From the PEP8 Style Guide:
PEP 257 describes good docstring conventions. Note that most
importantly, the """ that ends a multiline docstring should be on a
line by itself, e.g.:
"""Return a foobang
Optional plotz says to frobnicate the bizbaz first.
"""
For one liner docstrings, it's okay to keep the closing """ on the
same line.
PEP 257 recommends using triple quotes, even for one-line docstrings:
Triple quotes are used even though the string fits on one line. This
makes it easy to later expand it.
Note that not even the Python standard library itself follows these recommendations consistently. For example,
abcoll.py
ftplib.py
functools.py
inspect.py
They're both strings, so there is no difference. The preferred style is triple double quotes (PEP 257):
For consistency, always use """triple double quotes""" around docstrings.
Use r"""raw triple double quotes""" if you use any backslashes in your docstrings. For Unicode docstrings, use u"""Unicode triple-quoted strings""".
No, not really. If you are writing to a file, using triple quotes may be ideal, because you don't have to use "\n" in order to go a line down. Just make sure the quotes you start and end with are the same type(Double or Triple quotes). Here is a reliable resource if you have any more questions:
http://docs.python.org/release/1.5.1p1/tut/strings.html
You can also use triple-double quotes for a long SQL query to improve the readability and not to scroll right to see it as shown below:
query = """
SELECT count(*)
FROM (SELECT *
FROM student
WHERE grade = 2 AND major = 'Computer Science'
FOR UPDATE)
AS result;
"""
And, if using double quotes for the SQL query above, the readability is worse and you will need to scroll right to see it as shown below:
query = "SELECT count(*) FROM (SELECT * FROM student WHERE grade = 2 AND major = 'Computer Science' FOR UPDATE) AS result;"
In addition, you can also use triple-double quotes for a GraphQL query as shown below:
query = """
{
products(first: 5) {
edges {
node {
id
handle
}
}
}
}"""
This string:
"CREATE USER %s PASSWORD %s", (user, pw)
always gets expanded to:
CREATE USER E'someuser' PASSWORD E'somepassword'
Can anyone tell me why?
Edit:
The expanded string above is the string my database gives me back in the error message. I'm using psycopg2 to access my postgres database. The real code looks like this:
conn=psycopg2.connect(user=adminuser, password=adminpass, host=host)
cur = conn.cursor()
#user and pw are simple standard python strings the function gets as parameter
cur.execute("CREATE USER %s PASSWORD %s", (user, pw))
conn.commit()
To pass identifiers to postgresql through psycopg use AsIs from the extensions module
from psycopg2.extensions import AsIs
import psycopg2
connection = psycopg2.connect(database='db', user='user')
cur = connection.cursor()
cur.mogrify(
'CREATE USER %s PASSWORD %s', (AsIs('someuser'), AsIs('somepassword'))
)
'CREATE USER someuser PASSWORD somepassword'
That works also for passing conditions to clauses like order by:
cur.mogrify(
'select * from t order by %s', (AsIs('some_column, another column desc'),)
)
'select * from t order by some_column, another column desc'
As the OP's edit reveals he's using PostgreSQL, the docs for it are relevant, and they say:
PostgreSQL also accepts "escape"
string constants, which are an
extension to the SQL standard. An
escape string constant is specified by
writing the letter E (upper or lower
case) just before the opening single
quote, e.g. E'foo'.
In other words, psycopg is correctly generating escape string constants for your strings (so that, as the docs also say:
Within an escape string, a backslash
character () begins a C-like
backslash escape sequence, in which
the combination of backslash and
following character(s) represents a
special byte value.
(which as it happens are also the escape conventions of non-raw Python string literals).
The OP's error clearly has nothing to do with that, and, besides the excellent idea of studying PostgreSQL's excellent docs, he should not worry about that E'...' form in this case;-).
Not only the E but the quotes appear to come from whatever type user and pw have. %s simply does what str() does, which may fall back to repr(), both of which have corresponding methods __str__ and __repr__. Also, that isn't the code that generates your result (I'd assumed there was a %, but now see only a comma). Please expand your question with actual code, types and values.
Addendum: Considering that it looks like SQL, I'd hazard a guess that you're seeing escape string constants, likely properly generated by your database interface module or library.
Before attempting something like:
statement = "CREATE USER %s PASSWORD %s" % (user, pw)
Please ensure you read: http://www.initd.org/psycopg/docs/usage.html
Basically the issue is that if you are accepting user input (I assume so as someone is entering in the user & pw) you are likely leaving yourself open to SQL injection.
As PsyCopg2 states:
Warning Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
As has been identified, Postgres (or Psycopg2) doesn't seem to provide a good answer to escaping identifiers. In my opinion, the best way to resolve this is to provide a 'whitelist' filtering method.
ie: Identify what characters are allowed in a 'user' and a 'pw'. (perhaps A-Za-z0-9_). Be careful that you don't include escape characters (' or ;, etc..) or if you do, that you escape these values.
I have a python script that reads raw movie text files into an sqlite database.
I use re.escape(title) to add escape chars into the strings to make them db safe before executing the inserts.
Why does this not work:
In [16]: c.execute("UPDATE movies SET rating = '8.7' WHERE name='\'Allo\ \'Allo\!\"\ \(1982\)'")
--------------------------------------------------------------------------- OperationalError Traceback (most recent call last)
/home/rajat/Dropbox/amdb/<ipython console> in <module>()
OperationalError: near "Allo": syntax error
Yet this works (removed \' in two places) :
In [17]: c.execute("UPDATE movies SET rating = '8.7' WHERE name='Allo\ Allo\!\"\ \(1982\)'") Out[17]: <sqlite3.Cursor object at 0x9666e90>
I can't figure it out. I also can't ditch those leading quotes because they're actually part of the movie title.
Thank you.
You're doing it wrong. Literally. You should be using parameters, like this:
c.execute("UPDATE movies SET rating = ? WHERE name = ?", (8.7, "'Allo 'Allo! (1982)"))
Like that, you won't need to do any quoting at all and (if those values are coming from anyone untrusted) you'll be 100% safe (here) from SQL injection attacks too.
I use re.escape(title) to add escape
chars into the strings to make them db
safe
Note that re.escape makes a string re-safe -- nothing to do with making it db safe. Rather, as #Donal says, what you need is the parameter substitution concept of the Python DB API -- that makes things "db safe" as you need.
SQLite doesn't support backslash escape sequences. Apostrophes in string literals are indicated by doubling them: '''Allo ''Allo! (1982)'.
But, like Donal said, you should be using parameters.
I've one simple tip you could use to handle this problem:
When your SQL statement string has single quote:', then you could use double quote to enclose your statement string. And when your SQL statement string has double quotes:", then you could use single quote:" to enclose your statement string.
E.g.
sqlString="UPDATE movies SET rating = '8.7' WHERE name='Allo Allo !' (1982 )"
c.execute(sqlString)
Or,
sqlString='UPDATE movies SET rating = "8.7" WHERE name="Allo Allo !" (1982 )'
c.execute(sqlString)
This solution works for me in Python environment.