I have the following line of code that is supposed to build a Pandas DataFrame from a SQL query:
query_epd = pandas.read_sql_query("SELECT 'Department', COUNT('LastName') FROM thestaff.employees GROUP BY 'Department'", engine)
Yet when I run my code this line gives me the error:
SyntaxError: non-integer constant in GROUP BY
LINE 1: ...OUNT('LastName') FROM thestaff.employees GROUP BY 'Departmen...
^
I don't see where or how I am using constants, integer or not, and this is a very standard query for me on MSSQL, but running under PostgreSQL and Pandas this query is not valid. What is wrong with my query?
The single quotes around the identifiers turn them to literal strings, which is probably not what you want. You should write this query as:
SELECT department, COUNT(*) no_emp
FROM thestaff.employees
GROUP BY department
If your identifiers are case-sensitive, then you need to surround them with double quotes (this is the SQL standard, which Postgres complies to).
Note that I changed COUNT(lastname) to COUNT(*): unless you have null values in the lastname column, this is equivalent, and more efficient. I also gave an alias to this column in the resultset.
This link might be helpful Non-integer constants in the ORDER BY clause they explain what this error is and when it occurs
Related
This question's answers are a community effort. Edit existing answers to improve this post. It is not currently accepting new answers or interactions.
I'm trying to execute a simple MySQL query as below:
INSERT INTO user_details (username, location, key)
VALUES ('Tim', 'Florida', 42)
But I'm getting the following error:
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'key) VALUES ('Tim', 'Florida', 42)' at line 1
How can I fix the issue?
The Problem
In MySQL, certain words like SELECT, INSERT, DELETE etc. are reserved words. Since they have a special meaning, MySQL treats it as a syntax error whenever you use them as a table name, column name, or other kind of identifier - unless you surround the identifier with backticks.
As noted in the official docs, in section 10.2 Schema Object Names (emphasis added):
Certain objects within MySQL, including database, table, index, column, alias, view, stored procedure, partition, tablespace, and other object names are known as identifiers.
...
If an identifier contains special characters or is a reserved word, you must quote it whenever you refer to it.
...
The identifier quote character is the backtick ("`"):
A complete list of keywords and reserved words can be found in section 10.3 Keywords and Reserved Words. In that page, words followed by "(R)" are reserved words. Some reserved words are listed below, including many that tend to cause this issue.
ADD
AND
BEFORE
BY
CALL
CASE
CONDITION
DELETE
DESC
DESCRIBE
FROM
GROUP
IN
INDEX
INSERT
INTERVAL
IS
KEY
LIKE
LIMIT
LONG
MATCH
NOT
OPTION
OR
ORDER
PARTITION
RANK
REFERENCES
SELECT
TABLE
TO
UPDATE
WHERE
The Solution
You have two options.
1. Don't use reserved words as identifiers
The simplest solution is simply to avoid using reserved words as identifiers. You can probably find another reasonable name for your column that is not a reserved word.
Doing this has a couple of advantages:
It eliminates the possibility that you or another developer using your database will accidentally write a syntax error due to forgetting - or not knowing - that a particular identifier is a reserved word. There are many reserved words in MySQL and most developers are unlikely to know all of them. By not using these words in the first place, you avoid leaving traps for yourself or future developers.
The means of quoting identifiers differs between SQL dialects. While MySQL uses backticks for quoting identifiers by default, ANSI-compliant SQL (and indeed MySQL in ANSI SQL mode, as noted here) uses double quotes for quoting identifiers. As such, queries that quote identifiers with backticks are less easily portable to other SQL dialects.
Purely for the sake of reducing the risk of future mistakes, this is usually a wiser course of action than backtick-quoting the identifier.
2. Use backticks
If renaming the table or column isn't possible, wrap the offending identifier in backticks (`) as described in the earlier quote from 10.2 Schema Object Names.
An example to demonstrate the usage (taken from 10.3 Keywords and Reserved Words):
mysql> CREATE TABLE interval (begin INT, end INT);
ERROR 1064 (42000): You have an error in your SQL syntax.
near 'interval (begin INT, end INT)'
mysql> CREATE TABLE `interval` (begin INT, end INT);
Query OK, 0 rows affected (0.01 sec)
Similarly, the query from the question can be fixed by wrapping the keyword key in backticks, as shown below:
INSERT INTO user_details (username, location, `key`)
VALUES ('Tim', 'Florida', 42)"; ^ ^
In PgAdmin, I can do the following query successfully:
select * from "Faces" where "Face_Name" = 'Alex'
However, when I try to do the exact same query in python, I get endless syntax errors.
I am trying to write the line like this:
cursor.execute('SELECT * from "Faces" where ("Face_Name" = 'Alex')
I understand the table and column names need to be in double quotes, and the whole query needs to be in single quotes. Also seems the string (in this case 'Alex') that I am searching for needs to be in single quotes.
How do I put all this together into a single line?
Assuming you did need to escape the table and column names, you could use double quotes. In that case, just escape the double quotes inside the Python SQL string:
sql = "SELECT * FROM \"Faces\" WHERE \"Face_Name\" = 'Alex'"
cursor.execute(sql)
There are two issues here:
As others already wrote, you need to be careful not to mix up the Python and SQL quotes; depending on the field name you may need to have both in the query, and either escape one of them or use """ for the Python string.
If the name "Alex" comes from a variable in Python, rather than being a constant, you should use a placeholder and pass it separately. This will help avoid security problems (SQL Injection) and is a good habit to get into whether or not it's required in this particular case.
Putting these together, the query should be:
cursor.execute('SELECT * from "Faces" where "Face_Name" = %s', ('Alex',))
I am using hand crafted SQL to fetch data from a PG database, using SqlAlchemy. I am trying a query which contains the SQL like operator '%' and that seems to throw SqlAlcjhemy through a loop:
sql = """
SELECT DISTINCT u.name from user u
INNER JOIN city c ON u.city_id = c.id
WHERE c.designation=upper('fantasy')
AND c.id IN (select id from ref_geog where short_name LIKE '%opt')
"""
# The last line in the above statement throws the error mentioned in the title.
# However if the last line is change to:
# AND c.id IN (select id from ref_geog where short_name = 'helloopt')
# the script runs correctly.
#
# I also tried double escaping the '%' i.e. using '%%' instead - that generated the same error as previously.
connectDb()
res = executeSql(sql)
print res
closeDbConnection()
Any one knows what is causing this misleading error message and how I may fix it?
[[Edit]]
Before any one asks, there is nothing special or fancy about the functions included above. For example the function executeSql() simply invokes conn.execute(sql) and returns the results. The variable conn is simply the previously established connection to the database.
You have to give %% to use it as % because % in python is use as string formatting so when you write single % its assume that you are going to replace some value with this.
So when you want to place single % in string with query allways place double %.
SQLAlchemy has a text() function for wrapping text which appears to correctly escape the SQL for you.
I.e.
res = executeSql(sqlalchemy.text(sql))
should work for you and save you from having to do the manual escaping.
I cannot find the "executeSql" in sqlalchemy version 1.2 docs , but the below line worked for me
engine.execute(sqlalchemy.text(sql_query))
I found one more case when this error shows up:
c.execute("SELECT * FROM t WHERE a = %s")
In other words, if you provide parameter (%s) in query, but you forget to add query params. In this case error message is very misleading.
It seems like your problem may be related to this bug.
In which case, you should triple-escape as a workaround.
One more note- you must escape (or delete) % characters in comments as well. Unfortunately, sqlalchemy.text(query_string) does not escape the percent signs in the comments.
Another way of solving your problem, if you don't want to escape % characters or use sqlalchemy.text(), is to use a regular expression.
Instead of:
select id from ref_geog where short_name LIKE '%opt'
Try (for case-sensitive match):
select id from ref_geog where short_name ~ 'opt$'
or (for case-insensitive):
select id from ref_geog where short_name ~* 'opt$'
Both LIKE and regex are covered in the documentation on pattern matching.
Note that:
Unlike LIKE patterns, a regular expression is allowed to match anywhere within a string, unless the regular expression is explicitly anchored to the beginning or end of the string.
For an anchor, you can use the assertion $ for end of string (or ^ for beginning).
This could also result from the case - in case parameters to be passed onto the SQL are declared in DICT formate and are being manipulated in the SQL in the form of LIST or TUPPLE.
I used MySQL Connector/Python API, NOT MySQLdb.
I need to dynamically insert values into a sparse table so I wrote the Python code like this:
cur.executemany("UPDATE myTABLE SET %s=%s WHERE id=%s" % data)
where
data=[('Depth', '17.5cm', Decimal('3003')), ('Input_Voltage', '110 V AC', Decimal('3004'))]
But it resulted an error:
TypeError: not enough arguments for format string
Is there any solution for this problem? Is it possible to use executemany when there is a
substitution of a field in query?
Thanks.
Let's start with the original method:
As the error message suggests you have a problem with your SQL syntax (not Python). If you insert your values you are effectively trying to execute
UPDATE myTABLE SET 'Depth'='17.5cm' WHERE id='3003'
You should notice that you are trying to assign a value to a string 'Depth', not a database field. The reason for this is that the %s substitution of the mysql module is only possible for values, not for tables/fields or other object identifiers.
In the second try you are not using the substitution anymore. Instead you use generic python string interpolation, which however looks similar. This does not work for you because you have a , and a pair of brackets too much in your code. It should read:
cur.execute("UPDATE myTABLE SET %s=%s WHERE id=%s" % data)
I also replaced executemany with execute because this method will work only for a single row. However your example only has one row, so there is no need to use executemany anyway.
The second method has some drawbacks however. The substitution is not guaranteed to be quoted or formatted in a correct manner for the SQL query, which might cause unexpected behaviour for certain inputs and may be a security concern.
I would rather ask, why it is necessary to provide the field name dynamically in the first place. This should not be necessary and might cause some trouble.
I am using hand crafted SQL to fetch data from a PG database, using SqlAlchemy. I am trying a query which contains the SQL like operator '%' and that seems to throw SqlAlcjhemy through a loop:
sql = """
SELECT DISTINCT u.name from user u
INNER JOIN city c ON u.city_id = c.id
WHERE c.designation=upper('fantasy')
AND c.id IN (select id from ref_geog where short_name LIKE '%opt')
"""
# The last line in the above statement throws the error mentioned in the title.
# However if the last line is change to:
# AND c.id IN (select id from ref_geog where short_name = 'helloopt')
# the script runs correctly.
#
# I also tried double escaping the '%' i.e. using '%%' instead - that generated the same error as previously.
connectDb()
res = executeSql(sql)
print res
closeDbConnection()
Any one knows what is causing this misleading error message and how I may fix it?
[[Edit]]
Before any one asks, there is nothing special or fancy about the functions included above. For example the function executeSql() simply invokes conn.execute(sql) and returns the results. The variable conn is simply the previously established connection to the database.
You have to give %% to use it as % because % in python is use as string formatting so when you write single % its assume that you are going to replace some value with this.
So when you want to place single % in string with query allways place double %.
SQLAlchemy has a text() function for wrapping text which appears to correctly escape the SQL for you.
I.e.
res = executeSql(sqlalchemy.text(sql))
should work for you and save you from having to do the manual escaping.
I cannot find the "executeSql" in sqlalchemy version 1.2 docs , but the below line worked for me
engine.execute(sqlalchemy.text(sql_query))
I found one more case when this error shows up:
c.execute("SELECT * FROM t WHERE a = %s")
In other words, if you provide parameter (%s) in query, but you forget to add query params. In this case error message is very misleading.
It seems like your problem may be related to this bug.
In which case, you should triple-escape as a workaround.
One more note- you must escape (or delete) % characters in comments as well. Unfortunately, sqlalchemy.text(query_string) does not escape the percent signs in the comments.
Another way of solving your problem, if you don't want to escape % characters or use sqlalchemy.text(), is to use a regular expression.
Instead of:
select id from ref_geog where short_name LIKE '%opt'
Try (for case-sensitive match):
select id from ref_geog where short_name ~ 'opt$'
or (for case-insensitive):
select id from ref_geog where short_name ~* 'opt$'
Both LIKE and regex are covered in the documentation on pattern matching.
Note that:
Unlike LIKE patterns, a regular expression is allowed to match anywhere within a string, unless the regular expression is explicitly anchored to the beginning or end of the string.
For an anchor, you can use the assertion $ for end of string (or ^ for beginning).
This could also result from the case - in case parameters to be passed onto the SQL are declared in DICT formate and are being manipulated in the SQL in the form of LIST or TUPPLE.