So I'm using the PyMySQL package to query a remote database and I am finding it quite annoying to have to use the .execute() method.
I am looking for a wrapper that would be able to construct the raw MySQL queries in a more friendly fashion. Does anyone know of a package that would do this?
I have tried using pypika but the queries that it's building ('SELECT "id","username" FROM "users"') throw an error
pymysql.err.ProgrammingError: (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'"users"\' at line 1')
EDIT: this is a remote DB that I have no control over structure-wise and it is likely that the structure could change so I don't think I could use SQLAlchemy
EDIT 2: Here is the code that I used when I got that error above
from pypika import Query, Table, Field
import pymysql
users = Table('users')
q = Query.from_(users).select(users.id, users.username)
db = pymysql.connect(host=host,
port=3306,
user=user,
passwd=password,
db=db,
cursorclass=pymysql.cursors.DictCursor)
db.execute(str(q))
It looks like it's generating a query using double-quotes as identifier delimiters.
But MySQL uses back-ticks as the default identifier delimiter, so the query should look like this:
SELECT `id`,`username` FROM `users`
You can configure MySQL to use double-quotes, which is the proper identifier delimiter to comply with the ANSI SQL standard. To do this, you have to change the sql_mode to include the modes ANSI or ANSI_QUOTES.
Read https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html for full documentation on sql_mode behavior, and https://dev.mysql.com/doc/refman/8.0/en/identifiers.html for documentation on identifier delimiters.
I searched the pypika site and the docs are pretty sparse. But there's apparently a class for MySQLQuery that sets its own QUOTE_CHAR to ` which is what I would expect.
Are you using the proper query builder for MySQL?
You can use peewee a a simple and small ORM.
http://docs.peewee-orm.com/en/latest/peewee/quickstart.html#quickstart
Related
I'm using Python + MySQL and want to use parameterized query. I'm stuck. I've encountered an error and can't figure out how to solve it. I've spent a day, checked dozens of articles, used various options (sinle quotes, double quotes, prepared statements) and still no luck.
Requirements: use Parameterized Query
Here is basic demo of the issue:
#!/usr/bin/python3
import mysql.connector as mysql
conn = mysql.connect(host=server, user=username, passwd=password, autocommit=True)
try:
create_database_query = "CREATE DATABASE %s;"
db_name = "BOOKS"
cursor = conn.cursor()
print(f"Creating {db_name} database... ", end='')
cursor.execute(create_database_query, (db_name,))
print("Success")
except mysql.Error as error:
print("Parameterized query failed {}".format(error))
Output:
Creating BOOKS database... Parameterized query failed 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''BOOKS'' at line 1
So it looks like it uses too many quotes (2 single quotes on each side). The code above works fine if I change the following line:
create_database_query = "CREATE DATABASE %s;"
and put backtick around %s
The problem that now it creates a database but with invalid chars - 'BOOKS' (quotes are now part of db name). Duh...
If I use prepared statements then the same issue occurs but slightly different error message:
1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '?' at line 1
Environment:
MacOS Catalina
Python 3.8
PyCharm 2019.3 IDE
MySQL 8.0.19
mysql-connector-python module 8.0.19
What is going on? Any ideas?
Thanks
You can't use query parameters for identifiers (like a database name or table name or column name).
Query parameters can be used only in place of a constant value — a quoted string, quoted date/time, or a numeric value. Not identifiers, expressions, SQL keywords, etc.
To combine a database name with your CREATE DATABASE statement, you have to format it into the string in a way that forms the full statement before it is sent to MySQL.
db_name = "BOOKS"
create_database_query = "CREATE DATABASE %s;" % db_name
cursor.execute(create_database_query)
Because this creates a risk of SQL injection when you format variables into your string, it's up to you to make sure the db_name is safe.
Update: Thanks to #Parfait for the reminder about current best practices of string-formatting.
Prefer:
db_name = "BOOKS"
create_database_query = "CREATE DATABASE {};".format(db_name)
Or F-strings:
db_name = "BOOKS"
create_database_query = f"CREATE DATABASE {db_name};"
(In other words, Python has become Ruby ;-)
I'm using Python + MySQL and want to use parameterized query. I'm stuck. I've encountered an error and can't figure out how to solve it. I've spent a day, checked dozens of articles, used various options (sinle quotes, double quotes, prepared statements) and still no luck.
Requirements: use Parameterized Query
Here is basic demo of the issue:
#!/usr/bin/python3
import mysql.connector as mysql
conn = mysql.connect(host=server, user=username, passwd=password, autocommit=True)
try:
create_database_query = "CREATE DATABASE %s;"
db_name = "BOOKS"
cursor = conn.cursor()
print(f"Creating {db_name} database... ", end='')
cursor.execute(create_database_query, (db_name,))
print("Success")
except mysql.Error as error:
print("Parameterized query failed {}".format(error))
Output:
Creating BOOKS database... Parameterized query failed 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''BOOKS'' at line 1
So it looks like it uses too many quotes (2 single quotes on each side). The code above works fine if I change the following line:
create_database_query = "CREATE DATABASE %s;"
and put backtick around %s
The problem that now it creates a database but with invalid chars - 'BOOKS' (quotes are now part of db name). Duh...
If I use prepared statements then the same issue occurs but slightly different error message:
1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '?' at line 1
Environment:
MacOS Catalina
Python 3.8
PyCharm 2019.3 IDE
MySQL 8.0.19
mysql-connector-python module 8.0.19
What is going on? Any ideas?
Thanks
You can't use query parameters for identifiers (like a database name or table name or column name).
Query parameters can be used only in place of a constant value — a quoted string, quoted date/time, or a numeric value. Not identifiers, expressions, SQL keywords, etc.
To combine a database name with your CREATE DATABASE statement, you have to format it into the string in a way that forms the full statement before it is sent to MySQL.
db_name = "BOOKS"
create_database_query = "CREATE DATABASE %s;" % db_name
cursor.execute(create_database_query)
Because this creates a risk of SQL injection when you format variables into your string, it's up to you to make sure the db_name is safe.
Update: Thanks to #Parfait for the reminder about current best practices of string-formatting.
Prefer:
db_name = "BOOKS"
create_database_query = "CREATE DATABASE {};".format(db_name)
Or F-strings:
db_name = "BOOKS"
create_database_query = f"CREATE DATABASE {db_name};"
(In other words, Python has become Ruby ;-)
I keep running into an odd error when attempting to connect python sqlalchemy to a msssql server/database. I need to use sqlalchemy as it is (from what I've been told) the only way to connect pandas dataframes to mssql.
I have tried connecting sqlalchemy two different ways:
using full connection string:
import sqlalchemy as sa
import urllib.parse as ulp
usrCnnStr = r'DRIVER={SQL Server};SERVER=myVoid\MYINSTANCE;Trusted_Connection=yes;'
usrCnnStr = ulp.quote_plus(usrCnnStr)
usrCnnStr = "mssql+pyodbc:///?odbc_connect=%s" % usrCnnStr
engine = sa.create_engine(usrCnnStr)
connection = engine.connect()
connection.execute("select getdate() as dt from mydb.dbo.dk_rcdtag")
connection.close()
using DSN:
import sqlalchemy as sa
import urllib.parse as ulp
usrDsn = 'myDb'
params = ulp.quote_plus(usrDsn)
engine = sa.create_engine("mssql+pyodbc://cryo:pass#myDb")
conn = engine.connect()
conn.execute('select getdate() as dt')
conn.close()
Both methods return the same error:
sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('ODBC data type -150 is not supported. Cannot read column .', 'HY000') [SQL: "SELECT SERVERPROPERTY('ProductVersion')"]
I am not sure how to get around this error; when I execute the "SELECT SERVERPROPERTY('ProductVersion')" in mssql, it works fine but comes back with a data type of "sql_variant".
Is there any way to get around this?
This is most certainly a bug introduced in Issue 3814, new in SQLAlchemy 1.1.0, where they introduce SELECT SERVERPROPERTY('ProductVersion') to fetch server version for the pyodbc MSSQL driver. Downgrading to 1.0.15 will make the code work again, but hopefully the SQLAlchemy devs will make the new version lookup scheme work better in a new patch release.
(There is an issue already reported in the SQLAlchemy issue tracker, I would add this comment there, but bitbucket can't log me in.)
I upgraded to sqlalchemy 1.1 today and ran into a similar issue with connections that were working before. Bumped back to 1.0.15 and no problems. Not the best answer, more of a workaround, but it may work if you are on 1.1 and need to get rolling.
If you are unsure of your version:
>>import sqlalchemy
>>sqlalchemy.__version__
IIRC, this is because you can't select non-cast functions directory, since they don't return a datatype pyodbc recognizes.
Try this:
SELECT CAST(GETDATE() AS DATETIME) AS dt
Also, your may want to use CURRENT_TIMESTAMP, which is ANSI standard SQL, instead of GETDATE(): Retrieving date in sql server, CURRENT_TIMESTAMP vs GetDate()
I'm not sure where your product version select is coming from, but hopefully this gets you on the right path. I'll amend the answer if we figure out more.
I'm trying to use OPENJSON in a Python script to import some basic JSON into a SQL database. I initially tried with a more complex JSON file, but simplified it for the sake of this post. Here's what I have:
sql_statement = "declare #json nvarchar(max) = '{\"name\":\"James\"}'; SELECT * FROM OPENJSON(#json) WITH (name nvarchar(20))"
cursor.execute(sql_statement)
cursor.commit()
connection.close()
The error I receive:
pypyodbc.ProgrammingError: (u'42000', u"[42000] [Microsoft][ODBC SQL
Server Driver][SQL Server]Incorrect syntax near the keyword 'with'. If
this statement is a common table expression, an xmlnamespaces clause
or a change tracking context clause, the previous statement must be
terminated with a semicolon.")
Any thoughts on why I'm seeing this error? I was successfully able to execute other SQL queries with the same pypyodbc / database configuration.
The problem could be that your database is running in an older compatibility level, where OPEN JSON is not available.
To find the compatibility level of your database, run following SQL statement:
SELECT compatibility_level FROM sys.databases WHERE name = 'your_db_name';
If the result is 120 or lower, you'll need to update your compatibility level to 130, by running:
ALTER DATABASE your_db_name SET COMPATIBILITY_LEVEL = 130;
Note: In case your database is actually Azure SQL DB, you should check the version as well, as OPEN JSON is not available for versions prior to 12.x
I have seen this question asked in various ways on this website, but none of them exactly addressed my issue.
I have an sql statement with single quotes inside it, and am trying to use recommended practices before making database queries with it. So the statement is like
val2="abc 'dostuff'"
sql="INSERT INTO TABLE_A(COL_A,COL_B) VALUES(%s,'%s')" %(val1, val2)
a_cursor.execute(sql)
However, when I run this, I get..
ProgrammingError: (1064,"You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'dostuff'.
What am I doing wrong?
Thanks very much
Nupur
Use parameters instead of string interpolation to ensure that your values are properly escaped by the database connector:
sql = "INSERT INTO TABLE_A(COL_A,COL_B) VALUES(%s, %s)"
a_cursor.execute(sql, (val1, val2))
The mysqldb sql parameter style uses the same syntax as used by the python string formatting operator, which is a little confusing.