This question already has answers here:
How to use variables in SQL statement in Python?
(5 answers)
Closed 10 months ago.
I've recently started learning and trying to apply SQL into my python code and came across this problem; I can't figure out a way to properly insert variables into a table using the set up I have. This setup came from a SQL library tutorial on realpython.com (https://realpython.com/python-sql-libraries/), and judging by the different formats of sqlite setups, I feel like it isn't too efficient.
import sqlite3
from sqlite3 import Error
# - - - - SQL DATABASE SETUP - - - - #
def create_connection(path):
connection = None
try:
connection = sqlite3.connect(path)
print("Connection to SQLite DB successful")
except Error as e:
print(f"The error '{e}' occurred")
return connection
connection = create_connection("databse.sqlite") # Creates the connection
def execute_query(connection, query):
cursor = connection.cursor()
try:
cursor.execute(query)
connection.commit()
print("Query executed successfully")
except Error as e:
print(f"The error '{e}' occurred")
create_users_table = """
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
age INTEGER,
grade INTEGER,
gender TEXT,
nationality TEXT
);
"""
execute_query(connection, create_users_table)
# - - - - ^ SQL DATABSE SETUP ^ - - - - #
I believe I understand everything going on in these two functions, although I noticed that not many use this way, and I think that's why I couldn't find an answer to my problem, I would appreciate any advice on more efficient setups.
I tried using a solution that used the q-mark style in order to insert variables, but it didn't work for me. I think it could be because of the triple quotes strings, and the query statement being a parameter of a function within a function.
First attempt looked like this
name = str(input("Hi! What is your name?\n"))
store_name = ("""
INSERT INTO users (name)
VALUES (?)
""", name)
execute_query(connection, store_name)
Which gave me the error "ValueError: operation parameter must be str"
I later tried to remove the variable and just plug in the query as a single line statement, enclosed within a parameter
execute_query(connection, ("INSERT INTO users (name) VALUES (?)", name))
Same error, might be a simple mistake on my end that I'm not seeing.
And again if this setup isn't efficient, I would appreciate any advice.
you need to use same query as MySQL.
For example - "INSERT INTO table (column1,column2 ,..)
VALUES( value1, value2 ,...)"
Related
I use pyodbc in my python flask Project for the SQLite DB connection.
I know and understand SQL Injections but this is my first time dealing with it.
I tried to execute some
I have a function which concatenates the SQL String in my database.py file:
def open_issue(self, data_object):
cursor = self.conn.cursor()
# data_object is the issue i get from the user
name = data_object["name"]
text = data_object["text"]
rating_sum = 0
# if the user provides an issue
if name:
# check if issue is already in db
test = cursor.execute(f'''SELECT name FROM issue WHERE name = "{name}"''')
data = test.fetchall()
# if not in db insert
if len(data) == 0:
# insert the issue
cursor.executescript(f'''INSERT INTO issue (name, text, rating_sum)
VALUES ("{name}", "{text}", {rating_sum})''')
else:
print("nothing inserted!")
In the api.py file the open_issue() function gets called:
#self.app.route('/open_issue')
def insertdata():
# data sent from client
# data_object = flask.request.json
# unit test dictionary
data_object = {"name": "injection-test-table",
"text": "'; CREATE TABLE 'injected_table-1337';--"}
DB().open_issue(data_object)
The "'; CREATE TABLE 'injected_table-1337';--" sql injection has not created the injected_table-1337, instead it got inserted normally like a string into the text column of the injection-test-table.
So i don't really know if i am safe for the standard ways of SQL injection (this project will only be hosted locally but good security is always welcome)
And secondary: are there ways with pyodbc to check if a string contains sql syntax or symbols, so that nothing will get inserted in my example or do i need to check the strings manually?
Thanks a lot
As it turns out, with SQLite you are at much less risk of SQL injection issues because by default neither Python's built-in sqlite3 module nor the SQLite ODBC driver allow multiple statements to be executed in a single .execute call (commonly known as an "anonymous code block"). This code:
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = f"SELECT * FROM table1 WHERE txt='{thing}'"
crsr.execute(sql)
throws this for sqlite3
sqlite3.Warning: You can only execute one statement at a time.
and this for SQLite ODBC
pyodbc.Error: ('HY000', '[HY000] only one SQL statement allowed (-1) (SQLExecDirectW)')
Still, you should follow best practices and use a proper parameterized query
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = "SELECT * FROM table1 WHERE txt=?"
crsr.execute(sql, (thing, ))
because this will also correctly handle parameter values that would cause errors if injected directly, e.g.,
thing = "it's good to avoid SQL injection"
My objective is to store a JSON object into a MySQL database field of type json, using the mysql.connector library.
import mysql.connector
import json
jsonData = json.dumps(origin_of_jsonData)
cnx = mysql.connector.connect(**config_defined_elsewhere)
cursor = cnx.cursor()
cursor.execute('CREATE DATABASE dataBase')
cnx.database = 'dataBase'
cursor = cnx.cursor()
cursor.execute('CREATE TABLE table (id_field INT NOT NULL, json_data_field JSON NOT NULL, PRIMARY KEY (id_field))')
Now, the code below WORKS just fine, the focus of my question is the use of '%s':
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES (%s, %s)"
values_to_insert = (1, jsonData)
cursor.execute(insert_statement, values_to_insert)
My problem with that: I am very strictly adhering to the use of '...{}'.format(aValue) (or f'...{aValue}') when combining variable aValue(s) into a string, thus avoiding the use of %s (whatever my reasons for that, let's not debate them here - but it is how I would like to keep it wherever possible, hence my question).
In any case, I am simply unable, whichever way I try, to create something that stores the jsonData into the mySql dataBase using something that resembles the above structure and uses '...{}'.format() (in whatever shape or form) instead of %s. For example, I have (among many iterations) tried
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES ({}, {})".format(1, jsonData)
cursor.execute(insert_statement)
but no matter how I turn and twist it, I keep getting the following error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[some_content_from_jsonData})]' at line 1
Now my question(s):
1) Is there a way to avoid the use of %s here that I am missing?
2) If not, why? What is it that makes this impossible? Is it the cursor.execute() function, or is it the fact that it is a JSON object, or is it something completely different? Shouldn't {}.format() be able to do everything that %s could do, and more?
First of all: NEVER DIRECTLY INSERT YOUR DATA INTO YOUR QUERY STRING!
Using %s in a MySQL query string is not the same as using it in a python string.
In python, you just format the string and 'hello %s!' % 'world' becomes 'hello world!'. In SQL, the %s signals parameter insertion. This sends your query and data to the server separately. You are also not bound to this syntax. The python DB-API specification specifies more styles for this: DB-API parameter styles (PEP 249). This has several advantages over inserting your data directly into the query string:
Prevents SQL injection
Say you have a query to authenticate users by password. You would do that with the following query (of course you would normally salt and hash the password, but that is not the topic of this question):
SELECT 1 FROM users WHERE username='foo' AND password='bar'
The naive way to construct this query would be:
"SELECT 1 FROM users WHERE username='{}' AND password='{}'".format(username, password)
However, what would happen if someone inputs ' OR 1=1 as password. The formatted query would then become
SELECT 1 FROM users WHERE username='foo' AND password='' OR 1=1
which will allways return 1. When using parameter insertion:
execute('SELECT 1 FROM users WHERE username=%s AND password=%s', username, password)
this will never happen, as the query will be interpreted by the server separately.
Performance
If you run the same query many times with different data, the performance difference between using a formatted query and parameter insertion can be significant. With parameter insertion, the server only has to compile the query once (as it is the same every time) and execute it with different data, but with string formatting, it will have to compile it over and over again.
In addition to what was said above, I would like to add some details that I did not immediately understand, and that other (newbies like me ;)) may also find helpful:
1) "parameter insertion" is meant for only for values, it will not work for table names, column names, etc. - for those, the Python string substitution works fine in the sql syntax defintion
2) the cursor.execute function requires a tuple to work (as specified here, albeit not immediately clear, at least to me: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html)
EXAMPLE for both in one function:
def checkIfRecordExists(column, table, condition_name, condition_value):
...
sqlSyntax = 'SELECT {} FROM {} WHERE {} = %s'.format(column, table, condition_name)
cursor.execute(sqlSyntax, (condition_value,))
Note both the use of .format in the initial sql syntax definition and the use of (condition_value,) in the execute function.
I am using mysqldb to try to update a lot of records in a database.
cur.executemany("""UPDATE {} set {} =%s Where id = %s """.format(table, ' = %s, '.join(col)),updates.values.tolist())
I get the error message...
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near...
So I tried outputting the actual sql update statement as that error message wasn't helpful using the following code:
cur.execute('set profiling = 1')
try:
cur.executemany("""UPDATE {} set {} =%s Where id = %s """.format(table, ' = %s, '.join(col)),updates.values.tolist())
except Exception:
cur.execute('show profiles')
for row in cur:
print(row)
That print statement seems to cut off the update statement at 300 characters. I can't find anything in the documentation about limits so I am wondering is this the print statement limiting or is it mysqldb?
Is there a way I can generate the update statement with just python rather than mysqldb to see the full statement?
To see exactly what the cursor was executing, you can use the cursor.statement command as shown here in the API. That may help with the debugging.
I don't have experience with the mySQL adapter, but I work with the PostgreSQL adapter on a daily basis. At least in that context, it is recommended not to format your query string directly, but let the second argument in the cursor.execute statement do the substitution. This avoids problems with quoted strings and such. Here is an example, the second one is correct (at least for Postgres):
cur.execute("""UPDATE mytbl SET mycol = %s WHERE mycol2 = %s""".format(val, cond))
cur.execute("""UPDATE mytbl SET mycol = %(myval)s WHERE mycol2 = %(mycond)s""", {'myval': val, 'mycond': cond})
This can result in the query
UPDATE mytbl SET mycol = abc WHERE mycol2 = xyz
instead of
UPDATE mytbl SET mycol = 'abc' WHERE mycol2 = 'xyz'.
You would have needed to explicitly add those quotes if you do the value substitution in the query yourself, which becomes annoying and circumvents the type handling of the database adapter (keep in mind this was only a text example). See the API for a bit more information on this notation and the cursor.executemany command.
Could you please help me figure out what I am doing wrong?
I am trying to insert a new player into the players table.
this is the python code:
def registerPlayer(name):
code for connecting to db and cursor
c.execute("INSERT INTO players(player_name) VALUES({name});".format(name=name))
code for committing to db and closing the connection
here is my table schema:
CREATE TABLE players(
player_id serial PRIMARY KEY,
player_name varchar(50) NOT NULL
);
below is error:
psycopg2.ProgrammingError: syntax error at or near "Nalaar" LINE 1:
INSERT INTO players(player_name) VALUES(Chandra Nalaar);
You should never use string formatting for placing values into sql query. Instead, you should use %s and pass the name in the vars parameter. The reason behind doing it this way is because it helps you convert the parameters into the appropriate data types.
Btw, having a ; at the end in the sql string is redundant when called by cursor.execute, since it does it for you automatically.
c.execute("INSERT INTO players(player_name) VALUES(%(name)s)", {"name":name})
See this page for further details:
http://initd.org/psycopg/docs/usage.html#query-parameters
This question already has answers here:
How to UPSERT (MERGE, INSERT ... ON DUPLICATE UPDATE) in PostgreSQL?
(7 answers)
Closed 7 years ago.
I have a python script that is using the Psycopg adapter; I am parsing a JSON Array and inserting into my PostgreSQL database.
for item in data["SchoolJSONData"]:
mId = item.get("Id")
mNumofRooms = item.get("NumofRooms")
mFloors = item.get("Floors")
con = None
con = psycopg2.connect("dbname='database' user='dbuser'")
cur = con.cursor()
cur.execute('INSERT INTO Schools(Id, NumofRooms, Floors)VALUES(%s, %s, %s)',(mId, mNumofRooms, mFloors))
con.commit()
Everytime I run the script again, I get the following:
psycopg2.IntegrityError: duplicate key value violates unique constraint "schools_pkey"
How can I run the insert script so that it will ignore existing entries in the database?
EDIT: Thanks for the replies all... I am trying to NOT overwrite any data, only ADD (if the PK is not already in the table), and ignore any errors. In my case I will only be adding new entries, never updating data.
There is no single one way to solve this problem. As well this problem has little to do with python. It is valid exception generated by the database ( not just postgre all databases will do the same ).
But you can try - catch this exception and continue smoothly later.
OR
you can use "select count(*) where id = mId" to ensure it is not existing already.