I am learning myself some programming with python and make use of SQlite3
I keep running into the same problem, and I can't figure out what goes wrong.
My table setup
def user_table():
data = lite.connect(database)
dat = data.cursor()
with data:
dat.executescript("""CREATE TABLE IF NOT EXISTS Users(Id INTEGER PRIMARY KEY AUTOINCREMENT,
'Username' UNIQUE,
'Password',
'Email',
'UserCharacters_Id' INTEGER
)""");
Now my code to select a Username (the username 123 exists and tables seem right (checked with SQLite studio)
database = 'test.db'
data = lite.connect(database)
dat = data.cursor()
with data:
dat.execute("SELECT * FROM Users WHERE 'Username'='123'")
user = dat.fetchone()
print user
I tried a lot of different ways, but it keeps returning None.
The python part seems to be working, just the select part of SQL goes wrong (checked with prints)
Please help me out
In SQL, single quotes are used for strings, while table/columns names are quoted with double quotes.
(SQLite supports single quotes for the latter for compatibility with MySQL in some places.)
Your query compares the string Username against the string 123, and this comparison fails for every record.
Use this:
dat.execute('SELECT * FROM Users WHERE "Username" = \'123\'')
But to prevent string formatting problems and SQL injection attacks, you should use parameters:
username = "123"
dat.execute('SELECT * FROM Users WHERE "Username" = ?', (username,))
Related
I use pyodbc in my python flask Project for the SQLite DB connection.
I know and understand SQL Injections but this is my first time dealing with it.
I tried to execute some
I have a function which concatenates the SQL String in my database.py file:
def open_issue(self, data_object):
cursor = self.conn.cursor()
# data_object is the issue i get from the user
name = data_object["name"]
text = data_object["text"]
rating_sum = 0
# if the user provides an issue
if name:
# check if issue is already in db
test = cursor.execute(f'''SELECT name FROM issue WHERE name = "{name}"''')
data = test.fetchall()
# if not in db insert
if len(data) == 0:
# insert the issue
cursor.executescript(f'''INSERT INTO issue (name, text, rating_sum)
VALUES ("{name}", "{text}", {rating_sum})''')
else:
print("nothing inserted!")
In the api.py file the open_issue() function gets called:
#self.app.route('/open_issue')
def insertdata():
# data sent from client
# data_object = flask.request.json
# unit test dictionary
data_object = {"name": "injection-test-table",
"text": "'; CREATE TABLE 'injected_table-1337';--"}
DB().open_issue(data_object)
The "'; CREATE TABLE 'injected_table-1337';--" sql injection has not created the injected_table-1337, instead it got inserted normally like a string into the text column of the injection-test-table.
So i don't really know if i am safe for the standard ways of SQL injection (this project will only be hosted locally but good security is always welcome)
And secondary: are there ways with pyodbc to check if a string contains sql syntax or symbols, so that nothing will get inserted in my example or do i need to check the strings manually?
Thanks a lot
As it turns out, with SQLite you are at much less risk of SQL injection issues because by default neither Python's built-in sqlite3 module nor the SQLite ODBC driver allow multiple statements to be executed in a single .execute call (commonly known as an "anonymous code block"). This code:
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = f"SELECT * FROM table1 WHERE txt='{thing}'"
crsr.execute(sql)
throws this for sqlite3
sqlite3.Warning: You can only execute one statement at a time.
and this for SQLite ODBC
pyodbc.Error: ('HY000', '[HY000] only one SQL statement allowed (-1) (SQLExecDirectW)')
Still, you should follow best practices and use a proper parameterized query
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = "SELECT * FROM table1 WHERE txt=?"
crsr.execute(sql, (thing, ))
because this will also correctly handle parameter values that would cause errors if injected directly, e.g.,
thing = "it's good to avoid SQL injection"
The statement is set-up so that when a record already exists, it doesn't add a record, else, it does.
I've tried changing the query, even though I don't see anything wrong with it.
I've let the script run on python, and print the query it executed. Then I pasted that query in phpmyadmin, where it executed succesfully.
I have also double checked all parameters.
Query (blank params):
INSERT INTO users (uname,pass) SELECT * FROM (SELECT '{}','{}') AS tmp WHERE NOT EXISTS(SELECT uname FROM users WHERE uname = '{}') LIMIT 1;
Query (filled in parameters):
INSERT INTO users (uname,pass) SELECT * FROM (SELECT 'john_doe','password') AS tmp WHERE NOT EXISTS(SELECT uname FROM users WHERE uname = 'john_doe') LIMIT 1;
Python script (the important part)
if action == "add_user":
username = form.getvalue('username')
password = form.getvalue('password')
query = """
INSERT INTO users (uname,pass) SELECT * FROM
(SELECT '{}','{}') AS tmp WHERE NOT EXISTS(SELECT uname FROM users WHERE uname = '{}') LIMIT 1;
""".format(username, password, username)
mycursor.execute(query)
I know a couple of things.
There is nothing wrong with the database connection.
The parameters are not empty (ex. username="john_doe" & password="secret")
The query actually executes in that specific table.
The query seems to add a record and delete it directly afterwards (as AUTO_INCREMENT increases each time, even when the python script executes and doesn't add anything)
A try except doesn't do anything, as mysql.connector.Error doesn't report any error (which is obvious, since the query actually executes succesfully)
phpMyAdmin practical example:
(Removed INSERT INTO part in order to be able to show the resulting tables)
The first time you enter the query (above query as example), it will result in a table with both values as both column names as column values.
Screenshot of table output: http://prntscr.com/nkgaka
Once that result is entered once, next time you will try to insert it, it will simply result in only column names, no values. This means it will insert nothing, as there is nothing to insert as there are no actual values.
Screenshot of table output: http://prntscr.com/nkgbp3
Help is greatly appreciated.
If you want to ensure any field is unique in a table, make that field a PRIMARY KEY or UNIQUE KEY. In this case you want name to be unique.
CREATE UNIQUE INDEX name ON users(name)
With this in place you only need to INSERT. If a duplicate key error occurs the name already exists.
To avoid SQL injection, don't use """SELECT {}""".format. It will be SQL injection vulnerable.
Also don't store plain text passwords, ever. Salted hashes at least. There's plenty of frameworks that do this well already so you don't need to invent your own.
My objective is to store a JSON object into a MySQL database field of type json, using the mysql.connector library.
import mysql.connector
import json
jsonData = json.dumps(origin_of_jsonData)
cnx = mysql.connector.connect(**config_defined_elsewhere)
cursor = cnx.cursor()
cursor.execute('CREATE DATABASE dataBase')
cnx.database = 'dataBase'
cursor = cnx.cursor()
cursor.execute('CREATE TABLE table (id_field INT NOT NULL, json_data_field JSON NOT NULL, PRIMARY KEY (id_field))')
Now, the code below WORKS just fine, the focus of my question is the use of '%s':
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES (%s, %s)"
values_to_insert = (1, jsonData)
cursor.execute(insert_statement, values_to_insert)
My problem with that: I am very strictly adhering to the use of '...{}'.format(aValue) (or f'...{aValue}') when combining variable aValue(s) into a string, thus avoiding the use of %s (whatever my reasons for that, let's not debate them here - but it is how I would like to keep it wherever possible, hence my question).
In any case, I am simply unable, whichever way I try, to create something that stores the jsonData into the mySql dataBase using something that resembles the above structure and uses '...{}'.format() (in whatever shape or form) instead of %s. For example, I have (among many iterations) tried
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES ({}, {})".format(1, jsonData)
cursor.execute(insert_statement)
but no matter how I turn and twist it, I keep getting the following error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[some_content_from_jsonData})]' at line 1
Now my question(s):
1) Is there a way to avoid the use of %s here that I am missing?
2) If not, why? What is it that makes this impossible? Is it the cursor.execute() function, or is it the fact that it is a JSON object, or is it something completely different? Shouldn't {}.format() be able to do everything that %s could do, and more?
First of all: NEVER DIRECTLY INSERT YOUR DATA INTO YOUR QUERY STRING!
Using %s in a MySQL query string is not the same as using it in a python string.
In python, you just format the string and 'hello %s!' % 'world' becomes 'hello world!'. In SQL, the %s signals parameter insertion. This sends your query and data to the server separately. You are also not bound to this syntax. The python DB-API specification specifies more styles for this: DB-API parameter styles (PEP 249). This has several advantages over inserting your data directly into the query string:
Prevents SQL injection
Say you have a query to authenticate users by password. You would do that with the following query (of course you would normally salt and hash the password, but that is not the topic of this question):
SELECT 1 FROM users WHERE username='foo' AND password='bar'
The naive way to construct this query would be:
"SELECT 1 FROM users WHERE username='{}' AND password='{}'".format(username, password)
However, what would happen if someone inputs ' OR 1=1 as password. The formatted query would then become
SELECT 1 FROM users WHERE username='foo' AND password='' OR 1=1
which will allways return 1. When using parameter insertion:
execute('SELECT 1 FROM users WHERE username=%s AND password=%s', username, password)
this will never happen, as the query will be interpreted by the server separately.
Performance
If you run the same query many times with different data, the performance difference between using a formatted query and parameter insertion can be significant. With parameter insertion, the server only has to compile the query once (as it is the same every time) and execute it with different data, but with string formatting, it will have to compile it over and over again.
In addition to what was said above, I would like to add some details that I did not immediately understand, and that other (newbies like me ;)) may also find helpful:
1) "parameter insertion" is meant for only for values, it will not work for table names, column names, etc. - for those, the Python string substitution works fine in the sql syntax defintion
2) the cursor.execute function requires a tuple to work (as specified here, albeit not immediately clear, at least to me: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html)
EXAMPLE for both in one function:
def checkIfRecordExists(column, table, condition_name, condition_value):
...
sqlSyntax = 'SELECT {} FROM {} WHERE {} = %s'.format(column, table, condition_name)
cursor.execute(sqlSyntax, (condition_value,))
Note both the use of .format in the initial sql syntax definition and the use of (condition_value,) in the execute function.
I am building a registration system for my web application wherein users provide a username and a password. This data is stored in a postgresql database. I am using bcrypt to generate a salted hash of the user entered password as follows
import bcrypt
hashed = bcrypt.hashpw(PasswordFromWebForm.encode('UTF-8'), bcrypt.gensalt())
This creates a salted password that looks something like this - b'$2b$12$GskbcRCMFHGuXumrNt3FLO'
I am storing this value in a postgresql database. Next, when a user tries to login to the system, I want to verify his/her credentials. To do this, I intend to do something along the lines of -
import psycopg2
conn = psycopg2.connect("dbname=test user=me")
cur = conn.cursor()
saltedpassword = cur.execute("SELECT saltedpassword FROM test WHERE loginid = %s", (LoginIDFromWebForm,))
if bcrypt.hashpw(PasswordFromWebForm.encode('UTF-8'), saltedpassword) == saltedpassword:
print("Success")
This does not work. It throws the following TypeError: Unicode-objects must be encoded before hashing error.
I suspect that this error is because the variable saltedpassword stores the value as a string like this "b'$2b$12$GskbcRCMFHGuXumrNt3FLO'" instead of just plain b'$2b$12$GskbcRCMFHGuXumrNt3FLO' (Notice the quotes enclosing the salted password in the former)
How do I get around this problem? What is the best way to store the salted hashed password in a database and how do I go about retrieving it when needed for verification? Apologies for the rather longish question - please help.
psycopg2 stores Unicode text, you must decode the salted password before inserting into the database:
cur.execute("INSERT INTO test VALUES (%s, %s)",
(LoginIDFromWebForm, saltedpassword.decode('ascii')))
This prevents the str() conversion being inserted instead (giving you "b'....'" in the database).
Next, when querying saltedpassword is not a string, because cursor.execute() does not return the results; it returns None instead.
You'll need to fetch the result row:
cur.execute("SELECT saltedpassword FROM test WHERE loginid = %s", (LoginIDFromWebForm,))
row = cur.fetchone()
if not row:
# No such login ID, handle accordingly
saltedpassword = row[0].encode('ascii')
Since rows contain Unicode text, you need to first encode to a bytestring before passing it to bcrypt. You also want to use the bcrypt.checkpw() function to check the password:
if bcrypt.checkpw(PasswordFromWebForm.encode('UTF-8'), saltedpassword):
print("Success")
bcrypt.checkpw() avoids timing attacks when comparing strings, something your code is vulnerable to.
I'm creating a change-password page for a website, which requests the new password and the current password. The old password is hashed and salted using the scrypt library then compared to the password stored in the sqlite3 database, and if these are a match, the new password is hashed and the database is updated. However I am having difficulty executing the update command, as it throws a sqlite3.OperationalError: unrecognised token: "\" error. The execute statement currently has the following code:
c.execute("UPDATE users SET password = \'{0}\' WHERE memberID = \'{1}\'".format(newPas, memID))
Initially we believed this error to have been caused by the use of ' in the string formatting due to the presence of ' within the new password itself, so this was run again as:
c.execute("UPDATE users SET password = \"{0}\" WHERE memberID = \"{1}\"".format(newPas, memID))
This successfully runs, but doesn't actually change anything in the database. We also attempted to create a query string and then execute the string.
query = "UPDATE users SET password = {0} WHERE memberID = {1}".format(newPas, memID)
c.execute(query)
This caused a sqlite3.OperationalError: near "'\xa1\x91\x9f\x88\xfb\x81\x12\xd4\xc2\xf9\xce\x91y\xf0/\xe1*#\x8aj\xc7\x1d\xd3\x91\x14\xcb\xa4\xabaP[\x02\x1d\x1b\xabr\xc7\xe4\xee\x19\x80c\x8e|\xc0S\xaaX\xc6\x04\xab\x08\x9b\x8e\xd7zB\xc6\x84[\xfb\xbc\x8d\xfc'": syntax error. I believe that this is caused by the presence of ' and " characters within the password, but I am unsure how to get around this issue as these are added by the hashing process and thus removing them would change the password.
The password I would like to add is:
b'\xa1\x91\x9f\x88\xfb\x81\x12\xd4\xc2\xf9\xce\x91y\xf0/\xe1*#\x8aj\xc7\x1d\xd3\x91\x14\xcb\xa4\xabaP[\x02\x1d\x1b\xabr\xc7\xe4\xee\x19\x80c\x8e|\xc0S\xaaX\xc6\x04\xab\x08\x9b\x8e\xd7zB\xc6\x84[\xfb\xbc\x8d\xfc'
I was wondering if anyone could share some insights into why it isn't liking the "\" character or why it isn't updating the database, and point me in the right direction to making it work. If you need more information or code snippets or just want to yell at me, please don't hesitate to! Thank you in advance :)
A couple of things with your code:
You should not use format to build your queries like this. This leaves you liable to SQL injection and, whilst you might sanitise your inputs in this case, it's a bad habit that will bite you.
All changes need to be committed to the database to actually take effect. This is why your second query did not throw an error but equally did not make any changes to the database.
The correct formatting of this query would be:
conn = sqlite3.connect('my_db.db')
c = conn.cursor()
query = "UPDATE users SET password = ? WHERE memberID = ?"
c.execute(query, (newPas, memID))
conn.commit() # To finalise the alteration
As a side note, the cursor expects a tuple in this case, so a common stumbling block comes when passing single values:
query = "UPDATE users SET password = ? WHERE memberID = 'abc'"
c.execute(query, (newPas)) # Throws "incorrect number of bindings" error
# Use this instead i.e. pass single value as a tuple
c.execute(query, (newPas,))
You could use format to create variable field names in a query, since placeholders are not allowed in this case:
fields = ['a', 'b', 'c']
query = "UPDATE users SET {} = ?".format(random.choice(fields))
in addition to using it to help you build big queries where it would be tedious to manually type all the placeholders, and difficult to ensure that you had the correct number if your code changed:
my_list = ['a', 'b',...., n]
placeholders = ', '.join(['?' for item in my_list])
query = "INSERT .... VALUES = ({})".format(placeholders)
You should use parametrized queries something like this:
c.execute("""UPDATE users SET password = ? WHERE memberID = ?;""", (newPas, memID))
It will allow to avoid nasty things like SQL-injections.