The statement is set-up so that when a record already exists, it doesn't add a record, else, it does.
I've tried changing the query, even though I don't see anything wrong with it.
I've let the script run on python, and print the query it executed. Then I pasted that query in phpmyadmin, where it executed succesfully.
I have also double checked all parameters.
Query (blank params):
INSERT INTO users (uname,pass) SELECT * FROM (SELECT '{}','{}') AS tmp WHERE NOT EXISTS(SELECT uname FROM users WHERE uname = '{}') LIMIT 1;
Query (filled in parameters):
INSERT INTO users (uname,pass) SELECT * FROM (SELECT 'john_doe','password') AS tmp WHERE NOT EXISTS(SELECT uname FROM users WHERE uname = 'john_doe') LIMIT 1;
Python script (the important part)
if action == "add_user":
username = form.getvalue('username')
password = form.getvalue('password')
query = """
INSERT INTO users (uname,pass) SELECT * FROM
(SELECT '{}','{}') AS tmp WHERE NOT EXISTS(SELECT uname FROM users WHERE uname = '{}') LIMIT 1;
""".format(username, password, username)
mycursor.execute(query)
I know a couple of things.
There is nothing wrong with the database connection.
The parameters are not empty (ex. username="john_doe" & password="secret")
The query actually executes in that specific table.
The query seems to add a record and delete it directly afterwards (as AUTO_INCREMENT increases each time, even when the python script executes and doesn't add anything)
A try except doesn't do anything, as mysql.connector.Error doesn't report any error (which is obvious, since the query actually executes succesfully)
phpMyAdmin practical example:
(Removed INSERT INTO part in order to be able to show the resulting tables)
The first time you enter the query (above query as example), it will result in a table with both values as both column names as column values.
Screenshot of table output: http://prntscr.com/nkgaka
Once that result is entered once, next time you will try to insert it, it will simply result in only column names, no values. This means it will insert nothing, as there is nothing to insert as there are no actual values.
Screenshot of table output: http://prntscr.com/nkgbp3
Help is greatly appreciated.
If you want to ensure any field is unique in a table, make that field a PRIMARY KEY or UNIQUE KEY. In this case you want name to be unique.
CREATE UNIQUE INDEX name ON users(name)
With this in place you only need to INSERT. If a duplicate key error occurs the name already exists.
To avoid SQL injection, don't use """SELECT {}""".format. It will be SQL injection vulnerable.
Also don't store plain text passwords, ever. Salted hashes at least. There's plenty of frameworks that do this well already so you don't need to invent your own.
Related
Instead of using a JSON file to store data, I've decided I wanted to use a database instead. Here is how the data currently looks inside of the JSON file:
{"userID": ["reason 1", "reason 2", "reason 3"]}
I made it so that after a certain amount of time a reason is removed. For example, "reason 2" will be removed after 12 hours of it being added. However, I realised that if I terminate the process and then run it again the reason would just stay there until I manually remove it.
I've decided to use sqlite3 to make a database and have a discord.py task loop to remove it for me. How can I replicate the dictionary inside the database? Here is what I'm thinking at the moment:
c = sqlite3.connect('file_name.db')
cursor = c.cursor()
cursor.execute("""CREATE TABLE table_name (
userID text,
reason blob
)"""
Try the following table to store the reasons:
CREATE TABLE reasons (
reason_id PRIMARY KEY,
user_id,
reason,
is_visible,
created_at
)
Then the reasons could be soft deleted for every user by running:
UPDATE reasons
SET is_visible = 0
WHERE created_at + 3600 < CAST( strftime('%s', 'now') AS INT )
The example shows hiding reasons after 1 hour (3600 seconds).
The reasons can be hard deleted later by running the following query:
DELETE reasons
WHERE is_visible = 0
The soft delete comes in handy for verification and getting data back in case of a future bug in the software.
Simply use nested loops to insert each reason into a row of the table.
insert_query = """INSERT INTO table_name (userID, reason) VALUES (?, ?)"""
for user, reasons in json_data.items():
for reason in reasons:
cursor.execute(insert_query, (user, reason))
i wondering if it's possible to make a query with select with the user give the field of table and the value that want. For example:
field=input("Field: ")
value=input("Value: ")
cursorobject.execute('SELECT id FROM users WHERE {}=\'{}\'')
result=cursorobject.fetchall()
for x in result:
print(x)
and if it's not possible , there is any way to do it?
PS: this one not working
Of course you can construct the text of your query as you want using variables. E.g.
query = 'SELECT id FROM users WHERE {}=\'{}\''
print(query.format(field,value))
But, have in mind that you should validate very well the contents of the variables, before executing the query, to avoid SQL injections. For example the contents of the variables should not contain quotes.
E.g. the below code, with the specific values of the variables, will return the full list of users:
field='name'
value='name\' or \'1\'=\'1'
query = 'SELECT id FROM users WHERE {}=\'{}\''
print(query.format(field,value))
The produced query would be:
SELECT id FROM users WHERE name='name' or '1'='1'
Following your edit, you should replace your 3rd line with:
cursorobject.execute('SELECT id FROM users WHERE {}=\'{}\''.format(field,value))
And for making the best to avoid sql injections, you should use the built-in query parameterization features of your framework - pymysql:
cursorobject.execute('SELECT id FROM users WHERE {}=%s'.format(field),(value))
Simply format the query for field and pass value as a parameter in second argument of cursor.execute which must receive an iterable (i.e., tuple/list):
# PREPARED STATEMENT
sql = 'SELECT id FROM users WHERE {} = %s'
# EXECUTE QUERY
cursorobject.execute(sql.format(field), (value,))
result = cursorobject.fetchall()
I'm creating a change-password page for a website, which requests the new password and the current password. The old password is hashed and salted using the scrypt library then compared to the password stored in the sqlite3 database, and if these are a match, the new password is hashed and the database is updated. However I am having difficulty executing the update command, as it throws a sqlite3.OperationalError: unrecognised token: "\" error. The execute statement currently has the following code:
c.execute("UPDATE users SET password = \'{0}\' WHERE memberID = \'{1}\'".format(newPas, memID))
Initially we believed this error to have been caused by the use of ' in the string formatting due to the presence of ' within the new password itself, so this was run again as:
c.execute("UPDATE users SET password = \"{0}\" WHERE memberID = \"{1}\"".format(newPas, memID))
This successfully runs, but doesn't actually change anything in the database. We also attempted to create a query string and then execute the string.
query = "UPDATE users SET password = {0} WHERE memberID = {1}".format(newPas, memID)
c.execute(query)
This caused a sqlite3.OperationalError: near "'\xa1\x91\x9f\x88\xfb\x81\x12\xd4\xc2\xf9\xce\x91y\xf0/\xe1*#\x8aj\xc7\x1d\xd3\x91\x14\xcb\xa4\xabaP[\x02\x1d\x1b\xabr\xc7\xe4\xee\x19\x80c\x8e|\xc0S\xaaX\xc6\x04\xab\x08\x9b\x8e\xd7zB\xc6\x84[\xfb\xbc\x8d\xfc'": syntax error. I believe that this is caused by the presence of ' and " characters within the password, but I am unsure how to get around this issue as these are added by the hashing process and thus removing them would change the password.
The password I would like to add is:
b'\xa1\x91\x9f\x88\xfb\x81\x12\xd4\xc2\xf9\xce\x91y\xf0/\xe1*#\x8aj\xc7\x1d\xd3\x91\x14\xcb\xa4\xabaP[\x02\x1d\x1b\xabr\xc7\xe4\xee\x19\x80c\x8e|\xc0S\xaaX\xc6\x04\xab\x08\x9b\x8e\xd7zB\xc6\x84[\xfb\xbc\x8d\xfc'
I was wondering if anyone could share some insights into why it isn't liking the "\" character or why it isn't updating the database, and point me in the right direction to making it work. If you need more information or code snippets or just want to yell at me, please don't hesitate to! Thank you in advance :)
A couple of things with your code:
You should not use format to build your queries like this. This leaves you liable to SQL injection and, whilst you might sanitise your inputs in this case, it's a bad habit that will bite you.
All changes need to be committed to the database to actually take effect. This is why your second query did not throw an error but equally did not make any changes to the database.
The correct formatting of this query would be:
conn = sqlite3.connect('my_db.db')
c = conn.cursor()
query = "UPDATE users SET password = ? WHERE memberID = ?"
c.execute(query, (newPas, memID))
conn.commit() # To finalise the alteration
As a side note, the cursor expects a tuple in this case, so a common stumbling block comes when passing single values:
query = "UPDATE users SET password = ? WHERE memberID = 'abc'"
c.execute(query, (newPas)) # Throws "incorrect number of bindings" error
# Use this instead i.e. pass single value as a tuple
c.execute(query, (newPas,))
You could use format to create variable field names in a query, since placeholders are not allowed in this case:
fields = ['a', 'b', 'c']
query = "UPDATE users SET {} = ?".format(random.choice(fields))
in addition to using it to help you build big queries where it would be tedious to manually type all the placeholders, and difficult to ensure that you had the correct number if your code changed:
my_list = ['a', 'b',...., n]
placeholders = ', '.join(['?' for item in my_list])
query = "INSERT .... VALUES = ({})".format(placeholders)
You should use parametrized queries something like this:
c.execute("""UPDATE users SET password = ? WHERE memberID = ?;""", (newPas, memID))
It will allow to avoid nasty things like SQL-injections.
I have a table of three columnsid,word,essay.I want to do a query using (?). The sql sentence is sql1 = "select id,? from training_data". My code is below:
def dbConnect(db_name,sql,flag):
conn = sqlite3.connect(db_name)
cursor = conn.cursor()
if (flag == "danci"):
itm = 'word'
elif flag == "wenzhang":
itm = 'essay'
n = cursor.execute(sql,(itm,))
res1 = cursor.fetchall()
return res1
However, when I print dbConnect("data.db",sql1,"danci")
The result I obtained is [(1,'word'),(2,'word'),(3,'word')...].What I really want to get is [(1,'the content of word column'),(2,'the content of word column')...]. What should I do ? Please give me some ideas.
You can't use placeholders for identifiers -- only for literal values.
I don't know what to suggest in this case, as your function takes a database nasme, an SQL string, and a flag to say how to modify that string. I think it would be better to pass just the first two, and write something like
sql = {
"danci": "SELECT id, word FROM training_data",
"wenzhang": "SELECT id, essay FROM training_data",
}
and then call it with one of
dbConnect("data.db", sql['danci'])
or
dbConnect("data.db", sql['wenzhang'])
But a lot depends on why you are asking dbConnect to decide on the columns to fetch based on a string passed in from outside; it's an unusual design.
Update - SQL Injection
The problems with SQL injection and tainted data is well documented, but here is a summary.
The principle is that, in theory, a programmer can write safe and secure programs as long as all the sources of data are under his control. As soon as they use any information from outside the program without checking its integrity, security is under threat.
Such information ranges from the obvious -- the parameters passed on the command line -- to the obscure -- if the PATH environment variable is modifiable then someone could induce a program to execute a completely different file from the intended one.
Perl provides direct help to avoid such situations with Taint Checking, but SQL Injection is the open door that is relevant here.
Suppose you take the value for a database column from an unverfied external source, and that value appears in your program as $val. Then, if you write
my $sql = "INSERT INTO logs (date) VALUES ('$val')";
$dbh->do($sql);
then it looks like it's going to be okay. For instance, if $val is set to 2014-10-27 then $sql becomes
INSERT INTO logs (date) VALUES ('2014-10-27')
and everything's fine. But now suppose that our data is being provided by someone less than scrupulous or downright malicious, and your $val, having originated elsewhere, contains this
2014-10-27'); DROP TABLE logs; SELECT COUNT(*) FROM security WHERE name != '
Now it doesn't look so good. $sql is set to this (with added newlines)
INSERT INTO logs (date) VALUES ('2014-10-27');
DROP TABLE logs;
SELECT COUNT(*) FROM security WHERE name != '')
which adds an entry to the logs table as before, end then goes ahead and drops the entire logs table and counts the number of records in the security table. That isn't what we had in mind at all, and something we must guard against.
The immediate solution is to use placeholders ? in a prepared statement, and later passing the actual values in a call to execute. This not only speeds things up, because the SQL statement can be prepared (compiled) just once, but protects the database from malicious data by quoting every supplied value appropriately for the data type, and escaping any embedded quotes so that it is impossible to close one statement and another open another.
This whole concept was humourised in Randall Munroe's excellent XKCD comic
I'm using psycopg2 with Python.
I'd like to periodically flush data from my db. I've set up a task with Timer for this. I had asked this question before, but using the answer listed there freezes up my machine (keyboard stops responding and entire system grinds to halt). Instead, I would like to delete all entries in my table albeit the last N (Not sure that this is the right approach either).
Basically, there is another python process that is running (separate executable), which is populating the db that I wish to interrogate. It seems that if I delete all entries, and that other process is running, that it can lead to the freeze. I don't know of a safe way in which I can remove entries; it's almost as if the other process is relying on an incrementing ID as it writes to the db.
If anyone could help me work this out it'd be greatly appreciated. Thoughts?
A possible solution is to run a DELETE on all ids except those returned by select ... order by pk desc limit N given an autoincremental pk. If no such pk exists, having a created_date and ordering by it should do the same.
Non tested example:
import psycopg2
connection = psycopg2.connect('dbname=test user=postgres')
cursor = conn.cursor()
query = 'delete from my_table where id not in (
select id from my_table order by id desc limit 30)'
cursor.execute(query)
cursor.commit() #Don't know if necessary
cursor.close()
connection.close()
This is probably much faster:
CRETE TEMP TABLE tbl_tmp AS
SELECT * FROM tbl ORDER BY <undisclosed> LIMIT <N>;
TRUNCATE TABLE tbl;
INSERT INTO tbl SELECT * FROM tbl_tmp;
Do it all in one session. Specifics depend on additional circumstances you did not disclose.
Compare to this related, comprehensive answer (your case is simpler):
Remove duplicates from table based on multiple criteria and persist to other table