I’ve a sqlite3 db: phone book (name_id, phone_nb). I want to
insert (“Kyl”, +33661) if Kyl-entry doesn’t exist yet,
or, if Kyl already exists, I want to update his phone number to +33770. This is called upsert.
SQLite upsert is:
INSERT INTO table(...)
VALUES (...)
ON CONFLICT (...) DO UPDATE SET expression
My issue:
The above statement works perfectly when I use sqlite3, but it doesn’t work at all when I call the same from python.
On the other hand, if from python I use pure INSERT INTO table VALUES it works (without ON CONFLICT)
In addition, if from python I use classical UPDATE table SET col WHERE condition, it works too
Using SQLite upsert, I always have the same error: near "ON": syntax error
This is my table:
CREATE TABLE phone_book (
author_id INTEGER PRIMARY KEY
UNIQUE,
short_name TEXT NOT NULL,
join_date DATE NOT NULL,
email TEXT NOT NULL,
phone_nb STRING
);
From SQL Studio, I run
INSERT INTO phone_book(author_id, short_name, join_date, email, phone_nb)
VALUES (13, "kyl", "2020-12-20", "kyl#domain.net", 33670668832)
ON CONFLICT(author_id) DO UPDATE SET phone_nb=excluded.phone_nb;
This insert works. Then as Kyl changed his phone nb, I update his phone nb, using the same:
INSERT INTO phone_book(author_id, short_name, join_date, email, phone_nb)
VALUES (13, "kyl", "2020-12-20", "kyl#domain.net", 33677755231)
ON CONFLICT(author_id) DO UPDATE SET phone_nb=excluded.phone_nb;
This update work too. Everything’s in place! It’s time now to run all that from python. The bad news is that, when called from python, this precise statement doesn’t work at all.
What I’ve tried all the combinations:
cursor.execute(...)
cursor.executemany(...)
With explicit parameters
With ‘?’ placeholder
I always have the same error: near "ON": syntax error.
My non-working code with ‘?’ placeholder:
try:
sqliteConnection = sqlite3.connect('my.db')
cursor = sqliteConnection.cursor()
#print("Connected to SQLite")
author_id = 13
short_name = "mike"
join_date = "2021-01-12"
email = "mike#domain.net"
phone_nb = "00336"
tupple = []
tupple.append((author_id, short_name, join_date, email, phone_nb))
statement_ON_CONF = """INSERT INTO phone_book(author_id, short_name, join_date, email, phone_nb)
VALUES(?,?,?,?,?)
ON CONFLICT(author_id) DO UPDATE SET phone_nb=excluded.phone_nb;"""
print("statement_ON_CONF: " + statement_ON_CONF) # check my statement
cursor.executemany(statement_ON_CONF, tupple)
sqliteConnection.commit()
except sqlite3.Error as error:
print("Failed to insert or update into sqlite table: ", error)
finally:
if (sqliteConnection):
sqliteConnection.close()
#print("The SQLite connection is closed")
On the other hand, using pure INSERT and then UPDATE all's ok: my working code:
try:
sqliteConnection = sqlite3.connect('my.db')
cursor = sqliteConnection.cursor()
author_id = 2
short_name = "mike"
join_date = "2021-01-12"
email = "mike#domain.net"
phone_nb = "00336"
# Insert a new entry: Mike
statement = """INSERT INTO phone_book(author_id, short_name, join_date, email, phone_nb)
VALUES(?,?,?,?,?)"""
print("statement: " + statement)
cursor.execute(statement, (author_id, short_name, join_date, email, phone_nb))
sqliteConnection.commit()
# Update Mike phone nb
phone_nb = "+3310"
statement_ON_CONF = """INSERT INTO phone_book(author_id, short_name, join_date, email, phone_nb)
VALUES(?,?,?,?,?)
ON CONFLICT(author_id) DO UPDATE SET phone_nb=excluded.phone_nb;"""
statement_UPDATE = "UPDATE phone_book SET phone_nb=? WHERE author_id=?;"
cursor.execute(statement_UPDATE, (phone_nb, author_id))
sqliteConnection.commit()
except sqlite3.Error as error:
print("Failed to insert or update into sqlite table: ", error)
finally:
if (sqliteConnection):
sqliteConnection.close()
I use SQLite version 3.34.0 2020-12-01, and python version 3.7.2rc1, on Windows 7 Pro
Does anyone know why upsert always throws an error when called from python? Thanks!
According to a comment:
"SQLite supports UPSERT since version 3.24.0" – forpas.
You try to pass this tupple:
[(author_id, short_name, join_date, email, phone_nb)]
as VALUES in your statement, this results to error as it's not readable from sqlite
Try this instead:
statement_ON_CONF = """INSERT INTO phone_book(author_id, short_name, join_date, email, phone_nb)
VALUES ?
ON CONFLICT(author_id) DO UPDATE SET phone_nb=excluded.phone_nb;"""
print("statement_ON_CONF: " + statement_ON_CONF) # check my statement
cursor.executemany(statement_ON_CONF, tupple[0])
Related
I'm aware that the best way to prevent sql injection is to write Python queries of this form (or similar):
query = 'SELECT %s %s from TABLE'
fields = ['ID', 'NAME']
cur.execute(query, fields)
The above will work for a single query, but what if we want to do a UNION of 2 SQL commands? I've set this up via sqlite3 for sake of repeatability, though technically I'm using pymysql. Looks as follows:
import sqlite3
conn = sqlite3.connect('dummy.db')
cur = conn.cursor()
query = 'CREATE TABLE DUMMY(ID int AUTO INCREMENT, VALUE varchar(255))'
query2 = 'CREATE TABLE DUMMy2(ID int AUTO INCREMENT, VALUE varchar(255)'
try:
cur.execute(query)
cur.execute(query2)
except:
print('Already made table!')
tnames = ['DUMMY1', 'DUMMY2']
sqlcmds = []
for i in range(0,2):
query = 'SELECT %s FROM {}'.format(tnames[i])
sqlcmds.append(query)
fields = ['VALUE', 'VALUE']
sqlcmd = ' UNION '.join(sqlcmds)
cur.execute(sqlcmd, valid_fields)
When I run this, I get a sqlite Operational Error:
sqlite3.OperationalError: near "%": syntax error
I've validated the query prints as expected with this output:
INSERT INTO DUMMY VALUES(%s) UNION INSERT INTO DUMMY VALUES(%s)
All looks good there. What is the issue with the string substitutions here? I can confirm that running a query with direct string substitution works fine. I've tried it with both selects and inserts.
EDIT: I'm aware there are multiple ways to do this with executemany and a few other. I need to do this with UNION for the purposes I'm using this for because this is a very, very simplified example fo the operational code I'm using
The code below executes few INSERTS at once
import sqlite3
conn = sqlite3.connect('dummy.db')
cur = conn.cursor()
query = 'CREATE TABLE DUMMY(ID int AUTO INCREMENT NOT NULL, VALUE varchar(255))'
try:
cur.execute(query)
except:
print('Already made table!')
valid_fields = [('ya dummy',), ('stupid test example',)]
cur.executemany('INSERT INTO DUMMY (VALUE) VALUES (?)',valid_fields)
I am trying to update some values into a database. The user can give the row that should be changed. The input from the user, however is a string. When I try to parse this into the MySQL connector with python it gives an error because of the apostrophes. The code I have so far is:
import mysql.connector
conn = mysql.connector
conn = connector.connect(user=dbUser, password=dbPasswd, host=dbHost, database=dbName)
cursor = conn.cursor()
cursor.execute("""UPDATE Search SET %s = %s WHERE searchID = %s""", ('maxPrice', 300, 10,))
I get this error
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''maxPrice' = 300 WHERE searchID = 10' at line 1
How do I get rid of the apostrophes? Because I think they are causing problems.
As noted, you can't prepare it using a field.
Perhaps the safest way is to allow only those fields that are expected, e.g.
#!/usr/bin/python
import os
import mysql.connector
conn = mysql.connector.connect(user=os.environ.get('USER'),
host='localhost',
database='sandbox',
unix_socket='/var/run/mysqld/mysqld.sock')
cur = conn.cursor(dictionary=True)
query = """SELECT column_name
FROM information_schema.columns
WHERE table_schema = DATABASE()
AND table_name = 'Search'
"""
cur.execute(query)
fields = [x['column_name'] for x in cur.fetchall()]
user_input = ['maxPrice', 300, 10]
if user_input[0] in fields:
cur.execute("""UPDATE Search SET {0} = {1} WHERE id = {1}""".format(user_input[0], '%s'),
tuple(user_input[1:]))
print cur.statement
Prints:
UPDATE Search SET maxPrice = 300 WHERE id = 10
Where:
mysql> show create table Search\G
*************************** 1. row ***************************
Search
CREATE TABLE `Search` (
`id` int(11) DEFAULT NULL,
`maxPrice` float DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1
A column name is not a parameter. Put the column name maxPrice directly into your SQL.
cursor.execute("""UPDATE Search SET maxPrice = %s WHERE searchID = %s""", (300, 10))
If you want to use the same code with different column names, you would have to modify the string itself.
sql = "UPDATE Search SET {} = %s WHERE searchID = %s".format('maxPrice')
cursor.execute(sql, (300,10))
But bear in mind that this is not safe from injection the way parameters are, so make sure your column name is not a user-input string or anything like that.
You cannot do it like that. You need to place the column name in the string before you call cursor.execute. Column names cannot be used when transforming variables in cursor.execute.
Something like this would work:
sql = "UPDATE Search SET {} = %s WHERE searchID = %s".format('maxPrice')
cursor.execute(sql, (300, 10,))
You cannot dynamically bind object (e.g., column) names, only values. If that's the logic you're trying to achieve, you'd have to resort to string manipulation/formatting (with all the risks of SQL-injection attacks that come with it). E.g.:
sql = """UPDATE Search SET {} = %s WHERE searchID = %s""".format('maxPrice')
cursor.execute(sql, (300, 10,))
Guys I'm using sqlite3 with python tkinter as front end. The database is a simple one with two fields, username and password. I want to make a registration sign up page. where data given in the two fields to be stored in a sqlite3 database. Data is inserting properly. But I want to display a messagebox when the username provided already exist in the database. I tried the below code.
MY CODE:
def signup():
userID = username.get()
passwd = password.get()
conn = sqlite3.connect('test.db')
c = conn.cursor()
result = c.execute("SELECT * FROM userstable")
for i in result:
if i[0] == userID:
messagebox.showerror("DUPLICATE", "USER ALREADY EXISTS!")
else:
conn = sqlite3.connect('test.db')
c = conn.cursor()
c.execute("INSERT INTO userstable VALUES (?, ?)", (userID, passwd))
conn.commit()
c.close()
conn.close()
username.delete(0,END)
password.delete(0,END)
username.focus()
messagebox.showinfo("SUCCESS", "USER CREATED SUCCESSFULLY")
This works but still the duplicate data is being stored after the error message. My requirement is to throw the error and stop executing if the username is already available. If the username is not available already it should insert the data.
Where am I going wrong? could some one explain me by pointing out or is there any other way I can achieve this? It seems I need to modify something to my function. Please guide me.
EDIT 1
If i try to use three conditions the break is not working.
MY CODE
def data_entry():
conn = sqlite3.connect('test.db')
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS userstable(username TEXT, password TEXT)')
username = uname.get()
password = passwd.get()
result = c.execute("SELECT * FROM userstable")
if username != '' or password != '':
for i in result:
if i[0] == username:
tkinter.messagebox.showerror("DUPLICATE", "USER ALREADY EXISTS!")
break
else:
c.execute('INSERT INTO userstable (username, password) VALUES(?, ?)',(username,password))
conn.commit()
c.close()
conn.close()
another_clear()
tkinter.messagebox.showinfo("Success", "User Created Successfully,\nPlease restart application.")
else:
tkinter.messagebox.showerror("ERROR", "Fill both fields!")
A better way to approach this would be to create a unique constraint (index, in sqlite) on the table to prevent the insertion of a duplicate username. That way, you can try/except the insert statement instead of looping through a list of all users to see if it already exists (that's not scalable). This would also prevent you from having to "select *", which is generally bad practice (try to be explicit).
https://sqlite.org/lang_createtable.html
So, you could add the constraint either as a unique index, or as a primary key. If you only have 2 columns in this table, or if you have more than 2 but no additional IDs, your username can be your primary key. If you are introducing a system ID for your users, I'd use that as your primary key and username as a unique index. Either way, you'll need to alter your table to add the constraint.
CREATE UNIQUE INDEX username_uidx ON userstable (username);
Again, because you're not explicitly letting us know the column names, you'll have to fill that in.
After that:
try:
conn = sqlite3.connect('test.db')
c = conn.cursor()
c.execute("INSERT INTO userstable VALUES (?, ?)", (userID, passwd))
conn.commit()
except: # I'm not sure the exact error that's raised by SQLite
messagebox.showerror("DUPLICATE", "USER ALREADY EXISTS!")
finally:
c.close()
conn.close()
I typically wrap my cursor and connections in a finally so that they close even if there's an exception. That's not 100% of what you need, but it should get you there in one step with better DB design to enforce the uniqueness on a user.
I advise against using a loop with an else statement, it can be confusing.
See this post why-does-python-use-else-after-for-and-while-loops for more info.
If you want to use for - else you can add a break, so else will not be executed :
for i in result:
if i[0] == userID:
messagebox.showerror("DUPLICATE", "USER ALREADY EXISTS!")
break
else:
...
Or you can use a true / false flag :
user_exists = False
for i in result:
if i[0] == userID:
messagebox.showerror("DUPLICATE", "USER ALREADY EXISTS!")
user_exists = True
if not user_exists :
...
I have the following function:
def credential_check(username, password):
conn = sqlite3.connect('pythontkinter.db')
c = conn.cursor()
idvalue = c.execute('''SELECT ID FROM userdetails WHERE username = "{0}"'''.format(username)).fetchall()
print(idvalue)
I wish to assign the value of ID in my userdetails table to the variable idvalue in the row where the inputted username = userdetails username, however when I use this fetchall() I get [('0',)] printed out rather than just 0.
How do I go about doing this?
Thanks
You can use fetchone() if you only want one value. However, the result will still be returned as a tuple, just without the list.
import sqlite3
conn = sqlite3.connect('test.db')
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS testing(id TEXT)''')
conn.commit()
c.execute("""INSERT INTO testing (id) VALUES ('0')""")
conn.commit()
c.execute("""SELECT id FROM testing""")
data = c.fetchone()
print data
# --> (u'0',)
You can also use LIMIT if you want to restrict the number of returned values with fetchall().
More importantly, don't format your queries like that. Get used to using the ? placeholder as a habit so that you are not vulnerable to SQL injection.
idvalue = c.execute("""SELECT ID FROM userdetails WHERE username = ?""", (username,)).fetchone()
I am trying to query the records for a specific ID in an Oracle table based on what the user inputs.
Here is my code:
import cx_Oracle
con = cx_Oracle.connect('dbuser/dbpassword#oracle_host/service_ID')
cur = con.cursor()
id_number = raw_input('What is the ID Number?')
cur.execute('select id, info from oracle_table_name where id=:id_number')
for result in cur:
print "test", result
cur.close()
con.close()
The following error pops up: cx_Oracle.DatabaseError: ORA-01008: not all variables bound
When I remove the user input and the variable substitution and run the query, everything works fine.
:id_number in your SQL is a parameter (variable). You need to provide its value.
execute method accepts parameters as the second argument.
Example:
query = "select * from some_table where col=:my_param"
cursor.execute(query, {'my_param': 5})
Check the documentation at http://cx-oracle.readthedocs.org/en/latest/cursor.html#Cursor.execute
I assigned a name to the user_value:
user_value = raw_input('What is the ID Number?')
And then referenced it in the execute statement:
cur.execute(query, {'id': (user_value)})
Thanks to Radoslaw-Roszkowiak for the assist!!