I have the cursor with the query statement as follows:
cursor.execute("select rowid from components where name = ?", (name,))
I want to check for the existence of the components: name and return to a python variable.
How do I do that?
Since the names are unique, I really favor your (the OP's) method of using fetchone or Alex Martelli's method of using SELECT count(*) over my initial suggestion of using fetchall.
fetchall wraps the results (typically multiple rows of data) in a list. Since the names are unique, fetchall returns either a list with just one tuple in the list (e.g. [(rowid,),] or an empty list []. If you desire to know the rowid, then using fetchall requires you to burrow through the list and tuple to get to the rowid.
Using fetchone is better in this case since you get just one row, (rowid,) or None.
To get at the rowid (provided there is one) you just have to pick off the first element of the tuple.
If you don't care about the particular rowid and you just want to know there is a hit,
then you could use Alex Martelli's suggestion, SELECT count(*), which would return either (1,) or (0,).
Here is some example code:
First some boiler-plate code to setup a toy sqlite table:
import sqlite3
connection = sqlite3.connect(':memory:')
cursor=connection.cursor()
cursor.execute('create table components (rowid int,name varchar(50))')
cursor.execute('insert into components values(?,?)', (1,'foo',))
Using fetchall:
for name in ('bar','foo'):
cursor.execute("SELECT rowid FROM components WHERE name = ?", (name,))
data=cursor.fetchall()
if len(data)==0:
print('There is no component named %s'%name)
else:
print('Component %s found with rowids %s'%(name,','.join(map(str, next(zip(*data))))))
yields:
There is no component named bar
Component foo found with rowids 1
Using fetchone:
for name in ('bar','foo'):
cursor.execute("SELECT rowid FROM components WHERE name = ?", (name,))
data=cursor.fetchone()
if data is None:
print('There is no component named %s'%name)
else:
print('Component %s found with rowid %s'%(name,data[0]))
yields:
There is no component named bar
Component foo found with rowid 1
Using SELECT count(*):
for name in ('bar','foo'):
cursor.execute("SELECT count(*) FROM components WHERE name = ?", (name,))
data=cursor.fetchone()[0]
if data==0:
print('There is no component named %s'%name)
else:
print('Component %s found in %s row(s)'%(name,data))
yields:
There is no component named bar
Component foo found in 1 row(s)
I have found the answer.
exist = cursor.fetchone()
if exist is None:
... # does not exist
else:
... # exists
As both existing answers (your own and #unutbu's) point out, the trick is that you do need to do some sort of fetching, after executing the SELECT, to check whether there have been any results to the select or not (whether you do it with a single fetch and check for none, or a fetch-all and check for an empty list, is a marginal difference -- given that you mention a UNIQUE constraint they're basically equivalent approaches).
For a very direct answer, you could select count(*) from components where name = ?, rather than selecting rowid, if all you care is whether the given value for name is present or not (as opposed to, caring about what row id it's on, if at all;-). Executing this select, and fetching the result, gives you 0 if the value is absent, 1 if it's present (no other result is possible given what you mentioned in a comment about the UNIQUE constraint on column name;-).
Starting Python 3.8, and the introduction of assignment expressions (PEP 572) (:= operator), we can slightly modify previous answers by capturing the result of cursor.fetchone() within the condition check and re-use it to extract the result content:
# cursor.execute("SELECT rowid FROM components WHERE name = ?", (name,))
if item := cursor.fetchone():
print(f'Component {name} found with rowid {item[0]}')
else:
print(f'There is no component named {name}')
To make it even shorter...:
row = cursor.fetchone()
if row:
print "row is present"
else:
print "row is not present"
The reason mine was showing the "Lock" message was actually due to me having opened an SQLite3 IDE on my Windows and that was the reason it was locked. I assume I was playing around with the DB within the IDE and hadn't saved the changes and therefor a lock was placed.
This is what worked for me
exist = conn.execute("select rowid from rowid where name = ?", (name,)).fetchone()
if exist is None:
print("Doesn't exist")
else:
print("Yep exists")
Related
I have a large database with hundreds of tables. Each table has a functionally equivalent structure: one column is a unique identifier and the other a group membership indicator; that is, each row in the table has a unique identifier, but any number of rows can have the same group membership indicator. The tables are also created in pairs in the same schema, so the naming scheme for this database is project_abbreviation.<name>_<suffix>; for example, the pair proj_abc.original_a and proj_abc.original_b.
I inherited this database, and when the original developers constructed it, they did not add UNIQUE constraints to the unique identifier columns when the tables were created. As a result, whenever someone wants to change the group membership indicator for a row or set of rows in a given table, I have to add a UNIQUE constraint on the column if the table hasn't been modified since its creation. I do that programmatically:
#connect
def make_column_unique(self, cursor, connection, column, suffix):
sql = f"ALTER TABLE {self._schema}.{self._table}_{suffix} "
sql += f"ADD CONSTRAINT unique_{column} UNIQUE ({column});"
cursor.execute(sql)
connection.commit()
where #connect is a decorator function which connects to the db instance, and the cursor and connection parameters are psycopg2 Cursor and Connection objects, respectively. I then call this in a try/except block:
...
for suffix in ["a", "b"]:
try:
self.modify_table(...)
except (Exception, psycopg2.DatabaseError) as e:
self.make_column_unique("uid", suffix)
self.modify_table(...)
...
Here is the function signature for self.modify_table:
#connect
def modify_table(self, cursor, connection, data, suffix):
sql = f"INSERT INTO {self._schema}.{self._table}_{suffix} (uid, group) "
sql += "VALUES "
zipped = list(zip(list(data["uid"]), list(data["group"])))
row = 0
for uid, group in zipped:
row += 1
sql += f"({uid},'{group}')" + ("," if row < len(zipped) else " ")
sql += f"ON CONFLICT (uid) DO UPDATE "
sql += "SET group = EXCLUDED.group;"
cursor.execute(sql)
connection.commit()
This approach worked exceedingly well and modified table entries properly, and set the UNIQUE constraint when one needed to be set.
Now, when I attempt to modify a table which has yet to be modified, I get a There is no unique or exclusion constraint matching the ON CONFLICT specification error, which kicks off the call to make_column_unique. However, when the program attempts to make the provided column unique, I get back a relation "unique_<column>" already exists error. Furthermore, this only happens for tables of suffix a, not suffix b. I went into pgAdmin4 to verify, and the desired modification occurred on the table with suffix b, but before and after the database transaction, the table with suffix a had no constraints applied to it:
pgAdmin database viewer with no constraints
Why am I getting these contradictory errors for only one type of table? It makes no sense to me to be told that a UNIQUE constraint doesn't exist, and then when I alter the table to include the constraint, to be told that it already exists.
This is the dumbest possible answer I could give to this question.
It turns out that, in my predecessor's infinite wisdom, they decided to randomly sprinkle duplicate values into columns that were supposed to be unique. Easily fixed.
Question: Is it possible to use a variable as your table name without having to use string constructors to do so?
Info:
I'm working on a project right now that catalogs data from a star simulation of mine. To do so I'm loading all the data into a sqlite database. It's working pretty well, but I've decided to add a lot more flexibility, efficiency, and usability to my db. I plan on later adding planetoids to the simulation, and wanted to have a table for each star. This way I wouldn't have to query a table of 20m some planetoids for the 1-4k in each solar system.
I've been told using string constructors is bad because it leaves me vulnerable to a SQL injection attack. While that isn't a big deal here as I'm the only person with access to these dbs, I would like to follow best practices. And also this way if I do a project with a similar situation where it is open to the public, I know what to do.
Currently I'm doing this:
cursor.execute("CREATE TABLE StarFrame"+self.name+" (etc etc)")
This works, but I would like to do something more like:
cursor.execute("CREATE TABLE StarFrame(?) (etc etc)",self.name)
though I understand that this would probably be impossible. though I would settle for something like
cursor.execute("CREATE TABLE (?) (etc etc)",self.name)
If this is not at all possible, I'll accept that answer, but if anyone knows a way to do this, do tell. :)
I'm coding in python.
Unfortunately, tables can't be the target of parameter substitution (I didn't find any definitive source, but I have seen it on a few web forums).
If you are worried about injection (you probably should be), you can write a function that cleans the string before passing it. Since you are looking for just a table name, you should be safe just accepting alphanumerics, stripping out all punctuation, such as )(][;, and whitespace. Basically, just keep A-Z a-z 0-9.
def scrub(table_name):
return ''.join( chr for chr in table_name if chr.isalnum() )
scrub('); drop tables --') # returns 'droptables'
For people searching for a way to make the table as a variable, I got this from another reply to same question here:
It said the following and it works. It's all quoted from mhawke:
You can't use parameter substitution for the table name. You need to add the table name to the query string yourself. Something like this:
query = 'SELECT * FROM {}'.format(table)
c.execute(query)
One thing to be mindful of is the source of the value for the table name. If that comes from an untrusted source, e.g. a user, then you need to validate the table name to avoid potential SQL injection attacks. One way might be to construct a parameterised query that looks up the table name from the DB catalogue:
import sqlite3
def exists_table(db, name):
query = "SELECT 1 FROM sqlite_master WHERE type='table' and name = ?"
return db.execute(query, (name,)).fetchone() is not None
I wouldn't separate the data into more than one table. If you create an index on the star column, you won't have any problem efficiently accessing the data.
Try with string formatting:
sql_cmd = '''CREATE TABLE {}(id, column1, column2, column2)'''.format(
'table_name')
db.execute(sql_cmd)
Replace 'table_name' with your desire.
To avoid hard-coding table names, I've used:
table = "sometable"
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS {} (
importantdate DATE,
somename VARCHAR,
)'''.format(table))
c.execute('''INSERT INTO {} VALUES (?, ?)'''.format(table),
(datetime.strftime(datetime.today(), "%Y-%m-%d"),
myname))
As has been said in the other answers, "tables can't be the target of parameter substitution" but if you find yourself in a bind where you have no option, here is a method of testing if the table name supplied is valid.
Note: I have made the table name a real pig in an attempt to cover all of the bases.
import sys
import sqlite3
def delim(s):
delims="\"'`"
use_delim = []
for d in delims:
if d not in s:
use_delim.append(d)
return use_delim
db_name = "some.db"
db = sqlite3.connect(db_name)
mycursor = db.cursor()
table = 'so""m ][ `etable'
delimiters = delim(table)
if len(delimiters) < 1:
print "The name of the database will not allow this!"
sys.exit()
use_delimiter = delimiters[0]
print "Using delimiter ", use_delimiter
mycursor.execute('SELECT name FROM sqlite_master where (name = ?)', [table])
row = mycursor.fetchall()
valid_table = False
if row:
print (table,"table name verified")
valid_table = True
else:
print (table,"Table name not in database", db_name)
if valid_table:
try:
mycursor.execute('insert into ' +use_delimiter+ table +use_delimiter+ ' (my_data,my_column_name) values (?,?) ',(1,"Name"));
db.commit()
except Exception as e:
print "Error:", str(e)
try:
mycursor.execute('UPDATE ' +use_delimiter+ table +use_delimiter+ ' set my_column_name = ? where my_data = ?', ["ReNamed",1])
db.commit()
except Exception as e:
print "Error:", str(e)
db.close()
you can use something like this
conn = sqlite3.connect()
createTable = '''CREATE TABLE %s (# );''' %dateNow)
conn.execute(createTable)
basically, if we want to separate the data into several tables according to the date right now, for example, you want to monitor a system based on the date.
createTable = '''CREATE TABLE %s (# );''' %dateNow) means that you create a table with variable dateNow which according to your coding language, you can define dateNow as a variable to retrieve the current date from your coding language.
You can save your query in a .sql or txt file and use the open().replace() method to use variables in any part of your query. Long time reader but first time poster so I apologize if anything is off here.
```SQL in yoursql.sql```
Sel *
From yourdbschema.tablenm
```SQL to run```
tablenm = 'yourtablename'
cur = connect.cursor()
query = cur.execute(open(file = yoursql.sql).read().replace('tablenm',tablenm))
You can pass a string as the SQL command:
import sqlite3
conn = sqlite3.connect('db.db')
c = conn.cursor()
tablename, field_data = 'some_table','some_data'
query = 'SELECT * FROM '+tablename+' WHERE column1=\"'+field_data+"\""
c.execute(query)
My postgres query is:
query = """INSERT INTO statustable(value) SELECT '%s'
WHERE NOT EXISTS (SELECT id, value FROM statustable
WHERE value = '%s') RETURNING id""" % (status, status)
cursor_postgres.execute(query)
conn_postgres.commit()
statusId = cursor_postgres.fetchone()[0]
print "statusId" + str(statusId)
I need to get the freshly inserted status value id if it doesnt exist, or select its id if already exist. RETURNING id choked this query entirely so I had to remove it to atleast get the selective insertion working.
Any clues how to get the statusId here? In another instance I am doing an Upsert.(Insert if not exist, update otherwise) Here again, I need the inserted or updated row id. (No, I am not using stored procedures, if that was your first question...)
Thanks in advance
I can't say I fully understand your motivation for insisting on a single query. I think your best bet is to have two simple queries:
SELECT id FROM statustable WHERE value = '%s'. This gives you the id if the entry exists, in which case skip step 2;
INSERT INTO statustable(value) VALUES('%s') RETURNING id. This'll give you the id of the newly created entry.
Lastly -- although I haven't verified whether this is a problem -- fetchone() across a commit looks slightly suspect.
I have a simple table in mysql with the following fields:
id -- Primary key, int, autoincrement
name -- varchar(50)
description -- varchar(256)
Using MySQLdb, a python module, I want to insert a name and description into the table, and get back the id.
In pseudocode:
db = MySQLdb.connection(...)
queryString = "INSERT into tablename (name, description) VALUES" % (a_name, a_desc);"
db.execute(queryString);
newID = ???
I think it might be
newID = db.insert_id()
Edit by Original Poster
Turns out, in the version of MySQLdb that I am using (1.2.2)
You would do the following:
conn = MySQLdb(host...)
c = conn.cursor()
c.execute("INSERT INTO...")
newID = c.lastrowid
I am leaving this as the correct answer, since it got me pointed in the right direction.
I don't know if there's a MySQLdb specific API for this, but in general you can obtain the last inserted id by SELECTing LAST_INSERT_ID()
It is on a per-connection basis, so you don't risk race conditions if some other client performs an insert as well.
You could also do a
conn.insert_id
The easiest way of all is to wrap your insert with a select count query into a single stored procedure and call that in your code. You would pass in the parameters needed to the stored procedure and it would then select your row count.
Using SQLite3 with Python 2.5, I'm trying to iterate through a list and pull the weight of an item from the database based on the item's name.
I tried using the "?" parameter substitution suggested to prevent SQL injections but it doesn't work. For example, when I use:
for item in self.inventory_names:
self.cursor.execute("SELECT weight FROM Equipment WHERE name = ?", item)
self.cursor.close()
I get the error:
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 8 supplied.
I believe this is somehow caused by the initial creation of the database; the module I made that actually creates the DB does have 8 bindings.
cursor.execute("""CREATE TABLE Equipment
(id INTEGER PRIMARY KEY,
name TEXT,
price INTEGER,
weight REAL,
info TEXT,
ammo_cap INTEGER,
availability_west TEXT,
availability_east TEXT)""")
However, when I use the less-secure "%s" substitution for each item name, it works just fine. Like so:
for item in self.inventory_names:
self.cursor.execute("SELECT weight FROM Equipment WHERE name = '%s'" % item)
self.cursor.close()
I can't figure out why it thinks I have 8 bindins when I'm only calling one. How can I fix it?
The Cursor.execute() method expects a sequence as second parameter. You are supplying a string which happens to be 8 characters long.
Use the following form instead:
self.cursor.execute("SELECT weight FROM Equipment WHERE name = ?", [item])
Python library reference: sqlite3 Cursor Objects.
I have spent half a day trying to figure out why something like this would give me an error:
cursor.execute("SELECT * from ? WHERE name = ?", (table_name, name))
only to find out that table names cannot be parametrized. Hope this will help other people save some time.
The argument of cursor.execute that represents the values you need inserted in the database should be a tuple (sequence). However consider this example and see what's happening:
>>> ('jason')
'jason'
>>> ('jason',)
('jason',)
The first example evaluates to a string instead; so the correct way of representing single valued tuple is as in the second evaluation. Anyhow, the code below to fix your error.
self.cursor.execute("SELECT weight FROM Equipment WHERE name = ?", (item,))
Also giving the cursor.execute value arguments as strings,(which is what you are doing) results in the first evaluation in the example and results into the error you are getting.
The sqlite3 module supports two kinds of placeholders for parameters:
qmark style
Use one or more ? to mark the position of each parameter, and supply a list or tuple of parameters. E.g.:
curs.execute(
"SELECT weight FROM Equipment WHERE name = ? AND price = ?",
["lead", 24],
)
named style
Use :par placeholders for each named parameter, and supply a dict. E.g.:
curs.execute(
"SELECT weight FROM Equipment WHERE name = :name AND price = :price",
{"name": "lead", "price": 24},
)
Advantages of named style parameters is that you don't need to worry about the order of parameters, and each :par can be used multiple times in large/complex SQL queries.
have You tried this ? :
for item in self.inventory_names:
t = (item,)
self.cursor.execute("SELECT weight FROM Equipment WHERE name = ?", t)
self.cursor.close()
cursor.execute() expects a sequence (list,tuple) as second parameter. (-> ddaa )
Quoting (is that what the parens mean?) the ? with parens seems to work for me. I kept trying with (literally) '?' but I kept getting
ProgrammingError: Incorrect number of bindings supplied. The current statement uses 0, and there are 1 supplied.
When I did:
SELECT fact FROM factoids WHERE key LIKE (?)
instead of:
SELECT fact FROM factoids WHERE key LIKE '?'
It worked.
Is this some python 2.6 thing?
each element of items has to be a tuple.
assuming names looks something like this:
names = ['Joe', 'Bob', 'Mary']
you should do the following:
for item in self.inventory_names:
self.cursor.execute("SELECT weight FROM Equipment WHERE name = ?", (item, ))
by using (item, ) you are making it a tuple instead of a string.
Try
execute("select fact from factoids where key like ?", "%%s%" % val)
You don't wrap anything around the ? at all, Python sqlite will correctly convert it into a quoted entity.