Question: Is it possible to use a variable as your table name without having to use string constructors to do so?
Info:
I'm working on a project right now that catalogs data from a star simulation of mine. To do so I'm loading all the data into a sqlite database. It's working pretty well, but I've decided to add a lot more flexibility, efficiency, and usability to my db. I plan on later adding planetoids to the simulation, and wanted to have a table for each star. This way I wouldn't have to query a table of 20m some planetoids for the 1-4k in each solar system.
I've been told using string constructors is bad because it leaves me vulnerable to a SQL injection attack. While that isn't a big deal here as I'm the only person with access to these dbs, I would like to follow best practices. And also this way if I do a project with a similar situation where it is open to the public, I know what to do.
Currently I'm doing this:
cursor.execute("CREATE TABLE StarFrame"+self.name+" (etc etc)")
This works, but I would like to do something more like:
cursor.execute("CREATE TABLE StarFrame(?) (etc etc)",self.name)
though I understand that this would probably be impossible. though I would settle for something like
cursor.execute("CREATE TABLE (?) (etc etc)",self.name)
If this is not at all possible, I'll accept that answer, but if anyone knows a way to do this, do tell. :)
I'm coding in python.
Unfortunately, tables can't be the target of parameter substitution (I didn't find any definitive source, but I have seen it on a few web forums).
If you are worried about injection (you probably should be), you can write a function that cleans the string before passing it. Since you are looking for just a table name, you should be safe just accepting alphanumerics, stripping out all punctuation, such as )(][;, and whitespace. Basically, just keep A-Z a-z 0-9.
def scrub(table_name):
return ''.join( chr for chr in table_name if chr.isalnum() )
scrub('); drop tables --') # returns 'droptables'
For people searching for a way to make the table as a variable, I got this from another reply to same question here:
It said the following and it works. It's all quoted from mhawke:
You can't use parameter substitution for the table name. You need to add the table name to the query string yourself. Something like this:
query = 'SELECT * FROM {}'.format(table)
c.execute(query)
One thing to be mindful of is the source of the value for the table name. If that comes from an untrusted source, e.g. a user, then you need to validate the table name to avoid potential SQL injection attacks. One way might be to construct a parameterised query that looks up the table name from the DB catalogue:
import sqlite3
def exists_table(db, name):
query = "SELECT 1 FROM sqlite_master WHERE type='table' and name = ?"
return db.execute(query, (name,)).fetchone() is not None
I wouldn't separate the data into more than one table. If you create an index on the star column, you won't have any problem efficiently accessing the data.
Try with string formatting:
sql_cmd = '''CREATE TABLE {}(id, column1, column2, column2)'''.format(
'table_name')
db.execute(sql_cmd)
Replace 'table_name' with your desire.
To avoid hard-coding table names, I've used:
table = "sometable"
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS {} (
importantdate DATE,
somename VARCHAR,
)'''.format(table))
c.execute('''INSERT INTO {} VALUES (?, ?)'''.format(table),
(datetime.strftime(datetime.today(), "%Y-%m-%d"),
myname))
As has been said in the other answers, "tables can't be the target of parameter substitution" but if you find yourself in a bind where you have no option, here is a method of testing if the table name supplied is valid.
Note: I have made the table name a real pig in an attempt to cover all of the bases.
import sys
import sqlite3
def delim(s):
delims="\"'`"
use_delim = []
for d in delims:
if d not in s:
use_delim.append(d)
return use_delim
db_name = "some.db"
db = sqlite3.connect(db_name)
mycursor = db.cursor()
table = 'so""m ][ `etable'
delimiters = delim(table)
if len(delimiters) < 1:
print "The name of the database will not allow this!"
sys.exit()
use_delimiter = delimiters[0]
print "Using delimiter ", use_delimiter
mycursor.execute('SELECT name FROM sqlite_master where (name = ?)', [table])
row = mycursor.fetchall()
valid_table = False
if row:
print (table,"table name verified")
valid_table = True
else:
print (table,"Table name not in database", db_name)
if valid_table:
try:
mycursor.execute('insert into ' +use_delimiter+ table +use_delimiter+ ' (my_data,my_column_name) values (?,?) ',(1,"Name"));
db.commit()
except Exception as e:
print "Error:", str(e)
try:
mycursor.execute('UPDATE ' +use_delimiter+ table +use_delimiter+ ' set my_column_name = ? where my_data = ?', ["ReNamed",1])
db.commit()
except Exception as e:
print "Error:", str(e)
db.close()
you can use something like this
conn = sqlite3.connect()
createTable = '''CREATE TABLE %s (# );''' %dateNow)
conn.execute(createTable)
basically, if we want to separate the data into several tables according to the date right now, for example, you want to monitor a system based on the date.
createTable = '''CREATE TABLE %s (# );''' %dateNow) means that you create a table with variable dateNow which according to your coding language, you can define dateNow as a variable to retrieve the current date from your coding language.
You can save your query in a .sql or txt file and use the open().replace() method to use variables in any part of your query. Long time reader but first time poster so I apologize if anything is off here.
```SQL in yoursql.sql```
Sel *
From yourdbschema.tablenm
```SQL to run```
tablenm = 'yourtablename'
cur = connect.cursor()
query = cur.execute(open(file = yoursql.sql).read().replace('tablenm',tablenm))
You can pass a string as the SQL command:
import sqlite3
conn = sqlite3.connect('db.db')
c = conn.cursor()
tablename, field_data = 'some_table','some_data'
query = 'SELECT * FROM '+tablename+' WHERE column1=\"'+field_data+"\""
c.execute(query)
Related
im new in programing, starting with python
I found this script in a post (link below)
query = 'SELECT * FROM {}'.format(table)
c.execute(query)
How to use variable for SQLite table name
it worked, but now i have to add some complexity and dont know how...
i had this befour:
def add(order, theme):
parameters = [order, theme]
c.execute("INSERT INTO TABLE_NAME VALUES(NULL, ?,?)", parameters)
so im trying have the same, but with de name of the table free to choose.
trying things like this, but dont really know the syntax for it:
def add(order, theme):
table = input("with what table do you want to work with? ")
parameters = [order, theme]
insert = 'INSERT INTO {} VALUES(NULL, ?,?)'.format(table, parameters)
c.execute(insert)
i suspected it woudnt work Hope you can help! :)
The following line substitutes {} with the contents of the variable table and stores the result in the variable query. format works with any string:
query = 'SELECT * FROM {}'.format(table)
On the other hand the kind of substitution done by c.execute replaces the ? in the string with the values in the list parameters:
c.execute("INSERT INTO TABLE_NAME VALUES(NULL, ?,?)", parameters)
You could combine them both to achieve the effect you want:
table = input("with what table do you want to work with? ")
query = 'INSERT INTO {} VALUES(NULL, ?, ?)'.format(table)
parameters = [order, theme]
c.execute(query, parameters)
You are mixing things.
Format is a method of Python string object used to generate dynamically your string based on variables from your code. In your first example is correct to define the name of the database.
Questions marks as a placeholder for the values of your query is proper of SQL and must be used without format as you did in the second example.
To fix your code, first build the string template of the query using format to define the databasename. Then, execute the resultant string with execute and passing the arguments targeting the question marks.
I have an query string in Python as follows:
query = "select name from company where id = 13 order by name;"
I want to be able to change the id dynamically. Thus I want to find id = 13 and replace it with a new id.
I can do it as follows:
query.replace("id = 13", "id = {}".format(some_new_id))
But if in the query is id= 13 or id=13 or id =13, ... it will not work.
How to avoid that?
Gluing variables directly into your query leaves you vulnerable to SQL injection.
If you are passing your query to a function to be executed in your database, that function should accept additional parameters.
For instance,
query = "select name from company where id = %s order by name"
cursor.execute(query, params=(some_other_id,))
It is better to use formatted sql.
Ex:
query = "select name from company where id = %s order by name;".
cursor.execute(query, (id,))
The usual solution when it comes to dynamically building strings is string formatting, ie
tpl = "Hello {name}, how are you"
for name in ("little", "bobby", "table"):
print(tpl.format(name))
BUT (and that's a BIG "but"): you do NOT want to do this for SQL queries (assuming you want to pass this query to your db using your db's python api).
There are two reasons to not use string formatting here: the first one is that correctly handling quoting and escaping is tricky at best, the second and much more important one is that it makes your code vulnerable to SQL injections attacks.
So in this case, the proper solution is to use prepared statements instead:
# assuming MySQL which uses "%" as placeholder,
# consult your db-api module's documentation for
# the proper placeholder
sql = "select name from company where id=%s order by name"
cursor = yourdbconnection.cursor()
cursor.execute(sql, [your_id_here])
I have a sqlite database that I want to extract specific tables from.
The database has thousands of table names.
I'm interested only in tables that startwith "contact_"
However there are many that are contactOLD, contact#### you name it.
I then need to extract the row data from each "contact_########" table and create a CSV or spreadsheet type document. In total there are 1600 or so with unique names.
I had initially thought I could do this with a sqlite query but could not.
I then tried to write a small script to do this but I could not figure out how to setup conditionals for the cursor.execute to only grab the data from the tables of interest to me.
Any ideas?
Update**
import sqlite3
fname = raw_input("Enter your filename: ")
con = sqlite3.connect(fname)
cursor = con.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='table'")
mydata = cursor.fetchall()
for lines in mydata:
print lines
I have been able to get the tables to list. I still need to add a condition to my WHERE for "contact_" When I add it, I get only [] on my print. I think I'm messing something up here.
****Update 2***
Thanks to #Olver W. below who got me on the right track with this.
fname = raw_input("Enter your filename: ")
con = sqlite3.connect(fname)
cursor = con.cursor()
for tablename in cursor.execute("SELECT name FROM sqlite_master WHERE type='table';"):
if tablename[0].startswith('Contacts_'):
tablename = str(tablename[0])
query = "Select * FROM " + tablename
query = str(query)
data = cursor.execute(query)
for items in data:
print items
I'm going to output this to a spreadsheet in some cases or another SQLite database but in my test database it is selecting the appropriate criteria and outputting the rows correctly. I can and will condense it a bit to make it cleaner but it does the trick. Thanks
As mentioned in the comments, you could first query all the table names, then go through them and check if the name condition is fulfilled, which can be done in Python:
for tablename in cur.execute('SELECT name FROM sqlite_master WHERE type="table";'):
if tablename[0].startswith('conn_'):
execute_some_query_using_this_table()
I'm having problems with a search in a sqlite3's file, I have a database with columns below:
Title,artist,lyrics,tracking
My code is the is that:
def search(title,artist,query):
if title and artist and query:
db = sqlite3.connect('songs.db')
cursor = db.cursor()
cursor.execute('SELECT ? FROM song WHERE title=? and artist=?',(query,title,artist))
result = cursor.fetchall()
if result:
print result
else:
return False
I have set text_factory to str, but it fix another problem that I had, I have checked the database with a viewer like sqliteman, and tried to search through terminal and it works well. ¿Can anyone see what is wrong?
Parameters in SQL queries are always replaced as fixed values.
In other words, strings are always replaced as strings, not as column names.
To customize column names in your query, you must not use SQL parameters but do the replacement in Python:
cursor.execute('SELECT %s FROM song WHERE title = ? AND artist = ?' % (query),
(title, artist))
I have a simple table in mysql with the following fields:
id -- Primary key, int, autoincrement
name -- varchar(50)
description -- varchar(256)
Using MySQLdb, a python module, I want to insert a name and description into the table, and get back the id.
In pseudocode:
db = MySQLdb.connection(...)
queryString = "INSERT into tablename (name, description) VALUES" % (a_name, a_desc);"
db.execute(queryString);
newID = ???
I think it might be
newID = db.insert_id()
Edit by Original Poster
Turns out, in the version of MySQLdb that I am using (1.2.2)
You would do the following:
conn = MySQLdb(host...)
c = conn.cursor()
c.execute("INSERT INTO...")
newID = c.lastrowid
I am leaving this as the correct answer, since it got me pointed in the right direction.
I don't know if there's a MySQLdb specific API for this, but in general you can obtain the last inserted id by SELECTing LAST_INSERT_ID()
It is on a per-connection basis, so you don't risk race conditions if some other client performs an insert as well.
You could also do a
conn.insert_id
The easiest way of all is to wrap your insert with a select count query into a single stored procedure and call that in your code. You would pass in the parameters needed to the stored procedure and it would then select your row count.