I'm trying to insert a record into an sqlite database using named parameters in python (with the sqlite3 module).
The values I want to insert are in a dictionary, but the dictionary keys might contain dashes, for example {'request-id': 100, 'year': '2015'}.
I'm trying to execute the following:
import sqlite3
conn = sqlite3.connect('database.db')
cursor = conn.cursor()
cursor.execute('''CREATE TABLE IF NOT EXISTS requests (request_id text, year text)''')
query = '''INSERT INTO requests (request_id, year) VALUES (:request-id, :year)'''
cursor.execute(query, {'request-id': 100, 'year': '2015'})
conn.commit()
conn.close()
I get this error during the insert statement:
sqlite3.OperationalError: no such column: id
It seems like dashes are not well accepted as named parameters.
There are many workarounds for this, like creating a new dictionary where dashes in the keys are replaced by underscores, but I'd like to know if I could use some escaping technique or something else to avoid that.
Thanks for your help
The documentation for sqlite3_bind_* states that parameter names must be composed of alphanumeric characters, and doesn't mention a way of escaping them.
Your query is probably being parsed as :request - id, i.e. :request minus id, and since there's no such column id, SQLite throws an error.
(Also, as Prerak Sola points out, you create the table with a date column but try to insert to a year column which doesn't exist.)
SQL parameter names have no quoting or escaping mechanism; you have to use the same rules as for an unquoted identifier.
Related
When I'm trying to format an SQL statement this way:
cursor.execute('SELECT (%s) FROM table WHERE id = 12345', (column,))
it doesn't work properly. Instead of returning values from the specified column, it just returns the name of the column. Any way to fix that?
It only works with formatted string, but according to reviews it is not the safest approach
You can't bind column names like that -you're binding a string literal with that name.
If you want to dynamically set the column names, you'll have to resort to string manipulation of some sort, e.g.:
cursor.execute(f'SELECT {column} FROM table WHERE id = 12345')
Of course if column is created by user-input, you'll have to sanitize it somehow.
I keep getting an OperationalError: Unrecognized Token. The error hapens when I'm attempting to insert data into my SQLite database using an SQLite Insert command. What do I need to do to correct this error or is there a better way I should go about inserting data into my database? The data is water level data measured in meters above chart datum and is gathered from water level gauge data loggers throughout the Great Lakes region of Canada and the US. The script uses the Pandas library and is hardcoded to merge data from water level gauging stations that are located in close proximity to each other. I'd like to use the insert command so I can deal with overlapping data when adding future data to the database. I won't even begin to pretend I know what I'm talking about with databases and programming so any help would be appreciated in how I can solve this error!
I've tried altering my script in the parameterized query to try and solve the problem without any luck as my research has said this is the likely culprit
# Tecumseh. Merges station in steps due to inability of operation to merge all stations at once. Starts by merging PCWL station to hydromet station followed by remaining PCWL station and 3 minute time series
final11975 = pd.merge(hydrometDF["Station11975"], pcwlDF["station11995"], how='outer', left_index=True,right_index=True)
final11975 = pd.merge(final11975, pcwlDF["station11965"], how='outer', left_index=True,right_index=True)
final11975 = pd.merge(final11975, cts, how='outer', left_index=True,right_index=True)
final11975.to_excel("C:/Users/Andrew/Documents/CHS/SeasonalGaugeAnalysis_v2/SeasonalGaugeAnalysis/Output/11975_Tecumseh.xlsx")
print "-------------------------------"
print "11975 - Tecumseh"
print(final11975.info())
final11975.index = final11975.index.astype(str)
#final11975.to_sql('11975_Tecumseh', conn, if_exists='replace', index=True)
#Insert and Ignore data into database to eliminate overlaps
testvalues = (final11975.index, final11975.iloc[:,0], final11975.iloc[:,1], final11975.iloc[:,2])
c.execute("INSERT OR IGNORE INTO 11975_Tecumseh(index,11975_VegaRadar(m),11995.11965), testvalues")
conn.commit()
I'd like the data to insert into the database using the Insert And Ignore command as data is often overlapping when its downloaded. I'm new to databases but I'm under the impression that the Insert and Ignore command will illiminate overlapping data. The message I receive when running my script is:
</> <Exception has occurred: OperationalError
unrecognized token: "11975_Tecumseh"
File "C:\Users\Documents\CHS\SeasonalGaugeAnalysis_v2\SeasonalGaugeAnalysis\Script\CombineStations.py", line 43, in <module>>
c.execute("INSERT OR IGNORE INTO 11975_Tecumseh(index,11975_VegaRadar(m),11995.11965), testvalues") </>
As per SQL Standards, You can create table or column name such as "11975_Tecumseh" and also Tecumseh_11975, but cannot create table or column name begin with numeric without use of double quotes.
c.execute("INSERT OR IGNORE INTO '11975_Tecumseh'(index,'11975_VegaRadar(m)',11995.11965), testvalues")
The error you are getting is because the table name 11975_Tecumseh is invalid as it stands as it is not suitably enclosed.
If you want to use a keyword as a name, you need to quote it. There
are four ways of quoting keywords in SQLite:
'keyword' A keyword in single quotes is a string literal.
"keyword" A keyword in double-quotes is an identifier. [keyword] A
keyword enclosed in square brackets is an identifier.
This is not
standard SQL. This quoting mechanism is used by MS Access and SQL
Server and is included in SQLite for compatibility. keyword A
keyword enclosed in grave accents (ASCII code 96) is an identifier.
This is not standard SQL. This quoting mechanism is used by MySQL and
is included in SQLite for compatibility. For resilience when
confronted with historical SQL statements, SQLite will sometimes bend
the quoting rules above:
If a keyword in single quotes (ex: 'key' or 'glob') is used in a
context where an identifier is allowed but where a string literal is
not allowed, then the token is understood to be an identifier instead
of a string literal.
If a keyword in double quotes (ex: "key" or "glob") is used in a
context where it cannot be resolved to an identifier but where a
string literal is allowed, then the token is understood to be a string
literal instead of an identifier.
Programmers are cautioned not to use the two exceptions described in
the previous bullets. We emphasize that they exist only so that old
and ill-formed SQL statements will run correctly. Future versions of
SQLite might raise errors instead of accepting the malformed
statements covered by the exceptions above.
SQL As Understood By SQLite - SQLite Keywords
The above is applied to invalid names, which includes names that start with numbers and names that include a non numeric inside parenthesises.
If 11975_Tecumseh is the actual table name then it must be enclosed e.g. [11975_Tecumseh]
Likewise the columns
index
11975_VegaRadar(m)
and 11995.11965
Also have to be suitably enclosed.
Doing so you'd end up with
"INSERT OR IGNORE INTO [11975_Tecumseh]([index],[11975_VegaRadar(m)],[11995.11965]), testvalues"
The the issues is that ,testvalues is syntactically incorrect. after the columns to insert into i.e. ([index],[11975_VegaRadar(m)],[11995.11965]) the keyword VALUES with the three values should be used.
An example of a valid statement is :
"INSERT INTO [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965]) VALUES('value1','value2','value3')"
As such
c.execute("INSERT INTO [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965]) VALUES('value1','value2','value3')")
would insert a new row (unless a constrain conflict occurred)
However, I suspect that you want to insert values according to variables in which case you could use:
"INSERT INTO [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965]) VALUES(?,?,?)"
the question marks being place-holders/bind values
SQL As Understood By SQLite- INSERT
The above would then be invoked using :
c.execute("INSERT INTO [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965]) VALUES(?,?,?)",testvalues);
#Working Example :
import sqlite3
drop_sql = "DROP TABLE IF EXISTS [11975_Tecumseh]"
crt_sql = "CREATE TABLE IF NOT EXISTS [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965])"
testvalues = ("X","Y","Z")
c = sqlite3.connect("test.db")
c.execute(drop_sql)
c.execute(crt_sql)
insert_sql1 = "INSERT INTO [11975_Tecumseh] " \
"([index],[11975_VegaRadar(m)],[11995.11965]) " \
"VALUES('value1','value2','value3')"
c.execute(insert_sql1)
insert_sql2 = "INSERT OR IGNORE INTO '11975_Tecumseh'" \
"('index','11975_VegaRadar(m)',[11995.11965])" \
" VALUES(?,?,?)"
c.execute(insert_sql2,(testvalues))
cursor = c.cursor()
cursor.execute("SELECT * FROM [11975_Tecumseh]")
for row in cursor:
print(row[0], "\n" + row[1], "\n" + row[2])
c.commit()
cursor.close()
c.close()
#Result
##Row 1
value1
value2
value3
##Row 2
X
Y
Z
I am trying to load the data inside the table trial and it says Invalid Column name - Name.
I am passing values inside Name and Area dynamically.
cursor.execute("insert into trial (NameofTheProperty, AreaofTheProperty)
values (Name, Area)")
cnxn.commit()
You need to have quotes around the column values so that they are not gonna be interpreted as column names instead:
insert into
trial (NameofTheProperty, AreaofTheProperty)
values
("Name", "Area")
Now, since you mentioned that you dynamically insert these values into the query, you can just let your database driver handle the quotes and other things like type conversions:
property_name = "Name"
property_area = "Area"
cursor.execute("""
insert into
trial (NameofTheProperty, AreaofTheProperty)
values
(?, ?)""", (property_name, property_area))
cnxn.commit()
This is called query parameterization and is considered the safest and the most robust way to insert values into the SQL queries. These ? values are called "placeholders".
Note that the database driver is gonna put quotes around the string values automatically - no need to do it manually.
I am trying to search my SQLite3 database using a pythonic variable as a search term. The term I'm searching for is a part of the contents of the cell in the database (e.g. Smith in a cell: [Harrison GB, Smith JH]) and is often in the middle of the string in a cell.
I have tried to code it as shown below:
def read_from_db():
c.execute("SELECT authors, year, title, abstract FROM usertable WHERE authors LIKE (?)",(var1,))
data = c.fetchall()
print(data)
for row in data:
searchlist.append(row)
var1="Smith"
read_from_db()
This should show the results row after row. However, I get 0 results when var1 = "Smith". When I change its value to "Harrison GB, Smith JH", I get all the results.
When I try to solve it by changing the SQLite3 execute query I yield an error.
ERROR
c.execute("SELECT authors, year, title, abstract FROM usertable WHERE authors LIKE '%?%'",(var1,))
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 0, and there are 1 supplied.
I get syntax errors if I change the endings with: $?$, (%?%) etc. I tried this with:
...authors="%?%"
But this doesn't work either. There is a few similar questions on SO, but they don't exactly solve my issue...
I am running Python 3.4 and SQLite3 on Windows 10.
Consider concatenating the % wildcards to the binded value, var1:
c.execute("SELECT authors, year, title, abstract" +
" FROM usertable WHERE authors LIKE (?)", ('%'+var1+'%',))
The reason you need to do so is the ? placeholder substitutes a string literal in parameterized queries and for LIKE expressions, wildcards with values together are string literals as denoted by their enclosed single quotes:
SELECT authors, year, title, abstract FROM usertable WHERE authors LIKE '%Smith%'
Your initial attempt failed because you wrap ? with single quotes and the cursor cannot bind the param to prepared statement properly and so the symbol is taken literally as question mark.
I'm trying to insert strings read from a file into an sqlite database in Python. The strings have whitespace (newline, tab characters, and spaces) and also have appearances of single or double quotes. Here's how I try to do it:
import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()
# Create table
c.execute('''CREATE TABLE test
(a text, b text)''')
f = open("foo", "w")
f.write("hello\n\'world\'\n")
f.close()
testfield = open("foo").read()
# Insert a row of data
c.execute("INSERT INTO test VALUES ('%s', 'bar')" %(testfield))
# Save (commit) the changes
conn.commit()
I find that this fails with the error:
c.execute("INSERT INTO test VALUES ('%s', 'bar')" %(testfield))
sqlite3.OperationalError: near "world": syntax error
How can I achieve this? Do the strings need to be escaped before insertion in the db, and if so how? thanks.
You use SQL parameters instead of string formatting:
c.execute("INSERT INTO test VALUES (?, 'bar')", (testfield,))
When using SQL parameters you let the database library handle the quoting, and even better, give the database to optimize the query and reuse the optimized query plan for multiple executions of the same basic query (with different parameters).
Last but not least, you are much better defended against SQL injection attacks as the database library knows best how to escape dangerous SQL-like values.
To quote the sqlite3 documentation:
Usually your SQL operations will need to use values from Python variables. You shouldn’t assemble your query using Python’s string operations because doing so is insecure; it makes your program vulnerable to an SQL injection attack (see http://xkcd.com/327/ for humorous example of what can go wrong).
Instead, use the DB-API’s parameter substitution. Put ? as a placeholder wherever you want to use a value, and then provide a tuple of values as the second argument to the cursor’s execute() method.