I have three python lists with about 1.5 million entries each and would like to insert these into a new SQLite table. When doing this I get the error:
OperationalError: no such column: days
This is the code I have:
con = sqlite3.connect('database.db')
cur = con.cursor()
...
cur.execute("DROP TABLE IF EXISTS days")
cur.execute("CREATE TABLE IF NOT EXISTS days(DAYS_NEEDED integer, RAISED_TIME text, POSTED_TIME text)")
cur.execute("INSERT INTO days (DAYS_NEEDED, RAISED_TIME, POSTED_TIME) VALUES (days, rt_list, pt_list)")
con.commit()
"days" is a list of integers, rt_list and pt_list are both lists of strings. Does anyone know what I'm doing wrong here?
Any help is much appreciated!
That's not the way you can insert list of values in SQL. First, you must give a valid SQL instruction using ? as placeholders. Then if you want to insert more than one row at a time, you will need the executemany method. It is a true improvement because the SQL in only parsed and prepared once.
So you should have written:
cur.execute("DROP TABLE IF EXISTS days")
cur.execute("CREATE TABLE IF NOT EXISTS days(DAYS_NEEDED integer, RAISED_TIME text, POSTED_TIME text)")
cur.executemany("INSERT INTO days (DAYS_NEEDED, RAISED_TIME, POSTED_TIME) VALUES (?,?,?)",
zip(days, rt_list, pt_list))
con.commit()
BTW, the direct usage of zip is a Sqlite3 module extension, the DB-API 2.0 Python interface normally requires a sequence where zip returns an iterator, so the more portable way (any DB engine) would be:
cur.executemany("INSERT INTO days (DAYS_NEEDED, RAISED_TIME, POSTED_TIME) VALUES (?,?,?)",
tuple(zip(days, rt_list, pt_list)))
You have to use ? placeholders inside your VALUES() and then provide the actual values to the execute method.
Something along the lines should do the job:
con = sqlite3.connect('database.db')
cur = con.cursor()
...
cur.execute("DROP TABLE IF EXISTS days")
cur.execute("CREATE TABLE IF NOT EXISTS days(DAYS_NEEDED integer, RAISED_TIME text, POSTED_TIME text)")
def insert(days_needed, rt, pt):
cur.execute("INSERT INTO days (DAYS_NEEDED, RAISED_TIME, POSTED_TIME) VALUES (?, ?, ?)", (days_needed, rt, pt))
for d, rt, pt in zip(days, rt_list, pt_list):
insert(d, rt, pt)
con.commit()
Related
I am struggling with generating the delete query where parameters for the query is actually a set of values.
So I need to delete rows where parameters are a pair values for example:
delete from table where col1 = %s and col2 = %s
which can be executed in Python like:
cur = conn.cursor()
cur.execute(query, (col1_value, col2_value))
Now I would like to run a query:
delete from table where (col1, col2) in ( (col1_value1, col2_value1), (col1_value2, col2_value2) );
I can generate the queries and values and execute the exact SQL but I can't quite generate prepared statement.
I tried:
delete from table where (col1, col2) in %s
and
delete from table where (col1, col2) in (%s)
But when I try to execute:
cur.execute(query, list_of_col_value_tuples)
or
cur.execute(query, tuple_of_col_value_tuples)
I get an exception that indicates that psycopg2 cannot convert arguments to strings.
Is there any way to use psycopg2 to execute a query like this?
You could dynamically add %s placeholders to your query:
cur = con.cursor()
query = "delete from table where (role, username) in (%s)"
options = [('admin', 'foo'), ('user', 'bar')]
placeholders = '%s,' * len(options)
query = query % placeholders[:-1] # remove last comma
print(query)
print(cur.mogrify(query, options).decode('utf-8'))
Out:
delete from table where (role, user) in (%s,%s)
delete from table where (role, user) in (('admin', 'foo'),('user', 'bar'))
Alternatively, build the query using psycopg2.sql as answered there.
Actually the resolution is quite easy if carefully constructed.
In the miscellaneous goodies of psycopg2 there is a function execute_values.
While all the examples that are given by psycopg2 deal with inserts as the function basically converts the list of arguments into a VALUES list if the call to delete is formatted like so:
qry = "delete from table where (col1, col2) in (%s)"
The call:
execute_values(cur=cur, qry=qry, argslist=<list of value tuples>)
will make the delete perform exactly as required.
Age and phone_num are int values rest all are strings. when trying to push this into a DB using the code below am getting the following error
insert_query = "insert into employee.details (name,emp_id,age,contact,address) values('"+name+"','"+emp_id+"',"+age+","+phone_num+",'"+address+"')"
cursor = connection.cursor
result = cursor.execute(insert_query)
print("Table updated successfully ")
I think you were getting this error because python cannot combine integers and strings unless they are explicitly converted using str()
I assume you are using SQLite3? If so here is the proper syntax for a query.
insert_query = """INSERT INTO employee.details (name, emp_id, age, contact, address) VALUES (?, ?, ?, ?, ?)"""
cur = conn.cursor()
cur.execute(insert_query, (name, emp_id, age, phone_num, address))
one_row = cur.fetchone() # This will only get one row of the data
all_data = cur.fetchall() # This will get all rows of data in a list of tuples
conn.commit()
conn.close() # only if this is last db change
Templating into your query using a tuple will automatically escape strings and prevent SQL injection. It will also convert your integers to strings, fixing your error.
Why my code only inserts one line?
thewholeenchilada = ("SELECT SUBSTR(email, (SELECT INSTR(email,'#'))) AS org, SUM(count) as count FROM Em GROUP BY org ORDER BY count DESC")
for salida in cur.execute(thewholeenchilada):
cur.execute('''INSERT INTO Counts (org, count)
VALUES (?, ?)''', (salida[0],row[1]))
print((str(salida[0]), salida[1]))
conn.commit()
Avoid the loop and run one INSERT INTO ... SELECT query. Right now you re-use same cursor outside and inside loop causing issues with processing. Either use two different cursors or efficiently combine and have database engine run action query:
sql = '''INSERT INTO Counts (org, [count])
SELECT SUBSTR(email, INSTR(email, '#')+1) AS org,
SUM(count) as [count]
FROM Em
GROUP BY org
ORDER BY count DESC
'''
cur.execute(sql)
conn.commit()
What is the most used way to create a Sqlite query in Python?
query = 'insert into events (date, title, col3, col4, int5, int6)
values("%s", "%s", "%s", "%s", %s, %s)' % (date, title, col3, col4, int5, int6)
print query
c.execute(query)
Problem: it won't work for example if title contains a quote ".
query = 'insert into events (date, title, col3, col4, int5, int6)
values(?, ?, ?, ?, ?, ?)'
c.execute(query, (date, title, col3, col4, int5, int6))
Problem: in solution 1., we could display/print the query (to log it); here in solution 2. we can't log the query string anymore because the "replace" of each ? by a variable is done during the execute.
Another cleaner way to do it? Can we avoid to repeat ?, ?, ?, ..., ? and have one single values(?) and still have it replaced by all the parameters in the tuple?
You should always use parameter substitution of DB API, to avoid SQL injection, query logging is relatively trivial by subclassing sqlite3.Cursor:
import sqlite3
class MyConnection(sqlite3.Connection):
def cursor(self):
return super().cursor(MyCursor)
class MyCursor(sqlite3.Cursor):
def execute(self, sql, parameters=''):
print(f'statement: {sql!r}, parameters: {parameters!r}')
return super().execute(sql, parameters)
conn = sqlite3.connect(':memory:', timeout=60, factory=MyConnection)
conn.execute('create table if not exists "test" (id integer, value integer)')
conn.execute('insert into test values (?, ?)', (1, 0));
conn.commit()
yields:
statement: 'create table if not exists "test" (id integer, value integer)', parameters: ''
statement: 'insert into test values (?, ?)', parameters: (1, 0)
To avoid formatting problems and SQL injection attacks, you should always use parameters.
When you want to log the query, you can simply log the parameter list together with the query string.
(SQLite has a function to get the expanded query, but Python does not expose it.)
Each parameter markers corresponds to exactly one value. If writing many markers is too tedious for you, let the computer do it:
parms = (1, 2, 3)
markers = ",".join("?" * len(parms))
I am trying to insert values into database table using Python. I have connected to the database server and have gotten the INSERT INTO sql statement to work, however I cannot figure out how to insert values from a list into my database. I think my problem is simple, and just involves using the for-in loop correctly, but I do not know how to fix it.
The line I need help with is "for i in cur:
cur.execute("INSERT INTO Events2013 VALUES (i))")"
Here is part of my code:
import cx_Oracle
import fileDb
import g
fileDb.loadTeams()
fileDb.loadEvents()
for event in g.eventList:
print '%s, %s, %s' % (event.eventName, event.eventType, event.dates)
dsn_tns = cx_Oracle.makedsn('hostname', 1521, 'orcl')
con = cx_Oracle.connect('std15', 'std15', dsn_tns)
cur = con.cursor()
for i in cur:
cur.execute("INSERT INTO Events2013 VALUES (i))")
cur.execute("SELECT * FROM Events2013")
for result in cur:
print result
cur.execute("COMMIT")
cur.close()
con.close()
cur.execute("INSERT INTO Events2013 VALUES (i))")
that i doesn't automatically expand to the variables (it's passed as a literal i)
I imagine you need something like
for event in g.eventList:
cur.execute("INSERT INTO Events2013 VALUES ('%s','%s','%s')" % (event.eventName, event.eventType, event.dates))