I've got a wrapper I wrote around the sqlite3 module that lets me serialize access from multiple threads. It also lets me automatically migrate tables when I change their definition. I noticed when I drop a table and re-add it with more columns, I get an index out of range error. Something like this:
conn = sqlite3.connect("test.db", detect_types=sqlite3.PARSE_COLNAMES)
curs = conn.cursor()
curs.execute("CREATE TABLE test (derp TEXT);"); conn.commit()
curs.execute("INSERT INTO test (derp) VALUES ('deedle');"); conn.commit()
print curs.execute("SELECT * FROM test;").fetchall()
curs.execute("DROP TABLE test;"); conn.commit()
curs.execute("CREATE TABLE test (derp TEXT, val REAL);"); conn.commit()
curs.execute("INSERT INTO test (derp) VALUES ('deedle');"); conn.commit()
print curs.execute("SELECT * FROM test;").fetchall()
conn.close()
Will print this:
[(u'deedle',)] Traceback (most recent
call last): File "test.py", line 23,
in <module>
print curs.execute("SELECT * FROM test;").fetchall() IndexError: list
index out of range
When executing the second SELECT statement. Does anyone know why this is?
Well, it works just fine for me
[(u'deedle',)]
[(u'deedle', None)]
Related
I have a database, and I wish to add multiple values to the same row. I am somewhat new with sqlite and databases, but I am learning. I know I can do this:
conn = sqlite3.connect('sqlitedb.db')
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS threadTable(threadName varchar(30)')
c.execute('INSERT INTO threadTable (threadName) Values(?), x.Name')
This works for me, but I want to pass Multiple variables into the table, like so:
conn = sqlite3.connect('sqlitedb.db')
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS threadTable(threadName varchar(30),threadKey varchar(10),threadID varchar(1000))')
c.execute('INSERT INTO threadTable (threadName),(threadKey),(threadID) VALUES(?,?,?)', (x.Name, x.Key, x.ID))
When I try this, I get this error:
Traceback (most recent call last):
File "Files/main.py", line 39, in <module>
c.execute('INSERT INTO threadTable (threadName),(threadKey),(threadID) VALUES(?,?,?)', (x.Name, x.Key, x.ID))
sqlite3.OperationalError: near ",": syntax error
you don't need parantheses around each column :
c.execute('INSERT INTO threadTable (threadName,threadKey,threadID) VALUES(?,?,?)', (x.Name, x.Key, x.ID))
import sqlite3
def create_table():
connection = sqlite3.connect('lite.db')
cursor = connection.cursor()
cursor.execute('CREATE TABLE IF NOT EXISTS shop (item TEXT, quantity INTEGER, price REAL)') #you write the SQL code in between brackets
connection.commit()
connection.close()
create_table()
def insert(item,quantity,price):
connection = sqlite3.connect('lite.db')
cursor = connection.cursor()
cursor.execute("INSERT INTO shop VALUES (?,?,?)", (item,quantity,price)) # inserting data
connection.commit()
connection.close()
insert('Wine Glass', 10, 5)
insert('Coffe Cup', 5, 2)
insert('Plate', 20, 10)
def view():
connection = sqlite3.connect('lite.db')
cursor = connection.cursor()
cursor.execute('SELECT ALL FROM shop ')
rows = cursor.fetchall()
connection.close()
return rows
def delete_item(item):
connection = sqlite3.connect('lite.db')
cursor = connection.cursor()
cursor.execute("DELETE * FROM shop WHERE item = ?", (item,)) # inserting data
connection.commit()
connection.close()
print(view())
delete_item('Wine Glass')
print(view())
Error Message:
cursor.execute('SELECT ALL FROM shop ')
sqlite3.OperationalError: near "FROM": syntax error
It used to work and then I added the delete function and now it gives me this syntax error, I didn't even make any changes on that function. The code is based on a Udemy tutorial, and with the same changes applied on the video I got this error message but the tutor did not. As you can guess I am pretty new to this stuff and I cant decipher the error message, or at least if it means any more than the obvious. So yeah thanks in advance
SELECT ALL should be SELECT ALL * or just SELECT * to select all columns in all rows. See the syntax here.
DELETE * FROM shop should be DELETE FROM shop. DELETE deletes whole rows, it doesn't need a list of columns. See the syntax here.
I am struggling to establish a connection inside data iteration. Means I am running a select query to postgres and iterating the return data. after some transformation I am writing it to another table. But it is not working. Sample python code is below.
conn = pgconn(------)
cursor = pgconn.Cursor()
query1 = "select * from table"
query2 = "select * from table2 where Id=(%s);"
cursor.execute(query1)
result = query1.fetchall()
for row in result:
If row.a == 2:
cursor.execute(query2, [row.time])
In the above python code I can't able to extract the data by running query2 and passing query1 result as a parameter. It seems cursor is blocked by the query1 so query2 execution is not happening. Please some one help in this issue.
First of all you can write a join statement to do this and can get the data easily
select * from table join table2 where table2.id == table.time
Also why this is not working maybe because the cursor object is getting override inside the for loop and thus the query results get changed.
Use RealDictCursor, and correct the syntax on your inside call to execute():
import psycopg2
import psycopg2.extras
conn = pgconn(------)
cursor = conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
query1 = "select * from table"
query2 = "select * from table2 where Id=(%s);"
cursor.execute(query1)
result = query1.fetchall()
for row in result:
If row.a == 2:
cursor.execute(query2, (row['time'],))
1. install psycopg2 and psycopg2.extras. ( pip install)
Then set up your Postgres Connection like:
def Postgres_init(self):
try:
conn = psycopg2.connect(host=os.environ['SD_POSTGRES_SERVER'],
user=os.environ['SD_POSTGRES_USER'],
password=os.environ['SD_POSTGRES_PASSWORD'],
port=os.environ['SD_POSTGRES_PORT'],
database=os.environ['SD_POSTGRES_DATABASE'])
logging.info("Connected to PostgreSQL")
except (Exception, psycopg2.Error) as error:
logging.info(error)
2. Connect your Cursor with the defined connection
cursor = conn.cursor()
3. Execute your query:
cursor.execute("""SELECT COUNT (column1) from tablename WHERE column2 =%s""", (
Value,)) # Check if already exists
result = cursor.fetchone()
Now the value is stored in the "result" variable. Now you can execute the next query like:
cursor.execute("""
INSERT INTO tablename2
(column1, column2, column3)
VALUES
(%s, %s, %s)
ON CONFLICT(column1) DO UPDATE
SET
column2=excluded.column2,
column3=excluded.column3;
""", (result, column2, column3)
)
Now the result of query 1 is stored in the second table in the first column.
Now you can close your connection:
conn.close()
I am new to coding and databases, I can not get the query to work if I write it long hand but I have a lot to carry out and want it in a function but cannot get it to work, it returns a parameters error
import mysql.connector
def connection_check_1(query, value):
mydb = mysql.connector.connect(
host="******",
user="*****",
passwd="*****",
database="****"
)
mycursor = mydb.cursor()
mycursor.execute(query, (value))
myresult = mycursor.fetchall()
mydb.close()
return myresult
value = "sheep"
query = 'select inlicence from licence where animal = %s'
myresult = connection_check_1(query, value)
print(myresult)
Here is the SQL table I have
create table licence
(
animal varchar (20) primary key,
inlicence int (1)
);
This is the error I get
Traceback (most recent call last):
File "*******************", line 20, in
myresult = connection_check_1(query, value)
File "********************", line 13, in connection_check_1
mycursor.execute(query, (value))
File "********************************************88", line 246, in execute
prepared = self._cnx.prepare_for_mysql(params)
File "/home/kev/PycharmProjects/test bed/venv/lib/python3.5/site-packages/mysql/connector/connection_cext.py", line 535, in prepare_for_mysql
raise ValueError("Could not process parameters")
ValueError: Could not process parameters
I have tried changing the way the query is written, changing it to fetchall().
Wrapping a value with () doesn't turn it in to a tuple. You probably meant to add a comma there:
mycursor.execute(query, (value,))
# Creates a one-element tuple-^
I'm getting this error no matter what with python and sqlite.
File "addbooks.py", line 77, in saveBook
conn.commit()
sqlite3.OperationalError: cannot commit transaction - SQL statements in progress
The code looks like this:
conn = sqlite3.connect(fname)
cread = conn.cursor()
cread.execute('''select book_text from table''')
while True:
row = cread.fetchone()
if row is None:
break
....
for entry in getEntries(doc):
saveBook(entry, conn)
Can't do a fetchall() because table and column size are big, and the memory is scarce.
What can be done without resorting to dirty tricks(as getting the rowids in memory, which would probably fit, and then selecting the rows one by one)?.
The problem is that you've left the connection in auto-commit mode. Wrap a single transaction around the whole lot so that a commit only happens after you've done all the updates, and it should all work fine.
Don't know if this count as "dirty tricks" too ;-)
My solution to this problem is to use SELECT... LIMIT clause, assumed you have primary key integer field id
current_id = 0
while True:
cread.execute('''select book_text from table where id > %s limit 2''' % current_id)
results = cread.fetchall()
if results is None:
break;
for row in results:
... (save book) ...
current_id = row.id
The problem is that there should be no more than a single active cursor for a connection.
The solution is to use a new connection for the updates.
Unfortunatelly I do not remember the exact place in docs where I read it, so I can not prove it.
UPD:
The following code works on my Windows XP:
import sqlite3
import os
conn1 = sqlite3.connect('test.db')
cursor1 = conn1.cursor()
conn2 = sqlite3.connect('test.db')
cursor2 = conn2.cursor()
cursor1.execute("CREATE TABLE my_table (a INT, b TEXT)")
cursor1.executemany("INSERT INTO my_table (a, b) VALUES (?, NULL);", zip(range(5)))
conn1.commit()
cursor1.execute("SELECT * FROM my_table")
for a, b in cursor1:
cursor2.execute("UPDATE my_table SET b='updated' WHERE a = ?", (a, ))
conn2.commit()
print "results:"
print 10 * '-'
cursor1.execute("SELECT * FROM my_table")
for a, b in cursor1:
print a, b
cursor1.close()
conn1.close()
cursor2.close()
conn2.close()
os.unlink('test.db')
And returns the following as expected:
results:
----------
0 updated
1 updated
2 updated
3 updated
4 updated
If I move the conn2.commit() into the for loop, I get the same error as you mention:
Traceback (most recent call last):
File "concurent.py", line 16, in <module>
conn2.commit()
sqlite3.OperationalError: database is locked
Thus, the solution is to commit once at the end instead of committing after each line.