In my Python application I have been using sqlite3.Row as the row factory to index results by name for a while with no issues. Recently I moved my application to a new server (no code changes), and I discovered this method of indexing is now unexpectedly failing on the new server given quite a specific condition. I cannot see any explanation for it.
The problem seems to occur on the new server when I have the DISTINCT keyword in my select query:
import sqlite3
conn = sqlite3.connect(':memory:')
conn.row_factory = sqlite3.Row
c = conn.cursor()
c.execute('create table test ([name] text)')
c.execute("insert into test values ('testing')")
conn.commit()
c.execute('select [name] from test')
row = c.fetchone()
print row['name'] # works fine on both machines
c.execute('select distinct [name] from test') # add distinct keyword
row = c.fetchone()
print row['name'] # fails on new server (no item with that key)
As you can see I am able to sandbox this problem using an in-memory database, so the problem is nothing to do with my existing data. Both machines are Debian based (old: Ubuntu 8.10, new: Debian 5.0.3) and both machines are running Python 2.5.2. I believe the sqlite3 module is a core part of the Python install, so I do not know how this subtle breakage can be occurring since the python versions are identical.
Has anyone got any ideas, or seen anything like this before?
Thanks,
Chris
Try adding the line
print row.keys()
instead of "print row['name']" to see what column 0's actual name is in the second case (it's probably altered by the "DISTINCT" keyword).
Alternatively you can use row[0] in this case, but that's most likely not what you want. :)
I had a different, but similar, problem but googling "indexerror no item with that key" led me to this question. In my case the issue was that different sqlite versions appear to handle row key names in row_factory = sqlite3.Row mode differently. In sqlite 3.24.0, a query like:
select table.col
from table
...creates a key in the row dictionary like col. But older versions appear to use the qualified key like table.col. Providing an explicit alias or not qualifying the column is a workaround. e.g:
select table.col as "col"
from table
Or:
select col
from table
Related
I am trying to use the sqlite3 module in python to do a database lookup in a table that only has one column. The column contains phone numbers in the format of:
9545551212
???5551212
Here's what I am running in python:
import sqlite3
cti = '/home/user/test/cti.db'
conn = sqlite3.connect(cti)
c = conn.cursor()
c.execute('select * from ani_table_1 where number = 9545551212')
<sqlite3.Cursor object at 0x7f6b435316c0>
When I run that exact same select statement in sqlite3 I get the expected result:
sqlite> select * from ani_table_1 where number = 9545551212;
9545551212
I'm using python 3.6.5 and sqlite 3.7.17
What have I got wrong in my code? Any help is much appreciated.
You didn't iterate over the result. The sqlite3 command line tool is not the same thing as Python code; the latter always prints the results, because it is a command-line tool and will make sure you don't get flooded with large result sets.
When accessing a database in code, however, the library can't assume you want to print out all the rows to the end user. You maybe wanted to do something different with the data instead.
So you need to loop over the cursor and print each row:
c.execute('select * from ani_table_1 where number = 9545551212')
for row in c:
print(*row, sep='\t')
You may want to familiarise yourself with how the Python database API standard works; search around for a good tutorial. At a glance, this specific tutorial looks like it covers the most important basics.
I'm querying a json on a website for data, then saving that data into a variable so I can put it into a sqlite table. I'm 2 out of 3 for what I'm trying to do, but the sqlite side is just mystifying. I'm able to request the data, from there I can verify that the variable has data when I test it with a print, but all of my sqlite stuff is failing. It's not even creating a table, much less updating the table (but it is printing all the results to the buffer for some reason) Any idea what I'm doing wrong here? Disclaimer: Bit of a python noob. I've successfully created test tables just copying the stuff off of the python sqlite doc
# this is requesting the data and seems to work
for ticket in zenpy.search("bananas"):
id = ticket.id
subj = ticket.subject
created = ticket.created_at
for comment in zenpy.tickets.comments(ticket.id):
body = comment.body
# connecting to sqlite db that exists. things seem to go awry here
conn = sqlite3.connect('example.db')
c = conn.cursor()
# Creating the table table (for some reason table is not being created at all)
c.execute('''CREATE TABLE tickets_test
(ticket id, ticket subject, creation date, body text)''')
# Inserting the variables into the sqlite table
c.execute("INSERT INTO ticketstest VALUES (id, subj, created, body)")
# committing changes the changes and closing
c.commit()
c.close()
I'm on Windows 64bit and using pycharm to do this.
Your table likely isn't created because you haven't committed yet, and your sql fails before it commits. It should work when you fix your 2nd sql statement.
You're not inserting the variables you've created into the table. You need to use parameters. There are two ways of parameterizing your sql statement. I'll show the named placeholders one:
c.execute("INSERT INTO ticketstest VALUES (:id, :subj, :created, :body)",
{'id':id, 'subj':subj, 'created':created, 'body':body}
)
I'm inserting data from some csv files into my SQLite3 database with a python script I wrote. When I run the script, it inserts the first row into the database, but gives this error when trying to inset the second:
sqlite3.IntegrityError: columns column_name1, column_name2 are not unique.
It is true the values in column_name1 and column_name2 are same in the first two rows of the csv file. But, this seems a bit strange to me, because reading about this error indicated that it signifies a uniqueness constraint on one or more of the database's columns. I checked the database details using SQLite Expert Personal, and it does not show any uniqueness constraints on the current table. Also, none of the fields that I am entering specify the primary key. It seems that the database automatically assigns those. Any thoughts on what could be causing this error? Thanks.
import sqlite3
import csv
if __name__ == '__main__' :
conn = sqlite3.connect('ts_database.sqlite3')
c = conn.cursor()
fileName = "file_name.csv"
f = open(fileName)
csv_f = csv.reader(f)
for row in csv_f:
command = "INSERT INTO table_name(column_name1, column_name2, column_name3)"
command += " VALUES (%s, '%s', %s);" % (row[0],row[1],row[2])
print command
c.execute(command)
conn.commit()
f.close()
If SQLite is reporting an IntegrityError error it's very likely that there really is a PRIMARY KEY or UNIQUE KEY on those two columns and that you are mistaken when you state there is not. Ensure that you're really looking at the same instance of the database.
Also, do not write your SQL statement using string interpolation. It's dangerous and also difficult to get correct (as you probably know considering you have single quotes on one of the fields). Using parameterized statements in SQLite is very, very simple.
The error may be due to duplicate column names in the INSERT INTO statement. I am guessing it is a typo and you meant column_name3 in the INSERT INTO statement.
So, I have the following code that inserts the data of an old database to a new one:
...
cur_old.execute("""SELECT DISTINCT module FROM all_students_users_log_course266""")
module_rows = cur_old.fetchall()
for row in module_rows:
cur_new.execute("""INSERT INTO modules(label) SELECT %s WHERE NOT EXISTS (SELECT 1 FROM modules WHERE label=%s)""", (row[0], row[0]))
...
The last line executes a query where labels are inserted into the new database table. I tested this query on pgAdmin and it works as I want.
However, when execute the script, nothing is inserted on the modules table. (Actually the sequences are updated, but none data is stored on the table).
Do I need to do anything else after I call the execute method from the cursor?
(Ps. The script is running till the end without any errors)
You forgot to do connection.commit(). Any alteration in the database has to be followed by a commit on the connection. For example, the sqlite3 documentation states it clearly in the first example:
# Save (commit) the changes.
conn.commit()
And the first example in the psycopg2 documentation does the same:
# Make the changes to the database persistent
>>> conn.commit()
As Evert said, the commit() was missing. An alternative to always specifying it in your code is using the autocommit feature.
http://initd.org/psycopg/docs/connection.html#connection.autocommit
For example like this:
with psycopg2.connect("...") as dbconn:
dbconn.autocommit=True
I am using sqlite with python. When i insert into table A i need to feed it an ID from table B. So what i wanted to do is insert default data into B, grab the id (which is auto increment) and use it in table A. Whats the best way receive the key from the table i just inserted into?
As Christian said, sqlite3_last_insert_rowid() is what you want... but that's the C level API, and you're using the Python DB-API bindings for SQLite.
It looks like the cursor method lastrowid will do what you want (search for 'lastrowid' in the documentation for more information). Insert your row with cursor.execute( ... ), then do something like lastid = cursor.lastrowid to check the last ID inserted.
That you say you need "an" ID worries me, though... it doesn't matter which ID you have? Unless you are using the data just inserted into B for something, in which case you need that row ID, your database structure is seriously screwed up if you just need any old row ID for table B.
Check out sqlite3_last_insert_rowid() -- it's probably what you're looking for:
Each entry in an SQLite table has a
unique 64-bit signed integer key
called the "rowid". The rowid is
always available as an undeclared
column named ROWID, OID, or _ROWID_ as
long as those names are not also used
by explicitly declared columns. If the
table has a column of type INTEGER
PRIMARY KEY then that column is
another alias for the rowid.
This routine returns the rowid of the
most recent successful INSERT into the
database from the database connection
in the first argument. If no
successful INSERTs have ever occurred
on that database connection, zero is
returned.
Hope it helps! (More info on ROWID is available here and here.)
Simply use:
SELECT last_insert_rowid();
However, if you have multiple connections writing to the database, you might not get back the key that you expect.