I am trying to use the sqlite3 module in python to do a database lookup in a table that only has one column. The column contains phone numbers in the format of:
9545551212
???5551212
Here's what I am running in python:
import sqlite3
cti = '/home/user/test/cti.db'
conn = sqlite3.connect(cti)
c = conn.cursor()
c.execute('select * from ani_table_1 where number = 9545551212')
<sqlite3.Cursor object at 0x7f6b435316c0>
When I run that exact same select statement in sqlite3 I get the expected result:
sqlite> select * from ani_table_1 where number = 9545551212;
9545551212
I'm using python 3.6.5 and sqlite 3.7.17
What have I got wrong in my code? Any help is much appreciated.
You didn't iterate over the result. The sqlite3 command line tool is not the same thing as Python code; the latter always prints the results, because it is a command-line tool and will make sure you don't get flooded with large result sets.
When accessing a database in code, however, the library can't assume you want to print out all the rows to the end user. You maybe wanted to do something different with the data instead.
So you need to loop over the cursor and print each row:
c.execute('select * from ani_table_1 where number = 9545551212')
for row in c:
print(*row, sep='\t')
You may want to familiarise yourself with how the Python database API standard works; search around for a good tutorial. At a glance, this specific tutorial looks like it covers the most important basics.
Related
So I am trying to do an IF statement after the select is done in python. Hard to explain, better to provide an example of it:
customer = cursor.execute("SELECT * FROM customer")
try:
customer['name']="john"
(do something here)
except:
customer['name']!="john"
(do something else here)
I hope this makes sense and sorry if this is not enough information. Trying to think of how to explain this. I don't want it to do a WHERE statement in the select because I don't want it to NOT SELECT certain information just because their 'name' is not "john"
First of all, you are not telling us which library you are using for working with MySQL. I assume its MySQLdb.
After executing SQL query with cursor.execute you have to call fetchall or fetchone. In later case you will be able to do customer['name'].
Example:
results = cursor.execute("SELECT * FROM customers")
customer = cursor.fetchone()
You shouldn't be using try/except since there is not an obvious case when any exception is thrown or is expected.
Using:
Python3
SQLite
TKinter
I am currently trying to create a function to search for a keyword in a database, but as soon as I try to combine it with TKinter, it seems to fail.
Here are the relevant lines:
(I tried it in a lot of different ways, those 3 lines below seem to work with variables, but not with the input from TKinter, so I thought they might actually work, if I edit them a little.
The problem I got is, that I'm not experienced in TKinter and SQLite yet and worked with those 2 for about 3 days yet.
def searcher(column):
#Getting input from user (TKinter)
keyword = tk.Entry(self)
keyword.pack()
#Assigning the input to a variable
kword = keyword.get()
c.execute("SELECT * FROM my_lib WHERE {kappa} LIKE {%goal%}".format(kappa=column, goal=kword))
#c.execute("SELECT * FROM my_lib WHERE "+column+"=?", (kword,))
#c.execute("SELECT * FROM my_lib WHERE {} LIKE '%kword%'".format(column))
I want to check if any of the data CONTAINS the keyword, so basically:
k_word in column_data
and not
column_data == k_word
My question is:
Is there a way to take the user input (by TKinter) and search in the database (SQLite) and check, if any data in the database contains the keyword.
The SQLite docs explain that you can use ? as a placeholder in the query string, which allows you so substitute in a tuple of values. They also advise against ever assembling a full query using variables with Python's string operations (explained below):
c.execute("SELECT * FROM my_lib WHERE ? LIKE ?", (column, '%'+kword+'%'))
You can see above that I concatenated the % with kword, which will get substituted into the second ?. This also is secure, meaning it will protect against SQL Injection attacks if needed.
Docs: https://docs.python.org/2/library/sqlite3.html
Related posts: Escaping chars in Python and sqlite
Try:
kword = "%"+kword+"%"
c.execute("SELECT * FROM my_lib WHERE kappa LIKE '%s'" % kword)
After trying over and over again, I got the actual solution. It seems like I can't just add the '%' to the variable like a string, but rather:
c.execute("SELECT * FROM my_lib WHERE {} LIKE '%{}%'".format(column, kword))
In my Python application I have been using sqlite3.Row as the row factory to index results by name for a while with no issues. Recently I moved my application to a new server (no code changes), and I discovered this method of indexing is now unexpectedly failing on the new server given quite a specific condition. I cannot see any explanation for it.
The problem seems to occur on the new server when I have the DISTINCT keyword in my select query:
import sqlite3
conn = sqlite3.connect(':memory:')
conn.row_factory = sqlite3.Row
c = conn.cursor()
c.execute('create table test ([name] text)')
c.execute("insert into test values ('testing')")
conn.commit()
c.execute('select [name] from test')
row = c.fetchone()
print row['name'] # works fine on both machines
c.execute('select distinct [name] from test') # add distinct keyword
row = c.fetchone()
print row['name'] # fails on new server (no item with that key)
As you can see I am able to sandbox this problem using an in-memory database, so the problem is nothing to do with my existing data. Both machines are Debian based (old: Ubuntu 8.10, new: Debian 5.0.3) and both machines are running Python 2.5.2. I believe the sqlite3 module is a core part of the Python install, so I do not know how this subtle breakage can be occurring since the python versions are identical.
Has anyone got any ideas, or seen anything like this before?
Thanks,
Chris
Try adding the line
print row.keys()
instead of "print row['name']" to see what column 0's actual name is in the second case (it's probably altered by the "DISTINCT" keyword).
Alternatively you can use row[0] in this case, but that's most likely not what you want. :)
I had a different, but similar, problem but googling "indexerror no item with that key" led me to this question. In my case the issue was that different sqlite versions appear to handle row key names in row_factory = sqlite3.Row mode differently. In sqlite 3.24.0, a query like:
select table.col
from table
...creates a key in the row dictionary like col. But older versions appear to use the qualified key like table.col. Providing an explicit alias or not qualifying the column is a workaround. e.g:
select table.col as "col"
from table
Or:
select col
from table
I am using pythons built-in sqlite3 module to access a database. My query executes a join between a table of 150000 entries and a table of 40000 entries, the result contains about 150000 entries again. If I execute the query in the SQLite Manager it takes a few seconds, but if I execute the same query from Python, it has not finished after a minute. Here is the code I use:
cursor = self._connection.cursor()
annotationList = cursor.execute("SELECT PrimaryId, GOId " +
"FROM Proteins, Annotations " +
"WHERE Proteins.Id = Annotations.ProteinId")
annotations = defaultdict(list)
for protein, goterm in annotationList:
annotations[protein].append(goterm)
I did the fetchall just to measure the execution time. Does anyone have an explanation for the huge difference in performance? I am using Python 2.6.1 on Mac OS X 10.6.4.
I implemented the join manually, and this works much faster. The code looks like this:
cursor = self._connection.cursor()
proteinList = cursor.execute("SELECT Id, PrimaryId FROM Proteins ").fetchall()
annotationList = cursor.execute("SELECT ProteinId, GOId FROM Annotations").fetchall()
proteins = dict(proteinList)
annotations = defaultdict(list)
for protein, goterm in annotationList:
annotations[proteins[protein]].append(goterm)
So when I fetch the tables myself and then do the join in Python, it takes about 2 seconds. The code above takes forever. Am I missing something here?
I tried the same with apsw, and it works just fine (the code does not need to be changed at all), the performance it great. I'm still wondering why this is so slow with the sqlite3-module.
There is a discussion about it here: http://www.mail-archive.com/python-list#python.org/msg253067.html
It seems that there is a performance bottleneck in the sqlite3 module. There is an advice how to make your queries faster:
make sure that you do have indices on the join columns
use pysqlite
You haven't posted the schema of the tables in question, but I think there might be a problem with indexes, specifically not having an index on Proteins.Id or Annotations.ProteinId (or both).
Create the SQLite indexes like this
CREATE INDEX IF NOT EXISTS index_Proteins_Id ON Proteins (Id)
CREATE INDEX IF NOT EXISTS index_Annotations_ProteinId ON Annotations (ProteinId)
I wanted to update this because I am noticing the same issue and we are now 2022...
In my own application I am using python3 and sqlite3 to do some data wrangling on large databases (>100000 rows * >200 columns). In particular, I have noticed that my 3 table inner join clocks in around ~12 minutes of run time in python, whereas running the same join query in sqlite3 from the CLI runs in ~100 seconds. All the join predicates are properly indexed and the EXPLAIN QUERY PLAN indicates that the added time is most likely because I am using SELECT *, which is a necessary evil in my particular context.
The performance discrepancy caused me to pull my hair out all night until I realized there is a quick fix from here: Running a Sqlite3 Script from Command Line. This is definitely a workaround at best, but I have research due so this is my fix.
Write out the query to an .sql file (I am using f-strings to pass variables in so I used an example with {foo} here)
fi = open("filename.sql", "w")
fi.write(f"CREATE TABLE {Foo} AS SELECT * FROM Table1 INNER JOIN Table2 ON Table2.KeyColumn = Table1.KeyColumn INNER JOIN Table3 ON Table3.KeyColumn = Table1.KeyColumn;")
fi.close()
Run os.system from inside python and send the .sql file to sqlite3
os.system(f"sqlite3 {database} < filename.sql")
Make sure you close any open connection before running this so you don't end up locked out and you'll have to re-instantiate any connection objects afterward if you're going back to working in sqlite within python.
Hope this helps and if anyone has figured the source of this out, please link to it!
I am trying to write a simple query to an sqlite database in a python script. To test if my parameters were correct, I tried running the query from the ipython command line. It looked something like this:
import sqlite3
db = 'G:\path\to\db\file.sqlite'
conn = sqlite3.connect(db)
results = conn.execute('SELECT * FROM studies').fetchall()
for some reason, my results came back totally empty. Then I tried another test query:
results = conn.execute('SELECT id FROM studies').fetchall()
Which returned correctly. I figured there was a problem with the asterisk [WRONG, SEE SECOND UPDATE BELOW], so I tried the 'SELECT * FROM studies' query from a default python command line. Lo and behold, it returned correctly. I tried all the normal ways to escape the asterisk only to be met by a wide variety of error messages. Is there any way to run this query in IPython?
EDIT: Sorry, I incorrectly assumed IronPython and IPython were the same. What I meant was the IPython command line, not the IronPython framework.
EDIT2: Okay, it turns out the asterisk DOES work as shown by this successful query:
'SELECT COUNT(*) FROM studies'
From the suggestions posted here, it turns out the error results from trying to return records with multiple fields, i.e.:
'SELECT field1,field2 FROM studies'
which still results in to records being returned. I have changed the title of the question accordingly.
This is SQL. IronPython has little or nothing to do with the processing of the query. Are you using an unusual character encoding? (IE not UTF-8 or ASCII)?
What happens if you SELECT id,fieldname,fieldname FROM studies (In other words, simulating what '*' does.)
Some more debugging you could try:
s = 'SELEECT * from studies'
print s
conn.execute(s).fetchall()
or:
s = 'SELECT ' + chr(42) + ' from studies'
conn.execute(s).fetchall()
You might also try:
conn.execute('select count(*) from studies').fetchall()
if that comes back as [(0,)] then something really weird is going on :-)
Some more things you could try:
conn.execute('select id from (select * from studies)').fetchall()
or:
cur = conn.cursor()
cur.execute('select * from studies').fetchall()
I've tried all the things you've mentioned in IPython and sqlite without any problems (ipython 0.9.1, python 2.5.2).
Is there a chance this is some kind of version mismatch issue? Maybe your shells are referencing different libraries?
For example, does
import sqlite3; print sqlite3.version
return the same thing from both shells (i.e. ipython and the regular one where the sql query works)?
How about
conn.execute('select sqlite_version()').fetchall()
Does that return the same thing?
Just a wild guess, but please try to escape backslashes in the path to the database file. In other words instead of
db = 'G:\path\to\db\file.sqlite'
try
db = 'G:\\path\\to\\db\\file.sqlite'
I had a similar problem and found the only way to get the correct output was by assigning the results of the execute statement to another cursor object and calling fetchall on that:
In [1]: import sqlite3
In [2]: conn = sqlite3.connect('file.db')
In [3]: cur = conn.cursor()
In [4]: cur2 = cur.execute("<STATEMENT>")
In [5]: cur2.fetchall()