Develop Reference

cursor.execute returns 1 int

When running a query to a mysql database using MySqlHook, cursor.execute(query) returns int 1
My code is
import logging
from airflow.hooks.mysql_hook import MySqlHook
query = "SELECT col1, col2 FROM myschema.mytable LIMIT 1"
mysql = MySqlHook(mysql_conn_id=conn_id)
conn = mysql.get_conn()
cursor = conn.cursor()
result_cursor = cursor.execute(query)
logging.info(result_cursor) # this prints out "INFO - 1" in the log
df = pd.DataFrame(result_cursor.fetchall(), columns=result_cursor.keys()) # this triggers error "ERROR - 'int' object has no attribute 'fetchall'"
I would have expected result_cursor to return a "fetchable" result, since the query is working fine.

Cursor.execute() return value is not defined by the db-api spec, but for most implementations it returns the number of rows affected by the query.
To retrieve data, you have to either iterate over the cursor or call .fetchall().

It seems I cannot save cursor.execute(query) into variable result_cursor.
To make the code work, I simply needed to define the data for the data-frame as cursor.fetchall()
cursor.execute(query)
df = pd.DataFrame(list(cursor.fetchall()), column=[col[0] for col in cursor.description])

cx_oracle python iteration

I have my python script which reads an excel column row by row and returns all rows str(values).
I want to write another script which will allow put these values to sql db. I've already written connect method:
def db_connect():
adr = 'some_addr'
uid = 'some_uid'
pwd = 'pwd'
port = port
dsn_tns = cx_Oracle.makedsn(adr, port, SID)
db = cx_Oracle.connect('username', 'pass', dsn_tns)
cur = db.cursor()
cur.execute('update TABLE set ROW = 666 where ANOTHER_ROW is null')
db.commit()
This method does an update but it sets 666 for ALL rows. How to do it by kind of iteration in sql? For example, first row of output == 1, second == 23, third == 888.

If I understand correctly what you are trying to do here it should be done in two phases. First select all rows for update (based on chosen condition), then you can iteratively update each of these rows.
It cannot be done in single query (or on only single condition that does not change through a number of queries), because SQL works on sets, that's why each time your query is executed you are updating whole table, and in the end only getting result of the last query.

You can use the "rownum" expression, as in:
cur.execute("update TABLE set ROW = rownum where ANOTHER_ROW is null")
This will start with the value 1 and increment up by one for each row updated.
If you want more control over the value to set, you can also do the following in PL/SQL (untested):
cur.execute("""
declare
t_NewValue number;
cursor c_Data is
select ROW, ANOTHER_ROW
from TABLE
where ANOTHER_ROW is null
for update;
begin
t_NewValue := 1;
for row in c_Data loop
update TABLE set ROW = t_NewValue
where current of c_Data;
t_NewValue := t_NewValue + 1;
end loop;
end;""")
This gives you the most control. You can use whatever logic you require to control what the new value should be.

Please take a look at another method which is writing to excel:
adr = 'some_addr'
uid = 'some_uid'
pwd = 'pwd'
port = port
dsn_tns = cx_Oracle.makedsn(adr, port, SID)
db = cx_Oracle.connect('username', 'pass', dsn_tns)
cur = db.cursor()
cells = excel.read_from_cell()
indices_and_statuses = []
stat = execute_script(some_js)
for req_id in cells:
indices_and_statuses.append((cells.index(req_id), stat))
cur.execute("""update TABLE set ROW ="""+"'"+req_id+"'"+"""where ANOTHER_ROW is null""")
db.commit()
db.close()
And in this code when you put print(req_id) in this FOR statement, you will see that req_id is changing. But in DB only the last req_id is saved.

Counting Organizations by using Python and Sqlite

This application will read the mailbox data (mbox.txt) count up the number email messages per organization (i.e. domain name of the email address) using a database with the following schema to maintain the counts.
CREATE TABLE Counts (org TEXT, count INTEGER)
When you have run the program on mbox.txt upload the resulting database file above for grading.
If you run the program multiple times in testing or with different files, make sure to empty out the data before each run.
You can use this code as a starting point for your application: http://www.pythonlearn.com/code/emaildb.py. The data file for this application is the same as in previous assignments: http://www.pythonlearn.com/code/mbox.txt.
First time to learn Sqlite. I am very confused about this assignment although it seems to be easy. I don't know how can I connect Python codes to Sqlite. It seems that they don't need the code as assignment. All the need is database file. How should I solve this problem. Don't know how to start it. Much appreciated it!

The starting code you've been given is a really good template for what you want to do. The difference is that - in that example - you're counting occurences of email address, and in this problem you're counting domains.
First thing to do is think about how to get domain names from email addresses. Building from the code given (which sets email = pieces[1]):
domain = email.split('#')[1]
This will break the email on the # character, and return the second item (the part after the '#'), which is the domain - the thing you want to count.
After this, go through the SQL statements in the code and replace 'email' with 'domain', so that you're counting the right thing.
One last thing - the template code checks 'mbox-short.txt' - you'll need to edit that as well for the file you want.

import sqlite3
conn = sqlite3.connect('emaildb2.sqlite')
cur = conn.cursor()
cur.execute('''
DROP TABLE IF EXISTS Counts''')
cur.execute('''
CREATE TABLE Counts (org TEXT, count INTEGER)''')
fname = input('Enter file name: ')
if (len(fname) < 1): fname = 'mbox.txt'
fh = open(fname)
list_1 =[]
for line in fh:
if not line.startswith('From: '): continue
pieces = line.split()
email = pieces[1]
dom = email.find('#')
org = email[dom+1:len(email)]
cur.execute('SELECT count FROM Counts WHERE org = ? ', (org,))
row = cur.fetchone()
if row is None:
cur.execute('''INSERT INTO Counts (org, count)
VALUES (?, 1)''', (org,))
else:
cur.execute('UPDATE Counts SET count = count + 1 WHERE org = ?',
(org,))
conn.commit()
# https://www.sqlite.org/lang_select.html
sqlstr = 'SELECT org, count FROM Counts ORDER BY count DESC LIMIT 10'
for row in cur.execute(sqlstr):
print(str(row[0]), row[1])
cur.close()

I am still new here, but I want to thank Stidgeon for pointing me in the right direction. I suspect other Using Databases with Python students will end up here too.
There are two things you need to do with the source code.
domain = email.split('#')[1] http://www.pythonlearn.com/code/emaildb.py
Change from email TEXT to org TEXT when the database is generated.
That should get you on your way.

import sqlite3
conn = sqlite3.connect('emaildb.sqlite')
cur = conn.cursor()
cur.execute('DROP TABLE IF EXISTS Counts')
cur.execute('''
CREATE TABLE Counts (org TEXT, count INTEGER)''')
fname = input('Enter file name: ')
if (len(fname) < 1): fname = 'mbox-short.txt'
fh = open(fname)
for line in fh:
if not line.startswith('From: '): continue
pieces = line.split()
org = pieces[1].split('#')
cur.execute('SELECT count FROM Counts WHERE org = ? ', (org[1],))
row = cur.fetchone()
if row is None:
cur.execute('''INSERT INTO Counts (org, count)
VALUES (?, 1)''', (org[1],))
else:
cur.execute('UPDATE Counts SET count = count + 1 WHERE org = ?',
(org[1],))
conn.commit()
# https://www.sqlite.org/lang_select.html
sqlstr = 'SELECT org, count FROM Counts ORDER BY count DESC LIMIT 10'
for row in cur.execute(sqlstr):
print(str(row[0]), row[1])
cur.close()
print('-----------------done----------------')

mariadb for update doesn't work correctly

I was testing mariadb for update with python mysqldb and it doesn't works correctly for me. The same rows are getting twice when running two python scripts symultaneously as cronjob. When I testing on two mysql consoles everything going fine. Using innodb. My code:
cursor.execute("start transaction")
cursor.execute("SELECT id FROM t1 where tw1=2 AND tw2=1 AND tw3=3 order by id limit 1000 FOR UPDATE")
rows = [r for r in cursor.fetchall()]
ids = []
for row in rows:
ids.append(str(row['id']))
for i in range(2):
cursor.execute("SELECT id FROM t2 where tw1="sth" AND tw2 =0 AND tw3=0 LIMIT 1 FOR UPDATE")
d = cursor.fetchone()
if not d:
#here: insert to t2 and get last id as d_id
cursor.execute("update t2 SET tu1=1,tu2=500,tu3=0 WHERE id = %s" % d_id)
for row in rows:
#job with rows
cursor.execute( "Update t1 SET tw1=tw1+1 WHERE id IN (" + ','.join(ids) + ")")
cursor.execute("commit")
tw1 from t1 should be max 3 but it is very often 4. When I put ids to file I see that there are the same ids for 2 diffrent program runs. What I'm doing wrong?

Inserting rows while fetching(from another table) in SQLite

I'm getting this error no matter what with python and sqlite.
File "addbooks.py", line 77, in saveBook
conn.commit()
sqlite3.OperationalError: cannot commit transaction - SQL statements in progress
The code looks like this:
conn = sqlite3.connect(fname)
cread = conn.cursor()
cread.execute('''select book_text from table''')
while True:
row = cread.fetchone()
if row is None:
break
....
for entry in getEntries(doc):
saveBook(entry, conn)
Can't do a fetchall() because table and column size are big, and the memory is scarce.
What can be done without resorting to dirty tricks(as getting the rowids in memory, which would probably fit, and then selecting the rows one by one)?.

The problem is that you've left the connection in auto-commit mode. Wrap a single transaction around the whole lot so that a commit only happens after you've done all the updates, and it should all work fine.

Don't know if this count as "dirty tricks" too ;-)
My solution to this problem is to use SELECT... LIMIT clause, assumed you have primary key integer field id
current_id = 0
while True:
cread.execute('''select book_text from table where id > %s limit 2''' % current_id)
results = cread.fetchall()
if results is None:
break;
for row in results:
... (save book) ...
current_id = row.id

The problem is that there should be no more than a single active cursor for a connection.
The solution is to use a new connection for the updates.
Unfortunatelly I do not remember the exact place in docs where I read it, so I can not prove it.
UPD:
The following code works on my Windows XP:
import sqlite3
import os
conn1 = sqlite3.connect('test.db')
cursor1 = conn1.cursor()
conn2 = sqlite3.connect('test.db')
cursor2 = conn2.cursor()
cursor1.execute("CREATE TABLE my_table (a INT, b TEXT)")
cursor1.executemany("INSERT INTO my_table (a, b) VALUES (?, NULL);", zip(range(5)))
conn1.commit()
cursor1.execute("SELECT * FROM my_table")
for a, b in cursor1:
cursor2.execute("UPDATE my_table SET b='updated' WHERE a = ?", (a, ))
conn2.commit()
print "results:"
print 10 * '-'
cursor1.execute("SELECT * FROM my_table")
for a, b in cursor1:
print a, b
cursor1.close()
conn1.close()
cursor2.close()
conn2.close()
os.unlink('test.db')
And returns the following as expected:
results:
----------
0 updated
1 updated
2 updated
3 updated
4 updated
If I move the conn2.commit() into the for loop, I get the same error as you mention:
Traceback (most recent call last):
File "concurent.py", line 16, in <module>
conn2.commit()
sqlite3.OperationalError: database is locked
Thus, the solution is to commit once at the end instead of committing after each line.