I'm running into an error when I try to instantiate an object from a cursor in SQLite and I've exhausted my research and couldn't find a solution.
Premise: I cannot use SqlAlchemy or anything of that sorts.
Assumption: The database (SQLite) works, it contains a table named table_cars, and the table is populated with data in its single column: name.
So, I have a class lets say:
class Car():
def __init__(self, name):
self.name = name
#classmethod
def from_cursor(cls, c):
car = cls(c(0))
# this line breaks when called from the function below.
And I also have a db module, with the following function:
def get_cars_from_db():
sql = 'SELECT * FROM table_cars;'
conn = get_conn()
cur = conn.cursor()
cur.execute(sql)
data = cur.fetchall()
# at this point, if i print the cursor, I can see all data, so far so good.
cars = [Car.from_cursor(c) for c in data]
# the line above causes the code to break
return cars
The code breaks with the following error:
TypeError: 'tuple' object is not callable
What am I doing wrong here?
You can use cls(c[0]) or cls(*c) to unpack tuple to function arguments.
It's also worth to specify an exact order of your columns in query.
select name from table_cars
Related
I am writing a script in python 3.x using mysqlconnector.
What I am trying to achieve right now is to check if there is a record inside my db which may be a duplicate to the one I am analyzing right now.
I came up with such code:
def fill_data(self, db_name, data):
cursor = self.cnx.cursor(buffered=True)
isDuplicate = cursor.execute(("SELECT destination FROM {0} WHERE destination = '{1}';")
.format(db_name, data['destination']))
print(cursor.statement)
self.commit()
print(isDuplicate is None)
Though I still get isDuplicate as None object. I tried to check via cursor.statement what statement is being passed to my db: it turned out that while in script I get None obj while passed in db that query works fine.
I also tried SELECT COUNT(1) FROM db_name which also gave me different results.
I am out of ideas: maybe you guys can help me out?
Update:
The solution that works for me was:
q = ("SELECT * FROM {0} WHERE destination = %s AND countryCode = %s AND prefix = %s")
.format(db_name)
cursor.execute(q, (data['destination'], data['country_code'], data['prefix']))
self.cnx.commit()
isDoubled = cursor.fetchone()
So at the end of the day it was all about fetching data from the cursor :)
Maybe the reason of your issue is the way you use execute() method.
Try to make some changes and see what is printed out:
def fill_data(self, db_name, data):
cursor = self.cnx.cursor(buffered=True)
q = 'SELECT count(*) FROM {} WHERE destination = %s'.format(db_name)
duplicate_count = cursor.execute(q, (data['destination'], )).fetchall()
print(duplicate_count)
Why should I provide query parameters this way? (article is on psql, but the core principles are the same as in mysql)
update
If you are still receiving "NoneType" object has no atribute "fetchall", then the error is probably here:
cursor = self.cnx.cursor(buffered=True)
Looks like you are not creating cursor at all. I can take a look at it if you post some code about cnx creation.
I'm newer to OOP in Python, and have been trying for awhile to use my database class database within another class.
How can I do so?
class database(object):
def connect_db(self):
try:
import sqlite3 as sqli
connection = sqli.connect('pw.db')
cur = connection.cursor()
except:
print("There was an error connecting to the database.")
I've been trying to like this, but it doesnt work:
import db_helper as dbh
class account_settings(object):
def create_account(self):
setting_username = input('\nUsername?\n')
setting_password = input('\nPassword?\n')
cur = db.connect_db()
with cur:
cur.execute('''
CREATE TABLE IF NOT EXISTS my_passwords(
id INTEGER PRIMARY KEY AUTOINCREMENT NULL,
password text UNIQUE
)
''')
try:
cur.execute('INSERT INTO my_passwords(password) VALUES(?)', (self.create_password(),) )
except:
print("Error occurred trying to insert password into db. Please retry")
c = account_settings()
c.create_account()
new error:
File "settings.py", line 30, in <module>
c.create_account()
File "settings.py", line 15, in create_account
with cur:
AttributeError: __exit__
You need to learn about variable scope. db.connect_db() creates a cursor connection with the name cur, but does not do anything with it; when that method finishes, the object is destroyed. In particular, it never makes it back to the create_account method.
There is a simple way to solve this: return the object back to the method, and use it there.
def connect_db(self):
...
cur = connection.cursor()
return cur
...
def create_account(self):
cur = db.connect_db()
with cur:
or even beter:
with db.connect_db()
Note that really, neither of these should be classes. Classes in Python are really only useful when you're keeping some kind of state, which isn't happening here. connect_db and create_account should just be standalone functions in their respective modules.
In Python mysqldb I could declare a cursor as a dictionary cursor like this:
cursor = db.cursor(MySQLdb.cursors.DictCursor)
This would enable me to reference columns in the cursor loop by name like this:
for row in cursor: # Using the cursor as iterator
city = row["city"]
state = row["state"]
Is it possible to create a dictionary cursor using this MySQL connector?
http://dev.mysql.com/doc/connector-python/en/connector-python-example-cursor-select.html
Their example only returns a tuple.
I imagine the creators of MySQL would eventually do this for us?
According to this article it is available by passing in 'dictionary=True' to the cursor constructor:
http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursordict.html
so I tried:
cnx = mysql.connector.connect(database='bananas')
cursor = cnx.cursor(dictionary=True)
and got:
TypeError: cursor() got an unexpected keyword argument 'dictionary'
and I tried:
cnx = mysql.connector.connect(database='bananas')
cursor = cnx.cursor(named_tuple=True)
and got:
TypeError: cursor() got an unexpected keyword argument 'named_tuple'
and I tried this one too: cursor = MySQLCursorDict(cnx)
but to no avail. Clearly I'm on the wrong version here and I suspect we just have to be patient as the document at http://downloads.mysql.com/docs/connector-python-relnotes-en.a4.pdf suggests these new features are in alpha phase at point of writing.
A possible solution involves subclassing the MySQLCursor class like this:
class MySQLCursorDict(mysql.connector.cursor.MySQLCursor):
def _row_to_python(self, rowdata, desc=None):
row = super(MySQLCursorDict, self)._row_to_python(rowdata, desc)
if row:
return dict(zip(self.column_names, row))
return None
db = mysql.connector.connect(user='root', database='test')
cursor = db.cursor(cursor_class=MySQLCursorDict)
Now the _row_to_python() method returns a dictionary instead of a tuple.
I found this on the mysql forum, and I believe it was posted by the mysql developers themselves. I hope they add it to the mysql connector package some day.
I tested this and it does work.
UPDATE: As mentioned below by Karl M.W... this subclass is no longer needed in v2 of the mysql.connector. The mysql.connector has been updated and now you can use the following option to enable a dictionary cursor.
cursor = db.cursor(dictionary=True)
This example works:
cnx = mysql.connector.connect(database='world')
cursor = cnx.cursor(dictionary=True)
cursor.execute("SELECT * FROM country WHERE Continent = 'Europe'")
print("Countries in Europe:")
for row in cursor:
print("* {Name}".format(Name=row['Name']
Keep in mind that in this example, 'Name' is specific to the column name of the database being referenced.
Also, if you want to use stored procedures, do this instead:
cursor.callproc(stored_procedure_name, args)
result = []
for recordset in cursor.stored_results():
for row in recordset:
result.append(dict(zip(recordset.column_names,row)))
where stored_procedure_name is the name of the stored procedure to use and args is the list of arguments for that stored procedure (leave this field empty like [] if no arguments to pass in).
This is an example from the MySQL documentation found here: http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursordict.html
Using Python 3.6.2 and MySQLdb version 1.3.10, I got this to work with:
import MySQLdb
import MySQLdb.cursors
...
conn = MySQLdb.connect(host='...',
<connection info>,
cursorclass=MySQLdb.cursors.DictCursor)
try:
with conn.cursor() as cursor:
query = '<SQL>'
data = cursor.fetchall()
for record in data:
... record['<field name>'] ...
finally:
conn.close()
I'm using PyCharm, and simply dug into the MySQLdb modules connections.py and cursors.py.
I had the same problem with the default cursor returning tuples with no column names.
The answer is here:
Getting error while using MySQLdb.cursors.DictCursor in MYSQL_CURSORCLASS
app.config["MYSQL_CURSORCLASS"] = "DictCursor"
I'm trying to run the follwoing 'sqlite3_custom_type.py' example from the book "Python Standard Library by Example". The following code works 'straight out of the box':
import os
import sqlite3
db_filename = 'todo.db'
db_is_new = not os.path.exists(db_filename)
conn = sqlite3.connect(db_filename)
if db_is_new:
print('need to create schema')
else:
print('database exists, assume schema does to')
conn.close()
#import sqlite3
try:
import cPickle as pickle
except:
import pickle
db_filename = 'todo.db'
def adapter_func(obj):
"""Convert from in-memory to storage representation.
"""
print 'adapter_func(%s)\n' % obj
return pickle.dumps(obj)
def converter_func(data):
"""Convert from storage to in-memory representation.
"""
print 'converter_func(%r)\n' % data
return pickle.loads(data)
class MyObj(object):
def __init__(self, arg):
self.arg = arg
def __str__(self):
return 'MyObj(%r)' % self.arg
# Register the functions for manipulating the type.
sqlite3.register_adapter(MyObj, adapter_func)
sqlite3.register_converter("MyObj", converter_func)
# Create some objects to save. Use a list of tuples so
# the sequence can be passed directly to executemany().
to_save = [ (MyObj('this is a value to save'),),
(MyObj(42),),
]
with sqlite3.connect(db_filename,
detect_types=sqlite3.PARSE_DECLTYPES) as conn:
# Create a table with column of type "MyObj"
conn.execute("""
create table if not exists obj (
id integer primary key autoincrement not null,
data MyObj
)
""")
cursor = conn.cursor()
# Insert the objects into the database
cursor.executemany("insert into obj (data) values (?)", to_save)
# Query the database for the objects just saved
cursor.execute("select id, data from obj")
for obj_id, obj in cursor.fetchall():
print 'Retrieved', obj_id, obj, type(obj)
print
But if I put the all the code in a function such as
def stuff():
~same code as above but indented
if __name__=="__main__":
stuff()
then I get an error code:
cursor.executemany("insert into obj (data) values (?)", to_save)
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
Why doesn't the code work when it is in a function and how can I make it work?
As per other answers, it's good style to place classes in module scope. The real reason for failure in this particular case though, is because of the pickle.dumps(obj) call which tries to pickle a non-module level class.
Try the following code in your adapter_func:
def adapter_func(obj):
"""Convert from in-memory to storage representation.
"""
try:
return pickle.dumps(obj)
except Exception, arg:
print 'Failed to pickle object [%s]' % arg
You will see an error such as the following when MyObj is declared inside stuff:
Failed to pickle object [Can't pickle <class '__main__.MyObj'>: it's not found as __main__.MyObj]
It is a requirement of pickle that classes to be pickled are declared at the module level as described in the pickle documentation. The sqlite3 module appears to be squashing exceptions raised in the adapter functions, rather than propagating them through resulting in a silent failure.
You can declare and register your adapter and converter functions inside stuff. Style issues aside, you could also declare your MyObj inside your function and have it work, as long as you find some other way to serialise/deserialise your object.
It's the attempt to pickle a class which isn't at the top level which is the root of this problem.
As per Tichodroma's answer, you need to take out all of the classes and functions out of the stuff function, including sqlite3.register_adapter and sqlite3.register_converter. Also, as a general stylistic point, your imports should go at the top of the script.
The following code works:
import os
import sqlite3
try:
import cPickle as pickle
except:
import pickle
class MyObj(object):
def __init__(self, arg):
self.arg = arg
def __str__(self):
return 'MyObj(%r)' % self.arg
def adapter_func(obj):
"""Convert from in-memory to storage representation.
"""
print('adapter_func(%s)\n' % obj)
return pickle.dumps(obj)
def converter_func(data):
"""Convert from storage to in-memory representation.
"""
print('converter_func(%r)\n' % data)
return pickle.loads(data)
# Register the functions for manipulating the type.
sqlite3.register_adapter(MyObj, adapter_func)
sqlite3.register_converter("MyObj", converter_func)
def stuff():
db_filename = 'todo.db'
db_is_new = not os.path.exists(db_filename)
conn = sqlite3.connect(db_filename)
if db_is_new:
print('need to create schema')
else:
print('database exists, assume schema does to')
conn.close()
db_filename = 'todo.db'
# Create some objects to save. Use a list of tuples so
# the sequence can be passed directly to executemany().
to_save = [ (MyObj('this is a value to save'),),
(MyObj(42),),
]
with sqlite3.connect(db_filename,
detect_types=sqlite3.PARSE_DECLTYPES) as conn:
# Create a table with column of type "MyObj"
conn.execute("""
create table if not exists obj (
id integer primary key autoincrement not null,
data MyObj
)
""")
cursor = conn.cursor()
# Insert the objects into the database
cursor.executemany("insert into obj (data) values (?)", to_save)
# Query the database for the objects just saved
cursor.execute("select id, data from obj")
for obj_id, obj in cursor.fetchall():
print('Retrieved', obj_id, obj, type(obj))
print()
if __name__ == "__main__":
stuff()
Don't put the classes and functions inside of stuff. Especially, don't put MyObj inside there.
If you want to use the if __name__=="__main__": condition, only put the code that is not a class or a function inside of stuff.
I have this error when i perform the following task,
results = db1.executeSelectCommand(siteSql, (),)
TypeError: unbound method executeSelectCommand() must be called with dbConnn instance as first argument (got str instance instead)
My code is as follows:
class dbConnn:
db_con = None
execfile("/Users/usera/Documents/workspace/testing/src/db/db_config.py")
def executeSelectCommand(self,sql,ip):
#psycopg connection here.
I use this class here:
from db import dbConnections
db1 = dbConnections.dbConnn
siteSql = 'select post_content from post_content_ss order by RANDOM() limit 500' #order by year,month ASC'
results = db1.executeSelectCommand(siteSql, (),)
In windows, there don't seem to have a problem with this? God, it must be really elementary but I can't find it.
db1 = dbConnections.dbConnn
Here you assign the class dbConn to the variable db1. You probably wanted to create a new instance instead:
db1 = dbConnections.dbConnn()