I'm moving my code across from an sqlite database to mysql and I'm having a problem with the context manager, getting the following attribute error.
I've tried combinations of mydb.cursor() as cursor, mydb: etc...
mydb = mysql.connector.connect(
host="localhost",
user="root",
passwd="",
database="database_name"
cur = mydb.cursor()
with mydb as cursor:
AttributeError: __enter__
Python has a built-in way to implement a context manager if the object you're creating have a .close() method, by using the contextlib.closing context manager.
From the Python docs:
contextlib.closing(thing)
Return a context manager that closes thing upon completion of the block.
This is basically equivalent to:
from contextlib import contextmanager
#contextmanager
def closing(thing):
try:
yield thing
finally:
thing.close()
So, for your specific issue, you can use not only on the connection, but also the cursor.
Your code would be:
from contextlib import closing
import mysql.connector
query = "SELECT * FROM table"
db_conn_info = {
"user": "root",
"passwd": "",
"host": "localhost",
"port": 5000,
"database": "database_name"
}
with closing(mysql.connector.connect(**db_conn_info)) as conn:
with closing(conn.cursor()) as cur:
cur.execute(query)
result = cur.fetchall()
You have to define your own context manager as mysql.connector.connect is not a context manager.
Context managers have to be defined with __enter__ and __exit__ attributes.
It should be something like this. (Tested using psycopg2)
class DBConnection:
def __init__(self):
self.mydb = mysql.connector.connect(
host="localhost",
user="root",
passwd="",
database="database_name"
)
self.cur = self.mydb.cursor()
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
# close db connection
self.mydb.connection.close()
Related
im trying to develop a simple application where multiple business objects are saved to the sqlite database. I have several business objects for which I want to program save/update/delete methods. For each of the classes and methods within those classes I always create a new connection. For example:
import sqlite3
db = "mydb.db"
class BusinessObject1:
...
def save_to_db(self, db):
conn = sqlite3.connect(db)
cur = conn.cursor()
with conn:
cur.execute("...")
def delete_from_db(self, db):
conn = sqlite3.connect(db)
cur = conn.cursor()
with conn:
cur.execute("...")
class BusinessObject2:
...
def save_to_db(self, db):
conn = sqlite3.connect(db)
cur = conn.cursor()
with conn:
cur.execute("...")
def delete_from_db(self, db):
conn = sqlite3.connect(db)
cur = conn.cursor()
with conn:
cur.execute("...")
It doesn't feel like a good design solution (not DRY). Can someone propose a better design for this? I have around 20 Business objects and 4-8 methods for each and a record table where all these objects are included. Typing conn=sqlite3.connect(db) every time simply cannot be the right way. And if I'll decide to change to MySQL or PostgreSQL I would need to refactor the whole project.
Thanks!
Then you might want to separate the connection handler from the class object or you can do as the comment above suggesting to put the connection inside the class constructor.
If for some reason you can't do both you can still add a staticmethod to your class that will handle the connection. It's not ideal but it's still better than your solution:
#staticmethod
def create_connection(db_file):
conn = None
try:
with sqlite3.connect(db_file) as conn:
return conn
except sqlite3.Error as e:
print("Can't connect to database, error:", e, sep="\n")
return conn
Then you add a parameter to your other methods which will be the returned create_connection method:
def save_to_db(self, connection):
cur = connection.cursor()
cur.execute("...")
This way you can separate your connect object from your class object:
if __name__ == "__main__":
conn = BusinessObject1.create_connection(db)
b1 = BusinessObject1(...)
b1.save_to_db(conn)
When it comes to functions that require database access, I'm trying to figure out how to configure a connection between multiple MySQL databases; one existing in main.py and the other in test.py. So that when I import functions from main.py to test.py the functions execute against the test database.
This is my first time working with MySQL/Python Connector so I'm not sure how to achieve this.
main.py
import mysql.connector
def database_connect(func):
def wrapper_db_connect():
with mysql.connector.connect(user=user, password=password, database=db)
return func()
return wrapper_db_connect
#database_connect
def select_listing(x=None):
pass
#database_connect
def insert_record():
pass
It's actually simpler than you'd imagine, you'd have to create a global mysql connector variable like so,
mydb = mysql.connector.connect(
host="localhost",
user="yourusername",
password="yourpassword"
database = "databasename"
)
At which point you can use it as a parameter for your functions,
#main.py
def exampleFunc(mydb):
mycursor = mydb.cursor()
mycursor.execute("SELECT * FROM example")
return mycursor.fetchall()
#test.py
mydb = mysql.connector.connect(
host="localhost",
user="yourusername",
password="yourpassword"
database = "test_database"
)
val = exampleFunction(mydb)
I am currently following alongside a book (cpp for quantitative finance) and I am trying to import the symbols for S&P500 from wiki into a sql database I've created. However, I am getting the AttributeError: exit with regards to my "with con" statement (see below). I have read posts from similar errors but I cannot seem to fix mine. I am extremely new to python so perhaps there is some fundamental misunderstanding on my part. I have included the relevant code below, any advice would be hugely appreciated.
"""
Insert the S&P500 symbols into the MySQL database.
"""
# Connect to the MySQL instance
db_host = 'localhost'
db_user = 'sec_user'
db_pass = 'database_password'
db_name = 'database_name'
con = mdb.connect(
host=db_host, user=db_user, passwd=db_pass, db=db_name
)
# Create the insert strings
column_str = """ticker, instrument, name, sector,
currency, created_date, last_updated_date
"""
insert_str = ("%s, " * 7)[:-2]
final_str = "INSERT INTO symbol (%s) VALUES (%s)" % \
(column_str, insert_str)
# Using the MySQL connection, carry out
# an INSERT INTO for every symbol
with con:
cur = con.cursor()
cur.executemany(final_str, symbols)
if __name__ == "__main__":
symbols = obtain_parse_wiki_snp500()
insert_snp500_symbols(symbols)
print("%s symbols were successfully added." % len(symbols))
The error is telling you that the object returned by mdb.connect is not a context manager, that is it cannot be used in a with statement. You'll need to close the connection manually once you've finished with it (con.close()) or use a package that provides a connection that is a context manager.
A quick study of commonly used connectors suggests you want to use pymysql
>>> import MySQLdb
>>> import mysql.connector
>>> import pymysql
>>> params = {'host': 'localhost', 'user': 'root', 'password': '', 'database': 'test'}
>>> for pkg in (MySQLdb, mysql.connector, pymysql):
... conn = pkg.connect(**params)
... try:
... with conn:
... pass
... except AttributeError as ex:
... print(pkg.__name__, 'failed with', ex)
...
MySQLdb failed with __enter__
mysql.connector failed with __enter__
If you have to use a connection that is not a context manager, you can emulate it in a try/except/finally suite:
import MySQLdb
conn = MySQLdb.connect(host='localhost', user='root', password='', database='test')
try:
cursor = conn.cursor()
cursor.execute('SELECT * FROM my_table;')
for row in cursor.fetchall():
print(row)
cursor.close()
conn.commit()
except:
# log the error here
conn.rollback()
finally:
conn.close()
Or you can make your own context manager using the tools provided in contextlib:
import contextlib
import MySQLdb
#contextlib.contextmanager
def managed_connection(conn):
try:
yield
conn.commit()
except:
# log the error here
conn.rollback()
finally:
conn.close()
conn = MySQLdb.connect(host='localhost', user='root', password='', database='test')
with managed_connection(conn) as mc:
cursor = mc.cursor()
cursor.execute('SELECT * FROM my_table;')
for row in cursor.fetchall():
print(row)
cursor.close()
(You can make a cursor context manager too, or have the context manager yield a cursor rather than the connection).
I am starting to use the mysqldb module in python and I seem to have some issues with the "standard" way of calling queries.
I understand that the standard way is to create a cursor and then use it to execute queries.
However, when I try to instanciate one, it gives me the following error :
AttributeError: cursor
My Database class looks like :
class Database():
def __init__(self):
server = "localhost"
login = "login"
password = "passws"
database = "DB"
my_conv = { FIELD_TYPE.LONG: int }
self.conn = MySQLdb.connection(user=login, passwd=password, db=database, host=server, conv=my_conv)
self.cursor = self.conn.cursor()
def close(self):
self.conn.close()
def execute(self, query):
self.cursor.execute(query)
return self.cursor.fetchall()
For now I get it working by using the query method, but I feel not using the standard will give me trouble in the future.
Any idea ?
You are using wrong connection constructor.
MySQLdb.Connection instead of MySQLdb.connection should work.
I want my code to automatically try multiple ways to create a database connection. As soon as one works, the code needs to move on (i.e. it shouldn't try to other ways anymore). If they all fail well, then the script can just blow up.
So in - what I thought was, but most likely isn't - a stroke of genius I tried this:
import psycopg2
from getpass import getpass
# ouch, global variable, ooh well, it's just a simple script eh
CURSOR = None
def get_cursor():
"""Create database connection and return standard cursor."""
global CURSOR
if not CURSOR:
# try to connect and get a cursor
try:
# first try the bog standard way: db postgres, user postgres and local socket
conn = psycopg2.connect(database='postgres', user='postgres')
except psycopg2.OperationalError:
# maybe user pgsql?
conn = psycopg2.connect(database='postgres', user='pgsql')
except psycopg2.OperationalError:
# maybe it was postgres, but on localhost? prolly need password then
conn = psycopg2.connect(database='postgres', user='postgres', host='localhost', password=getpass())
except psycopg2.OperationalError:
# or maybe it was pgsql and on localhost
conn = psycopg2.connect(database='postgres', user='pgsql', host='localhost', password=getpass())
# allright, nothing blew up, so we have a connection
# now make a cursor
CURSOR = conn.cursor()
# return existing or new cursor
return CURSOR
But it seems that the second and subsequent except statements aren't catching the OperationalErrors anymore. Probably because Python only catches an exception once in a try...except statement?
Is that so? If not: is there anything else I'm doing wrong? If so: how do you do something like this then? Is there a standard idiom?
(I know there are ways around this problem, like having the user specify the connection parameters on the command line, but that's not my question ok :) )
EDIT:
I accepted retracile's excellent answer and I took in gnibbler's comment for using the for..else construct. The final code became (sorry, I'm not really following the max characters per line limit from pep8):
EDIT 2: As you can see from the comment on the Cursor class: I don't really know how to call this kind of class. It's not really a singleton (I can have multiple different instances of Cursor) but when calling get_cursor I do get the same cursor object everytime. So it's like a singleton factory? :)
import psycopg2
from getpass import getpass
import sys
class UnableToConnectError(Exception):
pass
class Cursor:
"""Cursor singleton factory?"""
def __init__(self):
self.CURSOR = None
def __call__(self):
if self.CURSOR is None:
# try to connect and get a cursor
attempts = [
{'database': 'postgres', 'user': 'postgres'},
{'database': 'postgres', 'user': 'pgsql'},
{'database': 'postgres', 'user': 'postgres', 'host': 'localhost', 'password': None},
{'database': 'postgres', 'user': 'pgsql', 'host': 'localhost', 'password': None},
]
for attempt in attempts:
if 'password' in attempt:
attempt['password'] = getpass(stream=sys.stderr) # tty and stderr are default in 2.6, but 2.5 uses sys.stdout, which I don't want
try:
conn = psycopg2.connect(**attempt)
attempt.pop('password', None)
sys.stderr.write("Succesfully connected using: %s\n\n" % attempt)
break # no exception raised, we have a connection, break out of for loop
except psycopg2.OperationalError:
pass
else:
raise UnableToConnectError("Unable to connect: exhausted standard permutations of connection dsn.")
# allright, nothing blew up, so we have a connection
# now make a cursor
self.CURSOR = conn.cursor()
# return existing or new cursor
return self.CURSOR
get_cursor = Cursor()
Approximately:
attempts = [
{ 'database'='postgres', 'user'='pgsql', ...},
{ 'database'='postgres', 'user'='postgres', 'host'='localhost', 'password'=getpass()},
...
]
conn = None
for attempt in attempts:
try:
conn = psycopg2.connect(**attempt)
break
except psycopg2.OperationalError:
pass
if conn is None:
raise a ruckus
CURSOR = conn.cursor()
Now, if you don't want to call getpass() unless it is necessary, you'd want to check if 'password' in attempt: attempt['password'] = getpass() or so.
Now about that global....
class MyCursor:
def __init__(self):
self.CURSOR = None
def __call__(self):
if self.CURSOR is None:
<insert logic here>
return self.CURSOR
get_cursor = MyCursor()
... though I think there are a couple of other ways to accomplish the same thing.
Bringing it all together:
class MyCursor:
def __init__(self):
self.CURSOR = None
def __call__(self):
if self.CURSOR is None:
attempts = [
{'database'='postgres', 'user'='postgres'},
{'database'='postgres', 'user'='pgsql'},
{'database'='postgres', 'user'='postgres', 'host'='localhost', 'password'=True},
{'database'='postgres', 'user'='pgsql', 'host'='localhost', 'password'=True},
]
conn = None
for attempt in attempts:
if 'password' in attempt:
attempt['password'] = getpass()
try:
conn = psycopg2.connect(**attempt)
break # that didn't throw an exception, we're done
except psycopg2.OperationalError:
pass
if conn is None:
raise a ruckus # nothin' worked
self.CURSOR = conn.cursor()
return self.CURSOR
get_cursor = MyCursor()
Note: completely untested
You're close. Probably the best thing to do in this case is nesting the second and subsequent attempts in the except block. Thus the critical part of your code would look like:
if not CURSOR:
# try to connect and get a cursor
try:
# first try the bog standard way: db postgres, user postgres and local socket
conn = psycopg2.connect(database='postgres', user='postgres')
except psycopg2.OperationalError:
# maybe user pgsql?
try:
conn = psycopg2.connect(database='postgres', user='pgsql')
except psycopg2.OperationalError:
# maybe it was postgres, but on localhost? prolly need password then
try:
conn = psycopg2.connect(database='postgres', user='postgres', host='localhost', password=getpass())
except psycopg2.OperationalError:
# or maybe it was pgsql and on localhost
conn = psycopg2.connect(database='postgres', user='pgsql', host='localhost', password=getpass())