As the title states I need some help with Python and MySQL. I am currently studying Python further and I am focusing hard on using Python and MySQL for database design, development, administration and applications.
I am familiar with MySQL and somewhat familiar with Python. Currently I am working on object orientated programming and I am trying my hand at setting up a database connection inside of a database class and then using the class to Create, Update, Delete and Read data.
I have created a new Python object:
import pymysql as MySQL
class Database(object):
Host = "127.0.0.1"
Database = "****"
user = "****"
password = "****"
#staticmethod
def initialize():
currentdb = MySQL.connect(Database.Host, Database.user, Database.password, Database.Database)
cursor = currentdb.cursor()
#staticmethod
def insert(Table, DataDict):
placeholders = ", ".join(["%s"] * len(DataDict))
columns = ", ".join(DataDict.keys())
sql = "INSERT INTO %s (%s) VALUES (%s)"%(Table, columns, placeholders)
cursor.execute(sql, DataDict.values())
I want to know, how do I work with the cursor inside of a object? I don't know if my current approach is even close to how it should be handled, I am really not sure.
Can the cursor be initialized in this way, and then used further in the object as I intend on doing in the above extract?
Any help would be highly appreciated.
The right way to work with cursors is like this:
import contextlib
def doSomething():
with contextlib.closing(database.cursor()) as cursor:
cursor.execute(...)
# At the end of the `with` statement, cursor is closed
Do not keep a cursor open for too long. Keeping a connection open for a long time, as you do, is fine. Also, read on transaction control.
If you're doing more than a handful of DB operations, consider using a library like SQLAlchemy or Pony ORM.
import contextlib
def doSomething():
with contextlib.closing(database.cursor()) as cursor:
cursor.execute(...)
library for db SQLAlchemy or Pony ORM.
Have you considered using SQLAlchemy? This gives you a mapping between Python classes and MySQL (or any other RDBMS) tables. I've recently been using it on a fairly hefty real-world project and it seems to do the job fairly well and is easy enough to learn.
Check out the following code. I added the content in your initialize() to the standard python class init method and made the database be initialized with different types of parameters:
import pymysql as MySQL
class Database(object):
def __init__(self, host, db, user, pw):
self.currentdb = MySQL.connect(Database.host, user, pw, db)
def insert(self, Table, DataDict):
placeholders = ", ".join(["%s"] * len(DataDict))
columns = ", ".join(DataDict.keys())
sql = "INSERT INTO %s (%s) VALUES (%s)"%(Table, columns, placeholders)
with self.currentdb.cursor() as db_cursor:
db_cursor.execute(sql, DataDict.values())
Once you are here, then you can initialize a Database object as below and insert data:
my_db = Database(host="127.0.0.1", user="****", pw="****", db="****")
my_db.insert('table_name', data_dict)
Please note, I haven't changed your code, only presenting an organization based on your initial post that could work.
Related
I was writing a few simple CRUD operations to try out sqlite3 with Python, and then I saw a nice function that executes queries and closes connection in this answer:
from contextlib import closing
import sqlite3
def query(self, db_name, sql):
with closing(sqlite3.connect(db_name)) as con, con, \
closing(con.cursor()) as cur:
cur.execute(sql)
return cur.fetchall()
I thought it would be nice to have something like this and call this function with whatever sql sentence I need whenever I want to query the database.
However, when I'm running an insert I'd need to return cur.lastrowid instead or cur.fetchall() and when deleting I'd like to know the cursor.rowcount instead. Also, sometimes I need to add parameters to the query, for instance sometimes I want to run select * from [some_table] and some other times I need select * from [some_table] where [some_column] = ?. So the function needs some tweaks depending on what kind of operation is being executed.
I could write one function for each kind of operation, with the same basic structure and the tweaks each query needs. But that sounds a bit repetitive since there would be duplicate chunks of code and these functions would look pretty similar to each other. So I'm not sure it's the right approach.
Is there another alternative to make this function a bit more "generic" to fit all cases?
One option is to have callouts in the with clause that let you customize program actions. There are many ways to do this. One is to write a class that calls methods to allow specialization. In this example, a class has pre and post processers. It does its work in __init__ and leaves its result in an instance variable which allows for terse usage.
from contextlib import closing
import sqlite3
class SqlExec:
def __init__(self, db_name, sql, parameters=()):
self.sql = sql
self.parameters = parameters
with closing(sqlite3.connect(db_name)) as self.con, \
closing(con.cursor()) as self.cur:
self.pre_process()
self.cur.execute(self.sql, parameters=self.parameters)
self.retval = self.post_process()
def pre_process(self):
return
def post_process_fetchall(self):
self.retval = self.cur.fetchall
post_process = post_process_fetchall
class SqlExecLastRowId(SqlExec):
def post_process(self):
self.retval = cur.lastrowid
last_row = SqlExecLastRowId("mydb.db", "DELETE FROM FOO WHERE BAR='{}'",
paramters=("baz",)).retval
I spent a lot of time looking for solution to parametrize table names and field names in SQL Alchemy plain textual SQL queries for SQL Server. I stumbled upon several stackoverflow questions and other resources like:
SQL Alchemy Parametrized Query , binding table name as parameter gives error
Answer to above which I don't like as it is just building query from string which is proun to SQL Injection attacks
I know it is possible (I was doing it that way in the past) to do by creating table objects from sqlalchemy.ext.declarative.declarative_base but it requires to declare whole schema of your database which is a lot of unscalable code.
Without much luck with SQL Server I found solution in Postgres psycopg2 using
psycopg2.sql.Identifier. So from here I started looking for equivalent in SQL Alchemy. I found quoted_name. Which I understand works as identifier preventing from SQL Injections. But is it really? Could somebody confirm that it is safe to use?
Code example below which returns number of rows in the passed in table:
def count_rows(self, table_name: str) -> int:
query_base = "SELECT COUNT(*) FROM {}"
query_params = [quoted_name(table_name, True)]
query = text((query_base).format(*query_params))
with self.engine.connect() as con:
result = con.execute(query).fetchone()
return result[0]
I don't get the impression from the documentation this is the purpose for which quoted_name is intended. My reading was that it's for cases where non-standard naming conventions for column or table names are in use, requiring quotation for them to work.
I think there are two possible solutions:
1. exercise total control over the allowed table names
f"SELECT COUNT(*) FROM {table_name}" is fine if you don't allow table_name to be provided by the user without filtering.
For example, you could simply have
...
allowed = ["table_1", ..., "table_N"]
if table_name not in allowed:
raise ValueError(f"Table name must be one of {allowed}. Received {table_name}.")
There are, of course, plenty of other ways to do this. But the idea is to either map user input to allowed values, reject disallowed values, or a mixture of both.
2. reflect the schema
You mentioned that
I know it is possible (I was doing it that way in the past) to do by creating table objects from sqlalchemy.ext.declarative.declarative_base but it requires to declare whole schema of your database which is a lot of unscalable code.
This is not true. You can 'reflect' the schema of an existing database as follows:
from sqlalchemy import create_engine, func, select, MetaData
class YourClass:
def __init__(self, db_connection_string: str):
"""
__init__ for YourClass
(for example)
"""
self.engine = create_engine(db_connection_string)
self.metadata = MetaData(bind=self.engine)
MetaData.reflect(self.metadata)
def count_rows(self, table_name: str) -> int:
"""
count_rows
Returns the COUNT of the rows for a given table
"""
table = self.metadata.tables[table_name]
result = select([func.count()]).select_from(table).scalar()
return result
Worth noting that this approach will also throw an exception if table_name doesn't exist in the database.
Alternative syntax - for full ORM-goodness, use a sessionmaker:
from sqlalchemy import create_engine, MetaData
from sqlalchemy.orm import sessionmaker
class YourClass:
def __init__(self, db_connection_string: str):
self.engine = create_engine(db_connection_string)
self.Session = sessionmaker(bind=self.engine)
self.metadata = MetaData(bind=self.engine)
MetaData.reflect(self.metadata)
def count_rows(self, table_name: str) -> int:
table = self.metadata.tables[table_name]
# if you want a new session every call:
with self.Session.begin() as session:
return session.query(table).count()
I'm creating my DB from an existing shema and it's stored in :memory:.
db = Database(filename=':memory:', schema='schema.sql')
db.recreate()
I now want to "link" this to SQL Alchemy. Followed different methods but could not get it right.
My current attempt stands as follow:
engine = create_engine('sqlite:///:memory:')
Base = automap_base()
Base.prepare(engine, reflect=True)
User = Base.classes.user
session = Session(engine)
Much like the other stuff I tried this will throw AttributeError: user.
How can I have this work together?
The relevant part of the documentation is here: https://sqlite.org/inmemorydb.html .
If you use :memory: then every connection will have its own memory database. The trick is to use a named in memory database with the URI format, like the following
import random
import string
import sqlite3
# creating a random name for the temporary memory DB
sqlite_shared_name = "test_db_{}".format(
random.sample(string.ascii_letters, k=4)
)
create_engine(
"sqlite:///file:{}?mode=memory&cache=shared&uri=true".format(
sqlite_shared_name))
the format is a URI as stated by the query string parameter uri=true (see SQLAlchemy documentation)
it is a memory DB with mode=memory
it can be shared among various connection with cache=shared
If you have another connection, then you can use more or less the same connection string. For instance, for getting the connection to that same DB in memory using python's sqlite module, you can drop the uri=true from the query string (and the dialect part sqlite:///) and pass it as argument:
dest = sqlite3.connect(
"file:{}?mode=memory&cache=shared".format(sqlite_shared_name),
uri=True)
I am writing a class for database queries with SQLite3 in my application. Most of the methods of the class are very similar to this:
def getPrice(self, symbol, date):
date = dt.datetime.strptime(date, '%Y-%m-%d')
conn = sqlite3.connect('stocks.db')
curs =conn.cursor()
curs.execute('''SELECT close FROM prices WHERE symbol = ? AND date = ?;''', (symbol, date))
close = curs.fetchall()
curs.close()
return close
The only difference is the database query and the number of arguments. Is there a possibility to abstract the opening and closing of the database connection away?
I know that it would be probably easier to use a ORM like SQLAlchemy. But I want to understand how I solve this kind of problem in general, not only in relation to databases.
Thanks for your suggestions!
EDIT: This post basically answers my question.
First. You'll be much, much happier with one -- and only one -- global connection. Configuration changes are much easier if you do this in exactly one place.
Second, use the with statement and the context manager library.
from contextlib import closing
from my_database_module import the_global_connection
def getPrice(
with closing(the_global_connection.cursor())
curs.execute('''SELECT close FROM prices WHERE symbol = ? AND date = ?;''', (symbol, date))
close = curs.fetchall()
return close
Your database module looks like this:
import sqlite3
the_global_connection = sqlite3.connect( "stocks.db" )
This gives you the ability to change databases, or database server technology in exactly one place.
Note that as of Python2.6, sqlite.connect returns a context manager:
Connection objects can be used as context managers that automatically
commit or rollback transactions. In the event of an exception, the
transaction is rolled back; otherwise, the transaction is committed:
Therefore, do not decorate the connection with contextlib.closing -- otherwise, you will lose the commit/rollback behavior and instead only get the connection.close() called upon exiting the with-statement.
Per PEP249:
... closing a connection without committing the changes first will cause
an implicit rollback to be performed.
So the commit/rollback behavior is much more useful than simply calling close.
You could use a context manager:
import contextlib
def query(sql,args):
with contextlib.closing(sqlite3.connect('stocks.db')) as conn:
curs = conn.cursor()
curs.execute(sql,args))
close = curs.fetchall()
return close
def getPrice(self, symbol, date):
date = dt.datetime.strptime(date, '%Y-%m-%d')
sql = '''SELECT close FROM prices WHERE symbol = ? AND date = ?'''
args = (symbol, date)
return query(sql, args)
Since you have many functions like getPrice which differ only by the SQL and arguments, you could reduce the repetitious boiler-plate code by defining the query function.
You could also define a context manager to rollback the connection on errors and commit as well as close upon exiting the with block. An example of this (for MySQL) can be found here, adapting it to sqlite3 should not be difficult..
Reference:
The contextlib.closing decorator
Encapsulate that logic into an object, pass that object to the data access object and ask it to call the methods.
Aspects or decorators might be a good way to do things.
You don't mention pooling or transactions. Think about those as well.
I am designing a fairly complex database, and know that some of my queries will be far outside the scope of Django's ORM. Has anyone integrated SP's with Django's ORM successfully? If so, what RDBMS and how did you do it?
We (musicpictures.com / eviscape.com) wrote that django snippet but its not the whole story (actually that code was only tested on Oracle at that time).
Stored procedures make sense when you want to reuse tried and tested SP code or where one SP call will be faster than multiple calls to the database - or where security requires moderated access to the database - or where the queries are very complicated / multistep. We're using a hybrid model/SP approach against both Oracle and Postgres databases.
The trick is to make it easy to use and keep it "django" like. We use a make_instance function which takes the result of cursor and creates instances of a model populated from the cursor. This is nice because the cursor might return additional fields. Then you can use those instances in your code / templates much like normal django model objects.
def make_instance(instance, values):
'''
Copied from eviscape.com
generates an instance for dict data coming from an sp
expects:
instance - empty instance of the model to generate
values - dictionary from a stored procedure with keys that are named like the
model's attributes
use like:
evis = InstanceGenerator(Evis(), evis_dict_from_SP)
>>> make_instance(Evis(), {'evi_id': '007', 'evi_subject': 'J. Bond, Architect'})
<Evis: J. Bond, Architect>
'''
attributes = filter(lambda x: not x.startswith('_'), instance.__dict__.keys())
for a in attributes:
try:
# field names from oracle sp are UPPER CASE
# we want to put PIC_ID in pic_id etc.
setattr(instance, a, values[a.upper()])
del values[a.upper()]
except:
pass
#add any values that are not in the model as well
for v in values.keys():
setattr(instance, v, values[v])
#print 'setting %s to %s' % (v, values[v])
return instance
# Use it like this:
pictures = [make_instance(Pictures(), item) for item in picture_dict]
# And here are some helper functions:
def call_an_sp(self, var):
cursor = connection.cursor()
cursor.callproc("fn_sp_name", (var,))
return self.fn_generic(cursor)
def fn_generic(self, cursor):
msg = cursor.fetchone()[0]
cursor.execute('FETCH ALL IN "%s"' % msg)
thing = create_dict_from_cursor(cursor)
cursor.close()
return thing
def create_dict_from_cursor(cursor):
rows = cursor.fetchall()
# DEBUG settings (used to) affect what gets returned.
if DEBUG:
desc = [item[0] for item in cursor.cursor.description]
else:
desc = [item[0] for item in cursor.description]
return [dict(zip(desc, item)) for item in rows]
cheers, Simon.
You have to use the connection utility in Django:
from django.db import connection
with connection.cursor() as cursor:
cursor.execute("SQL STATEMENT CAN BE ANYTHING")
data = cursor.fetchone()
If you are expecting more than one row, use cursor.fetchall() to fetch a list of them.
More info here: http://docs.djangoproject.com/en/dev/topics/db/sql/
Don't.
Seriously.
Move the stored procedure logic into your model where it belongs.
Putting some code in Django and some code in the database is a maintenance nightmare. I've spent too many of my 30+ years in IT trying to clean up this kind of mess.
There is a good example :
https://djangosnippets.org/snippets/118/
from django.db import connection
cursor = connection.cursor()
ret = cursor.callproc("MY_UTIL.LOG_MESSAGE", (control_in, message_in))# calls PROCEDURE named LOG_MESSAGE which resides in MY_UTIL Package
cursor.close()
If you want to look at an actual running project that uses SP, check out minibooks. A good deal of custom SQL and uses Postgres pl/pgsql for SP. I think they're going to remove the SP eventually though (justification in trac ticket 92).
I guess the improved raw sql queryset support in Django 1.2 can make this easier as you wouldn't have to roll your own make_instance type code.
Cx_Oracle can be used. Also, It is fairly helpful when we do not have access to production deployed code and need arises to make major changes in database.
import cx_Oracle
try:
db = dev_plng_con
con = cx_Oracle.connect(db)
cur = con.cursor()
P_ERROR = str(error)
cur.callproc('NAME_OF_PACKAGE.PROCEDURENAME', [P_ERROR])
except Exception as error:
error_logger.error(message)