Develop Reference

What is the type of table_name in select_from(table_name)?

I'm trying to get the number of rows from a SQL Server which consists of many tables by looping through it to see how much data they contain. However, I'm not sure what will go into select_from() function. As I currently supply Unicode for table names and it raised
NoInspectionAvailable: No inspection system is available for object of type <type 'unicode'>
The code that I used was
from sqlalchemy import create_engine
import urllib
from sqlalchemy import inspect
import sqlalchemy
from sqlalchemy import select, func, Integer, Table, Column, MetaData
from sqlalchemy.orm import sessionmaker
connection_string = "DRIVER={SQL Server}; *hidden*"
connection_string = urllib.quote_plus(connection_string)
connection_string = "mssql+pyodbc:///?odbc_connect=%s" % connection_string
engine = sqlalchemy.create_engine(connection_string)
Session = sessionmaker()
Session.configure(bind=engine)
session = Session()
connection = engine.connect()
inspector = inspect(engine)
for table_name in inspector.get_table_names():
print session.query(func.count('*')).select_from(table_name).scalar()

Typically, it's a class name that refers to a class that describes the database table.
In the sqlalchemy docs, http://docs.sqlalchemy.org/en/latest/orm/tutorial.html, they have you create a base class using declarative base and then create child classes for each table you want to query. You would then pass that class name into the select_from function unquoted.
The Flask framework provides a built-in base class that is ready for use called db.Model and Django has one called models.Model.
Alternatively, you can also pass queries. I use the Flask framework typically for python so I usually initiate queries like this:
my_qry = db.session.query(Cust).filter(Cust, Cust.cust == 'lolz')
results = my_qry.all()
On a side note, if you decide to look at .NET they also have nice ORMs. Personally, I favor Entity Framework, but Linq to SQL is out there, too.

Flask-Blogging error Table is already defined

I'm using a flask setup from a while and now trying to install Flask-Blogging module on it. Current modules:
- Flask-sqlalchemy with postgres
- Flask-login
- Flask-Blogging (new)
My application.py looks like this:
from flask import Flask
from flask import session
from flask.ext.blogging import SQLAStorage, BloggingEngine
from flask.ext.login import LoginManager
from flask.ext.sqlalchemy import SQLAlchemy
'''
The main application setup. The order of things is important
in this file.
'''
app = Flask(__name__)
app.config.from_object('config.base')
app.config.from_envvar('APP_CONFIG_FILE')
'''
Initialize database
'''
db = SQLAlchemy(app)
'''
Initialize blogger
'''
storage = SQLAStorage(db=db)
blog_engine = BloggingEngine(app, storage)
the last two lines are the only new things I added (other than the imports). Suddenly now I'm getting error about duplicate table names:
sqlalchemy.exc.InvalidRequestError: Table 'customer' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
Any ideas what am I doing wrong? I couldn't find much documentation about Flask-Blogging other than:
http://flask-blogging.readthedocs.org/en/latest/

You get this error because in SQLAStorage.__init__ there is this line:
self._metadata.reflect(bind=self._engine)
This will look at your database and create sqlalchemy table things for all the tables currently in your database.
Thus if your database contains a table called 'customer' the line in your code:
storage = SQLAStorage(db=db)
will automatically model an sqlalchemy table called 'customer' for you.
Now... no doubt you have your own database model definitions somewhere, probably in another python module, something like:
class Customer(db.Model):
id = db.Column(db.Integer, primary_key=True)
...
Since this class definition defines a table called 'customer' and since SQLAStorage has already defined a table called 'customer' you get the exception as soon as your class Customer is imported.
Some ways to work around this problem are:
Import your database definition modules before instantiating SQLAStorage
'''
Initialize database
'''
db = SQLAlchemy(app)
import ankit.db_models # import my db models here so SQLAStorage doesn't do it first
'''
Initialize blogger
'''
storage = SQLAStorage(db=db)
blog_engine = BloggingEngine(app, storage)
or
Tell SQLAStorage to use its own metadata
By passing the db param to SQLAStorage.__init__ you are telling it to use your metadata. You can instead just pass the engine parameter and it will create its own metadata.
storage = SQLAStorage(engine=db.engine)

SQLAlchemy freezing application

I am using SQLAlchemy in my python command line app. The app is basically reading a set of URLs and doing inserts into a postgreql database based on the data.
After about the same number of inserts (give or take a few), the entire app freezes.
Having seen python sqlalchemy + postgresql program freezes I am assuming I am doing something wrong with the SQLAlchemy Session (although I am not using drop_all(), which seemed to be the cause of that issue). I've tried a couple of things but thus far they have had no impact.
Any hints or help would be welcome. If my integration of SQLAlchemy into my app is incorrect, a pointer to a good example of doing it right would also be welcome.
My code is as follows:
Set up the sql alchemy base:
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
Create the session info and attach it to the Base
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
engine = create_engine("sqlite:///myapp.db")
db_session = scoped_session(sessionmaker(bind=engine))
Base.query = db_session.query_property()
Base.scoped_db_session = db_session
Create my model from Base and make use of the session
class Person(Base):
def store(self):
if self.is_new():
self.scoped_db_session.add(self)
self.scoped_db_session.commit()
If I create enough objects of type Person and call store(), the app eventually freezes.

Managed to solve the problem. Turns out that my implementation is specifically on the don't do it this way list (see http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#session-frequently-asked-questions E.g. don't do this) and I was not managing the session correctly
To solve my problem I moved the session out of the model into a separate class, so instead of having calls like:
mymodel.store()
I now have:
db.current_session.store(mymodel)
where db is an instance of my custom DBContext below:
from contextlib import contextmanager
from sqlalchemy.orm import scoped_session, sessionmaker
class DbContext(object):
def __init__(self, engine, session=None):
self._engine = engine
self._session = session or scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=self._engine))
self.query = self._session.query_property()
self.current_session = None
def start_session(self):
self.current_session = self._session()
def end_session(self):
if self.current_session:
self.current_session.commit()
self.current_session.close()
self.current_session = None
#contextmanager
def new_session(self):
try:
self.start_session()
yield
finally:
self.end_session()
When you want to store one or more model objects, call DBContext.start_session() to start a clean session. When you finally want to commit, call DBContext.end_session().

SqlAlchemy add tables versioning to existing tables

Imagine that I have one table in my project with some rows in it.
For example:
# -*- coding: utf-8 -*-
import sqlalchemy as sa
from app import db
class Article(db.Model):
__tablename__ = 'article'
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
name = sa.Column(sa.Unicode(255))
content = sa.Column(sa.UnicodeText)
I'm using Flask-SQLAlchemy, so db.session is scoped session object.
I saw in https://github.com/zzzeek/sqlalchemy/blob/master/examples/versioned_history/history_meta.py
but i can't understand how to use it with my existing tables and anymore how to start it. (I get ArgumentError: Session event listen on a scoped_session requires that its creation callable is associated with the Session class. error when I pass db.session in versioned_session func)
From versioning I need the following:
1) query for old versions of object
2) query old versions by date range when they changed
3) revert old state to existing object
4) add additional info to history table when version is creating (for example editor user_id, date_edit, remote_ip)
Please, tell me what are the best practicies for my case and if you can add a little working example for it.

You can work around that error by attaching the event handler to the SignallingSession class[1] instead of the created session object:
from flask.ext.sqlalchemy import SignallingSession
from history_meta import versioned_session, Versioned
# Create your Flask app...
versioned_session(SignallingSession)
db = SQLAlchemy(app)
class Article(Versioned, db.Model):
__tablename__ = 'article'
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
name = sa.Column(sa.Unicode(255))
content = sa.Column(sa.UnicodeText)
The sample code creates parallel tables with a _history suffix and an additional changed datetime column. Querying for old versions is just a matter of looking in that table.
For managing the extra fields, I would put them on your main table, and they'll automatically be kept track of in the history table.
[1] Note, if you override SQLAlchemy.create_session() to use a different session class, you should adjust the class you pass to versioned_session.

I think the problem is you're running into this bug: https://github.com/mitsuhiko/flask-sqlalchemy/issues/182
One workaround would be to stop using flask-sqlalchemy and configure sqlalchemy yourself.

How to structure a CherryPy app into a MVC architecture?

I was planning for my structure to be something similar to this:
appname/libs/user
/football
/tax
Where the library user, will have the models for the User, the controller which would show up the REST JSON API and the views.
The problems that I'm facing currently can be divided into two major questions, and which mainly stem from using Django for a while. I'm fairly new to CherryPy and SqlAlchemy.
How to define modles in each of this library? The problem I face is I've to inherit the Base Declarative and Engine in every model and run it as a standalone app for its models to be generated. Is there a mechanism where I can plug in the libraries and the database should pull all the models and create it? (Something that Django does.)
How to define routes/apis? (a urls.py)

How about defining the declarative base (sqlalchemy) in appname/db/__init__.py and for each of the libs import the base from appname in appname/libs/NAME/models.py:
import appname.db
Base = appname.db.Base
class MyUser(Base):
...
To get a database session just use a scoped session for example, this could be the basic content for appname/db/__init__.py (or just db.py if you don't want to define additional base models in appname/db/models.py)
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy import engine_from_config
__all__ = ['session', 'ses']
ses = session = scoped_session(sessionmaker())
def load_engine(config):
engine = engine_from_config(config, prefix='')
session.configure(bind=engine)
Then set a tool to remove the session from the thread locals when the request has ended:
import cherrypy
from appname import db
def remove_db_session():
db.session.remove()
cherrypy.tools.removedbs = cherrypy.Tool('on_end_request', remove_db_session)
from that point forward just use the session as normal from any part of your application, for example:
from appname import db
from appname.libs.user import models
class User:
exposed = True
def GET(self, id):
db.ses.query(models.User).filter_by(id=id)
# and let the tool to remove the session
# when the request finish
By the way to enable that removedbs tool, just make sure that you execute somewhere that cherrypy.tools.removedbs = .... I usually put that in: appname/cptools/tool and then in the config dictionary or file set tools.removedbs.on to True
Working with cherrypy means that you will build the application tree, there is no extra magic you need to have a central place to build the full tree, if you want to use the MethodDispatcher or the DefaultDispatcher.
In this case I recommend you the MethodDispatcher, and probably this post can give you a little more perspective and this is from my blog emulating the github api without any base handler.
There is an alternative to use more django like routes with the RoutesDispatcher, but you will lose a lot of functionality from the tools.
To show you an example with the MethodDispatcher and building your own object tree, from the current structure you can have a build function on the appname/builder.py and make something like this:
from appname.views import Main
from appname.libs import user, football
appmap = {'user': user,
'footbal': football}
def build_app(*apps):
root = Main()
for app in apps:
appmodule = appmap[app]
appmodule.attach(root)
return root
And inside the appname/libs/user/__init__.py
from appname.libs.user import views
def build_tree():
root = views.Main()
root.management = views.Management()
return root
def attach(root):
root.user = build_tree()
That's just a way to structure the application, there is also a repository with cherrypy recipes which are pretty helpful.