I am new in both flask and sqlalchemy, I just start working on a flask app, and I am using sqlalchemy for now. I was wondering if there is any significant benefit I can get from using flask-sqlalchemy vs sqlalchemy. I could not find enough motivations in http://packages.python.org/Flask-SQLAlchemy/index.html or maybe I did not understand the value!! I would appreciate your clarifications.
The main feature of the Flask-SQLAlchemy is proper integration with Flask application - it creates and configures engine, connection and session and configures it to work with the Flask app.
This setup is quite complex as we need to create the scoped session and properly handle it according to the Flask application request/response life-cycle.
In the ideal world that would be the only feature of Flask-SQLAlchemy, but actually, it adds few more things. Check out the docs for more info. Or see this blog post with the overview of them: Demystifying Flask-SQLAlchemy (update: the original article is not available at the moment, there is a snapshot on webarchive).
When I first worked with Flask and SQLAlchemy, I didn't like this overhead . I went over and extracted the session management code from the extension. This approach works, although I discovered that it is quite difficult to do this integration properly.
So the easier approach (which is used in another project I am working on) is to just drop the Flask-SQLAlchemy in and don't use any of additional features it provides. You will have the db.session and you can use it as if it was pure SQLAlchemy setup.
Flask-SQLAlchemy gives you a number of nice extra's you would else end up implementing yourself using SQLAlchemy.
Positive sides on using Flask-SQLAlchemy
Flask_SQLAlchemy handles session configuration, setup and teardown for you.
Gives you declarative base model that makes querying and pagination easier
Backend specific settings.Flask-SQLAlchemy scans installed libs for Unicode support and if fails automatically uses SQLAlchemy Unicode.
Has a method called apply_driver_hacks that automatically sets sane defaults to thigs like MySQL pool-size
Has nice build in methods create_all() and drop_all() for creating and dropping all tables. Useful for testing and in python command line if you did something stupid
It gives you get_or_404()instead of get() and find_or_404() instead of find() Code example at > http://flask-sqlalchemy.pocoo.org/2.1/queries/
Automatically set table names. Flask-SQLAlchemy automatically sets your table names converting your ClassName > class_name this can be overridden by setting __tablename__ class
List item
Negative sides on using Flask-SQLAlchemy
Using Flask-SQLAlchemy will make add additional difficulties to for
migrating from Flask to let's say Pyramid if you ever need to. This is mainly due to the custom declarative base model on Flask_SQLAchemy.
Using Flask-SQLAlchemy you risk using a package with a much smaller community than SQLAlchemy itself, which I cannot easily drop from active development any time soon.
Some nice extras Flask-SQLAlchemy has can make you confused if you do not know they are there.
To be honest, I don't see any benefits. IMHO, Flask-SQLAlchemy creates an additional layer you don't really need. In our case we have a fairly complex Flask application with multiple databases/connections (master-slave) using both ORM and Core where, among other things, we need to control our sessions / DB transactions (e.g. dryrun vs commit modes). Flask-SQLAlchemy adds some additional functionality such as automatic destruction of the session assuming some things for you which is very often not what you need.
The SQLAlchemy documentation clearly states that you should use Flask-SQLAlchemy (especially if you don't understand its benefits!):
[...] products such as Flask-SQLAlchemy [...] SQLAlchemy strongly recommends that these products be used as available.
This quote and a detailed motivation you can find in the second question of the Session FAQ.
As #schlamar suggests, Flask-SqlAlchemy is definitely a good thing. I'd just like to add some extra context to the point made there.
Don't feel like you are choosing one over the other. For example, let's say we want to grab all records from a table using a model using Flask-Sqlalchemy. It is as simple as
Model.query.all()
For a lot of the simple cases, Flask-Sqlalchemy is going to be totally fine. The extra point that I would like to make is, if Flask-Sqlalchemy is not going to do what you want, then there's no reason you can't use SqlAlchemy directly.
from myapp.database import db
num_foo = db.session.query(func.count(OtherModel.id)).filter(is_deleted=False).as_scalar()
db.session.query(Model.id, num_foo.label('num_foo')).order_by('num_foo').all()
As you can see, we can easily jump from one to the other with no trouble and in the second example we are in fact using the Flask-Sqlalchemy defined models.
Here is an example of a benefit flask-sqlalchemy gives you over plain sqlalchemy.
Suppose you're using flask_user.
flask_user automates creation and authentication of user objects, so it needs to access your database. The class UserManager does this by calling through to something called an "adapter" which abstracts the database calls. You provide an adapter in the UserManager constructor, and the adapter must implement these functions:
class MyAdapter(DBAdapter):
def get_object(self, ObjectClass, id):
""" Retrieve one object specified by the primary key 'pk' """
pass
def find_all_objects(self, ObjectClass, **kwargs):
""" Retrieve all objects matching the case sensitive filters in 'kwargs'. """
pass
def find_first_object(self, ObjectClass, **kwargs):
""" Retrieve the first object matching the case sensitive filters in 'kwargs'. """
pass
def ifind_first_object(self, ObjectClass, **kwargs):
""" Retrieve the first object matching the case insensitive filters in 'kwargs'. """
pass
def add_object(self, ObjectClass, **kwargs):
""" Add an object of class 'ObjectClass' with fields and values specified in '**kwargs'. """
pass
def update_object(self, object, **kwargs):
""" Update object 'object' with the fields and values specified in '**kwargs'. """
pass
def delete_object(self, object):
""" Delete object 'object'. """
pass
def commit(self):
pass
If you're using flask-sqlalchemy, you can use the built-in SQLAlchemyAdapter. If you're using sqlalchemy (not-flask-sqlalchemy) you might make different assumptions about the way in which objects are saved to the database (like the names of the tables) so you'll have to write your own adapter class.
Related
I have this code:
def advertiser_table(engine):
return Table('advertiser', metadata, autoload=True, autoload_with=engine)
And later I try this:
advertisers = advertiser_table(engine)
...
session.bulk_insert_mappings(
advertisers.name,
missing_advetisers.to_dict('records'),
)
where missing_adverisers is a Pandas DataFrame (but it's not important for this question).
The error this gives me is:
sqlalchemy.orm.exc.UnmappedClassError: Class ''advertiser'' is not mapped
From reading the documentation I could scramble enough to ask the question, but not much more than that... What is Mapper and why is it so detrimental to the functioning of this library?.. Why isn't "the class" mapped? Obviously, what am I to do to "map" it to whatever this library wants it to map?
A Mapper is the M in ORM. It is the thing that maps your table (advertisers in this case) to instances of a class (which you are missing in this case) in order for you to operate on it.
The reason it's confusing for you is because SQLAlchemy is actually two libraries in one -- one is called SQLAlchemy Core, and the other is the SQLAlchemy ORM. Core provides the ability to work with tables and to construct queries that return rows, while the ORM builds on top of Core to provide the ability to work with instances of classes and their relationships as an abstraction. Core roughly corresponds to things you can do on Connection and Engine, while ORM roughly corresponds to things you can do on Session.
So, all of that is to say, session.bulk_insert_mappings is an ORM functionality, and you cannot use it without having a mapped class.
What can you do instead? Use the equivalent Core functionality:
query = advertisers.insert().values(missing_advetisers.to_dict('records'))
engine.execute(query) # or session.execute(query)
Or even use the pandas-provided to_sql function:
missing_advetisers.to_sql("advertiser", engine, if_exists="append")
If you insist on using the ORM, you need to declare a mapped class for your table. The easiest way when using reflection is to use automap. The linked documentation has many examples, so I won't go into detail here.
I am writing a django project, that makes extensive use of a python package I'm writing (let's call it foo, for convenience).
The python package foo, will consist mainly of functions and classes that munge data obtained from a backend database. I want to write the package in such a way that it will have no dependency on django - and can be used in other projects outside of django.
I am thinking of writing the package so that functions accept a database connection - and classes uses IoC for the database connection - that way, I can obtain a database connection from django and pass it to the DBAL package - when using it in django, and instantiate a DB connection via other means when using the package outside django.
I have two questions:
Is this an acceptable way of approaching this problem (i.e. no gotchas)
Where/how do I obtain a database connection within django?
You can get a cursor object from Django like so:
from django.db import connection
def my_custom_sql(self):
with connection.cursor() as cursor:
cursor.execute("UPDATE bar SET foo = 1 WHERE baz = %s", [self.baz])
cursor.execute("SELECT foo FROM bar WHERE baz = %s", [self.baz])
row = cursor.fetchone()
return row
This example is taken from their docs for custom SQL queries here
In terms of being an acceptable way to write a package, sure. It's just a loose form of Dependency Injection and makes unit testing much easier, for one thing.
The only 'gotcha' is that the cursors passed in by users of your package may not present the same interface. SQLAlchemy cursors may not have the same properties and methods as Django ORM cursors.
I'd say ensuring that the object conforms to your required interface is an acceptable burden to push onto users, as long as you thoroughly document what that object should expose.
You can always provide custom workarounds for the most popular options in future, if you find it would be valuable to do so. (e.g. provide your own cursor adaptor classes).
I'm experiencing an issue with the Django ORM in our current application after upgrading from Django 1.4 to Django 1.6 recently. This issue is only showing up while running tests, not while running in development or production environments.
More specifically, we have a post_save hook which collects information from tables which subclass our main data object and coalesce the data stored in those objects into a SearchDocument object which is then stored and queried in order to provide full-text searching on the interface. This works in both development and production environments, but when attempting to run unit tests, we have issues where tests are failing.
Our method, which is in the process of being debugged, looks something like this right now:
def _update_search_document( self, doc ):
"""Updates and saves an existing search document for this model.
This will sync the search text and key/value attributes."""
# A bunch of code updating the model properties prior to saving.
doc.save()
print SearchDocument.objects.all()
print connection.queries:
This code saves correctly, but the subsequent call to SearchDocument.objects.all() returns [] and the connection.queries property contains the INSERT SQL from the doc.save() call but does not contain (at any point) SELECT SQL from any of the three extant SearchDocument SELECT queries which should have been performed to that point.
To the best of my knowledge, this code was working prior to our migration from Django 1.4 to Django 1.6. I know that some of the query logic changed between 1.4 and 1.6 and I'm wondering if there's a setting (perhaps surrounding caching?) that's causing problems with our tests now.
Edit: After a bit more hacking, I've been able to determine that moving this test class to a SimpleTestCase instead of the TestCase class which subclasses TransactionalTestCase (which wraps each test in a DB transaction) resolves the issue that I was seeing with items not showing up after they've been saved. It appears that something in that transaction management is causing the problems with the saving.
Quoting my edit because this is how we eventually solved it:
After a bit more hacking, I've been able to determine that moving this test class to a SimpleTestCase instead of the TestCase class which subclasses TransactionalTestCase (which wraps each test in a DB transaction) resolves the issue that I was seeing with items not showing up after they've been saved. It appears that something in that transaction management is causing the problems with the saving.
I have a Flask application with which I'd like to use SQLAlchemy Core (i.e. I explicitly do not want to use an ORM), similarly to this "fourth way" described in the Flask doc:
http://flask.pocoo.org/docs/patterns/sqlalchemy/#sql-abstraction-layer
I'd like to know what would be the recommended pattern in terms of:
How to connect to my database (can I simply store a connection instance in the g.db variable, in before_request?)
How to perform reflection to retrieve the structure of my existing database (if possible, I'd like to avoid having to explicitly create any "model/table classes")
Correct: You would create a connection once per thread and access it using a threadlocal variable. As usual, SQLAlchemy has thought of this use-case and provided you with a pattern: Using the Threadlocal Execution Strategy
db = create_engine('mysql://localhost/test', strategy='threadlocal')
db.execute('SELECT * FROM some_table')
Note: If I am not mistaken, the example seems to mix up the names db and engine (which should be db as well, I think).
I think you can safely disregard the Note posted in the documentation as this is explicitly what you want. As long as each transaction scope is linked to a thread (as is with the usual flask setup), you are safe to use this. Just don't start messing with threadless stuff (but flask chokes on that anyway).
Reflection is pretty easy as described in Reflecting Database Objects. Since you don't want to create all the tables manually, SQLAlchemy offers a nice way, too: Reflecting All Tables at Once
meta = MetaData()
meta.reflect(bind=someengine)
users_table = meta.tables['users']
addresses_table = meta.tables['addresses']
I suggest you check that complete chapter concerning reflection.
is there a way how to fully load some SQLAlchemy ORM mapped instance (together with its related objects) before detaching it from the Session? I want to send it via pipe into another processs and I don't want to merge it into session in this new process.
Thank you
Jan
I believe you'll want to use the options() method on the Query, with eagerload() or eagerload_all().
Here's an example of use from one of our apps, where the class Controlled has a relation called changes which brings in a bunch of DocumentChange records, which themselves have a relation dco that brings in one Dco object per instance. This is a two-level eager-load, thus the use of the eagerload_all(). We're using the declarative extension (in case that matters) and m.Session is a "scoped" (thread-local) session.
from sqlalchemy.orm import eagerload, eagerload_all
...
controlled_docs = (m.Session.query(m.Controlled)
.options(eagerload_all('changes.dco'))
.order_by('number')
.all())
If that's not sufficient, perhaps include a snippet or text showing how the relevant ORM classes are related and I could update the answer to show how those options would be used in your case.