is there a way how to fully load some SQLAlchemy ORM mapped instance (together with its related objects) before detaching it from the Session? I want to send it via pipe into another processs and I don't want to merge it into session in this new process.
Thank you
Jan
I believe you'll want to use the options() method on the Query, with eagerload() or eagerload_all().
Here's an example of use from one of our apps, where the class Controlled has a relation called changes which brings in a bunch of DocumentChange records, which themselves have a relation dco that brings in one Dco object per instance. This is a two-level eager-load, thus the use of the eagerload_all(). We're using the declarative extension (in case that matters) and m.Session is a "scoped" (thread-local) session.
from sqlalchemy.orm import eagerload, eagerload_all
...
controlled_docs = (m.Session.query(m.Controlled)
.options(eagerload_all('changes.dco'))
.order_by('number')
.all())
If that's not sufficient, perhaps include a snippet or text showing how the relevant ORM classes are related and I could update the answer to show how those options would be used in your case.
Related
So I have two table in a one-to-many relationship. When I make a new row of Table1, I want to populate Table2 with the related rows. However, this population actually involves computing the Table2 rows, using data in other related tables.
What's a good way to do that using the ORM layer? That is, assuming that that the Table1 mappings are created through the ORM, where/how should I call the code to populate Table2?
I thought about using the after_insert hook, but i want to have a session to pass to the population method.
Thanks.
You can use the before_flush or after_flush hook, it provides a session. You then check session.new objects for newly created models (tip: use isinstance(object, ModelClass)) and do your work here.
In fact, SQLAlchemy recommends before_flush for general on flush changes.
Mapper-level flush events only allow very limited operations, on attributes local to the row being operated upon only, as well as allowing any SQL to be emitted on the given Connection. Please read fully the notes at Mapper-level Events for guidelines on using these methods; generally, the SessionEvents.before_flush() method should be preferred for general on-flush changes.
After asking around in #sqlalchemy IRC, it was pointed out that this could be done using ORM-level relationships in an before_flush event listener.
It was explained that when you add a mapping through a relationship, the foreign key is automatically filled on flush, and the appropriate insert statement generated by the ORM.
Background
The get() method is special in SQLAlchemy's ORM because it tries to return objects from the identity map before issuing a SQL query to the database (see the documentation).
This is great for performance, but can cause problems for distributed applications because an object may have been modified by another process, so the local process has no ability to know that the object is dirty and will keep retrieving the stale object from the identity map when get() is called.
Question
How can I force get() to ignore the identity map and issue a call to the DB every time?
Example
I have a Company object defined in the ORM.
I have a price_updater() process which updates the stock_price attribute of all the Company objects every second.
I have a buy_and_sell_stock() process which buys and sells stocks occasionally.
Now, inside this process, I may have loaded a microsoft = Company.query.get(123) object.
A few minutes later, I may issue another call for Company.query.get(123). The stock price has changed since then, but my buy_and_sell_stock() process is unaware of the change because it happened in another process.
Thus, the get(123) call returns the stale version of the Company from the session's identity map, which is a problem.
I've done a search on SO(under the [sqlalchemy] tag) and read the SQLAlchemy docs to try to figure out how to do this, but haven't found a way.
Using session.expire(my_instance) will cause the data to be re-selected on access. However, even if you use expire (or expunge), the next data that is fetched will be based on the transaction isolation level. See the PostgreSQL docs on isolations levels (it applies to other databases as well) and the SQLAlchemy docs on setting isolation levels.
You can test if an instance is in the session with in: my_instance in session.
You can use filter instead of get to bypass the cache, but it still has the same isolation level restriction.
Company.query.filter_by(id=123).one()
I have a script which cleans the database, and this is widely used in our tests.
First, we tried to use SQLAlchemy Metadata.drop_all() thing, but it didn't resolve some foreign keys on deletion, which caused errors. Then, I found this script from #zzzeek, which does almost the same, but in a "smart" way. It handles all the issues with foreign keys, but now there are several issues regarding changed custom types (ENUMs). The question is, how can I drop them all them using SQLAlchemy? Execute DROP TYPE by hand only?
Tables in database are created with Alembic, and even while script above deletes all tables successfully, some custom ENUMs are still there and everything fails on attempt to recreate them.
Recreating the whole database is not a preferred solution, because default DB user for application shouldn't normally have rights to create databases.
Are you sure your Metadata instance fully describes all the tables?
Try:
Metadata.reflect()
Metadata.drop_all()
This is an ancient question, but it's still possible to come across this problem if create_type=False is on any postgresql.ENUM definitions.
Per the SQLAlchemy docs on create_type,
When False, no check will be performed and no CREATE TYPE or DROP TYPE is emitted, unless ENUM.create() or ENUM.drop() are called directly.
That means when running tests, while there may be a setup & teardown with create_all() and drop_all(), neither will affect custom enum types.
The solution is simply to remove create_type=False, since True is the default. Then all custom types will be created at the beginning of testing and dropped at the end, resulting in a perfectly clean test database.
I am new in both flask and sqlalchemy, I just start working on a flask app, and I am using sqlalchemy for now. I was wondering if there is any significant benefit I can get from using flask-sqlalchemy vs sqlalchemy. I could not find enough motivations in http://packages.python.org/Flask-SQLAlchemy/index.html or maybe I did not understand the value!! I would appreciate your clarifications.
The main feature of the Flask-SQLAlchemy is proper integration with Flask application - it creates and configures engine, connection and session and configures it to work with the Flask app.
This setup is quite complex as we need to create the scoped session and properly handle it according to the Flask application request/response life-cycle.
In the ideal world that would be the only feature of Flask-SQLAlchemy, but actually, it adds few more things. Check out the docs for more info. Or see this blog post with the overview of them: Demystifying Flask-SQLAlchemy (update: the original article is not available at the moment, there is a snapshot on webarchive).
When I first worked with Flask and SQLAlchemy, I didn't like this overhead . I went over and extracted the session management code from the extension. This approach works, although I discovered that it is quite difficult to do this integration properly.
So the easier approach (which is used in another project I am working on) is to just drop the Flask-SQLAlchemy in and don't use any of additional features it provides. You will have the db.session and you can use it as if it was pure SQLAlchemy setup.
Flask-SQLAlchemy gives you a number of nice extra's you would else end up implementing yourself using SQLAlchemy.
Positive sides on using Flask-SQLAlchemy
Flask_SQLAlchemy handles session configuration, setup and teardown for you.
Gives you declarative base model that makes querying and pagination easier
Backend specific settings.Flask-SQLAlchemy scans installed libs for Unicode support and if fails automatically uses SQLAlchemy Unicode.
Has a method called apply_driver_hacks that automatically sets sane defaults to thigs like MySQL pool-size
Has nice build in methods create_all() and drop_all() for creating and dropping all tables. Useful for testing and in python command line if you did something stupid
It gives you get_or_404()instead of get() and find_or_404() instead of find() Code example at > http://flask-sqlalchemy.pocoo.org/2.1/queries/
Automatically set table names. Flask-SQLAlchemy automatically sets your table names converting your ClassName > class_name this can be overridden by setting __tablename__ class
List item
Negative sides on using Flask-SQLAlchemy
Using Flask-SQLAlchemy will make add additional difficulties to for
migrating from Flask to let's say Pyramid if you ever need to. This is mainly due to the custom declarative base model on Flask_SQLAchemy.
Using Flask-SQLAlchemy you risk using a package with a much smaller community than SQLAlchemy itself, which I cannot easily drop from active development any time soon.
Some nice extras Flask-SQLAlchemy has can make you confused if you do not know they are there.
To be honest, I don't see any benefits. IMHO, Flask-SQLAlchemy creates an additional layer you don't really need. In our case we have a fairly complex Flask application with multiple databases/connections (master-slave) using both ORM and Core where, among other things, we need to control our sessions / DB transactions (e.g. dryrun vs commit modes). Flask-SQLAlchemy adds some additional functionality such as automatic destruction of the session assuming some things for you which is very often not what you need.
The SQLAlchemy documentation clearly states that you should use Flask-SQLAlchemy (especially if you don't understand its benefits!):
[...] products such as Flask-SQLAlchemy [...] SQLAlchemy strongly recommends that these products be used as available.
This quote and a detailed motivation you can find in the second question of the Session FAQ.
As #schlamar suggests, Flask-SqlAlchemy is definitely a good thing. I'd just like to add some extra context to the point made there.
Don't feel like you are choosing one over the other. For example, let's say we want to grab all records from a table using a model using Flask-Sqlalchemy. It is as simple as
Model.query.all()
For a lot of the simple cases, Flask-Sqlalchemy is going to be totally fine. The extra point that I would like to make is, if Flask-Sqlalchemy is not going to do what you want, then there's no reason you can't use SqlAlchemy directly.
from myapp.database import db
num_foo = db.session.query(func.count(OtherModel.id)).filter(is_deleted=False).as_scalar()
db.session.query(Model.id, num_foo.label('num_foo')).order_by('num_foo').all()
As you can see, we can easily jump from one to the other with no trouble and in the second example we are in fact using the Flask-Sqlalchemy defined models.
Here is an example of a benefit flask-sqlalchemy gives you over plain sqlalchemy.
Suppose you're using flask_user.
flask_user automates creation and authentication of user objects, so it needs to access your database. The class UserManager does this by calling through to something called an "adapter" which abstracts the database calls. You provide an adapter in the UserManager constructor, and the adapter must implement these functions:
class MyAdapter(DBAdapter):
def get_object(self, ObjectClass, id):
""" Retrieve one object specified by the primary key 'pk' """
pass
def find_all_objects(self, ObjectClass, **kwargs):
""" Retrieve all objects matching the case sensitive filters in 'kwargs'. """
pass
def find_first_object(self, ObjectClass, **kwargs):
""" Retrieve the first object matching the case sensitive filters in 'kwargs'. """
pass
def ifind_first_object(self, ObjectClass, **kwargs):
""" Retrieve the first object matching the case insensitive filters in 'kwargs'. """
pass
def add_object(self, ObjectClass, **kwargs):
""" Add an object of class 'ObjectClass' with fields and values specified in '**kwargs'. """
pass
def update_object(self, object, **kwargs):
""" Update object 'object' with the fields and values specified in '**kwargs'. """
pass
def delete_object(self, object):
""" Delete object 'object'. """
pass
def commit(self):
pass
If you're using flask-sqlalchemy, you can use the built-in SQLAlchemyAdapter. If you're using sqlalchemy (not-flask-sqlalchemy) you might make different assumptions about the way in which objects are saved to the database (like the names of the tables) so you'll have to write your own adapter class.
Yes, very basic question.
I've successfully created my db using declarative_base, and can perform inserts into the db too. I just have a few questions about SqlAlchemy sql statements.
I've create a table called Location.
A few issues/questions (see code below):
For statement, "print row", I have to specify each column name that I want to have output. i.e. "print row.name, row.lat, etc" Why? (Otherwise the print statement outputs "<classname.Location at <...>>"
Also, what is the preferred way to interact with a db and perform queries (select, insert, update, etc.)- there seem to be a bunch of options: using sqlalchemy.orm.select for example, or engine.text(<sql query>).execute().fetchall(), or even conn.execute(<select>). Options are great, but right now they're all just confusing me.
Thanks so much for the tips!
Here's my code:
from sqlalchemy import create_engine
from sqlalchemy.sql import select
from location_db_setup import *
db_path = "sqlite:////volumes/users/shared/programming/python/web/map.db"
engine = create_engine(db_path, echo= True)
Session = sessionmaker(bind= engine)
session = Session()
session.query(Location).fetchall()
for row in locations:
print row
You code in sample is incomplete and has errors. So it's impossible to say for sure what is Location here. I assume it's a mapped class, so you are requesting a list of all Location objects, not rows. When you print an object you get its string representation. String representation of objects can be changed by defining custom __str__ method.
Although ORM is the most important part of SQLAlchemy, it's not the only. It also expose a lot of functionality not related to ORM directly. When you work with objects the preferred way to create queries are corresponding session method. But sometimes you need selectable objects not bound to particular session (they are not executed directly, but are used in expressions passed to session methods). That's why there are functions in sqlalchemy.orm package.
The preferred way to interact with a db when using an ORM is not to use queries but to use objects that correspond to the tables you are manipulating, typically in conjunction with the session object. SELECT queries become get() or find() calls in some ORMs, query() calls in others. INSERT becomes creating a new object of the type you want (and maybe explicitly adding it, eg session.add() in sqlalchemy). UPDATE becomes editing such an object, and DELETE becomes deleting an object (eg. session.delete() ). The ORM is meant to handle the hard work of translating these operations into SQL for you.
Have you read the tutorial?
Denis and Kylotan gave you good answers. I'm just gonna focus on point 2.
Sometimes depends on your taste. There are times when you need database specific features that an ORM can't do, that's a case when you should use Session(<sql here>).execute() or conn.execute(<sql here>). Another case is when you have a very complex query which is beyond you and you don't find a suitable ORM expression.
Usually, using ORM features like select([...]).where(... or Session.query(<Model here>).filter(... (declarative base) are enough. Almost every sql query has an ORM equivalent.