it's the first time i am using this environment.
The part of SQLAlchemy i am willing to use is just the one that allows me to query the database using Table objects with autoload = True. I am doing this as my tables already exist in the DB (mysql server) and were not created by defining flask models.
I have gone through all the documentation and i don't seem to find an answer. Here is some code:
app = Flask(__name__)
app.config.from_object(__name__)
metadata = None
def connect_db():
engine = create_engine(app.config['DATABASE_URI'])
global metadata
metadata = MetaData(bind=engine)
return engine.connect()
#app.before_request
def before_request():
g.db = connect_db()
#app.teardown_request
def teardown_request(exception):
g.db.close()
Now you could be wondering why i use that global var named metadata. Ok some more code:
#app.route('/test/<int:id>')
def test(test_result_id):
testTable = Table('test_table', metadata , autoload=True)
As you can see i need that object to be global in order to access it from within a function.
Also I am declaring the same var testTable in each function that needs it. I have the feeling this is not the right approach. I coudn't find any best practice advice for a case like mine.
Thanks all!
Have you seen this snippet in the SQLAlchemy docs?
Maybe this would work:
# This is fine as a global global
metadata = MetaData()
#app.before_first_request
def autoload_tables():
meta.reflect(bind=g.db.bind)
#app.route('/')
def index():
users_table = meta.tables['users']
That way your tables are reflected only once per process which is probably what you want. Note that your engine should be a global too, so you needn't create a new engine in #app.before_request - app creation is a more appropriate place.
If your case is very special you might need one engine per request, in which case you should consider the ThreadLocalMetaData class.
Related
I have an application running in production that I've built for a single client that I want to convert to support multiple "tenants".
Currently I am using a Postgres database where all my data resides in a single database in the default public schema. I would like to isolate each tenant to a separate Postgres schema. Ideally, my application's UI would make a call to my API using the tenant's subdomain. In before_request I would somehow be able to set all database queries during the current request context to only query that tenant's schema, is this possible?
I envisage an ideal solution to be something similar to this contrived example:
from flask import Flask, request, jsonify
from pony.orm import Database, Required
app = Flask(__name__)
db = Database(**{<db_connection_dict>})
class User(db.Entity):
email = Required(str)
password = Required(str)
#classmethod
def login(cls, email: str, password: str) -> str:
user = cls.get(lambda u: u.email.lower() == email.lower())
if not user:
return None
password_is_valid = <method_to_check_hashed_pasword>
if not password_is_valid:
return None
return <method_to_generate_jwt>
db.generate_mapping()
#app.before_request
def set_tenant():
tenant_subdomain = request.host.split(".")[0]
// MISSING STEP.. set_schema is a fictitous method, does something similar to this exist?
db.set_schema(schema=tenant_subdomain)??
#app.route("auth/login", methods=["POST"]
def login_route():
data = request.get_json()
jwt = User.login(data["email"], data["password"])
if not jwt:
return make_response({}, 403)
return make_response(jsonify(data=jwt), 200)
I've come across an interesting/simple example using SQLAlchemy. If not possible with PonyORM I may consider porting my models over to SQLAlchemy but would miss the simplicity of Pony :(
I thought about possibly using the Database.on_connect method to do something as such but not sure if if anyone has any other ideas or if this would even work properly in production. I suspect not because if I had two separate tenants querying the database they would overwrite the search path..
#db.on_connect()
def set_request_context_tenant_schema(db, connection) -> None:
subdomain = request.host.split(".")[0]
cursor = connection.cursor()
cursor.execute(f"SET search_path TO {subdomain}, public;")
I wonder how SQLAlchemy tracks changes that are made outside of SQLAlchemy (manual change for example)?
Until now, I used to put db.session.commit() before each value that can be changed outside of SQLAlchemy. Is this a bad practice? If yes, is there a better way to make sure I'll have the latest value? I've actually created a small script below to check that and apparently, SQLAlchemy can detect external changes without db.session.commit() being called each time.
Thanks,
P.S: I really want to understand how all the magics happen behind SQLAlchemy work. Does anyone has a pointer to some docs explaining the behind-the-scenes work of SQLAlchemy?
import os
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
# Use SQLlite so this example can be run anywhere.
# On Mysql, the same behaviour is observed
basedir = os.path.abspath(os.path.dirname(__file__))
db_path = os.path.join(basedir, "app.db")
app.config["SQLALCHEMY_DATABASE_URI"] = 'sqlite:///' + db_path
db = SQLAlchemy(app)
# A small class to use in the test
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(100))
# Create all the tables and a fake data
db.create_all()
user = User(name="old name")
db.session.add(user)
db.session.commit()
#app.route('/')
def index():
"""The scenario: the first request returns "old name" as expected.
Then, I modify the name of User:1 to "new name" directly on the database.
On the next request, "new name" will be returned.
My question is: how SQLAlchemy knows that the value has been changed?
"""
# Before, I always use db.session.commit()
# to make sure that the latest value is fetched.
# Without db.session.commit(),
# SQLAlchemy still can track change made on User.name
# print "refresh db"
# db.session.commit()
u = User.query.filter_by(id=1).first()
return u.name
app.run(debug=True)
The "cache" of a session is a dict in its identity_map (session.identity_map.dict) that only caches objects for the time of "a single business transaction" , as answered here https://stackoverflow.com/a/5869795.
For different server requests, you have different identity_map. It is not a shared object.
In your scenario, you requested the server 2 separated times. The second time, the identity_map is a new one (you can easily check it by printing out its pointer), and has nothing in cache. Consequently the session will request the database and get you the updated answer. It does not "track change" as you might think.
So, to your question, you don't need to do session.commit() before a query if you have not done a query for the same object in the same server request.
Hope it helps.
I'm trying to test my flask application using unittest. I want to refrain from flask-testing because I don't like to get ahead of myself.
I've really been struggling with this unittest thing now. It is confusing because there's the request context and the app context and I don't know which one I need to be in when I call db.create_all().
It seems like when I do add to the database, it adds my models to the database specified in my app module (init.py) file, but not the database specified in the setUp(self) method.
I have some methods that must populate the database before every test_ method.
How can I point my db to the right path?
def setUp(self):
#self.db_gd, app.config['DATABASE'] = tempfile.mkstemp()
app.config['TESTING'] = True
# app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + app.config['DATABASE']
basedir = os.path.abspath(os.path.dirname(__file__))
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + \
os.path.join(basedir, 'test.db')
db = SQLAlchemy(app)
db.create_all()
#self.app = app.test_client()
#self.app.testing = True
self.create_roles()
self.create_users()
self.create_buildings()
#with app.app_context():
# db.create_all()
# self.create_roles()
# self.create_users()
# self.create_buildings()
def tearDown(self):
#with app.app_context():
#with app.request_context():
db.session.remove()
db.drop_all()
#os.close(self.db_gd)
#os.unlink(app.config['DATABASE'])
Here is one of the methods that populates my database:
def create_users(self):
#raise ValueError(User.query.all())
new_user = User('Some User Name','xxxxx#gmail.com','admin')
new_user.role_id = 1
new_user.status = 1
new_user.password = generate_password_hash(new_user.password)
db.session.add(new_user)
Places I've looked at:
http://kronosapiens.github.io/blog/2014/08/14/understanding-contexts-in-flask.html
http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvi-debugging-testing-and-profiling
And the flask documentation:
http://flask.pocoo.org/docs/0.10/testing/
one issue that your hitting is the limitations of flask contexts, this is the primary reason i think long and hard before including a flask extension into my project, and flask-sqlalchemy is one of the biggest offenders. i say this because in most cases it is completely unnecessary to depend on the flask app context when dealing with your database. Sure it can be nice, especially since flask-sqlalchemy does a lot behind the scenes for you, mainly you dont have to manually manage your session, metadata or engine, but keeping that in mind those things can easily be done on your own, and for doing that you get the benefit of unrestricted access to your database, with no worry about the flask context. here is an example of how to setup your db manually, first i will show the flask-sqlalchemy way, then the manual plain sqlalchemy way:
the flask-sqlalchemy way:
import flask
from flask_sqlalchemy import SQLAlchemy
app = flask.Flask(__name__)
db = SQLAlchemy(app)
# define your models using db.Model as base class
# and define columns using classes inside of db
# ie: db.Column(db.String(255),nullable=False)
# then create database
db.create_all() # <-- gives error if not currently running flask app
the standard sqlalchemy way:
import flask
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
# first we need our database engine for the connection
engine = sa.create_engine(MY_DB_URL,echo=True)
# the line above is part of the benefit of using flask-sqlalchemy,
# it passes your database uri to this function using the config value
# SQLALCHEMY_DATABASE_URI, but that config value is one reason we are
# tied to the application context
# now we need our session to create querys with
Session = sa.orm.scoped_session(sa.orm.sessionmaker())
Session.configure(bind=engine)
session = Session()
# now we need a base class for our models to inherit from
Model = declarative_base()
# and we need to tie the engine to our base class
Model.metadata.bind = engine
# now define your models using Model as base class and
# anything that would have come from db, ie: db.Column
# will be in sa, ie: sa.Column
# then when your ready, to create your db just call
Model.metadata.create_all()
# no flask context management needed now
if you set your app up like that, any context issues your having should go away.
as a separate answer, to actually just force what you need to work, you can just use the test_request_context function, ie: in setup do: self.ctx = app.test_request_context() then just activate it, self.ctx.push() and when your done get rid of it, ie in tearDown: self.ctx.pop()
I am trying to separate some of my database logic into its own helper module. This is because I have several routes that perform the same database functions, and I don't want to keep repeating the same code. I'm a bit confused on the db session scopes.
From the SQLAlchemy docs:
Some web frameworks include infrastructure to assist in the task of aligning the lifespan of a Session with that of a web request. This includes products such as Flask-SQLAlchemy, for usage in conjunction with the Flask web framework...
I think this means my db session scope is contained within a particular route since I'm using Flask and Flask-SQLAlchemy, so I came up with the following:
init.py
app = Flask(__name__)
db = SQLAlchemy(app)
routes.py
from init import db
#app.route('/one')
def one():
form = MyForm()
if form.validate_on_submit():
myhelper.saveStuff1(form.stuff1.data)
myhelper.saveStuff2(form.stuff2.data)
db.session.commit()
return render_template(...)
#app.route('/two')
def two():
form = MyForm()
if form.validate_on_submit():
myhelper.saveStuff1(form.stuff1.data)
myhelper.saveStuff2(form.stuff2.data)
myhelper.saveStuff3(form.stuff3.data)
db.session.commit()
return render_template(...)
myhelper.py
from init import db
# Add new Item
def saveStuff1(formdata):
db.session.add(Item(name=formdata))
# Update Item
def saveStuff2(formdata):
item = Item.query.filter_by(name=formdata).first()
item.description = 'default'
db.session.add(item)
# etc...
Would this be the correct way for structuring my helpers? I'm worried that from init import db will cause problems with scoping since it's imported in both files, or if this overall code pattern will cause other problems.
SQLAlchemy's session scope is not related to Python's variable scope. So no, importing db in multiple places as you've shown won't cause problems. Regarding the session scope, Flask-SQLAlchemy takes care of that for you, so you can ignore (or not worry about) the discussion of scope in the SQLAlchemy docs.
I want to use an explicit master-master DB setup together with Flask and SQLAlchemy, hopefully this is supported with Flask-SQLAlchemy.
I want to be able to do something like the following code snippet but I'm not sure if it's supported by Flask-SQLAlchemy
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
SQLALCHEMY_DATABASE_URI = 'default_DB_uri'
SQLALCHEMY_BINDS = { 'master1':'first_master_DB_uri', 'master2': 'second_master_DB_uri' }
app.config['SQLALCHEMY_DATABASE_URI'] = SQLALCHEMY_DATABASE_URI
app.config['SQLALCHEMY_BINDS'] = SQLALCHEMY_BINDS
db = SQLAlchemy(app)
#app.route('/some_endpoint')
def some_endpoint():
# read some data for the default DB
readData = db.session.query('select ...')
m = SomeModel()
masterSession1 = db.session(bind='master1')
# persist the data in m into the first master
masterSession1.add(m)
masterSession2 = db.session(bind='master2')
# persist the data into the second master
masterSession2.add(m)
return "some return value"
Is there a way to achieve this using Flask-SQLAlchemy and binds?
I guess that Flask-SQLAlchemy already handles more than one engine with the binds but I can't see how to use that for an explicit DB selection and not a model based selection like mentioned here: http://pythonhosted.org/Flask-SQLAlchemy/binds.html
Thanks for the help.
The code below is what I ended up with to have this functionality.
A few notes:
I changed get_table_for_bind to bind all tables without an explicit __bind_key__ to all the binds. This is done in order to be able to call db.create_all() or db.drop_all() and create/drop the tables in all the DBs. In order for this to work and not break the default DB selection, when not specifying a specific bind, get_binds was changed to map the None bind again after the original implementation, to override the Table->Bind mapping.
If you don't specify a using_bind everything should work with the default DB.
SQLAlchemy mapped objects keep a reference to the session and state so you can't really add the same object to two DBs. I made a copy of the object before adding it in order to persist it in two DBs. Not sure if there is some better way to do this.
I haven't fully tested this and this might break some other functionality I'm not using or not aware of.
flask-sqlalchemy overrides:
from flask_sqlalchemy import SQLAlchemy, SignallingSession, get_state
from flask_sqlalchemy._compat import itervalues
class UsingBindSignallingSession(SignallingSession):
def get_bind(self, mapper=None, clause=None):
if self._name:
_eng = get_state(self.app).db.get_engine(self.app,bind=self._name)
return _eng
else:
return super(UsingBindSignallingSession, self).get_bind(mapper, clause)
_name = None
def using_bind(self, name):
self._name = name
return self
class UsingBindSQLAlchemy(SQLAlchemy):
def create_session(self, options):
return UsingBindSignallingSession(self, **options)
def get_binds(self, app=None):
retval = super(UsingBindSQLAlchemy, self).get_binds(app)
# get the binds for None again in order to make sure that it is the default bind for tables
# without an explicit bind
bind = None
engine = self.get_engine(app, bind)
tables = self.get_tables_for_bind(bind)
retval.update(dict((table, engine) for table in tables))
return retval
def get_tables_for_bind(self, bind=None):
"""Returns a list of all tables relevant for a bind.
Tables without an explicit __bind_key__ will be bound to all binds.
"""
result = []
for table in itervalues(self.Model.metadata.tables):
# if we don't have an explicit __bind_key__ bind this table to all databases
if table.info.get('bind_key') == bind or table.info.get('bind_key') == None:
result.append(table)
return result
db = UsingBindSQLAlchemy()
Now you can do this:
# This is the default DB
SQLALCHEMY_DATABASE_URI=YOUR_MAIN_DB_URI_CONNECT_STRING
# Master1 and Master2
SQLALCHEMY_BINDS = { 'master1':YOUR_MASTER1_DB_URI_CONNECT_STRING, 'master2':YOUR_MASTER2_DB_URI_CONNECT_STRING }
# Tables without __bind_key__ will be dropped/created on all DBs (default, master1, master2)
db.drop_all()
db.create_all()
s = db.session().using_bind('master1')
s.add(SOME_OBJECT)
s.commit()
s = db.session().using_bind('master2')
s.add(SOME_OBJECT_CLONE) # a clone of the original object, before the first add()
s.commit()
# and the default DB, as always
db.session.add(SOME_OTHER_OBJECT)
db.session.commit()