I have an application running in production that I've built for a single client that I want to convert to support multiple "tenants".
Currently I am using a Postgres database where all my data resides in a single database in the default public schema. I would like to isolate each tenant to a separate Postgres schema. Ideally, my application's UI would make a call to my API using the tenant's subdomain. In before_request I would somehow be able to set all database queries during the current request context to only query that tenant's schema, is this possible?
I envisage an ideal solution to be something similar to this contrived example:
from flask import Flask, request, jsonify
from pony.orm import Database, Required
app = Flask(__name__)
db = Database(**{<db_connection_dict>})
class User(db.Entity):
email = Required(str)
password = Required(str)
#classmethod
def login(cls, email: str, password: str) -> str:
user = cls.get(lambda u: u.email.lower() == email.lower())
if not user:
return None
password_is_valid = <method_to_check_hashed_pasword>
if not password_is_valid:
return None
return <method_to_generate_jwt>
db.generate_mapping()
#app.before_request
def set_tenant():
tenant_subdomain = request.host.split(".")[0]
// MISSING STEP.. set_schema is a fictitous method, does something similar to this exist?
db.set_schema(schema=tenant_subdomain)??
#app.route("auth/login", methods=["POST"]
def login_route():
data = request.get_json()
jwt = User.login(data["email"], data["password"])
if not jwt:
return make_response({}, 403)
return make_response(jsonify(data=jwt), 200)
I've come across an interesting/simple example using SQLAlchemy. If not possible with PonyORM I may consider porting my models over to SQLAlchemy but would miss the simplicity of Pony :(
I thought about possibly using the Database.on_connect method to do something as such but not sure if if anyone has any other ideas or if this would even work properly in production. I suspect not because if I had two separate tenants querying the database they would overwrite the search path..
#db.on_connect()
def set_request_context_tenant_schema(db, connection) -> None:
subdomain = request.host.split(".")[0]
cursor = connection.cursor()
cursor.execute(f"SET search_path TO {subdomain}, public;")
Related
I am new to flask and python in general. I just need help accessing an already existing snowflake database with flask. I just want to query the data. This is my code thus far and its not working:
from flask import Flask, render_template, request, redirect, url_for
from model import InputForm
from compute import preprocess
from sqlalchemy import create_engine, MetaData, Table
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABSE_URI'] = 'snowflake://<user_login_name>:<password>#<account_name>'
db = SQLAlchemy(app)
engine = create_engine()
metadata = MetaData(bind=engine)
engagements = db.Table('ENGAGEMENTS', db.metadata, autoload=True, autoload_with=db.engine)
companies = db.Table('COMPANIES', db.metadata, autoload=True, autoload_with=db.engine)
#app.route('/')
def index():
results = db.session.query(engagements).all()
for r in results:
print(engagements)
return ''
if __name__ == '__main__':
app.run(debug=True)
There may be multiple things going on here.
First, if the error message you get is due to non-existent engagements table in the Snowflake db, it may be exactly that. Before you can execute queries, you need to bring your database at par with the models in your code. https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-iv-database is a great source on how to set up your db. It teaches how to set up a normal sqlite database for migrations using Alembic. Luckily snowflake-sqlalchemy has alembic support: https://github.com/snowflakedb/snowflake-sqlalchemy#alembic-support . You can use them to build and export migrations, and your db should be ready.
Second, there's two typos: one in your code, one in the documentation. Your code sets app.config['SQLALCHEMY_DATABSE_URI'] instead of app.config['SQLALCHEMY_DATABASE_URI']. And you also definitely need to mention your db and schema for snowflake as pointed out in https://stackoverflow.com/a/59204661/2928486. But the official documentation has incorrect uri.
'snowflake://<user_login_name>:<password>#<account_name>/<database_name>/<schema_name>?warehouse=<warehouse_name>?role=<role_name>'
Instead of the above, it should be the following:
'snowflake://<user_login_name>:<password>#<account_name>/<database_name>/<schema_name>?warehouse=<warehouse_name>&role=<role_name>'
Basically your connection params should be joined using & instead of ?. I was able to set up my flask application using your code, and it is migrating and updating the snowflake db fine, so hopefully it works for other ddl/dql scenarios as well!
I'm not a flask user, but it is possible you need to include the database and schema. I found a URL example below that includes snowflake account, database, schema, warehouse and role.
'snowflake://<user_login_name>:<password>#<account_name>/<database_name>/<schema_name>?warehouse=<warehouse_name>?role=<role_name>'
I want to override setup_db method for my add-on because in current situation you can't take argument from url with db name, and if the user has more than 1 database I can`t run my login link from incognito.
I don't want the user to go to /web/database/selector first.
I was thinking about the user going to /web/login?db=example_db_name and then somehow redirect to my login link.
("somehow" because if you type it, it redirects you to /web/login, so i cant add redirect from login page).
I'm doing that assuming that in odoo.conf user has
db_name = False, dbfilter = .
If you faced the same problem, here is my solution. It`s overriding the default method which is kinda bad usually, but in our situation there isnt much we can do.
from odoo import http
class Rooting(http.Root):
def setup_db(self, httprequest):
db = httprequest.session.db
# Check if session.db is legit
if db:
if db not in http.db_filter([db], httprequest=httprequest):
httprequest.session.logout()
db = None
if not db:
if 'db' in httprequest.args:
db = httprequest.args['db']
httprequest.session.db = db
if not db:
httprequest.session.db = http.db_monodb(httprequest)
http.Root.setup_db = Rooting.setup_db
I am not able to find good resources which can help me understand how can i migrate my Flask and sqlalchemy apps to AWS lambda and API gateway and make it serverless. Like for instance below is a sample code taken from the flask_sqlalchemy documentation :
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True, nullable=False)
email = db.Column(db.String(120), unique=True, nullable=False)
def __repr__(self):
return '<User %r>' % self.username
Now how can i migrate this code to AWS lambda . Is it even possible . For instance the line app = Flask(__name__) should not be there right ? If there is no app variable how am i going to initialize db variable ?
Please can someone give me some intro or a link to a good tutorial which will clear these concepts?
Many thanks in advance.
To use a Flask/sqlalchemy app with Lambda, you need to wrap Flask in the Lambda dispatch model, and make sure sqlalchemy can access its database.
Dispatching Lambda requests to Flask
You can integrate Chalice with Flask like this:
class ChaliceWithFlask(chalice.Chalice):
"""
Subclasses Chalice to host a Flask app, route and proxy requests to it.
"""
def __init__(self, flask_app, *args, **kwargs):
super().__init__(*args, **kwargs)
self.flask_app = flask_app
self.trailing_slash_routes = []
routes = collections.defaultdict(list)
for rule in self.flask_app.url_map.iter_rules():
route = re.sub(r"<(.+?)(:.+?)?>", r"{\1}", rule.rule)
if route.endswith("/"):
self.trailing_slash_routes.append(route.rstrip("/"))
routes[route.rstrip("/")] += rule.methods
for route, methods in routes.items():
self.route(route, methods=list(set(methods) - {"OPTIONS"}), cors=True)(self.dispatch)
def dispatch(self, *args, **kwargs):
uri_params = self.current_request.uri_params or {}
path = self.current_request.context["resourcePath"].format(**uri_params)
if self.current_request.context["resourcePath"] in self.trailing_slash_routes:
if self.current_request.context["path"].endswith("/"):
path += "/"
else:
return chalice.Response(status_code=requests.codes.found, headers={"Location": path + "/"}, body="")
req_body = self.current_request.raw_body if self.current_request._body is not None else None
base_url = "https://{}".format(self.current_request.headers["host"])
query_string = self.current_request.query_params or {}
with self.flask_app.test_request_context(path=path,
base_url=base_url,
query_string=list(query_string.items()),
method=self.current_request.method,
headers=list(self.current_request.headers.items()),
data=req_body,
environ_base=self.current_request.stage_vars):
flask_res = self.flask_app.full_dispatch_request()
res_headers = dict(flask_res.headers)
res_headers.pop("Content-Length", None)
res_body = b"".join([c for c in flask_res.response])
return chalice.Response(status_code=flask_res._status_code, headers=res_headers, body=res_body)
flask_app = flask.Flask(app_name)
# add routes, etc. to your Flask app here
app = ChaliceWithFlask(app_name, flask_app=flask_app)
Connecting sqlalchemy to the database
You could access the database directly, but that means opening the database port to the Internet or placing your Lambda in a VPC (which makes it impossible for the Lambda to be accessible over the Internet). Also, traditional database drivers make assumptions about persistence of their connections that are not satisfied in Lambda.
AWS recently came out with the perfect solution for this - the AWS Aurora RDS Data API. It's basically an AWS-authenticated SQL-over-HTTP tunnel. I wrote a SQLAlchemy adapter for it: sqlalchemy-aurora-data-api. After installing it, you can do:
from sqlalchemy import create_engine
cluster_arn = "arn:aws:rds:us-east-1:123456789012:cluster:my-aurora-serverless-cluster"
secret_arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:MY_DB_CREDENTIALS"
app = Flask(app_name)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql+auroradataapi://:#/my_db_name'
engine_options=dict(connect_args=dict(aurora_cluster_arn=cluster_arn, secret_arn=secret_arn))
db = flask_sqlalchemy.SQLAlchemy(app, engine_options=engine_options)
First of all, in AWS Lambda, you don't use your Flask for routing anymore. Instead use AWS API Gateway for routing. An example of the routing is shown below, from https://apievangelist.com/2017/10/23/a-simple-api-with-aws-dynamodb-lambda-and-api-gateway/
As you can see at the right end of the picture, the "Lambda" box shows the name of the Lambda function you have uploaded. For Lambda in Python, see https://docs.aws.amazon.com/lambda/latest/dg/python-programming-model-handler-types.html
Basically, the main thing in Python lambda is the:
def handler_name(event, context):
...
return some_value
From the event and context you can get everything: path, HTTP method, headers, params, body, etc (like flask.request). You might also need to know there are two ways of doing Lambda the LAMBDA and LAMBDA_PROXY (see the Integration Request box in the first picture).
Short version difference is:
LAMBDA mode will preprocess request body automatically and gives your Lambda function a Python object in event.
LAMBDA_PROXY will give you raw HTTP request, you need to convert the content yourself inside the Lambda function.
As for SQL Alchemy, all you need to do is to zip all the SQL Alchemy library code and its' dependency together with your Lambda function and upload it to the Lambda Console, it works without any modification.
Please note that SQLite will not work in Lambda, as Lambda function has no filesystem access. You should put the data somewhere else, e.g. Amazon RDS (with MySQL, PostgreSQL, whatever you like) and then make sure the Lambda is connected to the same VPC (Amazon internal router) with the RDS database.
In the event mapper level docs
it says that Session.add() is not supported, but when I tried to do db.session.add(some_object) inside after_insert event it worked, example:
def after_insert_listener(mapper, connection, user):
global_group = Group.query.filter_by(groupname='global').first()
a = Association(user,global_group)
db.session.add(a)
event.listen(User, 'after_insert', after_insert_listener)
Basically any new user should be part of global_group, so I added it in the after_insert event. I tried to insert a user, and then checked into my database and I found the user record, and the association record.
Let's check the diferences:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite://///Users/dedeco/Documents/tmp/testDb.db'
db = SQLAlchemy(app)
>>>type(db.session)
<class 'sqlalchemy.orm.scoping.scoped_session'>
or
from sqlalchemy import *
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
some_engine = create_engine('sqlite://///Users/dedeco/Documents/tmp/testDb.db')
Session = sessionmaker(bind=some_engine)
session = Session()
Base = declarative_base()
>>> type(session)
<class 'sqlalchemy.orm.session.Session'>
Basically the difference is:
In the first way you are using a API developed for the Flask framework, called Flask-SQLAlchemy. It's the option if you are creating a Flask application, because the scope of the Session can be managed automatically by your application. You have many benefits like a infrastructure to establish a single Session, associated with the request, which is correctly constructed and torn down corresponding torn down at the end of a request.
In the second way is a pure SQLAlchemy app, so if you are using a library to connect a particular database, you can use just a SQLAlchemy API, for example, for a command-line script, background daemon, GUI interface-driven application, etc.
So, in a both way you can add, like:
Using a Flask-SQLAlchemy:
class User(db.Model):
__tablename__ = 'users'
user_id = db.Column(db.Integer(), primary_key = True)
user_name = db.Column(db.String(80), unique=True)
def __init__(self, user_name):
self.user_name = user_name
>>> db.create_all()
>>> u = User('user1')
>>> db.session.add(u)
>>> db.session.commit()
>>> users = db.session.query(User).all()
>>> for u in users:
... print u.user_name
...
user1
Using just SQLAlchemy:
class User(Base):
__tablename__ = 'users'
user_id = Column(Integer(), primary_key = True)
user_name = Column(String(80), unique=True)
>>> u = User()
>>> u.user_name = 'user2'
>>> session.add(u)
>>> session.commit()
>>> users = session.query(User).all()
>>> for u in users:
... print u.user_name
...
user1
user2
Realize that I am connecting in the same database just for show that you can add using many ways.
server = Flask(__name__)
app = dash.Dash(__name__,server=server,external_stylesheets=[dbc.themes.LITERA], suppress_callback_exceptions = True)
app.server.config["SQLALCHEMY_DATABASE_URI"] = f'postgresql://postgres:.../...'
db = SQLAlchemy(app.server)
I have the same problem of not knowing at what point I should close the session of the database in my web application. I found this in the link that #GabrielChu shared so what I understood was if you are dealing with a web app the session is closed when the user close their tab
A web application is the easiest case because such an application is already constructed around a single, consistent scope - this is the request, which represents an incoming request from a browser, the processing of that request to formulate a response, and finally the delivery of that response back to the client. Integrating web applications with the Session is then the straightforward task of linking the scope of the Session to that of the request. The Session can be established as the request begins, or using a lazy initialization pattern which establishes one as soon as it is needed. The request then proceeds, with some system in place where application logic can access the current Session in a manner associated with how the actual request object is accessed. As the request ends, the Session is torn down as well, usually through the usage of event hooks provided by the web framework. The transaction used by the Session may also be committed at this point, or alternatively the application may opt for an explicit commit pattern, only committing for those requests where one is warranted, but still always tearing down the Session unconditionally at the end.
Some web frameworks include infrastructure to assist in the task of aligning the lifespan of a Session with that of a web request. This includes products such as Flask-SQLAlchemy, for usage in conjunction with the Flask web framework, and Zope-SQLAlchemy, typically used with the Pyramid framework. SQLAlchemy recommends that these products be used as available
I want to use an explicit master-master DB setup together with Flask and SQLAlchemy, hopefully this is supported with Flask-SQLAlchemy.
I want to be able to do something like the following code snippet but I'm not sure if it's supported by Flask-SQLAlchemy
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
SQLALCHEMY_DATABASE_URI = 'default_DB_uri'
SQLALCHEMY_BINDS = { 'master1':'first_master_DB_uri', 'master2': 'second_master_DB_uri' }
app.config['SQLALCHEMY_DATABASE_URI'] = SQLALCHEMY_DATABASE_URI
app.config['SQLALCHEMY_BINDS'] = SQLALCHEMY_BINDS
db = SQLAlchemy(app)
#app.route('/some_endpoint')
def some_endpoint():
# read some data for the default DB
readData = db.session.query('select ...')
m = SomeModel()
masterSession1 = db.session(bind='master1')
# persist the data in m into the first master
masterSession1.add(m)
masterSession2 = db.session(bind='master2')
# persist the data into the second master
masterSession2.add(m)
return "some return value"
Is there a way to achieve this using Flask-SQLAlchemy and binds?
I guess that Flask-SQLAlchemy already handles more than one engine with the binds but I can't see how to use that for an explicit DB selection and not a model based selection like mentioned here: http://pythonhosted.org/Flask-SQLAlchemy/binds.html
Thanks for the help.
The code below is what I ended up with to have this functionality.
A few notes:
I changed get_table_for_bind to bind all tables without an explicit __bind_key__ to all the binds. This is done in order to be able to call db.create_all() or db.drop_all() and create/drop the tables in all the DBs. In order for this to work and not break the default DB selection, when not specifying a specific bind, get_binds was changed to map the None bind again after the original implementation, to override the Table->Bind mapping.
If you don't specify a using_bind everything should work with the default DB.
SQLAlchemy mapped objects keep a reference to the session and state so you can't really add the same object to two DBs. I made a copy of the object before adding it in order to persist it in two DBs. Not sure if there is some better way to do this.
I haven't fully tested this and this might break some other functionality I'm not using or not aware of.
flask-sqlalchemy overrides:
from flask_sqlalchemy import SQLAlchemy, SignallingSession, get_state
from flask_sqlalchemy._compat import itervalues
class UsingBindSignallingSession(SignallingSession):
def get_bind(self, mapper=None, clause=None):
if self._name:
_eng = get_state(self.app).db.get_engine(self.app,bind=self._name)
return _eng
else:
return super(UsingBindSignallingSession, self).get_bind(mapper, clause)
_name = None
def using_bind(self, name):
self._name = name
return self
class UsingBindSQLAlchemy(SQLAlchemy):
def create_session(self, options):
return UsingBindSignallingSession(self, **options)
def get_binds(self, app=None):
retval = super(UsingBindSQLAlchemy, self).get_binds(app)
# get the binds for None again in order to make sure that it is the default bind for tables
# without an explicit bind
bind = None
engine = self.get_engine(app, bind)
tables = self.get_tables_for_bind(bind)
retval.update(dict((table, engine) for table in tables))
return retval
def get_tables_for_bind(self, bind=None):
"""Returns a list of all tables relevant for a bind.
Tables without an explicit __bind_key__ will be bound to all binds.
"""
result = []
for table in itervalues(self.Model.metadata.tables):
# if we don't have an explicit __bind_key__ bind this table to all databases
if table.info.get('bind_key') == bind or table.info.get('bind_key') == None:
result.append(table)
return result
db = UsingBindSQLAlchemy()
Now you can do this:
# This is the default DB
SQLALCHEMY_DATABASE_URI=YOUR_MAIN_DB_URI_CONNECT_STRING
# Master1 and Master2
SQLALCHEMY_BINDS = { 'master1':YOUR_MASTER1_DB_URI_CONNECT_STRING, 'master2':YOUR_MASTER2_DB_URI_CONNECT_STRING }
# Tables without __bind_key__ will be dropped/created on all DBs (default, master1, master2)
db.drop_all()
db.create_all()
s = db.session().using_bind('master1')
s.add(SOME_OBJECT)
s.commit()
s = db.session().using_bind('master2')
s.add(SOME_OBJECT_CLONE) # a clone of the original object, before the first add()
s.commit()
# and the default DB, as always
db.session.add(SOME_OTHER_OBJECT)
db.session.commit()