Develop Reference

How to set schema_translate_map in SQLAlchemy object in Flask app

My app.py file
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgres:////tmp/test.db'
db = SQLAlchemy(app) # refer https://flask-sqlalchemy.palletsprojects.com/en/2.x/api/#flask_sqlalchemy.SQLAlchemy
One of my model classes, where I imported db
from app import db
Base = declarative_base()
# User class
class User(db.Model, Base):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True, nullable=False)
email = db.Column(db.String(120), unique=True, nullable=False)
def __repr__(self):
return '<User %r>' % self.username
def get_user_by_id(self, id):
return self.query.get(id)
My database has the same set of tables in different schema (multi-tenancy) and there
I need to select the schema as per the request initiated by a particular tenant on the fly by using before_request (grabbing tenant_id from subdomain URL).
I found Postgres provides selecting the schema name on fly by using
schema_translate_map ref. https://docs.sqlalchemy.org/en/14/core/connections.html#translation-of-schema-names and that is under execution_options https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.Connection.execution_options
In my above code snippet where you see db = SQLAlchemy(app), as per official documentation, two parameters can be set in SQLAlchemy objct creation and they are - session_options and engine_options, but no execution_options ref. https://flask-sqlalchemy.palletsprojects.com/en/2.x/api/#flask_sqlalchemy.SQLAlchemy
But how do I set schema_translate_map setting when I am creating an object of SQLAlchemy
I tried this -
db = SQLAlchemy(app,
session_options={
"autocommit": True,
"autoflush": False,
"schema_translate_map": {
None: "public"
}
}
)
But obviously, it did not work, because schema_translate_map is under execution_options as mentioned here https://docs.sqlalchemy.org/en/14/core/connections.html#translation-of-schema-names
Anyone has an idea, how to set schema_translate_map at the time of creating SQLAlchemy object.
My goal is to set it dynamically for each request. I want to control it from this
centralized place, rather than going in each model file and specifying it when I execute
queries.
I am aware of doing this differently as suggested here https://stackoverflow.com/a/56490246/1560470
but my need is to set somewhere around db = SQLAlchemy(app) in app.py file only. Then after I import db in all my model classes (as shown above) and in those model classes, all queries execute under the selected schema.

I found a way to accomplish it. This is what needed
db = SQLAlchemy(app,
session_options={
"autocommit": True,
"autoflush": False
},
engine_options={
"execution_options":
{
"schema_translate_map": {
None: "public",
"abc": "xyz"
}
}
}
)

Threading and recursion problem using Sqlite with sqlalchemy in a flask api called by react app

I am working on a web app using Flask for API and data access and React with Material-UI for UI. In general, I have been successful in using React components in the UI, getting the data from Sqlite using the Flask API. The Flask app is set to proxy unknown requests to the Flask API.
I'm not able to create a code sandbox because I can't add dependencies for Flask or for SQLAlchemy.
The app was really almost complete; I was able to get data from several different tables using Flask and the API. Then I added a data filter selection component and a call to the API to get the last filter selection from a Settings table and that simple call is breaking in two ways:
The API call to getFilter is being called repeatedly. I can't find what to do to stop it. Loading the value is a change which calls the onChange which reloads the settings which causes a change. I am sure this is a bug in how I'm implementing it but I don't know what to do to fix it. The event handler is in the grandparent component, app.js, of the FilterSelector, where the change occurs. The error repeats until I get an error that Python could not create any more threads and then the Flask app is hung and has to be closed by closing the PowerShell window I am running it in, after which, the React app gives an http 500 that it could not proxy to the Flask app.
The calls to the API are coming in on different threads - a new thread is created for every second call and Sqlite is complaining about objects created in one thread not being available in another thread:
**sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 4852 and this is thread id 2824.
[SQL: SELECT settings.settingname AS settings_settingname, settings.settingvalue AS settings_settingvalue
FROM settings
WHERE settings.settingname = ?]
[parameters: [immutabledict({})]]
(Background on this error at: http://sqlalche.me/e/13/f405) // Werkzeug Debugger**
I don't know if the recursion is due to the thread error or if the threading error is due to the recursion. I wouldn't expect Python or sqlalchemy to create multiple threads for these calls.
I wish I could create a codeplaypen for this but I can't so here's the code:
React Code:
This line at the end of package.json lets the React app proxy to the API:
"proxy": "http://localhost:5000"
app.js
import React, { useState, useEffect } from 'react';
import Users from './Components/Users/Users';
export default function UserManager() {
const [Filter, setFilter] = useState('all');
useEffect(() => {
fetch('/api/getFilter').then(res => res.text()).then(data => {
console.log(new Date() + "Filter useEffect data = " + data);
setFilter(data);
console.log(new Date() + "Filter useEffect set to: " + Filter)
});
}
)
const filterChanged = (event, selectedFilter) => {
alert('app.js filter changed! Filter=' + Filter + ' and selectedFilter=' + selectedFilter);
setFilter(selectedFilter);
};
return (
<div>
<Users
filterChanged={filterChanged.bind(this)}
filter={ Filter }
/>
</div>
)
}
Users.js
import React, { Component } from 'react'
import FilterSelector from './FilterSelector'
export class Users extends Component {
render() {
const filter = this.props.filter;
console.log(this.props.filter)
console.log("Users render - " + this.props.filter);
return (<FilterSelector filter={filter} filterChanged={this.props.filterChanged.bind(this)}/>)
}
}
export default Users
FilterSelector.js
import React from 'react';
import Radio from '#material-ui/core/Radio';
import RadioGroup from '#material-ui/core/RadioGroup';
import FormControlLabel from '#material-ui/core/FormControlLabel';
import FormControl from '#material-ui/core/FormControl';
import FormLabel from '#material-ui/core/FormLabel';
export default function FilterSelector(props) {
return (
<FormControl component="fieldset">
<FormLabel component="legend">Filter</FormLabel>
<RadioGroup aria-label="filters" name="filter1" value={ props.filter } onChange={ props.filterChanged }>
<FormControlLabel value="favorites" control={<Radio />} label="Favorites" />
<FormControlLabel value="included" control={<Radio />} label="Included" />
<FormControlLabel value="all" control={<Radio />} label="All" />
</RadioGroup>
</FormControl>
);
}
Flask API Code:
api.py:
import time
from flask import Flask
import userdata
import json
import datetime
import subprocess
app = Flask(__name__)
#app.route('/api/getFilter')
#sample: http://localhost:5000/api/getFilter
def getFilter():
print ("In getFilter")
#return "favorites"
filter = userdata.getSetting('filter')
print ("{0}In api getFilter(). Filterdata from db is: {1}".format(datetime.datetime.now(), filter))
if filter is None:
#userdata.updateSetting('channelfilter', 'all')
filter = 'favorites'
print ("In getfilter2")
return filter
"""
#app.route('/api/saveFilter/<filterValue>')
#sample: http://localhost:5000/api/saveFilter/favorites
def saveFilter(filterValue):
userdata.updateSetting('filter', filterValue)
return filterValue
"""
#Below route is for development testing only, remove for production.
#app.route('/api/getUsers')
#sample: http://localhost:5000/api/getUsers
def getUsersFromDB():
return json.loads(userdata.GetUsers())
#app.route('/api/getUsers/<filter>')
#sample: http://localhost:5000/api/getUsers/favorites
def getUsersFromDBFiltered(filter):
print("In getUsersFiltered. filter = {0}".format(filter))
return json.loads(userdata.GetUsers())
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0')
userdata.py - shows there is no data changing or updating code so nothing is "created" in the DB in any thread.
from sqlalchemy import Column, ForeignKeyConstraint, UniqueConstraint
from sqlalchemy import PrimaryKeyConstraint, Integer, String, Boolean, DateTime, func
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy import text
import json
import os
Base = declarative_base()
Engine = create_engine('sqlite:///data/users.db')
Base.metadata.bind = Engine
DBSession = sessionmaker(bind=Engine)
session = DBSession()
class Settings(Base):
__tablename__ = "settings"
settingname = Column(String(80), primary_key=True)
settingvalue = Column(String(80), nullable=False)
def ResultSetQueryToJson(resultSet):
print ('Not Yet Implemented.')
return resultSet
def ResultSetDeclarativeToJson(resultSet):
jsontext = "{{\n\t\"{0}\": [".format(resultSet[0].__table__.name)
for datarow in resultSet:
rowsdata = ""
columncount = len(datarow.__table__.columns)
if jsontext.endswith("}"):
jsontext += ","
jsontext += "{\n"
for c in datarow.__table__.columns:
columncount -= 1
rowsdata = rowsdata + "\n\t\t\"{0}\": \"{1}\"".format(c.key, getattr(datarow, c.key))
if columncount > 0:
rowsdata += ","
jsontext += rowsdata
jsontext += "\n\t}"
return jsontext + "]\n}"
def getSettings():
print("Start getSettings())")
return session.query(Settings).all()
def getSetting(sname):
print("Starting userdata.getSetting() for {0}.".format(sname))
result = session.query(Settings).filter(Settings.settingname == sname).one_or_none()
print ("in getSetting. Query Result = {0}".format(result))
if result is not None:
svalue = result.settingvalue
print('Got setting {0}: {1}'.format(sname, svalue))
return svalue
return None
"""
def updateSetting(sname, svalue):
setting = Settings()
setting.settingname = sname
setting.settingvalue = svalue
session.merge(setting)
session.commit()
"""
class Apps(Base):
__tablename__ = 'apps'
appid = Column(String(24), primary_key=True)
name = Column(String(250), nullable=False)
vendor = Column(String(80), nullable=True)
version = Column(String(50), nullable=False)
isDeleted = Column(Boolean, nullable=False)
class Users(Base):
__tablename__ = 'users'
userid = Column(String(80), primary_key=True)
lastname = Column(String(40), nullable=False)
firstname = Column(String(40), nullable=False)
emailaddress = Column(String(80), nullable=False)
def ReflectTableColumns(DbEngine, meta, targetTable):
tableschema = Table(targetTable, meta, autoload=True, autoload_with=DbEngine)
cols = dict()
for c in tableschema.columns:
print("{0}\t|\t{1}".format(c.name, c.type))
cols[c.name] = c.type
return cols
def GetUsers():
DBSession = sessionmaker(bind=Engine)
session = DBSession()
results = session.query(Users).all()
ShowTableData(results, 'users')
return ResultSetDeclarativeToJson(results)
I've made sure no other instances of anything are touching the database file; closing all VSCode windows, all explorer windows, even rebooting and then starting the React and Flask apps anew. It's very odd that this code only causes the issue an none of my other database calling code does.
I know it's a complex scenario to evaluate without a codesandbox but I will definitely be appreciative of any help. I know appreciation isn't as good a beer but it's all I got for now.

Using Flask-SQLAlchemy without Flask

I had a small web service built using Flask and Flask-SQLAlchemy that only held one model. I now want to use the same database, but with a command line app, so I'd like to drop the Flask dependency.
My model looks like this:
class IPEntry(db.Model):
id = db.Column(db.Integer, primary_key=True)
ip_address = db.Column(db.String(16), unique=True)
first_seen = db.Column(db.DateTime(),
default = datetime.datetime.utcnow
)
last_seen = db.Column(db.DateTime(),
default = datetime.datetime.utcnow
)
#validates('ip')
def validate_ip(self, key, ip):
assert is_ip_addr(ip)
return ip
Since db will no longer be a reference to flask.ext.sqlalchemy.SQLAlchemy(app), how can I convert my model to use just SQLAlchemy. Is there a way for the two applications (one with Flask-SQLAlchemy the other with SQLAlchemy) to use the same database?

you can do this to replace db.Model:
from sqlalchemy import orm
from sqlalchemy.ext.declarative import declarative_base
import sqlalchemy as sa
base = declarative_base()
engine = sa.create_engine(YOUR_DB_URI)
base.metadata.bind = engine
session = orm.scoped_session(orm.sessionmaker())(bind=engine)
# after this:
# base == db.Model
# session == db.session
# other db.* values are in sa.*
# ie: old: db.Column(db.Integer,db.ForeignKey('s.id'))
# new: sa.Column(sa.Integer,sa.ForeignKey('s.id'))
# except relationship, and backref, those are in orm
# ie: orm.relationship, orm.backref
# so to define a simple model
class UserModel(base):
__tablename__ = 'users' #<- must declare name for db table
id = sa.Column(sa.Integer,primary_key=True)
name = sa.Column(sa.String(255),nullable=False)
then to create the tables:
base.metadata.create_all()

That is how to use SQLAlchemy without Flask (for example to write a bulk of objects to PostgreSQL database):
from sqlalchemy import Column, Integer, String
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
# Define variables DB_USERNAME, DB_PASSWORD, DB_HOST, DB_PORT, DB_NAME
SQLALCHEMY_DATABASE_URI = f'postgresql://{DB_USERNAME}:{DB_PASSWORD}#{DB_HOST}:
{DB_PORT}/{DB_NAME}'
# ----- This is related code -----
engine = create_engine(SQLALCHEMY_DATABASE_URI, echo=True)
Base = declarative_base()
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
Session.configure(bind=engine)
session = Session()
# ----- This is related code -----
class MyModel(Base):
__tablename__ = 'my_table_name'
id = Column(Integer, primary_key=True)
value = Column(String)
objects = [MyModel(id=0, value='a'), MyModel(id=1, value='b')]
session.bulk_save_objects(objects)
session.commit()

Check this one github.com/mardix/active-alchemy
Active-Alchemy is a framework agnostic wrapper for SQLAlchemy that makes it really easy to use by implementing a simple active record like api, while it still uses the db.session underneath. Inspired by Flask-SQLAlchemy

There is a great article about Flask-SQLAlchemy: how it works, and how to modify models to use them outside of Flask:
http://derrickgilland.com/posts/demystifying-flask-sqlalchemy/

The sqlalchemy docs has a good tutorial with examples that sound like what you want to do.
Shows how to connect to a db, mapping, schema creation, and querying/saving to the db.

This does not completely answer your question, because it does not remove Flask dependency, but you can use SqlAlchemy in scripts and tests by just not running the Flask app.
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import MetaData
test_app = Flask('test_app')
test_app.config['SQLALCHEMY_DATABASE_URI'] = 'database_uri'
test_app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
metadata = MetaData(schema='myschema')
db = SQLAlchemy(test_app, metadata=metadata)
class IPEntry(db.Model):
pass
One difficulty you may encounter is the requirement of using db.Model as a base class for your models if you want to target the web app and independent scripts using same codebase. Possible way to tackle it is using dynamic polymorphism and wrap the class definition in a function.
def get_ipentry(db):
class IPEntry(db.Model):
pass
return IPEntry
As you construct the class run-time in the function, you can pass in different SqlAlchemy instances. Only downside is that you need to call the function to construct the class before using it.
db = SqlAlchemy(...)
IpEntry = get_ipentry(db)
IpEntry.query.filter_by(id=123).one()

Flask (> 1.0) attempt to provide helpers for sharing code between an web application and a command line interface; i personally think it might be cleaner, lighter and easier to build libraries unbound to flask, but you might want to check:
https://flask.palletsprojects.com/en/2.1.x/cli/
https://flask.palletsprojects.com/en/2.1.x/api/#flask.Flask.cli

Create database and table
import os
from sqlalchemy import create_engine
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
if os.path.exists('test.db'):
os.remove('test.db')
Base = declarative_base()
class Person(Base):
__tablename__ = 'person'
id = Column(Integer(), primary_key=True)
name = Column(String())
engine = create_engine('sqlite:///test.db')
Base.metadata.create_all(engine)
Using Flask_SQLAlchemy directly
from flask import Flask
from sqlalchemy import MetaData
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import Column, Integer, String
app = Flask(__name__)
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db'
db = SQLAlchemy(app, metadata=MetaData())
class Person(db.Model):
__tablename__ = 'person'
id = Column(Integer(), primary_key=True)
name = Column(String())
person = Person(name='Bob')
db.session.add(person)
db.session.commit()
print(person.id)

Serializing Python Arrow objects for the Flask-Restless API

I am currently developing an application with Flask-Restless. When I substituted my SQLAlchemy models' typical DateTime fields with corresponding arrow fields, all went smoothly. This was due to the help of SQLAlchemy-Utils and its ArrowType field.
However, after using the API to return a JSON representation of these objects, I received the following error:
TypeError: Arrow [2015-01-05T01:17:48.074707] is not JSON serializable
Where would be the ideal place to modify how the model gets serialized? Do I modify Flask-Restless code to support Arrow objects or write a model method that Flask-Restless can identify and use to retrieve a JSON-compatible object?
I could also write an ugly post-processor function in the meantime but that solution seems like a terrible hack.
Below is an example model with the ArrowType field:
class Novel(db.Model):
id = db.Column(db.Integer, primary_key=True)
title = db.Column(db.Unicode, unique=True, nullable=False)
created_at = db.Column(ArrowType, nullable=False)
def __init__(self, title):
self.title = title
self.created_at = arrow.utcnow()

Arrow now has a for_json method. For example: arrow.utcnow().for_json()

How about a custom JSONEncoder which supports Arrow types? Looking at the Flask-Restless source code, it uses Flask's built in jsonify under the hood. See this snippet for an example which serializes regular datetime objects in a different format: http://flask.pocoo.org/snippets/119/
Here's a full self-contained example for good measure:
import flask
import flask.ext.sqlalchemy
import flask.ext.restless
from flask.json import JSONEncoder
import sqlalchemy_utils
import arrow
app = flask.Flask(__name__)
app.config['DEBUG'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db'
db = flask.ext.sqlalchemy.SQLAlchemy(app)
class Event(db.Model):
id = db.Column(db.Integer, primary_key=True)
timestamp = db.Column(sqlalchemy_utils.ArrowType)
class ArrowJSONEncoder(JSONEncoder):
def default(self, obj):
try:
if isinstance(obj, arrow.Arrow):
return obj.format('YYYY-MM-DD HH:mm:ss ZZ')
iterable = iter(obj)
except TypeError:
pass
else:
return list(iterable)
return JSONEncoder.default(self, obj)
app.json_encoder = ArrowJSONEncoder
db.create_all()
manager = flask.ext.restless.APIManager(app, flask_sqlalchemy_db=db)
manager.create_api(Event, methods=['GET','POST'])
app.run()
From the two options in your post, I'd suggest adding the method to your model class to retrieve a JSON-compatible object, only because it's simpler and more maintainable. If you want to modify Flask-Restless, you either need to fork it or monkey patch it.

Configuring Flask-SQLAlchemy to use multiple databases with Flask-Restless

I have a Flask app that uses Flask-SQLAlchemy and I'm trying to configure it to use multiple databases with the Flask-Restless package.
According to the docs, configuring your models to use multiple databases with __bind_key__ seems pretty straightforward.
However it doesn't seem to be working for me.
I create my app and initialise my database like this:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
SQLALCHEMY_DATABASE_URI = 'postgres://db_user:db_pw#localhost:5432/db_name'
SQLALCHEMY_BINDS = {
'db1': SQLALCHEMY_DATABASE_URI,
'db2': 'mysql://db_user:db_pw#localhost:3306/db_name'
}
app = Flask(__name__)
db = SQLALchemy(app)
Then define my models including __bind_key__, which should tell SQLAlchemy which DB it needs to use:
class PostgresModel(db.Model):
__tablename__ = 'postgres_model_table'
__bind_key__ = 'db1'
id = db.Column(db.Integer, primary_key=True)
...
class MySQLModel(db.Model):
__tablename__ = 'mysql_model_table'
__bind_key__ = 'db2'
id = db.Column(db.Integer, primary_key=True)
...
Then I fire up Flask-Restless like this:
manager = restless.APIManager(app, flask_sqlalchemy_db=db)
manager.init_app(app, db)
auth_func = lambda: is_authenticated(app)
manager.create_api(PostgresModel,
methods=['GET'],
collection_name='postgres_model',
authentication_required_for=['GET'],
authentication_function=auth_func)
manager.create_api(MySQLModel,
methods=['GET'],
collection_name='mysql_model',
authentication_required_for=['GET'],
authentication_function=auth_func)
The app runs fine and when I hit http://localhost:5000/api/postgres_model/[id] I get the expected JSON response of the object from the Postgres DB (I'm guessing this is because I have it's credentials in SQLALCHEMY_DATABASE_URI).
Although when I hit http://localhost:5000/api/mysql_model/[id], I get a mysql_model_table does not exist error, indicating that it's looking in the Postgres DB, not the MySQL one.
What am I doing wrong here?

This was not working because of a simple typo:
__bind_key = 'db1'
Should have been
__bind_key__ = 'db1'
I've updated the original question and fixed the typo as an example of how this can work for others.