I have a python app that uses Flask RESTful as well as Flask SQLAlchemy. Part of the API I'm writing has the side effect of spinning off Timer objects. When a Timer expires, it executes some database queries. I'm seeing an issue in which code that is supposed to update rows in the database (a sqlite backend) is actually not issuing any UPDATE statements. I have verified this by turning the SQLALCHEMY_ECHO flag on to log the SQL statements. Whether or not the code works seems to be random. About half the time it fails to issue the UPDATE statement. See full example below.
My guess here is that SQLAlchemy Flask does not work properly when called from a worker thread. I think part of the point of Flask SQLAlchemy is to manage the SQLAlchemy sessions for you per API request. Obviously since there are no API requests going on when the Timer expires, I could see where things may not work properly.
Just to test this, I went ahead and wrote a simple data access layer using python's sqlite3 interface and it seems to solve the problem.
I'd really rather not have to rewrite a bunch of data access code though. Is there a way to get Flask SQLAlchemy to work properly in this case?
Sample code
Here's where I set up the flask app and save off the SQLAlchemy db object:
from flask import Flask
from flask_restful import Api
from flask.ext.sqlalchemy import SQLAlchemy
from flask_cors import CORS
import db_conn
flask_app = Flask(__name__)
flask_app.config.from_object('config')
CORS(flask_app)
api = Api(flask_app)
db_conn.db = SQLAlchemy(flask_app)
api.add_resource(SomeClass, '/abc/<some_id>/def')
Here's how I create the ORM models:
import db_conn
db = db_conn.db
class MyTable(db.Model):
__tablename__ = 'my_table'
id = db.Column(db.Integer, primary_key=True)
phase = db.Column(db.Integer, nullable=False, default=0)
def set_phase(self, phase):
self.phase = phase
db.session.commit()
Here's the API handler with timer and the database call that is failing:
from flask_restful import Resource
from threading import Timer
from models import MyTable
import db_conn
import global_store
class SomeClass(Resource):
def put(self, some_id):
global_store.saved_id = some_id
self.timer = Timer(60, self.callback)
return '', 204
def callback(self):
row = MyTable.query.filter_by(id=global_store.saved_id).one()
# sometimes this works, sometimes it doesn't
row.set_phase(1)
db_conn.db.session.commit()
I'm guessing in your callback you aren't actually changing the value of the object. SQLAlchemey won't issue DB UPDATE calls if the session state is not dirty. So if the phase is already 1 for some reason there is nothing to do.
Related
I have a flask application, that accesses a DB via Class that encapsulates DB access. I need to use this same class outside of the flask application for some regular jobs that access the same db.
/databases/database.db
/website/application/myblueprint/views.py
/website/application/myblueprint/db_class.py
/scripts/log_reading.py
Both views.py and log_reading.py need to use db_class.py, but you can't import from above your own package.
I could make db_class.py it's own application and install it each
venv, but then every time I edit I have to reinstall it in each
place. Plus there's the overhead of the setup stuff for a single module.
I could put the file in python site path, either by moving
it or by adding to the path, but that feels wrong and I'm not sure
would work with venvs.
I could sym link, that also feels wrong.
I'm not using flask models for the DB, but I don't think that would solve my problem anyway.
if you want to use db or session then you have to create application context, outside of application context you can't use it.
from flask import Flask
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
import os
APP_SETTINGS = os.getenv("APP_SETTINGS", "your config path")
def _create_db_session(self, connection_string):
engine = create_engine(connection_string)
session_factory = sessionmaker(bind=self.engine)
session = scoped_session(session_factory)
return session
app = Flask(__name__)
app.config.from_object(APP_SETTINGS)
db_url = app.config.get("SQLALCHEMY_DATABASE_URI")
session =_create_db_session(db_url)
#then you can use
# session.query to query database
# session.commit(), session.remove()
# same operations you do normally with DB session
I am new to flask and python in general. I just need help accessing an already existing snowflake database with flask. I just want to query the data. This is my code thus far and its not working:
from flask import Flask, render_template, request, redirect, url_for
from model import InputForm
from compute import preprocess
from sqlalchemy import create_engine, MetaData, Table
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABSE_URI'] = 'snowflake://<user_login_name>:<password>#<account_name>'
db = SQLAlchemy(app)
engine = create_engine()
metadata = MetaData(bind=engine)
engagements = db.Table('ENGAGEMENTS', db.metadata, autoload=True, autoload_with=db.engine)
companies = db.Table('COMPANIES', db.metadata, autoload=True, autoload_with=db.engine)
#app.route('/')
def index():
results = db.session.query(engagements).all()
for r in results:
print(engagements)
return ''
if __name__ == '__main__':
app.run(debug=True)
There may be multiple things going on here.
First, if the error message you get is due to non-existent engagements table in the Snowflake db, it may be exactly that. Before you can execute queries, you need to bring your database at par with the models in your code. https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-iv-database is a great source on how to set up your db. It teaches how to set up a normal sqlite database for migrations using Alembic. Luckily snowflake-sqlalchemy has alembic support: https://github.com/snowflakedb/snowflake-sqlalchemy#alembic-support . You can use them to build and export migrations, and your db should be ready.
Second, there's two typos: one in your code, one in the documentation. Your code sets app.config['SQLALCHEMY_DATABSE_URI'] instead of app.config['SQLALCHEMY_DATABASE_URI']. And you also definitely need to mention your db and schema for snowflake as pointed out in https://stackoverflow.com/a/59204661/2928486. But the official documentation has incorrect uri.
'snowflake://<user_login_name>:<password>#<account_name>/<database_name>/<schema_name>?warehouse=<warehouse_name>?role=<role_name>'
Instead of the above, it should be the following:
'snowflake://<user_login_name>:<password>#<account_name>/<database_name>/<schema_name>?warehouse=<warehouse_name>&role=<role_name>'
Basically your connection params should be joined using & instead of ?. I was able to set up my flask application using your code, and it is migrating and updating the snowflake db fine, so hopefully it works for other ddl/dql scenarios as well!
I'm not a flask user, but it is possible you need to include the database and schema. I found a URL example below that includes snowflake account, database, schema, warehouse and role.
'snowflake://<user_login_name>:<password>#<account_name>/<database_name>/<schema_name>?warehouse=<warehouse_name>?role=<role_name>'
I have a web application that is on top of mysql. When I started it, I built the mysql database from scratch, and connected to it using pymysql.
Fast forward...
I've rewritten everything using sqlalchemy to connect to the db (non-declarative?) I can connect to the db, read the db, update, etc. I also use MySQLWorkbench to view the database in a graphic way.
My webapp has a few tables that will poll the database for changes, and update the table. For instance, I have a job queue which will update the percentage-done. Here's the kicker:
When I send a job through, I will watch the database in MySQLWorkbench. I can verify that the job is going through (I see the percentage climbing). This confirms that the webapp is writing to the database. But! On the other end, where the table is asking for the status, it is not recieving the updated information (I've done print outs in the models, as well as in the flask app, as well as the FE js).
What is going on here? Even though I can see it in MySQLWorkbench is the connection not 'releasing' or something? Here is some of my code:
#models.py
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker, Query
import os
import json
Base = automap_base()
engine = create_engine("mysql+pymysql://user:asdas758ef10d2364d54d7e8#localhost:{}/metadatacontroller".format(3306))
# reflect the tables
Base.prepare(engine, reflect=True)
EncodeQueue = Base.classes.encode_queue
db_session = scoped_session(sessionmaker(bind=engine))
class DataStore:
def __init__(self):
self.db = db_session()
#receive status
def queue_rendering_items(self):
query = self.db.query(EncodeQueue).filter(EncodeQueue.status.in_(['queue', 'rendering', 'rendering thumbnails'])).all()
return_query = []
for item in query:
row = {'artist_id': item.artist_id,
'art_id': item.art_id,
'status': item.status,
'progress': item.errors,
'artwork_title': item.artwork_title,
'artist_name': item.artist_name,
}
return_query.append(row)
return return_query
#update status
def update_render_progress(self, job_id, progress):
self.db.query(EncodeQueue).filter_by(job_id=job_id).update\({'errors': json.dumps(progress)})
self.db.commit()
(ignore the back slash in update_render_progress ... weird SO formatting)
This is how 'status' code is being called on a flask end point:
#app.route('/api/jobs', methods=["GET"])
def get_all_jobs():
db = models.DataStore()
all_art = db.queue_rendering_items()
print(all_art)
return jsonify({'data':all_art})
Lastly, the status is being updated by a different file, on a different machine:
import models
db = models.DataStore()
db.update_render_progress(job_id, {'percentage': percentage_complete, 'time_remain': render.render_estimated_seconds_remaining})
What the heck is happening?
I wonder how SQLAlchemy tracks changes that are made outside of SQLAlchemy (manual change for example)?
Until now, I used to put db.session.commit() before each value that can be changed outside of SQLAlchemy. Is this a bad practice? If yes, is there a better way to make sure I'll have the latest value? I've actually created a small script below to check that and apparently, SQLAlchemy can detect external changes without db.session.commit() being called each time.
Thanks,
P.S: I really want to understand how all the magics happen behind SQLAlchemy work. Does anyone has a pointer to some docs explaining the behind-the-scenes work of SQLAlchemy?
import os
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
# Use SQLlite so this example can be run anywhere.
# On Mysql, the same behaviour is observed
basedir = os.path.abspath(os.path.dirname(__file__))
db_path = os.path.join(basedir, "app.db")
app.config["SQLALCHEMY_DATABASE_URI"] = 'sqlite:///' + db_path
db = SQLAlchemy(app)
# A small class to use in the test
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(100))
# Create all the tables and a fake data
db.create_all()
user = User(name="old name")
db.session.add(user)
db.session.commit()
#app.route('/')
def index():
"""The scenario: the first request returns "old name" as expected.
Then, I modify the name of User:1 to "new name" directly on the database.
On the next request, "new name" will be returned.
My question is: how SQLAlchemy knows that the value has been changed?
"""
# Before, I always use db.session.commit()
# to make sure that the latest value is fetched.
# Without db.session.commit(),
# SQLAlchemy still can track change made on User.name
# print "refresh db"
# db.session.commit()
u = User.query.filter_by(id=1).first()
return u.name
app.run(debug=True)
The "cache" of a session is a dict in its identity_map (session.identity_map.dict) that only caches objects for the time of "a single business transaction" , as answered here https://stackoverflow.com/a/5869795.
For different server requests, you have different identity_map. It is not a shared object.
In your scenario, you requested the server 2 separated times. The second time, the identity_map is a new one (you can easily check it by printing out its pointer), and has nothing in cache. Consequently the session will request the database and get you the updated answer. It does not "track change" as you might think.
So, to your question, you don't need to do session.commit() before a query if you have not done a query for the same object in the same server request.
Hope it helps.
In the event mapper level docs
it says that Session.add() is not supported, but when I tried to do db.session.add(some_object) inside after_insert event it worked, example:
def after_insert_listener(mapper, connection, user):
global_group = Group.query.filter_by(groupname='global').first()
a = Association(user,global_group)
db.session.add(a)
event.listen(User, 'after_insert', after_insert_listener)
Basically any new user should be part of global_group, so I added it in the after_insert event. I tried to insert a user, and then checked into my database and I found the user record, and the association record.
Let's check the diferences:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite://///Users/dedeco/Documents/tmp/testDb.db'
db = SQLAlchemy(app)
>>>type(db.session)
<class 'sqlalchemy.orm.scoping.scoped_session'>
or
from sqlalchemy import *
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
some_engine = create_engine('sqlite://///Users/dedeco/Documents/tmp/testDb.db')
Session = sessionmaker(bind=some_engine)
session = Session()
Base = declarative_base()
>>> type(session)
<class 'sqlalchemy.orm.session.Session'>
Basically the difference is:
In the first way you are using a API developed for the Flask framework, called Flask-SQLAlchemy. It's the option if you are creating a Flask application, because the scope of the Session can be managed automatically by your application. You have many benefits like a infrastructure to establish a single Session, associated with the request, which is correctly constructed and torn down corresponding torn down at the end of a request.
In the second way is a pure SQLAlchemy app, so if you are using a library to connect a particular database, you can use just a SQLAlchemy API, for example, for a command-line script, background daemon, GUI interface-driven application, etc.
So, in a both way you can add, like:
Using a Flask-SQLAlchemy:
class User(db.Model):
__tablename__ = 'users'
user_id = db.Column(db.Integer(), primary_key = True)
user_name = db.Column(db.String(80), unique=True)
def __init__(self, user_name):
self.user_name = user_name
>>> db.create_all()
>>> u = User('user1')
>>> db.session.add(u)
>>> db.session.commit()
>>> users = db.session.query(User).all()
>>> for u in users:
... print u.user_name
...
user1
Using just SQLAlchemy:
class User(Base):
__tablename__ = 'users'
user_id = Column(Integer(), primary_key = True)
user_name = Column(String(80), unique=True)
>>> u = User()
>>> u.user_name = 'user2'
>>> session.add(u)
>>> session.commit()
>>> users = session.query(User).all()
>>> for u in users:
... print u.user_name
...
user1
user2
Realize that I am connecting in the same database just for show that you can add using many ways.
server = Flask(__name__)
app = dash.Dash(__name__,server=server,external_stylesheets=[dbc.themes.LITERA], suppress_callback_exceptions = True)
app.server.config["SQLALCHEMY_DATABASE_URI"] = f'postgresql://postgres:.../...'
db = SQLAlchemy(app.server)
I have the same problem of not knowing at what point I should close the session of the database in my web application. I found this in the link that #GabrielChu shared so what I understood was if you are dealing with a web app the session is closed when the user close their tab
A web application is the easiest case because such an application is already constructed around a single, consistent scope - this is the request, which represents an incoming request from a browser, the processing of that request to formulate a response, and finally the delivery of that response back to the client. Integrating web applications with the Session is then the straightforward task of linking the scope of the Session to that of the request. The Session can be established as the request begins, or using a lazy initialization pattern which establishes one as soon as it is needed. The request then proceeds, with some system in place where application logic can access the current Session in a manner associated with how the actual request object is accessed. As the request ends, the Session is torn down as well, usually through the usage of event hooks provided by the web framework. The transaction used by the Session may also be committed at this point, or alternatively the application may opt for an explicit commit pattern, only committing for those requests where one is warranted, but still always tearing down the Session unconditionally at the end.
Some web frameworks include infrastructure to assist in the task of aligning the lifespan of a Session with that of a web request. This includes products such as Flask-SQLAlchemy, for usage in conjunction with the Flask web framework, and Zope-SQLAlchemy, typically used with the Pyramid framework. SQLAlchemy recommends that these products be used as available