I am building an app where users will occasionally initiate a longer-running process. While running, the process will commit updates to a database entry.
Since the process takes some time, I am using the threading module to execute it. But values updated while in the thread are never actually committed.
An example:
from flask import Flask, url_for, redirect
from flask_sqlalchemy import SQLAlchemy
import time, threading, os
if os.path.exists('test.db'): os.remove('test.db')
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db'
db = SQLAlchemy(app)
class Item(db.Model):
id = db.Column(db.Integer, primary_key=True)
value = db.Column(db.Integer)
def __init__(self, value): self.value = value
db.create_all()
item = Item(1)
db.session.add(item)
db.session.commit()
#app.route('/go', methods=['GET'])
def go():
def fun(item):
time.sleep(2)
item.value += 1
db.session.commit()
thr = threading.Thread(target=fun, args=(item,))
# thr.daemon = True
thr.start()
return redirect(url_for('view'))
#app.route('/view', methods=['GET'])
def view(): return str(Item.query.get(1).value)
app.run(host='0.0.0.0', port=8080, debug=True)
My expectation was that the item's value would be asynchronously updated after two seconds (when the fun completes), and that additional requests to /view would reveal the updated value. But this never occurs. I am not an expert on what is going on in the threading module; am I missing something?
I have tried setting thr.daemon=True as pointed out in some posts; but that is not it. The closest SO post I have found is this one; that question does not have a minimal and verifiable example and has not been answered.
I guess this is due to the fact that sessions are local threaded, as mentioned in the documentation. In your case, item was created in one thread and then passed to a new thread to be modified directly.
You can either use scoped sessions as suggested in the documentation, or simply change your URI config to bypass this behavior:
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db?check_same_thread=False'
After some debugging I figured out a solution; though I still do not understand the problem. It has to do with referencing a variable for the database object. If fun updates an object returned by a query, it works as expected:
def fun(item_id):
time.sleep(2)
Item.query.get(item_id).value += 1
db.session.commit()
In context:
from flask import Flask, url_for, redirect
from flask_sqlalchemy import SQLAlchemy
import time, threading, os
if os.path.exists('test.db'): os.remove('test.db')
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db'
db = SQLAlchemy(app)
class Item(db.Model):
id = db.Column(db.Integer, primary_key=True)
value = db.Column(db.Integer)
def __init__(self, value): self.value = value
db.create_all()
item = Item(1)
db.session.add(item)
db.session.commit()
#app.route('/go', methods=['GET'])
def go():
def fun(item_id):
time.sleep(2)
Item.query.get(item_id).value += 1
db.session.commit()
thr = threading.Thread(target=fun, args=(item.id,))
# thr.daemon = True
thr.start()
return redirect(url_for('view'))
#app.route('/view', methods=['GET'])
def view(): return str(Item.query.get(1).value)
app.run(host='0.0.0.0', port=8080, debug=True)
I would be very pleased to hear from anyone knows what exactly is going on here!
Related
I've looked at the other posts with this same title, but they did not fix my problem. The error message in the above title is what I am getting.
I've made a simple database and constructor for the database. Just to test, I'm trying to add the word "hello" to the column "words" in the class User. But I keep getting this error. I've already tried putting autoincrement in the class column, but I ended it up getting another error. What is going wrong? Please see the code below. Python and Flask.
from flask import Flask, request, redirect, url_for, render_template
from datetime import datetime
from flask_sqlalchemy import SQLAlchemy
import sqlite3
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE URI"] = 'sqlite:///test.db'
db = SQLAlchemy(app)
class User(db.Model):
__tablename__ = 'user'
words = db.Column(db.String(200), primary_key=True)
def __init__(self, thewords):
self.thewords = thewords
db.create_all()
db.session.commit()
texts = "hello"
textstuffent = User(texts)
db.session.add(textstuffent)
db.session.commit()
results = User.query.all()
print(results)
#app.route('/', methods = ['POST', 'GET'])
def hello():
return render_template('Textarea.html')
app.run(debug=True)
For anyone who visits this page in the future, I've found what was wrong. I did not reference "words" as "self.words". My code now works and I am relieved to finally continue coding so that some other bug can confound me later down the road. Please see below. You can see the difference in code in the User class. I used query.all() to prove that data has made it into the database.
from flask import Flask, request, redirect, url_for, render_template
from datetime import datetime
from flask_sqlalchemy import SQLAlchemy
import sqlite3
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE URI"] = 'sqlite:///test.db'
db = SQLAlchemy(app)
class User(db.Model):
words = db.Column(db.String(200), primary_key=True, nullable=False)
def __init__(self, thewords):
self.words = thewords
db.create_all()
db.session.commit()
texts = "hello"
textstuffent = User(texts)
db.session.add(textstuffent)
db.session.commit()
print(User.query.all())
#app.route('/', methods = ['POST', 'GET'])
def hello():
return render_template('Textarea.html')
app.run(debug=True)
when I run tests It succeeds to connect to the database, but it does not create tables. I think maybe there is a different way to create tables when I use flask-sqlalchemy, but I can't find the solution.
This is app.py
db = SQLAlchemy()
def create_app(config_name):
app = Flask(__name__, template_folder='templates')
app.wsgi_app = ProxyFix(app.wsgi_app)
app.config.from_object(config_name)
app.register_blueprint(api)
db.init_app(app)
#app.route('/ping')
def health_check():
return jsonify(dict(ok='ok'))
#app.errorhandler(404)
def ignore_error(err):
return jsonify()
app.add_url_rule('/urls', view_func=Shorty.as_view('urls'))
return app
This is run.py
environment = environ['TINY_ENV']
config = config_by_name[environment]
app = create_app(config)
if __name__ == '__main__':
app.run()
This is config.py
import os
basedir = os.path.abspath(os.path.dirname(__file__))
class Config:
"""
set Flask configuration vars
"""
# General config
DEBUG = True
TESTING = False
# Database
SECRET_KEY = os.environ.get('SECRET_KEY', 'my_precious_secret_key')
SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://root#localhost:3306/tiny'
SQLALCHEMY_TRACK_MODIFICATIONS = False
SERVER_HOST = 'localhost'
SERVER_PORT = '5000'
class TestConfig(Config):
"""
config for test
"""
TESTING = True
SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://root#localhost:3306/test_tiny'
config_by_name = dict(
test=TestConfig,
local=Config
)
key = Config.SECRET_KEY
This is models.py
from datetime import datetime
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
class URLS(db.Model):
__tablename__ = 'urls'
id = db.Column(db.Integer, primary_key=True)
original_url = db.Column(db.String(400), nullable=False)
short_url = db.Column(db.String(200), nullable=False)
created_at = db.Column(db.DateTime, default=datetime.utcnow()
This is test config setting.
db = SQLAlchemy()
#pytest.fixture(scope='session')
def app():
test_config = config_by_name['test']
app = create_app(test_config)
app.app_context().push()
return app
#pytest.fixture(scope='session')
def client(app):
return app.test_client()
#pytest.fixture(scope='session')
def init_db(app):
db.init_app(app)
db.create_all()
yield db
db.drop_all()
The following might be the problem that is preventing your code from running multiple times and/or preventing you from dropping/creating your tables. Regardless if it solves your problem, it is something one might not be aware of and quite important to keep in mind. :)
When you are running your tests multiple times, db.drop_all() might not be called (because one of your tests failed) and therefore, it might not be able to create the tables on the next run (since they are already existing). The problem lies in using a context manager without a try: finally:. (NOTE: Every fixture using yield is a context manager).
from contextlib import contextmanager
def test_foo(db):
print('begin foo')
raise RuntimeError()
print('end foo')
#contextmanager
def get_db():
print('before')
yield 'DB object'
print('after')
This code represents your code, but without using the functionality of pytest. Pytest is running it more or less like
try:
with get_db(app) as db:
test_foo(db)
except Exception as e:
print('Test failed')
One would expect an output similar to:
before
begin_foo
after
Test failed
but we only get
before
begin_foo
Test failed
While the contextmanager is active (yield has been executed), our test method is running. If an exception is raised during the execution of our test function, the execution is stopped WITHOUT running any code after the yield statement. To prevent this, we have to wrap our fixture/contextmanager in a try: ... finally: block. As finally is ALWAYS executed regardless of what has happened.
#contextmanager
def get_db():
print('before')
try:
yield 'DB object'
finally:
print('after')
The code after the yield statement is now executed as expected.
before
begin foo
after
Test failed
If you want to learn more, see the relevant section in the contextmanager docs:
At the point where the generator yields, the block nested in the with statement is
executed. The generator is then resumed after the block is exited. If an unhandled
exception occurs in the block, it is reraised inside the generator at the point
where the yield occurred. Thus, you can use a try…except…finally statement to trap
the error (if any), or ensure that some cleanup takes place.
So, I'm trying to rollback the database session in case of an HTTP error like a bad_request, unauthorized, forbidden, or not_found happens.
it is a serverless application with wsgi and flask.
The scenario is: I create an entry to be saved in the database, but if something wrong happens, I want it to roll_back the session.
If, I raise an exception, the rollback happens, but if I use abort(make_response(jsonify(message=message, **kwargs), 400)) an HTTPException is raised, but the teardown_appcontext kind of ignores it.
I also tried application.config['PRESERVE_CONTEXT_ON_EXCEPTION'] = True #and false too but it didn't solve my problem.
In my app:
def database(application, engine=None):
sqlalchemy_url = os.environ.get('SQLALCHEMY_URL')
set_session(sqlalchemy_url, engine=engine)
#application.teardown_appcontext
def finish_session(exception=None):
commit_session(exception)
def commit_session(exception=None):
if exception:
_dbsession.rollback()
else:
_dbsession.commit()
_dbsession.remove()
if hasattr(_engine, 'dispose'):
_engine.dispose()
And here, the function that is called if I want to return an bad_request response. The abort function raises an HTTPException that is ignored by the teardown function
def badrequest(message='bad request.', **kwargs):
abort(make_response(jsonify(message=message, **kwargs), 400))
I want the teardown_appcontext to recognize the HTTPException too, not only an Exception. In this way, if the abort function is called, the rollback will be done.
I think this is because teardown_appcontext called when the request context is popped. An exception was init in context of request. You can rollback session using errorhandler() or register_error_handler(). Here is an example:
from flask import Flask, abort, jsonify
from flask_sqlalchemy import SQLAlchemy
from werkzeug.exceptions import BadRequest
app = Flask(__name__)
app.config.update(dict(SQLALCHEMY_DATABASE_URI='...'))
db = SQLAlchemy(app)
class Node(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(100), nullable=False)
#app.errorhandler(BadRequest)
def handle_bad_request(e):
db.session.rollback()
return 'session has been rolled back!', 400
#app.teardown_appcontext
def finish_session(exception=None):
if not exception:
db.session.commit()
#app.route('/bad-node')
def bad():
# add into session without commit and abort(see: handle_bad_request)
db.session.add(Node(name='bad node'))
abort(400)
#app.route('/good-node')
def good():
# without exceptions - see: finish_session
db.session.add(Node(name='good node'))
return '<good node> was saved'
#app.route('/nodes')
def all_nodes():
# just list of items from db
return jsonify([i.name for i in Node.query.all()])
if __name__ == '__main__':
db.create_all()
db.session.commit()
app.run(debug=True)
Open /good-node and /bad-node a few times. After that open /nodes you will see that 'bad nodes' were not saved(was rollback).
Hope this helps.
I have a heroku app that I am trying to add a database to. I am using Flask-SLAlchemy, PostgreSQL (and psql). I've already created a table in the database, but I cannot add any rows to it. Here is what I believe to be all relevant code:
import flask
import keys
import requests_oauthlib
import json
import os
import psycopg2
import urlparse
from flask import (Flask, jsonify, render_template, redirect, url_for, request, make_response)
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'heroku-url-here'
db = SQLAlchemy(app)
class Page (db.Model):
__tablename__ = "pages"
title = db.Column('Title', db.String)
date = db.Column('Date', db.String, primary_key=True)
writing = db.Column('Writing', db.String)
def __init__(self, title, date, writing):
self.title = title
self.date = date
self.writing = writing
def __repr__(self):
return '<Page %r>' % self.date
app.secret_key = keys.secret_key
# db.create_all()
# this created the database already after I ran it once, it made a psycopg2 error after that first time.
#app.route('/db', methods=['GET', 'POST'])
def db():
if request.method == 'POST':
title = request.form['title']
date = request.form['date']
writing = request.form['writing']
newest = Page(title, date, writing)
print newest
db.session.add(newest)
db.session.commit()
else:
title = None
date = None
writing = None
return flask.redirect(flask.url_for('home'))
In my heroku logs, there are no errors shown. The code runs to the the print newest line, and the newly created Page is printed as <Page u'whatever-the-date-was'>. When a form is submitted in my html template, it calls the function by using the action {{url_for('db')}}.
This is my first time using heroku and flask and basically doing any back-end stuff, so please explain thoroughly if you have an answer. Thanks in advance!
Take advantage of your Page model here.
db.session.add(Page(
title = request.form['title']
date = request.form['date']
writing = request.form['writing']
))
db.session.commit()
You'll probably also run into trouble with your conditional - if the method isn't POST then nothing will happen, and there won't be any message logged about it. If you remove the 'GET' from the methods in the route declaration you won't need that conditional at all.
I'd recommend taking a look at the Flask-WTF extension, as well as breaking out your form validation and redirect steps into separate functions. Flask works best by breaking down elements to their smallest usable components and then reassembling them in many different ways.
For more info on form handling, check out Miguel Grinberg's Flask Mega-Tutorial (if you haven't already).
I am trying to set up Celery tasks. Our main app is Pyramid with SQLAlchemy.
So I have a task defined as:
from celery.contrib.methods import task
from apipython.celerytasks import celery
class Email():
def __init__(self, from_name, from_email, to_name, to_email, subject, html_body,
sendgrid_category=None):
self.from_name = from_name
self.from_email = from_email
self.to_name = to_name
self.to_email = to_email
self.subject = subject
self.body = None
self.html_body = html_body
self.sendgrid_category = sendgrid_category
class EmailService():
#task()
def task__send_smtp(self, email, from_user_id=None, to_user_id=None):
# send the email, not shown here
# EmailLog is a SQLAlchemy model
email_log = EmailLog(
email.subject,
email.html_body,
from_user_id=from_user_id,
to_user_id=to_user_id,
action_type=email.sendgrid_category)
DBSession.add(email_log)
transaction.commit()
And celerytasks.py I have:
from celery import Celery
celery = Celery('apipython.celery',
broker='sqla+mysql+mysqldb://root:notarealpassword#127.0.0.1/gs?charset=utf8',
backend=None,
include=['apipython.services.NotificationService'])
if __name__ == '__main__':
celery.start()
It works - the task gets serialized and picked up.
However when I try to use SQLAlchemy / DBSession inside the task, I get an error:
UnboundExecutionError: Could not locate a bind configured on mapper Mapper|EmailLog|emaillogs or this Session
I understand the worker task is running on a separate process and need to have its settings, session, engine etc set up. So I have this:
#worker_init.connect
def bootstrap_pyramid(signal, sender):
import os
from pyramid.paster import bootstrap
sender.app.settings = bootstrap('development.ini')['registry'].settings
customize_settings(sender.app.settings)
engine = sqlalchemy.create_engine('mysql+mysqldb://root:notarealpassword#127.0.0.1/gs?charset=utf8')
DBSession.configure(bind=engine)
Base.metadata.bind = engine
However I am still getting the same error.
DBSession and Base are defined in models.py as
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base()
What step am I missing to make the models binding work?
Second question, can this code for creating session / binding work in celery's init, vs worker init?
(BTW I did try pyramid_celery but prefer to make plain celery work)
Thanks,
My colleague tried the exact same code and it worked. Strange