I'm using Flask-Testing which says:
Another gotcha is that Flask-SQLAlchemy also removes the session
instance at the end of every request (as should any threadsafe
application using SQLAlchemy with scoped_session). Therefore the
session is cleared along with any objects added to it every time you
call client.get() or another client method.
However, I'm not seeing that. This test fails:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.testing import TestCase
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)
#app.route('/')
def index():
print 'before request:', `db.session`
u = db.session.query(User).first()
u.name = 'bob'
return ''
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String)
class SessionTest(TestCase):
def create_app(self):
return app
def test_remove(self):
db.drop_all()
db.create_all()
u = User()
u.name = 'joe'
db.session.add(u)
db.session.commit()
client = app.test_client()
client.get('/')
print 'after request:', `db.session`
print u.name
assert u not in db.session
(Run with $ nosetests test_file.py to see it in action.)
stdout:
-------------------- >> begin captured stdout << ---------------------
before request: <sqlalchemy.orm.scoping.ScopedSession object at 0x10224c610>
after request: <sqlalchemy.orm.scoping.ScopedSession object at 0x10224c610>
bob
--------------------- >> end captured stdout << ----------------------
According to the docs, user u should not be in the session after a get request, but it is! Does anybody know why this is happening?
Furthermore, u.name is bob and not joe, even though the request never committed! (So I'm convinced it's the same session.)
For the record,
$ pip freeze | grep Flask
Flask==0.10.1
Flask-Bcrypt==0.5.2
Flask-DebugToolbar==0.8.0
Flask-Failsafe==0.1
Flask-SQLAlchemy==0.16
Flask-Script==0.6.2
Flask-Testing==0.4
Flask-Uploads==0.1.3
Flask-WTF==0.8
I'm pretty sure the confusion comes from the fact that sessions in SQLAlchemy are scoped, meaning that each request handler creates and destroys its own session.
This is necessary because web servers can be multi-threaded, so multiple requests might be served at the same time, each working with a different database session.
For this reason, the session that you used outside of the context of a request is likely not the same session that the view function that handles the '/' route gets and then destroys at the end.
UPDATE: I dug around a bit and figured this thing out.
Flask-SQLAlchemy installs a hook on app.teardown_appcontext, and here is where it calls db.session.remove().
The testing environment does not fully replicate the environment of a real request because it does not push/pop the application context. Because of that the session is never removed at the end of the request.
As a side note, keep in mind that functions registered with before_request and after_request are also not called when you call client.get().
You can force an application context push and pop with a small change to your test:
def test_remove(self):
db.drop_all()
db.create_all()
u = User()
u.name = 'joe'
db.session.add(u)
db.session.commit()
with app.app_context():
client = app.test_client()
client.get('/')
print 'after request:', `db.session`
print u.name
assert u not in db.session
with this change the test passes.
The documentation for Flask-Testing seems to be wrong or more likely outdated. Maybe things worked like they describe at some point, but that isn't accurate for current Flask and Flask-SQLAlchemy versions.
I hope this helps!
FlaskClient works with request context while Flask-SQLAlchemy calls it's shutdown_session on app.teardown_appcontext since Flask version 0.9. Thats why nothing happens after test client call, bacause app context started by flask.ext.testing.TestCase even before test's setUp and will be closed after tearDown.
I got the same problem when tried to use Flask-Manage to run my tests. Running tests in a separate thread solved the problem.
import threading
# some code omited
runner = unittest.TextTestRunner(verbosity=2)
t = threading.Thread(target=runner.run, args=[test_suite])
t.start()
t.join()
# other code omited
Related
I have couple of simple tasks which could take maximum 20 seconds to complete, so I decided to use separate thread to accomplish them. I want thread to do the job and update database with result.
While it works (no exceptions yet) I have lack of understanding Flask internals and how it works with WSGI server. I'm not quite sure that on some amount of parallel requests it won't end with some database access error.
Simplified code:
from time import time, sleep
from threading import Thread
from peewee import *
from playhouse.shortcuts import model_to_dict
from flask import Flask, abort, jsonify
db = SqliteDatabase("test.db")
Flask(__name__)
class Task(Model):
status = IntegerField(default=0)
result = TextField(null=True)
class Meta:
database = db
def do_smth(task_id):
start = time()
sleep(10)
# DATABASE UPDATE HERE
Task.update({Task.status: 1, Task.result: f"{start} - {time()}"})\
.where(Task.id == task_id).execute()
#app.route("/new")
def new_task():
try:
task = Task.create()
except IntegrityError:
abort(500)
else:
Thread(target=do_smth, args=(task.id,)).start()
return jsonify(model_to_dict(task))
#app.route("/get/<int:task_id>")
def get_task(task_id):
try:
task = Task.get(Task.id == task_id)
except Task.DoesNotExist:
abort(404)
else:
return jsonify(model_to_dict(task))
#app.before_request
def before_request():
db.connect()
#app.after_request
def after_request(response):
db.close()
return response
if __name__ == "__main__":
with db:
db.create_tables([Task])
app.run(host="127.0.0.1", port=5000)
As it suggested in peewee tutorial I added custom Flask.before_request and Flask.after_request which open and close database connection.
So the question is how to update database from separate thread safely? I have had an idea to add route which will update database and send request from thread, but I find it kinda dumb.
P.S. I've tried my best trying to be precise, but if something is unclear I will try to clarify it, just ask it in comments section.
This is a good question:
how to update database from separate thread safely?
With Sqlite you have to remember that it only allows one writer at a time. So you have to manage your connections carefully to ensure that you are only doing a write txn when you have to, and that you're committing it as soon as you're done with it.
Since you're opening and closing the DB during the lifetime of a request, and running your DB operations in separate thread(s), you should be OK for a smallish number of operations (100?). I think the main thing I'd be careful about is, during your task body, be sure you're only holding that write txn open for as short a time as possible:
def do_smth(task_id):
# open a database connection. it will be read-only for now.
with db.connection_context():
start = time()
sleep(10)
with db.atomic() as txn: # here is write tx, keep this brief!
Task.update({Task.status: 1, Task.result: f"{start} - {time()}"})\
.where(Task.id == task_id).execute()
See the first section on transactions: https://charlesleifer.com/blog/going-fast-with-sqlite-and-python/
For a more drastic approach, you can try this: https://charlesleifer.com/blog/multi-threaded-sqlite-without-the-operationalerrors/
Before asking you, I tried to search as best as I could in SO for similar questions. I've read https://docs.pytest.org/en/latest/explanation/fixtures.html page twice. However, I could not solve the problem I am facing with. I am new to Python and Flask. So I thought that I am missing something obvious.
I have a small micky-mouse Flask application. I am trying to write tests for this. I voted for using Pytest because fixtures seemed more flexible than unittest. To my surprise, first test case runs with no problem at all. However the exact same test case fails to run for the second time.
Here is my fixtures.py file:
from config import Config
import pytest
from app import create_app, db
from app.models import User, Role
class TestConfig(Config):
SECRET_KEY = 'Shhhhh 007! Be quiet.!'
SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'
SQLALCHEMY_TRACK_MODIFICATIONS = False
EXPLAIN_TEMPLATE_LOADING = True
DEBUG = True
API_VERSION = 'temp'
#pytest.fixture
def client():
testConfig = TestConfig()
app = create_app(testConfig)
from app import routes, models
with app.test_client() as client:
with app.app_context():
db.create_all()
yield client
#pytest.fixture(autouse=True)
def role(client):
role = Role()
role.name = 'test role'
db.session.add(role)
db.session.commit()
yield role
db.session.delete(role)
db.session.commit()
#pytest.fixture(autouse=True)
def user(role):
user = User()
user.role_id = role.id
user.name = 'test name'
user._normalized_name = 'test name'
user.lastname = 'test lastname'
user._normalized_lastname = 'test lastname'
user.phonenumber = '+11231234567'
user.setPassword('testPassword')
db.session.add(user)
db.session.commit()
yield user
db.session.delete(user)
db.session.commit()
And here is my test_login.py file:
import logging
from tests.fixtures import client, role, user
def test_login_page_Once(client):
rv = client.get('/login')
decodedRv = rv.data.decode('utf-8')
print(decodedRv)
assert '<title>Signin' in decodedRv
def test_login_page_Twice(client):
rv = client.get('/login')
decodedRv = rv.data.decode('utf-8')
print(decodedRv)
assert '<title>Signin' in decodedRv
I run my tests using:
pytest -v -k test_login
And here is the outcome:
tests/functional/test_login.py::test_login_page_Once PASSED [ 50%]
tests/functional/test_login.py::test_login_page_Twice FAILED [100%]
========================================== FAILURES ===========================================
____________________________________ test_login_page_Twice ____________________________________
client = <FlaskClient <Flask 'app'>>
def test_login_page_Twice(client):
rv = client.get('/login')
decodedRv = rv.data.decode('utf-8')
print(decodedRv)
> assert '<title>Signin' in decodedRv
E assert '<title>Signin' in '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.
If you entered the URL manually please check your spelling and try again.</p>\n'
tests\functional\test_login.py:17: AssertionError
------------------------------------ Captured stdout call -------------------------------------
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
=================================== short test summary info ===================================
FAILED tests/functional/test_login.py::test_login_page_Twice - assert '<title>Signin' in '<!D...
========================== 1 failed, 1 passed, 2 deselected in 1.57s ==========================
What am I missing? Any help would be highly appreciated. Thanks very much.
Well, I found a solution to the problem for which I was banging my head against the wall for a while. Though I reached that solution by try and error, it is still beyond me why/how it works. If any experts of pytest/flask would give me an explanation, that would be lovely.
I would also appreciate if someone tells me what the differences are among fixture scopes. ( I have read https://docs.pytest.org/en/latest/reference/reference.html#pytest.fixture already but to me it is not clear.)
Thanks.
The only change I made in the code is that I added
scope='session'
to the client fixture.
#pytest.fixture(scope='session')
def client():
testConfig = TestConfig()
app = create_app(testConfig)
from app import routes, models
with app.test_client() as client:
with app.app_context():
db.create_all()
yield client
ADDITION #2:
Further down the road, another test of mine failed with similar symptoms. The difference was this test case living in another file (I encountered previous problems while I was testing login page/behavior. Now I am dealing with registration page/behavior). I could not find my way around this error for a while. (I even considered dropping of pytest and going back to unittest. But I am still here because I am a bit of a stubborn. That's a different story!)
Anyways, I realized that one way of grouping fixtures is to place them in a file named conftest.py. Mine was named fixtures.py. In my head, it was just a name and I thought mine was clearer. Well, I was wrong. conftest.py seems to be a magic file. Once I renamed the fixtures.py to conftest.py test all marched pass in shining green armors.
Adding my two cents here just in case anyone else feels lost in the depths of fixtures/routes/imports forests.
I had a similar issue and it turned out that I was accidentally mutating an object inside a function and that was having side effects outside the function.
First time I ran the fixture it worked fine.
Second time I ran the fixture, the function that defined the fixture was working with different data because of a side effect of the first time I ran the function. This gave an error trying to run the function.
Not sure if that is your problem, but this might help someone facing a similar issue.
I know it is possible to create session object using session_transaction() method. However, is there a way to access the current session object which gets created when for example "/" route gets hit? I did from flask import session to access the session but it's empty. Let me know if it is possible. Thanks.
This is what you're looking for. As it says however, you'd have to use the instantiation you create in your with statement.
with app.test_client() as c:
with c.session_transaction() as sess:
sess['a_key'] = 'a value'
# once this is reached the session was stored
result = app.test_client.get('/a_url')
# NOT part of the 2nd context
Note that this won't work if you run your test within the scope of the with c.session_transaction() as sess statement, it needs to be run after that block.
If you want to read the session data written in your view from the test, one way is to mock the session view as a dict and verify the session in your test. Here's an example using Python's unittest.mock:
app.py
from flask import Flask, session, request
app = Flask(__name__)
app.config["SECRET_KEY"] = "my secret key"
#app.route("/", methods=["POST"])
def index():
session["username"] = request.form["username"]
return "Username saved in session"
test_index.py
from unittest.mock import patch
from app import app
def test_index():
with patch("app.session", dict()) as session:
client = app.test_client()
response = client.post("/", data={
"username": "test"
})
assert session.get("username") == "test"
assert response.data == b"Username saved in session"
You can use any mocking solution you prefer, of course.
I have a Flask REST API, running with a gunicorn/nginx stack. There is global SQLAlchemy session set up once for each thread that the API runs on. I set up an endpoint /test/ for running the unit tests for the API. One test makes a POST request to add something to the database, then has a finally: clause to clean up:
def test_something():
try:
url = "http://myposturl"
data = {"content" : "test post"}
headers = {'content-type': 'application/json'}
result = requests.post(url, json=data, headers=headers).json()
validate(result, myschema)
finally:
db.sqlsession.query(MyTable).filter(MyTable.content == "test post").delete()
db.sqlsession.commit()
The problem is that the thread to which the POST request is made now has a "test post" object in its session, but the database has no such object because the thread on which the tests ran deleted that thing from the database. So when I make a GET request to the server, about 1 in 4 times (I have 4 gunicorn workers), I get the "test post" object, and 3 in 4 times I do not. This is because the threads each have their own session object, and they are getting out of sync, but I don't really know what to do about it....
Here is my setup for my SQLAlchemy session:
def connectSQLAlchemy():
import sqlalchemy
import sqlalchemy.orm
engine = sqlalchemy.create_engine(connection_string(DBConfig.USER, DBConfig.PASSWORD, DBConfig.HOST, DBConfig.DB))
session_factory = sqlalchemy.orm.sessionmaker(bind=engine)
Session = sqlalchemy.orm.scoped_session(session_factory)
return Session()
# Create a global session for everyone
sqlsession = connectSQLAlchemy()
Please use flask-sqlalchemy if you're using flask, it takes care of the lifecycle of the session for you.
If you insist on doing it yourself, the correct pattern is to create a session for each request instead of having a global session. You should be doing
Session = scoped_session(session_factory, scopefunc=flask._app_ctx_stack.__ident_func__)
return Session
instead of
Session = scoped_session(session_factory)
return Session()
And do
session = Session()
every time you need a session. By virtue of the scoped_session and the scopefunc, this will return you a different session in each request, but the same session in the same request.
Figured it out. What I did was to add a setup and teardown to the request in my app's __init__.py:
#app.before_request
def startup_session():
db.session = db.connectSQLAlchemy()
#app.teardown_request
def shutdown_session(exception=None):
db.session.close()
still using the global session object in my db module:
db.py:
....
session = None
....
The scoped_session handles the different threads, I think...
Please advise if this is a terrible way to do this for some reason. =c)
I'm making some Django projects using Redis as Backend Cache[1], but I can't be sure that the Redis Server will be On all the time, then I'm trying to use Redis "if" it's available otherwise use some other Backend like LocMem and so on.
The Redis Backend that I'm using[1] is full compatible so I can use Django Decorations.
I was think to create a function to be called like that:
from django.views.decorators.cache import cache_page
from utils import PingBackend
from time import time
#cache_page(60, cache=PingBackend(time()))
def index(request):
artigos = Artigo.objects.filter(ativo=1)
return render_to_response('index.html', {'artigos':artigos}, RequestContext(request))
The problem is that Django (Internals I guess) Caches the response of PingBackend() and call it just the first time, even if I drop the RedisServer Django tells that the ping process was successfully.
It occurs even with DEBUG=True and 'default' CacheBackend to dummy.
def PingBackend(time):
print time
response = None
try:
con = StrictRedis(host=settings.REDIS_HOST, port=settings.REDIS_PORT, db=0)
# Execute some action
con.ping()
# If not give an exception, use redis
response = 'redis'
except:
response = 'default' #dummy
return last_response
I'm passing time() just to create some differentiation as a try solve the cache problm.
The big picture is that the function PingBackend() aren't executing for each request, just for the first the I can't monitor the Redis Server.
Thank you!
[1] - https://github.com/niwibe/django-redis
This is not about Django internals, this is about how decorators work. When you define your view like this:
#cache_page(60, cache=PingBackend(time()))
def index(request):
blah blah
it is exactly equivalent to this:
def index(request):
blah blah
index = cache_page(60, cache=PingBackend(time()))(index)
You are invoking cache_page only one, passing it a cache argument that you got by invoking PingBackend once. It isn't even executing just for the first request, it is executing once when the view function is defined.
You should write your own cache backend that uses Redis if it's available, or something else if it is not.