I have a Flask app that takes parameters from a web form, queries a DB with SQL Alchemy and returns Jinja-generated HTML showing a table with the results. I want to cache the calls to the DB. I looked into Redis (Using redis as an LRU cache for postgres), which led me to http://pythonhosted.org/Flask-Cache/.
Now I am trying to use Redis + Flask-Cache to cache the calls to the DB. Based on the Flask-Cache docs, it seems like I need to set up a custom Redis cache.
class RedisCache(BaseCache):
def __init__(self, servers, default_timeout=500):
pass
def redis(app, config, args, kwargs):
args.append(app.config['REDIS_SERVERS'])
return RedisCache(*args, **kwargs)
From there I would need to something like:
# not sure what to put for args or kwargs
cache = redis(app, config={'CACHE_TYPE': 'redis'})
app = Flask(__name__)
cache.init_app(app)
I have two questions:
What do I put for args and kwargs? What do these mean? How do I set up a Redis cache with Flask-Cache?
Once the cache is set up, it seems like I would want to somehow "memoize" the calls the DB so that if the method gets the same query it has the output cached. How do I do this? My best guess would be to wrap the call the SQL Alchemy in a method that could then be given memoize decorator? That way if two identical queries were passed to the method, Flask-Cache would recognize this and return to the appropriate response. I'm guessing that it would look like this:
#cache.memoize(timeout=50)
def queryDB(q):
return q.all()
This seems like a fairly common use of Redis + Flask + Flask-Cache + SQL Alchemy, but I am unable to find a complete example to follow. If someone could post one, that would be super helpful -- but for me and for others down the line.
You don't need to create custom RedisCache class. The docs is just teaching how you would create new backends that are not available in flask-cache. But RedisCache is already available in werkzeug >= 0.7, which you might have already installed because it is one of the core dependencies of flask.
This is how I could run the flask-cache with redis backend:
import time
from flask import Flask
from flask_cache import Cache
app = Flask(__name__)
cache = Cache(app, config={'CACHE_TYPE': 'redis'})
#cache.memoize(timeout=60)
def query_db():
time.sleep(5)
return "Results from DB"
#app.route('/')
def index():
return query_db()
app.run(debug=True)
The reason you're getting "ImportError: redis is not a valid FlaskCache backend" is probably because you don't have redis (python library) installed which you can simply install by:
pip install redis.
your redis args would look something like this:
cache = Cache(app, config={
'CACHE_TYPE': 'redis',
'CACHE_KEY_PREFIX': 'fcache',
'CACHE_REDIS_HOST': 'localhost',
'CACHE_REDIS_PORT': '6379',
'CACHE_REDIS_URL': 'redis://localhost:6379'
})
Putting the #cache.memoize over a method that grabs the info from the DB should work.
Related
Trying to test an endpoint that has a celery task. Celery tasks don't seem to run in the test.
django==4.1.5
celery==5.2.7
pytest==7.2.1
pytest-django==4.5.2
An endpoint:
def do_some_stuff(blah: Blah) -> Blah:
res = cool_task.apply_async(kwargs={
'cool_id': int(pk),
'config': config,
'name': RESUBMIT.value,
},
link=update_status.si(
cool_id=int(pk),
new_status="why is this so hard",
)
)
[...]
A test:
#pytest.mark.django_db
def test_my_thing(django_client: Client) -> None:
[...]
response = django_client.post(f"/api/myendpoint/{mything.id}/do_some_stuff/")
It hits the endpoint. Gets a 202 back as expected. But celery doesn't seem to be picking up the task in the test. The update_status method updates the db, and I'm not seeing that happen.
I've tried creating a celery app in the test, creating a worker in the test, changing the test to use the main DB instead of the test db, setting the override_settings to "BROKER_BANDEND='memory'".
I'd like a full working example. It seems kind of basic but it's eluding me. I don't understand what combination of fixtures and overrides I need for this to work.
Seems to work when I actually call the application.
I am running Pyramid as my API server. Recently we started getting query string parameters out of order when handed to the RESTful API server. For example, a GET to /v1/finishedGoodRequests?exact=true&id=39&join=OR&exact=false&name=39
is logged by the RESTful api module upon init as request.url:
v1/finishedGoodRequests?join=OR&name=39&exact=true&exact=false&id=39
with request.query_string: join=OR&name=39&exact=true&exact=false&id=39
I process the query params in order to qualify the search, in this case id exactly 39 or 39 anywhere in the name. What kind of possible server setting or bug could have crept in to the server code to cause such a thing? It is still a MultiDict...
As a simple example, this works fine for me, and the MultiDict has always preserved the order and so I suspect something is getting rewritten by something you're using in your stack.
from pyramid.config import Configurator
from pyramid.view import view_config
from waitress import serve
#view_config(renderer='json')
def view(request):
return list(request.GET.items())
config = Configurator()
config.scan(__name__)
app = config.make_wsgi_app()
serve(app, listen='127.0.0.1:8080')
$ curl http://localhost:8080\?join=OR\&name=39\&exact=true\&exact=false\&id=39
[["join", "OR"], ["name", "39"], ["exact", "true"], ["exact", "false"], ["id", "39"]]
Depending on which WSGI server you are using, often you can view environ vars to see the original url which may be handy. Waitress does not, so instead just put something high up in the pipeline (wsgi middleware) that can log out the environ['QUERY_STRING'] and see if it doesn't match somewhere lower down in your stack.
I'm trying to set up a Flask API Server from which I can data from local database via an ongoing HTTP Request to another database.
In the local code, I run a thread that is running and updating the local DB every 1 minute.
app = Flask(__name__)
cached_event_log = None
#app.route('/event_log', methods=['GET'])
def get_event_log():
if cached_event_log != None and .get_latest_event_time == cached_event_log[-1]:
return jsonify(cached_event_log)
#MAKE CONNECTION TO DB AND GET THE DATA
return jsonify(event_log)
if(__name__ == '__main__'):
app.run(Debug=True)
I'm struggling to find a "standard" way to set up A request.
Any opinion would be highly appreciated- Thank you,
You could set up a cron job to run a script at a specified interval
Or use something like Advanced Python Scheduler
https://apscheduler.readthedocs.io/en/3.0/
Advanced Python Scheduler support in Flask
https://github.com/viniciuschiele/flask-apscheduler
I've been thinking about the factory pattern for WSGI applications, as recommended by the Flask docs, for a while now. Specifically about those functions usually being shown to make use of objects that have been created at module import time, like db in the example, as opposed to having been created in the factory function.
Would the factory function ideally create _everything_ anew or wouldn't that make sense for objects like the db engine?
(I'm thinking cleaner separation and better testability here.)
Here is some code, where I'm trying to accomplish creating all needed objects for the wsgi app. in its factory function.
# factories.py
def create_app(config, engine=None):
"""Create WSGI application to be called by WSGI server. Full factory function
that takes care to deliver entirely new WSGI application instance with all
new member objects like database engine etc.
Args:
config (dict): Dict to update the wsgi app. configuration.
engine (SQLAlchemy engine): Database engine to use.
"""
# flask app
app = Flask(__name__) # should be package name instead of __name__ acc. to docs
app.config.update(config)
# create blueprint
blueprint = ViewRegistrationBlueprint('blueprint', __name__, )
# request teardown behaviour, always called, even on unhandled exceptions
# register views for blueprint
from myapp.views import hello_world
# dynamically scrapes module and registers methods as views
blueprint.register_routes(hello_world)
# create engine and request scoped session for current configuration and store
# on wsgi app
if (engine is not None):
# delivers transactional scope when called
RequestScopedSession = scoped_session(
sessionmaker(bind=engine),
scopefunc=flask_request_scope_func
)
def request_scoped_session_teardown(*args, **kwargs):
"""Function to register and call by the framework when a request is finished
and the session should be removed.
"""
# wrapped in try/finally to make sure no error collapses call stack here
try:
RequestScopedSession.remove() # rollback all pending changes, close and return conn. to pool
except Exception as exception_instance:
msg = "Error removing session in request teardown.\n{}"
msg = msg.format(exception_instance)
logger.error(msg)
finally:
pass
app.config["session"] = RequestScopedSession
blueprint.teardown_request(request_scoped_session_teardown)
# register blueprint
app.register_blueprint(blueprint)
return app
def create_engine(config):
"""Create database engine from configuration
Args:
config (dict): Dict used to assemble the connection string.
"""
# connection_string
connection_string = "{connector}://{user}:{password}#{host}/{schema}"
connection_string = connection_string.format(**config)
# database engine
return sqlalchemy_create_engine(
connection_string,
pool_size=10,
pool_recycle=7200,
max_overflow=0,
echo=True
)
# wsgi.py (served by WSGI server)
from myapp.factories import create_app
from myapp.factories import create_engine
from myapp.configuration.config import Config
config = Config()
engine = create_engine(config.database_config)
app = create_app(config.application_config, engine=engine)
# conftest.py
from myapp.factories import create_app
from myapp.factories import create_engine
from myapp.configuration.config import Config
#pytest.fixture
def app():
config = TestConfig()
engine = create_engine(config.database_config)
app = create_app(config.application_config, engine=engine)
with app.app_context():
yield app
As you also tagged this with sanic I'll answer with that background. Sanic is async and thus relies on an event loop. An event loop is a resource and thus must not be shared between tests but created anew for each one. Hence, the database connection etc also need to be created for each test and cannot be re-used as it is async and depends on the event loop. Even without the async nature it would be cleanest to create db connections per test because they have state (like temp tables).
So I ended up with a create_app() that creates everything which allows me to create an arbitrary number of independent apps in a test run. (To be honest there are some global resources like registered event listeners, but tearing those down is easy with py.test factories.) For testability I'd try to avoid global resources that are created on module import. Although I've seen differently in big and successful projects.
That's not really a definite answer, I know...
I'm making some Django projects using Redis as Backend Cache[1], but I can't be sure that the Redis Server will be On all the time, then I'm trying to use Redis "if" it's available otherwise use some other Backend like LocMem and so on.
The Redis Backend that I'm using[1] is full compatible so I can use Django Decorations.
I was think to create a function to be called like that:
from django.views.decorators.cache import cache_page
from utils import PingBackend
from time import time
#cache_page(60, cache=PingBackend(time()))
def index(request):
artigos = Artigo.objects.filter(ativo=1)
return render_to_response('index.html', {'artigos':artigos}, RequestContext(request))
The problem is that Django (Internals I guess) Caches the response of PingBackend() and call it just the first time, even if I drop the RedisServer Django tells that the ping process was successfully.
It occurs even with DEBUG=True and 'default' CacheBackend to dummy.
def PingBackend(time):
print time
response = None
try:
con = StrictRedis(host=settings.REDIS_HOST, port=settings.REDIS_PORT, db=0)
# Execute some action
con.ping()
# If not give an exception, use redis
response = 'redis'
except:
response = 'default' #dummy
return last_response
I'm passing time() just to create some differentiation as a try solve the cache problm.
The big picture is that the function PingBackend() aren't executing for each request, just for the first the I can't monitor the Redis Server.
Thank you!
[1] - https://github.com/niwibe/django-redis
This is not about Django internals, this is about how decorators work. When you define your view like this:
#cache_page(60, cache=PingBackend(time()))
def index(request):
blah blah
it is exactly equivalent to this:
def index(request):
blah blah
index = cache_page(60, cache=PingBackend(time()))(index)
You are invoking cache_page only one, passing it a cache argument that you got by invoking PingBackend once. It isn't even executing just for the first request, it is executing once when the view function is defined.
You should write your own cache backend that uses Redis if it's available, or something else if it is not.