I have one DB query that takes a couple of seconds in production. I have also a DRF ViewSet action that returns this query.
I'm already caching this action using cache_page.
#method_decorator(cache_page(settings.DEFAULT_CACHE_TIMEOUT))
#action(detail=False)
def home(self, request) -> Response:
articles = Article.objects.home()
return Response(serializers.ArticleListSerializer(articles, many=True).data,
headers={'Access-Control-Allow-Origin': '*'})
The problem is that after 15 minutes, at least one user needs to wait 15 seconds for the response. I want to pre-cache this every 5 minutes in background so that no user will need to wait.
I use the default caching mechanism.
My idea is to create a management command that will be executed using crontab. Every 5 minutes it will call the Article.objects.home() or the ViewSet.action and change it's value in the cache.
As this is only one entry, I don't hesitate to use database caching.
How would you do that?
EDIT: as the default LocMemCache is single-threaded, I'll go with the database caching. I just don't know how to manually cache the view or QuerySet.
A cron or Celery beat task (if you already use celery) looks like the best option.
Calling Article.objects.home() would not do much unless you cache in home() method of the manager (which could be a valid option that could simplify automated cache refresh).
To automate the refresh of view cache you better send actual requests to the URL from the management command. You will also want to invalidate the cache before sending the request, in order to update it.
Also, keep in mind the cache timeout when planning the job frequency. You wouldn't want to refresh too early nor too late.
Related
Two questions regarding session timeouts in cherrypy:
1) Is there a way to determine the remaining time in a session? This is related to trying to use http://plugins.jquery.com/epilgrim.sessionTimeoutHandler/
2) Is there a way to make a call to cherrypy NOT reset the timeout, such that the plugin above could call a URL to determine the time remaining in the session without resetting said time
Edit to help clarify: The purpose here is to be able to have a client-side process that periodically queries the server via AJAX to determine the amount of time left in a users session. This is to overcome difficulties with keeping a client side session timeout timer in-sync with the server-side timer - I'd like to simply have the client ask the server "how much time do I have left?" and act accordingly. Of course, if the act of asking resets the timeout, then this won't work, as the AJAX "time left" requests would essentially become a session keep-alive. So I need to be able to make an AJAX query to the server without resetting the session timeout timer for the user.
I believe cherrypy uses the expiration time in the cookie with the key session_id. Mine says:
Wed 22 Jan 2014 03:44:31 PM EST
You could extend the expiration with your set of circumstances and edit the session cookie.
EDIT: You will also need to extend the server timeout...
cherrypy.request.config.update({'tools.sessions.timeout': 60})
https://groups.google.com/forum/#!topic/cherrypy-users/2yrG79QoYFQ
Hope this helps!
You need to subclass the session and add a "stats" function to it and a flag to prevent saving in the session "stats" request handler. Or disable sessions in the config for the "stats" path and load session exp info directly from your storage without using normal session class.
I have found the answer to 2) question while going through source code of cherrypy session class. Apparently, you do not want to save session after serving such requests - this will then also not update expiration time (and will not save any changes to session object).
I found in the source code that setting cherrypy.serving.request._sessionsaved = True does exactly that. And added decorator for convinience:
def nosessionsave( func ):
"""
Decorator to avoid session saving and thus not resetting session timeout.
"""
def decorate( *args, **data ):
cherrypy.serving.request._sessionsaved = True
return func( *args, **data )
return decorate
Just add #nosessionsave before method def.
How do I create a new clean session and invalidate the current one in Flask?
Do I use make_null_session() or open_session()?
I do this by calling session.clear().
EDIT:
After reading your comment in another answer, I see that you're trying to prevent a replay attack that might be made using a cookie that was issued in the past. I solved that problem as much as possible* with this approach:
Override SecureCookieSessionInterface.save_session(), copying the code from the overridden version rather than calling it.
When the overridden version of save_session() calls save_cookie(), make it pass a session_expires argument 30 minutes in the future. This causes cookies more than 30 minutes old to be considered invalid.
Make the overridden version of save_session() update a session variable every so often, to make sure the cookie and its session_expires time get rewritten regularly. (I name this session variable '_refresh' and store the current time in it, then rewrite it only if more than a few seconds have passed since the last-stored time. This optimization avoids rewriting the cookie on every HTTP request.)
Duplicating Flask code in the custom save_session() makes this approach a bit ugly and brittle, but it is necessary in order to change the arguments passed to save_cookie(). It would be nice if Flask made this easier, or at least implemented its own safeguard against replay attacks.
*WARNING: This approach by itself will not stop replay attacks that might happen during a session cookie's valid lifetime. This fundamental problem with cookie-based sessions is discussed in RFC 6896 and A Secure Cookie Protocol by Liu, Kovacs, Huang, Gouda.
If you have security concerns (and everyone should have) There is the answer:
This is not REALLY possible
Flask uses cookie-based sessions. When you edit or delete session, you send a REQUEST to CLIENT to remove the cookie, normal clients (browsers) will do. But if session hijacked by an attacker, the attacker's session remains valid.
You can add an after_request callback to remove the session cookie if a particular flag is set:
#app.after_request
def remove_if_invalid(response):
if "__invalidate__" in session:
response.delete_cookie(app.session_cookie_name)
return response
Then you simply set that session key whenever you want to invalidate the session:
#app.route("/logout")
def logout():
session["__invalidate__"] = True
return redirect(url_for("index"))
See also: http://werkzeug.pocoo.org/docs/wrappers/#werkzeug.wrappers.BaseResponse.delete_cookie
If you use default flask sessions and set the app.permanent_session_lifetime, then the session will not work if a user tries to replay the same session as long as the session has expired.If you look at the source code for open_session, there is line:
max_age = total_seconds(app.permanent_session_lifetime)
try:
data = s.loads(val, max_age=max_age)
return self.session_class(data)
except BadSignature:
return self.session_class()
I have the following code:
#task()
def handle_upload(title, temp_file, user_id):
.
.
.
photo.save()
#if i insert here "photo2 = Photo.objects.get(pk=photo.pk)" it works, including the view function
return photo.pk
#view function
def upload_status(request):
task_id = request.POST['task_id']
async_result = AsyncResult(task_id)
photo_id = async_result.get()
if async_result.successful():
photo = Photo.objects.get(pk=photo_id)
I use an ajax request to check for the uploaded file but after the celery task finishes i get a Photo matching query does not exist. The photo pk does exist and gets returned. If i query the database manually it works. Is this some sort of database lag? How can I fix it?
I'm using Django 1.4 and Celery 3.0
You can confirm if it is a lag issue by adding a delay to your django view to wait after the task has successfully finished for a a few seconds. If that resolves the problem you might want to wrap the handle_upload in a transaction to block until the db has completely confirmed it has finished before returning.
Beside Django, DB too has its own caches. When django invokes the queryset, it gets stale data either from its own caches (unlikely unless you were reusing querysets, which I didn't see in the portion of the code you posted) or the DB is caching results for the same Django connection.
For example if you were to invoke post processing after the celery task has finished in a completely new django request/view you would probably see the new changes in DB just fine. However, since your view was blocked while the task was executing (which defeats the purpose of celery btw) internally django only keeps the snapshot of the DB at the time the view was entered. Therefore your get fails and you confirmed this behavior directly when simply entering the django shell.
You can fix this like you already did by either:
invoking transactional management which will refresh the snapshot
changing on your DB endpoint caching and autocommit policies
have celery make a callback to django (web request) once it is done to finalize processing (which is likely what you want to do anyway because blocking django defeats the purpose)
Every time a user logs in to the application, I want to perform a certain task, say, record the time of login. So I wanted to know if a hook is fired on login by default? If yes, how can I make my module respond to it.
Edit - Assume there are multiple entry points in the application to login.
While there may well be multiple points of entry, it's crucial that your auth/session code conform to the DRY principle.
Once/if you're down to a single code path for logging in, you should be able to find an appropriate place in that code path to do something as simple as this:
user.last_login = time
user.num_logins++
user.save()
Additionally, you could use a memcache cooldown to make sure this only happens once every, say, 30 minutes:
cooldown_memcache_key = "login_cooldown_%s" % user.id
cooldown = memcache.get(cooldown_memcache_key)
if cooldown is None:
user.last_login = time
user.num_logins++
user.save()
memcache.add(cooldown_key, True, 1800)
I'm using Python on GAE (so it may be different for Java) but have seen no documentation about such a hook for a user logging in. If you used one of the session management frameworks you'd probably get some indication for that, but otherwise I do this kind of house keeping on my opening page itself which requires login. (What do you want to do about an already logged in user returning to your site a few days later... that is, do you really want to record logins or the start time of a visit/session??)
If I wanted to do this but with multiple landing pages, and without using a session framework,
I'd use memcache to do a quick check on every page request and then only write to the datastore when a new visit starts.
I have a running django/apache2 + memcached app (ubuntu) and would like to keep track of logged in users that are online.
What would be the best way to track this?
I would prefer not writing to the database each time a logged in user loads a page; but what other options are there?
An approach might be:
you create a middleware that does the following on process_response:
check for a cookie called 'online', but only if the user is authenticated
if the cookie is not there,
set a cookie called 'online' with value '1'
set the lifespan of the cookie to 10 minutes
update the 'last_login' field of auth.User for this user with the current datetime
now you have all currently logged in users in your auth.User table. All Users that have a last_login newer than datetime.now()-interval(15minutes) might be considered "online".
The database will be written for every logged in user about every 10 minutes. Adjust the values "10" and "15" to your needs.
The advantage here is that database writes are rare (according to your two numeric settings 10/15). And for speed optimization make sure that last_login is indexed, so a filter on this field including Count is really fast.
Hope this helps.
A hashmap or a queue in memory with a task running every hour or so to persist it.
You need to persist the info server-side, integrity isn't critical, throughput and latency are important. That means you should use some sort of key-value store.
Memcached and redis have keys that expire. You probably have memcached already installed, so use that.
You can reset expiry time of the user:last-seen:$username key every visit, or you can use mawimawi's cookie technique and have expiry = 4 * cookie-lifetime.
You can't do that in django without using a database/persistent-storage because of the same reason why django sessions are stored in database: There can be multiple instances of your applications running and the must synchronize their states+data through a single persistence source [1]
Alternatively, you might want to write this information in a folder in a file named with user id and then check its create/modified date to find the required information.