I want to operate on cache values using Redis cache system. Redis has many good operations itself, but cache module in Django doesn't support them. I know for using Redis methods I can import Redis and create a client and use these methods, but using this method we must create client each time and try to connect to the Redis server. Is this an efficient way to use Redis in Django in large scale requests?
Other people have faced the same issue and have created Redis cache backends for Django so you can just import one of those, eg
https://django-redis.readthedocs.org/en/latest/
Related
I am debugging a slow API endpoint in a web app which uses mongodb as storage. It turns out the request send 8 different queries to MongoDB, and group the data together to return. The MongoDB lives on another host, so the request involves 8 roundtrips.
These 8 requests don't have any dependency among themselves, so if I can send the 8 queries in a batch, or in parallel, a lot of time can be saved.
I am wondering if Mongo supports something like Redis's pipeline, or maybe send a script (like a lua script in Redis) for fetching data, so that I can get all data in one go?
If not, is there a way to send the querys in parallel? (The app is based on python/tornado/pymongo)
There are a number of options for this kind of thing in mongodb all of which can be accessed by the python driver pymonogo.
The best way (IMHO) is the aggregation framework which allows you to build a pipeline. However, some of the functionality is limited by mongodb version and whether you have sharded clusters.
Other options include map-reduce or simple operators.
AFAIK, there is no pipeline-like mechanism in MongoDB.
I would try using server-side scripts.
I'm running Django (1.5) on App Engine and I need to use some kind of key-value cache. I know App Engine's memcache API and also the Django's cache framework. I wonder which one should I use.
On one hand I would like my code to be as portable as possible for migrating it to another cloud platform. But on the other hand I would like to fully utilize the services offered by App Engine.
Is writing a custom cache backend for Django that will use the App Engine memcache is the best solution?
Tzach, I think you're already answering your question.
Putting your app in GAE and not using the services provided by Google it doesn't look to me as a wise decision, even more, when those features are key for performance at the same time free or very cheap.
On the other hand, the embedded default cache in Python is not guaranteed to give its best results under GAE, as GAE instances are not a normal server where you'd run your django instance, e.g. instances can be shutdown at any time.
These special characteristics found in Django are tuned in the django for GAE versions.
For that reason, and taking into account that using the GAE memcache is also straightforward, I'd recommend you using the easiest ones to add to your application.
And, if in the future, you move to another platform, there will be more things to change than the key-value cache.
My two cents on that is to focus firstly in getting the job done and secondly in optimizing the performance on GAE and only afterwards to start thinking on things to improve.
I am trying to develop a multithreaded web server, it has the following task:
Collect data from various data sources (API calls), I was planning to do this using multiple threads.
Store the collected data in a memory data structure
Do some processing on the data structure using another thread
This data structure would be queried by the multiple clients; maybe I could also make separate threads for each client request.
Now regarding language and platform, I was considering either python or JAVA. I did some research on Flask framework for python, but I do not know how it will accommodate the multithreaded nature of web server.
Please suggest how I could achieve the above functionality in my project.
Flask, with some of the available addons, is very suited for what you want to do. Keep in mind that flask is pure python, and therefore you can access any of the excellent available python libraries.
As far as I understand what you have in mind, you can:
1- define a url that, when visited, executes the data gathering from external sources by means of, e.g. python-requests (http://docs.python-requests.org/en/latest/)
2- do the same periodically by scheduling the function above
3- store the collected data in a (e.g.) Redis database (which is memory based) or one of the many available databases (all of the nosql dbs have python bindings that you can access from a flask application)
4- define urls for the visiting clients to access the latest versions of the data. You will just need to define the data extraction functions (from redis or whatever you decide to use) and design a nice template to show them.
Flask/Werkzeug will take care of the multithreading necessary to handle simultaneous requests from different clients.
I've heard of redis-cache but how exactly does it work? Is it used as a layer between django and my rdbms, by caching the rdbms queries somehow?
Or is it supposed to be used directly as the database? Which I doubt, since that github page doesn't cover any login details, no setup.. just tells you to set some config property.
This Python module for Redis has a clear usage example in the readme: http://github.com/andymccurdy/redis-py
Redis is designed to be a RAM cache. It supports basic GET and SET of keys plus the storing of collections such as dictionaries. You can cache RDBMS queries by storing their output in Redis. The goal would be to speed up your Django site. Don't start using Redis or any other cache until you need the speed - don't prematurely optimize.
Just because Redis stores things in-memory does not mean that it is meant to be a cache. I have seen people using it as a persistent store for data.
That it can be used as a cache is a hint that it is useful as a high-performance storage. If your Redis system goes down though you might loose data that was not been written back onto the disk again. There are some ways to mitigate such dangers, e.g. a hot-standby replica.
If your data is 'mission-critical', like if you run a bank or a shop, Redis might not be the best pick for you. But if you write a high-traffic game with persistent live data or some social-interaction stuff and manage the probability of data-loss to be quite acceptable, then Redis might be worth a look.
Anyway, the point remains, yes, Redis can be used as a database.
Redis is basically an 'in memory' KV store with loads of bells and whistles. It is extremely flexible. You can use it as a temporary store, like a cache, or a permanent store, like a database (with caveats as mentioned in other answers).
When combined with Django the best/most common use case for Redis is probably to cache 'responses' and sessions.
There's a backend here https://github.com/sebleier/django-redis-cache/ and excellent documentation in the Django docs here: https://docs.djangoproject.com/en/1.3/topics/cache/ .
I've recently started using https://github.com/erussell/django-redis-status to monitor my cache - works a charm. (Configure maxmemory on redis or the results aren't so very useful).
You can also use Redis as a queue for distributed tasks in your Django app. You can use it as a message broker for Celery or Python RQ.
Redis as a Primary database
Yes you can use Redis key-value store as a primary database.
Redis not only store key-value pairs it also support different data structures like
List
Set
Sorted set
Hashes
Bitmaps
Hyperloglogs
Redis Data types Official doc
Redis is in memory key-value store so you must aware of it if Redis server failure occurred your data will be lost.
Redis can also persist data check official doc.
Redis Persistence Official doc
Redis as a Cache
Yes Redis reside between in Django and RDBMS.
How it works
given a URL, try finding that page in the cache if the page is in the cache: return the cached page else: generate the page save the generated page in the cache (for next time) return the generated page
Django’s cache framework Official Doc
How can use Redis with Django
We can use redis python client redis-py for Django application.
Redis python client redis-py Github
We can use Django-redis for django cache backend.
Django-redis build on redis-py and added extra features related to django application.
Django-redis doc Github
Other libraries also exists.
Redis use cases and data types
Some use cases
Session cache
Real time analytics
Web caching
Leaderboards
Top Redis Use Cases by Core Data structure types
Big Tech companies using Redis
Twitter GitHub Weibo Pinterest Snapchat Craigslist Digg StackOverflow Flickr
I'm writing a simple app with AppEngine, using Python. After a successful insert by a user and redirect, I'd like to display a flash confirmation message on the next page.
What's the best way to keep state between one request and the next? Or is this not possible because AppEngine is distributed? I guess, the underlying question is whether AppEngine provides a persistent session object.
Thanks
Hannes
No session support is included in App Engine itself, but you can add your own session support.
GAE Utilities is one library made specifically for this; a more heavyweight alternative is to use django sessions through App Engine Patch.
The ways to reliable keep state between requests are memcache, the datastore or through the user (cookies or post/get).
You can use the runtime cache too, but this is very unreliable as you don't know if a request will end up in the same runtime or the runtime can drop it's entire cache if it feels like it.
I really wouldn't use the runtime cache except for very specific situations, for example I use it to cache the serialization of objects to json as that is pretty slow and if the caching is gone I can regenerate the result easily.