I have a template that I would like to cache for django. The url is something like this:
/cache_my_page/<object_number>/
There are about a million objects here -- so 1M pages I would like to cache. Is there a way to pre-cache all these pages, before a user loads it? Or does django not offer that, and I need to dive into something like redis or memcache to do that?
You can compile your templates outside a request-response, but that's probably not what you want. Django is for dynamic content. Though if your objects hardly ever changes you could certainly loop over your objects, dump the resulting content of the request into a static file and let it be served by nginx.
https://docs.djangoproject.com/en/1.7/ref/templates/api/#compiling-a-string
Though there are more elegant solutions for dynamic content as well.
How you can avoid rerendering a url which was requested before, is described here in detail:
http://www.djangobook.com/en/2.0/chapter15.html
Or have a look at a solution like Varnish taking Django out of the equation:
Varnish is a piece of software that sits between our load balancers
and our Django backends and acts as an HTTP caching layer. What this
means is that it can cache the entire HTTP response without even
hitting a Django server, if we know that request won’t be unique.
http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views
Related
I am using django on the server side and obviously javascript on the client side. Now i want to use the plate template engine on the client.
What's the best way so serve django templates to the client? We taught of some ways doing that.
Create a view that serves the raw templates.
probably not the best method
Copy the needed templates to the static folder.
this could be done with a custom static files finder
the broser is able to cache the templates
Provide the templates using a template tag which puts the raw template into a javascript variables.
templates received this way can not be cached seperatly
is a django app out there that makes this easyer?
The reason i need the templates on the client is, that i want to use the same clients on the server and the client side. When the page is first loaded, the full template is rendered on the server, when navigating trough the application only the needed data gets loaded and the page change is done using push state.
If you need to be able to have A) dynamically generated plate templates, or B) dynamically created plate templates (e.g., entered into the DB via the admin, etc.), You'll want to go with 1 (not a bad thing - django is made for serving text content, so as long as you need to have it in a dynamic manner, there's no problem doing it). 3 is a bad choice, because it means that a browser can't cache the static resource (if it's output into each page)... unless you need different plate templates for each page of course.
If you don't need A or B from above, I'd just stick the templates in your static dir, as you mentioned (e.g., collectstatic or simply add them to your repo, if they're a part of your app).
Regarding an app that makes this easy - you could look at Django Chunks (output a static chunk into a place in the page, like `{% chunk "header-snippet" %}), but I don't think you need that.
I'm planning an iOS app that requires a server backend capable of efficiently serving image files and performing some dynamic operations based on the requests it gets (like reading and writing into a data store, such as Redis). I'm most comfortable with, and would thus prefer to write the backend in Python.
I've looked at a lot of Python web framework/server options, Flask, Bottle, static and Tornado among them. The common thread seems to be that either they support serving static files as a development-time convenience only, discouraging it in production, or are efficient static file servers but not really geared towards the dynamic framework-like side of things. This is not to say they couldn't function as the backend, but at a quick glance they all seem a bit awkward at it.
In short, I need a web framework that specializes in serving JPEGs instead of generating HTML. I'm pretty certain no such thing exists, but right now I'm hoping that someone could suggest a solution that works without bending the used Python applications in ways they are not meant for.
Specifications and practical requirements
The images I'd be serving to the clients live in the file system in a shallow directory hierarchy. The actual file names would be invisible to the clients. The server would essentially read the directory hierarchy at startup, assigning a numeric ID for each file, and would route the requests to controller methods that then actually serve the image files. Here are a few examples of ways the client would want to access the images in different circumstances:
Randomly (example URL path: /image/random)
Randomly, each file only once (/image/random_unique), produces some suitable non-200 HTTP status code when the files are exhausted
Sequentially in either direction (/image/0, /image/1, /image/2 etc.)
and so on. In addition, there would be URL endpoints for things like ratings, image info and other metadata, some client-specific information as well (the client would "register" with the server, so that needs some logic, too). This data would live in a Redis datastore, most likely.
All in all, the backend needs to be good at serving image/jpeg and application/json (which it would also generate). The scalability and concurrency requirements are modest, at least to start with (this is not an App Store app, going for ad-hoc or enterprise distribution).
I don't want the app to rely on redirects. That is, I don't want a model where a request to a URL would return a redirect to another URL that is backed by, say, nginx as a separate static file server, leaving only the image selection logic for the Python backend. Instead, a request to a URL from the client should always return image/jpeg, with metadata in custom HTTP headers where necessary. I specify this because it is a way of avoiding serving static files from Python that I thought of, and someone else might think of too ;-)
Given this information, what sort of solution would you consider a good choice, and why? Or is this something for which I need to code non-trivial extensions to existing projects?
EDIT: I've been thinking about this a bit more. I don't want redirects due to the delay inherent in the multiple requests they entail, plus I'd like to abstract out the file names from the client, but I was wondering if something like this would be possible:
It's pretty self-explanatory, but the idea is that the Python program is given the request info by nginx (or whatever serves the role), mulls it over and then tells nginx to respond to the client's request with a specific file from the file system. It does so. The client is none the wiser about how the request was fulfilled, it just receives a response with the correct content type.
This would be pretty optimal in my view, but is it possible? If not with nginx, perhaps something else?
I've been using Django for well over a year now, and it is the hammer I use for all my nails. You could probably do this with a bit of database-image storage and django's builtin orm and url routing (with regex). If you store the images in the database, you will automatically get the unique-id's set. According to this stackoverflow answer, you can use redis with django.
I don't want a model where a request to a URL would return a redirect to another URL that is backed by, say, nginx as a separate static file server, leaving only the image selection logic for the Python backend.
I think Nginx for serving static and python for figuring out the image url is the better solution.
Still if you do not want to do that I would suggest you use any Python web framework (like Django) and write your models and convert them into REST resources (Eg. Using django-tastypie) and/or return a base64 encoded image which you can then decode in your iOS client.
Refs:
Decoding a Base64 image
TastyPie returns the path as default, you might have to do extra work to either store the image blob in the table or write more code to return a base64 encoded image string
You might want to look at one of the async servers like Tornado or Twisted.
I love django-staticgenerator. Unfortunately you can't use it in cases in which you have to display different content depending on the user, plus is not being actively developed.
Is it possible to return the cached response only to non-logged in users?
I saw an app that used nginx to do it, is it possible to do it with apache?
Has anybody tried this solution and can you suggest if there is any actively developed django app that serves this purpose?
Thanks!
Have you tried CACHE_MIDDLEWARE_ANONYMOUS_ONLY = True? Assuming whatever caching you're using utilizes Django's cache framework, that should do the trick.
I've been doing some research regarding file downloads with access control, using Django. My goal is to completely block access to a file, except when accessed by a specific user. I've read that when using Django, X-Sendfile is one of the methods of choice for achieving this (based on other SO questions, etc). My rudimentary understanding of using X-Sendfile with Django is:
User requests URI to get a protected file
Django app decides which file to return based on URL, and checks user permission, etc.
Django app returns an HTTP Response with the 'X-Sendfile' header set to the server's file path
The web server finds the file and returns it to the requester (I assume the webs server also strips out the 'X-Sendfile' header along the way)
Compared with chucking the file directly from Django, X-Sendfile seems likely to be a more efficient method of achieving protected downloads (since I can rely on Nginx to serve files, vs Django), but leaves 2 questions for me:
Is my explanation of X-Sendfile at least abstractly correct?
Is it really secure, assuming I don't provide normal, front-end HTTP access (e.g. http://www.example.com/downloads/secret-file.jpg) to the directory that the file is stored (ie, don't keep it in my public_html directory)? Or, could a tech-savvy user examine headers, etc. and reverse engineer a way to access a file (to then distribute)?
Is it really a big difference in performance. Am I going to bog my application server down by providing 8b chunked downloads of 150Mb files directly from Django, or is this sort-of a non-issue? The reason I ask is because if both versions are near equal, the Django version would be preferable due to my ability to do things in Python, like log the number of completed downloads, tally bandwidth of downloads etc.
Thanks in advance.
Yes, that's just how it works.
The exact implementation depends on the webserver but in the case of nginx, it's recommended to mark the location as internal to prevent external access.
Nginx can asynchronously serve files while with Django you need one thread per request which can get problematic for higher numbers of parallel requests.
Remember to send a X-Accel-Redirect header for nginx instead of X-Sendfile.
See http://wiki.nginx.org/XSendfile for more information.
I'm building a website that doesn't require a database because a REST API "is the database". (Except you don't want to be putting site-specific things in there, since the API is used by mostly mobile clients)
However there's a few things that normally would be put in a database, for example the "jobs" page. You have master list view, and the detail views for each job, and it should be easy to add new job entries. (not necessarily via a CMS, but that would be awesome)
e.g. example.com/careers/ and example.com/careers/77/
I could just hardcode this stuff in templates, but that's no DRY- you have to update the master template and the detail template every time.
What do you guys think? Maybe a YAML file? Or any better ideas?
Thx
Why not still keep it in a database? Your remote REST store is all well and funky, but if you've got local data, there's nothing (unless there's spec saying so) to stop you storing some stuff in a local db. Doesn't have to be anything v glamorous - could be sqlite, or you could have some fun with redis, etc.
You could use the Memcachedb via the Django cache interface.
For example:
Set the cache backend as memcached in your django settings, but install/use memcachedb instead.
Django can't tell the difference between the two because the provide the same interface (at least last time I checked).
Memcachedb is persistent, safe for multithreaded django servers, and won't lose data during server restarts, but it's just a key value store. not a complete database.
Some alternatives built into the Python library are listed in the Data Persistence chapter of the documentation. Still another is JSON.