Develop Reference

Django: how can I automatically generate and cache my urls.py?

My project's URLs are automatically generated in urls.py using a for loop (the URLs look like AppName/ViewName). According to the docs, urls.py is loaded upon every request. This appears to be slowing my site down since it requires a bunch of introspection code, so I want to generate the URLs less frequently. I could of course manually run a script to re-generate urls.py (or a file imported by urls.py) as needed, but would prefer if it happened automatically as part of project validation/startup (like the server starting up or the database being synced). I'm open-sourcing this project, and many people will be running it on their own servers, so I want to do this in a robust way. Any recommendations?

The docs do not say what you claim they do (or rather, you're reading too much into a phrase which only means "loads that Python module (if it has not already been loaded)".
Generally, the only things that happen on every request are running the middleware and the specific view code associated with that request. Even then, nothing is reloaded on every request. URLs, like all Python code, is only loaded when a new process is started, and when that happens depends on your server setup. Your problem is elsewhere: you should profile your application carefully to find out exactly where.

For example you can look for django-json-rpc where author has realized url-generating via decorators. There are main controller which receive all requests and urls dict {'pattern': method}. urls dict filled automatically by decorators like #jsonrpc_method which receive a function and put them to urls.
I think it must run faster than the for and I believe that this approach will be able to apply for django.urls

Flask templates including incorrect files

I'm having a rather strange problem using a jinja2.ChoiceLoader (also tried with multiple paths with FileSystemLoader, no joy) in Flask.
I have several "theme" directories, like so.
/templates/
themes/
default/
layout.html
menu.html
blue/
layout.html
grey/
menu.html
...
And I'd like to fallback to default/ if the selected theme doesn't have the required template, so I used a ChoiceLoader, like so.
#app.before_request
def setup_request():
current_theme = get_theme()
logging.info('Using theme %s'%(current_theme))
app.jinja_loader = jinja2.ChoiceLoader([
jinja2.FileSystemLoader('/templates/themes/%s/'%(current_theme)),
jinja2.FileSystemLoader('/templates/themes/default/')
])
That's great, but if I change the <current_theme> it still loads the theme from the old folder until I reload Apache or restart the Flask development server.
It should be using the new theme. Logging says that it's using the changed theme, but apparently app.jinja_loader is a bit like honey badger... it's completely ignoring it until I reload Apache.
Edit: This appears to be related to Flask considering all files of the same name to be the same file. I can reproduce with the builtin server (with DEBUG=True), Cherry, and mod_wsgi. This person seems to have a similar problem, but no simple solution: flask blueprint template folder My situation is different in the sense that I require cascading templates for a single app. His problem is related to cascading templates between blueprints, but it may be the same issue under the hood.
Here's the code that's in the "get_theme()" call:
def get_theme():
# I know this is unsafe, testing only
return request.args.get('theme','default')
Edit 2: I need to change the HTML and JavaScript between themes, not just the CSS. This is why I'm not just loading different CSS files. Also, some of these themes are for mobile devices, and have very little in common with the other themes.
Edit 3: Two solutions. Solution 1: Name the files uniquely, like "blue.layout.html" and "default.layout.html". This works perfectly, but it doesn't cascade as required. Solution 2: Use relative paths, so instead of include 'file.html', use include 'theme/blue/file.html. I achieved cascading by creating a get_theme_file() function that checks for the active theme, checks if the file exists (if not, fallback to "default" theme), and returns the relative path. I just have to make sure everything I include looks like {% include get_theme_file('file.html') %}. This is not elegant, but I find it to be more elegant that the low-level fiddling with Flask used here.

By the way, you can pass multiple locations to FileSystemLoader, and it is the recommended way to load templates
This is expected behavior in Apache with mod_wsgi (which I assume you are using). File system changes don't trigger a reload of all the processes. See this entry in the Flask docs that talks about this, and offers a workaround which is to add:
WSGIScriptReloading On
To the configuration section for your application and then touching the wsgi file to trigger a reload of the child processes.
Are you sure this is what you intend? Most theme switching tricks rely on the cascading part of cascading style sheets (CSS) to control themes.

Well, I'm not the only one to encounter this problem. The issue is that Flask caches based on filename, and if you don't include the relative path it just caches the first one to be loaded. There are three ways to accomplish dynamic, cascading, templates.
Override Jinja builtins. Which I found to be quite confusing. I'm not smart enough for this solution.
Serve a different WSGI process for each file. The setup here seems a bit too much for a dynamic site. Flask caches the filenames per WSGI process, so you can do something with multiple Cherry WSGI servers, for example.
Include the theme in the load path. Create a function, load it in with context_processor, and only load template files using that function. The function needs to check for a non-default theme, check if the file exists, and return it. So a template call would be get_theme_file('layout.html') which would return the relative path (like themes/blue/layout.html).
An example of Option 3.
def get_theme_file(fname):
theme = get_theme()
if os.path.exists(os.path.join(theme.theme_dir, fname)):
return os.path.join('themes', theme.name, fname)
return os.path.join('themes', 'default', fname)
...
# Each render_template should reference this
return render_template(get_theme_file('layout.html'))
If you include theme files in templates:
{% include get_theme_file('layout.html') %}
Unfortunately, this doesn't cache, but I can see a few ways to optimize it. Maybe cache an os.listdir of the get_theme().theme_dir, and instead of reading the filesystem for each get_theme_file call, just do an in check against the cached listdir list.
It's worth noting that this is Flask specific. I was unable to reproduce this behavior with plain Jinja2 and my own WSGI server. One might say that Flask was a poor choice for this particular project, but I'd argue that the savings from everything else Flask does was well worth it.

Using external URLs in Django's TEMPLATE_DIRS

Django's TEMPLATE_DIRS in Settings.py calls for unix style slashes.
Because of this, when I call
get_template('some/template.html')
in a view, the result always starts at the root, and results in a call to
/home/username/projectname/public/some/template.html
The problem is that I'd like to use templates hosted on an entirely different site. This works fine for other Settings.py fields (MEDIA_URL and STATIC_URL), where it will take an absolute http path with no objection.
Given an http path,
TEMPLATE_DIRS ('http://example.com/',)
in Settings.py will force
get_template('some/template.html')
in a view to try and find
/home/username/projectname/public/http://example.com/some/template.html
I've tried to circumvent this like so
TEMPLATE_DIRS ('../../../../http://example.com/',)
But it still forces a leading slash, so I get "/http://example.com", which is useless.
My questions:
Is there a way to trick this into pulling the template files from
another server?
Is that even feasible, given that the template files need to be
processed for the view?
Is it possible to create an alternate to 'django.template.loaders.filesystem.Loader' that doesn't call for unix style slashes?

You don't need to use the template directory is you dont want to. If you have a server that is serving template files, you can simply fetch them remotely using urllib2 and create and render the template with a context manually:
import urllib2
from django.template import Context, Template
tpl_html = urllib2.urlopen("http://mysite.com")
tpl = Template(tpl_html)
return tpl.render(Context({
'some_variable' : 'some_val',
})
If you are going to do this, you have to incorporate some caching, as for every request to using this template, you need to make an external request. Alternatively you could write this into a custom loader but it will suffer the same limitations.

You can't do this.
It has nothing to do with path names. It's just that the filesystem template loader needs to load things from the filesystem, hence the name.
This is completely different from the case of MEDIA_URL: that simply adds a path into your HTML, which your browser then loads. Django doesn't care where that file lives: although in fact the opposite applies, in that if you pass it a filepath that isn't a URL (ie served by a webserver somewhere), it simply won't work.
Now, you could write a template loader that gets its templates from another server. Template loaders are pluggable - you just need to put the name of your new loader in the TEMPLATE_LOADERS setting. The loader itself would need to use something like urllib.urlopen to get the template from the external server.
But think very carefully before you do this. This means that every single template request now requires a call to an external server before you can serve the page. In the typical case of a template that extends other templates and includes calls to included template tags, that might be five or ten calls. And, unlike media files, it can't be done in parallel: the page simply won't be served until the whole process is finished. This is likely to make your webserver very very slow.
I don't know why you think you need to do this. Templates are part of your application code, so they would normally live on the same server as your Python code. If you really have some reason to keep them externally, one solution might be to mount the external filesystem onto your webserver via something like sshfs. It's still likely to be very slow though. Think again.

No - it's not possible to trick it into pulling files from another server via http
Yes - you certainly could subclass django.template.loaders.filesystem.Loader (and by altering the load_template_source method appropriately) so that you could load templates via http
Once you had done 3 then the answer to 2 would be yes - it would be feasible - ultimately Django's template language doesn't care where it gets the file from, as long it's in the right format.
However, it seems like a very inefficient way of loading templates and more likely there is a much better way of achieving the same result.

Is there a way to have something execute for every view in django?

I'm new to Django and coming from Rails, so that may explain my odd questions below:
I have a main layout that has a sidebar that lists the latest updates to the site. That main layout is used for every page in my webapp so every template that is created extends main.html.
For the latest updates section, I just want to get the last 5 updates from posts to the web app every time a page is rendered. I thought about making the sidebar do this through an ajax call once the page is loaded, but I thought this may not be my best option.
I also considered creating a tag to do this for me and then just calling the tag inside of main.html. This is simple enough, but I'd have to write a lot of HTML inside of the tag code, which seems to be a little annoying (a lot of string appending and such, unless I'm wrong and there is a better way?)
I have read about Context Processors. This seemed to be exactly what I wanted, but it appears this may cause another issue where I have to pass a context_instance to every single render_to_response? This appears to be a lot of code repeat and I'm trying to avoid that if possible. Is there a way to just make render_to_response always take the RequestContext object without always having to specify it?
Are there any other ways to achieve having some code run for every view and eliminate the need to always pass data to a view?

Django 1.3 added the render shortcut which is the same as render_to_response but with RequestContext automatically used.

Templates is a appropriate place for this: the variant with custom tag and template inheritance is simple and convenient. To avoid string appending use mini-template just for your tag: it is called inclusion tags.

I would definitely go for the Ajax call, it is as simple as to create an small view which queries the model for the 5 latest posts, serializes them into json or xml data, and returns them in your HttpRequest object.

You can use direct_to_template instead of render_to_response.

How to externally populate a Django model?

What is the best idea to fill up data into a Django model from an external source?
E.g. I have a model Run, and runs data in an XML file, which changes weekly.
Should I create a view and call that view URL from a curl cronjob (with the advantage that that data can be read anytime, not only when the cronjob runs), or create a python script and install that script as a cron (with DJANGO _SETTINGS _MODULE variable setup before executing the script)?

There is excellent way to do some maintenance-like jobs in project environment- write a custom manage.py command. It takes all environment configuration and other stuff allows you to concentrate on concrete task.
And of course call it directly by cron.

You don't need to create a view, you should just trigger a python script with the appropriate Django environment settings configured. Then call your models directly the way you would if you were using a view, process your data, add it to your model, then .save() the model to the database.

I've used cron to update my DB using both a script and a view. From cron's point of view it doesn't really matter which one you choose. As you've noted, though, it's hard to beat the simplicity of firing up a browser and hitting a URL if you ever want to update at a non-scheduled interval.
If you go the view route, it might be worth considering a view that accepts the XML file itself via an HTTP POST. If that makes sense for your data (you don't give much information about that XML file), it would still work from cron, but could also accept an upload from a browser -- potentially letting the person who produces the XML file update the DB by themselves. That's a big win if you're not the one making the XML file, which is usually the case in my experience.

"create a python script and install that script as a cron (with DJANGO _SETTINGS _MODULE variable setup before executing the script)?"
First, be sure to declare your Forms in a separate module (e.g. forms.py)
Then, you can write batch loaders that look like this. (We have a LOT of these.)
from myapp.forms import MyObjectLoadForm
from myapp.models import MyObject
import xml.etree.ElementTree as ET
def xmlToDict( element ):
return dict(
field1= element.findtext('tag1'),
field2= element.findtext('tag2'),
)
def loadRow( aDict ):
f= MyObjectLoadForm( aDict )
if f.is_valid():
f.save()
def parseAndLoad( someFile ):
doc= ET.parse( someFile ).getroot()
for tag in doc.getiterator( "someTag" )
loadRow( xmlToDict(tag) )
Note that there is very little unique processing here -- it just uses the same Form and Model as your view functions.
We put these batch scripts in with our Django application, since it depends on the application's models.py and forms.py.
The only "interesting" part is transforming your XML row into a dictionary so that it works seamlessly with Django's forms. Other than that, this command-line program uses all the same Django components as your view.
You'll probably want to add options parsing and logging to make a complete command-line app out of this. You'll also notice that much of the logic is generic -- only the xmlToDict function is truly unique. We call these "Builders" and have a class hierarchy so that our Builders are all polymorphic mappings from our source documents to Python dictionaries.