I'm having a rather strange problem using a jinja2.ChoiceLoader (also tried with multiple paths with FileSystemLoader, no joy) in Flask.
I have several "theme" directories, like so.
/templates/
themes/
default/
layout.html
menu.html
blue/
layout.html
grey/
menu.html
...
And I'd like to fallback to default/ if the selected theme doesn't have the required template, so I used a ChoiceLoader, like so.
#app.before_request
def setup_request():
current_theme = get_theme()
logging.info('Using theme %s'%(current_theme))
app.jinja_loader = jinja2.ChoiceLoader([
jinja2.FileSystemLoader('/templates/themes/%s/'%(current_theme)),
jinja2.FileSystemLoader('/templates/themes/default/')
])
That's great, but if I change the <current_theme> it still loads the theme from the old folder until I reload Apache or restart the Flask development server.
It should be using the new theme. Logging says that it's using the changed theme, but apparently app.jinja_loader is a bit like honey badger... it's completely ignoring it until I reload Apache.
Edit: This appears to be related to Flask considering all files of the same name to be the same file. I can reproduce with the builtin server (with DEBUG=True), Cherry, and mod_wsgi. This person seems to have a similar problem, but no simple solution: flask blueprint template folder My situation is different in the sense that I require cascading templates for a single app. His problem is related to cascading templates between blueprints, but it may be the same issue under the hood.
Here's the code that's in the "get_theme()" call:
def get_theme():
# I know this is unsafe, testing only
return request.args.get('theme','default')
Edit 2: I need to change the HTML and JavaScript between themes, not just the CSS. This is why I'm not just loading different CSS files. Also, some of these themes are for mobile devices, and have very little in common with the other themes.
Edit 3: Two solutions. Solution 1: Name the files uniquely, like "blue.layout.html" and "default.layout.html". This works perfectly, but it doesn't cascade as required. Solution 2: Use relative paths, so instead of include 'file.html', use include 'theme/blue/file.html. I achieved cascading by creating a get_theme_file() function that checks for the active theme, checks if the file exists (if not, fallback to "default" theme), and returns the relative path. I just have to make sure everything I include looks like {% include get_theme_file('file.html') %}. This is not elegant, but I find it to be more elegant that the low-level fiddling with Flask used here.
By the way, you can pass multiple locations to FileSystemLoader, and it is the recommended way to load templates
This is expected behavior in Apache with mod_wsgi (which I assume you are using). File system changes don't trigger a reload of all the processes. See this entry in the Flask docs that talks about this, and offers a workaround which is to add:
WSGIScriptReloading On
To the configuration section for your application and then touching the wsgi file to trigger a reload of the child processes.
Are you sure this is what you intend? Most theme switching tricks rely on the cascading part of cascading style sheets (CSS) to control themes.
Well, I'm not the only one to encounter this problem. The issue is that Flask caches based on filename, and if you don't include the relative path it just caches the first one to be loaded. There are three ways to accomplish dynamic, cascading, templates.
Override Jinja builtins. Which I found to be quite confusing. I'm not smart enough for this solution.
Serve a different WSGI process for each file. The setup here seems a bit too much for a dynamic site. Flask caches the filenames per WSGI process, so you can do something with multiple Cherry WSGI servers, for example.
Include the theme in the load path. Create a function, load it in with context_processor, and only load template files using that function. The function needs to check for a non-default theme, check if the file exists, and return it. So a template call would be get_theme_file('layout.html') which would return the relative path (like themes/blue/layout.html).
An example of Option 3.
def get_theme_file(fname):
theme = get_theme()
if os.path.exists(os.path.join(theme.theme_dir, fname)):
return os.path.join('themes', theme.name, fname)
return os.path.join('themes', 'default', fname)
...
# Each render_template should reference this
return render_template(get_theme_file('layout.html'))
If you include theme files in templates:
{% include get_theme_file('layout.html') %}
Unfortunately, this doesn't cache, but I can see a few ways to optimize it. Maybe cache an os.listdir of the get_theme().theme_dir, and instead of reading the filesystem for each get_theme_file call, just do an in check against the cached listdir list.
It's worth noting that this is Flask specific. I was unable to reproduce this behavior with plain Jinja2 and my own WSGI server. One might say that Flask was a poor choice for this particular project, but I'd argue that the savings from everything else Flask does was well worth it.
Related
I have an idea to put the templates in the database, and offer the possibility for the designer to edit the templates direct from CMS panel. But what is haunted me is the security question. How could it be any secure if we have ability to put python commands directly in the templates. If I have something like this in a mako template:
<%!
import os
os.system('rm /var/www/env/harmless.txt')
%>
it will performed successful and harmless.txt will be removed. Should I find for another template engine except Mako or could I somehow configure Mako to prevent harmful code injection? On the other hand, some python commands incredibly helpful used in templates, inline if statement for example.
If they have their own separated instance of a CMS it doesn't matter like Loic points out. But if they are in some shared environment it is best to use another template engine. The question Untrusted templates in Python - what is a safe library to use? recommends Django templates and Jinja2.
There is still a possibility to do what you want. The mako engine lets you add a preprocessor to the template. I can't say for sure as I can test at the moment, but you could have something like this:
Here: http://docs.makotemplates.org/en/latest/usage.html#mako.template.Template.params.preprocessor
def removeImports(source):
# remove whatever you don't want inside the template
return new_source
tpl = Template(...., preprocessor=removeImports)
If you're using a framework like pyramid you can make it a new renderer to do that for you on all templates.
As a security concern, if you want to got that way as Mako is a terribly good engine in my own opinion. You should do a good set of test case on your preprocessor. I'm pretty sure that imports can be achieved inside inline python <% %>. Actually imports can be achieved almost anywhere inside the template as you can use inline python almost everywhere.
So if you're conerned about running python in a template, you can consider other templates such as Chameleon, Jinja2 ...
Django's TEMPLATE_DIRS in Settings.py calls for unix style slashes.
Because of this, when I call
get_template('some/template.html')
in a view, the result always starts at the root, and results in a call to
/home/username/projectname/public/some/template.html
The problem is that I'd like to use templates hosted on an entirely different site. This works fine for other Settings.py fields (MEDIA_URL and STATIC_URL), where it will take an absolute http path with no objection.
Given an http path,
TEMPLATE_DIRS ('http://example.com/',)
in Settings.py will force
get_template('some/template.html')
in a view to try and find
/home/username/projectname/public/http://example.com/some/template.html
I've tried to circumvent this like so
TEMPLATE_DIRS ('../../../../http://example.com/',)
But it still forces a leading slash, so I get "/http://example.com", which is useless.
My questions:
Is there a way to trick this into pulling the template files from
another server?
Is that even feasible, given that the template files need to be
processed for the view?
Is it possible to create an alternate to 'django.template.loaders.filesystem.Loader' that doesn't call for unix style slashes?
You don't need to use the template directory is you dont want to. If you have a server that is serving template files, you can simply fetch them remotely using urllib2 and create and render the template with a context manually:
import urllib2
from django.template import Context, Template
tpl_html = urllib2.urlopen("http://mysite.com")
tpl = Template(tpl_html)
return tpl.render(Context({
'some_variable' : 'some_val',
})
If you are going to do this, you have to incorporate some caching, as for every request to using this template, you need to make an external request. Alternatively you could write this into a custom loader but it will suffer the same limitations.
You can't do this.
It has nothing to do with path names. It's just that the filesystem template loader needs to load things from the filesystem, hence the name.
This is completely different from the case of MEDIA_URL: that simply adds a path into your HTML, which your browser then loads. Django doesn't care where that file lives: although in fact the opposite applies, in that if you pass it a filepath that isn't a URL (ie served by a webserver somewhere), it simply won't work.
Now, you could write a template loader that gets its templates from another server. Template loaders are pluggable - you just need to put the name of your new loader in the TEMPLATE_LOADERS setting. The loader itself would need to use something like urllib.urlopen to get the template from the external server.
But think very carefully before you do this. This means that every single template request now requires a call to an external server before you can serve the page. In the typical case of a template that extends other templates and includes calls to included template tags, that might be five or ten calls. And, unlike media files, it can't be done in parallel: the page simply won't be served until the whole process is finished. This is likely to make your webserver very very slow.
I don't know why you think you need to do this. Templates are part of your application code, so they would normally live on the same server as your Python code. If you really have some reason to keep them externally, one solution might be to mount the external filesystem onto your webserver via something like sshfs. It's still likely to be very slow though. Think again.
No - it's not possible to trick it into pulling files from another server via http
Yes - you certainly could subclass django.template.loaders.filesystem.Loader (and by altering the load_template_source method appropriately) so that you could load templates via http
Once you had done 3 then the answer to 2 would be yes - it would be feasible - ultimately Django's template language doesn't care where it gets the file from, as long it's in the right format.
However, it seems like a very inefficient way of loading templates and more likely there is a much better way of achieving the same result.
I'm a beginner to Python and Django.
When starting a new project what do you do first before diving into the code?
For example, one could take the following steps:
Configure the settings.py file first
Configure models.py to lay out data structure
Create template files
Define the views/pages
Syncdb
etc
So my question is, what is a good workflow to get through the required steps for a Django application? This also serves as a checklist of things to do. In the definitive guide to Django, the author talks about approaching top down or bottom up. Can anyone expand further on this and perhaps share their process?
Thanks.
Follow the Agile approach. Finish one small case, from the start to the end. From the models to the tests to user experience. Then build on it. Iterate.
Thats the right way to software development.
To do it efficiently, you need: (don't bother right away, you will need it.)
Automated schema migration, automated build system, auto updating and deployment. - None of these, django has got anything to do with. Use pip, fabric, hudson, twill and south appropriately.
Take care not to over burden yourself with all these right away, particularly since you say, you are beginning.
the required steps for a Django application?
There are two required steps.
Write the settings. Write the urls.py
The rest of the steps are optional.
This also serves as a checklist of things to do.
Bad policy. You don't need a checklist of Django features. You need a collection of use cases or user stories which you must implement.
For some reason, you've omitted the two most important and valuable features of Django. Configure the default admin interface and write unit tests. The default admin interface is very high value. Unit testing is absolutely central.
You do it like this.
Gather use cases.
Prioritize the use cases.
Define the actors. The classes of actors becomes groups in the security model.
Define enough "applications" to satisfy the first release of use cases. Define the url structure. Cool URL's don't change.
Build the first use case: models (including security), admin, urls, tests, forms, views and templates. Note that these are the file names (models.py, admin.py, ...) except for templates. Also note that forms and admin should be defined in separate modules even though this isn't required. Also note that templates will be split between a generic templates directory for top-level stuff and application-specific templates.
Build the second use case: models (including security), admin, urls, tests, forms, views and templates.
...
n. Package for release. Tweak up the settings. Configure database and mod-wsgi
I personally can't make a template without writing the views (unless it's a photoshop draft) but in general that's the way I go after I have a plan.
What's extremely important for me is that I don't dive head-first into the code, and that I spend time mocking up the model structure based on the "screens" or "pages" that the user will see.
Once I have a user experience defined, I make sure the backend is robust enough to handle that experience. If I don't visualize the user experience, details get left out that are certainly accomplishable in the shell but not ideal for the website, default django admin, etc.
There are always tradeoffs between agile development and a huge spec: I think there's an important balance. Agile is good: there's no point planning every detail before writing your first line of code, as your needs will change by the time you get to the end. You don't know how your users will really use the site.
On the other hand, without a plan, you can end up with a messy foundation that affects all future code.
An educated guess is a good start. Don't think or assume too much, but definitely have a clear idea how your users will interact with your site for stage 1.
Always try to remember about a DRY rule. For example, why to write RequestContext every new view is defined, where you can simply write a function once, which will add it for you. Good description is here in another topic.
Try to keep a code written one way. Each time you upgrade a schema of your view, edit it in all already written views. That will help keep your code clear and save a lot time for you in future.
Generally good rule, and how do I write my applications is the rule of small steps. Start with writing a settings and urls, then add one model and one view. When it works, modify - add another models or another views. You won't even notice, when your project becomes bigger and bigger.
And the last useful rule for clarity of all the source. Keep files in folders. If you have two subsites based one one (for example "accounts" and "blogs") create two directories names the same. Remeber to put init.py file in each directory. It's really easy to forget. With this practice it's easy to write models and views dedicated to each category. By the way it's a good practice to keep urls like in a tree structure. Main urls.py should contain only links like this one:
(r'^accounts/', include('your_main_name.accounts.urls')),
and of course all media, static, css and so on. In accounts directory urls keep:
urlpatterns = patterns('your_main_name.accounts.views',
url(r'^$', 'index', name='index'),
)
with all views subdirectories.
Last one - keep code clear with actuall django version. Remeber, that the 3.0 release is comming soon.
Hope this will help.
I find that my process varies depending on a lot of variables, mainly whether I know something will work or if I'm experimenting and also whether I'm developing on my production server or in a development environment.
For example, I often do my development directly on the deployment server (most of my work is for intranet projects so there isn't any security risk, etc). But when I do this I really need to make sure the settings and urls are setup first and that gunicorn and nginx are configured.
If I know something should work, or am setting up a generic base set of code, sometimes I'll do all that coding for views and models before I even get enough setup to even run the development server. But when experimenting with new code I find it's good to be able to test every step of the way, so in that case you need your servers running.
In general I do settings, models, syncdb, views, urls, templates, collectstatic, graphics/aesthetics
In general I leave my base.html very plain until the everything else is working, then I add css/js etc.
I guess my point here is that there isn't really a wrong answer for how you do it, and there isn't even only one best practice (as far as I'm concerned). When you do more work, you'll find what you are comfortable with and it'll even vary from project to project.
Good luck, hopefully you learn to love django!
here is something I do in general,
configure basic settings
configure root url.py
configure settings, url.py for static (media) files
create model
sync db
write views (use simple template, if needed)
once you are done with back end implementation
think about UI
prepare styles, scripts
start working on template implementation
I have a fairly complex "product" I'm getting ready to build using Django. I'm going to avoid using the terms "project" and "application" in this context, because I'm not clear on their specific meaning in Django.
Projects can have many apps. Apps can be shared among many projects. Fine.
I'm not reinventing the blog or forum - I don't see any portion of my product being reusable in any context. Intuitively, I would call this one "application." Do I then do all my work in a single "app" folder?
If so... in terms of Django's project.app namespace, my inclination is to use myproduct.myproduct, but of course this isn't allowed (but the application I'm building is my project, and my project is an application!). I'm therefore lead to believe that perhaps I'm supposed to approach Django by building one app per "significant" model, but I don't know where to draw the boundaries in my schema to separate it into apps - I have a lot of models with relatively complex relationships.
I'm hoping there's a common solution to this...
Once you graduate from using startproject and startapp, there's nothing to stop you from combining a "project" and "app" in the same Python package. A project is really nothing more than a settings module, and an app is really nothing more than a models module—everything else is optional.
For small sites, it's entirely reasonable to have something like:
site/
models.py
settings.py
tests.py
urls.py
views.py
Try to answer question: "What does my
application do?". If you cannot answer
in a single sentence, then maybe you can
split it into several apps with cleaner
logic.
I read this thought somewhere soon after I've started to work with django and I find that I ask this question of myself quite often and it helps me.
Your apps don't have to be reusable, they can depend on each other, but they should do one thing.
What is to stop you using myproduct.myproduct? What you need to achieve that roughly consists of doing this:
django-admin.py startproject myproduct
cd myproduct
mkdir myproduct
touch myproduct/__init__.py
touch myproduct/models.py
touch myproduct/views.py
and so on. Would it help if I said views.py doesn't have to be called views.py? Provided you can name, on the python path, a function (usually package.package.views.function_name) it will get handled. Simple as that. All this "project"/"app" stuff is just python packages.
Now, how are you supposed to do it? Or rather, how might I do it? Well, if you create a significant piece of reusable functionality, like say a markup editor, that's when you create a "top level app" which might contain widgets.py, fields.py, context_processors.py etc - all things you might want to import.
Similarly, if you can create something like a blog in a format that is pretty generic across installs, you can wrap it up in an app, with its own template, static content folder etc, and configure an instance of a django project to use that app's content.
There are no hard and fast rules saying you must do this, but it is one of the goals of the framework. The fact that everything, templates included, allows you to include from some common base means your blog should fit snugly into any other setup, simply by looking after its own part.
However, to address your actual concern, yes, nothing says you can't work with the top level project folder. That's what apps do and you can do it if you really want to. I tend not to, however, for several reasons:
Django's default setup doesn't do it.
Often, I want to create a main app, so I create one, usually called website. However, at a later date I might want to develop original functionality just for this site. With a view to making it removable (whether or not I ever do) I tend to then create a separate directory. This also means I can drop said functionality just by unlinking that package from the config and removing the folder, rather than a complex delete the right urls from a global urls.py folder.
Very often, even when I want to make something independent, it needs somewhere to live whilst I look after it / make it independent. Basically the above case, but for stuff I do intend to make generic.
My top level folder often contains a few other things, including but not limited to wsgi scripts, sql scripts etc.
django's management extensions rely on subdirectories. So it makes sense to name packages appropriately.
In short, the reason there is a convention is the same as any other convention - it helps when it comes to others working with your project. If I see fields.py I immediately expect code in it to subclass django's field, whereas if I see inputtypes.py I might not be so clear on what that means without looking at it.
I've found the following blog posts very useful about django applications and projects:
http://www.b-list.org/weblog/2006/sep/10/django-tips-laying-out-application/
http://web.archive.org/web/20080302205555/www.pointy-stick.com/blog/2007/11/09/django-tip-developing-without-projects/
In principle, you have a lot of freedom with django for organizing the source code of your product.
If so... in terms of Django's project.app namespace, my inclination is to usemyproduct.myproduct, but of course this isn't allowed
There is nothing like not allowed. Its your project, no one is restricting you. It is advisable to keep a reasonable name.
I don't see any portion of my product being reusable in any context. Intuitively, I would call this one "application." Do I then do all my work in a single "app" folder?
In a general django project there are many apps (contrib apps) which are used really in every project.
Let us say that your project does only one task and has only a single app (I name it main as thethe project revolves around it and is hardly pluggable). This project too still uses some other apps generally.
Now if you say that your project is using just the one app (INSTALLED_APPS='myproduct') so what is use of project defining the project as project.app, I think you should consider some points:
There are many other things that the code other than the app in a project handles (base static files, base templates, settings....i.e. provides the base).
In the general project.app approach django automatically defines sql schema from models.
Your project would be much easier to be built with the conventional approach.
You may define some different names for urls, views and other files as you wish, but I don't see the need.
You might need to add some applications in future which would be real easy with the conventional django projects which otherwise it may become equally or more difficult and tedious to do.
As far as most of the work being done in the app is concerned, I think that is the case with most of django projects.
Here Django creators points out that difference themselves.
I think that thinking about Apps as they have to be reusable in other projects is good. Also a good way of thinking about Apps in Django provide modern web applications.
Imagine that you are creating big dynamic web app basing on JavaScript.
You can create then in django App named e.g "FrontEnd" <-- in thins app you will display content.
Then you create some backend Apps. E.g App named "Comments" that will store user comments. And "Comments" App will not display anything itself. It will be just API for AJAX requests of your dynamic JS website.
In this way you can always reuse your "Comments" app. You can make it open source without opening source of whole project. And you keep clean logic of your project.
It is sometimes beneficial to share certain constants between various code files in a django application.
Examples:
- Name or location of dump file used in various modules\commands etc
- Debug mode on\off for the entire app
- Site specific configuration
What would be the elegant\pythonic way of doing this?
There's already a project-wide settings.py file. This is the perfect place to put your own custom setttings.
you can provide settings in your settings.py like
MY_SETTING = 'value'
and in any module you can fetch it like
from django.conf import settings
settings.MY_SETTING
Create a configuration module.
Configuration.py: (in your project/app source directory)
MYCONST1 = 1
MYCONST2 = "rabbit"
Import it from other source files:
from Configuration import MYCONST1,MYCONST2
...
Django apps are meant to be (more or less) pluggable. Therefore, you are not supposed to hack into the code of an app in order to parametrize what you want (it would be quite a mess if you had to do this ! Imagine you want to upgrade an app you downloaded on internet... you would have to re-hack into the code of the new version ?!?).
For this reason you shouldn't add app-level settings at the app level, but rather put them together somewhere in your project-wide settings.py.