We are building uServices based on tornado. There are some routes which are common for all uServices, for the time being health and docs. The docs route is built using Swagger. That means, the swagger route, and associated assets, are part of our common library (but not the documentation itself, which is uService related), which is simply a requirement to our uServices.
Since swagger needs static assets, and each uService needs its own static assets too, I have a problem: my static assets are coming from two completely different places. I have found a (very inconvenient) hack to solve this:
track static assets in the common repo
track static assets in the uService repo
when deploying, copy those static assets, from both sources, to a deployment static folder (and hope that there are no clashes)
specify the deployment static folder as the static_path
Step 3 is quite complex, because it involves pip-installing the common library, finding the assets there (already a hack), and copying them around:
install the common library with pip
look for the location of the installed library (site-packages)
copy the static assets to the deployment static folder
It would be much easier if, as happens with the template_path, it would be possible to have a per-application static folder:
> Return None to load templates relative to the calling file.
I have been searching the web, and it seems this is not possible, but I wanted to double check.
An alternative would be to simplify asset extracting from a packaged python module, but not sure what is the preferred method to do this, without having to recur to hacks searching in the virtualenv for the site-packages.
It's not possible to do this with the static_path setting, but as long as you don't require the static_url() function, you can create multiple StaticFileHandler entries in your URLSpec list:
Application([
('/static1/(.*)', tornado.web.StaticFileHandler, dict(path='static1')),
('/static2/(.*)', tornado.web.StaticFileHandler, dict(path='static1')),
])
As a substitute for static_url, you can call StaticFileHandler.make_static_url and pass it the arguments that would have been global if you were using static_path.
Related
Currently, collecstatic gathers all files of the project's Apps in one directory (STATIC_ROOT).
What should be done to keep the project structure for the static files:
STATIC_ROOT/App1
..
STATIC_ROOT/Appn
STATIC_ROOT/App3
thanks
The documentation recommends namespacing static files:
Static file namespacing
Now we might be able to get away with putting our static files directly in my_app/static/ (rather than creating another my_app subdirectory), but it would actually be a bad idea. Django will use the first static file it finds whose name matches, and if you had a static file with the same name in a different application, Django would be unable to distinguish between them. We need to be able to point Django at the right one, and the easiest way to ensure this is by namespacing them. That is, by putting those static files inside another directory named for the application itself.
Make sure to create an inner directory named after your app and put your files in there, e.g.
my_app/
static/
my_app/ ← This directory
file1
file1
...
Thank you chris,
I find exactly the solution as indicated here: How should I structure my static app files in Django?
I have just replace STATICDIR_FILES by :
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
]
I'm having a rather strange problem using a jinja2.ChoiceLoader (also tried with multiple paths with FileSystemLoader, no joy) in Flask.
I have several "theme" directories, like so.
/templates/
themes/
default/
layout.html
menu.html
blue/
layout.html
grey/
menu.html
...
And I'd like to fallback to default/ if the selected theme doesn't have the required template, so I used a ChoiceLoader, like so.
#app.before_request
def setup_request():
current_theme = get_theme()
logging.info('Using theme %s'%(current_theme))
app.jinja_loader = jinja2.ChoiceLoader([
jinja2.FileSystemLoader('/templates/themes/%s/'%(current_theme)),
jinja2.FileSystemLoader('/templates/themes/default/')
])
That's great, but if I change the <current_theme> it still loads the theme from the old folder until I reload Apache or restart the Flask development server.
It should be using the new theme. Logging says that it's using the changed theme, but apparently app.jinja_loader is a bit like honey badger... it's completely ignoring it until I reload Apache.
Edit: This appears to be related to Flask considering all files of the same name to be the same file. I can reproduce with the builtin server (with DEBUG=True), Cherry, and mod_wsgi. This person seems to have a similar problem, but no simple solution: flask blueprint template folder My situation is different in the sense that I require cascading templates for a single app. His problem is related to cascading templates between blueprints, but it may be the same issue under the hood.
Here's the code that's in the "get_theme()" call:
def get_theme():
# I know this is unsafe, testing only
return request.args.get('theme','default')
Edit 2: I need to change the HTML and JavaScript between themes, not just the CSS. This is why I'm not just loading different CSS files. Also, some of these themes are for mobile devices, and have very little in common with the other themes.
Edit 3: Two solutions. Solution 1: Name the files uniquely, like "blue.layout.html" and "default.layout.html". This works perfectly, but it doesn't cascade as required. Solution 2: Use relative paths, so instead of include 'file.html', use include 'theme/blue/file.html. I achieved cascading by creating a get_theme_file() function that checks for the active theme, checks if the file exists (if not, fallback to "default" theme), and returns the relative path. I just have to make sure everything I include looks like {% include get_theme_file('file.html') %}. This is not elegant, but I find it to be more elegant that the low-level fiddling with Flask used here.
By the way, you can pass multiple locations to FileSystemLoader, and it is the recommended way to load templates
This is expected behavior in Apache with mod_wsgi (which I assume you are using). File system changes don't trigger a reload of all the processes. See this entry in the Flask docs that talks about this, and offers a workaround which is to add:
WSGIScriptReloading On
To the configuration section for your application and then touching the wsgi file to trigger a reload of the child processes.
Are you sure this is what you intend? Most theme switching tricks rely on the cascading part of cascading style sheets (CSS) to control themes.
Well, I'm not the only one to encounter this problem. The issue is that Flask caches based on filename, and if you don't include the relative path it just caches the first one to be loaded. There are three ways to accomplish dynamic, cascading, templates.
Override Jinja builtins. Which I found to be quite confusing. I'm not smart enough for this solution.
Serve a different WSGI process for each file. The setup here seems a bit too much for a dynamic site. Flask caches the filenames per WSGI process, so you can do something with multiple Cherry WSGI servers, for example.
Include the theme in the load path. Create a function, load it in with context_processor, and only load template files using that function. The function needs to check for a non-default theme, check if the file exists, and return it. So a template call would be get_theme_file('layout.html') which would return the relative path (like themes/blue/layout.html).
An example of Option 3.
def get_theme_file(fname):
theme = get_theme()
if os.path.exists(os.path.join(theme.theme_dir, fname)):
return os.path.join('themes', theme.name, fname)
return os.path.join('themes', 'default', fname)
...
# Each render_template should reference this
return render_template(get_theme_file('layout.html'))
If you include theme files in templates:
{% include get_theme_file('layout.html') %}
Unfortunately, this doesn't cache, but I can see a few ways to optimize it. Maybe cache an os.listdir of the get_theme().theme_dir, and instead of reading the filesystem for each get_theme_file call, just do an in check against the cached listdir list.
It's worth noting that this is Flask specific. I was unable to reproduce this behavior with plain Jinja2 and my own WSGI server. One might say that Flask was a poor choice for this particular project, but I'd argue that the savings from everything else Flask does was well worth it.
I'm new to Django and to Python, and I've just completed Django's "build your first app" tutorial. I found it fantastic. However it never spoke about stylesheets, javascripts or anything of the like.
Where should they go? Currently my project directory looks like this:
myproject
+app1
+app2
+myproject
+public
+css
+img
+js
+templates
Am I doing it correctly? I would like to stick to Django/Python standards as much as possible. Also, if I would like to include certain styles and scripts in all of my templates, how can I do that? Let's say I'm using Bootstrap.
You should probably take a closer look at Managing static files.
Basically you can put a static folder in each of your apps (similar to your template folders) and the more general stuff (base.css, project-wide icons, jquery-plugins) in a static folder in your project root.
So the layout might look like this:
myproject
+app1
+static
+app1
+css
+js
+app2
+static
+css
+img
Once you deploy your project you can use, the collectstatic-command from contrib.staticfiles to collect all your static files to a single directory (STATIC_ROOT) where you can serve it from.
I prefer to call the +public folder +static, and put the +template (app specific) folder inside the application that is using them (so to avoid confusion and to be able to use a template with a different name twice), also I don't understand why you put +myproject inside your root project.
I have a fairly complex "product" I'm getting ready to build using Django. I'm going to avoid using the terms "project" and "application" in this context, because I'm not clear on their specific meaning in Django.
Projects can have many apps. Apps can be shared among many projects. Fine.
I'm not reinventing the blog or forum - I don't see any portion of my product being reusable in any context. Intuitively, I would call this one "application." Do I then do all my work in a single "app" folder?
If so... in terms of Django's project.app namespace, my inclination is to use myproduct.myproduct, but of course this isn't allowed (but the application I'm building is my project, and my project is an application!). I'm therefore lead to believe that perhaps I'm supposed to approach Django by building one app per "significant" model, but I don't know where to draw the boundaries in my schema to separate it into apps - I have a lot of models with relatively complex relationships.
I'm hoping there's a common solution to this...
Once you graduate from using startproject and startapp, there's nothing to stop you from combining a "project" and "app" in the same Python package. A project is really nothing more than a settings module, and an app is really nothing more than a models module—everything else is optional.
For small sites, it's entirely reasonable to have something like:
site/
models.py
settings.py
tests.py
urls.py
views.py
Try to answer question: "What does my
application do?". If you cannot answer
in a single sentence, then maybe you can
split it into several apps with cleaner
logic.
I read this thought somewhere soon after I've started to work with django and I find that I ask this question of myself quite often and it helps me.
Your apps don't have to be reusable, they can depend on each other, but they should do one thing.
What is to stop you using myproduct.myproduct? What you need to achieve that roughly consists of doing this:
django-admin.py startproject myproduct
cd myproduct
mkdir myproduct
touch myproduct/__init__.py
touch myproduct/models.py
touch myproduct/views.py
and so on. Would it help if I said views.py doesn't have to be called views.py? Provided you can name, on the python path, a function (usually package.package.views.function_name) it will get handled. Simple as that. All this "project"/"app" stuff is just python packages.
Now, how are you supposed to do it? Or rather, how might I do it? Well, if you create a significant piece of reusable functionality, like say a markup editor, that's when you create a "top level app" which might contain widgets.py, fields.py, context_processors.py etc - all things you might want to import.
Similarly, if you can create something like a blog in a format that is pretty generic across installs, you can wrap it up in an app, with its own template, static content folder etc, and configure an instance of a django project to use that app's content.
There are no hard and fast rules saying you must do this, but it is one of the goals of the framework. The fact that everything, templates included, allows you to include from some common base means your blog should fit snugly into any other setup, simply by looking after its own part.
However, to address your actual concern, yes, nothing says you can't work with the top level project folder. That's what apps do and you can do it if you really want to. I tend not to, however, for several reasons:
Django's default setup doesn't do it.
Often, I want to create a main app, so I create one, usually called website. However, at a later date I might want to develop original functionality just for this site. With a view to making it removable (whether or not I ever do) I tend to then create a separate directory. This also means I can drop said functionality just by unlinking that package from the config and removing the folder, rather than a complex delete the right urls from a global urls.py folder.
Very often, even when I want to make something independent, it needs somewhere to live whilst I look after it / make it independent. Basically the above case, but for stuff I do intend to make generic.
My top level folder often contains a few other things, including but not limited to wsgi scripts, sql scripts etc.
django's management extensions rely on subdirectories. So it makes sense to name packages appropriately.
In short, the reason there is a convention is the same as any other convention - it helps when it comes to others working with your project. If I see fields.py I immediately expect code in it to subclass django's field, whereas if I see inputtypes.py I might not be so clear on what that means without looking at it.
I've found the following blog posts very useful about django applications and projects:
http://www.b-list.org/weblog/2006/sep/10/django-tips-laying-out-application/
http://web.archive.org/web/20080302205555/www.pointy-stick.com/blog/2007/11/09/django-tip-developing-without-projects/
In principle, you have a lot of freedom with django for organizing the source code of your product.
If so... in terms of Django's project.app namespace, my inclination is to usemyproduct.myproduct, but of course this isn't allowed
There is nothing like not allowed. Its your project, no one is restricting you. It is advisable to keep a reasonable name.
I don't see any portion of my product being reusable in any context. Intuitively, I would call this one "application." Do I then do all my work in a single "app" folder?
In a general django project there are many apps (contrib apps) which are used really in every project.
Let us say that your project does only one task and has only a single app (I name it main as thethe project revolves around it and is hardly pluggable). This project too still uses some other apps generally.
Now if you say that your project is using just the one app (INSTALLED_APPS='myproduct') so what is use of project defining the project as project.app, I think you should consider some points:
There are many other things that the code other than the app in a project handles (base static files, base templates, settings....i.e. provides the base).
In the general project.app approach django automatically defines sql schema from models.
Your project would be much easier to be built with the conventional approach.
You may define some different names for urls, views and other files as you wish, but I don't see the need.
You might need to add some applications in future which would be real easy with the conventional django projects which otherwise it may become equally or more difficult and tedious to do.
As far as most of the work being done in the app is concerned, I think that is the case with most of django projects.
Here Django creators points out that difference themselves.
I think that thinking about Apps as they have to be reusable in other projects is good. Also a good way of thinking about Apps in Django provide modern web applications.
Imagine that you are creating big dynamic web app basing on JavaScript.
You can create then in django App named e.g "FrontEnd" <-- in thins app you will display content.
Then you create some backend Apps. E.g App named "Comments" that will store user comments. And "Comments" App will not display anything itself. It will be just API for AJAX requests of your dynamic JS website.
In this way you can always reuse your "Comments" app. You can make it open source without opening source of whole project. And you keep clean logic of your project.
I started an application in Google App Engine right when it came out, to play with the technology and work on a pet project that I had been thinking about for a long time but never gotten around to starting. The result is BowlSK. However, as it has grown, and features have been added, it has gotten really difficult to keep things organized - mainly due to the fact that this is my first python project, and I didn't know anything about it until I started working.
What I have:
Main Level contains:
all .py files (didn't know how to make packages work)
all .html templates for main level pages
Subdirectories:
separate folders for css, images, js, etc.
folders that hold .html templates for subdirecty-type urls
Example:
http://www.bowlsk.com/ maps to HomePage (default package), template at "index.html"
http://www.bowlsk.com/games/view-series.html?series=7130 maps to ViewSeriesPage (again, default package), template at "games/view-series.html"
It's nasty. How do I restructure? I had 2 ideas:
Main Folder containing: appdef, indexes, main.py?
Subfolder for code. Does this have to be my first package?
Subfolder for templates. Folder heirarchy would match package heirarchy
Individual subfolders for css, images, js, etc.
Main Folder containing appdef, indexes, main.py?
Subfolder for code + templates. This way I have the handler class right next to the template, because in this stage, I'm adding lots of features, so modifications to one mean modifications to the other. Again, do I have to have this folder name be the first package name for my classes? I'd like the folder to be "src", but I don't want my classes to be "src.WhateverPage"
Is there a best practice? With Django 1.0 on the horizon, is there something I can do now to improve my ability to integrate with it when it becomes the official GAE templating engine? I would simply start trying these things, and seeing which seems better, but pyDev's refactoring support doesn't seem to handle package moves very well, so it will likely be a non-trivial task to get all of this working again.
First, I would suggest you have a look at "Rapid Development with Python, Django, and Google App Engine"
GvR describes a general/standard project layout on page 10 of his slide presentation.
Here I'll post a slightly modified version of the layout/structure from that page. I pretty much follow this pattern myself. You also mentioned you had trouble with packages. Just make sure each of your sub folders has an __init__.py file. It's ok if its empty.
Boilerplate files
These hardly vary between projects
app.yaml: direct all non-static requests to main.py
main.py: initialize app and send it all requests
Project lay-out
static/*: static files; served directly by App Engine
myapp/*.py: app-specific python code
views.py, models.py, tests.py, __init__.py, and more
templates/*.html: templates (or myapp/templates/*.html)
Here are some code examples that may help as well:
main.py
import wsgiref.handlers
from google.appengine.ext import webapp
from myapp.views import *
application = webapp.WSGIApplication([
('/', IndexHandler),
('/foo', FooHandler)
], debug=True)
def main():
wsgiref.handlers.CGIHandler().run(application)
myapp/views.py
import os
import datetime
import logging
import time
from google.appengine.api import urlfetch
from google.appengine.ext.webapp import template
from google.appengine.api import users
from google.appengine.ext import webapp
from models import *
class IndexHandler(webapp.RequestHandler):
def get(self):
date = "foo"
# Do some processing
template_values = {'data': data }
path = os.path.join(os.path.dirname(__file__) + '/../templates/', 'main.html')
self.response.out.write(template.render(path, template_values))
class FooHandler(webapp.RequestHandler):
def get(self):
#logging.debug("start of handler")
myapp/models.py
from google.appengine.ext import db
class SampleModel(db.Model):
I think this layout works great for new and relatively small to medium projects. For larger projects I would suggest breaking up the views and models to have their own sub-folders with something like:
Project lay-out
static/: static files; served directly by App Engine
js/*.js
images/*.gif|png|jpg
css/*.css
myapp/: app structure
models/*.py
views/*.py
tests/*.py
templates/*.html: templates
My usual layout looks something like this:
app.yaml
index.yaml
request.py - contains the basic WSGI app
lib
__init__.py - common functionality, including a request handler base class
controllers - contains all the handlers. request.yaml imports these.
templates
all the django templates, used by the controllers
model
all the datastore model classes
static
static files (css, images, etc). Mapped to /static by app.yaml
I can provide examples of what my app.yaml, request.py, lib/init.py, and sample controllers look like, if this isn't clear.
I implemented a google app engine boilerplate today and checked it on github. This is along the lines described by Nick Johnson above (who used to work for Google).
Follow this link gae-boilerplate
I think the first option is considered the best practice. And make the code folder your first package. The Rietveld project developed by Guido van Rossum is a very good model to learn from. Have a look at it: http://code.google.com/p/rietveld
With regard to Django 1.0, I suggest you start using the Django trunk code instead of the GAE built in django port. Again, have a look at how it's done in Rietveld.
I like webpy so I've adopted it as templating framework on Google App Engine.
My package folders are typically organized like this:
app.yaml
application.py
index.yaml
/app
/config
/controllers
/db
/lib
/models
/static
/docs
/images
/javascripts
/stylesheets
test/
utility/
views/
Here is an example.
I am not entirely up to date on the latest best practices, et cetera when it comes to code layout, but when I did my first GAE application, I used something along your second option, where the code and templates are next to eachother.
There was two reasons for this - one, it kept the code and template nearby, and secondly, I had the directory structure layout mimic that of the website - making it (for me) a bit easier too remember where everything was.