I'm working on some universal solution for problem with static files and updates in it.
Example: let's say there was site with /static/styles.css file - and site was used for a long time - so a lot of visitors cached this file in browser
Now we doing changes in this css file, and update on server, but some users still have old version (despite modification date returned by server)
The obvious solution is to add some version to file /static/styles.css?v=1.1 but in this case developer must track changes in this file and manually increase version
A second solution is to count the md5 hash of the file and add it to the url /static/styels.css/?v={mdp5hashvalue} which looks much better, but md5 should be calculated automatically somehow.
they possible way I see it - create some template tag like this
{% static_file "style.css" %}
which will render
<link src="/static/style.css?v=md5hash">
BUT, I do not want this tag to calculate md5 on every page load, and I do not want to store hash in django-cache, because then we will have to clear after updating file...
any thoughts ?
Django 1.4 now includes CachedStaticFilesStorage which does exactly what you need (well... almost).
Since Django 2.2 ManifestStaticFilesStorage should be used instead of CachedStaticFilesStorage.
You use it with the manage.py collectstatic task. All static files are collected from your applications, as usual, but this storage manager also creates a copy of each file with the MD5 hash appended to the name. So for example, say you have a css/styles.css file, it will also create something like css/styles.55e7cbb9ba48.css.
Of course, as you mentioned, the problem is that you don't want your views and templates calculating the MD5 hash all the time to find out the appropriate URLs to generate. The solution is caching. Ok, you asked for a solution without caching, I'm sorry, that's why I said almost. But there's no reason to reject caching, really. CachedStaticFilesStorage uses a specific cache named staticfiles. By default, it will use your existing cache system, and voilà! But if you don't want it to use your regular cache, perhaps because it's a distributed memcache and you want to avoid the overhead of network queries just to get static file names, then you can setup a specific RAM cache just for staticfiles. It's easier than it sounds: check out this excellent blog post. Here's what it would look like:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
'LOCATION': '127.0.0.1:11211',
},
'staticfiles': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'staticfiles-filehashes'
}
}
I would suggest using something like django-compressor. In addition to automatically handling this type of stuff for you, it will also automatically combine and minify your files for fast page load.
Even if you don't end up using it in entirety, you can inspect their code for guidance in setting up something similar. It's been better vetted than anything you'll ever get from a simple StackOverflow answer.
I use my own templatetag which add file modification date to url: https://bitbucket.org/ad3w/django-sstatic
Is reinventing the wheel and creating own implementation that bad? Furthermore I would like low level code (nginx for example) to serve my staticfiles in production instead of python application, even with backend. And one more thing: I'd like links stay the same after recalculation, so browser fetches only new files. So here's mine point of view:
template.html:
{% load md5url %}
<script src="{% md5url "example.js" %}"/>
out html:
static/example.js?v=5e52bfd3
settings.py:
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(PROJECT_DIR, 'static')
appname/templatetags/md5url.py:
import hashlib
import threading
from os import path
from django import template
from django.conf import settings
register = template.Library()
class UrlCache(object):
_md5_sum = {}
_lock = threading.Lock()
#classmethod
def get_md5(cls, file):
try:
return cls._md5_sum[file]
except KeyError:
with cls._lock:
try:
md5 = cls.calc_md5(path.join(settings.STATIC_ROOT, file))[:8]
value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
except IsADirectoryError:
value = settings.STATIC_URL + file
cls._md5_sum[file] = value
return value
#classmethod
def calc_md5(cls, file_path):
with open(file_path, 'rb') as fh:
m = hashlib.md5()
while True:
data = fh.read(8192)
if not data:
break
m.update(data)
return m.hexdigest()
#register.simple_tag
def md5url(model_object):
return UrlCache.get_md5(model_object)
Note, to apply changes an uwsgi application (to be specific a process) should be restarted.
Django 1.7 added ManifestStaticFilesStorage, a better alternative to CachedStaticFilesStorage that doesn't use the cache system and solves the problem of the hash being computed at runtime.
Here is an excerpt from the documentation:
CachedStaticFilesStorage isn’t recommended – in almost all cases ManifestStaticFilesStorage is a better choice. There are several performance penalties when using CachedStaticFilesStorage since a cache miss requires hashing files at runtime. Remote file storage require several round-trips to hash a file on a cache miss, as several file accesses are required to ensure that the file hash is correct in the case of nested file paths.
To use it, simply add the following line to settings.py:
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
And then, run python manage.py collectstatic; it will append the MD5 to the name of each static file.
How about you always have a URL Parameter in your URL with a version and whenever you have a major release you change the version in your URL Parameter. Even in the DNS. So if www.yourwebsite.com loads up www.yourwebsite.com/index.html?version=1.0 then after the major release the browser should load www.yourwebsite.com/index.html?version=2.0
I guess this is similar to your solution 1. Instead of tracking files can you track whole directories? For example ratehr than /static/style/css?v=2.0 can you do /static-2/style/css or to make it even granular /static/style/cssv2/.
There is an update for #deathangel908 code. Now it works well with S3 storage also (and with any other storage I think). The difference is using of static file storage for getting file content. Original doesn't work on S3.
appname/templatetags/md5url.py:
import hashlib
import threading
from django import template
from django.conf import settings
from django.contrib.staticfiles.storage import staticfiles_storage
register = template.Library()
class UrlCache(object):
_md5_sum = {}
_lock = threading.Lock()
#classmethod
def get_md5(cls, file):
try:
return cls._md5_sum[file]
except KeyError:
with cls._lock:
try:
md5 = cls.calc_md5(file)[:8]
value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
except OSError:
value = settings.STATIC_URL + file
cls._md5_sum[file] = value
return value
#classmethod
def calc_md5(cls, file_path):
with staticfiles_storage.open(file_path, 'rb') as fh:
m = hashlib.md5()
while True:
data = fh.read(8192)
if not data:
break
m.update(data)
return m.hexdigest()
#register.simple_tag
def md5url(model_object):
return UrlCache.get_md5(model_object)
The major advantage of this solution: you dont have to modify anything in the templates.
This will add the build version into the STATIC_URL, and then the webserver will remove it with a Rewrite rule.
settings.py
# build version, it's increased with each build
VERSION_STAMP = __versionstr__.replace(".", "")
# rewrite static url to contain the number
STATIC_URL = '%sversion%s/' % (STATIC_URL, VERSION_STAMP)
So the final url would be for example this:
/static/version010/style.css
And then Nginx has a rule to rewrite it back to /static/style.css
location /static {
alias /var/www/website/static/;
rewrite ^(.*)/version([\.0-9]+)/(.*)$ $1/$3;
}
Simple templatetag vstatic that creates versioned static files urls that extends Django's behaviour:
from django.conf import settings
from django.contrib.staticfiles.templatetags.staticfiles import static
#register.simple_tag
def vstatic(path):
url = static(path)
static_version = getattr(settings, 'STATIC_VERSION', '')
if static_version:
url += '?v=' + static_version
return url
If you want to automatically set STATIC_VERSION to the current git commit hash, you can use the following snippet (Python3 code adjust if necessary):
import subprocess
def get_current_commit_hash():
try:
return subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).strip().decode('utf-8')
except:
return ''
At settings.py call get_current_commit_hash(), so this will be calculated only once:
STATIC_VERSION = get_current_commit_hash()
I use a global base context in all my views, where I set the static version to be the millisecond time (that way, it will be a new version every time I restart my application):
# global base context
base_context = {
"title": settings.SITE_TITLE,
"static_version": int(round(time.time() * 1000)),
}
# function to merge context with base context
def context(items: Dict) -> Dict:
return {**base_context, **items}
# view
def view(request):
cxt = context({<...>})
return render(request, "page.html", cxt)
my page.html extends my base.html template, where I use it like this:
<link rel="stylesheet" type="text/css" href="{% static 'style.css' %}?v={{ static_version }}">
fairly simple and does the job
Related
When using:
#route('/<filename>')
def server_static(filename):
return static_file(filename, root='.')
it allows to serve the request www.example.com/helloworld with the static HTML file /myapp/helloworld.
How to make www.example.com/anything be served by the static HTML file /myapp/html/anything.html, without having to hardcode each static filename anything in the Python code?
Note: the tricky part is that the request is /anything (and not /anything.html), and the static file is /myapp/html/anything.html (there are 20 or more such files)
If it's always .html you could consider simply doing filename += '.html'.
If your needs are more complex, you could write code that examines the directory to match various extensions. For example, if you wanted to match .html files, something like this would work and would be adjustable to work with various/multiple extensions or other conditions.
def is_html(filename):
return filename.lower().endswith('.html')
#route('/<filename>')
def server_static(filename):
root = '.'
all_filenames = os.listdir(root)
html_files = filter(is_html, all_filenames)
for fname in html_files:
if fname.startswith(filename):
return static_file(fname, root=root)
else:
return "No such file"
This can probably be abbreviated using fnmatch or something similar.
Although, you may want to simply consider simply having a webserver like Apache simply serve that path instead of using Flask. This would probably be a more secure option, too.
In Django we have ManifestStaticFilesStorage for caching static files, but it works between Django and browser, but I want right cache between user and browser.
I want: every time static file is changed, hash of file is recalculated and browser cache is invalidated and user see new static file without F5 adn without running --collectstatic --no-input.
My code now isn't working:
settings.py
STATICFILES_STORAGE = 'auth.utils.HashPathStaticFilesStorage'
CACHES = {
'staticfiles': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'staticfiles',
'TIMEOUT': 3600 * 24 * 7,
'MAX_ENTRIES': 100,
}
}
and auth.utils.py:
# -*- coding: utf-8 -*-
import time
from hashlib import sha384
from django.conf import settings
from django.core.cache import cache
from django.contrib.staticfiles.storage import ManifestStaticFilesStorage
try:
ACCURACY = settings.STATICFILES_HASH_ACCURACY
except AttributeError:
ACCURACY = 12
try:
KEY_PREFIX = settings.STATICFILES_HASH_KEY_PREFIX
except AttributeError:
KEY_PREFIX = 'staticfiles_hash'
class HashPathStaticFilesStorage(ManifestStaticFilesStorage):
"""A static file storage that returns a unique url based on the contents
of the file. When a static file is changed the url will also change,
forcing all browsers to download the new version of the file.
The uniqueness of the url is a GET parameter added to the end of it. It
contains the first 12 characters of the SHA3 sum of the contents of the
file.
Example: {% static "image.jpg" %} -> /static/image.jpg?4e1243
The accuracy of the hash (number of characters used) can be set in
settings.py with STATICFILES_HASH_ACCURACY. Setting this value too low
(1 or 2) can cause different files to get the same hash and is not
recommended. SHA3 hashes are 40 characters long so all accuracy values
above 40 have the same effect as 40.
The values can be cached for faster performance. All keys in the cache have
the prefix specified in STATICFILES_HASH_KEY_PREFIX in setings.py. This
value defaults to 'staticfiles_hash'
"""
#property
def prefix_key(self):
return "%s:%s" % (KEY_PREFIX, 'prefix')
def invalidate_cache(self, nocache=False):
"""Invalidates the cache. Run this when one or more static files change.
If called with nocache=True the cache will not be used.
"""
value = int(time.time())
if nocache:
value = None
cache.set(self.prefix_key, value)
def get_cache_key(self, name):
hash_prefix = cache.get(self.prefix_key)
if not hash_prefix:
return None
key = "%s:%s:%s" % (KEY_PREFIX, hash_prefix, name)
return key
def set_cached_hash(self, name, the_hash):
key = self.get_cache_key(name)
if key:
cache.set(key, the_hash)
def get_cached_hash(self, name):
key = self.get_cache_key(name)
if not key:
return None
the_hash = cache.get(key)
return the_hash
def calculate_hash(self, name):
path = self.path(name)
try:
the_file = open(path, 'rb')
the_hash = sha384(the_file.read()).hexdigest()[:ACCURACY]
the_file.close()
except IOError:
return ""
return the_hash
def get_hash(self, name):
the_hash = self.get_cached_hash(name)
if the_hash:
return the_hash
the_hash = self.calculate_hash(name)
self.set_cached_hash(name, the_hash)
return the_hash
def url(self, name):
base_url = super(HashPathStaticFilesStorage, self).url(name)
the_hash = self.get_hash(name)
if "?" in base_url:
return "%s&%s" % (base_url, the_hash)
return "%s?%s" % (base_url, the_hash)
I just use this very simple idea
<img src="{{ company.logo.url }}?v={% now 'U' %}" />
Force a version with ?v= and set the version to the current timestamp {% now 'U' %} so it will change with every request
A common and simple approach to avoid users having to reload the page to get fresh static content is to append some mutable value in the inclusion of the static files in the HTML markup, something like this:
<script src="{% static 'js/library.js' %}?{{ version }}"></script>
In this way when the variable version assumes a different value, the browser is forced to download a new version of the static files from the server.
You can set version using a custom context processor, for instance reading the project version from settings. Something like this:
from django.conf import settings
def version(request):
return {
'version': settings.VERSION
}
If you are using git as VCS, another approach would be writing the last commit hash of your project in a file, when you push your modifications to the server. The file should be in a format that is readable by Python. In this way you can use the git commit hash as the version variable mentioned before.
You can do this using a GIT post-receive hook:
#!/bin/bash
WORKDIR=/path/to/project/
VERSION_MODULE=${WORKDIR}django_project/project/version.py
# for every branch which has been pushed
while read oldrev newrev ref
do
# if branch pushed is master, update version.py file in the django project
if [[ $ref =~ .*/master$ ]]; then
GIT_WORK_TREE=$WORKDIR git checkout -f master
echo "GIT_REF = 'master'" > $VERSION_MODULE
echo "GIT_REV = '$newrev'" >> $VERSION_MODULE
fi
done
Then your context processor could be:
from project.version import GIT_REV
def version(request):
return {
'version': GIT_REV[:7]
}
Super simple idea: change the STATIC_URL variable in your Django settings.
STATIC_URL = /static/ # before
STATIC_URL = /static/v2/ # after
This will change the path of all static files, forcing browsers to reload the content.
I have tried to get the autoprefixer filter to work with flask_assets by following the instructions in the Flask_Assets documentation, but it does not appear to apply the filter. Here is my code:
# construct flask app object
from flask import Flask, render_template_string
flask_args = { 'import_name': __name__ }
flask_app = Flask(**flask_args)
from flask_assets import Environment, Bundle
assets = Environment(flask_app)
assets.config['AUTOPREFIXER_BIN'] = 'postcss'
assets.config['AUTOPREFIXER_BROWSERS'] = [ '> 1%' ]
css_min = Bundle('../styles/mycss.css', filters='autoprefixer', output='styles/test.css')
assets.register('css_assets', css_min)
#flask_app.route('/')
def landing_page():
html = '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\
<head>{% assets "css_assets" %}\
<link rel="stylesheet" href="{{ ASSET_URL }}" type="text/css">\
{% endassets %}\
<title>Hello</title>\
</head>\
<h1>Hello World</h1>\
<p>Just a test of flask</p>'
return render_template_string(html), 200
if __name__ == '__main__':
flask_app.run(host='0.0.0.0', port=5000)
I have been able to apply the cssmin, pyscss, uglifyjs and jsmin filters successfully. I can also run autoprefixer on the command line to successfully compile a transformed output:
postcss --use autoprefixer --autoprefixer.browsers "> 1%" -o test.css mycss.css
However, when trying to run autoprefixer through flask_assets registration, the process neither throws an error nor does it seem to take the required time to compile. It does produce the output file but when I examine the resulting file, none of the prefixes have been applied.
UPDATE: This problem seems to occur whenever attempting to configure options for ANY filter. I have not been able to get uglifyjs to accept 'UGLIFYJS_EXTRA_ARGS' or for the pyscss filter to adopt a new style using 'PYSCSS_STYLE' either. I have tried to set these configuration as environmental variables using os.environ['AUTOPREFIXER_BIN'] as well as attempting to pass them through flask.config['AUTOPREFIXER_BIN']. But none of the configuration settings have been applied when the filter is run. It is also not clear to me where in the code itself the configuration options are constructed by either Bundle or Environment.
One SO post claims to have found a way to get a configuration setting to work, but the post does not show the entire workflow of how flask_assets needs to be setup to ingest these options.
Perhaps someone can help me understand what I am doing wrong?
Autoprefixer:
There is nothing wrong with your code1. You are just not using the correct filter for the latest version of Autoprefixer. If you look at the history of the releases in that link, since version 6.0.0, it started using postcss. Your code will work for versions older than 6.0.0.
Webassets has provided support for versions after 6.0.0 (inclusive), by providing the autoprefixer6 filter.
Therefore all you have to do is change the filter(s) while initializing your bundle, like so:
css_min = Bundle('../styles/mycss.css', filters='autoprefixer6', output='styles/test.css')
Other Filters' Configurations:
Don't use os.environ, that is not the way to set configuration variables for Flask and flask-extensions. The most common (and preferred) way to set configuration for extensions is by using the flask Config itself, and in large projects this is done using a separate config file. The extensions will pickup its configuration options from flask's config.
Depending on which extension you use, you can also set the config separately like you have done, but that is rarely used, from what I have seen so far.
Please check the Flask's Configuration related documentation for some good examples on how to setup configuration for your app "properly".
from flask import Flask, render_template_string
from flask_assets import Environment, Bundle
# construct flask app object
flask_args = {'import_name': __name__}
flask_app = Flask(**flask_args)
assets = Environment(flask_app)
# specify the bin path (optional), required only if not globally installed
assets.config['AUTOPREFIXER_BIN'] = 'path/to/postcss'
assets.config['AUTOPREFIXER_BROWSERS'] = ['> 1%', ]
# use the autoprefixer6 updated filter
css_min = Bundle('../styles/mycss.css', filters='autoprefixer6',
output='styles/test.css')
assets.register('css_assets', css_min)
#flask_app.route('/')
def landing_page():
html = '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\
<head>{% assets "css_assets" %}\
<link rel="stylesheet" href="{{ ASSET_URL }}" type="text/css">\
{% endassets %}\
<title>Hello</title>\
</head>\
<h1>Hello World</h1>\
<p>Just a test of flask</p>'
return render_template_string(html), 200
if __name__ == '__main__':
flask_app.run(host='0.0.0.0', port=5000)
Remember to clean out the previously generated files, if the source css/js has not changed, i.e remove the output files, and the .webassets-cache folder.
1Except for code style & formatting conventions!
I'm using django-pipeline with s3. I'm successfully using collectstatic to combined my Javascript files and store them in my s3 bucket, but they are not getting compressed for some reason (verified by looking at the file, its size, and its content-encoding). Otherwise things are working correctly with the combined scripts.js that is produced.
Here are the changes I made to use django-pipeline:
Added pipeline to installed apps.
Added 'pipeline.finders.PipelineFinder' to STATICFILES_FINDERS.
Set STATICFILES_STORAGE = 'mysite.custom_storages.S3PipelineManifestStorage' where this class is as defined in the documentation, as seen below.
Set PIPELINE_JS as seen below, which works but just isn't compressed.
PIPELINE_ENABLED = True since DEBUG = True and I'm running locally.
PIPELINE_JS_COMPRESSOR = 'pipeline.compressors.yuglify.YuglifyCompressor' even though this should be default.
Installed the Yuglify Compressor with npm -g install yuglify.
PIPELINE_YUGLIFY_BINARY = '/usr/local/bin/yuglify' even though the default with env should work.
Using the {% load pipeline %} and {% javascript 'scripts' %} which work.
More detail:
PIPELINE_JS = {
'scripts': {
'source_filenames': (
'lib/jquery-1.11.1.min.js',
...
),
'output_filename': 'lib/scripts.js',
}
}
class S3PipelineManifestStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
location = settings.STATICFILES_LOCATION
As mentioned, collectstatic does produce scripts.js just not compressed. The output of that command includes:
Post-processed 'lib/scripts.js' as 'lib/scripts.js'
I'm using Django 1.8, django-pipeline 1.5.2, and django-storages 1.1.8.
Similar questions:
django-pipeline not compressing
django pipeline with S3 storage not compressing
The missing step was to also extend GZipMixin, AND, it has to be first in the list of parents:
from pipeline.storage import GZIPMixin
class S3PipelineManifestStorage(GZIPMixin, PipelineMixin, ManifestFilesMixin, S3BotoStorage):
location = settings.STATICFILES_LOCATION
Now collectstatic produces a .gz version of each file as well, but my templates still weren't referencing the .gz version.
To address this the author says:
To make it work with S3, you would need to change the staticfiles
storage url method to return .gz urls (and staticfiles/pipeline
template tags depending if you care for clients that don't support
gzip). Also don't forget to setup the proper header on s3 to serve
theses assets as being gzipped.
I adapted an example he provided elsewhere, which overrides the url method:
class S3PipelineManifestStorage(GZIPMixin, PipelineMixin, ManifestFilesMixin, S3BotoStorage):
location = settings.STATICFILES_LOCATION
def url(self, name, force=False):
# Add *.css if you are compressing those as well.
gzip_patterns = ("*.js",)
url = super(GZIPMixin, self).url(name, force)
if matches_patterns(name, gzip_patterns):
return "{0}.gz".format(url)
return url
This still doesn't handle setting the Content-Encoding header.
A simpler alternative is to use the S3Boto Storages option AWS_IS_GZIPPED which performs gzipping AND sets the appropriate header.
More is required to support clients without gzip, however.
Also useful are these instructions from Amazon on serving compressed files from S3.
I want to add set of css files to head. For instance,
python file:
modules.py
class SomeModule(tornado.web.UIModule):
def css_files(self):
return [self.handler.static_url('css/modules/some-module.css'),]
def render(self, some_data=None):
result = ''
if some_data is not None:
"""to do something with data"""
return result
server.py
app = Application(
...
ui_module=modules
...
)
template file:
...
{% module SomeModule(some_data=put_data_here) %}
As the result, I see only data that were returned from render. But css files weren't set between head tags.
What's the result of the template file depends on how you generate the template file. As far as I can see, you didn't include the code for that in your question.
If you want to use css_files() or similar, you need to use self.render_string() in order to give Tornado a chance to insert the CSS tags in the proper places.
For an example of how to use tornado.web.UIModule, see this Slideshare presentation