I'm using Django 1.8 and I want to add a parameter to my static files to cache bust.
This is what I'm doing right now, setting a manual parameter:
<link href="{% static 'css/openprescribing.css' %}?q=0.1.1" rel="stylesheet">
But I feel there must be a better way to update the parameter.
I guess it would be a bit neater to have a setting passed through the template (and that would save having to update it in multiple places).
But what would be really nice is if Django could update it automatically for me.
The notes on django-cachebuster suggest that it's now possible to do this automatically in staticfiles, but I can't find anything in the staticfiles docs about it.
Anyone know a way to do this?
Yes this can be done automatically with contrib.staticfiles. There are two additional provided storage classes which will rename files using a hash. These
are documented here: ManifestStaticFilesStorage and CachedStaticFilesStorage
From the docs:
A subclass of the StaticFilesStorage storage backend which stores the file names it handles by appending the MD5 hash of the file’s content to the filename. For example, the file css/styles.css would also be saved as css/styles.55e7cbb9ba48.css.
The purpose of this storage is to keep serving the old files in case some pages still refer to those files, e.g. because they are cached by you or a 3rd party proxy server. Additionally, it’s very helpful if you want to apply far future Expires headers to the deployed files to speed up the load time for subsequent page visits.
The main difference is
CachedStaticFilesStorage is a similar class like the ManifestStaticFilesStorage class but uses Django’s caching framework for storing the hashed names of processed files instead of a static manifest file called staticfiles.json. This is mostly useful for situations in which you don’t have access to the file system.
To enable them you need to change your STATICFILES_STORAGE setting is set to 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage' or 'django.contrib.staticfiles.storage.CachedStaticFilesStorage'. The file names are only changed when DEBUG=False as it would be in production.
I'm no expert in Caching, but I think letting nginx handle caching might be better than using Django. Django has a lot to handle, so you can let the lightweight static server do that nasty job.
I do not use cloudflare, but I use this line to cache my statics, however, immediately the file changes, Nginx propagates the most recent file (thats the same file):
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
expires 30d;
}
which is a snippet from this gist I'm currently using that conf in production, so I know it works.
One thing I'll point out is, make sure MemCached isn't working and connected to your django as a Caching Backend. I say this, because I have spent many hours in time past hitting my head against the wall, simply because MemCached was caching my page with every content in it for up to 10 minutes.
with this nginx location conf, Whenever I change my .css, or upload a new file (static), the new file immediately takes over, as long as I've python manage.py collectstatic'ed them into the appropriate directory
I stand to be corrected though, if that's not actually the part doing the trick.
Proof that above works with Cache-busting (as you call it)
I went into server
Deleted my static folder (nginx still running) sudo rm -rf static/
Accessed my site
No static loaded
Went back, and python manage.py collectstatic
Accessed my site again. Statics loaded
No browser hard refresh was used. No nginx reload|restart whatever was used.
Nginx is smart enough to cache your statics, but reload the static when the file is new and serve it.
If you use a CDN, cache busting becomes more complex. If you just want to bust the cache for a specific image, just create a simple tag that does the trick....
import time
from django import template
register = template.Library()
#register.simple_tag()
def cache_bust():
return int(time.time())
Then in your template just do something like this...
{% load cache_app %}
<img src="/captcha/?cache_bust={% cache_bust %}" class="captcha"/>
And you have cache busting the simple way.
Related
How do you dynamically access the domain name URL in the settings.py file of Django? (ie "http://localhost:8000")
I am trying to overwrite a package CDN while the internet is unavailable during development, and want to point to the local file in the static files directory. While os.path.join(BASE_DIR, "path/to/local.file") should work, it is context-dependent as to which app/url (ie "http://localhost:8000/app/static/css/bootstrap.min.css
"), and not just the main domain with the static file location appended to the starting server with ./manage.py runserver 0:8000 (ie " http://localhost:8000/static/css/bootstrap.min.css").
Notes:
Because this is in the settings.py, I can't load any apps or reverse because of the error *** django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
I am not in a template, so I can't use the static url
statically defining it won't allow for different port loadings when starting via ./manage.py runserver 0:8000
settings.py is basically a python module, but how can you get the domain within it?
Basically in the settings.py file:
# If in local dev
if "RDS_DB_NAME" not in os.environ:
# the setting for the package I am pointing to a local version
BOOTSTRAP5 = {
"css_url": {
### dynamically get domain here ###
# "href": os.path.join(LOCAL_DIR, "static/css/bootstrap.min.css"),
"href": "static/css/bootstrap.min.css",
}
You can't access the domain in settings.py. When you run ./manage.py runserver 0:8000 you are telling Django to listen on port 8000 of whatever localhost is. But you can't tell from this machine which requests are going to come to this machine. For instance, you could have DNS configured to send www.domain1.com and www.domain2.com to come to this machine which looks like localhost from your perspective. So you can't know "domain" until a requests comes in.
Your machine does however have a local machine name which you could figure out from settings.py. Different OSes do this differently but on Macs you can get it with scutil --get LocalHostName and on other Unixes you can cat /etc/hostname.
Chris Curvey is right that the canonical way to differentiate between environments is with different settings files. Most devs differentiate between local and production environments but you are proposing a third: local with no internet access.
There are 2 steps to making this work.
Detect when to use the "local with no internet" environment.
The du jour way (and easiest way, IMO) to do this is with an environment variable. There are lots of ways to do this. You could, for instance, set LOCAL_NO_INTERNET=True in the environment and then check that variable anytime you need to do something special in your code.
If you really need to detect this automatically you could:
Query for the local machine's name (machine name, not domain, as described above)
Check if the name matches your local dev machine (so that you don't do this thing in prod, etc.)
Check if you don't have internet (ping google.com or something and see if it comes back successfully or times out)
I do not recommend this approach; there are many edge cases. Plus you probably can determine for yourself if you would like to be in "local with no internet" environment and set the environment variable manually.
Serve cached files usually provided by CDN
I think the easiest way, although a little verbose, is to put an if statement in your templates to either target the CDN or your local version (note that you will have to pass LOCAL_NO_INTERNET in the context of the view):
{% if LOCAL_NO_INTERNET %}
<script src="{% static "cdncache/bootstrap.min.js" %}"></script>
{% else %}
<script src="https://cdn.jsdelivr.net/npm/bootstrap#5.1.3/dist/js/bootstrap.min.js" integrity="..." crossorigin="anonymous"></script>
{% endif %}
Or you could do something way more complicated here like a middleware to somehow replace references to CDNs.
I would suggest that instead of defining this new "local with no internet" environment you could change your local setup to always assume "local with no internet". The CDN should speed up requests in production but doesn't do you much good in local development compared with always using your own version of those files.
I've never found a way to find out details about the Django listener.
The canonical way to have different settings in different environments is to have different settings files, and then set the DJANGO_SETTINGS_MODULE environment variable to control which one is used.
My first question ever, so go easy. I'll give as much detail as possible.
My setup
django 1.10 (I know, I need to upgrade)
python 3.5.[something]
postgres DB
gunicorn
nginx
The problem and what I've tried
The problem is that I pulled a recent commit that was working fine locally and suddenly none of a model's images, which previously rendered fine, are working - I'm getting 404s for all of them. I've tried the following:
Checking out previous commits
Restarting gunicorn (sudo systemctl restart gunicorn)
Restarting nginx
Restarting postgresql
Using python manage.py shell to access objects and check that they're still associating with a URL to the file
My code works when run locally - none of the uploaded images/files are causing 404s. As with my production environment, the logos folder sits in the main directory of my project, rather than being appended to a media folder (which other answers and the django docs suggested would be the case - this could be a red herring though).
My browser console shows the 404s and that it's trying to fetch from <domain.com>/media/logos/<filename>, even though they're stored (I've checked the file structure) in <project_folder>/logos/<filename> (i.e. without the media folder), but this was never previously a problem. Again, locally this is also the case but is completely fine.
My code and stuff
In my models.py I have this field:
class Thing(models.Model):
...
logo = models.FileField(upload_to='logos/', blank=True)
...which I then render in my HTML file with:
<img class="..." src="{{ thing.logo.url }}>
...which, again, is how django docs says to do it (rather than hard-coding a URL). I read in the docs that the file is stored as part of the object in the database, so referring to a filepath with filename wouldn't necessarily work (or something similar), therefore that this is the best option.
As far as I can see my urls are set up fine:
urlpatterns = [
...
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
...as per this answer (and probably others).
In settings.py no filepaths or URLs are hard-coded: they're all relative to the os.path....
MEDIA_URL = '/media/' and MEDIA_ROOT is unset (default is '')
I can't think of any other information that would be helpful, but if there is then let me know. Please help! I currently have a live website with ugly alt-text :(
The static(...) only works in development. In production you should configure your server (e.g. Apache or Nginx) to serve the files. See the example for Apache and modwsgi. – Alasdair Jan 9 at 22:12
Thanks #Alasdair for your above comment - unfortunately I can't mark it as the answer, but it led to it.
I hadn't setup Nginx to handle media files - only static files. So my Nginx conf file now looks like this:
location /static/ {
root /home/<path>/<to>/<folder>;
}
location /media/ {
root /home/<path>/<to>/<folder>;
}
My Django app was then uploading files to a slightly unexpected location, so I had to copy those files across to the correct media file location and set that right.
Up till now, I have been using Pyramid to serve static assets from a folder inside the python package, as specified in their documentation:
config.add_static_view('static', 'myapp:static')
and loading them from the templates as follows:
<script type="text/javascript" src="{{ request.application_url }}/static/js/jquery-1.7.1.min.js"></script>
However, I noticed that Chrome spits out warnings like this:
Resource interpreted as Stylesheet but transferred with MIME type apache/2.2.14: "http://mydomain.com/static/js/jquery-1.7.1.min.js"
or
Resource interpreted as Stylesheet but transferred with MIME type text/plain: "http://mydomain.com/static/js/jquery-1.7.1.min.js"
This happens on hard refresh, and it seems that 3-4 random resources which are loaded in the <head> are served with the wrong Content-Type header (according to Pyramid docs, the header is determined by the file extension).
There is no pattern I was able to deduce as to how the wrong header is set. Sometimes, it is text/plain for javascript/CSS files, sometimes it is a path like /static/js/something.js (and this path is in no way related to the request URL), and sometimes it is the value of the Server header, as with apache/2.2.14 above.
This is a big problem since when CSS is returned with a bad Content-Type, it is not rendered, and this breaks the entire page. I have solved this by catching requests to /static with Apache, and using it to serve the static assets, while letting all other requests go through to Pyramid. I no longer see the bad MIME type warnings in Chrome. However, I was wondering if anyone has run into this issue, and whether it is a Pyramid bug, or whether I am doing something else wrong.
EDIT: I forgot to provide the specs of how I deploy my app. The production server runs Apache 2.2, and the app runs under mod_wsgi. The process I followed is almost verbatim described in this tutorial: http://docs.pylonsproject.org/projects/pyramid/en/1.0-branch/tutorials/modwsgi/index.html. IMPORTANT: the issue only occurs when running on Apache via mod_wsgi. When I run the app locally on waitress, the Content-Type headers are always correct.
I am writing a web-app in web.py (a rewrite/extension of mongs) which I want to function both as a standalone application, and as a sub-app that requests can be forwarded to. The issue that I am having is that when it is used as a sub-app, static files cannot easily be served from its own static directory. Since I intend to distribute this (and not require users to combine the files into their project's static directory) I want the directory structure to be:
app_that_is_using_mongs (not mine)
static (which holds the app's static files - also not mine)
mongs (my subapp)
main.py (the code for mongs)
view (holds templates)
static (the static folder for mongs)
main.py (the code for the app that is using mongs)
...so that the entire mongs directory is separated from whatever app is using it.
I have considered a few possibilities for getting this to work:
Using a request handler that reads and outputs the files from the static directory, like:
cwd = os.path.dirname(__file__) + '/' # get current working directory
class Static:
def GET(self, filename):
"""searches for and returns a requested static file or 404s out"""
try:
return open(cwd + 'static/' + filename, 'r').read()
except:
web.application.notfound(app) # file not found
I am not sure about the performance of this solution for large files, and it seems like this should be something web.py can do on its own.
Adding another static directory by accessing the cherry.py staticdir tool through web.py... I'm not sure how to do something like this (interacting directly with the server that web.py is running on), and I don't think it would still work if I switched to a Gunicorn server (or any server but cherry.py).
Fixing the way that web.py handles static files to make it more extendable... If there is no other way, then rewriting this portion of web.py and maybe getting it pushed into the main repo is probably the best way.
So, what is the best way to do this?
In web.py the static assets aren't served through the application router. Instead the http server has a check weather the request url starts with /static. This means that weather you have a sub-application or not, the /static/... maps directly to the static directory in the root application.
Your first idea of building a static class will definitely work, but you are right that there is a definite performance implication - though, you'd have to benchmark it to really know how bad it is.
Another option, which is operationally worse, but is a temporary fix is to create a soft-link from the static directory of the parent app, into the sub-application's static directory. i.e.
parent_app/
static/
sub_app/ -> parent_app/sub_app/static/sub_app
...
sub_app/
static/
sub_app/
...
Then, when you want to access a static asset from sub_app, you would hit a url like: /static/sub_app/asset. Since this url starts with /static, it will get caught by the http server and redirect to the static directory, following the soft link, and resolves to the actual asset. Because of the sub_app directory, this solution will work when running the sub_app directly, or running the parent_app. You will have to setup this soft link on every server you deploy to, and for every development environment, which makes this less than ideal.
I can't, for the life of me, get Django to find my JavaScript files! I am trying to plug in a custom widget to the admin page, like so:
class MarkItUpWidget(forms.Textarea):
class Media:
js = (
'js/jquery.js',
'js/markitup/jquery.markitup.js',
'js/markitup/sets/markdown/set.js',
'js/markitup.init.js',
)
css = {
'screen': (
'js/markitup/skins/simple/style.css',
'js/markitup/sets/markdown/style.css',
)
}
However, when this code is inserted on my admin page, all of the links are broken, I can't get to any of my JavaScript files. My settings file looks like this: http://pastebin.com/qFVqUXyG
Is there something I am doing wrong? I'm somewhat new to Django.
I guess you're using django-admin runserver to test your website. In that case, have a look at "How to serve static files" (but don't ignore the big fat disclaimer).
Once you're ready to deploy, this chapter contains all the information (provided you go the standard route of Apache/mod_wsgi)
I don't know what the "django way" to include js files is, but I just use a simple regular include in the template, its great since you can dynamically generate the location / filename for these.. oh and if you are using django's test server I don't know how to get it to recognize these or if it even can but all you need is to run an apache server on your local machine and then put the files in the localhost directory and you can include them through localhost by including their full path in the template (i.e., http://localhost/myfiledirectory/myfile.js), also something I do is use amazon s3 to host files since you then get a url for them on there (not that you asked about this but its a quick way to host files if you don't have apache running locally)
Basically, my understanding of Django is that it works differently than a PHP framework, for example, as the files for Django don't sit (or don't need to at least) in the web path (i.e. they do not need to be accessible through the host path but can be anywhere on the machine) this is good for providing extra security, etc but it also means, that to give files a url to download them they need to be in the web hosting path, others may know how to make django do this directly but my quick and dirty method of putting them on the localhost path will work if thats all you're after.