My first question ever, so go easy. I'll give as much detail as possible.
My setup
django 1.10 (I know, I need to upgrade)
python 3.5.[something]
postgres DB
gunicorn
nginx
The problem and what I've tried
The problem is that I pulled a recent commit that was working fine locally and suddenly none of a model's images, which previously rendered fine, are working - I'm getting 404s for all of them. I've tried the following:
Checking out previous commits
Restarting gunicorn (sudo systemctl restart gunicorn)
Restarting nginx
Restarting postgresql
Using python manage.py shell to access objects and check that they're still associating with a URL to the file
My code works when run locally - none of the uploaded images/files are causing 404s. As with my production environment, the logos folder sits in the main directory of my project, rather than being appended to a media folder (which other answers and the django docs suggested would be the case - this could be a red herring though).
My browser console shows the 404s and that it's trying to fetch from <domain.com>/media/logos/<filename>, even though they're stored (I've checked the file structure) in <project_folder>/logos/<filename> (i.e. without the media folder), but this was never previously a problem. Again, locally this is also the case but is completely fine.
My code and stuff
In my models.py I have this field:
class Thing(models.Model):
...
logo = models.FileField(upload_to='logos/', blank=True)
...which I then render in my HTML file with:
<img class="..." src="{{ thing.logo.url }}>
...which, again, is how django docs says to do it (rather than hard-coding a URL). I read in the docs that the file is stored as part of the object in the database, so referring to a filepath with filename wouldn't necessarily work (or something similar), therefore that this is the best option.
As far as I can see my urls are set up fine:
urlpatterns = [
...
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
...as per this answer (and probably others).
In settings.py no filepaths or URLs are hard-coded: they're all relative to the os.path....
MEDIA_URL = '/media/' and MEDIA_ROOT is unset (default is '')
I can't think of any other information that would be helpful, but if there is then let me know. Please help! I currently have a live website with ugly alt-text :(
The static(...) only works in development. In production you should configure your server (e.g. Apache or Nginx) to serve the files. See the example for Apache and modwsgi. – Alasdair Jan 9 at 22:12
Thanks #Alasdair for your above comment - unfortunately I can't mark it as the answer, but it led to it.
I hadn't setup Nginx to handle media files - only static files. So my Nginx conf file now looks like this:
location /static/ {
root /home/<path>/<to>/<folder>;
}
location /media/ {
root /home/<path>/<to>/<folder>;
}
My Django app was then uploading files to a slightly unexpected location, so I had to copy those files across to the correct media file location and set that right.
Related
How do you dynamically access the domain name URL in the settings.py file of Django? (ie "http://localhost:8000")
I am trying to overwrite a package CDN while the internet is unavailable during development, and want to point to the local file in the static files directory. While os.path.join(BASE_DIR, "path/to/local.file") should work, it is context-dependent as to which app/url (ie "http://localhost:8000/app/static/css/bootstrap.min.css
"), and not just the main domain with the static file location appended to the starting server with ./manage.py runserver 0:8000 (ie " http://localhost:8000/static/css/bootstrap.min.css").
Notes:
Because this is in the settings.py, I can't load any apps or reverse because of the error *** django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
I am not in a template, so I can't use the static url
statically defining it won't allow for different port loadings when starting via ./manage.py runserver 0:8000
settings.py is basically a python module, but how can you get the domain within it?
Basically in the settings.py file:
# If in local dev
if "RDS_DB_NAME" not in os.environ:
# the setting for the package I am pointing to a local version
BOOTSTRAP5 = {
"css_url": {
### dynamically get domain here ###
# "href": os.path.join(LOCAL_DIR, "static/css/bootstrap.min.css"),
"href": "static/css/bootstrap.min.css",
}
You can't access the domain in settings.py. When you run ./manage.py runserver 0:8000 you are telling Django to listen on port 8000 of whatever localhost is. But you can't tell from this machine which requests are going to come to this machine. For instance, you could have DNS configured to send www.domain1.com and www.domain2.com to come to this machine which looks like localhost from your perspective. So you can't know "domain" until a requests comes in.
Your machine does however have a local machine name which you could figure out from settings.py. Different OSes do this differently but on Macs you can get it with scutil --get LocalHostName and on other Unixes you can cat /etc/hostname.
Chris Curvey is right that the canonical way to differentiate between environments is with different settings files. Most devs differentiate between local and production environments but you are proposing a third: local with no internet access.
There are 2 steps to making this work.
Detect when to use the "local with no internet" environment.
The du jour way (and easiest way, IMO) to do this is with an environment variable. There are lots of ways to do this. You could, for instance, set LOCAL_NO_INTERNET=True in the environment and then check that variable anytime you need to do something special in your code.
If you really need to detect this automatically you could:
Query for the local machine's name (machine name, not domain, as described above)
Check if the name matches your local dev machine (so that you don't do this thing in prod, etc.)
Check if you don't have internet (ping google.com or something and see if it comes back successfully or times out)
I do not recommend this approach; there are many edge cases. Plus you probably can determine for yourself if you would like to be in "local with no internet" environment and set the environment variable manually.
Serve cached files usually provided by CDN
I think the easiest way, although a little verbose, is to put an if statement in your templates to either target the CDN or your local version (note that you will have to pass LOCAL_NO_INTERNET in the context of the view):
{% if LOCAL_NO_INTERNET %}
<script src="{% static "cdncache/bootstrap.min.js" %}"></script>
{% else %}
<script src="https://cdn.jsdelivr.net/npm/bootstrap#5.1.3/dist/js/bootstrap.min.js" integrity="..." crossorigin="anonymous"></script>
{% endif %}
Or you could do something way more complicated here like a middleware to somehow replace references to CDNs.
I would suggest that instead of defining this new "local with no internet" environment you could change your local setup to always assume "local with no internet". The CDN should speed up requests in production but doesn't do you much good in local development compared with always using your own version of those files.
I've never found a way to find out details about the Django listener.
The canonical way to have different settings in different environments is to have different settings files, and then set the DJANGO_SETTINGS_MODULE environment variable to control which one is used.
I'm using Django 1.8 and I want to add a parameter to my static files to cache bust.
This is what I'm doing right now, setting a manual parameter:
<link href="{% static 'css/openprescribing.css' %}?q=0.1.1" rel="stylesheet">
But I feel there must be a better way to update the parameter.
I guess it would be a bit neater to have a setting passed through the template (and that would save having to update it in multiple places).
But what would be really nice is if Django could update it automatically for me.
The notes on django-cachebuster suggest that it's now possible to do this automatically in staticfiles, but I can't find anything in the staticfiles docs about it.
Anyone know a way to do this?
Yes this can be done automatically with contrib.staticfiles. There are two additional provided storage classes which will rename files using a hash. These
are documented here: ManifestStaticFilesStorage and CachedStaticFilesStorage
From the docs:
A subclass of the StaticFilesStorage storage backend which stores the file names it handles by appending the MD5 hash of the file’s content to the filename. For example, the file css/styles.css would also be saved as css/styles.55e7cbb9ba48.css.
The purpose of this storage is to keep serving the old files in case some pages still refer to those files, e.g. because they are cached by you or a 3rd party proxy server. Additionally, it’s very helpful if you want to apply far future Expires headers to the deployed files to speed up the load time for subsequent page visits.
The main difference is
CachedStaticFilesStorage is a similar class like the ManifestStaticFilesStorage class but uses Django’s caching framework for storing the hashed names of processed files instead of a static manifest file called staticfiles.json. This is mostly useful for situations in which you don’t have access to the file system.
To enable them you need to change your STATICFILES_STORAGE setting is set to 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage' or 'django.contrib.staticfiles.storage.CachedStaticFilesStorage'. The file names are only changed when DEBUG=False as it would be in production.
I'm no expert in Caching, but I think letting nginx handle caching might be better than using Django. Django has a lot to handle, so you can let the lightweight static server do that nasty job.
I do not use cloudflare, but I use this line to cache my statics, however, immediately the file changes, Nginx propagates the most recent file (thats the same file):
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
expires 30d;
}
which is a snippet from this gist I'm currently using that conf in production, so I know it works.
One thing I'll point out is, make sure MemCached isn't working and connected to your django as a Caching Backend. I say this, because I have spent many hours in time past hitting my head against the wall, simply because MemCached was caching my page with every content in it for up to 10 minutes.
with this nginx location conf, Whenever I change my .css, or upload a new file (static), the new file immediately takes over, as long as I've python manage.py collectstatic'ed them into the appropriate directory
I stand to be corrected though, if that's not actually the part doing the trick.
Proof that above works with Cache-busting (as you call it)
I went into server
Deleted my static folder (nginx still running) sudo rm -rf static/
Accessed my site
No static loaded
Went back, and python manage.py collectstatic
Accessed my site again. Statics loaded
No browser hard refresh was used. No nginx reload|restart whatever was used.
Nginx is smart enough to cache your statics, but reload the static when the file is new and serve it.
If you use a CDN, cache busting becomes more complex. If you just want to bust the cache for a specific image, just create a simple tag that does the trick....
import time
from django import template
register = template.Library()
#register.simple_tag()
def cache_bust():
return int(time.time())
Then in your template just do something like this...
{% load cache_app %}
<img src="/captcha/?cache_bust={% cache_bust %}" class="captcha"/>
And you have cache busting the simple way.
I have a django web app on A2hosting, where I am using .htaccess files and passenger_wsgi.py. It was working fine the last time I touched it, but then someone who had a look at it later informed me that it was broken.
I created a test situation to find the problem and here's the gist of it.
When I do a GET (to www.geo4ce.com/quiz/test_weird/), it goes to a page with a simple form that just has one input and a submit and an action that has "/quiz/test_weird/" and method="post". When I submit the form, the server expects the "quiz" part of the url to be referring to a path on the file server, can't find it and then logs an error that it can't find it. But, then it checks the test_weird part of the url against my django urls.py file, finds a different view for that and displays it.
A scenario that almost works properly is with www.geo4ce.com/quiz/test_hacked/, that has the same set up, except the form has action = "/anythinghere/quiz/test_hacked/". In this case, the "anythinghere" part of the url gets an error logged, since it doesn't exist on the file server, and then the /quiz/test_hacked/ part of the url works normally to get back to the original web page.
Anyone have any idea how I might be able to fix or debug this?
[EDIT]
I don't think it's the .htaccess file that's the cause. It looks something like this.
PassengerEnabled On
PassengerAppRoot /path/to/app/folder/
# Prevent Apache from serving .htaccess files:
<FilesMatch "^\.htaccess">
Order allow,deny
Deny from all
</FilesMatch>
deny from xxx.yyy.zzz
Apparently the issue is caused by a certain version of Passenger (with RoR). I've been told to switch to using FCGI.
I'm trying to set a reasonable cache expiry for my JS files while in development. I have the standard setup, where HTML, CSS and JS are living under the static directory.
The docs do mention this, but for the life of me I cannot get this to work. I've tried both methods implied, first
class MyFlask(flask.Flask):
def get_send_file_max_age(self, name):
if name.lower().endswith('.js'):
return 60
return flask.Flask.get_send_file_max_age(self, name)
app = MyFlask(__name__)
and
app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 60
Both have had no effect, my JS files under /static are still coming back with the default cache timeout,
Cache-Control: public, max-age=43200
Any pointers appreciated.
You may want to look at webassets to manage the cache expiry. It works in both development and production environment.
I had this problem and couldn't find an answer online that worked for me.
Then I realised that my static files are not being served from Flask at all! Flask only generates my HTML. The static files are served directly by my web server (Apache in my case, yours might be Nginx or something else).
Here are the instructions for Apache.
First install the mod_expires module:
sudo a2enmod expires
Then add something like this to your .htaccess file:
ExpiresByType text/css "access plus 1 year"
ExpiresByType application/javascript "access plus 1 year"
ExpiresByType image/* "access plus 1 year"
More details on how to configure it in the Apache manual.
I can't, for the life of me, get Django to find my JavaScript files! I am trying to plug in a custom widget to the admin page, like so:
class MarkItUpWidget(forms.Textarea):
class Media:
js = (
'js/jquery.js',
'js/markitup/jquery.markitup.js',
'js/markitup/sets/markdown/set.js',
'js/markitup.init.js',
)
css = {
'screen': (
'js/markitup/skins/simple/style.css',
'js/markitup/sets/markdown/style.css',
)
}
However, when this code is inserted on my admin page, all of the links are broken, I can't get to any of my JavaScript files. My settings file looks like this: http://pastebin.com/qFVqUXyG
Is there something I am doing wrong? I'm somewhat new to Django.
I guess you're using django-admin runserver to test your website. In that case, have a look at "How to serve static files" (but don't ignore the big fat disclaimer).
Once you're ready to deploy, this chapter contains all the information (provided you go the standard route of Apache/mod_wsgi)
I don't know what the "django way" to include js files is, but I just use a simple regular include in the template, its great since you can dynamically generate the location / filename for these.. oh and if you are using django's test server I don't know how to get it to recognize these or if it even can but all you need is to run an apache server on your local machine and then put the files in the localhost directory and you can include them through localhost by including their full path in the template (i.e., http://localhost/myfiledirectory/myfile.js), also something I do is use amazon s3 to host files since you then get a url for them on there (not that you asked about this but its a quick way to host files if you don't have apache running locally)
Basically, my understanding of Django is that it works differently than a PHP framework, for example, as the files for Django don't sit (or don't need to at least) in the web path (i.e. they do not need to be accessible through the host path but can be anywhere on the machine) this is good for providing extra security, etc but it also means, that to give files a url to download them they need to be in the web hosting path, others may know how to make django do this directly but my quick and dirty method of putting them on the localhost path will work if thats all you're after.