I am trying to use django-storages with s3boto in my app and trying to serve media and static files from s3.
I have the following settings in my settings file:
AWS_STORAGE_BUCKET_NAME = '<bucket_name>'
AWS_S3_ACCESS_KEY_ID = '<access_key>'
AWS_S3_SECRET_ACCESS_KEY = '<secret>'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = '<custom_storage_satic>'
MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = '<custom_storage_media>'
And my custom_storages.py is
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
When I create an image in django, instead of getting the relative path to my image starting with
image.url
'/media/image/<rest_of_the_path>.jpg'
I am getting the absolute url, which is something like
image.url
'https://<s3_bucket_name>.s3.amazonaws.com/media/image/original/'
When I use local storage instead of s3boto, it works as expected and gives me the relative path. Am I missing something here?
I struck the same issue when attempting to use the Imgix CDN for my S3 media (I suspect we're both using the same tutorial based on your use of the custom_storages.py override).
Here is an abridged version of the S3BotoStorage class in the django-storages framework. This excerpt highlights the important properties and methods for this issue, namely the custom-domain property.
class S3BotoStorage(Storage):
location = setting('AWS_LOCATION', '')
custom_domain = setting('AWS_S3_CUSTOM_DOMAIN')
def url(self, name, headers=None, response_headers=None, expire=None):
# Preserve the trailing slash after normalizing the path.
name = self._normalize_name(self._clean_name(name))
if self.custom_domain:
return "%s//%s/%s" % (self.url_protocol, self.custom_domain, filepath_to_uri(name))
As you can see in the url method, a URL is generated to override the STATIC_URL and MEDIA_URL Django settings. Currently the domain of the URL is created with the AWS_S3_CUSTOM_DOMAIN setting, which is why you continue to see the static S3 URL for media files.
So first, in your Django settings file, add a setting describing your CDN's domain.
IMGIX_DOMAIN = 'example.imgix.net'
Then, similar to the override of the location property, add an override to the custom_domain property in your MediaStorage class.
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
custom_domain = settings.IMGIX_DOMAIN
Now the final URL to your media files should begin with your CDN's domain, followed by the relative path to your file on the S3 bucket.
If you are serving static media from an S3 bucket, you must use an absolute URL, since the media is being served from a wholly different server.
Related
I'm trying to set up Django so that all static files are uploaded to s3, but for whatever reason, it's not working. Here is the relevant section in settings.py:
AWS_STORAGE_BUCKET_NAME = "bucket_name"
AWS_ACCESS_KEY_ID = os.environ.get("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.environ.get("AWS_SECRET_ACCESS_KEY")
AWS_S3_FILE_OVERWRITE = False
AWS_S3_REGION_NAME = "us-east-2"
AWS_S3_CUSTOM_DOMAIN = (
f"{AWS_STORAGE_BUCKET_NAME}.s3.{AWS_S3_REGION_NAME}.amazonaws.com"
)
AWS_S3_ENDPOINT_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}"
AWS_LOCATION = "static"
STATIC_URL = f"{AWS_S3_ENDPOINT_URL}/{AWS_LOCATION}/"
STATICFILES_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
Supposing that my bucket is called bucket_name, this is what will happen:
In my bucket_name bucket, there will be a folder called "bucket_name" with a static folder inside which contains all of my files.
On the server, none of the assets will load. This is because they are looking for the url https://bucket_name.s3.us-east-2.amazonaws.com/static whereas it's written in https://bucket_name.s3.us-east-2.amazonaws.com/bucket_name/static.
How do I either get the django assets to use this new address, or change the address on aws so that it aligns with django?
I've done some things to debug this.
It seems that if I redefine AWS_STORAGE_BUCKET_NAME after all of these lines, all that will change is the folder name will change from bucket_name to whatever else. Except I don't want to rename this folder, I want to either remove it entirely or get django to understand this folder.
Changing STATICFILES_STORAGE doesn't seem to do anything, unless it gets removed, which will cause an error
Changing the AWS_LOCATION just changes the folder to bucket_name/<new_thing>/static, it has no effect on the base folder. This will make the assets use <new_thing>/static, but the bucket_name still isn't there.
Temporarily removing my bucket policy doesn't fix any problems.
I imagine my problem is similar or the same as the one here: why static files and images are not working on my django s3 bucket project?. But it did not get a solution.
My problem was that I was using both AWS_S3_CUSTOM_DOMAIN and AWS_S3_ENDPOINT_URL. I had to pick just one of them.
When AWS_S3_CUSTOM_DOMAIN is specified and not AWS_S3_ENDPOINT_URL , S3 will contain a folder according to AWS_LOCATION, so in my case, the static files are stored in https://bucket_name.s3.us-east-2.amazonaws.com/static. Django will access this URL and the assets will be found.
When AWS_S3_ENDPOINT_URL is specified and not AWS_S3_CUSTOM_DOMAIN, S3 will contain a folder with the bucket name, which contains a folder according to AWS_LOCATION. So I can access it with https://bucket_name.s3.us-east-2.amazonaws.com/bucket_name/static. Django again will be able to access this URL as would be expected.
The problem arises when I have both at the same time. The domain for AWS_S3_CUSTOM_DOMAIN will be used on django and AWS_S3_ENDPOINT_URL will be used on S3, resulting in the disconnect that I was having.
I'm using django-oscar,and wanted to serve my static files with AWS S3.
To config my s3 bucket I've created a module called aws with conf.py and utils.py files.
On my website when I upload an image to the product it gets uploaded well with the correct path to my aws s3 bucket, but then after very short time the path changes from https://mybucketname.s3.amazonaws.com/media/cache/..../image.jpg to https://mybucketname.s3.amazonaws.com/cache/..../image.jpg
The images are in the media folder in my bucket.
I'm hosting my web app on heroku, the static files are served correctly but the issue happen in media folder.
Here is my code -
utils.py file
from storages.backends.s3boto3 import S3Boto3Storage
StaticRootS3BotoStorage = lambda: S3Boto3Storage(location='static')
MediaRootS3BotoStorage = lambda: S3Boto3Storage(location='media')
as static and media are the folders on my s3 bucket
conf.py
import datetime
AWS_ACCESS_KEY_ID = "xxx"
AWS_SECRET_ACCESS_KEY = "yyy"
AWS_PRELOAD_METADATA = True
AWS_QUERYSTRING_AUTH = False
AWS_DEFAULT_ACL = None
DEFAULT_FILE_STORAGE =
'myproject.aws.utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE =
'myproject.aws.utils.StaticRootS3BotoStorage'
AWS_STORAGE_BUCKET_NAME = 'mybucket-name'
S3DIRECT_REGION = 'us-east-2'
S3_URL = '//%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = '//%s.s3.amazonaws.com/media/' % AWS_STORAGE_BUCKET_NAME
MEDIA_ROOT = MEDIA_URL
STATIC_URL = S3_URL + 'static/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
two_months = datetime.timedelta(days=61)
date_two_months_later = datetime.date.today() + two_months
expires = date_two_months_later.strftime("%A, %d %B %Y 20:00:00
GMT")
AWS_HEADERS = {
'Expires': expires,
'Cache-Control': 'max-age=%d' %
(int(two_months.total_seconds()), ),
}
and my settings.py I added this
from myproject.aws.conf import *
What should I do to resolve this issue?
The file storage system configured for your Django app should be a class that implements django.core.files.storage.Storage [1]
storages.backends.s3boto3.S3Boto3Storage already implements this storage interface. [2]
Setting StaticRootS3BotoStorage in utils.py to a lambda, the Storage system is instantiated lazily with the proper location value; but the location attribute in the storage class itself is never changes. [3]
location = setting('AWS_LOCATION', '')
Django clears properties of storage instance when the project settings changes. [4] So that when the location attribute is resolved on the storage system, it effectively looks up the class attribute one (location value is shown in above snippet) because location attribute is missing in the instance.
This situation can be solved by subclassing storages.backends.s3boto3.S3Boto3Storage instead. This guarantees that location value never changes regardless of changes to project settings.
class StaticRootS3BotoStorage(S3Boto3Storage):
location = 'static'
class MediaRootS3BotoStorage(S3Boto3Storage):
location = 'media'
I'm attempting to use http://django-storages.readthedocs.org/en/latest/backends/amazon-S3.html for serving both static files and uploaded media, but I'm not certain it's possible. Is there a documented way that I'm missing? Also, I would assume (hope) that you could configure a separate bucket for each, but I can't find any info on that.
Yes this is possible by configuring both DEFAULT_FILE_STORAGE and STATICFILES_STORAGE to use the S3 storage. However if you set
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
then these will both use the default settings for the S3 storage, meaning they will both use the AWS_STORAGE_BUCKET_NAME bucket. The way to work around this is to create a small subclass of S3BotoStorage in your project which uses a different setting for the bucket name.
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class S3StaticStorage(S3BotoStorage):
def __init__(self, *args, **kwargs):
kwargs['bucket'] = settings.AWS_STATIC_BUCKET_NAME
super(S3StaticStorage, self).__init__(*args, **kwargs)
You would then define the AWS_STATIC_BUCKET_NAME setting to be whatever you want for your static bucket and change AWS_STATIC_BUCKET_NAME to the path for this custom storage class.
STATICFILES_STORAGE = 'dotted.path.to.storage.S3StaticStorage'
If you wanted to change other settings such as AWS_QUERYSTRING_AUTH, AWS_S3_CUSTOM_DOMAIN, AWS_PRELOAD_METADATA, etc then you would change them in this subclass as well.
I'm writing a view that displays a set of images on a page.
This is the model
#models.py
class Movie(models.Model):
title = models.CharField(max_length = 500)
poster = models.ImageField(upload_to = 'qanda/static/movie_posters')
#index.html
<img src = "{{ STATIC_URL }}movie_posters/{{ movie.poster }}"></a>
When I runserver, the image doesn't appear. The URL the image is trying to load is
http://127.0.0.1:8000/static/movie_posters/qanda/static/movie_posters/image.jpg
When the URL it should be trying to load is
http://127.0.0.1:8000/static/movie_posters/image.jpg
My assumption is that since movie.poster is located at 'qanda/static/movie_posters', when I render it on HTML, it is loading the Static URL (127.0.0:8000/static) and then the location 'qanda/static/movie_posters' at the end. How do I make the image render correctly?
There are two pieces of how image url is calculated.
First in your settings.py you define MEDIA_ROOT. MEDIA_ROOT specifies an absolute folder on your computer where media will be stored. So for example for these settings:
MEDIA_ROOT = '/abs/path/to/media/'
and if you have a field
models.ImageField(upload_to='movie_posters')
then images will be stored at:
/abs/path/to/media/movie_posters/
/abs/path/to/media/movie_posters/poster.jpg <- example
This deals with where media is stored on your hard drive.
Second piece is how to calculate urls for these media files. For that you define MEDIA_URL in your settings.py. That essentially maps a URL to your MEDIA_ROOT location. So then if your MEDIA_URL is:
MEDIA_URL = 'http://localhost/media/'
Then if you want to access an image stored at movie_posters/poster.jpg which has an absolute path of /abs/path/to/media/movie_posters/poster.jpg, its URL should be http://localhost/media/movie_posters/poster.jpg. You can compute the URL by doing:
{{ MEDIA_URL }}{{ movie.poster }}
Please note that I am using MEDIA_URL instead of STATIC_URL. Those are not the same thing. Computing urls like that however is not very neat. Thats why Django's ImageField and FileField have an url attribute:
{{ movie.poster.url }}
Django will then compute the proper url depending on your MEDIA_URL setting.
Note:
For all of this to work, you have to have a separate media server running. Django does not serve any media files. In development it is only capable of serving static file (not same as media files). So in development one nice trick to serve media files is to use Python's simple web server. For that, open a new terminal (on Linux and Mac) or Command Prompt (on Windows) window/tab and navigate to your media folder. Then just execute the following command there:
python -m SimpleHTTPServer 8090
and make sure your setting is:
MEDIA_URL = 'http://localhost:8090/'
and then Python will serve your media files. That works nice for development.
If you want to serve your media just using the development server you can add this for the time being to your urls.py
urlpatterns = patterns( ...all your awesome urls...) + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
you can define a custom template tag which returns the basename of the URL like this:
from django import template
import os
register = template.Library()
#register.filter
def getBasename(myURL):
return os.path.basename(myURL)
this should go in your custom template tags (eg. customTemplateTags.py) file within your templatetags directory of your app.
Then you can use the filter in order to get only the image filename, not the entire URL.
{% load customTemplateTags %}
<img src = "{{ STATIC_URL }}movie_posters/{{ movie.poster.url|getBasename }}"></a>
I'm configuring a Django project that were using the server filesystem for storing the apps static files (STATIC_ROOT) and user uploaded files (MEDIA_ROOT).
I need now to host all that content on Amazon's S3, so I have created a bucket for this. Using django-storages with the boto storage backend, I managed to upload collected statics to the S3 bucket:
MEDIA_ROOT = '/media/'
STATIC_ROOT = '/static/'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'KEY_ID...'
AWS_SECRET_ACCESS_KEY = 'ACCESS_KEY...'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
Then, I got a problem: the MEDIA_ROOT and STATIC_ROOT are not used within the bucket, so the bucket root contains both the static files and user uploaded paths.
So then I could set:
S3_URL = 'http://s3.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = S3_URL + STATIC_ROOT
MEDIA_URL = 'S3_URL + MEDIA_ROOT
And use those settings in the templates, but there is no distinction of static/media files when storing in S3 with django-storages.
How this can be done?
Thanks!
I think the following should work, and be simpler than Mandx's method, although it's very similar:
Create a s3utils.py file:
from storages.backends.s3boto import S3BotoStorage
StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')
Then in your settings.py:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3BotoStorage'
A different but related example (that I've actually tested) can be seen in the two example_ files here.
I'm currently using this code in a separated s3utils module:
from django.core.exceptions import SuspiciousOperation
from django.utils.encoding import force_unicode
from storages.backends.s3boto import S3BotoStorage
def safe_join(base, *paths):
"""
A version of django.utils._os.safe_join for S3 paths.
Joins one or more path components to the base path component intelligently.
Returns a normalized version of the final path.
The final path must be located inside of the base path component (otherwise
a ValueError is raised).
Paths outside the base path indicate a possible security sensitive operation.
"""
from urlparse import urljoin
base_path = force_unicode(base)
paths = map(lambda p: force_unicode(p), paths)
final_path = urljoin(base_path + ("/" if not base_path.endswith("/") else ""), *paths)
# Ensure final_path starts with base_path and that the next character after
# the final path is '/' (or nothing, in which case final_path must be
# equal to base_path).
base_path_len = len(base_path) - 1
if not final_path.startswith(base_path) \
or final_path[base_path_len:base_path_len + 1] not in ('', '/'):
raise ValueError('the joined path is located outside of the base path'
' component')
return final_path
class StaticRootS3BotoStorage(S3BotoStorage):
def __init__(self, *args, **kwargs):
super(StaticRootS3BotoStorage, self).__init__(*args, **kwargs)
self.location = kwargs.get('location', '')
self.location = 'static/' + self.location.lstrip('/')
def _normalize_name(self, name):
try:
return safe_join(self.location, name).lstrip('/')
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
class MediaRootS3BotoStorage(S3BotoStorage):
def __init__(self, *args, **kwargs):
super(MediaRootS3BotoStorage, self).__init__(*args, **kwargs)
self.location = kwargs.get('location', '')
self.location = 'media/' + self.location.lstrip('/')
def _normalize_name(self, name):
try:
return safe_join(self.location, name).lstrip('/')
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
Then, in my settings module:
DEFAULT_FILE_STORAGE = 'myproyect.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproyect.s3utils.StaticRootS3BotoStorage'
I got to redefine the _normalize_name() private method to use a "fixed" version of the safe_join() function, since the original code is giving me SuspiciousOperation exceptions for legal paths.
I'm posting this for consideration, if anyone can give a better answer or improve this one, it will be very welcome.
File: PROJECT_NAME/custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
File: PROJECT_NAME/settings.py
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
if not DEBUG:
STATICFILES_STORAGE = 'PROJECT_NAME.custom_storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'PROJECT_NAME.custom_storages.MediaStorage'
AWS_ACCESS_KEY_ID = 'KEY_XXXXXXX'
AWS_SECRET_ACCESS_KEY = 'SECRET_XXXXXXXXX'
AWS_STORAGE_BUCKET_NAME = 'BUCKET_NAME'
AWS_HEADERS = {'Cache-Control': 'max-age=86400',}
AWS_QUERYSTRING_AUTH = False
And run: python manage.py collectstatic
I think the answer is pretty simple and done by default. This is working for me on AWS Elastic Beanstalk with Django 1.6.5 and Boto 2.28.0:
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_KEY']
The AWS keys are passed in from the container config file and I have no STATIC_ROOT or STATIC_URL set at all. Also, no need for the s3utils.py file. These details are handled by the storage system automatically. The trick here is that I needed to reference this unknown path in my templates correctly and dynamically. For example:
<link rel="icon" href="{% static "img/favicon.ico" %}">
That is how I address my favicon which lives locally (pre-deployment) in ~/Projects/my_app/project/my_app/static/img/favicon.ico.
Of course I have a separate local_settings.py file for accessing this stuff locally in dev environment and it does have STATIC and MEDIA settings. I had to do a lot of experimenting and reading to find this solution and it works consistently with no errors.
I understand that you need the static and root separation and considering that you can only provide one bucket I would point out that this method takes all the folders in my local environment under ~/Projects/my_app/project/my_app/static/and creates a folder in the bucket root (ie: S3bucket/img/ as in the example above). So you do get separation of files. For example you could have a media folder in the static folder and access it via templating with this:
{% static "media/" %}
I hope this helps. I came here looking for the answer and pushed a bit harder to find a simpler solution than to extend the storage system. Instead, I read the documentation about the intended use of Boto and I found that a lot of what I needed was built-in by default. Cheers!
If you want to have subfolders even before media or static seperations, you can use AWS_LOCATION on top of bradenm answer.
Reference: https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#usage
AWS_STORAGE_BUCKET_NAME = 'bucket_name'
AWS_LOCATION = 'path1/path2/'
Bradenm's answer is outdated and doesn't work so I updated it in March 2021.
Updated One:
Create a s3utils.py in the same folder of "settings.py":
from storages.backends.s3boto3 import S3Boto3Storage
StaticRootS3Boto3Storage = lambda: S3Boto3Storage(location='static')
MediaRootS3Boto3Storage = lambda: S3Boto3Storage(location='media')
Then, add 2 lines of code to settings.py and change "myproject" to your folder name:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto3Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto3Storage'
The updated one has multiple "3s" as I emphasize below.
s3utils.py:
from storages.backends.s3boto"3" import S3Boto"3"Storage
StaticRootS3Boto"3"Storage = lambda: S3Boto"3"Storage(location='static')
MediaRootS3Boto"3"Storage = lambda: S3Boto"3"Storage(location='media')
settings.py:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto"3"Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto"3"Storage'
Check and compare with Bradenm's (outdated) answer.
"I respect Bradenm's answer."