Develop Reference

Static images disappeared after using AWS for Heroku site

I made a site for Heroku using Django and I got it to the point where it kept all the static images and files on Heroku just fine but the images the user uploads got deleted on dyno reset; that's why I wanted to use AWS to host the files the user uploads.
This is the code I am using in my settings:
AWS_ACCESS_KEY_ID = os.environ.get('my key')
AWS_SECRET_ACCESS_KEY = os.environ.get('my secret key')
AWS_STORAGE_BUCKET_NAME = 'my bucket name'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'
STATIC_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.eu-west-1.amazonaws.com/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
AWS_QUERYSTRING_AUTH = False
I added the "eu-west-1" part in the static URL because it was in the bucket's URL but not in my site's src for the image.
The problem is that now most of my JavaScript and CSS have disappeared as have all my static files that were previously just on Heroku and worked fine, furthermore the files that the user uploads also don't show up and the src doesn't containt the "eu-west-1" that I added (and it doesn't work without that part either).
Can somebody help me make it so that my static files are on Heroku as before while user uploaded files are on AWS?

'python manage.py collectstatic' is still collecting static to C drive instead of s3 bucket, am I missing something?

I am trying to use s3 bucket with django (I have done this like twice before) but this time, after installing boto3 and django-storages and assigning correct values to necessary variables in settings.py, python manage.py collectstatic is still collecting static files to a local directory on my computer instead of s3 bucket. Below is my settings.py...
settings.py
INSTALLED_APPS = [
...
"storages",
]
AWS_ACCESS_KEY_ID = "*****"
AWS_SECRET_ACCESS_KEY = "******"
AWS_STORAGE_BUCKET_NAME = "****"
AWS_S3_CUSTOM_DOMAIN = "%s.s3.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {"CacheControl": "max-age=86400"}
AWS_DEFAULT_ACL = None
AWS_LOCATION = 'static'
STATICFILES_DIRS = [
BASE_DIR / "build/static", #this is the correct path by the way!
]
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
terminal
(env) C:\Users\LENOVO\Desktop\project> python manage.py collectstatic
You have requested to collect static files at the destination
location as specified in your settings:
C:\Users\LENOVO\Desktop\project\staticfiles
This will overwrite existing files!
Are you sure you want to do this?
According to all tutorials and my expectation, collectstatic is supposed to be collecting my static files into my s3 bucket.. Am I missing something??
Thanks for your time!

I tend to have define my own subclass of S3Boto3Storage and use that
# ./storage_backends.py - lives in the same dir as the config file.
from storages.backends.s3boto3 import S3Boto3Storage
from django.conf import settings
class StaticRootS3BotoStorage(S3Boto3Storage):
location = settings.STATIC_DIRECTORY
Then
# config.py
STATIC_DIRECTORY = BASE_DIR + "/build/static" # You will need to change this to your path
STATICFILES_STORAGE = 'config.settings.storage_backends.StaticRootS3BotoStorage' # you also need to change this to the path to the file we created above.
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, STATIC_DIRECTORY)
Now you can override methods for debugging and see where the problem might be.

Django Storages using s3boto ignoring MEDIA_URL

I am trying to use django-storages with s3boto in my app and trying to serve media and static files from s3.
I have the following settings in my settings file:
AWS_STORAGE_BUCKET_NAME = '<bucket_name>'
AWS_S3_ACCESS_KEY_ID = '<access_key>'
AWS_S3_SECRET_ACCESS_KEY = '<secret>'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = '<custom_storage_satic>'
MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = '<custom_storage_media>'
And my custom_storages.py is
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
When I create an image in django, instead of getting the relative path to my image starting with
image.url
'/media/image/<rest_of_the_path>.jpg'
I am getting the absolute url, which is something like
image.url
'https://<s3_bucket_name>.s3.amazonaws.com/media/image/original/'
When I use local storage instead of s3boto, it works as expected and gives me the relative path. Am I missing something here?

I struck the same issue when attempting to use the Imgix CDN for my S3 media (I suspect we're both using the same tutorial based on your use of the custom_storages.py override).
Here is an abridged version of the S3BotoStorage class in the django-storages framework. This excerpt highlights the important properties and methods for this issue, namely the custom-domain property.
class S3BotoStorage(Storage):
location = setting('AWS_LOCATION', '')
custom_domain = setting('AWS_S3_CUSTOM_DOMAIN')
def url(self, name, headers=None, response_headers=None, expire=None):
# Preserve the trailing slash after normalizing the path.
name = self._normalize_name(self._clean_name(name))
if self.custom_domain:
return "%s//%s/%s" % (self.url_protocol, self.custom_domain, filepath_to_uri(name))
As you can see in the url method, a URL is generated to override the STATIC_URL and MEDIA_URL Django settings. Currently the domain of the URL is created with the AWS_S3_CUSTOM_DOMAIN setting, which is why you continue to see the static S3 URL for media files.
So first, in your Django settings file, add a setting describing your CDN's domain.
IMGIX_DOMAIN = 'example.imgix.net'
Then, similar to the override of the location property, add an override to the custom_domain property in your MediaStorage class.
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
custom_domain = settings.IMGIX_DOMAIN
Now the final URL to your media files should begin with your CDN's domain, followed by the relative path to your file on the S3 bucket.

If you are serving static media from an S3 bucket, you must use an absolute URL, since the media is being served from a wholly different server.

Upload Media from Heroku to Amazon S3

New to Heroku & Amazon S3, so bear with me. Uploaded my Django app onto Heroku, and having a problem with user media uploads. The model is below:
#models.py
class Movie(models.Model):
title = models.CharField(max_length = 500)
poster = models.ImageField(upload_to = 'storages.backends.s3boto')
pub_date = models.DateTimeField(auto_now_add = True)
author = models.ForeignKey(User)
The poster attribute is the one where the image is uploaded. I had it running fine locally, and now on Heroku there is an error. So I added 'storages.backends.s3boto', as numerous other posts have told me to. (not sure if right).
My Settings.py file looks like this right now, kind of a mess:
#settings.py
PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))
PROJECT_DIR = os.path.join(PROJECT_ROOT, '../qanda')
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = '****************'
AWS_SECRET_ACCESS_KEY = '************'
AWS_STORAGE_BUCKET_NAME = 'mrt-assets'
AWS_PRELOAD_METADATA = True
MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'qanda/media/movie_posters/)
MEDIA_URL = '/media'
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticfiles')
STATIC_URL = 'https://mrt-assets.s3.amazonaws.com/static/'
STATICFILES_DIRS = (os.path.join(PROJECT_DIR, 'static'),)
My bucket is called mrt-assets, and there are 2 folders in there static (css, js, images and media. I'm not too worried about the static files for now, as I've hardcoded the CSS/JS files into my HTML files*, but how do I get my user uploaded media (images of any kind) into mrt-assets/media?
*although if someone wanted to help with STATIC files too that would be great. But user uploaded media more urgent.
EDIT (per Yuji's comment):
Have tried a number of options, and none of them working. I've gone back and deleted a lot of changes, and this is now my Settings
#settings.py
PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))
MEDIA_ROOT = 'http://s3.amazonaws.com/mrt-assets/media/'
MEDIA_URL = '/media/'
STATIC_ROOT = 'http://s3.amazonaws.com/mrt-assets/static/'
STATIC_URL = '/static/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
TEMPLATE_DIRS = (os.path.join(PROJECT_ROOT, "templates"),)
#models.py
#same as before, but now have changed the poster directory
poster = models.ImageField().
Not really sure what to do, need to connect my Heroku app to S3 so user media uploads are saved there.
Have now changed my S3 Bucket to this
mrt-assets
static
css
js
images
media
(empty)

The trick of getting your media to upload into <bucket>/media and your static assets into <bucket>/static is to create two different default storage backends for the two asset types or to explicitly instantiate your model fields with a storage object taking a location parameter.
Instantiating model field with custom storage
from storages.backends.s3boto import S3BotoStorage
class Movie(models.Model):
title = models.CharField(max_length=500)
poster = models.ImageField(storage=S3BotoStorage(location='media'))
pub_date = models.DateTimeField(auto_now_add=True)
author = models.ForeignKey(User)
Giving S3BotoStorage a location will prefix all uploads with its path.
Creating custom storage backends for media and static assets
This is almost the same as explicitly defining a storage backend with
location, but instead we'll be using settings.MEDIA_ROOT and
settings.STATIC_ROOT to apply a path prefix globally.
# settings.py
STATIC_ROOT = '/static/'
MEDIA_ROOT = '/media/'
DEFAULT_FILE_STORAGE = 'app.storage.S3MediaStorage'
STATICFILES_STORAGE = 'app.storage.S3StaticStorage'
# app/storage.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class S3MediaStorage(S3BotoStorage):
def __init__(self, **kwargs):
kwargs['location'] = kwargs.get('location',
settings.MEDIA_ROOT.replace('/', ''))
super(S3MediaStorage, self).__init__(**kwargs)
class S3StaticStorage(S3BotoStorage):
def __init__(self, **kwargs):
kwargs['location'] = kwargs.get('location',
settings.STATIC_ROOT.replace('/', ''))
super(S3StaticStorage, self).__init__(**kwargs)
Refining it
You can refine the above code to take advantage of
Heroku config vars
to make it more portable:
# settings.py
import os
STATIC_ROOT = os.environ.get('STATIC_ROOT',
os.path.join(os.path.dirname(__file__), 'static'))
MEDIA_ROOT = os.environ.get('MEDIA_ROOT',
os.path.join(os.path.dirname(__file__), 'media'))
DEFAULT_FILE_STORAGE = os.environ.get('DEFAULT_FILE_STORAGE',
'django.core.files.storage.FileSystemStorage')
STATICFILES_STORAGE = os.environ.get('STATICFILES_STORAGE',
'django.contrib.staticfiles.storage.StaticFilesStorage')
Couple the above settings with a .env file and you can use the
default storage backends locally for development and testing and when
deploying on Heroku you'll automatically switch to
app.storage.S3MediaStorage and app.storage.S3StaticStorage respectively:
# .env
STATIC_ROOT=static
MEDIA_ROOT=media
DEFAULT_FILE_STORAGE=app.storage.S3MediaStorage
STATICFILES_STORAGE=app.storage.S3StaticStorage

How to set-up a Django project with django-storages and Amazon S3, but with different folders for static files and media files?

I'm configuring a Django project that were using the server filesystem for storing the apps static files (STATIC_ROOT) and user uploaded files (MEDIA_ROOT).
I need now to host all that content on Amazon's S3, so I have created a bucket for this. Using django-storages with the boto storage backend, I managed to upload collected statics to the S3 bucket:
MEDIA_ROOT = '/media/'
STATIC_ROOT = '/static/'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'KEY_ID...'
AWS_SECRET_ACCESS_KEY = 'ACCESS_KEY...'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
Then, I got a problem: the MEDIA_ROOT and STATIC_ROOT are not used within the bucket, so the bucket root contains both the static files and user uploaded paths.
So then I could set:
S3_URL = 'http://s3.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = S3_URL + STATIC_ROOT
MEDIA_URL = 'S3_URL + MEDIA_ROOT
And use those settings in the templates, but there is no distinction of static/media files when storing in S3 with django-storages.
How this can be done?
Thanks!

I think the following should work, and be simpler than Mandx's method, although it's very similar:
Create a s3utils.py file:
from storages.backends.s3boto import S3BotoStorage
StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')
Then in your settings.py:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3BotoStorage'
A different but related example (that I've actually tested) can be seen in the two example_ files here.

I'm currently using this code in a separated s3utils module:
from django.core.exceptions import SuspiciousOperation
from django.utils.encoding import force_unicode
from storages.backends.s3boto import S3BotoStorage
def safe_join(base, *paths):
"""
A version of django.utils._os.safe_join for S3 paths.
Joins one or more path components to the base path component intelligently.
Returns a normalized version of the final path.
The final path must be located inside of the base path component (otherwise
a ValueError is raised).
Paths outside the base path indicate a possible security sensitive operation.
"""
from urlparse import urljoin
base_path = force_unicode(base)
paths = map(lambda p: force_unicode(p), paths)
final_path = urljoin(base_path + ("/" if not base_path.endswith("/") else ""), *paths)
# Ensure final_path starts with base_path and that the next character after
# the final path is '/' (or nothing, in which case final_path must be
# equal to base_path).
base_path_len = len(base_path) - 1
if not final_path.startswith(base_path) \
or final_path[base_path_len:base_path_len + 1] not in ('', '/'):
raise ValueError('the joined path is located outside of the base path'
' component')
return final_path
class StaticRootS3BotoStorage(S3BotoStorage):
def __init__(self, *args, **kwargs):
super(StaticRootS3BotoStorage, self).__init__(*args, **kwargs)
self.location = kwargs.get('location', '')
self.location = 'static/' + self.location.lstrip('/')
def _normalize_name(self, name):
try:
return safe_join(self.location, name).lstrip('/')
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
class MediaRootS3BotoStorage(S3BotoStorage):
def __init__(self, *args, **kwargs):
super(MediaRootS3BotoStorage, self).__init__(*args, **kwargs)
self.location = kwargs.get('location', '')
self.location = 'media/' + self.location.lstrip('/')
def _normalize_name(self, name):
try:
return safe_join(self.location, name).lstrip('/')
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
Then, in my settings module:
DEFAULT_FILE_STORAGE = 'myproyect.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproyect.s3utils.StaticRootS3BotoStorage'
I got to redefine the _normalize_name() private method to use a "fixed" version of the safe_join() function, since the original code is giving me SuspiciousOperation exceptions for legal paths.
I'm posting this for consideration, if anyone can give a better answer or improve this one, it will be very welcome.

File: PROJECT_NAME/custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
File: PROJECT_NAME/settings.py
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
if not DEBUG:
STATICFILES_STORAGE = 'PROJECT_NAME.custom_storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'PROJECT_NAME.custom_storages.MediaStorage'
AWS_ACCESS_KEY_ID = 'KEY_XXXXXXX'
AWS_SECRET_ACCESS_KEY = 'SECRET_XXXXXXXXX'
AWS_STORAGE_BUCKET_NAME = 'BUCKET_NAME'
AWS_HEADERS = {'Cache-Control': 'max-age=86400',}
AWS_QUERYSTRING_AUTH = False
And run: python manage.py collectstatic

I think the answer is pretty simple and done by default. This is working for me on AWS Elastic Beanstalk with Django 1.6.5 and Boto 2.28.0:
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_KEY']
The AWS keys are passed in from the container config file and I have no STATIC_ROOT or STATIC_URL set at all. Also, no need for the s3utils.py file. These details are handled by the storage system automatically. The trick here is that I needed to reference this unknown path in my templates correctly and dynamically. For example:
<link rel="icon" href="{% static "img/favicon.ico" %}">
That is how I address my favicon which lives locally (pre-deployment) in ~/Projects/my_app/project/my_app/static/img/favicon.ico.
Of course I have a separate local_settings.py file for accessing this stuff locally in dev environment and it does have STATIC and MEDIA settings. I had to do a lot of experimenting and reading to find this solution and it works consistently with no errors.
I understand that you need the static and root separation and considering that you can only provide one bucket I would point out that this method takes all the folders in my local environment under ~/Projects/my_app/project/my_app/static/and creates a folder in the bucket root (ie: S3bucket/img/ as in the example above). So you do get separation of files. For example you could have a media folder in the static folder and access it via templating with this:
{% static "media/" %}
I hope this helps. I came here looking for the answer and pushed a bit harder to find a simpler solution than to extend the storage system. Instead, I read the documentation about the intended use of Boto and I found that a lot of what I needed was built-in by default. Cheers!

If you want to have subfolders even before media or static seperations, you can use AWS_LOCATION on top of bradenm answer.
Reference: https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#usage
AWS_STORAGE_BUCKET_NAME = 'bucket_name'
AWS_LOCATION = 'path1/path2/'

Bradenm's answer is outdated and doesn't work so I updated it in March 2021.
Updated One:
Create a s3utils.py in the same folder of "settings.py":
from storages.backends.s3boto3 import S3Boto3Storage
StaticRootS3Boto3Storage = lambda: S3Boto3Storage(location='static')
MediaRootS3Boto3Storage = lambda: S3Boto3Storage(location='media')
Then, add 2 lines of code to settings.py and change "myproject" to your folder name:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto3Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto3Storage'
The updated one has multiple "3s" as I emphasize below.
s3utils.py:
from storages.backends.s3boto"3" import S3Boto"3"Storage
StaticRootS3Boto"3"Storage = lambda: S3Boto"3"Storage(location='static')
MediaRootS3Boto"3"Storage = lambda: S3Boto"3"Storage(location='media')
settings.py:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto"3"Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto"3"Storage'
Check and compare with Bradenm's (outdated) answer.
"I respect Bradenm's answer."