I'm configuring my Django project to use S3 buckets to store static and media files, both for local and production settings.
My project tree is as follows:
src/
blog/
settings/
__init__
local.py
production.py
s3utils.py
[..]
[..]
My local.py:
access_key = "xx"
secret_key = "yy"
AWS_ACCESS_KEY_ID = access_key
AWS_SECRET_ACCESS_KEY = secret_key
AWS_STORAGE_BUCKET_NAME = 'zz'
STATICFILES_STORAGE = 'blog.s3utils.StaticRootS3BotoStorage'
DEFAULT_FILE_STORAGE = 'blog.s3utils.MediaRootS3BotoStorage'
S3DIRECT_REGION = 'us-west-2'
S3_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = 'http://%s.s3.amazonaws.com/media/' % AWS_STORAGE_BUCKET_NAME
MEDIA_ROOT = MEDIA_URL
STATIC_URL = S3_URL + "/static/"
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
import datetime
two_months = datetime.timedelta(days=61)
date_two_months_later = datetime.date.today() + two_months
expires = date_two_months_later.strftime("%A, %d %B %Y 20:00:00 GMT")
AWS_HEADERS = {
'Expires': expires,
'Cache-Control': 'max-age=%d' % (int(two_months.total_seconds()), ),
}
my s3utils.py:
from storages.backends.s3boto import S3BotoStorage
StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')
When I run:
python manage.py collectstatic
only (django) admin static files are copied to my s3 bucket.
I thought that the problem was that I misconfigured the permission of my IAM user, but actually I do have the permissions to copy the 'admin/' files to the s3 bucket.
Thank you for any help you can provide.
Add STATICFILES_DIRS to your settings.
STATICFILES_DIRS = [
"/path/to/your/static",
]
https://docs.djangoproject.com/en/1.10/ref/settings/#std:setting-STATICFILES_DIRS
Related
I am trying to use s3 bucket with django (I have done this like twice before) but this time, after installing boto3 and django-storages and assigning correct values to necessary variables in settings.py, python manage.py collectstatic is still collecting static files to a local directory on my computer instead of s3 bucket. Below is my settings.py...
settings.py
INSTALLED_APPS = [
...
"storages",
]
AWS_ACCESS_KEY_ID = "*****"
AWS_SECRET_ACCESS_KEY = "******"
AWS_STORAGE_BUCKET_NAME = "****"
AWS_S3_CUSTOM_DOMAIN = "%s.s3.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {"CacheControl": "max-age=86400"}
AWS_DEFAULT_ACL = None
AWS_LOCATION = 'static'
STATICFILES_DIRS = [
BASE_DIR / "build/static", #this is the correct path by the way!
]
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
terminal
(env) C:\Users\LENOVO\Desktop\project> python manage.py collectstatic
You have requested to collect static files at the destination
location as specified in your settings:
C:\Users\LENOVO\Desktop\project\staticfiles
This will overwrite existing files!
Are you sure you want to do this?
According to all tutorials and my expectation, collectstatic is supposed to be collecting my static files into my s3 bucket.. Am I missing something??
Thanks for your time!
I tend to have define my own subclass of S3Boto3Storage and use that
# ./storage_backends.py - lives in the same dir as the config file.
from storages.backends.s3boto3 import S3Boto3Storage
from django.conf import settings
class StaticRootS3BotoStorage(S3Boto3Storage):
location = settings.STATIC_DIRECTORY
Then
# config.py
STATIC_DIRECTORY = BASE_DIR + "/build/static" # You will need to change this to your path
STATICFILES_STORAGE = 'config.settings.storage_backends.StaticRootS3BotoStorage' # you also need to change this to the path to the file we created above.
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, STATIC_DIRECTORY)
Now you can override methods for debugging and see where the problem might be.
Intro: I have made my application in Django I am trying to get my static and media files hosted in aws s3. My Django project is on AWS Lambda and AWS Api gateway using Zappa. below is my settings.py
AWS_DEFAULT_ACL = None
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static'),
]
DEFAULT_FILE_STORAGE = 'aws_storage_classes.MediaStorage'
AWS_ACCESS_KEY_ID = os.getenv("ACCESS_KEY")
AWS_SECRET_ACCESS_KEY = os.getenv("ACCESS_SECRET_KEY")
AWS_STORAGE_BUCKET_NAME = os.getenv("AWS_STORAGE_BUCKET_NAME")
STATICFILES_STORAGE = 'aws_storage_classes.StaticStorage'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_S3_DOMAIN = "%s.s3.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
STATIC_URL = 'https://%s.static/' % AWS_S3_DOMAIN
MEDIA_URL = 'https://%s.media/' % AWS_S3_DOMAIN
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
I then created a file in my project folder the same as my manage.py called aws_storage_classes.py
Below are the contents of my file aws_storage_classes.py
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
location = 'static'
class MediaStorage(S3Boto3Storage):
location = 'media'
Below are my urls
if settings.DEBUG:
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
Also in my settings.py
DEBUG = False
When I do python manage.py collectstatic all the staticfiles are downloaded and I don't get any errors. But when go on admin page the static files are not uploaded. See images below
Below is the image of my S3 bucket
Below is what is inside the static folder
Static files not loading
<link rel="stylesheet" type="text/css" href="https://<bucketname>.s3.amazonaws.com/static/style.css?AWSAccessKeyId=AKIAJHJGHJGJGJGJJGJGJMHIFQ&Signature=yfS3%2BvA0q15aUxw7OBySuQWZfjg%3D&x-amz-security-token=FQoGZXIvYXdgljdaksfgdjsgfkjertert;hre;thtires=151232173">
Updated settings.py
AWS_DEFAULT_ACL = None
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static'),
]
DEFAULT_FILE_STORAGE = 'aws_storage_classes.MediaStorage'
AWS_ACCESS_KEY_ID = os.getenv("ACCESS_KEY")
AWS_SECRET_ACCESS_KEY = os.getenv("ACCESS_SECRET_KEY")
AWS_STORAGE_BUCKET_NAME = os.getenv("AWS_STORAGE_BUCKET_NAME")
STATICFILES_STORAGE = 'aws_storage_classes.StaticStorage'
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_S3_DOMAIN = "%s.s3.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
STATIC_URL = 'https://%s.static/' % AWS_S3_DOMAIN
MEDIA_URL = 'https://%s.media/' % AWS_S3_DOMAIN
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
Below is the link after adding ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
<link rel="stylesheet" type="text/css" href="https://django-static-media.s3.amazonaws.com/static/style.css?AWSAccessKeyId=AKIAGGGGGGGGGMHIFQ&Signature=6gFQTsOSDFSDFA%3D&x-amz-security-token=FQoGZvwbJhd9amp;Expires=1549168642">
The issue is your url should be /static/admin/style.css/.... to fix this add the following to your settings.py file.
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
I tried uploading my static and media files onto AWS S3 bucket but, the css is not being rendered.
settings.py
AWS_HEADERS = { # see http://developer.yahoo.com/performance /rules.html#expires
'Expires': 'Thu, 31 Dec 2099 20:00:00 GMT',
'Cache-Control': 'max-age=94608000',
}
AWS_STORAGE_BUCKET_NAME = '*'
AWS_ACCESS_KEY_ID = '*'
AWS_SECRET_ACCESS_KEY = '*'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = "https://%s/" % AWS_S3_CUSTOM_DOMAIN
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = 'custom_storages.StaticStorage'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
MEDIAFILES_LOCATION = 'media'
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'
And when I try to access the css file through its source I get the following message -
<Error><Code>AccessDenied</Code><Message>Access Denied</Message>
<RequestId>BD606AFA0AF488B7</RequestId>
<HostId>2aZOfc5qdORtn2VhfDQsJ2gZHPoYBeBV9ciAdOnk+wGuK7azaWngXwfv+rEy1XLMFOrOgs+qDpI= </HostId>
</Error>
Sorry was making a silly mistake. The root of the static folders was not specified in STATICFILES_DIRS so collectstatic command couldn't collect those files.
I have successfully set up my app to use S3 for storing all static and media files. However, I would like to upload to S3 (current operation), but serve from a cloudfront instance I have set up. I have tried adjusting settings to the cloudfront url but it does not work. How can I upload to S3 and serve from Cloudfront please?
Settings
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
DEFAULT_FILE_STORAGE = 'app.custom_storages.MediaStorage'
STATICFILES_STORAGE = 'app.custom_storages.StaticStorage'
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
STATIC_URL = "https://s3-eu-west-1.amazonaws.com/app/%s/" % (STATICFILES_LOCATION)
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
Your code is almost complete except you are not adding your cloudfront domain to STATIC_URL/MEDIA_URL and your custom storages.
In detail, you must first install the dependencies
pip install django-storages-redux boto
Add the required settings to your django settings file
INSTALLED_APPS = (
...
'storages',
...
)
AWS_STORAGE_BUCKET_NAME = 'mybucketname'
AWS_CLOUDFRONT_DOMAIN = 'xxxxxxxx.cloudfront.net'
AWS_ACCESS_KEY_ID = get_secret("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = get_secret("AWS_SECRET_ACCESS_KEY")
MEDIAFILES_LOCATION = 'media'
MEDIA_ROOT = '/%s/' % MEDIAFILES_LOCATION
MEDIA_URL = '//%s/%s/' % (AWS_CLOUDFRONT_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'app.custom_storages.MediaStorage'
STATICFILES_LOCATION = 'static'
STATIC_ROOT = '/%s/' % STATICFILES_LOCATION
STATIC_URL = '//%s/%s/' % (AWS_CLOUDFRONT_DOMAIN, STATICFILES_LOCATION)
STATICFILES_STORAGE = 'app.custom_storages.StaticStorage'
Your custom storages need some modification to present the cloudfront domain for the resources, instead of the S3 domain:
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
"""uploads to 'mybucket/static/', serves from 'cloudfront.net/static/'"""
location = settings.STATICFILES_LOCATION
def __init__(self, *args, **kwargs):
kwargs['custom_domain'] = settings.AWS_CLOUDFRONT_DOMAIN
super(StaticStorage, self).__init__(*args, **kwargs)
class MediaStorage(S3BotoStorage):
"""uploads to 'mybucket/media/', serves from 'cloudfront.net/media/'"""
location = settings.MEDIAFILES_LOCATION
def __init__(self, *args, **kwargs):
kwargs['custom_domain'] = settings.AWS_CLOUDFRONT_DOMAIN
super(MediaStorage, self).__init__(*args, **kwargs)
And that is all you need, assuming your bucket and cloudfront domain are correctly linked and the user's AWS_ACCESS_KEY has access permissions to your bucket. Additionally, based on your use case, you may wish to make your s3 bucket items read-only accessible by everyone.
I had a similar issue and just setting AWS_S3_CUSTOM_DOMAIN to the Cloudfront url in Django's settings.py worked for me. You can check the code here.
I have installed easy_thumbnails and am trying to deploy my solution on to S3. I'm using https://github.com/jamstooks/django-s3-folder-storage to separate my /media/ and /static/ folders, with media containing uploaded content.
My settings file works like this:
# static file config
DEFAULT_FILE_STORAGE = 's3_folder_storage.s3.DefaultStorage'
DEFAULT_S3_PATH = "media"
STATICFILES_STORAGE = 's3_folder_storage.s3.StaticStorage'
STATIC_S3_PATH = "static"
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
MEDIA_ROOT = '/media/'
MEDIA_URL = 'https://%s.s3.amazonaws.com/media/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = "/%s/" % STATIC_S3_PATH
STATIC_URL = '//%s.s3.amazonaws.com/static/' % AWS_STORAGE_BUCKET_NAME
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
AWS_PRELOAD_METADATA = True
CKEDITOR_UPLOAD_PATH = 'uploads'
AWS_DEFAULT_ACL = 'public-read'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
But I'm encountering this error:
TemplateSyntaxError at /
Couldn't get the thumbnail teams/alumni/images/thumbs/alumni.png: [Errno 30] Read-only file system: '/media'
After doing a lot of research I discovered http://gibuloto.com/blog/easy-thumbnails-with-amazon-s3/. This should resolve any issues one may be having in implementing easy_thumbnails using S3.