I made a site for Heroku using Django and I got it to the point where it kept all the static images and files on Heroku just fine but the images the user uploads got deleted on dyno reset; that's why I wanted to use AWS to host the files the user uploads.
This is the code I am using in my settings:
AWS_ACCESS_KEY_ID = os.environ.get('my key')
AWS_SECRET_ACCESS_KEY = os.environ.get('my secret key')
AWS_STORAGE_BUCKET_NAME = 'my bucket name'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'
STATIC_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.eu-west-1.amazonaws.com/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
AWS_QUERYSTRING_AUTH = False
I added the "eu-west-1" part in the static URL because it was in the bucket's URL but not in my site's src for the image.
The problem is that now most of my JavaScript and CSS have disappeared as have all my static files that were previously just on Heroku and worked fine, furthermore the files that the user uploads also don't show up and the src doesn't containt the "eu-west-1" that I added (and it doesn't work without that part either).
Can somebody help me make it so that my static files are on Heroku as before while user uploaded files are on AWS?
Related
We have 2 servers. One of them is for media files and the other one is for django project server(ngnx+gunicorne).
Our media server is local(internal). We want to access to media server from inside the project with sftp storage package which includes paramiko. we don't want to access media server via URL (http,https).
HttpResponse(file, content_type=type) can display the image file as a big picture but we want to pass the image file to django template for display in html file like <img src="{{images}}" alt="">
We know HttpResponse is not a good solution but we use it below code for explained our problem.
# view
def coursesPageView(request):
courses = Course.objects.filter(is_published=True)
image_data =[imageRespone(data) for data in courses]
data = {
'published_courses_list':courses,
'images' : image_data
}
return render(request, 'pages/course2.html', data)
def imageRespone(valid_image):
if sfs.exists(valid_image.image.name):
file = sfs._read(valid_image.image.name)
type, encoding = mimetypes.guess_type(valid_image.image.name)
response = HttpResponse(file, content_type=type)
return response
else:
return HttpResponse('404 Not Found')
#course2.html
<img src="{{images}}" alt="">
What you can do is mounting a shared drive into your webserver pointing to your media server (Samba for linux). Then with Django you can specify the localisation of your statics file.
For example :
import os
MEDIA_URL = "/media/"
MEDIA_ROOT = os.path.abspath(os.path.join(os.sep, 'srv', 'media'))
STATIC_URL = '/app/static/'
STATIC_ROOT = os.path.abspath(os.path.join(os.sep, 'srv', 'static'))
Pay attention to the permissions between the web server and the mounted folder.
I'm trying to set up Django so that all static files are uploaded to s3, but for whatever reason, it's not working. Here is the relevant section in settings.py:
AWS_STORAGE_BUCKET_NAME = "bucket_name"
AWS_ACCESS_KEY_ID = os.environ.get("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.environ.get("AWS_SECRET_ACCESS_KEY")
AWS_S3_FILE_OVERWRITE = False
AWS_S3_REGION_NAME = "us-east-2"
AWS_S3_CUSTOM_DOMAIN = (
f"{AWS_STORAGE_BUCKET_NAME}.s3.{AWS_S3_REGION_NAME}.amazonaws.com"
)
AWS_S3_ENDPOINT_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}"
AWS_LOCATION = "static"
STATIC_URL = f"{AWS_S3_ENDPOINT_URL}/{AWS_LOCATION}/"
STATICFILES_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
Supposing that my bucket is called bucket_name, this is what will happen:
In my bucket_name bucket, there will be a folder called "bucket_name" with a static folder inside which contains all of my files.
On the server, none of the assets will load. This is because they are looking for the url https://bucket_name.s3.us-east-2.amazonaws.com/static whereas it's written in https://bucket_name.s3.us-east-2.amazonaws.com/bucket_name/static.
How do I either get the django assets to use this new address, or change the address on aws so that it aligns with django?
I've done some things to debug this.
It seems that if I redefine AWS_STORAGE_BUCKET_NAME after all of these lines, all that will change is the folder name will change from bucket_name to whatever else. Except I don't want to rename this folder, I want to either remove it entirely or get django to understand this folder.
Changing STATICFILES_STORAGE doesn't seem to do anything, unless it gets removed, which will cause an error
Changing the AWS_LOCATION just changes the folder to bucket_name/<new_thing>/static, it has no effect on the base folder. This will make the assets use <new_thing>/static, but the bucket_name still isn't there.
Temporarily removing my bucket policy doesn't fix any problems.
I imagine my problem is similar or the same as the one here: why static files and images are not working on my django s3 bucket project?. But it did not get a solution.
My problem was that I was using both AWS_S3_CUSTOM_DOMAIN and AWS_S3_ENDPOINT_URL. I had to pick just one of them.
When AWS_S3_CUSTOM_DOMAIN is specified and not AWS_S3_ENDPOINT_URL , S3 will contain a folder according to AWS_LOCATION, so in my case, the static files are stored in https://bucket_name.s3.us-east-2.amazonaws.com/static. Django will access this URL and the assets will be found.
When AWS_S3_ENDPOINT_URL is specified and not AWS_S3_CUSTOM_DOMAIN, S3 will contain a folder with the bucket name, which contains a folder according to AWS_LOCATION. So I can access it with https://bucket_name.s3.us-east-2.amazonaws.com/bucket_name/static. Django again will be able to access this URL as would be expected.
The problem arises when I have both at the same time. The domain for AWS_S3_CUSTOM_DOMAIN will be used on django and AWS_S3_ENDPOINT_URL will be used on S3, resulting in the disconnect that I was having.
I'm using django-oscar,and wanted to serve my static files with AWS S3.
To config my s3 bucket I've created a module called aws with conf.py and utils.py files.
On my website when I upload an image to the product it gets uploaded well with the correct path to my aws s3 bucket, but then after very short time the path changes from https://mybucketname.s3.amazonaws.com/media/cache/..../image.jpg to https://mybucketname.s3.amazonaws.com/cache/..../image.jpg
The images are in the media folder in my bucket.
I'm hosting my web app on heroku, the static files are served correctly but the issue happen in media folder.
Here is my code -
utils.py file
from storages.backends.s3boto3 import S3Boto3Storage
StaticRootS3BotoStorage = lambda: S3Boto3Storage(location='static')
MediaRootS3BotoStorage = lambda: S3Boto3Storage(location='media')
as static and media are the folders on my s3 bucket
conf.py
import datetime
AWS_ACCESS_KEY_ID = "xxx"
AWS_SECRET_ACCESS_KEY = "yyy"
AWS_PRELOAD_METADATA = True
AWS_QUERYSTRING_AUTH = False
AWS_DEFAULT_ACL = None
DEFAULT_FILE_STORAGE =
'myproject.aws.utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE =
'myproject.aws.utils.StaticRootS3BotoStorage'
AWS_STORAGE_BUCKET_NAME = 'mybucket-name'
S3DIRECT_REGION = 'us-east-2'
S3_URL = '//%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = '//%s.s3.amazonaws.com/media/' % AWS_STORAGE_BUCKET_NAME
MEDIA_ROOT = MEDIA_URL
STATIC_URL = S3_URL + 'static/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
two_months = datetime.timedelta(days=61)
date_two_months_later = datetime.date.today() + two_months
expires = date_two_months_later.strftime("%A, %d %B %Y 20:00:00
GMT")
AWS_HEADERS = {
'Expires': expires,
'Cache-Control': 'max-age=%d' %
(int(two_months.total_seconds()), ),
}
and my settings.py I added this
from myproject.aws.conf import *
What should I do to resolve this issue?
The file storage system configured for your Django app should be a class that implements django.core.files.storage.Storage [1]
storages.backends.s3boto3.S3Boto3Storage already implements this storage interface. [2]
Setting StaticRootS3BotoStorage in utils.py to a lambda, the Storage system is instantiated lazily with the proper location value; but the location attribute in the storage class itself is never changes. [3]
location = setting('AWS_LOCATION', '')
Django clears properties of storage instance when the project settings changes. [4] So that when the location attribute is resolved on the storage system, it effectively looks up the class attribute one (location value is shown in above snippet) because location attribute is missing in the instance.
This situation can be solved by subclassing storages.backends.s3boto3.S3Boto3Storage instead. This guarantees that location value never changes regardless of changes to project settings.
class StaticRootS3BotoStorage(S3Boto3Storage):
location = 'static'
class MediaRootS3BotoStorage(S3Boto3Storage):
location = 'media'
I am trying to use django-storages with s3boto in my app and trying to serve media and static files from s3.
I have the following settings in my settings file:
AWS_STORAGE_BUCKET_NAME = '<bucket_name>'
AWS_S3_ACCESS_KEY_ID = '<access_key>'
AWS_S3_SECRET_ACCESS_KEY = '<secret>'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = '<custom_storage_satic>'
MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = '<custom_storage_media>'
And my custom_storages.py is
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
When I create an image in django, instead of getting the relative path to my image starting with
image.url
'/media/image/<rest_of_the_path>.jpg'
I am getting the absolute url, which is something like
image.url
'https://<s3_bucket_name>.s3.amazonaws.com/media/image/original/'
When I use local storage instead of s3boto, it works as expected and gives me the relative path. Am I missing something here?
I struck the same issue when attempting to use the Imgix CDN for my S3 media (I suspect we're both using the same tutorial based on your use of the custom_storages.py override).
Here is an abridged version of the S3BotoStorage class in the django-storages framework. This excerpt highlights the important properties and methods for this issue, namely the custom-domain property.
class S3BotoStorage(Storage):
location = setting('AWS_LOCATION', '')
custom_domain = setting('AWS_S3_CUSTOM_DOMAIN')
def url(self, name, headers=None, response_headers=None, expire=None):
# Preserve the trailing slash after normalizing the path.
name = self._normalize_name(self._clean_name(name))
if self.custom_domain:
return "%s//%s/%s" % (self.url_protocol, self.custom_domain, filepath_to_uri(name))
As you can see in the url method, a URL is generated to override the STATIC_URL and MEDIA_URL Django settings. Currently the domain of the URL is created with the AWS_S3_CUSTOM_DOMAIN setting, which is why you continue to see the static S3 URL for media files.
So first, in your Django settings file, add a setting describing your CDN's domain.
IMGIX_DOMAIN = 'example.imgix.net'
Then, similar to the override of the location property, add an override to the custom_domain property in your MediaStorage class.
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
custom_domain = settings.IMGIX_DOMAIN
Now the final URL to your media files should begin with your CDN's domain, followed by the relative path to your file on the S3 bucket.
If you are serving static media from an S3 bucket, you must use an absolute URL, since the media is being served from a wholly different server.
I'm developing a Django website and I have a media folder where users can upload some stuff. This folder is in my root folder and when I run the server in local (with the ./manage.py runserver) it works fine and put the files in MyApp/media/
The problem is that I have a production server Apache with the website running via mod_wsgi. The folder of my project is in /var/www/MyApp/ and it is creating my media folder in /var/www/media instead of /var/www/MyApp/media.
In my settings I have
STATIC_URL = 'static/'
MEDIA_URL = 'media/'
And the way I'm creating the path for my uploaded files is this:
def generate_path(self, filename):
url = "media/files/users/%s/%s" % (self.user.username, filename)
return url
Any idea of what in production it is changing the directory?
Thanks in advance
Configure your MEDIA_ROOT:
# Project root is intended to be used when building paths,
# e.g. ``os.path.join(PROJECT_ROOT, 'relative/path')``.
PROJECT_ROOT = os.path.abspath(os.path.dirname(__name__))
# Absolute path to the directory that will hold uploaded files.
#
# For more information on ``MEDIA_ROOT``, visit
# https://docs.djangoproject.com/en/1.8/ref/settings/#media-root
MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'uploads/')
Then, use upload_to to specify path relative to your MEDIA_ROOT.