Develop Reference

Uploading images in django

I have being using django for sometime but recently noticed this. Before now I thought images in django, by default, gets uploaded in the
path specified in STATIC_URL but I just saw that the bahaviour is diffrent in my app. I have this set up in settings.py:
class BlogPost(models.Model):
author = models.ForeignKey(settings.AUTH_USER_MODEL, null=True, on_delete=models.CASCADE)
category = models.CharField(max_length=50, choices=Categories.choices, default=Categories.medications)
title = models.CharField(max_length=50)
slug = models.SlugField()
image = ResizedImageField(upload_to='images/blog/', null=True, blank=True)
introduction = models.CharField(max_length=255)
body = models.TextField()
settings.py
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static',),
os.path.join(BASE_DIR, 'frontend/build/static'),
]
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
The static, staticfiles, and media are within the root directory. In my django app, if media directory isn't present the
images get uploaded in static directory. However, when both static and media directorie are present preference is given to the media
directory (the images get uploaded in media directory). Did I make a mistake somewhere?

STATIC_ROOT was used to store all kinds of files like JavaScript, CSS and media files. Bot now the MEDIA_ROOT is used explicit for images and other media. That's where all you medias should go, you can also create additional directories in MEDIA_ROOT ex. for each model you have.
More about MEDIA_ROOT here

Django storages backend url pattern error when using dropbox

I am trying to use dropbox as a backend storage solution for django using django-storages, i followed the documentation but i am getting this error:
'C:/media/post_pics/profile_pic.jpeg' did not match pattern '(/(.|[\r\n])*|id:.*)|(rev:[0-9a-f]{9,})|(ns:[0-9]+(/.*)?)'
This is my models.py:
class Post(models.Model):
title = models.CharField(max_length=64)
date = models.DateField(timezone.now())
picture = models.ImageField(upload_to='post_pics')
...and my settings.py:
STATIC_URL = '/static/'
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
# Dropbox media file storage
DEFAULT_FILE_STORAGE = 'storages.backends.dropbox.DropBoxStorage'
DROPBOX_OAUTH2_TOKEN = 'myToken'
DROPBOX_ROOT_PATH = '/media/'

It turns out it was just a problem with how windows path works, using a linux system solved it

Django + Heroku: Images Work Until Next Push [duplicate]

This question already has an answer here:
Deploying on Heroku - Images disappears after upload
(1 answer)
Closed 6 years ago.
I have a basic learning blog app that I am running into an issue with: a "blog post" with an image uploaded via the app's admin form works fine, until I push another version via heroku. Then where the image was displays the broken img icon, and the app cannot find the image even though the picture target when I look at the picture address is the same.
I feel like I'm missing something fundamentally important about something - hence why I can't seem to fix this by google. Does anyone know why I'm tripping up here, and am I even approaching this right?
I'm using Django + Gunicorn + dj-static to serve the static files.
Here is the model:
class Post(models.Model):
title = models.CharField(max_length=100)
content = models.TextField()
picture = models.ImageField(upload_to='blog/static/blog/images')
section = models.ForeignKey('Section')
posted = models.DateTimeField(auto_now_add=True)
posted_by = models.TextField()
tags = models.ManyToManyField('Tag')
def __str__(self):
return self.title
the view:
def detail(request, post_id):
post = Post.objects.get(id=post_id)
picture = post.picture
context = {'post': post, 'picture': picture}
return render(request, 'blog/detail.html', context)
snip containing the image html:
<hr>
<img src="{{ picture.url }}">
<hr>
settings.py dealing with static files:
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.10/howto/static-files/
STATIC_ROOT = 'staticfiles'
STATIC_URL = '/static/'
BOOTSTRAP3 = {'include_jquery': True}
LOGIN_URL = '/users/login'
MEDIA_ROOT = 'media'
MEDIA_URL = '/media/'
#Herokusettings
if os.getcwd() == '/app':
import dj_database_url
DATABASES = {
'default' : dj_database_url.config(default='postgres://localhost')
}
DEBUG = False
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
ALLOWED_HOSTS = ['*']
PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
# Static files (CSS, JavaScript, Images)
# https://docs.djangoprojectcom/en/1.9/howto/static-files/
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticfiles')
STATIC_URL = '/static/'
MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'media')
MEDIA_URL = '/media/'
# Extra places for collectstatic to find static files.
STATICFILES_DIRS = (
os.path.join(PROJECT_ROOT, 'static'),
)
and wsgi.py:
import os
from django.core.wsgi import get_wsgi_application
from dj_static import Cling, MediaCling
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "bss.settings")
application = Cling(get_wsgi_application())
application = Cling(MediaCling(get_wsgi_application()))
#application = get_wsgi_application()
Thank you for giving this a look, I feel like I'm missing something fundamental about how all this works.

The default upload location on Heroku is into temporary storage. This is because you will get a different web worker each time you deploy.
You need to use S3 or another location to store your files. Luckily this is well documented for Paperclip on Heroku.

Using Cloudfront with Django S3Boto

I have successfully set up my app to use S3 for storing all static and media files. However, I would like to upload to S3 (current operation), but serve from a cloudfront instance I have set up. I have tried adjusting settings to the cloudfront url but it does not work. How can I upload to S3 and serve from Cloudfront please?
Settings
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
DEFAULT_FILE_STORAGE = 'app.custom_storages.MediaStorage'
STATICFILES_STORAGE = 'app.custom_storages.StaticStorage'
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
STATIC_URL = "https://s3-eu-west-1.amazonaws.com/app/%s/" % (STATICFILES_LOCATION)
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION

Your code is almost complete except you are not adding your cloudfront domain to STATIC_URL/MEDIA_URL and your custom storages.
In detail, you must first install the dependencies
pip install django-storages-redux boto
Add the required settings to your django settings file
INSTALLED_APPS = (
...
'storages',
...
)
AWS_STORAGE_BUCKET_NAME = 'mybucketname'
AWS_CLOUDFRONT_DOMAIN = 'xxxxxxxx.cloudfront.net'
AWS_ACCESS_KEY_ID = get_secret("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = get_secret("AWS_SECRET_ACCESS_KEY")
MEDIAFILES_LOCATION = 'media'
MEDIA_ROOT = '/%s/' % MEDIAFILES_LOCATION
MEDIA_URL = '//%s/%s/' % (AWS_CLOUDFRONT_DOMAIN, MEDIAFILES_LOCATION)
DEFAULT_FILE_STORAGE = 'app.custom_storages.MediaStorage'
STATICFILES_LOCATION = 'static'
STATIC_ROOT = '/%s/' % STATICFILES_LOCATION
STATIC_URL = '//%s/%s/' % (AWS_CLOUDFRONT_DOMAIN, STATICFILES_LOCATION)
STATICFILES_STORAGE = 'app.custom_storages.StaticStorage'
Your custom storages need some modification to present the cloudfront domain for the resources, instead of the S3 domain:
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
"""uploads to 'mybucket/static/', serves from 'cloudfront.net/static/'"""
location = settings.STATICFILES_LOCATION
def __init__(self, *args, **kwargs):
kwargs['custom_domain'] = settings.AWS_CLOUDFRONT_DOMAIN
super(StaticStorage, self).__init__(*args, **kwargs)
class MediaStorage(S3BotoStorage):
"""uploads to 'mybucket/media/', serves from 'cloudfront.net/media/'"""
location = settings.MEDIAFILES_LOCATION
def __init__(self, *args, **kwargs):
kwargs['custom_domain'] = settings.AWS_CLOUDFRONT_DOMAIN
super(MediaStorage, self).__init__(*args, **kwargs)
And that is all you need, assuming your bucket and cloudfront domain are correctly linked and the user's AWS_ACCESS_KEY has access permissions to your bucket. Additionally, based on your use case, you may wish to make your s3 bucket items read-only accessible by everyone.

I had a similar issue and just setting AWS_S3_CUSTOM_DOMAIN to the Cloudfront url in Django's settings.py worked for me. You can check the code here.

How to set-up a Django project with django-storages and Amazon S3, but with different folders for static files and media files?

I'm configuring a Django project that were using the server filesystem for storing the apps static files (STATIC_ROOT) and user uploaded files (MEDIA_ROOT).
I need now to host all that content on Amazon's S3, so I have created a bucket for this. Using django-storages with the boto storage backend, I managed to upload collected statics to the S3 bucket:
MEDIA_ROOT = '/media/'
STATIC_ROOT = '/static/'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'KEY_ID...'
AWS_SECRET_ACCESS_KEY = 'ACCESS_KEY...'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
Then, I got a problem: the MEDIA_ROOT and STATIC_ROOT are not used within the bucket, so the bucket root contains both the static files and user uploaded paths.
So then I could set:
S3_URL = 'http://s3.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = S3_URL + STATIC_ROOT
MEDIA_URL = 'S3_URL + MEDIA_ROOT
And use those settings in the templates, but there is no distinction of static/media files when storing in S3 with django-storages.
How this can be done?
Thanks!

I think the following should work, and be simpler than Mandx's method, although it's very similar:
Create a s3utils.py file:
from storages.backends.s3boto import S3BotoStorage
StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')
Then in your settings.py:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3BotoStorage'
A different but related example (that I've actually tested) can be seen in the two example_ files here.

I'm currently using this code in a separated s3utils module:
from django.core.exceptions import SuspiciousOperation
from django.utils.encoding import force_unicode
from storages.backends.s3boto import S3BotoStorage
def safe_join(base, *paths):
"""
A version of django.utils._os.safe_join for S3 paths.
Joins one or more path components to the base path component intelligently.
Returns a normalized version of the final path.
The final path must be located inside of the base path component (otherwise
a ValueError is raised).
Paths outside the base path indicate a possible security sensitive operation.
"""
from urlparse import urljoin
base_path = force_unicode(base)
paths = map(lambda p: force_unicode(p), paths)
final_path = urljoin(base_path + ("/" if not base_path.endswith("/") else ""), *paths)
# Ensure final_path starts with base_path and that the next character after
# the final path is '/' (or nothing, in which case final_path must be
# equal to base_path).
base_path_len = len(base_path) - 1
if not final_path.startswith(base_path) \
or final_path[base_path_len:base_path_len + 1] not in ('', '/'):
raise ValueError('the joined path is located outside of the base path'
' component')
return final_path
class StaticRootS3BotoStorage(S3BotoStorage):
def __init__(self, *args, **kwargs):
super(StaticRootS3BotoStorage, self).__init__(*args, **kwargs)
self.location = kwargs.get('location', '')
self.location = 'static/' + self.location.lstrip('/')
def _normalize_name(self, name):
try:
return safe_join(self.location, name).lstrip('/')
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
class MediaRootS3BotoStorage(S3BotoStorage):
def __init__(self, *args, **kwargs):
super(MediaRootS3BotoStorage, self).__init__(*args, **kwargs)
self.location = kwargs.get('location', '')
self.location = 'media/' + self.location.lstrip('/')
def _normalize_name(self, name):
try:
return safe_join(self.location, name).lstrip('/')
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
Then, in my settings module:
DEFAULT_FILE_STORAGE = 'myproyect.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproyect.s3utils.StaticRootS3BotoStorage'
I got to redefine the _normalize_name() private method to use a "fixed" version of the safe_join() function, since the original code is giving me SuspiciousOperation exceptions for legal paths.
I'm posting this for consideration, if anyone can give a better answer or improve this one, it will be very welcome.

File: PROJECT_NAME/custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
File: PROJECT_NAME/settings.py
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
if not DEBUG:
STATICFILES_STORAGE = 'PROJECT_NAME.custom_storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'PROJECT_NAME.custom_storages.MediaStorage'
AWS_ACCESS_KEY_ID = 'KEY_XXXXXXX'
AWS_SECRET_ACCESS_KEY = 'SECRET_XXXXXXXXX'
AWS_STORAGE_BUCKET_NAME = 'BUCKET_NAME'
AWS_HEADERS = {'Cache-Control': 'max-age=86400',}
AWS_QUERYSTRING_AUTH = False
And run: python manage.py collectstatic

I think the answer is pretty simple and done by default. This is working for me on AWS Elastic Beanstalk with Django 1.6.5 and Boto 2.28.0:
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_KEY']
The AWS keys are passed in from the container config file and I have no STATIC_ROOT or STATIC_URL set at all. Also, no need for the s3utils.py file. These details are handled by the storage system automatically. The trick here is that I needed to reference this unknown path in my templates correctly and dynamically. For example:
<link rel="icon" href="{% static "img/favicon.ico" %}">
That is how I address my favicon which lives locally (pre-deployment) in ~/Projects/my_app/project/my_app/static/img/favicon.ico.
Of course I have a separate local_settings.py file for accessing this stuff locally in dev environment and it does have STATIC and MEDIA settings. I had to do a lot of experimenting and reading to find this solution and it works consistently with no errors.
I understand that you need the static and root separation and considering that you can only provide one bucket I would point out that this method takes all the folders in my local environment under ~/Projects/my_app/project/my_app/static/and creates a folder in the bucket root (ie: S3bucket/img/ as in the example above). So you do get separation of files. For example you could have a media folder in the static folder and access it via templating with this:
{% static "media/" %}
I hope this helps. I came here looking for the answer and pushed a bit harder to find a simpler solution than to extend the storage system. Instead, I read the documentation about the intended use of Boto and I found that a lot of what I needed was built-in by default. Cheers!

If you want to have subfolders even before media or static seperations, you can use AWS_LOCATION on top of bradenm answer.
Reference: https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#usage
AWS_STORAGE_BUCKET_NAME = 'bucket_name'
AWS_LOCATION = 'path1/path2/'

Bradenm's answer is outdated and doesn't work so I updated it in March 2021.
Updated One:
Create a s3utils.py in the same folder of "settings.py":
from storages.backends.s3boto3 import S3Boto3Storage
StaticRootS3Boto3Storage = lambda: S3Boto3Storage(location='static')
MediaRootS3Boto3Storage = lambda: S3Boto3Storage(location='media')
Then, add 2 lines of code to settings.py and change "myproject" to your folder name:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto3Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto3Storage'
The updated one has multiple "3s" as I emphasize below.
s3utils.py:
from storages.backends.s3boto"3" import S3Boto"3"Storage
StaticRootS3Boto"3"Storage = lambda: S3Boto"3"Storage(location='static')
MediaRootS3Boto"3"Storage = lambda: S3Boto"3"Storage(location='media')
settings.py:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto"3"Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto"3"Storage'
Check and compare with Bradenm's (outdated) answer.
"I respect Bradenm's answer."