Develop Reference

Django - Can't read media files

I'm using Django to create a website where you can upload an image on that website and check if the image contains Moire pattern. Here is the project structure:
In file settings.py, I specified the following directory for media files:
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
and in file views.py, I implemented the API that will receive the image and detect moire pattern like this:
from django.core.files.storage import default_storage
def index(request):
if request.method == 'POST':
f = request.FILES['sentFile']
response = {}
fname = 'pic.jpg'
fname2 = default_storage.save(fname, f)
file_url = default_storage.url(fname2)
image = Image.open(file_url)
pred_label = moire_detect(image)
response['label'] = pred_label
return render(request, 'frontpage.html', response)
else:
return render(request, 'frontpage.html')
However, when I try to open the image using Image module of Pillow, I got this error "No such file or directory: '/media/pic_O1TOyCK.jpg'".
I don't really understand what is happening here, because the path is correct. Any help would be really appreciated.

Image.open doesn't work with url but with normal path.
But file_url has /media/pic_O1TOyCK.jpg which is relative url and it can be used on HTML (to get http://localhost/media/pic_O1TOyCK.jpg) but it can't be used as normal path to access directly local file.
You should rather use fname2 with Image.open()

Django Media Files (Images) Security

I am working on image handling in Django. I am using normal image model storing method.
So, my model is something like,
class PictureModel(models.Model):
def user_directory_path(instance, filename):
return 'images/{0}/{1}'.format(instance.user.username, filename)
user = models.ForeignKey(User, on_delete=models.CASCADE)
image = models.ImageField(upload_to=user_directory_path)
The view.py is somewhat like,
#login_required(login_url='/accounts/login/')
def myMethod(request):
user = request.user
myImage = PictureModel.objects.get(user=request.user)
return render(request, 'main/myPage.html', {'myImage': myImage})
I have a media root setup as,
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'
The url.py setup is as follows,
url(r'^static/(?P<path>.*)$', django.views.static.serve, {'document_root': settings.STATIC_ROOT, 'show_indexes': settings.DEBUG})
Now in normal flow, the image uploading and fetching are working as expected. But the issue is regarding the user validation/authentication while fetching the media file.
Suppose, I am getting an image by the URL,
media/images/user_1/mypic.jpg
But the whole media folder gets exposed without any validations and I can access,
media/images/user_2/mypic.jpg
also through the browser. I have searched on this over the net and found there are some random third party libraries are available but they are not so standard/popular. Can anyone suggest the best practices and libraries to handle the situation.

Add this in your template
it solves the problem
<img style="height:200px; width:200px; border-radius:50%; margin:auto" src="{{ appname.function_name.image.url }}" alt="Photo">

How to to make a file private by securing the url that only authenticated users can see

I was wondering if there is a way to secure an image or a file to be hidden when it is not authenticated.
Suppose there is an image in my website which can only be seen if that user is authenticated. But the thing is I can copy the url or open the image in the new tab.
http://siteis.com/media/uploaded_files/1421499811_82_Chrysanthemum.jpg
And again, even if I am not authenticated, I can view that particular image by going to that url. So, my my problem is, how do I secure the files, so that only authenticated users will see?
Update:
view:
def pictures(request, user_id):
user = User.objects.get(id=user_id)
all = user.photo_set.all()
return render(request, 'pictures.html',{
'pictures': all
})
models:
def get_upload_file_name(instance, filename):
return "uploaded_files/%s_%s" %(str(time()).replace('.','_'), filename)
class Photo(models.Model):
photo_privacy = models.CharField(max_length=1,choices=PRIVACY, default='F')
user = models.ForeignKey(User)
image = models.ImageField(upload_to=get_upload_file_name)
settings:
if DEBUG:
MEDIA_URL = '/media/'
STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR), "myproject", "static", "static-only")
MEDIA_ROOT = os.path.join(os.path.dirname(BASE_DIR), "myproject", "static", "media")
STATICFILES_DIRS = (
os.path.join(os.path.dirname(BASE_DIR), "myproject", "static", "static"),
)
Update:
template:
{% if pictures %}
{% for photo in pictures %}
<img src="/media/{{ photo.image }}" width="300" alt="{{ photo.caption }}"/>
{% endfor %}
{% else %}
<p>You have no picture</p>
{% endif %}
url:
url(r'^(?P<user_name>[\w#%.]+)/photos/$', 'pictures.views.photos', name='photos'),
if settings.DEBUG:
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

By securing any media file not to serve by anonymous user, better way url protection.
Code ( Updated ):
from django.conf.urls import patterns, include, url
from django.contrib.auth.decorators import login_required
from django.views.static import serve
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist
from django.shortcuts import HttpResponse
#login_required
def protected_serve(request, path, document_root=None):
try:
obj = Photobox.objects.get(user=request.user.id)
obj_image_url = obj.image.url
correct_image_url = obj_image_url.replace("/media/", "")
if correct_image_url == path:
return serve(request, path, document_root)
except ObjectDoesNotExist:
return HttpResponse("Sorry you don't have permission to access this file")
url(r'^{}(?P<path>.*)$'.format(settings.MEDIA_URL[1:]), protected_serve, {'file_root': settings.MEDIA_ROOT}),
Note: previously any logged in user can access any page, now this update restrict non user to view other files...

It would be better to handle just the authentication, and let your webserver handle the serving of files. It's probably good to put them in a different directory than your settings.MEDIA_ROOT, to prevent your webserver from serving the files before you handle the request, e.g. project_root/web-private/media/.
import os
#login_required
def protected_file(request, path):
# set PRIVATE_MEDIA_ROOT to the root folder of your private media files
name = os.path.join(settings.PRIVATE_MEDIA_ROOT, path)
if not os.path.isfile(name):
raise Http404("File not found.")
# set PRIVATE_MEDIA_USE_XSENDFILE in your deployment-specific settings file
# should be false for development, true when your webserver supports xsendfile
if settings.PRIVATE_MEDIA_USE_XSENDFILE:
response = HttpResponse()
response['X-Accel-Redirect'] = filename # Nginx
response['X-Sendfile'] = filename # Apache 2 with mod-xsendfile
del response['Content-Type'] # let webserver regenerate this
return response
else:
# fallback method
from django.views.static import serve
return serve(request, path, settings.PRIVATE_MEDIA_ROOT)
As your webserver is way better at serving static files than Django, this will speed up your website. Check django.views.static.serve for an idea how to sanitize file names etc.

The easiest option is to serve the file from django, and then add the #login_required decorator to the view, like this:
import os
import mimetypes
from django.core.servers.basehttp import FileWrapper
from django.contrib.auth.decorators import login_required
#login_required
def sekret_view(request, path=None):
filename = os.path.basename(path)
response = HttpResponse(FileWrapper(open(path)),
content_type=mimetypes.guess_type(path)[0])
response['Content-Length'] = os.path.getsize(path)
return response

django static_root understanding - IOError - ??

i am trying to use some image in my views.py, i did this,
from django.conf import settings
image = settings.STATIC_ROOT + "images/test.png"
fp = open(image, 'rb')
but it is saying:
IOError at /
(2, 'No such file or directory')
in my settings.py, i have:
STATIC_ROOT = os.path.join(PROJECT_PATH, "static")
STATIC_URL = '/static/'
what do i miss here ? STATICFILES_DIRS is empty but in templates, i has been working till now, but now i want to get image in my views.py.

You could try:
from django.contrib.staticfiles.views import serve
serve(request, 'images/test.png')
If memory serves me correct tho this will only work if debug is true.
But it sounds like its not finding the path, to find te project path...
Print(normpath(join(dirname(__file__), '..')))
This should help you identify and correct the problem.

How to set-up a Django project with django-storages and Amazon S3, but with different folders for static files and media files?

I'm configuring a Django project that were using the server filesystem for storing the apps static files (STATIC_ROOT) and user uploaded files (MEDIA_ROOT).
I need now to host all that content on Amazon's S3, so I have created a bucket for this. Using django-storages with the boto storage backend, I managed to upload collected statics to the S3 bucket:
MEDIA_ROOT = '/media/'
STATIC_ROOT = '/static/'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'KEY_ID...'
AWS_SECRET_ACCESS_KEY = 'ACCESS_KEY...'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
Then, I got a problem: the MEDIA_ROOT and STATIC_ROOT are not used within the bucket, so the bucket root contains both the static files and user uploaded paths.
So then I could set:
S3_URL = 'http://s3.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = S3_URL + STATIC_ROOT
MEDIA_URL = 'S3_URL + MEDIA_ROOT
And use those settings in the templates, but there is no distinction of static/media files when storing in S3 with django-storages.
How this can be done?
Thanks!

I think the following should work, and be simpler than Mandx's method, although it's very similar:
Create a s3utils.py file:
from storages.backends.s3boto import S3BotoStorage
StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')
Then in your settings.py:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3BotoStorage'
A different but related example (that I've actually tested) can be seen in the two example_ files here.

I'm currently using this code in a separated s3utils module:
from django.core.exceptions import SuspiciousOperation
from django.utils.encoding import force_unicode
from storages.backends.s3boto import S3BotoStorage
def safe_join(base, *paths):
"""
A version of django.utils._os.safe_join for S3 paths.
Joins one or more path components to the base path component intelligently.
Returns a normalized version of the final path.
The final path must be located inside of the base path component (otherwise
a ValueError is raised).
Paths outside the base path indicate a possible security sensitive operation.
"""
from urlparse import urljoin
base_path = force_unicode(base)
paths = map(lambda p: force_unicode(p), paths)
final_path = urljoin(base_path + ("/" if not base_path.endswith("/") else ""), *paths)
# Ensure final_path starts with base_path and that the next character after
# the final path is '/' (or nothing, in which case final_path must be
# equal to base_path).
base_path_len = len(base_path) - 1
if not final_path.startswith(base_path) \
or final_path[base_path_len:base_path_len + 1] not in ('', '/'):
raise ValueError('the joined path is located outside of the base path'
' component')
return final_path
class StaticRootS3BotoStorage(S3BotoStorage):
def __init__(self, *args, **kwargs):
super(StaticRootS3BotoStorage, self).__init__(*args, **kwargs)
self.location = kwargs.get('location', '')
self.location = 'static/' + self.location.lstrip('/')
def _normalize_name(self, name):
try:
return safe_join(self.location, name).lstrip('/')
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
class MediaRootS3BotoStorage(S3BotoStorage):
def __init__(self, *args, **kwargs):
super(MediaRootS3BotoStorage, self).__init__(*args, **kwargs)
self.location = kwargs.get('location', '')
self.location = 'media/' + self.location.lstrip('/')
def _normalize_name(self, name):
try:
return safe_join(self.location, name).lstrip('/')
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
Then, in my settings module:
DEFAULT_FILE_STORAGE = 'myproyect.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproyect.s3utils.StaticRootS3BotoStorage'
I got to redefine the _normalize_name() private method to use a "fixed" version of the safe_join() function, since the original code is giving me SuspiciousOperation exceptions for legal paths.
I'm posting this for consideration, if anyone can give a better answer or improve this one, it will be very welcome.

File: PROJECT_NAME/custom_storages.py
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
File: PROJECT_NAME/settings.py
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
if not DEBUG:
STATICFILES_STORAGE = 'PROJECT_NAME.custom_storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'PROJECT_NAME.custom_storages.MediaStorage'
AWS_ACCESS_KEY_ID = 'KEY_XXXXXXX'
AWS_SECRET_ACCESS_KEY = 'SECRET_XXXXXXXXX'
AWS_STORAGE_BUCKET_NAME = 'BUCKET_NAME'
AWS_HEADERS = {'Cache-Control': 'max-age=86400',}
AWS_QUERYSTRING_AUTH = False
And run: python manage.py collectstatic

I think the answer is pretty simple and done by default. This is working for me on AWS Elastic Beanstalk with Django 1.6.5 and Boto 2.28.0:
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_KEY']
The AWS keys are passed in from the container config file and I have no STATIC_ROOT or STATIC_URL set at all. Also, no need for the s3utils.py file. These details are handled by the storage system automatically. The trick here is that I needed to reference this unknown path in my templates correctly and dynamically. For example:
<link rel="icon" href="{% static "img/favicon.ico" %}">
That is how I address my favicon which lives locally (pre-deployment) in ~/Projects/my_app/project/my_app/static/img/favicon.ico.
Of course I have a separate local_settings.py file for accessing this stuff locally in dev environment and it does have STATIC and MEDIA settings. I had to do a lot of experimenting and reading to find this solution and it works consistently with no errors.
I understand that you need the static and root separation and considering that you can only provide one bucket I would point out that this method takes all the folders in my local environment under ~/Projects/my_app/project/my_app/static/and creates a folder in the bucket root (ie: S3bucket/img/ as in the example above). So you do get separation of files. For example you could have a media folder in the static folder and access it via templating with this:
{% static "media/" %}
I hope this helps. I came here looking for the answer and pushed a bit harder to find a simpler solution than to extend the storage system. Instead, I read the documentation about the intended use of Boto and I found that a lot of what I needed was built-in by default. Cheers!

If you want to have subfolders even before media or static seperations, you can use AWS_LOCATION on top of bradenm answer.
Reference: https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#usage
AWS_STORAGE_BUCKET_NAME = 'bucket_name'
AWS_LOCATION = 'path1/path2/'

Bradenm's answer is outdated and doesn't work so I updated it in March 2021.
Updated One:
Create a s3utils.py in the same folder of "settings.py":
from storages.backends.s3boto3 import S3Boto3Storage
StaticRootS3Boto3Storage = lambda: S3Boto3Storage(location='static')
MediaRootS3Boto3Storage = lambda: S3Boto3Storage(location='media')
Then, add 2 lines of code to settings.py and change "myproject" to your folder name:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto3Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto3Storage'
The updated one has multiple "3s" as I emphasize below.
s3utils.py:
from storages.backends.s3boto"3" import S3Boto"3"Storage
StaticRootS3Boto"3"Storage = lambda: S3Boto"3"Storage(location='static')
MediaRootS3Boto"3"Storage = lambda: S3Boto"3"Storage(location='media')
settings.py:
DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto"3"Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto"3"Storage'
Check and compare with Bradenm's (outdated) answer.
"I respect Bradenm's answer."