I've been running into a problem while trying to delete uploaded images.
The error is along these lines:
SuspiciousOperation: Attempted access to '/media/artists/12-stones/154339.jpg' denied.
After reading around it looks like the error is due to the fact that it's looking for the image in the wrong place (notice first slash, /media/ doesn't exist on the filesystem)
My MEDIA_ROOT and MEDIA_URL are:
MEDIA_ROOT = '/home/tsoporan/site/media/'
MEDIA_URL = "/media/
My models upload_to parameter is passed this function:
def get_artist_path(instance, filename):
return os.path.join('artists', slugify(instance.name), filename)
My questions are:
1) How can I fix this problem for future uploads?
2) Is it possible to fix my current images' paths without having to reupload?
Regards,
Titus
I got this error when I put a leading slash in the upload_to definition.
BAD
pic = models.ImageField(upload_to="/uploads/product_images/")
GOOD
pic = models.ImageField(upload_to="uploads/product_images/")
Well, a little grepping around in the code shows that there may be a deeper error message that got homogenized along the way.
in django/core/files/storage.py, line 210 (this is in 1.1.1) we have:
def path(self, name):
try:
path = safe_join(self.location, name)
except ValueError:
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
return smart_str(os.path.normpath(path))
So the error has to be coming out of safe_join().
In django/utils/_os.py, we have the following. Note the ValueError it throws on the third to last line:
===========================
def safe_join(base, *paths):
"""
Joins one or more path components to the base path component intelligently.
Returns a normalized, absolute version of the final path.
The final path must be located inside of the base path component (otherwise
a ValueError is raised).
"""
# We need to use normcase to ensure we don't false-negative on case
# insensitive operating systems (like Windows).
base = force_unicode(base)
paths = [force_unicode(p) for p in paths]
final_path = normcase(abspathu(join(base, *paths)))
base_path = normcase(abspathu(base))
base_path_len = len(base_path)
# Ensure final_path starts with base_path and that the next character after
# the final path is os.sep (or nothing, in which case final_path must be
# equal to base_path).
if not final_path.startswith(base_path) \
or final_path[base_path_len:base_path_len+1] not in ('', sep):
raise ValueError('the joined path is located outside of the base path'
' component')
return final_path
==================
Hmmm, "The joined path is located outside of the base path component". Now there are a couple of calls to abspathu() in there (which is defined just above this routine and is different for NT than for other OSes). abspathu() converts all non-absolute paths to absolute by tacking on os.cwdu(), the current working directory.
Question: By any chance do you have a symlink (symbolic link) to your media directory? In other words, it's not a direct child of the project directory? I don't know if this is a valid question, it just popped out of my head.
Question: What are the values of self.location and name that are being passed to safe_join()?
Wild-ass-guess: is self.location empty?
Another wild-ass-guess: did MEDIA_ROOT somehow get changed to /media/?
If you have your own copy of Django installed (it's not hard to do), trying putting some print statements in these routines and then run it as the development server. The print output will go to the console.
Update: Hmmm. You said "2) The values for self.location and name are: /home/tsoporan/site/media and /media/albums/anthem-for-the-underdog/30103635.jpg"
Does the following path make any sense?
"/home/tsoporan/site/media/media/albums/anthem-for-the-underdog"
Note the .../media/media/... in there.
Also, what OS is this? Django rev?
As a note for others this issue can be caused when you have a double '//' in the static file resource you are looking for.
{{ STATIC_URL }}/style.css # Causes the issue it should be
{{ STATIC_URL }}style.css
Ah figured it out, slightly embarrassing, but it turns out the error was higher up. I was plugging these images in by a script and while going over it again realized that my paths started with /media/.
Now I have about 4000 images with wrong paths ... is there a way to somehow amend the paths for all these images? Or will the need to be reuploaded?
Thanks everyone, apologies for my mistake.
You really should just ask a new question on this. Try the following:
Write a standalone django script that looks something like this:
from django.core.management import setup_environ
from mysite import settings
setup_environ(settings)
from django.db import transaction
from app.models import Album # or whatever your model name is
for a in Album.objects.all():
# Do something to cleanup the filename.
# NOTE! This will not move the files, just change the value in the field.
a.filename = re.sub(r'^/media', '', a.filename)
a.save()
transaction.commit_unless_managed() # flush all changes
Use SimpleUploadedFile if your temporary file is not part of the MEDIA_ROOT folder. This won't throw a SuspiciousOperation error:
upload_file = SimpleUploadedFile(name=basename(out_file), content=open(out_file, 'rb').read())
object = YourModel.objects.create(file=upload_file)
Use File if your temporary file is already a part of MEDIA_ROOT. This is useful if you want to link an existing Django file to an object.
object = YourModel.objects.create(file=File(open(file_path, 'rb')))
if you want use other location ，such as /data/images/myfile/ , you should set you MEDIA_ROOT to /data/images in django settings.py file .
I found out, by using dumb print statements, that some media files have /media prefixed in their url paths. While the default storage option handles that, there is a problem if you use S3BotoStorage from django-storages.
So I fixed it by overriding the _normalize_name (guided by the answer by #peter-rowell):
class MediaStorage(FixedUrlBotoStorage):
location = settings.MEDIAFILES_LOCATION
# Overriding function because some media files are stored with '/media' prefixed (which causes problems)
def _normalize_name(self, name):
if name.startswith('/media'):
name = name.lstrip('/media')
return super(MediaStorage, self)._normalize_name(name)
I fixed this in a very simple way, go to utils.py in this folder
djk\lib\site-packages\django\core\files\utils.py
(djk is the name of the virtualenv)
In the file just make line 7 and line 8 python comments and that's it, job's done.
I got this error too. Debugging through I found that the following exception is being raised.
SuspiciousOperation(u"Attempted access to '2015-03-19-08:29:51-2-f8945842891244629dfd0c0af4c72a9c.pdf' denied.",)
BTW, I am using django-storages (v1.1.8) to store my media files onto S3 (using S3boto backend). I am using django 1.7.6.
But if I switch to storing with file name with out colons (:) it seems to work. I haven't figured out yet what is the root cause. Just posting this in case this is helpful to somebody else. Apparently, django or django-storages does not like filenames with colons.
This bug is fixed in django 1.2.5. See http://docs.djangoproject.com/en/dev/releases/1.2.5/#filefield-no-longer-deletes-files
Related
Working with scientific data, specifically climate data, I am constantly hard-coding paths to data directories in my Python code. Even if I were to write the most extensible code in the world, the hard-coded file paths prevent it from ever being truly portable. I also feel like having information about the file system of your machine coded in your programs could be security issue.
What solutions are out there for handling the configuration of paths in Python to avoid having to code them out explicitly?
One of the solution rely on using configuration files.
You can store all your path in a json file like so :
{
"base_path" : "/home/bob/base_folder",
"low_temp_area_path" : "/home/bob/base/folder/low_temp"
}
and then in your python code, you could just do :
import json
with open("conf.json") as json_conf :
CONF = json.load(json_conf)
and then you can use your path (or any configuration variable you like) like so :
print "The base path is {}".format(CONF["base_path"])
First off its always good practise to add a main function to go with each class to test that class or functions in the file. Along with this you determine the current working directory. This becomes incredibly important when running python from a cron job or from a directory that is not the current working directory. No JSON files or environment variables are then needed and you will obtain interoperation across Mac, RHEL and Debian distributions.
This is how you do it, and it will work on windows also if you use '\' instead of '/' (if that is even necessary, in your case).
if "__main__" == __name__:
workingDirectory = os.path.realpath(sys.argv[0])
As you can see when you run your command, the working directory is calculated if you provide a full path or relative path, meaning it will work in a cron job automatically.
After that if you want to work with data that is stored in the current directory use:
fileName = os.path.join( workingDirectory, './sub-folder-of-current-directory/filename.csv' )
fp = open( fileName,'r')
or in the case of the above working directory (parallel to your project directory):
fileName = os.path.join( workingDirectory, '../folder-at-same-level-as-my-project/filename.csv' )
fp = open( fileName,'r')
I believe there are many ways around this, but here is what I would do:
Create a JSON config file with all the paths I need defined.
For even more portability, I'd have a default path where I look for this config file but also have a command line input to change it.
In my opinion passing arguments from command line would be best solution. You should take a look at argparse . This allows you to create nice way to handle arguments from the command line. for example:
myDataScript.py /home/userName/datasource1/
#view_config(route_name='home_page', renderer='templates/edit.pt')
def home_page(request):
if 'form.submitted' in request.params:
name= request.params['name']
body = request.params['body']
renderer_dict = dict(name=name,body=body)
new_comment = render('new_page.pt', renderer_dict, request=request)
with open('tutorial:templates/{name}.html','w') as file:
file.write(new_comment)
return HTTPFound(location=request.static_url('tutorial:pages/{pagename}.html',pagename=name))
return {}
Right now this is a view callable I have in my pyramid app that is for my apps home page. I am concerned about the line where file is created (with open...). I want the name of the file to be the same name defined by the request.params in the code above but I am not sure how to pass the variable (I doubt brackets are the right solution). I then want .html to be added to that name to make it a full file name. I am not sure what syntax to use in order to do this
Edit: I also would like advice on how to correctly do this on the return HTTPFound line. I would like it to redirect to that new file. Right now I have {pagename}.html but doubt that this is sufficient. I feel like the solution to this is the same as to the with open line but please correct me if Im wrong.
first off, i think you probably should NOT be doing whatever it is that you're trying to do.
second, to open the file...
name = request.params['name']
app_dir = SEE_BELOW
filename = "%(app_dir)s/templates/%(name)s" % { 'app_dir':app_dir , 'name':name }
filename = "%s/templates/%s" % ( app_dir , name )
with open(filename,'w') as file:
file.write(new_comment)
i'm going to note a few things:
app_dir - i forget how to get the actual pyramid app dir. i usually get spooked by this stuff, so only use specific subdirectories like such:
env.ini
templates_writable_dir = %(here)s/app/templates/writable/
then i can access it via:
request.registry.settings['templates_writable_dir']
note that i made a specific writable subfolder. i don't want the main stuff writable. i'll chmod/grp that writable folder so the user pyramid runs as can ed it. i won't allow that user to write to anything else.
"tutorial:templates/{name}.html" that is using the templated syntax, which only works in the templates. one of your pyramid plugins injects the renderer_dict into the template and renders it for you. you need to use normal python string formatting, as i showed above using two options.
More importantly...
based on your question, you're not just new to pyramid but to python too. i'd suggest doing a few quick python tutorials before jumping into Pyramid - or any other framework.
I don't think so that standard open understand tutorial:templtes and even {name}
use request.static_path there before use open
I chrooted directory using following commands:
os.chroot("/mydir")
How to return to directory to previous - before chrooting?
Maybe it is possible to unchroot directory?
SOLUTION:
Thanks to Phihag. I found a solution. Simple example:
import os
os.mkdir('/tmp/new_dir')
dir1 = os.open('.', os.O_RDONLY)
dir2 = os.open('/tmp/new_dir', os.O_RDONLY)
os.getcwd() # we are in 'tmp'
os.chroot('/tmp/new_dir') # chrooting 'new_dir' directory
os.fchdir(dir2)
os.getcwd() # we are in chrooted directory, but path is '/'. It's OK.
os.fchdir(dir1)
os.getcwd() # we came back to not chrooted 'tmp' directory
os.close(dir1)
os.close(dir2)
More info
If you haven't changed your current working directory, you can simply call
os.chroot('../..') # Add '../' as needed
Of course, this requires the CAP_SYS_CHROOT capability (usually only given to root).
If you have changed your working directory, you can still escape, but it's harder:
os.mkdir('tmp')
os.chroot('tmp')
os.chdir('../../') # Add '../' as needed
os.chroot('.')
If chroot changes the current working directory, you can get around that by opening the directory, and using fchdir to go back.
Of course, if you intend to go out of a chroot in the course of a normal program (i.e. not a demonstration or security exploit), you should rethink your program. First of all, do you really need to escape the chroot? Why can't you just copy the required info into it beforehand?
Also, consider using a second process that stays outside of the chroot and answers to the requests of the chrooted one.
I see that if we change the HOME (linux) or USERPROFILE (windows) environmental variable and run a python script, it returns the new value as the user home when I try
os.environ['HOME']
os.exp
Is there any way to find the real user home directory without relying on the environmental variable?
edit:
Here is a way to find userhome in windows by reading in the registry,
http://mail.python.org/pipermail/python-win32/2008-January/006677.html
edit:
One way to find windows home using pywin32,
from win32com.shell import shell,shellcon
home = shell.SHGetFolderPath(0, shellcon.CSIDL_PROFILE, None, 0)
I think os.path.expanduser(path) could be helpful.
On Unix and Windows, return the argument with an initial component of ~ or ~user replaced by that user‘s home directory.
On Unix, an initial ~ is replaced by the environment variable HOME if it is set; otherwise the current user’s home directory is looked up in the password directory through the built-in module pwd. An initial ~user is looked up directly in the password directory.
On Windows, HOME and USERPROFILE will be used if set, otherwise a combination of HOMEPATH and HOMEDRIVE will be used. An initial ~user is handled by stripping the last directory component from the created user path derived above.
If the expansion fails or if the path does not begin with a tilde, the path is returned unchanged.
So you could just do:
os.path.expanduser('~user')
from pathlib import Path
str(Path.home())
works in Python 3.5 and above. Path.home() returns a Path object providing an API I find very useful.
I think os.path.expanduser(path) is the best answer to your question, but there's an alternative that may be worth mentioning in the Unix world: the pwd package. e.g.
import os, pwd
pwd.getpwuid(os.getuid()).pw_dir
For windows;
import os
homepath = os.path.expanduser(os.getenv('USERPROFILE'))
will give you a handle to current user's home directory and
filepath = os.path.expanduser(os.getenv('USERPROFILE'))+'\\Documents\\myfile.txt'
will give you a handle to below file;
C:\Users\urUserName\Documents\myfile.txt
home_folder = os.getenv('HOME')
This should work on Windows and Mac OS too, works well on Linux.
Really, a change in environment variable indicates that the home must be changed. So every program/script should have the new home in context; also the consequences are up to the person who changed it.
I would still stick with
home = os.getenv('USERPROFILE') or os.getenv('HOME')
what exactly is required?
I realize that this is an old question that has been answered but I thought I would add my two cents. The accepted answer was not working for me. I needed to find the user directory and I wanted it to work with and without sudo. In Linux, my user directory is "/home/someuser" but my root directory is "/root/". However, on my Mac, the user directory is "/Users/someuser". Here is what I ended up doing:
_USERNAME = os.getenv("SUDO_USER") or os.getenv("USER")
_HOME = os.path.expanduser('~'+_USERNAME)
This worked both with and without sudo on Mac and Linux.
get (translated) user folder names on Linux:
from gi.repository import GLib
docs = GLib.get_user_special_dir(GLib.USER_DIRECTORY_DOCUMENTS)
desktop = GLib.get_user_special_dir(GLib.USER_DIRECTORY_DESKTOP)
pics = GLib.get_user_special_dir(GLib.USER_DIRECTORY_PICTURES)
videos = GLib.get_user_special_dir(GLib.USER_DIRECTORY_VIDEOS)
music = GLib.get_user_special_dir(GLib.USER_DIRECTORY_MUSIC)
downloads = GLib.get_user_special_dir(GLib.USER_DIRECTORY_DOWNLOAD)
public = GLib.get_user_special_dir(GLib.USER_DIRECTORY_PUBLIC_SHARE)
templates = GLib.get_user_special_dir(GLib.USER_DIRECTORY_TEMPLATES)
print(docs)
print(desktop)
print(pics)
print(videos)
print(music)
print(downloads)
print(public)
print(templates)
On Linux and other UNIXoids you can always take a peek in /etc/passwd. The home directory is the sixth colon-separated field in there. No idea on how to do better than the environment variable on Windows though. There'll be a system call for it, but if it's available from Python, ...
Simply moving the file to ~/.Trash/ will not work, as if the file os on an external drive, it will move the file to the main system drive..
Also, there are other conditions, like files on external drives get moved to /Volumes/.Trash/501/ (or whatever the current user's ID is)
Given a file or folder path, what is the correct way to determine the trash folder? I imagine the language is pretty irrelevant, but I intend to use Python
Based upon code from http://www.cocoadev.com/index.pl?MoveToTrash I have came up with the following:
def get_trash_path(input_file):
path, file = os.path.split(input_file)
if path.startswith("/Volumes/"):
# /Volumes/driveName/.Trashes/<uid>
s = path.split(os.path.sep)
# s[2] is drive name ([0] is empty, [1] is Volumes)
trash_path = os.path.join("/Volumes", s[2], ".Trashes", str(os.getuid()))
if not os.path.isdir(trash_path):
raise IOError("Volume appears to be a network drive (%s could not be found)" % (trash_path))
else:
trash_path = os.path.join(os.getenv("HOME"), ".Trash")
return trash_path
Fairly basic, and there's a few things that have to be done seperatly, particularly checking if the filename already exist in trash (to avoid overwriting) and the actual moving to trash, but it seems to cover most things (internal, external and network drives)
Update: I wanted to trash a file in a Python script, so I re-implemented Dave Dribin's solution in Python:
from AppKit import NSURL
from ScriptingBridge import SBApplication
def trashPath(path):
"""Trashes a path using the Finder, via OS X's Scripting Bridge.
"""
targetfile = NSURL.fileURLWithPath_(path)
finder = SBApplication.applicationWithBundleIdentifier_("com.apple.Finder")
items = finder.items().objectAtLocation_(targetfile)
items.delete()
Usage is simple:
trashPath("/tmp/examplefile")
Alternatively, if you're on OS X 10.5, you could use Scripting Bridge to delete files via the Finder. I've done this in Ruby code here via RubyCocoa. The the gist of it is:
url = NSURL.fileURLWithPath(path)
finder = SBApplication.applicationWithBundleIdentifier("com.apple.Finder")
item = finder.items.objectAtLocation(url)
item.delete
You could easily do something similar with PyObjC.
A better way is NSWorkspaceRecycleOperation, which is one of the operations you can use with -[NSWorkspace performFileOperation:source:destination:files:tag:]. The constant's name is another artifact of Cocoa's NeXT heritage; its function is to move the item to the Trash.
Since it's part of Cocoa, it should be available to both Python and Ruby.
In Python, without using the scripting bridge, you can do this:
from AppKit import NSWorkspace, NSWorkspaceRecycleOperation
source = "path holding files"
files = ["file1", "file2"]
ws = NSWorkspace.sharedWorkspace()
ws.performFileOperation_source_destination_files_tag_(NSWorkspaceRecycleOperation, source, "", files, None)
The File Manager API has a pair of functions called FSMoveObjectToTrashAsync and FSPathMoveObjectToTrashSync.
Not sure if that is exposed to Python or not.
Another one in ruby:
Appscript.app('Finder').items[MacTypes::Alias.path(path)].delete
You will need rb-appscript gem, you can read about it here