I want to do a SuspiciousFileOperation which django disallows by default.
I am writing a command (to run via manage.py importfiles) to import a given directory structure on the real file system in my self written filestorage in Django.
I think, this is my relevant code:
def _handle_directory(self, directory_path, directory):
for root, subFolders, files in os.walk(directory_path):
for filename in files:
self.cnt_files += 1
new_file = File(directory=directory, filename=filename, file=os.path.join(root, filename),
uploader=self.uploader)
new_file.save()
The backtrace is:
Traceback (most recent call last):
File ".\manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 399, in execute_from_command_line
utility.execute()
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 285, in execute
output = self.handle(*args, **options)
File "D:\Development\github\Palco\engine\filestorage\management\commands\importfiles.py", line 53, in handle
self._handle_directory(args[0], root)
File "D:\Development\github\Palco\engine\filestorage\management\commands\importfiles.py", line 63, in _handle_directory
new_file.save()
File "D:\Development\github\Palco\engine\filestorage\models.py", line 157, in save
self.sha512 = hashlib.sha512(self.file.read()).hexdigest()
File "C:\Python27\lib\site-packages\django\core\files\utils.py", line 16, in <lambda>
read = property(lambda self: self.file.read)
File "C:\Python27\lib\site-packages\django\db\models\fields\files.py", line 46, in _get_file
self._file = self.storage.open(self.name, 'rb')
File "C:\Python27\lib\site-packages\django\core\files\storage.py", line 33, in open
return self._open(name, mode)
File "C:\Python27\lib\site-packages\django\core\files\storage.py", line 160, in _open
return File(open(self.path(name), mode))
File "C:\Python27\lib\site-packages\django\core\files\storage.py", line 261, in path
raise SuspiciousFileOperation("Attempted access to '%s' denied." % name)
django.core.exceptions.SuspiciousFileOperation: Attempted access to 'D:\Temp\importme\readme.html' denied.
The full model can be found at GitHub. The full command is currently on gist.github.com available.
If you do not want to check the model: the attribute file of my File class is a FileField.
I assume, this problem happens, because I am just "linking" to the file found. But I need to copy it, huh? How can I copy the file into the file?
In Django, SuspiciousFileOperation can be avoid by read the file from external dir and make a tmp file within the project media then save in the appropriate file filed as below
import tempfile
file_name="file_name.pdf"
EXT_FILE_PATH = "/home/somepath/"
file_path = EXT_FILE_PATH + file_name
if exists(file_path):
#create a named temporary file within the project base , here in media
lf = tempfile.NamedTemporaryFile(dir='media')
f = open(file_path, 'rb')
lf.write(f.read())
#doc object with file FileField.
doc.file.save(file_name, File(lf), save=True)
lf.close()
I haven't faced similar problem but related issue. I have recently upgraded Django 1.8 to 1.11.
Now I am getting the following error if try to save a file in a model having FileField field:
SuspiciousFileOperation at /api/send_report/
The joined path (/vagrant/tmp/test_file.pdf) is located outside of the base path component (/vagrant/media)
My model where I want to save the file:
class Report(BaseModel):
file = models.FileField(max_length=200, upload_to=os.path.join(settings.REPORTS_URL, '%Y/week_%W/'))
type = models.CharField(max_length=20, verbose_name='Type', blank=False, default='', db_index=True)
I am trying following codes to save the file from tmp folder which is not located in MEDIA_ROOT:
from django.core.files import File
filepath = "/vagrant/tmp/test_file.pdf"
file = File(open(filepath, "rb"))
report_type = "My_report_type"
report = Report.objects.create(
file=file,
type=report_type,
)
What I have done to solve the issue:
import os
from django.core.files import File
filepath = "/vagrant/tmp/test_file.pdf"
file = File(open(filepath, "rb"))
file_name = os.path.basename(file.name)
report_type = "My_report_type"
report = Report.objects.create(
type=report_type,
)
report.file.save(file_name, file, save=True)
Hope it will help someone.
Analyzing this part of stacktrace:
File "C:\Python27\lib\site-packages\django\core\files\storage.py", line 261, in path
raise SuspiciousFileOperation("Attempted access to '%s' denied." % name)
leads to the standard Django FileSystemStorage. It expects files to be within your MEDIA_ROOT. Your files can be anywhere in the file system, therefore this problem occurs.
You should pass file-like object instead of a path to your File model. The easiest way to achieve that would be to use Django File class, which is a wrapper around python file-like objects. See File object documentation for more details.
Update:
Ok, I am suggesting here a route taken from the docs:
from django.core.files import File as FileWrapper
def _handle_directory(self, directory_path, directory):
for root, subFolders, files in os.walk(directory_path):
for filename in files:
self.cnt_files += 1
new_file = File(
directory=directory, filename=filename,
file=os.path.join(root, filename),
uploader=self.uploader)
with open(os.path.join(root, filename), 'r') as f:
file_wrapper = FileWrapper(f)
new_file = File(
directory=directory, filename=filename,
file=file_wrapper,
uploader=self.uploader)
new_file.save()
If it works it should copy the file to the location provided by your secure_storage callable.
Check if there is slash before your filepath
file_item = models.FileField(upload_to=content_file_name)
def content_file_name(username, filename):
return '/'.join(['content', username, filename])
Note here "content" not "/content". That was the problem for me.
Related
I have this code :
import gensim
filename = 'GoogleNews-vectors-negative300.bin'
model = gensim.models.KeyedVectors.load_word2vec_format(filename, binary=True)
and this is my folder organization thing :
image of my folder tree that shows that the .bin file is in the same directory as the file calling it, the file being ai_functions
But sadly I'm not sure why I'm having an error saying that it can't find it. Btw I checked, I am sure the file is not corrupted. Any thoughts?
Full traceback :
File "/Users/Ile-Maurice/Desktop/Flask/flaskapp/run.py", line 1, in <module>
from serv import app
File "/Users/Ile-Maurice/Desktop/Flask/flaskapp/serv/__init__.py", line 13, in <module>
from serv import routes
File "/Users/Ile-Maurice/Desktop/Flask/flaskapp/serv/routes.py", line 7, in <module>
from serv.ai_functions import checkplagiarism
File "/Users/Ile-Maurice/Desktop/Flask/flaskapp/serv/ai_functions.py", line 31, in <module>
model = gensim.models.KeyedVectors.load_word2vec_format(filename, binary=True)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gensim/models/keyedvectors.py", line 1629, in load_word2vec_format
return _load_word2vec_format(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gensim/models/keyedvectors.py", line 1955, in _load_word2vec_format
with utils.open(fname, 'rb') as fin:
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/smart_open/smart_open_lib.py", line 188, in open
fobj = _shortcut_open(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/smart_open/smart_open_lib.py", line 361, in _shortcut_open
return _builtin_open(local_path, mode, buffering=buffering, **open_kwargs)
FileNotFoundError: [Errno 2] No such file or directory: 'GoogleNews-vectors-negative300.bin'
The 'current working directory' that the Python process will consider active, and thus will use as the expected location for your plain relative filename GoogleNews-vectors-negative300.bin, will depend on how you launched Flask.
You could print out the directory to be sure – see some ways at How do you properly determine the current script directory? – but I suspect it may just be the /Users/Ile-Maurice/Desktop/Flask/flaskapp/ directory.
If so, you could relatively-reference your file with the path relative to the above directory...
serv/GoogleNews-vectors-negative300.bin
...or you could use a full 'absolute' path...
/Users/Ile-Maurice/Desktop/Flask/flaskapp/serv/GoogleNews-vectors-negative300.bin
...or you could move the file up to its parent directory, so that it is alonside your Flask run.py.
I have this problem, I'm trying to run the script to download Springers free books [https://towardsdatascience.com/springer-has-released-65-machine-learning-and-data-books-for-free-961f8181f189], but many things start to go wrong.
I solved some of the problems but now I'm stuck.
C:\Windows\system32>python C:\Users\loren\Desktop\springer_free_books-master\main.py
Traceback (most recent call last):
File "C:\Users\loren\Desktop\springer_free_books-master\main.py", line 42, in <module>
books.to_excel(table_path)
File "C:\Users\loren\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\core\generic.py", line 2175, in to_excel
formatter.write(
File "C:\Users\loren\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\io\formats\excel.py", line 738, in write
writer.save()
File "C:\Users\loren\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\io\excel\_openpyxl.py", line 43, in save
return self.book.save(self.path)
File "C:\Users\loren\AppData\Local\Programs\Python\Python38-32\lib\site-packages\openpyxl\workbook\workbook.py", line 392, in save
save_workbook(self, filename)
File "C:\Users\loren\AppData\Local\Programs\Python\Python38-32\lib\site-packages\openpyxl\writer\excel.py", line 291, in save_workbook
archive = ZipFile(filename, 'w', ZIP_DEFLATED, allowZip64=True)
File "C:\Users\loren\AppData\Local\Programs\Python\Python38-32\lib\zipfile.py", line 1251, in __init__
self.fp = io.open(file, filemode)
FileNotFoundError: [Errno 2] No such file or directory: 'downloads\\table_v4.xlsx'
This is part of the code, were table_path is introduced.
table_url = 'https://resource-cms.springernature.com/springer-cms/rest/v1/content/17858272/data/v4'
table = 'table_' + table_url.split('/')[-1] + '.xlsx'
table_path = os.path.join(folder, table)
if not os.path.exists(table_path):
books = pd.read_excel(table_url)
# Save table
books.to_excel(table_path)
else:
books = pd.read_excel(table_path, index_col=0, header=0)
Try to create the destination directory before calling .to_excel() to ensure a valid writable directory exists. Make sure the os module is imported:
import os # add to your imports
and replace
books.to_excel(table_path)
with
os.makedirs(folder, exist_ok=True)
books.to_excel(table_path)
def generate_w2vModel(decTokenFlawPath, w2vModelPath):
print("training...")
model = Word2Vec(sentences= DirofCorpus(decTokenFlawPath), size=30, alpha=0.01, window=5, min_count=0, max_vocab_size=None, sample=0.001, seed=1, workers=1, min_alpha=0.0001, sg=1, hs=0, negative=10, iter=5)
model.save(w2vModelPath)
def evaluate_w2vModel(w2vModelPath):
print("\nevaluating...")
model = Word2Vec.load(w2vModelPath)
for sign in ['(', '+', '-', '*', 'main']:
print(sign, ":")
print(model.most_similar_cosmul(positive=[sign], topn=10))
def main():
dec_tokenFlaw_path = ['./data/cdg_ddg/corpus/']
w2v_model_path = "./w2v_model/wordmodel3"
generate_w2vModel(dec_tokenFlaw_path, w2v_model_path)
evaluate_w2vModel(w2v_model_path)
print("success!")
this the python that i was running. This file is used to train word2vec model. The inputs are corpus files, and the output is the word2vec model. i got the following error :
Traceback (most recent call last):
File "create_w2vmodel.py", line 67, in <module>
main()
File "create_w2vmodel.py", line 62, in main
generate_w2vModel(dec_tokenFlaw_path, w2v_model_path)
File "create_w2vmodel.py", line 50, in generate_w2vModel
model.save(w2vModelPath)
File "/usr/local/lib/python2.7/dist-packages/gensim-3.4.0-py2.7-linux-x86_64.egg/gensim/models/word2vec.py", line 930, in save
super(Word2Vec, self).save(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gensim-3.4.0-py2.7-linux-x86_64.egg/gensim/models/base_any2vec.py", line 281, in save
super(BaseAny2VecModel, self).save(fname_or_handle, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gensim-3.4.0-py2.7-linux-x86_64.egg/gensim/utils.py", line 691, in save
self._smart_save(fname_or_handle, separately, sep_limit, ignore, pickle_protocol=pickle_protocol)
File "/usr/local/lib/python2.7/dist-packages/gensim-3.4.0-py2.7-linux-x86_64.egg/gensim/utils.py", line 550, in _smart_save
pickle(self, fname, protocol=pickle_protocol)
File "/usr/local/lib/python2.7/dist-packages/gensim-3.4.0-py2.7-linux-x86_64.egg/gensim/utils.py", line 1311, in pickle
with smart_open(fname, 'wb') as fout: # 'b' for binary, needed on Windows
File "build/bdist.linux-x86_64/egg/smart_open/smart_open_lib.py", line 89, in smart_open
File "build/bdist.linux-x86_64/egg/smart_open/smart_open_lib.py", line 301, in file_smart_open
IOError: [Errno 21] Is a directory: './w2v_model/wordmodel3'
please help me to change this particular error. i think there is no folder like this, but i had already created the w2v_model/wordmodel3 in my folder. I had tried it in many ways. I will provide smart_open_lib.py program file below :
def file_smart_open(fname, mode='rb'):
"""
Stream from/to local filesystem, transparently (de)compressing gzip and bz2
files if necessary.
"""
_, ext = os.path.splitext(fname)
if ext == '.bz2':
PY2 = sys.version_info[0] == 2
if PY2:
from bz2file import BZ2File
else:
from bz2 import BZ2File
return make_closing(BZ2File)(fname, mode)
if ext == '.gz':
from gzip import GzipFile
return make_closing(GzipFile)(fname, mode)
return open(fname, mode)
this is the trace back code which they tell. kindly request to help me fr change this error!!!
The Word2Vec .save() method needs a new path, to the filename (not directory!) to which you want to save the model's main file. (For most models of significant size, there will be extra files alongside that main file, with other extensions, that are also part of the model-save.)
If you're providing a path to an existing directory, you'll get an error like this.
Change your w2v_model_path to be a path to a desired file name, not an already-existing directory. (Perhaps that path will be inside that directory. But it shouldn't be the directory itself!)
Through the metadata of one folder, I am able to get relative file paths of the files I want to download to my local machine. When I give this path to source code of do_get(), it gives me permission denied error. Here is the code which is supposed to download files and decrypt them but its not able to download the files on the first hand.
#command
def do_decryptFiles(self, from_path, to_path, key):
"""
Decrypt all the files given in the folder and subfolders of from_path
Examples:
Dropbox> decryptFiles '/Photos' 'E:\temp' 'a13223132323232'
"""
folder_metadata = self.api_client.metadata(from_path)
print "metadata:", folder_metadata
for s in folder_metadata['contents']:
if(s['is_dir'] == True):
print "directory:", s['path']
else:
FFPath = s['path']
print FFPath
do_get(self, from_path, to_path)
to_file = open(os.path.abspath(to_path), "wb")
f, metadata = self.api_client.get_file_and_metadata(self.current_path + FFPath)
to_file.write(f.read())
When it calls open(), command line gives me Permission Denied error. Any help would be appreciated.
Traceback (most recent call last):
File "example/cli_client.py", line 397, in <module>
main()
File "example/cli_client.py", line 394, in main
term.cmdloop()
File "C:\Python27\lib\cmd.py", line 142, in cmdloop
stop = self.onecmd(line)
File "C:\Python27\lib\cmd.py", line 219, in onecmd
return func(arg)
File "example/cli_client.py", line 77, in wrapper
return f(self, *args)
File "example/cli_client.py", line 315, in do_decryptFiles
to_file = open(os.path.abspath(to_path), "wb")
IOError: [Errno 13] Permission denied: 'E:\\proto'
Sounds a local directory permissions issue? I had similar problem recently, if it is there are some possible solutions here.
It sounds to me like this isn't a Dropbox API issue, it's a local IO Error.
I have the following situation. Project files was stored in ~/django-apps/app-old/app. For some reasons I move files to ~/django-apps/app . Application have some images, stored with use of ImageField. In database images have paths like so:
~/django-apps/app-old/app/media/images/blabla.jpeg
So, I need to fix this paths in database to look like this:
~django-apps/app/media/images/blabla.jpeg
I try to write management command for to do this:
from django.core.management.base import BaseCommand, CommandError
from books.models import Book
import string
class Command(BaseCommand):
help = ''
def handle(self, *args, **options):
books = Book.objects.all()
total = len(books)
curr = 1
for book in books:
print "%d/%d" % (curr, total)
if book.cover_url != "":
book.cover_url.path = string.replace(book.cover_url.path, "app-old/", "")
book.save()
curr+=1
By using this command I get following error:
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 220, in execute
output = self.handle(*args, **options)
File "/var/www/dizpers/data/django-apps/app/books/management/commands/fix-covers-path.py", line 23, in handle
book.cover_url.path = string.replace(book.cover_url.path, "bparser-old/", "")
File "/usr/local/lib/python2.6/dist-packages/django/db/models/fields/files.py", line 65, in _get_path
return self.storage.path(self.name)
File "/usr/local/lib/python2.6/dist-packages/django/core/files/storage.py", line 234, in path
raise SuspiciousOperation("Attempted access to '%s' denied." % name)
django.core.exceptions.SuspiciousOperation: Attempted access to '/var/www/dizpers/data/django-apps/app-old/app/media/covers/a3d9545d3a17bb68a91749019c95357d.jpeg' denied.
Why I get this error message? How I can fix image's path?
UPD1
My model contain ImageField like this:
cover_url = models.ImageField(upload_to=os.path.join(MEDIA_ROOT, "covers"), null=True, default=None)
UPD2
Path property is readonly
This should work
for book in books:
print "%d/%d" % (curr, total)
if book.cover_url != "":
new_path = string.replace(book.cover_url.path, "app-old/", "")
book.cover_url = new_path
book.save()
curr+=1
You get this message because the default file system storage prevents usage of absolute path for security reasons.
If you really want to use an absolute path
django-documents forces absolute paths for security reasons, this is how:
from django.core.files.storage import FileSystemStorage
fs = FileSystemStorage(location=UPLOAD_TO)
class Document(models.Model):
file = models.FileField(storage=fs, upload_to=UPLOAD_TO)
Where UPLOAD_TO can be an absolute path.
-- UPDATE IN REPLY TO QUESTION UPDATE --
If you don't need such security
Then you should read the documentation on FileField:
FileField.upload_to
A local filesystem path that will be appended to your MEDIA_ROOT setting to determine the value of the url attribute.
In your case, upload_to='covers' is sufficient.
Note that relative paths should be stored in the database.