Develop Reference

How to export .csv file from python and using pandas DataFrame

I am trying to export some filtered data from Python using Pandas DF to .csv file (Personal Learning project)
Code : df5.to_csv(r'/C:/Users/j/Downloads/data1/export.csv')
Error:
Traceback (most recent call last):
File "C:\Users\jansa\PycharmProjects\bbb\main.py", line 62, in <module>
df5.to_csv(r'/C:/Users/jansa/Downloads/data1/export.csv')
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\core\generic.py", line 3551, in to_csv
return DataFrameRenderer(formatter).to_csv(
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\io\formats\format.py", line 1180, in to_csv
csv_formatter.save()
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\io\formats\csvs.py", line 241, in save with get_handle(
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\io\common.py", line 697, in get_handle
check_parent_directory(str(handle))
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\io\common.py", line 571, in check_parent_directory
raise OSError(rf"Cannot save file into a non-existent directory: '{parent}'")
OSError: Cannot save file into a non-existent directory: '\C:\Users\jansa\Downloads\data1'
I am researching, but cannot pinpoint the error.

Try
df.to_csv(r'C:\path\to\directory\filename.csv')

Generally, in Linux/Mac environment path separator is '/' but in windows, it is '\'. Also, the absolute path starts with '/' in Linux/Mac, while in windows, it starts with / So, using arguments in to_csv with C:\Users\j\Downloads\data1\export.csv' will resolve your issue.
In addition, if you want to get rid of such situations, you can do this:
import os
path = os.path.join('.', 'export.csv') #will save the file in current directory
Also, this returns the os path separator:
print(os.sep)

python giving error in extracting a file from the file

Python code:
import tarfile,os
import sys
def extract_system_report (tar_file_path):
extract_path = ""
tar = tarfile.open(sys.argv[1])
for member in tar.getmembers():
print ("member_name is: ")
print (member.name)
tar.extract(member.name, "./crachinfo/")
extract_system_report(sys.argv[1])
while extracting the file getting below error:
>> python tar_read.py /nobackup/deepakhe/POLARIS_POJECT_09102019/hackathon_2021/a.tar.gz
member_name is:
/bootflash/.prst_sync/reload_info
Traceback (most recent call last):
File "tar_read.py", line 38, in <module>
extract_system_report(sys.argv[1])
File "tar_read.py", line 10, in extract_system_report
tar.extract(member.name, "./crachinfo/")
File "/auto/pysw/cel8x/python64/3.6.10/lib/python3.6/tarfile.py", line 2052, in extract
numeric_owner=numeric_owner)
File "/auto/pysw/cel8x/python64/3.6.10/lib/python3.6/tarfile.py", line 2114, in _extract_member
os.makedirs(upperdirs)
File "/auto/pysw/cel8x/python64/3.6.10/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/auto/pysw/cel8x/python64/3.6.10/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/bootflash'
I am specifying the folder to extract but still it seems trying to create a folder in the root directory. is this behavior expected? I can extract this tar file fine in file manager. is there a way to handle this in python?

I am specifying the folder to extract but still it seems trying to create a folder in the root directory. is this behavior expected?
Yes. The path provided is just a "current directory", it's not a "jail". The documentation even specifically warns about it:
It is possible that files are created outside of path, e.g. members that have absolute filenames starting with "/" or filenames with two dots "..".
I can extract this tar file fine in file manager. is there a way to handle this in python?
Use extractfile to get a pseudo-file handle on the contents of the archived file, then copy that wherever you want.

Owlrady2 Load imported onotologies from file not from url

I'm trying to load an ontology that imports some other ontologies. These other ontologies are located on my machine and not on the web. If I try my_ontology.load() it tries to load these other ontologies from their URI. But that fails.
How can I tell owlready2 where these ontologies are located?
The traceback looks like this:
Traceback (most recent call last):
File "~Software/PyOntologyChecker/app.py", line 12, in <module>
covid_onto = get_ontology_from_file(COVID_ONTOLOGY)
File "~Software/PyOntologyChecker/app.py", line 8, in get_ontology_from_file
return get_ontology("file://" + file_path).load()
File "~.local/lib/python3.8/site-packages/owlready2/namespace.py", line 813, in load
imported_ontologies = [self.world.get_ontology(self._unabbreviate(abbrev_iri)).load() for abbrev_iri in self.world._get_obj_triples_sp_o(self.storid, owl_imports)]
File "~.local/lib/python3.8/site-packages/owlready2/namespace.py", line 813, in <listcomp>
imported_ontologies = [self.world.get_ontology(self._unabbreviate(abbrev_iri)).load() for abbrev_iri in self.world._get_obj_triples_sp_o(self.storid, owl_imports)]
File "~.local/lib/python3.8/site-packages/owlready2/namespace.py", line 773, in load
except: raise OwlReadyOntologyParsingError("Cannot download '%s'!" % f)
owlready2.base.OwlReadyOntologyParsingError: Cannot download 'http://ontofox.hegroup.org/CHEBI_module11.owl'!

Assuming you have a local copy of CHEBI_module11.owl dependence (ex. SOME_DIR/HEBI_module11.owl), add its location into onto_path (as stated in the docs):
onto_path.append("SOME_DIR")
before doing my_ontology.load(). That tells Owlready2 to look for files in SOME_DIR. The path can be relative or absolute (like c:/data/ontologies).
The current Python's working directory is not scanned by default but can be set so with onto_path.append(".").
If a required ontology is described by URI not ending with a usual filename, like "http://rdf.webofcode.org/woc/", the local filename (woc in this case) should not have an extension.

How to get content of file inside package

I am writing something in Python where I want to use predefined texts from files within the package. Somehow I can't manage to get it to work in Eclipse PyDev Console.
This is my path structure. From "story.py" I want to use the content of "starttext".
I tried open() with multiple variations of os.getcwd() and os.path.dirname(sys.argv[0]) which resulted in
FileNotFoundError: [Errno 2] No such file or directory: '..\starttext'
My last attempt was trying something like
import pkg_resources
resource_package = __name__
resource_path = '/'.join(('.', 'starttext'))
template = pkg_resources.resource_stream(resource_package, resource_path)
resulting in:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\Program Files\Python\Python36-64\lib\site-packages\pkg_resources\__init__.py", line 1232, in resource_stream
self, resource_name
File "C:\Program Files\Python\Python36-64\lib\site-packages\pkg_resources\__init__.py", line 1479, in get_resource_stream
return io.BytesIO(self.get_resource_string(manager, resource_name))
File "C:\Program Files\Python\Python36-64\lib\site-packages\pkg_resources\__init__.py", line 1482, in get_resource_string
return self._get(self._fn(self.module_path, resource_name))
File "C:\Program Files\Python\Python36-64\lib\site-packages\pkg_resources\__init__.py", line 1560, in _get
"Can't perform this operation for loaders without 'get_data()'"
NotImplementedError: Can't perform this operation for loaders without 'get_data()'
which appears to have something to do with python 3.x?
This seems to be such an easy task and I don't understand whats wrong.
Any help is appreciated.
Thank you.
update
Thanks to ShmulikA I changed it to:
from os.path import dirname, join, abspath
filename = join(dirname(abspath(communitybot.anthology.teststory.story.__file__)), 'starttext')
file = open(filename, 'r')
content = file.read()
This works although I think it is a little bit long, but I'm certain I am still doing something wrong there.

seems like you are missing a \ - use os.path.join:
from os.path import dirname, join, abspath
filename = join(dirname(abspath(__file__)), 'starttext')
file = open(filename, 'r')
__file__ - the path to the module's source file (you can also do import requests;requests.__file__)
os.path.abspath - returns the absolute filename (e.g. abspath('..') returns /home)
os.path.dirname - returns the dirname of a file
os.path.join - joins a file parts compatible on both linux and windows

Django: How to allow a Suspicious File Operation / copy a file

I want to do a SuspiciousFileOperation which django disallows by default.
I am writing a command (to run via manage.py importfiles) to import a given directory structure on the real file system in my self written filestorage in Django.
I think, this is my relevant code:
def _handle_directory(self, directory_path, directory):
for root, subFolders, files in os.walk(directory_path):
for filename in files:
self.cnt_files += 1
new_file = File(directory=directory, filename=filename, file=os.path.join(root, filename),
uploader=self.uploader)
new_file.save()
The backtrace is:
Traceback (most recent call last):
File ".\manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 399, in execute_from_command_line
utility.execute()
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 285, in execute
output = self.handle(*args, **options)
File "D:\Development\github\Palco\engine\filestorage\management\commands\importfiles.py", line 53, in handle
self._handle_directory(args[0], root)
File "D:\Development\github\Palco\engine\filestorage\management\commands\importfiles.py", line 63, in _handle_directory
new_file.save()
File "D:\Development\github\Palco\engine\filestorage\models.py", line 157, in save
self.sha512 = hashlib.sha512(self.file.read()).hexdigest()
File "C:\Python27\lib\site-packages\django\core\files\utils.py", line 16, in <lambda>
read = property(lambda self: self.file.read)
File "C:\Python27\lib\site-packages\django\db\models\fields\files.py", line 46, in _get_file
self._file = self.storage.open(self.name, 'rb')
File "C:\Python27\lib\site-packages\django\core\files\storage.py", line 33, in open
return self._open(name, mode)
File "C:\Python27\lib\site-packages\django\core\files\storage.py", line 160, in _open
return File(open(self.path(name), mode))
File "C:\Python27\lib\site-packages\django\core\files\storage.py", line 261, in path
raise SuspiciousFileOperation("Attempted access to '%s' denied." % name)
django.core.exceptions.SuspiciousFileOperation: Attempted access to 'D:\Temp\importme\readme.html' denied.
The full model can be found at GitHub. The full command is currently on gist.github.com available.
If you do not want to check the model: the attribute file of my File class is a FileField.
I assume, this problem happens, because I am just "linking" to the file found. But I need to copy it, huh? How can I copy the file into the file?

In Django, SuspiciousFileOperation can be avoid by read the file from external dir and make a tmp file within the project media then save in the appropriate file filed as below
import tempfile
file_name="file_name.pdf"
EXT_FILE_PATH = "/home/somepath/"
file_path = EXT_FILE_PATH + file_name
if exists(file_path):
#create a named temporary file within the project base , here in media
lf = tempfile.NamedTemporaryFile(dir='media')
f = open(file_path, 'rb')
lf.write(f.read())
#doc object with file FileField.
doc.file.save(file_name, File(lf), save=True)
lf.close()

I haven't faced similar problem but related issue. I have recently upgraded Django 1.8 to 1.11.
Now I am getting the following error if try to save a file in a model having FileField field:
SuspiciousFileOperation at /api/send_report/
The joined path (/vagrant/tmp/test_file.pdf) is located outside of the base path component (/vagrant/media)
My model where I want to save the file:
class Report(BaseModel):
file = models.FileField(max_length=200, upload_to=os.path.join(settings.REPORTS_URL, '%Y/week_%W/'))
type = models.CharField(max_length=20, verbose_name='Type', blank=False, default='', db_index=True)
I am trying following codes to save the file from tmp folder which is not located in MEDIA_ROOT:
from django.core.files import File
filepath = "/vagrant/tmp/test_file.pdf"
file = File(open(filepath, "rb"))
report_type = "My_report_type"
report = Report.objects.create(
file=file,
type=report_type,
)
What I have done to solve the issue:
import os
from django.core.files import File
filepath = "/vagrant/tmp/test_file.pdf"
file = File(open(filepath, "rb"))
file_name = os.path.basename(file.name)
report_type = "My_report_type"
report = Report.objects.create(
type=report_type,
)
report.file.save(file_name, file, save=True)
Hope it will help someone.

Analyzing this part of stacktrace:
File "C:\Python27\lib\site-packages\django\core\files\storage.py", line 261, in path
raise SuspiciousFileOperation("Attempted access to '%s' denied." % name)
leads to the standard Django FileSystemStorage. It expects files to be within your MEDIA_ROOT. Your files can be anywhere in the file system, therefore this problem occurs.
You should pass file-like object instead of a path to your File model. The easiest way to achieve that would be to use Django File class, which is a wrapper around python file-like objects. See File object documentation for more details.
Update:
Ok, I am suggesting here a route taken from the docs:
from django.core.files import File as FileWrapper
def _handle_directory(self, directory_path, directory):
for root, subFolders, files in os.walk(directory_path):
for filename in files:
self.cnt_files += 1
new_file = File(
directory=directory, filename=filename,
file=os.path.join(root, filename),
uploader=self.uploader)
with open(os.path.join(root, filename), 'r') as f:
file_wrapper = FileWrapper(f)
new_file = File(
directory=directory, filename=filename,
file=file_wrapper,
uploader=self.uploader)
new_file.save()
If it works it should copy the file to the location provided by your secure_storage callable.

Check if there is slash before your filepath
file_item = models.FileField(upload_to=content_file_name)
def content_file_name(username, filename):
return '/'.join(['content', username, filename])
Note here "content" not "/content". That was the problem for me.