Develop Reference

How to export .csv file from python and using pandas DataFrame

I am trying to export some filtered data from Python using Pandas DF to .csv file (Personal Learning project)
Code : df5.to_csv(r'/C:/Users/j/Downloads/data1/export.csv')
Error:
Traceback (most recent call last):
File "C:\Users\jansa\PycharmProjects\bbb\main.py", line 62, in <module>
df5.to_csv(r'/C:/Users/jansa/Downloads/data1/export.csv')
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\core\generic.py", line 3551, in to_csv
return DataFrameRenderer(formatter).to_csv(
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\io\formats\format.py", line 1180, in to_csv
csv_formatter.save()
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\io\formats\csvs.py", line 241, in save with get_handle(
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\io\common.py", line 697, in get_handle
check_parent_directory(str(handle))
File "C:\Users\jansa\PycharmProjects\bbb\venv\lib\site-packages\pandas\io\common.py", line 571, in check_parent_directory
raise OSError(rf"Cannot save file into a non-existent directory: '{parent}'")
OSError: Cannot save file into a non-existent directory: '\C:\Users\jansa\Downloads\data1'
I am researching, but cannot pinpoint the error.

Try
df.to_csv(r'C:\path\to\directory\filename.csv')

Generally, in Linux/Mac environment path separator is '/' but in windows, it is '\'. Also, the absolute path starts with '/' in Linux/Mac, while in windows, it starts with / So, using arguments in to_csv with C:\Users\j\Downloads\data1\export.csv' will resolve your issue.
In addition, if you want to get rid of such situations, you can do this:
import os
path = os.path.join('.', 'export.csv') #will save the file in current directory
Also, this returns the os path separator:
print(os.sep)

No such file or directory: 'GoogleNews-vectors-negative300.bin'

I have this code :
import gensim
filename = 'GoogleNews-vectors-negative300.bin'
model = gensim.models.KeyedVectors.load_word2vec_format(filename, binary=True)
and this is my folder organization thing :
image of my folder tree that shows that the .bin file is in the same directory as the file calling it, the file being ai_functions
But sadly I'm not sure why I'm having an error saying that it can't find it. Btw I checked, I am sure the file is not corrupted. Any thoughts?
Full traceback :
File "/Users/Ile-Maurice/Desktop/Flask/flaskapp/run.py", line 1, in <module>
from serv import app
File "/Users/Ile-Maurice/Desktop/Flask/flaskapp/serv/__init__.py", line 13, in <module>
from serv import routes
File "/Users/Ile-Maurice/Desktop/Flask/flaskapp/serv/routes.py", line 7, in <module>
from serv.ai_functions import checkplagiarism
File "/Users/Ile-Maurice/Desktop/Flask/flaskapp/serv/ai_functions.py", line 31, in <module>
model = gensim.models.KeyedVectors.load_word2vec_format(filename, binary=True)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gensim/models/keyedvectors.py", line 1629, in load_word2vec_format
return _load_word2vec_format(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/gensim/models/keyedvectors.py", line 1955, in _load_word2vec_format
with utils.open(fname, 'rb') as fin:
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/smart_open/smart_open_lib.py", line 188, in open
fobj = _shortcut_open(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/smart_open/smart_open_lib.py", line 361, in _shortcut_open
return _builtin_open(local_path, mode, buffering=buffering, **open_kwargs)
FileNotFoundError: [Errno 2] No such file or directory: 'GoogleNews-vectors-negative300.bin'

The 'current working directory' that the Python process will consider active, and thus will use as the expected location for your plain relative filename GoogleNews-vectors-negative300.bin, will depend on how you launched Flask.
You could print out the directory to be sure – see some ways at How do you properly determine the current script directory? – but I suspect it may just be the /Users/Ile-Maurice/Desktop/Flask/flaskapp/ directory.
If so, you could relatively-reference your file with the path relative to the above directory...
serv/GoogleNews-vectors-negative300.bin
...or you could use a full 'absolute' path...
/Users/Ile-Maurice/Desktop/Flask/flaskapp/serv/GoogleNews-vectors-negative300.bin
...or you could move the file up to its parent directory, so that it is alonside your Flask run.py.

makedirs cannot create directory because it contains a question mark

Someone created an awesome script to download videos for personal use which is permitted under their TOS. However when the script tries to create a directory based on the title name, it creates an error due to a question mark "?" in the title. Windows does not permit special characters in the directory name. Is there a way to tell makedirs to ignore special character?
Traceback (most recent call last):
File "D:\nick.py", line 91, in <module>
main()
File "D:\nick.py", line 88, in main
episode.download()
File "D:\nick.py", line 37, in download
os.makedirs(dirname)
File "C:\Users\*****\AppData\Local\Programs\Python\Python38\lib\os.py", line 223, in makedirs
mkdir(name, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: "The Crystal Maze\\What's The Sign For Winning?"

You can remove the symbols using this line:
''.join([i for i in name if i.isalnum() or i.isspace()])
You'll have to change dirname in the source code for what I wrote above:
os.makedirs(dirname) # in line 37, in download
It will save as the name, but without the symbols:
Out[58]: 'The Crystal Maze Whats The Sign For Winning'

Permission denied when trying to copy files

Using the script:
file = os.path.join(subfolder_name, list_of_files[i])
for dest_folder_finetune, dest_folder_relab in zip(finetune_datasets, relab_datasets):
copy(file, dest_folder_finetune)
copy(file, dest_folder_relab)
every 20 iterations, I am having a permission denied problem. The stack, looks something like:
Traceback (most recent call last):
File "/home/revan/boosting_classifier_with_games/dataset_creator.py", line 72, in <module>
main()
File "/home/revan/boosting_classifier_with_games/dataset_creator.py", line 24, in main
create_test_and_relab(list_of_subfolders)
File "/home/revan/boosting_classifier_with_games/dataset_creator.py", line 66, in create_test_and_relab
copy(file, dest_folder_finetune)
File "/home/revan/anaconda2/envs/pytorch/lib/python2.7/shutil.py", line 119, in copy
copyfile(src, dst)
File "/home/revan/anaconda2/envs/pytorch/lib/python2.7/shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: '/sun_btbivuchmkkzetpo.jpg'
The 'funny' thing is that I definitely have all permissions for that file. Furthermore, I tried to copy it manually, and I can do it without problem. If I randomize the process, the same file gets copied, but then some other file (after 20 iterations) cannot be copied.
PS: Changing copy to copy2, gives the exact same problem.
Has anyone experienced anything similar in the past? Could it be a Python or a Linux problem?

One of your datasets has a bad folder name, which is setting the destination to the root of the file system.
To just get it working, you can skip when it's trying to copy to the root:
import os
file = os.path.join(subfolder_name, list_of_files[i])
for dest_folder_finetune, dest_folder_relab in zip(finetune_datasets, relab_datasets):
if os.path.abspath(dest_folder_finetune) != "/":
copy(file, dest_folder_finetune)
else:
print("Warning, path {} for dest_folder_finetune writes to the root of the filesystem".format(dest_folder_finetune))
if os.path.abspath(dest_folder_relab) != "/":
copy(file, dest_folder_relab)
else:
print("Warning, path {} for dest_folder_relab writes to the root of the filesystem".format(dest_folder_relab ))
However, if this is more than a live once script, I suggest scrubbing and verifying the dataset beforehand instead.

try with "sudo python filename.py"

Python Zipfile - Invalid Argument Errno 22

I have a .zip file and would like to know the names of the files within it. Here's the code:
zip_path = glob.glob(path + '/*.zip')[0]
file = open(zip_path, 'r') # opens without error
if zipfile.is_zipfile(file):
print str(file) # prints to console
my_zipfile = zipfile.ZipFile(zip_path) # throws IOError
Here is the traceback:
<open file u'/Users/me/Documents/project/uploads/assets/peter/offline_message/offline_imgs.zip', mode 'r' at 0x107b2a150>
Traceback (most recent call last):
File "/Users/me/Documents/project/admin_dev/proj_name/views.py", line 1680, in get_dps_app_builder_assets
link_to_assets_zip = zip_dps_app_builder_assets(server_url, app_slug, button_slugs)
File "/Users/me/Documents/project/admin_dev/proj_name/views.py", line 1724, in zip_dps_app_builder_assets
my_zipfile = zipfile.ZipFile(zip_path)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 712, in __init__
self._GetContents()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 746, in _GetContents
self._RealGetContents()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 779, in _RealGetContents
fp.seek(self.start_dir, 0)
IOError: [Errno 22] Invalid argument
I am very confused as to why this is happening since the file is clearly there and is a valid .zip file. The documentation clearly states that you can pass it either the path to the file or a file-like object, neither of which work in my case:
http://docs.python.org/2/library/zipfile#zipfile-objects

I was not able to figure this issue out and ended up doing it a different way entirely.
EDIT: In the Django app I work with, users needed to be able to upload assets in the form of .zip files and later download everything they had uploaded (plus other content we generate dynamically) in another zip with a different structure. So, I wanted to unzip a previously uploaded file and zip up the contents of that file up in another zip, which I couldn't do because of the error. Instead of reading the zip file when the user requested the download, I ended up unzipping it from a Django InMemoryUploadedFile (whose contents I was able to successfully read) and just leaving the unzipped files on the file system to work with later. The contents of the zip are only two smallish image files, so this workaround of unzipping the zip ahead of time to be used later worked OK for my purposes.