Develop Reference

shutil.unpack_archive and ziplib both return FileNotFoundError: [Errno 2] No such file or directory

I'm trying to unzip some zip files in Python:
print(f"Unzipping:\n{os.getcwd()}\\{constants.DOWNLOADS_FOLDER}\\{case_number}\\{username_hostname}.zip")

# with zipfile.ZipFile(f"{os.getcwd()}\\{constants.DOWNLOADS_FOLDER}\\{case_number}\\{username_hostname}.zip") as z:
# z.extractall(path=f"{os.getcwd()}\\{constants.DOWNLOADS_FOLDER}\\{case_number}\\{username_hostname}")
shutil.unpack_archive(f"{os.getcwd()}\\{constants.DOWNLOADS_FOLDER}\\{case_number}\\{username_hostname}.zip",

 f"{os.getcwd()}\\{constants.DOWNLOADS_FOLDER}\\{case_number}\\{username_hostname}")
print(f"Unzipped:\n{os.getcwd()}\\{constants.DOWNLOADS_FOLDER}\\{case_number}\\{username_hostname}")
It works for most of the zip files (case_number being the same, and username_hostname being different), but I got only 2~3 files which always return:
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\current_workdirectory\\downloads\\case_number\\username_hostname\\path\\OmniDesk \\OmniDesk.pdf'
... or ...
'D:\\current_workdirectory\\downloads\\case_number\\username_hostname\\path\\Users\\username\\Desktop\\path\\OCT 11 to 16th \\2022-10-11.xlsx'
I tried to manually unzip the file (in Windows) and it works fine.
I do notice a strange space at the end of these files' parent foldername, and when I manually unzip the archive, the space is NOT in the foldername of the unpacked archive.
Is this what caused the problem?
case_number and username_hostname is created from sys.argv:
case_number = sys.argv[1] # e.g., XXX-YYY
username_hostname = sys.argv[2] # e.g., johnsmith_PC12345
printout:
DEBUG: This is case_number -->F23-003<--
Unzipping:
D:\path\F23-003\username_hostname.zip
...
FileNotFoundError: [Errno 2] No such file or directory:
...
I saw a discussion on cpython' github repo: https://github.com/python/cpython/issues/94018#issuecomment-1160309581
But, according to the repo, this issue should has been solved: gvanrossum pushed a commit to gvanrossum/cpython that referenced this issue on Jul 1, 2022
I'm using Python 3.11.1

I confirmed that Python 3.11.1 hasn't included this commit (mentioned in the question) yet, so I edited Python's built-in zipfile myself:
(line 1689, original)
arcname = (x.rstrip('.') for x in arcname.split(pathsep))
to...
(line 1689)
arcname = (x.rstrip(' .') for x in arcname.split(pathsep))
... which solved the problem.
So, if anyone encounters the same problem, please copy Python's built-in zipfile.py, edit it as described above, and put it into your project folder.
By importing the edited zipfile.py, you can overwrite Python's built-in troublesome zipfile.py.

creating/deleting folders in runtime using heroku/django

I have developed a Django app where I am uploading a file, doing some processing using a project folder name media.
Process:
user uploads a csv file, python code treats the csv data by creating temp folders in Media folder. After processing is complete, these temp folders are deleted and processed data is downloaded through browser.
I am using the below lines of code to make and delete temp file after processing
temp = 'media/temp3'
os.mkdir(temp)
shutil.copyfile('media/' + file_name, temp + '/' + file_name)
shutil.rmtree(temp, ignore_errors=True)
To set the media root, I used the below lines in settings.py which I am sure I am not using in other parts of the code.
MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
MEDIA_URL = "/media/"
Everything works fine when I run the app on local host. but as soon as i deployed it to heroku, it seems like these folders were not created/not found.
I am looking for:
Either a solution to create, read and delete folders/files in runtime using heroku,
or
a better way to manage files/folders in runtime.

Chron - [Permission Denied] when I try to save & remove file from directory

Issue: Unable to save file in directory (/root/Notion/Image) when using Cron schedule
This is what my code is trying to do:
Check email
Download image attachment
Store in a directory - root/Notion/Image
Retrieve file path
The script is working when I run it manually in Google Cloud terminal. The problem is when I try to schedule it on Cron, it's unable to access the folder to save the file locally.
This is the error when the script failed and require permission:
Traceback (most recent call last):
File "Notion/test.py", line 121, in <module>
path = get_attachments(email.message_from_bytes(msg[0][1]))
File "Notion/test.py", line 47, in get_attachments
with open(filePath, 'wb') as f:
PermissionError: [Errno 13] Permission denied: '/root/Notion/Image/3.jpeg'
This is the code to retrieve attachment from email
def get_attachments(msg):
for part in msg.walk():
if part.get_content_maintype()=='multipart':
continue
if part.get('Content-Disposition') is None:
continue
fileName = part.get_filename()
if bool(fileName):
filePath = os.path.join(attachment_dir, fileName)
with open(filePath, 'wb') as f:
f.write(part.get_payload(decode=True))
return str(filePath)
Resolved:
The problem is that I shouldn't use root directory since it requires permission. I've changed it to home directory instead.
attachment_dir = '/home/dev_thomas_yang/folder_name/folder_name'
For people who needs to check their home direction, simply run this script.
from pathlib import Path
home= str(Path.home())
print(home)
Thanks Triplee for the patience to breakdown my issue despite my sloppy ways of presenting it!

The easiest fix hands down is to change the code so it doesn't try to write to /root. Have it write to the invoking user's home directory instead.
Your question doesn't show the relevant parts of the code, but just change attachment_dir so it's not an absolute path. Maybe separately take care of creating the directory if it doesn't already exist.
import pathlib
# ...
attachment_dir = pathlib.Path("cron/whatever/attachments").mkdir(parents=True, exist_ok=True)
# ...
for loop in circumstances:
get_attachments(something)
A better design altogether would be to have get_attachments accept the directory name as a parameter, so you can make this configurable from the code which calls it. Global variables are a nuisance and cause hard-to-debug problems because they hide information which is important for understanding the code, and tricky to change when you try to debug that code and don't know which parts of the code depend on the old value.

OSError: [Errno 30] Read-only file system in Django on Heroku

I'm using Django 2.0 and Heroku to host the application.
My media directory settings are like
App/settings/production.py:
BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(os.path.dirname(BASE_DIR), 'static_cdn', 'media_root')
I'm using gTTS to convert text to speech and save .mp3 file in the media directory:
tts_file_name = str(int(time.time())) + '.mp3'
joined_path = os.path.join(settings.MEDIA_ROOT, 'tts')
joined_path_with_file = os.path.join(joined_path, tts_file_name)
# create directory if does not exists
if not os.path.exists(joined_path):
os.makedirs(joined_path)
tts = gTTS(text='Good morning', lang='en')
tts.save(joined_path_with_file)
# tts path to send to template
tts_media_url = os.path.join(settings.MEDIA_URL, 'tts', tts_file_name)
It is working fine on local system as I can change file permissions manually also.
But It is not working on Heroku and giving error:
OSError: [Errno 30] Read-only file system: '/static_cdn'
I tried to locate static_cdn by running heroku shell, but could not even found static_cdn in application path and root path. But it seems to be working as other uploading through form is working perfectly.
using Django model's upload_to is working and even directory is created in static_cdn.
How can I create directory in static_cdn on Heroku the same way Django does using model's upload_to?

Changed MEDIA_ROOT path by removing additional os.path.dirname() and it is working now.
MEDIA_ROOT = os.path.join(BASE_DIR, 'static_cdn', 'media_root')

In my case, this error occurred because I set the STATIC_ROOT = '/static/'
This means it's looking at / root folder of the system and then static, which is obviously read-only,
changing it to STATIC_ROOT = 'static/' fixed my issue.

I'm using gTTS to convert text to speech and save .mp3 file in the media directory
I'm not sure what's causing your immediate error, but this isn't going to work very well on Heroku. Its filesystem is ephemeral: you can write to it, but whatever you write will be lost when the dyno restarts. This happens frequently (at least once per day).
Heroku recommends using a third-party file or object store like Amazon S3 for storing generated files, uploaded files, etc. I recommend gong down this path. There are many Django libraries for using S3, and other services, as storage backends.

matplotlib with Django fails on print_figure throwing [Errno 2] No such file or directory

I am trying to create a png via matplotlib but I get:
[Errno 2] No such file or directory
The same code works on unit tests. print_figure call is in
# creates a canvas from figure
canvas = FigureCanvasAgg(fig)
filename = "directory" + os.sep + name_prefix + ".png"
# saves figure to filesystem in png format
canvas.print_figure(filename)
I am thinking it can be a permission issue, but it seems weird to me that the same code works via manage.py test
Thanks

My recommendation is to use fully qualified path names. For example: you could determine the MEDIA_ROOT from your django settings, write a snippet of code that ensures that a subdirectory for the graphs exists, then save the images there.
Your current code seems to rely on finding a subdirectory with the appropriate name in the "current working directory". The "current working directory" is a finicky thing - it will be different in testing, dev, production...
# import settings
from django.conf import settings
...
# ensure that a subdirectory with the appropriate name exists
if not os.path.exists(directory):
os.makedirs(directory)
# save the plots
canvas = FigureCanvasAgg(fig)
filename = settings.MEDIA_ROOT + os.sep + directory + os.sep + name_prefix + ".png"
# saves figure to filesystem in png format
canvas.print_figure(filename)
...
The actual location that you will save at should be determined by your needs. The key points are to use fully qualified paths and to check for the existence of the directory / subdirectory before attempting to save the image.