Develop Reference

Python tempfile.TemporaryDirectory() cleanup crashes with PermissionError and NotADirectoryError

Premise
I'm trying to convert some PDF to images via pdf2image and poppler, to then run some computervision tasks on.
The conversion itsself works fine.
However, the conversion creates some artifacts for each page in the pdf as it is being converted, which I would like to be deleted at the end of the function. To facilitate this, I am using tempfile.TemporaryDirectory(). The function looks as follow:
with tempfile.TemporaryDirectory() as path:
images_from_path: [Image] = convert_from_path(
os.path.join(path_superfolder, "calibration_target.pdf"),
size=(2480, 3508),
output_folder=path, poppler_path=r'E:\poppler-22.04.0\Library\bin')
if len(images_from_path) >= page:
images_from_path[page - 1].save(os.path.join(path_superfolder, "result.jpg"))
Problem
The trouble is, that the program always crashes with the following errors, after transforming the PDF and writing the required image to a file.
Traceback (most recent call last):
File "C:\Python310\lib\shutil.py", line 617, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file, because it is being used by another process: 'C:\\Users\\tobia\\AppData\\Local\\Temp\\tmp24c4bmzv\\bd76d834-672e-49fc-ac30-7751b7b660d0-01.ppm'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python310\lib\tempfile.py", line 843, in onerror
_os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file, because it is being used by another process: 'C:\\Users\\tobia\\AppData\\Local\\Temp\\tmp24c4bmzv\\bd76d834-672e-49fc-ac30-7751b7b660d0-01.ppm'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python310\lib\code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "E:\PyCharm 2022.2.3\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "E:\PyCharm 2022.2.3\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "D:\Dokumente\Uni\Informatik\BA_Thesis\tumexam-scheduling-codebase\generate_data.py", line 393, in <module>
extract_calibration_page_as_image_from_pdf()
File "D:\Dokumente\Uni\Informatik\BA_Thesis\tumexam-scheduling-codebase\generate_data.py", line 190, in extract_calibration_page_as_image_from_pdf
tmp_dir.cleanup()
File "C:\Python310\lib\tempfile.py", line 873, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
File "C:\Python310\lib\tempfile.py", line 855, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "C:\Python310\lib\shutil.py", line 749, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Python310\lib\shutil.py", line 619, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Python310\lib\tempfile.py", line 846, in onerror
cls._rmtree(path, ignore_errors=ignore_errors)
File "C:\Python310\lib\tempfile.py", line 855, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "C:\Python310\lib\shutil.py", line 749, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Python310\lib\shutil.py", line 600, in _rmtree_unsafe
onerror(os.scandir, path, sys.exc_info())
File "C:\Python310\lib\shutil.py", line 597, in _rmtree_unsafe
with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] Directory name invalid: 'C:\\Users\\tobia\\AppData\\Local\\Temp\\tmp24c4bmzv\\bd76d834-672e-49fc-ac30-7751b7b660d0-01.ppm'
When stepping through the cleanup routine, everything seems fine, the path is correct and it starts deleting files, until at some point the internal path variable gets jumbled up and the routine crashes, because obviously a file is not a directory. To me it seems like a race condition is causing problems here.
What I have already tried
Rewriting the function to not use with and instead explicitly call the routine with tmp_dir.cleanup()
Just creating the directory without populating it with the conversion artifacts. The cleanup works in this case.
The documentation for tempfile mentions Permission errors occuring when files are open. The files are however only used in this function and if this is what is causing the error, I am unsure where the files are still opened or which function is causing this. My suspicion of course would be the conversion function.

While experimenting some more and writing this question, I found a working solution:
with tempfile.TemporaryDirectory() as path:
images_from_path: [Image] = convert_from_path(
os.path.join(path_superfolder, f"calibration_target_{exam_type}.pdf"),
size=(2480, 3508),
output_folder=path, poppler_path=r'E:\poppler-22.04.0\Library\bin')
if len(images_from_path) >= page:
images_from_path[page - 1].save(os.path.join(path_superfolder, "result.jpg"))
images_from_path = []
It seems that somehow, the routine had trouble cleaning up, because the converted images, are actually the artifacts created by pdf2image and were still being held by my data structure. Resetting the data structure, before implicitly initiating the cleanup fixed the issue.
If there is a better way of tackling this issue, please do not hesitate to inform me.

Is there a workaround for an [Error 18]: invalid cross-device link while calling 'pip' with Popen?

I am trying to create a GUI for working with the pip command with a text editor. Currently I am receiving an [Error 18]: Invalid cross-device link which traces back to os.rename which my program does not explicitly call.
I believe that this error occurs because pip is trying to call on a temporary file located on the /tmp/ file system and the actual folders are on the /usr/ file system (same computer different mount points). I have tried to resolve this by creating a new temporary directory in my current working directory via tempfile.TemporaryDirectory(dir='./') and feeding that into my Popen(..., cwd=NewTempDir). I have also tried setting the os.environ['TMPDIR'] = NewTempDir and both of these still lead to the Error 18.
self.tmp = tempfile.TemporaryDirectory(dir='./')
os.environ['TMPDIR'] = self.tmp
....
self.sp = subprocess.Popen([sys.executable, '-m', 'pip', \
pip_dict['process'], pip_dict['package']], \
stdout=subprocess.PIPE, stdin=subprocess.PIPE, \
cwd=self.tmp.name)
....
while True:
stdout = self.sp.stdout.read(1)
self.stdout += stdout
print(self.stdout)
if b'Proceed (y/n)?' in self.stdout:
try:
print('attempting response')
self.sp.stdin.write(b'y\n')
self.sp.stdin.flush()
break
except:
print('stdin write failed')
I expect the output to be the uninstalling of the named package.
My current output is:
Exception:
Traceback (most recent call last):
File "/usr/lib/python3.7/shutil.py", line 563, in move
os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link:
'/usr/lib/python3.7/site-packages/anytree' -> '/tmp/pip-
uninstall-ye0w908_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-\
packages/pip/_internal/cli/base_command.py", line 179, in
main
status = self.run(options, args)
File "/usr/lib/python3.7/site-
packages/pip/_internal/commands/uninstall.py", line 75, in run
auto_confirm=options.yes, verbose=self.verbosity > 0,
File "/usr/lib/python3.7/site-
packages/pip/_internal/req/req_install.py", line 817, in uninstall
uninstalled_pathset.remove(auto_confirm, verbose)
File "/usr/lib/python3.7/site-
packages/pip/_internal/req/req_uninstall.py", line 360, in remove
moved.stash(path)
File "/usr/lib/python3.7/site-
packages/pip/_internal/req/req_uninstall.py", line 257, in stash
renames(path, new_path)
File "/usr/lib/python3.7/site-packages/pip/_internal/utils/misc.py",
line 303, in renames
shutil.move(old, new)
File "/usr/lib/python3.7/shutil.py", line 575, in move
rmtree(src)
File "/usr/lib/python3.7/shutil.py", line 491, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.7/shutil.py", line 449, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/usr/lib/python3.7/shutil.py", line 447, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'search.py'
I do receive print("attempting response") before the code breaks.
Here is the link to the full program file... https://github.com/rscales02/porcupine/blob/pipgui/porcupine/plugins/pip_plugin.py
Thanks in advance!

IO Error python PIL image preprocessing script

I am following this tutorial and specifically going through the "generate own data" section:
https://github.com/surfertas/deep_learning/tree/master/projects/imdbwiki-challenge
https://github.com/surfertas/deep_learning/blob/master/projects/imdbwiki-challenge/imdb_preprocess.py
and i am facing this issue running the imdb_preprocess.py script;
Dictionary created...
Converting 1000 samples. (0=all samples)
Traceback (most recent call last):
File "imdb_preprocess.py", line 137, in <module>
main()
File "imdb_preprocess.py", line 131, in main
create_and_dump(imdb_dict, args.partial)
File "imdb_preprocess.py", line 106, in create_and_dump
for img_path in imgs
File "/usr/lib64/python2.7/site-packages/scipy/misc/pilutil.py", line 156, in imread
im = Image.open(name)
File "/usr/lib64/python2.7/site-packages/PIL/Image.py", line 2477, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: u'/path/48/10000548_1925-04-04_1964.jpg'
Now i manually checked folder 48 and checked that the image is complaining about is indeed there.
Any hints on where the fault is?
path was replaced

copyfile(), ValueError: stat: embedded null character in path

I know my tkinter buttons are horribly inefficient but i don't care so please don't comment on that.
https://pastebin.com/4e7LXY2t
copyfile() gives me the error
Traceback (most recent call last):
File "C:\Users\Jack\Desktop\PKSun\test.py", line 80, in <module>
copyfile(str(root)+'/romfs/a'+paths[i],a+'/'+dirs[i])
File "C:\Program Files (x86)\Python36-32\lib\shutil.py", line 103, in copyfile
if _samefile(src, dst):
File "C:\Program Files (x86)\Python36-32\lib\shutil.py", line 88, in _samefile
return os.path.samefile(src, dst)
File "C:\Program Files (x86)\Python36-32\lib\genericpath.py", line 96, in samefile
s1 = os.stat(f1)
ValueError: stat: embedded null character in path
Im using windows 7 and python 3.6.1, i also intend to build this into an EXE file if that makes any difference.

IsADirectoryError: [Errno 21] Is a directory: '/home/cali/Dropbox/'

I have a function where I am trying to copy an XML document to my Dropbox folder, but I am getting:
/usr/bin/python3.5 /home/cali/PycharmProjects/Vocabulary/Vocabulary.py
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python3.5/tkinter/__init__.py", line 1553, in __call__
return self.func(*args)
File "/home/cali/PycharmProjects/Vocabulary/Vocabulary.py", line 232, in add_item
self.sync()
File "/home/cali/PycharmProjects/Vocabulary/Vocabulary.py", line 287, in sync
copyfile(vocabulary_path, destination_path)
File "/usr/lib/python3.5/shutil.py", line 115, in copyfile
with open(dst, 'wb') as fdst:
IsADirectoryError: [Errno 21] Is a directory: '/home/cali/Dropbox/'
Process finished with exit code 0
Here is the function:
def sync(self):
path = os.path.expanduser('~/Desktop')
vocabulary_path = os.path.join(path, 'Vocabulary', 'Words.xml')
destination_path = os.path.expanduser('~/Dropbox/')
copyfile(vocabulary_path, destination_path)
How can I overcome the issue?

Perhaps by copying the file to another file, rather than a directory.
...
destination_path = os.path.expanduser('~/Dropbox/Words.xml')
...