Develop Reference

How to deal with WhiteSpaces in directory when calling a function from a module

I have a code where I call a Node script. The javaScript named preprocess_latex.js is in the same directory as filter_tokenize.py. Every directory has __init__.py so I can call the modules.
My code looks like this:
def tokenize(latex:str,kind:str='normalize')->list:
'''
Tokenize the incoming LaTex. This uses subscripts based on Node and Perl which uses bash call to KaTex. It also creates 2 .lst files in the same directory where the function is being called
args:
latex: Latex string
out:
List of cleaned tokens per LaTex
'''
path = pathlib.Path(__file__).parent.absolute() # When you call this module, it'll give the path of THIS file instead of the Current Working Directory
output_file = os.path.join(path,'out.lst')
input_file = os.path.join(path,'input_file.lst')
js_file_path = os.path.join(path,"preprocess_latex.js").replace(' ','\\ ')
with open(input_file,'w') as f:
f.write(latex+'\n')
cmd = "perl -pe 's|hskip(.*?)(cm\\|in\\|pt\\|mm\\|em)|hspace{\\1\\2}|g' %s > %s"%(input_file, output_file) # Call Perl subscript
ret = subprocess.call(cmd, shell=True)
cmd = f"cat {temp_file} | node {js_file_path} {kind} > {output_file} "
ret = subprocess.call(cmd, shell=True)
Problem is that I have spaces in my directories and when I do:
from latex_utils.filter_tokenize import *
tokenize('some_string')
it gives me error as:
Can't open Improve/Latex/latex_simialarity/latex_utils/input_file.lst: No such file or directory.
Can't open Improve/Latex/latex_simialarity/latex_utils/out.lst: No such file or directory.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/admin1/Desktop/Work/Search Improve/Latex/latex_simialarity/latex_utils/filter_tokenize.py", line 31, in tokenize
with open(output_file) as fin:
FileNotFoundError: [Errno 2] No such file or directory: '/home/admin1/Desktop/Work/Search Improve/Latex/latex_simialarity/latex_utils/out.lst'
I tried with output_file.replace(' ','\\ ') and there was a different error as:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/admin1/Desktop/Work/Search Improve/Latex/latex_simialarity/latex_utils/filter_tokenize.py", line 23, in tokenize
with open(input_file,'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/admin1/Desktop/Work/Search\\ Improve/Latex/latex_simialarity/latex_utils/input_file.lst'

FileNotFoundError when trying to rename all the files in a directory

I am writing a python script to rename all files in a given folder. The python script exists in my j-l-classifier along with images/jaguar file. I'm trying to run the following script to take each file in the folder and rename it to this format:
jaguar_[#].jpg
But its throwing the following error:
Traceback (most recent call last):
File "/home/onur/jaguar-leopard-classifier/file.py", line 14, in <module>
main()
File "/home/onur/jaguar-leopard-classifier/file.py", line 9, in main
os.rename(filename, "Jaguar_" + str(x) + file_ext)
FileNotFoundError: [Errno 2] No such file or directory: '406.Black+Leopard+Best+Shot.jpg' -> 'Jaguar_0.jpg'
This is my code:
import os
def main():
x = 0
file_ext = ".jpg"
for filename in os.listdir("images/jaguar"):
os.rename(filename, "Jaguar_" + str(x) + file_ext)
x += 1
if __name__ == '__main__':
main()

In order to use os.rename(), you need to provide absolute paths.
I would suggest replacing line 9 with os.rename(os.path.expanduser(f"~/{whatever folders you have here}/images/jaguar/{filename}"), os.path.expanduser(f"~/{whatever folders you have here}/images/jaguar/Jaguar_{str(x)}{file_ext}")
os.path.expanduser() allows you to use the "~" syntax to aid the abs file path.

os.listdir only returns the filename (not the file path...)
try the following
for filename in os.listdir("images/jaguar"):
filepath = os.path.join("images/jaguar",filename)
new_filepath = os.path.join("images/jaguar","Jaguar_{0}{1}".format(x,file_ext))
os.rename(filepath, new_filepath)
being explicit is almost always a path to a happier life

Extract a .TAR file within a .gz file

I have to untar around fifty *.gz files in a directory. Inside each *.gz file there is a *.TAR file and some other files.
I am trying a python script which extracts the contents of the *.gz files to a directory. But, I am not able to extract the *.TAR files inside the same directory to which the contents of *.gz are extracted.
This is how the script looks:
import tarfile
import os
import glob
basedir = "path_to _dir"
for i in glob.glob(basedir +"*.gz"):
a = os.path.basename(i)
b = os.path.splitext(a)[0]
c = os.path.splitext(b)[0]
os.mkdir(os.path.join(basedir,c))
t1 = tarfile.open(i)
t1.extractall(c)
for j in os.listdir(c):
if j.endswith('.TAR'):
print(j)
t2 = tarfile.open(j)
t2.extractall()
t2.close()
t1.close()
Its giving me the error:
Traceback (most recent call last):
File "./untar.py", line 16, in <module>
t2 = tarfile.open(j)
File "/usr/lib64/python2.7/tarfile.py", line 1660, in open
return func(name, "r", fileobj, **kwargs)
File "/usr/lib64/python2.7/tarfile.py", line 1722, in gzopen
fileobj = bltn_open(name, mode + "b")
IOError: [Errno 2] No such file or directory: '0299_0108060501.TAR'
0299_0108060501.TAR is the file contained inside the *.gz file
It seems to me that I am doing something very wrong fundamentally, but I don't know what.

Since tar.gz files are TAR archives compressed with gzip one should use
t1 = tarfile.open(i, 'r:gz')
as per documentation.
Also, you need to combine the path of the inner file with the directory being inspected, like so:
t2 = tarfile.open(os.path.join(c, j))

Renaming files recursively with Python

I have a large directory structure, each directory containing multiple sub-directories, multiple .mbox files, or both. I need to rename all the .mbox files to the respective file name without the extension e.g.
bar.mbox -> bar
foo.mbox -> foo
Here is the script I've written:
# !/usr/bin/python
import os, sys
def walktree(top, callback):
for path, dirs, files in os.walk(top):
for filename in files:
fullPath = os.path.join(path, filename)
callback(fullPath)
def renameFile(file):
if file.endswith('.mbox'):
fileName, fileExt = os.path.splitext(file)
print file, "->", fileName
os.rename(file,fileName)
if __name__ == '__main__':
walktree(sys.argv[1], renameFile)
When I run this using:
python walktrough.py "directory"
I get the error:
Traceback (most recent call last):
File "./walkthrough.py", line 18, in <module>
walktree(sys.argv[1], renameFile)
File "./walkthrough.py", line 9, in walktree
callback(fullPath)
File "./walkthrough.py", line 15, in renameFile
os.rename(file,fileName)
OSError: [Errno 21] Is a directory

This was solved by adding an extra conditional statement to test if the name the file was to be changed to, was a current directory.
If this was true, the filename to-be had an underscore added to.
Thanks to WKPlus for the hint on this.
BCvery1

How can I appropriately include external resources with cxfreeze?

I'm trying to use cxfreeze to build my Python scripts into an .exe file. However my scripts use some external data files which aren't being packaged into the libary.zip file created.
For example, my scripts are located in src/, and the external data is located in src/data/. I've specified the include_files property in the build_exe_options, but this only copies the directory and files into the built directory; it doesn't add them to library.zip, which is where the scripts end up looking for the files.
Even if I go in to the created library.zip and manually add the data directory, I receive the same error. Any idea how to get cxfreeze to package these external resources appropriately?
setup.py
from cx_Freeze import setup, Executable
build_exe_options = {"includes" : ["re"], "include_files" : ["data/table_1.txt", "data/table_2.txt"]}
setup(name = "My Script",
version = "0.8",
description = "My Script",
options = { "build_exe" : build_exe_options },
executables = [Executable("my_script.py")])
fileutil.py (where it tries to read the resource files)
def read_file(filename):
path, fl = os.path.split(os.path.realpath(__file__))
filename = os.path.join(path, filename)
with open(filename, "r") as file:
lines = [line.strip() for line in file]
return [line for line in lines if len(line) == 0 or line[0] != "#"]
... called with ...
read_file("data/table_1.txt")
Error Traceback
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\cx_Freeze\initscripts\Console3.py", line 2
7, in <module> exec(code, m.__dict__)
File "my_script.py", line 94, in <module>
File "my_script.py", line 68, in run
File "C:\workspaces\py\test_script\src\tables.py", line 12, in load_data
raw_gems = read_file("data/table_1.txt")
File "C:\workspaces\py\test_script\src\fileutil.py", line 8, in read_file
with open(filename, "r") as file:
FileNotFoundError: [Errno 2] No such file or directory:
'C:\\workspaces\\py\\test_script\\src\\build\\exe.win32-3.3\\library.zip\\data/table_1.txt'

The following structure worked for me:
|-main.py
|-src
|-utils.py (containing get_base_dir())
|-data
then refer to your data always relative to the location of main.py that you receive through the following function within the src directory:
import os, sys, inspect
def get_base_dir():
if getattr(sys,"frozen",False):
# If this is running in the context of a frozen (executable) file,
# we return the path of the main application executable
return os.path.dirname(os.path.abspath(sys.executable))
else:
# If we are running in script or debug mode, we need
# to inspect the currently executing frame. This enable us to always
# derive the directory of main.py no matter from where this function
# is being called
thisdir = os.path.dirname(inspect.getfile(inspect.currentframe()))
return os.path.abspath(os.path.join(thisdir, os.pardir))
If you include the data according to the cx_Freeze documentation, it will be in the same directory as the .exefile (i.e. not in the zipfile), which will work with this solution.