Develop Reference

Python os.chdir() not working with path with spaces and special characters

I'm trying to run some files sequentially (scrape.py, tag.py, save.py and select.py) that are located in a folder named 'cargen'. However, when I try to make os.chdir(path) to access this 'cargen' folder, I'm getting an Exception message because in the path to 'cargen' folder there's a directory with spaces and special characters on it.
The code to run the files sequentially looks like the following:
import os
path = "C:/Users/Desktop/repl/Special Cháracters/cargen/"
os.chdir(path)
directory = 'C:/Users/Desktop/scrap/'
files = ['scrape', 'tag', 'save', 'select']
if __name__ == '__main__':
if not os.path.isdir(directory):
os.mkdir(directory)
[os.system('python ' + path + f'{file}.py ' + directory) for file in files]
The message that I'm getting looks like this:
python: can't open file 'C:/Users/Desktop/repl/\Special': [Errno 2] No such file or directory
I've tried to move to files to a path where there aren't any special characters or spaces in the path and the code works perfectly. Could anybody please help me with this? How should I define the path to 'cargen' to be able to access these files?
NOTE: I'm using Windows 10 with Anaconda

When windows reads comands it uses spaces as separators. e.g.:
my folder -> command1: my, command2: folder
But you can join them adding quotation marks you can join the separated commands in one.
"my folder" -> command1: my folder
I think something similiar is appening to you, try to declare your path like that:
'"your path"'

Finally, the problem wasn't the os.chdir(). The problem was related to os.system() function as #user2357112 mentioned.
The os.system() command was giving me problems due to the spaces in the path. Thus, as the files are all located in the same folder, I have just eliminated that reference to path, resulting in something like:
import os
path = os.getcwd()
os.chdir(path)
directory = 'C:/Users/Desktop/scrap/'
files = ['scrape', 'tag', 'save', 'select']
if __name__ == '__main__':
if not os.path.isdir(directory):
os.mkdir(directory)
[os.system('python ' + f'{file}.py ' + directory) for file in files]

How to convert multiple .doc files to .docx using antiword?

This manual command is working:
!antiword "test" > "test.docx"
but the following script convert files to empty .docx files:
for file in os.listdir(directory):
subprocess.run(["bash", "-c", "antiword \"$1\" > \"$1\".docx", "_", file])
also it stores the .docx file in the previous directly e-g file is in \a\b this command will store the files to \a
I have tried many different ways including running directly on terminal adn bash loops. ony the manual way works.

Something like this should work (adjust dest_path etc. accordingly).
import os
import shlex
for filename in os.listdir(directory):
if ".doc" not in filename:
continue
path = os.path.join(directory, filename)
dest_path = os.path.splitext(path)[0] + ".txt"
cmd = "antiword %s > %s" % (shlex.quote(path), shlex.quote(dest_path))
print(cmd)
# If the above seems to print correct commands, add:
# os.system(cmd)

Passing a variable in shell/python

I'm using a python script where I'm using a shell command to copy from local to hdfs.
import os
import logging
import subprocess
filePath = "/tmp"
keyword = "BC10^Dummy-Segment"
for root, dirs, files in os.walk(filePath):
for file in files:
if keyword in file:
subprocess.call(["hadoop fs -copyFromLocal /tmp/BC10%5EDummy-Segment* /user/app"], shell=True)
subprocess.call(["hadoop fs -rm /tmp/BC10%5EDummy-Segment*"], shell=True)
I'm seeing this error:
copyFromLocal: `/tmp/BC10^Dummy-Segment*': No such file or directory
rm: `/tmp/BC10^Dummy-Segment_2019': No such file or directory
Updated code:
import glob
import subprocess
import os
from urllib import urlencode, quote_plus
filePath = "/tmp"
keyword = "BC10^Dummy-Segment"
wildcard = os.path.join(filePath, '{0}*'.format(keyword))
print(wildcard)
files = [urlencode(x, quote_via=quote_plus) for x in glob.glob(wildcard)]
subprocess.check_call(["hadoop", "fs", "-copyFromLocal"] + files + ["/user/app"])
#subprocess.check_call(["hadoop", "fs", "-rm"] + files)
Seeing error when I run:
Traceback (most recent call last):
File "ming.py", line 11, in <module>
files = [urlencode(x, quote_via=quote_plus) for x in glob.glob(wildcard)]
TypeError: urlencode() got an unexpected keyword argument 'quote_via'

I'm guessing you are URL-encoding the path to pass it properly to Hadoop, but in doing so you basically hide it from the shell. There really are no files matching the wildcard /tmp/BC10%5EDummy-Segment* where % etc are literal characters.
Try handling the glob from Python instead. With that, you can also get rid of that pesky shell=True; and with that change, it is finally actually correct and useful to pass the commands as a list of strings (never a list of a singe space-separated string, and with shell=True, don't pass a list at all). Notice also the switch to check_call so we trap errors and don't delete the source files if copying them failed. (See also https://stackoverflow.com/a/51950538/874188 for additional rationale.)
import glob
import subprocess
import os
from urllib import quote_plus
filePath = "/tmp"
keyword = "BC10^Dummy-Segment"
wildcard = os.path.join(filePath, '{0}*'.format(keyword))
files = [quote_plus(x) for x in glob.glob(wildcard)]
subprocess.check_call(["hadoop", "fs", "-copyFromLocal"] + files + ["/user/app"])
subprocess.check_call(["hadoop", "fs", "-rm"] + files)
This will not traverse subdirectories; but neither would your attempt with os.walk() do anything actually useful if it found files in subdirectories. If you actually want that to happen, please explain in more detail what the script should do.

Problems running batch files with Python

I am rather new to Python and have been trying to run a .cmd file with it, but it won't run it from the correct location. My file Run_setup.cmd, is setting up another a different software with a bunch of related files and so I have sequestered them to their own folder for my sanity.
Currently I can get the .cmd file to run from the same location as my source code. I know I am messing up the file path for it with cwd=r'%s' based on what the documentation says, but I don't get how.
If cwd is not None, the function changes the working directory to cwd before executing the child. cwd can be a str and path-like object. In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path.
I currently have it using cwd=r' C:\LargeFolder\Files\CorrectFolder' based off this post, and it seems that it works for any file path, but I can't seem to get it to work for me.
from subprocess import Popen
def runCmdfile():
# File Path to source code: 'C:\LargeFolder\Files'
myDir = os.getcwd()
# File Path to .cmd file: 'C:\LargeFolder\Files\CorrectFolder'
myDir = myDir + '\CorrectFolder'
runThis = Popen('Run_setup.cmd', cwd=r'%s' % myDir)
stdout, stderr = runThis.communicate()
What am I missing here, and furthermore what is the purpose of using cwd=r' ' ?

this one works for me:
def runCmdfile():
# File Path to source code: 'C:\LargeFolder\Files'
myDir = os.getcwd()
# File Path to .cmd file: 'C:\LargeFolder\Files\CorrectFolder'
myDir = os.path.join(myDir, 'CorrectFolder')
# Popen does not take cwd into account for the file to execute
# so we build the FULL PATH on our own
runThis = Popen(os.path.join(myDir, 'Run_setup.cmd'), cwd=myDir)
stdout, stderr = runThis.communicate()

the parameter is cwd=. The r"" part only needs to exist in the definition of your string, to have a raw string and make python ignore special sequences using backslashes.
Since your string comes from os.getcwd, you don't need it.
def runCmdfile():
# File Path to source code: 'C:\LargeFolder\Files'
myDir = os.getcwd()
# File Path to .cmd file: 'C:\LargeFolder\Files\CorrectFolder'
myDir = os.path.join(myDir, 'CorrectFolder')
runThis = Popen('Run_setup.cmd', cwd=myDir)
stdout, stderr = runThis.communicate()

Your error is due to not escaping your \.
You need to escape your "\" where you're adding in your subfolder and then you should be good to go.
myDir = myDir + '\CorrectFolder'
should be
myDir = myDir + '\\CorrectFolder'

os.path.join issue Windows

I am having some issues with os.path.join and a Windows system. I have created a script that recursively reads files containing unstructured JSON data, creates a directory named "converted_json", and prints the content of each unstructured JSON file in a structured format into a new file within the "converted_json" directory.
I have tested the script below on macOS and upon execution, the structured JSON data is printed to new files and the new files are output to the "converted_json" directory. However, when I execute the script on a Windows system, the JSON data is printed to new files, but the files are not output to the "converted_json" directory.
Essentially, the following os.path.join code does not appear to be working on Windows in the following section:
conv_json = open(os.path.join(converted_dir, str(file_name[-1]) + '_converted'), 'wb')
The files are created, however they are not stored within the "converted_json" directory that is specified by the converted_dir variable.
The following output is from printing the "conv_json" variable:
open file 'C:\Users\test\Desktop\test\file_name.json.gz.json_converted', mode 'wb' at 0x0000000002617930
As seen from above, the file path contained within the "conv_json" variable does not contain the "converted_json" directory (it should be there from using os.path.join and the converted_dir variable.
Any assistance as to how to get the structured data to output to the "converted_json" directory would be greatly appreciated.
Code below:
argparser = argparse.ArgumentParser()
argparser.add_argument('-d', '--d', dest='dir_path', type=str, default=None, required=True, help='Directory path to Archive/JSON files')
args = argparser.parse_args()
dir_path = args.dir_path
converted_dir = os.path.join(dir_path, 'converted_json')
os.mkdir(converted_dir, 0777)
for subdir1, dirs1, files1 in os.walk(dir_path):
for file in files1:
try:
if file.endswith(".json"):
file = open(os.path.join(subdir1, file))
file_name = str.split(file.name, '/')
conv_json = open(os.path.join(converted_dir, str(file_name[-1]) + '_converted'), 'wb')
conv_json.write('#################################################################################################################################')
conv_json.write('\n')
conv_json.write('File Name: ' + file_name[-1])
conv_json.write('\n')
conv_json.write('#################################################################################################################################')
conv_json.write('\n')
parsed_json = json.load(file)
s = cStringIO.StringIO()
pprint.pprint(parsed_json, s)
conv_json.write(s.getvalue())
conv_json.close()
except:
print 'JSON Files Not Found'
print 'JSON Processing Completed: ' + str(datetime.datetime.now())

I think that this line is bad on Windows:
file_name = str.split(file.name, '/')
The split on '/' will not split at all. You should use os.path.sep instead.
I think os.path.join reacts so confusing since the second part you try to join is already a full file path (since the split failed).