This question already has answers here:
subprocess.call
(3 answers)
Closed 4 years ago.
I wanted to create some folders with subfolders and have this python3 script that would read a csv file and create directories for me.
from subprocess import call
import csv
def csv_dict_reader(file_obj):
"""
Read a CSV file using csv.DictReader
"""
reader = csv.DictReader(file_obj, delimiter=',')
for line in reader:
sanitized_name = line["course"].replace(" ", "-").lower()
command = 'mkdir -p ' + sanitized_name + '/week_{1..'+ line["week"] +'}/{slides,exercise,assignment}'
call(command)
if __name__ == "__main__":
with open("courses.csv") as f_obj:
csv_dict_reader(f_obj)
I was expecting that it executes commands of this sort
mkdir -p algorithmic-toolbox/week_{1..6}/{slides,exercise,assignment}
which works fine from the terminal and creates a folder called algorithmic-toolbox with 6 folders from week1 to week6 and then each folder contains slides, exercises and assignment folders.
A sample csv would be like this:
course,week
Algorithmic Toolbox,6
Data Structures,4
Algorithms on Graphs,5
I am getting this error when I run the python script.
Traceback (most recent call last):
File "course-folder.py", line 18, in <module>
csv_dict_reader(f_obj)
File "course-folder.py", line 13, in csv_dict_reader
call(command)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 267, in call
with Popen(*popenargs, **kwargs) as p:
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'mkdir -p algorithmic-toolbox/week_{1..6}/{slides,exercise,assignment}': 'mkdir -p algorithmic-toolbox/week_{1..6}/{slides,exercise,assignment}'
Not sure why it says FilenotFoundError when I am creating directories with -p which is supposed to create the parent directories if they don't exist.
mkdir -p algorithmic-toolbox/week_{1..6}/{slides,exercise,assignment}
I believe parameter expansion used in the above command will not work without passing the shell=True argument to subprocess.call. This way Python will execute the command in an intermediate shell process.
If you want your script to be platform independent you could instead use os.mkdir and handle the logic in Python.
Related
As part of larger code, I am trying to make a function that calls to a latexmk compiler using subprocess, but I consistently get FileNotFoundError: [Errno 2] No such file or directory: 'latexmk': 'latexmk'
However, If I write the command directly in the terminal, everything works: latexmk --pdf test.tex
In case it is important, I am on MacOS Mojave 10.14.6, running python 3.6 spyder through anaconda
I have checked the following links:
https://askubuntu.com/questions/801493/python-subprocess-call-not-working-as-expected
OSError: [Errno 2] No such file or directory while using python subprocess in Django
Running Bash commands in Python
If anything there solves the problem, I missed it.
To make everyone's life easier, here's a link to a .tex file [you can use your own]:
https://drive.google.com/open?id=1DoJnvg2BmbRCzmRmqFYRVybyTQUtyS-h
Afer putting type latexmk to terminal it outputs:
latexmk is hashed (/Library/TeX/texbin/latexmk)
Here is the minimal reproducible example (you do need latexmk on your computer though):
import os, subprocess
def pdf(file_path):
cur_dir = os.getcwd()
dest_dir = os.path.dirname(file_path)
basename = os.path.basename(file_path)
os.chdir(dest_dir)
main_arg = [basename]
command = ["latexmk", "--pdf"] + main_arg
try:
output = subprocess.check_output(command)
except subprocess.CalledProcessError as e:
print(e.output.decode())
raise
os.chdir(cur_dir)
pdf("path to your .tex file")
I have a feeling that I am grossly misunderstanding the way subprocess works. Any ideas?
Update: In case neccessary, the full traceback:
Traceback (most recent call last):
File "<ipython-input-90-341a2810ccbf>", line 1, in <module>
pdf('/Users/sergejczan/Desktop/untitled folder/test.tex')
File "/Users/sergejczan/Desktop/Lab/subprocess error reproduction.py", line 23, in pdf
output = subprocess.check_output(command)
File "/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/anaconda3/lib/python3.6/subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "/anaconda3/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/anaconda3/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'latexmk': 'latexmk'
New Update
Changing the output = subprocess.check_output(command) line with the hardcoded envirnoment that I got from echo $PATH worked wonderfully.
output = subprocess.check_output(command,env = {'PATH': '/anaconda3/bin:/Users/sergejczan/anaconda3/bin:/Users/sergejczan/Desktop/Lab/anaconda2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin'})
Would you think that there is a way to make the code find the PATH automatically?
This question already has answers here:
Why can't I change directories using "cd" in a script?
(33 answers)
Closed 6 years ago.
I am trying to execute some shell commands using python :
The command is cd /home/n1603031f/Desktop/parsec/wd/
It works fine through the shell, but when executed through python it does not work :
path_to_wd = "/home/n1603031f/Desktop/parsec/wd/"
call(["cd",path_to_wd])
Error :
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
I need this command to work as the original command I want to execute is :
cd ./parsec/wd/ && tar -cf ../abcd.tar *
which works correctly only when you change directories to not create the top-level folders in the .tar file
Even if you had the right call to change directories it wouldn't accomplish what you want because each subprocess.call creates a separate process.
What you really need is the cwd argument to subprocess.Popen to say what directory you want to work in. Additionally you need to use os.listdir since the subprocess call won't go through a shell to expand the * glob. This is the right way to do what you're trying to do:
d = './parsec/wd'
subprocess.Popen(['tar', '-cf', '../abcd.tar'] + os.listdir(d), cwd=d).wait()
However, os.listdir will list hidden files as well, if you want to you can filter them out beforehand:
files = [f for f in os.listdir(d) if not f.startswith('.')]
If you really need to (and you don't,) you can use shell=True to get this to work with *. Though unless you're working with trusted input shell=True is widely considered a security vulnerability.
subprocess.Popen('tar -cf ../abcd.tar *', shell=True, cwd='./parsec/wd').wait()
If you need your python process to change it's current working directory, use
os.chdir('./parsec/wd')
This question already has answers here:
Can't execute shell script from python subprocess: permission denied
(2 answers)
Closed 6 years ago.
Currently I am testing a very simple piece of code. I simply want to write a python script that sets a variable and then pass that variable into a bash script for use.
Python Script:
from subprocess import check_call
a = str(3)
check_call(["/home/desktop/bash2pyTest/test.sh", a], shell=False)
Bash Script:
#!/bin/bash
echo "This number was sent from the py script: " $1
I have read other Q&As that are related to this topic; however, I am not finding a solution that I am conceptually understand; thus, the syntax above might be incorrect. I have tried a few other methods as well; however, I keep receiving the following error:
Traceback (most recent call last):
File "/home/cassandra/desktop/bash2pyTest/callVar.py", line 3, in <module>
check_call(["/home/cassandra/desktop/bash2pyTest/test.sh", a], shell=False)
File "/usr/lib64/python2.7/subprocess.py", line 537, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
Process finished with exit code 1
Any help would be greatly appreciated. I'm stumped.
Try
chmod +x /home/desktop/bash2pyTest/test.sh
in shell. The file you are trying to execute is not executable.
Or another option in python script:
check_call(["sh","/home/desktop/bash2pyTest/test.sh", a], shell=False)
The error says that permission is denied. I tested your code, and it works fine on my machine. However, you will need to be sure that the user running the command has sufficient privileges for the test.sh script. Most importantly, be sure that test.sh has execute permissions set. That's often the most easily missed permission.
I am trying to convert a pdf document to text document using pdftotext software.
I need to call this application inc command prompt from python script to convert the file.
I have following code:
import os
import subprocess
path = "C:\\Users\\..."
pdffname = "pdffilename.pdf"
txtfname = "txtfilename.txt"
subprocess.call(['pdftotext', '-layout',
os.path.join(path, pdffname),
os.path.join(path, txtfname)])
When I run this code, I get error
File "C:/Users/.../code-1.py", line 44, in <module>
os.path.join(path, txtfname)])
File "C:\Anaconda\lib\subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Anaconda\lib\subprocess.py", line 710, in __init__
errread, errwrite)
File "C:\Anaconda\lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Can you help to call pdftotext application from python to convert pdf to text file.
I had this same error, except with Popen. I fixed it by providing the full path to pdftotext.exe in the subprocess call. Don't forget to escape your backslashes.
I do not know much about Anaconda, and I have not tested this myself, but I believe Conda may have an issue referencing scripts on Windows: fix references to scripts on windows
I am trying to automate some tasks - the process requires I call some exe's and pass parameters. The particular directories for the exe's are in the PATH variable for windows. However, I consistently get a
WindowsError: [Error 2] The system cannot find the file specified
My current workaround is to set the os.cwd to the directory with the exe but that imposes some other limits on how we distribute the code. I want to note that in every case if I start a cmd window and type the same code I am passing to subprocess.check_output the code works no matter what directory I am in on the computer.
Just to be clear I am afraid for example of trying to automate a WinRAR task and WinRAR.exe is in a different folder on their computer.
Okay in response to the comment below here is the input and the output after I changed the cwd to root (c:)
The call to subprocess
rarProcess = check_output('''WinRAR a -r -v700m -sfx -agYYYYMMDD-NN -iiconD:\\RarResources\\de96.ico -iimgd:\\RarResources\\accounting2013.bmp d:\\testFTP\\compressed_test_ d:\\files_to_compress''')
and here is the Traceback message in all of it's glory
Traceback (most recent call last):
File "<pyshell#93>", line 1, in <module>
rarProcess = check_output('''WinRAR a -r -v700m -sfx -agYYYYMMDD-NN -iiconD:\\RarResources\\de96.ico -iimgd:\\RarResources\\accounting2013.bmp d:\\testFTP\\compressed_test_ d:\\files_to_compress''')
File "C:\Program Files (x86)\python\lib\subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "C:\Program Files (x86)\python\lib\subprocess.py", line 679, in __init__
errread, errwrite)
File "C:\Program Files (x86)\python\lib\subprocess.py", line 893, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Now I can't prove that this is not a hypothetical question/problem. I get the intended results when I use the same command (adjusting for path separators) through the cmd window and if I change the directory to the directory with the exe before running the command as pasted above.
You don't need to set os.cwd and run the process. Instead you pass the location of your "Winrar.exe" file to the subprocess as a dict.
proc = subprocess.Popen(args, env={'PATH': '/path/to/winrar.exe'})