I'm trying to make a bash file executable via a python program. Right now it looks like this:
p = subprocess.Popen(chmod u+x, bashName)
bashName being the name of the bash file I'm making executable, and I'm receiving the error:
FileNotFoundError: [Errno 2] No such file or directory: 'chmod u+x
/home/#####/Desktop/music/addSong/bashFileName'
I've tried this and it didn't fare any better
subprocess.call('chmod u+x /home/stoplight25/Desktop/music/addSong/'+bashName)
I've tried reading the documentation on subprocess but it's a bit beyond my comprehension. Could someone explain how to make a file executable with subprocess.
Expected:
make a new bash file with the correct contents and name, make it executable
Result:
a bash file with the right contents and name but isn't executable.
You have to pass the arguments as a list, not as a string or python tries to pass the whole string with spaces & args & all to the system as the executable (or use shell=True which I don't recommend). Also check return code just in case:
subprocess.check_call(['chmod','u+x','/home/stoplight25/Desktop/music/addSong/'+bashName])
Or you could use pure python to access the file permissions (get file current permissions, add user execute mask, apply os.chmod):
import os
my_file = os.path.join('/home/stoplight25/Desktop/music/addSong',bashName)
new_mode = os.stat(my_file).st_mode | 0o100
os.chmod(my_file,new_mode)
This should work:
import subprocess
command = 'chmod u+x /home/stoplight25/Desktop/music/addSong/' + bashName
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
I am trying to execute the following command programatically:
~$ obabel -:"ccco" -O /home/user/output.png
obabel is a chemistry library, and basically if the string in the "" is complete nonsense chemically, it won't be able to generate the PNG file, and if it is a legitimate chemical structure the PNG file will be generated. This works in the terminal.
However, if I call the same command with Python, PNG files are generated for complete nonsense input strings which don't generate a PNG when the command is executed in the terminal.
I'm using subprocess like this:
cmd = 'obabel -:"ccco" -O /home/user/output.png'
proc = sub.Popen([cmd], shell=True, stderr=sub.PIPE)
res = proc.communicate()
I have also tried this:
os.system(cmd)
And tried Python2 and Python3. This happens when running scripts from the terminal or iPython.
I have also tried using C++ and running the cmd like this:
std::string cmd = "obabel -:\"ccco\" -O /home/user/output.png";
system(cmd.c_str());
By default Popen expects a list of string arguments, but if you pass shell=True, you can supply the command as a simple string (it will be executed in a shell). Currently you are passing in a list with one string that contains the entirety of the command, instead you can use either of these:
proc = subprocess.Popen('obabel -:"ccco" -O output.png', shell=True, stderr=subprocess.PIPE)
proc = subprocess.Popen(['obabel', '-:ccco', '-O', 'output.png'], stderr=subprocess.PIPE)
Escaping the SMILES string with quotes seems to be done to protect it from the shell, and you don't need it when passing the input directly (otherwise the " characters will be a part of the string and cause invalid syntax).
I need to run the command date | grep -o -w '"+tz+"'' | wc -w using Python on my localhost. I am using subprocess module for the same and using the check_output method as I need to capture the output for the same.
However it is throwing me an error :
Traceback (most recent call last):
File "test.py", line 47, in <module>
check_timezone()
File "test.py", line 40, in check_timezone
count = subprocess.check_output(command)
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception-
OSError: [Errno 2] No such file or directory
You have to add shell=True to execute a shell command. check_output is trying to find an executable called: date | grep -o -w '"+tz+"'' | wc -w and he cannot find it. (no idea why you removed the essential information from the error message).
See the difference between:
>>> subprocess.check_output('date | grep 1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.4/subprocess.py", line 603, in check_output
with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
File "/usr/lib/python3.4/subprocess.py", line 848, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.4/subprocess.py", line 1446, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'date | grep 1'
And:
>>> subprocess.check_output('date | grep 1', shell=True)
b'gio 19 giu 2014, 14.15.35, CEST\n'
Read the documentation about the Frequently Used Arguments for more information about the shell argument and how it changes the interpretation of the other arguments.
Note that you should try to avoid using shell=True since spawning a shell can be a security hazard (even if you do not execute untrusted input attacks like Shellshock can still be performed!).
The documentation for the subprocess module has a little section about replacing the shell pipeline.
You can do so by spawning the two processes in python and use subprocess.PIPE:
date_proc = subprocess.Popen(['date'], stdout=subprocess.PIPE)
grep_proc = subprocess.check_output(['grep', '1'], stdin=date_proc.stdout, stdout=subprocess.PIPE)
date_proc.stdout.close()
output = grep_proc.communicate()[0]
You can write some simple wrapper function to easily define pipelines:
import subprocess
from shlex import split
from collections import namedtuple
from functools import reduce
proc_output = namedtuple('proc_output', 'stdout stderr')
def pipeline(starter_command, *commands):
if not commands:
try:
starter_command, *commands = starter_command.split('|')
except AttributeError:
pass
starter_command = _parse(starter_command)
starter = subprocess.Popen(starter_command, stdout=subprocess.PIPE)
last_proc = reduce(_create_pipe, map(_parse, commands), starter)
return proc_output(*last_proc.communicate())
def _create_pipe(previous, command):
proc = subprocess.Popen(command, stdin=previous.stdout, stdout=subprocess.PIPE)
previous.stdout.close()
return proc
def _parse(cmd):
try:
return split(cmd)
except Exception:
return cmd
With this in place you can write pipeline('date | grep 1') or pipeline('date', 'grep 1') or pipeline(['date'], ['grep', '1'])
The most common cause of FileNotFound with subprocess, in my experience, is the use of spaces in your command. If you have just a single command (not a pipeline, and no redirection, wildcards, etc), use a list instead.
# Wrong, even with a valid command string
subprocess.run(['grep -o -w "+tz+"'])
# Fixed; notice also
subprocess.run(["grep", "-o", "-w", '"+tz+"'])
This change results in no more FileNotFound errors, and is a nice solution if you got here searching for that exception with a simpler command.
If you need a pipeline or other shell features, the simple fix is to add shell=True:
subprocess.run(
'''date | grep -o -w '"+tz+"'' | wc -w''',
shell=True)
However, if you are using python 3.5 or greater, try using this approach:
import subprocess
a = subprocess.run(["date"], stdout=subprocess.PIPE)
print(a.stdout.decode('utf-8'))
b = subprocess.run(["grep", "-o", "-w", '"+tz+"'],
input=a.stdout, stdout=subprocess.PIPE)
print(b.stdout.decode('utf-8'))
c = subprocess.run(["wc", "-w"],
input=b.stdout, stdout=subprocess.PIPE)
print(c.stdout.decode('utf-8'))
You should see how one command's output becomes another's input just like using a shell pipe, but you can easily debug each step of the process in python. Using subprocess.run is recommended for python > 3.5, but not available in prior versions.
The FileNotFoundError happens because - in the absence of shell=True - Python tries to find an executable whose file name is the entire string you are passing in. You need to add shell=True to get the shell to parse and execute the string, or figure out how to rearticulate this command line to avoid requiring a shell.
As an aside, the shell programming here is decidedly weird. On any normal system, date will absolutely never output "+tz+" and so the rest of the processing is moot.
Further, using wc -w to count the number of output words from grep is unusual. The much more common use case (if you can't simply use grep -c to count the number of matching lines) would be to use wc -l to count lines of output from grep.
Anyway, if you can, you want to avoid shell=True; if the intent here is to test the date command, you should probably replace the rest of the shell script with native Python code.
Pros:
The person trying to understand the program only needs to understand Python, not shell script.
The script will have fewer external dependencies (here, date) rather than require a Unix-like platform.
Cons:
Reimplementing standard Unix tools in Python is tiresome and sometimes rather verbose.
With that out of the way, if the intent is simply to count how wany times "+tz+" occurs in the output from date, try
p = subprocess.run(['date'],
capture_output=True, text=True,
check=True)
result = len(p.stdout.split('"+tz+"'))-1
The keyword argument text=True requires Python 3.7; for compatibility back to earlier Python versions, try the (misnomer) legacy synonym universal_newlines=True. For really old Python versions, maybe fall back to subprocess.check_output().
If you really need the semantics of the -w option of grep, you need to check if the characters adjacent to the match are not alphabetic, and exclude those which are. I'm leaving that as an exercise, and in fact would assume that the original shell script implementation here was not actually correct. (Maybe try re.split(r'(?<=^|\W)"\+tz\+"(?=\W|$)', p.stdout).)
In more trivial cases (single command, no pipes, wildcards, redirection, shell builtins, etc) you can use Python's shlex.split() to parse a command into a correctly quoted list of arguments. For example,
>>> import shlex
>>> shlex.split(r'''one "two three" four\ five 'six seven' eight"'"nine'"'ten''')
['one', 'two three', 'four five', 'six seven', 'eight\'nine"ten']
Notice how the regular string split() is completely unsuitable here; it simply splits on every whitespace character, and doesn't support any sort of quoting or escaping. (But notice also how it boneheadedly just returns a list of tokens from the original input:
>>> shlex.split('''date | grep -o -w '"+tz+"' | wc -w''')
['date', '|', 'grep', '-o', '-w', '"+tz+"', '|', 'wc', '-w']
(Even more parenthetically, this isn't exactly the original input, which had a superfluous extra single quote after '"+tz+"').
This is in fact passing | and grep etc as arguments to date, not implementing a shell pipeline! You still have to understand what you are doing.)
The question already has an answer above but just in case the solutions are not working for you; Please check the path itself and if all the environment variables are set for the process to locate the path.
what worked for me on python 3.8.10 (inspired by #mightypile solution here: https://stackoverflow.com/a/49986004/12361522), was removed splits of parametres and i had to enable shell, too:
this:
c = subprocess.run(["wc -w"], input=b.stdout, stdout=subprocess.PIPE, shell=True)
instead of:
c = subprocess.run(["wc", "-w"], input=b.stdout, stdout=subprocess.PIPE)
if anyone wanted to try my solution (at least for v3.8.10), here is mine:
i have directory with multiple files of at least 2 file-types (.jpg and others). i needed to count specific file type (.jpg) and not all files in the directory, via 1 pipe:
ls *.jpg | wc -l
so eventually i got it working like here:
import subprocess
proc1 = subprocess.run(["ls *.jpg"], stdout=subprocess.PIPE, shell=True)
proc2 = subprocess.run(['wc -l'], input=proc1.stdout, stdout=subprocess.PIPE, shell=True)
print(proc2.stdout.decode())
it would not work with splits:
["ls", "*.jpg"] that would make ls to ignore contraint *.jpg
['wc', '-l'] that would return correct count, but will all 3 outputs and not just one i was after
all that would not work without enabled shell shell=True
I had this error too and what worked for me was setting the line endings of the .sh file - that I was calling with subprocess - to Unix (LF) instead of Windows CRLF.
I have a format holding paths to files and command line arguments to pass to those files when they are opened in Windows.
For example I might have a path to a javascript file and a list of command line arguments to pass it, in such a case I want to open the javascript file in the same way you might with os.startfile and pass it the command line arguments - since the arguments are saved as a string I would like to pass it as a string but I can also pass it as a list if need be.
I am not quite sure what I should be using for this since a .js is not an executable, and thus will raise errors in Popen while startfile only takes verbs as its second command.
This problem can be extended to an arbitrary number of file extensions that need to be opened, and passed command line arguments, but will be interpreted by a true executable when opening.
If windows has registered the .js extension to open with wscript, you can do this, by leaving that decision up to the windows shell.
You can just use os.system() to do the same thing as you would do when you type it at the command prompt, for example:
import os
os.system('example.js arg1 arg2')
You can also use the start command:
os.system('start example.js arg1 arg2')
If you need more power, for example to get results, you can use subprocess.Popen(), but make sure to use shell=True (so that the shell can call the right application):
from subprocess import Popen
p = Popen('example.js arg1 arg2', shell=True)
# you can also do pass the filename and arguments separately:
# p = Popen(['example.js', 'arg1', 'arg2'], shell=True)
stdoutdata, stderrdata = p.communicate()
(Although this would probably require cscript instead of wscript)
If Windows doesn't have any default application to open the file with (or if it's not the one you want), well, you're on your own of course...