I'm having some strange issues using subprocess.check_output(). At first I was just using subprocess.call() and everything was working fine. However when I simply switch out call() for check_output(), I receive a strange error.
Before code (works fine):
def execute(hosts):
''' Using psexec, execute the batch script on the list of hosts '''
successes = []
wd = r'c:\\'
file = r'c:\\script.exe'
for host in hosts:
res = subprocess.call(shlex.split(r'psexec \\\\%s -e -s -d -w %s %s' % (host,wd,file)))
if res.... # Want to check the output here
successes.append(host)
return successes
After code (doesn't work):
def execute(hosts):
''' Using psexec, execute the batch script on the list of hosts '''
successes = []
wd = r'c:\\'
file = r'c:\\script.exe'
for host in hosts:
res = subprocess.check_output(shlex.split(r'psexec \\\\%s -e -s -d -w %s %s' % (host,wd,file)))
if res.... # Want to check the output here
successes.append(host)
return successes
This gives the error:
I couldnt redirect this because the program hangs here and I can't ctrl-c out. Any ideas why this is happening? What's the difference between subprocess.call() and check_output() that could be causing this?
Here is the additional code including the multiprocessing portion:
PROCESSES = 2
host_sublists_execute = [.... list of hosts ... ]
poolE = multiprocessing.Pool(processes=PROCESSES)
success_executions = poolE.map(execute,host_sublists_execute)
success_executions = [entry for sub in success_executions for entry in sub]
poolE.close()
poolE.join()
Thanks!
You are encountering Python Issue 9400.
There is a key distinction you have to understand about subprocess.call() vs subprocess.check_output(). subprocess.call() will execute the command you give it, then provide you with the return code. On the other hand, subprocess.check_output() returns the program's output to you in a string, but it tries to do you a favor and check the program's return code and will raise an exception (subprocess.CalledProcessError) if the program did not execute successfully (returned a non-zero return code).
When you call pool.map() with a multiprocessing pool, it will try to propagate exceptions in the subprocesses back to main and raise an exception there. Apparently there is an issue with how the subprocess.CalledProcessError exception class is defined, so the pickling fails when the multiprocessing library tries to propagate the exception back to main.
The program you're calling is returning a non-zero return code, which makes subprocess.check_output() raise an exception, and pool.map() can't handle it properly, so you get the TypeError that results from the failed attempt to retrieve the exception.
As a side note, the definition of subprocess.CalledProcessError must be really screwed up, because if I open my 2.7.6 terminal, import subprocess, and manuallly raise the error, I still get the TypeError, so I don't think it's merely a pickling problem.
Related
I am making a program that adds additional functionality to the standard command shell in Windows. For instance, typing google followed by keywords will open a new tab with Google search for those keywords, etc. Whenever the input doesn't refer to a custom function I've created, it gets processed as a shell command using subprocess.call(rawCommand, shell=True).
Since I'd like to anticipate when my input isn't a valid command and return something like f"Invalid command: {rawCommand}", how should I go about doing that?
So far I've tried subprocess.call(rawCommand) which also return the standard output as well as the exit code. So that looks like this:
>>> from subprocess import call
>>> a, b = call("echo hello!", shell=1), call("xyz arg1 arg2", shell=1)
hello!
'xyz' is not recognized as an internal or external command,
operable program or batch file.
>>> a
0
>>> b
1
I'd like to simply recieve that exit code. Any ideas on how I can do this?
Should you one day want deal with encoding errors, get back the result of the command you're running, have a timeout or decide which exit codes other than 0 may not trigger errors (i'm looking at you, java runtime !), here's a complete function that does that job:
import os
from logging import getLogger
import subprocess
logger = getLogger()
def command_runner(command, valid_exit_codes=None, timeout=300, shell=False, encoding='utf-8',
windows_no_window=False, **kwargs):
"""
Whenever we can, we need to avoid shell=True in order to preseve better security
Runs system command, returns exit code and stdout/stderr output, and logs output on error
valid_exit_codes is a list of codes that don't trigger an error
windows_no_window will hide the command window (works with Microsoft Windows only)
Accepts subprocess.check_output arguments
"""
# Set default values for kwargs
errors = kwargs.pop('errors', 'backslashreplace') # Don't let encoding issues make you mad
universal_newlines = kwargs.pop('universal_newlines', False)
creationflags = kwargs.pop('creationflags', 0)
if windows_no_window:
creationflags = creationflags | subprocess.CREATE_NO_WINDOW
try:
# universal_newlines=True makes netstat command fail under windows
# timeout does not work under Python 2.7 with subprocess32 < 3.5
# decoder may be unicode_escape for dos commands or utf-8 for powershell
output = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=shell,
timeout=timeout, universal_newlines=universal_newlines, encoding=encoding,
errors=errors, creationflags=creationflags, **kwargs)
except subprocess.CalledProcessError as exc:
exit_code = exc.returncode
try:
output = exc.output
except Exception:
output = "command_runner: Could not obtain output from command."
if exit_code in valid_exit_codes if valid_exit_codes is not None else [0]:
logger.debug('Command [%s] returned with exit code [%s]. Command output was:' % (command, exit_code))
if isinstance(output, str):
logger.debug(output)
return exc.returncode, output
else:
logger.error('Command [%s] failed with exit code [%s]. Command output was:' %
(command, exc.returncode))
logger.error(output)
return exc.returncode, output
# OSError if not a valid executable
except (OSError, IOError) as exc:
logger.error('Command [%s] failed because of OS [%s].' % (command, exc))
return None, exc
except subprocess.TimeoutExpired:
logger.error('Timeout [%s seconds] expired for command [%s] execution.' % (timeout, command))
return None, 'Timeout of %s seconds expired.' % timeout
except Exception as exc:
logger.error('Command [%s] failed for unknown reasons [%s].' % (command, exc))
logger.debug('Error:', exc_info=True)
return None, exc
else:
logger.debug('Command [%s] returned with exit code [0]. Command output was:' % command)
if output:
logger.debug(output)
return 0, output
Usage:
exit_code, output = command_runner('whoami', shell=True)
Some shells have a syntax-checking mode (e.g., bash -n), but that’s the only form of error that’s separable from “try to execute the commands and see what happens”. Defining a larger class of “immediate” errors is a fraught proposition: if echo hello; ./foo is invalid because foo can’t be found as a command, what about false && ./foo, which will never try to run it, or cp /bin/ls foo; ./foo, which may succeed (or might fail to copy)? What about eval $(configure_shell); foo which might or might not manipulate PATH so as to find foo? What about foo || install_foo, where the failure might be anticipated?
As such, anticipating failure is not possible in any meaningful sense: your only real option is to capture the command’s output/error (as mentioned in the comments) and report them in some useful way.
I'm creating a Python program to monitor the processes on a server once an hour, to see if it can return the PID. To do this, I made a function that uses subprocess to call pgrep -f on any name submitted to it. If it returns a process, the function evaluates to true; otherwise, it returns false.
import subprocess
import psutil
def check_essentials(name):
child = subprocess.Popen(['pgrep', '-f', name], stdout=subprocess.PIPE, shell=False)
response = child.communicate()[0]
pid = response.split()
if len(pid) == 0:
print("unable to find PID")
return False
else:
print("PID is %s" % pid)
return True
essentialApps = ['ProfileService','aflaf']
sendEmail=False
for x in essentialApps:
check_essentials(x)
if check_essentials == False:
print("Unable to find PID for %s. Sending email alert" % x)
sendEmail = True
else:
print("Found PID for %s" % x)
I then set up a for loop to have it iterate through a list of process names(essentialApps) and see if it can return anything for them. If not, sendEmail is set to true.
In testing this, however, I'm finding that the else statement is always being called, regardless of whether or not the app exists. When I call this program (python alert.py), I get the following output:
PID is [b'11111']
Found PID for ProfileService
unable to find PID #This is expected
Found PID for aflaf #This should be "Unable to find PID for aflaf"
I'm sure it is something simple, but can anyone tell me why it is not evaluating check_essential correctly?
Also, is there anyway to do this with psutil? I'm reading that this should be used over subprocess, but I'm not able to find anyway to specifically mimic pgrep -f name or ps -aux | grep name. Thats important, as I have multiple Java applications running on the machine, and the program name that psutil seemed to see was always 'java', not 'ProfileService'.
You're not using the result of the function, you're checking if the check_essentials function itself is False.
Which it's not, because it's a function.
You need to you the result of check_essentials in your condition, check_essentials is always True, because it is a Python object:
for x in essentialApps:
check_result = check_essentials(x)
if check_result == False:
I'm trying to run some commands using the python library subprocess. Some of my commands might get stuck in loops and block the python script. Therefore I'm using check_output() with a timeout argument to limit the commands in time. When the command takes too much time, the function raise a TimeoutExpired error. What I want to do is get what the command has been able to run before being killed by the timeout.
I've except the error and tried "except sp.TimeoutExpired as e:". I read on the doc that if I do e.output it should give me what I want. "Output of the child process if this exception is raised by check_output(). Otherwise, None.". However I don't get anything in the output.
Here is what I did:
import subprocess as sp
try:
out = sp.check_output('ls', stderr=sp.STDOUT, universal_newlines=True, timeout=1)
except sp.TimeoutExpired as e:
print ('output: '+ e.output)
else:
return out
Let say the folder I'm working with is huge and so 1 second isn't enough to ls all its files. Therefore the TimeoutExpired error will be raised. However, I'd like to store what the script was able to get at least. Does someone have an idea?
Found a solution, posting it here in case someone is interested.
In Python 3, the run method allows to get the output.
Using the parameters as shown in the example, TimeoutExpired returns the output before the timeout in stdout:
import subprocess as sp
for cmd in [['ls'], ['ls', '/does/not/exist'], ['sleep', '5']]:
print('Running', cmd)
try:
out = sp.run(cmd, timeout=3, check=True, stdout=sp.PIPE, stderr=sp.STDOUT)
except sp.CalledProcessError as e:
print(e.stdout.decode() + 'Returned error code ' + str(e.returncode))
except sp.TimeoutExpired as e:
print(e.stdout.decode() + 'Timed out')
else:
print(out.stdout.decode())
Possible output:
Running ['ls']
test.py
Running ['ls', '/does/not/exist']
ls: cannot access '/does/not/exist': No such file or directory
Returned error code 2
Running ['sleep', '5']
Timed out
I hope it helps someone.
How can I get the exist status (if command succeed or not) of any line in Python?
For example in bash, $? will tell me the last exist status of any command.
I need it to know if my connection to FTP server was successful or not.
Have you tried using a try/catch? If there was an error while executing the command, an exception will be raised. You can retrieve it with the sys module.
Example code:
import sys
try:
run_command()
except:
e = sys.exc_info()[0]
print(e)
If it is a function you call, it should have a retrun code you can collect like
retVal = doSomething()
Then you can check what happened.
I wrote a python script to run a command called "gtdownload" on a bunch of files with multiprocessing. The function "download" is where I am having trouble with.
#/usr/bin/env python
import os, sys, subprocess
from multiprocessing import Pool
def cghub_dnld_file(file1, file2, file3, np):
<open files>
<read in lines>
p = Pool(int(np))
map_args = [(line.rstrip(),name_lines[i].rstrip(),bar_lines[i].rstrip()) for i, line in enumerate(id_lines)]
p.map(download_wrapper,map_args)
def download(id, name, bar):
<check if file has been downloaded, if not download>
<.....>
link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
dnld_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
subprocess.call(dnld_cmd,shell=True)
def download_wrapper(args):
return download(*args)
def main():
<read in arguments>
<...>
cghub_dnld_file(file1,file2,file3,threads)
if __name__ == "__main__":
main()
If this file does not exist in the database, gtdownload would quit which also kills my python job with the following error:
Traceback (most recent call last):
File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 102, in <module>
main()
File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 98, in main
cghub_dnld_file(file1,file2,file3,threads)
File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 22, in cghub_dnld_file
p.map(download_wrapper,map_args)
File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 250, in map
return self.map_async(func, iterable, chunksize).get()
File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 554, in get
raise self._value
OSError: [Errno 2] No such file or directory
The actual error message from gtdownload :
Welcome to gtdownload-3.8.5a.
Ready to download
Communicating with GT Executive ...
Headers received from the client: 'HTTP/1.1 100 Continue
HTTP/1.1 404 Not Found
Date: Tue, 29 Jul 2014 18:49:57 GMT
Server: Apache/2.2.15 (Red Hat and CGHub)
Strict-Transport-Security: max-age=31536000
X-Powered-By: PHP/5.3.3
Content-Length: 669
Connection: close
Content-Type: text/xml
'
Error: You have requested to download a uuid which either does not exist within the system, or has not yet reached the 'live' state. The requested action will not be performed. Please double check the supplied uuid or contact thelpdesk for further assistance.
I would like the script to skip the one that does not exist and start gtdownload on the next one. I tried to output the stderr of subprocess.call to a pipe and see if there is the "error" keyword. But it seems it stops at the exact subprocess.call command. Same thing with os.system.
I made a MCV case without the multiprocessing and subprocess did not kill the main process at all. Looks like multiprocessing messes things up although I had it run with 1 thread just for testing.
#!/usr/bin/env python
import subprocess
#THis is the id that gtdownload had problem with
id = "df1e073f-4485-4d5f-8659-cd8eaac17329"
link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
print dlnd_cmd
subprocess.call(dlnd_cmd,shell=True)
print "done"
Clearly multiprocessing conflicts subprocess.call but it is not clear to me why.
What is the best way to avoid the failure of subprocess killing the main process?
Handle the exception in some appropriate way and move on, of course.
try:
subprocess.call(dlnd_cmd)
except OSError as e:
print 'failed to download: {!r}'.format(e)
However, this may not be appropriate here. The kinds of exceptions that subprocess.call raises are usually not transient things that you can just log and work around; if it's not working now, it will continue to not work forever until you fix the underlying problem (a bug in your script, or gtdownload not being installed right, or whatever).
For example, if the code you showed us is your actual code:
dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
subprocess.call(dlnd_cmd)
… then this is guaranteed to raise an OSError for the reason explained in dano's answer: call (without shell=True) will try to take that entire string—spaces, shell-redirection, etc.—as the name of an executable program to find on your $PATH. And there is no such program. So it will raise an OSError(errno.ENOENT). (Which is exactly what you're seeing.) Just logging that doesn't do you any good; it's a good thing that your entire process is exiting, so you can debug that problem.
subprocess.call should not kill the main process. Something else must be wrong with your script or your conclusions about the script's behaviour is wrong. Did you try printing some trace output after the subprocess call?
You have to use shell=True to use subprocess.call with a string for an argument (and with shell redirection):
subprocess.call(dlnd_cmd, shell=True)
Without shell=True, subprocess tries to treat your entire command string like a single executable name, which of course doesn't exist, and leads to the No such file or directory exception.
See this answer for more info on when to use a string vs. when to use a sequence with subprocess.