I apologize if this is a dumb question, however, I am not very fluent in Python yet.
In regards to the Python Subprocess function...
I've seen that when you use sp = subprocess.Popen(...) people close/terminate it when it's finished running the command. Example:
sp = subprocess.Popen(['powershell.exe', '-ExecutionPolicy', 'Unrestricted', 'cp', '-r', 'ui', f'..\\{name}'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd='UI Boiler')
sp.wait()
sp.terminate()
However, my question is, do you need to close any subprocess.run() functions? Or do those processes close automatically once they are finished running their commands?
The project I am working on requires a lot of those to be run and I do not wish to have 10+ shells/powershells/processes open because I didn't close them.
**Yes, on both windows and posix implementations, subprocess.run() as well as subprocess.call() will both block until completion e.g. via Process.wait() internally. Since this is a blocking call it will wait until process completion to return, so you should not need to do anything special to close processes.
To wit, here's the relevant snippets from subprocess source in cpython-3.10 (amended for brevity):
def call(*popenargs, timeout=None, **kwargs):
"""..."""
with Popen(*popenargs, **kwargs) as p:
try:
return p.wait(timeout=timeout)
except: # Including KeyboardInterrupt, wait handled that.
p.kill()
raise
# ...
def run(*popenargs,
input=None, capture_output=False, timeout=None, check=False, **kwargs):
"""..."""
# ...
with Popen(*popenargs, **kwargs) as process:
# communicate (as well as the with statement will both wait() internally
try:
stdout, stderr = process.communicate(input, timeout=timeout)
except TimeoutExpired as exc:
process.kill()
# ... additional handling here
raise
except: # Including KeyboardInterrupt, communicate handled that.
process.kill()
# We don't call process.wait() as .__exit__ does that for us.
raise
retcode = process.poll()
if check and retcode:
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
return CompletedProcess(process.args, retcode, stdout, stderr)
If however you want to have more control over if and when the subprocess blocks, e.g. such that you can run other code on the same thread while the other process is running, then you should use the internal sp = supbrocess.Popen() directly
As to the call to terminate() - note this would always be a no-op in your example of waiting first and then terminating without a catch. Reason being, terminate as implemented will never even bother sending a TERM signal to your subprocess because wait() is a blocking call that will not exit until the child process completes or throws an exception (e.g. on timeout). Again, if you are calling subprocesses that might hang and you want to run in the background, e.g. so you can terminate yourself if it hasn't completed after a certain amount of time e.g. you will need to manage the subprocess yourself and subprocess.run() is probably not suitable for your needs.
A note on terminate(): subprocess.run and subprocess.call do both properly support automatically sending a kill to an erroring or timed out process, so if that's all you need, you can stick with one of those. In fact, on windows, kill() and terminate() are identical. On posix, a SIGKILL will be sent if the subprocess throws or times out.
If on POSIX, you would want to send a SIGTERM instead so that you give the subprocess the opportunity to try to terminate gracefully or cleanup then again it's best to interact with the Process object directly via Popen
Related
I am calling a shell script fom within Python, that spawns multiple child processes. I want to terminate that process and all of its children, if it did not finish after two minutes.
Is there any way I can do that with subprocess.run or do I have to go back to using Popen? Since run is blocking, I am not able to save the pid somewhere to kill the children in an extra command. A short code example:
try:
subprocess.run(["my_shell_script"], stderr=subprocess.STDOUT, timeout=120)
except subprocess.TimeoutExpired:
print("Timeout during execution")
This problem was reported as a bug to the Python developers. It seems to happen specifically when stderr or stdout is redirected.
Here is a more correct version of #Tanu's code.
import subprocess as sp
try:
proc = sp.Popen(['ls', '-l'], stdout=sp.PIPE, stderr=sp.PIPE)
outs, errs = proc.communicate(timeout=120)
except sp.TimeoutExpired:
proc.terminate()
Popen doesn't accept timeout as a parameter. It must be passed to communicate.
On Posix OSs, terminate is more gentle than kill, in that it reduces the risk of creating zombie processes.
Quoting from the docs:
subprocess.run - This does not capture stdout or stderr by default. To do so, pass PIPE for the stdout and/or stderr arguments.
Don't have to use Popen() if you don't want to. The other functions in the module, such as .call(), .Popen().
There are three 'file' streams: stdin for input, and stdout and stderr for output. The application decides what to write where; usually error and diagnostic information to stderr, the rest to stdout. To capture the output for either of these outputs, specify the subprocess.PIPE argument so that the 'stream' is redirected into your program.
To kill the child process after timeout:
import os
import signal
import subprocess
try:
proc = subprocess.Popen(["ls", "-l"], stdout=PIPE, stderr=PIPE, timeout=120)
except subprocess.TimeoutExpired:
os.kill(proc.pid, signal.SIGTERM)
I want to use a timeout on a subprocess
from subprocess32 import check_output
output = check_output("sleep 30", shell=True, timeout=1)
Unfortunately, whilst this raises a timeout error, it does so after 30 seconds. It seems that check_output cannot interrupt the shell command.
What can I do on on the Python side to stop this?
I suspect that subprocess32 fails to kill the timed out process.
check_output() with timeout is essentially:
with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
try:
output, unused_err = process.communicate(inputdata, timeout=timeout)
except TimeoutExpired:
process.kill()
output, unused_err = process.communicate()
raise TimeoutExpired(process.args, timeout, output=output)
There are two issues:
[the second] .communicate() may wait for descendant processes, not just for the immediate child, see Python subprocess .check_call vs
.check_output
process.kill() might not kill the whole process tree, see How to terminate a python subprocess launched with shell=True
It leads to the behaviour that you observed: the TimeoutExpired happens in a second, the shell is killed, but check_output() returns only in 30 seconds after the grandchild sleep process exits.
To workaround the issues, kill the whole process tree (all subprocesses that belong to the same group):
#!/usr/bin/env python3
import os
import signal
from subprocess import Popen, PIPE, TimeoutExpired
from time import monotonic as timer
start = timer()
with Popen('sleep 30', shell=True, stdout=PIPE, preexec_fn=os.setsid) as process:
try:
output = process.communicate(timeout=1)[0]
except TimeoutExpired:
os.killpg(process.pid, signal.SIGINT) # send signal to the process group
output = process.communicate()[0]
print('Elapsed seconds: {:.2f}'.format(timer() - start))
Output
Elapsed seconds: 1.00
Update for Python 3.6.
This is still happening but I have tested a lot of combinations of check_output, communicate and run methods and now I have a clear knowledge about where is the bug and how to avoid it in a easy way on Python 3.5 and Python 3.6.
My conclusion: It happens when you mix the use shell=True and any PIPE on stdout, stderr or stdin parameters (used in Popen and run methods).
Be careful: check_output uses PIPE inside.
If you look at the code inside on Python 3.6 it is basically a call to run with stdout=PIPE: https://github.com/python/cpython/blob/ae011e00189d9083dd84c357718264e24fe77314/Lib/subprocess.py#L335
So, to solve #innisfree problem on Python 3.5 or 3.6 just do this:
check_output(['sleep', '30'], timeout=1)
And for other cases, just avoid mixing shell=True and PIPE, keeping in mind that check_output uses PIPE.
I want to use a timeout on a subprocess
from subprocess32 import check_output
output = check_output("sleep 30", shell=True, timeout=1)
Unfortunately, whilst this raises a timeout error, it does so after 30 seconds. It seems that check_output cannot interrupt the shell command.
What can I do on on the Python side to stop this?
I suspect that subprocess32 fails to kill the timed out process.
check_output() with timeout is essentially:
with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
try:
output, unused_err = process.communicate(inputdata, timeout=timeout)
except TimeoutExpired:
process.kill()
output, unused_err = process.communicate()
raise TimeoutExpired(process.args, timeout, output=output)
There are two issues:
[the second] .communicate() may wait for descendant processes, not just for the immediate child, see Python subprocess .check_call vs
.check_output
process.kill() might not kill the whole process tree, see How to terminate a python subprocess launched with shell=True
It leads to the behaviour that you observed: the TimeoutExpired happens in a second, the shell is killed, but check_output() returns only in 30 seconds after the grandchild sleep process exits.
To workaround the issues, kill the whole process tree (all subprocesses that belong to the same group):
#!/usr/bin/env python3
import os
import signal
from subprocess import Popen, PIPE, TimeoutExpired
from time import monotonic as timer
start = timer()
with Popen('sleep 30', shell=True, stdout=PIPE, preexec_fn=os.setsid) as process:
try:
output = process.communicate(timeout=1)[0]
except TimeoutExpired:
os.killpg(process.pid, signal.SIGINT) # send signal to the process group
output = process.communicate()[0]
print('Elapsed seconds: {:.2f}'.format(timer() - start))
Output
Elapsed seconds: 1.00
Update for Python 3.6.
This is still happening but I have tested a lot of combinations of check_output, communicate and run methods and now I have a clear knowledge about where is the bug and how to avoid it in a easy way on Python 3.5 and Python 3.6.
My conclusion: It happens when you mix the use shell=True and any PIPE on stdout, stderr or stdin parameters (used in Popen and run methods).
Be careful: check_output uses PIPE inside.
If you look at the code inside on Python 3.6 it is basically a call to run with stdout=PIPE: https://github.com/python/cpython/blob/ae011e00189d9083dd84c357718264e24fe77314/Lib/subprocess.py#L335
So, to solve #innisfree problem on Python 3.5 or 3.6 just do this:
check_output(['sleep', '30'], timeout=1)
And for other cases, just avoid mixing shell=True and PIPE, keeping in mind that check_output uses PIPE.
I've been using subprocess.check_output() for some time to capture output from subprocesses, but ran into some performance problems under certain circumstances. I'm running this on a RHEL6 machine.
The calling Python environment is linux-compiled and 64-bit. The subprocess I'm executing is a shell script which eventually fires off a Windows python.exe process via Wine (why this foolishness is required is another story). As input to the shell script, I'm piping in a small bit of Python code that gets passed off to python.exe.
While the system is under moderate/heavy load (40 to 70% CPU utilization), I've noticed that using subprocess.check_output(cmd, shell=True) can result in a significant delay (up to ~45 seconds) after the subprocess has finished execution before the check_output command returns. Looking at output from ps -efH during this time shows the called subprocess as sh <defunct>, until it finally returns with a normal zero exit status.
Conversely, using subprocess.call(cmd, shell=True) to run the same command under the same moderate/heavy load will cause the subprocess to return immediately with no delay, all output printed to STDOUT/STDERR (rather than returned from the function call).
Why is there such a significant delay only when check_output() is redirecting the STDOUT/STDERR output into its return value, and not when the call() simply prints it back to the parent's STDOUT/STDERR?
Reading the docs, both subprocess.call and subprocess.check_output are use-cases of subprocess.Popen. One minor difference is that check_output will raise a Python error if the subprocess returns a non-zero exit status. The greater difference is emphasized in the bit about check_output (my emphasis):
The full function signature is largely the same as that of the Popen constructor, except that stdout is not permitted as it is used internally. All other supplied arguments are passed directly through to the Popen constructor.
So how is stdout "used internally"? Let's compare call and check_output:
call
def call(*popenargs, **kwargs):
return Popen(*popenargs, **kwargs).wait()
check_output
def check_output(*popenargs, **kwargs):
if 'stdout' in kwargs:
raise ValueError('stdout argument not allowed, it will be overridden.')
process = Popen(stdout=PIPE, *popenargs, **kwargs)
output, unused_err = process.communicate()
retcode = process.poll()
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise CalledProcessError(retcode, cmd, output=output)
return output
communicate
Now we have to look at Popen.communicate as well. Doing this, we notice that for one pipe, communicate does several things which simply take more time than simply returning Popen().wait(), as call does.
For one thing, communicate processes stdout=PIPE whether you set shell=True or not. Clearly, call does not. It just lets your shell spout whatever... making it a security risk, as Python describes here.
Secondly, in the case of check_output(cmd, shell=True) (just one pipe)... whatever your subprocess sends to stdout is processed by a thread in the _communicate method. And Popen must join the thread (wait on it) before additionally waiting on the subprocess itself to terminate!
Plus, more trivially, it processes stdout as a list which must then be joined into a string.
In short, even with minimal arguments, check_output spends a lot more time in Python processes than call does.
Let's look at the code. The .check_output has the following wait:
def _internal_poll(self, _deadstate=None, _waitpid=os.waitpid,
_WNOHANG=os.WNOHANG, _os_error=os.error, _ECHILD=errno.ECHILD):
"""Check if child process has terminated. Returns returncode
attribute.
This method is called by __del__, so it cannot reference anything
outside of the local scope (nor can any methods it calls).
"""
if self.returncode is None:
try:
pid, sts = _waitpid(self.pid, _WNOHANG)
if pid == self.pid:
self._handle_exitstatus(sts)
except _os_error as e:
if _deadstate is not None:
self.returncode = _deadstate
if e.errno == _ECHILD:
# This happens if SIGCLD is set to be ignored or
# waiting for child processes has otherwise been
# disabled for our process. This child is dead, we
# can't get the status.
# http://bugs.python.org/issue15756
self.returncode = 0
return self.returncode
The .call waits using the following code:
def wait(self):
"""Wait for child process to terminate. Returns returncode
attribute."""
while self.returncode is None:
try:
pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
except OSError as e:
if e.errno != errno.ECHILD:
raise
# This happens if SIGCLD is set to be ignored or waiting
# for child processes has otherwise been disabled for our
# process. This child is dead, we can't get the status.
pid = self.pid
sts = 0
# Check the pid and loop as waitpid has been known to return
# 0 even without WNOHANG in odd situations. issue14396.
if pid == self.pid:
self._handle_exitstatus(sts)
return self.returncode
Notice that bug related to internal_poll. It is viewable at http://bugs.python.org/issue15756. Pretty much exactly the issue you are running into.
Edit: The other potential issue between .call and .check_output is that .check_output actually cares about stdin and stdout and will try to perform IO against both pipes. If you are running into a process that get's itself into a zombie state it is possible that a read against a pipe in a defunct state is causing the hang you are experiencing.
In most cases zombie states get cleaned up pretty quickly, but, they will not if for instance they are interrupted while in a system call (like read or write). Of course the read/write system call should itself be interrupted as soon as the IO can no longer be performed, but, it is possible that you are hitting some sort of race condition where things are getting killed in a bad order.
The only way that I can think of to determine which is the cause in this case is for you to either add debugging code to the subprocess file or to invoke the python debugger and initiate a backtrace when you run into the condition you are experiencing.
I've got a command that I'm wrapping in script and spawning from a Python script using subprocess.Popen. I'm trying to make sure it dies if the user issues a SIGINT.
I could figure out if the process was interrupted in a least two ways:
A. Die if the wrapped command has a non-zero exit status (doesn't work, because script seems to always return 0)
B. Do something special with SIGINT in the parent Python script rather than simply interrupting the subprocess. I've tried the following:
import sys
import signal
import subprocess
def interrupt_handler(signum, frame):
print "While there is a 'script' subprocess alive, this handler won't executes"
sys.exit(1)
signal.signal(signal.SIGINT, interrupt_handler)
for n in range( 10 ):
print "Going to sleep for 2 second...Ctrl-C to exit the sleep cycles"
# exit 1 if we make it to the end of our sleep
cmd = [ 'script', '-q', '-c', "sleep 2 && (exit 1)", '/dev/null']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
if p.poll() != None :
break
else :
pass
# Exiting on non-zero exit status would suffice
print "Exit status (script always exits zero, despite what happened to the wrapped command):", p.returncode
I'd like hitting Ctrl-C to exit the python script. What's happening instead is the subprocess dies and the script continues.
The subprocess is by default part of the same process group, and only one can control and receive signals from the terminal, so there are a couple of different solutions.
Setting stdin as a PIPE (in contrast to inheriting from the parent process), this will prevent the child process from receiving signals associated to it.
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
Detaching from the parent process group, the child will no longer receive signals
def preexec_function():
os.setpgrp()
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=preexec_function)
Explicitly ignoring signals in the child process
def preexec_function():
signal.signal(signal.SIGINT, signal.SIG_IGN)
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=preexec_function)
This might however be overwritten by the child process.
Fist thing; there is a send_signal() method on the Popen object. If you want to send a signal to one you've launched, use this method to send it.
Second thing; a deeper problem with the way you're setting up communication with your subprocess and then, um, not communicating with it. You cannot safely tell the subprocess to send its output to subprocess.PIPE and then not read from the pipes. UNIX pipes are buffered (typically a 4K buffer?), and if the subprocess fills up the buffer and the process on the other end of the pipe doesn't read the buffered data, the subprocess will pend (locking up, from an observer's perspective) on its next write to the pipe. So, the usual pattern when using subprocess.PIPE is to call communicate() on the Popen object.
It is not mandatory to use subprocess.PIPE if you want data back from the subprocess. A cool trick is to use the tempfile.TemporaryFile class to make an unnamed temp file (really it opens a file and immediately deletes the inode from the file system, so you have access to the file but no-one else can open one. You can do something like:
with tempfile.TemporaryFile() as iofile:
p = Popen(cmd, stdout=iofile, stderr=iofile)
while True:
if p.poll() is not None:
break
else:
time.sleep(0.1) # without some sleep, this polling is VERY busy...
Then you can read the contents of your temporary file (seek to the beginning of it before you do, to be sure you're at the beginning) when you know the subprocess has exited, instead of using pipes. The pipe buffering problem won't be a problem if the subprocess's output is going to a file (temporary or not).
Here is a riff on your code sample that I think does what you want. The signal handler just repeats the signals being trapped by the parent process (in this example, SIGINT and SIGTERM) to all current subprocesses (there should only ever be one in this program) and sets a module-level flag saying to shutdown at the next opportunity. Since I'm using subprocess.PIPE I/O, I call communicate() on the Popen object.
#!/usr/bin/env python
from subprocess import Popen, PIPE
import signal
import sys
current_subprocs = set()
shutdown = False
def handle_signal(signum, frame):
# send signal recieved to subprocesses
global shutdown
shutdown = True
for proc in current_subprocs:
if proc.poll() is None:
proc.send_signal(signum)
signal.signal(signal.SIGINT, handle_signal)
signal.signal(signal.SIGTERM, handle_signal)
for _ in range(10):
if shutdown:
break
cmd = ["sleep", "2"]
p = Popen(cmd, stdout=PIPE, stderr=PIPE)
current_subprocs.add(p)
out, err = p.communicate()
current_subprocs.remove(p)
print "subproc returncode", p.returncode
And calling it (with a Ctrl-C in the third 2 second interval):
% python /tmp/proctest.py
subproc returncode 0
subproc returncode 0
^Csubproc returncode -2
This hack will work, but it's ugly...
Change the command to this:
success_flag = '/tmp/success.flag'
cmd = [ 'script', '-q', '-c', "sleep 2 && touch " + success_flag, '/dev/null']
And put
if os.path.isfile( success_flag ) :
os.remove( success_flag )
else :
return
at the end of the for loop
If you have no python processing to do after your process is spawned (like in your example), then the easiest way is to use os.execvp instead of the subprocess module. Your subprocess is going to completely replace your python process, and will be the one catching SIGINT directly.
I found a -e switch in the script man page:
-e Return the exit code of the child process. Uses the same format
as bash termination on signal termination exit code is 128+n.
Not too sure what the 128+n is all about but it seems to return 130 for ctrl-c. So modifying your cmd to be
cmd = [ 'script', '-e', '-q', '-c', "sleep 2 && (exit 1)", '/dev/null']
and putting
if p.returncode == 130:
break
at the end of the for loop seems to do what you want.