I am calling a shell script fom within Python, that spawns multiple child processes. I want to terminate that process and all of its children, if it did not finish after two minutes.
Is there any way I can do that with subprocess.run or do I have to go back to using Popen? Since run is blocking, I am not able to save the pid somewhere to kill the children in an extra command. A short code example:
try:
subprocess.run(["my_shell_script"], stderr=subprocess.STDOUT, timeout=120)
except subprocess.TimeoutExpired:
print("Timeout during execution")
This problem was reported as a bug to the Python developers. It seems to happen specifically when stderr or stdout is redirected.
Here is a more correct version of #Tanu's code.
import subprocess as sp
try:
proc = sp.Popen(['ls', '-l'], stdout=sp.PIPE, stderr=sp.PIPE)
outs, errs = proc.communicate(timeout=120)
except sp.TimeoutExpired:
proc.terminate()
Popen doesn't accept timeout as a parameter. It must be passed to communicate.
On Posix OSs, terminate is more gentle than kill, in that it reduces the risk of creating zombie processes.
Quoting from the docs:
subprocess.run - This does not capture stdout or stderr by default. To do so, pass PIPE for the stdout and/or stderr arguments.
Don't have to use Popen() if you don't want to. The other functions in the module, such as .call(), .Popen().
There are three 'file' streams: stdin for input, and stdout and stderr for output. The application decides what to write where; usually error and diagnostic information to stderr, the rest to stdout. To capture the output for either of these outputs, specify the subprocess.PIPE argument so that the 'stream' is redirected into your program.
To kill the child process after timeout:
import os
import signal
import subprocess
try:
proc = subprocess.Popen(["ls", "-l"], stdout=PIPE, stderr=PIPE, timeout=120)
except subprocess.TimeoutExpired:
os.kill(proc.pid, signal.SIGTERM)
Related
I want to use a timeout on a subprocess
from subprocess32 import check_output
output = check_output("sleep 30", shell=True, timeout=1)
Unfortunately, whilst this raises a timeout error, it does so after 30 seconds. It seems that check_output cannot interrupt the shell command.
What can I do on on the Python side to stop this?
I suspect that subprocess32 fails to kill the timed out process.
check_output() with timeout is essentially:
with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
try:
output, unused_err = process.communicate(inputdata, timeout=timeout)
except TimeoutExpired:
process.kill()
output, unused_err = process.communicate()
raise TimeoutExpired(process.args, timeout, output=output)
There are two issues:
[the second] .communicate() may wait for descendant processes, not just for the immediate child, see Python subprocess .check_call vs
.check_output
process.kill() might not kill the whole process tree, see How to terminate a python subprocess launched with shell=True
It leads to the behaviour that you observed: the TimeoutExpired happens in a second, the shell is killed, but check_output() returns only in 30 seconds after the grandchild sleep process exits.
To workaround the issues, kill the whole process tree (all subprocesses that belong to the same group):
#!/usr/bin/env python3
import os
import signal
from subprocess import Popen, PIPE, TimeoutExpired
from time import monotonic as timer
start = timer()
with Popen('sleep 30', shell=True, stdout=PIPE, preexec_fn=os.setsid) as process:
try:
output = process.communicate(timeout=1)[0]
except TimeoutExpired:
os.killpg(process.pid, signal.SIGINT) # send signal to the process group
output = process.communicate()[0]
print('Elapsed seconds: {:.2f}'.format(timer() - start))
Output
Elapsed seconds: 1.00
Update for Python 3.6.
This is still happening but I have tested a lot of combinations of check_output, communicate and run methods and now I have a clear knowledge about where is the bug and how to avoid it in a easy way on Python 3.5 and Python 3.6.
My conclusion: It happens when you mix the use shell=True and any PIPE on stdout, stderr or stdin parameters (used in Popen and run methods).
Be careful: check_output uses PIPE inside.
If you look at the code inside on Python 3.6 it is basically a call to run with stdout=PIPE: https://github.com/python/cpython/blob/ae011e00189d9083dd84c357718264e24fe77314/Lib/subprocess.py#L335
So, to solve #innisfree problem on Python 3.5 or 3.6 just do this:
check_output(['sleep', '30'], timeout=1)
And for other cases, just avoid mixing shell=True and PIPE, keeping in mind that check_output uses PIPE.
I want to use a timeout on a subprocess
from subprocess32 import check_output
output = check_output("sleep 30", shell=True, timeout=1)
Unfortunately, whilst this raises a timeout error, it does so after 30 seconds. It seems that check_output cannot interrupt the shell command.
What can I do on on the Python side to stop this?
I suspect that subprocess32 fails to kill the timed out process.
check_output() with timeout is essentially:
with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
try:
output, unused_err = process.communicate(inputdata, timeout=timeout)
except TimeoutExpired:
process.kill()
output, unused_err = process.communicate()
raise TimeoutExpired(process.args, timeout, output=output)
There are two issues:
[the second] .communicate() may wait for descendant processes, not just for the immediate child, see Python subprocess .check_call vs
.check_output
process.kill() might not kill the whole process tree, see How to terminate a python subprocess launched with shell=True
It leads to the behaviour that you observed: the TimeoutExpired happens in a second, the shell is killed, but check_output() returns only in 30 seconds after the grandchild sleep process exits.
To workaround the issues, kill the whole process tree (all subprocesses that belong to the same group):
#!/usr/bin/env python3
import os
import signal
from subprocess import Popen, PIPE, TimeoutExpired
from time import monotonic as timer
start = timer()
with Popen('sleep 30', shell=True, stdout=PIPE, preexec_fn=os.setsid) as process:
try:
output = process.communicate(timeout=1)[0]
except TimeoutExpired:
os.killpg(process.pid, signal.SIGINT) # send signal to the process group
output = process.communicate()[0]
print('Elapsed seconds: {:.2f}'.format(timer() - start))
Output
Elapsed seconds: 1.00
Update for Python 3.6.
This is still happening but I have tested a lot of combinations of check_output, communicate and run methods and now I have a clear knowledge about where is the bug and how to avoid it in a easy way on Python 3.5 and Python 3.6.
My conclusion: It happens when you mix the use shell=True and any PIPE on stdout, stderr or stdin parameters (used in Popen and run methods).
Be careful: check_output uses PIPE inside.
If you look at the code inside on Python 3.6 it is basically a call to run with stdout=PIPE: https://github.com/python/cpython/blob/ae011e00189d9083dd84c357718264e24fe77314/Lib/subprocess.py#L335
So, to solve #innisfree problem on Python 3.5 or 3.6 just do this:
check_output(['sleep', '30'], timeout=1)
And for other cases, just avoid mixing shell=True and PIPE, keeping in mind that check_output uses PIPE.
I'm very new to python. I would like to know, when using subprocess.Popen do we need to close the connection or subprocess automatically closes the connection?
process = subprocess.Popen(["mysql", "-uroot", "-ppassword", "database"],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
process_out, process_err = process.communicate(file("test.sql").read())
print process_out
.communicate() call closes all the pipes (if it is what you mean by "connection") and reaps the child process. You don't need to do anything after that.
#!/usr/bin/env python3
from subprocess import Popen, PIPE
with open('test.sql', 'rb', 0) as input_file, \
Popen([cmd] + args, stdin=input_file, stdout=PIPE, stderr=PIPE) as p:
output, errors = p.communicate()
if p.returncode != 0:
raise Error
process is the object which represents the subprocess. With it, you can do everything you want, but at the end, after having processed all communication, you should .wait() for it in order not to have a zombie process for a too long time.
Only after .wait() the subprocess will be really gone from the view of the OS.
The said is valid if you handle the communication on your own. But if your circumstances allow you to use the .communicate() method and you do so, you don't have to call .wait(), as it does so for you.
I've got a command that I'm wrapping in script and spawning from a Python script using subprocess.Popen. I'm trying to make sure it dies if the user issues a SIGINT.
I could figure out if the process was interrupted in a least two ways:
A. Die if the wrapped command has a non-zero exit status (doesn't work, because script seems to always return 0)
B. Do something special with SIGINT in the parent Python script rather than simply interrupting the subprocess. I've tried the following:
import sys
import signal
import subprocess
def interrupt_handler(signum, frame):
print "While there is a 'script' subprocess alive, this handler won't executes"
sys.exit(1)
signal.signal(signal.SIGINT, interrupt_handler)
for n in range( 10 ):
print "Going to sleep for 2 second...Ctrl-C to exit the sleep cycles"
# exit 1 if we make it to the end of our sleep
cmd = [ 'script', '-q', '-c', "sleep 2 && (exit 1)", '/dev/null']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
if p.poll() != None :
break
else :
pass
# Exiting on non-zero exit status would suffice
print "Exit status (script always exits zero, despite what happened to the wrapped command):", p.returncode
I'd like hitting Ctrl-C to exit the python script. What's happening instead is the subprocess dies and the script continues.
The subprocess is by default part of the same process group, and only one can control and receive signals from the terminal, so there are a couple of different solutions.
Setting stdin as a PIPE (in contrast to inheriting from the parent process), this will prevent the child process from receiving signals associated to it.
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
Detaching from the parent process group, the child will no longer receive signals
def preexec_function():
os.setpgrp()
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=preexec_function)
Explicitly ignoring signals in the child process
def preexec_function():
signal.signal(signal.SIGINT, signal.SIG_IGN)
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=preexec_function)
This might however be overwritten by the child process.
Fist thing; there is a send_signal() method on the Popen object. If you want to send a signal to one you've launched, use this method to send it.
Second thing; a deeper problem with the way you're setting up communication with your subprocess and then, um, not communicating with it. You cannot safely tell the subprocess to send its output to subprocess.PIPE and then not read from the pipes. UNIX pipes are buffered (typically a 4K buffer?), and if the subprocess fills up the buffer and the process on the other end of the pipe doesn't read the buffered data, the subprocess will pend (locking up, from an observer's perspective) on its next write to the pipe. So, the usual pattern when using subprocess.PIPE is to call communicate() on the Popen object.
It is not mandatory to use subprocess.PIPE if you want data back from the subprocess. A cool trick is to use the tempfile.TemporaryFile class to make an unnamed temp file (really it opens a file and immediately deletes the inode from the file system, so you have access to the file but no-one else can open one. You can do something like:
with tempfile.TemporaryFile() as iofile:
p = Popen(cmd, stdout=iofile, stderr=iofile)
while True:
if p.poll() is not None:
break
else:
time.sleep(0.1) # without some sleep, this polling is VERY busy...
Then you can read the contents of your temporary file (seek to the beginning of it before you do, to be sure you're at the beginning) when you know the subprocess has exited, instead of using pipes. The pipe buffering problem won't be a problem if the subprocess's output is going to a file (temporary or not).
Here is a riff on your code sample that I think does what you want. The signal handler just repeats the signals being trapped by the parent process (in this example, SIGINT and SIGTERM) to all current subprocesses (there should only ever be one in this program) and sets a module-level flag saying to shutdown at the next opportunity. Since I'm using subprocess.PIPE I/O, I call communicate() on the Popen object.
#!/usr/bin/env python
from subprocess import Popen, PIPE
import signal
import sys
current_subprocs = set()
shutdown = False
def handle_signal(signum, frame):
# send signal recieved to subprocesses
global shutdown
shutdown = True
for proc in current_subprocs:
if proc.poll() is None:
proc.send_signal(signum)
signal.signal(signal.SIGINT, handle_signal)
signal.signal(signal.SIGTERM, handle_signal)
for _ in range(10):
if shutdown:
break
cmd = ["sleep", "2"]
p = Popen(cmd, stdout=PIPE, stderr=PIPE)
current_subprocs.add(p)
out, err = p.communicate()
current_subprocs.remove(p)
print "subproc returncode", p.returncode
And calling it (with a Ctrl-C in the third 2 second interval):
% python /tmp/proctest.py
subproc returncode 0
subproc returncode 0
^Csubproc returncode -2
This hack will work, but it's ugly...
Change the command to this:
success_flag = '/tmp/success.flag'
cmd = [ 'script', '-q', '-c', "sleep 2 && touch " + success_flag, '/dev/null']
And put
if os.path.isfile( success_flag ) :
os.remove( success_flag )
else :
return
at the end of the for loop
If you have no python processing to do after your process is spawned (like in your example), then the easiest way is to use os.execvp instead of the subprocess module. Your subprocess is going to completely replace your python process, and will be the one catching SIGINT directly.
I found a -e switch in the script man page:
-e Return the exit code of the child process. Uses the same format
as bash termination on signal termination exit code is 128+n.
Not too sure what the 128+n is all about but it seems to return 130 for ctrl-c. So modifying your cmd to be
cmd = [ 'script', '-e', '-q', '-c', "sleep 2 && (exit 1)", '/dev/null']
and putting
if p.returncode == 130:
break
at the end of the for loop seems to do what you want.
I'm using the following to execute a process and hide its output from Python. It's in a loop though, and I need a way to block until the sub process has terminated before moving to the next iteration.
subprocess.Popen(["scanx", "--udp", host], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Use subprocess.call(). From the docs:
subprocess.call(*popenargs, **kwargs)
Run command with arguments. Wait for command to complete, then
return the returncode attribute.
The arguments are the same as for the
Popen constructor.
Edit:
subprocess.call() uses wait(), and wait() is vulnerable to deadlocks (as Tommy Herbert pointed out). From the docs:
Warning: This will deadlock if the
child process generates enough output
to a stdout or stderr pipe such that
it blocks waiting for the OS pipe
buffer to accept more data. Use
communicate() to avoid that.
So if your command generates a lot of output, use communicate() instead:
p = subprocess.Popen(
["scanx", "--udp", host],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = p.communicate()
If you don't need output at all you can pass devnull to stdout and stderr. I don't know if this can make a difference but pass a bufsize. Using devnull now subprocess.call doesn't suffer of deadlock anymore
import os
import subprocess
null = open(os.devnull, 'w')
subprocess.call(['ls', '-lR'], bufsize=4096, stdout=null, stderr=null)