Develop Reference

Opens a process with Popen cant close it ( need to run Ros command in cmd)

I need to save some image files from my simulation at different times. So my idea was to open a subprocess save some image files and close it .
import subprocess
cmd = "rosrun pcl_ros pointcloud_to_pcd input:=camera/depth/points"
proc = subprocess.Popen(cmd, shell=True)
When it comes to closing I tried different things:
import os
import signal
import subprocess
cmd = "rosrun pcl_ros pointcloud_to_pcd input:=camera/depth/points"
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
os.killpg(os.getpgid(pro.pid), signal.SIGTERM)
command did not execute , so it doesn't work for me. I also tried a solution with psutil and it didn't work neither...

you probably don't need shell=True here, which is the cause of your problems. I suspect that when you kill the process group in your second snippet, the shell process is killed before the process you want to run has a chance to start...
Try to pass the parameters as a list of strings (so you don't need shell=True), wait a bit, and use terminate on the Popen object. You don't need process group, or psutil to kill the process & its children, just plain old terminate() on the process object does the trick.
cmd = ["rosrun","pcl_ros","pointcloud_to_pcd","input:=camera/depth/points"]
proc = subprocess.Popen(cmd)
time.sleep(1) # maybe needed to wait the process to do something useful
proc.terminate()
Note that proc.terminate() tries to exit gracefully where proc.kill() would have just killed the process (there's a difference under Un*x systems, not under Windows)
Another plea for "do not use shell=True unless forced at gunpoint".

Kill children of Python subprocess after subprocess.TimeoutExpired is thrown

I am calling a shell script fom within Python, that spawns multiple child processes. I want to terminate that process and all of its children, if it did not finish after two minutes.
Is there any way I can do that with subprocess.run or do I have to go back to using Popen? Since run is blocking, I am not able to save the pid somewhere to kill the children in an extra command. A short code example:
try:
subprocess.run(["my_shell_script"], stderr=subprocess.STDOUT, timeout=120)
except subprocess.TimeoutExpired:
print("Timeout during execution")

This problem was reported as a bug to the Python developers. It seems to happen specifically when stderr or stdout is redirected.
Here is a more correct version of #Tanu's code.
import subprocess as sp
try:
proc = sp.Popen(['ls', '-l'], stdout=sp.PIPE, stderr=sp.PIPE)
outs, errs = proc.communicate(timeout=120)
except sp.TimeoutExpired:
proc.terminate()
Popen doesn't accept timeout as a parameter. It must be passed to communicate.
On Posix OSs, terminate is more gentle than kill, in that it reduces the risk of creating zombie processes.

Quoting from the docs:
subprocess.run - This does not capture stdout or stderr by default. To do so, pass PIPE for the stdout and/or stderr arguments.
Don't have to use Popen() if you don't want to. The other functions in the module, such as .call(), .Popen().
There are three 'file' streams: stdin for input, and stdout and stderr for output. The application decides what to write where; usually error and diagnostic information to stderr, the rest to stdout. To capture the output for either of these outputs, specify the subprocess.PIPE argument so that the 'stream' is redirected into your program.
To kill the child process after timeout:
import os
import signal
import subprocess
try:
proc = subprocess.Popen(["ls", "-l"], stdout=PIPE, stderr=PIPE, timeout=120)
except subprocess.TimeoutExpired:
os.kill(proc.pid, signal.SIGTERM)

Making sure a Python script with subprocesses dies on SIGINT

I've got a command that I'm wrapping in script and spawning from a Python script using subprocess.Popen. I'm trying to make sure it dies if the user issues a SIGINT.
I could figure out if the process was interrupted in a least two ways:
A. Die if the wrapped command has a non-zero exit status (doesn't work, because script seems to always return 0)
B. Do something special with SIGINT in the parent Python script rather than simply interrupting the subprocess. I've tried the following:
import sys
import signal
import subprocess
def interrupt_handler(signum, frame):
print "While there is a 'script' subprocess alive, this handler won't executes"
sys.exit(1)
signal.signal(signal.SIGINT, interrupt_handler)
for n in range( 10 ):
print "Going to sleep for 2 second...Ctrl-C to exit the sleep cycles"
# exit 1 if we make it to the end of our sleep
cmd = [ 'script', '-q', '-c', "sleep 2 && (exit 1)", '/dev/null']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
if p.poll() != None :
break
else :
pass
# Exiting on non-zero exit status would suffice
print "Exit status (script always exits zero, despite what happened to the wrapped command):", p.returncode
I'd like hitting Ctrl-C to exit the python script. What's happening instead is the subprocess dies and the script continues.

The subprocess is by default part of the same process group, and only one can control and receive signals from the terminal, so there are a couple of different solutions.
Setting stdin as a PIPE (in contrast to inheriting from the parent process), this will prevent the child process from receiving signals associated to it.
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
Detaching from the parent process group, the child will no longer receive signals
def preexec_function():
os.setpgrp()
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=preexec_function)
Explicitly ignoring signals in the child process
def preexec_function():
signal.signal(signal.SIGINT, signal.SIG_IGN)
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=preexec_function)
This might however be overwritten by the child process.

Fist thing; there is a send_signal() method on the Popen object. If you want to send a signal to one you've launched, use this method to send it.
Second thing; a deeper problem with the way you're setting up communication with your subprocess and then, um, not communicating with it. You cannot safely tell the subprocess to send its output to subprocess.PIPE and then not read from the pipes. UNIX pipes are buffered (typically a 4K buffer?), and if the subprocess fills up the buffer and the process on the other end of the pipe doesn't read the buffered data, the subprocess will pend (locking up, from an observer's perspective) on its next write to the pipe. So, the usual pattern when using subprocess.PIPE is to call communicate() on the Popen object.
It is not mandatory to use subprocess.PIPE if you want data back from the subprocess. A cool trick is to use the tempfile.TemporaryFile class to make an unnamed temp file (really it opens a file and immediately deletes the inode from the file system, so you have access to the file but no-one else can open one. You can do something like:
with tempfile.TemporaryFile() as iofile:
p = Popen(cmd, stdout=iofile, stderr=iofile)
while True:
if p.poll() is not None:
break
else:
time.sleep(0.1) # without some sleep, this polling is VERY busy...
Then you can read the contents of your temporary file (seek to the beginning of it before you do, to be sure you're at the beginning) when you know the subprocess has exited, instead of using pipes. The pipe buffering problem won't be a problem if the subprocess's output is going to a file (temporary or not).
Here is a riff on your code sample that I think does what you want. The signal handler just repeats the signals being trapped by the parent process (in this example, SIGINT and SIGTERM) to all current subprocesses (there should only ever be one in this program) and sets a module-level flag saying to shutdown at the next opportunity. Since I'm using subprocess.PIPE I/O, I call communicate() on the Popen object.
#!/usr/bin/env python
from subprocess import Popen, PIPE
import signal
import sys
current_subprocs = set()
shutdown = False
def handle_signal(signum, frame):
# send signal recieved to subprocesses
global shutdown
shutdown = True
for proc in current_subprocs:
if proc.poll() is None:
proc.send_signal(signum)
signal.signal(signal.SIGINT, handle_signal)
signal.signal(signal.SIGTERM, handle_signal)
for _ in range(10):
if shutdown:
break
cmd = ["sleep", "2"]
p = Popen(cmd, stdout=PIPE, stderr=PIPE)
current_subprocs.add(p)
out, err = p.communicate()
current_subprocs.remove(p)
print "subproc returncode", p.returncode
And calling it (with a Ctrl-C in the third 2 second interval):
% python /tmp/proctest.py
subproc returncode 0
subproc returncode 0
^Csubproc returncode -2

This hack will work, but it's ugly...
Change the command to this:
success_flag = '/tmp/success.flag'
cmd = [ 'script', '-q', '-c', "sleep 2 && touch " + success_flag, '/dev/null']
And put
if os.path.isfile( success_flag ) :
os.remove( success_flag )
else :
return
at the end of the for loop

If you have no python processing to do after your process is spawned (like in your example), then the easiest way is to use os.execvp instead of the subprocess module. Your subprocess is going to completely replace your python process, and will be the one catching SIGINT directly.

I found a -e switch in the script man page:
-e Return the exit code of the child process. Uses the same format
as bash termination on signal termination exit code is 128+n.
Not too sure what the 128+n is all about but it seems to return 130 for ctrl-c. So modifying your cmd to be
cmd = [ 'script', '-e', '-q', '-c', "sleep 2 && (exit 1)", '/dev/null']
and putting
if p.returncode == 130:
break
at the end of the for loop seems to do what you want.

Running shell command from Python script

I'm trying to run a shell command from within a python script which needs to do several things
1. The shell command is 'hspice tran.deck >! tran.lis'
2. The script should wait for the shell command to complete before proceeding
3. I need to check the return code from the command and
4. Capture STDOUT if it completed successfully else capture STDERR
I went through the subprocess module and tried out a couple of things but couldn't find a way to do all of the above.
- with subprocess.call() I could check the return code but not capture the output.
- with subprocess.check_output() I could capture the output but not the code.
- with subprocess.Popen() and Popen.communicate(), I could capture STDOUT and STDERR but not the return code.
I'm not sure how to use Popen.wait() or the returncode attribute. I also couldn't get Popen to accept '>!' or '|' as arguments.
Can someone please point me in the right direction? I'm using Python 2.7.1
EDIT: Got things working with the following code
process = subprocess.Popen('ls | tee out.txt', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = process.communicate()
if(process.returncode==0):
print out
else:
print err
Also, should I use a process.wait() after the process = line or does it wait by default?

Just use .returncode after .communicate(). Also, tell Popen that what you're trying to run is a shell command, rather than a raw command line:
p = subprocess.Popen('ls | tee out.txt', shell=True, ...)
p.communicate()
print p.returncode

From the docs:
Popen.returncode
The child return code, set by poll() and wait() (and indirectly by communicate()). A None value indicates that the process hasn’t terminated yet.
A negative value -N indicates that the child was terminated by signal N (Unix only).

Here is example how to interact with shell:
>>> process = subprocess.Popen(['/bin/bash'], shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> process.stdin.write('echo it works!\n')
>>> process.stdout.readline()
'it works!\n'
>>> process.stdin.write('date\n')
>>> process.stdout.readline()
'wto, 13 mar 2012, 17:25:35 CET\n'
>>>

Keep a subprocess alive and keep giving it commands? Python

If I spawn a new subprocess in python with a given command (let's say I start the python interpreter with the python command), how can I send new data to the process (via STDIN)?

Use the standard subprocess module. You use subprocess.Popen() to start the process, and it will run in the background (i.e. at the same time as your Python program). When you call Popen(), you probably want to set the stdin, stdout and stderr parameters to subprocess.PIPE. Then you can use the stdin, stdout and stderr fields on the returned object to write and read data.
Untested example code:
from subprocess import Popen, PIPE
# Run "cat", which is a simple Linux program that prints it's input.
process = Popen(['/bin/cat'], stdin=PIPE, stdout=PIPE)
process.stdin.write(b'Hello\n')
process.stdin.flush()
print(repr(process.stdout.readline())) # Should print 'Hello\n'
process.stdin.write(b'World\n')
process.stdin.flush()
print(repr(process.stdout.readline())) # Should print 'World\n'
# "cat" will exit when you close stdin. (Not all programs do this!)
process.stdin.close()
print('Waiting for cat to exit')
process.wait()
print('cat finished with return code %d' % process.returncode)

Don't.
If you want to send commands to a subprocess, create a pty and then fork the subprocess with one end of the pty attached to its STDIN.
Here is a snippet from some of my code:
RNULL = open('/dev/null', 'r')
WNULL = open('/dev/null', 'w')
master, slave = pty.openpty()
print parsedCmd
self.subp = Popen(parsedCmd, shell=False, stdin=RNULL,
stdout=WNULL, stderr=slave)
In this code, the pty is attached to stderr because it receives error messages rather than sending commands, but the principle is the same.