I'm trying to write some basic tests for a piece of code that normally accepts input endlessly through stdin until given a specific exit command.
I want to check if the program crashes on being given some input string (after some amount of time to account for processing), but can't seem to figure out how to send data and not be stuck waiting for output which I don't care about.
My current code looks like this (using cat as an example of the program):
myproc = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
myproc.communicate(input=inputdata.encode("utf-8"))
time.sleep(0.1)
if myproc.poll() != None:
print("not running")
else:
print("still running")
How can I modify this to allow the program to proceed to the polling instead of hanging after the communicate() call?
You are using the wrong tool here with communicate which waits for the end of the program. You should simply feed the standard input of the subprocess:
myproc = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
myproc.stdin.write(inputdata.encode("utf-8"))
time.sleep(0.1)
if myproc.poll() != None:
print("not running")
else:
print("still running")
But beware: you cannot be sure that the output pipes will contain anything before the end of the subprocess...
You could set a timeout in the Popen.communicate(input=None, timeout=None) function. After the timeout the process is still running and I think but you have to test it you can still send in input with communicate.
From the docs:
If the process does not terminate after timeout seconds, a TimeoutExpired exception will be raised. Catching this exception and retrying communication will not lose any output.
The child process is not killed if the timeout expires, so in order to
cleanup properly a well-behaved application should kill the child
process and finish communication:
I think I understand what you want here. If you know an existing command that will crash your program, you can use subprocess.Popen.wait() and it'll still block, but it'll return a tuple of the output message and the error associated with it, if any.
Then you can note the error and catch it in a try exception statement.
This was really helpful when I was working with sub processes:
https://docs.python.org/3/library/asyncio-subprocess.html
Related
I'm having a problem with subprocess poll not returning the return code when the process has finished.
I found out how to set a timeout on subprocess.Popen and used that as the basis for my code. However, I have a call that uses Java that doesn't correctly report the return code so each call "times out" even though it is actually finished. I know the process has finished because when removing the poll timeout check, the call runs without issue returning a good exit code and within the time limit.
Here is the code I am testing with.
import subprocess
import time
def execute(command):
print('start command: {}'.format(command))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print('wait')
wait = 10
while process.poll() is None and wait > 0:
time.sleep(1)
wait -= 1
print('done')
if wait == 0:
print('terminate')
process.terminate()
print('communicate')
stdout, stderr = process.communicate()
print('rc')
exit_code = process.returncode
if exit_code != 0:
print('got bad rc')
if __name__ == '__main__':
execute(['ping','-n','15','127.0.0.1']) # correctly times out
execute(['ping','-n','5','127.0.0.1']) # correctly runs within the time limit
# incorrectly times out
execute(['C:\\dev\\jdk8\\bin\\java.exe', '-jar', 'JMXQuery-0.1.8.jar', '-url', 'service:jmx:rmi:///jndi/rmi://localhost:18080/jmxrmi', '-json', '-q', 'java.lang:type=Runtime;java.lang:type=OperatingSystem'])
You can see that two examples are designed to time out and two are not to time out and they all work correctly. However, the final one (using jmxquery to get tomcat metrics) doesn't return the exit code and therefore "times out" and has to be terminated, which then causes it to return an error code of 1.
Is there something I am missing in the way subprocess poll is interacting with this Java process that is causing it to not return an exit code? Is there a way to get a timeout option to work with this?
This has the same cause as a number of existing questions, but the desire to impose a timeout requires a different answer.
The OS deliberately gives only a small amount of buffer space to each pipe. When a process writes to one that is full (because the reader has not yet consumed the previous output), it blocks. (The reason is that a producer that is faster than its consumer would otherwise be able to quickly use a great deal of memory for no gain.) Therefore, if you want to do more than one of the following with a subprocess, you have to interleave them rather than doing each in turn:
Read from standard output
Read from standard error (unless it’s merged via subprocess.STDOUT)
Wait for the process to exit, or for a timeout to elapse
Of course, the subprocess might close its streams before it exits, write useful output after you notice the timeout and before you kill it, and/or start additional processes that keep the pipe open indefinitely, so you might want to have multiple timeouts. Probably what’s most informative is the EOF on the pipe, so repeatedly use something like select to wait for (however much is left of) the timeout, issue single reads on the streams that are ready, and wait (with another timeout if you’re concerned about hangs after an early stream closure) on EOF. If the timeout occurs instead, (try to) kill the subprocess, and consider issuing non-blocking reads (or another timeout loop) to get any last available output before closing the pipes.
Using the other answer by #DavisHerring as the basis for more research, I came across a concept that worked for my original case. Here is the code that came out of that.
import subprocess
import threading
import time
def execute(command):
print('start command: {}'.format(command))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
timer = threading.Timer(10, terminate_process, [process])
timer.start()
print('communicate')
stdout, stderr = process.communicate()
print('rc')
exit_code = process.returncode
timer.cancel()
if exit_code != 0:
print('got bad rc')
def terminate_process(p):
try:
p.terminate()
except OSError:
pass # ignore error
It uses the threading.Timer to make sure that the process doesn't go over the time limit and terminates the process if it does. It otherwise waits for a response back and cancels the timer once it finishes.
I have a thread which handles commands sent to a device. It opens a subprocess, sends the command to qmicli application (https://sigquit.wordpress.com/2012/08/20/an-introduction-to-libqmi/), gets a reply and the reply is dealt with.
This generally works fine for days/weeks of running. However I noticed that sometimes the thread would sometimes just stop doing anything when I make a subprocess.Popen call (the next lines of code do not run), the simplified code looks like this:
try:
self.qmi_process = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
# Log value of self.qmi_process happens here
if self.qmi_process:
out = self.qmi_process.communicate()
else:
return "ERROR: no qmi_process"
self.qmi_process = None
ret = ''.join(str(e) for e in out if e)
except:
return "ERROR: Caught unhandled exception"
I have started logging the value of the subprocess.Popen call to see if the communicate() call was blocking or was it failing before this when the subprocess call is created. It turns out that for some reason the subprocess.Popen fails and self.qmi_process value is not logged, but my Exception code is not being called, any idea how that could happen?
subprocess.Popen does not return.
I have multiple threads calling popen, I've read this can cause deadlock in 2.7?
I'm writing a python script that launches programs in the background and then monitors to see if they encounter an error. I am using the subprocess module to start the process and keep a list of running programs.
processes.append((subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE), command))
I have found that when I try to monitor the programs by calling communicate on the subprocess object, the main program waits for the program to finish. I have tried to use poll(), but that doesn't give me access to the error code that caused the crash and I would like to address the issue and retry opening the process.
runningProcesses is a list of tuples containing the subprocess object and the command associated with it.
def monitorPrograms(runningProcesses):
for program in runningProcesses:
temp = program[0].communicate()
if program[0].returncode:
if program[0].returncode == 1:
print "Program exited successfully."
else:
print "Whoops, something went wrong. Program %s crashed." % program[0].pid
When I tried to get the return code without using communicate, the crash of the program didn't register.
Do I have to use threads to run the communication in parallel or is there a simpler way that I am missing?
No need to use threads, to monitor multiple processes, especially if you don't use their output (use DEVNULL instead of PIPE to hide the output), see Python threading multiple bash subprocesses?
Your main issue is incorrect Popen.poll() usage. If it returns None; it means that the process is still running -- you should call it until you get non-None value. Here's a similar to your case code example that prints ping processes statuses.
If you do want to get subprocess' stdout/stderr as a string then you could use threads, async.io.
If you are on Unix and you control all the code that may spawn subprocesses then you could avoid polling and handle SIGCHLD yourself. asyncio stdlib library may handle SIGCHLD. You could also implement it manually, though it might be complicated.
Based on my research, the best way to do this is with threads. Here's an article that I referenced when creating my own package to solve this problem.
The basic method used here is to spin of threads that constantly request log output (and finally the exit status) of the subprocess call.
Here's an example of my own "receiver" which listens for logs:
class Receiver(threading.Thread):
def __init__(self, stream, stream_type=None, callback=None):
super(Receiver, self).__init__()
self.stream = stream
self.stream_type = stream_type
self.callback = callback
self.complete = False
self.text = ''
def run(self):
for line in iter(self.stream.readline, ''):
line = line.rstrip()
if self.callback:
line = self.callback(line, msg_type=self.stream_type)
self.text += line + "\n"
self.complete = True
And now the code that spins the receiver off:
def _execute(self, command):
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
out = Receiver(process.stdout, stream_type='out', callback=self.handle_log)
err = Receiver(process.stderr, stream_type='err', callback=self.handle_log)
out.start()
err.start()
try:
self.wait_for_complete(out)
except CommandTimeout:
os.killpg(process.pid, signal.SIGTERM)
raise
else:
status = process.poll()
output = CommandOutput(status=status, stdout=out.text, stderr=err.text)
return output
finally:
out.join(timeout=1)
err.join(timeout=1)
CommandOutput is simply a named tuple that makes it easy to reference the data I care about.
You'll notice I have a method 'wait_for_complete' which waits for the receiver to set complete = True. Once complete, the execute method calls process.poll() to get the exit code. We now have all stdout/stderr and the status code of the process.
I'm using python to benchmark something. This can take a large amount of time, and I want to set a (global) timeout. I use the following script (summarized):
class TimeoutException(Exception):
pass
def timeout_handler(signum, frame):
raise TimeoutException()
# Halt problem after half an hour
signal.alarm(1800)
try:
while solution is None:
guess = guess()
try:
with open(solutionfname, 'wb') as solutionf:
solverprocess = subprocess.Popen(["solver", problemfname], stdout=solutionf)
solverprocess.wait()
finally:
# `solverprocess.poll() == None` instead of try didn't work either
try:
solverprocess.kill()
except:
# Solver process was already dead
pass
except TimeoutException:
pass
# Cancel alarm if it's still active
signal.alarm(0)
However it keeps spawning orphan processes sometimes, but I can't reliably recreate the circumstances. Does anyone know what the correct way to prevent this is?
You simply have to wait after killing the process.
The documentation for the kill() method states:
Kills the child. On Posix OSs the function sends SIGKILL to the child.
On Windows kill() is an alias for terminate().
In other words, if you aren't on Windows, you are only sending a signal to the subprocess.
This will create a zombie process because the parent process didn't read the return value of the subprocess.
The kill() and terminate() methods are just shortcuts to send_signal(SIGKILL) and send_signal(SIGTERM).
Try adding a call to wait() after the kill(). This is even shown in the example under the documentation for communicate():
proc = subprocess.Popen(...)
try:
outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
note the call to communicate() after the kill(). (It is equivalent to calling wait() and also erading the outputs of the subprocess).
I want to clarify one thing: it seems like you don't understand exactly what a zombie process is. A zombie process is a terminated process. The kernel keeps the process in the process table until the parent process reads its exit status. I believe all memory used by the subprocess is actually reused; the kernel only has to keep track of the exit status of such a process.
So, the zombie processes you see aren't running. They are already completely dead, and that's why they are called zombie. They are "alive" in the process table, but aren't really running at all.
Calling wait() does exactly this: wait till the subprocess ends and read the exit status. This allows the kernel to remove the subprocess from the process table.
On linux, you can use python-prctl.
Define a preexec function such as:
def pre_exec():
import signal
prctl.set_pdeathsig(signal.SIGTERM)
And have your Popen call pass it.
subprocess.Popen(..., preexec_fn=pre_exec)
That's as simple as that. Now the child process will die rather than become orphan if the parent dies.
If you don't like the external dependency of python-prctl you can also use the older prctl. Instead of
prctl.set_pdeathsig(signal.SIGTERM)
you would have
prctl.prctl(prctl.PDEATHSIG, signal.SIGTERM)
I am trying to write a small app that uses the subprocess module.
My program calls an external Bash command that takes some time to process. During this time, I would like to show the user a series of messages like this:
Processing. Please wait...
The output is foo()
How can I do this using Popen.wait() or Popen.poll(). I have read that I need to use the Popen.returncode, but how I can get it to actively check the state, I don't know.
Both wait() (with timeout specified) and poll() return None if the process has not yet finished, and something different if the process has finished (I think an integer, the exit code, hopefully 0).
Edit:
wait() and poll() have different behaviors:
wait (without the timeout argument) will block and wait for the process to complete.
wait with the timeout argument will wait timeout seconds for the process to complete. If it doesn't complete, it will throw the TimeoutExpired exception. If you catch the exception, you're then welcome to go on, or to wait again.
poll always returns immediately. It effectively does a wait with a timeout of 0, catches any exception, and returns None if the process hasn't completed.
With either wait or poll, if the process has completed, the popen object's returncode will be set (otherwise it's None - you can check for that as easily as calling wait or poll), and the return value from the function will also be the process's return code.
</Edit>
So I think you should do something like:
while myprocess.poll() is None:
print("Still working...")
# sleep a while
Be aware that if the bash script creates a lot of output you must use communicate() or something similar to prevent stdout or stderr to become stuffed.
#extraneon's answer is a little backwards. Both wait() and poll() return the process's exit code if the process has finished. The poll() method will return None if the process is still running and the wait() method will block until the process exits:
Check out the following page: https://docs.python.org/3.4/library/subprocess.html#popen-objects
Popen.poll()
Check if child process has terminated. Set and return returncode attribute.
Popen.wait()
Wait for child process to terminate. Set and return returncode attribute.