child = subprocess.Popen(command,
shell=True,
env=environment,
close_fds=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
stdin=sys.stdin,
preexec_fn=os.setsid
)
child_interrupted = False
while child.poll() is None:
if Signal.isInterrupted():
child_interrupted = True
os.killpg(os.getpgid(child.pid), signal.SIGTERM)
break
time.sleep(0.1)
subout = child.communicate()[0]
logging.info(subout)
the above works for most command it executes (90%) but for some commands it hangs
for those command that repeatedly hangs, if i get rid of the below, it works fine:
child_interrupted = False
while child.poll() is None:
if Signal.isInterrupted():
child_interrupted = True
os.killpg(os.getpgid(child.pid), signal.SIGTERM)
break
time.sleep(0.1)
im assuming for those hanging commands, child.poll() is None even though the job is finished??
communicate() can tell the process is finished but poll() cant?
i've executed ps -ef on those processes
and they are defunct only when child.poll() code is in place
any idea why?
it looks like defunct means "That's a zombie process, it's finished but the parent hasn't wait()ed for it yet."
well, im polling to see if i can call wait/communitcate...
You've set the Popen object to receive the subprocess's stdout via pipe. Problem is, you're not reading from that pipe until the process exits. If the process produces enough output to fill the OS level pipe buffers, and you don't drain the pipe, then you're deadlocked; the subprocess wants you to read the output its writing so it can continue to write, then exit, while you're waiting for it to exit before you'll read the output.
If your explicit poll and interrupt checking is necessary, the easiest solution to this deadlock is probably to launch a thread that drains the pipe:
... launch the thread just after Popen called ...
draineddata = []
# Trivial thread just reads lines from stdout into the list
drainerthread = threading.Thread(target=draineddata.extend, args=(child.stdout,))
drainerthread.daemon = True
drainerthread.start()
... then where you had been doing communicate, change it to: ...
child.wait()
drainerthread.join()
subout = b''.join(draineddata) # Combine the data read back to a single output
Related
I was have to call a shell command
subprocess.Popen(command, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
I did it. and:
and, that command is prints lots of things like verbose is on, and then when its done it's job it prints (writes) blah blah : Ready
I have to call this command, wait for the 'Ready' text and leave it running on background, then let the rest of the code run
I tried this and things like this, didn't work
...
done=False
with subprocess.Popen(command, stderr=subprocess.STDOUT, stdout=subprocess.PIPE) as proc:
while not done:
x=proc.stdout.read()
x=x.find('Ready')
if x > -1:
done=True
print("YaaaY, this subprocess is done it's job and now running on background")
#rest of the code
i ran similar (edited) code on python terminal and I think I can't even access (read) the terminal of the subprocess. because...
I was expecting it will show every line that this subprocess print but. its just waiting.
Your problem is proc.stdout.read(). This reads the entire output of your subprocess, which is not known until it has terminated (usually). Try something like:
output = b''
while not done:
output += proc.stdout.read(1)
x = output.find(b'Ready')
if x > -1:
done = True
This reads the process's stdout one character at a time, so it doesn't have to wait for it to finish.
Note that using Popen in a context manager (with block) will cause your program to wait for the child process to terminate before it exits the with block, so it will not leave it running past that point. It's unclear if that is desired behaviour or not.
Trying to use Python subprocesses to play audio on a Raspberry Pi and kill the subprocess when a GPIO-connected button is pressed.
The issue I'm having is that the subprocess won't kill when there's a subprocess Popen.wait() command between playback files. If there's only one file to play, there's no need to wait for it to finish, and a p.kill() command works just fine. As soon as a p.wait() is inserted so that the files don't play on top of each other, Popen.kill() no longer works.
Already tried using os.kill() in
Kill a running subprocess call. Can't find any other resources about how subprocesses behave under Popen.wait(). I'm looking for a way to force the aplay function in the code below to close on a button press at any time during the three play_wav commands in the second code snippet instead of having just one play_wav command.
Here's the play_wav function utilizing subprocess.Popen():
# play wav file on the attached system sound device
def play_wav(wav_filename):
global p
msg = "playing " + wav_filename
logger.debug(msg)
p = subprocess.Popen(
['aplay','-i','-D','plughw:1', wav_filename],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.STDOUT,
shell = False
)
Here's a snippet of code calling the play_wav function:
# determine what to do when a button is pressed
def button_pressed(channel):
global MENU
btnval = readadc(0, SPICLK, SPIMOSI, SPIMISO, SPICS) # check value of ADC
if btnval > 980: # 1
if p.poll() == None:
p.kill()
if MENU == 1:
play_wav("/media/pi/WONDERPHONE/prompts/en/Menu1.wav")
p.wait()
play_wav("/media/pi/WONDERPHONE/stories/1/PersonalStory.wav")
p.wait()
play_wav("/media/pi/WONDERPHONE/prompts/en/returntomain.wav")
How I'm checking for button presses:
GPIO.add_event_detect(PRESSED, GPIO.RISING, callback=button_pressed, bouncetime=500) # look for button presses
From the subprocess module docs:
Popen.wait()
Wait for child process to terminate. Set and return returncode attribute.
Warning - This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.
Try using communicate() before you kill the subprocess
You can use terminal to do that.
ps aux | grep taskname
sudo kill -9 taskid
I've been reading up on a lot of documentations but am still not sure what I'm doing wrong.
So I have a separate shell script that fires up a separate server then the one I'm working on. Once the server is connected, I want to run ls and that's it. However, for some reason stdin=subprocess.PIPE is preventing the Popen command from terminating so that the next line could execute. For example because the code is stuck I'll Ctrl+C but I'll get an error saying that wait() got a keyboard interrupt. Here's an example code:
import subprocess
from time import sleep
p1 = subprocess.Popen("run_server",
stdout = subprocess.PIPE,
stdin = subprocess.PIPE)
#sleep(1)
p1.wait()
p1.communicate(input = "ls")[0]"
If I replace p1.wait() with sleep(1), the communicate command does run and displays ls, but the script that runs the server detects eof on tty and terminates it self. I must have some kind of wait between Popen and communicate because the server script will terminate for the same reason.
p.wait() does not return until the child process is dead. While the parent script is stuck on p.wait() call; your child process expects input at the same time -- deadlock. Then you press Ctrl+C in the shell; it sends SIGINT signal to all processes in the foreground process group that kills both your parent Python script and run_server subprocess.
You should drop the .wait() call:
#!/usr/bin/env python
from subprocess import Popen, PIPE
p = Popen(["run_server"], stdout=PIPE, stdin=PIPE)
output = p.communicate(b"ls")[0]
Or in Python 3.4+:
#!/usr/bin/env python3
from subprocess import check_output
output = check_output(["run_server"], input=b"ls")
If you want to run several commands then pass them all at once:
input = "\n".join(["ls", "cmd2", "etc"]) # with universal_newlines=True
As you know from reading the subprocess docs, p.communicate() waits for the child process to exit and therefore it should be called at most once. As well as with .wait(), the child process is dead after .communicate() has returned.
The fact that when you Ctrl+C and your traceback says you were stuck in wait() means the next line is executing, the next line is wait(). wait() won't return until your p1 process returns. However, it seems your p1 process won't return until you send it a command, 'ls' in your case. Try sending the command then calling wait().:
import subprocess
from time import sleep
p1 = subprocess.Popen("run_server",
stdout = subprocess.PIPE,
stdin = subprocess.PIPE)
#sleep(1)
p1.communicate(input = "ls")[0]"
p1.wait()
Otherwise, make sure your "run_server" script terminates so your script can advance past p1.wait()
I'm using the following command to run a shell command (creating a subprocess):
cmd = "ls"
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
Then, I want to get its return code when it's finished. I should use wait() or poll()? It seems to me wait() is equal to a poll() enclosed in a busy wait. Something like:
while process.poll() == None:
time.sleep(0.5)
I read that wait() could generate a deadlock if stdout/stderr buffer is filled. process.poll() used like above also could generate a deadlock? If this is true,
I should use process.comunicate() to solve the problem? Nowadays, I only use
process.comunicate() when I'm interested in the subprocess stdout/stderr.
Thanks in advance.
Yes. subprocess.poll, when used in a loop like that, will cause a deadlock if the stdout is piped into your process and you aren't reading from it. If you don't pipe stdout or you're consistently reading from it, neither poll nor wait will deadlock. subprocess.communicate will solve the deadlock in the cases it would occur. However, if you just want to run a command, check its return code, and get its output, use subprocess.check_output, which wraps all of that.
I've got a command that I'm wrapping in script and spawning from a Python script using subprocess.Popen. I'm trying to make sure it dies if the user issues a SIGINT.
I could figure out if the process was interrupted in a least two ways:
A. Die if the wrapped command has a non-zero exit status (doesn't work, because script seems to always return 0)
B. Do something special with SIGINT in the parent Python script rather than simply interrupting the subprocess. I've tried the following:
import sys
import signal
import subprocess
def interrupt_handler(signum, frame):
print "While there is a 'script' subprocess alive, this handler won't executes"
sys.exit(1)
signal.signal(signal.SIGINT, interrupt_handler)
for n in range( 10 ):
print "Going to sleep for 2 second...Ctrl-C to exit the sleep cycles"
# exit 1 if we make it to the end of our sleep
cmd = [ 'script', '-q', '-c', "sleep 2 && (exit 1)", '/dev/null']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
if p.poll() != None :
break
else :
pass
# Exiting on non-zero exit status would suffice
print "Exit status (script always exits zero, despite what happened to the wrapped command):", p.returncode
I'd like hitting Ctrl-C to exit the python script. What's happening instead is the subprocess dies and the script continues.
The subprocess is by default part of the same process group, and only one can control and receive signals from the terminal, so there are a couple of different solutions.
Setting stdin as a PIPE (in contrast to inheriting from the parent process), this will prevent the child process from receiving signals associated to it.
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
Detaching from the parent process group, the child will no longer receive signals
def preexec_function():
os.setpgrp()
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=preexec_function)
Explicitly ignoring signals in the child process
def preexec_function():
signal.signal(signal.SIGINT, signal.SIG_IGN)
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=preexec_function)
This might however be overwritten by the child process.
Fist thing; there is a send_signal() method on the Popen object. If you want to send a signal to one you've launched, use this method to send it.
Second thing; a deeper problem with the way you're setting up communication with your subprocess and then, um, not communicating with it. You cannot safely tell the subprocess to send its output to subprocess.PIPE and then not read from the pipes. UNIX pipes are buffered (typically a 4K buffer?), and if the subprocess fills up the buffer and the process on the other end of the pipe doesn't read the buffered data, the subprocess will pend (locking up, from an observer's perspective) on its next write to the pipe. So, the usual pattern when using subprocess.PIPE is to call communicate() on the Popen object.
It is not mandatory to use subprocess.PIPE if you want data back from the subprocess. A cool trick is to use the tempfile.TemporaryFile class to make an unnamed temp file (really it opens a file and immediately deletes the inode from the file system, so you have access to the file but no-one else can open one. You can do something like:
with tempfile.TemporaryFile() as iofile:
p = Popen(cmd, stdout=iofile, stderr=iofile)
while True:
if p.poll() is not None:
break
else:
time.sleep(0.1) # without some sleep, this polling is VERY busy...
Then you can read the contents of your temporary file (seek to the beginning of it before you do, to be sure you're at the beginning) when you know the subprocess has exited, instead of using pipes. The pipe buffering problem won't be a problem if the subprocess's output is going to a file (temporary or not).
Here is a riff on your code sample that I think does what you want. The signal handler just repeats the signals being trapped by the parent process (in this example, SIGINT and SIGTERM) to all current subprocesses (there should only ever be one in this program) and sets a module-level flag saying to shutdown at the next opportunity. Since I'm using subprocess.PIPE I/O, I call communicate() on the Popen object.
#!/usr/bin/env python
from subprocess import Popen, PIPE
import signal
import sys
current_subprocs = set()
shutdown = False
def handle_signal(signum, frame):
# send signal recieved to subprocesses
global shutdown
shutdown = True
for proc in current_subprocs:
if proc.poll() is None:
proc.send_signal(signum)
signal.signal(signal.SIGINT, handle_signal)
signal.signal(signal.SIGTERM, handle_signal)
for _ in range(10):
if shutdown:
break
cmd = ["sleep", "2"]
p = Popen(cmd, stdout=PIPE, stderr=PIPE)
current_subprocs.add(p)
out, err = p.communicate()
current_subprocs.remove(p)
print "subproc returncode", p.returncode
And calling it (with a Ctrl-C in the third 2 second interval):
% python /tmp/proctest.py
subproc returncode 0
subproc returncode 0
^Csubproc returncode -2
This hack will work, but it's ugly...
Change the command to this:
success_flag = '/tmp/success.flag'
cmd = [ 'script', '-q', '-c', "sleep 2 && touch " + success_flag, '/dev/null']
And put
if os.path.isfile( success_flag ) :
os.remove( success_flag )
else :
return
at the end of the for loop
If you have no python processing to do after your process is spawned (like in your example), then the easiest way is to use os.execvp instead of the subprocess module. Your subprocess is going to completely replace your python process, and will be the one catching SIGINT directly.
I found a -e switch in the script man page:
-e Return the exit code of the child process. Uses the same format
as bash termination on signal termination exit code is 128+n.
Not too sure what the 128+n is all about but it seems to return 130 for ctrl-c. So modifying your cmd to be
cmd = [ 'script', '-e', '-q', '-c', "sleep 2 && (exit 1)", '/dev/null']
and putting
if p.returncode == 130:
break
at the end of the for loop seems to do what you want.