Develop Reference

subprocess.popen is failing to retry

What specific syntax must be changed below in order to get the call to subprocess.popen to retry if no response is received in n seconds?
def runShellCommand(cmd):
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
process.wait()
The problem we are having is that the command is succeeding but the command is not receiving a response. This means that the runShellCommand(cmd) function is just hanging forever.
If the process.wait() lasted only n seconds and then retried running the same cmd, then repeated the call/wait cycle 3 or 4 times, then the function could either receive a response from one of the subsequent tries and return successful, or could fail gracefully within a specified maximum period of time.

Your process is probably deadlocking due to the STDOUT buffer filling up.
Setting stdout=subprocess.PIPE makes the process redirect STDOUT to a file called process.stdout instead of the terminal output. However, process.wait() doesn't ever read from process.stdout. Therefor, when process.stdout fills up (usually after a few megabytes of output), then the process deadlocks. The process is waiting for STDOUT (directed to process.stdout) to get read, but it will never get read because process.wait() is waiting for the process to finish, which it can't do because it's waiting to print to STDOUT... and that's the deadlock.
To solve this and read the output, use something like:
def runShellCommand(cmd):
return subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, text=True).stdout
Note that text=True requires Python 3.7 or later. Before that, use universal_newlines=True for the same effect, or leave that argument out to get the results as bytes instead.
Security note:
Please consider removing shell=True. It's horribly unsafe (subject to the variable expansion whims of your shell, which could be almost anything from a simple POSIX sh or bash, but also something more unusual like tcsh, zsh, or even a totally unexpected custom shell compiled by the user or their sysadmin).
E.g. instead of
subprocess.run('echo "Hello, World!"', shell=True)
You can use this more safely:
subprocess.run(['echo', 'Hello, World!'])

Child Process is failing using Popen

I'm trying to get a program named xselect to run using the Popen construct in python. If I run xselect from the terminal manually by typing in the commands by hand, It runs all the way through. However, when done from the python script, it freezes at a certain command and will not continue. When I check the log file, all of the output is captured, but none of the error messages are captured.
I'm thinking that Popen may not know what to do with the errors from the output of xselect, and its causing xselect to freeze. To counter this, I tried to add a timeout so that it kills xselect after 5 seconds, but this hasn't worked either.
Can anyone help me get this running?
with subprocess.Popen(args, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, universal_newlines=True) as proc:
proc.wait(timeout=5)
out = proc.stdout.read()
if TimeoutExpired:
proc.kill()

See the warning for proc.wait() here: https://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait. Basically, you should either be using proc.communicate(), or should be reading from proc.stdout instead of waiting.

What is the difference between subprocess.popen and subprocess.run

I'm new to the subprocess module and the documentation leaves me wondering what the difference is between subprocess.popen and subprocess.run. Is there a difference in what the command does? Is one just newer? Which is better to use?

subprocess.run() was added in Python 3.5 as a simplification over subprocess.Popen when you just want to execute a command and wait until it finishes, but you don't want to do anything else in the mean time. For other cases, you still need to use subprocess.Popen.
The main difference is that subprocess.run() executes a command and waits for it to finish, while with subprocess.Popen you can continue doing your stuff while the process finishes and then just repeatedly call Popen.communicate() yourself to pass and receive data to your process. Secondly, subprocess.run() returns subprocess.CompletedProcess.
subprocess.run() just wraps Popen and Popen.communicate() so you don't need to make a loop to pass/receive data or wait for the process to finish.
Check the official documentation for info on which params subprocess.run() pass to Popen and communicate().

Both available in Python by default.
The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.
-Subprocess.run:
import subprocess
import sys
result = subprocess.run([sys.executable, "-c", "print('ocean')"])
-Subprocess.popen: run multiple command line with subprocess, communicate method waits for the process to finish and finally prints the stdout and stderr as a tuple
EX:
import subprocess
process = subprocess.Popen(shell_cmd,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE,
text = True,
shell = True
)
std_out, std_err = process.communicate()
std_out.strip(), std_err

have a .exe run in the background and type things into that through python

I have a program, myshell.exe, that i need to interact with through python (send commands to it and read results back).
The catch is that i can only run myshell.exe once (cannot enclose popen and communicate in a loop)
I have tried popen and popen.communicate() but that seems to run myshell.exe, send my commands and then exits the process.
# settin up the command
p = Popen("myshell.exe", stdout=PIPE, stdin=PIPE, stderr=PIPE, shell=True)
# sending something (and getting output)
print p.communicate("run");
At this point, from the print output i can see that my myshell.exe has exited (i have a goodbye message that is printed).
Any ideas if there is any way around it ?
Thanks.

As you can read in Popen.communicate docs, it will wait until myshell.exe exits before returning.
Use p.stdout and p.stdin to communicate with the process instead:
p.stdin.write("run")
print p.stdout.read(1024)
p.stdin and p.stdout are regular file objects. You can read and write to them in a loop, just leave the p = Popen(...) part outside:
p = Popen("myshell.exe", stdout=PIPE, stdin=PIPE, stderr=PIPE, shell=True)
for i in range(3):
p.stdin.write("run")
print p.stdout.read(16)
p.terminate()
This assuming that myshell.exe is behaving as you expect (e.g. does not exit after first command is sent).

Python - Execute Process -> Block till it exits & Suppress Output

I'm using the following to execute a process and hide its output from Python. It's in a loop though, and I need a way to block until the sub process has terminated before moving to the next iteration.
subprocess.Popen(["scanx", "--udp", host], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Use subprocess.call(). From the docs:
subprocess.call(*popenargs, **kwargs)
Run command with arguments. Wait for command to complete, then
return the returncode attribute.
The arguments are the same as for the
Popen constructor.
Edit:
subprocess.call() uses wait(), and wait() is vulnerable to deadlocks (as Tommy Herbert pointed out). From the docs:
Warning: This will deadlock if the
child process generates enough output
to a stdout or stderr pipe such that
it blocks waiting for the OS pipe
buffer to accept more data. Use
communicate() to avoid that.
So if your command generates a lot of output, use communicate() instead:
p = subprocess.Popen(
["scanx", "--udp", host],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = p.communicate()

If you don't need output at all you can pass devnull to stdout and stderr. I don't know if this can make a difference but pass a bufsize. Using devnull now subprocess.call doesn't suffer of deadlock anymore
import os
import subprocess
null = open(os.devnull, 'w')
subprocess.call(['ls', '-lR'], bufsize=4096, stdout=null, stderr=null)