What specific syntax must be changed below in order to get the call to subprocess.popen to retry if no response is received in n seconds?
def runShellCommand(cmd):
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
process.wait()
The problem we are having is that the command is succeeding but the command is not receiving a response. This means that the runShellCommand(cmd) function is just hanging forever.
If the process.wait() lasted only n seconds and then retried running the same cmd, then repeated the call/wait cycle 3 or 4 times, then the function could either receive a response from one of the subsequent tries and return successful, or could fail gracefully within a specified maximum period of time.
Your process is probably deadlocking due to the STDOUT buffer filling up.
Setting stdout=subprocess.PIPE makes the process redirect STDOUT to a file called process.stdout instead of the terminal output. However, process.wait() doesn't ever read from process.stdout. Therefor, when process.stdout fills up (usually after a few megabytes of output), then the process deadlocks. The process is waiting for STDOUT (directed to process.stdout) to get read, but it will never get read because process.wait() is waiting for the process to finish, which it can't do because it's waiting to print to STDOUT... and that's the deadlock.
To solve this and read the output, use something like:
def runShellCommand(cmd):
return subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, text=True).stdout
Note that text=True requires Python 3.7 or later. Before that, use universal_newlines=True for the same effect, or leave that argument out to get the results as bytes instead.
Security note:
Please consider removing shell=True. It's horribly unsafe (subject to the variable expansion whims of your shell, which could be almost anything from a simple POSIX sh or bash, but also something more unusual like tcsh, zsh, or even a totally unexpected custom shell compiled by the user or their sysadmin).
E.g. instead of
subprocess.run('echo "Hello, World!"', shell=True)
You can use this more safely:
subprocess.run(['echo', 'Hello, World!'])
Related
I am trying to run a python file that prints something, waits 2 seconds, and then prints again. I want to catch these outputs live from my python script to then process them. I tried different things but nothing worked.
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if process.poll() is not None and output == '':
break
if output:
print(output.strip())
I'm at this point but it doesn't work. It waits until the code finishes and then prints all the outputs.
I just need to run a python file and get live outputs from it, if you have other ideas for doing it, without using the print function let me know, just know that I have to run the file separately. I just thought of the easiest way possible but, from what I'm seeing it can't be done.
There are three layers of buffering here, and you need to limit all three of them to guarantee you get live data:
Use the stdbuf command (on Linux) to wrap the subprocess execution (e.g. run ['stdbuf', '-oL'] + cmd instead of just cmd), or (if you have the ability to do so) alter the program itself to either explicitly change the buffering on stdout (e.g. using setvbuf for C/C++ code to switch stdout globally to line-buffered mode, rather than the default block buffering it uses when outputting to a non-tty) or to insert flush statements after critical output (e.g. fflush(stdout); for C/C++, fileobj.flush() for Python, etc.) the buffering of the program to line-oriented mode (or add fflushs); without that, everything is stuck in user-mode buffers of the sub-process.
Add bufsize=0 to the Popen arguments (probably not needed since you don't send anything to stdin, but harmless) so it unbuffers all piped handles. If the Popen is in text=True mode, switch to bufsize=1 (which is line-buffered, rather than unbuffered).
Add flush=True to the print arguments (if you're connected to a terminal, the line-buffering will flush it for you, so it's only if stdout is piped to a file that this will matter), or explicitly call sys.stdout.flush().
Between the three of these, you should be able to guarantee no data is stuck waiting in user-mode buffers; if at least one line has been output by the sub-process, it will reach you immediately, and any output triggered by it will also appear immediately. Item #1 is the hardest in most cases (when you can't use stdbuf, or the process reconfigures its own buffering internally and undoes the effect of stdbuf, and you can't modify the process executable to fix it); you have complete control over #2 and #3, but #1 may be outside your control.
This is the code I use for that same purpose:
def run_command(command, **kwargs):
"""Run a command while printing the live output"""
process = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
**kwargs,
)
while True: # Could be more pythonic with := in Python3.8+
line = process.stdout.readline()
if not line and process.poll() is not None:
break
print(line.decode(), end='')
An example of usage would be:
run_command(['git', 'status'], cwd=Path(__file__).parent.absolute())
I'm creating a program where I need to use a powershell session and I found out how I could have a persistent session using the below code. However I want to loop through the new lines of the output of powershell when a command has been run. The for loop below is the only way i've found to do so but it expects an EOF and doesn't get it so it just lingers and the program never exits. How can I get the amount of new lines in stdout so I can properly loop through them?
from subprocess import Popen, PIPE
process = Popen(["powershell"], stdin=PIPE, stdout=PIPE)
def ps(command):
command = bytes("{}\n".format(command), encoding='utf-8')
process.stdin.write(command)
process.stdin.flush()
process.stdout.readline()
return process.stdout.readline().decode("utf-8")
ps("echo hello world")
for line in process.stdout:
print(line.strip().decode("utf-8"))
process.stdin.close()
process.wait()
You need the Powershell command to know when to exit. Typically, the solution is to not just flush, but close the stdin for the child process; when it's done with its work and finds EOF on its input, it should exit on its own. Just change:
process.stdin.flush()
to:
process.stdin.close()
which implies a flush and also ensures the child process knows input is done. If that doesn't work on its own, you might explicitly add a quit or exit (whatever Powershell uses to terminate the session manually) command after the command you're actually running.
If you must run multiple commands in the single subprocess, and each command must be fully consumed before the next one is sent, there are terrible heuristic solutions available, e.g. sending three commands at once, where the second simply echoes a sentinel string and the third explicitly flushes stdout (to ensure block buffering doesn't mean you deadlock waiting for the sentinel when its stuck in subprocess's internal buffers), and your loop can terminate once it sees the sentinel. Without a sentinel, it's worse, because you basically can't tell when the command is done, and just have to use the select/selectors module to poll the process's stdout with a timeout, reading lines whenever there is available data, and assuming the process is done if no new input is available without the expected timeout window.
I'm trying to get a program named xselect to run using the Popen construct in python. If I run xselect from the terminal manually by typing in the commands by hand, It runs all the way through. However, when done from the python script, it freezes at a certain command and will not continue. When I check the log file, all of the output is captured, but none of the error messages are captured.
I'm thinking that Popen may not know what to do with the errors from the output of xselect, and its causing xselect to freeze. To counter this, I tried to add a timeout so that it kills xselect after 5 seconds, but this hasn't worked either.
Can anyone help me get this running?
with subprocess.Popen(args, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, universal_newlines=True) as proc:
proc.wait(timeout=5)
out = proc.stdout.read()
if TimeoutExpired:
proc.kill()
See the warning for proc.wait() here: https://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait. Basically, you should either be using proc.communicate(), or should be reading from proc.stdout instead of waiting.
I am using subprosess to ping a server, but I want to receive the full response. In the past I have used os to ping but this only returned 1 or 0.
The code that I am using is:
import subprocess
p = subprocess.Popen(['ping', '8.8.8.8'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print out
I would like if if the response that I would see if I had ran it from the terminal was visible. Please note that I am using -c and not -n because I am using linux as the OS.
I am confused as to why this does't work because when I ran similar code, it printed out the expected response:
import subprocess
p = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print out
The code above printed out the list of files and folders in the directory which the python script was saved in, so the code that I am using seems to be correct.
My question is, how can I have the response from pinging a server assigned to a variable that I can then print out like I can when I run ls.
The default ping is a continuous process; it doesn't stop until you interrupt it, e.g. with ctrl-C. Of course, with subprocess, ctrl-C is not possible.
subprocess.communicate will buffer all the output in memory until the program ends (that is, never); you could use it in fact to create an out-of-memory error ;-).
If you just like a few pings, or even 1 ping, use the -c option to ping:
p = subprocess.Popen(['ping', '-c1', '8.8.8.8'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
If you'd like continuous polling, you could wrap this in a while loop inside Python.
.communicate() waits for the child process to finish. ping with the given arguments does not exit unless you explicitly stop it e.g., send it Ctrl+C.
Here're varous methods to stop reading process output in Python without hang.
If want to see the output while the process is still running; see Python: read streaming input from subprocess.communicate()
I'm using the following command to run a shell command (creating a subprocess):
cmd = "ls"
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
Then, I want to get its return code when it's finished. I should use wait() or poll()? It seems to me wait() is equal to a poll() enclosed in a busy wait. Something like:
while process.poll() == None:
time.sleep(0.5)
I read that wait() could generate a deadlock if stdout/stderr buffer is filled. process.poll() used like above also could generate a deadlock? If this is true,
I should use process.comunicate() to solve the problem? Nowadays, I only use
process.comunicate() when I'm interested in the subprocess stdout/stderr.
Thanks in advance.
Yes. subprocess.poll, when used in a loop like that, will cause a deadlock if the stdout is piped into your process and you aren't reading from it. If you don't pipe stdout or you're consistently reading from it, neither poll nor wait will deadlock. subprocess.communicate will solve the deadlock in the cases it would occur. However, if you just want to run a command, check its return code, and get its output, use subprocess.check_output, which wraps all of that.