Develop Reference

Why doesn't subprocess.Popen() give live feed?

I'm working on an automated framework for a bioinformatics tool. As most software that my program will use is written for Linux and not written in python, I use subprocess to invoke the processes.
The problem I have is that many steps in the pipeline takes very long time and I want to see the live output so I know that it's still working and has not hung or something. But I will also need to capture the output to log any unexpected errors after the process is done.
I found that subprocces.Popen() is what I need for this issue.
This is the code I use (found here: https://fabianlee.org/2019/09/15/python-getting-live-output-from-subprocess-using-poll/):
# invoke process
process = subprocess.Popen("./test.sh", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# print stdout while process is still working
while True:
output = process.stdout.readline()
if process.poll() is not None:
break
if output:
print("out:", output.strip())
rc = process.poll()
if rc == 0:
print("Process ended with rc:", rc, "output:", output)
else:
print("Process ended with rc:", rc, "error:", process.stderr.readline())
It works like a charm when I use this simple bash script as argument:
#!/bin/bash
for i in $(seq 1 5); do
echo "iteration" $i
sleep 1
done
which gives the output:
out: iteration 1
out: iteration 2
out: iteration 3
out: iteration 4
out: iteration 5
Process ended with rc: 0 output:
or this if i deliberately insert an error in the script, e.g.:
Process ended with rc: 2 error: ./test.sh: line 7: syntax error: unexpected end of file
Hovever, when I try it with (in this case picard ValidateSamFile) it does not give me any livefeed no matter what I have tried:
# invoke process
process = subprocess.Popen("picard ValidateSamFile -I dna_seq/aligned/2064-01/AHWM2NCCXY.RJ-1967-2064-01.6.bam", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# print stdout while process is still working
while True:
output = process.stdout.readline()
if process.poll() is not None:
break
if output:
print("out:", output.strip())
rc = process.poll()
if rc == 0:
print("Process ended with rc:", rc, "output:", output)
else:
print("Process ended with rc:", rc, "error:", process.stderr.readline())
I get this after the process is completed:
out: No errors found
Process ended with rc: 0 output:
Any ideas?

python - read linux command output continuously

I have a command that provides event stream - new message every few second.
How do I read this as it comes with python?
The standard approach with
def getMessage(command):
lines = os.popen(command).readlines()
return lines
waits for the command to complete, but in this command run forever. It will continue on and print new message to stdout every few seconds.
How do I pass it on to python? I want to capture all messages in the stream.

You can read the output line by line and process/print it. Meanwhile use p.poll to check if the process has ended.
def get_message(command):
p = subprocess.Popen(
command,
stdout=subprocess.PIPE,
)
while True:
output = p.stdout.readline()
if output == '' and p.poll() is not None:
break
if output:
yield output.strip()

Running a process as a different user from Python *and* print exit code

I am running a python script as root, from this script I want to run on linux process as userA.
There is many answers on how to do that
but I also need to print the exit code recived from the process and that the real isuue here.
Is there a way of running a process as userA and also to print the return value ?
os.system("su userA -c 'echo $USER'")
answer = subprocess.call(["su userA -c './my/path/run.sh'"])
print(answer)

When you're running a script and you want to print its return code, you must wait until its execution is done before executing the print command.
The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.
http://docs.python.org/library/subprocess.html
In your case:
import subprocess
process = subprocess.Popen('su userA -c ./my/path/run.sh', shell=True, stdout=subprocess.PIPE)
process.wait()
print process.returncode
Reference:
https://stackoverflow.com/a/325474/13798864

If you use os.fork (with an os.setuid inside the child) you can collect the status using os.waitpid.
pid = os.fork()
if pid == 0:
os.setgid(...)
os.setgroups(...)
os.setuid(...)
# do something and collect the exit status... for example
#
# using os.system:
#
# return_value = os.system(.....) // 256
#
# or using subprocess:
#
# p = subprocess.Popen(....)
# out, err = p.communicate()
# return_value = p.returncode
#
# Or you can simply exec (this will not return to python but the
# exit status will still be visible in the parent). Note there are
# several os.exec* calls, so choose the one which you want.
#
# os.exec...
#
os._exit(return_value)
pid, status = os.waitpid(pid, 0)
print(f"Child exit code was {status // 256}")
Here I recently posted an answer to a related question which was not so much focused on the return value but does include some more details of the values that you might pass to the os.setuid etc calls.

Display process output incrementally using Python subprocess

I'm trying to run "docker-compose pull" from inside a Python automation script and to incrementally display the same output that Docker command would print if it was run directly from the shell. This command prints a line for each Docker image found in the system, incrementally updates each line with the Docker image's download progress (a percentage) and replaces this percentage with a "done" when the download has completed. I first tried getting the command output with subprocess.poll() and (blocking) readline() calls:
import shlex
import subprocess
def run(command, shell=False):
p = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell)
while True:
# print one output line
output_line = p.stdout.readline().decode('utf8')
error_output_line = p.stderr.readline().decode('utf8')
if output_line:
print(output_line.strip())
if error_output_line:
print(error_output_line.strip())
# check if process finished
return_code = p.poll()
if return_code is not None and output_line == '' and error_output_line == '':
break
if return_code > 0:
print("%s failed, error code %d" % (command, return_code))
run("docker-compose pull")
The code gets stuck in the first (blocking) readline() call. Then I tried to do the same without blocking:
import select
import shlex
import subprocess
import sys
import time
def run(command, shell=False):
p = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell)
io_poller = select.poll()
io_poller.register(p.stdout.fileno(), select.POLLIN)
io_poller.register(p.stderr.fileno(), select.POLLIN)
while True:
# poll IO for output
io_events_list = []
while not io_events_list:
time.sleep(1)
io_events_list = io_poller.poll(0)
# print new output
for event in io_events_list:
# must be tested because non-registered events (eg POLLHUP) can also be returned
if event[1] & select.POLLIN:
if event[0] == p.stdout.fileno():
output_str = p.stdout.read(1).decode('utf8')
print(output_str, end="")
if event[0] == p.stderr.fileno():
error_output_str = p.stderr.read(1).decode('utf8')
print(error_output_str, end="")
# check if process finished
# when subprocess finishes, iopoller.poll(0) returns a list with 2 select.POLLHUP events
# (one for stdout, one for stderr) and does not enter in the inner loop
return_code = p.poll()
if return_code is not None:
break
if return_code > 0:
print("%s failed, error code %d" % (command, return_code))
run("docker-compose pull")
This works, but only the final lines (with "done" at the end) are printed to the screen, when all Docker images downloads have been completed.
Both methods work fine with a command with simpler output such as "ls". Maybe the problem is related with how this Docker command prints incrementally to screen, overwriting already written lines ? Is there a safe way to incrementally show the exact output of a command in the command line when running it via a Python script?
EDIT: 2nd code block was corrected

Always openSTDIN as a pipe, and if you are not using it, close it immediately.
p.stdout.read() will block until the pipe is closed, so your polling code does nothing useful here. It needs modifications.
I suggest not to use shell=True
Instead of *.readline(), try with *.read(1) and wait for "\n"
Of course you can do what you want in Python, the question is how. Because, a child process might have different ideas about how its output should look like, that's when trouble starts. E.g. the process might want explicitly a terminal at the other end, not your process. Or a lot of such simple nonsense. Also, a buffering may also cause problems. You can try starting Python in unbuffered mode to check. (/usr/bin/python -U)
If nothing works, then use pexpect automation library instead of subprocess.

I have found a solution, based on the first code block of my question:
def run(command,shell=False):
p = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell)
while True:
# read one char at a time
output_line = p.stderr.read(1).decode("utf8")
if output_line != "":
print(output_line,end="")
else:
# check if process finished
return_code = p.poll()
if return_code is not None:
if return_code > 0:
raise Exception("Command %s failed" % command)
break
return return_code
Notice that docker-compose uses stderr to print its progress instead of stdout. #Dalen has explained that some applications do it when they want that their results are pipeable somewhere, for instance a file, but also want to be able to show their progress.

How to control background process in linux

I need to write a script in Linux which can start a background process using one command and stop the process using another.
The specific application is to take userspace and kernel logs for android.
following command should start taking logs
$ mylogscript start
following command should stop the logging
$ mylogscript stop
Also, the commands should not block the terminal. For example, once I send the start command, the script run in background and I should be able to do other work on terminal.
Any pointers on how to implement this in perl or python would be helpful.
EDIT:
Solved: https://stackoverflow.com/a/14596380/443889

I got the solution to my problem. Solution essentially includes starting a subprocess in python and sending a signal to kill the process when done.
Here is the code for reference:
#!/usr/bin/python
import subprocess
import sys
import os
import signal
U_LOG_FILE_PATH = "u.log"
K_LOG_FILE_PATH = "k.log"
U_COMMAND = "adb logcat > " + U_LOG_FILE_PATH
K_COMMAND = "adb shell cat /proc/kmsg > " + K_LOG_FILE_PATH
LOG_PID_PATH="log-pid"
def start_log():
if(os.path.isfile(LOG_PID_PATH) == True):
print "log process already started, found file: ", LOG_PID_PATH
return
file = open(LOG_PID_PATH, "w")
print "starting log process: ", U_COMMAND
proc = subprocess.Popen(U_COMMAND,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
print "log process1 id = ", proc.pid
file.write(str(proc.pid) + "\n")
print "starting log process: ", K_COMMAND
proc = subprocess.Popen(K_COMMAND,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
print "log process2 id = ", proc.pid
file.write(str(proc.pid) + "\n")
file.close()
def stop_log():
if(os.path.isfile(LOG_PID_PATH) != True):
print "log process not started, can not find file: ", LOG_PID_PATH
return
print "terminating log processes"
file = open(LOG_PID_PATH, "r")
log_pid1 = int(file.readline())
log_pid2 = int(file.readline())
file.close()
print "log-pid1 = ", log_pid1
print "log-pid2 = ", log_pid2
os.killpg(log_pid1, signal.SIGTERM)
print "logprocess1 killed"
os.killpg(log_pid2, signal.SIGTERM)
print "logprocess2 killed"
subprocess.call("rm " + LOG_PID_PATH, shell=True)
def print_usage(str):
print "usage: ", str, "[start|stop]"
# Main script
if(len(sys.argv) != 2):
print_usage(sys.argv[0])
sys.exit(1)
if(sys.argv[1] == "start"):
start_log()
elif(sys.argv[1] == "stop"):
stop_log()
else:
print_usage(sys.argv[0])
sys.exit(1)
sys.exit(0)

There are a couple of different approaches you can take on this:
1. Signal - you use a signal handler, and use, typically "SIGHUP" to signal the process to restart ("start"), SIGTERM to stop it ("stop").
2. Use a named pipe or other IPC mechanism. The background process has a separate thread that simply reads from the pipe, and when something comes in, acts on it. This method relies on having a separate executable file that opens the pipe, and sends messages ("start", "stop", "set loglevel 1" or whatever you fancy).
I'm sorry, I haven't implemented either of these in Python [and perl I haven't really written anything in], but I doubt it's very hard - there's bound to be a ready-made set of python code to deal with named pipes.
Edit: Another method that just struck me is that you simply daemonise the program at start, and then let the "stop" version find your deamonized process [e.g. by reading the "pidfile" that you stashed somewhere suitable], and then sends a SIGTERM for it to terminate.

I don't know if this is the optimum way to do it in perl, but for example:
system("sleep 60 &")
This starts a background process that will sleep for 60 seconds without blocking the terminal. The ampersand in shell means to do something in the background.
A simple mechanism for telling the process when to stop is to have it periodically check for the existence of a certain file. If the file exists, it exits.