I'm wrestling with the concepts behind subprocesses and pipes, and working with them in a Python context. If anybody could shed some light on these questions it would really help me out.
Say I have a pipeline set up as follows
createText.py | processText.py | cat
processText.py is receiving data through stdin, but how is this implemented? How does it know that no more data will be coming and that it should exit? My guess is that it could look for an EOF and terminate based on that, but what if createText.py never sends one? Would that be considered an error on createText.py's part?
Say parent.py starts a child subprocess (child.py) and calls wait() to wait for the child to complete. If parent is capturing child's stdout and stderr as pipes, is it still safe to read from them after child has terminated? Or are the pipes (and data in them) destroyed when one end terminates?
The general problem that I want to solve is to write a python script that calls rsync several times with the Popen class. I want my program to wait until rsync has completed, then I want to check the return status to see if it exited correctly. If it didn't, I want to read the child's stderr to see what the error was. Here is what I have so far
# makes the rsync call. Will block until the child
# process is finished. Returns the exit code for rsync
def performRsync(src, dest):
print "Pushing " + src + " to " + dest
child = Popen(['rsync', '-av', src, dest], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
child.wait()
## check for success or failure
## 0 is a successful exit code here
if not child.returncode:
return True
else:#ballz
stout, sterr = child.communicate()
print "ERR pushing " + src + ". " + sterr
return False
Update: I also came across this problem. Consider these two simple files:
# createText.py
for x in range(1000):
print "creating line " + str(x)
time.sleep(1)
# processText.py
while True:
line = sys.stdin.readline()
if not line:
break;
print "I modified " + line
Why does processText.py in this case not start printing as it gets data from stdin? Does a pipe collect some amount of buffered data before it passes it along?
This assumes a UNIXish/POSIXish environment.
EOF in a pipeline is signaled by no more data to read, that is, read() returns a length of 0. This normally occurs when the left-hand process exits and closes its stdout. Since you can't read from a pipe whose other end is closed the read in processText indicates EOF.
If createText were to not exit thus closing its output it would be a non-ending program which in a pipeline is a Bad Thing. Even if not in pipeline, a program that never ends usually incorrect (odd cases like yes(1) excepted).
You can read from a pipe as long as you don't get EOF or an IOError(errno.EPIPE) indication which would also indicate there is nothing left to read.
I've not tested your code, does it do something unexpected?
Related
I'm running a python script that triggers a script inside a docker container, which prints out its start time, starts another process with arguments, and then prints out its end time, after the process it triggered is done.
I've been using this code to trigger the script in the container and get its output:
pr = subprocess.Popen('docker exec ' + docker_container + ' /scripts/sct.sh', cwd=os.path.dirname('/'),
shell=True, stdout=subprocess.PIPE, stderr = subprocess.PIPE )
if debug:
while True:
output = pr.stdout.readline().decode('utf-8')
if output == '' and pr.poll() is not None:
break
if output:
print(output.strip())
else:
(out, error) = pr.communicate()
print("### re-index: error message: " + error.decode('utf-8'))
print("### re-index: message: " + out.decode('utf-8'))
When I run in debug mode I want to be able to see all of the output lines from the process triggered by the script inside the container, as it runs, so I could see if it hangs, stop it running, adjust its settings, and rerun to see if I solved thing.
Problem is, when I try it, I only get the start time line from the script inside the container, on the fly. The rest of the lines (the output from the process the script inside the container triggered) only come through after the process the script inside the container triggered is done running.
The process takes hours, whether it works well or badly, so just waiting for it to finish and then analyzing the results is not an option.
Is there any way I could adjust my code to get the output of BOTH my child process (= the script inside the container) and its child process (=the process it triggers) as they run, and not after the child's child process ends?
Other than cutting out the middleman and triggering the child's child's process directly, that is, which I really would like to avoid.
I'm having a problem with subprocess poll not returning the return code when the process has finished.
I found out how to set a timeout on subprocess.Popen and used that as the basis for my code. However, I have a call that uses Java that doesn't correctly report the return code so each call "times out" even though it is actually finished. I know the process has finished because when removing the poll timeout check, the call runs without issue returning a good exit code and within the time limit.
Here is the code I am testing with.
import subprocess
import time
def execute(command):
print('start command: {}'.format(command))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print('wait')
wait = 10
while process.poll() is None and wait > 0:
time.sleep(1)
wait -= 1
print('done')
if wait == 0:
print('terminate')
process.terminate()
print('communicate')
stdout, stderr = process.communicate()
print('rc')
exit_code = process.returncode
if exit_code != 0:
print('got bad rc')
if __name__ == '__main__':
execute(['ping','-n','15','127.0.0.1']) # correctly times out
execute(['ping','-n','5','127.0.0.1']) # correctly runs within the time limit
# incorrectly times out
execute(['C:\\dev\\jdk8\\bin\\java.exe', '-jar', 'JMXQuery-0.1.8.jar', '-url', 'service:jmx:rmi:///jndi/rmi://localhost:18080/jmxrmi', '-json', '-q', 'java.lang:type=Runtime;java.lang:type=OperatingSystem'])
You can see that two examples are designed to time out and two are not to time out and they all work correctly. However, the final one (using jmxquery to get tomcat metrics) doesn't return the exit code and therefore "times out" and has to be terminated, which then causes it to return an error code of 1.
Is there something I am missing in the way subprocess poll is interacting with this Java process that is causing it to not return an exit code? Is there a way to get a timeout option to work with this?
This has the same cause as a number of existing questions, but the desire to impose a timeout requires a different answer.
The OS deliberately gives only a small amount of buffer space to each pipe. When a process writes to one that is full (because the reader has not yet consumed the previous output), it blocks. (The reason is that a producer that is faster than its consumer would otherwise be able to quickly use a great deal of memory for no gain.) Therefore, if you want to do more than one of the following with a subprocess, you have to interleave them rather than doing each in turn:
Read from standard output
Read from standard error (unless it’s merged via subprocess.STDOUT)
Wait for the process to exit, or for a timeout to elapse
Of course, the subprocess might close its streams before it exits, write useful output after you notice the timeout and before you kill it, and/or start additional processes that keep the pipe open indefinitely, so you might want to have multiple timeouts. Probably what’s most informative is the EOF on the pipe, so repeatedly use something like select to wait for (however much is left of) the timeout, issue single reads on the streams that are ready, and wait (with another timeout if you’re concerned about hangs after an early stream closure) on EOF. If the timeout occurs instead, (try to) kill the subprocess, and consider issuing non-blocking reads (or another timeout loop) to get any last available output before closing the pipes.
Using the other answer by #DavisHerring as the basis for more research, I came across a concept that worked for my original case. Here is the code that came out of that.
import subprocess
import threading
import time
def execute(command):
print('start command: {}'.format(command))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
timer = threading.Timer(10, terminate_process, [process])
timer.start()
print('communicate')
stdout, stderr = process.communicate()
print('rc')
exit_code = process.returncode
timer.cancel()
if exit_code != 0:
print('got bad rc')
def terminate_process(p):
try:
p.terminate()
except OSError:
pass # ignore error
It uses the threading.Timer to make sure that the process doesn't go over the time limit and terminates the process if it does. It otherwise waits for a response back and cancels the timer once it finishes.
I am running this code in a loop - in Python 3.6:
# previous code: copy certain files to the working folder
shellCmd = "myCmd " + file1 + " " + file2
# myCmd works on file1 and file2
result = subprocess.check_output(myCmd, shell=True)
# delete the files
Every now and then I get a fail caused by access denied to some of the files. I am guessing that the subprocess is running in the background and the loop continues, spawning other subprocesses. Sometimes this will cause one subprocess trying to copy (or delete) files that myCmd is still busy with in another subprocess.
How do I stop and wait for subprocess.check_output() to complete?
I saw that subprocess.Popen has a wait() function, but I need the result string from the myCmd process, so want to use subprocess.check_output().
But any solution that will (a) get me the string output of myCmd and (b) ensure the subprocesses happen in series, is OK.
Thanks!
The situation you describe is far from satisfactory, because it appears that multiple subprocesses will, if I understand your description correctly, sometimes create race conditions. The logical thing to do would be to have your program read the output from the subprocess as it appears.
If you want better control over subprocesses your are safer using subprocess.Popen objects, which have a more usable interface. By reading the output from one command until you get to end-of-file you know that you won't be creating additional processes to interfere. Send the command's standard output to a pipe with stdout=subprocess.PIPE and then you can read the process's standard output as the Popen object's stdout attribute as shown below.
>>> process = subprocess.Popen("getmac", stdout=subprocess.PIPE)
>>> for line in process.stdout:
... print(line)
...
b'\r\n'
b'Physical Address Transport Name \r\n'
b'=================== ==========================================================\r\n'
b'94-C6-91-1B-56-A4 \\Device\\Tcpip_{023B9717-B878-43D4-A0BE-28A4295785FA} \r\n'
b'68-EC-C5-52-14-AD Media disconnected \r\n'
b'68-EC-C5-52-14-B1 Media disconnected \r\n'
b'0A-00-27-00-00-0E \\Device\\Tcpip_{89DD54F9-0C99-4F5B-8376-45598FB4C0FD} \r\n'
>>>
No, check_output only returns after the subprocess has finished running. Your problem is caused by something else.
This code create pty (pseudo-terminals) in Python. I have commented the parts that I do not understand
import os,select
pid, master_fd =os.forkpty() #I guess this function return the next available pid and fd
args=['/bin/bash']
if pid == 0:#I have no I idea what this if statement does, however I have noticed that it get executed twice
os.execlp('/bin/bash',*args)
while 1:
r,w,e=select.select([master_fd,0], [], [])
for i in r:
if i==master_fd:
data=os.read(master_fd, 1024)
"""Why I cannot do something like
f=open('/dev/pts/'+master_fd,'r')
data=f.read()"""
os.write(1, data) # What does 1 mean???
elif i==0:
data = os.read(0, 1024)
while data!='':
n = os.write(master_fd, data)
data = data[n:]
In Unix-like operating systems, the way to start a new process is a fork. That is accomplished with fork() or its several cousins. What this does is it duplicates the calling process, in effect having two exactly the same programs.
The only difference is the return value from fork(). The parent process gets the PID of the child, and the child gets 0. What usually happens is that you have an if statement like the one that you're asking about.
If the returned PID is 0 then you're "in the child". In this case the child is supposed to be a shell, so bash is executed.
Else, you're "in the parent". In this case the parent makes sure that the child's open file descriptors (stdin, stdout, stderr and any open files) do what they're supposed to.
If you ever take an OS class or just try to write your own shell you'll be following this pattern a lot.
As for your other question, what does the 1 mean in os.write(1, data)?
The file descriptors are integer offsets into an array inside the kernel:
0 is stdin
1 is stdout
2 is stderr
i.e. that line just writes to stdout.
When you want to set up pipes or redirections then you just change the meaning of those three file descriptors (look up dup2()).
I have a large file that needs to be processed before feeding to another command. I could save the processed data as a temporary file but would like to avoid it. I wrote a generator that processes each line at a time then following script to feed to the external command as input. however I got "I/O operation on closed file" exception at the second round of the loop:
cmd = ['intersectBed', '-a', 'stdin', '-b', bedfile]
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for entry in my_entry_generator: # <- this is my generator
output = p.communicate(input='\t'.join(entry) + '\n')[0]
print output
I read another similar question that uses p.stdin.write. but still had the same problem.
What I did wrong?
[edit]
I replaced last two statements with following (thanks SpliFF):
output = p.communicate(input='\t'.join(entry) + '\n')
if output[1]: print "error:", output[1]
else: print output[0]
to see if there was any error by the external program. But no.
Still have the same exception at p.communicate line.
The communicate method of subprocess.Popen objects can only be called once. What it does is it sends the input you give it to the process while reading all the stdout and stderr output. And by "all", I mean it waits for the process to exit so that it knows it has all output. Once communicate returns, the process no longer exists.
If you want to use communicate, you have to either restart the process in the loop, or give it a single string that is all the input from the generator. If you want to do streaming communication, sending data bit by bit, then you have to not use communicate. Instead, you would need to write to p.stdin while reading from p.stdout and p.stderr. Doing this is tricky, because you can't tell which output is caused by which input, and because you can easily run into deadlocks. There are third-party libraries that can help you with this, like Twisted.
If you want to do this interactively, sending some data and then waiting for and processing the result before sending more data, things get even harder. You should probably use a third-party library like pexpect for that.
Of course, if you can get away with just starting the process inside the loop, that would be a lot easier:
cmd = ['intersectBed', '-a', 'stdin', '-b', bedfile]
for entry in my_entry_generator:
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output = p.communicate(input='\t'.join(entry) + '\n')[0]
print output
Probably your intersectBed application is exiting with an error but since you aren't printing any stderr data you can't see it. Try:
result = p.communicate(input='\t'.join(entry) + '\n')
if result[1]:
print "error:", result[1]
else:
print result[0]