How can I spawn or Popen a subprocess in python and process its output in realtime?
The subprocess prints output randomly depending on other system events.
This "example" hangs:
$./print.sh | ./echo.py hangs.
print.sh
#!/bin/bash
while [ 1 ]; do
echo 'A'
sleep 1
done
echo.py
#!/usr/bin/python
import sys
for line in sys.stdin:
print line
It doesn't hang. echo/the shell decides that, because it's writing to a pipe, it will perform I/O in a block-buffered rather than line-buffered mode. If you wait long enough, or remove the sleep 1 from the shell script, you'll see that the output from A does come through.
There are two possible solutions:
Modify the subprocess's program so that it flushes its buffers when it's written enough output for the Python program to process.
Use pseudo-terminals (PTYs) instead of pipes. pexpect does that, hiding most of the complexity from you. It's not a drop-in replacement for subprocess, though.
check whether this coding snippet works.
cat ech.sh
#!/bin/bash
while [ 1 ]; do
echo -n 'A'
sleep 1
done
cat in_read.py
#!/usr/bin/python
import sys
import os
while True:
print os.read(0,1)
Related
nohup python test.py &
In the generated nohup.out file, 'print' output are not written into the it, and some log info are not written the nohup.out either. Is this normal? I just want to leave the program run background and periodically I can check the progress of the program by open the nohup.out. For example, my code has this line:
with open(file_name, 'r', encoding='utf8') :
for index, line in enumerate(f):
logger.info(index)
#print(index)
By opening nohup.out, I want to see the current value of 'index', so that I know how much content it has processed. However, in this nohup.out, I can not see any 'index' info in it. Why is that?
I used to run programs in a similar way and can see the index in the nohup.out sometimes.
What might be wrong for my running?
Based on running a quick test, I saw printing to stderr happens first, then stdout at the end of my script. I suspect you need to explicitly flush stdout every once in a while or you can print to stderr.
To print to stderr, define this eprint function and call that instead of printing.
import sys
def eprint(*args,**kwrgs):
print(*args,file=sys.stderr,**kwargs)
To flush stdout, do
import sys
sys.stdout.flush()#call this periodically
My test:
#test.py
import sys
for x in range(0,10,2):
sys.stdout.write(str(x)+'\n')
sys.stderr.write(str(x+1)+'\n')
nohup python test.py
$ cat nohup.out
1
3
5
7
9
0
2
4
6
8
$
Description:
I was trying to make a shell that can be interactive on a chatting software, so I need a cmd.exe as a subprocess and pass strings into the process.
I have this:
from subprocess import Popen
from subprocess import PIPE as p
proc = Popen("cmd",stdout=p,stdin=p,shell=True)
so usually what we do if we need to pass input to the process is by using proc.stdin.write()
but it seems that the string will only pass in and work after the python script is complete
for example, I have
#same thing above
proc.stdin.write("ping 127.0.0.1".encode())
time.sleep(10)
the script will wait for 10 sec then pass and run the ping command.
which means it's impossible to get the result stdout.read() because there is nothing.
I have tried to use subprocess.Popen.communicate() but it closes the pipe after one input.
Is there any way to solve the "only run the command after script finish" thing, or make communicate() not close the pipe?
Writes to pipes are buffered, you need to flush the buffer.
proc.stdin.write("ping 127.0.0.1".encode())
proc.stdin.flush()
I need to write a script that will receive several parameters, from which the most important one
Is a string that contains a command (in linux).
I need to be able to run it, keep the output in STDOUT (the usual), but also time it, and later output some .csv file.
Say it looks something like this:
timing_script.py param "echo hello world; cat /tmp/foo_bar"
The command will output stuff to STDOUT every couple of milliseconds, which I need it to stay there. I'm saying this because my previous attempt at this script was in bash and I had to cut from the time command to actually time that, which also meant having to disregard the output of the command.
I'll also have to append something like param,0.345 to a csv file.
How do I execute a command from a string and also time it?
You can use subprocess to run linux command from string and time to calculate execution time:
import time
from subprocess import Popen, PIPE
start = time.time()
p1 = Popen(["my_linux_cmd"], stdout=PIPE)
print(p1.communicate()) # sdout
end = time.time()
exec_time = end - start
print(exec_time) # exeution time
Check subprocess.Popen fro more details about the available options
Warning: to print the stdout you can also use Popen.stdout.read but use communicate() rather to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
A simpler way which stays in the shell uses the formatting option -f of the time command. You can use it like that :
$ param="foo"
$ command="echo bar ; cat /tmp/foobar"
$ /usr/bin/time -f "$param,%e" bash -c "$command"
bar
#Beginning of foobar file
#End of foobar file
foo,0.00
Please have a look at man time for further examples about formatting the output of time
Of course, you can also directly run the following command (i.e. without using variables) :
/usr/bin/time -f "myparam,%e" bash -c "echo bar ; cat /tmp/foobar"
Have fun
When piping printed output from a python script to a command like grep, the output from the script seems to only be piped to the follow-up command after completion of the entire script.
For example, in a script test_grep.py like the following:
#!/usr/bin/env python
from time import sleep
print "message1"
sleep(5)
print "message2"
sleep(5)
print "message3"
when called with ./test_grep.py | grep message, nothing will appear for 10 seconds, at which time all three lines will appear.
Compare this to a script test_grep.sh:
#!/usr/bin/env bash
echo "message1"
sleep 5
echo "message2"
sleep 5
echo "message3"
./test_grep.sh | grep message will immediately output message1, followed at 5 second intervals by message2 and message3.
I expect this is because only once the python interpreter finishes executing is the output available for the next command. Is there any way to alter this behavior?
You can do it:
By flushing every print in python
By setting stdout to be unbuffered
By setting stdout to be line-buffered
You can even call python -u to disable buffering.
I would go for the line-buffering option as it seems most natural.
open(file, mode='r', buffering=-1 ....)
buffering is an optional integer used to set the buffering policy.
Pass 0 to switch buffering off (only allowed in binary mode), 1 to
select line buffering (only usable in text mode), and an integer > 1
to indicate the size of a fixed-size chunk buffer.
When you don't specify buffering (the typical "open") it will use line-buffering if it detects the output is going directly do a TTY, i.e. to your screen console. If you pipe output or redirect it to a file it will switch back to a large (4K / 8K) buffer.
How do you "set stdout to be line-buffered"?
You can reopen stdout via sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1).
I've got a script parent.py trying to to read stdout from a subprocess sub.py in Python.
The parent parent.py:
#!/usr/bin/python
import subprocess
p = subprocess.Popen("sub.py", stdout=subprocess.PIPE)
print p.stdout.read(1)
And the subprocess, sub.py:
#!/usr/bin/python
print raw_input( "hello world!" )
I would expect running parent.py to print the 'h' from "hello world!". Actually, it hangs. I can only get my expected behaviour by adding -u to sub.py's she-bang line.
This confuses me because the -u switch makes no difference when sub.py is run directly from a shell; the shell is somehow privy to the un-flushed output stream, unlike parent.py.
My goal is to run a C program as the subprocess, so I won't be able to control whether or not it flushes stdout. How is it that a shell has better access to a process's stdout than Python running the same thing from subprocess.Popen? Am I going to be able to read such a stdout stream from a C program that doesn't flush its buffers?
EDIT:
Here is an updated example based on korylprince's comment...
## capitalize.sh ##
#!/bin/sh
while [ 1 ]; do
read s
echo $s | tr '[:lower:]' '[:upper:]'
done
########################################
## parent.py ##
#!/usr/bin/python
from subprocess import Popen, PIPE
# cmd = [ 'capitalize.sh' ] # This would work
cmd = [ 'script', '-q', '-f', '-c', 'capitalize.sh', '/dev/null']
p = Popen(cmd, stdin=PIPE)
p.stdin.write("some string\n")
p.wait()
When running through script, I get steady printing of newlines (and if this were a Python, subprocess, it'd raise an EOFerror).
An alternative is
p = subprocess.Popen(["python", "-u", "sub.py"], stdout=subprocess.PIPE)
or the suggestions here.
My experience is that yes, you will be able to read from most C programs without any extra effort.
The Python interpreter takes extra steps to buffer its output which is why it needs the -u switch to disable output buffering. Your typical C program won't do this.
I haven't run into any program (C or otherwise) other than the Python interpreter that I expected to work and didn't within a subshell.
The reason the shell can read output immediately, regardless of "-u" is because the program you're launching from the shell has its output connected to a TTY. When the stdout is connected to a TTY, it is unbuffered (because it is up to the TTY to buffer). When you launch the python subprocess from within python, you're connecting stdout to a pipe, which means you're at the mercy of the subprocess to flush its output when it feels like it.
If you're looking to do complicated interactions with a subprocess, look into this tutorial.