I want to read on-going output of started process from code (running in background) with subprocess.Popen() and subprocess.communicate()
Starting the process:
import subprocess
process_params = ['/usr/bin/tcpdump', '-n', 'dst port 80']
proc = subprocess.Popen(
process_params,
stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
(stdout, stderr) = proc.communicate()
The tcpdump process is running in background but proc.communicate() waits till end of file and only when process is killed it produces some output.
>>> (stdout, stderr) = proc.communicate()
I would like to achieve something like receiving data from process' stdout at the moment when output is produced by the process.
I think I need some thread that looks if some output is generated from process and then for example append it to log file.
I don't know how to get down to it, so any ideas and suggestions will be much appreciated.
tcpdump is probably buffering its output. It will only write output when it has a buffer full of data (typically 8KB) to write. This is common behavior for programs which are not writing to a TTY.
Tcpdump has a command-line option to line-buffer its output. This causes it to write its output every time it has a full line of text. Try this:
process_params = ['/usr/bin/tcpdump', '-l', '-n', 'dst port 80']
^^--Line buffer output
Alternately, see if you have programs named stdbuf or unbuffer installed on your system. They can be used to adjust the buffering behavior of another process. Or see these questions:
How to make output of any shell command unbuffered?
How to unbuffer stdout of legacy running binary without stdbuf and similar tools
Related
I'm having a problem getting two subprocesses to run together.
The first subprocesss is a transcode of a video stream:
subprocess.Popen("ffmpeg -i input output", shell=True)
I need this to run in the background of my program, transcoding video from my IP camera into a mjpeg stream.
The second subprocess is the Openalpr Daemon that looks at the mjpeg stream and returns car license plates it sees in the stream.
subprocess.Popen("alprd -f", shell=True)
Here is a sample piece of python test code that tries to run both subprocesses:
import subprocess
subprocess.Popen("ffmpeg -i input output", shell=True)
subprocess.Popen("alprd -f", shell=True)
When i do this, the ffmpeg transcoding works fine, i can view the transcoded mjpeg stream and i can see ffmpegs verbose output in the console. However, the alprd daemon seems to not return any number plates as expected. In fact, i can't see any output from alprd in the console window.
If i run the above code with just one subprocess it works e.g.
import subprocess
subprocess.Popen("ffmpeg -i input output", shell=True)
works fine, as does:
import subprocess
subprocess.Popen("alprd -f", shell=True)
If i run either of the two working code snippets above - and at the same time run the other command in a separate linux terminal, it all works.
I'm clearly not understanding something with subprocesses, They are clearly conflicting with each other, but Can anyone explain what is happening and how to resolve the problem?
Thanks!
It's likely that ffmpeg and alprd are both trying to interact with the same stdin/stdout file handles. To solve this, create separate pipes for one or both subprocesses to use as stdin/stdout. Then they can interact with them without interfering with each other.
import subprocess
with open('ffmpeg-output.txt', 'w') as ffmpeg_output:
ffmpeg = subprocess.Popen(
['ffmpeg', '-i', 'input', 'output'],
stdin=subprocess.PIPE,
stdout=ffmpeg_output,
stderr=subprocess.STDOUT)
# We won't be sending any input into ffmpeg's stdin, so close it.
ffmpeg.stdin.close()
# alprd inherits stdin, stdout, and stderr from the current process.
alprd = subprocess.Popen(['alprd', '-f'])
# Wait for the subprocesses to finish.
ffmpeg.wait()
alprd.wait()
My python script (python 3.4.3) calls a bash script via subprocess.
OutPST = subprocess.check_output(cmd,shell=True)
It works, but the problem is, that I only get half of the data. The subprocess I call, calls a different subprocess and I have the guess, that if the "sub subprocess" sends the EOF, my programm thinks, that that´s it and ends the check_output.
Has someone an idea how to get all the data?
You should use subprocess.run() unless you really need that fine grained of control over talking to the processing via its stdin (or doing something else while the process is running instead of blocking for it to finish). It makes capturing output super easy:
from subprocess import run, PIPE
result = run(cmd, stdout=PIPE, stderr=PIPE)
print(result.stdout)
print(result.stderr)
If you want to merge stdout and stderr (like how you'd see it in your terminal if you didn't do any redirection), you can use the special destination STDOUT for stderr:
from subprocess import STDOUT
result = run(cmd, stdout=PIPE, stderr=STDOUT)
print(result.stdout)
I have a Python script to capture network traffic with tcpdump in a subprocess:
p = subprocess.Popen(['tcpdump', '-I', '-i', 'en1',
'-w', 'cap.pcap'], stdout=subprocess.PIPE)
time.sleep(10)
p.kill()
When this script completes its work, I'm trying to open output .pcap file in Wireshark and getting this error:
"The capture file appears to have been cut short in the middle of a packet."
What solution could be applied for "proper" closing of tcpdump's subprocess?
Instead of p.kill(), you can use p.send_signal(subprocess.signal.SIGTERM) to send a terminate signal rather than a kill (p.terminate() does the same).
The Popen docs describe the send_signal() command. The documentation on signals is a bit weak, but a dir(subprocess.signal) will list all the signals you may send to the process, but terminate should allow it some time to clean up.
Found working solution:
I've changed p.kill() to p.terminate().
After this change the subprocess is "properly" finished (output of tcpdump subprocess with statistics available in console) and output .pcap file not damaged.
I had the same problem about closing subprocesses. This thread really helped, so thanks, especially to https://stackoverflow.com/users/3583715/rkh. My solution:
Before:
output = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
message = output.stdout.read()
output.stdout.close()
After reading the Popen docs:
output = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
message = output.stdout.read()
output.TerminateProcess()
For some reason, calling output.kill(), and/or output.terminate() or sending output.send_signal(subprocess.signal.SIGTERM) didn't work, but output.TerminateProcess() did.
We are having some problems with the dreaded "too many open files" on our Ubuntu Linux machine rrunning a python Twisted application. In many places in our program, we are using subprocess Popen, something like this:
Popen('ifconfig ' + iface, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = process.stdout.read()
while in other places we use subprocess communicate:
process = subprocess.Popen(['/usr/bin/env', 'python', self._get_script_path(script_name)],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
close_fds=True)
out, err = process.communicate(data)
What exactly do I need to do in both cases in order to close any open file descriptors? Python documentation is not clear on this. From what I gather (which could be wrong) both communicate() and wait() will indeed clean up any open fds on their own. But what about Popen? Do I need to close stdin, stdout, and stderr explicitly after calling Popen if I don't call communicate or wait?
According to this source for the subprocess module (link) if you call communicate you should not need to close the stdout and stderr pipes.
Otherwise I would try:
process.stdout.close()
process.stderr.close()
after you are done using the process object.
For instance, when you call .read() directly:
output = process.stdout.read()
process.stdout.close()
Look in the above module source for how communicate() is defined and you'll see that it closes each pipe after it reads from it, so that is what you should also do.
If you're using Twisted, don't use subprocess. If you were using spawnProcess instead, you wouldn't need to deal with annoying resource-management problems like this.
Im trying to handle tcpdump output in python.
What I need is to run tcpdump (which captures the packets and gives me information) and read the output and process it.
The problem is that tcpdump keeps running forever and I need to read the packet info as soon as it outputs and continue doing it.
I tried looking into subprocess of python and tried calling tcpdump using popen and piping the stdout but it doesnt seem to work.
Any directions on how to proceed with this.
import subprocess
def redirect():
tcpdump = subprocess.Popen("sudo tcpdump...", stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
while True:
s = tcpdump.stdout.readline()
# do domething with s
redirect()
You can make tcpdump line-buffered with "-l". Then you can use subprocess to capture the output as it comes out.
import subprocess as sub
p = sub.Popen(('sudo', 'tcpdump', '-l'), stdout=sub.PIPE)
for row in iter(p.stdout.readline, b''):
print row.rstrip() # process here
By default, pipes are block buffered and interactive output is line buffered. It sounds like you need a line buffered pipe - coming from tcpdump in a subprocess.
In the old days, we'd recommend Dan Bernstein's "pty" program for this kind of thing. Today, it appears that pty hasn't been updated in a long time, but there's a new program called "emtpy" which is more or less the same idea:
http://empty.sourceforge.net/
You might try running tcpdump under empty in your subprocess to make tcpdump line buffered even though it's writing to a pipe.