I'm having a problem getting two subprocesses to run together.
The first subprocesss is a transcode of a video stream:
subprocess.Popen("ffmpeg -i input output", shell=True)
I need this to run in the background of my program, transcoding video from my IP camera into a mjpeg stream.
The second subprocess is the Openalpr Daemon that looks at the mjpeg stream and returns car license plates it sees in the stream.
subprocess.Popen("alprd -f", shell=True)
Here is a sample piece of python test code that tries to run both subprocesses:
import subprocess
subprocess.Popen("ffmpeg -i input output", shell=True)
subprocess.Popen("alprd -f", shell=True)
When i do this, the ffmpeg transcoding works fine, i can view the transcoded mjpeg stream and i can see ffmpegs verbose output in the console. However, the alprd daemon seems to not return any number plates as expected. In fact, i can't see any output from alprd in the console window.
If i run the above code with just one subprocess it works e.g.
import subprocess
subprocess.Popen("ffmpeg -i input output", shell=True)
works fine, as does:
import subprocess
subprocess.Popen("alprd -f", shell=True)
If i run either of the two working code snippets above - and at the same time run the other command in a separate linux terminal, it all works.
I'm clearly not understanding something with subprocesses, They are clearly conflicting with each other, but Can anyone explain what is happening and how to resolve the problem?
Thanks!
It's likely that ffmpeg and alprd are both trying to interact with the same stdin/stdout file handles. To solve this, create separate pipes for one or both subprocesses to use as stdin/stdout. Then they can interact with them without interfering with each other.
import subprocess
with open('ffmpeg-output.txt', 'w') as ffmpeg_output:
ffmpeg = subprocess.Popen(
['ffmpeg', '-i', 'input', 'output'],
stdin=subprocess.PIPE,
stdout=ffmpeg_output,
stderr=subprocess.STDOUT)
# We won't be sending any input into ffmpeg's stdin, so close it.
ffmpeg.stdin.close()
# alprd inherits stdin, stdout, and stderr from the current process.
alprd = subprocess.Popen(['alprd', '-f'])
# Wait for the subprocesses to finish.
ffmpeg.wait()
alprd.wait()
Related
I have created a simple method that executes a command like you do in the terminal
from subprocess import PIPE, run
class Command_Line():
#staticmethod
def execute(command):
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=True)
print(result)
return result.stdout
My problem with the code above is it does not wait until the task/process is done. Lets say i use ffmpeg to change the frame rate of a video via the following code
import Command_Line as cmd
cmd.execute('ffmpeg -i "000000004.avi" -c copy -y -r 30 "000000004.avi"')
The problem is the output video because it does not complete the process. I've search how to wait like Is there a way to check if a subprocess is still running?
but could not incorporate it with my code. Can you share your experience with this.
Thanks
According to the python documentation subprocess.run waits for the process to end.
The problem is that ffmpeg overwrites the input file if the input and output files are the same and therefore the output video becomes unusable.
subprocess.run() is synchronous which means that the system will wait till it finishes before moving on to the next command. subprocess.Popen() does the same thing but it is asynchronous (the system will not wait for it to finish). You can try reloading your file using importlib.reload command. It may find your generated file then.
Description:
I was trying to make a shell that can be interactive on a chatting software, so I need a cmd.exe as a subprocess and pass strings into the process.
I have this:
from subprocess import Popen
from subprocess import PIPE as p
proc = Popen("cmd",stdout=p,stdin=p,shell=True)
so usually what we do if we need to pass input to the process is by using proc.stdin.write()
but it seems that the string will only pass in and work after the python script is complete
for example, I have
#same thing above
proc.stdin.write("ping 127.0.0.1".encode())
time.sleep(10)
the script will wait for 10 sec then pass and run the ping command.
which means it's impossible to get the result stdout.read() because there is nothing.
I have tried to use subprocess.Popen.communicate() but it closes the pipe after one input.
Is there any way to solve the "only run the command after script finish" thing, or make communicate() not close the pipe?
Writes to pipes are buffered, you need to flush the buffer.
proc.stdin.write("ping 127.0.0.1".encode())
proc.stdin.flush()
I want to read on-going output of started process from code (running in background) with subprocess.Popen() and subprocess.communicate()
Starting the process:
import subprocess
process_params = ['/usr/bin/tcpdump', '-n', 'dst port 80']
proc = subprocess.Popen(
process_params,
stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
(stdout, stderr) = proc.communicate()
The tcpdump process is running in background but proc.communicate() waits till end of file and only when process is killed it produces some output.
>>> (stdout, stderr) = proc.communicate()
I would like to achieve something like receiving data from process' stdout at the moment when output is produced by the process.
I think I need some thread that looks if some output is generated from process and then for example append it to log file.
I don't know how to get down to it, so any ideas and suggestions will be much appreciated.
tcpdump is probably buffering its output. It will only write output when it has a buffer full of data (typically 8KB) to write. This is common behavior for programs which are not writing to a TTY.
Tcpdump has a command-line option to line-buffer its output. This causes it to write its output every time it has a full line of text. Try this:
process_params = ['/usr/bin/tcpdump', '-l', '-n', 'dst port 80']
^^--Line buffer output
Alternately, see if you have programs named stdbuf or unbuffer installed on your system. They can be used to adjust the buffering behavior of another process. Or see these questions:
How to make output of any shell command unbuffered?
How to unbuffer stdout of legacy running binary without stdbuf and similar tools
Im trying to handle tcpdump output in python.
What I need is to run tcpdump (which captures the packets and gives me information) and read the output and process it.
The problem is that tcpdump keeps running forever and I need to read the packet info as soon as it outputs and continue doing it.
I tried looking into subprocess of python and tried calling tcpdump using popen and piping the stdout but it doesnt seem to work.
Any directions on how to proceed with this.
import subprocess
def redirect():
tcpdump = subprocess.Popen("sudo tcpdump...", stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
while True:
s = tcpdump.stdout.readline()
# do domething with s
redirect()
You can make tcpdump line-buffered with "-l". Then you can use subprocess to capture the output as it comes out.
import subprocess as sub
p = sub.Popen(('sudo', 'tcpdump', '-l'), stdout=sub.PIPE)
for row in iter(p.stdout.readline, b''):
print row.rstrip() # process here
By default, pipes are block buffered and interactive output is line buffered. It sounds like you need a line buffered pipe - coming from tcpdump in a subprocess.
In the old days, we'd recommend Dan Bernstein's "pty" program for this kind of thing. Today, it appears that pty hasn't been updated in a long time, but there's a new program called "emtpy" which is more or less the same idea:
http://empty.sourceforge.net/
You might try running tcpdump under empty in your subprocess to make tcpdump line buffered even though it's writing to a pipe.
ok, so my google-fu really kind of sucks and I was unable to find an answer, hopefully you folks can help me ^_^
ok, so what I thought would be a simple script is seemingly not communicating with its subprocess correctly, I'm running this line by line. I'm also using the mpg123 player, this is a Linux system (well, Raspberry Pi)
from subprocess import Popen, PIPE, STDOUT
p = Popen(["mpg123", "-C", "test.mp3"], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
#wait a few seconds to enter this, "q" without a newline is how the controls for the player work to quit out if it were ran like "mpg123 -C test.mp3" on the command line
p.communicate(input='q')[0]
I can run stdout.read() on it just fine, but using communicate for input just makes it hang and p.stdin.write('q') does seemingly nothing at all. This is python related though I have a feeling I'm also not looking in the right place in the mpg123 documentation. Please be kind as I'm exceptionally new to this ^_^
Check what arguments your mpg123 version understands. The following works on my machine:
#!/usr/bin/env python3
import time
from subprocess import Popen, PIPE, DEVNULL, STDOUT
p = Popen(["mpg123", "-K", "test.mp3"], stdin=PIPE, stdout=DEVNULL, stderr=STDOUT)
# wait a little
time.sleep(3)
# send command "Skip song", wait for the player to exit
p.communicate(b'n')[0]
It starts playing the file, waits ~3 seconds, and exits.
This is an awful solution, but it works in a pinch. I'm using this as a patch because for some reason, I cannot get Python libraries to play properly on my Raspberry Pi.
If you start mpg123 in remote mode (mpg123 -R), you can send commands to it far more easily:
proc = sp.Popen(["mpg123", "-R"], stdin=sp.PIPE)
Then, you can send commands to its stdin attribute.
Note:
The commands are different. To pause, it's "pause", not " " for example. Run mpg123 -R in a console, then send it the help command to see a list of commands.
The commands need to be newline terminated.
From the docs:
-R, --remote
Activate generic control interface. mpg123 will then read and execute commands from stdin. Basic usage is ''load '' to play some file and the obvious ''pause'', ''command. ''jump '' will jump/seek to a given point (MPEG frame number). Issue ''help'' to get a full list of commands and syntax.