I have a script that saves 5 seconds length of videos locally and it omits the file name.
Here's the bash command
ffmpeg -i http://0.0.0.0:8080/stream/video.mjpeg -vcodec copy -map 0 -f segment -segment_time 2 -loglevel 40 -segment_format mp4 capture-%05d.mp4 2>&1 | grep --line-buffered -Eo "segment:.+ended" | gawk -F "'" '{print $2; system("")}' | xargs -n1
If I run this command in the terminal, it will return the expected file name, as such
capture-00001.mp4
Notice that the xargs command at the end easily lets me to pass the file name to a python script as a new argument. But now I want to execute this command within Python itself, specifically getting the file name with subprocess.
Here's what I've done so far. When running the script, as expected the terminal will print the file name, but it never pass it as a string to fName. I've tried subprocess.check_output but it never passes anything as the command continuously capture videos and save it locally.
FFMPEG_SCRIPT = r"""ffmpeg -i http://0.0.0.0:8080/stream/video.mjpeg -vcodec copy -map 0 -f segment -segment_time 2 -loglevel 40 -segment_format mp4 capture-%05d.mp4 2>&1 | grep --line-buffered -Eo "segment:.+ended" | gawk -F "'" '{print $2; system("")}' | xargs -n1 """
try:
fName = subprocess.check_call(FFMPEG_SCRIPT, stderr=subprocess.STDOUT, shell=True).decode('utf-8')
print(">>> {}".format(fName))
except subprocess.CalledProcessError as e:
print(e.output)
import subprocess
from subprocess import PIPE
fName = subprocess.Popen("test.bat", stdin=PIPE, stdout=PIPE)
(stdout, stderr) = fName.communicate()
print(">>> {}".format(fName))
print(stdout)
test.bat is a simple echo yes since I can't test with your script, and the resulting output of print(stdout) is b'yes\r\n'. If the file name is the only thing the scripts prints, it shouldn't be too hard to extract it.
That is a common problem when you pipe commands. The IO subsystem processes differently output on terminal and on files or pipes. On terminal, output is flushed on each newline, which does not append on files or pipes unless the program specifically asks for an explicit flush.
It is not a problem on pipelines where the first command ends on an end of file, because everything is flushed before the command exists, and the write end of the pipe is closed. So next commands sees an end of file and everything propagates smoothly.
But when the first program contains an infinite reading loop, it does queue its output, but nothing is flushed until the buffer is full, and buffers are huge on modern systems. In that case, everything works fine but you cannot see any output.
Here is a version based on Alexandre Cox proposal with a piped command to check a mount:
import subprocess
fName = subprocess.Popen('mount | grep sda3 | cut -d " " -f 3', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
(stdout, stderr) = fName.communicate()
print(">>> {}".format(fName))
print(stdout)
If the mount exists, the output is:
>>> <subprocess.Popen object at 0x7f9b5eb2fdd8>
b'/mnt/sda3\n'
Related
I have a command that slowly outputs a list. I want this list to be saved on a file and also see it slowly being generated on the terminal.
python script.py 2>&1 | tee File.txt
This does not work for me. While the command is saved, I don't see the list of websites appearing on the terminal.
By default stdout is line buffered when going to a terminal, but uses a larger buffer when being redirected, hence tee and the terminal don't see the output until later.
For ways to get script.py to not buffer the output see the answers to this question Disable output buffering
For example if script.py is:
#!/usr/bin/python3
import time
for i in range(5):
print('This is line', i, flush=True)
time.sleep(1)
Running ./script.py | tee File.txt will print each line to the terminal as the line is executed - one second apart.
If you remove flush=True then the entire output is buffered, and nothing is printed until the script finishes 5 seconds later when everything is printed.
2>&1 redirects stderr to stdout, so you may need to apply the same buffering to stderr as well as stdout.
Per the Linux Documentation Project (TLDP),
2>&1
# Redirects stderr to stdout.
# Error messages get sent to same place as standard output.
And,
&>filename
# Redirect both stdout and stderr to file "filename."
So to pipe both to a file,
Command &> | tee File.txt
Or just stdout,
Command | tee File.txt
This question already has answers here:
How do I use subprocess.Popen to connect multiple processes by pipes?
(9 answers)
Closed 7 years ago.
I want to run this command using call subprocess
ls -l folder | wc -l
My code in Python file is here:
subprocess.call(["ls","-l","folder","|","wc","-l"])
I got an error message like this:
ls: cannot access |: No such file or directory
ls: cannot access wc: No such file or directory
It's like command |wc can't be read by call subprocess.
How can i fix it?
Try out the shell option using a string as first parameter:
subprocess.call("ls -l folder | wc -l",shell=True)
Although this work, note that using shell=True is not recommended since it can introduce a security issue through shell injection.
You can setup a command pipeline by connecting one process's stdout with another's stdin. In your example, errors and the final output are written to the screen, so I didn't try to redirect them. This is generally preferable to something like communicate because instead of waiting for one program to complete before starting another (and encouring the expense of moving the data into the parent) they run in parallel.
import subprocess
p1 = subprocess.Popen(["ls","-l"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["wc","-l"], stdin=p1.stdout)
# close pipe in parent, its still open in children
p1.stdout.close()
p2.wait()
p1.wait()
You'll need to implement the piping logic yourself to make it work properly.
def piped_call(prog1, prog2):
out, err = subprocess.call(prog1).communicate()
if err:
print(err)
return None
else:
return subprocess.call(prog2).communicate(out)
You could try using subprocess.PIPE, assuming you wanted to avoid using subprocess.call(..., shell=True).
import subprocess
# Run 'ls', sending output to a PIPE (shell equiv.: ls -l | ... )
ls = subprocess.Popen('ls -l folder'.split(),
stdout=subprocess.PIPE)
# Read output from 'ls' as input to 'wc' (shell equiv.: ... | wc -l)
wc = subprocess.Popen('wc -l'.split(),
stdin=ls.stdout,
stdout=subprocess.PIPE)
# Trap stdout and stderr from 'wc'
out, err = wc.communicate()
if err:
print(err.strip())
if out:
print(out.strip())
For Python 3 keep in mind the communicate() method used here will return a byte object instead of a string. :
In this case you will need to convert the output to a string using decode():
if err:
print(err.strip().decode())
if out:
print(out.strip().decode())
I need to get the duration of an Video for an application for Django. So I'll have to do this in python. But I'm really a beginner in this. So it would be nice, if you can help.
This is what I got so far:
import subprocess
task = subprocess.Popen("avconv -i video.mp4 2>&1 | grep Duration | cut -d ' ' -f 4 | sed -r 's/([^\.]*)\..*/\1/'", shell=True, stdout=subprocess.PIPE)
time = task.communicate()[0]
print time
I want to solve it with avconv because I'm allready using this at another point. The shell-command works well so far and gives me an output like:
HH:MM:SS.
But when I'm executing the python-code I just get an non-interpretable symbol on the shell.
Thanks a lot allready for your help!
Found a solution. Problem was the sed-part:
import os
import subprocess
task = subprocess.Popen("avconv -i video.mp4 2>&1 | grep Duration | cut -d ' ' -f 4 | sed -e 's/.\{4\}$//'", shell=True, stdout=subprocess.PIPE)
time = task.communicate()[0]
print time
Because it is allways the same part, it was enought to just cut the last 4 characters.
From python documentation:
Warning
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
So you should really user communicate for that:
import subprocess
task = subprocess.Popen("avconv -i video.mp4 2>&1 | grep Duration | cut -d ' ' -f 4 | sed -r 's/([^\.]*)\..*/\1/'", shell=True, stdout=subprocess.PIPE)
time = task.communicate()[0]
print time
That way you can also catch stderr message, if any.
I want to record voice, process it, than play it on the fly. The code below should describe more.
arecord -D plughw:USER -f S16_LE -t raw | python my_app.py | aplay -D plughw:USER -f S16_LE
I was able to capture input from arecord for my_app.py by using raw_input(). Now, I have problem on how to send my_app.py output to aplay.
I've tried using subprocess inside my_app.py to run aplay,
cmdPlay = 'aplay -D plughw:USER -f S16_LE'
p = subprocess.Popen(
cmdPlay,
stdout = subprocess.PIPE,
stdin = subprocess.PIPE,
stderr = subprocess.PIPE,
shell = True
)
p.communicate(input=dataHex.decode('hex'))
But, I heard no voice played on speaker.
So, how do you piping between python app and other processes (as input and output)?
n.b.: Please also suggest a better method, if any, than raw_input() for capture data from arecord.
I have been writing some python code and in my code I was using "command"
The code was working as I intended but then I noticed in the Python docs that command has been deprecated and will be removed in Python 3 and that I should use "subprocess" instead.
"OK" I think, "I don't want my code to go straight to legacy status, so I should change that right now.
The thing is that subprocess.Popen seems to prepend a nasty string to the start of any output e.g.
<subprocess.Popen object at 0xb7394c8c>
All the examples I see have it there, it seems to be accepted as given that it is always there.
This code;
#!/usr/bin/python
import subprocess
output = subprocess.Popen("ls -al", shell=True)
print output
produces this;
<subprocess.Popen object at 0xb734b26c>
brettg#underworld:~/dev$ total 52
drwxr-xr-x 3 brettg brettg 4096 2011-05-27 12:38 .
drwxr-xr-x 21 brettg brettg 4096 2011-05-24 17:40 ..
<trunc>
Is this normal? If I use it as part of a larger program that outputs various formatted details to the console it messes everything up.
I'm using the command to obtain the IP address for an interface by using ifconfig along with various greps and awks to scrape the address.
Consider this code;
#!/usr/bin/python
import commands,subprocess
def new_get_ip (netif):
address = subprocess.Popen("/sbin/ifconfig " + netif + " | grep inet | grep -v inet6 | awk '{print $2}' | sed 's/addr://'i", shell=True)
return address
def old_get_ip (netif):
address = commands.getoutput("/sbin/ifconfig " + netif + " | grep inet | grep -v inet6 | awk '{print $2}' | sed 's/addr://'i")
return address
print "OLD IP is :",old_get_ip("eth0")
print ""
print "NEW IP is :",new_get_ip("eth0")
This returns;
brettg#underworld:~/dev$ ./IPAddress.py
OLD IP is : 10.48.16.60
NEW IP is : <subprocess.Popen object at 0xb744270c>
brettg#underworld:~/dev$ 10.48.16.60
Which is fugly to say the least.
Obviously I am missing something here. I am new to Python of course so I'm sure it is me doing the wrong thing but various google searches have been fruitless to this point.
What if I want cleaner output? Do I have to manually trim the offending output or am I invoking subprocess.Popen incorrectly?
The "ugly string" is what it should be printing. Python is correctly printing out the repr(subprocess.Popen(...)), just like what it would print if you said print(open('myfile.txt')).
Furthermore, python has no knowledge of what is being output to stdout. The output you are seeing is not from python, but from the process's stdout and stderr being redirected to your terminal as spam, that is not even going through the python process. It's like you ran a program someprogram & without redirecting its stdout and stderr to /dev/null, and then tried to run another command, but you'd occasionally see spam from the program. To repeat and clarify:
<subprocess.Popen object at 0xb734b26c> <-- output of python program
brettg#underworld:~/dev$ total 52 <-- spam from your shell, not from python
drwxr-xr-x 3 brettg brettg 4096 2011-05-27 12:38 . <-- spam from your shell, not from python
drwxr-xr-x 21 brettg brettg 4096 2011-05-24 17:40 .. <-- spam from your shell, not from python
...
In order to capture stdout, you must use the .communicate() function, like so:
#!/usr/bin/python
import subprocess
output = subprocess.Popen(["ls", "-a", "-l"], stdout=subprocess.PIPE).communicate()[0]
print output
Furthermore, you never want to use shell=True, as it is a security hole (a major security hole with unsanitized inputs, a minor one with no input because it allows local attacks by modifying the shell environment). For security reasons and also to avoid bugs, you generally want to pass in a list rather than a string. If you're lazy you can do "ls -al".split(), which is frowned upon, but it would be a security hole to do something like ("ls -l %s"%unsanitizedInput).split().
See the subprocess module documentation for more information.
Here is how to get stdout and stderr from a program using the subprocess module:
from subprocess import Popen, PIPE, STDOUT
cmd = 'echo Hello World'
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = p.stdout.read()
print output
results:
b'Hello\r\n'
you can run commands with PowerShell and see results:
from subprocess import Popen, PIPE, STDOUT
cmd = 'powershell.exe ls'
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = p.stdout.read()
useful link
The variable output does not contain a string, it is a container for the subprocess.Popen() function. You don't need to print it. The code,
import subprocess
output = subprocess.Popen("ls -al", shell=True)
works perfectly, but without the ugly : <subprocess.Popen object at 0xb734b26c> being printed.