I have the following command:
$ ffmpeg -i http://url/1video.mp4 2>&1 | perl -lane 'print $1 if /(\d+x\d+)/'
640x360
I'm trying to set the output of this command into a python variable. Here is what I have so far:
>>> from subprocess import Popen, PIPE
>>> p1 = Popen(['ffmpeg', '-i', 'http://url/1video.mp4', '2>&1'], stdout=PIPE)
>>> p2=Popen(['perl','-lane','print $1 if /(\d+x\d+)/'], stdin=p1.stdout, stdout=PIPE)
>>> dimensions = p2.communicate()[0]
''
What am I doing incorrectly here, and how would I get the correct value for dimensions?
In general, you can replace a shell pipeline with this pattern:
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]
However, in this case, no pipeline is necessary:
import subprocess
import shlex
import re
url='http://url/1video.mp4'
proc=subprocess.Popen(shlex.split('ffmpeg -i {f}'.format(f=url)),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
dimensions=None
for line in proc.stderr:
match=re.search(r'(\d+x\d+)',line)
if match:
dimensions=match.group(1)
break
print(dimensions)
No need to call perl from within python.
If you have the output from ffmpeg in a variable, you can do something like this:
print re.search(r'(\d+x\d+)', str).group()
Note the “shell” argument to subprocess.Popen: this specifies whether the command you pass is parsed by the shell or not.
That “2>&1” is one of those things that needs to be parsed by a shell, otherwise FFmpeg (like most programs) will try to treat it as a filename or option value.
The Python sequence that most closely mimics the original would probably be more like
p1 = subprocess.Popen("ffmpeg -i http://url/1video.mp4 2>&1", shell = True, stdout = subprocess.PIPE)<BR>
p2 = subprocess.Popen(r"perl -lane 'print $1 if /(\d+x\d+)/'", shell = True, stdin = p1.stdout, stdout = subprocess.PIPE)<BR>
dimensions = p2.communicate()[0]
Related
How would one call a shell command from Python which contains a pipe and capture the output?
Suppose the command was something like:
cat file.log | tail -1
The Perl equivalent of what I am trying to do would be something like:
my $string = `cat file.log | tail -1`;
Use a subprocess.PIPE, as explained in the subprocess docs section "Replacing shell pipeline":
import subprocess
p1 = subprocess.Popen(["cat", "file.log"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["tail", "-1"], stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output,err = p2.communicate()
Or, using the sh module, piping becomes composition of functions:
import sh
output = sh.tail(sh.cat('file.log'), '-1')
import subprocess
task = subprocess.Popen("cat file.log | tail -1", shell=True, stdout=subprocess.PIPE)
data = task.stdout.read()
assert task.wait() == 0
Note that this does not capture stderr. And if you want to capture stderr as well, you'll need to use task.communicate(); calling task.stdout.read() and then task.stderr.read() can deadlock if the buffer for stderr fills. If you want them combined, you should be able to use 2>&1 as part of the shell command.
But given your exact case,
task = subprocess.Popen(['tail', '-1', 'file.log'], stdout=subprocess.PIPE)
data = task.stdout.read()
assert task.wait() == 0
avoids the need for the pipe at all.
This:
import subprocess
p = subprocess.Popen("cat file.log | tail -1", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
#for shell=False use absolute paths
p_stdout = p.stdout.read()
p_stderr = p.stderr.read()
print p_stdout
Or this should work:
import os
result = os.system("cat file.log | tail -1")
Another way similar to Popen would be:
command=r"""cat file.log | tail -1 """
output=subprocess.check_output(command, shell=True)
This is a fork from #chown with some improvements:
an alias for import subprocess, makes easier when setting parameters
if you just want the output, you don't need to set stderr or stdin when calling Popen
for better formatting, it's recommended to decode the output
shell=True is necessary, in order to call an interpreter for the command line
#!/usr/bin/python3
import subprocess as sp
p = sp.Popen("cat app.log | grep guido", shell=True, stdout=sp.PIPE)
output = p.stdout.read()
print(output.decode('utf-8'))
$ cat app.log
2017-10-14 22:34:12, User Removed [albert.wesker]
2017-10-26 18:14:02, User Removed [alexei.ivanovich]
2017-10-28 12:14:56, User Created [ivan.leon]
2017-11-14 09:22:07, User Created [guido.rossum]
$ python3 subproc.py
2017-11-14 09:22:07, User Created [guido.rossum]
Simple function for run shell command with many pipes
Using
res, err = eval_shell_cmd('pacman -Qii | grep MODIFIED | grep -v UN | cut -f 2')
Function
import subprocess
def eval_shell_cmd(command, debug=False):
"""
Eval shell command with pipes and return result
:param command: Shell command
:param debug: Debug flag
:return: Result string
"""
processes = command.split(' | ')
if debug:
print('Processes:', processes)
for index, value in enumerate(processes):
args = value.split(' ')
if debug:
print(index, args)
if index == 0:
p = subprocess.Popen(args, stdout=subprocess.PIPE)
else:
p = subprocess.Popen(args, stdin=p.stdout, stdout=subprocess.PIPE)
if index == len(processes) - 1:
result, error = p.communicate()
return result.decode('utf-8'), error
I tested following command on bash(Linux) and it works fine:
awk '/string1\/parameters\/string2/' RS= myfile | grep Value | sed 's/.*"\(.*\)"[^"]*$/\1/'
Now I have to call it in a python script, while string1 and string2 are python variables.
I tried it with os.popen but I didn't figure out how to concatenate the characters.
Any ideas how to solve this issue?
Thank you in advance for your help!
You can replace shell pipeline with Popen:
from subprocess import PIPE,Popen
from shlex import split
p1 = Popen(split("awk /string1\/parameters\/string2 RS=myfile"), stdout=PIPE)
p2 = Popen(["grep", "Value"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
p3 = Popen(split("""sed 's/.*"\(.*\)"[^"]*$/\1/'"""), stdin=p2.stdout, stdout=PIPE)
p2.stdout.close() # Allow p2 to receive a SIGPIPE if p3 exits.
output = p3.communicate()[0]
You can use subprocess.check_output() with the variables being substituted into the command using format():
cmd = """awk '/{}\/parameters\/{}/' RS= myfile | grep Value | sed 's/.*"\(.*\)"[^"]*$/\1/'""".format('string1', 'string2')
cmd_output = subprocess.check_output(cmd, shell=True)
But note the warnings regarding the use of shell=True in the referenced documentation.
An alternative is to set up the pipeline yourself using Popen():
import shlex
from subprocess import Popen, PIPE
awk_cmd = """awk '/{}\/parameters\/{}/' RS= myfile""".format('s1', 's2')
grep_cmd = 'grep Value'
sed_cmd = """sed 's/.*"\(.*\)"[^"]*$/\1/'"""
p_awk = Popen(shlex.split(awk_cmd), stdout=PIPE)
p_grep = Popen(shlex.split(grep_cmd), stdin=p_awk.stdout, stdout=PIPE)
p_sed = Popen(shlex.split(sed_cmd), stdin=p_grep.stdout, stdout=PIPE)
for p in p_awk, p_grep:
p.stdout.close()
stdout, stderr = p_sed.communicate()
print stdout
It seems that using shell=True in the first process of a chain somehow drops the stdout from downstream tasks:
p1 = Popen(['echo','hello'], stdout=PIPE)
p2 = Popen('cat', stdin=p1.stdout, stdout=PIPE)
p2.communicate()
# outputs correctly ('hello\n', None)
Making the first process use shell=True kills the output somehow...
p1 = Popen(['echo','hello'], stdout=PIPE, shell=True)
p2 = Popen('cat', stdin=p1.stdout, stdout=PIPE)
p2.communicate()
# outputs incorrectly ('\n', None)
shell=True on the second process doesn't seem to matter. Is this expected behavior?
When you pass shell=True, Popen expects a single string argument, not a list. So when you do this:
p1 = Popen(['echo','hello'], stdout=PIPE, shell=True)
What happens is this:
execve("/bin/sh", ["/bin/sh", "-c", "echo", "hello"], ...)
That is, it calls sh -c "echo", and hello is effectively ignored (technically it becomes a positional argument to the shell). So the shell runs echo, which prints \n, which is why you see that in your output.
If you use shell=True, you need to do this:
p1 = Popen('echo hello', stdout=PIPE, shell=True)
I have a case to want to execute the following shell command in Python and get the output,
echo This_is_a_testing | grep -c test
I could use this python code to execute the above shell command in python,
>>> import subprocess
>>> subprocess.check_output("echo This_is_a_testing | grep -c test", shell=True)
'1\n'
However, as I do not want to use the "shell=True" option, I tried the following python code,
>>> import subprocess
>>> p1 = subprocess.Popen(["echo", "This_is_a_testing"], stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout)
>>> p1.stdout.close()
>>> p2.communicate()
(None, None)
I wonder why the output is "None" as I have referred to the descriptions in the webpage : http://docs.python.org/library/subprocess.html#subprocess.PIPE
Had I missed some points in my code ? Any suggestion / idea ? Thanks in advance.
>>> import subprocess
>>> mycmd=subprocess.getoutput('df -h | grep home | gawk \'{ print $1 }\' | cut -d\'/\' -f3')
>>> mycmd
'sda6'
>>>
Please look here:
>>> import subprocess
>>> p1 = subprocess.Popen(["echo", "This_is_a_testing"], stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout)
>>> 1
p1.stdout.close()
>>> p2.communicate()
(None, None)
>>>
here you get 1 as output after you write p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout), Do not ignore this output in the context of your question.
If this is what you want, then pass stdout=subprocess.PIPE as argument to the second Popen:
>>> p1 = subprocess.Popen(["echo", "This_is_a_testing"], stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(["grep", "test"], stdin=p1.stdout, stdout=subprocess.PIPE)
>>> p2.communicate()
('This_is_a_testing\n', None)
>>>
From the manual:
to get anything other than None in the result tuple, you need to give
stdout=PIPE and/or stderr=PIPE
p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout, stdout=subprocess.PIPE)
While the accepted answer is correct/working, another option would be to use the Popen.communicate() method to pass something to a process' stdin:
>>> import subprocess
>>> p2 = subprocess.Popen(["grep", "-c", "test"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> p2.communicate("This_is_a_testing")
('1\n', None)
>>> print p2.returncode
0
>>>>
This resolves the need to execute another command just to redirect it's output, if the output is already known in the python script itself.
However communicate has the side-effect, that it waits for the process to terminate. If asynchronous execution is needed/desired using two processes might be the better option.
Answer is similar to mentioned earlier, with little formatting. I wanted to get exactly same output as normal shell command with pipe on python 3.
import subprocess
p1 = subprocess.Popen(["ls", "-l", "."], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "May"], stdin=p1.stdout, stdout=subprocess.PIPE)
for s in (str(p2.communicate())[2:-10]).split('\\n'):
print(s)
I would like to replicate this in python:
gvimdiff <(hg cat file.txt) file.txt
(hg cat file.txt outputs the most recently committed version of file.txt)
I know how to pipe the file to gvimdiff, but it won't accept another file:
$ hg cat file.txt | gvimdiff file.txt -
Too many edit arguments: "-"
Getting to the python part...
# hgdiff.py
import subprocess
import sys
file = sys.argv[1]
subprocess.call(["gvimdiff", "<(hg cat %s)" % file, file])
When subprocess is called it merely passes <(hg cat file) onto gvimdiff as a filename.
So, is there any way to redirect a command as bash does?
For simplicity's sake just cat a file and redirect it to diff:
diff <(cat file.txt) file.txt
It can be done. As of Python 2.5, however, this mechanism is Linux-specific and not portable:
import subprocess
import sys
file = sys.argv[1]
p1 = subprocess.Popen(['hg', 'cat', file], stdout=subprocess.PIPE)
p2 = subprocess.Popen([
'gvimdiff',
'/proc/self/fd/%s' % p1.stdout.fileno(),
file])
p2.wait()
That said, in the specific case of diff, you can simply take one of the files from stdin, and remove the need to use the bash-alike functionality in question:
file = sys.argv[1]
p1 = subprocess.Popen(['hg', 'cat', file], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['diff', '-', file], stdin=p1.stdout)
diff_text = p2.communicate()[0]
There is also the commands module:
import commands
status, output = commands.getstatusoutput("gvimdiff <(hg cat file.txt) file.txt")
There is also the popen set of functions, if you want to actually grok the data from a command as it is running.
This is actually an example in the docs:
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
which means for you:
import subprocess
import sys
file = sys.argv[1]
p1 = Popen(["hg", "cat", file], stdout=PIPE)
p2 = Popen(["gvimdiff", "file.txt"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
This removes the use of the linux-specific /proc/self/fd bits, making it probably work on other unices like Solaris and the BSDs (including MacOS) and maybe even work on Windows.
It just dawned on me that you are probably looking for one of the popen functions.
from: http://docs.python.org/lib/module-popen2.html
popen3(cmd[, bufsize[, mode]])
Executes cmd as a sub-process. Returns the file objects (child_stdout, child_stdin, child_stderr).
namaste,
Mark