Develop Reference

converting command line syntax to os.system call or subprocess cal; in python [duplicate]

How would one call a shell command from Python which contains a pipe and capture the output?
Suppose the command was something like:
cat file.log | tail -1
The Perl equivalent of what I am trying to do would be something like:
my $string = `cat file.log | tail -1`;

Use a subprocess.PIPE, as explained in the subprocess docs section "Replacing shell pipeline":
import subprocess
p1 = subprocess.Popen(["cat", "file.log"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["tail", "-1"], stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output,err = p2.communicate()
Or, using the sh module, piping becomes composition of functions:
import sh
output = sh.tail(sh.cat('file.log'), '-1')

import subprocess
task = subprocess.Popen("cat file.log | tail -1", shell=True, stdout=subprocess.PIPE)
data = task.stdout.read()
assert task.wait() == 0
Note that this does not capture stderr. And if you want to capture stderr as well, you'll need to use task.communicate(); calling task.stdout.read() and then task.stderr.read() can deadlock if the buffer for stderr fills. If you want them combined, you should be able to use 2>&1 as part of the shell command.
But given your exact case,
task = subprocess.Popen(['tail', '-1', 'file.log'], stdout=subprocess.PIPE)
data = task.stdout.read()
assert task.wait() == 0
avoids the need for the pipe at all.

This:
import subprocess
p = subprocess.Popen("cat file.log | tail -1", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
#for shell=False use absolute paths
p_stdout = p.stdout.read()
p_stderr = p.stderr.read()
print p_stdout
Or this should work:
import os
result = os.system("cat file.log | tail -1")

Another way similar to Popen would be:
command=r"""cat file.log | tail -1 """
output=subprocess.check_output(command, shell=True)

This is a fork from #chown with some improvements:
an alias for import subprocess, makes easier when setting parameters
if you just want the output, you don't need to set stderr or stdin when calling Popen
for better formatting, it's recommended to decode the output
shell=True is necessary, in order to call an interpreter for the command line
#!/usr/bin/python3
import subprocess as sp
p = sp.Popen("cat app.log | grep guido", shell=True, stdout=sp.PIPE)
output = p.stdout.read()
print(output.decode('utf-8'))
$ cat app.log
2017-10-14 22:34:12, User Removed [albert.wesker]
2017-10-26 18:14:02, User Removed [alexei.ivanovich]
2017-10-28 12:14:56, User Created [ivan.leon]
2017-11-14 09:22:07, User Created [guido.rossum]
$ python3 subproc.py
2017-11-14 09:22:07, User Created [guido.rossum]

Simple function for run shell command with many pipes
Using
res, err = eval_shell_cmd('pacman -Qii | grep MODIFIED | grep -v UN | cut -f 2')
Function
import subprocess
def eval_shell_cmd(command, debug=False):
"""
Eval shell command with pipes and return result
:param command: Shell command
:param debug: Debug flag
:return: Result string
"""
processes = command.split(' | ')
if debug:
print('Processes:', processes)
for index, value in enumerate(processes):
args = value.split(' ')
if debug:
print(index, args)
if index == 0:
p = subprocess.Popen(args, stdout=subprocess.PIPE)
else:
p = subprocess.Popen(args, stdin=p.stdout, stdout=subprocess.PIPE)
if index == len(processes) - 1:
result, error = p.communicate()
return result.decode('utf-8'), error

Python, evaluation output of OS command in an if statement

I want to convert the following shell evaluation to python2.6(can't upgrade). I can't figure out how to evaluate the output of the command.
Here's the shell version:
status=`$hastatus -sum |grep $hostname |grep Grp| awk '{print $6}'`
if [ $status != "ONLINE" ]; then
exit 1
fi
I tried os.popen and it returns ['ONLINE\n'].
value = os.popen("hastatus -sum |grep `hostname` |grep Grp| awk '{print $6}'".readlines()
print value

Try the subprocess module:
import subprocess
value = subprocess.call("hastatus -sum |grep `hostname` |grep Grp| awk '{print $6}'")
print(value)
Documentation is found here:
https://docs.python.org/2.6/library/subprocess.html?highlight=subprocess#module-subprocess

The recommended way is to use the subprocess module.
The following section of the documentation is instructive:
replacing shell pipeline
I report here for reference:
output=dmesg | grep hda
becomes:
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]
The p1.stdout.close() call after starting the p2 is important in order for p1 to receive a SIGPIPE if p2 exits before p1.
Alternatively, for trusted input, the shell’s own pipeline support may still be used directly:
output=dmesg | grep hda
becomes:
output=check_output("dmesg | grep hda", shell=True)
And here the recipe to translate os.popen to the subprocess module:
replacing os.popen()
So in your case you could do something like
import subprocess
output=check_output("hastatus -sum |grep `hostname` |grep Grp| awk '{print $6}'", shell=True)
or
concatenating the Popens as showed in the documentation above (probably what I would do).
Then to test the output you could just use, assuming you're using the first approach:
import sys
import subprocess
....
if 'ONLINE' in output:
sys.exit(1)

Python - how to execute shell commands with pipe, but without 'shell=True'?

I have a case to want to execute the following shell command in Python and get the output,
echo This_is_a_testing | grep -c test
I could use this python code to execute the above shell command in python,
>>> import subprocess
>>> subprocess.check_output("echo This_is_a_testing | grep -c test", shell=True)
'1\n'
However, as I do not want to use the "shell=True" option, I tried the following python code,
>>> import subprocess
>>> p1 = subprocess.Popen(["echo", "This_is_a_testing"], stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout)
>>> p1.stdout.close()
>>> p2.communicate()
(None, None)
I wonder why the output is "None" as I have referred to the descriptions in the webpage : http://docs.python.org/library/subprocess.html#subprocess.PIPE
Had I missed some points in my code ? Any suggestion / idea ? Thanks in advance.

>>> import subprocess
>>> mycmd=subprocess.getoutput('df -h | grep home | gawk \'{ print $1 }\' | cut -d\'/\' -f3')
>>> mycmd
'sda6'
>>>

Please look here:
>>> import subprocess
>>> p1 = subprocess.Popen(["echo", "This_is_a_testing"], stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout)
>>> 1
p1.stdout.close()
>>> p2.communicate()
(None, None)
>>>
here you get 1 as output after you write p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout), Do not ignore this output in the context of your question.
If this is what you want, then pass stdout=subprocess.PIPE as argument to the second Popen:
>>> p1 = subprocess.Popen(["echo", "This_is_a_testing"], stdout=subprocess.PIPE)
>>> p2 = subprocess.Popen(["grep", "test"], stdin=p1.stdout, stdout=subprocess.PIPE)
>>> p2.communicate()
('This_is_a_testing\n', None)
>>>

From the manual:
to get anything other than None in the result tuple, you need to give
stdout=PIPE and/or stderr=PIPE
p2 = subprocess.Popen(["grep", "-c", "test"], stdin=p1.stdout, stdout=subprocess.PIPE)

While the accepted answer is correct/working, another option would be to use the Popen.communicate() method to pass something to a process' stdin:
>>> import subprocess
>>> p2 = subprocess.Popen(["grep", "-c", "test"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> p2.communicate("This_is_a_testing")
('1\n', None)
>>> print p2.returncode
0
>>>>
This resolves the need to execute another command just to redirect it's output, if the output is already known in the python script itself.
However communicate has the side-effect, that it waits for the process to terminate. If asynchronous execution is needed/desired using two processes might be the better option.

Answer is similar to mentioned earlier, with little formatting. I wanted to get exactly same output as normal shell command with pipe on python 3.
import subprocess
p1 = subprocess.Popen(["ls", "-l", "."], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "May"], stdin=p1.stdout, stdout=subprocess.PIPE)
for s in (str(p2.communicate())[2:-10]).split('\\n'):
print(s)

Convert complex command into python subprocess

I have the following command:
$ ffmpeg -i http://url/1video.mp4 2>&1 | perl -lane 'print $1 if /(\d+x\d+)/'
640x360
I'm trying to set the output of this command into a python variable. Here is what I have so far:
>>> from subprocess import Popen, PIPE
>>> p1 = Popen(['ffmpeg', '-i', 'http://url/1video.mp4', '2>&1'], stdout=PIPE)
>>> p2=Popen(['perl','-lane','print $1 if /(\d+x\d+)/'], stdin=p1.stdout, stdout=PIPE)
>>> dimensions = p2.communicate()[0]
''
What am I doing incorrectly here, and how would I get the correct value for dimensions?

In general, you can replace a shell pipeline with this pattern:
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]
However, in this case, no pipeline is necessary:
import subprocess
import shlex
import re
url='http://url/1video.mp4'
proc=subprocess.Popen(shlex.split('ffmpeg -i {f}'.format(f=url)),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
dimensions=None
for line in proc.stderr:
match=re.search(r'(\d+x\d+)',line)
if match:
dimensions=match.group(1)
break
print(dimensions)

No need to call perl from within python.
If you have the output from ffmpeg in a variable, you can do something like this:
print re.search(r'(\d+x\d+)', str).group()

Note the “shell” argument to subprocess.Popen: this specifies whether the command you pass is parsed by the shell or not.
That “2>&1” is one of those things that needs to be parsed by a shell, otherwise FFmpeg (like most programs) will try to treat it as a filename or option value.
The Python sequence that most closely mimics the original would probably be more like
p1 = subprocess.Popen("ffmpeg -i http://url/1video.mp4 2>&1", shell = True, stdout = subprocess.PIPE)<BR>
p2 = subprocess.Popen(r"perl -lane 'print $1 if /(\d+x\d+)/'", shell = True, stdin = p1.stdout, stdout = subprocess.PIPE)<BR>
dimensions = p2.communicate()[0]

Redirect command to input of another in Python

I would like to replicate this in python:
gvimdiff <(hg cat file.txt) file.txt
(hg cat file.txt outputs the most recently committed version of file.txt)
I know how to pipe the file to gvimdiff, but it won't accept another file:
$ hg cat file.txt | gvimdiff file.txt -
Too many edit arguments: "-"
Getting to the python part...
# hgdiff.py
import subprocess
import sys
file = sys.argv[1]
subprocess.call(["gvimdiff", "<(hg cat %s)" % file, file])
When subprocess is called it merely passes <(hg cat file) onto gvimdiff as a filename.
So, is there any way to redirect a command as bash does?
For simplicity's sake just cat a file and redirect it to diff:
diff <(cat file.txt) file.txt

It can be done. As of Python 2.5, however, this mechanism is Linux-specific and not portable:
import subprocess
import sys
file = sys.argv[1]
p1 = subprocess.Popen(['hg', 'cat', file], stdout=subprocess.PIPE)
p2 = subprocess.Popen([
'gvimdiff',
'/proc/self/fd/%s' % p1.stdout.fileno(),
file])
p2.wait()
That said, in the specific case of diff, you can simply take one of the files from stdin, and remove the need to use the bash-alike functionality in question:
file = sys.argv[1]
p1 = subprocess.Popen(['hg', 'cat', file], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['diff', '-', file], stdin=p1.stdout)
diff_text = p2.communicate()[0]

There is also the commands module:
import commands
status, output = commands.getstatusoutput("gvimdiff <(hg cat file.txt) file.txt")
There is also the popen set of functions, if you want to actually grok the data from a command as it is running.

This is actually an example in the docs:
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
which means for you:
import subprocess
import sys
file = sys.argv[1]
p1 = Popen(["hg", "cat", file], stdout=PIPE)
p2 = Popen(["gvimdiff", "file.txt"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
This removes the use of the linux-specific /proc/self/fd bits, making it probably work on other unices like Solaris and the BSDs (including MacOS) and maybe even work on Windows.

It just dawned on me that you are probably looking for one of the popen functions.
from: http://docs.python.org/lib/module-popen2.html
popen3(cmd[, bufsize[, mode]])
Executes cmd as a sub-process. Returns the file objects (child_stdout, child_stdin, child_stderr).
namaste,
Mark