In Python I need to get the version of an external binary I need to call in my script.
Let's say that I want to use Wget in Python and I want to know its version.
I will call
os.system( "wget --version | grep Wget" )
and then I will parse the outputted string.
How to redirect the stdout of the os.command in a string in Python?
One "old" way is:
fin,fout=os.popen4("wget --version | grep Wget")
print fout.read()
The other modern way is to use a subprocess module:
import subprocess
cmd = subprocess.Popen('wget --version', shell=True, stdout=subprocess.PIPE)
for line in cmd.stdout:
if "Wget" in line:
print line
Use the subprocess module:
from subprocess import Popen, PIPE
p1 = Popen(["wget", "--version"], stdout=PIPE)
p2 = Popen(["grep", "Wget"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
Use subprocess instead.
If you are on *nix, I would recommend you to use commands module.
import commands
status, res = commands.getstatusoutput("wget --version | grep Wget")
print status # Should be zero in case of of success, otherwise would have an error code
print res # Contains stdout
Related
How do I place output of bash command to Python variable?
I am writing a Python script, which I want to enter the output of
bash command:
rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | grep -v 'Red Hat'|wc -l, and place it to Python variable, let say R.
After that I want to do, Python if R != 0
then run some Linux command.
How do I achieve that?
There are various options, but the easiest is probably using subprocess.check_output() with shell=True although this can be security hazard if you don't fully control what command is passed in.
import subprocess
var = subprocess.check_output('rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | grep -v 'Red Hat'|wc -l', shell = True)
var = int(var)
You need to use shell=True as otherwise the pipes would not be interpreted.
If you need more control you might want to look at plumbum where you can do:
from plumbum.cmd import rpm, grep, wc
chain = rpm["-qa", "--qf", r"%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n"] | grep["-v", "Red Hat"] | wc["-l"]
R = int(chain())
Although I would probably not invoke wc and get the whole output and count its length within python (easier to check that you got just the lines that you expected, piping through wc -l throws away all of the details)
I would recommend envoy primarily because the API is much more intuitive to use for 90% of the use cases.
r = envoy.run('ls ', data='data to pipe in', timeout=2)
print r.status_code # returns status code
print r.std_out # returns the output.
See the Envoy Github page for more details.
You can use stdin.
#!/usr/bin/python
import sys
s = sys.stdin.read()
print s
Then you will run a bash command like this
echo "Hello" | ./myscript.py
Output
Hello
You can replace shell pipeline using Popen:
from subprocess import PIPE,Popen
p1 = Popen(["rpm", "-qa", "--qf", '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n'],stdout=PIPE)
p2 = Popen(["grep", "-v", 'Red Hat'],stdin=p1.stdout,stdout=PIPE)
p1.stdout.close()
p3 = Popen(["wc", "-l"],stdin=p2.stdout,stdout=PIPE)
p2.stdout.close()
out,err = p3.communicate()
If you just want to check if grep returned any matches then forget the wc - l and just check what grep returns:
p1 = Popen(["rpm", "-qa", "--qf", '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n'],stdout=PIPE)
p2 = Popen(["grep", "-v", 'Red Hat'],stdin=p1.stdout,stdout=PIPE)
p1.stdout.close()
out,err = p2.communicate()
if out:
...
Or just use check_output to run the rpm command and check the string for "Red Hat":
out = check_output(["rpm", "-qa", "--qf", '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n'])
if "Red Hat" not in out:
....
Which is the same as inverting the search with grep -v then checking if there are any matches with wc.
im having an issue trying to get a simple grep command into python. I want to take the output of the following command in a file or a list.
grep -c 'some thing' /home/user/* | grep -v :0
This is what I have, but its not working at all...
thing = str(subprocess.Popen(['grep', '-c', 'some thing', '/home/user/*', '|', 'grep', '-v', ':0'], stdout=subprocess.PIPE)
Basically I need to search files in a directory and return a result if my string is missing from any of the files in the directory.
Working Code (Thanks!!):
thing = subprocess.Popen(('grep -c "some thing" /home/user/* | grep -v ":0"' ),shell=True, stdout=subprocess.PIPE)
The pipe | is a shell feature. You have to use Popen with shell=True to use it.
To emulate the shell pipeline in Python, see How do I use subprocess.Popen to connect multiple processes by pipes?:
#!/usr/bin/env python
import os
from glob import glob
from subprocess import Popen, PIPE
p1 = Popen(["grep", "-c", 'some thing'] + glob(os.path.expanduser('~/*')),
stdout=PIPE)
p2 = Popen(["grep", "-v", ":0"], stdin=p1.stdout)
p1.stdout.close()
p2.wait()
p1.wait()
To get output as a string, set stdout=PIPE and call output = p2.communicate()[0] instead of p2.wait().
To suppress error messages such as "grep: /home/user/dir: Is a directory", you could set stderr=DEVNULL.
You could implement the pipeline in pure Python:
import os
from glob import glob
for name in glob(os.path.expanduser('~/*')):
try:
count = sum(1 for line in open(name, 'rb') if b'some thing' in line)
except IOError:
pass # ignore
else:
if count: # don't print zero counts
print("%s:%d" % (name, count))
I can run this normally on the command line in Linux:
$ tar c my_dir | md5sum
But when I try to call it with Python I get an error:
>>> subprocess.Popen(['tar','-c','my_dir','|','md5sum'],shell=True)
<subprocess.Popen object at 0x26c0550>
>>> tar: You must specify one of the `-Acdtrux' or `--test-label' options
Try `tar --help' or `tar --usage' for more information.
You have to use subprocess.PIPE, also, to split the command, you should use shlex.split() to prevent strange behaviours in some cases:
from subprocess import Popen, PIPE
from shlex import split
p1 = Popen(split("tar -c mydir"), stdout=PIPE)
p2 = Popen(split("md5sum"), stdin=p1.stdout)
But to make an archive and generate its checksum, you should use Python built-in modules tarfile and hashlib instead of calling shell commands.
Ok, I'm not sure why but this seems to work:
subprocess.call("tar c my_dir | md5sum",shell=True)
Anyone know why the original code doesn't work?
What you actually want is to run a shell subprocess with the shell command as a parameter:
>>> subprocess.Popen(['sh', '-c', 'echo hi | md5sum'], stdout=subprocess.PIPE).communicate()
('764efa883dda1e11db47671c4a3bbd9e -\n', None)
>>> from subprocess import Popen,PIPE
>>> import hashlib
>>> proc = Popen(['tar','-c','/etc/hosts'], stdout=PIPE)
>>> stdout, stderr = proc.communicate()
>>> hashlib.md5(stdout).hexdigest()
'a13061c76e2c9366282412f455460889'
>>>
i would try your on python v3.8.10 :
import subprocess
proc1 = subprocess.run(['tar c my_dir'], stdout=subprocess.PIPE, shell=True)
proc2 = subprocess.run(['md5sum'], input=proc1.stdout, stdout=subprocess.PIPE, shell=True)
print(proc2.stdout.decode())
key points (like outline in my solution on related https://stackoverflow.com/a/68323133/12361522):
subprocess.run()
no splits of bash command and parameters, i.e. ['tar c my_dir']or ["tar c my_dir"]
stdout=subprocess.PIPE for all processes
input=proc1.stdout chain of output of previous one into input of the next one
enable shell shell=True
How do I run a command with a pipe | in it?
The subprocess module seems complex...
Is there something like
output,error = `ps cax | grep something`
as in shell script?
See Replacing shell pipeline:
import subprocess
proc1 = subprocess.Popen(['ps', 'cax'], stdout=subprocess.PIPE)
proc2 = subprocess.Popen(['grep', 'python'], stdin=proc1.stdout,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc1.stdout.close() # Allow proc1 to receive a SIGPIPE if proc2 exits.
out, err = proc2.communicate()
print('out: {0}'.format(out))
print('err: {0}'.format(err))
PS. Using shell=True can be dangerous. See for example the warning in the docs.
There is also the sh module which can make subprocess scripting in Python a lot more pleasant:
import sh
print(sh.grep(sh.ps("cax"), 'something'))
You've already accepted an answer, but:
Do you really need to use grep? I'd write something like:
import subprocess
ps = subprocess.Popen(('ps', 'cax'), stdout=subprocess.PIPE)
output = ps.communicate()[0]
for line in output.split('\n'):
if 'something' in line:
...
This has the advantages of not involving shell=True and its riskiness, doesn't fork off a separate grep process, and looks an awful lot like the kind of Python you'd write to process data file-like objects.
import subprocess
process = subprocess.Popen("ps cax | grep something",
shell=True,
stdout=subprocess.PIPE,
)
stdout_list = process.communicate()[0].split('\n')
Drop that 'ps' subprocess and back away slowly! :)
Use the psutil module instead.
import os
os.system('ps -cax|grep something')
If you wanna replace grep argument with some variable:
os.system('ps -cax|grep '+your_var)
I would like to replicate this in python:
gvimdiff <(hg cat file.txt) file.txt
(hg cat file.txt outputs the most recently committed version of file.txt)
I know how to pipe the file to gvimdiff, but it won't accept another file:
$ hg cat file.txt | gvimdiff file.txt -
Too many edit arguments: "-"
Getting to the python part...
# hgdiff.py
import subprocess
import sys
file = sys.argv[1]
subprocess.call(["gvimdiff", "<(hg cat %s)" % file, file])
When subprocess is called it merely passes <(hg cat file) onto gvimdiff as a filename.
So, is there any way to redirect a command as bash does?
For simplicity's sake just cat a file and redirect it to diff:
diff <(cat file.txt) file.txt
It can be done. As of Python 2.5, however, this mechanism is Linux-specific and not portable:
import subprocess
import sys
file = sys.argv[1]
p1 = subprocess.Popen(['hg', 'cat', file], stdout=subprocess.PIPE)
p2 = subprocess.Popen([
'gvimdiff',
'/proc/self/fd/%s' % p1.stdout.fileno(),
file])
p2.wait()
That said, in the specific case of diff, you can simply take one of the files from stdin, and remove the need to use the bash-alike functionality in question:
file = sys.argv[1]
p1 = subprocess.Popen(['hg', 'cat', file], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['diff', '-', file], stdin=p1.stdout)
diff_text = p2.communicate()[0]
There is also the commands module:
import commands
status, output = commands.getstatusoutput("gvimdiff <(hg cat file.txt) file.txt")
There is also the popen set of functions, if you want to actually grok the data from a command as it is running.
This is actually an example in the docs:
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
which means for you:
import subprocess
import sys
file = sys.argv[1]
p1 = Popen(["hg", "cat", file], stdout=PIPE)
p2 = Popen(["gvimdiff", "file.txt"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
This removes the use of the linux-specific /proc/self/fd bits, making it probably work on other unices like Solaris and the BSDs (including MacOS) and maybe even work on Windows.
It just dawned on me that you are probably looking for one of the popen functions.
from: http://docs.python.org/lib/module-popen2.html
popen3(cmd[, bufsize[, mode]])
Executes cmd as a sub-process. Returns the file objects (child_stdout, child_stdin, child_stderr).
namaste,
Mark