Whenever I use a command in a subprocess with "|" in it doesn't work it has an output of
Command "|" is unknown, try "in link help".
Or when I put this:
#!/usr/bin/python
from subprocess import call
from shlex import split
interface = call(split("ip -o link show | awk '{print $2}' | grep wl"))
It is giving the output of:
Error: either "dev" is duplicate, or "awk" is a garbage.
You can use subprocess.check_output method and Popen class though I wasn't able to chain both pipe operations. Partial solution:
from subprocess import check_output, Popen, PIPE
from shlex import split
process = Popen(split('ip -o link show'), stdout=PIPE)
output = check_output(('awk', '{print $2}'), stdin=process.stdout)
return_code = process.wait()
print(output, return_code)
So basically, awk is taking the process standard output, and result is saved in the output variable.
Related
I want only the wlan device name at a linux system with python. I could get the device name with shell scripting:
echo /sys/class/net/*/wireless | awk -F'/' '{ print $5 }'
So i want to use this at python with subprocess.
import shlex
import subprocess
def main():
echo = shlex.split('echo /sys/class/net/*/wireless')
echo_proc = subprocess.Popen(echo, shell=True, stdout=subprocess.PIPE)
awk = shlex.split("awk -F'/' '{ print $5 }'")
awk_proc = subprocess.Popen(awk, stdin=echo_proc.stdout)
print(awk_proc.stdout)
But I get only None as output. If it is possible, I would prefer a solution with subprocess.run(). So I replaced Popen with run. But then I get the error message AttributeError: 'bytes' object has no attribute 'fileno'.
A type glob and a pathname expansion by shell will be a headache.
In my environment, the following snippet works:
import subprocess
subprocess.run('echo /sys/class/net/*/wireless', shell=True)
But the following returns an empty string:
import subprocess
subprocess.run(['echo', '/sys/class/net/*/wireless'], shell=True)
Then please try the following as a starting point:
import subprocess
subprocess.run('echo /sys/class/net/*/wireless | awk -F"/" "{ print \$5 }"', shell=True)
which will bring your desired output.
[Update]
If you want to assign a variable to the output above, please try:
import subprocess
proc = subprocess.run('echo /sys/class/net/*/wireless | awk -F"/" "{ print \$5 }"', shell=True, stdout = subprocess.PIPE)
wlan = proc.stdout.decode("utf8").rstrip("\n")
print(wlan)
BTW if you don't stick to the subprocess module, why don't you go with a native way as:
import glob
list = glob.glob('/sys/class/net/*/wireless')
for elm in list:
print(elm.split('/')[4])
Hope this helps.
I know there are posts already on how to use subprocess in python to run linux commands but I just cant get the syntax correct for this one. please help. This is the command I need to run...
/sbin/ifconfig eth1 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'
Ok this is what I have at the moment that gives a syntax error...
import subprocess
self.ip = subprocess.Popen([/sbin/ifconfig eth1 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'])
Any help greatly appreciated.
This has been gone over many, many times before; but here is a simple pure Python replacement for the inefficient postprocessing.
from subprocess import Popen, PIPE
eth1 = subprocess.Popen(['/sbin/ifconfig', 'eth1'], stdout=PIPE)
out, err = eth1.communicate()
for line in out.split('\n'):
line = line.lstrip()
if line.startswith('inet addr:'):
ip = line.split()[1][5:]
Here's how to construct the pipe in Python (rather than reverting to Shell=True, which is more difficult to secure).
from subprocess import PIPE, Popen
# Do `which` to get correct paths
GREP_PATH = '/usr/bin/grep'
IFCONFIG_PATH = '/usr/bin/ifconfig'
AWK_PATH = '/usr/bin/awk'
awk2 = Popen([AWK_PATH, '{print $1}'], stdin=PIPE)
awk1 = Popen([AWK_PATH, '-F:', '{print $2}'], stdin=PIPE, stdout=awk2.stdin)
grep = Popen([GREP_PATH, 'inet addr'], stdin=PIPE, stdout=awk1.stdin)
ifconfig = Popen([IFCONFIG_PATH, 'eth1'], stdout=grep.stdin)
procs = [ifconfig, grep, awk1, awk2]
for proc in procs:
print(proc)
proc.wait()
It'd be better to do the string processing in Python using re. Do this to get the stdout of ifconfig.
from subprocess import check_output
stdout = check_output(['/usr/bin/ifconfig', 'eth1'])
print(stdout)
My subprocess call should be calling tabix 1kg.phase1.snp.bed.gz -B test.bed | awk '{FS="\t";OFS="\t"} $4 >= 10' but is giving me errors because it has both " and ' in it. I have tried using r for a raw string but I can't figure out the right combination to prevent errors. My current call looks like:
snp_tabix = subprocess.Popen(["tabix", tgp_snp, "-B", infile, "|", "awk", """'{FS="\t";OFS="\t"}""", "$4", ">=", maf_cut_off, r"'"], stdout=subprocess.PIPE)
Which gives the error TypeError: execv() arg 2 must contain only strings
r"'" is not the issue. Most likely you're passing maf_cut_off as an integer, which is incorrect. You should use str(maf_cut_off).
There are several issues. You are trying to execute a shell command (there is a pipe | in the command). So it won't work even if you convert all variables to strings.
You could execute it using shell:
from pipes import quote
from subprocess import check_output
cmd = r"""tabix %s -B %s | awk '{FS="\t";OFS="\t"} $4 >= %d'""" % (
quote(tgp_snp), quote(infile), maf_cut_off)
output = check_output(cmd, shell=True)
Or you could use the pipe recipe from subprocess' docs:
from subprocess import Popen, PIPE
tabix = Popen(["tabix", tgp_snp, "-B", infile], stdout=PIPE)
awk = Popen(["awk", r'{FS="\t";OFS="\t"} $4 >= %d' % maf_cut_off],
stdin=tabix.stdout, stdout=PIPE)
tabix.stdout.close() # allow tabix to receive a SIGPIPE if awk exits
output = awk.communicate()[0]
tabix.wait()
Or you could use plumbum that provides some syntax sugar for shell commands:
from plumbum.cmd import tabix, awk
cmd = tabix[tgp_snp, '-B', infile]
cmd |= awk[r'{FS="\t";OFS="\t"} $4 >= %d' % maf_cut_off]
output = cmd() # run it and get output
Another option is to reproduce the awk command in pure Python. To get all lines that have 4th field larger than or equal to maf_cut_off numerically (as an integer):
from subprocess import Popen, PIPE
tabix = Popen(["tabix", tgp_snp, "-B", infile], stdout=PIPE)
lines = []
for line in tabix.stdout:
columns = line.split(b'\t', 4)
if len(columns) > 3 and int(columns[3]) >= maf_cut_off:
lines.append(line)
output = b''.join(lines)
tabix.communicate() # close streams, wait for the subprocess to exit
tgp_snp, infile should be strings and maf_cut_off should be an integer.
You could use bufsize=-1 (Popen()'s parameter) to improve time performance.
How do I run a command with a pipe | in it?
The subprocess module seems complex...
Is there something like
output,error = `ps cax | grep something`
as in shell script?
See Replacing shell pipeline:
import subprocess
proc1 = subprocess.Popen(['ps', 'cax'], stdout=subprocess.PIPE)
proc2 = subprocess.Popen(['grep', 'python'], stdin=proc1.stdout,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc1.stdout.close() # Allow proc1 to receive a SIGPIPE if proc2 exits.
out, err = proc2.communicate()
print('out: {0}'.format(out))
print('err: {0}'.format(err))
PS. Using shell=True can be dangerous. See for example the warning in the docs.
There is also the sh module which can make subprocess scripting in Python a lot more pleasant:
import sh
print(sh.grep(sh.ps("cax"), 'something'))
You've already accepted an answer, but:
Do you really need to use grep? I'd write something like:
import subprocess
ps = subprocess.Popen(('ps', 'cax'), stdout=subprocess.PIPE)
output = ps.communicate()[0]
for line in output.split('\n'):
if 'something' in line:
...
This has the advantages of not involving shell=True and its riskiness, doesn't fork off a separate grep process, and looks an awful lot like the kind of Python you'd write to process data file-like objects.
import subprocess
process = subprocess.Popen("ps cax | grep something",
shell=True,
stdout=subprocess.PIPE,
)
stdout_list = process.communicate()[0].split('\n')
Drop that 'ps' subprocess and back away slowly! :)
Use the psutil module instead.
import os
os.system('ps -cax|grep something')
If you wanna replace grep argument with some variable:
os.system('ps -cax|grep '+your_var)
In Python I need to get the version of an external binary I need to call in my script.
Let's say that I want to use Wget in Python and I want to know its version.
I will call
os.system( "wget --version | grep Wget" )
and then I will parse the outputted string.
How to redirect the stdout of the os.command in a string in Python?
One "old" way is:
fin,fout=os.popen4("wget --version | grep Wget")
print fout.read()
The other modern way is to use a subprocess module:
import subprocess
cmd = subprocess.Popen('wget --version', shell=True, stdout=subprocess.PIPE)
for line in cmd.stdout:
if "Wget" in line:
print line
Use the subprocess module:
from subprocess import Popen, PIPE
p1 = Popen(["wget", "--version"], stdout=PIPE)
p2 = Popen(["grep", "Wget"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
Use subprocess instead.
If you are on *nix, I would recommend you to use commands module.
import commands
status, res = commands.getstatusoutput("wget --version | grep Wget")
print status # Should be zero in case of of success, otherwise would have an error code
print res # Contains stdout