Develop Reference

subprocess.Popen() sends awk and grep lines differently than expected

On a CentOS 7.2 I have a file called cpuload, which contains the latest CPU load data in the following format:
last 30 sec:
average load: 0
cpu0 total load: 0
cpu1 total load: 0
cpu2 total load: 0
cpu3 total load: 1
cpu4 total load: 0
cpu5 total load: 0
cpu6 total load: 0
cpu7 total load: 0
last sec:
average load: 1
cpu0 total load: 5
cpu1 total load: 1
cpu2 total load: 1
cpu3 total load: 3
cpu4 total load: 2
cpu5 total load: 1
cpu6 total load: 0
cpu7 total load: 0
I want to get the number after the "average load:" of the "last sec" bit.
Two cli commands give me that information when I run them as shell commands on the terminal:
grep 'average load:' cpuload | sed -n 's/.*load: //p' | tail -n1
and
awk 'NR > 2 && /average load:/ {print $3}' cpuload
But when I run them in subprocess.Popen() with Shell=True I only get stderr:
for:
import subprocess
cmd = ["grep", "'average load:'", "cpuload", "|", "sed", "-n", "'s/.*load: //p'", "|", "tail", "-n1"]
test = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell=True)
test.stderr.read()
I get:
b"Usage: grep [OPTION]... PATTERN [FILE]...\nTry 'grep --help' for more information.\n"
and for:
import subprocess
cmd = cmd = ["awk", "'NR > 2 && /average load:/ {print $3}'", "cpuload"]
test = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
test.stderr.read()
I also get:
b"awk: cmd. line:1: 'NR > 2 && /average load:/ {print $3}'\nawk: cmd. line:1: ^ invalid char ''' in expression\n"
Even though I avoided using a |
or if shell=True I get:
b"Usage: awk [POSIX or GNU style options] -f progfile [--] file ...\nUsage: awk [POSIX or GNU style options] [--] 'program' file ...\nPOSIX options:\t\tGNU long options: (standard)\n\t-f progfile\t\t--file=progfile\n\t-F fs\t\t\t--field-separator=fs\n\t-v var=val\t\t--assign=var=val\nShort options:\t\tGNU long options: (extensions)\n\t-b\t\t\t--characters-as-bytes\n\t-c\t\t\t--traditional\n\t-C\t\t\t--copyright\n\t-d[file]\t\t--dump-variables[=file]\n\t-e 'program-text'\t--source='program-text'\n\t-E file\t\t\t--exec=file\n\t-g\t\t\t--gen-pot\n\t-h\t\t\t--help\n\t-L [fatal]\t\t--lint[=fatal]\n\t-n\t\t\t--non-decimal-data\n\t-N\t\t\t--use-lc-numeric\n\t-O\t\t\t--optimize\n\t-p[file]\t\t--profile[=file]\n\t-P\t\t\t--posix\n\t-r\t\t\t--re-interval\n\t-S\t\t\t--sandbox\n\t-t\t\t\t--lint-old\n\t-V\t\t\t--version\n\nTo report bugs, see node `Bugs' in `gawk.info', which is\nsection `Reporting Problems and Bugs' in the printed version.\n\ngawk is a pattern scanning and processing language.\nBy default it reads standard input and writes standard output.\n\nExamples:\n\tgawk '{ sum += $1 }; END { print sum }' file\n\tgawk -F: '{ print $1 }' /etc/passwd\n"
What am I doing wrong?

I have a file called cpuload, which contains the latest CPU load data ...I want to get the number after the "average load:" of the "last sec" bit
why not just to use simple python code in order to get the value you are looking for?
with open('cpuload') as f:
lines = [l.strip() for l in f.readlines()]
got_it = False
for line in lines:
if got_it:
parts = line.split(':')
result = parts[-1].strip()
print(result)
break
if line == 'last sec:':
got_it = True
output
1

First case with grep, sed, tail... and pipes.
You need to use shell = True parameter for Popen method and a single string for the command. We need to put cotes around parameters:
import subprocess
cmd = "grep 'average load:' cpuload | sed -n 's/.*load: //p' | tail -n1"
output = ""
test = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True)
while True:
output += test.stdout.readline().decode("utf-8")
if test.poll() is not None:
break
print("output=<%s>" % (output))
Second case, without pipe:
You don't need to use shell = True parameter for Popen method and a single string for the command. We don't put cotes around parameters:
import subprocess
cmd = ["/usr/bin/awk", "NR > 2 && /^average load:/ {print $3}", "cpuload"]
output = ""
test = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
while True:
output += test.stdout.readline().decode("utf-8")
if test.poll() is not None:
break
print("output=<%s>" % (output))

The issue was with passing the awk parameter to subprocess with ' around them, as detailed here
Did not accept Ed Morton's comment as it did not specify what should have been done.

greping with os command assigning it in tupe want to remove /t

args = parser.parse_args()
find = args.data_variable
cmd = commands.getoutput("cat /cc/ddd '"+find+"' |grep -A 3 node | tail -n +2")
r = tuple(cldb.split('\n'))
print r
i get the out put in tuples but it has /t please help how to remove it.

Use str.strip()
Ex:
cldb = ('\t\tm\mrcg1234', '\t\tmrcg1235', '\t\tmrcg1235.me.com')
cldb = tuple([i.strip() for i in cldb])
print(cldb)
Output:
('m\\mrcg1234', 'mrcg1235', 'mrcg1235.me.com')

How to port python2 code to python3?

I find out the following code that print shellcode or machine code of the objdump in the output but in python3 i can't run it. How can I port it to python3:
import subprocess
import sys
from subprocess import Popen
num_of_args = len(sys.argv)
binary_file = sys.argv[1]
#| awk -F'[:]' '{print $2}' | awk -F'[ ]' '{print $1}' | tr -d '[[:space:]]'
proc = subprocess.Popen(['arm-linux-gnueabi-objdump','-d',binary_file], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if line != '':
array = line.rstrip().split(':')
if len(array) > 1:
if array[1]:
array2 = array[1].split(' ')
array2 = array2[0].lstrip().rstrip()
if array2:
sc_part = '"'
sc_part += '\\x'
sc_part += '\\x'.join(a+b for a,b in zip(array2[::2], array2[1::2]))
sc_part += '"'
print sc_part
else:
break

Use the 2to3 tool that comes bundled with the python distribution. An example of its usage is this:
foo.py
def foo:
for i in xrange(5):
print i,
foo()
In the command line, you'll type:
$ 2to3 -w foo.py
foo.py (post command):
def foo():
for i in range(5):
print(i, end=' ')
foo()
Running 2to3 on your code, it appears only the print needs changing: print(sc_part)

print(sc_part) may well be the only thing you need to change.

It would be helpful if you pasted the stacktrace.
Anyways, try replacing print sc_part with print(sc_part)
Furthermore, you can read all of the changes in Python3 here.

Getting Python to print to the command line (the output of the graphviz command gvpr)

I'm trying to get Python to run this command, which runs fine from my command prompt:
ccomps -x rel_graph.dot | gvpr -c "N[nNodes($G)<5]{delete(0,$)}" | dot | gvpack | sfdp -Goverlap=prism | gvmap -e | gvpr "BEGIN{int m,w,e = 0} N[fontcolor=='blue']{m += 1} N[fontcolor=='green']{e += 1} N[fontcolor=='red']{w += 1} END{print(m); print(w); print(e);}"
In Python, I'm using:
temp = subprocess.Popen("""ccomps -x rel_graph.dot | gvpr -c \
"N[nNodes($G)<5]{delete(0,$)}" | dot | gvpack | sfdp -Goverlap=prism \
| gvmap -e | gvpr 'BEGIN{int m,w,e = 0} \
N[fontcolor=="blue"]{m += 1} \
N[fontcolor=="green"]{e += 1} \
N[fontcolor=="red"]{w += 1} \
END{print(m); print(w); print(e);}'
""", shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
...and then read/print lines from temp. The issue is that Python doesn't print the three last print statements (all are integers) to standard output, or at least I wasn't able to find it. The rest of the gvpr program works fine from Python.
Thanks!

After some more work I changed the BEGIN quotation marks to double, and all the internal arguments to single, and that seems to have solved the issue.

You can send the stdout \ stderr to files likes this -
from subprocess import Popen
std_out_file = "/tmp/stdout.log"
std_err_file = "/tmp/stderr.log"
command_to_execute = "<your-command>"
with open(std_out_file, "wb") as out, open(std_err_file, "wb") as err:
p = Popen(command_to_execute, shell=True, cwd=<folder>, stdout=out, stderr=err)
p.communicate()
Then you read the stdout \ stderr from the files, for example:
f = open(std_out_file, "r")
stdout = f.readlines()
f.close()
You can check the return code of the command, to check if you also need to print the stderr like this -
if p.returncode == 0:

Determine total size of SVN directory and store in python variable

this prints the directory size, but how can i save the output to a python variable, instead of print.
svn list -vR http://myIP/repos/test | awk '{sum+=$3; i++} END {print sum/1024000}'
but i need to store this print in a python variable;
proc = subprocess.Popen(svnproc, stdout=subprocess.PIPE, shell=True)
output = proc.stdout.read()
Print str(output)
nasty workaround is the push it out to a file and cat the file
svn list -vR http://myIP/repos/test | awk '{sum+=$3; i++} END {> /tmp/output.txt}'

From the fine docstring of "subprocess" I can read:
Replacing shell pipe line
output=dmesg | grep hda
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
so that in your case I'd try the following code
switches = ...
directory = ...
p1 = Popen(["svn", "list", switches, directory], stdout=PIPE)
p2 = Popen(["awk", "{sum+=$3; i++} END {print sum/1024/1024}", stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0].strip()
ps: I have changed from sum/1024000 to sum/1024/1024 assuming that you want to count in megabytes
"

svnproc = "svn list -vR " + repoURL + " | awk '{sum+=$3; i++} END {print sum/1073741824}'"
proc = subprocess.Popen(svnproc, shell=True,
stdout=subprocess.PIPE)
svnbackupsize = float(proc.stdout.read())
The only problematic part of this script is that Popen does not wait till the process is DONE, however subprocess.call does wait till the process is completed.