Develop Reference

Execute df | grep -w "/" not parsing output correctly

I am trying to run the shell command df -h | grep -w "/" using python to watch the root partition usage and wanted to avoid shell=True option for security.
The code I tried as follows:
import subprocess
p1 = subprocess.Popen(['df', '-h'], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['grep', '-w', '"/"'], stdin=p1.stdout, stdout=subprocess.PIPE)
output=p2.communicate()[0]
print(output)
The output I get is:
$ ./subprocess_df_check.py
b''
Expected output is:
$ df -h | grep -w "/"
/dev/sdd 251G 4.9G 234G 3% /

The immediate problem is the unnecessary quotes being added.
p2 = subprocess.Popen(['grep', '-w', '"/"'], stdin=p1.stdout, stdout=subprocess.PIPE)
is not equivalent to the shell command grep -w "/". Instead, it's equivalent to the shell command grep -w '"/"', (or grep -w \"/\", or any other means of writing an argument vector that passes literal double-quote characters on the last non-NUL element of grep's argument vector) and wrong for the same reasons.
Use '/', not '"/"'.

Don't use subprocess with df and / or grep. If you already use python, you can use the statvfs function call like:
import os
import time
path = "/"
while True:
info = os.statvfs(path)
print("Block size [%d] Free blocks [%d] Free inodes [%d]"
% (info.f_bsize, info.f_bfree, info.f_ffree))
time.sleep(15)

Running grep in a separate subprocess is certainly unnecessary. If you are using Python, you already have an excellent tool for looking for substrings within strings.
df = subprocess.run(['df', '-h'],
capture_output=True, text=True, check=True)
for line in df.stdout.split('\n')[1:]:
if '/' in line:
print(line)
Notice also how you basically always want to prefer subprocess.run over Popen when you can, and how you want text=True to get text rather than bytes. Usually you also want check=True to ensure that the subprocess completed successfully.

Ok figured out the whole thing.
import subprocess
p1 = subprocess.Popen(['df', '-h'], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['grep', '-w', '/'], stdin=p1.stdout, stdout=subprocess.PIPE)
output=p2.communicate()[0].split()[4]
print("Root partition is of", output.decode(), "usage now")
Removed unnecessary double quotes, changed from subprocess.Popen(['grep', '-w', '"/"'] to subprocess.Popen(['grep', '-w', '/']. The double quotes are for the shell, not for df. When you have no shell, you need no shell syntax.
On output=p2.communicate()[0].split()[4], the [0] picks only stdout, not the stderr, which is None if no error. Then split()[4] cuts the 4th column which is disk usage percent value from shell df command.
output.decode(), the decode() is to decode the encoded bytes string format and avoid outputting character b in front of the result. Refer here
So the output of the script is:
$ ./subprocess_df_check.py
Root partition is of 3% usage now

Python os.system timeout with strings

I'm unable to execute the following line:
os.system("timeout 1s bash -c \"ffmpeg -i \""+path+\"+" | <some_<other_cmd>\"")
So the purpose of this command is to set a timeout for the whole command, i.e. pipelining some ffmpeg information from a path.
The problem is because bash -c "CMD" is expected, but the command also contains " ".
Is there another way of defining the \"path\", because the path can contain spaces? Or another solution which can resolve my problem?
Thanks in advance!

Triple sinqle quotes can do the trick (so that you do not have to escape doublequotes):
os.system('''timeout 1s bash -c "ffmpeg -i "+path+"+" | cat''')
But in general.. Why not use subprocess.call that has saner syntax?

Has answered similar question in other posts: 1 and 2
you can use subprocess related functions, which all support timeout parameter, to replace os.system
such as subprocess.check_output
ffmpegCmd = "ffmpeg -I %s | %s" % (path, someOtherCmd)
outputBytes = subprocess.check_output(ffmpegCmd, shell=True, timeout=1)
outputStr = outputBytes.decode("utf-8") # change utf-8 to your console encoding if necessary

Grep stops suddenly

I am trying to parse a file in Python, using grep but it always stops at the same line and I am enable to understand why. I tried three different ways :
process = os.popen("grep -A1 "+name+" "+new_hairpins+" | grep -oE '.{0,6}"+seq+".{0,6}'")
results = process.readlines()
process.close()
then
process = subprocess.Popen("grep -A1 "+name+" "+new_hairpins+" | grep -oE '.{0,6}"+seq+".{0,6}'",stdout=PIPE, stderr=PIPE, shell=True)
process.wait()
process_result = process.communicate()
results = filter(None, process_result[0].split("\n"))
and through a temp file
os.system("grep -A1 "+name+" "+new_hairpins+" | grep -oE '.{0,6}"+seq+".{0,6}' > tmp.txt")
with open("tmp.txt","r") as f :
results = f.readlines()
but the script always fails at the same line.
I manually tried this line directly in bash, and it worked....!
So could it be a memory issue from grep and how could I fix this problem ?
Thanks a lot !

You need to quote command line argument because there's a space in between:
"grep -A1 '"+name+" "+new_hairpins+"' | grep ....
^ ^
Otherwise, name, new_hairpins will be treated as separated arguments.

I finally found the problem : my fault !
I realized that the file new_hairpins I used in the grep and that I generated just before in the code wasn't closed with .close()....
As it was working on the 1879 first lines, I didn't think the problem could come from this file...
Anyway, thanks for your help guys!

Realize padding and rounding up values via awk out of a python script with subprocess

I use this command string to get the percentage value of the CPU utilization.
top -d 0.5 -b -n2 | grep 'Cpu(s)'|tail -n 1 | awk '{result = $2 + $4} END {printf "%3.0f\n", result'}
In the shell it works, but I need to execute it out of a python script.
This does not work because of the "%3.0f\n" part.
p = subprocess.Popen("top -d 0.5 -b -n2 | grep 'Cpu(s)'|tail -n 1 | awk '{result = $2 + $4} END {printf "%3.0f\n", result'}", stdout=subprocess.PIPE, shell=True)
How can I realize the padding to 3 characters and rounding up in this scenario?
I think the masking is the problem.
But maybe there is another way, which is better suited?
Thanks for any help!
Update: The solution was using tripple quotes:
p = subprocess.Popen('''top -d 0.5 -b -n2 | grep 'Cpu(s)'|tail -n 1 | awk '{result = $2 + $4} END {printf "%3.0f", result'}''', stdout=subprocess.PIPE, shell=True)

Either escape the " quotes inside your command string, i.e., replace them with \" or quote the command string with triple quotes `'''.
Using Jotne's version
cmd = '''top -d 0.5 -b -n2 | awk '/Cpu\(s\)/ {result=$2+$4} END {printf "%3.0f\n",result'}'''

In my opinion and as you suggested you may try other solutions, there already exists some tools which will fit your needs. For instance the psutil library.
Here is a very simple example which I think does what you want:
import psutil
print psutil.cpu_percent(interval=1)
If you still want to use your solution, you should read the python subprocess documentations about how to replace shell pipelines, and/or the existing stackoverflow topics about that (for instance this one)

Escaping escape sequence in Python

I am kind of new to python. Goal is to execute a shell command using subprocess parse & retrive the printed output from shell. The execution errors out as shown in the sample output msg below. Also shown below is the sample code snippet
Code snippet:
testStr = "cat tst.txt | grep Location | sed -e '/.*Location: //g' "
print "testStr = "+testStr
testStrOut = subprocess.Popen([testStr],shell=True,stdout=subprocess.PIPE).communicate()[0]
Output:
testStr = cat tst.txt | grep Location | sed -e '/.*Location: //g'
cat: tst.txt: No such file or directory
sed: -e expression #1, char 15: unknown command: `/'
Is there a workaround or a function that could be used ?
Appreciate your help
Thanks

I suppose your main error is not python related. To be more precise, there are 3 of them:
You forgot to import subprocess.
It should be sed -e 's/.*Location: //g'. You wrote ///g instead of s///g.
tst.txt does not exist.

You should be passing testStr directly as the first argument, rather than enclosing it in a list. See subprocess.Popen, the paragraph that starts "On Unix, with shell=True: ...".