Develop Reference

Python Script that takes in Linux Commands to return Server Health

For my job, I'm developing a small script that users can run to essentially do a check of log files for errors. I'm familiar with both Python and cmd prompts but not running one inside the other.
I read around quite a bit but can't really find the best process for this. Many seem a bit more complex for my intentions.
Ideally what I want to build is a program that follows this process:
For all dirs in host:
cd into directory then grep log file for a certain string
-print errors
-return back to a dir
cd into directory:
df -h
My personal preference would be to perform like so:
def myFirstCheck():
file_result = cat a/b/c/LogFile.log | awk /ERROR\|/FATAL\
file0_result = cat a/b/c/LogFile2.log | awk /ERROR\|/FATAL\
return file_result, file0_result
def mySecondCheck():
print('Server 2 check:')
file2_result = cat d/e/f/LogFile3.log | awk /ERROR\|/FATAL\
file3_result = cat d/e/f/LogFile4.log | awk /ERROR\|/FATAL\
return file2_result, file3_result
file_result, file0_result = myFirstCheck()
print('Server 1 check:')
print('df -h') #I want this to return the output from cmd 'df -h'
print(file_result)
print(file0_result)
file2_result, file3_result = mySecondCheck()
print('Server 2 check:')
print('df -h') #I want this to return the output from cmd 'df -h'
print(file2_result)
print(file3_result)
#exit
I know this is widely inefficient and may be a simple way of thinking for something that may be a bit more complex. I just want to see if there's any helpful discussion from those with previous experience trying to do the similar thing.

Looks like you try to do alot of things with the cmd, but you can do it pythonicly.
you can read file like this:
with open("/path/to/log/file", "r") as f:
data = f.read()
error_lines = [line for line in data.splitlines() if "ERROR" in line]
secondly, for the df -h, you can simple do it as:
import subprocess
subprocess.check_output("df -h")
or if you want to do it pythonic and know the disks path you can do it as:
import psutil
hdd = psutil.disk_usage('/')

Redirecting Output From a Program to a File with Python: Specific Bug

I've been trying to run a Java program and capture it's STDOUT output to a file from the Python script. The idea is to run test files through my program and check if it matches the answers.
Per this and this SO questions, using subprocess.call is the way to go. In the code below, I am doing subprocess.call(command, stdout=f) where f is the file I opened.
The resulted file is empty and I can't quite understand why.
import glob
test_path = '/path/to/my/testfiles/'
class_path = '/path/to/classfiles/'
jar_path = '/path/to/external_jar/'
test_pattern = 'test_case*'
temp_file = 'res'
tests = glob.glob(test_path + test_pattern) # find all test files
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# cd into directory where the class files are and run the program
command = 'cd {p} ; java -cp {cp} package.MyProgram {tc_p}'
.format(p=class_path,
cp=jar_path,
tc_p=test_path + tc)
# execute the command and direct all STDOUT to file
subprocess.call(command.split(), stdout=f, stderr=subprocess.STDOUT)
# diff is just a lambda func that uses os.system('diff')
exec_code = diff(answers[i], test_path + temp_file)
if exec_code == BAD:
scream(':(')

I checked the docs for subprocess and they recommended using subprocess.run (added in Python 3.5). The run method returns the instance of CompletedProcess, which has a stdout field. I inspected it and the stdout was an empty string. This explained why the file f I tried to create was empty.
Even though the exit code was 0 (success) from the subprocess.call, it didn't mean that my Java program actually got executed. I ended up fixing this bug by breaking down command into two parts.
If you notice, I initially tried to cd into correct directory and then execute the Java file -- all in one command. I ended up removing cd from command and did the os.chdir(class_path) instead. The command now contained only the string to run the Java program. This did the trick.
So, the code looked like this:
good_code = 0
# Assume the same variables defined as in the original question
os.chdir(class_path) # get into the class files directory first
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# run the program
command = 'java -cp {cp} package.MyProgram {tc_p}'
.format(cp=jar_path,
tc_p=test_path + tc)
# runs the command and redirects it into the file f
# stores the instance of CompletedProcess
out = subprocess.run(command.split(), stdout=f)
# you can access useful info now
assert out.returncode == good_code

Print file contents using cat in subprocess

I am using a subprocess call in python where I have to print a file contents using cat. The file name is a variable that I generate in the python code itself. This is my code:
pid = str(os.getpid())
tmp_file_path = "/tmp/" + pid + "/data_to_synnet"
synnet_output = subprocess.check_output(["cat echo '%s'"%tmp_file_path], shell=True)
The above code throws an error saying cat: echo: No such file or directory.
However, when I use only subprocess.check_output(["echo '%s'"%tmp_file_path], shell=True), the variable name is printed correctly.
Also, I tried doing this (cat echo $tmp_file_name) in the command line and it works. Can someone please tell what is wrong?

The command you want is this:
"cat '%s'"%tmp_file_path
Just get rid of the "echo" word.
Alternatively,
synnet_output = subprocess.check_output(["cat", tmp_file_path], shell=False)

shell multipipe broken with multiple python scripts

I am trying to get the stdout of a python script to be shell-piped in as stdin to another python script like so:
find ~/test -name "*.txt" | python my_multitail.py | python line_parser.py
It should print an output but nothing comes out of it.
Please note that this works:
find ~/test -name "*.txt" | python my_multitail.py | cat
And this works too:
echo "bla" | python line_parser.py
my_multitail.py prints out the new content of the .txt files:
from multitail import multitail
import sys
filenames = sys.stdin.readlines()
# we get rid of the trailing '\n'
for index, filename in enumerate(filenames):
filenames[index] = filename.rstrip('\n')
for fn, line in multitail(filenames):
print '%s: %s' % (fn, line),
sys.stdout.flush()
When a new line is added to the .txt file ("hehe") then my_multitail.py prints:
/home/me/test2.txt: hehe
line_parser.py simply prints out what it gets on stdin:
import sys
for line in sys.stdin:
print "line=", line
There is something I must be missing. Please community help me :)

There's a hint if you run your line_parser.py interactively:
$ python line_parser.py
a
b
c
line= a
line= b
line= c
Note that I hit ctrl+D to provoke an EOF after entering the 'c'. You can see that it's slurping up all the input before it starts iterating over the lines. Since this is a pipeline and you're continuously sending output through to it, this doesn't happen and it never starts processing. You'll need to choose a different way of iterating over stdin, for example:
import sys
line = sys.stdin.readline()
while line:
print "line=", line
line = sys.stdin.readline()

Why I got "sh: 1: Syntax error: Unterminated quoted string" when I run my Python program?

I want to count how many lines of code I have written.
Here is the Python code:
import os
import sys
EXT = ['.c','.cpp','.java','.py']
def main():
l = []
if os.path.isdir(sys.argv[1]):
for root, dirs, files in os.walk(sys.argv[1]):
l.extend([os.path.join(root, name) for name in files])
else:
l.append(sys.argv[1])
params = ["'"+p+"'" for p in l if os.path.splitext(p)[1] in EXT]
result = os.popen("wc -l %s "%" ".join(params)).read()
print result
if __name__ == '__main__':
main()
Before this, it was running as expected. But today, it give me this error:
sh: 1: Syntax error: Unterminated quoted string
I don't know what happened.

Your Python script is missing a shebang line. Add the following to the top of your file:
#!/usr/bin/env python
Then you should be able to run the following, assuming your script is at /path/to/your_script.py and it has the executable bit set:
/path/to/your_script.py arg1 arg2 [...]
Alternatively:
python /path/to/your_script.py arg1 arg2 [...]
Update following comments
I suspect what has changed is that a source file containing a ' in its name has been added to the directory you are checking and the shell is choking on this.
You could add the following function to your program:
def shellquote(s):
return "'" + s.replace("'", "'\\''") + "'"
[Lifted from Greg Hewgill's answer to How to escape os.system() calls in Python? .]
And call it like this:
params = [shellquote(p) for p in l if os.path.splitext(p)[1] in EXT]

#Johnsyweb's updated answer seems to have the correct diagnostic, but the correct fix is to not use a shell to invoke wc. Try something like this instead:
cmd = ['/bin/wc', '-l'] # Need full path!
[cmd.extend(p) for p in l if os.path.splitext(p)[1] in EXT]
result = os.popen2(cmd).read()
Note that the subprocess module is the recommended solution now. Switching to that requires a less intrusive change to your current code, though; see http://docs.python.org/2/library/subprocess.html#replacing-os-popen-os-popen2-os-popen3

Looks like your Python program was parsed like a shell script. Add something like this at the header to indicate where your Python is:
#!/usr/bin/python
or you just run python a.py.