The output of
ps uaxw | egrep 'kms' | grep -v 'grep'
yields:
user1 8148 0.0 0.0 128988 3916 pts/8 S+ 18:34 0:00 kms
user2 11782 0.7 0.3 653568 56564 pts/14 Sl+ 20:29 0:01 kms
Clearly two processes running the program. I want to store this number (2 here) as a variable. Any suggestions on how to do this in python?
I tried the following:
procs = subprocess.check_output("ps uaxw | egrep 'kmns' |grep -v 'grep'",shell=True)
But i get the following (I think when the jobs are not currently running, so number of processes running the jobs is zero):
Traceback (most recent call last): File "", line 1, in
File "/usr/lib64/python2.7/subprocess.py", line 573, in
check_output
raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command 'ps uaxw | egrep 'kmns' |grep
-v 'grep'' returned non-zero exit status 1
How do I get around this?
Btw, here is the function I wrote to detect if my system was busy (which means if the number of cpus > total installed, and if load avg > 0.9 per cpu):
def busy():
import subprocess
output = subprocess.check_output("uptime", shell=False)
words = output.split()
sys.stderr.write("%s\n"%(output))
procs = subprocess.check_output("ps uaxw | egrep '(kmns)' | grep -v 'grep'", shell=True)
kmns_wrds = procs.split("\n")
wrds=words[9]
ldavg=float(wrds.strip(','))+0.8
sys.stderr.write("%s %s\n"%(ldavg,len(kmns_wrds)))
return max(ldavg, len(kmns_wrds)) > ncpus
The above is called by:
def wait_til_free(myseconds):
while busy():
import time
import sys
time.sleep(myseconds)
""" sys.stderr.write("Waiting %s seconds\n"%(myseconds)) """
which basically tells the system to wait while all cpus are taken.
Any suggestions?
Many thanks!
If you're going to do this all with a big shell command, just add the -c argument to grep, so it gives you a count of lines instead of the actual lines:
$ ps uaxw |grep python |grep -v grep
abarnert 1028 0.0 0.3 2529488 55252 s000 S+ 9:46PM 0:02.80 /Library/Frameworks/Python.framework/Versions/3.4/Resources/Python.app/Contents/MacOS/Python /Library/Frameworks/Python.framework/Versions/3.4/bin/ipython3
abarnert 9639 0.0 0.1 2512928 19228 s002 T 3:06PM 0:00.40 /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /usr/local/bin/ipython2
$
$ ps uaxw |grep python |grep -c -v grep
2
Of course you could make this more complicated by adding a | wc -l to the end, or by counting the lines in Python, but why?
Alternatively, why even involve the shell? You can search within Python just as easily as you can run grep—and then you don't have the problem that you've accidentally created a grep process that ps will repeat as matching your search and then need to grep -v it back out:
procs = subprocess.check_output(['ps', 'uaxw']).splitlines()
kms_procs = [proc for proc in procs if 'kms' in proc]
count = len(kms_procs)
Or, even more simply, don't ask ps to give you a whole bunch of information that you don't want and then figure out how to ignore it, just ask for the information you want:
procs = subprocess.check_output(['ps', '-a', '-c', '-ocomm=']).splitlines()
count = procs.count('kms')
Or, even more more simplierly, install psutil and don't even try to run subprocesses and parse their output:
count = sum(1 for proc in psutil.process_iter() if proc.name() == 'kms')
If you want to simulate pipes you can use Popen:
p1 = Popen(["ps", "uaxw"], stdout=PIPE)
p2 = Popen(["grep", 'kms'], stdout=PIPE, stdin=p1.stdout)
p1.stdout.close()
out,_ = p2.communicate()
print(len(out.splitlines()))
Or use pgrep if it is available:
count = check_output(["pgrep", "-c", "kms"])
You may get different output from both as pgrep only gets the executable's names but so will ps -aux vs ps -a.
Related
I run vim inside gnome-terminal in a Python subprocess:
>>> import subprocess
>>> cmd=['gnome-terminal', '--', 'vim']
>>> p = subprocess.Popen(cmd)
It is possible to get the process ID for gnome-terminal with p.pid, but how can I get the process ID for vim from within the Python script?
Even though pstree in Bash shows vim as a child process of gnome-terminal, psutils does not list it.
>>> import psutil
>>> terminal_process = psutil.Process(p.pid)
>>> terminal_process.children()
[]
This behavior is caused by gnome-terminal.
If you type the command ps | grep <pid> inside your shell, you will see something similar to <pid> pts/0 00:00:00 gnome-terminal <defunct>.
A process being defunct means it has finished its task and is waiting to get killed (or is misbehaving which isn't the case here).
This means the process you launched from python has completed its job and is waiting for python to kill it.
Now if you look at pstree, you will see that another gnome-terminal process has been spawn at the root level. This means that the gnome-terminal process you launched in Python simply launched the "real terminal process" at the root level and exited.
If you further investigate and look for processes starting with gnome-terminal using ps aux | grep gnome-terminal, you will see output like :
root 5047 0.0 0.0 0 0 pts/0 Z 10:07 0:00 [gnome-terminal] <defunct>
root 5053 0.0 0.3 595276 30468 ? Sl 10:07 0:00 /usr/lib/gnome-terminal/gnome-terminal-server
root 7147 0.0 0.0 12780 972 pts/0 S+ 10:17 0:00 grep gnome-terminal
There is your now defunct process, and a new gnome-terminal-server process. gnome-terminal-server is the process you are looking for.
Long story short pgrep -f gnome-terminal-server will return the pid you want.
I think this works fine
import time
import subprocess
cmd=['gnome-terminal','--', 'vim']
p = subprocess.Popen(cmd)
time.sleep(10)
a = subprocess.Popen(['ps', '-eo', 'pid,ppid,command'], stdout = subprocess.PIPE)
b = subprocess.Popen(['grep', 'vim'], stdin = a.stdout, stdout = subprocess.PIPE)
output, error = b.communicate()
output = output.decode("utf-8").split('\n')
print(output)
The reason I used time.sleep(10) is because for some reason vim was not getting forked that fast so, I delayed it for 10 seconds.
Here we create 2 process for getting the ID of vim editor, we give the output of process a to b using stdout and stdin.
Then we use .communicate() to get stdout of process b into output.
Now our output is in form of bytes so we decode it to UTF-8 using .decode("utf-8") and then split on every new line.
It produces the output:
rahul#RNA-HP:~$ python3 so.py
# _g_io_module_get_default: Found default implementation gvfs (GDaemonVfs) for ‘gio-vfs’
# _g_io_module_get_default: Found default implementation dconf (DConfSettingsBackend) for ‘gsettings-backend’
# watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0)
# unwatch_fast: "/org/gnome/terminal/legacy/" (active: 0, establishing: 1)
# watch_established: "/org/gnome/terminal/legacy/" (establishing: 0)
['21325 21093 vim', '21330 21318 grep vim', '']
rahul#RNA-HP:~$
To verify this:
rahul#RNA-HP:~$ ps aux | grep gnome-terminal
rahul 21093 1.7 2.4 978172 45096 ? Ssl 19:55 0:02 /usr/lib/gnome-terminal/gnome-terminal-server
rahul 21374 0.0 0.0 8988 840 pts/0 S+ 19:57 0:00 grep --color=auto gnome-terminal
rahul#RNA-HP:~$ ps -eo pid,ppid,command | grep vim
21325 21093 vim
21376 21104 grep --color=auto vim
rahul#RNA-HP:~$
Here we can see that vim is forked from gnome-terminal 21093 is the id of gnome-terminal which is the ppid of vim.
Now, this happened if I didn't use time.sleep(10)
rahul#RNA-HP:~$ python3 so.py
['21407 21406 /usr/bin/python3 /usr/bin/gnome-terminal -- vim', '21409 21406 grep vim', '']
If we try to verify if those PID exist:
rahul#RNA-HP:~$ kill 21407
bash: kill: (21407) - No such process
rahul#RNA-HP:~$
Those ID dont exist for some reason.
If there are multiple instances of vim:
It produces:
['21416 21093 vim', '21736 21093 vim', '21738 21728 grep vim', '']
To get the latest instantiated vim's pid:
output = output[len(output) - 3]
Our output is sorted in ascending order of pid's and our last and second last values are and grep vim so we need the third last argument for getting the pid of vim.
Comment if something can be improved.
Here is my own Python only take, which works well so far. Any issues with this code?
import psutil, subprocess
cmd=['gnome-terminal', '--', 'vim']
editor_cmd=cmd[-1] # vim
proc = subprocess.Popen(cmd)
proc.wait()
# find the latest editor process in the list of all running processes
editor_processes = []
for p in psutil.process_iter():
try:
process_name = p.name()
if editor_cmd in process_name:
editor_processes.append((process_name, p.pid))
except:
pass
editor_proc = psutil.Process(editor_processes[-1][1])
print(editor_proc)
Here is a workaround. Name the vims by symbolic links, and find their pids:
import subprocess as sub,time,os,signal
N=5
vims= [ sub.Popen(f'ln -fs $(which vim) /dev/shm/vim{vn} && gnome-terminal -- /dev/shm/vim{vn} -c "s/$/Welcome to vim{vn}/"', shell=True) for vn in range(N) ]
time.sleep(1)
for vn in range(N):
# Get the pids of vims. Vim pid is not equal to proc.pid!
phelper= sub.Popen(f'ps -o pid= -C vim{vn}',shell=True, stdout=sub.PIPE, stderr=sub.PIPE)
try:
out,err= phelper.communicate(timeout=1)
vims[vn]= (vims[vn],int(out.decode(encoding="utf8"))) # proc_object --> (proc_object,vim pid)
except TimeoutExpired:
pass
phelper.kill()
# do something:
time.sleep(10)
for proc,vimpid in vims:
os.kill(vimpid,signal.SIGTERM)
I am trying to store the output of a cmd command as a variable in python.
To achieve this i am using os.system() but os.system() just runs the process,it doesn't capture the output.
import os
PlatformName = os.system("adb shell getprop | grep -e 'bt.name'")
DeviceName = os.system("adb shell getprop | grep -e '.product.brand'")
DeviceID = os.system("adb shell getprop | grep -e 'serialno'")
Version = os.system("adb shell getprop | grep -e 'version.release'")
print(PlatformName)
print(DeviceName)
print(DeviceID)
print(Version)
Then i tried to use the subprocess module.
import subprocess
import os
PlatformName = subprocess.check_output(["adb shell getprop | grep -e 'bt.name'"])
DeviceName = subprocess.check_output(["adb shell getprop | grep -e '.product.brand'"])
DeviceID = subprocess.check_output(["adb shell getprop | grep -e 'serialno'"])
Version = subprocess.check_output(["adb shell getprop | grep -e 'version.release'"])
print(PlatformName)
print(DeviceName)
print(DeviceID)
print(Version)
I am getting the following error
FileNotFoundError: [WinError 2] The system cannot find the file
specified
How can I store the output of the command as a variable?
The issues here:
passing arguments like this (string in a list, with spaces) is really not recommended
passing arguments like this need shell=True for it to have a slight chance to work, and shell=True is known for security issues (and other issues as well, like non-portability)
grep is not standard on windows, and the pattern is a regex which means you'd probably have to escape . ("bt\.name").
when not found grep returns 1 and would make check_output fail.
when found grep returns match(es), and a newline, that you'd have to strip
I'd rewrite this:
PlatformName = subprocess.check_output(["adb shell getprop | grep -e 'bt.name'"])
as:
output = subprocess.check_output(["adb","shell","getprop"])
platform_name = next((line for line in output.decode().splitlines() if "bt.name" in line),"")
The second line is a "native" version of grep (without regexes). It returns the first occurrence of "bt.line" in the output lines or empty string if not found.
You don't need grep here (the above is not strictly equivalent, as it yields the first occurrence, not all the occurrences but that should be okay on your case). And your clients may not have grep installed on Windows.
Hey I got the same problem as you. Sub-process can do what you want even with the shell=False. The trick is the communicate() method.
with subprocess.Popen(cmdCode,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
cwd = workingDir,
bufsize=1,
universal_newlines = True) as proc:
#output is stored in proc.stdout
#errors are stored in proc.stderr
Now you just need a little function to scan the proc.stdout for the information you need: PlatformName, etc
I want to convert the following shell evaluation to python2.6(can't upgrade). I can't figure out how to evaluate the output of the command.
Here's the shell version:
status=`$hastatus -sum |grep $hostname |grep Grp| awk '{print $6}'`
if [ $status != "ONLINE" ]; then
exit 1
fi
I tried os.popen and it returns ['ONLINE\n'].
value = os.popen("hastatus -sum |grep `hostname` |grep Grp| awk '{print $6}'".readlines()
print value
Try the subprocess module:
import subprocess
value = subprocess.call("hastatus -sum |grep `hostname` |grep Grp| awk '{print $6}'")
print(value)
Documentation is found here:
https://docs.python.org/2.6/library/subprocess.html?highlight=subprocess#module-subprocess
The recommended way is to use the subprocess module.
The following section of the documentation is instructive:
replacing shell pipeline
I report here for reference:
output=dmesg | grep hda
becomes:
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]
The p1.stdout.close() call after starting the p2 is important in order for p1 to receive a SIGPIPE if p2 exits before p1.
Alternatively, for trusted input, the shell’s own pipeline support may still be used directly:
output=dmesg | grep hda
becomes:
output=check_output("dmesg | grep hda", shell=True)
And here the recipe to translate os.popen to the subprocess module:
replacing os.popen()
So in your case you could do something like
import subprocess
output=check_output("hastatus -sum |grep `hostname` |grep Grp| awk '{print $6}'", shell=True)
or
concatenating the Popens as showed in the documentation above (probably what I would do).
Then to test the output you could just use, assuming you're using the first approach:
import sys
import subprocess
....
if 'ONLINE' in output:
sys.exit(1)
I know there are posts already on how to use subprocess in python to run linux commands but I just cant get the syntax correct for this one. please help. This is the command I need to run...
/sbin/ifconfig eth1 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'
Ok this is what I have at the moment that gives a syntax error...
import subprocess
self.ip = subprocess.Popen([/sbin/ifconfig eth1 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'])
Any help greatly appreciated.
This has been gone over many, many times before; but here is a simple pure Python replacement for the inefficient postprocessing.
from subprocess import Popen, PIPE
eth1 = subprocess.Popen(['/sbin/ifconfig', 'eth1'], stdout=PIPE)
out, err = eth1.communicate()
for line in out.split('\n'):
line = line.lstrip()
if line.startswith('inet addr:'):
ip = line.split()[1][5:]
Here's how to construct the pipe in Python (rather than reverting to Shell=True, which is more difficult to secure).
from subprocess import PIPE, Popen
# Do `which` to get correct paths
GREP_PATH = '/usr/bin/grep'
IFCONFIG_PATH = '/usr/bin/ifconfig'
AWK_PATH = '/usr/bin/awk'
awk2 = Popen([AWK_PATH, '{print $1}'], stdin=PIPE)
awk1 = Popen([AWK_PATH, '-F:', '{print $2}'], stdin=PIPE, stdout=awk2.stdin)
grep = Popen([GREP_PATH, 'inet addr'], stdin=PIPE, stdout=awk1.stdin)
ifconfig = Popen([IFCONFIG_PATH, 'eth1'], stdout=grep.stdin)
procs = [ifconfig, grep, awk1, awk2]
for proc in procs:
print(proc)
proc.wait()
It'd be better to do the string processing in Python using re. Do this to get the stdout of ifconfig.
from subprocess import check_output
stdout = check_output(['/usr/bin/ifconfig', 'eth1'])
print(stdout)
I am trying to automate the process of executing a command. When I this command:
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
Into a termianl I get the response:
%CPU PID USER COMMAND
5.7 25378 stackusr whttp
4.8 25656 stackusr tcpproxy
But when I execute this section of code I get an error regarding the format specifier:
if __name__ == '__main__':
fullcmd = ['ps','-eo','pcpu,pid,user,args | sort -k 1 -r | head -10']
print fullcmd
sshcmd = subprocess.Popen(fullcmd,
shell= False,
stdout= subprocess.PIPE,
stderr= subprocess.STDOUT)
out = sshcmd.communicate()[0].split('\n')
#print 'here'
for lin in out:
print lin
This is the error showen:
ERROR: Unknown user-defined format specifier "|".
********* simple selection ********* ********* selection by list *********
-A all processes -C by command name
-N negate selection -G by real group ID (supports names)
-a all w/ tty except session leaders -U by real user ID (supports names)
-d all except session leaders -g by session OR by effective group name
-e all processes -p by process ID
T all processes on this terminal -s processes in the sessions given
a all w/ tty, including other users -t by tty
g OBSOLETE -- DO NOT USE -u by effective user ID (supports names)
r only running processes U processes for specified users
x processes w/o controlling ttys t by tty
I have tryed placing a \ before the | but this has not effect.
You would need to use shell=True to use the pipe character, if you are going to go down that route then using check_output would be the simplest approach to get the output:
from subprocess import check_output
out = check_output("ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10",shell=True,stderr=STDOUT)
You can also simulate a pipe with Popen and shell=False, something like:
from subprocess import Popen, PIPE, STDOUT
sshcmd = Popen(['ps', '-eo', "pcpu,pid,user,args"],
stdout=PIPE,
stderr=STDOUT)
p2 = Popen(["sort", "-k", "1", "-r"], stdin=sshcmd.stdout, stdout=PIPE)
sshcmd.stdout.close()
p3 = Popen(["head", "-10"], stdin=p2.stdout, stdout=PIPE,stderr=STDOUT)
p2.stdout.close()
out, err = p3.communicate()