I run vim inside gnome-terminal in a Python subprocess:
>>> import subprocess
>>> cmd=['gnome-terminal', '--', 'vim']
>>> p = subprocess.Popen(cmd)
It is possible to get the process ID for gnome-terminal with p.pid, but how can I get the process ID for vim from within the Python script?
Even though pstree in Bash shows vim as a child process of gnome-terminal, psutils does not list it.
>>> import psutil
>>> terminal_process = psutil.Process(p.pid)
>>> terminal_process.children()
[]
This behavior is caused by gnome-terminal.
If you type the command ps | grep <pid> inside your shell, you will see something similar to <pid> pts/0 00:00:00 gnome-terminal <defunct>.
A process being defunct means it has finished its task and is waiting to get killed (or is misbehaving which isn't the case here).
This means the process you launched from python has completed its job and is waiting for python to kill it.
Now if you look at pstree, you will see that another gnome-terminal process has been spawn at the root level. This means that the gnome-terminal process you launched in Python simply launched the "real terminal process" at the root level and exited.
If you further investigate and look for processes starting with gnome-terminal using ps aux | grep gnome-terminal, you will see output like :
root 5047 0.0 0.0 0 0 pts/0 Z 10:07 0:00 [gnome-terminal] <defunct>
root 5053 0.0 0.3 595276 30468 ? Sl 10:07 0:00 /usr/lib/gnome-terminal/gnome-terminal-server
root 7147 0.0 0.0 12780 972 pts/0 S+ 10:17 0:00 grep gnome-terminal
There is your now defunct process, and a new gnome-terminal-server process. gnome-terminal-server is the process you are looking for.
Long story short pgrep -f gnome-terminal-server will return the pid you want.
I think this works fine
import time
import subprocess
cmd=['gnome-terminal','--', 'vim']
p = subprocess.Popen(cmd)
time.sleep(10)
a = subprocess.Popen(['ps', '-eo', 'pid,ppid,command'], stdout = subprocess.PIPE)
b = subprocess.Popen(['grep', 'vim'], stdin = a.stdout, stdout = subprocess.PIPE)
output, error = b.communicate()
output = output.decode("utf-8").split('\n')
print(output)
The reason I used time.sleep(10) is because for some reason vim was not getting forked that fast so, I delayed it for 10 seconds.
Here we create 2 process for getting the ID of vim editor, we give the output of process a to b using stdout and stdin.
Then we use .communicate() to get stdout of process b into output.
Now our output is in form of bytes so we decode it to UTF-8 using .decode("utf-8") and then split on every new line.
It produces the output:
rahul#RNA-HP:~$ python3 so.py
# _g_io_module_get_default: Found default implementation gvfs (GDaemonVfs) for ‘gio-vfs’
# _g_io_module_get_default: Found default implementation dconf (DConfSettingsBackend) for ‘gsettings-backend’
# watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0)
# unwatch_fast: "/org/gnome/terminal/legacy/" (active: 0, establishing: 1)
# watch_established: "/org/gnome/terminal/legacy/" (establishing: 0)
['21325 21093 vim', '21330 21318 grep vim', '']
rahul#RNA-HP:~$
To verify this:
rahul#RNA-HP:~$ ps aux | grep gnome-terminal
rahul 21093 1.7 2.4 978172 45096 ? Ssl 19:55 0:02 /usr/lib/gnome-terminal/gnome-terminal-server
rahul 21374 0.0 0.0 8988 840 pts/0 S+ 19:57 0:00 grep --color=auto gnome-terminal
rahul#RNA-HP:~$ ps -eo pid,ppid,command | grep vim
21325 21093 vim
21376 21104 grep --color=auto vim
rahul#RNA-HP:~$
Here we can see that vim is forked from gnome-terminal 21093 is the id of gnome-terminal which is the ppid of vim.
Now, this happened if I didn't use time.sleep(10)
rahul#RNA-HP:~$ python3 so.py
['21407 21406 /usr/bin/python3 /usr/bin/gnome-terminal -- vim', '21409 21406 grep vim', '']
If we try to verify if those PID exist:
rahul#RNA-HP:~$ kill 21407
bash: kill: (21407) - No such process
rahul#RNA-HP:~$
Those ID dont exist for some reason.
If there are multiple instances of vim:
It produces:
['21416 21093 vim', '21736 21093 vim', '21738 21728 grep vim', '']
To get the latest instantiated vim's pid:
output = output[len(output) - 3]
Our output is sorted in ascending order of pid's and our last and second last values are and grep vim so we need the third last argument for getting the pid of vim.
Comment if something can be improved.
Here is my own Python only take, which works well so far. Any issues with this code?
import psutil, subprocess
cmd=['gnome-terminal', '--', 'vim']
editor_cmd=cmd[-1] # vim
proc = subprocess.Popen(cmd)
proc.wait()
# find the latest editor process in the list of all running processes
editor_processes = []
for p in psutil.process_iter():
try:
process_name = p.name()
if editor_cmd in process_name:
editor_processes.append((process_name, p.pid))
except:
pass
editor_proc = psutil.Process(editor_processes[-1][1])
print(editor_proc)
Here is a workaround. Name the vims by symbolic links, and find their pids:
import subprocess as sub,time,os,signal
N=5
vims= [ sub.Popen(f'ln -fs $(which vim) /dev/shm/vim{vn} && gnome-terminal -- /dev/shm/vim{vn} -c "s/$/Welcome to vim{vn}/"', shell=True) for vn in range(N) ]
time.sleep(1)
for vn in range(N):
# Get the pids of vims. Vim pid is not equal to proc.pid!
phelper= sub.Popen(f'ps -o pid= -C vim{vn}',shell=True, stdout=sub.PIPE, stderr=sub.PIPE)
try:
out,err= phelper.communicate(timeout=1)
vims[vn]= (vims[vn],int(out.decode(encoding="utf8"))) # proc_object --> (proc_object,vim pid)
except TimeoutExpired:
pass
phelper.kill()
# do something:
time.sleep(10)
for proc,vimpid in vims:
os.kill(vimpid,signal.SIGTERM)
Related
I need to write a piece of code that will check if another instance of a python script with the same name is running, kill it and all its children and do the job of the script. On my way to solution I have this code:
import os, sys
import time
import subprocess
from subprocess import PIPE,Popen
import os
import signal
if sys.argv[1] == 'boss':
print 'I am boss', \
"pid:", os.getpid(), \
"pgid:", os.getgid()
# kill previous process with same name
my_pid=os.getpid()
pgrep = Popen(['pgrep', '-f', 'subp_test.py'], stdout=PIPE)
prev_pids=pgrep.communicate()[0].split()
print 'previous processes:' , prev_pids
for pid in prev_pids:
if int(pid) !=int(my_pid):
print 'killing', pid
os.kill(int(pid), signal.SIGKILL)
# do the job
subprocess.call('python subp_test.py 1', shell=True)
subprocess.call('python subp_test.py 2', shell=True)
subprocess.call('python some_other_script.py', shell=True)
else:
p_num = sys.argv[1]
for i in range(20):
time.sleep(1)
print 'child', p_num, \
"pid:", os.getpid(), \
"pgid:", os.getgid(), \
":", i
This will kill all processes that have the substring 'subp_test.py' in its command but will not kill some_other_script.py or other programs without the 'subp_test.py' in it.
The calls the script subp_test.py will execute are unexpected but as I understand, they are supposed to be under it in the process tree.
So how do I access all the children of subp_test.py in order to kill them when a new instance of subp_test.py begins to run?
Also, are there better approaches to implement this logic?
I use Python 2.6.5 on Ubuntu 10.04.
Run your program under a new session group and write the session group id to a file. When your program starts, kill -SIGNAL -prevsessiongroup. This will kill all processes and their children etc unless one of them explicitly changes the session group. I have included urls which contain snippets of code you can use.
https://docs.python.org/2/library/os.html#os.setsid
http://web.archive.org/web/20131017130434/http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
you can use this command to kill all resources allocated to the python script:
kill -9 `ps -ef | grep your_python_script.py | grep -v grep | awk '{print $2}'`
I'm observing the following script hang under a particular scenario described below:
#!/usr/bin/env python
import os
import subprocess as sp
import time
tres = 0.1
time.sleep(2)
user_at_host = "user#host"
child_proc = sp.Popen(["ssh", "-q", user_at_host, "ls"], preexec_fn=os.setpgrp)
while(child_proc.poll() is None):
time.sleep(tres)
When I run the script with a valid "user#host", I normally just get the output from "ls", but if I push the Enter/Return key before the "sleep(2)" ends, it hangs (with no output) when we get to the "ls" sub-process and I have to interrupt it.
ps aux | grep ls
user 14216 0.0 0.0 6844 2856 pts/27 T 18:18 0:00 ssh -q user#host ls
As you can see it's in the "stopped" state.
If I don't use "preexec_fn=os.setpgrp", there is no hang.
Also, if I replace the remote ssh/ls call with a local ls call, there is no hang.
Any thoughts on why this is happening?
EDIT:
Authentication of ssh is performed via public key.
There is no output when the process hangs/stops.
I'm using python 2.4.3, but have observed the problem in 2.6.6 as well.
I am trying to automate the process of executing a command. When I this command:
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
Into a termianl I get the response:
%CPU PID USER COMMAND
5.7 25378 stackusr whttp
4.8 25656 stackusr tcpproxy
But when I execute this section of code I get an error regarding the format specifier:
if __name__ == '__main__':
fullcmd = ['ps','-eo','pcpu,pid,user,args | sort -k 1 -r | head -10']
print fullcmd
sshcmd = subprocess.Popen(fullcmd,
shell= False,
stdout= subprocess.PIPE,
stderr= subprocess.STDOUT)
out = sshcmd.communicate()[0].split('\n')
#print 'here'
for lin in out:
print lin
This is the error showen:
ERROR: Unknown user-defined format specifier "|".
********* simple selection ********* ********* selection by list *********
-A all processes -C by command name
-N negate selection -G by real group ID (supports names)
-a all w/ tty except session leaders -U by real user ID (supports names)
-d all except session leaders -g by session OR by effective group name
-e all processes -p by process ID
T all processes on this terminal -s processes in the sessions given
a all w/ tty, including other users -t by tty
g OBSOLETE -- DO NOT USE -u by effective user ID (supports names)
r only running processes U processes for specified users
x processes w/o controlling ttys t by tty
I have tryed placing a \ before the | but this has not effect.
You would need to use shell=True to use the pipe character, if you are going to go down that route then using check_output would be the simplest approach to get the output:
from subprocess import check_output
out = check_output("ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10",shell=True,stderr=STDOUT)
You can also simulate a pipe with Popen and shell=False, something like:
from subprocess import Popen, PIPE, STDOUT
sshcmd = Popen(['ps', '-eo', "pcpu,pid,user,args"],
stdout=PIPE,
stderr=STDOUT)
p2 = Popen(["sort", "-k", "1", "-r"], stdin=sshcmd.stdout, stdout=PIPE)
sshcmd.stdout.close()
p3 = Popen(["head", "-10"], stdin=p2.stdout, stdout=PIPE,stderr=STDOUT)
p2.stdout.close()
out, err = p3.communicate()
The output of
ps uaxw | egrep 'kms' | grep -v 'grep'
yields:
user1 8148 0.0 0.0 128988 3916 pts/8 S+ 18:34 0:00 kms
user2 11782 0.7 0.3 653568 56564 pts/14 Sl+ 20:29 0:01 kms
Clearly two processes running the program. I want to store this number (2 here) as a variable. Any suggestions on how to do this in python?
I tried the following:
procs = subprocess.check_output("ps uaxw | egrep 'kmns' |grep -v 'grep'",shell=True)
But i get the following (I think when the jobs are not currently running, so number of processes running the jobs is zero):
Traceback (most recent call last): File "", line 1, in
File "/usr/lib64/python2.7/subprocess.py", line 573, in
check_output
raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command 'ps uaxw | egrep 'kmns' |grep
-v 'grep'' returned non-zero exit status 1
How do I get around this?
Btw, here is the function I wrote to detect if my system was busy (which means if the number of cpus > total installed, and if load avg > 0.9 per cpu):
def busy():
import subprocess
output = subprocess.check_output("uptime", shell=False)
words = output.split()
sys.stderr.write("%s\n"%(output))
procs = subprocess.check_output("ps uaxw | egrep '(kmns)' | grep -v 'grep'", shell=True)
kmns_wrds = procs.split("\n")
wrds=words[9]
ldavg=float(wrds.strip(','))+0.8
sys.stderr.write("%s %s\n"%(ldavg,len(kmns_wrds)))
return max(ldavg, len(kmns_wrds)) > ncpus
The above is called by:
def wait_til_free(myseconds):
while busy():
import time
import sys
time.sleep(myseconds)
""" sys.stderr.write("Waiting %s seconds\n"%(myseconds)) """
which basically tells the system to wait while all cpus are taken.
Any suggestions?
Many thanks!
If you're going to do this all with a big shell command, just add the -c argument to grep, so it gives you a count of lines instead of the actual lines:
$ ps uaxw |grep python |grep -v grep
abarnert 1028 0.0 0.3 2529488 55252 s000 S+ 9:46PM 0:02.80 /Library/Frameworks/Python.framework/Versions/3.4/Resources/Python.app/Contents/MacOS/Python /Library/Frameworks/Python.framework/Versions/3.4/bin/ipython3
abarnert 9639 0.0 0.1 2512928 19228 s002 T 3:06PM 0:00.40 /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /usr/local/bin/ipython2
$
$ ps uaxw |grep python |grep -c -v grep
2
Of course you could make this more complicated by adding a | wc -l to the end, or by counting the lines in Python, but why?
Alternatively, why even involve the shell? You can search within Python just as easily as you can run grep—and then you don't have the problem that you've accidentally created a grep process that ps will repeat as matching your search and then need to grep -v it back out:
procs = subprocess.check_output(['ps', 'uaxw']).splitlines()
kms_procs = [proc for proc in procs if 'kms' in proc]
count = len(kms_procs)
Or, even more simply, don't ask ps to give you a whole bunch of information that you don't want and then figure out how to ignore it, just ask for the information you want:
procs = subprocess.check_output(['ps', '-a', '-c', '-ocomm=']).splitlines()
count = procs.count('kms')
Or, even more more simplierly, install psutil and don't even try to run subprocesses and parse their output:
count = sum(1 for proc in psutil.process_iter() if proc.name() == 'kms')
If you want to simulate pipes you can use Popen:
p1 = Popen(["ps", "uaxw"], stdout=PIPE)
p2 = Popen(["grep", 'kms'], stdout=PIPE, stdin=p1.stdout)
p1.stdout.close()
out,_ = p2.communicate()
print(len(out.splitlines()))
Or use pgrep if it is available:
count = check_output(["pgrep", "-c", "kms"])
You may get different output from both as pgrep only gets the executable's names but so will ps -aux vs ps -a.
how to perform
echo xyz | ssh [host]
(send xyz to host)
with python?
I have tried pexpect
pexpect.spawn('echo xyz | ssh [host]')
but it's performing
echo 'xyz | ssh [host]'
maybe other package will be better?
http://pexpect.sourceforge.net/pexpect.html#spawn
Gives an example of running a command with a pipe :
shell_cmd = 'ls -l | grep LOG > log_list.txt'
child = pexpect.spawn('/bin/bash', ['-c', shell_cmd])
child.expect(pexpect.EOF)
Previous incorrect attempt deleted to make sure no-one is confused by it.
You don't need pexpect to simulate a simple shell pipeline. The simplest way to emulate the pipeline is the os.system function:
os.system("echo xyz | ssh [host]")
A more Pythonic approach is to use the subprocess module:
p = subprocess.Popen(["ssh", "host"],
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
p.stdin.write("xyz\n")
output = p.communicate()[0]