I have been having an issue whereby I can kill the processes that spawns the nodes but the nodes do not get killed. Does anyone have any suggest how I can do this?
Some of my latest failed attempts to accomplish this are:
node.terminate()
and
node.send_signal(signal.SIGINT)
below is the code:
from subprocess import Popen
import json
import sys
import os
import signal
import requests
FNULL = open(os.devnull, 'w')
json_data = open('nodes.json', 'r').read()
data = json.loads(json_data)
port = data['port']
# launch hub
hub = Popen('java -jar selenium-server-standalone-2.37.0.jar -role hub -port %s' % port, stdout=FNULL, stderr=FNULL, shell=True)
#launch nodes
nodes = []
for node in data['nodes']:
options = ''
if node['name'] == 'CHROME':
options += '-Dwebdriver.chrome.driver=../grid/chromedriver '
#options += ' -browser browserName='+node['name']+' maxInstances='+str(node['maxInstances'])
nodes.append(Popen('java -jar selenium-server-standalone-2.37.0.jar -role node -hub http://localhost:%i/grid/register %s' % (port, options), stdout=FNULL, stderr=FNULL, shell=True))
# wait for user input
print "type 'q' and ENTER to close the grid:"
while True:
line = sys.stdin.readline()
if line == 'q\n':
break
# close nodes
for node in nodes:
#node.terminate()
node.send_signal(signal.SIGINT)
# close hub
r = requests.get('http://localhost:'+str(port)+'/lifecycle-manager?action=shutdown')
As far as im aware, I'm basically forced to use shell=True, to get redirections to work
Processing the child's stdout/stderr in the parent python process is not an option, since I couldn't find functionality for doing it in a non-waiting way (and the parent python process must do other things while the child is running)
# close nodes
for node in nodes:
node.send_signal(signal.SIGINT)
node.terminate()
this seems to kill all the processes except for 1 of the nodes. Not always the same one
You could try using os.killpg. This function sends the signal to the process group, it should work if your processes do not change process group.
import os
import signal
os.killpg(os.getpgid(pid), signal.SIGINT)
Note, that process group will be changed if you are creating process under shell (bash, zsh, etc.), in that case more complicated technique should be used.
Related
I'm trying to integrate ESA'2 sen2cor python-script into my workflow. To do this I create a subprocess with which I call the "L2A_Process.bat" file, which in turn calls the "L2A_Process.py" script.
I want to launch the sen2cor-script with a timeout since it gets stuck and freezes from time to time, so as to automatically re-launch it if it failed.
To launch it and catch a timeout I successfully used the following script:
import os, subprocess
from signal import CTRL_BREAK_EVENT
timeout = 3600 #1hour
l1c_safe_path = "path/to/my/input/file.SAFE" #product that I want to process
command = ["L2A_process.bat", l1c_safe_path]
p = subprocess.Popen(command,shell=False, creationflags=subprocess.CREATE_NEW_PROCESS_GROUP)
try:
p.wait(timeout)
except subprocess.TimeoutExpired:
os.kill(p.pid, CTRL_BREAK_EVENT)
This is as far as I got. It results in the sen2cor-script being paused giving the following output:
Terminate batch job (Y/N)
I'd like to know how I can properly terminate my subprocess "p" with all it own child-subprocesses (i.e. "L2A_Process.py").
Some observations:
This script needs to run on Windows;
I've tried to kill the subprocess without the creationflag I've used in the example above: this results in the subprocess being killed but the "L2A_Process.py" script deteaches an keeps running (which is the core of my problem);
I cannot use a CTRL_C_EVENT since I want to re-launch the failed "L2A_Process.py" in a loop until it succeeds.
This code works for me to monitor Sen2cor status while converting L1C to L2A for Sentinel 2 data. The Sen2cor process is time and cpu consuming so be patient. It took half an hour to create L2A with DEM, DDV, etc. Hope it helps
from subprocess import Popen, PIPE
import os
pathtoprodS1C = "path_to_L1C_product" // safe file
outdirS2A = "output_dir" // where L2A files will be placed
pathtoL2Process = "path_to_L2A_Process" //if not in path
pathtoGIPP = "path_to_your_GIPP/L2A_GIPP.xml"
procName = "./L2A_Process"
os.chdir(pathtoL2Process)
import shlex
pcall = "./{} {} --output_dir {} --tif --GIP_L2A {}".format(procName,
pathtoprodS1C,
pathtoprodS2A,
pathtoGIPP)
args = shlex.split(pcall)
print(args)
try:
p = Popen(args, stdout=PIPE)
eut = p.stdout.readline()
while eut:
eut = p.stdout.readline()
print(eut)
finally:
print('Sen2Cor is Done')
exit()
I'm using the following script for connecting two compute nodes in a slurm cluster.
from getpass import getuser
from socket import gethostname
from pexpect import pxssh
import sys
python = sys.executable
worker_command = "%s -m worker" % python + " %i " + server_socket
pid = 0
children = []
for node, ntasks in node_list.items():
if node == gethostname():
continue
if node != gethostname():
pid_range = range(pid, pid + ntasks)
pid += ntasks
ssh = pxssh.pxssh()
ssh.login(node, getuser())
for worker in pid_range:
ssh.sendline(worker_command % worker + '&')
children.append(ssh)
node_list is a dictionary {'cn000': 28, 'cn001': 28}. worker is a python file placed in the working dictionary.
I expect ssh.sendline to be the same as pexpect.spawn. However, nothing happened after I ran the script.
Although an ssh session was built by ssh.login(node, getuser()), it seems the line ssh.sendline(worker_command % worker) has no effect, because the script to be run by worker_command is not run.
How can I fix this? Or should I try something else?
How can I create one socket on one compute node and connect it with a socket on another compute node?
There is missing a '%s' from the content of worker_command. It contains something like this: "/usr/bin/python3 -m worker" -> worker_command%worker should result in error.
If not (it is possible, because this source looks like a short part of the original program), then add ">>workerprocess.log 2>&1" string before '&', then try to run your program and take a look at workerprocess.log on the server! If your $HOME is writable on the server, you should find the error message(s) in it.
Okay I'm officially out of ideas after running each and every sample I could find on google up to 19th page. I have a "provider" script. The goal of this python script is to start up other services that run indefinitely even after this "provider" stopped running. Basically, start the process then forget about it but continue the script and not stopping it...
My problem: python-daemon... I have actions (web-service calls to start/stop/get status from the started services). I create the start commands on the fly and perform variable substitution on the config files as required.
Let's start from this point: I have a command to run (A bash script that executes a java process - a long running service that will be stopped sometime later).
def start(command, working_directory):
pidfile = os.path.join(working_directory, 'application.pid')
# I expect the pid of the started application to be here. The file is not created. Nothing is there.
context = daemon.DaemonContext(working_directory=working_directory,
pidfile=daemon.pidfile.PIDLockFile(pidfile))
with context:
psutil.Popen(command)
# This part never runs. Even if I put a simple print statement at this point, that never appears. Debugging in pycharms shows that my script returns with 0 on with context
with open(pidfile, 'r') as pf:
pid = pf.read()
return pid
From here on in my caller to this method I prepare a json object to return to the client which essentially contains an instance_id (don't mind it) and a pid (that'll be used to stop this process in another request.
What happens? After with context my application exits with status 0, nothing is returned, no json response gets created, no pidfile gets created only the executed psutil.Popen command runs. How can I achieve what I need? I need an independently running process and need to know its PID in order to stop it later on. The executed process must run even if the current python script stops for some reason. I can't get around the shell script as that application is not mine I have to use what I have.
Thanks for any tip!
#Edit:
I tried using simply the Popen from psutil/subprocess with somewhat more promising result.
def start(self, command):
import psutil/subprocess
proc = psutil.Popen(command)
return str(proc.pid)
Now If I debug the application and wait some undefined time on the return statement everything is working great! The service is running the pid is there, I can stop later on. Then I simply ran the provider without debugging. It returns the pid but the process is not running. Seems like Popen has no time to start the service because the whole provider stops faster.
#Update:
Using os.fork:
#staticmethod
def __start_process(command, working_directory):
pid = os.fork()
if pid == 0:
os.chdir(working_directory)
proc = psutil.Popen(command)
with open('application.pid', 'w') as pf:
pf.write(proc.pid)
def start(self):
...
__start_process(command, working_directory)
with open(os.path.join(working_directory, 'application.pid'), 'r') as pf:
pid = int(pf.read())
proc = psutil.Process(pid)
print("RUNNING" if proc.status() == psutil.STATUS_RUNNING else "...")
After running the above sample, RUNNING is written on console. After the main script exits because I'm not fast enough:
ps auxf | grep
No instances are running...
Checking the pidfile; sure it's there it was created
cat /application.pid
EMPTY 0bytes
From multiple partial tips i got, finally managed to get it working...
def start(command, working_directory):
pid = os.fork()
if pid == 0:
os.setsid()
os.umask(0) # I'm not sure about this, not on my notebook at the moment
os.execv(command[0], command) # This was strange as i needed to use the name of the shell script twice: command argv[0] [args]. Upon using ksh as command i got a nice error...
else:
with open(os.path.join(working_directory, 'application.pid'), 'w') as pf:
pf.write(str(pid))
return pid
That together solved the issue. The started process is not a child process of the running python script and won't stop when the script terminates.
Have you tried with os.fork()?
In a nutshell, os.fork() spawns a new process and returns the PID of that new process.
You could do something like this:
#!/usr/bin/env python
import os
import subprocess
import sys
import time
command = 'ls' # YOUR COMMAND
working_directory = '/etc' # YOUR WORKING DIRECTORY
def child(command, directory):
print "I'm the child process, will execute '%s' in '%s'" % (command, directory)
# Change working directory
os.chdir(directory)
# Execute command
cmd = subprocess.Popen(command
, shell=True
, stdout=subprocess.PIPE
, stderr=subprocess.PIPE
, stdin=subprocess.PIPE
)
# Retrieve output and error(s), if any
output = cmd.stdout.read() + cmd.stderr.read()
print output
# Exiting
print 'Child process ending now'
sys.exit(0)
def main():
print "I'm the main process"
pid = os.fork()
if pid == 0:
child(command, working_directory)
else:
print 'A subprocess was created with PID: %s' % pid
# Do stuff here ...
time.sleep(5)
print 'Main process ending now.'
sys.exit(0)
if __name__ == '__main__':
main()
Further info:
Documentation: https://docs.python.org/2/library/os.html#os.fork
Examples: http://www.python-course.eu/forking.php
Another related-question: Regarding The os.fork() Function In Python
I am writing a program which initiates a connection to a remote machine, then dynamically sending multiple commands to it by monitoring the response. Instead of using pexpect, what else can I use? I am trying to use subprocess.Popen, but communicate() method will kill the process.
Pexpect version: 2.4, http://www.bx.psu.edu/~nate/pexpect/pexpect.html
Referring to the API for subprocess in:
https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate
Popen.communicate(input=None)
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate. The optional input argument should be a string to be sent to the child process, or None, if no data should be sent to the child.
Thanks
Refer the subprocess documentation to understand the basics here
You could do something like this ...
Again, this is just a pointer... this approach may/may not be a best fit for your use case.
Explore -> and Test to find what works for you!
import shlex
import subprocess
import sys
class Command(object):
""" Generic Command Interface ."""
def execute(self, cmd):
proc = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE)
stdout_value = proc.communicate()[0]
exit_value = proc.poll()
if exit_value:
logger.error('Command execution failed. Command : %s' % cmd)
return exit_value, stdout_value
if __name__ == '__main__':
cmd = Command()
host = '' # HOSTNAME GOES HERE
cmd_str = '' # YOUR COMMAND GOES HERE
cmdline = 'ksh -c "ssh root#{0} "{1}""'.format(host, cmd_str)
exit_value, stdout_value = cmd.execute(cmdline)
if exit_value == 0:
# execute other command/s
# you basically use the same logic as above
else:
# return Or execute other command/s
I need to write a script in Linux which can start a background process using one command and stop the process using another.
The specific application is to take userspace and kernel logs for android.
following command should start taking logs
$ mylogscript start
following command should stop the logging
$ mylogscript stop
Also, the commands should not block the terminal. For example, once I send the start command, the script run in background and I should be able to do other work on terminal.
Any pointers on how to implement this in perl or python would be helpful.
EDIT:
Solved: https://stackoverflow.com/a/14596380/443889
I got the solution to my problem. Solution essentially includes starting a subprocess in python and sending a signal to kill the process when done.
Here is the code for reference:
#!/usr/bin/python
import subprocess
import sys
import os
import signal
U_LOG_FILE_PATH = "u.log"
K_LOG_FILE_PATH = "k.log"
U_COMMAND = "adb logcat > " + U_LOG_FILE_PATH
K_COMMAND = "adb shell cat /proc/kmsg > " + K_LOG_FILE_PATH
LOG_PID_PATH="log-pid"
def start_log():
if(os.path.isfile(LOG_PID_PATH) == True):
print "log process already started, found file: ", LOG_PID_PATH
return
file = open(LOG_PID_PATH, "w")
print "starting log process: ", U_COMMAND
proc = subprocess.Popen(U_COMMAND,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
print "log process1 id = ", proc.pid
file.write(str(proc.pid) + "\n")
print "starting log process: ", K_COMMAND
proc = subprocess.Popen(K_COMMAND,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
print "log process2 id = ", proc.pid
file.write(str(proc.pid) + "\n")
file.close()
def stop_log():
if(os.path.isfile(LOG_PID_PATH) != True):
print "log process not started, can not find file: ", LOG_PID_PATH
return
print "terminating log processes"
file = open(LOG_PID_PATH, "r")
log_pid1 = int(file.readline())
log_pid2 = int(file.readline())
file.close()
print "log-pid1 = ", log_pid1
print "log-pid2 = ", log_pid2
os.killpg(log_pid1, signal.SIGTERM)
print "logprocess1 killed"
os.killpg(log_pid2, signal.SIGTERM)
print "logprocess2 killed"
subprocess.call("rm " + LOG_PID_PATH, shell=True)
def print_usage(str):
print "usage: ", str, "[start|stop]"
# Main script
if(len(sys.argv) != 2):
print_usage(sys.argv[0])
sys.exit(1)
if(sys.argv[1] == "start"):
start_log()
elif(sys.argv[1] == "stop"):
stop_log()
else:
print_usage(sys.argv[0])
sys.exit(1)
sys.exit(0)
There are a couple of different approaches you can take on this:
1. Signal - you use a signal handler, and use, typically "SIGHUP" to signal the process to restart ("start"), SIGTERM to stop it ("stop").
2. Use a named pipe or other IPC mechanism. The background process has a separate thread that simply reads from the pipe, and when something comes in, acts on it. This method relies on having a separate executable file that opens the pipe, and sends messages ("start", "stop", "set loglevel 1" or whatever you fancy).
I'm sorry, I haven't implemented either of these in Python [and perl I haven't really written anything in], but I doubt it's very hard - there's bound to be a ready-made set of python code to deal with named pipes.
Edit: Another method that just struck me is that you simply daemonise the program at start, and then let the "stop" version find your deamonized process [e.g. by reading the "pidfile" that you stashed somewhere suitable], and then sends a SIGTERM for it to terminate.
I don't know if this is the optimum way to do it in perl, but for example:
system("sleep 60 &")
This starts a background process that will sleep for 60 seconds without blocking the terminal. The ampersand in shell means to do something in the background.
A simple mechanism for telling the process when to stop is to have it periodically check for the existence of a certain file. If the file exists, it exits.