I am trying to write a python program to test a server written in C. The python program launches the compiled server using the subprocess module:
pid = subprocess.Popen(args.server_file_path).pid
This works fine, however if the python program terminates unexpectedly due to an error, the spawned process is left running. I need a way to ensure that if the python program exits unexpectedly, the server process is killed as well.
Some more details:
Linux or OSX operating systems only
Server code can not be modified in any way
I would atexit.register a function to terminate the process:
import atexit
process = subprocess.Popen(args.server_file_path)
atexit.register(process.terminate)
pid = process.pid
Or maybe:
import atexit
process = subprocess.Popen(args.server_file_path)
#atexit.register
def kill_process():
try:
process.terminate()
except OSError:
pass #ignore the error. The OSError doesn't seem to be documented(?)
#as such, it *might* be better to process.poll() and check for
#`None` (meaning the process is still running), but that
#introduces a race condition. I'm not sure which is better,
#hopefully someone that knows more about this than I do can
#comment.
pid = process.pid
Note that this doesn't help you if you do something nasty to cause python to die in a non-graceful way (e.g. via os._exit or if you cause a SegmentationFault or BusError)
Related
I am scripting GDB with Python 2.7.
I am simply stepping instructions with gdb.execute("stepi"). If the debugged program is idling and waiting for user interaction, gdb.execute("stepi") doesn't return. If there is such a situation, I want to stop the debugging session without terminating gdb.
To do so, I create a thread that will kill the debugged process if the current instruction ran for more than x seconds:
from ctypes import c_ulonglong, c_bool
from os import kill
from threading import Thread
from time import sleep
import signal
# We need mutable primitives in order to update them in the thread
it = c_ulonglong(0) # Instructions counter
program_exited = c_bool(False)
t = Thread(target=check_for_idle, args=(pid,it,program_exited))
t.start()
while not program_exited.value:
gdb.execute("si") # Step instruction
it.value += 1
# Threaded function that will kill the loaded program if it's idling
def check_for_idle(pid, it, program_exited):
delta_max = 0.1 # Max delay between 2 instructions, seconds
while not program_exited.value:
it_prev = c_ulonglong(it.value) # Previous value of instructions counter
sleep(delta_max)
# If previous instruction lasted for more than 'delta_max', kill debugged process
if (it_prev.value == it.value):
# Process pid has been retrieved before
kill(pid, signal.SIGTERM)
program_exited.value = True
print("idle_process_end")
However, gdb.execute is pausing my thread... Is there another way to kill the debugged process if it is idling?
However, gdb.execute is pausing my thread
What is happening here is that gdb.execute does not release Python's global lock when calling into gdb. So, while the gdb command executes, other Python threads are stuck.
This is just an oversight in gdb. I've filed a bug for it.
Is there another way to kill the debugged process if it is idling?
There is one other technique you can try -- I am not certain it will work. Unfortunately this part of gdb is not fully fleshed out (at the present moment); so also feel free to file bug reports.
The main idea is to run gdb commands on the main thread -- but not from Python. So, try writing your stepping loop using the gdb CLI, maybe like:
(gdb) while 1
> stepi
> end
Then your thread should be able to kill the inferior. Another approach might be for your thread to inject a gdb command into the main loop using gdb.post_event.
I have a script that is supposed to run 24/7 unless interrupted. This script is script A.
I want script A to call Script B, and have script A exit while B is running. Is this possible?
This is what I thought would work
#script_A.py
while(1)
do some stuff
do even more stuff
if true:
os.system("python script_B.py")
sys.exit(0)
#script_B.py
time.sleep(some_time)
do something
os.system("python script_A.py")
sys.exit(0)
But it seems as if A doesn't actually exit until B has finished executing (which is not what I want to happen).
Is there another way to do this?
What you are describing sounds a lot like a function call:
def doScriptB():
# do some stuff
# do some more stuff
def doScriptA():
while True:
# do some stuff
if Your Condition:
doScriptB()
return
while True:
doScriptA()
If this is insufficient for you, then you have to detach the process from you python process. This normally involves spawning the process in the background, which is done by appending an ampersand to the command in bash:
yes 'This is a background process' &
And detaching said process from the current shell, which, in a simple C program is done by forking the process twice. I don't know how to do this in python, but would bet, that there is a module for this.
This way, when the calling python process exits, it won't terminate the spawned child, since it is now independent.
It seems you want to detach a system call to another thread.
script_A.py
import subprocess
import sys
while(1)
do some stuff
do even more stuff
if true:
pid = subprocess.Popen([sys.executable, "python script_B.py"]) # call subprocess
sys.exit(0)
Anyway it does not seem a good practice at all. Why do you not try the script A listens the Process Stack and if it finds script B running then stops. This is another example how you could do it.
import subprocess
import sys
import psutil
while(1)
#This sections queries the current processes running
for proc in psutil.process_iter():
pinfo = proc.as_dict(attrs=['pid', 'name'])
if pinfo[ 'name' ] == "script_B.py":
sys.exit(0)
do some stuff
do even more stuff
if true:
pid = subprocess.Popen([sys.executable, "python script_B.py"]) # call subprocess
sys.exit(0)
I'm using python to benchmark something. This can take a large amount of time, and I want to set a (global) timeout. I use the following script (summarized):
class TimeoutException(Exception):
pass
def timeout_handler(signum, frame):
raise TimeoutException()
# Halt problem after half an hour
signal.alarm(1800)
try:
while solution is None:
guess = guess()
try:
with open(solutionfname, 'wb') as solutionf:
solverprocess = subprocess.Popen(["solver", problemfname], stdout=solutionf)
solverprocess.wait()
finally:
# `solverprocess.poll() == None` instead of try didn't work either
try:
solverprocess.kill()
except:
# Solver process was already dead
pass
except TimeoutException:
pass
# Cancel alarm if it's still active
signal.alarm(0)
However it keeps spawning orphan processes sometimes, but I can't reliably recreate the circumstances. Does anyone know what the correct way to prevent this is?
You simply have to wait after killing the process.
The documentation for the kill() method states:
Kills the child. On Posix OSs the function sends SIGKILL to the child.
On Windows kill() is an alias for terminate().
In other words, if you aren't on Windows, you are only sending a signal to the subprocess.
This will create a zombie process because the parent process didn't read the return value of the subprocess.
The kill() and terminate() methods are just shortcuts to send_signal(SIGKILL) and send_signal(SIGTERM).
Try adding a call to wait() after the kill(). This is even shown in the example under the documentation for communicate():
proc = subprocess.Popen(...)
try:
outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
note the call to communicate() after the kill(). (It is equivalent to calling wait() and also erading the outputs of the subprocess).
I want to clarify one thing: it seems like you don't understand exactly what a zombie process is. A zombie process is a terminated process. The kernel keeps the process in the process table until the parent process reads its exit status. I believe all memory used by the subprocess is actually reused; the kernel only has to keep track of the exit status of such a process.
So, the zombie processes you see aren't running. They are already completely dead, and that's why they are called zombie. They are "alive" in the process table, but aren't really running at all.
Calling wait() does exactly this: wait till the subprocess ends and read the exit status. This allows the kernel to remove the subprocess from the process table.
On linux, you can use python-prctl.
Define a preexec function such as:
def pre_exec():
import signal
prctl.set_pdeathsig(signal.SIGTERM)
And have your Popen call pass it.
subprocess.Popen(..., preexec_fn=pre_exec)
That's as simple as that. Now the child process will die rather than become orphan if the parent dies.
If you don't like the external dependency of python-prctl you can also use the older prctl. Instead of
prctl.set_pdeathsig(signal.SIGTERM)
you would have
prctl.prctl(prctl.PDEATHSIG, signal.SIGTERM)
I've got a program that implements a buggy library that occasionally hangs due to improperly implementing parallisation.
I don't have the time to fix the core issue, so I'm looking for a hack to figure out when the process is hanging and not doing it's job.
Are there any OS X or python specific APIs to do this? Is it possible to use another thread or even the main thread to repeatedly parse stdout so that when the last few lines haven't changed in a certain duration, the other thread is notified and can kill the misbehaving thread? (and then restart?
Basically you are looking for a monitor process. It will run a command (or set of commands) and watch their execution looking for specific things (in your case, silence on stdout). Referencing the 2 SO questions below (and a brief look at some docs), you can quickly build a super simple monitor.
https://stackoverflow.com/questions/2804543/read-subprocess-stdout-line-by-line
https://stackoverflow.com/questions/3471461/raw-input-and-timeout
# monitor.py
import subprocess
TIMEOUT = 10
while True:
# start a new process to monitor
# you could also run sys.argv[1:] for a more generic monitor
child = subprocess.Popen(['python','other.py','arg'], stdout=subprocess.PIPE)
while True:
rlist,_,_ = select([child.stdout], [], [], TIMEOUT)
if rlist:
child.stdout.read() # do you need to save the output?
else:
# timeout occurred, did the process finish?
if child.poll() is not None:
# child process completed (or was killed, but didn't hang), we are done
sys.exit()
else:
# otherwise, kill the child and start a new one
child.kill()
break
I am developing a wrapper around gdb using python. Basically, I just want to be able to detect a few setup annoyances up-front and be able to run a single command to invoke gdb, rather than a huge string I have to remember each time.
That said, there are two cases that I am using. The first, which works fine, is invoking gdb by creating a new process and attaching to it. Here's the code that I have for this one:
def spawnNewProcessInGDB():
global gObjDir, gGDBProcess;
from subprocess import Popen
from os.path import join
import subprocess
binLoc = join(gObjDir, 'dist');
binLoc = join(binLoc, 'bin');
binLoc = join(binLoc, 'mycommand')
profileDir = join(gObjDir, '..')
profileDir = join(profileDir, 'trash-profile')
try:
gGDBProcess = Popen(['gdb', '--args', binLoc, '-profile', profileDir], cwd=gObjDir)
gGDBProcess.wait()
except KeyboardInterrupt:
# Send a termination signal to the GDB process, if it's running
promptAndTerminate(gGDBProcess)
Now, if the user presses CTRL-C while this is running, it breaks (i.e. it forwards the CTRL-C to GDB). This is the behavior I want.
The second case is a bit more complicated. It might be the case that I already had this program running on my system and it crashed, but was caught. In this case, I want to be able to connect to it using gdb to get a stack trace (or perhaps I was already running it, and I simply now want to connect to the process that's already in memory).
As a convenience feature, I've created a mirror function, which will connect to a running process using gdb:
def connectToProcess(procNum):
global gObjDir, gGDBProcess
from subprocess import Popen
import subprocess
import signal
print("Connecting to mycommand process number " + str(procNum) + "...")
try:
gGDBProcess = Popen(['gdb', '-p', procNum], cwd=gObjDir)
gGDBProcess.wait()
except KeyboardInterrupt:
promptAndTerminate(gGDBProcess)
Again, this seems to work as expected. It starts gdb, I can set breakpoints, run the program, etc. The only catch is that it doesn't forward CTRL-C to gdb if I press it while the program is running. Instead, it jumps immediately to promptAndTerminate().
I'm wondering if anyone can see why this is happening - the two calls to subprocess.Popen() seem identical to me, albeit that one is running gdb in a different mode.
I have also tried replacing the call to subprocess.Popen() with the following:
gGDBProcess = Popen(['gdb', '-p', procNum], cwd=gObjDir, stdin=subprocess.PIPE)
but this leads to undesirable results as well, because it doesn't actually communicate anything to the child gdb process (e.g. if I type in c to start the program running again after it is broken upon connection from gdb, it doesn't do anything). Again, it terminates the running python process when I type CTRL-C.
Any help would be appreciated!