Develop Reference

Cannot get subprocess return code in python3

I'm trying to make something like supervisor for my python daemon process and found out that same code works in python2 and doesn't work in python3.
Generally, I've come to this minimal example code.
daemon.py
#!/usr/bin/env python
import signal
import sys
import os
def stop(*args, **kwargs):
print('daemon exited', os.getpid())
sys.exit(0)
signal.signal(signal.SIGTERM, stop)
print('daemon started', os.getpid())
while True:
pass
supervisor.py
import os
import signal
import subprocess
from time import sleep
parent_pid = os.getpid()
commands = [
[
'./daemon.py'
]
]
popen_list = []
for command in commands:
popen = subprocess.Popen(command, preexec_fn=os.setsid)
popen_list.append(popen)
def stop_workers(*args, **kwargs):
for popen in popen_list:
print('send_signal', popen.pid)
popen.send_signal(signal.SIGTERM)
while True:
popen_return_code = popen.poll()
if popen_return_code is not None:
break
sleep(5)
signal.signal(signal.SIGTERM, stop_workers)
for popen in popen_list:
print('wait_main', popen.wait())
If you run supervisor.py and then call kill -15 on its pid, then it will hang in infinite loop, because popen_return_code will never be not None. I discovered, that it's basically because of adding threading.Lock for wait_pid operation (source), but how can I rewrite code so it'll handle child exit correctly?

This is an interesting case.
I've spent few hours trying to figure out the reason why this happens and the only thing I came up with at this moment is that the implementation of wait() and poll() have been changed in python3 versus python2.7.
Looking into the source code of python3/suprocess.py implementation, we can see that there is a lock acquire happens when you call wait() method of Popen object, see
https://github.com/python/cpython/blob/master/Lib/subprocess.py#L1402.
This lock prevents further poll() calls to work as expected until the lock acquired by wait() will be released, see
https://github.com/python/cpython/blob/master/Lib/subprocess.py#L1355
and comment there
Something else is busy calling waitpid. Don't allow two
at once. We know nothing yet.
There is no such a lock in python2.7/subprocess.py so this looks like a reason why it works in python2.7 and doesn't work in python3.
However I don't see a reason why are you trying to poll() inside the signal handler, try rewrite your supervisor.py as following, this should work as expected both on python3 and python2.7
supervisor.py
import os
import signal
import subprocess
from time import sleep
parent_pid = os.getpid()
commands = [
[
'./daemon.py'
]
]
popen_list = []
for command in commands:
popen = subprocess.Popen(command, preexec_fn=os.setsid)
popen_list.append(popen)
def stop_workers(*args, **kwargs):
for popen in popen_list:
print('send_signal', popen.pid)
popen.send_signal(signal.SIGTERM)
signal.signal(signal.SIGTERM, stop_workers)
for popen in popen_list:
print('wait_main', popen.wait())
Hope this helps

Generally, I agree with answer from #risboo6909, but also have some thoughts, how to fix this situation.
You can change subproccess.Popen to psutil.Popen.
In main loop instead of popen.wait() you can just do infinite loop, because process will exit in signal handler.

python kill parent process but child process left

While I try to kill a python process, the child process started via os.system won't be terminated at the same time.
Killing child process when parent crashes in python and
Python Process won't call atexit
(atexit looks like not work with signal)
Does that mean I need to handle this situation by myself? If so, what is the preferred way to do so?
> python main.py
> ps
4792 ttys002 0:00.03 python run.py
4793 ttys002 0:00.03 python loop.py
> kill -15 4792
> ps
4793 ttys002 0:00.03 python loop.py
Sample Code:
main.py
import os
os.system('python loop.py')
loop.py
import time
while True:
time.sleep(1000)
UPDATE1
I did some experiment, and find out a workable version but still confuse about the logic.
import os
import sys
import signal
import subprocess
def sigterm_handler(_signo, _stack_frame):
# it raises SystemExit(0):
print 'go die'
sys.exit(0)
signal.signal(signal.SIGTERM, sigterm_handler)
try:
# os.system('python loop.py')
# use os.system won't work, it will even ignore the SIGTERM entirely for some reason
subprocess.call(['python', 'loop.py'])
except:
os.killpg(0, signal.SIGKILL)

kill -15 4792 sends SIGTERM to run.py in your example -- it sends nothing to loop.py (or its parent shell). SIGTERM is not propagated to other processes in the process tree by default.
os.system('python loop.py') starts at least two processes the shell and python process. You don't need it; use subprocess.check_call(), to run a single child process without the implicit shell. btw, if your subprocess is a Python script; consider importing it and running corresponding functions instead.
os.killpg(0, SIGKILL) sends SIGKILL signal to the current process group. A shell creates a new process group (a job) for each pipeline and therefore the os.killpg() in the parent has no effect on the child (see the update). See How to terminate a python subprocess launched with shell=True.
#!/usr/bin/env python
import subprocess
import sys
try:
p = subprocess.Popen([executable, 'loop'])
except EnvironmentError as e: #
sys.exit('failed to start %r, reason: %s' % (executable, e))
else:
try: # wait for the child process to finish
p.wait()
except KeyboardInterrupt: # on Ctrl+C (SIGINT)
#NOTE: the shell sends SIGINT (on CtrL+C) to the executable itself if
# the child process is in the same foreground process group as its parent
sys.exit("interrupted")
Update
It seems os.system(cmd) doesn't create a new process group for cmd:
>>> import os
>>> os.getpgrp()
16180
>>> import sys
>>> cmd = sys.executable + ' -c "import os; print(os.getpgrp())"'
>>> os.system(cmd) #!!! same process group
16180
0
>>> import subprocess
>>> import shlex
>>> subprocess.check_call(shlex.split(cmd))
16180
0
>>> subprocess.check_call(cmd, shell=True)
16180
0
>>> subprocess.check_call(cmd, shell=True, preexec_fn=os.setpgrp) #!!! new
18644
0
and therefore os.system(cmd) in your example should be killed by the os.killpg() call.
Though if I run it in bash; it does create a new process group for each pipeline:
$ python -c "import os; print(os.getpgrp())"
25225
$ python -c "import os; print(os.getpgrp())"
25248

how to open a cmd shell in windows and issue commands to that shell using python

I've tried
import os
os.system('cmd.exe')
great it opens up a cmd shell
but how do I write commands from my python script so that the opened shell executes it?
basically it's like this
open up a cmd shell and somehow get a hold of the instance of the opened shell and issue commands to it. the subprocess module doesn't seem to open a cmd shell im not trying to view the contents of the shell via the interpreter but actually
but that's what subprocess does?
so how can we open up a cmd shell and pass commands to that opened shell?

to anyone having the same problems as me
I needed to get a handle on the command prompt window and and wanted to activate my virtualenv and run my .py file programatically.
I used pywin32com and after hours of researching stackoverflow and the web
I managed to get a working solution
I don't know much about the subprocess module but and I don't know if it let's you send different commands to the opened command prompt
but here is my working solution
import time
import os
from win32com import client
from win32gui import GetWindowText, GetForegroundWindow, SetForegroundWindow, EnumWindows
from win32process import GetWindowThreadProcessId
class ActivateVenv:
def set_cmd_to_foreground(self, hwnd, extra):
"""sets first command prompt to forgeround"""
if "cmd.exe" in GetWindowText(hwnd):
SetForegroundWindow(hwnd)
return
def get_pid(self):
"""gets process id of command prompt on foreground"""
window = GetForegroundWindow()
return GetWindowThreadProcessId(window)[1]
def activate_venv(self, shell, venv_location):
"""activates venv of the active command prompt"""
shell.AppActivate(self.get_pid())
shell.SendKeys("cd \ {ENTER}")
shell.SendKeys(r"cd %s {ENTER}" % venv_location)
shell.SendKeys("activate {ENTER}")
def run_py_script(self,shell):
"""runs the py script"""
shell.SendKeys("cd ../..{ENTER}")
shell.SendKeys("python run.py {ENTER}")
def open_cmd(self, shell):
""" opens cmd """
shell.run("cmd.exe")
time.sleep(1)
if __name__ == "__main__":
shell = client.Dispatch("WScript.Shell")
run_venv = ActivateVenv()
run_venv.open_cmd(shell)
EnumWindows(run_venv.set_cmd_to_foreground, None)
run_venv.activate_venv(shell, "flask3.5/venv/scripts")
run_venv.run_py_script(shell)

import subprocess
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
process.wait()
print process.returncode
The command variable should be for example: cmd /k. You can also add a stdin=subprocess.PIPE to the Popen argument list and write commands to cmd:
subprocess.Popen(command, shell=True, stdout=subprocess.PIPE,stdin=subprocess.PIPE) the final code:
import subprocess
process = subprocess.Popen('cmd /k ', shell=True, stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=None)
process.stdin.write("dir") #passing command
stdOutput,stdError = process.communicate()
print stdOutput
process.stdin.close()
Or alternatively:
from subprocess import *
Popen("cmd /k dir")

Trapping and handling taskkill in Windows Python

I am using Python 2.6.6 for Windows (on Windows XP SP3) with pywin32-218.
In my Python application, I have a second thread (apart from the main thread) which spawns a subprocess to run another Windows executable.
My problem is that when the main process (python.exe) is killed (e.g. using taskkill), I want to terminate the subprocess (calc.exe) and perform some cleaning up.
I tried various methods (atexit, signal and win32api.handleConsoleCtrl), but none seem to be able to trap the taskkill signal.
My code as follows (test.py):
import sys
import os
import signal
import win32api
import atexit
import time
import threading
import subprocess
class SecondThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.secondProcess = None
def run(self):
secondCommand = ['C:\WINDOWS\system32\calc.exe']
self.secondProcess = subprocess.Popen(secondCommand)
print 'calc.exe running'
self.secondProcess.wait()
print 'calc.exe stopped'
# do cleanup here
def stop(self):
if self.secondProcess and self.secondProcess.returncode == None:
self.secondProcess.kill()
secondThread = SecondThread()
def main():
secondThread.start()
def cleanup():
print 'cleaning up'
secondThread.stop()
print 'cleaned up'
atexit.register(cleanup)
def handleSignal(signalNum, frame):
print 'handleSignal'
cleanup()
sys.exit(0)
for signalNum in (signal.SIGINT, signal.SIGILL, signal.SIGABRT, signal.SIGFPE, signal.SIGSEGV, signal.SIGTERM):
signal.signal(signalNum, handleSignal)
def handleConsoleCtrl(signalNum):
print ('handleConsoleCtrl')
cleanup()
win32api.SetConsoleCtrlHandler(handleConsoleCtrl, True)
main()
The application is launched using
python.exe test.py
The console then prints "calc.exe running", and the Calculator application runs, and using Process Explorer, I can see calc.exe as a sub-process of python.exe
Then I kill the main process using
taskkill /pid XXXX /f
(where XXXX is the PID for python.exe)
What happens after this is that the command prompt returns without further output (i.e. none of "cleaning up", "handleSignal" or "handleConsoleCtrl" gets printed), the Calculator application continues running, and from Process Explorer, python.exe is no longer running but calc.exe has re-parented itself.

Taskkill (normally) sends WM_CLOSE. If your application is console only and has no window, while you can get CTRL_CLOSE_EVENT via a handler set by SetConsoleCtrlHandler (which happens if your controlling terminal window is closed) you can't receive a bare WM_CLOSE message.
If you have to stick with taskkill (rather than using a different program to send a Ctrl-C) one solution is to set the aforementioned handler and ensure your application has its own terminal window (e.g. by usingstart.exe "" <yourprog> to invoke it). See https://stackoverflow.com/a/23197789/4513656 for details an alternatives.

Process closing

can I use Popen from python subprocess to close started process? For example, from popen I run some application. In some part of my code I have to close that ran app.
For example, from console in Linux I do:
./some_bin
... It works and logs stdout here ...
Ctrl + C and it breaks
I need something like Ctrl + C but in my program code.

from subprocess import Popen
process = Popen(['slow', 'running', 'program'])
while process.poll():
if raw_input() == 'Kill':
if process.poll(): process.kill()
kill() will kill a process. See more here: Python subprocess module

Use the subprocess module.
import subprocess
# all arguments must be passed one at a time inside a list
# they must all be string elements
arguments = ["sleep", "3600"] # first argument is the program's name
process = subprocess.Popen(arguments)
# do whatever you want
process.terminate()

Some time ago I needed a 'gentle' shutdown for a process by sending CTRL+C in Windows console.
Here's what I have:
import win32api
import win32con
import subprocess
import time
import shlex
cmdline = 'cmd.exe /k "timeout 60"'
args = shlex.split(cmdline)
myprocess = subprocess.Popen(args)
pid = myprocess.pid
print(myprocess, pid)
time.sleep(5)
win32api.GenerateConsoleCtrlEvent(win32con.CTRL_C_EVENT, pid)
# ^^^^^^^^^^^^^^^^^^^^ instead of myprocess.terminate()