Develop Reference

Python child process doesn't seem to function

I run this python file to spawn a process:
import os
import pwd
import subprocess
import sys
p = subprocess.Popen(['python', 'process_script.py'],
cwd="/execute",
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
process_script.py looks like this:
import time
import random
import string
import helper
#
def run():
while True:
filename = "/execute/" + "".join([random.choice(string.ascii_letters) for j in range(8)]) + ".txt"
helper.execute(f"echo foo > {filename}")
time.sleep(10)
#
run()
[EDIT] In fact ps shows no other processess, so it looks like the thread terminates... but how and why?
If I run process_script.py directly, the files are created.

in Popen child process dies when the parent exits you can add p.wait() at the end of your first script to prevent parent from exiting.
also this link is useful check it out!
subprocess gets killed even with nohup

python kill parent process but child process left

While I try to kill a python process, the child process started via os.system won't be terminated at the same time.
Killing child process when parent crashes in python and
Python Process won't call atexit
(atexit looks like not work with signal)
Does that mean I need to handle this situation by myself? If so, what is the preferred way to do so?
> python main.py
> ps
4792 ttys002 0:00.03 python run.py
4793 ttys002 0:00.03 python loop.py
> kill -15 4792
> ps
4793 ttys002 0:00.03 python loop.py
Sample Code:
main.py
import os
os.system('python loop.py')
loop.py
import time
while True:
time.sleep(1000)
UPDATE1
I did some experiment, and find out a workable version but still confuse about the logic.
import os
import sys
import signal
import subprocess
def sigterm_handler(_signo, _stack_frame):
# it raises SystemExit(0):
print 'go die'
sys.exit(0)
signal.signal(signal.SIGTERM, sigterm_handler)
try:
# os.system('python loop.py')
# use os.system won't work, it will even ignore the SIGTERM entirely for some reason
subprocess.call(['python', 'loop.py'])
except:
os.killpg(0, signal.SIGKILL)

kill -15 4792 sends SIGTERM to run.py in your example -- it sends nothing to loop.py (or its parent shell). SIGTERM is not propagated to other processes in the process tree by default.
os.system('python loop.py') starts at least two processes the shell and python process. You don't need it; use subprocess.check_call(), to run a single child process without the implicit shell. btw, if your subprocess is a Python script; consider importing it and running corresponding functions instead.
os.killpg(0, SIGKILL) sends SIGKILL signal to the current process group. A shell creates a new process group (a job) for each pipeline and therefore the os.killpg() in the parent has no effect on the child (see the update). See How to terminate a python subprocess launched with shell=True.
#!/usr/bin/env python
import subprocess
import sys
try:
p = subprocess.Popen([executable, 'loop'])
except EnvironmentError as e: #
sys.exit('failed to start %r, reason: %s' % (executable, e))
else:
try: # wait for the child process to finish
p.wait()
except KeyboardInterrupt: # on Ctrl+C (SIGINT)
#NOTE: the shell sends SIGINT (on CtrL+C) to the executable itself if
# the child process is in the same foreground process group as its parent
sys.exit("interrupted")
Update
It seems os.system(cmd) doesn't create a new process group for cmd:
>>> import os
>>> os.getpgrp()
16180
>>> import sys
>>> cmd = sys.executable + ' -c "import os; print(os.getpgrp())"'
>>> os.system(cmd) #!!! same process group
16180
0
>>> import subprocess
>>> import shlex
>>> subprocess.check_call(shlex.split(cmd))
16180
0
>>> subprocess.check_call(cmd, shell=True)
16180
0
>>> subprocess.check_call(cmd, shell=True, preexec_fn=os.setpgrp) #!!! new
18644
0
and therefore os.system(cmd) in your example should be killed by the os.killpg() call.
Though if I run it in bash; it does create a new process group for each pipeline:
$ python -c "import os; print(os.getpgrp())"
25225
$ python -c "import os; print(os.getpgrp())"
25248

Daemon thread launching software won't die

I'm trying to write a small script which will use plink.exe (from the same folder) to create a ssh tunnel (on windows).
I'm basically using os.system to launch the the command:
import time
import threading
from os.path import join, dirname, realpath
pc_tunnel_command = '-ssh -batch -pw xxxx -N -L 1234:host1:5678 user#host2'
if __name__ == '__main__':
t = threading.Thread(target = os.system, \
args = (join(dirname(realpath(__file__)), 'plink.exe ') + \
pc_tunnel_command,))
t.daemon = True
t.start()
#without this line it will die. I guess that plink doesn't have enough time to start.
time.sleep(5)
print 'Should die now'
However, it seems that the thread (and plink.exe) keep running. Why is this happening? Any way to force the thread to close? Better way to launch plink?
I want plink.exe to die when my program ends. Using a daemon thread was my plan of having the tunnel run in the background, and then dying when my main code exits.
BTW - same thing happens with subprocess.call.

You can use the atexit and signal modules to register calls back that will explicitly kill the process when your program exits normally or receives SIGTERM, respectively:
import sys
import time
import atexit
import signal
import subprocess
from functools import partial
from os.path import join, dirname, realpath
pc_tunnel_command = '-ssh -batch -pw xxxx -N -L 1234:host1:5678 user#host2'
def handle_exit(p, *args):
print("killing it")
p.terminate()
sys.exit(0)
if __name__ == '__main__':
p = subprocess.Popen(join(dirname(realpath(__file__)), 'plink.exe ') + pc_tunnel_command, shell=True)
func = partial(handle_exit, p)
signal.signal(signal.SIGTERM, func)
atexit.register(func)
print 'Should die now'
The one thing that is odd about the behavior your desrcibed is that I would have expected your program to exit after your sleep call, but leave plink running in the background, rather than having your program hang until the os.system call completes. That's the behavior I see on Linux, at least. In any case, explicitly terminating the child process should solve the issue for you.

os.system does not return until the child process exits. The same is true for subprocess.call. That's why your thread is sitting there, waiting for plink to finish. You can probably use subprocess.Popen to launch the process asynchronously and then exit. In any case, the additional thread you are creating is unnecessary.

Python subprocess communicate kills my process

Why does communicate kill my process? I want an interactive process but communicate does something so that I cannot take raw_input any more in my process.
from sys import stdin
from threading import Thread
from time import sleep
if __name__ == '__main__':
print("Still Running\n")
x = raw_input()
i = 0
while ('n' not in x ) :
print("Still Running " + str(i) + " \r\n")
x = raw_input()
i += 1
print("quit")
print(aSubProc.theProcess.communicate('y'))
print(aSubProc.theProcess.communicate('y'))
exception!
self.stdin.write(input)
ValueError: I/O operation on closed file

communicate and wait methods of Popen objects, close the PIPE after the process returns. If you want stay in communication with the process try something like this:
import subprocess
proc = subprocess.Popen("some_process", stdout=subprocess.PIPE, stdin=subprocess.PIPE)
proc.stdin.write("input")
proc.stdout.readline()

Why does communicate kill my process?
From the docs for Popen.communicate(input=None, timeout=None):
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
emphasize mine
You may call .communicate() only once. It means that you should provide all input at once:
#!/usr/bin/env python
import os
import sys
from subprocess import Popen, PIPE
p = Popen([sys.executable, 'child.py'], stdin=PIPE, stdout=PIPE)
print p.communicate(os.linesep.join('yyn'))[0]
Output
Still Running
Still Running 0
Still Running 1
quit
Notice the doubled newlines: one from '\r\n' and another from print statement itself in your script for the child process.
Output shows that the child process received three input lines successfully ('y', 'y', and 'n').
Here's a similar code using subprocess.check_output()'s input parameter from Python3.4:
#!/usr/bin/env python3.4
import os
import sys
from subprocess import check_output
output = check_output(['python2', 'child.py'], universal_newlines=True,
input='\n'.join('yyn'))
print(output, end='')
It produces the same output.
If you want to provide a different input depending on responses from the child processes then use pexpect module or its analogs to avoid issues mentioned in Why not just use a pipe (popen())?

hcitool lescan subprocess python do not produce output

I have a problem with sub-process code. The subprocess.Popen() works fine but when I try to read its output through stdout.read() there is no value to read.
**import os
import signal
import subprocess
import threading
import sys
import commands
print commands.getoutput("hcitool dev")
print 'down'
commands.getoutput('hciconfig hci1 down')
print 'up'
commands.getoutput('hciconfig hci1 up')
commands.getoutput('killall hcitool')
stop = False
ping = subprocess.call('hcitool lescan', shell = False,
stdout=subprocess.PIPE,executable='/bin/bash')
for i in ping.stdout:
print i
def kill():
global stop
stop = True
os.kill(ping.pid, signal.SIGTERM)
threading.Timer(5, kill).start()
#while not stop:
# print 'now in while not loop'
# sys.stdout.write(ping.stdout.read(1))
print 'trying to print stdout'
out, err = ping.communicate()
print "out",out
#result = out.decode()
print "Result : ",result**
This code works fine when I change hcitool lescan to ping www.google.com, and produces output but when I try with hcitool lescan it either hangs forever or produces no output. Help is appreciated!

Any of the above answers didn't work for me. Was hung up in the forever scan of hcitool. So finally i wrote a shell script and called it by my python code. This is working fine for me and i am reading the output from the file "result.txt".
hcitool lescan>result.txt &
sleep 5
pkill --signal SIGINT hcitool

There are multiple errors in your code e.g., subprocess.call() returns an integer (exit status of the program) and an integer has no .stdout attribute; also the combination of shell=False and non-None executable is very rarely useful (and it is probably used incorrectly in this case).
The simplest way to fix the code is to use check_output():
from subprocess import check_output as qx
output = qx(["hcitool", "lescan"]) # get all output at once
print output,
As an alternative, you could print program's output line by line as soon as its stdout is flushed:
from subprocess import Popen, PIPE
proc = Popen(["hcitool", "lescan"], stdout=PIPE, bufsize=1) # start process
for line in iter(proc.stdout.readline, b''): # read output line-by-line
print line,
# reached EOF, nothing more to read
proc.communicate() # close `proc.stdout`, wait for child process to terminate
print "Exit status", proc.returncode
To kill a subprocess, you could use its .kill() method e.g.:
from threading import Timer
def kill(process):
try:
process.kill()
process.wait() # to avoid zombies
except OSError: # ignore errors
pass
Timer(5, kill, [proc]).start() # kill in 5 seconds

thank you very much..but the problem is hcitool lescan never stops,and hence hangs out in the very next line of your code.,
and i found similar solution here it is.this works fine and i dont have to kill subprocess,this code takes some extra time to pour output,but this following code works preciesly,
from os import kill
import signal
import subprocess
import threading
import tempfile
import sys
import time
from tempfile import TemporaryFile
import commands
t = TemporaryFile()
global pipe_output
print commands.getoutput("hcitool dev")
print 'down'
commands.getoutput('hciconfig hci0 down')
print 'up'
commands.getoutput('hciconfig hci0 up')
print commands.getoutput("hcitool dev")
commands.getoutput('killall hcitool')
p = subprocess.Popen('hcitool lescan', bufsize = 0,shell = True, stdout =subprocess.PIPE,stderr = subprocess.STDOUT)
time.sleep(10)
#os.kill(p.pid,signal.SIGTERM)
for i in range(0,30,1):
print 'for'
inchar = p.stdout.readline()
i+=1
if inchar:
print 'loop num:',i
print str(inchar)
t.write(str(inchar))
print 'out of loop'
t.seek(0)
print t.read()
any help how to reduce waiting time,other than just changing time.sleep() ,is appreciated
thank you all

Use Popen class instead of the call class. hcitool lescan will run forever. subprocess.call waits for the call to be finished to return. Popen does not wait.