Develop Reference

Python: Process in one Thread stopping a Process in another Thread from finishing

I'd appreciate some help with threading, which I pretty new to.
The example code is not exactly what I’m doing (‘notepad’ and ‘calc’ are just example commands), but a simplified version that shows my problem.
I want to run two seperate threads that each run a different command a number of times. I would like the code to do this:
Start the first instance of ‘notepad’ and ‘calc’ simultaneously
(which it does)
When I close an instance of ‘notepad’, to open the
next instance of ‘notepad’.
When I close an instance of ‘calc’, to
open the next instance of ‘calc’.
[edit] I want the script to wait until both threads have finished, as it needs to do some processing of the output from these.
However, when I close an instance of ‘notepad’, the next instance of ‘notepad’ does not start until I’ve closed the current instance of ‘calc’ and vice versa. With a bit of de-bugging, it looks like the process (from Popen) for the closed instance of 'notepad' doesn't finish until the current 'calc' is closed.
Running Python 2.7 on Windows 7
Example Code:
from subprocess import Popen, PIPE, STDOUT
from threading import Thread
def do_commands(command_list):
for command in command_list:
proc = Popen("cmd.exe", stdin=PIPE, stdout=PIPE, stderr=STDOUT)
stdout_value, stderr_value = proc.communicate(input=command)
# MAIN CODE
A_command_list = ["notepad\n", "notepad\n", "notepad\n" ]
B_command_list = ["calc\n", "calc\n", "calc\n" ]
A_args = [A_command_list]
B_args = [B_command_list]
A_thread = Thread(target=do_commands, args=(A_args))
B_thread = Thread(target=do_commands, args=(B_args))
A_thread.start()
B_thread.start()
A_thread.join()
B_thread.join()
Thanks in advance :-)
Nick

So the communicate() method is apparently waiting for all processes created by Popen and executing cmd.exe and started at nearly the same time to terminate. Since the cmd.exe that runs calculator starts at nearly the same time as the cmd.exe that runs Notepad, both communicate() calls (one in A_thread and one in B_thread) wait until both processes term. Thus neither for loop advances until both processes term.
Adding a delay between starting the two threads fixes the problem.
So, leaving your original code unchanged and adding
sleep(1)
between the two Thread starts produces the desired behavior.
On my system, adding a delay of 0.0001 seconds reliably fixed the problem whereas a delay of 0.00001 did not.

How to check if a shell command is over in Python

Let's say that I have this simple line in python:
os.system("sudo apt-get update")
of course, apt-get will take some time untill it's finished, how can I check in python if the command had finished or not yet?
Edit: this is the code with Popen:
os.environ['packagename'] = entry.get_text()
process = Popen(['dpkg-repack', '$packagename'])
if process.poll() is None:
print "It still working.."
else:
print "It finished"
Now the problem is, it never print "It finished" even when it really finish.

As the documentation states it:
This is implemented by calling the Standard C function system(), and
has the same limitations
The C call to system simply runs the program until it exits. Calling os.system blocks your python code until the bash command has finished thus you'll know that it is finished when os.system returns. If you'd like to do other stuff while waiting for the call to finish, there are several possibilities. The preferred way is to use the subprocessing module.
from subprocess import Popen
...
# Runs the command in another process. Doesn't block
process = Popen(['ls', '-l'])
# Later
# Returns the return code of the command. None if it hasn't finished
if process.poll() is None:
# Still running
else:
# Has finished
Check the link above for more things you can do with Popen
For a more general approach at running code concurrently, you can run that in another thread or process. Here's example code:
from threading import Thread
...
thread = Thread(group=None, target=lambda:os.system("ls -l"))
thread.run()
# Later
if thread.is_alive():
# Still running
else:
# Has finished
Another option would be to use the concurrent.futures module.

os.system will actually wait for the command to finish and return the exit status (format dependent format).

os.system is blocking; it calls the command waits for its completion, and returns its return code.
So, it'll be finished once os.system returns.
If your code isn't working, I think that could be caused by one of sudo's quirks, it refuses to give rights on certain environments(I don't know the details tho.).

Python daemon thread does not exit when parent thread exits

I have some Python code that creates a demon thread. The parent thread ends almost immediately, but the daemon thread keeps printing sleep.
import threading
import time
def int_sleep():
for _ in range(1, 600):
time.sleep(1)
print("sleep")
def main():
thread = threading.Thread(target=int_sleep)
thread.daemon = True
thread.start()
time.sleep(2)
print("main thread end...")
thread = threading.Thread(target=main)
thread.start()
sys.version:
'3.3.3 (v3.3.3:c3896275c0f6, Nov 18 2013, 21:19:30) [MSC v.1600 64 bit (AMD64)]'
Prints:
sleep
main thread end...
sleep
sleep
sleep
Why doesn't the Python daemon thread exit when parent thread exits?

If you specify thread.daemon = True for your python thread, then the program will halt immediately when only the daemon is left. The the commands sent to stdout are lost.
Add this to a file called main.py
import threading
import time
def int_sleep():
for _ in range(1, 600):
time.sleep(1)
print("sleep")
def main():
thread = threading.Thread(target=int_sleep)
thread.daemon = True
thread.start()
time.sleep(2)
print("main thread end...")
thread = threading.Thread(target=main)
thread.daemon = True
thread.start()
Run it like this:
el#apollo:~/code/python/run01$ python --version
Python 2.7.6
el#apollo:~$ python main.py
el#apollo:~$
See it prints nothing because the thread started. You set it to be a daemon and started it. Then the program ended.
Extra notes: If you paste this code into a python interpreter, all the print statements will appear on the terminal because the daemon never loses hold of its connection to stdout.
Read more: http://docs.python.org/2/library/threading.html

I can only reproduce the behavior described by OP (unending output of 'sleep') if done from the python shell. If run from a file it works as expected (a few lines of 'sleep' and a single line of 'main thread end ...' )
Similarly, the second program exits immediately if run as a file, BUT also prints unending 'sleep' statements when run from the python shell.
My conclusion: since the thread that is the python shell continues to run even after "main" finishes, preventing the daemon(s) from being terminated when run from the python shell.
Could this be considered a bug (i.e that Python's behavior is different depending on how the script is run) or is it expected ? I defer to more experienced Pythonistas...
BTW - tested with Python 3.2.3

For completeness check out this article.
https://joeshaw.org/2009/02/24/605/
The monitoring was done inside a daemon thread. The Python docs say
only:
A thread can be flagged as a “daemon thread”. The significance
of this flag is that the entire Python program exits when only
daemon threads are left.
Which sounds pretty good, right? This thread is just occasionally
grabbing some data, and we don’t need to do anything special when the
program shuts down. Yeah, I remember when I used to believe in things
too.
Despite a global interpreter lock that prevents Python from being
truly concurrent anyway, there is a very real possibility that the
daemon threads can still execute after the Python runtime has started
its own tear-down process. One step of this process appears to be to
set the values inside globals() to None, meaning that any module
resolution results in an AttributeError attempting to dereference
NoneType. Other variations on this cause TypeError to be thrown.
I'm not sure whether that's a bug that's been fixed or a bug still in existence or behaviour as per design. But if you see weirdness keep this in the back of your head.
So an alternative is to loop in the child thread on an exit flag which you can set in the main when you're done. Then wait in the main for the child thread to die and then clean up.

Detect hanging python shell in OS X

I've got a program that implements a buggy library that occasionally hangs due to improperly implementing parallisation.
I don't have the time to fix the core issue, so I'm looking for a hack to figure out when the process is hanging and not doing it's job.
Are there any OS X or python specific APIs to do this? Is it possible to use another thread or even the main thread to repeatedly parse stdout so that when the last few lines haven't changed in a certain duration, the other thread is notified and can kill the misbehaving thread? (and then restart?

Basically you are looking for a monitor process. It will run a command (or set of commands) and watch their execution looking for specific things (in your case, silence on stdout). Referencing the 2 SO questions below (and a brief look at some docs), you can quickly build a super simple monitor.
https://stackoverflow.com/questions/2804543/read-subprocess-stdout-line-by-line
https://stackoverflow.com/questions/3471461/raw-input-and-timeout
# monitor.py
import subprocess
TIMEOUT = 10
while True:
# start a new process to monitor
# you could also run sys.argv[1:] for a more generic monitor
child = subprocess.Popen(['python','other.py','arg'], stdout=subprocess.PIPE)
while True:
rlist,_,_ = select([child.stdout], [], [], TIMEOUT)
if rlist:
child.stdout.read() # do you need to save the output?
else:
# timeout occurred, did the process finish?
if child.poll() is not None:
# child process completed (or was killed, but didn't hang), we are done
sys.exit()
else:
# otherwise, kill the child and start a new one
child.kill()
break

Python: How to determine subprocess children have all finished running

I am trying to detect when an installation program finishes executing from within a Python script. Specifically, the application is the Oracle 10gR2 Database. Currently I am using the subprocess module with Popen. Ideally, I would simply use the wait() method to wait for the installation to finish executing, however, the documented command actually spawns child processes to handle the actual installation. Here is some sample code of the failing code:
import subprocess
OUI_DATABASE_10GR2_SUBPROCESS = ['sudo',
'-u',
'oracle',
os.path.join(DATABASE_10GR2_TMP_PATH,
'database',
'runInstaller'),
'-ignoreSysPrereqs',
'-silent',
'-noconfig',
'-responseFile '+ORACLE_DATABASE_10GR2_SILENT_RESPONSE]
oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
oracle_subprocess.wait()
There is a similar question here: Killing a subprocess including its children from python, but the selected answer does not address the children issue, instead it instructs the user to call directly the application to wait for. I am looking for a specific solution that will wait for all children of the subprocess. What if there are an unknown number of subprocesses? I will select the answer that addresses the issue of waiting for all children subprocesses to finish.
More clarity on failure: The child processes continue executing after the wait() command since that command only waits for the top level process (in this case it is 'sudo'). Here is a simple diagram of the known child processes in this problem:
Python subprocess module -> Sudo -> runInstaller -> java -> (unknown)

Ok, here is a trick that will work only under Unix. It is similar to one of the answers to this question: Ensuring subprocesses are dead on exiting Python program. The idea is to create a new process group. You can then wait for all processes in the group to terminate.
pid = os.fork()
if pid == 0:
os.setpgrp()
oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
oracle_subprocess.wait()
os._exit(0)
else:
os.waitpid(-pid)
I have not tested this. It creates an extra subprocess to be the leader of the process group, but avoiding that is (I think) quite a bit more complicated.
I found this web page to be helpful as well. http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

You can just use os.waitpid with the the pid set to -1, this will wait for all the subprocess of the current process until they finish:
import os
import sys
import subprocess
proc = subprocess.Popen([sys.executable,
'-c',
'import subprocess;'
'subprocess.Popen("sleep 5", shell=True).wait()'])
pid, status = os.waitpid(-1, 0)
print pid, status
This is the result of pstree <pid> of different subprocess forked:
python───python───sh───sleep
Hope this can help :)

Check out the following link http://www.oracle-wiki.net/startdocsruninstaller which details a flag you can use for the runInstaller command.
This flag is definitely available for 11gR2, but I have not got a 10g database to try out this flag for the runInstaller packaged with that version.
Regards

Everywhere I look seems to say it's not possible to solve this in the general case. I've whipped up a library called 'pidmon' that combines some answers for Windows and Linux and might do what you need.
I'm planning to clean this up and put it on github, possibly called 'pidmon' or something like that. I'll post a link if/when I get it up.
EDIT: It's available at http://github.com/dbarnett/python-pidmon.
I made a special waitpid function that accepts a graft_func argument so that you can loosely define what sort of processes you want to wait for when they're not direct children:
import pidmon
pidmon.waitpid(oracle_subprocess.pid, recursive=True,
graft_func=(lambda p: p.name == '???' and p.parent.pid == ???))
or, as a shotgun approach, to just wait for any processes started since the call to waitpid to stop again, do:
import pidmon
pidmon.waitpid(oracle_subprocess.pid, graft_func=(lambda p: True))
Note that this is still barely tested on Windows and seems very slow on Windows (but did I mention it's on github where it's easy to fork?). This should at least get you started, and if it works at all for you, I have plenty of ideas on how to optimize it.