I wrote a test program, which has two processes. The father process gets data from a Queue, and the child puts data into it. There is a signal handler which tells the program to exit. However, it does not exit sometimes when I send the signal SIGTERM to the pid(child process) I printed, and it seems to be having a deadlock.
import os
import sys
import multiprocessing
import time
import signal
bStop = False
def worker(que):
signal.signal(signal.SIGTERM,sighandler)
print 'worker:',os.getpid()
for i in range(100000000):
que.put(i)
print 'STOP'
def sighandler(num,frame):
print 'catch signal'
q.put('STOP')
sys.exit(0)
q = multiprocessing.Queue(100)
p = multiprocessing.Process(target=worker,args=(q,))
p.start()
for item in iter(q.get,'STOP'):
print 'get',item
pass
print 'main stop'
p.join()
Unless you are running python 3 you should be using xrange instead of range for a loop that large. Python tends to choke once it exceeds a certain list size and so you really really need to move to generators by that point.
That very well could be the issue your seeing right now.
Related
tl;dr: I have several threads, one being a thread listening to input() to keep the program running/exit on keypress. But at one time in the program I need to stop this listener or it will intercept the input for a subprocessed program.
Long version:
- Program should download some data, then hand this over to some other console program to be processed.
- Program should either run until download is finished or until ENTER-keypress has been sent.
- In both cases the download thread will be ended gracefully and the external processing should be done.
- Problem: The input() function is still listening and intercepting the first input to the subprocess'ed console program.
import os
import subprocess
import threading
import time
def thread_do_downloads():
# does some downloads and will set the flag "flag_download_completed=True"
# eventually to signal download completed
# for this example just set the flag
global flag_download_completed
flag_download_completed = True
def do_stuff_with_downloaded_data():
# this is of course not the program I would call,
# but this example should show how the input would be intercepted
if os.name == 'nt':
parameters = ["set", "/p", "variable=Press Enter"] # for this example (Windows) call "set", this program will wait for a user input
else:
parameters = ["read", "variable"] # hope this works for linux...
p1 = subprocess.Popen(parameters, shell=True)
p1.communicate()
def listen_for_keypress():
input()
print("keypress intercepted")
def main():
dl = threading.Thread(target=thread_do_downloads)
dl.start()
kill_listener = threading.Thread(target=listen_for_keypress, daemon=True) # daemon: to not have it lingering after main thread is done
kill_listener.start()
print("Press ENTER to stop downloading.")
while True:
if not kill_listener.is_alive() or flag_download_completed:
break
time.sleep(1)
# here are some lines to make sure the download thread above completes gracefully
do_stuff_with_downloaded_data()
print("All done")
if __name__ == '__main__':
flag_download_completed = False
main()
Will result in:
Press ENTER to stop downloading.
Press Enter << stopped here until I pressed ENTER
keypress intercepted << stopped here until I pressed ENTER
All done
If you can keep the main thread on top of the console, maybe you could take advantage of the fact that input() is going to block the main thread until Enter is pressed. Once the execution continues (because Enter was pressed), communicate to the running threads that they have to stop using an Event (another example here). If you do want to listen for S.O. signals, I suggest you take a look to the signal module (watch out, some features may be O.S dependent).
import threading
import time
def thread_do_downloads(stop_activated):
# does some downloads and will set the flag "flag_download_completed=True"
# eventually to signal download completed
# for this example just set the flag
global flag_download_completed
while not stop_activated.is_set():
time.sleep(0.5)
print("ZZZZZZZ")
def do_stuff_with_downloaded_data():
print("doing stuff with downloaded data")
def main():
stop_activated = threading.Event()
dl = threading.Thread(target=thread_do_downloads, args=(stop_activated,))
dl.start()
input("Press ENTER to stop downloading.")
stop_activated.set()
print("stopping (waiting for threads to finish...)")
dl.join()
# here are some lines to make sure the download thread above completes gracefully
do_stuff_with_downloaded_data()
print("All done")
if __name__ == '__main__':
main()
EDIT (as per the OP's comment):
One of the complications that the original question has is how to communicate the termination request to a subprocess. Because processes don't share memory with the parent process (the process who spawned it) this can, indeed, only (or almost only) be done through actual SO signals. Because of this memory isolation, any flags set on the parent process will have no effect in the spawned subprocesses: the only way of inter process communication is either through OS signals, or through files (or file-like structures) that both parent and child process "known about" and use to share information. Also, calling an input() in the parent binds the standard input (stdin) to that process which means by default, the subprocesses are unaware about the keys pressed in the parent (you could always bind the stdin of the child process to the stdin of the parent, but that would complicate a bit more the code)
Fortunately, the instances of Popen do offer a nice way to send signals to the child process: the TERM signal, which the subprocess could catch and is supposed to interpret as "Hey, you're gonna be stopped real soon, so do your clean-up things, close files and so on and exit" and the KILL signal that doesn't really tell anything to the subprocess (can't be caught): it just kills it (In Linux, for instance a KILL signal removes all access to memory from the killed process so any action that uses memory, such as a seek for next operation will cause an error. More info here)
To demonstrate that, let's say we have a simple script.py file in the same directory where our main program is located that looks like this:
script.py >
#!/usr/bin/env python
import sys
import random
import time
def main():
done = False
while not done:
time.sleep(0.5)
print("I'm busy doing things!!")
done = random.randint(0, 15) == 1
if __name__ == "__main__":
main()
sys.exit(0) # This is pretty much unnecessary, though
A script that would take a random time to process and that can, potentially, be quite long (at least long enough to demonstrate)
Now, we could create one (or many) subprocesses in a tread that run that script.py file, regularly check their status (using poll()) and if the user has requested the forced output send a TERM signal and a bit later a KILL if necessary.
import threading
import time
import subprocess
def thread_do_downloads(stop_activated):
p = subprocess.Popen('./script.py', stdout=subprocess.PIPE)
while p.poll() is None:
time.sleep(0.5)
print("Subprocess still running... Slepping a bit... ZzzzzzZZZ")
if stop_activated.is_set():
print("Forcing output requested!!!")
print("Trying to terminate the process nicely, which a SIGTERM:")
p.terminate()
time.sleep(0.5)
if p.poll() is None:
print("Not being nice anymore... Die, die die!!")
p.kill()
print("This is what the subprocess 'said':\n%s" % p.stdout.read())
return
print("stopping normally")
def do_stuff_with_downloaded_data():
print("doing stuff with downloaded data")
def listen_for_keypress(stop_activated):
input("Press ENTER to stop downloading.")
print("keypress intercepted")
stop_activated.set()
def main():
stop_activated = threading.Event()
dl = threading.Thread(target=thread_do_downloads, args=(stop_activated,))
dl.start()
kill_listener = threading.Thread(target=listen_for_keypress, args=(stop_activated,), daemon=True)
kill_listener.start()
dl.join()
print("Finished downloading data")
# here are some lines to make sure the download thread above completes gracefully
do_stuff_with_downloaded_data()
print("All done")
if __name__ == '__main__':
main()
I'm trying to create a script in Python. The idea is to start 3 processes, 2 of them constantly print a message, and the third is there to kill them after a few seconds. The problem is that I don't know how to tell that third which processes should be terminated.
from multiprocessing import *
import time
def OkreciLevi():
while 1:
print "okrecem levi"
time.sleep(3)
def OkreciDesni():
while 1:
print "okrecem desni"
time.sleep(3)
def Koci(levi,desni):
for vrednost in range(2):
print str(vrednost)
time.sleep(3)
levi.terminate()
desni.terminate()
print "kocim"
if __name__== '__main__':
levi=Process(target=OkreciLevi)
desni=Process(target=OkreciDesni)
koci=Process(target=Koci, args=(levi,desni))
koci.start()
levi.start()
desni.start()
levi.join()
desni.join()
koci.join()
Assuming that you're on *nix-like operating system I guess that you need to:
Get the PID of the multiprocessing worker;
Send SIGTERM to them. For instanse use os.kill.
Also this information may be useful for you.
I have a piece of code below that creates a few threads to perform a task, which works perfectly well on its own. However I'm struggling to understand why the print statements I call in my function do not execute until all threads complete and the print 'finished' statement is called. I would expect them to be called as the thread executes. Is there any simple way to accomplish this, and why does this work this way in the first place?
def func(param):
time.sleep(.25)
print param*2
if __name__ == '__main__':
print 'starting execution'
launchTime = time.clock()
params = range(10)
pool=multiprocessing.Pool(processes=100) #use N processes to download the data
_=pool.map(func,params)
print 'finished'
For python 3 you can now use the flush param like that:
print('Your text', flush=True)
This happens due to stdout buffering. You still can flush the buffers:
import sys
print 'starting'
sys.stdout.flush()
You can find more info on this issue here and here.
Having run into plenty of issues around this and garbled outputs (especially under Windows when adding colours to the output..), my solution has been to have an exclusive printing thread which consumes a queue
If this still doesn't work, also add flush=True to your print statement(s) as suggested by #Or Duan
Further, you may find the "most correct", but a heavy-handed approach to displaying messages with threading is to use the logging library which can wrap a queue (and write to many places asynchronously, including stdout) or write to a system-level queue (outside Python; availability depends greatly on OS support)
import threading
from queue import Queue
def display_worker(display_queue):
while True:
line = display_queue.get()
if line is None: # simple termination logic, other sentinels can be used
break
print(line, flush=True) # remove flush if slow or using Python2
def some_other_worker(display_queue, other_args):
# NOTE accepts queue reference as an argument, though it could be a global
display_queue.put("something which should be printed from this thread")
def main():
display_queue = Queue() # synchronizes console output
screen_printing_thread = threading.Thread(
target=display_worker,
args=(display_queue,),
)
screen_printing_thread.start()
### other logic ###
display_queue.put(None) # end screen_printing_thread
screen_printing_thread.stop()
I have a piece of code below that creates a few threads to perform a task, which works perfectly well on its own. However I'm struggling to understand why the print statements I call in my function do not execute until all threads complete and the print 'finished' statement is called. I would expect them to be called as the thread executes. Is there any simple way to accomplish this, and why does this work this way in the first place?
def func(param):
time.sleep(.25)
print param*2
if __name__ == '__main__':
print 'starting execution'
launchTime = time.clock()
params = range(10)
pool=multiprocessing.Pool(processes=100) #use N processes to download the data
_=pool.map(func,params)
print 'finished'
For python 3 you can now use the flush param like that:
print('Your text', flush=True)
This happens due to stdout buffering. You still can flush the buffers:
import sys
print 'starting'
sys.stdout.flush()
You can find more info on this issue here and here.
Having run into plenty of issues around this and garbled outputs (especially under Windows when adding colours to the output..), my solution has been to have an exclusive printing thread which consumes a queue
If this still doesn't work, also add flush=True to your print statement(s) as suggested by #Or Duan
Further, you may find the "most correct", but a heavy-handed approach to displaying messages with threading is to use the logging library which can wrap a queue (and write to many places asynchronously, including stdout) or write to a system-level queue (outside Python; availability depends greatly on OS support)
import threading
from queue import Queue
def display_worker(display_queue):
while True:
line = display_queue.get()
if line is None: # simple termination logic, other sentinels can be used
break
print(line, flush=True) # remove flush if slow or using Python2
def some_other_worker(display_queue, other_args):
# NOTE accepts queue reference as an argument, though it could be a global
display_queue.put("something which should be printed from this thread")
def main():
display_queue = Queue() # synchronizes console output
screen_printing_thread = threading.Thread(
target=display_worker,
args=(display_queue,),
)
screen_printing_thread.start()
### other logic ###
display_queue.put(None) # end screen_printing_thread
screen_printing_thread.stop()
How can I exit my entire Python application from one of its threads? sys.exit() only terminates the thread in which it is called, so that is no help.
I would not like to use an os.kill() solution, as this isn't very clean.
Short answer: use os._exit.
Long answer with example:
I yanked and slightly modified a simple threading example from a tutorial on DevShed:
import threading, sys, os
theVar = 1
class MyThread ( threading.Thread ):
def run ( self ):
global theVar
print 'This is thread ' + str ( theVar ) + ' speaking.'
print 'Hello and good bye.'
theVar = theVar + 1
if theVar == 4:
#sys.exit(1)
os._exit(1)
print '(done)'
for x in xrange ( 7 ):
MyThread().start()
If you keep sys.exit(1) commented out, the script will die after the third thread prints out. If you use sys.exit(1) and comment out os._exit(1), the third thread does not print (done), and the program runs through all seven threads.
os._exit "should normally only be used in the child process after a fork()" -- and a separate thread is close enough to that for your purpose. Also note that there are several enumerated values listed right after os._exit in that manual page, and you should prefer those as arguments to os._exit instead of simple numbers like I used in the example above.
If all your threads except the main ones are daemons, the best approach is generally thread.interrupt_main() -- any thread can use it to raise a KeyboardInterrupt in the main thread, which can normally lead to reasonably clean exit from the main thread (including finalizers in the main thread getting called, etc).
Of course, if this results in some non-daemon thread keeping the whole process alive, you need to followup with os._exit as Mark recommends -- but I'd see that as the last resort (kind of like a kill -9;-) because it terminates things quite brusquely (finalizers not run, including try/finally blocks, with blocks, atexit functions, etc).
Using thread.interrupt_main() may not help in some situation. KeyboardInterrupts are often used in command line applications to exit the current command or to clean the input line.
In addition, os._exit will kill the process immediately without running any finally blocks in your code, which may be dangerous (files and connections will not be closed for example).
The solution I've found is to register a signal handler in the main thread that raises a custom exception. Use the background thread to fire the signal.
import signal
import os
import threading
import time
class ExitCommand(Exception):
pass
def signal_handler(signal, frame):
raise ExitCommand()
def thread_job():
time.sleep(5)
os.kill(os.getpid(), signal.SIGUSR1)
signal.signal(signal.SIGUSR1, signal_handler)
threading.Thread(target=thread_job).start() # thread will fire in 5 seconds
try:
while True:
user_input = raw_input('Blocked by raw_input loop ')
# do something with 'user_input'
except ExitCommand:
pass
finally:
print('finally will still run')
Related questions:
Why does sys.exit() not exit when called inside a thread in Python?
Python: How to quit CLI when stuck in blocking raw_input?
The easiest way to exit the whole program is, we should terminate the program by using the process id (pid).
import os
import psutil
current_system_pid = os.getpid()
ThisSystem = psutil.Process(current_system_pid)
ThisSystem.terminate()
To install psutl:- "pip install psutil"
For Linux you can use the kill() command and pass the current process' ID and the SIGINT signal to start the steps to exit the app.
import signal
os.kill(os.getpid(), signal.SIGINT)