I am having the Python Multi-threaded program as below. If I press ctrl+c within 5 seconds (approx), It is going inside the KeyboardInterrupt exception.
Running the code longer than 15 seconds failed to respond to ctrl+c. If I press ctrl+c after 15 seconds, It is not working. It is not throwing KeyboardInterrupt exception. What could be the reason ? I tested this on Linux.
#!/usr/bin/python
import os, sys, threading, time
class Worker(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
# A flag to notify the thread that it should finish up and exit
self.kill_received = False
def run(self):
while not self.kill_received:
self.do_something()
def do_something(self):
[i*i for i in range(10000)]
time.sleep(1)
def main(args):
threads = []
for i in range(10):
t = Worker()
threads.append(t)
t.start()
while len(threads) > 0:
try:
# Join all threads using a timeout so it doesn't block
# Filter out threads which have been joined or are None
threads = [t.join(1) for t in threads if t is not None and t.isAlive()]
except KeyboardInterrupt:
print "Ctrl-c received! Sending kill to threads..."
for t in threads:
t.kill_received = True
if __name__ == '__main__':
main(sys.argv)
After the first execution of
threads = [t.join(1) for t in threads if t is not None and t.isAlive()]
your variable threads contains
[None, None, None, None, None, None, None, None, None, None]
after the second execution, the same variable threads contains:
[]
At this point, len(threads) > 0 is False and you get out of the while loop. Your script is still running since you have 10 threads still active, but since you're not anymore in your try / except block (to catch KeyboardInterrupt), you can't stop using Ctrl + C
Add some prints to your script to see what I described:
#!/usr/bin/python
import os, sys, threading, time
class Worker(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
# A flag to notify the thread that it should finish up and exit
self.kill_received = False
def run(self):
while not self.kill_received:
self.do_something()
def do_something(self):
[i*i for i in range(10000)]
time.sleep(1)
def main(args):
threads = []
for i in range(10):
t = Worker()
threads.append(t)
t.start()
print('thread {} started'.format(i))
while len(threads) > 0:
print('Before joining')
try:
# Join all threads using a timeout so it doesn't block
# Filter out threads which have been joined or are None
threads = [t.join(1) for t in threads if t is not None and t.isAlive()]
print('After join() on threads: threads={}'.format(threads))
except KeyboardInterrupt:
print("Ctrl-c received! Sending kill to threads...")
for t in threads:
t.kill_received = True
print('main() execution is now finished...')
if __name__ == '__main__':
main(sys.argv)
And the result:
$ python thread_test.py
thread 0 started
thread 1 started
thread 2 started
thread 3 started
thread 4 started
thread 5 started
thread 6 started
thread 7 started
thread 8 started
thread 9 started
Before joining
After join() on threads: threads=[None, None, None, None, None, None, None, None, None, None]
Before joining
After join() on threads: threads=[]
main() execution is now finished...
Actually, Ctrl + C doesn't stop to work after 15 seconds, but after 10 or 11 seconds. This is the time needed to create and start the 10 threads (less than a second) and to execute join(1) on each thread (about 10 seconds).
Hint from the doc:
As join() always returns None, you must call isAlive() after join() to decide whether a timeout happened – if the thread is still alive, the join() call timed out.
to follow up on the poster above, isAlive() got renamed to is_alive()
tried on Python 3.9.6
full code:
#!/usr/bin/python
import os, sys, threading, time
class Worker(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
# A flag to notify the thread that it should finish up and exit
self.kill_received = False
def run(self):
while not self.kill_received:
self.do_something()
def do_something(self):
[i*i for i in range(10000)]
time.sleep(1)
def main(args):
threads = []
for i in range(10):
t = Worker()
threads.append(t)
t.start()
print('thread {} started'.format(i))
while len(threads) > 0:
print('Before joining')
try:
# Join all threads using a timeout so it doesn't block
# Filter out threads which have been joined or are None
threads = [t.join(1) for t in threads if t is not None and t.is_alive()]
print('After join() on threads: threads={}'.format(threads))
except KeyboardInterrupt:
print("Ctrl-c received! Sending kill to threads...")
for t in threads:
t.kill_received = True
print('main() execution is now finished...')
if __name__ == '__main__':
main(sys.argv)
Related
I recently started studying threads in python, and I ran into this problem: I need the "two" function to finish executing after executing the function one in the thread, but the join method does not work, apparently because of the while true loop in the third function. I tried using queue, but it didn't work either.
the code itself:
from threading import Thread,Event
def one():
event.set()
thr.join()
for i in range(3):
print('some print')
time.sleep(1)
def two():
t = Thread(target=one)
t.start()
#with t.join() here the program does not work at all, same thing with event.set()
print('func two finished')
def three(callback, event):
c = 0
while True:
c += 1
time.sleep(1)
print('func 3 is working')
if c == 5:
two()
if event.is_set():
callback(c)
print('func 3 is stopped')
break
def callback(t):
print('callback ',t)
def thread(callback):
global event, thr
event = Event()
thr = Thread(target=three, args=(callback, event,))
thr.start()
thr.join()
thread(callback)
current output:
func 3 is working
func 3 is working
func 3 is working
func 3 is working
func 3 is working
func two finished
callback 5
func 3 is stopped
some print
some print
some print
expected:
func 3 is working
func 3 is working
func 3 is working
func 3 is working
func 3 is working
callback 5
func 3 is stopped
some print
some print
some print
func two finished
After running the code I understand by "not working" you mean the program finished before all prints are printed.
The reason is that you join the thr thread twice, one of them by the main thread.
The sequence of return of join is not guaranteed.
When the main thread finished, all threads created by the program also finish, so they terminated no matter what.
Same thing when setting the event, it makes the main thread exit and kill the remaining threads.
To do what you intend, you should wait for the one thread in the main thread.
from threading import Thread,Event
def one():
event.set()
thr.join()
for i in range(3):
print('some print')
time.sleep(1)
def two():
t = Thread(target=one)
t.start()
#with t.join() here the program does not work at all, same thing with event.set()
print('func two finished')
def three(callback, event):
c = 0
while True:
c += 1
time.sleep(1)
print('func 3 is working')
if c == 5:
two()
if event.is_set():
callback(c)
print('func 3 is stopped')
break
def callback(t):
print('callback ',t)
def thread(callback):
global event, thr
event = Event()
thr = Thread(target=three, args=(callback, event,))
thr.start()
thr.join()
thread(callback)
Note that as other said, this might be nice for learning purpesses but has to be modified if you want to actually use it in real life code.
Your program creates a deadlock if you un-comment that t.join() call in function two;
The thr thread cannot finish until after the t thread has finished because the thr thread calls t.join() in function two.
The t thread cannot finish until after the thr thread has finished because the t thread calls thr.join() in function one.
Neither thread can finish until after the other thread finishes. Therefore, neither thread can ever finish.
Why does one join the thr thread?
def one():
event.set()
thr.join() # What were you trying to do here?
for i in range(3):
print('some print')
time.sleep(1)
Your program will give the output you wanted if you comment out that join call, and uncomment the t.join() call in function two.
The sequence you need is obtained by a small permutation of commands. But it is not clear why you need threads if everything is done sequentially.
from threading import Thread, Event
import time
def one(event):
event.set()
for i in range(3):
print('some print')
time.sleep(1)
def two(event):
t = Thread(target=one, args=(event,))
t.start()
t.join()
print('func two finished')
def three(event):
c = 0
while True:
c += 1
time.sleep(1)
print('func 3 is working')
if c == 5:
callback(c)
print('func 3 is stopped')
two(event)
break
def callback(t):
print('callback ', t)
def thread():
event = Event()
thr = Thread(target=three, args=(event,))
thr.start()
thread()
--------------------------------
func 3 is working
func 3 is working
func 3 is working
func 3 is working
func 3 is working
callback 5
func 3 is stopped
some print
some print
some print
func two finished
This is a comment, not an answer.
This makes no sense:
t = Thread(target=one, args=(event,))
t.start()
t.join()
There's no point in starting a new thread if you aren't going to do something concurrently with the thread. Either do this,
t = Thread(target=one, args=(event,))
t.start()
do_something_else_concurrently_with_thread_t(...)
t.join()
Or just just call the function instead of creating a new thread to call it:
one(event)
If you don't want concurrency, then you don't need threads.
I am confused for showing waiting message.
For my program,
Step 1, it keep looping to show the waiting message.
Step 2, If the trigger_file exist, it stop the waiting message and run main_process()
Step 3, After finishing main_process, it show the waiting message again.
I tried to use variable waiting to stop the waiting message but it is not working
I am not sure how to use async/await function and multithreadubg for this case.
Thank you
import os
import time
import threading
waiting = True
trigger_file = r'D:\Desktop\OK'
def main_process():
print('1')
time.sleep(5)
print('2')
time.sleep(5)
print('3')
time.sleep(5)
print('4')
time.sleep(5)
print('5')
def print_waiting(): # animation of waiting
while(waiting):
for loading_symbol in ['|','/','-','\\','|','/','-','\\','|']:
print('\r[INFO] Waiting for trigger... '+loading_symbol,end="")
time.sleep(0.2)
def triggerListener(): # trigger a function if the file exist
while(True):
if os.path.exists(trigger_file):
global waiting
waiting=False
print('\n[INFO] main process start')
main_process()
waiting=True
if __name__ == "__main__":
# creating thread
t1 = threading.Thread(target=print_waiting)
t2 = threading.Thread(target=triggerListener)
# starting thread 1
t1.start()
# starting thread 2
t2.start()
# wait until thread 1 is completely executed
t1.join()
# wait until thread 2 is completely executed
t2.join()
# both threads completely executed
print("Done!")
Expected Output:
[INFO] Waiting for trigger... -
[INFO] main process start
1
2
3
4
5
[INFO] Waiting for trigger... -
Finally, I use threading.Lock to solve the problem. acquire the lock and release the lock after finishing the function.
class Print_waiting(threading.Thread):
def __init__(self,lock):
threading.Thread.__init__(self)
self.running = True
self.lock = lock
def run(self): # animation of waiting
while self.running:
for loading_symbol in ['|','/','-','\\','|','/','-','\\','|']:
self.lock.acquire()
print('\r[INFO] Waiting for trigger... '+loading_symbol,end="")
self.lock.release()
time.sleep(0.2)
def stop(self):
self.running = False
class TriggerListener(threading.Thread):
def __init__(self,lock):
threading.Thread.__init__(self)
self.running = True
self.lock = lock # for mutex
def run(self): # trigger a function if the file exist
while(self.running):
if os.path.exists(trigger_file):
self.lock.acquire()
print('\n[INFO] main process start')
Program.main()
self.lock.release()
def stop(self):
self.running = False
if __name__ == "__main__":
lock = threading.Lock()
waiting_anime = Print_waiting(lock)
Trigger_RPA = TriggerListener(lock)
Trigger_RPA.start()
waiting_anime.start()
Trigger_RPA.join()
waiting_anime.join()
I try threading in python. I have some code and hear that my program is waiting for threads if i use the .join method. But in the following code I get the print 'done' earlier then the prints in my thread. But why?
def getresults(seed):
print("get results now")
results[seed]
i = 0
threads = []
for suggestengine in suggestengines.keys():
i += 1
t = threading.Thread(target=getSuggestengineResult, args = (suggestengine, seed, i))
threads.append(t)
print('threads initialized')
for thread in threads:
thread.start()
for thread in threads:
thread.join
print('done')
I am building a multi threading application.
I have setup a threadPool.
[ A Queue of size N and N Workers that get data from the queue]
When all tasks are done I use
tasks.join()
where tasks is the queue .
The application seems to run smoothly until suddently at some point (after 20 minutes in example) it terminates with the error
thread.error: can't start new thread
Any ideas?
Edit: The threads are daemon Threads and the code is like:
while True:
t0 = time.time()
keyword_statuses = DBSession.query(KeywordStatus).filter(KeywordStatus.status==0).options(joinedload(KeywordStatus.keyword)).with_lockmode("update").limit(100)
if keyword_statuses.count() == 0:
DBSession.commit()
break
for kw_status in keyword_statuses:
kw_status.status = 1
DBSession.commit()
t0 = time.time()
w = SWorker(threads_no=32, network_server='http://192.168.1.242:8180/', keywords=keyword_statuses, cities=cities, saver=MySqlRawSave(DBSession), loglevel='debug')
w.work()
print 'finished'
When the daemon threads are killed?
When the application finishes or when the work() finishes?
Look at the thread pool and the worker (it's from a recipe )
from Queue import Queue
from threading import Thread, Event, current_thread
import time
event = Event()
class Worker(Thread):
"""Thread executing tasks from a given tasks queue"""
def __init__(self, tasks):
Thread.__init__(self)
self.tasks = tasks
self.daemon = True
self.start()
def run(self):
'''Start processing tasks from the queue'''
while True:
event.wait()
#time.sleep(0.1)
try:
func, args, callback = self.tasks.get()
except Exception, e:
print str(e)
return
else:
if callback is None:
func(args)
else:
callback(func(args))
self.tasks.task_done()
class ThreadPool:
"""Pool of threads consuming tasks from a queue"""
def __init__(self, num_threads):
self.tasks = Queue(num_threads)
for _ in range(num_threads): Worker(self.tasks)
def add_task(self, func, args=None, callback=None):
''''Add a task to the queue'''
self.tasks.put((func, args, callback))
def wait_completion(self):
'''Wait for completion of all the tasks in the queue'''
self.tasks.join()
def broadcast_block_event(self):
'''blocks running threads'''
event.clear()
def broadcast_unblock_event(self):
'''unblocks running threads'''
event.set()
def get_event(self):
'''returns the event object'''
return event
ALSo maybe the problem it's because I create SWorker objects in a loop?
What happens with the old SWorker (garbage collection ?) ?
There is still not enough code for localize the problem, but I'm sure that this is because you don't utilize the threads and start too much of them. Did you see canonical example from Queue python documentation http://docs.python.org/library/queue.html (bottom of the page)?
I can reproduce your problem with the following code:
import threading
import Queue
q = Queue.Queue()
def worker():
item = q.get(block=True) # sleeps forever for now
do_work(item)
q.task_done()
# create infinite number of workers threads and fails
# after some time with "error: can't start new thread"
while True:
t = threading.Thread(target=worker)
t.start()
q.join() # newer reached this
Instead you must create the poll of threads with known number of threads and put your data to queue like:
q = Queue()
def worker():
while True:
item = q.get()
do_work(item)
q.task_done()
for i in range(num_worker_threads):
t = Thread(target=worker)
t.daemon = True
t.start()
for item in source():
q.put(item)
q.join() # block until all tasks are done
UPD: In case you need to stop some thread, you can add a flag to it or send a special mark means "stop" for break while loop:
class Worker(Thread):
break_msg = object() # just uniq mark sign
def __init__(self):
self.continue = True
def run():
while self.continue: # can stop and destroy thread, (var 1)
msg = queue.get(block=True)
if msg == self.break_msg:
return # will stop and destroy thread (var 2)
do_work()
queue.task_done()
workers = [Worker() for _ in xrange(num_workers)]
for w in workers:
w.start()
for task in tasks:
queue.put(task)
for _ in xrange(num_workers):
queue.put(Worker.break_msg) # stop thread after all tasks done. Need as many messages as many threads you have
OR
queue.join() # wait until all tasks done
for w in workers:
w.continue = False
w.put(None)
Why doesn't this code "act" threaded? (Please see the output.)
import time
from threading import Thread
def main():
for nums in [range(0,5), range(5,10)]:
t = Spider(nums)
t.start()
print 'started a thread'
t.join()
print "done"
class Spider(Thread):
def __init__(self, nums):
Thread.__init__(self)
self.nums = nums
def run(self): # this is an override
for num in self.nums:
time.sleep(3) # or do something that takes a while
print 'finished %s' % (num, )
if __name__ == '__main__':
main()
Output:
started a thread
finished 0
finished 1
finished 2
finished 3
finished 4
started a thread
finished 5
finished 6
finished 7
finished 8
finished 9
done
When you say t.join(), you're telling it to wait for the thread to end.
This means, you're asking it to make a thread, start it, then wait for the thread to end before making a new one.
If you want it to act multithreaded, you'll need to move the join()s outside of the loop.
def main():
# We will store the running threads in this
threads = []
# Start the threads
for nums in [range(0,5), range(5,10)]:
t = Spider(nums)
t.start()
print 'started a thread'
threads.append(t)
# All the threads have been started
# Now we wait for them to finish
for t in threads:
t.join()
print "done"
See also:
Documentation of Thread.join()
Your Thread join t.join blocks the main thread until the thread completes execution ( http://docs.python.org/library/threading.html#threading.Thread.join ). Change your code to look something like this:
def main():
threads = []
for nums in [range(0,5), range(5,10)]:
t = Spider(nums)
t.start()
print 'started a thread'
threads.append(t)
for t in threads: t.join()
print "done"
You need to start both the threads first, and then join with them once they are both running.