my tool stops randomly and it seems like all threads are 'ghosts'.
How does it work:
The tool loops until the max number of allowed threads at the same time are running, in this case 20. When a thread finishes it starts the next one.
Problem:
After like an hour of doing this, the tool is stuck at 20 Threads running but nothing happens anymore.
Thanks in advance everyone!
maxthreadcount = 20
while True:
if threading.active_count() < maxthreadcount:
threading.Thread(target=Dealer).start()
Dealer:
def Dealer():
print("thread started")
return
You need to terminate previously created threads after their job (print command in this case) is done.
Take a look at this example from this article:
class CountdownTask:
def __init__(self):
self._running = True
def terminate(self):
self._running = False
def run(self, n):
while self._running and n > 0:
print('T-minus', n)
n -= 1
time.sleep(5)
c = CountdownTask()
t = Thread(target = c.run, args =(10, ))
t.start()
...
# Signal termination
c.terminate()
# Wait for actual termination (if needed)
t.join()
I think you should call self.terminate() after doing the print. Something like below:
class Dealer():
def __init__(self):
self._running = True
def run(self):
print("thread started")
return self.terminate()
def terminate(self):
self._running = False
Edit
I also believe you can make use of python's ThreadPool to this extent. Instead of spawning threads yourself, you might be able to reuse threads after their assigned task is over, for the new tasks.
Related
How can I start and stop a thread with my poor thread class?
It is in loop, and I want to restart it again at the beginning of the code. How can I do start-stop-restart-stop-restart?
My class:
import threading
class Concur(threading.Thread):
def __init__(self):
self.stopped = False
threading.Thread.__init__(self)
def run(self):
i = 0
while not self.stopped:
time.sleep(1)
i = i + 1
In the main code, I want:
inst = Concur()
while conditon:
inst.start()
# After some operation
inst.stop()
# Some other operation
You can't actually stop and then restart a thread since you can't call its start() method again after its run() method has terminated. However you can make one pause and then later resume its execution by using a threading.Condition variable to avoid concurrency problems when checking or changing its running state.
threading.Condition objects have an associated threading.Lock object and methods to wait for it to be released and will notify any waiting threads when that occurs. Here's an example derived from the code in your question which shows this being done. In the example code I've made the Condition variable a part of Thread subclass instances to better encapsulate the implementation and avoid needing to introduce additional global variables:
from __future__ import print_function
import threading
import time
class Concur(threading.Thread):
def __init__(self):
super(Concur, self).__init__()
self.iterations = 0
self.daemon = True # Allow main to exit even if still running.
self.paused = True # Start out paused.
self.state = threading.Condition()
def run(self):
self.resume()
while True:
with self.state:
if self.paused:
self.state.wait() # Block execution until notified.
# Do stuff...
time.sleep(.1)
self.iterations += 1
def pause(self):
with self.state:
self.paused = True # Block self.
def resume(self):
with self.state:
self.paused = False
self.state.notify() # Unblock self if waiting.
class Stopwatch(object):
""" Simple class to measure elapsed times. """
def start(self):
""" Establish reference point for elapsed time measurements. """
self.start_time = time.time()
return self
#property
def elapsed_time(self):
""" Seconds since started. """
try:
return time.time() - self.start_time
except AttributeError: # Wasn't explicitly started.
self.start_time = time.time()
return 0
MAX_RUN_TIME = 5 # Seconds.
concur = Concur()
stopwatch = Stopwatch()
print('Running for {} seconds...'.format(MAX_RUN_TIME))
concur.start()
while stopwatch.elapsed_time < MAX_RUN_TIME:
concur.resume()
# Can also do other concurrent operations here...
concur.pause()
# Do some other stuff...
# Show Concur thread executed.
print('concur.iterations: {}'.format(concur.iterations))
This is David Heffernan's idea fleshed-out. The example below runs for 1 second, then stops for 1 second, then runs for 1 second, and so on.
import time
import threading
import datetime as DT
import logging
logger = logging.getLogger(__name__)
def worker(cond):
i = 0
while True:
with cond:
cond.wait()
logger.info(i)
time.sleep(0.01)
i += 1
logging.basicConfig(level=logging.DEBUG,
format='[%(asctime)s %(threadName)s] %(message)s',
datefmt='%H:%M:%S')
cond = threading.Condition()
t = threading.Thread(target=worker, args=(cond, ))
t.daemon = True
t.start()
start = DT.datetime.now()
while True:
now = DT.datetime.now()
if (now-start).total_seconds() > 60: break
if now.second % 2:
with cond:
cond.notify()
The implementation of stop() would look like this:
def stop(self):
self.stopped = True
If you want to restart, then you can just create a new instance and start that.
while conditon:
inst = Concur()
inst.start()
#after some operation
inst.stop()
#some other operation
The documentation for Thread makes it clear that the start() method can only be called once for each instance of the class.
If you want to pause and resume a thread, then you'll need to use a condition variable.
This simple code example:
import threading
import time
class Monitor():
def __init__(self):
self.stop = False
self.blocked_emails = []
def start_monitor(self):
print("Run start_monitor")
rows = []
while not self.stop:
self.check_rows(rows)
print("inside while")
time.sleep(1)
def check_rows(self, rows):
print('check_rows')
def stop_monitoring(self):
print("Run stop_monitoring")
self.stop = True
if __name__ == '__main__':
monitor = Monitor()
b = threading.Thread(name='background_monitor', target=monitor.start_monitor())
b.start()
b.join()
for i in range(0, 10):
time.sleep(2)
print('Wait 2 sec.')
monitor.stop_monitoring()
How can I run background thread, in mine case background_monitor without blocking main thread?
I wanted to background_monitor thread stopped on after stop_monitoring will be called
I mine example, the for loop from main thread never called and the background is running forever.
There are two issues with your current code. Firstly, you're calling monitor.start_monitor on this line, whereas according to the docs
target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called
This means that you need to pass it as a function rather than calling it. To fix this, you should change the line
b = threading.Thread(name='background_monitor', target=monitor.start_monitor())
to
b = threading.Thread(name='background_monitor', target=monitor.start_monitor)
which passes the function as an argument.
Secondly, you use b.join() before stopping the thread, which waits for the second thread to finish before continuing. Instead, you should place that below the monitor.stop_monitoring().
The corrected code looks like this:
import threading
import time
class Monitor():
def __init__(self):
self.stop = False
self.blocked_emails = []
def start_monitor(self):
print("Run start_monitor")
rows = []
while not self.stop:
self.check_rows(rows)
print("inside while")
time.sleep(1)
def check_rows(self, rows):
print('check_rows')
def stop_monitoring(self):
print("Run stop_monitoring")
self.stop = True
if __name__ == '__main__':
monitor = Monitor()
b = threading.Thread(name='background_monitor', target=monitor.start_monitor)
b.start()
for i in range(0, 10):
time.sleep(2)
print('Wait 2 sec.')
monitor.stop_monitoring()
b.join()
I want to limit the number of active threads. What i have seen is, that a finished thread stays alive and does not exit itself, so the number of active threads keep growing until an error occours.
The following code starts only 8 threads at a time but they stay alive even when they finished. So the number keeps growing:
class ThreadEx(threading.Thread):
__thread_limiter = None
__max_threads = 2
#classmethod
def max_threads(cls, thread_max):
ThreadEx.__max_threads = thread_max
ThreadEx.__thread_limiter = threading.BoundedSemaphore(value=ThreadEx.__max_threads)
def __init__(self, target=None, args:tuple=()):
super().__init__(target=target, args=args)
if not ThreadEx.__thread_limiter:
ThreadEx.__thread_limiter = threading.BoundedSemaphore(value=ThreadEx.__max_threads)
def run(self):
ThreadEx.__thread_limiter.acquire()
try:
#success = self._target(*self._args)
#if success: return True
super().run()
except:
pass
finally:
ThreadEx.__thread_limiter.release()
def call_me(test1, test2):
print(test1 + test2)
time.sleep(1)
ThreadEx.max_threads(8)
for i in range(0, 99):
t = ThreadEx(target=call_me, args=("Thread count: ", str(threading.active_count())))
t.start()
Due to the for loop, the number of threads keep growing to 99.
I know that a thread has done its work because call_me has been executed and threading.active_count() was printed.
Does somebody know how i make sure, a finished thread does not stay alive?
This may be a silly answer but to me it looks you are trying to reinvent ThreadPool.
from multiprocessing.pool import ThreadPool
from time import sleep
p = ThreadPool(8)
def call_me(test1):
print(test1)
sleep(1)
for i in range(0, 99):
p.apply_async(call_me, args=(i,))
p.close()
p.join()
This will ensure only 8 concurrent threads are running your function at any point of time. And if you want a bit more performance, you can import Pool from multiprocessing and use that. The interface is exactly the same but your pool will now be subprocesses instead of threads, which usually gives a performance boost as GIL does not come in the way.
I have changed the class according to the help of Hannu.
I post it for reference, maybe it's useful for others that come across this post:
import threading
from multiprocessing.pool import ThreadPool
import time
class MultiThread():
__thread_pool = None
#classmethod
def begin(cls, max_threads):
MultiThread.__thread_pool = ThreadPool(max_threads)
#classmethod
def end(cls):
MultiThread.__thread_pool.close()
MultiThread.__thread_pool.join()
def __init__(self, target=None, args:tuple=()):
self.__target = target
self.__args = args
def run(self):
try:
result = MultiThread.__thread_pool.apply_async(self.__target, args=self.__args)
return result.get()
except:
pass
def call_me(test1, test2):
print(test1 + test2)
time.sleep(1)
return 0
MultiThread.begin(8)
for i in range(0, 99):
t = MultiThread(target=call_me, args=("Thread count: ", str(threading.active_count())))
t.run()
MultiThread.end()
The maximum of threads is 8 at any given time determined by the method begin.
And also the method run returns the result of your passed function if it returns something.
Hope that helps.
How can I start and stop a thread with my poor thread class?
It is in loop, and I want to restart it again at the beginning of the code. How can I do start-stop-restart-stop-restart?
My class:
import threading
class Concur(threading.Thread):
def __init__(self):
self.stopped = False
threading.Thread.__init__(self)
def run(self):
i = 0
while not self.stopped:
time.sleep(1)
i = i + 1
In the main code, I want:
inst = Concur()
while conditon:
inst.start()
# After some operation
inst.stop()
# Some other operation
You can't actually stop and then restart a thread since you can't call its start() method again after its run() method has terminated. However you can make one pause and then later resume its execution by using a threading.Condition variable to avoid concurrency problems when checking or changing its running state.
threading.Condition objects have an associated threading.Lock object and methods to wait for it to be released and will notify any waiting threads when that occurs. Here's an example derived from the code in your question which shows this being done. In the example code I've made the Condition variable a part of Thread subclass instances to better encapsulate the implementation and avoid needing to introduce additional global variables:
from __future__ import print_function
import threading
import time
class Concur(threading.Thread):
def __init__(self):
super(Concur, self).__init__()
self.iterations = 0
self.daemon = True # Allow main to exit even if still running.
self.paused = True # Start out paused.
self.state = threading.Condition()
def run(self):
self.resume()
while True:
with self.state:
if self.paused:
self.state.wait() # Block execution until notified.
# Do stuff...
time.sleep(.1)
self.iterations += 1
def pause(self):
with self.state:
self.paused = True # Block self.
def resume(self):
with self.state:
self.paused = False
self.state.notify() # Unblock self if waiting.
class Stopwatch(object):
""" Simple class to measure elapsed times. """
def start(self):
""" Establish reference point for elapsed time measurements. """
self.start_time = time.time()
return self
#property
def elapsed_time(self):
""" Seconds since started. """
try:
return time.time() - self.start_time
except AttributeError: # Wasn't explicitly started.
self.start_time = time.time()
return 0
MAX_RUN_TIME = 5 # Seconds.
concur = Concur()
stopwatch = Stopwatch()
print('Running for {} seconds...'.format(MAX_RUN_TIME))
concur.start()
while stopwatch.elapsed_time < MAX_RUN_TIME:
concur.resume()
# Can also do other concurrent operations here...
concur.pause()
# Do some other stuff...
# Show Concur thread executed.
print('concur.iterations: {}'.format(concur.iterations))
This is David Heffernan's idea fleshed-out. The example below runs for 1 second, then stops for 1 second, then runs for 1 second, and so on.
import time
import threading
import datetime as DT
import logging
logger = logging.getLogger(__name__)
def worker(cond):
i = 0
while True:
with cond:
cond.wait()
logger.info(i)
time.sleep(0.01)
i += 1
logging.basicConfig(level=logging.DEBUG,
format='[%(asctime)s %(threadName)s] %(message)s',
datefmt='%H:%M:%S')
cond = threading.Condition()
t = threading.Thread(target=worker, args=(cond, ))
t.daemon = True
t.start()
start = DT.datetime.now()
while True:
now = DT.datetime.now()
if (now-start).total_seconds() > 60: break
if now.second % 2:
with cond:
cond.notify()
The implementation of stop() would look like this:
def stop(self):
self.stopped = True
If you want to restart, then you can just create a new instance and start that.
while conditon:
inst = Concur()
inst.start()
#after some operation
inst.stop()
#some other operation
The documentation for Thread makes it clear that the start() method can only be called once for each instance of the class.
If you want to pause and resume a thread, then you'll need to use a condition variable.
I need to wait in a script until a certain number of conditions become true?
I know I can roll my own eventing using condition variables and friends, but I don't want to go through all the trouble of implementing it, since some object property changes come from external thread in a wrapped C++ library (Boost.Python), so I can't just hijack __setattr__ in a class and put a condition variable there, which leaves me with either trying to create and signal a Python condition variable from C++, or wrap a native one and wait on it in Python, both of which sound fiddly, needlessly complicated and boring.
Is there an easier way to do it, barring continuous polling of the condition?
Ideally it would be along the lines of
res = wait_until(lambda: some_predicate, timeout)
if (not res):
print 'timed out'
Unfortunately the only possibility to meet your constraints is to periodically poll, e.g....:
import time
def wait_until(somepredicate, timeout, period=0.25, *args, **kwargs):
mustend = time.time() + timeout
while time.time() < mustend:
if somepredicate(*args, **kwargs): return True
time.sleep(period)
return False
or the like. This can be optimized in several ways if somepredicate can be decomposed (e.g. if it's known to be an and of several clauses, especially if some of the clauses are in turn subject to optimization by being detectable via threading.Events or whatever, etc, etc), but in the general terms you ask for, this inefficient approach is the only way out.
Another nice package is waiting - https://pypi.org/project/waiting/
install:
pip install waiting
Usage:
You pass a function that will be called every time as a condition, a timeout, and (this is useful) you can pass a description for the waiting, which will be displayed if you get TimeoutError.
using function:
from waiting import wait
def is_something_ready(something):
if something.ready():
return True
return False
# wait for something to be ready
something = # whatever
wait(lambda: is_something_ready(something), timeout_seconds=120, waiting_for="something to be ready")
# this code will only execute after "something" is ready
print("Done")
Note: the function must return a boolean - True when the wait is over, False otherwise
Here is another solution. The goal was to make threads to wait on each other before doing some work in a very precise order. The work can take unknown amount of time. Constant polling is not good for two reasons: it eats CPU time and action does not start immediately after condition is met.
class Waiter():
def __init__(self, init_value):
self.var = init_value
self.var_mutex = threading.Lock()
self.var_event = threading.Event()
def WaitUntil(self, v):
while True:
self.var_mutex.acquire()
if self.var == v:
self.var_mutex.release()
return # Done waiting
self.var_mutex.release()
self.var_event.wait(1) # Wait 1 sec
def Set(self, v):
self.var_mutex.acquire()
self.var = v
self.var_mutex.release()
self.var_event.set() # In case someone is waiting
self.var_event.clear()
And the way to test it
class TestWaiter():
def __init__(self):
self.waiter = Waiter(0)
threading.Thread(name='Thread0', target=self.Thread0).start()
threading.Thread(name='Thread1', target=self.Thread1).start()
threading.Thread(name='Thread2', target=self.Thread2).start()
def Thread0(self):
while True:
self.waiter.WaitUntil(0)
# Do some work
time.sleep(np.random.rand()*2)
self.waiter.Set(1)
def Thread1(self):
while True:
self.waiter.WaitUntil(1)
# Do some work
time.sleep(np.random.rand())
self.waiter.Set(2)
def Thread2(self):
while True:
self.waiter.WaitUntil(2)
# Do some work
time.sleep(np.random.rand()/10)
self.waiter.Set(0)
Waiter for multiprocessing:
import multiprocessing as mp
import ctypes
class WaiterMP():
def __init__(self, init_value, stop_value=-1):
self.var = mp.Value(ctypes.c_int, init_value)
self.stop_value = stop_value
self.event = mp.Event()
def Terminate(self):
self.Set(self.stop_value)
def Restart(self):
self.var.value = self.init_value
def WaitUntil(self, v):
while True:
if self.var.value == v or self.var.value == self.stop_value:
return
# Wait 1 sec and check aiagn (in case event was missed)
self.event.wait(1)
def Set(self, v):
exit = self.var.value == self.stop_value
if not exit: # Do not set var if threads are exiting
self.var.value = v
self.event.set() # In case someone is waiting
self.event.clear()
Please comment if this is still not the best solution.
You've basically answered your own question: no.
Since you're dealing with external libraries in boost.python, which may change objects at their leisure, you need to either have those routines call an event handler refresh, or work with a condition.
Here is the threading extention to Alex's solution:
import time
import threading
# based on https://stackoverflow.com/a/2785908/1056345
def wait_until(somepredicate, timeout, period=0.25, *args, **kwargs):
must_end = time.time() + timeout
while time.time() < must_end:
if somepredicate(*args, **kwargs):
return True
time.sleep(period)
return False
def wait_until_par(*args, **kwargs):
t = threading.Thread(target=wait_until, args=args, kwargs=kwargs)
t.start()
print ('wait_until_par exits, thread runs in background')
def test():
print('test')
wait_until_par(test, 5)
From the computational perspective there must be a check for all conditions somewhere, sometime. If you have two parts of code, one that generates conditions changes and the other one that should be executed when some are true, you can do the following:
Have the code that changes conditions in, say, main thread, and the code that should be launched when some conditions are true, in a worker thread.
from threading import Thread,Event
locker = Event()
def WhenSomeTrue(locker):
locker.clear() # To prevent looping, see manual, link below
locker.wait(2.0) # Suspend the thread until woken up, or 2s timeout is reached
if not locker.is_set(): # when is_set() false, means timeout was reached
print('TIMEOUT')
else:
#
# Code when some conditions are true
#
worker_thread = Thread(target=WhenSomeTrue, args=(locker,))
worker_thread.start()
cond1 = False
cond2 = False
cond3 = False
def evaluate():
true_conditions = 0
for i in range(1,4):
if globals()["cond"+str(i)]: #access a global condition variable one by one
true_conditions += 1 #increment at each true value
if true_conditions > 1:
locker.set() # Resume the worker thread executing the else branch
#Or just if true_conditions > 1: locker.set();
#true_conditions would need be incremented when 'True' is written to any of those variables
#
# some condition change code
#
evaluate()
For more information concerning this method, visit: https://docs.python.org/3/library/threading.html#event-objects
Proposed solution:
def wait_until(delegate, timeout: int):
end = time.time() + timeout
while time.time() < end:
if delegate():
return True
else:
time.sleep(0.1)
return False
Usage:
wait_until(lambda: True, 2)
I once used this in my code:
while not condition:
pass
Hope this helps
In 2022 now you could use https://trio-util.readthedocs.io/en/latest/#trio_util.AsyncValue
I think this comes closest to what you want in its "smoothest" form
This worked for me
direction = ''
t = 0
while direction == '' and t <= 1:
sleep(0.1)
t += 0.1
This is for waiting for a signal while making sure time limit of 1 second
here's how:
import time
i = false
while i == false:
if (condition):
i = true
break
Here's my Code I used during one of my Projects :
import time
def no() :
if (Condition !!!) :
it got true
oh()
else:
time.sleep(1) /Don't remove or don't blame me if ur system gets ""DEAD""
no()
def oh() : /Ur main program
while True:
if(bla) :
.......
no()
else :
time.sleep(1)
oh()
oh()
Hope it Helps