Develop Reference

Python 3 Limit count of active threads (finished threads do not quit)

I want to limit the number of active threads. What i have seen is, that a finished thread stays alive and does not exit itself, so the number of active threads keep growing until an error occours.
The following code starts only 8 threads at a time but they stay alive even when they finished. So the number keeps growing:
class ThreadEx(threading.Thread):
__thread_limiter = None
__max_threads = 2
#classmethod
def max_threads(cls, thread_max):
ThreadEx.__max_threads = thread_max
ThreadEx.__thread_limiter = threading.BoundedSemaphore(value=ThreadEx.__max_threads)
def __init__(self, target=None, args:tuple=()):
super().__init__(target=target, args=args)
if not ThreadEx.__thread_limiter:
ThreadEx.__thread_limiter = threading.BoundedSemaphore(value=ThreadEx.__max_threads)
def run(self):
ThreadEx.__thread_limiter.acquire()
try:
#success = self._target(*self._args)
#if success: return True
super().run()
except:
pass
finally:
ThreadEx.__thread_limiter.release()
def call_me(test1, test2):
print(test1 + test2)
time.sleep(1)
ThreadEx.max_threads(8)
for i in range(0, 99):
t = ThreadEx(target=call_me, args=("Thread count: ", str(threading.active_count())))
t.start()
Due to the for loop, the number of threads keep growing to 99.
I know that a thread has done its work because call_me has been executed and threading.active_count() was printed.
Does somebody know how i make sure, a finished thread does not stay alive?

This may be a silly answer but to me it looks you are trying to reinvent ThreadPool.
from multiprocessing.pool import ThreadPool
from time import sleep
p = ThreadPool(8)
def call_me(test1):
print(test1)
sleep(1)
for i in range(0, 99):
p.apply_async(call_me, args=(i,))
p.close()
p.join()
This will ensure only 8 concurrent threads are running your function at any point of time. And if you want a bit more performance, you can import Pool from multiprocessing and use that. The interface is exactly the same but your pool will now be subprocesses instead of threads, which usually gives a performance boost as GIL does not come in the way.

I have changed the class according to the help of Hannu.
I post it for reference, maybe it's useful for others that come across this post:
import threading
from multiprocessing.pool import ThreadPool
import time
class MultiThread():
__thread_pool = None
#classmethod
def begin(cls, max_threads):
MultiThread.__thread_pool = ThreadPool(max_threads)
#classmethod
def end(cls):
MultiThread.__thread_pool.close()
MultiThread.__thread_pool.join()
def __init__(self, target=None, args:tuple=()):
self.__target = target
self.__args = args
def run(self):
try:
result = MultiThread.__thread_pool.apply_async(self.__target, args=self.__args)
return result.get()
except:
pass
def call_me(test1, test2):
print(test1 + test2)
time.sleep(1)
return 0
MultiThread.begin(8)
for i in range(0, 99):
t = MultiThread(target=call_me, args=("Thread count: ", str(threading.active_count())))
t.run()
MultiThread.end()
The maximum of threads is 8 at any given time determined by the method begin.
And also the method run returns the result of your passed function if it returns something.
Hope that helps.

Is there any replacement for empty while loops?

I'm using empty while loops a lot, for example:
I have a thread running in the background that will change a value called "a" in 5 seconds. however, I'm using a different function at the same time, and I want to let the second function know that the value has changed, so what I always did was:
import threading, time
class example:
def __init__(self):
self.a = 0
def valchange(self):
time.sleep(5)
self.a += 1
time.sleep(1)
print("im changing the a value to " + str(self.a))
print("those print commands needs to run after notifier stopped his while and started printing")
def notifier(exam :example, num :int):
while(exam.a != num):
pass
print("it changed to " + str(num))
exa = example()
i = 1
while(i <= 16):
temp= threading.Thread(target=notifier, args=(exa, i, ))
temp.start()
i += 3
i = 1
while(i <= 16):
exa.valchange()
i += 1
It's important to mention, that example could not use wait and set to an event, because there is no indication to when you need to run set, and how much threads are running in the background, and even what numbers will have a thread waiting for them to change.
And also you can't use join because changing 'a' is not a sign to print, only the condition is the sign.
Async and select can't help me as well because of the last reason.
Is there any way to create something, that will stop the program fromrunning until the condition will become true? you can provide your solution with any programming language you want, but mainly I'm using python 3.
EDIT: please remember that I need it to work with every condition. And my code example- is only an example, so if something works there, it doesn't necessarily will work with a different condition.
Thank you very much in advance :)
Idea:
wait(a == 5) // will do nothing until a == 5

You need to use select or epoll system calls if you're waiting for some system operation to finish. In case you're waiting for a certain IO event, then you can use asyncio (provided your Python version > 3.3), otherwise you could consider twisted.
If you're doing some CPU bound operations you need to consider multiple processes or threads, only then you can do any such monitoring effectively. Having a while loop running infinitely without any interruption is a disaster waiting to happen.

If your thread only changes a's value once, at the end of its life, then you can use .join() to wait for the thread to terminate.
import threading
import time
class example:
def __init__(self):
self.a = 0
self.temp = threading.Thread(target=self.valchange)
self.temp.start()
self.notifier()
def valchange(self):
time.sleep(5)
self.a = 1
def notifier(self):
self.temp.join()
print("the value of a has changed")
example()
If the thread might change a's value at any point in its lifetime, then you can use one of the threading module's more generalized control flow objects to coordinate execution. For instance, the Event object.
import threading
import time
class example:
def __init__(self):
self.a = 0
self.event = threading.Event()
temp = threading.Thread(target=self.valchange)
temp.start()
self.notifier()
def valchange(self):
time.sleep(5)
self.a = 1
self.event.set()
def notifier(self):
self.event.wait()
print("the value of a has changed")
example()
One drawback to this Event approach is that the thread target has to explicitly call set() whenever it changes the value of a, which can be irritating if you change a several times in your code. You could automate this away using a property:
import threading
import time
class example(object):
def __init__(self):
self._a = 0
self._a_event = threading.Event()
temp = threading.Thread(target=self.valchange)
temp.start()
self.notifier()
#property
def a(self):
return self._a
#a.setter
def a(self, value):
self._a = value
self._a_event.set()
def valchange(self):
time.sleep(5)
self.a = 1
def notifier(self):
self._a_event.wait()
print("the value of a has changed")
example()
Now valchange doesn't have to do anything special after setting a's value.

What you are describing is a spin lock, and might be fine, depending on your use case.
The alternative approach is to have the code you are waiting on call you back when it reaches a certain condition. This would require an async framework such as https://docs.python.org/3/library/asyncio-task.html
There are some nice simple examples in those docs so I won't insult your intelligence by pasting them here.

Alternative to time.sleep

INTRO: It is well known that the accuracy of time.sleep is OS and computation load dependent. The accuracy in Windows is very poor.
Similarly to /questions/17499837 a method can implement a busy wait using the time.clock method as an alternative to time.sleep. Such an approach creates unnecessary load affecting other modules in the system. That is not desirable while doing simulations.
To reduce the amount of time spent in busy wait and not relying on the time.sleep, a class employs the method select.select and exploits the timeout attribute. See code below:
from sys import platform as _platform
import time, select, socket
class HighResolutionTimeStamp():
__init = time.clock()
__base = time.time()
def __init__(self):
self.__fd = socket.socket()
self.dtts = time.clock if _platform == 'win32' else time.time
def __del__(self):
self.__fd.close()
def get_high_resolution_dt(self):
return HighResolutionTimeStamp.__base + self.dtts() if _platform == 'win32' else time.time()
def busy_wait(self, wait_time):
currentTime = self.dtts()
while (self.dtts() <= currentTime + wait_time):
pass
def sleep(self, wait_time):
currentTime = self.dtts()
while (self.dtts() < (currentTime + wait_time - 0.001)):
select.select([self.__fd], [], [], 0.001)
while (self.dtts() < currentTime + wait_time):
select.select([self.__fd], [], [], 0.0)
if __name__ == '__main__':
st = 1.0/80.0
it = 10
ts = 1
time.sleep(ts)
hrdr = HighResolutionTimeStamp()
total = hrdr.get_high_resolution_dt()
for i in range(it):
hrdr.busy_wait(st)
print 'Ellapsed:', hrdr.get_high_resolution_dt() - total
time.sleep(ts)
total = hrdr.get_high_resolution_dt()
for i in range(it):
hrdr.sleep(st)
print 'Ellapsed:', hrdr.get_high_resolution_dt() - total
time.sleep(ts)
total = hrdr.get_high_resolution_dt()
for i in range(it):
time.sleep(st)
print 'Ellapsed:', hrdr.get_high_resolution_dt() - total
ENVIRONMENT: I'm using PortablePython2.7.6.1
PROBLEM: When the code is executed at the PyScripter or in the command line with PyScripter open in the background, the script above performs very accurate. Once the PyScripter is closed, the method sleep becomes inaccurate. I'm aware that the timeout for select.select should be inaccurate as time.sleep but in all cases, not as described above.
RESULTS:
Without PyScripter running in the background
C:\..\PortablePython2.7.6.1\App\python.exe highresolutiondt.py
Busy wait. Ellapsed: 0.125249385834
Sleep. Ellapsed: 0.15624165535
Time.sleep. Ellapsed: 0.156844139099
With PyScripter running in the background
C:\..\PortablePython2.7.6.1\App\python.exe highresolutiondt.py
Busy wait. Ellapsed: 0.125702142715
Sleep. Ellapsed: 0.125874519348
Time.sleep. Ellapsed: 0.120799064636

This uses time since unix epoch, which, I'm pretty sure is more accurate, I don't use windows though, so I didn't test this out.
from time import time
def pause(secs):
init_time = time()
while time() < init_time+secs: pass
print("See ya in 10 seconds")
pause(10)
print("Heeeeeelooooo there")
Hope it helped

Is it possible to execute function every x seconds in python, when it is performing pool.map?

I am running pool.map on big data array and i want to print report in console every minute.
Is it possible? As i understand, python is synchronous language, it can't do this like nodejs.
Perhaps it can be done by threading.. or how?
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
# I want to call this function every minute
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
results = pool.map(make_job, data)
pool.close()
pool.join()

You can use a permanent threaded timer, like those from this question: Python threading.timer - repeat function every 'n' seconds
from threading import Timer,Event
class perpetualTimer(object):
# give it a cycle time (t) and a callback (hFunction)
def __init__(self,t,hFunction):
self.t=t
self.stop = Event()
self.hFunction = hFunction
self.thread = Timer(self.t,self.handle_function)
def handle_function(self):
self.hFunction()
self.thread = Timer(self.t,self.handle_function)
if not self.stop.is_set():
self.thread.start()
def start(self):
self.stop.clear()
self.thread.start()
def cancel(self):
self.stop.set()
self.thread.cancel()
Basically this is just a wrapper for a Timer object that creates a new Timer object every time your desired function is called. Don't expect millisecond accuracy (or even close) from this, but for your purposes it should be ideal.
Using this your example would become:
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
# set up the monitor to make run the function every minute
monitor = PerpetualTimer(60,display_status)
monitor.start()
results = pool.map(make_job, data)
pool.close()
pool.join()
monitor.cancel()
EDIT:
A cleaner solution may be (thanks to comments below):
from threading import Event,Thread
class RepeatTimer(Thread):
def __init__(self, t, callback, event):
Thread.__init__(self)
self.stop = event
self.wait_time = t
self.callback = callback
self.daemon = True
def run(self):
while not self.stop.wait(self.wait_time):
self.callback()
Then in your code:
def main():
data = [...]
pool = ThreadPool(45)
stop_flag = Event()
RepeatTimer(60,display_status,stop_flag).start()
results = pool.map(make_job, data)
pool.close()
pool.join()
stop_flag.set()

One way to do this, is to use main thread as the monitoring one. Something like below should work:
def main():
data = [...]
results = []
step = 0
pool = ThreadPool(16)
pool.map_async(make_job, data, callback=results.extend)
pool.close()
while True:
if results:
break
step += 1
sleep(1)
if step % 60 == 0:
print "status update" + ...
I've used .map() instead of .map_async() as the former is synchronous one. Also you probably will need to replace results.extend with something more efficient. And finally, due to GIL, speed improvement may be much smaller than expected.
BTW, it is little bit funny that you wrote that Python is synchronous in a question that asks about ThreadPool ;).

Consider using the time module. The time.time() function returns the current UNIX time.
For example, calling time.time() right now returns 1410384038.967499. One second later, it will return 1410384039.967499.
The way I would do this would be to use a while loop in the place of results = pool(...), and on every iteration to run a check like this:
last_time = time.time()
while (...):
new_time = time.time()
if new_time > last_time+60:
print "status update" + ...
last_time = new_time
(your computation here)
So that will check if (at least) a minute has elapsed since your last status update. It should print a status update approximately every sixty seconds.
Sorry that this is an incomplete answer, but I hope this helps or gives you some useful ideas.

Is there an easy way in Python to wait until certain condition is true?

I need to wait in a script until a certain number of conditions become true?
I know I can roll my own eventing using condition variables and friends, but I don't want to go through all the trouble of implementing it, since some object property changes come from external thread in a wrapped C++ library (Boost.Python), so I can't just hijack __setattr__ in a class and put a condition variable there, which leaves me with either trying to create and signal a Python condition variable from C++, or wrap a native one and wait on it in Python, both of which sound fiddly, needlessly complicated and boring.
Is there an easier way to do it, barring continuous polling of the condition?
Ideally it would be along the lines of
res = wait_until(lambda: some_predicate, timeout)
if (not res):
print 'timed out'

Unfortunately the only possibility to meet your constraints is to periodically poll, e.g....:
import time
def wait_until(somepredicate, timeout, period=0.25, *args, **kwargs):
mustend = time.time() + timeout
while time.time() < mustend:
if somepredicate(*args, **kwargs): return True
time.sleep(period)
return False
or the like. This can be optimized in several ways if somepredicate can be decomposed (e.g. if it's known to be an and of several clauses, especially if some of the clauses are in turn subject to optimization by being detectable via threading.Events or whatever, etc, etc), but in the general terms you ask for, this inefficient approach is the only way out.

Another nice package is waiting - https://pypi.org/project/waiting/
install:
pip install waiting
Usage:
You pass a function that will be called every time as a condition, a timeout, and (this is useful) you can pass a description for the waiting, which will be displayed if you get TimeoutError.
using function:
from waiting import wait
def is_something_ready(something):
if something.ready():
return True
return False
# wait for something to be ready
something = # whatever
wait(lambda: is_something_ready(something), timeout_seconds=120, waiting_for="something to be ready")
# this code will only execute after "something" is ready
print("Done")
Note: the function must return a boolean - True when the wait is over, False otherwise

Here is another solution. The goal was to make threads to wait on each other before doing some work in a very precise order. The work can take unknown amount of time. Constant polling is not good for two reasons: it eats CPU time and action does not start immediately after condition is met.
class Waiter():
def __init__(self, init_value):
self.var = init_value
self.var_mutex = threading.Lock()
self.var_event = threading.Event()
def WaitUntil(self, v):
while True:
self.var_mutex.acquire()
if self.var == v:
self.var_mutex.release()
return # Done waiting
self.var_mutex.release()
self.var_event.wait(1) # Wait 1 sec
def Set(self, v):
self.var_mutex.acquire()
self.var = v
self.var_mutex.release()
self.var_event.set() # In case someone is waiting
self.var_event.clear()
And the way to test it
class TestWaiter():
def __init__(self):
self.waiter = Waiter(0)
threading.Thread(name='Thread0', target=self.Thread0).start()
threading.Thread(name='Thread1', target=self.Thread1).start()
threading.Thread(name='Thread2', target=self.Thread2).start()
def Thread0(self):
while True:
self.waiter.WaitUntil(0)
# Do some work
time.sleep(np.random.rand()*2)
self.waiter.Set(1)
def Thread1(self):
while True:
self.waiter.WaitUntil(1)
# Do some work
time.sleep(np.random.rand())
self.waiter.Set(2)
def Thread2(self):
while True:
self.waiter.WaitUntil(2)
# Do some work
time.sleep(np.random.rand()/10)
self.waiter.Set(0)
Waiter for multiprocessing:
import multiprocessing as mp
import ctypes
class WaiterMP():
def __init__(self, init_value, stop_value=-1):
self.var = mp.Value(ctypes.c_int, init_value)
self.stop_value = stop_value
self.event = mp.Event()
def Terminate(self):
self.Set(self.stop_value)
def Restart(self):
self.var.value = self.init_value
def WaitUntil(self, v):
while True:
if self.var.value == v or self.var.value == self.stop_value:
return
# Wait 1 sec and check aiagn (in case event was missed)
self.event.wait(1)
def Set(self, v):
exit = self.var.value == self.stop_value
if not exit: # Do not set var if threads are exiting
self.var.value = v
self.event.set() # In case someone is waiting
self.event.clear()
Please comment if this is still not the best solution.

You've basically answered your own question: no.
Since you're dealing with external libraries in boost.python, which may change objects at their leisure, you need to either have those routines call an event handler refresh, or work with a condition.

Here is the threading extention to Alex's solution:
import time
import threading
# based on https://stackoverflow.com/a/2785908/1056345
def wait_until(somepredicate, timeout, period=0.25, *args, **kwargs):
must_end = time.time() + timeout
while time.time() < must_end:
if somepredicate(*args, **kwargs):
return True
time.sleep(period)
return False
def wait_until_par(*args, **kwargs):
t = threading.Thread(target=wait_until, args=args, kwargs=kwargs)
t.start()
print ('wait_until_par exits, thread runs in background')
def test():
print('test')
wait_until_par(test, 5)

From the computational perspective there must be a check for all conditions somewhere, sometime. If you have two parts of code, one that generates conditions changes and the other one that should be executed when some are true, you can do the following:
Have the code that changes conditions in, say, main thread, and the code that should be launched when some conditions are true, in a worker thread.
from threading import Thread,Event
locker = Event()
def WhenSomeTrue(locker):
locker.clear() # To prevent looping, see manual, link below
locker.wait(2.0) # Suspend the thread until woken up, or 2s timeout is reached
if not locker.is_set(): # when is_set() false, means timeout was reached
print('TIMEOUT')
else:
#
# Code when some conditions are true
#
worker_thread = Thread(target=WhenSomeTrue, args=(locker,))
worker_thread.start()
cond1 = False
cond2 = False
cond3 = False
def evaluate():
true_conditions = 0
for i in range(1,4):
if globals()["cond"+str(i)]: #access a global condition variable one by one
true_conditions += 1 #increment at each true value
if true_conditions > 1:
locker.set() # Resume the worker thread executing the else branch
#Or just if true_conditions > 1: locker.set();
#true_conditions would need be incremented when 'True' is written to any of those variables
#
# some condition change code
#
evaluate()
For more information concerning this method, visit: https://docs.python.org/3/library/threading.html#event-objects

Proposed solution:
def wait_until(delegate, timeout: int):
end = time.time() + timeout
while time.time() < end:
if delegate():
return True
else:
time.sleep(0.1)
return False
Usage:
wait_until(lambda: True, 2)

I once used this in my code:
while not condition:
pass
Hope this helps

In 2022 now you could use https://trio-util.readthedocs.io/en/latest/#trio_util.AsyncValue
I think this comes closest to what you want in its "smoothest" form

This worked for me
direction = ''
t = 0
while direction == '' and t <= 1:
sleep(0.1)
t += 0.1
This is for waiting for a signal while making sure time limit of 1 second

here's how:
import time
i = false
while i == false:
if (condition):
i = true
break

Here's my Code I used during one of my Projects :
import time
def no() :
if (Condition !!!) :
it got true
oh()
else:
time.sleep(1) /Don't remove or don't blame me if ur system gets ""DEAD""
no()
def oh() : /Ur main program
while True:
if(bla) :
.......
no()
else :
time.sleep(1)
oh()
oh()
Hope it Helps