I have a program that takes in a list. For each value in this list, it retrieves another list and processes this other list.
Basically, it's a 3-depth tree which need to do possibly expensive processing at each node.
Each nodes needs to be able to process the results of its children.
What I'd like to be able to do is to map from the inputs in the first layer list to the results of each node. In each of these processes though, I would like to map the result from the next layer down.
What I'm worried about is that each layer will have its own number of max workers. I would like them to share a process pool if possible, otherwise there are performance hits for all of the process switching.
Is there a way to, using concurrency.futures or some other method, have each layer share the same process pool?
An example would be:
def main():
my_list = [1,2,3,4]
with concurrent.futures.ProcessPoolExecutor(max_workers = 4) as executor:
results = executor.map(my_function, zip(my_list, [executor] * len(my_list)))
#process results
def my_function(args):
list = args[0]
executor = args[1]
new_list = process(list)
results = executor.map(second_function, new_list)
#process results
#return processed results
def second_function(values):
...
In this way, each child process will draw from the same pool.
Or, can I do something like (but not exactly)
import concurrent.futures.ProcessPoolExecutor(max_workers = 4) as executor
and have each call to executor pull from the same process pool?
The problem is that you process pool has 4 threads and you try to wait in maybe 20 threads.. so that there are not enough threads to do what you want.
In other words: the my_function is executed in a worker thread. This thread blocks when map is called. There is one thread less available to exeucte the calls to map. The futures block this thread.
My solution is to use the yield and yield from statements that return futures. So my solution is to remove the blocking of the futures and the thread. All futures are created and then a yield occurs to interrut the currrent execution and free the thread. This thread can then execute the map futures. Onc a future is done the registered callbac executes the next() generator step.
To solve the Proxy problem to exissting objects this question has to be solved first: How to properly set up multiprocessing proxy objects for objects that already exist
So given we have the following recursion to execute: [1,[2,[3,3,3],2],1],0,0] A recursive parallel sum of the lists.
We can expect the following output:
tasks: [[1, [2, [3, 3, 3], 2], 1], 0, 0]
tasks: [1, [2, [3, 3, 3], 2], 1]
tasks: 0
tasks: 0
tasks: 1
tasks: [2, [3, 3, 3], 2]
tasks: 1
tasks: 2
tasks: [3, 3, 3]
tasks: 2
tasks: 3
tasks: 3
tasks: 3
v: 15
This code here introduces a Recursion enabled ThreadPoolExecutor:
import traceback
from concurrent.futures.thread import *
from concurrent.futures import *
from concurrent.futures._base import *
##import hanging_threads
class RecursiveThreadPoolExecutor(ThreadPoolExecutor):
# updated version here: https://gist.github.com/niccokunzmann/9170072
def _submit(self, fn, *args, **kwargs):
return ThreadPoolExecutor.submit(self, fn, *args, **kwargs)
def submit(self, fn, *args, **kwargs):
"""Submits a callable to be executed with the given arguments.
Schedules the callable to be executed as fn(*args, **kwargs) and returns
a Future instance representing the execution of the callable.
Returns:
A Future representing the given call.
"""
real_future = Future()
def generator_start():
try:
## print('start', fn, args, kwargs)
generator = fn(*args, **kwargs)
## print('generator:', generator)
def generator_next():
try:
## print('next')
try:
future = next(generator)
except StopIteration as stop:
real_future.set_result(stop.args[0])
else:
if future is None:
self._submit(generator_next)
else:
future.add_done_callback(lambda future: generator_next())
except:
traceback.print_exc()
self._submit(generator_next)
## print('next submitted 1')
except:
traceback.print_exc()
self._submit(generator_start)
return real_future
def recursive_map(self, fn, *iterables, timeout=None):
"""Returns a iterator equivalent to map(fn, iter).
Args:
fn: A callable that will take as many arguments as there are
passed iterables.
timeout: The maximum number of seconds to wait. If None, then there
is no limit on the wait time.
Returns:
An iterator equivalent to: map(func, *iterables) but the calls may
be evaluated out-of-order.
Raises:
TimeoutError: If the entire result iterator could not be generated
before the given timeout.
Exception: If fn(*args) raises for any values.
"""
if timeout is not None:
end_time = timeout + time.time()
fs = [self.submit(fn, *args) for args in zip(*iterables)]
# Yield must be hidden in closure so that the futures are submitted
# before the first iterator value is required.
def result_iterator():
yield from fs
return fs
return result_iterator()
if __name__ == '__main__':
def f(args):
executor, tasks = args
print ('tasks:', tasks)
if type(tasks) == int:
return tasks
# waiting for all futures without blocking the thread
futures = yield from executor.recursive_map(f, [(executor, task) for task in tasks])
return sum([future.result() for future in futures])
with RecursiveThreadPoolExecutor(max_workers = 1) as executor:
r = executor.map(f, [(executor, [[1,[2,[3,3,3],2],1],0,0],)] * 1)
import time
time.sleep(0.1)
for v in r:
print('v: {}'.format(v))
An updated version can be found here: https://gist.github.com/niccokunzmann/9170072
Sadly, I am not able to implement this for Processes using some multiprocessing stuff now. You can do it and the only thing that should be necessairy is to create a proxy object to the generator_start and generator_next functions. If you do so, please let me know.
To solve the proxy problem to the methods this question would also be answered: How to properly set up multiprocessing proxy objects for objects that already exist
Related
I want to create a list of futures running on ThreadPoolExecutor, then display each one of them as soon as they finist evaluating.
The expected result is: each of 0, 2, 6, 12 will printed every 3 seconds.
However, I'm getting a result only after 12 seconds and the numbers are displayed simulatenously.
from concurrent.futures import ThreadPoolExecutor
import time
def fnc(x, y):
time.sleep(3)
return x*y
futures = []
with ThreadPoolExecutor(max_workers=1) as executor:
for i in range(0, 4):
print(f"Submitting {i}")
futures += [executor.submit(fnc, i, i+1)]
for f in futures:
print(f.result())
Build your list of submitted futures then use as_completed() to know when a thread has finished and its result is available.
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
def fnc(x, y):
time.sleep(3)
return x*y
futures = []
with ThreadPoolExecutor() as executor:
for i in range(0, 4):
print(f"Submitting {i}")
futures += [executor.submit(fnc, i, i+1)]
for future in as_completed(futures):
print(future.result())
You are calling the result method on each future outside the ThreadPoolExecutor context manager, when you exit, it, it calls the __exit__ method:
def __exit__(self, exc_type, exc_val, exc_tb):
self.shutdown(wait=True)
return False
The shutdown method signature is:
shutdown(self, wait=True, *, cancel_futures=False)
And the docs says:
Args:
wait: If True then shutdown will not return until all running
futures have finished executing and the resources used by the
executor have been reclaimed.
cancel_futures: If True then shutdown will cancel all pending
futures. Futures that are completed or running will not be
cancelled.
We can see that by default it will wait until all running futures and their resources have stopped running as well, and that cancel_futures is by default getting the value False, thus we are not canceling pending futures.
We can prove it by changing the fnc to print the values instead of returning them, and do nothing after the ThreadPoolExecutor context manager block:
def fnc(x, y):
time.sleep(3)
print(x * y)
futures = []
with ThreadPoolExecutor(max_workers=1) as executor:
for i in range(0, 4):
print(f"Submitting {i}")
futures += [executor.submit(fnc, i, i + 1)]
print("Blah!")
Still prints the values 0, 2, 6, 12! even though we only submitted the function to an executor list...
Fix it by moving the for loop block inside the context manager:
from concurrent.futures import ThreadPoolExecutor
import time
def fnc(x, y):
time.sleep(3)
return x*y
futures = []
with ThreadPoolExecutor(max_workers=1) as executor:
for i in range(0, 4):
print(f"Submitting {i}")
futures += [executor.submit(fnc, i, i+1)]
for f in futures:
print(f.result())
Note that setting max_workers=1 is essentially forcing the program to run consecutively rather than concurrently, in this program, setting max_workers=X will print the results of X returns of fnc at a time.
If you want to wait three seconds between each result, then either set max_workers to 1 or remove it at all. If you want to print two results at a time each three seconds - set max_workers=2 etc..
I have a function that shows a progress bar while another function runs, using concurrent.futures.ThreadPoolExecutor.
def run_with_loading(function, args=[], kwargs={}, phrase="Loading...", bar_length=5):
'''
Run a function while showing a loading bar.
'''
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
f = executor.submit(function, *args, **kwargs)
while True:
for i in range(bar_length):
if f.done():
result = f.result()
if f.exception() or not result[0]:
c = "✗"
else:
c = "✔"
print(f"\33[2K\r{phrase} {c}")
return result
sys.stdout.write(
f"\r{phrase} {'□' * i}{'■'}{'□' * (bar_length - i - 1)}")
sys.stdout.flush()
time.sleep(0.2)
Yet this still polls to see if the function spawned is done, every 0.2 seconds. While this works, I am wondering if there is any more efficient way to notify the run_with_loading function that the function it started has finished. I need to retain whether there is an exception or not, which is made clear in the code, so it can print ✗.
Instead of polling for the result you should use concurrent.futures.as_completed to loop through the results:
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
futures = []
futures.append(executor.submit(function, *args, **kwargs))
for future in concurrent.futures.as_completed(futures):
result = f.result()
if f.exception() or not result[0]:
c = "✗"
else:
c = "✔"
print(f"\33[2K\r{phrase} {c}")
You can find the documentation here: Futures doc The function as_completed an iterator to go through all finished futures, which has ended successfully.
You need to adapt your bar_length approach, but hopefully, this helps you to get another idea of how to wait for your results.
After encountering some probable memory leaks in a long running multi threaded script I found out about maxtasksperchild, which can be used in a Multi process pool like this:
import multiprocessing
with multiprocessing.Pool(processes=32, maxtasksperchild=x) as pool:
pool.imap(function,stuff)
Is something similar possible for the Threadpool (multiprocessing.pool.ThreadPool)?
As the answer by noxdafox said, there is no way in the parent class, you can use threading module to control the max number of tasks per child. As you want to use multiprocessing.pool.ThreadPool, threading module is similar, so...
def split_processing(yourlist, num_splits=4):
'''
yourlist = list which you want to pass to function for threading.
num_splits = control total units passed.
'''
split_size = len(yourlist) // num_splits
threads = []
for i in range(num_splits):
start = i * split_size
end = len(yourlist) if i+1 == num_splits else (i+1) * split_size
threads.append(threading.Thread(target=function, args=(yourlist, start, end)))
threads[-1].start()
# wait for all threads to finish
for t in threads:
t.join()
Lets say
yourlist has 100 items, then
if num_splits = 10; then threads = 10, each thread has 10 tasks.
if num_splits = 5; then threads = 5, each thread has 20 tasks.
if num_splits = 50; then threads = 50, each thread has 2 tasks.
and vice versa.
Looking at multiprocessing.pool.ThreadPool implementation it becomes evident that the maxtaskperchild parameter is not propagated to the parent multiprocessing.Pool class. The multiprocessing.pool.ThreadPool implementation has never been completed, hence it lacks few features (as well as tests and documentation).
The pebble package implements a ThreadPool which supports workers restart after a given amount of tasks have been processed.
I wanted a ThreadPool that will run a new task as soon as another task in the pool completes (i.e. maxtasksperchild=1). I decided to write a small "ThreadPool" class that creates a new thread for every task. As soon a task in the pool completes, another thread is created for the next value in the iterable passed to the map method. The map method blocks until all values in the passed iterable have been processed and their threads returned.
import threading
class ThreadPool():
def __init__(self, processes=20):
self.processes = processes
self.threads = [Thread() for _ in range(0, processes)]
def get_dead_threads(self):
dead = []
for thread in self.threads:
if not thread.is_alive():
dead.append(thread)
return dead
def is_thread_running(self):
return len(self.get_dead_threads()) < self.processes
def map(self, func, values):
attempted_count = 0
values_iter = iter(values)
# loop until all values have been attempted to be processed and
# all threads are finished running
while (attempted_count < len(values) or self.is_thread_running()):
for thread in self.get_dead_threads():
try:
# run thread with the next value
value = next(values_iter)
attempted_count += 1
thread.run(func, value)
except StopIteration:
break
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, exc_tb):
pass
class Thread():
def __init__(self):
self.thread = None
def run(self, target, *args, **kwargs):
self.thread = threading.Thread(target=target,
args=args,
kwargs=kwargs)
self.thread.start()
def is_alive(self):
if self.thread:
return self.thread.is_alive()
else:
return False
You can use it like this:
def run_job(self, value, mp_queue=None):
# do something with value
value += 1
with ThreadPool(processes=2) as pool:
pool.map(run_job, [1, 2, 3, 4, 5])
I'm running a thread pool that is giving a random bug. Sometimes it works, sometimes it gets stuck at the pool.join part of this code. I've been at this several days, yet cannot find any difference between when it works or when it gets stuck. Please help...
Here's the code...
def run_thread_pool(functions_list):
# Make the Pool of workers
pool = ThreadPool() # left blank to default to machine number of cores
pool.map(run_function, functions_list)
# close the pool and wait for the work to finish
pool.close()
pool.join()
return
Similarly, this code is also randomly getting stuck at q.join(:
def run_queue_block(methods_list, max_num_of_workers=20):
from views.console_output_handler import add_to_console_queue
'''
Runs methods on threads. Stores method returns in a list. Then outputs that list
after all methods in the list have been completed.
:param methods_list: example ((method name, args), (method_2, args), (method_3, args)
:param max_num_of_workers: The number of threads to use in the block.
:return: The full list of returns from each method.
'''
method_returns = []
log = StandardLogger(logger_name='run_queue_block')
# lock to serialize console output
lock = threading.Lock()
def _output(item):
# Make sure the whole print completes or threads can mix up output in one line.
with lock:
if item:
add_to_console_queue(item)
msg = threading.current_thread().name, item
log.log_debug(msg)
return
# The worker thread pulls an item from the queue and processes it
def _worker():
log = StandardLogger(logger_name='_worker')
while True:
try:
method, args = q.get() # Extract and unpack callable and arguments
except:
# we've hit a nonetype object.
break
if method is None:
break
item = method(*args) # Call callable with provided args and store result
method_returns.append(item)
_output(item)
q.task_done()
num_of_jobs = len(methods_list)
if num_of_jobs < max_num_of_workers:
max_num_of_workers = num_of_jobs
# Create the queue and thread pool.
q = Queue()
threads = []
# starts worker threads.
for i in range(max_num_of_workers):
t = threading.Thread(target=_worker)
t.daemon = True # thread dies when main thread (only non-daemon thread) exits.
t.start()
threads.append(t)
for method in methods_list:
q.put(method)
# block until all tasks are done
q.join()
# stop workers
for i in range(max_num_of_workers):
q.put(None)
for t in threads:
t.join()
return method_returns
I never know when it's going to work. It works most the time, but most the time is not good enough. What might possibly cause a bug like this?
You have to call shutdown on the concurrent.futures.ThreadPoolExecutor object. Then return the result of pool.map.
def run_thread_pool(functions_list):
# Make the Pool of workers
pool = ThreadPool() # left blank to default to machine number of cores
result = pool.map(run_function, functions_list)
# close the pool and wait for the work to finish
pool.shutdown()
return result
I've simplified your code without a Queue object and daemon Thread. Check if it fits your requirement.
def run_queue_block(methods_list):
from views.console_output_handler import add_to_console_queue
'''
Runs methods on threads. Stores method returns in a list. Then outputs that list
after all methods in the list have been completed.
:param methods_list: example ((method name, args), (method_2, args), (method_3, args)
:param max_num_of_workers: The number of threads to use in the block.
:return: The full list of returns from each method.
'''
method_returns = []
log = StandardLogger(logger_name='run_queue_block')
# lock to serialize console output
lock = threading.Lock()
def _output(item):
# Make sure the whole print completes or threads can mix up output in one line.
with lock:
if item:
add_to_console_queue(item)
msg = threading.current_thread().name, item
log.log_debug(msg)
return
# The worker thread pulls an item from the queue and processes it
def _worker(method, *args, **kwargs):
log = StandardLogger(logger_name='_worker')
item = method(*args, **kwargs) # Call callable with provided args and store result
with lock:
method_returns.append(item)
_output(item)
threads = []
# starts worker threads.
for method, args in methods_list:
t = threading.Thread(target=_worker, args=(method, args))
t.start()
threads.append(t)
# stop workers
for t in threads:
t.join()
return method_returns
To allow your queue to join in your second example, you need to ensure that all tasks are removed from the queue.
So in your _worker function, mark tasks as done even if they could not be processed, otherwise the queue will never be emptied, and your program will hang.
def _worker():
log = StandardLogger(logger_name='_worker')
while True:
try:
method, args = q.get() # Extract and unpack callable and arguments
except:
# we've hit a nonetype object.
q.task_done()
break
if method is None:
q.task_done()
break
item = method(*args) # Call callable with provided args and store result
method_returns.append(item)
_output(item)
q.task_done()
I was trying to create a function which can run multiple processes asynchronous and will send the response. Since multiprocessing.Process() do not return the response, I thought of creating a function as:
from multiprocessing import Process
def async_call(func_list):
"""
Runs the list of function asynchronously.
:param func_list: Expects list of lists to be of format
[[func1, args1, kwargs1], [func2, args2, kwargs2], ...]
:return: List of output of the functions
[output1, output2, ...]
"""
response_list = []
def worker(function, f_args, f_kwargs, response_list):
"""
Runs the function and appends the output to list
"""
response = function(*f_args, **f_kwargs)
response_list.append(response)
processes = [Process(target=worker, args=(func, args, kwargs, response_list)) \
for func, args, kwargs in func_list]
for process in processes:
process.start()
for process in processes:
process.join()
return response_list
Within this function, I call worker asynchronously which accepts additional parameter as list. Since, lists are passed as reference, I thought I can append the response of actual function within the list. And async_call will return me the response of all the function.
But this is not behaving the way I expected. Value gets appended to the list within the worker(), but outside the worker response_list list remains empty.
Any idea what I am doing wrong? And, is there any alternative to achieve what I am doing?
You can't share objects directly across processes. You need to use one of the classes especially designed for communicating values, Queue and Pipe; see the documentation.
As mentioned in Daniel's Answer, objects can not be directly shared between the processes. However, multiprocessing library provides Queues and Pipes as communication channel between processes. (Read documentation for more details)
Here is the function I created using multiprocessing.Queue():
def async_call(func_list):
"""
Runs the list of function asynchronously.
:param func_list: Expects list of lists to be of format
[[func1, args1, kwargs1], [func2, args2, kwargs2], ...]
:return: List of output of the functions
[output1, output2, ...]
"""
def worker(function, f_args, f_kwargs, queue, index):
"""
Runs the function and appends the output to list, and the Exception in the case of error
"""
response = {
'index': index, # For tracking the index of each function in actual list.
# Since, this function is called asynchronously, order in
# queue may differ
'data': None,
'error': None
}
# Handle error in the function call
try:
response['data'] = function(*f_args, **f_kwargs)
except Exception as e:
response['error'] = e # send back the exception along with the queue
queue.put(response)
queue = Queue()
processes = [Process(target=worker, args=(func, args, kwargs, queue, i)) \
for i, (func, args, kwargs) in enumerate(func_list)]
for process in processes:
process.start()
response_list = []
for process in processes:
# Wait for process to finish
process.join()
# Get back the response from the queue
response = queue.get()
if response['error']:
raise response['error'] # Raise exception if the function call failed
response_list.append(response)
return [content['data'] for content in sorted(response_list, key=lambda x: x['index'])]
Sample run:
def my_sum(x, y):
return x + y
def your_mul(x, y):
return x*y
my_func_list = [[my_sum, [1], {'y': 2}], [your_mul, [], {'x':1, 'y':2}]]
async_call(my_func_list)
# Value returned: [3, 2]