Let's consider the following example:
from pathos.pools import ProcessPool
class A:
def run(self, arg: int):
shared_variable = 100
def __run_parallel(arg: int):
local_variable = 0
# ...
if local_variable > shared_variable:
shared_variable = local_variable
ProcessPool(4).map(__run_parallel, range(1000))
It's quite obvious to see that there's a data race in if local_variable > shared_variable: with shared_variable = local_variable when using four processes.
Consequently, I'd like to introduce a locking mechanism around the if block, so I tried the following:
from pathos.pools import ProcessPool
from multiprocessing import Lock
class A:
def run(self, arg: int):
lock = Lock()
shared_variable = 100
def __run_parallel(arg: int):
local_variable = 0
# ...
lock.acquire()
if local_variable > shared_variable:
shared_variable = local_variable
lock.release()
ProcessPool(4).map(__run_parallel, range(1000))
However, I get the error RuntimeError: Lock objects should only be shared between processes through inheritance.
In the multiprocessing library, it seems as if the canonical way to achieve the desired mutual exclusion would be to use a Manager object.
However, how to do this idiomatically in pathos?
pathos leverages multiprocess, which has the same interface as multiprocessing, but uses dill. You can access it either of these ways.
>>> import pathos as pa
>>> import multiprocess as mp
>>> mp.Manager is pa.helpers.mp.Manager
True
Related
Would like to know if you can call an existing class function using Threading or Concurrent Futures without any issues? It works, but I'm curious about the implication in doing this. The reason I want to do this is because I want to keep class state information.
This is a sample with the general idea.
import concurrent.futures
import time
class Test:
def __init__(self) -> None:
super().__init__()
self.counter = 0
def run(self):
print('Test')
self.counter += 1
test1 = Test()
test2 = Test()
with concurrent.futures.ThreadPoolExecutor() as executer:
while True:
t1 = executer.submit(test1.run)
t2 = executer.submit(test2.run)
time.sleep(1.0)
Edit: the threads will use shared information downstream
My question is around Queues and using ThreadPoolExecutor. If I understand the Python docs for Queues I can have code somewhat like this and not have to worry about needing another lock in Class B to control which thread is adding in items in to the queue? Since the Queue implments multiproducer, multiconsumer
class A:
def __init__(max_worker = 1):
pool = ThreadPoolExecutor(max_worker)
buffer = {}
_lock = threading.RLock()
def add_record_id(id, item):
with self._lock:
buffer[id].add(item, pool)
class B:
def __init__():
q = queue.Queue()
def add(item, pool):
if id >= 0:
q.put(item)
pool.submit(background_remover)
I'm trying to use a defaultdict with multiprocessing, as described in Using defaultdict with multiprocessing?.
Example code:
from collections import defaultdict
from multiprocessing import Pool
from multiprocessing.managers import BaseManager, DictProxy
class DictProxyManager(BaseManager):
"""Support a using a defaultdict with multiprocessing"""
DictProxyManager.register('defaultdict', defaultdict, DictProxy)
class Test:
my_dict: defaultdict
def run(self):
for i in range(10):
self.my_dict['x'] += 1
def main():
test = Test()
mgr = DictProxyManager()
mgr.start()
test.my_dict = mgr.defaultdict(int)
p = Pool(processes=5)
for _ in range(10):
p.apply_async(test.run)
p.close()
p.join()
print(test.my_dict['x'])
if __name__ == '__main__':
main()
Expected output: 100
Actual output: Varies per run, usually somewhere in the 40-50 range.
For certain reasons I need to set the dict on an object rather than passing it as a parameter to the function in the Pool, but I don't think that should matter.
Why is it behaving this way? Thank you in advance!
The problem has nothing to do with defaultdict per se running as a manged object. The problem is that the operation being performed by method run on the defaultdict instance, namely self.my_dict['x'] += 1, is not atomic; it consists of first fetching the current value of key 'x' (if it exists) and then incrementing it and then finally storing it back. That is two separate method calls on the managed dictionary. In between those two calls another process could be running and retrieving the same value and incrementing and storing the same value.
You need to perform this non-atomic operation under a lock to ensure it is serialized across all processes as done below. I have also moved the call to DictProxyManager.register to inside function main for if you are running under Windows (you did not specify your platform but I inferred that possibility), that call will be issued needlessly by every process in the pool.
from collections import defaultdict
from multiprocessing import Pool, Lock
from multiprocessing.managers import BaseManager, DictProxy
class DictProxyManager(BaseManager):
"""Support a using a defaultdict with multiprocessing"""
def init_pool(the_lock):
global lock
lock = the_lock
class Test:
my_dict: defaultdict
def run(self):
for i in range(10):
with lock:
self.my_dict['x'] += 1
def main():
DictProxyManager.register('defaultdict', defaultdict, DictProxy)
test = Test()
mgr = DictProxyManager()
mgr.start()
test.my_dict = mgr.defaultdict(int)
lock = Lock()
p = Pool(processes=5, initializer=init_pool, initargs=(lock,))
for _ in range(10):
p.apply_async(test.run)
p.close()
p.join()
print(test.my_dict['x'])
if __name__ == '__main__':
main()
Prints:
100
I'm working on code where I frequently have to use python's multiprocessing Pool class. This results in a ton of code that looks like this:
import time
from multiprocessing import Pool
from functools import partial
def test_func(x):
time.sleep(1)
return x
def test_func_parallel(iterable, processes):
p = Pool(processes=processes)
output = p.map(test_func, iterable)
p.close()
return output
This can be made more general:
def parallel(func, iterable, **kwargs):
func = partial(func, **kwargs)
p = Pool(processes=6)
out = p.map(func, iterable)
p.close()
return out
This works, but adding a parallel wrapper to every other function complicates the code. What I'd really like is to get this working as a decorator. Something like this:
def parallel(num_processes):
def parallel_decorator(func, num_processes=num_processes):
def parallel_wrapper(iterable, **kwargs):
func = partial(func, **kwargs)
p = Pool(processes=num_processes)
output = p.map(func, iterable)
p.close()
return output
return parallel_wrapper
return parallel_decorator
Which could be used as follows:
#parallel(6)
def test_func(x):
time.sleep(1)
return x
This fails for pickle reasons
Can't pickle <function test1 at 0x117473268>: it's not the same object as __main__.test1
I've read a few posts on related issues, but they all implement a solution where the multiprocessing is executed outside the decorator. Does anyone know a way to make this work?
If you don't mind to not use the syntactic sugar for decorators (# symbol), something like this should work:
import functools
import time
from multiprocessing import Pool
def parallel(func=None, **options):
if func is None:
return functools.partial(parallel, **options)
def wrapper(iterable, **kwargs):
processes = options["processes"]
with Pool(processes) as pool:
result = pool.map(func, iterable)
return result
return wrapper
def test(i):
time.sleep(1)
print(f"{i}: {i * i}")
test_parallel = parallel(test, processes=6)
def main():
test_parallel(range(10))
if __name__ == "__main__":
main()
I have the same problem. It revolves around how Pool() objects are implemented. So, it is going to work fine with a normal wrapper but not with a Decorator. The workaround is to define your own Pool()-like implementation using Process().
This can be very tricky to optimize but if you are a Decorator enthusiast here is a (dirty) example:
# something to do
args = range(10)
def parallel(function):
""" An alternative implementation to
multiprocessing.Pool().map() using
multiprocessing.Process(). """
def interfacer(args):
""" The wrapper function. """
# required libraries
from multiprocessing import (Queue, Process)
from os import cpu_count
# process control
## maximum number of processes required
max_processes = len(args)
## maximum numer of processes running
max_threads = cpu_count() - 1
""" Since there is no Pool() around
we need to take care of the processes
ourselves. If there is nothing for a
processes to do, it is going to await
for an input, if there are too many of
them, the processor shall suffer. """
# communications
## things to do
inbasket = Queue()
## things done
outbasket = Queue()
""" I am thinking asynchronouly,
there is probably a better way of
doing this. """
# populate inputs
for each in args:
## put arguments into the basket
inbasket.put(each)
def doer():
""" Feeds the targeted/decorated
'function' with data from the baskets and
collets the results.
This blind function helps the
implementation to generalize over any
iterable. """
outbasket.put(function(inbasket.get()))
return(True)
def run(processes = max_threads):
""" Create a certain number of
Process()s and runs each one.
There is room for improvements here. """
# the process pool
factory = list()
# populate the process pool
for each in range(processes):
factory.append(Process(target = doer))
# execute in process pool
for each in factory:
each.start()
each.join()
each.close()
return(True)
""" Now we need to manage the processes,
and prevent them for overwhelm the CPU.
That is the tricky part that Pool() does
so well. """
while max_processes:
# as long as there is something to do
if (max_processes - max_threads) >= 0:
run(max_threads)
max_processes -= max_threads
else:
# play it safe
run(1)
max_processes -= 1
# undo the queue and give me back the list of 'dones'
return([outbasket.get() for each in range(outbasket.qsize())])
return(interfacer)
#parallel
def test(x):
return(x**2)
print(test(args))
Probably this code is inefficient, but gives an idea.
How can I get the following to work? The main point is that I want to run a method (and not a function) asynchronously.
from multiprocessing import Pool
class Async:
def __init__(self, pool):
self.pool = pool
self.run()
def run(self):
p.apply_async(self.f, (10, ))
def f(self, x):
print x*x
if __name__ == '__main__':
p = Pool(5)
a = Async(p)
p.close()
p.join()
This prints nothing.
The problem appears to be due to the fact that multiprocessing needs to pickle self.f while bound methods are not picklable. There is a discussion on how to solve the problem here.
The apply_async apparently creates an exception which is put inside the future returned. That's why nothing is printed. If a get is executed on the future, then the exception is raised.
Its definitely possible to thread class methods using a threadpool in python 2 - the following programme did what I would expect.
#!/usr/bin/env python
from multiprocessing.pool import ThreadPool
class TestAsync():
def __init__(self):
pool = ThreadPool(processes = 2)
async_completions = []
for a in range(2):
async_completions.append(pool.apply_async(self.print_int, ( a,)))
for completion in async_completions:
res = completion.get()
print("res = %d" % res)
def print_int(self, value):
print(value)
return (value*10)
a = TestAsync()