Code:
import multiprocessing
import multiprocessing.managers
class A:
name = "A name"
arr_b = []
def __init__(self, num):
for i in range(5):
self.arr_b.append(B())
class B:
name = "B name"
def __init__(self):
pass
def func(num):
return A(num)
if __name__ == '__main__':
pool = multiprocessing.Pool()
result = pool.map(func, range(5))
for res in result:
print(res.name)
print(res.arr_b)
Result:
A name
[]
A name
[]
A name
[]
A name
[]
A name
[]
How i can share array of B class objects correctly?
I tried to used Manager and BaseManager, but it allows me to use created in main object. But i need to create object in func and return it into main.
Related
I wish to have a list of Queue's shared between processes. The idea is from a "main" process, I can pipe whatever information I want to one of the other processes, but the number of other processes aren't determined.
I cannot create the Queue in the "main" process. I am simulating a decentralised system and creating the Queue in the main process does not fit this paradigm. As such, the Queue's must be created within the other processes.
This poses a difficulty, as I can't find how to share these Queue's with the main process. I have a managed list using multiprocessing.Manager, but if I append a multiprocess.Queue to it, I get:
RuntimeError: Queue objects should only be shared between processes
through inheritance
Appending a standard data type such as an integer works just fine.
MRE below:
import multiprocessing as mp
from time import sleep
class test:
def __init__(self, qlist):
self.qlist = qlist
self.q = mp.Queue()
qlist.append(4)
self.next = None
self.run()
def run(self):
while True:
val = self.q.get()
if val == 1:
p = mp.Process(target = test, args=(qlist, ))
p.start()
else:
print(val)
if __name__ == '__main__':
manager = mp.Manager()
qlist = manager.list()
p = mp.Process(target = test, args=(qlist, ))
p.start()
sleep(0.5)
print(qlist)
p.join()
The idea would be in the if __name__ == '__main__': code, I could look through the qlist and select one of the Queues to pipe information to, such as: qlist[2].put(1) to add a test object or qlist[3].put("Hello") to print "Hello".
The best case scenario would rather be to have a list of test objects (where the test object has its self.q attribute for accessing it's Queue) that I could access from the "main" process, but I'm even less sure of how to do that hence why I'm asking about the Queue's.
Any help with this would be greatly appreciated
You can definitely create queue instances in the main process; this occurs in your test.__init__ method with the statement self.q = mp.Queue(), which is running in the main process. The problem is that a multiprocessing queue cannot be added to a managed list. Here is your program, slightly modified where it does not attempt to add the queues to a managed list. I have also made your test class (now renamed Test) to be a subclass of Process and it will now terminate:
import multiprocessing as mp
class Test(mp.Process):
def __init__(self, value):
mp.Process.__init__(self)
self.value = value
self.q = mp.Queue()
self.q.put(value)
self.next = None
def run(self):
value = self.q.get()
print('value = ', value)
value -= 1
if value > 0:
p = Test(value).start()
if __name__ == '__main__':
p = Test(4).start()
Prints:
value = 4
value = 3
value = 2
value = 1
If you want to maintain a list of objects, then it would be better if Test is not a subclass of Process:
import multiprocessing as mp
class Test():
def __init__(self, lst, value):
lst.append(self)
self.lst = lst
self.value = value
self.q = mp.Queue()
self.q.put(value)
self.next = None
def run(self):
value = self.q.get()
print('value = ', value)
value -= 1
if value > 0:
test = Test(self.lst, value)
p = mp.Process(target=test.run).start()
if __name__ == '__main__':
manager = mp.Manager()
lst = manager.list()
test = Test(lst, 4)
p = mp.Process(target=test.run).start()
import time
time.sleep(3)
print(lst)
Prints:
value = 4
value = 3
value = 2
value = 1
[<__mp_main__.Test object at 0x0000028E6DAD5DC0>, <__mp_main__.Test object at 0x0000028E6DAD5FA0>, <__mp_main__.Test object at 0x0000028E6DAD5E50>, <__mp_main__.Test object at 0x0000028E6DAD5D90>]
But here is a big BUT:
Each of those objects "live" in a different address space and the references can only have meaning when accessed from the original address space they were created in. So this is pretty useless:
import multiprocessing as mp
class Test():
def __init__(self, lst, value):
lst.append(self)
self.lst = lst
self.value = value
self.q = mp.Queue()
self.q.put(value)
self.next = None
def run(self):
value = self.q.get()
print('value = ', value)
value -= 1
if value > 0:
test = Test(self.lst, value)
p = mp.Process(target=test.run).start()
if __name__ == '__main__':
manager = mp.Manager()
lst = manager.list()
test = Test(lst, 4)
p = mp.Process(target=test.run).start()
import time
time.sleep(3)
print(test, test.__class__, test.value)
print(lst)
for elem in lst:
print(type(elem))
print(elem.value)
Prints:
value = 4
value = 3
value = 2
value = 1
<__main__.Test object at 0x0000020E52E6A640> <class '__main__.Test'> 4
[<__mp_main__.Test object at 0x0000016827704DC0>, <__mp_main__.Test object at 0x0000016827704FA0>, <__mp_main__.Test object at 0x0000016827704250>, <__mp_main__.Test object at 0x0000016827704D90>]
<class '__main__.Test'>
Traceback (most recent call last):
File "C:\Ron\test\test.py", line 31, in <module>
print(elem.value)
AttributeError: 'Test' object has no attribute 'value'
I want to send tasks to the POOL inside the shared class based on some conditions. But I got some unexpected result, Which are shown below.
• Why the len(self.map) is 0, not 100.
• Do I have to reconstruct my code to achieve this goal.
from multiprocessing import Pool
from multiprocessing.managers import BaseManager
pool = None
def doSomething(obj, *args):
obj.doSomething(*args)
class SharedClass:
def __init__(self):
global pool
self.map = set()
pool = Pool(4)
def someCondition(self):
# the condition is rely on the instance, here is just an example
return True
def go(self, n):
global pool
for i in xrange(n):
if self.someCondition():
# pass the shared class to other process
pool.apply_async(doSomething, (self, i))
pool.close()
pool.join()
# got AssertionError here
# why the len of self.map is 0
assert len(self.map) == 100
def doSomething(self, n):
# this should change the same SharedClass instance?
self.map.add(n)
class MyManager(BaseManager):
pass
MyManager.register("SharedClass", SharedClass)
def main():
manager = MyManager()
manager.start()
obj = manager.SharedClass()
obj.go(100)
if __name__ == "__main__":
main()
I want to test async_who function by pytest.
How do I test callback is called and the return value is 'Bob'
import threading
def async_who(callback):
t = threading.Thread(target=_who, args=(callback,))
t.start()
def _who(callback):
return callback('Bob')
def callback(name):
print(name)
return name
async_who(callback)
Because the async_who didn't return value. I can't do this,
def test_async_who():
res = async_who(callback)
assert res == 'Bob'
ThreadPool from multiprocessing module or ThreadPoolExecutor (for python version >= 3.2)
are ways to get the return value of a thread.
With concurrent.futures.ThreadPoolExecutor
from concurrent.futures import ThreadPoolExecutor
def async_who(callback):
executor = ThreadPoolExecutor(max_workers=2)
res = executor.submit(_who, callback)
return res.result()
def _who(callback):
return callback('Bob')
def callback(name):
print(name)
return name
def test_async_who():
res = async_who(callback)
assert res == 'Bob'
With multiprocessing.pool.ThreadPool
from multiprocessing.pool import ThreadPool
pool = ThreadPool(processes=2)
def async_who(callback):
res = pool.apply_async(_who, args=(callback,))
return res.get()
def _who(callback):
return callback('Bob')
def callback(name):
print(name)
return name
def test_async_who():
res = async_who(callback)
assert res == 'Bob'
I want to get return values from multiple processes initialised in one function and started in another function.
import multiprocessing
import time
class Auto:
def __init__(self):
self.msf = 0
def auto(self, return_dict, i):
# print "hello"
return_dict["hello"] = "hello{}".format(i)
def msf1(self):
man = multiprocessing.Manager()
self.return_dict = man.dict()
self.a= multiprocessing.Process(target=self.auto, args=(self.return_dict, 1, ))
self.b= multiprocessing.Process(target=self.auto, args=(self.return_dict, 1, ))
self.c= multiprocessing.Process(target=self.auto, args=(self.return_dict, 1, ))
def msf2(self):
self.a.start()
self.b.start()
self.c.start()
return self.return_dict.values()
You can use a Queue() to collect items from multiple processes. [docs]
Here is a very simple example of how it can work. See this part of the docs for a more in depth example of how it works.
def number(done_queue):
done_queue.put(5)
done_queue = multiprocessing.Queue()
x = Process(target=number, args=(done_queue))
x.start()
x.join()
y = [i for i in done_queue]
print(y)
I'm trying to run a function function of a class Foo in multiple subprocesses created by multiprocessing.pool in my main fileMain.py.
import sys
import multiprocessing
from Foo import Foo
f = Foo()
def get(n):
print f.ID
sys.stdout.flush() # Print the used instance's ID immediately
return f.function(n)
def calculate():
s = pool.map_async(get,range(4))
return s.get()
if __name__ == '__main__':
pool = multiprocessing.Pool(processes=4)
result = calculate()
pool.close()
The class Foois defined in Foo.py like
import random
class Foo():
def __init__(self):
print 'Initialized!'
self.ID = random.random() # Assign an unique ID to spot instance
pass
def function(self,x):
return x**2
The output I get is (with the ID obviously randomly)
Initialized!
Initialized!
0.955181146828
0.955181146828
0.955181146828
0.955181146828
>>> Initialized!
Initialized!
Initialized!
I want to avoid that a new instance of Foo is created by every subprocess. Why is this happening although all subprocesses then use the same instance as desired?
Notice also that f = Foo() cannot be placed after if __name__ == '__main__': or else
NameError: global name 'f' is not defined