I feel like I am missing something very simple but still cannot figure out how to achieve the result after reading docs of the multiprocessing package. All I want is to set a class object (property) in a separate process and return it back to the main process. What I tried:
from multiprocessing import Process, Queue
class B:
def __init__(self):
self.attr = 'hello'
def worker(queue):
b = B()
setattr(b.__class__, 'prop', property(lambda b: b.attr))
assert b.prop
queue.put(b)
queue = Queue()
p = Process(target=worker, args=(queue,))
p.start()
res = queue.get()
p.join()
assert hasattr(res, 'prop')
So property "prop" just disappears. What is the proper way to return it ? I am using Windows 10.
Related
Here is some code
How do I change the current multi-process part to a single process?
import multiprocessing
def process(self):
if product != 0:
if self.mlist.check(file_dictionary):
self.mlist.patch(file_dictionary)
process = multiprocessing.Process(target = self.mlist.job, args = (file_dictionary, targ))
self.multiprocess_list.append(process)
process.start()
def wait(self):
process_list = self.multiprocess_list
for i in process_list:
i.join(2)
Next time, please provide a minimal reproducible example like this:
import multiprocessing
def doSomething(k: str, v: int):
print(f"key={k}, value={v}")
if __name__ == "__main__":
data = {"a": 1, "b": 2, "c": 3}
processes = []
for key, value in data.items():
process = multiprocessing.Process(target=doSomething, args=(key, value))
processes.append(process)
for process in processes:
process.start()
for process in processes:
process.join()
Now, replace the import multiprocessing by
class multiprocessing: # Fake multiprocessing, not multiprocessing at all
def __init__(self, target, args):
self.target = target
self.args = args
#staticmethod
def Process(target, args):
return multiprocessing(target, args)
def start(self):
self.target(*self.args)
def join(self):
pass
The idea is that you provide the exact same interface (same names) but do not do multiprocessing. You can switch from and to multiprocessing easily.
multiprocessing is no longer a library but a class
multiprocessing.Process is no longer the constructor of a process object, but a static method
start() just calls the method
join() does nothing, since it ran synchronously already
Here is my code below , I put string in queue , and hope dowork2 to do something work , and return char in shared_queue
but I always get nothing at while not shared_queue.empty()
please give me some point , thanks.
import time
import multiprocessing as mp
class Test(mp.Process):
def __init__(self, **kwargs):
mp.Process.__init__(self)
self.daemon = False
print('dosomething')
def run(self):
manager = mp.Manager()
queue = manager.Queue()
shared_queue = manager.Queue()
# shared_list = manager.list()
pool = mp.Pool()
results = []
results.append(pool.apply_async(self.dowork2,(queue,shared_queue)))
while True:
time.sleep(0.2)
t =time.time()
queue.put('abc')
queue.put('def')
l = ''
while not shared_queue.empty():
l = l + shared_queue.get()
print(l)
print( '%.4f' %(time.time()-t))
pool.close()
pool.join()
def dowork2(queue,shared_queue):
while True:
path = queue.get()
shared_queue.put(path[-1:])
if __name__ == '__main__':
t = Test()
t.start()
# t.join()
# t.run()
I managed to get it work by moving your dowork2 outside the class. If you declare dowork2 as a function before Test class and call it as
results.append(pool.apply_async(dowork2, (queue, shared_queue)))
it works as expected. I am not 100% sure but it probably goes wrong because your Test class is already subclassing Process. Now when your pool creates a subprocess and initialises the same class in the subprocess, something gets overridden somewhere.
Overall I wonder if Pool is really what you want to use here. Your worker seems to be in an infinite loop indicating you do not expect a return value from the worker, only the result in the return queue. If this is the case, you can remove Pool.
I also managed to get it work keeping your worker function within the class when I scrapped the Pool and replaced with another subprocess:
foo = mp.Process(group=None, target=self.dowork2, args=(queue, shared_queue))
foo.start()
# results.append(pool.apply_async(Test.dowork2, (queue, shared_queue)))
while True:
....
(you need to add self to your worker, though, or declare it as a static method:)
def dowork2(self, queue, shared_queue):
Could someone please shed some light on why this threaded code to call a classes' method never completes?
from Queue import Queue
from threading import Thread
class SimpleThing(object):
def __init__(self, name):
self.name = name
def print_name(self):
print self.name
class ThingSpawner(object):
def __init__(self, name_list):
self.things = [SimpleThing(name) for name in name_list]
self.thread_queue = Queue()
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
self.thread_queue.put(t)
self.thread_queue.join()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
The code will clearly run the print_name method, but does not join() and exit.
Also, what is the neatest way to modify this code so that the join() completes? The motivation is to use an existing python control class for a bit of hardware, and allows you to call a (very slow) method of the control class in parallel. Thanks!
When you are doing
self.thread_queue.put(t)
You are putting some threads into the Queue, obviously. However, i'm not really sure why. You never use that queue again for anything, and it's completely unnecessary. To make matters worse, you then call
self.thread_queue.join()
Which basically waits forever for the queue to empty, which never happens, because you never empty it or do anything with it.
If I copy paste all your code, but without any Queue at all, everything is fine...
from threading import Thread
class SimpleThing(object):
def __init__(self, name):
self.name = name
def print_name(self):
print self.name
class ThingSpawner(object):
def __init__(self, name_list):
self.things = [SimpleThing(name) for name in name_list]
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
However that's not what you want! Because your threads are daemons they will exit when the main program exits, even if they are not done yet (if I add some delay like sleep(1) before printing the name for example). You should call join() on the threads, not the queue, if you want to wait for them to finish. So we'll return the threads first:
def run(self):
all_threads = []
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
all_threads.append(t)
return all_threads
And when we run we'll do this:
threads = sp.run()
for t in threads:
t.join()
Thanks Ofer for the clear answer, which I've just accepted -- I am indeed not using the queue properly! Having reacquainted myself with queues now you've pointed out my error, for prosperity, here's an alternative approach using a queue:
from Queue import Queue
from threading import Thread
class SimpleThing(object):
def __init__(self, name, q):
self.name = name
def print_name(self, q):
print self.name
q.get()
q.task_done()
class ThingSpawner(object):
def __init__(self, name_list):
self.thread_queue = Queue()
self.things = [SimpleThing(name, self.thread_queue) for name in name_list]
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name, args=(self.thread_queue,))
t.daemon = True
t.start()
self.thread_queue.put(t)
self.thread_queue.join()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
This tutorial on threading and queues was useful once I understood my mistake.
I'm trying to inherit a sub class from multiprocessing.Process, which will have a queue for each instant, so that the queue can be use to catch the return value of the target.
But the problem is the multiprocessing.Process.start() uses subprocess.Popen (https://github.com/python/cpython/blob/master/Lib/multiprocessing/process.py) to create a process and run the target inside it. Is there a way to overload this without defining/overloading the entire Process module.
This is what I'm trying to do:
class Mprocessor(multiprocessing.Process):
def __init__(self, **kwargs):
multiprocessing.Process.__init__(self, **kwargs)
self._ret = Queue.Queue()
def run(self):
self._ret.put(multiprocessing.Process.run(self))
def getReturn(self):
if self._ret.empty:
return None
return self._ret.get()
Here I try to create a multiprocessig.Queue inside the class.
I override the 'run' method so when it is executed the return value/s of the target is put inside the queue.
I have a 'getReturn' method which is called in the main function using the Mprocess class. This method should only be called when 'Mprocess.isalive()' method(which is defined for multiprocessing.Process) returns false.
But this mechanism is not working because when I call 'Mprocess.start()' it creates a subprocess which runs the target in its own environment.
I want to know if there's a way to use the queue in the start method to get the return value, and avoid the target to have a queue argument to communicate.
I wanted to generalize this module.
I don't want my methods to be defined to have a queue to get return value.
I want to have a module so that it can be applicable to any function, because I am planning to have a manager method, which takes a dict["process_name/ID" : methods/targets], a dict["process name/ID" : [argument_list]] and create a process for each of this targets and return a dict["process_name/ID" : (return tuple, ).
Any ideas will be welcomed.
EDIT
Manager function:
def Processor_call(func = None, func_args = None):
if sorted(func.keys()) != sorted(func_args()):
print "Names in func dict and args dict doesn't match"
return None
process_list = multiprocessing.Queue()
for i in func.keys():
p = Mprocessor(name = i, target = func[i], args = tuple(func_args[i]))
process_list.put(p)
p.start()
return_dict = {}
while not process_list.empty():
process_wait = process_list.get()
if not process_wait.is_alive():
process_wait.join()
if process_wait.exitcode == 0:
return_dict[process_wait.name] = process_wait.getReturn()
else:
print "Error in process %s, status not availabe" %process_wait.name
else:
join_process.put(process_wait)
return return_dict
EDIT: The target function should look like this.
def sum(a , b):
return a + b
I don't want to pass a queue into the function, and return with queue.
I want to make a common module so that, any existing methods can use multiprocessing without any change to its definition, So the interface with other modules are maintained.
I don't want a function to be designed only to be run as a process, I want to have the common interface so that other modules can also use this function as a normal method, without bothering to read from the queue to get the return value.
Comment: ... so that I'll get the return value from the process started from start method
This will work for me, for instance:
class Mprocessor
class Mprocessor(multiprocessing.Process):
def __init__(self, queue, **kwargs):
multiprocessing.Process.__init__(self, **kwargs)
self._ret = queue
def run(self):
return_value = self._target( *self._args )
self._ret.put((self.name, return_value))
time.sleep(0.25)
exit(0)
Start processes and wait for return values
def Processor_call(func=None, func_args=None):
print('func=%s, func_args=%s' % (func, func_args))
ret_q = multiprocessing.Manager().Queue()
process_list = []
for i in func.keys():
p = Mprocessor(name=i, target=func[i], args=(func_args[i],), queue=ret_q)
p.start()
process_list.append(p)
time.sleep(0.1)
print('Block __main__ until all process terminated')
for p in process_list:
p.join()
print('Aggregate alle return values')
return_dict = {}
while not ret_q.empty():
p_name, value = ret_q.get()
return_dict[p_name] = value
return return_dict
__main__
if __name__ == '__main__':
rd = Processor_call({'f1':f1, 'f2':f1}, {'f1':1, 'f2':2})
print('rd=%s' % rd)
Output:
func={'f1': , 'f2': }, func_args={'f1': 1, 'f2': 2}
pid:4501 start 2
pid:4501 running
pid:4500 start 1
pid:4500 running
Block __main__ until all process terminated
pid:4501 running
pid:4500 running
pid:4501 running
pid:4500 running
pid:4501 Terminate
pid:4500 Terminate
Aggregate alle return values
rd={'f1': 1, 'f2': 2}
Tested with Python:3.4.2 and 2.7.9
Question: Is it possible to inherit multiprocessing.Process to communicate with the main process
Yes, it's possible. But not useing a class object, as your process use it's own copy of the class object .
You have to use a global Queue object and pass it to your process .
I was wondering if multiprocessing can be confined in separate python modules. For example, if I have a python module with multiprocessing as so:
#example.py
def f(q, a, m='No'):
print m
q.put(a)
if __name__ == '__main__':
a = '123'
m = 'Yes'
q = Queue()
p = Process(target=f, args=(q, a, m))
p.start()
print q.get()
p.join()
Is there anyway of using this in another script as a module using import, whilst still retaining the multiprocessing:
#Call from another script
import example
example.f(q, a)
>>> 'Yes' #Confirmation that multiprocessing was used
Yes, you can accomplish this by either creating a class or a function. You can import either into another script.
Here is an example with a class:
# example.py
from multiprocessing import Process
class Example(object):
def __init__(self, queue):
"""
#type queue: multiprocessing.Queue
"""
self.q = queue
def run(self, a, m=None):
p = Process(target=self.f, args=(a, m))
p.start()
print self.q.get()
p.join()
def f(self, a, m='No'):
print m
self.q.put(a)
Then import from your example:
>>> from multiprocessing import Queue
>>> from example import Example
>>> q = Queue()
>>> e = Example(q)
>>> e.run('123', m='Yes')
Yes
123