Short question: Is it possible to have N work processes and a balancer process that will find worker that does nothing at this time and pass UnitOfWork to it?
Long question:
Imagine class like this, witch will be subclassed for certain tasks:
class UnitOfWork:
def __init__(self, **some_starting_parameters):
pass
def init(self):
# open connections, etc.
def run(self):
# do the job
Start the balancer and worker process:
balancer = LoadBalancer()
workers = balancer.spawn_workers(10)
Deploy work (balancer should find a lazy worker, and pass a task to it, or else if every worker is busy, add UOW to queue and wait till free worker):
balancer.work(UnitOfWork(some=parameters))
# internally, find free worker, pass UOW, ouw.init() + ouw.run()
Is this possible (or is it crazy)?
PS I'm familiar with multiprocessing Process class, and process pools, but:
Every Process instance starts a process (yep :) ) - I want fixed num of workers
I want Process instance that can make generic work
I suggest you take a look at multiprocessing.Pool() because I believe it exactly solves your problem. It runs N "worker processes" and as each worker finishes a task, another task is provided. And there is no need for "poison pills"; it is very simple.
I have always used the .map() method on the pool.
Python multiprocessing.Pool: when to use apply, apply_async or map?
EDIT: Here is an answer I wrote to another question, and I used multiprocessing.Pool() in my answer.
Parallel file matching, Python
You don't need any smarts in the balancer; the Queue alone will do what you want. Throw each unit of work into the queue, and have the workers loop, taking a single work unit from the queue and processing it on each iteration. I don't think there's any problem passing an instance of UnitOfWork through the queue.
If you have a fixed amount of work to be done, you can create a "no more work to be done" work unit (a "poison pill") that tells a worker to shut down, and after all the regular work is put into the queue, put as many poison pills into the queue as you have workers.
Related
Lets assume I'm working with Python although it's not really relevant.
I have a big array and I want to find whether element x is in the array.
However, when one of the threads finds the element, I want that all other threads will stop,
there is no point for them to continue running. I want to continue with main program with the result.
What would be the right way for doing this?
I want to minimize the cpu time of the other threads after I already found that the element is truly exist.
In Python, you can create a thread-safe queue in the main thread and pass it to each worker thread. Each worker should search while the queue is empty() and then terminate. If the result is found, the lucky worker should put() it into the queue, causing all other workers to stop after their current iteration.
Example code (untested):
from Queue import Queue
from Threading import Thread
class Worker(Thread):
def __init__(self, queue):
self.queue=queue
def run(self):
while self.queue.empty():
result=search( ... )
if result:
queue.put(result)
def main():
queue=Queue()
workers=[]
for i in xrange(0,5):
workers.append(Worker(queue))
result=queue.get()
print result
There are multiple ways, one of them is polling a queue in caller's thread, where spawned threads store their results. As soon as there first result appears, terminate all running threads.
Just note, in CPython only one thread can run at the same time due to Global Interpreter Lock limitation (unless in C-extension which can free the lock). Also note, for searching in large data more appropriate data structure then array should be used, like a binary tree.
I want to spawn X number of Pool workers and give each of them X% of the work to do. My issue is that the work takes about 20 minutes to exhaust, longer for each extra process running, due to the type of calculations being done my answer may be found within minutes or hours. What I would like to do is implement some way for a single worker to go "HEY I FOUND IT" and use that signal to kill the remainder of the pool and move on with my calculations.
Key points:
I have tried callbacks, they don't seem to run on a starmap_async until the entire pool finishes.
I only care about the first suitable answer found.
I am not sharing resources and surprise process death, albeit rude, is perfectly acceptable.
I've also considered using a Queue, but it wouldn't make since because the scope of work I'm passing to each is already built into the parameters of the function.
Below is a very dulled down version of what I'm working with (the calculations I'm working with can take hours to finish over a 4.2 billion complex iterable.)
def doWork():
workers = Pool(2)
results = workers.starmap_async( func = distSearch , iterable = Sections1_5, callback = killPool )
workers.close()
print("Found answer : {}".format(results.get()))
workers.join()
def killPool():
workers.terminate()
print("Worker Pool Terminated")
I should probably specify that my process only returns if it finds an answer otherwise it just exits once done. I have looked at this thread but it has my completely lost and seems like a lot of overhead to consistently check for the win condition when that should come in the return/callback of the Worker Pool.
All the answers I've found result in significant overhead by supervising the worker pool, I'm looking for a solution that sources the kill signal at the worker level, autonomously.
I'm looking for a solution that sources the kill signal at the worker level, autonomously.
AFAIK, that doesn't exist. The methods of the Pool object (like Pool.terminate) should only be used in the process that created the pool.
What you could do is use Pool.imap_unordered. This returns an iterator in the parent process over the results which yields results as soon as they become available. As soon as the desired result pops up, you could then use Pool.terminate().
Edit:
From looking at the 3.5 implementation starmap_async returns a MapResult instance, which is not an iterator.
You can wrap multiple inputs in a tuple and use imap_unordered over a list of those.
I want to use the remote manager functions in the multiprocessing module to distribute work among many machines. I know there are 3rd party modules, but I want to stick with core as much as possible. I know for desktop (single machine), you can use the multiprocessing.Pool class to limit the number of CPUs, but have a couple of questions with remote managers.
I have the following code for the remote manager:
from multiprocessing.managers import BaseManager
import Queue
queue = Queue.Queue()
class QueueManager(BaseManager): pass
QueueManager.register('get_queue', callable=lambda:queue)
m = QueueManager(address=('', 50000), authkey='abracadabra')
s = m.get_server()
s.serve_forever()
This works great, and I can even submit a job into the Queue using the following code:
QueueManager.register('get_queue')
m = QueueManager(address=('machinename', 50000), authkey='abracadabra')
m.connect()
queue = m.get_queue()
queue.put('hello')
You can also the queue.get() to get a single entry in the queue.
How do you get the items in the queue? When I tried to iterate through the queue, I enter an infinite loop.
On the workers, can you limit each machine to 1 job per machine?
Since this method seems to be a pull method, where the workers need to examine if a job exists, can there be a push method where the multiprocessing server can be triggered?
Iterating over a queue is the same as doing:
while True:
elem = queue.get() #queue empty -> it blocks!!!
An elegant way to "iterate" over a queue and block your worker process when there are no more jobs to execute is to use None(or something else) as a sentinel and use iter(callable, sentinel):
for job in iter(queue.get, None):
# execute the calculation
output_queue.put(result)
#shutdown the worker process
Which is equivalent to:
while True:
job = queue.get()
if job is None:
break
#execute the calculation
output_queue.put(result)
#shutdown the worker process
Note that you have to insert in the queu a sentinel for each worker subprocess, otherwise there will be subprocesses waiting for it.
Regarding your second question, I don't understand what you are asking. The BaseManager provides one server that executes the calls from the clients, so, obviously, all requests are satisfied by the same machine.
Or do you mean allow each client to do only a request? I don't see any option for this, even though it could be implemented "by hand".
I don't understand your question. What is like a pull method? Can you rephrase your question with a bit more details on what you mean by "a push method where the multiprocessing server can be triggered"?
How do I add more threads (and remove threads) to the current multiprocessing pool, from within a task (i.e. celeryd was run with CELERYD_CONCURRENCY = 10 but I want to change it on-the-fly to CELERYD_CONCURRENCY = 15)?
There is a function called celery.concurrency.processes.TaskPool.Pool.grow but I have no idea how to call that from a running task or whether it is the correct function to do that.
Read the source:
https://github.com/ask/celery/blob/master/celery/concurrency/processes/__init__.py
there's both grow() and shrink(), although the latter seems a tad fishy.
you'd need to keep a reference to the pool somewhere, if you have only one pool, keep it global.
caveat poster: if multiprocessing actually means running multiple separate processes, you might already be in a child process when you try to shrink or grow, and obviously that won't work.
I need it to open 10 processes, and each time one of them finishes I want to wait few seconds and start another one.
It seems pretty simple, but somehow I can't get it to work.
I'm not 100% clear on what you're trying to accomplish, but have you looked at the multiprocessing module, specifically using a pool of workers?
I've done this same thing to process web statistics using a semaphore. Essentially, as processes are created, the semaphore is incremented. When they exit, it's decremented. The creation process is blocked when the semaphore blocks.
This actually fires off threads, which run external processes down execution path a bit.
Here's an example.
thread_sem = threading.Semaphore(int(cfg.maxthreads))
for k,v in log_data.items():
thread_list.append(ProcessorThread(int(k), v, thread_sem))
thread_list[-1].start()
And then in the constructor for ProcessorThread, I do this:
def __init__(self, siteid, data, lock_object):
threading.Thread.__init__(self)
self.setDaemon(False)
self.lock_object = lock_object
self.data = data
self.siteid = siteid
self.lock_object.acquire()
When the thread finishes it's task (whether successfully or not), the lock_object is released which allows for another process to begin.
HTH