I am trying to learn the multiprocessing library in Python but I cannot get my code to work with queue.Queue. Simply put, I have no idea where to put the queue.Queue.join() method in my code. Does it go IN the while loop or outside of it? If it goes outside of the while loop, do I write while q.not_empty? Why would I use q.not_empty when it the docs it is explicitly mentioned to use join()?
Here is my code. I am expecting my 4 cores, simultaneously to return the amount of prime numbers computed by my function, 2 times per core for the total amount of 8 computations. The prime computing functions works without problem.
import queue
def main():
q = queue.Queue()
[q.put((compute_primes, (1, 30000))) for _ in range(8)]
with multiprocessing.Pool(processes=4) as pool:
while q.not_empty:
result = q.get()
function = pool.apply_async(result[0], args=(result[1][0], result[1][1]))
function.get()
q.join()
With the code above, I break out of the loop if the queue is empty. But this is supposed to be unrealiable and why would I need q.join() afterwards?
With the code below, I can't break out of the loop. Changes are while True and position of q.join()
def main():
q = queue.Queue()
[q.put((compute_primes, (1, 30000))) for _ in range(8)]
with multiprocessing.Pool(processes=4) as pool:
while True:
result = q.get()
function = pool.apply_async(result[0], args=(result[1][0], result[1][1]))
function.get()
q.join()
Where should I put q.join?
P.S. This code also does not parrallelise the tasks effectively, it essentially computes the functions one by one and I cannot understand why, but this is a different problem.
P.S. 2
Code for prime function
def compute_primes(start, end):
start_time = time.time()
primes = []
for number in range(start, end + 1):
flag = True
for i in range(2, number):
if (number % i) == 0:
flag = False
break
if flag:
primes.append(number)
end_time = time.time()
print(f"Time taken: {end_time - start_time}\n"
f"Amount primes: {len(primes)}")
return primes
Queues and Pools
Running one at a time... separate problem.
Actually, this is part of the same problem. All this implies that you are not
using a multiprocessing pool managed by Pool. What you do at the moment is to
put all your tasks in a queue, get them straight back out again, and then
process them one at a time using a pool, which only ever gets one task at a
time. These two paradigms are mutually exclusive: if you want to use a pool to
do the work for you, you don't need queue; if you need to handle the queue
yourself, you probably don't want to use pool.
Pool
multiprocessing.Pool and accompanying methods spawn the right number of worker
processes, serialise your function to them, and then set up a queue internally
and handle sending tasks and getting results. This is much easier than doing it
manually, and is normally the right way to go about things:
When you use pool, you do something like this:
results = pool.map(compute_primes, [(0,100_000) for _ in range(8)])
Which will block for you until all the pool has finished, or:
results = pool.map_async(compute_primes, [(0, 100_000) for _ in range(8)])
results.wait() # wait
unless you plan to process the results as they come in, in which case you don't
use results.wait() at all:
for _ in range(8):
result = results.get()
do_stuff(result)
You do use pool.join() or pool.close() just to make sure the pool is shut
down gracefully, but this has nothing to do with getting your results.
Your examples
Your first example works, because you do this:
put tasks in a queue
get them out one by one and process them
join an empty queue -> leave immediately
Your second example fails, because you do this:
put tasks in a queue
get one task out
wait for queue to be empty or done -> blocks indefinitely
In this case, you don't want a queue at all.
Using Queue manually
Aside: where are you getting your Queue from? multiprocessing.Queue is not
joinable; you need multiprocessing.JoinableQueue. threading.Queue should
not be used with multiprocessing. queue.Queue, likewise, should not be used
with `multiprocessing.
When do you use a task queue? When you don't just want to apply a bunch of
arguments to a bunch of functions. Perhaps you want to use a custom class.
Perhaps you want to do something funny. Perhaps you want to do some things with
one type of argument, but something else if the argument is of a certain kind,
and code is better organised this way. In these cases subclassing Process (or
Thread for multithreading) yourself may be clearer and cleaner. None of that
seems to apply in this case.
Using join with queues
.join() is for task queues. It blocks until every task in the queue has
been marked as done. This is handy for when you want to offload some processing
to a bunch of processes, but wait for them before you do anything. so you
normally do something like:
tasks = JoinableQueue()
for t in qs:
tasks.put(t)
start_multiprocessing() # dummy fn
tasks.join() # wait for everything to be done
However in this case you don't do that, or want to do it.
I prefer not to specify an argument to the Pool constructor unless, for some reason, I want very few concurrent processes. By constructing Pool without an argument, the number of potential concurrent processes will vary from computer to computer depending on its CPU architecture. Here's how I would implement your task (assuming I understand your use-case completely):
from multiprocessing import Pool
def genPrime(): # prime number generator
D = {}
q = 2
while True:
if q not in D:
yield q
D[q * q] = [q]
else:
for p in D[q]:
D.setdefault(p + q, []).append(p)
del D[q]
q += 1
def compute_primes(n):
g = genPrime()
return [next(g) for _ in range(n)]
NCOMPUTATIONS = 8
NPRIMES = 30_000
def main():
with Pool() as pool:
ar = []
for _ in range(NCOMPUTATIONS):
ar.append(pool.apply_async(compute_primes, [NPRIMES]))
for _ar in ar:
result = _ar.get() # waits for process to terminate and get its return value
assert len(result) == NPRIMES
if __name__ == '__main__':
main()
[ Please note that I am not the author of the genPrime function ]
Related
So I want to run a function which can either search for information on the web or directly from my own mysql database.
The first process will be time-consuming, the second relatively fast.
With this in mind I create a process which starts this compound search (find_compound_view). If the process finishes relatively fast it means it's present on the database so I can render the results immediately. Otherwise, I will render "drax_retrieving_data.html".
The stupid solution I came up with was to run the function twice, once to check if the process takes a long time, the other to actually get the return values of the function. This is pretty much because I don't know how to return the values of my find_compound_view function. I've tried googling but I can't seem to find how to return the values from the class Process specifically.
p = Process(target=find_compound_view, args=(form,))
p.start()
is_running = p.is_alive()
start_time=time.time()
while is_running:
time.sleep(0.05)
is_running = p.is_alive()
if time.time() - start_time > 10 :
print('Timer exceeded, DRAX is retrieving info!',time.time() - start_time)
return render(request,'drax_internal_dbs/drax_retrieving_data.html')
compound = find_compound_view(form,use_email=False)
if compound:
data=*****
return render(request, 'drax_internal_dbs/result.html',data)
You will need a multiprocessing.Pipe or a multiprocessing.Queue to send the results back to your parent-process. If you just do I/0, you should use a Thread instead of a Process, since it's more lightweight and most time will be spend on waiting. I'm showing you how it's done for Process and Threads in general.
Process with Queue
The multiprocessing queue is build on top of a pipe and access is synchronized with locks/semaphores. Queues are thread- and process-safe, meaning you can use one queue for multiple producer/consumer-processes and even multiple threads in these processes. Adding the first item on the queue will also start a feeder-thread in the calling process. The additional overhead of a multiprocessing.Queue makes using a pipe for single-producer/single-consumer scenarios preferable and more performant.
Here's how to send and retrieve a result with a multiprocessing.Queue:
from multiprocessing import Process, Queue
SENTINEL = 'SENTINEL'
def sim_busy(out_queue, x):
for _ in range(int(x)):
assert 1 == 1
result = x
out_queue.put(result)
# If all results are enqueued, send a sentinel-value to let the parent know
# no more results will come.
out_queue.put(SENTINEL)
if __name__ == '__main__':
out_queue = Queue()
p = Process(target=sim_busy, args=(out_queue, 150e6)) # 150e6 == 150000000.0
p.start()
for result in iter(out_queue.get, SENTINEL): # sentinel breaks the loop
print(result)
The queue is passed as argument into the function, results are .put() on the queue and the parent get.()s from the queue. .get() is a blocking call, execution does not resume until something is to get (specifying timeout parameter is possible). Note the work sim_busy does here is cpu-intensive, that's when you would choose processes over threads.
Process & Pipe
For one-to-one connections a pipe is enough. The setup is nearly identical, just the methods are named differently and a call to Pipe() returns two connection objects. In duplex mode, both objects are read-write ends, with duplex=False (simplex) the first connection object is the read-end of the pipe, the second is the write-end. In this basic scenario we just need a simplex-pipe:
from multiprocessing import Process, Pipe
SENTINEL = 'SENTINEL'
def sim_busy(write_conn, x):
for _ in range(int(x)):
assert 1 == 1
result = x
write_conn.send(result)
# If all results are send, send a sentinel-value to let the parent know
# no more results will come.
write_conn.send(SENTINEL)
if __name__ == '__main__':
# duplex=False because we just need one-way communication in this case.
read_conn, write_conn = Pipe(duplex=False)
p = Process(target=sim_busy, args=(write_conn, 150e6)) # 150e6 == 150000000.0
p.start()
for result in iter(read_conn.recv, SENTINEL): # sentinel breaks the loop
print(result)
Thread & Queue
For use with threading, you want to switch to queue.Queue. queue.Queue is build on top of a collections.deque, adding some locks to make it thread-safe. Unlike with multiprocessing's queue and pipe, objects put on a queue.Queue won't get pickled. Since threads share the same memory address-space, serialization for memory-copying is unnecessary, only pointers are transmitted.
from threading import Thread
from queue import Queue
import time
SENTINEL = 'SENTINEL'
def sim_io(out_queue, query):
time.sleep(1)
result = query + '_result'
out_queue.put(result)
# If all results are enqueued, send a sentinel-value to let the parent know
# no more results will come.
out_queue.put(SENTINEL)
if __name__ == '__main__':
out_queue = Queue()
p = Thread(target=sim_io, args=(out_queue, 'my_query'))
p.start()
for result in iter(out_queue.get, SENTINEL): # sentinel-value breaks the loop
print(result)
Read here why for result in iter(out_queue.get, SENTINEL):
should be prefered over a while True...break setup, where possible.
Read here why you should use if __name__ == '__main__': in all your scripts and especially in multiprocessing.
More about get()-usage here.
I have a Pool of workers and use apply_async to submit work to them.
I do not care for the result of the function applied to each item.
The pool seems to accept any number of apply_async calls, no matter how large the data or how quickly the workers can keep up with the work.
Is there a way to make apply_async block as soon as a certain number of items are waiting to be processed? I am sure internally, the pool is using a Queue, so it would be trivial to just use a maximum size for the Queue?
If this is not supported, would it make sense to submit a big report because this look like very basic functionality and rather trivial to add?
It would be a shame if one had to essentially re-implement the whole logic of Pool just to make this work.
Here is some very basic code:
from multiprocessing import Pool
dowork(item):
# process the item (for side effects, no return value needed)
pass
pool = Pool(nprocesses)
for work in getmorework():
# this should block if we already have too many work waiting!
pool.apply_async(dowork, (work,))
pool.close()
pool.join()
So something like this?
import multiprocessing
import time
worker_count = 4
mp = multiprocessing.Pool(processes=worker_count)
workers = [None] * worker_count
while True:
try:
for i in range(worker_count):
if workers[i] is None or workers[i].ready():
workers[i] = mp.apply_async(dowork, args=next(getmorework()))
except StopIteration:
break
time.sleep(1)
I dunno how fast you're expecting each worker to finish, the time.sleep may or may not be necessary or might need to be a different time or whatever.
an alternative might be to use Queue's directly:
from multiprocessing import Process, JoinableQueue
from time import sleep
from random import random
def do_work(i):
print(f"worker {i}")
sleep(random())
print(f"done {i}")
def worker():
while True:
item = q.get()
if item is None:
break
do_work(item)
q.task_done()
def generator(n):
for i in range(n):
print(f"gen {i}")
yield i
# 1 = allow generator to get this far ahead
q = JoinableQueue(1)
# 2 = maximum amount of parallelism
procs = [Process(target=worker) for _ in range(2)]
# and get them running
for p in procs:
p.daemon = True
p.start()
# schedule 10 items for processing
for item in generator(10):
q.put(item)
# wait for jobs to finish executing
q.join()
# signal workers to finish up
for p in procs:
q.put(None)
# wait for workers to actually finish
for p in procs:
p.join()
mostly stolen from example Python's queue module:
https://docs.python.org/3/library/queue.html#queue.Queue.join
I have a function that encrypts a number and stores it in an list
encrypted = [[0]*10]*1000
def encrypt(i):
encrypted[i]=bin(i)[2:].zfill(10).decode('hex')
The expression is much more complex than this. I am just stating an example.
Now I want to call the encrypt function inside a for loop with multiple calls in different processes or threads - however due to GIL for CPU bound process, threads wont help - correct me if i am wrong.
for i in xrange(1000):
encrypt(i)
So The loop should not wait for the encryption of one value to get over, for the next to start.
So when i=1 and encryption of 1 is taking place, For loop should increment and start encrypting 2, and then 3 simultaneously.
The results of encryption should be stored in encrypted list (order of results is not important).
You can use multithreading.Pool
from multiprocessing import Pool
def encrypt(i):
return bin(i)[2:].zfill(10).decode('hex')
if __name__ == '__main__':
pool = Pool(processes=4) # adjust to number of cores
result = pool.map(encrypt, range(1000))
print result
Alright, first some advice. Depending on the number of threads you need to run you should check out PyPy this sounds like the kind of project that could benefit heavily from pypy's features.
Here Is an edited example from the Queue docs if I understand what you need than this should point you in the right direction.
This code assumes that you have a list of encrypted numbers and that
your encrypt function handles adding the results to a list or storing them somehow.
def worker():
while True:
number = q.get()
encrypt(number)
q.task_done()
q = Queue()
for i in range(num_worker_threads):
t = Thread(target=worker)
t.daemon = True
t.start()
for number in numbers:
q.put(number)
q.join() # block until all tasks are done
I have a problem running multiple processes in python3 .
My program does the following:
1. Takes entries from an sqllite database and passes them to an input_queue
2. Create multiple processes that take items off the input_queue, run it through a function and output the result to the output queue.
3. Create a thread that takes items off the output_queue and prints them (This thread is obviously started before the first 2 steps)
My problem is that currently the 'function' in step 2 is only run as many times as the number of processes set, so for example if you set the number of processes to 8, it only runs 8 times then stops. I assumed it would keep running until it took all items off the input_queue.
Do I need to rewrite the function that takes the entries out of the database (step 1) into another process and then pass its output queue as an input queue for step 2?
Edit:
Here is an example of the code, I used a list of numbers as a substitute for the database entries as it still performs the same way. I have 300 items on the list and I would like it to process all 300 items, but at the moment it just processes 10 (the number of processes I have assigned)
#!/usr/bin/python3
from multiprocessing import Process,Queue
import multiprocessing
from threading import Thread
## This is the class that would be passed to the multi_processing function
class Processor:
def __init__(self,out_queue):
self.out_queue = out_queue
def __call__(self,in_queue):
data_entry = in_queue.get()
result = data_entry*2
self.out_queue.put(result)
#Performs the multiprocessing
def perform_distributed_processing(dbList,threads,processor_factory,output_queue):
input_queue = Queue()
# Create the Data processors.
for i in range(threads):
processor = processor_factory(output_queue)
data_proc = Process(target = processor,
args = (input_queue,))
data_proc.start()
# Push entries to the queue.
for entry in dbList:
input_queue.put(entry)
# Push stop markers to the queue, one for each thread.
for i in range(threads):
input_queue.put(None)
data_proc.join()
output_queue.put(None)
if __name__ == '__main__':
output_results = Queue()
def output_results_reader(queue):
while True:
item = queue.get()
if item is None:
break
print(item)
# Establish results collecting thread.
results_process = Thread(target = output_results_reader,args = (output_results,))
results_process.start()
# Use this as a substitute for the database in the example
dbList = [i for i in range(300)]
# Perform multi processing
perform_distributed_processing(dbList,10,Processor,output_results)
# Wait for it all to finish.
results_process.join()
A collection of processes that service an input queue and write to an output queue is pretty much the definition of a process pool.
If you want to know how to build one from scratch, the best way to learn is to look at the source code for multiprocessing.Pool, which is pretty simply Python, and very nicely written. But, as you might expect, you can just use multiprocessing.Pool instead of re-implementing it. The examples in the docs are very nice.
But really, you could make this even simpler by using an executor instead of a pool. It's hard to explain the difference (again, read the docs for both modules), but basically, a future is a "smart" result object, which means instead of a pool with a variety of different ways to run jobs and get results, you just need a dumb thing that doesn't know how to do anything but return futures. (Of course in the most trivial cases, the code looks almost identical either way…)
from concurrent.futures import ProcessPoolExecutor
def Processor(data_entry):
return data_entry*2
def perform_distributed_processing(dbList, threads, processor_factory):
with ProcessPoolExecutor(processes=threads) as executor:
yield from executor.map(processor_factory, dbList)
if __name__ == '__main__':
# Use this as a substitute for the database in the example
dbList = [i for i in range(300)]
for result in perform_distributed_processing(dbList, 8, Processor):
print(result)
Or, if you want to handle them as they come instead of in order:
def perform_distributed_processing(dbList, threads, processor_factory):
with ProcessPoolExecutor(processes=threads) as executor:
fs = (executor.submit(processor_factory, db) for db in dbList)
yield from map(Future.result, as_completed(fs))
Notice that I also replaced your in-process queue and thread, because it wasn't doing anything but providing a way to interleave "wait for the next result" and "process the most recent result", and yield (or yield from, in this case) does that without all the complexity, overhead, and potential for getting things wrong.
Don't try to rewrite the whole multiprocessing library again. I think you can use any of multiprocessing.Pool methods depending on your needs - if this is a batch job you can even use the synchronous multiprocessing.Pool.map() - only instead of pushing to input queue, you need to write a generator that yields input to the threads.
I have two different functions f, and g that compute the same result with different algorithms. Sometimes one or the other takes a long time while the other terminates quickly. I want to create a new function that runs each simultaneously and then returns the result from the first that finishes.
I want to create that function with a higher order function
h = firstresult(f, g)
What is the best way to accomplish this in Python?
I suspect that the solution involves threading. I'd like to avoid discussion of the GIL.
I would simply use a Queue for this. Start the threads and the first one which has a result ready writes to the queue.
Code
from threading import Thread
from time import sleep
from Queue import Queue
def firstresult(*functions):
queue = Queue()
threads = []
for f in functions:
def thread_main():
queue.put(f())
thread = Thread(target=thread_main)
threads.append(thread)
thread.start()
result = queue.get()
return result
def slow():
sleep(1)
return 42
def fast():
return 0
if __name__ == '__main__':
print firstresult(slow, fast)
Live demo
http://ideone.com/jzzZX2
Notes
Stopping the threads is an entirely different topic. For this you need to add some state variable to the threads which needs to be checked in regular intervals. As I want to keep this example short I simply assumed that part and assumed that all workers get the time to finish their work even though the result is never read.
Skipping the discussion about the Gil as requested by the questioner. ;-)
Now - unlike my suggestion on the other answer, this piece of code does exactly what you are requesting:
from multiprocessing import Process, Queue
import random
import time
def firstresult(func1, func2):
queue = Queue()
proc1 = Process(target=func1,args=(queue,))
proc2 = Process(target=func2, args=(queue,))
proc1.start();proc2.start()
result = queue.get()
proc1.terminate(); proc2.terminate()
return result
def algo1(queue):
time.sleep(random.uniform(0,1))
queue.put("algo 1")
def algo2(queue):
time.sleep(random.uniform(0,1))
queue.put("algo 2")
print firstresult(algo1, algo2)
Run each function in a new worker thread, the 2 worker threads send the result back to the main thread in a 1 item queue or something similar. When the main thread receives the result from the winner, it kills (do python threads support kill yet? lol.) both worker threads to avoid wasting time (one function may take hours while the other only takes a second).
Replace the word thread with process if you want.
You will need to run each function in another process (with multiprocessing) or in a different thread.
If both are CPU bound, multithread won help much - exactly due to the GIL -
so multiprocessing is the way.
If the return value is a pickleable (serializable) object, I have this decorator I created that simply runs the function in background, in another process:
https://bitbucket.org/jsbueno/lelo/src
It is not exactly what you want - as both are non-blocking and start executing right away. The tirck with this decorator is that it blocks (and waits for the function to complete) as when you try to use the return value.
But on the other hand - it is just a decorator that does all the work.