I try to implement a multiprocessing code for generating some dictionary interested.
Here is my logic:
from multiprocessing import Manager, Queue, Process
input_list = Queue()
for x in my_input: # my_input is my input data
input_list.put(x)
output = Manager().dict()
def job():
while input_list.qsize()>0:
x = input_list.get()
result = my_func(x) # Do something here
output[x] = result
def monitor():
while True:
if input_list.qsize() > 0:
time.sleep(1)
print("Item List is Empty")
print("Does all the result being save?", len(output.keys()) == len(my_input))
job_list = [Process(target=monitor)]
for _ in range(num_of_worker):
job_list.append(Process(target=job))
for j in job_list:
j.start()
for j in job_list:
j.join()
print("The script is finished")
The logic of my code is quite simple.
Initialize a queue and put my input in.
Define two functions, job (doing something and save it to a dict) and monitor (print when everything inside queue is being processed and print how many results are being saved).
Then standard multiprocessing start and join.
The output I am getting:
Item List is Empty
Does all the result being save? True
...
Some child process did not finish and not yet join.
The script is stuck here and did not print "The script is finished".
My script will get stuck at the join statement, despite the monitor telling me that everything is finished (by checking number of items left in input_list and number of results stored in output).
Moreover, this error is not replicable. If I see my script stuck for more than 5 minutes, I will terminate it manually and restart it. I found that the script could finish properly (like 3 out of 10 times).
What could be happening?
Remark:
Since I suspect the error is some child process did not join, I tried something with Event. When the monitor found that the input_list is empty and output is completely filled, it will kill all the process. But the script is also stuck at the event triggering. (And same as above, the code does not get stuck every time, it works 3 out of 10 times).
#Homer512 comments gives me insight on wher is the mistake in the code.
switch from
def job():
while input_list.qsize>0:
x = input_list.get()
...
to
def job():
while input_list.qsize>0:
try:
x = input_list.get(True,5)
...
except Empty:
return 0
The reason for my script stuck at join because when input_list got only 1 element left, it trigger the while statement of job but only one process can get something from the queue. The other process will just stuck at get without suitable timeout.
Related
After some time trying to figure out how the Tkinter library works I have run into a problem. The script i wrote uses multiprocessing, and because the script needs to be as fast as possible I minimized the amount of traffic between the processes. This means that it takes about a minute to complete an enormous amount of tasks.
(If this task gets aborted halfway through, the used files will get corrupted).
The problem is that i want a stop button in my GUI to stop the script the proper way. After some research i haven't made any progress in finding a solution, so maybe some of you could help. I basically need a way to tell the script halfway through a task that it has to stop, after which the script will continue until the task is finished.
Edit:
The way my script is set up:
(This is missing the Tkinter part, because i don't know the solution to it yet).
from multiprocessing import Pool
def Setup():
#defines all paths of the files that are edited (and a whole lot more)
def Calculation(x, y, Primes):
#takes an x and y value, calculates the value of that coordinate and determines
#if the value is prime. Returns True of False, and the calculated value.
def Quadrant(List):
#takes a huge list of coordinates that have to be calculated. These
#coordinates (x and y) are passed to the 'Calculation' function, one by one.
#Returns all the calculated coordinates and if they are prime or not (boolean)
if __name__ == "__main__":
Filenames = Setup()
Process = Pool(4)
while True:
#Loop the main bit of the code to keep expanding the generated image
Input = [List of all coordinates, split into 4 quadrants (seperate lists) evenly]
Output = Process.map(Quadrant, Input)
#Combine all data and update list of primes
#Detects if escape is pressed, stops if true.
I am basically looking for a way to stop the while loop above, or an alternative to this loop.
I basically meant that the task has to stop, without aborting it suddenly. The script has to wait untill it's task is finished, and then look if a button is pressed to decide if it has to continue or not
We have no code from you to respond to, so if you are using a while() (note that you can also just issue a return from the function if some condition is True/False).
import time
from multiprocessing import Process, Manager
def test_f(test_d):
""" frist process to run
exit this process when dictionary's 'QUIT' == True
"""
while not test_d["QUIT"]:
print " test_f", test_d["QUIT"]
time.sleep(1.0)
def test_f2(name):
""" second process to run. Runs until the for loop exits
"""
for j in range(0, 10):
print name, j
time.sleep(0.5)
print "second process finished"
if __name__ == '__main__':
##--- create a dictionary via Manager
manager = Manager()
test_d = manager.dict()
test_d["QUIT"] = False
##--- start first process and send dictionary
p = Process(target=test_f, args=(test_d,))
p.start()
##--- start second process
p2 = Process(target=test_f2, args=('P2',))
p2.start()
##--- sleep 2 seconds and then change dictionary
## to exit first process
time.sleep(2.0)
print "\nterminate first process"
test_d["QUIT"] = True
print "test_d changed"
print "data from first process", test_d
##--- may not be necessary, but I always terminate to be sure
time.sleep(5.0)
p.terminate()
p2.terminate()
""" Thanks Doug Hellmann
Note: It is important to join() the process after terminating it.
in order to give the background machinery time to update the.
status of the object to reflect the termination
"""
p.join()
p2.join()
From the following code I would expect that the length of the resulting list were the same as the one of the range of items with which the multiprocess is feed:
import multiprocessing as mp
def worker(working_queue, output_queue):
while True:
if working_queue.empty() is True:
break #this is supposed to end the process.
else:
picked = working_queue.get()
if picked % 2 == 0:
output_queue.put(picked)
else:
working_queue.put(picked+1)
return
if __name__ == '__main__':
static_input = xrange(100)
working_q = mp.Queue()
output_q = mp.Queue()
for i in static_input:
working_q.put(i)
processes = [mp.Process(target=worker,args=(working_q, output_q)) for i in range(mp.cpu_count())]
for proc in processes:
proc.start()
for proc in processes:
proc.join()
results_bank = []
while True:
if output_q.empty() is True:
break
else:
results_bank.append(output_q.get())
print len(results_bank) # length of this list should be equal to static_input, which is the range used to populate the input queue. In other words, this tells whether all the items placed for processing were actually processed.
results_bank.sort()
print results_bank
Has anyone any idea about how to make this code to run properly?
This code will never stop:
Each worker gets an item from the queue as long as it is not empty:
picked = working_queue.get()
and puts a new one for each that it got:
working_queue.put(picked+1)
As a result the queue will never be empty except when the timing between the process happens to be such that the queue is empty at the moment one of the processes calls empty(). Because the queue length is initially 100 and you have as many processes as cpu_count() I would be surprised if this ever stops on any realistic system.
Well executing the code with slight modification proves me wrong, it does stop at some point, which actually surprises me. Executing the code with one process there seems to be a bug, because after some time the process freezes but does not return. With multiple processes the result is varying.
Adding a short sleep period in the loop iteration makes the code behave as I expected and explained above. There seems to be some timing issue between Queue.put, Queue.get and Queue.empty, although they are supposed to be thread-safe. Removing the empty test also gives the expected result (without ever getting stuck at an empty queue).
Found the reason for the varying behaviour. The objects put on the queue are not flushed immediately. Therefore empty might return False although there are items in the queue waiting to be flushed.
From the documentation:
Note: When an object is put on a queue, the object is pickled and a
background thread later flushes the pickled data to an underlying
pipe. This has some consequences which are a little surprising, but
should not cause any practical difficulties – if they really bother
you then you can instead use a queue created with a manager.
After putting an object on an empty queue there may be an infinitesimal delay before the queue’s empty() method returns False and get_nowait() can return without raising Queue.Empty.
If multiple processes are enqueuing objects, it is possible for the objects to be received at the other end out-of-order. However, objects enqueued by the same process will always be in the expected order with respect to each other.
I am relatively new to programming, and what I am asking might be a task that is not possible. What I want to do is start a parallel process, which will continuously run until requested to stop by the user. Once the process has started, I would like to update one of the local variables inside the parallel process from the parent program without stopping the process. For a very simple example, the parallel process I would like to execute would be the following:
import time
def loop(i):
while 1:
print i
i+=1
time.sleep(1)
Which continuously iterates and updates i. For clarity the parent program will contain:
from multiprocessing import Process
from loop import loop
i = 1
p = Process(target = loop, args = (i))
p.start()
From the parent program, once "loop" has be initiated, I would like to be able to change the input "i" to another number and have loop continue to iterate given the next starting value. If anybody has any ideas on how to do this, it would be greatly appreciated.
You can use a queue to pass data between the processes:
from multiprocessing import Process, Queue
from loop import loop
i = 1
q = Queue()
q.put(i)
p = Process(target=loop, args=(q, ))
p.start()
Whenever you want to transmit a new value of i to the other process, just put it
in the queue.
Change your loop.py module accordingly:
def loop(q):
while True:
i = q.get()
print i
In your main code, you can put new values for the process:
while True:
i+=1
q.put(i)
time.sleep(1)
Nothing wrong with a queue here, but it's probably more idiomatic to use "shared memory" instead. Less overhead. Here's an example self-contained program:
import time
def loop(i):
while 1:
print i.value
i.value += 1
time.sleep(1)
if __name__ == "__main__":
from multiprocessing import Process, Value
i = Value("i", 1) # "i" for integer, initial value 1
p = Process(target=loop, args=(i,))
p.start()
for base in range(100, 600, 100):
time.sleep(2)
i.value = base
That will probably ;-) display:
1
2
100
101
200
201
300
301
400
401
500
501
502
503
504
505
506
...
But caution: in return for being speedier, this is also more brittle. The kinds of data you can share this way are basically inherited from the C language, and aren't generally as rich as Python data types. In the example above, the type code "i" means we're sharing a signed C int, which is usually a 32-bit signed integer. If, for example, i.value reaches 2147483647, and we add 1 to it, it will suddenly become -2147483648 (yup, negative!). Python ints are unbounded, but C ints aren't.
Use a queue. (python 2 is Queue)
There are a few ways to consume from the queue. The naive implementation (i.e. the one that has a race condition) is simply to do if q.empty(). This is bad practice but wouldn't hurt you since it's not mission critical if you miss a tick.
The better method is to use exceptions for flow control:
q = queue.Queue()
try:
q.get(False) #non-blocking get()
#update value of i, etc
except queue.Empty:
#do stuff here as before
I have a problem running multiple processes in python3 .
My program does the following:
1. Takes entries from an sqllite database and passes them to an input_queue
2. Create multiple processes that take items off the input_queue, run it through a function and output the result to the output queue.
3. Create a thread that takes items off the output_queue and prints them (This thread is obviously started before the first 2 steps)
My problem is that currently the 'function' in step 2 is only run as many times as the number of processes set, so for example if you set the number of processes to 8, it only runs 8 times then stops. I assumed it would keep running until it took all items off the input_queue.
Do I need to rewrite the function that takes the entries out of the database (step 1) into another process and then pass its output queue as an input queue for step 2?
Edit:
Here is an example of the code, I used a list of numbers as a substitute for the database entries as it still performs the same way. I have 300 items on the list and I would like it to process all 300 items, but at the moment it just processes 10 (the number of processes I have assigned)
#!/usr/bin/python3
from multiprocessing import Process,Queue
import multiprocessing
from threading import Thread
## This is the class that would be passed to the multi_processing function
class Processor:
def __init__(self,out_queue):
self.out_queue = out_queue
def __call__(self,in_queue):
data_entry = in_queue.get()
result = data_entry*2
self.out_queue.put(result)
#Performs the multiprocessing
def perform_distributed_processing(dbList,threads,processor_factory,output_queue):
input_queue = Queue()
# Create the Data processors.
for i in range(threads):
processor = processor_factory(output_queue)
data_proc = Process(target = processor,
args = (input_queue,))
data_proc.start()
# Push entries to the queue.
for entry in dbList:
input_queue.put(entry)
# Push stop markers to the queue, one for each thread.
for i in range(threads):
input_queue.put(None)
data_proc.join()
output_queue.put(None)
if __name__ == '__main__':
output_results = Queue()
def output_results_reader(queue):
while True:
item = queue.get()
if item is None:
break
print(item)
# Establish results collecting thread.
results_process = Thread(target = output_results_reader,args = (output_results,))
results_process.start()
# Use this as a substitute for the database in the example
dbList = [i for i in range(300)]
# Perform multi processing
perform_distributed_processing(dbList,10,Processor,output_results)
# Wait for it all to finish.
results_process.join()
A collection of processes that service an input queue and write to an output queue is pretty much the definition of a process pool.
If you want to know how to build one from scratch, the best way to learn is to look at the source code for multiprocessing.Pool, which is pretty simply Python, and very nicely written. But, as you might expect, you can just use multiprocessing.Pool instead of re-implementing it. The examples in the docs are very nice.
But really, you could make this even simpler by using an executor instead of a pool. It's hard to explain the difference (again, read the docs for both modules), but basically, a future is a "smart" result object, which means instead of a pool with a variety of different ways to run jobs and get results, you just need a dumb thing that doesn't know how to do anything but return futures. (Of course in the most trivial cases, the code looks almost identical either way…)
from concurrent.futures import ProcessPoolExecutor
def Processor(data_entry):
return data_entry*2
def perform_distributed_processing(dbList, threads, processor_factory):
with ProcessPoolExecutor(processes=threads) as executor:
yield from executor.map(processor_factory, dbList)
if __name__ == '__main__':
# Use this as a substitute for the database in the example
dbList = [i for i in range(300)]
for result in perform_distributed_processing(dbList, 8, Processor):
print(result)
Or, if you want to handle them as they come instead of in order:
def perform_distributed_processing(dbList, threads, processor_factory):
with ProcessPoolExecutor(processes=threads) as executor:
fs = (executor.submit(processor_factory, db) for db in dbList)
yield from map(Future.result, as_completed(fs))
Notice that I also replaced your in-process queue and thread, because it wasn't doing anything but providing a way to interleave "wait for the next result" and "process the most recent result", and yield (or yield from, in this case) does that without all the complexity, overhead, and potential for getting things wrong.
Don't try to rewrite the whole multiprocessing library again. I think you can use any of multiprocessing.Pool methods depending on your needs - if this is a batch job you can even use the synchronous multiprocessing.Pool.map() - only instead of pushing to input queue, you need to write a generator that yields input to the threads.
I'm trying to understand multiprocessing in python.
from multiprocessing import Process
def multiply(a,b):
print(a*b)
return a*b
if __name__ == '__main__':
p = Process(target= multiply, args= (5,4))
p.start()
p.join()
print("ok.")
In this codeblock, for example, if there was an variable that called "result". How can we assign return value of multiply function to "result"?
And a little problem about IDLE: when i'm tried to run this sample with Python Shell, it doesn't work properly? If i double click .py file, output is like that:
20
ok.
But if i try to run this in IDLE:
ok.
Thanks...
Ok, i somehow managed this. I looked to python documentation, and i learnt that: with using Queue class, we can get return values from a function. And final version of my code is like this:
from multiprocessing import Process, Queue
def multiply(a,b,que): #add a argument to function for assigning a queue
que.put(a*b) #we're putting return value into queue
if __name__ == '__main__':
queue1 = Queue() #create a queue object
p = Process(target= multiply, args= (5,4,queue1)) #we're setting 3rd argument to queue1
p.start()
print(queue1.get()) #and we're getting return value: 20
p.join()
print("ok.")
And there is also a pipe() function, i think we can use pipe() function,too. But Queue worked for me, now.
Does this help? This takes a list of functions (and their arguments), runs them in parallel,
and returns their outputs.: (This is old. Much newer version of this is at https://gitlab.com/cpbl/cpblUtilities/blob/master/parallel.py )
def runFunctionsInParallel(listOf_FuncAndArgLists):
"""
Take a list of lists like [function, arg1, arg2, ...]. Run those functions in parallel, wait for them all to finish, and return the list of their return values, in order.
(This still needs error handling ie to ensure everything returned okay.)
"""
from multiprocessing import Process, Queue
def storeOutputFFF(fff,theArgs,que): #add a argument to function for assigning a queue
print 'MULTIPROCESSING: Launching %s in parallel '%fff.func_name
que.put(fff(*theArgs)) #we're putting return value into queue
queues=[Queue() for fff in listOf_FuncAndArgLists] #create a queue object for each function
jobs = [Process(target=storeOutputFFF,args=[funcArgs[0],funcArgs[1:],queues[iii]]) for iii,funcArgs in enumerate(listOf_FuncAndArgLists)]
for job in jobs: job.start() # Launch them all
for job in jobs: job.join() # Wait for them all to finish
# And now, collect all the outputs:
return([queue.get() for queue in queues])