I am relatively new to programming, and what I am asking might be a task that is not possible. What I want to do is start a parallel process, which will continuously run until requested to stop by the user. Once the process has started, I would like to update one of the local variables inside the parallel process from the parent program without stopping the process. For a very simple example, the parallel process I would like to execute would be the following:
import time
def loop(i):
while 1:
print i
i+=1
time.sleep(1)
Which continuously iterates and updates i. For clarity the parent program will contain:
from multiprocessing import Process
from loop import loop
i = 1
p = Process(target = loop, args = (i))
p.start()
From the parent program, once "loop" has be initiated, I would like to be able to change the input "i" to another number and have loop continue to iterate given the next starting value. If anybody has any ideas on how to do this, it would be greatly appreciated.
You can use a queue to pass data between the processes:
from multiprocessing import Process, Queue
from loop import loop
i = 1
q = Queue()
q.put(i)
p = Process(target=loop, args=(q, ))
p.start()
Whenever you want to transmit a new value of i to the other process, just put it
in the queue.
Change your loop.py module accordingly:
def loop(q):
while True:
i = q.get()
print i
In your main code, you can put new values for the process:
while True:
i+=1
q.put(i)
time.sleep(1)
Nothing wrong with a queue here, but it's probably more idiomatic to use "shared memory" instead. Less overhead. Here's an example self-contained program:
import time
def loop(i):
while 1:
print i.value
i.value += 1
time.sleep(1)
if __name__ == "__main__":
from multiprocessing import Process, Value
i = Value("i", 1) # "i" for integer, initial value 1
p = Process(target=loop, args=(i,))
p.start()
for base in range(100, 600, 100):
time.sleep(2)
i.value = base
That will probably ;-) display:
1
2
100
101
200
201
300
301
400
401
500
501
502
503
504
505
506
...
But caution: in return for being speedier, this is also more brittle. The kinds of data you can share this way are basically inherited from the C language, and aren't generally as rich as Python data types. In the example above, the type code "i" means we're sharing a signed C int, which is usually a 32-bit signed integer. If, for example, i.value reaches 2147483647, and we add 1 to it, it will suddenly become -2147483648 (yup, negative!). Python ints are unbounded, but C ints aren't.
Use a queue. (python 2 is Queue)
There are a few ways to consume from the queue. The naive implementation (i.e. the one that has a race condition) is simply to do if q.empty(). This is bad practice but wouldn't hurt you since it's not mission critical if you miss a tick.
The better method is to use exceptions for flow control:
q = queue.Queue()
try:
q.get(False) #non-blocking get()
#update value of i, etc
except queue.Empty:
#do stuff here as before
Related
So basically, I've this function th() which counts till certain number and then prints "done".
I'd want to start n number of such threads at the same time, running simultaneously.
So I wrote:
thread_num = 3 #here n is 3, but I'd normally want something way higher
thrds = []
i = 0
while i < thread_num:
thr = Thread(target=th, args=())
thrds.append(thr)
i += 1
print("thread", str(i), "added")
for t in thrds:
t.start()
t.join()
I'd want all the threads to print "done" at the same time, but they have a noticeable lag in between of them. They print "thread i started" at seemingly the same time, but print "done" with quite a bit of time lag.
Why is this happening?
Edit: Since someone asked me to add th() function as well, here it is:
def th():
v = 0
num = 10**7
while v < num:
v += 1
print("done")
This is happening because of the t.join() method that you are calling on each thread before start the next one. t.join() blocks the execution of the current thread until the thread t has completed execution. So, each thread is starting after the previous one has finished.
You first have to start all the threads, then join all the threads in separate for loops; otherwise, each thread starts but runs to completion due to join before starting another thread.
for t in thrds: # start all the threads
t.start()
for t in thrds: # wait for all threads to finish
t.join()
If you only have a simple counting thread, you may need to add some short sleep to actually see the threads output intermingle as they may still run fast enough to complete before another thread starts.
Because you start and join each thread sequentially, one thread will run to completion before the next even starts. You'd be better off running a thread pool which is a more comprehensive implementation that handles multiple issues in multithreading.
Because of memory management and object reference count issues, python only lets a single thread execute byte code at a time. Periodically, each thread will release and reacquire the Global Interpreter Lock (GIL) to let other threads run. Exactly which thread runs at any given time is up to the operating system and you may find one gets more slices than another, causing staggered results.
To get them all to print "done" at the same time, you could use a control structure like a barrier for threads to wait until all are done. With a barrier, all threads must call wait before any can continue.
thread_num = 3 #here n is 3, but I'd normally want something way higher
wait_done = threading.Barrier(thread_num)
def th(waiter):
x = 1 # to what you want
waiter.wait()
print("done")
thrds = []
i = 0
while i < thread_num:
thr = Thread(target=th, args=(wait_done,))
thrds.append(thr)
i += 1
print("thread", str(i), "added")
for t in thrds:
t.start()
for t in thrds:
t.join()
I have a basic question that ragards the Python multiprocessing method, how different processes, which use queues to transfer data, could optimally be started.
For that I use a simple example where
Data is received
Data is processed
Data is send
All of the upper steps should happen in parallel through three different processes.
Here the example code:
import multiprocessing
import keyboard
import time
def getData(queue_raw):
for num in range(1000):
queue_raw.put(num)
print("getData: put "+ str(num)+" in queue_raw")
while True:
if keyboard.read_key() == "s":
break
def calcFeatures(queue_raw, queue_features):
while not queue_raw.empty():
data = queue_raw.get()
queue_features.put(data**2)
print("calcFeatures: put "+ str(data**2)+" in queue_features")
def sendFeatures(queue_features):
while not queue_features.empty():
feature = queue_features.get()
print("sendFeatures: put "+ str(feature)+" out")
if __name__ == "__main__":
queue_raw = multiprocessing.Queue()
queue_features = multiprocessing.Queue()
processes = [
multiprocessing.Process(target=getData, args=(queue_raw,)),
multiprocessing.Process(target=calcFeatures, args=(queue_raw, queue_features,)),
multiprocessing.Process(target=sendFeatures, args=(queue_features,))
]
processes[0].start()
time.sleep(0.1)
processes[1].start()
time.sleep(0.1)
processes[2].start()
#for p in processes:
# p.start()
for p in processes:
p.join()
This program works, but my question is regarding the start of the different processes.
Ideally process[1] should start only if process[0] put data in the queue_raw; while process[2] should only start if process[1] put the calculated features in queue_features.
Right now I did that through time.sleep() function, which is suboptimal, since I don't necessarily know how long the processes will take.
I also tried something like:
processes[0].start()
while queue_raw.empty():
time.sleep(0.5)
processes[1].start()
But it won't work, since only the first process is estimated. Any method how this process depending starts could be done?
#moooeeeep pointed out the right comment.
Checking with while not queue.empty(): is not waiting till data is actually in the queue!
An approach via a sentinel object (here None) and a while True loop will enforce that the process waits till the other processes put data in the queue:
FLAG_STOP=False
while FLAG_STOP is False:
data = queue_raw.get() # get will wait
if data is None:
# Finish analysis
FLAG_STOP = True
else:
# work with data
I am trying to learn the multiprocessing library in Python but I cannot get my code to work with queue.Queue. Simply put, I have no idea where to put the queue.Queue.join() method in my code. Does it go IN the while loop or outside of it? If it goes outside of the while loop, do I write while q.not_empty? Why would I use q.not_empty when it the docs it is explicitly mentioned to use join()?
Here is my code. I am expecting my 4 cores, simultaneously to return the amount of prime numbers computed by my function, 2 times per core for the total amount of 8 computations. The prime computing functions works without problem.
import queue
def main():
q = queue.Queue()
[q.put((compute_primes, (1, 30000))) for _ in range(8)]
with multiprocessing.Pool(processes=4) as pool:
while q.not_empty:
result = q.get()
function = pool.apply_async(result[0], args=(result[1][0], result[1][1]))
function.get()
q.join()
With the code above, I break out of the loop if the queue is empty. But this is supposed to be unrealiable and why would I need q.join() afterwards?
With the code below, I can't break out of the loop. Changes are while True and position of q.join()
def main():
q = queue.Queue()
[q.put((compute_primes, (1, 30000))) for _ in range(8)]
with multiprocessing.Pool(processes=4) as pool:
while True:
result = q.get()
function = pool.apply_async(result[0], args=(result[1][0], result[1][1]))
function.get()
q.join()
Where should I put q.join?
P.S. This code also does not parrallelise the tasks effectively, it essentially computes the functions one by one and I cannot understand why, but this is a different problem.
P.S. 2
Code for prime function
def compute_primes(start, end):
start_time = time.time()
primes = []
for number in range(start, end + 1):
flag = True
for i in range(2, number):
if (number % i) == 0:
flag = False
break
if flag:
primes.append(number)
end_time = time.time()
print(f"Time taken: {end_time - start_time}\n"
f"Amount primes: {len(primes)}")
return primes
Queues and Pools
Running one at a time... separate problem.
Actually, this is part of the same problem. All this implies that you are not
using a multiprocessing pool managed by Pool. What you do at the moment is to
put all your tasks in a queue, get them straight back out again, and then
process them one at a time using a pool, which only ever gets one task at a
time. These two paradigms are mutually exclusive: if you want to use a pool to
do the work for you, you don't need queue; if you need to handle the queue
yourself, you probably don't want to use pool.
Pool
multiprocessing.Pool and accompanying methods spawn the right number of worker
processes, serialise your function to them, and then set up a queue internally
and handle sending tasks and getting results. This is much easier than doing it
manually, and is normally the right way to go about things:
When you use pool, you do something like this:
results = pool.map(compute_primes, [(0,100_000) for _ in range(8)])
Which will block for you until all the pool has finished, or:
results = pool.map_async(compute_primes, [(0, 100_000) for _ in range(8)])
results.wait() # wait
unless you plan to process the results as they come in, in which case you don't
use results.wait() at all:
for _ in range(8):
result = results.get()
do_stuff(result)
You do use pool.join() or pool.close() just to make sure the pool is shut
down gracefully, but this has nothing to do with getting your results.
Your examples
Your first example works, because you do this:
put tasks in a queue
get them out one by one and process them
join an empty queue -> leave immediately
Your second example fails, because you do this:
put tasks in a queue
get one task out
wait for queue to be empty or done -> blocks indefinitely
In this case, you don't want a queue at all.
Using Queue manually
Aside: where are you getting your Queue from? multiprocessing.Queue is not
joinable; you need multiprocessing.JoinableQueue. threading.Queue should
not be used with multiprocessing. queue.Queue, likewise, should not be used
with `multiprocessing.
When do you use a task queue? When you don't just want to apply a bunch of
arguments to a bunch of functions. Perhaps you want to use a custom class.
Perhaps you want to do something funny. Perhaps you want to do some things with
one type of argument, but something else if the argument is of a certain kind,
and code is better organised this way. In these cases subclassing Process (or
Thread for multithreading) yourself may be clearer and cleaner. None of that
seems to apply in this case.
Using join with queues
.join() is for task queues. It blocks until every task in the queue has
been marked as done. This is handy for when you want to offload some processing
to a bunch of processes, but wait for them before you do anything. so you
normally do something like:
tasks = JoinableQueue()
for t in qs:
tasks.put(t)
start_multiprocessing() # dummy fn
tasks.join() # wait for everything to be done
However in this case you don't do that, or want to do it.
I prefer not to specify an argument to the Pool constructor unless, for some reason, I want very few concurrent processes. By constructing Pool without an argument, the number of potential concurrent processes will vary from computer to computer depending on its CPU architecture. Here's how I would implement your task (assuming I understand your use-case completely):
from multiprocessing import Pool
def genPrime(): # prime number generator
D = {}
q = 2
while True:
if q not in D:
yield q
D[q * q] = [q]
else:
for p in D[q]:
D.setdefault(p + q, []).append(p)
del D[q]
q += 1
def compute_primes(n):
g = genPrime()
return [next(g) for _ in range(n)]
NCOMPUTATIONS = 8
NPRIMES = 30_000
def main():
with Pool() as pool:
ar = []
for _ in range(NCOMPUTATIONS):
ar.append(pool.apply_async(compute_primes, [NPRIMES]))
for _ar in ar:
result = _ar.get() # waits for process to terminate and get its return value
assert len(result) == NPRIMES
if __name__ == '__main__':
main()
[ Please note that I am not the author of the genPrime function ]
I have a class that starts multiple threads upon initialization. Originally I was using threading, but I learned the hard way how painfully slow it can get. As I researched this, it seems that multiprocessing would be faster because it actually utilizes multiple cores. The only hard part is the fact that it doesn't automatically share values. How could I make the following code share self across all processes?
Ideally, it would also share across processes outside of the class as well.
Also, I would rather share the entire class than share each individual value, if possible.
import multiprocessing as mp
from time import sleep
class ThreadedClass:
def __init__(self):
self.var = 0
#Here is where I would want to tell multiprocessing to share 'self'
change_var = mp.Process(target=self.change_var, args=())
print_var = mp.Process(target=self.print_var, args=())
change_var.start()
sleep(0.5)
print_var.start()
def change_var(self):
while True:
self.var += 1
print("Changed var to ", self.var)
sleep(1)
def print_var(self):
while True:
print("Printing var: ", self.var)
sleep(1)
ThreadedClass()
I also included output of the above code below:
Changed var to 1
Printing var: 0
Changed var to 2
Printing var: 0
Changed var to 3
Printing var: 0
Changed var to 4
Printing var: 0
Changed var to 5
Printing var: 0
Changed var to 6
Printing var: 0
Changed var to 7
Printing var: 0
Changed var to 8
Printing var: 0
Changed var to 9
Printing var: 0
Changed var to 10
Thanks in advance.
First of all, multiprocessing means that you are making sub-processes. This means that in general, they have their own space in memory and don't talk to each other. To be clear, when you start a new multiprocessing thread, python copies all your global variables into that thread and then runs that thread separate from everything else. So, when you spawned your two processes, change_var and print_var, each of them received a copy of self, and since their are two copies of self, neither of them is talking to each. One thread is updating it's own copy of self and producing answers that are counting, the other is not updating self. You can easily test this yourself:
import multiprocessing as mp
LIST = [] # This list is in parent process.
def update(item):
LIST.append(item)
p = mp.Process(target=update, args=(5,)) # Copies LIST, update, and anything else that is global.
p.start()
p.join()
# The LIST in the sub-process is cleaned up in memory when the process ends.
print(LIST) # The LIST in the parent process is not updated.
It would be very dangerous if different processes were updating each other's variables while they were trying to process with them; hence, naturally to isolate them (and prevent "segmentation faults"), the entire namespace is copied. If you want sub-processes to talk to each other, you need to communicate with a manager and Queue that is designed for that.
I personally recommend to write your code around things like a Pool() instead. Very clean, input an array, get back an array, done. But if you want to go down the rabbit hole, here is what I read on the multiprocessing website.
import multiprocessing as mp
def f(queue):
queue.put(['stuff',15])
def g(queue):
queue.put(['other thing'])
queue = mp.Queue()
p = mp.Process(target=f,args=(queue,))
q = mp.Process(target=g,args=(queue,))
p.start()
q.start()
for _ in range(2):
print(queue.get())
p.join()
q.join()
The main idea is that the queue does not get copied and instead allows things to be left in the queue. When the you run queue.get() it waits for something in the queue to be gotten that was left by some other process. queue.get() blocks and waits. This means you could have one process read the contents of the other process, like:
import multiprocessing as mp
def f(queue):
obj = queue.get() # Blocks this sub-process until something shows up.
if obj:
print('Something was in the queue from some other process.')
print(obj)
def g(queue):
queue.put(['leaving information here in queue'])
queue = mp.Queue()
p = mp.Process(target=f,args=(queue,))
q = mp.Process(target=g,args=(queue,))
p.start()
This is kindof cool, so I recommend waiting here a second to think about what is waiting to process. Next start the q process.
q.start()
Notice that p didn't get to finish processing until q was started. This is because the Queue blocked and waited for something to show up.
# clean up
p.join()
q.join()
You can read more at: https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#multiprocessing.Queue
I would like to control the number of Processes spawned while using the multiprocessing package.
Say I only want three processes active at the same time. The only way I know how to do this is:
import multiprocessing
import Queue
def worker(arg):
## Do stuff
return returnvalue
argument = list(1,2,3,4,5,6)
aliveprocesses = 0
jobs = Queue.Queue()
for arg in argument:
while jobs.qsize() > 2:
jobs.get().join()
p = multiprocessing.Process(target=worker,args=(arg,))
jobs.put(p)
p.start()
Basically I only know how to monitor one process at a time using the Process.join() function. I monitor the oldest process until it is done and then create a new process. For my program the oldest process should finish before the others, on average. But who knows? Maybe another process finishes first and I would have no way of knowing.
The only alternative I can think of is something like this:
import multiprocessing
import time
def worker(arg):
## Do stuff
return returnvalue
argument = list(1,2,3,4,5,6)
aliveprocesses = 0
jobs = set()
for arg in argument:
while aliveprocesses > 2:
for j in jobs:
if not j.is_alive():
aliveprocesses -= 1
break
time.sleep(1)
p = multiprocessing.Process(target=worker,args=(arg,))
jobs.put(p)
p.start()
aliveprocesses += 1
In the above function you are checking all of processes if they are still alive. If they are all still alive you sleep for a bit and then check again until there is a dead process after which you spawn a new process. The problem here is that from what I understand the time.sleep() function is not a particularly efficient way to wait for a process to end.
Ideally I would like a function "superjoin()" like Process.join() only it uses a set of Process objects and when one Process within the set returns then superjoin() returns. And superjoin() does not itself use the time.sleep() function ie it's not being "passed the buck"
Since you seem to have a single (parallel) task, instead of managing processes individually, you should use the higher-level multiprocessing.Pool, which makes managing the number of processes easier.
You can't join a pool, but you have blocking calls (such as Pool.map) that perform this kind of task.
If you need finer-grained control, you may want to adapt Pool's source code