Could someone please shed some light on why this threaded code to call a classes' method never completes?
from Queue import Queue
from threading import Thread
class SimpleThing(object):
def __init__(self, name):
self.name = name
def print_name(self):
print self.name
class ThingSpawner(object):
def __init__(self, name_list):
self.things = [SimpleThing(name) for name in name_list]
self.thread_queue = Queue()
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
self.thread_queue.put(t)
self.thread_queue.join()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
The code will clearly run the print_name method, but does not join() and exit.
Also, what is the neatest way to modify this code so that the join() completes? The motivation is to use an existing python control class for a bit of hardware, and allows you to call a (very slow) method of the control class in parallel. Thanks!
When you are doing
self.thread_queue.put(t)
You are putting some threads into the Queue, obviously. However, i'm not really sure why. You never use that queue again for anything, and it's completely unnecessary. To make matters worse, you then call
self.thread_queue.join()
Which basically waits forever for the queue to empty, which never happens, because you never empty it or do anything with it.
If I copy paste all your code, but without any Queue at all, everything is fine...
from threading import Thread
class SimpleThing(object):
def __init__(self, name):
self.name = name
def print_name(self):
print self.name
class ThingSpawner(object):
def __init__(self, name_list):
self.things = [SimpleThing(name) for name in name_list]
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
However that's not what you want! Because your threads are daemons they will exit when the main program exits, even if they are not done yet (if I add some delay like sleep(1) before printing the name for example). You should call join() on the threads, not the queue, if you want to wait for them to finish. So we'll return the threads first:
def run(self):
all_threads = []
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
all_threads.append(t)
return all_threads
And when we run we'll do this:
threads = sp.run()
for t in threads:
t.join()
Thanks Ofer for the clear answer, which I've just accepted -- I am indeed not using the queue properly! Having reacquainted myself with queues now you've pointed out my error, for prosperity, here's an alternative approach using a queue:
from Queue import Queue
from threading import Thread
class SimpleThing(object):
def __init__(self, name, q):
self.name = name
def print_name(self, q):
print self.name
q.get()
q.task_done()
class ThingSpawner(object):
def __init__(self, name_list):
self.thread_queue = Queue()
self.things = [SimpleThing(name, self.thread_queue) for name in name_list]
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name, args=(self.thread_queue,))
t.daemon = True
t.start()
self.thread_queue.put(t)
self.thread_queue.join()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
This tutorial on threading and queues was useful once I understood my mistake.
Related
Here is my code below , I put string in queue , and hope dowork2 to do something work , and return char in shared_queue
but I always get nothing at while not shared_queue.empty()
please give me some point , thanks.
import time
import multiprocessing as mp
class Test(mp.Process):
def __init__(self, **kwargs):
mp.Process.__init__(self)
self.daemon = False
print('dosomething')
def run(self):
manager = mp.Manager()
queue = manager.Queue()
shared_queue = manager.Queue()
# shared_list = manager.list()
pool = mp.Pool()
results = []
results.append(pool.apply_async(self.dowork2,(queue,shared_queue)))
while True:
time.sleep(0.2)
t =time.time()
queue.put('abc')
queue.put('def')
l = ''
while not shared_queue.empty():
l = l + shared_queue.get()
print(l)
print( '%.4f' %(time.time()-t))
pool.close()
pool.join()
def dowork2(queue,shared_queue):
while True:
path = queue.get()
shared_queue.put(path[-1:])
if __name__ == '__main__':
t = Test()
t.start()
# t.join()
# t.run()
I managed to get it work by moving your dowork2 outside the class. If you declare dowork2 as a function before Test class and call it as
results.append(pool.apply_async(dowork2, (queue, shared_queue)))
it works as expected. I am not 100% sure but it probably goes wrong because your Test class is already subclassing Process. Now when your pool creates a subprocess and initialises the same class in the subprocess, something gets overridden somewhere.
Overall I wonder if Pool is really what you want to use here. Your worker seems to be in an infinite loop indicating you do not expect a return value from the worker, only the result in the return queue. If this is the case, you can remove Pool.
I also managed to get it work keeping your worker function within the class when I scrapped the Pool and replaced with another subprocess:
foo = mp.Process(group=None, target=self.dowork2, args=(queue, shared_queue))
foo.start()
# results.append(pool.apply_async(Test.dowork2, (queue, shared_queue)))
while True:
....
(you need to add self to your worker, though, or declare it as a static method:)
def dowork2(self, queue, shared_queue):
in my project i have a class of threading.Thread like this:
class MakeHtml(threading.Thread):
def __init__(self, *rstext):
self.outhtml = [x for x in rstext]
self.retval = ''
threading.Thread.__init__(self)
def run(self):
...do something
in another file i call, every 10 seconds MakeHtml class
t = MakeHtml(mrr1, mrr2, mrr3, mrr4)
for create a thread but in this way i see that the thread is the same every time.
I need a new thread every time i call the MakeHtml Threading class, how can i do this?
Thanks in advance
MakeHtml extends Thread, but if you have only 1 instance of MakeHtml, you will have only one thread
For instance if you want 2 different thread you will have to do
t = MakeHtml(mrr1, mrr2, mrr3, mrr4) # one thread
t1 = MakeHtml(mrr1, mrr2, mrr3, mrr4) # another one
You can use :
import threading
def afunction(mm):
# do job
pass
threads = []
for mm in [mmr1, mmr2, mmr3n mmr4]:
t = threading.Thread(target=afunction, args=[mm,])
threads.append(t)
t.start()
[t.join() for t in threads]
I have a boolean that I want to pass to different threads that are executing methods from different modules. This boolean acts as a cancellation token so if set, the thread should exit. It seems to be passed by value since if I set it in another thread it doesn't change in the other threads. Thanks.
import module2
from threading import Thread
cancellationToken = False
def main:
thread2 = Thread(target = module2.method2, args (on_input, cancellationToken, ))
thread2.start()
...
thread2.join()
def on_input(command):
global cancellationToken
...
if(...):
cancellationToken = True
...
method2 in module2 is just a simple infinite while loop that checks the cancellation token and responds to user input.
def method2(on_input, cancellationToken):
while(True):
if(cancellationToken):
return
...
on_input(...)
When you do this:
thread2 = Thread(target = module2.method2, args (on_input, cancellationToken, ))
You're essentially passing the value False for the 2nd argument to the thread method.
But when you do this after that:
cancellationToken = True
You're replacing the reference represented by cancellationToken, but not the value that was originally passed to thread2.
To achieve what you want to do, you'll need to create a mutable object wrapper for your cancellation state:
class CancellationToken:
def __init__(self):
self.is_cancelled = False
def cancel(self):
self.is_cancelled = True
cancellationToken = CancellationToken()
thread2 = Thread(target = module2.method2, args (on_input, cancellationToken, ))
# then later on
cancellationToken.cancel()
Your thread code becomes:
def method2(on_input, cancellationToken):
while(True):
if(cancellationToken.is_cancelled):
return
...
on_input(...)
I have a little doubt if one could solve my issue, and create successful communication between threads.
First example and this is how it should be working, but does not work well:
import Queue,threading,time
class th(threading.Thread):
def __init__(self,q):
threading.Thread.__init__(self)
self.q = q
self.t = time
def run(self):
for i in range(5):
self.q.put(i)
self.t.sleep(0.5) # <----------
self.q.put('end')
class main(object):
def __init__(self):
self.q = Queue.Queue()
self.thread = th(self.q)
self.thread.setDaemon(True)
self.thread.run()
self.call()
def call(self):
while True:
recv = self.q.get();
if recv == 'end':
break
else:
print recv
if __name__ == '__main__':
root = main()
root.call()
In this example, all printed at the same time:
0,1,2,3,4
Second example:
import Queue,threading,time
class th(threading.Thread):
def __init__(self,q):
threading.Thread.__init__(self);
self.q = q
self.t = time
def run(self):
for i in range(5):
self.q.put(i) # <------ no sleep()
self.q.put('end')
class main(object):
def __init__(self):
self.q = Queue.Queue()
self.thread = th(self.q)
self.thread.setDaemon(True)
self.thread.run()
self.call()
def call(self):
while True:
recv = self.q.get()
if recv == 'end':
break
else:
print recv
if __name__ == '__main__':
root = main()
root.call()
the code is printed as it has to
0,
1
2
3
4
one to one
is there any way that the sleep function in the same way?
You don't want to call the run method on a thread directly. Call start instead, which will kick off the child thread, which will in turn run the run method.
Your current code is essentially single threaded, since the run call does the work of the child thread in the parent instead. The child thread is never actually started! (You're also calling your main.call method twice, which I'd expect to block or raise an exception, but that's a separate issue.)
sorry, it was something very simple, really simple, just had to replace
self.thread.run()
by
self.threat.start()
I want two objects of the same class to operate concurrently. The class "MyClass" has a function that connects an instance to another instance of the class. I also need to keep track of the objects that have been created (oList). what I am trying is:
main.py:
from MyClass import MyClass
import time
oList = []
class oCreator1(Thread):
def __init__(self):
Thread.__init__(self)
self.o1 = MyClass()
def run(self):
while 1:
time.sleep(1)
print "Hi"
def getO1(self):
return self.o1
class oCreator2(Thread):
def __init__(self):
Thread.__init__(self)
self.o2 = MyClass()
def run(self):
while 1:
time.sleep(1)
print "Bye!"
def getO2(self):
return self.o2
main():
threadList = []
global oList
oc1 = oCreator1()
threadList.append(oc1)
o1 = oc1.getO1()
oList.append(o1)
oc2 = oCreator2()
threadList.append(oc2)
o2 = oc2.getO2()
oList.append(o2)
o1.connToAnotherO(o2)
print oList
for t in threadList:
t.start()
t.join()
if __name__ == '__main__':
main()
But the only thing that is printed is "Hi". I really want to know the things I'm doing wrong and the right way to do it. Thank you in advance.
for t in threadList:
t.start()
t.join()
The t.join() call waits for the thread t to finish. That means when you start the first thread, you wait for it to finish before starting the second, but the first thread is just going to keep printing Hi forever. It'll never finish.
Don't join, or don't start joining until all threads have started.