I want two objects of the same class to operate concurrently. The class "MyClass" has a function that connects an instance to another instance of the class. I also need to keep track of the objects that have been created (oList). what I am trying is:
main.py:
from MyClass import MyClass
import time
oList = []
class oCreator1(Thread):
def __init__(self):
Thread.__init__(self)
self.o1 = MyClass()
def run(self):
while 1:
time.sleep(1)
print "Hi"
def getO1(self):
return self.o1
class oCreator2(Thread):
def __init__(self):
Thread.__init__(self)
self.o2 = MyClass()
def run(self):
while 1:
time.sleep(1)
print "Bye!"
def getO2(self):
return self.o2
main():
threadList = []
global oList
oc1 = oCreator1()
threadList.append(oc1)
o1 = oc1.getO1()
oList.append(o1)
oc2 = oCreator2()
threadList.append(oc2)
o2 = oc2.getO2()
oList.append(o2)
o1.connToAnotherO(o2)
print oList
for t in threadList:
t.start()
t.join()
if __name__ == '__main__':
main()
But the only thing that is printed is "Hi". I really want to know the things I'm doing wrong and the right way to do it. Thank you in advance.
for t in threadList:
t.start()
t.join()
The t.join() call waits for the thread t to finish. That means when you start the first thread, you wait for it to finish before starting the second, but the first thread is just going to keep printing Hi forever. It'll never finish.
Don't join, or don't start joining until all threads have started.
Related
Here is my code below , I put string in queue , and hope dowork2 to do something work , and return char in shared_queue
but I always get nothing at while not shared_queue.empty()
please give me some point , thanks.
import time
import multiprocessing as mp
class Test(mp.Process):
def __init__(self, **kwargs):
mp.Process.__init__(self)
self.daemon = False
print('dosomething')
def run(self):
manager = mp.Manager()
queue = manager.Queue()
shared_queue = manager.Queue()
# shared_list = manager.list()
pool = mp.Pool()
results = []
results.append(pool.apply_async(self.dowork2,(queue,shared_queue)))
while True:
time.sleep(0.2)
t =time.time()
queue.put('abc')
queue.put('def')
l = ''
while not shared_queue.empty():
l = l + shared_queue.get()
print(l)
print( '%.4f' %(time.time()-t))
pool.close()
pool.join()
def dowork2(queue,shared_queue):
while True:
path = queue.get()
shared_queue.put(path[-1:])
if __name__ == '__main__':
t = Test()
t.start()
# t.join()
# t.run()
I managed to get it work by moving your dowork2 outside the class. If you declare dowork2 as a function before Test class and call it as
results.append(pool.apply_async(dowork2, (queue, shared_queue)))
it works as expected. I am not 100% sure but it probably goes wrong because your Test class is already subclassing Process. Now when your pool creates a subprocess and initialises the same class in the subprocess, something gets overridden somewhere.
Overall I wonder if Pool is really what you want to use here. Your worker seems to be in an infinite loop indicating you do not expect a return value from the worker, only the result in the return queue. If this is the case, you can remove Pool.
I also managed to get it work keeping your worker function within the class when I scrapped the Pool and replaced with another subprocess:
foo = mp.Process(group=None, target=self.dowork2, args=(queue, shared_queue))
foo.start()
# results.append(pool.apply_async(Test.dowork2, (queue, shared_queue)))
while True:
....
(you need to add self to your worker, though, or declare it as a static method:)
def dowork2(self, queue, shared_queue):
import threading, time
class A():
def __init__(self,a ):
self.a = a
def run(self):
print(self.a)
if __name__=='__main__':
index = [0]
thread_list = []
for i in range(10):
index[0] = i
thread_list.append(threading.Thread(target=A(index).run))
for thread in thread_list:
thread.start()
time.sleep(0.5)
for thread in thread_list:
thread.join()
This piece of code doesn't show the correct result with [0] - [9] sequence but all results are [9]. However, if change it to thread_list.append(threading.Thread(target=A(tuple(index)).run))
or start directly rather than start it in another loop, or put index definition inside the loop the result is OK.
Here's another two correct version:
import threading, time
class A():
def __init__(self,a ):
self.a = a
def run(self):
print(self.a)
if __name__=='__main__':
index = [0]
thread_list = []
for i in range(10):
index[0] = i
thread = threading.Thread(target=A(index).run)
thread_list.append(thread_list)
thread.start()
time.sleep(0.5)
for thread in thread_list:
thread.join()
import threading, time
class A():
def __init__(self,a ):
self.a = a
def run(self):
print(self.a)
if __name__=='__main__':
thread_list = []
for i in range(10):
index = [0]
index[0] = i
thread_list.append(threading.Thread(target=A(index).run))
for thread in thread_list:
thread.start()
time.sleep(0.5)
for thread in thread_list:
thread.join()
Somebody could explain the mechanism behind how Python initialize a Thread object and call start() method. Why is's always the last one when a mutable variable pass into function?
The reason your first snippet of code does what it does is because you're passing a mutable object (the list named index) when creating each of the class A instances, so when they print its value they all display whatever is currently in it—which will be the last thing was that was assigned to index[0] just before the Thread instance itself is created in the following line.
So a workaround would be to avoid passing the function a mutable object. The change indicated below is a trivial way of doing that in this case:
class A():
def __init__(self, a):
self.a = a
def run(self):
print(self.a)
if __name__=='__main__':
index = [0]
thread_list = []
for i in range(10):
index[0] = i
# thread_list.append(threading.Thread(target=A(index).run))
thread_list.append(threading.Thread(target=A(index[0]).run()))
for thread in thread_list:
thread.start()
time.sleep(0.5)
for thread in thread_list:
thread.join()
Could someone please shed some light on why this threaded code to call a classes' method never completes?
from Queue import Queue
from threading import Thread
class SimpleThing(object):
def __init__(self, name):
self.name = name
def print_name(self):
print self.name
class ThingSpawner(object):
def __init__(self, name_list):
self.things = [SimpleThing(name) for name in name_list]
self.thread_queue = Queue()
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
self.thread_queue.put(t)
self.thread_queue.join()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
The code will clearly run the print_name method, but does not join() and exit.
Also, what is the neatest way to modify this code so that the join() completes? The motivation is to use an existing python control class for a bit of hardware, and allows you to call a (very slow) method of the control class in parallel. Thanks!
When you are doing
self.thread_queue.put(t)
You are putting some threads into the Queue, obviously. However, i'm not really sure why. You never use that queue again for anything, and it's completely unnecessary. To make matters worse, you then call
self.thread_queue.join()
Which basically waits forever for the queue to empty, which never happens, because you never empty it or do anything with it.
If I copy paste all your code, but without any Queue at all, everything is fine...
from threading import Thread
class SimpleThing(object):
def __init__(self, name):
self.name = name
def print_name(self):
print self.name
class ThingSpawner(object):
def __init__(self, name_list):
self.things = [SimpleThing(name) for name in name_list]
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
However that's not what you want! Because your threads are daemons they will exit when the main program exits, even if they are not done yet (if I add some delay like sleep(1) before printing the name for example). You should call join() on the threads, not the queue, if you want to wait for them to finish. So we'll return the threads first:
def run(self):
all_threads = []
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
all_threads.append(t)
return all_threads
And when we run we'll do this:
threads = sp.run()
for t in threads:
t.join()
Thanks Ofer for the clear answer, which I've just accepted -- I am indeed not using the queue properly! Having reacquainted myself with queues now you've pointed out my error, for prosperity, here's an alternative approach using a queue:
from Queue import Queue
from threading import Thread
class SimpleThing(object):
def __init__(self, name, q):
self.name = name
def print_name(self, q):
print self.name
q.get()
q.task_done()
class ThingSpawner(object):
def __init__(self, name_list):
self.thread_queue = Queue()
self.things = [SimpleThing(name, self.thread_queue) for name in name_list]
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name, args=(self.thread_queue,))
t.daemon = True
t.start()
self.thread_queue.put(t)
self.thread_queue.join()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
This tutorial on threading and queues was useful once I understood my mistake.
I have a little doubt if one could solve my issue, and create successful communication between threads.
First example and this is how it should be working, but does not work well:
import Queue,threading,time
class th(threading.Thread):
def __init__(self,q):
threading.Thread.__init__(self)
self.q = q
self.t = time
def run(self):
for i in range(5):
self.q.put(i)
self.t.sleep(0.5) # <----------
self.q.put('end')
class main(object):
def __init__(self):
self.q = Queue.Queue()
self.thread = th(self.q)
self.thread.setDaemon(True)
self.thread.run()
self.call()
def call(self):
while True:
recv = self.q.get();
if recv == 'end':
break
else:
print recv
if __name__ == '__main__':
root = main()
root.call()
In this example, all printed at the same time:
0,1,2,3,4
Second example:
import Queue,threading,time
class th(threading.Thread):
def __init__(self,q):
threading.Thread.__init__(self);
self.q = q
self.t = time
def run(self):
for i in range(5):
self.q.put(i) # <------ no sleep()
self.q.put('end')
class main(object):
def __init__(self):
self.q = Queue.Queue()
self.thread = th(self.q)
self.thread.setDaemon(True)
self.thread.run()
self.call()
def call(self):
while True:
recv = self.q.get()
if recv == 'end':
break
else:
print recv
if __name__ == '__main__':
root = main()
root.call()
the code is printed as it has to
0,
1
2
3
4
one to one
is there any way that the sleep function in the same way?
You don't want to call the run method on a thread directly. Call start instead, which will kick off the child thread, which will in turn run the run method.
Your current code is essentially single threaded, since the run call does the work of the child thread in the parent instead. The child thread is never actually started! (You're also calling your main.call method twice, which I'd expect to block or raise an exception, but that's a separate issue.)
sorry, it was something very simple, really simple, just had to replace
self.thread.run()
by
self.threat.start()
I have a simple example script constructed that defines three separate processes using multiprocessing in python. My objective is to have one parent thread that spawns two smaller threads that will collect and process data.
Currently, my implementation looks like this:
from Queue import Queue,Empty
from multiprocessing import Process
import time
import hashlib
class FillQueue(Process):
def __init__(self,q):
Process.__init__(self)
self.q = q
def run(self):
i = 0
while i is not 5:
print 'putting'
self.q.put('foo')
i+=1
self.q.put('|STOP|')
class ConsumeQueue(Process):
def __init__(self,q):
Process.__init__(self)
self.q = q
def run(self):
print 'Consume'
while True:
try:
value = self.q.get(False)
print value
if value == '|STOP|':
print 'done'
break;
except Empty:
print 'Nothing to process atm'
class Ripper(Process):
q = Queue()
def __init__(self):
self.fq = FillQueue(self.q)
self.cq = ConsumeQueue(self.q)
self.fq.daemon = True
self.cq.daemon = True
def run(self):
try:
self.fq.start()
self.cq.start()
except KeyboardInterrupt:
print 'exit'
if __name__ == '__main__':
r = Ripper()
r.start()
As it runs presently, the output from the script on CLI looks like this:
putting
putting
putting
putting
putting
Consume
foo
foo
foo
foo
foo
|STOP|
done
Obviously, the way I am starting my two threads is blocking, since the consumer doesn't even begin to process the items in the queue until the filler finishes adding items.
How should I rewrite this to make both threads begin immediately and not block, so the consumer will simply pass to the Empty except block while there is no work to process, but will exit completely when it receives the stop message?
EDIT: typo, had the start and run methods mixed up
You seem to be starting multiple processes using multiprocessing.Process.
However, you are using Queue.Queue which is only threadsafe, and not designed to be used by multiple processes.
shevek's answer is valid as well, but as a start, you should replace Queue.Queue with multiprocessing.Queue.
try this:
from Queue import Empty
from multiprocessing import Process, Queue
import time
import hashlib
class FillQueue(object):
def __init__(self, q):
self.q = q
def run(self):
i = 0
while i < 5:
print 'putting'
self.q.put('foo %d' % i )
i+=1
time.sleep(.5)
self.q.put('|STOP|')
class ConsumeQueue(object):
def __init__(self, q):
self.q = q
def run(self):
while True:
try:
value = self.q.get(False)
print value
if value == '|STOP|':
print 'done'
break;
except Empty:
print 'Nothing to process atm'
time.sleep(.2)
if __name__ == '__main__':
q = Queue()
f = FillQueue(q)
c = ConsumeQueue(q)
p1 = Process(target=f.run)
p1.start()
p2 = Process(target=c.run)
p2.start()
p1.join()
p2.join()
I think your program works fine. The CPU processes only one thing at a time, for a short time. However, the time required to put all your stuff in the queue is very short. So there is no reason that the filler cannot do this in one time slice.
If you add some delays in the filler, I think you should see that it actually works as you expect.