in my project i have a class of threading.Thread like this:
class MakeHtml(threading.Thread):
def __init__(self, *rstext):
self.outhtml = [x for x in rstext]
self.retval = ''
threading.Thread.__init__(self)
def run(self):
...do something
in another file i call, every 10 seconds MakeHtml class
t = MakeHtml(mrr1, mrr2, mrr3, mrr4)
for create a thread but in this way i see that the thread is the same every time.
I need a new thread every time i call the MakeHtml Threading class, how can i do this?
Thanks in advance
MakeHtml extends Thread, but if you have only 1 instance of MakeHtml, you will have only one thread
For instance if you want 2 different thread you will have to do
t = MakeHtml(mrr1, mrr2, mrr3, mrr4) # one thread
t1 = MakeHtml(mrr1, mrr2, mrr3, mrr4) # another one
You can use :
import threading
def afunction(mm):
# do job
pass
threads = []
for mm in [mmr1, mmr2, mmr3n mmr4]:
t = threading.Thread(target=afunction, args=[mm,])
threads.append(t)
t.start()
[t.join() for t in threads]
Related
Here is my code below , I put string in queue , and hope dowork2 to do something work , and return char in shared_queue
but I always get nothing at while not shared_queue.empty()
please give me some point , thanks.
import time
import multiprocessing as mp
class Test(mp.Process):
def __init__(self, **kwargs):
mp.Process.__init__(self)
self.daemon = False
print('dosomething')
def run(self):
manager = mp.Manager()
queue = manager.Queue()
shared_queue = manager.Queue()
# shared_list = manager.list()
pool = mp.Pool()
results = []
results.append(pool.apply_async(self.dowork2,(queue,shared_queue)))
while True:
time.sleep(0.2)
t =time.time()
queue.put('abc')
queue.put('def')
l = ''
while not shared_queue.empty():
l = l + shared_queue.get()
print(l)
print( '%.4f' %(time.time()-t))
pool.close()
pool.join()
def dowork2(queue,shared_queue):
while True:
path = queue.get()
shared_queue.put(path[-1:])
if __name__ == '__main__':
t = Test()
t.start()
# t.join()
# t.run()
I managed to get it work by moving your dowork2 outside the class. If you declare dowork2 as a function before Test class and call it as
results.append(pool.apply_async(dowork2, (queue, shared_queue)))
it works as expected. I am not 100% sure but it probably goes wrong because your Test class is already subclassing Process. Now when your pool creates a subprocess and initialises the same class in the subprocess, something gets overridden somewhere.
Overall I wonder if Pool is really what you want to use here. Your worker seems to be in an infinite loop indicating you do not expect a return value from the worker, only the result in the return queue. If this is the case, you can remove Pool.
I also managed to get it work keeping your worker function within the class when I scrapped the Pool and replaced with another subprocess:
foo = mp.Process(group=None, target=self.dowork2, args=(queue, shared_queue))
foo.start()
# results.append(pool.apply_async(Test.dowork2, (queue, shared_queue)))
while True:
....
(you need to add self to your worker, though, or declare it as a static method:)
def dowork2(self, queue, shared_queue):
Could someone please shed some light on why this threaded code to call a classes' method never completes?
from Queue import Queue
from threading import Thread
class SimpleThing(object):
def __init__(self, name):
self.name = name
def print_name(self):
print self.name
class ThingSpawner(object):
def __init__(self, name_list):
self.things = [SimpleThing(name) for name in name_list]
self.thread_queue = Queue()
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
self.thread_queue.put(t)
self.thread_queue.join()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
The code will clearly run the print_name method, but does not join() and exit.
Also, what is the neatest way to modify this code so that the join() completes? The motivation is to use an existing python control class for a bit of hardware, and allows you to call a (very slow) method of the control class in parallel. Thanks!
When you are doing
self.thread_queue.put(t)
You are putting some threads into the Queue, obviously. However, i'm not really sure why. You never use that queue again for anything, and it's completely unnecessary. To make matters worse, you then call
self.thread_queue.join()
Which basically waits forever for the queue to empty, which never happens, because you never empty it or do anything with it.
If I copy paste all your code, but without any Queue at all, everything is fine...
from threading import Thread
class SimpleThing(object):
def __init__(self, name):
self.name = name
def print_name(self):
print self.name
class ThingSpawner(object):
def __init__(self, name_list):
self.things = [SimpleThing(name) for name in name_list]
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
However that's not what you want! Because your threads are daemons they will exit when the main program exits, even if they are not done yet (if I add some delay like sleep(1) before printing the name for example). You should call join() on the threads, not the queue, if you want to wait for them to finish. So we'll return the threads first:
def run(self):
all_threads = []
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name)
t.daemon = True
t.start()
all_threads.append(t)
return all_threads
And when we run we'll do this:
threads = sp.run()
for t in threads:
t.join()
Thanks Ofer for the clear answer, which I've just accepted -- I am indeed not using the queue properly! Having reacquainted myself with queues now you've pointed out my error, for prosperity, here's an alternative approach using a queue:
from Queue import Queue
from threading import Thread
class SimpleThing(object):
def __init__(self, name, q):
self.name = name
def print_name(self, q):
print self.name
q.get()
q.task_done()
class ThingSpawner(object):
def __init__(self, name_list):
self.thread_queue = Queue()
self.things = [SimpleThing(name, self.thread_queue) for name in name_list]
def run(self):
for thing in self.things:
t = Thread(target=thing.print_name, name=thing.name, args=(self.thread_queue,))
t.daemon = True
t.start()
self.thread_queue.put(t)
self.thread_queue.join()
thing_list = ['cat', 'dog', 'llama', 'bat']
sp = ThingSpawner(thing_list)
sp.run()
This tutorial on threading and queues was useful once I understood my mistake.
I am trying to find a way to compare between different objects (inherited from Thread class) in a way that keep parallilsm (real-time processing).
Every worker has three fields (message, count, n ). I am updating Count everytime. Let's say that I have three threads workers. I need to compare in my server based on the field count, how can I do access and compare between Worker.count of every worker, in a way that I keep parallelism
from Queue import Queue
from threading import Thread
import time
class Worker(Thread):
def __init__(self, message, n):
Thread.__init__(self)
self.message = message
self.count= 0
self.n = n
def run(self):
while True:
print(self.message)
self.count+=1
time.sleep(self.n)
class Comparator(Thread):
def __init__(self, message, n):
Thread.__init__(self)
self.message = message
self.n = n
def run(self):
while True:
max= max([x.count for x in threads]) # how can I access to other threads
print "max", max
time.sleep(self.n)
thread1 = Worker("Test-1", 1)
thread2 = Worker("Test-2", 3)
s = Comparator("Test-3", 2)
s.start()
s.join()
threads = [thread1, thread2]
for g in threads:
g.start()
for worker in threads:
# wait for workers
worker.join()
NOTE Using shared object here is not a good solution for me, using Queue() for example is not what I want, I need to do comparision based on updated field in the object that I update on the go (for simplicity, I use max() ).
you can pass the threads list to Comparator __init__() method :
[...]
class Comparator(Thread):
def __init__(self, message, n, threads):
Thread.__init__(self)
self.threads = threads
def run(self):
while True:
max= max([x.count for x in self.threads])
print("max", max)
time.sleep(self.n)
[...]
threads = [thread1, thread2]
s = Comparator("Test-3", 2, threads)
I want to do a infinite loop function.
Here is my code
def do_request():
# my code here
print(result)
while True:
do_request()
When use while True to do this, it's a little slow, so I want to use a thread pool to concurrently execute the function do_request(). How to do this ?
Just like use ab (Apache Bench) to test HTTP server.
Finally, I've solved this problem. I use a variable to limit the thread number.
Here is my final code, solved my problem.
import threading
import time
thread_num = 0
lock = threading.Lock()
def do_request():
global thread_num
# -------------
# my code here
# -------------
with lock:
thread_num -= 1
while True:
if thread_num <= 50:
with lock:
thread_num += 1
t = threading.Thread(target=do_request)
t.start()
else:
time.sleep(0.01)
Thanks for all replies.
You can use threading in Python to implement this.
Can be something similar to this (when using two extra threads only):
import threading
# define threads
task1 = threading.Thread(target = do_request)
task2 = threading.Thread(target = do_request)
# start both threads
task1.start()
task2.start()
# wait for threads to complete
task1.join()
task2.join()
Basically, you start as many threads as you need (make sure you don't get too many, so your system can handle it), then you .join() them to wait for tasks to complete.
Or you can get fancier with multiprocessing Python module.
Try the following code:
import multiprocessing as mp
import time
def do_request():
while(True):
print('I\'m making requests')
time.sleep(0.5)
p = mp.Process(target=do_request)
p.start()
for ii in range(10):
print 'I\'m also doing other things though'
time.sleep(0.7)
print 'Now it is time to kill the service thread'
p.terminate()
The main thread stars a service thread that does the request and goes on until it has to, and then it finishes up the service thread.
Maybe you can use the concurrent.futures.ThreadPoolExecutor
from concurrent.futures import ThreadPoolExecutor
import time
def wait_on_b(hello):
time.sleep(1)
print(hello) # b will never complete because it is waiting on a.
return 5
def wait_on_a():
time.sleep(1)
print(a.result()) # a will never complete because it is waiting on b.
return 6
executor = ThreadPoolExecutor(max_workers=2)
a = executor.submit(wait_on_b, 3)
b = executor.submit(wait_on_a)
How about this?
from threading import Thread, Event
class WorkerThread(Thread):
def __init__(self, logger, func):
Thread.__init__(self)
self.stop_event = Event()
self.logger = logger
self.func = func
def run(self):
self.logger("Going to start the infinite loop...")
#Your code
self.func()
concur_task = WorkerThread(logger, func = do_request)
concur_task.start()
To end this thread...
concur_task.stop_event.set()
concur_task.join(10) #or any value you like
I want two objects of the same class to operate concurrently. The class "MyClass" has a function that connects an instance to another instance of the class. I also need to keep track of the objects that have been created (oList). what I am trying is:
main.py:
from MyClass import MyClass
import time
oList = []
class oCreator1(Thread):
def __init__(self):
Thread.__init__(self)
self.o1 = MyClass()
def run(self):
while 1:
time.sleep(1)
print "Hi"
def getO1(self):
return self.o1
class oCreator2(Thread):
def __init__(self):
Thread.__init__(self)
self.o2 = MyClass()
def run(self):
while 1:
time.sleep(1)
print "Bye!"
def getO2(self):
return self.o2
main():
threadList = []
global oList
oc1 = oCreator1()
threadList.append(oc1)
o1 = oc1.getO1()
oList.append(o1)
oc2 = oCreator2()
threadList.append(oc2)
o2 = oc2.getO2()
oList.append(o2)
o1.connToAnotherO(o2)
print oList
for t in threadList:
t.start()
t.join()
if __name__ == '__main__':
main()
But the only thing that is printed is "Hi". I really want to know the things I'm doing wrong and the right way to do it. Thank you in advance.
for t in threadList:
t.start()
t.join()
The t.join() call waits for the thread t to finish. That means when you start the first thread, you wait for it to finish before starting the second, but the first thread is just going to keep printing Hi forever. It'll never finish.
Don't join, or don't start joining until all threads have started.