What is the correct way to send a disconnect signal to a thread containing a SingleServerIRCBot?
I am instantiating bots that connect to twitch with
import threading
import irc.bot
class MyBot(irc.bot.SingleServerIRCBot):
...
bot = MyBot(...)
threads = []
t = threading.Thread(target=bot.start()
threads.append(t)
t.start()
When the stream no longer exists, no matter what I've tried, I haven't been able to get the thread to successfully end. How should I go about sending a signal to the thread that tells it to exit the channel kill the bot and then itself?
The code for the .start method can be found here https://github.com/jaraco/irc/blob/master/irc/bot.py#L331
My first thought is to override that method with a while loop that has an exit condition. I haven't had any luck with that so far though.
Furthermore, there is a .die method here https://github.com/jaraco/irc/blob/master/irc/bot.py#L269 but how can I call that method when the thread is executing an infinite loop?
Trying to kill the threads directly ends up with them persisting, and eventually throwing errors about the total number of threads that my process is running.
Edit for the bounty: I would also accept an answer that describes a better way to handle multiple IRC bots at once.
I don't think you could (or should) kill a thread directly, but you could stop the task running on that thread. Then the thread would be inactive and you could remove it from the threads list, if you like. I'm not familiar with SingleServerIRCBot, but I'll use the class below as an example.
import time
class MyTask:
def __init__(self):
self._active = True
def start(self):
while self._active:
print('running')
time.sleep(1)
def die(self):
self._active = False
In Python3, threads have a _target attribute, from which we can access the target function/method. We could use this attribute to access the target's object and call the die method (eg: thread._target.__self__.die()). However I think it would be best to subclass Thread and store the the target object in a variable, as _target is a private attribute, and also for compatibility reasons.
import threading
class MyThread(threading.Thread):
def __init__(self, target, args=()):
super(MyThread, self).__init__()
self.target = target
self.args = args
def run(self):
self.target.start(*self.args)
def stop_task(self):
self.target.die()
Using this class we would pass a MyTask object as a target, and the start method would be called from MyThread.run. Now we can use MyThread.stop_task to stop the task running on this thread.
o = MyTask()
t = MyThread(target=o)
t.start()
t.stop_task()
time.sleep(1.1)
print(t.is_alive())
Note that I'm waiting 1.1 sec to test if the thread is alive. That's because the target (MyTask.start) will take up to one second to stop. This method doesn't kill the thread, but calls MyTask.die and waits for the task to finish. If you want to end the task immediately (and loose any resources used by the task) you could use a Process and end it with .terminate. You should also choose multiprocessing over multithreading if your task is performing more CPU operations than IO operations, because processes are not limited by the GIL.
Afrer stydying the source code, I noticed that .die() calls sys.exit, so we can't use it to terminate the task because it would stop the program. It seems the reason for this is that .start() calls the parent object's .start(), which then calls the .process_forever() method of a Reactor object. This method starts running Reactor.process_once() in an infinite loop with no break condition.
A possible solution is to subclass SingleServerIRCBot and use a boolean variabe to break the loop. This class should override .start() and .die(), in order to stop the bot running on a thread. The .die() method would set the flag to false, and .start() would call Reactor.process_once() in a loop.
import irc.bot
class MyBot(irc.bot.SingleServerIRCBot):
def __init__(self, channel, nickname, server, port=6667):
super(MyBot, self).__init__([(server, port)], nickname, nickname)
self.channel = channel
self._active = True
def start(self):
self._connect()
while self._active:
self.reactor.process_once(timeout=0.2)
def die(self, msg="Bye, cruel world!"):
self.connection.disconnect(msg)
self._active = False
Now we can stop the bot either by calling .stop_task() on the thread running the bot, or by calling the .die() method of the bot directly.
host, port = 'irc.freenode.net', 6667
nick = 'My-Bot'
channel = '#python'
bot = MyBot(channel, nick, host, port)
t = MyThread(bot)
t.start()
t.stop_task()
#bot.die()
Related
I have a Python asyncio script that needs to run a long running task in a thread. During the operation of the thread, it needs to make network connections to another server. Is there any problem calling network/socket write functions in a thread as opposed to doing it in the main thread?
I know that in the Tiwsted library for example, one must always do network operations in the main thread. Are there any such limitations in asyncio? And if so, how does one get around this problem.
Here's my sample code:
import asyncio
import threading
#
# global servers dict keeps track of connected instances of each protocol
#
servers={}
class SomeOtherServer(asyncio.Protocol):
def __init__(self):
self.transport = None
def connection_made(self,transport):
self.transport=transport
servers["SomeOtherServer"] = self
def connection_lost(self):
self.transport=None
class MyServer(asyncio.Protocol):
def __init__(self):
self.transport = None
def connection_made(self,transport);
self.transport=transport
servers["MyServer"] = self
def connection_lost(self):
self.transport=None
def long_running_task(self,data):
# some long running operations here, then write data to other server
# other_server is also an instance of some sort of asyncio.Protocol
# is it ok to call this like this, even though this method is running in a thread?
other_server = servers["SomeOtherServer"]
other_server.transport.write(data)
def data_received(self,data):
task_thread = threading.Thread(target=self.long_running_task,args=[data])
task_thread.start()
async def main():
global loop
loop = asyncio.get_running_loop()
other_server_obj = await loop.create_server(lambda: SomeOtherServer(),"localhost",9001)
my_server_obj = await loop.create_server(lambda: MyServer(),"localhost",9002)
async with other_server_obj, my_server_obj:
while True:
await asyncio.sleep(3600)
asyncio.run(main())
Note that data_received will set up and call long_running_task in a thread, and long running_task makes a network connection to another server, and does so in the task thread, not the main thread. Is this ok or is there some other way this must be done?
When using a class Pup for creating stoppable threads that are meant to be running in the background until .stop() is called:
What happens when pup.join() is not called after pup.stop()? Will the following result in a leak:
pup = Pup()
pup.start()
time.sleep(5)
pup.stop()
pup2 = Pup()
pup2.start()
time.sleep(5)
pup2.stop()
pup3 = Pup()
pup3.start()
time.sleep(5)
pup3.stop()
Must pup be a daemonized thread since we are running it in the background?
Main code below is borrowed from this SO answer
import time
import threading
class StoppableThread(threading.Thread):
"""Thread class with a stop() method. The thread itself has to check
regularly for the stopped() condition."""
def __init__(self, *args, **kwargs):
super(StoppableThread, self).__init__(*args, **kwargs)
self._stopper = threading.Event()
def stop(self):
self._stopper.set()
def stopped(self):
return self._stopper.isSet()
class Pup(StoppableThread):
def __init__(self, i, *args, **kwargs):
super(Pup, self).__init__(*args, **kwargs)
self.i = i
def run(self):
while True:
if self.stopped():
return
print("Hello, world!", i)
time.sleep(1)
for i in range(100):
pup = Pup(i)
pup.start()
time.sleep(5)
pup.stop()
The StoppableThread should be joined.
Because it is just a thin wrapper about threading.Thread giving you a possibility of setting and checking flag stopper.
In this case, there must be a code that regularly checks this flag. The amount of delay between checks depends on the user of the class.
And given the fact that it is assumed that the thread should be correctly stopped, you must use join. Because if you make the thread as daemon and try to stop it before the application finishing:
Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
A leak is possible only if your code is responsible for checking the stopper flag and subsequent exiting from the thread does not work correctly. Otherwise, there is no leaks, because the app, even join is not called, will wait for the completion of all non-daemon threads. But using join will give more control over the program flow.
Taking all of the above into consideration, i think that making StoppableThread as daemon is bad idea.
I have a producer thread that produces data from a serial connection and puts them into multiple queues that will be used by different consumer threads. However, I'd like to be able to add in additional queues (additional consumers) from the main thread after the producer thread has already started running.
I.e. In the code below, how could I add a Queue to listOfQueues from the main thread while this thread is running? Can I add in a method such as addQueue(newQueue) to this class which appends to it listOfQueues? This doesn't seem likely as the thread will be in the run method. Can I create some sort of Event similar to the stop event?
class ProducerThread(threading.Thread):
def __init__(self, listOfQueues):
super(ProducerThread, self).__init__()
self.listOfQueues = listOfQueues
self._stop_event = threading.Event() # Flag to be set when the thread should stop
def run(self):
ser = serial.Serial() # Some serial connection
while(not self.stopped()):
try:
bytestring = ser.readline() # Serial connection or "producer" at some rate
for q in self.listOfQueues:
q.put(bytestring)
except serial.SerialException:
continue
def stop(self):
'''
Call this function to stop the thread. Must also use .join() in the main
thread to fully ensure the thread has completed.
:return:
'''
self._stop_event.set()
def stopped(self):
'''
Call this function to determine if the thread has stopped.
:return: boolean True or False
'''
return self._stop_event.is_set()
Sure, you can simply have an append function that adds to your list. E.g.
def append(self, element):
self.listOfQueues.append(element)
That will work even after your thread's start() method has been called.
Edit: for non thread-safe procedures you can use a lock, e.g.:
def unsafe(self, element):
with self.lock:
# do stuff
You would then also need to add the lock inside your run method, e.g.:
with lock:
for q in self.listOfQueues:
q.put(bytestring)
Any code acquiring a lock will wait for the lock to be released elsewhere.
Consider a hypothetical threaded Python application that runs each thread in an infinite loop:
import signal
import sys
import threading
import time
class CallSomebody (threading.Thread):
def __init__(self, target, *args):
self._target = target
self._args = args
threading.Thread.__init__(self)
def run (self):
self._target(*self._args)
def call (who):
while True:
print "Who you gonna call? %s" % (str(who))
def signal_handler(signal, frame):
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
a=CallSomebody(call, 'Ghostbusters!')
a.daemon=True
b=CallSomebody(call, 'The Exorcist!')
b.daemon=True
a.start()
b.start()
a.join()
b.join()
When running the application, sending SIGINT by pressing CtrlC does not stop the application. I tried removing the daemon statements but that did not help. What fundamental idea am I missing?
Thanks.
When you join a thread the active thread blocks until the joined thread returns. Yours never do. You won't want to join them in this way.
Generally, background threads that are daemon threads and that perform infinite loops should be marked daemon, never joined, and then allowed to expire when your main thread does so. If you happened to be using wx for example, you'd make your call to AppInstance.MainLoop() after starting the daemonic threads, then when your Frame or whatever other top level instances you had were closed, the program execution would be concluded, and the daemons would be addressed appropriately.
I am developing a multi-threaded application in python. I have following scenario.
There are 2-3 producer threads which communicate with DB and get some data in large chunks and fill them up in a queue
There is an intermediate worker which breaks large chunks fetched by producer threads into smaller ones and fill them up in another queue.
There are 5 consumer threads which consume queue created by intermediate worker thread.
objects of data sources are accessed by producer threads through their API. these data sources are completely separate. So these producer understands only presence or absence of data which is supposed to be given out by data source object.
I create threads of these three types and i make main thread wait for completion of these threads by calling join() on them.
Now for such a setup I want a common error handler which senses failure of any thread, any exception and decides what to do. For e.g if I press ctrl+c after I start my application, main thread dies but producer, consumer threads continue to run. I would like that once ctrl+c is pressed entire application should shut down. Similarly if some DB error occurs in data source module, then producer thread should get notified of that.
This is what I have done so far:
I have created a class ThreadManager, it's object is passed to all threads. I have written an error handler method and passed it to sys.excepthook. This handler should catch exceptions, error and then it should call methods of ThreadManager class to control the running threads. Here is snippet:
class Producer(threading.Thread):
....
def produce():
data = dataSource.getData()
class DataSource:
....
def getData():
raise Exception("critical")
def customHandler(exceptionType, value, stackTrace):
print "In custom handler"
sys.excepthook = customHandler
Now when a thread of producer class calls getData() of DataSource class, exception is thrown. But this exception is never caught by my customHandler method.
What am I missing? Also in such scenario what other strategy can I apply? Please help. Thank you for having enough patience to read all this :)
What you need is a decorator. In essence you are modifying your original function and putting in inside a try-except:
def exception_decorator(func):
def _function(*args):
try:
result = func(*args)
except:
print('*** ESC default handler ***')
os._exit(1)
return result
return _function
If your thread function is called myfunc, then you add the following line above your function definition
#exception_decorator
def myfunc():
pass;
Can't you just catch "KeyboardInterrupt" when pressing Ctrl+C and do:
for thread in threading.enumerate():
thread._Thread__stop()
thread._Thread__delete()
while len(threading.enumerate()) > 1:
time.sleep(1)
os._exit(0)
and have a flag in each threaded class which is self.alive
you could theoretically call thread.alive = False and have it stop gracefully?
for thread in threading.enumerate():
thread.alive = False
time.sleep(5) # Grace period
thread._Thread__stop()
thread._Thread__delete()
while len(threading.enumerate()) > 1:
time.sleep(1)
os._exit(0)
example:
import os
from threading import *
from time import sleep
class worker(Thread):
def __init__(self):
self.alive = True
Thread.__init__(self)
self.start()
def run(self):
while self.alive:
sleep(0.1)
runner = worker()
try:
raw_input('Press ctrl+c!')
except:
pass
for thread in enumerate():
thread.alive = False
sleep(1)
try:
thread._Thread__stop()
thread._Thread__delete()
except:
pass
# There will always be 1 thread alive and that's the __main__ thread.
while len(enumerate()) > 1:
sleep(1)
os._exit(0)
Try going about it by changing the internal system exception handler?
import sys
origExcepthook = sys.excepthook
def uberexcept(exctype, value, traceback):
if exctype == KeyboardInterrupt:
print "Gracefully shutting down all the threads"
# enumerate() thingie here.
else:
origExcepthook(exctype, value, traceback)
sys.excepthook = uberexcept