How can I create a class, that starts a blocking process in the background without needing to specifically design consumers/peers of that class around asyncio/threading? For example, how can we start a websocket connection to run alongside an event-loop, without specific support for threading/asyncio in the event-loop?
I have solved the conflict by automatically starting a threading.Thread on object creation. The thread will run blocking-processes without interruption, and can reference the self of its parent. This allows me to wrap any blocking process into a supporting object, avoiding the need to specifically design around the implementation of said process.
An example is provided below. On creation, the blocking process self.process will start on a separate thread, allowing the main thread to continue operation.
class SmartThread(threading.Thread):
def __init__(self):
super().__init__(target=self.process)
self.start()
def process(self):
while True:
print(self.clock)
time.sleep(1)
#property
def clock(self):
return time.time()
Related
The following code is the init function from a Kivy app I'm coding. The app utilises Kivy's built-in Clock method to call an update function every 10 seconds. The update function runs cpu intensive code so I use a function within the init function to run the update function in it's own thread. This code does what I want it to do but it occurred to me that each time the update function gets called, a new unique thread is created.
My Questions:
Are there any problems or issues associated with a potentially infinite number of threads being created?
Is there a method that stops or destroys a thread before the new one is created? If so is that approach advisable or does it matter if infinite threads get created?
Is there a better way to code this?
def __init__(self, **kwargs):
super().__init__(**kwargs)
def start_thread(dt):
t = threading.Thread(target=self.update)
t.start()
Clock.schedule_interval(start_thread, 10)
def update(self):
"Does some stuff in new thread every time it's called"
Is there a better way to code this?
Use a thread pool. Pooling is when we re-use objects instead of continually creating and destroying new ones. A thread pool uses a small collection of "worker" threads to perform tasks (callable objects) that your program submits to it.
https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
I'm replacing part of an existing program. That original program uses threads. There's this particular class which inherits from threading.Thread which functionality I need to replace but I need to keep the interface the same.
The functionality I'm integrating is packaged in a library which uses asyncio a lot.
The original calls to the class I'm replacing go something like this:
network = Network()
network.start()
network.fetch_something() # crashes!
network.stop()
I've gotten to a point where my replacing class inherits from threading.Thread too and I can connect, from within the run method to my backends via the client library:
class Network(threading.Thread):
def __init__(self):
self._loop = asyncio.new_event_loop()
self._client = Client() # this is the library
def run(self):
self._loop.run_until_complete(self.__connect()) # works dandy, implementation not shown
self._loop.run_forever()
def fetch_something(self):
return self._loop.run_until_complete(self._client.fetch_something())
Running this code throws an exception:
RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one
I sort of get what's going on here. In the run method things worked out because the same thread running the event loop was the caller. In the other case an other thread was the caller hence the problem.
As you might have noticed I was hoping the problem would have been solved by using the same event loop. Alas, that didn't work out.
I really want to keep the interface exactly as it is otherwise I'm refactoring for the remainder of the year. I could relatively easily pass arguments to the constructor of the Network class. I've tried passing in an event loop created on the main thread but the result was the same.
(Note that this is the opposite problem this author has: Call coroutine within Thread)
When scheduling a coroutine from a different thread, you must use asyncio.run_coroutine_threadsafe. For example:
def fetch_something(self):
future = asyncio.run_coroutine_threadsafe(
self._client.fetch_something(), loop)
return future.result()
run_coroutine_threadsafe schedules the coroutine with the event loop in a thread-safe way and returns a concurrent.futures.Future. You can use the returned future to simply wait for the result as shown above, but you can also pass it to other functions, poll whether the result has arrived, or implement timeouts.
When combining threads and asyncio, remember to make sure that all interfacing with the event loop from other threads (even to call something as simple as loop.stop to implement Network.stop) is done using loop.call_soon_threadsafe and asyncio.run_coroutine_threadsafe.
So through a lot of help in my previous questions
(Interrupting QThread sleep
and PySide passing signals from QThread to a slot in another QThread) I decided to attempt to change from the inherited QThread model to the Worker model. I am thinking I should stay with the QThread model as I had that working, and the other model is not. However I am not sure why the Worker model isn't working for me.
I am attempting to do this please let me know if there is something inherently wrong in my methodology?
I have a QtGui.QWidget that is my main GUI. I am using a QPushButton to signal
I have attempted to reduce the code to the basics of where I believe the issue is. I have verified that datagramHandled Signal gets emitted but the packet_handled Slot doesn't seem to get called.
class myObject(QtCore.QObject):
def __init__(self):
super(myObject, self).__init__()
self.ready=False
#QtCore.Slot()
def do_work(self):
#send a packet
self.ready=False
while not self.ready:
time.sleep(0.01)
#QtCore.Slot(int)
def packet_handled(self, errorCode):
print "Packet received."
self.ready = True
class myWidget(QtGui.QWidget):
datagramHandled = QtCore.Signal(int)
startRunThread = QtCore.Signal()
def __init__(self,parent=None, **kwargs):
super(myWidget, self).__init__(parent=parent)
# Bunch of GUI setup stuff (working)
self.myRunThread = QtCore.QThread()
#QtCore.Slot()
def run_command(self):
self.myRunObj = myObject()
self.myRunObj.moveToThread(self.myRunThread)
self.datagramHandled.connect(self.myRunObj.packet_handled)
self.startRunThread.connect(self.myRunObj.do_work)
self.myRunThread.start()
self.startRunThread.emit()
#QtCore.Slot()
def handle_datagram(self):
#handle the incoming datagram
errorCode = 0
self.datagramHandled.emit(errorCode)
The first issue is that you need to connect your myObject.do_work method to QThread.started:
self.myRunThread.started.connect(self.myRunObj.do_work)
Secondly, your do_work method should include something along these lines to enable event processing (please forgive my rusty PyQt and pseudocode):
def do_work(self):
while someCondition:
#The next two lines are critical for events and queued signals
if self.thread().eventDispatcher().hasPendingEvents():
self.thread().eventDispatcher().processEvents(QEventLoop.AllEvents)
if not self.meetsSomeConditionToContinueRunning():
break
elif self.hasWorkOfSomeKind():
self.do_something_here()
else:
QThread.yieldCurrentThread()
For more on this, check out the docs for QAbstractEventDispatcher.
The logic here is that when a signal is emitted from one thread (myWidget.datagramHandled), it gets queued in your worker thread's event loop. Calling processEvents processes any pending events (including queued signals, which are really just events), invoking the appropriate slots for any queued signals (myRunObj.packet_handled).
Further reading:
How To Really, Truly Use QThreads; The Full Explanation
Threading Basics
There 3 possible ways of distributing the computation/other load with Qt:
Explicitly putting the load to concrete QThread instance. That is thread-based concurrency.
Implicitly putting the load to pooled QThread instance. That is closer to task-based concurrency yet 'manually' managed with your own logic. QThreadPool class is used for maintaining the pool of threads.
Starting the task in own threading context we never explicitly manage. That is task-based concurrency and QtConcurrent namespace used. My guess is that task-based concurrency and "worker model" is the same thing (observed your changes). Mind that QtConcurrent does offer parallelization for tasks and uses exceptions (which may affect the way you write the code) unlike the rest of Qt.
Given you use PyQt you can also take an advantage of the feature designated for the pattern you want to implement with QtConcurrent for PyQt.
P.S. I see use thread.sleep( interval ) and that is not a good practice and one more indication that the proper technique should be used for implementing 'Worker model'.
An alternative to the solution provided by #JonHarper is to replace your while loop with a QTimer. Because you have an event loop running in your worker process now, it can handle QTimer events correctly (as long as you construct the QTimer in the relevant thread).
This way, control is returned to the event loop periodically so that other slots can be run when required.
I am currently subclassing python's threading.Thread class in order to add additional logging features to it. At the moment I am trying to get it to report both when the thread is started, and when it has finished. Reporting the thread has started is easy enough since I can just extend the start() function. However reporting exit has been more difficult. I tried to extend the _bootstrap and _boothstrap_inner functions to add logging after they were complete, however that seems to have no effect. I can not modify those functions at all.
Does anyone know of a way to add the ability for a thread to report that it has finished?
I usually use the target function argument to the Thread constructor, so I'd do it this way:
class MyThread(Thread):
def __init__(self, target):
Thread.__init__(self, target=self._target, args=(target,))
def _target(self, target):
print "thread starting"
target()
print "thread ended"
Now, it does seem like you're used to using Thread the other way, by overriding its run() method, but maybe this will be of some use anyway.
Seems that your only option is requiring your users to override some other method instead of run. This way you'll have run in your CustomThread that invokes that other method and reports when done.
This has an extra benefit: start function is non-trivial, you'll be able to report successful start at the beginning of run instead of carefully dealing with overriden start.
If you have a lot of asynchronous threads maybe you should consider a message queue for inter communication instead? have the thread post messages to an exchange and then exit. Then let the calling thread decide when to poll for messages. Kinda depends on your workload though.
This has the advantage that you can go multi process rather than multi thread if you want later.
I understand this suggestion may not be what you were wanted.
I am trying to write a unix client program that is listening to a socket, stdin, and reading from file descriptors. I assign each of these tasks to an individual thread and have them successfully communicating with the "main" application using synchronized queues and a semaphore. The problem is that when I want to shutdown these child threads they are all blocking on input. Also, the threads cannot register signal handlers in the threads because in Python only the main thread of execution is allowed to do so.
Any suggestions?
There is no good way to work around this, especially when the thread is blocking.
I had a similar issue ( Python: How to terminate a blocking thread) and the only way I was able to stop my threads was to close the underlying connection. Which resulted in the thread that was blocking to raise and exception and then allowed me to check the stop flag and close.
Example code:
class Example(object):
def __init__(self):
self.stop = threading.Event()
self.connection = Connection()
self.mythread = Thread(target=self.dowork)
self.mythread.start()
def dowork(self):
while(not self.stop.is_set()):
try:
blockingcall()
except CommunicationException:
pass
def terminate():
self.stop.set()
self.connection.close()
self.mythread.join()
Another thing to note is commonly blocking operations generally offer up a timeout. If you have that option I would consider using it. My last comment is that you could always set the thread to deamonic,
From the pydoc :
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property.
Also, the threads cannot register signal handlers
Signals to kill threads is potentially horrible, especially in C, especially if you allocate memory as part of the thread, since it won't be freed when that particular thread dies (as it belongs to the heap of the process). There is no garbage collection in C, so if that pointer goes out of scope, it's gone out of scope, the memory remains allocated. So just be careful with that one - only do it that way in C if you're going to actually kill all the threads and end the process so that the memory is handed back to the OS - adding and removing threads from a threadpool for example will give you a memory leak.
The problem is that when I want to shutdown these child threads they are all blocking on input.
Funnily enough I've been fighting with the same thing recently. The solution is literally don't make blocking calls without a timeout. So, for example, what you want ideally is:
def threadfunc(running):
while running:
blockingcall(timeout=1)
where running is passed from the controlling thread - I've never used threading but I have used multiprocessing and with this you actually need to pass an Event() object and check is_set(). But you asked for design patterns, that's the basic idea.
Then, when you want this thread to end, you run:
running.clear()
mythread.join()
and your main thread should then allow your client thread to handle its last call, and return, and the whole program folds up nicely.
What do you do if you have a blocking call without a timeout? Use the asynchronous option, and sleep (as in call whatever method you have to suspend the thread for a period of time so you're not spinning) if you need to. There's no other way around it.
See these answers:
Python SocketServer
How to exit a multithreaded program?
Basically, don't block on recv() by using select() with a timeout to check for readability of the socket, and poll a quit flag when select() times out.