I have an PySide application. In this application, the main function runs in a thread (AnalysisThread). In this thread, I log some stuff with the python logger. I then add a custom logger which essentially triggers a signal with a string. This signal may be handled in the main thread (the GUI thread), but obviously the slots is never triggered, but I am sure that the signal self.messageWritten.emit function is called (debugger confirms this). What am I doing wrong ?
class LogStream(QtCore.QObject):
messageWritten = QtCore.Signal(str)
signal_test = QtCore.Signal()
def write(self, msg):
if not self.signalsBlocked():
self.messageWritten.emit(msg)
class QtHandler(logging.Handler):
def __init__(self, stream):
logging.Handler.__init__(self)
self.stream = stream
def emit(self, record):
record = self.format(record)
if record:
self.stream.write('%s\n' % record)
class AnalysisThread(QtCore.QThread):
processing_ended = QtCore.Signal()
processing_failed = QtCore.Signal(Exception, list)
def __init__(self, analysis):
super(AnalysisThread, self).__init__()
self.analysis = analysis
def run(self):
try:
process = Process(target=self.analysis.analyze)
process.start()
process.join()
except Exception as err:
exec_info = sys.exc_info()
self.processing_failed.emit(err, exec_info)
finally:
self.processing_ended.emit()
class ProcessView(QtGui.QMainWindow):
def __init__(self):
super(ProcessView, self).__init__()
# Log Stream
self.stream = LogStream()
self.stream.messageWritten.connect(self.on_log_written)
def go(self):
analysis = MyAnalysis()
# Handler
handler = QtHandler(self.stream)
handler.setFormatter(logging.Formatter('(%(levelname)s-%(name)s) %(message)s'))
analysis.log.addHandler(handler)
self.processing = AnalysisThread(analysis)
self.processing.processing_ended.connect(self.on_processing_ended)
self.processing.processing_failed.connect(self.on_processing_failed)
self.processing.start()
def on_log_written(self, msg):
print('Message: {}'.format(msg)) # never called
EDIT
For clarification, it is a multithread application, but also a multiprocess one...
Since you're working in a multithreaded environment, try specifying a Qt.QueuedConnection when you're hooking up your signal and slots.
For example:
self.stream.messageWritten.connect(self.on_log_written, QtCore.Qt.QueuedConnection)
ANSWER
I finally found the solution. My application as noted uses multiprocessing, and the log occurs in the child process so the parent process is never alert of the signal. The solution is to create a 'link' between the child and the parent process with a multiprocessing.Pipe. An implementation could be
class Streamer(QtCore.QThread):
messageWritten = QtCore.Signal(str)
def __init__(self, pipe):
super(Streamer, self).__init__()
self.pipe = pipe
def run(self):
while True:
try:
msg = self.pipe.recv()
except EOFError:
break
else:
self.messageWritten.emit(msg)
class QtHandler(logging.Handler):
def __init__(self, stream):
"""Instantiate handler
:param stream: multiprocessing.Pipe
"""
logging.Handler.__init__(self)
self.stream = stream
def emit(self, record):
record = self.format(record)
if record:
self.stream.send('%s\n' % record)
class ProcessView(QtGui.QMainWindow):
def __init__(self):
super(ProcessView, self).__init__()
# Log Stream
mother_pipe, child_pipe = Pipe()
self.stream = child_pipe
self.streamer = Streamer(mother_pipe)
self.streamer.daemon = True
self.streamer.start()
self.streamer.messageWritten.connect(self.on_log_written)
def on_log_written(self, msg):
self.ui.textBrowser_log.insertPlainText(msg)
txt_max = self.ui.textBrowser_log.verticalScrollBar().maximum()
self.ui.textBrowser_log.verticalScrollBar().setValue(txt_max)
Explanations:
The main thread (GUI) starts. At initialization, it creates a multiprocesing.Pipe to communicate with a child proess, and a daemon that listens to the end of the pipe. So after the child process is launched, it eventually logs something. The QtHandler intercepts the message and sends it through the pipe. The daemon receives the message at the other end of the pipe and forwards it to the GUI thread via the signal messageWritten. Finally, this signal is processed by a slot that writes it in a QTextBrowser which scrollbar is refreshed to
Related
Suppose I have some consumer daemon threads that constantly take objects from a queue whenever the main thread puts them there and performs some long operation (a couple of seconds) with them.
The problem is that whenever the main thread is done, the daemon threads are killed before they finish processing whatever is left in the queue.
I know that one way to solve this could be to wait for the daemon threads to finish processing whatever is left in the queue and then exit, but I am curious if there is any way for the daemon threads to "clean up" after themselves (i.e. finish processing whatever is left in the queue) when the main thread exits, without explicitly having the main thread tell the daemon threads to start cleaning up.
The motivation behind this is that I made a python package that has a logging handler class that puts items into a queue whenever the user tries to log something (e.g. with logging.info("message")), and the handler has a daemon thread that sends the logs over the network. I'd prefer if the daemon thread could clean up by itself, so users of the package wouldn't have to manually make sure to make their main thread wait for the log handler to finish its processing.
Minimal working example
# this code is in my package
class MyHandler(logging.Handler):
def __init__(self, level):
super().__init__(level=level)
self.queue = Queue()
self.thread = Thread(target=self.consume, daemon=True)
self.thread.start()
def emit(self, record):
# This gets called whenever the user does logging.info, or similar
self.queue.put(record)
def consume(self):
while True:
record = self.queue.get()
send(record) # send record over network, can take a few seconds (assume it never raises)
self.queue.task_done()
# This is user's main code
# user will have to keep a reference to the handler for later. I want to avoid this.
my_handler = MyHandler()
# set up logging
logging.basicConfig(..., handlers=[..., my_handler])
# do some stuff...
logging.info("this will be sent over network")
# some more stuff...
logging.error("also sent over network")
# even more stuff
# before exiting must wait for handler to finish sending
# I don't want user to have to do this
my_hanler.queue.join()
You can use threading.main_thread.join() which will wait until shutdown like so:
import threading
import logging
import queue
class MyHandler(logging.Handler):
def __init__(self, level):
super().__init__(level=level)
self.queue = queue.Queue()
self.thread = threading.Thread(target=self.consume) # Not daemon
# Shutdown thread
threading.Thread(
target=lambda: threading.main_thread().join() or self.queue.put(None)
).start()
self.thread.start()
def emit(self, record):
# This gets called whenever the user does logging.info, or similar
self.queue.put(record)
def consume(self):
while True:
record = self.queue.get()
if record is None:
print("cleaning")
return # Cleanup
print(record) # send record over network, can take a few seconds (assume it never raises)
self.queue.task_done()
Quick test code:
logging.getLogger().setLevel(logging.INFO)
logging.getLogger().addHandler(MyHandler(logging.INFO))
logging.info("Hello")
exit()
You can use atexit to wait until the daemon thread shuts down:
import queue, threading, time, logging, atexit
class MyHandler(logging.Handler):
def __init__(self, level):
super().__init__(level=level)
self.queue = queue.Queue()
self.thread = threading.Thread(target=self.consume, daemon=True)
# Right before main thread exits, signal cleanup and wait until done
atexit.register(lambda: self.queue.put(None) or self.thread.join())
self.thread.start()
def emit(self, record):
# This gets called whenever the user does logging.info, or similar
self.queue.put(record)
def consume(self):
while True:
record = self.queue.get()
if record is None: # Cleanup requested
print("cleaning")
time.sleep(5)
return
print(record) # send record over network, can take a few seconds (assume it never raises)
self.queue.task_done()
# Test code
logging.getLogger().setLevel(logging.INFO)
logging.getLogger().addHandler(MyHandler(logging.INFO))
logging.info("Hello")
I have my own Thread called TimeBasedLogThread. I would like to fire a function my_function when the TimeBasedLogThread is being killed because the main process is exiting. I would like to do it from within this object. Is it possible to do so?
Here is my current approach:
class TimeBasedBufferingHandler(MemoryHandler):
# This is a logging-based handler that buffers logs to send
# them as emails
# the target of this handler is a SMTPHandler
def __init__(self, capacity=10, flushLevel=logging.ERROR, target=None,
flushOnClose=True, timeout=60):
MemoryHandler.__init__(self, capacity=capacity, flushLevel=flushLevel,
target=target, flushOnClose=flushOnClose)
self.timeout = timeout # in seconds (as time.time())
def flush(self):
# Send the emails that are younger than timeout, all together
# in the same email
class TimeBasedLogThread(Thread):
def __init__(self, handler, timeout=60):
Thread.__init__(self)
self.handler = handler
self.timeout = timeout
def run(self):
while True:
self.handler.flush()
time.sleep(self.timeout)
def my_function(self):
print("my_function is being called")
self.handler.flush()
def setup_thread():
smtp_handler = SMTPHandler()
new_thread = TimeBasedLogThread(smtp_handler, timeout=10)
new_thread.start()
In my main thread, I have:
setup_thread()
logging.error("DEBUG_0")
time.sleep(5)
logging.error("DEBUG_1")
time.sleep(5)
logging.error("DEBUG_2")
The time.sleep(5) releases the main thread 5 seconds before the timeout of my other thread. So, I receive the first 2 emails with "DEBUG_0" and "DEBUG_1", but not the last one "DEBUG_2" because the main process exits before the timeout has finished.
I would like to link the class TimeBasedLogThread and the function my_function that will flush (send the emails) before exiting. How can I do that? I looked at the source code of threading but I did not understand what method I could use.
Build your function as a Thread too. (Ex: AfterDeadThread)
You have two strategy here:
TimeBasedLogThread call AfterDeadThread before die
AfterDeadThread check if TimeBasedLogThread is alive or not, if not it will run some methods
Extend run() method (representing the thread’s activity) to fire the on_terminate handler passed to custom thread’s constructor as keyword argument.
On a slightly changed custom thread class (for demonstration):
from threading import Thread
import time, random
class TimeBasedLogThread(Thread):
def __init__(self, handler, timeout=2, on_terminate=None):
Thread.__init__(self)
self.handler = handler
self.timeout = timeout
self.terminate_handler = on_terminate
def run(self):
while True:
num = self.handler()
if num > 5:
break
time.sleep(self.timeout)
print(num)
if self.terminate_handler:
self.terminate_handler()
def my_term_function():
print("my_function is being called")
f = lambda: random.randint(3, 10)
tlog_thread = TimeBasedLogThread(f, on_terminate=my_term_function)
tlog_thread.start()
tlog_thread.join()
Sample output:
3
4
5
4
5
my_function is being called
I'm trying to build a little python application, where a simple webserver runs in the background, and you can the use GUI to send different messages.
I'm using PyQt5 and Python3.6, and I've managed to pass data from the working thread to the GUI, but I don't know how to do that the other way around.
Here's a skeleton of my code:
MainWindow:
class Ui_MainWindow(object):
def __init__(self):
super().__init__()
self.input = True
# 1 - create Worker and Thread inside the Form
self.obj = WebServer.WebServer(self.input) # no parent!
self.thread = QtCore.QThread() # no parent!
# 2 - Connect Worker`s Signals to Form method slots to post data.
self.obj.dataReady.connect(self.onDataReady)
# 3 - Move the Worker object to the Thread object
self.obj.moveToThread(self.thread)
# 4 - Connect Worker Signals to the Thread slots
self.obj.finished.connect(self.thread.quit)
# 5 - Connect Thread started signal to Worker operational slot method
self.thread.started.connect(self.obj.startServer)
# 6 - Start the thread
self.thread.start()
# 7 - Start the form
self.setupUi()
def setupUi(self):
# Set up the GUI
#...
self.MainWindow.show()
def onDataReady(self, data):
# Data received from working thread
# Do stuff...
def on_click_set2(self):
self.input = not self.input
print(self.input)
if __name__ == "__main__":
import sys
app = QtWidgets.QApplication(sys.argv)
ui = Ui_MainWindow()
sys.exit(app.exec_())
WebServer:
class WebServer(QObject):
finished = pyqtSignal()
dataReady = pyqtSignal(dict)
def __init__(self, input):
super().__init__()
self.input = input
#pyqtSlot()
def startServer(self): # A slot takes no params
# self.loop = asyncio.get_event_loop()
self.loop = asyncio.new_event_loop()
asyncio.set_event_loop(self.loop)
coro = asyncio.start_server(self.handle_update, '192.168.2.1', 8888, loop=self.loop)
self.server = self.loop.run_until_complete(coro)
# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(self.server.sockets[0].getsockname()))
try:
self.loop.run_forever()
except KeyboardInterrupt:
pass
self.finished.emit()
async def handle_update(self, reader, writer):
data = await reader.read(100)
message = data.decode()
addr = writer.get_extra_info('peername')
print(f'Received: {message} from {addr}')
if self.input:
print("Input is True")
else:
print("Input is False")
reply = self.input
print(f'Send: {reply}')
writer.write(str(reply).encode())
await writer.drain()
print("Close the client socket")
writer.close()
self.dataReady.emit(reply)
So for example I want to pass input to the thread, and if I do like above (obviously) input always stays the initial value, and won't change in the thread when I hit the button on the GUI.
How can I do it, that the value of the input is updated whenever I hit the button (so passing a value to the thread from GUI during runtime)? I assume is similar to passing from the thread to the GUI, so emitting a signal from GUI and connecting to it on the tread, but I don't know how to find a reference to the GUI from the working thread.
So any advice on how to do that? And of course any other input regarding the code or approach to the application/background server solution is welcomed! Thanks for the help in advance!
UPDATE:
Maybe it wasn't clear what my question was, so here it is:
How can I send a value from the GUI thread to the worker thread while both of them are running parallel? (If the above code makes any sense to you use that as an example, otherwise a general example would be appreciated)
It is not necessary that the WebServer live in another thread, it is only necessary that the eventloop be executed in another thread. In this case, the WebServer exchanges the data with the server thread through a queue. Although the QObjects are not thread-safe, the signals are so, there is no problem in emitting the signal from a thread other than the one that the QObject lives
import asyncio
import threading
import queue
from functools import partial
from PyQt5 import QtCore, QtGui, QtWidgets
class WebServer(QtCore.QObject):
dataReady = QtCore.pyqtSignal(object)
def startServer(self):
self.m_loop = asyncio.new_event_loop()
self.m_queue = queue.Queue()
asyncio.set_event_loop(self.m_loop)
coro = asyncio.start_server(
self.handle_update, "127.0.0.1", 10000, loop=self.m_loop
)
self.server = self.m_loop.run_until_complete(coro)
print("Serving on {}".format(self.server.sockets[0].getsockname()))
threading.Thread(target=self.m_loop.run_forever, daemon=True).start()
#QtCore.pyqtSlot(object)
def setData(self, data):
if hasattr(self, "m_queue"):
self.m_queue.put(data)
def stop(self):
if hasattr(self, "m_loop"):
self.m_loop.stop()
async def handle_update(self, reader, writer):
reply = ""
data = await reader.read(100)
message = data.decode()
addr = writer.get_extra_info("peername")
print(f"Received: {message} from {addr}")
if not self.m_queue.empty():
data = self.m_queue.get(block=False)
reply = data
print(f"Send: {reply}")
writer.write(str(reply).encode())
await writer.drain()
print("Close the client socket")
writer.close()
self.dataReady.emit(reply)
class Widget(QtWidgets.QWidget):
dataChanged = QtCore.pyqtSignal(object)
def __init__(self, parent=None):
super().__init__(parent)
self.m_lineedit = QtWidgets.QLineEdit()
button = QtWidgets.QPushButton("Send", clicked=self.onClicked)
lay = QtWidgets.QVBoxLayout(self)
lay.addWidget(self.m_lineedit)
lay.addWidget(button)
self.m_web = WebServer()
self.m_web.startServer()
self.dataChanged.connect(self.m_web.setData)
#QtCore.pyqtSlot()
def onClicked(self):
text = self.m_lineedit.text()
self.dataChanged.emit(text)
#QtCore.pyqtSlot(object)
def onDataReady(self, data):
print(data)
def closeEvent(self, event):
self.m_web.stop()
super().closeEvent(event)
if __name__ == "__main__":
import sys
app = QtWidgets.QApplication(sys.argv)
w = Widget()
w.show()
sys.exit(app.exec_())
I am trying to create an abort button in a GUI for stopping threads that are blocking on tcl.eval().
Class Application:
def __init__(self):
self.tcl = Tkinter.Tcl()
def eval(self, cmd):
return self.tcl.eval(cmd)
def stop(self):
# How can I interrupt a thread which is blocking in
# self.eval() from here?
pass
I am working with sockets. When I receive info from the server I handle it with a method listen that is in a thread. I want to pop up windows from here, so I use signals.
The problem is that the signal does not trigger the function. Here is a working example:
class Client(QtCore.QObject):
signal = QtCore.pyqtSignal()
def __init__(self):
super(Client, self).__init__()
self.thread_wait_server = threading.Thread(target=self.wait_server)
self.thread_wait_server.daemon = True
self.thread_wait_server.start()
def wait_server(self):
print('waiting')
self.signal.emit()
print("'signal emited")
class Main:
def Do(self):
print("'Do' starts")
self.Launch()
time.sleep(2)
print("'Do' ends")
def Launch(self):
print("'Launch' starts")
self.client = Client()
self.client.signal.connect(self.Tester)
print("'Launch' ends")
def Tester(self):
print("Tester Fired!!")
m = Main()
m.Do()
Tester function is never triggered.
The problem with your code is that, you are emitting the signal before connecting it to the slot! Add two print statements like this:
print("connecting the signal")
self.client.signal.connect(self.Tester)
print("signal connected")
You will notice that the signal gets emitted before it gets connected! That's why the slot is not triggering.