Develop Reference

Python multiprocessing asynchronous callback

I'm writing a program with a GUI using TKinter, in which the user can click a button and a new process is started to perform work using multiprocess.Process. This is necessary so the GUI can still be used while the work is being done, which can take several seconds.
The GUI also has a text box where the status of the program is displayed when things happen. This is often straight forward, with each function calling an add_text() function which just prints text in the text box. However, when add_text() is called in the separate process, the text does not end up in the text box.
I've thought about using a Pipe or Queue, but that would require using some sort of loop to check if anything has been returned from the process and that would also cause the main (GUI) process to be unusable. Is there some way to call a function in one process that will do work in another?
Here's an simple example of what I'm trying to do
import time
import multiprocessing as mp
import tkinter as tk
textbox = tk.Text()
def add_text(text):
# Insert text into textbox
textbox.insert(tk.END, text)
def worker():
x = 0
while x < 10:
add_text('Sleeping for {0} seconds'.format(x)
x += 1
time.sleep(1)
proc = mp.Process(target=worker)
# Usually happens on a button click
proc.start()
# GUI should still be usable here

The asyncronous things actually require loop.
You could attach function to the TkInter's loop by using Tk.after() method.
import Tkinter as tk
class App():
def __init__(self):
self.root = tk.Tk()
self.check_processes()
self.root.mainloop()
def check_processes(self):
if process_finished:
do_something()
else:
do_something_else()
self.after(1000, check_processes)
app=App()

I ended up using a multiprocessing.Pipe by using TKinter's after() method to perform the looping. It loops on an interval and checks the pipe to see if there's any messages from the thread, and if so it inserts them into the text box.
import tkinter
import multiprocessing
def do_something(child_conn):
while True:
child_conn.send('Status text\n')
class Window:
def __init__(self):
self.root = tkinter.Tk()
self.textbox = tkinter.Text()
self.parent_conn, child_conn = multiprocessing.Pipe()
self.process = multiprocessing.Process(target=do_something, args=(child_conn,))
def start(self):
self.get_status_updates()
self.process.start()
self.root.mainloop()
def get_status_updates()
status = self.check_pipe()
if status:
self.textbox.add_text(status)
self.root.after(500, self.get_status_updates) # loop every 500ms
def check_pipe():
if self.parent_conn.poll():
status = self.parent_conn.recv()
return status
return None

Creating multiple instances of wx.App - is it ok?

so I need to implement a following scenario:
- Several tasks are running simultaneously as processes.
- Each task should display a progress bar with a "Cancel" button, clicking on which should terminate it.
To achieve responsive GUI, I run the task for each process in a separate thread, and it seems that I need to create a separate wx.App for each process as well, otherwise the thread seems to be not running. This setup works fine, however:
a) I am not sure whether multiple wx.App's is a good idea or
b) If there is a better way of achieving my goal.
MWE below (note: in this sample code I could have used Update method of wx.ProgressDialog to identify whether the "Cancel" button has been pressed, but can't do so for my real application).
import wx, multiprocessing, time, psutil
from multiprocessing import Queue
from threading import Thread
from wx.lib.pubsub import pub as Publisher
#runs the task
def task_runner(q):
pid = multiprocessing.current_process().pid
q.put(pid)
while True:
print("Process Running")
time.sleep(1)
wx.CallAfter(Publisher.sendMessage, "update") #call to update the bar
class TestPanel():
def __init__(self,name):
self.q = Queue()
self.count=0
max = 80
# dialog to see progress and cancel the task
self.dlg = wx.GenericProgressDialog(name,
"An informative message",
maximum = max,
parent=None,
style = wx.PD_CAN_ABORT
| wx.PD_APP_MODAL
| wx.PD_ELAPSED_TIME
)
#set listener to dialog's "Cancel" button
for child in self.dlg.GetChildren():
if isinstance(child, wx.Button):
cancel_function = lambda evt, parent=self.dlg: self.onClose(evt, parent)
child.Bind(wx.EVT_BUTTON, cancel_function)
#subscribe to update the progress bar from the thread
Publisher.subscribe(self.updateProgress, "update")
# start thread which runs some task
p = Thread(target=task_runner, args=(self.q,))
p.start()
#updates the progress bar
def updateProgress(self):
print("updating progress")
self.count=self.count+10
self.dlg.Update(self.count)
#kills the process
def kill(self, proc_pid):
process = psutil.Process(proc_pid)
for proc in process.children(recursive=True):
proc.kill()
process.kill()
#closing the dialog event
def onClose(self, event, dialog):
""""""
print "Closing dialog!"
pid = self.q.get()
self.kill(pid)
dialog.Destroy()
# run process, each process creates its own wx.App
def runProcess(name):
app = wx.App(False)
TestPanel(name)
app.MainLoop()
# worker class to use for multiprocessing pool
class Worker():
def __call__(self, name):
return runProcess(name)
if __name__ == '__main__':
items=['Bar1', 'Bar2']
pool = multiprocessing.Pool(processes=2)
result = pool.map(Worker(), items) #create two processes
pool.close()

No, having more than one wx.App in a single process is not a good idea. Even creating a new one after the prior is finished can sometimes be problematic.
However since you are using multiprocess it is not quite the same. Unless I'm missing something, each OS process does have just one wx.App in your case, and since the parent process is not creating a wx.App then they are not trying to inherit that one (which would likely cause even more problems.)

How to signal from a running QThread back to the PyQt Gui that started it?

I am trying to understand how to use signaling from a Qthread back to the Gui interface that started.
Setup: I have a process (a simulation) that needs to run almost indefinitely (or at least for very long stretches of time)., While it runs, it carries out various computations, amd some of the results must be sent back to the GUI, which will display them appropriately in real time.
I am using PyQt for the GUI. I originally tried using python's threading module, then switched to QThreads after reading several posts both here on SO and elsewhere.
According to this post on the Qt Blog You're doing it wrong, the preferred way to use QThread is by creating a QObject and then moving it to a Qthread. So I followed the advice inBackground thread with QThread in PyQt"> this SO question and tried a simple test app (code below): it opens up a simple GUI, let you start the background process, and it issupposed to update the step value in a spinbox.
But it does not work. The GUI is never updated. What am I doing wrong?
import time, sys
from PyQt4.QtCore import *
from PyQt4.QtGui import *
class SimulRunner(QObject):
'Object managing the simulation'
stepIncreased = pyqtSignal(int, name = 'stepIncreased')
def __init__(self):
super(SimulRunner, self).__init__()
self._step = 0
self._isRunning = True
self._maxSteps = 20
def longRunning(self):
while self._step < self._maxSteps and self._isRunning == True:
self._step += 1
self.stepIncreased.emit(self._step)
time.sleep(0.1)
def stop(self):
self._isRunning = False
class SimulationUi(QDialog):
'PyQt interface'
def __init__(self):
super(SimulationUi, self).__init__()
self.goButton = QPushButton('Go')
self.stopButton = QPushButton('Stop')
self.currentStep = QSpinBox()
self.layout = QHBoxLayout()
self.layout.addWidget(self.goButton)
self.layout.addWidget(self.stopButton)
self.layout.addWidget(self.currentStep)
self.setLayout(self.layout)
self.simulRunner = SimulRunner()
self.simulThread = QThread()
self.simulRunner.moveToThread(self.simulThread)
self.simulRunner.stepIncreased.connect(self.currentStep.setValue)
self.connect(self.stopButton, SIGNAL('clicked()'), self.simulRunner.stop)
self.connect(self.goButton, SIGNAL('clicked()'), self.simulThread.start)
self.connect(self.simulRunner,SIGNAL('stepIncreased'), self.currentStep.setValue)
if __name__ == '__main__':
app = QApplication(sys.argv)
simul = SimulationUi()
simul.show()
sys.exit(app.exec_())

The problem here is simple: your SimulRunner never gets sent a signal that causes it to start its work. One way of doing that would be to connect it to the started signal of the Thread.
Also, in python you should use the new-style way of connecting signals:
...
self.simulRunner = SimulRunner()
self.simulThread = QThread()
self.simulRunner.moveToThread(self.simulThread)
self.simulRunner.stepIncreased.connect(self.currentStep.setValue)
self.stopButton.clicked.connect(self.simulRunner.stop)
self.goButton.clicked.connect(self.simulThread.start)
# start the execution loop with the thread:
self.simulThread.started.connect(self.simulRunner.longRunning)
...

multiprocessing and GUI updating - Qprocess or multiprocessing?

After reading the literature on QProcesses and the multiprocessing module for python, I am still having trouble creating a working and responsive GUI throughout having large processes ongoing in the background.
So far, I have come up with this simplified version of my application, which still shows similar problems to what many have described.
from PyQt4 import QtCore, QtGui
import multiprocessing as mp
import numpy as np
import sys
class Spectra:
def __init__(self, spectra_name, X, Y):
self.spectra_name = spectra_name
self.X = X
self.Y = Y
self.iteration = 0
def complex_processing_on_spectra(self, pipe_conn):
self.iteration += 1
pipe_conn.send(self.iteration)
class Spectra_Tab(QtGui.QTabWidget):
def __init__(self, parent, spectra):
self.parent = parent
self.spectra = spectra
QtGui.QTabWidget.__init__(self, parent)
self.treeWidget = QtGui.QTreeWidget(self)
self.properties = QtGui.QTreeWidgetItem(self.treeWidget, ["Properties"])
self.step = QtGui.QTreeWidgetItem(self.properties, ["Iteration #"])
self.consumer, self.producer = mp.Pipe()
# Make process associated with tab
self.process = mp.Process(target=self.spectra.complex_processing_on_spectra, args=(self.producer,))
def update_GUI(self, iteration):
self.step.setText(1, str(iteration))
def start_computation(self):
self.process.start()
while(True):
message = self.consumer.recv()
if message == 'done':
break
self.update_GUI(message)
self.process.join()
return
class MainWindow(QtGui.QMainWindow):
def __init__(self, parent = None):
QtGui.QMainWindow.__init__(self)
self.setTabShape(QtGui.QTabWidget.Rounded)
self.centralwidget = QtGui.QWidget(self)
self.top_level_layout = QtGui.QGridLayout(self.centralwidget)
self.tabWidget = QtGui.QTabWidget(self.centralwidget)
self.top_level_layout.addWidget(self.tabWidget, 1, 0, 25, 25)
process_button = QtGui.QPushButton("Process")
self.top_level_layout.addWidget(process_button, 0, 1)
QtCore.QObject.connect(process_button, QtCore.SIGNAL("clicked()"), self.process)
self.setCentralWidget(self.centralwidget)
self.centralwidget.setLayout(self.top_level_layout)
# Open several files in loop from button - simplifed to one here
X = np.arange(0.1200,.2)
Y = np.arange(0.1200,.2)
self.spectra = Spectra('name', X, Y)
self.spectra_tab = Spectra_Tab(self.tabWidget, self.spectra)
self.tabWidget.addTab(self.spectra_tab, 'name')
def process(self):
self.spectra_tab.start_computation()
return
if __name__ == "__main__":
app = QtGui.QApplication([])
win = MainWindow()
win.show()
sys.exit(app.exec_())
This should be fully capable of executing if you have the dependencies.
At the moment I have a QThreaded version of my program which works with signals and slots; Hwoever, I think it is important to have the ability to use all of a computers processors, since most users have ~8 cores available to them. So, I would like to expand this signal/slot threaded approach to the multiprocessed version using multiprocessing or QProcesses.
Does anyone have suggestions for whether or not to use QProcess or multiprocessing? While they are both complicated to me, QProcess seems as though it has less forums of people using pyQt, So I went with multiprocessing. Would it be simpler to go with QProcess since I already have signals/slots working with threads?
EDIT: Should I add a class like this as suggested?
class My_Process(QtCore.QProcess):
def __init__(self, spectra):
QtCore.QProcess.__init__(self)
self.spectra = spectra
def worker(self):
QtConcurrent.run(self.spectra, self.spectra.complex_processing_on_spectra)
def run(self):
QtCore.QObject.connect(self, QtCore.SIGNAL(QTimer.timeout()), self.worker)

Even though the question is old and has been answered I would like add some clarification:
AFAIK QtConcurrent is not available in PyQt.
Threads in Qt with C++ are different from PyQt or python threads. The latter need to acquire python's GIL (global interpreter lock) when they run which effectively means that there is no real concurrency between python/pyqt threads.
Python threads are ok to keep the gui responsive, but you won't see any performance improvements for cpu bound tasks. I recommend using multiprocessing together with a QThread on your main process that handles the communication with the child process. You can use signals and slots between your main (gui) and your communication thread.
edit: I just had the same problem and did something along these lines:
from multiprocessing import Process, Queue
from PyQt4 import QtCore
from MyJob import job_function
# Runner lives on the runner thread
class Runner(QtCore.QObject):
"""
Runs a job in a separate process and forwards messages from the job to the
main thread through a pyqtSignal.
"""
msg_from_job = QtCore.pyqtSignal(object)
def __init__(self, start_signal):
"""
:param start_signal: the pyqtSignal that starts the job
"""
super(Runner, self).__init__()
self.job_input = None
start_signal.connect(self._run)
def _run(self):
queue = Queue()
p = Process(target=job_function, args=(queue, self.job_input))
p.start()
while True:
msg = queue.get()
self.msg_from_job.emit(msg)
if msg == 'done':
break
# Things below live on the main thread
def run_job(input):
""" Call this to start a new job """
runner.job_input = input
runner_thread.start()
def handle_msg(msg):
print(msg)
if msg == 'done':
runner_thread.quit()
runner_thread.wait()
# Setup the OQ listener thread and move the OQ runner object to it
runner_thread = QtCore.QThread()
runner = Runner(start_signal=runner_thread.started)
runner.msg_from_job.connect(handle_msg)
runner.moveToThread(runner_thread)

QProcess is for reading and writing to pipes (shell/cmd). Use one of these.
Create a class with worker function and run it as thread by QtConcurrent.run(object, method, args)
connect QTimer.timeout() with your worker function.

pygtk and multiprocessing - Constantly update textview from another process running infinite loop

I'm building a python app with pygtk. It consists of some buttons that activate/deactivate some infinite looped processes and a textview that should keep showing whats going on inside each process. Like verbose stuff.
These processeses hasn't an end. They stop only when the user hit it's corresponding button (or close the app).
What's going wrong: I cant print stuff in the textview from these processes. Maybe because they haven't an end...
Actually the app is too big to show the whole code here. So I've made a simple and little example of what I'm doing.
import pygtk
pygtk.require("2.0")
import gtk
import time
import glib
from multiprocessing import Process
gtk.threads_init()
class Test(gtk.Window):
def delete_event(self, widget, event, data=None):
if isinstance(self.my_process, Process):
if self.my_process.is_alive():
self.my_process.terminate()
gtk.main_quit()
return False
def __init__(self):
gtk.Window.__init__(self)
self.set_default_size(500, 400)
self.set_title(u"Test")
self.connect("delete_event", self.delete_event)
self.mainBox = gtk.VBox(False, 5)
self.text = gtk.TextView()
self.text.set_wrap_mode(gtk.WRAP_WORD)
self.button = gtk.Button("Start")
self.add(self.mainBox)
self.mainBox.pack_start(self.text, True, True, 0)
self.mainBox.pack_start(self.button, False, True, 0)
self.button.connect("clicked", self.start_clicked)
self.show_all()
def start_clicked(self, widget):
self.register_data("Starting...")
self.my_process = Process(target=self.do_something)
self.my_process.start()
def do_something(self):
while True:
time.sleep(0.5)
#get a list of a lot of things
#Do stuff with each item in the list
#show me on the gui whats going on
glib.idle_add(self.register_data, "Yo! Here I'm")
print "Hello, boy."
def register_data(self, data):
data = data + "\r\n"
#gtk.gdk.threads_enter()
buff = self.text.get_buffer()
biter = buff.get_start_iter()
buff.insert(biter, data)
#gtk.gdk.threads_leave()
if __name__ == "__main__":
mnc = Test()
mnc.set_position(gtk.WIN_POS_CENTER)
gtk.threads_enter()
gtk.main()
gtk.threads_leave()

Remove all .threads_init(), .threads_enter(), .threads_leave(). multiprocessing is not threading.
Put data you'd like to display into multiprocessing.Queue() in your child process:
def do_something(self):
while True:
#get a list of a lot of things
#Do stuff with each item in the list
#show me on the gui whats going on
self.data_queue.put("Yo! Here I'm")
and poll it in GUI loop:
def __init__(self, ...):
# ...
self.data_queue = Queue()
gobject.timeout_add(100, self.update_text)
where:
def update_text(self):
# receive updates from the child process here
try:
data = self.data_queue.get_nowait()
except Empty:
pass # nothing at this time
else:
self.register_data(data)
return True
To avoid polling you could write to multiprocessing.Pipe in your child process and setup GUI callback using gobject.io_add_watch(). Here's a complete code example:
#!/usr/bin/python3
from multiprocessing import Pipe, Process
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import GObject, Gtk
# create GUI to show multiprocessing output
win = Gtk.Window()
win.set_default_size(640, 480)
label = Gtk.Label('process output')
win.add(label)
# start dummy infinite loop in a child process
def loop(conn):
import itertools, sys, time
for i in itertools.count():
conn.send(i)
time.sleep(0.1 - time.monotonic() % 0.1)
parent_conn, child_conn = Pipe(duplex=False)
Process(target=loop, args=[child_conn], daemon=True).start()
child_conn.close()
# read values from the child
def read_data(source, condition):
assert parent_conn.poll()
try:
i = parent_conn.recv()
except EOFError:
return False # stop reading
# update text
label.set_text('Result from the child: %03d' % (i,))
return True # continue reading
# . configure the callback
GObject.io_add_watch(parent_conn.fileno(), GObject.IO_IN, read_data)
win.connect('delete-event', Gtk.main_quit)
win.show_all()
Gtk.main()
You can also do it with an arbitrary subprocess (not just a python child process).

You should use gtk.threads_enter() when you are inside a thread to every call to
gtk and close with gtk.threads_leave() after call him.
Something like:
def do_something(self):
while True:
time.sleep(0.5)
gtk.threads_enter()
#get a list of a lot of things
#Do stuff with each item in the list
#show me on the gui whats going on
glib.idle_add(self.register_data, "Yo! Here I'm")
gtk.threads_leave()
print "Hello, boy."
and sometimes you need to use:
gtk.gdk.threads_init()
gtk.gdk.threads_enter()
#code section
gtk.gdk.threads_leave()

Here's a threaded example. But a non-threaded, non polling approach would be better, because in GTK 3, the threads_enter/threads_leave have been deprecated, so your program would be harder to port to GTK 3 + PyGObject.
In C, one would probably use g_spawn_async_with_pipes. An equivalent in python would be glib.spawn_async I guess, and you'd use glib.io_add_watch to be notified when there's data to read in the standard output.