Develop Reference

Python 3, Tkinter and threading a long-running process, OOP style

My little program has a potentially long running process. That's not a problem when doing it from the console, but now I want to add a GUI. Ideally I want to use Tkinter (a) because it's simple, and (b) because it might be easier to implement across platforms. From what I've read and experienced, (almost) all GUIs suffer the same issue anyway.
Through all my reading on the subject of threading and GUI there seem to be two streams. 1 - where the underlying worker process is polling (eg waiting to fetch data), and 2 - where the worker process is doing a lot of work (eg copying files in a for loop). My program falls into the latter.
My code has a "hierarchy" of classes.
The MIGUI class handles the GUI and interacts with the interface class MediaImporter.
The MediaImporter class is the interface between the user interface (console or GUI) and the worker classes.
The Import class is the long-running worker. It does not know that the interface or GUI classes exist.
The problem: After clicking the Start button, the GUI is blocked, so I can't click the Abort button. It is as if I'm not using threading at all. I suspect the issue is with the way I am starting the threading in startCallback method.
I've also tried the approach of threading the entire MediaImporter class. See the commented-out lines.
import tkinter as tk
from tkinter import ttk
from tkinter import filedialog
import threading
import time
class MIGUI():
def __init__(self, master):
self.master = master
self.mediaImporter = MediaImporter()
self.startButton = ttk.Button(self.master, text='Start', command=self.startCallback)
self.startButton.pack()
self.abortButton = ttk.Button(self.master, text='Abort', command=self.abortCallback)
self.abortButton.state(['disabled'])
self.abortButton.pack()
def startCallback(self):
print('startCallback')
self.abortButton.state(['!disabled'])
self.startButton.state(['disabled'])
self.abortButton.update() # forcing the update seems unnecessary
self.startButton.update()
#print(self.startButton.state())
#print(self.abortButton.state())
self.x = threading.Thread(target=self.mediaImporter.startImport)
self.x.start()
self.x.join()
#self.mediaImporter.startImport()
self.startButton.state(['!disabled'])
self.abortButton.state(['disabled'])
self.abortButton.update()
self.startButton.update()
#print(self.startButton.state())
#print(self.abortButton.state())
def abortCallback(self):
print('abortCallback')
self.mediaImporter.abortImport()
self.startButton.state(['!disabled'])
self.abortButton.state(['disabled'])
class MediaImporter():
#class MediaImporter(threading.Thread):
""" Interface between user (GUI / console) and worker classes """
def __init__(self):
#threading.Thread.__init__(self)
self.Import = Import()
#other worker classes exist too
def startImport(self):
print('mediaImporter - startImport')
self.Import.start()
def abortImport(self):
print('mediaImporter - abortImport')
self.Import.abort()
class Import():
""" Worker
Does not know anything about other non-worker classes or UI.
"""
def __init__(self):
self._wantAbort = False
def start(self):
print('import - start')
self._wantAbort = False
self.doImport()
def abort(self):
print('import - abort')
self._wantAbort = True
def doImport(self):
print('doImport')
for i in range(0,10):
#actual code has nested for..loops
print(i)
time.sleep(.25)
if self._wantAbort:
print('doImport - abort')
return
def main():
gui = True
console = False
if gui:
root = tk.Tk()
app = MIGUI(root)
root.mainloop()
if console:
#do simple console output without tkinter - threads not necessary
pass
if __name__ == '__main__':
main()

The reason your GUI is blocked is because you call self.x.join(), which blocks until the doImport function is complete, see the join documentation. Instead I would call join() in your abortCallback() function, since that is what will cause the thread to stop running.

Thank you again XORNAND. The join() was definitely part of the problem. The other part of the problem was that there was no means of the MIGUI class knowing when the long-running process was complete (either because it had run its course, or because it was aborted.) An additional layer of messaging is required between the low-level worker, and the UI layer. I did try to use threading.Event without success, and did consider using Queues.
My solution is to use pubsub. (https://github.com/schollii/pypubsub) The worker layer can sendMessage on various topics, and the UI and interface layers can set up Listeners to perform actions with received data.
In my case, the Import.doImport method sends a STATUS message when it is completed. The MIGUI listener can then flip-flop the Start/Abort buttons accordingly.
To make sure the implementation of pubsub was going to work as planned I also set up a tkinter Progressbar. The doImport method sends a PROGESS message with the percent complete. This is reflected in the on-screen Progressbar.
A side note - in my original issue I had to use .update() on the buttons to get them to display. Now that we're not blocking anymore, this is not necessary.
Posting the complete working solution here, showing the pubsub implementation.
import tkinter as tk
from tkinter import ttk
import threading
import time
from pubsub import pub
class MIGUI():
def __init__(self, master):
self.master = master
self.mediaImporter = MediaImporter()
self.startButton = ttk.Button(self.master, text='Start', command=self.startCallback)
self.startButton.pack()
self.abortButton = ttk.Button(self.master, text='Abort', command=self.abortCallback)
self.abortButton.state(['disabled'])
self.abortButton.pack()
self.progress = ttk.Progressbar(self.master, length=300)
self.progress.pack()
pub.subscribe(self.statusListener, 'STATUS')
pub.subscribe(self.progressListener, 'PROGRESS')
def statusListener(self, status, value):
print('MIGUI', status, value)
if status == 'copying' and (value == 'finished' or value == 'aborted'):
self.startButton.state(['!disabled'])
self.abortButton.state(['disabled'])
def progressListener(self, value):
print('Progress %d' % value)
self.progress['maximum'] = 100
self.progress['value'] = value
def startCallback(self):
print('startCallback')
self.abortButton.state(['!disabled'])
self.startButton.state(['disabled'])
self.x = threading.Thread(target=self.mediaImporter.startImport)
self.x.start()
# original issue had join() here, which was blocking.
def abortCallback(self):
print('abortCallback')
self.mediaImporter.abortImport()
class MediaImporter():
""" Interface between user (GUI / console) and worker classes """
def __init__(self):
self.Import = Import()
#other worker classes exist too
pub.subscribe(self.statusListener, 'STATUS')
def statusListener(self, status, value):
#perhaps do something
pass
def startImport(self):
self.Import.start()
def abortImport(self):
self.Import.abort()
class Import():
""" Worker
Does not know anything about other non-worker classes or UI.
It does use pubsub to publish messages - such as the status and progress.
The UI and interface classes can subsribe to these messages and perform actions. (see listener methods)
"""
def __init__(self):
self._wantAbort = False
def start(self):
self._wantAbort = False
self.doImport()
def abort(self):
pub.sendMessage('STATUS', status='abort', value='requested')
self._wantAbort = True
def doImport(self):
self.max = 13
pub.sendMessage('STATUS', status='copying', value='started')
for i in range(1,self.max):
#actual code has nested for..loops
progress = ((i+1) / self.max * 100.0)
pub.sendMessage('PROGRESS', value=progress)
time.sleep(.1)
if self._wantAbort:
pub.sendMessage('STATUS', status='copying', value='aborted')
return
pub.sendMessage('STATUS', status='copying', value='finished')
def main():
gui = True
console = False
if gui:
root = tk.Tk()
app = MIGUI(root)
root.mainloop()
if console:
#do simple console output without tkinter - threads not necessary
pass
if __name__ == '__main__':
main()

Python multiprocessing asynchronous callback

I'm writing a program with a GUI using TKinter, in which the user can click a button and a new process is started to perform work using multiprocess.Process. This is necessary so the GUI can still be used while the work is being done, which can take several seconds.
The GUI also has a text box where the status of the program is displayed when things happen. This is often straight forward, with each function calling an add_text() function which just prints text in the text box. However, when add_text() is called in the separate process, the text does not end up in the text box.
I've thought about using a Pipe or Queue, but that would require using some sort of loop to check if anything has been returned from the process and that would also cause the main (GUI) process to be unusable. Is there some way to call a function in one process that will do work in another?
Here's an simple example of what I'm trying to do
import time
import multiprocessing as mp
import tkinter as tk
textbox = tk.Text()
def add_text(text):
# Insert text into textbox
textbox.insert(tk.END, text)
def worker():
x = 0
while x < 10:
add_text('Sleeping for {0} seconds'.format(x)
x += 1
time.sleep(1)
proc = mp.Process(target=worker)
# Usually happens on a button click
proc.start()
# GUI should still be usable here

The asyncronous things actually require loop.
You could attach function to the TkInter's loop by using Tk.after() method.
import Tkinter as tk
class App():
def __init__(self):
self.root = tk.Tk()
self.check_processes()
self.root.mainloop()
def check_processes(self):
if process_finished:
do_something()
else:
do_something_else()
self.after(1000, check_processes)
app=App()

I ended up using a multiprocessing.Pipe by using TKinter's after() method to perform the looping. It loops on an interval and checks the pipe to see if there's any messages from the thread, and if so it inserts them into the text box.
import tkinter
import multiprocessing
def do_something(child_conn):
while True:
child_conn.send('Status text\n')
class Window:
def __init__(self):
self.root = tkinter.Tk()
self.textbox = tkinter.Text()
self.parent_conn, child_conn = multiprocessing.Pipe()
self.process = multiprocessing.Process(target=do_something, args=(child_conn,))
def start(self):
self.get_status_updates()
self.process.start()
self.root.mainloop()
def get_status_updates()
status = self.check_pipe()
if status:
self.textbox.add_text(status)
self.root.after(500, self.get_status_updates) # loop every 500ms
def check_pipe():
if self.parent_conn.poll():
status = self.parent_conn.recv()
return status
return None

Tkinter: Creating async timer, but both time and tk.after don't work, causing infinite loop

I'm trying to create a timer that updates the Tkinter GUI every second, but neither time nor tk.after are actually delaying the time, causing my program to crash. I'm using this timer to update my widgets in the main thread, while another worker Process runs a test. Here's what I tried (relevant parts only):
import time
import threading
from Tkinter import *
class View(Frame):
def __init__(self, top):
Frame.__init__(self, top)
self.top = top
self.async_update_thread = threading.Thread(target=self.async_update, args=(self.top,))
self.async_update_thread.start()
def async_update(self, top):
top.update()
top.after(1000, self.async_update(top))
'''
while True:
top.update()
time.sleep(1)
'''
if (__name__ == '__main__'):
top = Tk()
top.geometry("1250x750")
View(top)
top.mainloop()
This code results in the following error:
TclError: out of stack space (infinite loop?)
How should I handle this error that seems to ignore my delay settings?

Consider this code:
top.after(1000, self.async_update(top))
It is exactly the same as this code:
result = self.async_update(top)
top.after(1000, result)
See the problem? after requires a callable. Instead, you are calling the function immediately.
The proper way to call after is like this:
top.after(1000, self.async_update, top)

How to handle errors in tkinter mainloop?

I have a python program which is scraping web data for a client. tkinter is used for the interface. Outline is:
Window 1 lets the user select what information to scrape.
Window 1 closes
Separate thread is started for the scraper. This thread will in turn spawn more threads to allow multiple pages to be downloaded at once.
Window 2 opens to show download progress (e.g. "downloading client 5 of 17")
User closes Window 2 to end program.
The program will work for the first few hundred pages, but then it starts spitting out the error message:
Traceback (most recent call last):
File "C:\Users\Me\AppData\Local\Programs\Python\Python35-32\lib\tkinter\__init__.py", line 248, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <bound method Variable.__del__ of <tkinter.IntVar object at 0x03245510>>
multiple times until all the threads have been stopped. No idea what could be causing this error. The actual code is:
import scraper, threading
import tkinter as tk
from queue import Queue
outputQueue = Queue()
class OutputRedirect(object):
def __init__():
super().__init__()
def write(self, string):
outputQueue.put(string)
def getInformation():
stdout = sys.stdout
sys.stdout = OutputRedirect()
scraper.startThreads()
scraper.startPulling()
sys.stdout = stdout
def updateTextField(window, root):
if not outputQueue.empty():
string = outputQueue.get()
window.textArea.insert("insert", string)
outputQueue.task_done()
root.after(1, updateTextField, window, root)
'''widget/window definitions - not important'''
guiInfo = {"stuff1": [], "stuff2": []}
root = tk.Tk()
window1 = Window1(root, guiInfo)
window1.mainloop()
pullThread = threading.Thread(target=pullClaims,
args=(list(guiInfo["stuff1"]),
list(guiInfo["stuff2"])), daemon=True)
pullThread.start()
root = tk.Tk()
window2 = Window2(root)
root.after(0, updateTextField, window2, root)
window2.mainloop()
The scraper program (which works fine on its own) uses print statements for user feedback. Rather than re-write everything, I just pointed stdout to a queue. The main thread uses the "after" function to check on the queue a few times a second. If there is anything in it then it gets printed to the Text widget on the window.
I've put try/catch just about everywhere in the code, but they haven't caught a thing. I'm convinced the problem is in the mainloop itself, but I can't find any up to date information for how to stick something new in it. Any help would be greatly appreciated.

To handle tkinter errors you do the following
class TkErrorCatcher:
'''
In some cases tkinter will only print the traceback.
Enables the program to catch tkinter errors normally
To use
import tkinter
tkinter.CallWrapper = TkErrorCatcher
'''
def __init__(self, func, subst, widget):
self.func = func
self.subst = subst
self.widget = widget
def __call__(self, *args):
try:
if self.subst:
args = self.subst(*args)
return self.func(*args)
except SystemExit as msg:
raise SystemExit(msg)
except Exception as err:
raise err
import tkinter
tkinter.CallWrapper = TkErrorCatcher
But in your case please do not do this. This should only ever be done in the case of wanting to hide error messages from your users in production time. As commented above you have a nono's going on.
To spawn multiple windows you can use tkinter.Toplevel
I would recommend in general to read
http://www.tkdocs.com/tutorial/index.html
http://effbot.org/tkinterbook/tkinter-hello-tkinter.htm
http://infohost.nmt.edu/tcc/help/pubs/tkinter/web/index.html
and for your specific problem of threading in tkinter this blog post nails it. Basically you need to have the tkinter mainloop blocking the programs main thread, then use after calls from other threads to run other code in the mainloop.
http://stupidpythonideas.blogspot.de/2013/10/why-your-gui-app-freezes.html

How can I embed a python interpreter frame in python using tkinter?

I want to add a control terminal widget to my pure python+tkinter application similar to the python interpreter provided in Blender. It should be running within the same context (process) so the user can add features and control the application that is currently running from the control widget. Ideally I'd like it to also "hijack" stdout and stderr of the current application so it will report any problems or debugging information within the running application.
This is what I have come up with so far. The only problems are that it isn't responding to commands, and the thread doesn't stop when the user closes the window.
import Tkinter as tk
import sys
import code
from threading import *
class Console(tk.Frame):
def __init__(self,parent=None):
tk.Frame.__init__(self, parent)
self.parent = parent
sys.stdout = self
sys.stderr = self
self.createWidgets()
self.consoleThread = ConsoleThread()
self.after(100,self.consoleThread.start)
def write(self,string):
self.ttyText.insert('end', string)
self.ttyText.see('end')
def createWidgets(self):
self.ttyText = tk.Text(self.parent, wrap='word')
self.ttyText.grid(row=0,column=0,sticky=tk.N+tk.S+tk.E+tk.W)
class ConsoleThread(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
vars = globals().copy()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root)
main_window.mainloop()
try:
if root.winfo_exists():
root.destroy()
except:
pass

I have the answer in case anyone still cares! (I have also changed to python 3, hence the import tkinter rather than import Tkinter)
I have changed the approach slightly from the original by using a separate file to run the InteractiveConsole, and then making the main file open this other file (which I have called console.py and is in the same directory) in a subprocess, linking the stdout, stderr, and stdin of this subprocess to the tkinter Text widget programatically.
Here is the code in the for the console file (if this is run normally, it acts like a normal console):
# console.py
import code
if __name__ == '__main__':
vars = globals().copy()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()
And here is the code for the python interpreter, that runs the console inside the Text widget:
# main.py
import tkinter as tk
import subprocess
import queue
import os
from threading import Thread
class Console(tk.Frame):
def __init__(self,parent=None):
tk.Frame.__init__(self, parent)
self.parent = parent
self.createWidgets()
# get the path to the console.py file assuming it is in the same folder
consolePath = os.path.join(os.path.dirname(__file__),"console.py")
# open the console.py file (replace the path to python with the correct one for your system)
# e.g. it might be "C:\\Python35\\python"
self.p = subprocess.Popen(["python3",consolePath],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
# make queues for keeping stdout and stderr whilst it is transferred between threads
self.outQueue = queue.Queue()
self.errQueue = queue.Queue()
# keep track of where any line that is submitted starts
self.line_start = 0
# make the enter key call the self.enter function
self.ttyText.bind("<Return>",self.enter)
# a daemon to keep track of the threads so they can stop running
self.alive = True
# start the functions that get stdout and stderr in separate threads
Thread(target=self.readFromProccessOut).start()
Thread(target=self.readFromProccessErr).start()
# start the write loop in the main thread
self.writeLoop()
def destroy(self):
"This is the function that is automatically called when the widget is destroyed."
self.alive=False
# write exit() to the console in order to stop it running
self.p.stdin.write("exit()\n".encode())
self.p.stdin.flush()
# call the destroy methods to properly destroy widgets
self.ttyText.destroy()
tk.Frame.destroy(self)
def enter(self,e):
"The <Return> key press handler"
string = self.ttyText.get(1.0, tk.END)[self.line_start:]
self.line_start+=len(string)
self.p.stdin.write(string.encode())
self.p.stdin.flush()
def readFromProccessOut(self):
"To be executed in a separate thread to make read non-blocking"
while self.alive:
data = self.p.stdout.raw.read(1024).decode()
self.outQueue.put(data)
def readFromProccessErr(self):
"To be executed in a separate thread to make read non-blocking"
while self.alive:
data = self.p.stderr.raw.read(1024).decode()
self.errQueue.put(data)
def writeLoop(self):
"Used to write data from stdout and stderr to the Text widget"
# if there is anything to write from stdout or stderr, then write it
if not self.errQueue.empty():
self.write(self.errQueue.get())
if not self.outQueue.empty():
self.write(self.outQueue.get())
# run this method again after 10ms
if self.alive:
self.after(10,self.writeLoop)
def write(self,string):
self.ttyText.insert(tk.END, string)
self.ttyText.see(tk.END)
self.line_start+=len(string)
def createWidgets(self):
self.ttyText = tk.Text(self, wrap=tk.WORD)
self.ttyText.pack(fill=tk.BOTH,expand=True)
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root)
main_window.pack(fill=tk.BOTH,expand=True)
root.mainloop()
The reason that reading from stdout and stderr is in separate threads is because the read method is blocking, which causes the program to freeze until the console.py subprocess gives more output, unless these are in separate threads. The writeLoop method and the queues are needed to write to the Text widget since tkinter is not thread safe.
This certainly still has problems to be ironed out, such as the fact that any code on the Text widget is editable even once already submitted, but hopefully it answers your question.
EDIT: I've also neatened some of the tkinter such that the Console will behave more like a standard widget.

it isn't responding to commands
The reason it isn't responding to commands is because you haven't linked the Text widget (self.ttyText) into stdin. Currently when you type it adds text into the widget and nothing else. This linking can be done similarly to what you've already done with stdout and stderr.
When implementing this, you need to keep track of which part of the text in the widget is the text being entered by the user - this can be done using marks (as described here).
the thread doesn't stop when the user closes the window.
I don't think there is a "clean" way to solve this issue without a major code re-write, however a solution that seems to work well enough is it simply detect when the widget is destroyed and write the string "\n\nexit()" to the interpreter. This calls the exit function inside the interpreter, which causes the call to shell.interact to finish, which makes the thread finish.
So without further ado, here is the modified code:
import tkinter as tk
import sys
import code
from threading import Thread
import queue
class Console(tk.Frame):
def __init__(self, parent, _locals, exit_callback):
tk.Frame.__init__(self, parent)
self.parent = parent
self.exit_callback = exit_callback
self.destroyed = False
self.real_std_in_out = (sys.stdin, sys.stdout, sys.stderr)
sys.stdout = self
sys.stderr = self
sys.stdin = self
self.stdin_buffer = queue.Queue()
self.createWidgets()
self.consoleThread = Thread(target=lambda: self.run_interactive_console(_locals))
self.consoleThread.start()
def run_interactive_console(self, _locals):
try:
code.interact(local=_locals)
except SystemExit:
if not self.destroyed:
self.after(0, self.exit_callback)
def destroy(self):
self.stdin_buffer.put("\n\nexit()\n")
self.destroyed = True
sys.stdin, sys.stdout, sys.stderr = self.real_std_in_out
super().destroy()
def enter(self, event):
input_line = self.ttyText.get("input_start", "end")
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.mark_gravity("input_start", "left")
self.stdin_buffer.put(input_line)
def write(self, string):
self.ttyText.insert('end', string)
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.see('end')
def createWidgets(self):
self.ttyText = tk.Text(self.parent, wrap='word')
self.ttyText.grid(row=0, column=0, sticky=tk.N + tk.S + tk.E + tk.W)
self.ttyText.bind("<Return>", self.enter)
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.mark_gravity("input_start", "left")
def flush(self):
pass
def readline(self):
line = self.stdin_buffer.get()
return line
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root, locals(), root.destroy)
main_window.mainloop()
This code has few changes other than those that solve the problems stated in the question.
The advantage of this code over my previous answer is that it works inside a single process, so can be created at any point in the application, giving the programmer more control.
I have also written a more complete version of this which also prevents the user from editing text which shouldn't be editable (e.g. the output of a print statement) and has some basic coloring: https://gist.github.com/olisolomons/e90d53191d162d48ac534bf7c02a50cd