Develop Reference

multiprocessing with tkinter won't spawn multiple GUI's

I have functioning code that displays data in a GUI which is periodically updated with new information downloaded from the web. (The base code for the threaded approach was sourced from https://www.oreilly.com/library/view/python-cookbook/0596001673/ch09s07.html) I am using a threaded solution so as to improve blocking IO issues (IO code not included in the simplified code example below, as the IO does not appear to be the problem). The code runs fine if I run it as a single instance. However, it would be most convenient if I could use multiprocessing to run several instances of the code in parallel, using a different input list for each instance. When I try to implement the multiprocessing version, each separate process hangs during the attempt to create the root window: "window = tk.Tk()". Here is the working single instance version:
import threading
import random
import tkinter as tk
import random
import queue #Queue
import multiprocessing
import psutil
class GuiPartBase:
def __init__(self, master, queue, myList, endCommand):
self.queue = queue
# Set up the GUI
a = Label(master, text="Test Tkinter Display!")
a.pack()
## etc
def processIncoming(self):
"""Handle all messages currently in the queue, if any."""
while self.queue.qsize():
try:
result = (self.queue.get(0))
## do stuff with incoming data...
print('result =', result)
except queue.Empty:
# just on general principles...
pass
class ThreadedClientBase:
"""
Launch the main part of the GUI and the worker thread. periodicCall and
endApplication could reside in the GUI part, but putting them here
means that you have all the thread controls in a single place.
"""
def __init__(self, master, mylist):
"""
Start the GUI and the asynchronous threads. We are in the main
(original) thread of the application, which will later be used by
the GUI as well. We spawn a new thread for the worker (I/O).
"""
self.master = master
self.mylist = mylist
# Create the queue
self.queue = queue.Queue()
# Set up the GUI part
self.gui = GuiPartBase(self.master, self.queue, mylist, self.endApplication)
# Set up the thread to do asynchronous I/O
# More threads can also be created and used, if necessary
self.running = 1
self.thread1 = threading.Thread(target=self.workerThread1)
self.thread1.start()
# Start the periodic call in the GUI to check if the queue contains
# anything
self.periodicCall()
def periodicCall(self):
"""
Check every 200 ms if there is something new in the queue.
"""
self.gui.processIncoming()
if not self.running:
# This is the brutal stop of the system. You may want to do
# some cleanup before actually shutting it down.
import sys
sys.exit(1)
self.master.after(200, self.periodicCall)
def workerThread1(self):
"""
This is where we handle the asynchronous I/O. For example, it may be
a 'select( )'. One important thing to remember is that the thread has
to yield control pretty regularly, by select or otherwise.
"""
while self.running:
# simulate asynchronous I/O,
time.sleep(rand.random() * 1.5)
msg = rand.random()
self.queue.put(msg)
def endApplication(self):
self.running = 0
def runGUIthread(threadedList2Get):
print('entering runGUIthread...')
print('list2Get = ', threadedList2Get)
window = tk.Tk()
print('type of window = ', type(window))
print('window = ', window)
client = ThreadedClientBase(window, threadedList2Get)
print('type of client = ', type(client))
print('client = ', client)
window.mainloop()
if __name__ == '__main__':
rand = random.Random()
testList2a = ['abc','def','geh']
testList2b = ['xyz', 'lmn', 'opq']
allLists = [testList2a,testList2b]
runGUIthread(testList2a)
So, like I said, the above works - a single tkinter GUI is displayed appropriately without errors. However, if I attempt to implement multiprocessing with the following code below, the code spawns two processes as expected, and as documented by the printout of pid. However, each process prints the 'list2Get' (in runGUIthread), and then there is nothing else. There is no error message and the python code seems to have exited as there is no persistent process listed in the system activity monitor. Presumably the code is "hanging"/ exiting at the line "window = tk.TK()", as the line "print('type of window=',type(window))" is never executed:
if __name__ == '__main__':
rand = random.Random()
testList2a = ['abc','def','geh']
testList2b = ['xyz', 'lmn', 'opq']
allLists = [testList2a,testList2b]
#runGUIthread(testList2a)
for list in allLists:
p = multiprocessing.Process(target=runGUIthread, args=(list,))
p.start()
ps = psutil.Process(p.pid)
print('pid = ', ps)
#with multiprocessing.Pool(processes=2) as pool:
# pool.map(runGUIthread, allLists)
I am not experienced with multiprocessing, so perhaps I have implemented it incorrectly. I tried using multiprocessing.Pool(), with the same results.
I have not been able to find info indicating that tkinter can't spawn multiple GUI displays in the same program. In fact I found an instance of somebody accidentally spawning multiple GUI's, although this appears to be with Python 3.8 using concurrent.futures.ProcessPoolExecutor (Concurrent.futures opens new windows in tkinter instead of running the function). I am currently on Python 3.7, and was hoping not to have to reinstall a new enviroment to make this multiprocessing code work, although perhaps that is necessary...?
Other info: using python 3.7.6, tkinter 8.6.8, Eclipse 4.11.0, macOS10.13.6.
Any help appreciated.

You cannot use tkinter code across multiple processes. At least, you can't run the same tkinter code. It simply isn't designed to be used that way. When you create a root window, underneath the covers a tcl interpreter is created, and this interpreter can't be pickled or shared between processes, and doesn't use python's global interpreter lock.
In short, all of your GUI code needs to be in a single thread in a single process.
The following answer is a slightly better explanation, written by one of the developers on the Tcl core team: https://stackoverflow.com/a/38767665/7432. Here is the opening paragraph of that answer:
Each Tcl interpreter object (i.e., the context that knows how to run a Tcl procedure) can only be safely used from the OS thread that creates it. This is because Tcl doesn't use a global interpreter lock like Python, and instead makes extensive use of thread-specific data to reduce the number of locks required internally. (Well-written Tcl code can take advantage of this to scale up very large on suitable hardware.)

I found that this has been reported as part of a bug related to tkinter, python, and macOSX: https://bugs.python.org/issue33111.
(The bug was reported for python 2.7 and 3.6.4 and OSX 10.11.6, but apparently is still a problem with python 3.7.6 and OSX 10.13.6.)
However, there is a partial workaround (also reported at the same site), which for my case seems to work perfectly:
import multiprocessing
multiprocessing.set_start_method("spawn", force=True)
...
... other code same as initial ...
...
if __name__ == '__main__':
testList2a = ['abc','def','geh']
testList2b = ['xyz', 'lmn', 'opq']
allLists = [testList2a,testList2b]
with multiprocessing.Pool(processes=2) as pool:
pool.map(runGUIthread, allLists)
The result is the spawning of multiple GUIs, one for each process.
Per the bug report, in python 3.8, the default start method for multiprocessing when run on MacOS has been changed and is now "spawn" instead of "fork", so the problem will not reveal itself (unless you change the start method to be "fork", in which case the code will fail).

Python loading animation in a thread giving segmentation fault

Here's the code I have created to load animations in background using pyglet.resource.animation() function, while the app does some other things.
import pyglet
import threading
animations = dict()
class LoadingThread(object):
def __init__(self):
thread = threading.Thread(target=self.run, args=())
thread.start() # Start the execution
def run(self):
""" Method that runs forever """
loadAnimations()
print("Loaded all animations.")
def loadAnimations():
global animations
print("In loadAnimations")
for animation in os.listdir(os.getcwd()):
if animation.endswith(".gif"):
print(animation)
#Gives segmentation fault here
animations[animation] = pyglet.resource.animation(animation)
print("Loaded animations")
thread = LoadingThread()
Runs well when called normally without a thread.
If there is any other way to deal with loading animations in background in pyglet, please suggest.
Thanks.

As suggested by #Frank Merrow. The problem here was that I was using the function
pyglet.resource.animation("filename.gif") function in my main thread too. So it was creating a segmentation fault.
I found another function that can also load animation.
pyglet.image.load_animation("filename.gif")
Using this solved my problem.
Also this problem can be solved by starting two threads synchronously running main flow and the background work.

Avoiding PyQt5 GUI freeze during threaded operation?

I am having some trouble with a GUI that is freezing during a file save operation that is taking some time, and I'd love to understand why that is.
I've followed the instructions of Schollii's wonderful answer on a similar question, but there must be something I'm missing because I cannot get the GUI behaving as I expect.
The below example is not runnable, since it shows only the relevant parts, but hopefully it's enough to get a discussion going. Basically I have a main application class that generates some large data, and I need to save it to HDF5 format, but this takes some time. To leave the GUI responsive, the main class creates an object of the Saver class and a QThread to do the actual data saving (using moveToThread).
The output of this code is pretty much what I would expect (i.e. I see a message that the "saving thread" has a different thread id than the "main" thread) so I know that another thread is being created. The data is successfully saved, too, so that part is working correctly.
During the actual data saving however (which can take some minutes), the GUI freezes up and goes "Not responding" on Windows. Any clues as to what is going wrong?
Stdout during running:
outer thread "main" (#15108)
<__main__.Saver object at 0x0000027BEEFF3678> running SaveThread
Saving data from thread "saving_thread" (#13624)
Code sample:
from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtCore import QThread, pyqtSignal, pyqtSlot, QObject
class MyApp(QtWidgets.QMainWindow, MyAppDesign.Ui_MainWindow):
def save_file(self):
self.save_name, _ = QtWidgets.\
QFileDialog.getSaveFileName(self)
QThread.currentThread().setObjectName('main')
outer_thread_name = QThread.currentThread().objectName()
outer_thread_id = int(QThread.currentThreadId())
# print debug info about main app thread:
print('outer thread "{}" (#{})'.format(outer_thread_name,
outer_thread_id))
# Create worker and thread to save the data
self.saver = Saver(self.data,
self.save_name,
self.compressionSlider.value())
self.save_thread = QThread()
self.save_thread.setObjectName('saving_thread')
self.saver.moveToThread(self.save_thread)
# Connect signals
self.saver.sig_done.connect(self.on_saver_done)
self.saver.sig_msg.connect(print)
self.save_thread.started.connect(self.saver.save_data)
self.save_thread.start())
#pyqtSlot(str)
def on_saver_done(self, filename):
print('Finished saving {}'.format(filename))
''' End Class '''
class Saver(QObject):
sig_done = pyqtSignal(str) # worker id: emitted at end of work()
sig_msg = pyqtSignal(str) # message to be shown to user
def __init__(self, data_to_save, filename, compression_level):
super().__init__()
self.data = data_to_save
self.filename = filename
self.compression_level = compression_level
#pyqtSlot()
def save_data(self):
thread_name = QThread.currentThread().objectName()
thread_id = int(QThread.currentThreadId())
self.sig_msg.emit('Saving data '
'from thread "{}" (#{})'.format(thread_name,
thread_id))
print(self, "running SaveThread")
h5f = h5py.File(self.filename, 'w')
h5f.create_dataset('data',
data=self.data,
compression='gzip',
compression_opts=self.compression_level)
h5f.close()
self.sig_done.emit(self.filename)
''' End Class '''

There are actually two issues here: (1) Qt's signals and slots mechanisms, and (2) h5py.
First, the signals/slots. These actually work by copying arguments passed to the signal, to avoid any race conditions. (This is just one of the reasons you see so many signals with pointer arguments in the Qt C++ code: copying a pointer is cheap.) Because you're generating the data in the main thread, it must be copied in the main thread's event loop. The data is obviously big enough for this to take some time, blocking the event loop from handling GUI events. If you instead (for testing purposes) generate the data inside the Saver.save_data() slot, the GUI remains responsive.
However, you'll now notice a small lag after the first "Saving data from thread..." message is printed, indicating that the main event loop is blocked during the actual save. This is where h5py comes in.
You're presumably importing h5py at the top of your file, which is the "correct" thing to do. I noticed that if you instead import h5py directly before you create the file, this goes away. My best guess is that the global interpreter lock is involved, as the h5py code is visible from both the main and saving threads. I would have expected that the main thread would be entirely inside Qt module code at this point, however, over which the GIL has no control. So, like I said, I'm not sure what causes the blocking here.
As far as solutions, to the extent you can do what I described here, that will alleviate the problem. Generating the data outside the main thread, if possible, is advisable. It may also be possible to pass some memoryview object, or a numpy.view object, to the saving thread, though you'll then have to deal with thread-synchronization yourself. Also, importing h5py inside the Saver.save_data() slot will help, but isn't feasible if you need the module elsewhere in the code.
Hope this helps!

Multiple python loops in same process

I have a project that I'm writing in Python that will be sending hardware (Phidgets) commands. Because I'll be interfacing with more than one hardware component, I need to have more than one loop running concurrently.
I've researched the Python multiprocessing module, but it turns out that the hardware can only be controlled by one process at a time, so all my loops need to run in the same process.
As of right now, I've been able to accomplish my task with a Tk() loop, but without actually using any of the GUI tools. For example:
from Tk import tk
class hardwareCommand:
def __init__(self):
# Define Tk object
self.root = tk()
# open the hardware, set up self. variables, call the other functions
self.hardwareLoop()
self.UDPListenLoop()
self.eventListenLoop()
# start the Tk loop
self.root.mainloop()
def hardwareLoop(self):
# Timed processing with the hardware
setHardwareState(self.state)
self.root.after(100,self.hardwareLoop)
def UDPListenLoop(self):
# Listen for commands from UDP, call appropriate functions
self.state = updateState(self.state)
self.root.after(2000,self.UDPListenLoop)
def eventListenLoop(self,event):
if event == importantEvent:
self.state = updateState(self.event.state)
self.root.after(2000,self.eventListenLoop)
hardwareCommand()
So basically, the only reason for defining the Tk() loop is so that I can call the root.after() command within those functions that need to be concurrently looped.
This works, but is there a better / more pythonic way of doing it? I'm also wondering if this method causes unnecessary computational overhead (I'm not a computer science guy).
Thanks!

The multiprocessing module is geared towards having multiple separate processes. Although you can use Tk's event loop, that is unnecessary if you don't have a Tk based GUI, so if you just want multiple tasks to execute in the same process you can use the Thread module. With it you can create specific classes which encapsulate a separate thread of execution, so you can have many "loops" executing simultaneously in the background. Think of something like this:
from threading import Thread
class hardwareTasks(Thread):
def hardwareSpecificFunction(self):
"""
Example hardware specific task
"""
#do something useful
return
def run(self):
"""
Loop running hardware tasks
"""
while True:
#do something
hardwareSpecificTask()
class eventListen(Thread):
def eventHandlingSpecificFunction(self):
"""
Example event handling specific task
"""
#do something useful
return
def run(self):
"""
Loop treating events
"""
while True:
#do something
eventHandlingSpecificFunction()
if __name__ == '__main__':
# Instantiate specific classes
hw_tasks = hardwareTasks()
event_tasks = eventListen()
# This will start each specific loop in the background (the 'run' method)
hw_tasks.start()
event_tasks.start()
while True:
#do something (main loop)
You should check this article to get more familiar with the threading module. Its documentation is a good read too, so you can explore its full potential.

python console intrupt? and cross platform threads

I want my app to loop in python but have a way to quit. Is there a way to get input from the console, scan it for letter q and quick when my app is ready to quit? in C i would just create a pthread that waits for cin, scans, locks a global quit var, change, unlock and exit the thread allowing my app to quit when its done dumping a file or w/e it is doing. DO i do this the same way in python and will it be cross platform? (i see a global single instance in python that was windows specific)

use the threading module to make a thread class.
import threading;
class foo(threading.Thread):
def __init__(self):
#initialize anything
def run(self):
while True:
str = raw_input("input something");
class bar:
def __init__(self)
self.thread = foo(); #initialize the thread (foo) class and store
self.thread.start(); #this command will start the loop in the new thread (the run method)
if(quit):
#quit

Creating a new thread is easy enough – the threading module will help you out. You may want to make it daemonic (if you have other ways of exiting your program). I think you can change a variable without locking, too – python implements its own threads, and I'm fairly sure something like self.running = False will be atomic.
The simplest way to kick off a new thread is with threading.Thread(target=):
# inside your class definition
def signal_done(self):
self.done = True
def watcher(self):
while True:
if q_typed_in_console():
self.signal_done()
return
def start_watcher(self):
t = threading.Thread(target=self.watcher)
t.setDaemon(True) # Optional; means thread will exit when main thread does
t.start()
def main(self):
while not self.done:
# etc.
If you want your thread to be smarter, have its own state, etc. you can subclass threading.Thread yourself. The docs have more.
[related to this: the python executable itself is single-threaded, even if you have multiple python threads]