I have functioning code that displays data in a GUI which is periodically updated with new information downloaded from the web. (The base code for the threaded approach was sourced from https://www.oreilly.com/library/view/python-cookbook/0596001673/ch09s07.html) I am using a threaded solution so as to improve blocking IO issues (IO code not included in the simplified code example below, as the IO does not appear to be the problem). The code runs fine if I run it as a single instance. However, it would be most convenient if I could use multiprocessing to run several instances of the code in parallel, using a different input list for each instance. When I try to implement the multiprocessing version, each separate process hangs during the attempt to create the root window: "window = tk.Tk()". Here is the working single instance version:
import threading
import random
import tkinter as tk
import random
import queue #Queue
import multiprocessing
import psutil
class GuiPartBase:
def __init__(self, master, queue, myList, endCommand):
self.queue = queue
# Set up the GUI
a = Label(master, text="Test Tkinter Display!")
a.pack()
## etc
def processIncoming(self):
"""Handle all messages currently in the queue, if any."""
while self.queue.qsize():
try:
result = (self.queue.get(0))
## do stuff with incoming data...
print('result =', result)
except queue.Empty:
# just on general principles...
pass
class ThreadedClientBase:
"""
Launch the main part of the GUI and the worker thread. periodicCall and
endApplication could reside in the GUI part, but putting them here
means that you have all the thread controls in a single place.
"""
def __init__(self, master, mylist):
"""
Start the GUI and the asynchronous threads. We are in the main
(original) thread of the application, which will later be used by
the GUI as well. We spawn a new thread for the worker (I/O).
"""
self.master = master
self.mylist = mylist
# Create the queue
self.queue = queue.Queue()
# Set up the GUI part
self.gui = GuiPartBase(self.master, self.queue, mylist, self.endApplication)
# Set up the thread to do asynchronous I/O
# More threads can also be created and used, if necessary
self.running = 1
self.thread1 = threading.Thread(target=self.workerThread1)
self.thread1.start()
# Start the periodic call in the GUI to check if the queue contains
# anything
self.periodicCall()
def periodicCall(self):
"""
Check every 200 ms if there is something new in the queue.
"""
self.gui.processIncoming()
if not self.running:
# This is the brutal stop of the system. You may want to do
# some cleanup before actually shutting it down.
import sys
sys.exit(1)
self.master.after(200, self.periodicCall)
def workerThread1(self):
"""
This is where we handle the asynchronous I/O. For example, it may be
a 'select( )'. One important thing to remember is that the thread has
to yield control pretty regularly, by select or otherwise.
"""
while self.running:
# simulate asynchronous I/O,
time.sleep(rand.random() * 1.5)
msg = rand.random()
self.queue.put(msg)
def endApplication(self):
self.running = 0
def runGUIthread(threadedList2Get):
print('entering runGUIthread...')
print('list2Get = ', threadedList2Get)
window = tk.Tk()
print('type of window = ', type(window))
print('window = ', window)
client = ThreadedClientBase(window, threadedList2Get)
print('type of client = ', type(client))
print('client = ', client)
window.mainloop()
if __name__ == '__main__':
rand = random.Random()
testList2a = ['abc','def','geh']
testList2b = ['xyz', 'lmn', 'opq']
allLists = [testList2a,testList2b]
runGUIthread(testList2a)
So, like I said, the above works - a single tkinter GUI is displayed appropriately without errors. However, if I attempt to implement multiprocessing with the following code below, the code spawns two processes as expected, and as documented by the printout of pid. However, each process prints the 'list2Get' (in runGUIthread), and then there is nothing else. There is no error message and the python code seems to have exited as there is no persistent process listed in the system activity monitor. Presumably the code is "hanging"/ exiting at the line "window = tk.TK()", as the line "print('type of window=',type(window))" is never executed:
if __name__ == '__main__':
rand = random.Random()
testList2a = ['abc','def','geh']
testList2b = ['xyz', 'lmn', 'opq']
allLists = [testList2a,testList2b]
#runGUIthread(testList2a)
for list in allLists:
p = multiprocessing.Process(target=runGUIthread, args=(list,))
p.start()
ps = psutil.Process(p.pid)
print('pid = ', ps)
#with multiprocessing.Pool(processes=2) as pool:
# pool.map(runGUIthread, allLists)
I am not experienced with multiprocessing, so perhaps I have implemented it incorrectly. I tried using multiprocessing.Pool(), with the same results.
I have not been able to find info indicating that tkinter can't spawn multiple GUI displays in the same program. In fact I found an instance of somebody accidentally spawning multiple GUI's, although this appears to be with Python 3.8 using concurrent.futures.ProcessPoolExecutor (Concurrent.futures opens new windows in tkinter instead of running the function). I am currently on Python 3.7, and was hoping not to have to reinstall a new enviroment to make this multiprocessing code work, although perhaps that is necessary...?
Other info: using python 3.7.6, tkinter 8.6.8, Eclipse 4.11.0, macOS10.13.6.
Any help appreciated.
You cannot use tkinter code across multiple processes. At least, you can't run the same tkinter code. It simply isn't designed to be used that way. When you create a root window, underneath the covers a tcl interpreter is created, and this interpreter can't be pickled or shared between processes, and doesn't use python's global interpreter lock.
In short, all of your GUI code needs to be in a single thread in a single process.
The following answer is a slightly better explanation, written by one of the developers on the Tcl core team: https://stackoverflow.com/a/38767665/7432. Here is the opening paragraph of that answer:
Each Tcl interpreter object (i.e., the context that knows how to run a Tcl procedure) can only be safely used from the OS thread that creates it. This is because Tcl doesn't use a global interpreter lock like Python, and instead makes extensive use of thread-specific data to reduce the number of locks required internally. (Well-written Tcl code can take advantage of this to scale up very large on suitable hardware.)
I found that this has been reported as part of a bug related to tkinter, python, and macOSX: https://bugs.python.org/issue33111.
(The bug was reported for python 2.7 and 3.6.4 and OSX 10.11.6, but apparently is still a problem with python 3.7.6 and OSX 10.13.6.)
However, there is a partial workaround (also reported at the same site), which for my case seems to work perfectly:
import multiprocessing
multiprocessing.set_start_method("spawn", force=True)
...
... other code same as initial ...
...
if __name__ == '__main__':
testList2a = ['abc','def','geh']
testList2b = ['xyz', 'lmn', 'opq']
allLists = [testList2a,testList2b]
with multiprocessing.Pool(processes=2) as pool:
pool.map(runGUIthread, allLists)
The result is the spawning of multiple GUIs, one for each process.
Per the bug report, in python 3.8, the default start method for multiprocessing when run on MacOS has been changed and is now "spawn" instead of "fork", so the problem will not reveal itself (unless you change the start method to be "fork", in which case the code will fail).
Related
I have two python programs which one of them connects to a bluetooth device(socket package), it receives and saves data from device, and another one read the stored data and draw a real time plot. I should make one application from these two programs.
I tried to mix these two python programs, but since bluetooth should wait to receive data (through a while loop), the other parts of program does not work. I tried to solve this problem using Clock.schedule_interval, but the program will hang after a period of time. So I decided to run these two programs simultaneously. I read, we can run some python programs at a same time using a python script. Is there any trick to join these two programs and build one application?
Any help would be greatly appreciated.
Install threaded:
pip install threaded
Create a new python file:
from threading import Thread
def runFile1(): import file1
def runFile2(): import file2
Thread(target=runFile1).start()
runFile2()
Run the new python file.
It can be done with threading. To do communication between the threaded function and your main function, use objects such as queue.Queue and threading.Event.
the bluetooth functions can be placed into a function that is the target of the thread
import time
from threading import Thread
from queue import Queue
class BlueToothFunctions(Thread):
def __init__(self, my_queue):
super().__init__()
self.my_queue = my_queue
# optional: causes this thread to end immediately if the main program is terminated
self.daemon = True
def run(self) -> None:
while True:
# do all the bluetooth stuff foreverer
g = self.my_queue.get()
if g == (None, None):
break
print(g)
time.sleep(1.0)
print("bluetooth closed")
if __name__ == '__main__':
_queue = Queue() # just one way to communicate to a thread
# pass an object reference so both main and thread have a way to communicate on this common queue
my_bluetooth = BlueToothFunctions(_queue)
my_bluetooth.start() # creates the thread and executes run() method
for i in range(5):
# communicate to the threaded functions
_queue.put(i)
_queue.put((None, None)) # optional, a way to cause the thread to end
my_bluetooth.join(timeout=5.0) # optional, pause here until thread ends
print('program complete')
I'm trying to create a chess game using tkinter. I don't have a huge experience in python programming, but I kind of find weird the philosophy of tkinter : if my assumptions are correct, it seems to me that using tkinter means setting it as the base of the project, and everything has to work around it. And what I mean by that is that using whatever code that is not 'wrapped' in the tkinter framework is a pain to deal with (you have to use the event system, you have to use the after method if you want to perform an action after starting the main loop, etc.)
I have a rather different view on that, and in my chess project I simply consider the tkinter display as a part of my rendering system, and the event system provided by tkinter as a part of my input parser system. That being said, I want to be able to easily change the renderer or the input parser, which means that I could want to detect input from the terminal (for instance by writing D2 D3) instead of moving the objects on the screen. I could also want to print the chessboard on the terminal instead of having a GUI.
More to the point, because tkinter blocks the thread through the mainloop method instead of looping in another thread, I have to put my Tk object in a different thread, so that I can run the rest of my program in parallel. And I'm having a tough time doing it, because my Tk variable contained by my thread needs to be accessed by my program, to update it for instance.
After quite a bit of research, I found that queues in python were synchronized, which means that if I put my Tk object in a queue, I could access it without any problem from the main thread. I tried to see if the following code was working :
import threading, queue
class VariableContainer(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.queue = queue.Queue()
def run(self):
self.queue.put("test")
container = VariableContainer()
container.start()
print(container.queue.get(False))
and it does ! The output is test.
However, if I replace my test string by a Tk object, like below :
import threading, queue
import tkinter
class VariableContainer(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.queue = queue.Queue()
def run(self):
root = tkinter.Tk()
self.queue.put(root)
root.mainloop() # whether I call the mainloop or not doesn't change anything
container = VariableContainer()
container.start()
print(container.queue.get(False))
then the print throws an error, stating that the queue is empty.
(Note that the code above is not the code of my program, it is just an exemple since posting sample codes from my project might be less clear)
Why?
The answer to the trivial question you actually asked: you have a race condition because you call Queue.get(block=False). Tk taking a lot longer to initialize, the main thread almost always wins and finds the queue still empty.
The real question is “How do I isolate my logic from the structure of my interface library?”. (While I understand the desire for a simple branch point between “read from the keyboard” and “wait for a mouse event”, it is considered more composable, in the face of large numbers of event types, sources, and handlers, to have one event dispatcher provided by the implementation. It can sometimes be driven one event at a time, but that composes less well with other logic than one might think.)
The usual answer to that is to make your logic a state machine rather than an algorithm. Mechanically, this means replacing local variables with attributes on an object and dividing the code into methods on its class (e.g., one call per “read from keyboard” in a monolithic implementation). Sometimes language features like coroutines can be used to make this transformation mostly transparent/automatic, but they’re not always a good fit. For example:
def algorithm(n):
tot=0
for i in range(n):
s=int(input("#%s:"%i))
tot+=(i+1)*(n-i)*s
print(tot)
class FSM(object):
def __init__(self,n):
self.n=n
self.i=self.tot=0
def send(self,s):
self.tot+=(self.i+1)*(self.n-self.i)*s
self.i+=1
def count(self): return self.i
def done(self): return self.i>=self.n
def get(self):
return self.tot
def coroutine(n): # old-style, not "async def"
tot=0
for i in range(n):
s=(yield)
tot+=(i+1)*(n-i)*s
yield tot
Having done this, it’s trivial to layer the traditional stream-driven I/O back on top, or to connect it to an event-driven system (be it a GUI or asyncio). For example:
def fsmClient(n):
fsm=FSM(n)
while not fsm.done():
fsm.send(int(input("#%s:"%fsm.count())))
return fsm.get()
def coClient(n):
co=coroutine(n)
first=True
while True:
ret=co.send(None if first else
int(input("#%s:"%fsm.count())))
if ret is not None:
co.close()
return ret
first=False
These clients can work with any state machine/coroutine using the same interface; it should be obvious how to instead supply values from a Tkinter input box or so.
The context for this is much, much too big for an SO question so the code below is a extremely simplified demonstration of the actual implementation.
Generally, I've written an extensive module for academic contexts that launches a subprocess at runtime to be used for event scheduling. When a script or program using this module closes on pre-El Capitan machines my efforts to join the child process fail, as do my last-ditch efforts to just kill the process; OS X gives a "Python unexpectedly quit" error and the the orphaned process persists. I am very much a nub to multiprocessing, without a CS background; diagnosing this is beyond me.
If I am just too ignorant, I'm more than willing to go RTFM; specific directions welcome.
I'm pretty sure this example is coherent & representative, but, know that the actual project works flawlessly on El Capitan, works during runtime on everything else, but consistently crashes as described when quitting. I've tested it with absurd time-out values (30 sec+); always the same result.
One last note: I started this with python's default multiprocessing libraries, then switched to billiard as a dev friend suggested it might run smoother. To date, I've not experienced any difference.
UPDATE:
Had omitted the function that gives the #threaded decorator purpose; now present in code.
Generally, we have:
shared_queue = billiard.Queue() # or multiprocessing, have used both
class MainInstanceParent(object):
def __init__(self):
# ..typically init stuff..
self.event_ob = EventClass(self) # gets a reference to parent
def quit():
try:
self.event_ob.send("kkbai")
started = time.time()
while time.time - started < 1: # or whatever
self.event_ob.recieve()
if self.event_ob.event_p.is_alive():
raise RuntimeError("Little bugger still kickin'")
except RuntimeError:
os.kill(self.event_on.event_p.pid, SIGKILL)
class EventClass(object):
def __init__(self, parent):
# moar init stuff
self.parent = parent
self.pipe, child = Pipe()
self.event_p = __event_process(child)
def receive():
self.pipe.poll()
t = self.pipe.recv()
if isinstance(t, Exception):
raise t
return t
def send(deets):
self.pipe.send(deets)
def threaded(func):
def threaded_func(*args, **kwargs):
p = billiard.Process(target=func, args=args, kwargs=kwargs)
p.start()
return p
return threaded_func
#threaded
def __event_process(pipe):
while True:
if pipe.poll():
inc = pipe.recv()
# do stuff conditionally on what comes through
if inc == "kkbai":
return
if inc == "meets complex condition to pass here":
shared_queue.put("stuff inferred from inc")
Before exiting the main program, call multiprocessing.active_children() to see how many child processes are still running. This will also join the processes that have already quit.
If you would need to signal the children that it's time to quit, create a multiprocessing.Event before starting the child processes. Give it a meaningful name like children_exit. The child processes should regularly call children_exit.is_set() to see if it is time for them to quit. In the main program you call children_exit.set() to signal the child processes.
Update:
Have a good look through the Programming guidelines in the multiprocessing documentation;
It is best to provide the abovementioned Event objects as argument to the target of the Process initializer for reasons mentioned in those guidelines.
If your code also needs to run on ms-windows, you have to jump through some extra hoop, since that OS doesn't do fork().
Update 2:
On your PyEval_SaveThread error; could you modify your question to show the complete trace or alternatively could you post it somewhere?
Since multiprocessing uses threads internally, this is probably the culprit, unless you are also using threads somewhere.
If you also use threads note that GUI toolkits in general and tkinter in particular are not thread safe. Tkinter calls should therefore only be made from one thread!
How much work would it be to port your code to Python 3? If it is a bug in Python 2.7, it might be already fixed in the current (as of now) Python 3.5.1.
I have a Tkinter wrapper written over robocopy.exe, the code is organized as shown below :-
Tkinter Wrapper :
Spawns a new thread and pass the arguments, which includes source/destination and other parameters
(Note : Queue object is also passed to thread, since the thread will read the output from robocopy and will put in queue, main tkinter thread will keep on polling queue and will update Tkinter text widget with the output)
Code Snippet
... Code to poll queue and update tk widget ...
q = Queue.Queue()
t1 = threading.Thread(target=CopyFiles,args=(q,src,dst,), kwargs={"ignore":ignore_list})
t1.daemon = True
t1.start()
Thread : (In a separate file)
Below is the code snippet from thread
def CopyFiles(q,src,dst,ignore=None):
extra_args = ['/MT:15', '/E', '/LOG:./log.txt', '/tee', '/r:2', '/w:2']
if len(ignore) > 0:
extra_args.append('/xf')
extra_args.extend(ignore)
extra_args.append('/xd')
extra_args.extend(ignore)
command_to_pass = ["robocopy",src, dst]
command_to_pass.extend(extra_args)
proc = subprocess.Popen(command_to_pass,stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if line == '':
break
q.put(line.strip())
Code which is called, when tkinter application is closed :-
def onQuit(self):
global t1
if t1.isAlive():
pass
if tkMessageBox.askyesno("Title", "Do you really want to exit?"):
self.destroy()
self.master.destroy()
Problem
Whenever, I close the tkinter application when robocopy is running, python application closes but the robocopy.exe keeps on running.
I have tried setting the thread as daemon, but it has no effect. How can I stop robocopy.exe when onQuit method is called?
To simplify things let's ignore Tkinter and the fact that a separate thread is used.
The situation is that your app spawns a subprocess to execute an external program (robocopy.exe in this question), and you'd need to stop the spawned program from you're application on a certain event (when the Tkinter app is closing in this question).
This requires an inter process communication mechanism, so the spawned process would be notified of the event, and reacts accordingly. A common mechanism is to use signals provided by the OS.
You could send a signal (SIGTERM) to the external process and ask from it to quit. Assuming that the program reacts to signal as expected (most well written applications do) you'll get the desired behavior (the process will terminate).
Using the terminate method on the subprocess will send the proper signal of current platform to the subprocess.
You'd need a reference to the subprocess object proc in the onQuit function (from the provided code I see onQuit is a function and not an object method, so it could use a global variable to access proc), so you can call the terminate method of the process:
def onQuit(self):
global t1, proc
if t1.isAlive():
pass
# by the way I'm not sure about the logic, but I guess this
# below statement should be an elif instead of if
if tkMessageBox.askyesno("Title", "Do you really want to exit?"):
proc.terminate()
self.destroy()
self.master.destroy()
This code assumes you're storing the reference to the subprocess in the global scope, so you'd have to modify the CopyFiles as well.
I'm not sure how robocopy handles terminate signals and I'm guessing that's not something that we have any control over.
If you had more control on the external program (could modify the source), there might have been more options, for example sending messages using stdio, or using a shared memory, etc.
I have a project that I'm writing in Python that will be sending hardware (Phidgets) commands. Because I'll be interfacing with more than one hardware component, I need to have more than one loop running concurrently.
I've researched the Python multiprocessing module, but it turns out that the hardware can only be controlled by one process at a time, so all my loops need to run in the same process.
As of right now, I've been able to accomplish my task with a Tk() loop, but without actually using any of the GUI tools. For example:
from Tk import tk
class hardwareCommand:
def __init__(self):
# Define Tk object
self.root = tk()
# open the hardware, set up self. variables, call the other functions
self.hardwareLoop()
self.UDPListenLoop()
self.eventListenLoop()
# start the Tk loop
self.root.mainloop()
def hardwareLoop(self):
# Timed processing with the hardware
setHardwareState(self.state)
self.root.after(100,self.hardwareLoop)
def UDPListenLoop(self):
# Listen for commands from UDP, call appropriate functions
self.state = updateState(self.state)
self.root.after(2000,self.UDPListenLoop)
def eventListenLoop(self,event):
if event == importantEvent:
self.state = updateState(self.event.state)
self.root.after(2000,self.eventListenLoop)
hardwareCommand()
So basically, the only reason for defining the Tk() loop is so that I can call the root.after() command within those functions that need to be concurrently looped.
This works, but is there a better / more pythonic way of doing it? I'm also wondering if this method causes unnecessary computational overhead (I'm not a computer science guy).
Thanks!
The multiprocessing module is geared towards having multiple separate processes. Although you can use Tk's event loop, that is unnecessary if you don't have a Tk based GUI, so if you just want multiple tasks to execute in the same process you can use the Thread module. With it you can create specific classes which encapsulate a separate thread of execution, so you can have many "loops" executing simultaneously in the background. Think of something like this:
from threading import Thread
class hardwareTasks(Thread):
def hardwareSpecificFunction(self):
"""
Example hardware specific task
"""
#do something useful
return
def run(self):
"""
Loop running hardware tasks
"""
while True:
#do something
hardwareSpecificTask()
class eventListen(Thread):
def eventHandlingSpecificFunction(self):
"""
Example event handling specific task
"""
#do something useful
return
def run(self):
"""
Loop treating events
"""
while True:
#do something
eventHandlingSpecificFunction()
if __name__ == '__main__':
# Instantiate specific classes
hw_tasks = hardwareTasks()
event_tasks = eventListen()
# This will start each specific loop in the background (the 'run' method)
hw_tasks.start()
event_tasks.start()
while True:
#do something (main loop)
You should check this article to get more familiar with the threading module. Its documentation is a good read too, so you can explore its full potential.