Develop Reference

Python multiprocessing asynchronous callback

I'm writing a program with a GUI using TKinter, in which the user can click a button and a new process is started to perform work using multiprocess.Process. This is necessary so the GUI can still be used while the work is being done, which can take several seconds.
The GUI also has a text box where the status of the program is displayed when things happen. This is often straight forward, with each function calling an add_text() function which just prints text in the text box. However, when add_text() is called in the separate process, the text does not end up in the text box.
I've thought about using a Pipe or Queue, but that would require using some sort of loop to check if anything has been returned from the process and that would also cause the main (GUI) process to be unusable. Is there some way to call a function in one process that will do work in another?
Here's an simple example of what I'm trying to do
import time
import multiprocessing as mp
import tkinter as tk
textbox = tk.Text()
def add_text(text):
# Insert text into textbox
textbox.insert(tk.END, text)
def worker():
x = 0
while x < 10:
add_text('Sleeping for {0} seconds'.format(x)
x += 1
time.sleep(1)
proc = mp.Process(target=worker)
# Usually happens on a button click
proc.start()
# GUI should still be usable here

The asyncronous things actually require loop.
You could attach function to the TkInter's loop by using Tk.after() method.
import Tkinter as tk
class App():
def __init__(self):
self.root = tk.Tk()
self.check_processes()
self.root.mainloop()
def check_processes(self):
if process_finished:
do_something()
else:
do_something_else()
self.after(1000, check_processes)
app=App()

I ended up using a multiprocessing.Pipe by using TKinter's after() method to perform the looping. It loops on an interval and checks the pipe to see if there's any messages from the thread, and if so it inserts them into the text box.
import tkinter
import multiprocessing
def do_something(child_conn):
while True:
child_conn.send('Status text\n')
class Window:
def __init__(self):
self.root = tkinter.Tk()
self.textbox = tkinter.Text()
self.parent_conn, child_conn = multiprocessing.Pipe()
self.process = multiprocessing.Process(target=do_something, args=(child_conn,))
def start(self):
self.get_status_updates()
self.process.start()
self.root.mainloop()
def get_status_updates()
status = self.check_pipe()
if status:
self.textbox.add_text(status)
self.root.after(500, self.get_status_updates) # loop every 500ms
def check_pipe():
if self.parent_conn.poll():
status = self.parent_conn.recv()
return status
return None

Python/ Update value of variable threading and tkinter

I encounter a problem that slows me down a lot in my development ..
Indeed, I can not update the Queue with the Threading module of python!
I have searched several sites, and I could not find fault that could prevent my variable from updating.
My tkinter button should allow me to run another python script. To do this, I use Threading so I can use the GUI without interrupting it.
I explain my problem:
My tkinter button should allow me to run another python script. To do this, I use Threading so I can use the GUI without it being interrupted. Another button should allow me to update the Queue, and this is what the action does not do.
My main script with Tkinter:
import Tkinter, cv2
from Tkinter import *
from threading import Thread
import threading, Queue
import pyautogui, os, time, urllib2, urlparse
import cv2
from yes2 import *
def print1():
global kill, q
kill = []
q = Queue.Queue()
q.put("True")
thread = Thread(target = main, args=(kill, q))
thread.start()
def stop():
global q
q.put("False")
print q.get()
root = Tkinter.Tk()
root.title('breakable')
bouton= Button(root, text="Run", command=print1)
bouton.grid(row=3, column=0)
bouton= Button(root, text="stop", command=stop)
bouton.grid(row=3, column=1)
root.mainloop()
I want to open this other script:
def main(kill, q):
while True:
try:
get = q.get(timeout=2)
print get
except Empty as error:
print("Error too many times")
The value that comes out is "True", but when I click on my stop button, which is supposed to update my Queue in "False", well it does nothing
Thanks in advance :)

It does update your Queue, but on the very next line you get the value out again. Remove the print q.get() from the "stop" function.

Python win32gui SetAsForegroundWindow function not working properly

I am trying to write a program that finds a window by searching for its title. Once it has found the window, it will attempt to bring it to front. I am using win32gui API to achieve this. I am able to get it to work for the most part, but for some reason it does not work if the taskmanager is in front. I have the follow sample code.
import win32gui, win32con
import re, traceback
from time import sleep
class cWindow:
def __init__(self):
self._hwnd = None
def BringToTop(self):
win32gui.BringWindowToTop(self._hwnd)
def SetAsForegroundWindow(self):
win32gui.SetForegroundWindow(self._hwnd)
def Maximize(self):
win32gui.ShowWindow(self._hwnd, win32con.SW_MAXIMIZE)
def setActWin(self):
win32gui.SetActiveWindow(self._hwnd)
def _window_enum_callback(self, hwnd, wildcard):
'''Pass to win32gui.EnumWindows() to check all the opened windows'''
if re.match(wildcard, str(win32gui.GetWindowText(hwnd))) != None:
self._hwnd = hwnd
def find_window_wildcard(self, wildcard):
self._hwnd = None
win32gui.EnumWindows(self._window_enum_callback, wildcard)
def main():
sleep(5)
try:
wildcard = ".*Building Operation WorkStation.*"
cW = cWindow()
cW.find_window_wildcard(wildcard)
cW.Maximize()
cW.BringToTop()
cW.SetAsForegroundWindow()
except:
f = open("log.txt", "w")
f.write(traceback.format_exc())
print traceback.format_exc()
main()
I pieced this together from multiple online sources. It seems to work for the most part but for some windows like the task manager, it'll work sometimes but fails the rest. When it doesnt work properly, all I notice is the application icon blinks yellow. Is there a proper way of doing this to make sure the window that I am interested in is set to foreground 100% of the times? I am not sure if this is relevant but I am using Windows 7 Professional (32-bit) with Service Pack 1.

I found a solution: if taskmanager, then kill it. I added a method to cWindow:
def kill_task_manager(self):
# Here I use your method to find a window because of an accent in my french OS,
# but you should use win32gui.FindWindow(None, 'Task Manager complete name').
wildcard = 'Gestionnaire des t.+ches de Windows'
self.find_window_wildcard(wildcard)
if self._hwnd:
win32gui.PostMessage(self._hwnd, win32con.WM_CLOSE, 0, 0) # kill it
sleep(0.5) # important to let time for the window to be closed
Call this method just after cW = cWindow().
Another bug trap is to prevent this exception in SetAsForegroundWindow:
error: (0, 'SetForegroundWindow', 'No error message is available')
just send an alt key before the win32gui call:
# Add this import
import win32com.client
# Add this to __ini__
self.shell = win32com.client.Dispatch("WScript.Shell")
# And SetAsForegroundWindow becomes
def SetAsForegroundWindow(self):
self.shell.SendKeys('%')
win32gui.SetForegroundWindow(self._hwnd)
Last, if I may, do not compare != None but is not None. More pythonic ;)
This is the full code:
# coding: utf-8
import re, traceback
import win32gui, win32con, win32com.client
from time import sleep
class cWindow:
def __init__(self):
self._hwnd = None
self.shell = win32com.client.Dispatch("WScript.Shell")
def BringToTop(self):
win32gui.BringWindowToTop(self._hwnd)
def SetAsForegroundWindow(self):
self.shell.SendKeys('%')
win32gui.SetForegroundWindow(self._hwnd)
def Maximize(self):
win32gui.ShowWindow(self._hwnd, win32con.SW_MAXIMIZE)
def setActWin(self):
win32gui.SetActiveWindow(self._hwnd)
def _window_enum_callback(self, hwnd, wildcard):
'''Pass to win32gui.EnumWindows() to check all the opened windows'''
if re.match(wildcard, str(win32gui.GetWindowText(hwnd))) is not None:
self._hwnd = hwnd
def find_window_wildcard(self, wildcard):
self._hwnd = None
win32gui.EnumWindows(self._window_enum_callback, wildcard)
def kill_task_manager(self):
wildcard = 'Gestionnaire des t.+ches de Windows'
self.find_window_wildcard(wildcard)
if self._hwnd:
win32gui.PostMessage(self._hwnd, win32con.WM_CLOSE, 0, 0)
sleep(0.5)
def main():
sleep(5)
try:
wildcard = ".*Building Operation WorkStation.*"
cW = cWindow()
cW.kill_task_manager()
cW.find_window_wildcard(wildcard)
cW.BringToTop()
cW.Maximize()
cW.SetAsForegroundWindow()
except:
f = open("log.txt", "w")
f.write(traceback.format_exc())
print(traceback.format_exc())
if __name__ == '__main__':
main()
Sources: how do I close window with handle using win32gui in Python and win32gui.SetActiveWindow() ERROR : The specified procedure could not be found.

Note: The following deals only with making sure that always-on-top windows such as Task Manager are hidden before activating the window - it assumes that the activation part itself works fine, which may not be the case. The conditions under which a process is allowed to call the SetForegroundWindow Windows API function are listed here.
Task Manager is special in two respects:
By default, it is set to display always on top, i.e., above all other windows.
Even when that is turned off (Options > Always on Top unchecked), you can still make it display on top of other always-on-top windows (something that ordinary windows seemingly cannot do).
Your code:
is working - in my tests - in the sense that the target window does become the active window.
is not working in the sense that the Task Manager window still stays on top of the (maximized) window.
even trying to make your window an always-on-top window as well wouldn't help, unfortunately.
Specifically checking for the presence of the Task Manager window and minimizing it is an option, but note that there may be other always-on-top windows, so for a robust solution you have to identify all open always-on-top windows and minimize them:
The following tries hard to identify all always-on-top windows except the Task Bar and Start button, and minimizes (effectively hides) any such windows.
The new methods are hide_always_on_top_windows and _window_enum_callback_hide.
import win32gui, win32con
import re, traceback
from time import sleep
class cWindow:
def __init__(self):
self._hwnd = None
def SetAsForegroundWindow(self):
# First, make sure all (other) always-on-top windows are hidden.
self.hide_always_on_top_windows()
win32gui.SetForegroundWindow(self._hwnd)
def Maximize(self):
win32gui.ShowWindow(self._hwnd, win32con.SW_MAXIMIZE)
def _window_enum_callback(self, hwnd, regex):
'''Pass to win32gui.EnumWindows() to check all open windows'''
if self._hwnd is None and re.match(regex, str(win32gui.GetWindowText(hwnd))) is not None:
self._hwnd = hwnd
def find_window_regex(self, regex):
self._hwnd = None
win32gui.EnumWindows(self._window_enum_callback, regex)
def hide_always_on_top_windows(self):
win32gui.EnumWindows(self._window_enum_callback_hide, None)
def _window_enum_callback_hide(self, hwnd, unused):
if hwnd != self._hwnd: # ignore self
# Is the window visible and marked as an always-on-top (topmost) window?
if win32gui.IsWindowVisible(hwnd) and win32gui.GetWindowLong(hwnd, win32con.GWL_EXSTYLE) & win32con.WS_EX_TOPMOST:
# Ignore windows of class 'Button' (the Start button overlay) and
# 'Shell_TrayWnd' (the Task Bar).
className = win32gui.GetClassName(hwnd)
if not (className == 'Button' or className == 'Shell_TrayWnd'):
# Force-minimize the window.
# Fortunately, this seems to work even with windows that
# have no Minimize button.
# Note that if we tried to hide the window with SW_HIDE,
# it would disappear from the Task Bar as well.
win32gui.ShowWindow(hwnd, win32con.SW_FORCEMINIMIZE)
def main():
sleep(5)
try:
regex = ".*Building Operation WorkStation.*"
cW = cWindow()
cW.find_window_regex(regex)
cW.Maximize()
cW.SetAsForegroundWindow()
except:
f = open("log.txt", "w")
f.write(traceback.format_exc())
print(traceback.format_exc())
main()

How can I embed a python interpreter frame in python using tkinter?

I want to add a control terminal widget to my pure python+tkinter application similar to the python interpreter provided in Blender. It should be running within the same context (process) so the user can add features and control the application that is currently running from the control widget. Ideally I'd like it to also "hijack" stdout and stderr of the current application so it will report any problems or debugging information within the running application.
This is what I have come up with so far. The only problems are that it isn't responding to commands, and the thread doesn't stop when the user closes the window.
import Tkinter as tk
import sys
import code
from threading import *
class Console(tk.Frame):
def __init__(self,parent=None):
tk.Frame.__init__(self, parent)
self.parent = parent
sys.stdout = self
sys.stderr = self
self.createWidgets()
self.consoleThread = ConsoleThread()
self.after(100,self.consoleThread.start)
def write(self,string):
self.ttyText.insert('end', string)
self.ttyText.see('end')
def createWidgets(self):
self.ttyText = tk.Text(self.parent, wrap='word')
self.ttyText.grid(row=0,column=0,sticky=tk.N+tk.S+tk.E+tk.W)
class ConsoleThread(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
vars = globals().copy()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root)
main_window.mainloop()
try:
if root.winfo_exists():
root.destroy()
except:
pass

I have the answer in case anyone still cares! (I have also changed to python 3, hence the import tkinter rather than import Tkinter)
I have changed the approach slightly from the original by using a separate file to run the InteractiveConsole, and then making the main file open this other file (which I have called console.py and is in the same directory) in a subprocess, linking the stdout, stderr, and stdin of this subprocess to the tkinter Text widget programatically.
Here is the code in the for the console file (if this is run normally, it acts like a normal console):
# console.py
import code
if __name__ == '__main__':
vars = globals().copy()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()
And here is the code for the python interpreter, that runs the console inside the Text widget:
# main.py
import tkinter as tk
import subprocess
import queue
import os
from threading import Thread
class Console(tk.Frame):
def __init__(self,parent=None):
tk.Frame.__init__(self, parent)
self.parent = parent
self.createWidgets()
# get the path to the console.py file assuming it is in the same folder
consolePath = os.path.join(os.path.dirname(__file__),"console.py")
# open the console.py file (replace the path to python with the correct one for your system)
# e.g. it might be "C:\\Python35\\python"
self.p = subprocess.Popen(["python3",consolePath],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
# make queues for keeping stdout and stderr whilst it is transferred between threads
self.outQueue = queue.Queue()
self.errQueue = queue.Queue()
# keep track of where any line that is submitted starts
self.line_start = 0
# make the enter key call the self.enter function
self.ttyText.bind("<Return>",self.enter)
# a daemon to keep track of the threads so they can stop running
self.alive = True
# start the functions that get stdout and stderr in separate threads
Thread(target=self.readFromProccessOut).start()
Thread(target=self.readFromProccessErr).start()
# start the write loop in the main thread
self.writeLoop()
def destroy(self):
"This is the function that is automatically called when the widget is destroyed."
self.alive=False
# write exit() to the console in order to stop it running
self.p.stdin.write("exit()\n".encode())
self.p.stdin.flush()
# call the destroy methods to properly destroy widgets
self.ttyText.destroy()
tk.Frame.destroy(self)
def enter(self,e):
"The <Return> key press handler"
string = self.ttyText.get(1.0, tk.END)[self.line_start:]
self.line_start+=len(string)
self.p.stdin.write(string.encode())
self.p.stdin.flush()
def readFromProccessOut(self):
"To be executed in a separate thread to make read non-blocking"
while self.alive:
data = self.p.stdout.raw.read(1024).decode()
self.outQueue.put(data)
def readFromProccessErr(self):
"To be executed in a separate thread to make read non-blocking"
while self.alive:
data = self.p.stderr.raw.read(1024).decode()
self.errQueue.put(data)
def writeLoop(self):
"Used to write data from stdout and stderr to the Text widget"
# if there is anything to write from stdout or stderr, then write it
if not self.errQueue.empty():
self.write(self.errQueue.get())
if not self.outQueue.empty():
self.write(self.outQueue.get())
# run this method again after 10ms
if self.alive:
self.after(10,self.writeLoop)
def write(self,string):
self.ttyText.insert(tk.END, string)
self.ttyText.see(tk.END)
self.line_start+=len(string)
def createWidgets(self):
self.ttyText = tk.Text(self, wrap=tk.WORD)
self.ttyText.pack(fill=tk.BOTH,expand=True)
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root)
main_window.pack(fill=tk.BOTH,expand=True)
root.mainloop()
The reason that reading from stdout and stderr is in separate threads is because the read method is blocking, which causes the program to freeze until the console.py subprocess gives more output, unless these are in separate threads. The writeLoop method and the queues are needed to write to the Text widget since tkinter is not thread safe.
This certainly still has problems to be ironed out, such as the fact that any code on the Text widget is editable even once already submitted, but hopefully it answers your question.
EDIT: I've also neatened some of the tkinter such that the Console will behave more like a standard widget.

it isn't responding to commands
The reason it isn't responding to commands is because you haven't linked the Text widget (self.ttyText) into stdin. Currently when you type it adds text into the widget and nothing else. This linking can be done similarly to what you've already done with stdout and stderr.
When implementing this, you need to keep track of which part of the text in the widget is the text being entered by the user - this can be done using marks (as described here).
the thread doesn't stop when the user closes the window.
I don't think there is a "clean" way to solve this issue without a major code re-write, however a solution that seems to work well enough is it simply detect when the widget is destroyed and write the string "\n\nexit()" to the interpreter. This calls the exit function inside the interpreter, which causes the call to shell.interact to finish, which makes the thread finish.
So without further ado, here is the modified code:
import tkinter as tk
import sys
import code
from threading import Thread
import queue
class Console(tk.Frame):
def __init__(self, parent, _locals, exit_callback):
tk.Frame.__init__(self, parent)
self.parent = parent
self.exit_callback = exit_callback
self.destroyed = False
self.real_std_in_out = (sys.stdin, sys.stdout, sys.stderr)
sys.stdout = self
sys.stderr = self
sys.stdin = self
self.stdin_buffer = queue.Queue()
self.createWidgets()
self.consoleThread = Thread(target=lambda: self.run_interactive_console(_locals))
self.consoleThread.start()
def run_interactive_console(self, _locals):
try:
code.interact(local=_locals)
except SystemExit:
if not self.destroyed:
self.after(0, self.exit_callback)
def destroy(self):
self.stdin_buffer.put("\n\nexit()\n")
self.destroyed = True
sys.stdin, sys.stdout, sys.stderr = self.real_std_in_out
super().destroy()
def enter(self, event):
input_line = self.ttyText.get("input_start", "end")
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.mark_gravity("input_start", "left")
self.stdin_buffer.put(input_line)
def write(self, string):
self.ttyText.insert('end', string)
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.see('end')
def createWidgets(self):
self.ttyText = tk.Text(self.parent, wrap='word')
self.ttyText.grid(row=0, column=0, sticky=tk.N + tk.S + tk.E + tk.W)
self.ttyText.bind("<Return>", self.enter)
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.mark_gravity("input_start", "left")
def flush(self):
pass
def readline(self):
line = self.stdin_buffer.get()
return line
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root, locals(), root.destroy)
main_window.mainloop()
This code has few changes other than those that solve the problems stated in the question.
The advantage of this code over my previous answer is that it works inside a single process, so can be created at any point in the application, giving the programmer more control.
I have also written a more complete version of this which also prevents the user from editing text which shouldn't be editable (e.g. the output of a print statement) and has some basic coloring: https://gist.github.com/olisolomons/e90d53191d162d48ac534bf7c02a50cd

Python/Tkinter: Loop iterations not completing

In the following case, I am trying to print all output of temp2 script (which is run through subprocess) on to a textbox widget in realtime basis.
The problem I am facing is this, in temp2, for i <= 468, the script is working fine and to me it appears to be real time.
However, if i put i = 469 or above, the execution halts after many iterations without finishing.
So for example, for i = 469, the log file has entries for i = 469 to i = 3. There after the whole process halts.
Please note: the value i = 469 may not be the same for your machine. If i = 469 works fine for you, try some higher value.
Temp1.py is the main script.
#temp1.py
from Tkinter import *
import Tkinter as tk
import os
import ttk
os.system('echo OS ready')
os.system('chmod 755 *')
import subprocess
import sys
#Creating new Window to display output
t = Tk()
t.title('output Run Display')
t.geometry('800x1000-5+40')
t.state('normal')
little = Label(t, text="NRUNTEST OUTPUT LOG").grid(column = 0, row = 0)
log = Text(t, state='disabled', width=115, height=150, wrap='none')
log.grid(row = 1, column = 0)
test=subprocess.Popen('temp2',shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
#stdout
while True:
line_out = test.stdout.readline()
line_er = test.stderr.readline()
if line_out == "" and line_er == "":
break
else:
log['state'] = 'normal'
log.insert('end', line_out)
log.insert('end', line_er)
log['state'] = 'disabled'
print line_out
print line_er
t.update()
t.mainloop()
And below is the script i am running through the subprocess.
#temp2 #csh script
set i = 469
while ($i > 0)
echo i is $i | tee -a log
set i = `expr "$i" - 1`
end

Your problem is that the call to test.stdout.readline is blocking - which means processing will stop there and only resume when there is a new line of data avalilable which is returned.
The same is true for test.stderr.readline of course.
I think the simplest way of dealing with what you want is having your subprocess write to a file on the filesystem, you open that file on your main process, and try to read from it inside a tkinter callback function called regularly with the .after tkinter method.
(witha file on filesystem you can use seek and tell methods to check if you are at the end of file)
Note that in the code you placed as example, your call to Tkinter.mainloop is only reached after the subprocess is exausted already.
A better yet solution would be to read the logs you want entirely in Python without relying to a shell script.
Tkinter.after is similar to javascript's settimeout - it is a method on a widget (say, your "t" object), and you pass it the number of miliseconds to wait, and the name of the function to be called -
like in
def verify_output():
# read subprocess output file and update the window if needed
...
# call self after 1 second
t.after(1000, verify_output)
# callreader callback for the first time:
t.after(10, verify_output)
Tkinter.mainloop()

import os
import ConfigParser
from Tkinter import *
import Tkinter as tk
import os
import ttk
os.system('echo OS ready')
os.system('chmod 755 *')
import subprocess
from subprocess import call
import sys
os.system('rm stdout')
#Creating new Window to display output
t = Tk()
t.title('output Run Display')
t.geometry('100x100')
t.state('normal')
little = Label(t, text="NRUNTEST OUTPUT LOG").grid(column = 0, row = 0)
log = Text(t, state='disabled', width=50, height=50, wrap='none')
log.grid(row = 1, column = 0,sticky=(N,W,E,S))
s = ttk.Scrollbar(t,orient=VERTICAL,command=log.yview)
s.grid(column=1,row=1,sticky=(N,S))
log.configure(yscrollcommand=s.set)
ttk.Sizegrip().grid(column=1, row=1, sticky=(S,E))
with open("stdout","wb") as out:
with open("stderr","wb") as err:
test=subprocess.Popen('tem',shell=True,stdout=out,stderr=err)
fout = open('stdout','r')
ferr = open('stderr','r')
def verify():
data_out = fout.readlines()
data_out = ''.join(data_out)
log['state'] = 'normal'
log.insert('end',data_out)
log['state'] = 'disabled'
#print data_out
t.after(1000,verify)
fout.close()
verify()
t.mainloop()

As jsbueno answer, you problem come from blocking call to readline. You may instead use a file event source to get notified when data are available (through createfilehandler tkinter method). See this previous answer for details.