I'm trying to do what sounds fairly simple but I keep running in to all sorts of problems.
I'm trying to create a GUI that can tail several files at the same time using PyQt.
I saw this answer on how to tail a file in pure Python
How can I tail a log file in Python?
I have tried using this code inside of a QThread.
The issues I'm having here are that the tail process never stops by itself; it needs to be killed.
It should be killed when the GUI is closed.
The other issues I'm getting with this specific solution below is
QThread: Destroyed while thread is still running
and
QWaitCondition::wakeAll(): mutex lock failure:
and
QThread: Destroyed while thread is still running
Traceback (most recent call last):
File "./tailer.py", line 27, in run
self.emit(SIGNAL('newline'), line.rstrip())
RuntimeError: underlying C/C++ object has been deleted
Other implementations I've tried have had the tail process complaining about a broken pipe but those stopped appearing once I did stderr=PIPE as well.
I'm worried now that I could be missing errors since I never read from stderr (since it would block and there shouldn't be any output).
To get the errors fire this up trying to tail 3 different file.
I wrote another script that loops and writes to those 3 files doing a sleep of 0.1 seconds.
I close the GUI and start it up over and over again. Sometimes I get errors sometimes I don't.
Please tell me what I'm doing wrong here.
#!/usr/bin/env python
from PyQt4.QtCore import *
from PyQt4.QtGui import *
import os
from subprocess import Popen, PIPE
class Tailer(QThread):
def __init__(self, fname, parent=None):
super(Tailer, self).__init__(parent)
self.fname = fname
self.connect(self, SIGNAL('finished()'), self.cleanup)
def cleanup(self):
print 'CLEANING UP'
self.p.kill()
print 'killed'
def run(self):
command = ["tail", "-f", self.fname]
print command
self.p = Popen(command, stdout=PIPE, stderr=PIPE)
while True:
line = self.p.stdout.readline()
self.emit(SIGNAL('newline'), line.rstrip())
if not line:
print 'BREAKING'
break
def foo(self):
self.p.kill()
class TailWidget(QWidget):
def __init__(self, fnames, parent=None):
super(TailWidget, self).__init__(parent)
layout = QGridLayout()
self.threads = {}
self.browsers = {}
for i, fname in enumerate(fnames):
if not os.path.exists(fname):
print fname, "doesn't exist; creating"
p = Popen(['touch', fname], stdout=PIPE, stderr=PIPE)
out, err = p.communicate()
ret = p.wait()
assert ret == 0
t = Tailer(fname, self)
self.threads[fname] = t
b = QTextBrowser()
self.browsers[fname] = b
layout.addWidget(QLabel('Tail on %s' % fname), 0, i)
layout.addWidget(b, 1, i)
self.connect(t, SIGNAL("newline"), b.append)
t.start()
self.setLayout(layout)
def closeEvent(self, event):
for fname, t in self.threads.items():
t.foo()
if __name__ == '__main__':
import sys
app = QApplication(sys.argv)
tw = TailWidget(sys.argv[1:])
tw.show()
sys.exit(app.exec_())
The problem is that the main thread isn't waiting on the background threads.
You tell them to stop here:
def closeEvent(self, event):
for fname, t in self.threads.items():
t.foo()
So, this kills all of the subprocesses, which will make all of the background threads quit eventually. But it won't make them stop immediately. That won't happen until the next time each one gets to its readline.
After killing the subprocesses, you return, letting Qt immediately close your window and destroy your widget. Any background threads that try to send a signal to that widget will then fail.
Imagine that thread 1 has done a readline, and is in the middle of its rstrip when thread 0 tries to close down. So, thread 0 kills thread 1's subprocess, then deletes the main widget. Thread 1 finishes its rstrip and calls emit, and it's now sending to a deleted widget.
Related
I am writing a python app to take some inputs from the user and call a shell script based on these inputs.
This shell script can run for quite sometime, and I want to redirect the output it produces (in realtime) to tkinter.
I managed to do that, but it only happens after the shell script is completely finished, not as soon as the shell script "echos" something for example.
So main problem:
1. Output appears in the text widget only after shellScript.sh is exited (although if I run the same in the terminal manually, I see continuous output).
2. Side problem: all the "\n" printed in the output are just printed as "\n" and no new lines are added in the text widget.
here is what I have:
class RedirectText(object):
def __init__(self, text_ctrl):
"""Constructor"""
self.output = text_ctrl
def write(self, string):
self.output.insert(tk.END, string[2:])
class Gui(GuiApp):
def __init__(self, master=None):
redir = RedirectText(self.text_Output)
sys.stdout = redir
def testRun(self, event=None):
p = subprocess.Popen(["./shellScript.sh"], stdout=subprocess.PIPE)
print(p.communicate()[0])
As p.communicate() will wait for the process to complete, so use p.poll() and p.stdout.readline() instead. Also put the process in a thread to avoid blocking the main thread:
import threading
...
class Gui(GuiApp):
...
def runScript(self):
print('started')
p = subprocess.Popen(['./shellScript.sh'], stdout=subprocess.PIPE, bufsize=1, text=True)
while p.poll() is None: # check whether process is still running
msg = p.stdout.readline().strip() # read a line from the process output
if msg:
print(msg)
print('finished')
def testRun(self, event=None):
# create a thread to run the script
threading.Thread(target=self.runScript, daemon=True).start()
Segmentation Fault in subprocess.stdout while displaying on QPlainTextEdit
Hi,
I am starting a thread to the function shown and streaming its result to a QTextEdit object in the thread. The function sometimes crashes with a segmentation fault for unknown reasons.
self.plainTextEdit = QPlainTextEdit()
self.thread = Thread(target = runcmd, args = ("make dc",))
self.thread.start()
self.thread.join()
def runcmd(self,cmd):
process = subprocess.Popen(shlex.split(cmd),stdout=subprocess.PIPE, bufsize=-1)
while True:
line = process.stdout.readline()
if not line:
break
self.plainTextEdit.moveCursor(QTextCursor.End)
self.plainTextEdit.insertPlainText(line.strip())
process.terminate()
The
make dc
command is a call to the design compiler synthesis tool. If I try to print the
line
variable instead of writing to the plainTextEdit Object the thread runs fine, displaying the result in the terminal window. Any help/advice is welcome......
Thank You
You can't use QT things from a python thread. Your options:
Send the data back from the thread to your main program, using a queue.Queue or some other similar synchronization object. Then the main program adds the data to the text widget. Documentation here.
Use a QThread, which is the QT equivalent of threads. Documentation here.
You cannot update the Qt GUI from another thread. Fortunately for us, Qt gave us signals, and PyQt gave us pyqtSignal, just for this situation. There are a few ways to do it, but I prefer the following style.
class YourClass(QtCore.QObject):
appendSignal = pyqtSignal(str)
def __init__(self):
super(YourClass, self).__init__()
self.plainTextEdit = QPlainTextEdit()
self.appendSignal.connect(self.reallyAppendToTextEdit)
self.appendToTextEdit = self.appendSignal.emit
self.thread = Thread(target = runcmd, args = ("make dc",))
self.thread.start()
self.thread.join()
def reallyAppendToTextEdit(self, txt):
self.plainTextEdit.moveCursor(QTextCursor.End)
self.plainTextEdit.insertPlainText(txt)
def runcmd(self,cmd):
process = subprocess.Popen(shlex.split(cmd),stdout=subprocess.PIPE, bufsize=-1)
while True:
line = process.stdout.readline()
if not line:
break
self.appendToTextEdit(line.strip())
process.terminate()
Instead of using subprocess.Popen() you should use QProcess which is GUI friendly so it is not necessary to use a new thread.
from PyQt5.QtCore import *
from PyQt5.QtWidgets import *
import shlex
class LogWidget(QWidget):
def __init__(self, parent=None):
super(LogWidget, self).__init__(parent)
lay = QVBoxLayout(self)
self.plainTextEdit = QPlainTextEdit()
lay.addWidget(self.plainTextEdit)
self.runcmd("make dc")
def runcmd(self, cmd):
process = QProcess(self)
process.readyReadStandardOutput.connect(self.onReadyReadStandardOutput)
process.readyReadStandardError.connect(self.onReadyReadStandardError)
program, *arguments = shlex.split(cmd)
process.start(program, arguments)
def onReadyReadStandardOutput(self):
process = self.sender()
self.plainTextEdit.appendPlainText(str(process.readAllStandardOutput()))
def onReadyReadStandardError(self):
process = self.sender()
self.plainTextEdit.appendPlainText(str(process.readAllStandardError()))
if __name__ == '__main__':
import sys
app = QApplication(sys.argv)
w = LogWidget()
w.show()
sys.exit(app.exec_())
When closing my application, all threads, and Tkinter Threads close successfully but a subprocess that I have refuses to close on exit.
class ThreadedTask(Thread):
def __init__(self, queue):
Thread.__init__(self)
self.queue = queue
def run(self):
proc = Popen("receivetest -f=/dev/pcan33".split(), stdout = PIPE)
payload = iter(proc.stdout.readline, "")
for line in payload:
if line[0].isdigit():
splitline = line.split()
self.dictAdd(splitline)
This is the Class containing the subprocess.
And this is the call at the beginning:
if __name__ == "__main__":
root = tk.Tk()
Data = Queue.Queue()
DataThread = ThreadedTask(Data)
DataThread.daemon = True
DataThread.start()
myapp = BaudWindow(root)
root.mainloop()
As I say everything else closes correctly. Is this due to the fact I have nested a subprocess into a thread?
Child processes do not die automatically if the parent process dies by default. See Python: how to kill child process(es) when parent dies?
You could call proc.terminate() explicitly in your case e.g., in atexit handler.
That is because you are not starting a subprocess but a separate thread. Try subprocess.popen() instead. It works as per what you want.
I want to add a control terminal widget to my pure python+tkinter application similar to the python interpreter provided in Blender. It should be running within the same context (process) so the user can add features and control the application that is currently running from the control widget. Ideally I'd like it to also "hijack" stdout and stderr of the current application so it will report any problems or debugging information within the running application.
This is what I have come up with so far. The only problems are that it isn't responding to commands, and the thread doesn't stop when the user closes the window.
import Tkinter as tk
import sys
import code
from threading import *
class Console(tk.Frame):
def __init__(self,parent=None):
tk.Frame.__init__(self, parent)
self.parent = parent
sys.stdout = self
sys.stderr = self
self.createWidgets()
self.consoleThread = ConsoleThread()
self.after(100,self.consoleThread.start)
def write(self,string):
self.ttyText.insert('end', string)
self.ttyText.see('end')
def createWidgets(self):
self.ttyText = tk.Text(self.parent, wrap='word')
self.ttyText.grid(row=0,column=0,sticky=tk.N+tk.S+tk.E+tk.W)
class ConsoleThread(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
vars = globals().copy()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root)
main_window.mainloop()
try:
if root.winfo_exists():
root.destroy()
except:
pass
I have the answer in case anyone still cares! (I have also changed to python 3, hence the import tkinter rather than import Tkinter)
I have changed the approach slightly from the original by using a separate file to run the InteractiveConsole, and then making the main file open this other file (which I have called console.py and is in the same directory) in a subprocess, linking the stdout, stderr, and stdin of this subprocess to the tkinter Text widget programatically.
Here is the code in the for the console file (if this is run normally, it acts like a normal console):
# console.py
import code
if __name__ == '__main__':
vars = globals().copy()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()
And here is the code for the python interpreter, that runs the console inside the Text widget:
# main.py
import tkinter as tk
import subprocess
import queue
import os
from threading import Thread
class Console(tk.Frame):
def __init__(self,parent=None):
tk.Frame.__init__(self, parent)
self.parent = parent
self.createWidgets()
# get the path to the console.py file assuming it is in the same folder
consolePath = os.path.join(os.path.dirname(__file__),"console.py")
# open the console.py file (replace the path to python with the correct one for your system)
# e.g. it might be "C:\\Python35\\python"
self.p = subprocess.Popen(["python3",consolePath],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
# make queues for keeping stdout and stderr whilst it is transferred between threads
self.outQueue = queue.Queue()
self.errQueue = queue.Queue()
# keep track of where any line that is submitted starts
self.line_start = 0
# make the enter key call the self.enter function
self.ttyText.bind("<Return>",self.enter)
# a daemon to keep track of the threads so they can stop running
self.alive = True
# start the functions that get stdout and stderr in separate threads
Thread(target=self.readFromProccessOut).start()
Thread(target=self.readFromProccessErr).start()
# start the write loop in the main thread
self.writeLoop()
def destroy(self):
"This is the function that is automatically called when the widget is destroyed."
self.alive=False
# write exit() to the console in order to stop it running
self.p.stdin.write("exit()\n".encode())
self.p.stdin.flush()
# call the destroy methods to properly destroy widgets
self.ttyText.destroy()
tk.Frame.destroy(self)
def enter(self,e):
"The <Return> key press handler"
string = self.ttyText.get(1.0, tk.END)[self.line_start:]
self.line_start+=len(string)
self.p.stdin.write(string.encode())
self.p.stdin.flush()
def readFromProccessOut(self):
"To be executed in a separate thread to make read non-blocking"
while self.alive:
data = self.p.stdout.raw.read(1024).decode()
self.outQueue.put(data)
def readFromProccessErr(self):
"To be executed in a separate thread to make read non-blocking"
while self.alive:
data = self.p.stderr.raw.read(1024).decode()
self.errQueue.put(data)
def writeLoop(self):
"Used to write data from stdout and stderr to the Text widget"
# if there is anything to write from stdout or stderr, then write it
if not self.errQueue.empty():
self.write(self.errQueue.get())
if not self.outQueue.empty():
self.write(self.outQueue.get())
# run this method again after 10ms
if self.alive:
self.after(10,self.writeLoop)
def write(self,string):
self.ttyText.insert(tk.END, string)
self.ttyText.see(tk.END)
self.line_start+=len(string)
def createWidgets(self):
self.ttyText = tk.Text(self, wrap=tk.WORD)
self.ttyText.pack(fill=tk.BOTH,expand=True)
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root)
main_window.pack(fill=tk.BOTH,expand=True)
root.mainloop()
The reason that reading from stdout and stderr is in separate threads is because the read method is blocking, which causes the program to freeze until the console.py subprocess gives more output, unless these are in separate threads. The writeLoop method and the queues are needed to write to the Text widget since tkinter is not thread safe.
This certainly still has problems to be ironed out, such as the fact that any code on the Text widget is editable even once already submitted, but hopefully it answers your question.
EDIT: I've also neatened some of the tkinter such that the Console will behave more like a standard widget.
it isn't responding to commands
The reason it isn't responding to commands is because you haven't linked the Text widget (self.ttyText) into stdin. Currently when you type it adds text into the widget and nothing else. This linking can be done similarly to what you've already done with stdout and stderr.
When implementing this, you need to keep track of which part of the text in the widget is the text being entered by the user - this can be done using marks (as described here).
the thread doesn't stop when the user closes the window.
I don't think there is a "clean" way to solve this issue without a major code re-write, however a solution that seems to work well enough is it simply detect when the widget is destroyed and write the string "\n\nexit()" to the interpreter. This calls the exit function inside the interpreter, which causes the call to shell.interact to finish, which makes the thread finish.
So without further ado, here is the modified code:
import tkinter as tk
import sys
import code
from threading import Thread
import queue
class Console(tk.Frame):
def __init__(self, parent, _locals, exit_callback):
tk.Frame.__init__(self, parent)
self.parent = parent
self.exit_callback = exit_callback
self.destroyed = False
self.real_std_in_out = (sys.stdin, sys.stdout, sys.stderr)
sys.stdout = self
sys.stderr = self
sys.stdin = self
self.stdin_buffer = queue.Queue()
self.createWidgets()
self.consoleThread = Thread(target=lambda: self.run_interactive_console(_locals))
self.consoleThread.start()
def run_interactive_console(self, _locals):
try:
code.interact(local=_locals)
except SystemExit:
if not self.destroyed:
self.after(0, self.exit_callback)
def destroy(self):
self.stdin_buffer.put("\n\nexit()\n")
self.destroyed = True
sys.stdin, sys.stdout, sys.stderr = self.real_std_in_out
super().destroy()
def enter(self, event):
input_line = self.ttyText.get("input_start", "end")
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.mark_gravity("input_start", "left")
self.stdin_buffer.put(input_line)
def write(self, string):
self.ttyText.insert('end', string)
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.see('end')
def createWidgets(self):
self.ttyText = tk.Text(self.parent, wrap='word')
self.ttyText.grid(row=0, column=0, sticky=tk.N + tk.S + tk.E + tk.W)
self.ttyText.bind("<Return>", self.enter)
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.mark_gravity("input_start", "left")
def flush(self):
pass
def readline(self):
line = self.stdin_buffer.get()
return line
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root, locals(), root.destroy)
main_window.mainloop()
This code has few changes other than those that solve the problems stated in the question.
The advantage of this code over my previous answer is that it works inside a single process, so can be created at any point in the application, giving the programmer more control.
I have also written a more complete version of this which also prevents the user from editing text which shouldn't be editable (e.g. the output of a print statement) and has some basic coloring: https://gist.github.com/olisolomons/e90d53191d162d48ac534bf7c02a50cd
Is there a way that I can run a script from another while getting output as readily as I get by executing it by itself.
For example:
I use the os.popen3 command to execute abc.py, but I am unable to get output from abc.py as readily as I would with doing python abc.py; it seems that I need to wait for os.popen3 command to finish:
fin, fout, ferr=os.popen3("abc.py")
out = fout.read()
err = ferr.read()
fo.write(out)
fe.write(err)
print out
print err
[EDIT]:fo and fe here are file handles to the output and error logs, respectively.
Also, what widget do I use to populate the output in, in pygtk?
import subprocess
pro = subprocess.Popen('abc.py')
Is a much better way of fiddling with another scripts ins, outs, and errors.
The subprocess module is the option, but the tricky part is to follow the output un parallel with your main loop of gtk, to accomplish that goal you must have to consider the platform that you are dealing, if you are in linux you can easily run another thread and use gtk.gdk.threads_init to use threads in pygtk, but if you are planing to run your application on windows, then you should use generators and gobject.idle_add.
About the widget, use gtk.TextBuffer associated with a gtk.TextView
Here is an example with threads
import gtk
import subprocess
import threading
gtk.gdk.threads_init()
class FollowProcess(threading.Thread):
def __init__(self, cmd, text_buffer):
self.tb = text_buffer
self.child = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
super(FollowProcess, self).__init__()
def run(self):
while not self.child.poll():
out = self.child.stdout.read(1)
if out != '':
gtk.gdk.threads_enter()
self.tb.insert_at_cursor(out)
gtk.gdk.threads_leave()
def destroy(w, cmd):
cmd.child.terminate()
gtk.main_quit()
i = 0
def click_count(btn):
global i
message.set_text('Calling button %d' %i)
i += 1
other_command = 'python extranger.py'
w = gtk.Window()
w.resize(400, 400)
message = gtk.Label('Nothing')
tb = gtk.TextBuffer()
tv = gtk.TextView(tb)
scroll = gtk.ScrolledWindow()
scroll.set_policy(gtk.POLICY_AUTOMATIC, gtk.POLICY_AUTOMATIC)
scroll.add(tv)
box = gtk.VBox()
button = gtk.Button('Active button')
cmd = FollowProcess('python extranger.py', tb)
button.connect('clicked', click_count )
w.connect('destroy', destroy, cmd)
box.pack_start(button, False)
box.pack_start(message, False)
box.pack_start(scroll)
w.add(box)
w.show_all()
cmd.start()
gtk.main()
And in extranger.py
import time
import sys
i = 0
while True:
print 'some output %d' % i
sys.stdout.flush() # you need this to see the output
time.sleep(.5)
i += 1
Note how the button stills responsive even with the update in parallel.
You mentioned PyGtk but you can give a try to PyQt and particularly QProcess class wich has some nice signals like :
readyReadStandardError
readyReadStandardOutput
Look for a similar tool with PyGtk.