I'm writing a Python application with opens several text files in different threads. I want each thread to make some operations on the file while it is open. When the external text editor closes the file the processing of the corresponding thread should stop. So I need to check on whether the text file is still opened or not.
class MyThread (Thread):
def __init__(self, sf):
Thread.__init__(self)
self.sf = sf # sourcefile
def run (self):
subprocess.Popen([EDITOR, self.sf])
The problem now is that subprocess opens the file in the editor and run() terminates. How can I keep the run() open until I close the file externally?
Did you try the subprocess.call method?
subprocess.call('notepad')
This waits for the command to complete before the process terminates and returns control back to the program.
This is what i wrote:
import subprocess
from threading import Thread
class MyThread(Thread):
def __init__(self, sf):
Thread.__init__(self)
self.sf = sf
def run(self):
subprocess.call('notepad ' + self.sf)
def main():
myThread = MyThread('test.txt')
myThread.start()
while myThread.isAlive():
print "Still Alive"
if __name__ == '__main__':
main()
The program keeps printing "Still Alive" till I close the notepad window.
You have two events: FileOpen and FileClose. FileOpen happens when the editor is opened. FileClose is done when the editor closes? Or when the editor closes the file? If it's just on closing the editor then it should be straightforward to find out when the subprocess closes and then kill the worker threads.
Unless an editor provides a particular API, it's going to be a really nasty hack to find out if it closed the file but the editor remains open. E.g. You may have to poll lsof and process the output.
It sounds similar to what a revision control system does when you commit a file and it opens an editor for you. Maybe you should check the hg or bzr code bases. Here is how mercurial handles it. And here is how bazaar handles it.
I changed Shekar's version to work with vim and use the list as a command in the subprocess.call command (and added imports) . It works as he describes.
from threading import Thread
import subprocess
class MyThread(Thread):
def __init__(self, sf):
Thread.__init__(self)
self.sf = sf
def run(self):
subprocess.call(['/usr/bin/vim', self.sf])
def main():
myThread = MyThread('test.txt')
myThread.start()
while myThread.isAlive():
print "Still Alive"
if __name__ == '__main__':
main()
I found the problem. I was calling the editor MacVim via command line with /usr/bin/mvim. Now I changed it to /Applications/MacVim.app/Contents/MacOS/MacVim and it works as expected. I think that is because it checks the only process which it handles directly, in the first case it was the terminal and not the editor.
Related
I am using python 2.7 and Python thread doesn't kill its process after the main program exits. (checking this with the ps -ax command on ubuntu machine)
I have the below thread class,
import os
import threading
class captureLogs(threading.Thread):
'''
initialize the constructor
'''
def __init__(self, deviceIp, fileTag):
threading.Thread.__init__(self)
super(captureLogs, self).__init__()
self._stop = threading.Event()
self.deviceIp = deviceIp
self.fileTag = fileTag
def stop(self):
self._stop.set()
def stopped(self):
return self._stop.isSet()
'''
define the run method
'''
def run(self):
'''
Make the thread capture logs
'''
cmdTorun = "adb logcat > " + self.deviceIp +'_'+self.fileTag+'.log'
os.system(cmdTorun)
And I am creating a thread in another file sample.py,
import logCapture
import os
import time
c = logCapture.captureLogs('100.21.143.168','somefile')
c.setDaemon(True)
c.start()
print "Started the log capture. now sleeping. is this a dameon?", c.isDaemon()
time.sleep(5)
print "Sleep tiime is over"
c.stop()
print "Calling stop was successful:", c.stopped()
print "Thread is now completed and main program exiting"
I get the below output from the command line:
Started the log capture. now sleeping. is this a dameon? True
Sleep tiime is over
Calling stop was successful: True
Thread is now completed and main program exiting
And the sample.py exits.
But when I use below command on a terminal,
ps -ax | grep "adb"
I still see the process running. (I am killing them manually now using the kill -9 17681 17682)
Not sure what I am missing here.
My question is,
1) why is the process still alive when I already killed it in my program?
2) Will it create any problem if I don't bother about it?
3) is there any other better way to capture logs using a thread and monitor the logs?
EDIT: As suggested by #bug Killer, I added the below method in my thread class,
def getProcessID(self):
return os.getpid()
and used os.kill(c.getProcessID(), SIGTERM) in my sample.py . The program doesn't exit at all.
It is likely because you are using os.system in your thread. The spawned process from os.system will stay alive even after the thread is killed. Actually, it will stay alive forever unless you explicitly terminate it in your code or by hand (which it sounds like you are doing ultimately) or the spawned process exits on its own. You can do this instead:
import atexit
import subprocess
deviceIp = '100.21.143.168'
fileTag = 'somefile'
# this is spawned in the background, so no threading code is needed
cmdTorun = "adb logcat > " + deviceIp +'_'+fileTag+'.log'
proc = subprocess.Popen(cmdTorun, shell=True)
# or register proc.kill if you feel like living on the edge
atexit.register(proc.terminate)
# Here is where all the other awesome code goes
Since all you are doing is spawning a process, creating a thread to do it is overkill and only complicates your program logic. Just spawn the process in the background as shown above and then let atexit terminate it when your program exits. And/or call proc.terminate explicitly; it should be fine to call repeatedly (much like close on a file object) so having atexit call it again later shouldn't hurt anything.
I want to add a control terminal widget to my pure python+tkinter application similar to the python interpreter provided in Blender. It should be running within the same context (process) so the user can add features and control the application that is currently running from the control widget. Ideally I'd like it to also "hijack" stdout and stderr of the current application so it will report any problems or debugging information within the running application.
This is what I have come up with so far. The only problems are that it isn't responding to commands, and the thread doesn't stop when the user closes the window.
import Tkinter as tk
import sys
import code
from threading import *
class Console(tk.Frame):
def __init__(self,parent=None):
tk.Frame.__init__(self, parent)
self.parent = parent
sys.stdout = self
sys.stderr = self
self.createWidgets()
self.consoleThread = ConsoleThread()
self.after(100,self.consoleThread.start)
def write(self,string):
self.ttyText.insert('end', string)
self.ttyText.see('end')
def createWidgets(self):
self.ttyText = tk.Text(self.parent, wrap='word')
self.ttyText.grid(row=0,column=0,sticky=tk.N+tk.S+tk.E+tk.W)
class ConsoleThread(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
vars = globals().copy()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root)
main_window.mainloop()
try:
if root.winfo_exists():
root.destroy()
except:
pass
I have the answer in case anyone still cares! (I have also changed to python 3, hence the import tkinter rather than import Tkinter)
I have changed the approach slightly from the original by using a separate file to run the InteractiveConsole, and then making the main file open this other file (which I have called console.py and is in the same directory) in a subprocess, linking the stdout, stderr, and stdin of this subprocess to the tkinter Text widget programatically.
Here is the code in the for the console file (if this is run normally, it acts like a normal console):
# console.py
import code
if __name__ == '__main__':
vars = globals().copy()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()
And here is the code for the python interpreter, that runs the console inside the Text widget:
# main.py
import tkinter as tk
import subprocess
import queue
import os
from threading import Thread
class Console(tk.Frame):
def __init__(self,parent=None):
tk.Frame.__init__(self, parent)
self.parent = parent
self.createWidgets()
# get the path to the console.py file assuming it is in the same folder
consolePath = os.path.join(os.path.dirname(__file__),"console.py")
# open the console.py file (replace the path to python with the correct one for your system)
# e.g. it might be "C:\\Python35\\python"
self.p = subprocess.Popen(["python3",consolePath],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
# make queues for keeping stdout and stderr whilst it is transferred between threads
self.outQueue = queue.Queue()
self.errQueue = queue.Queue()
# keep track of where any line that is submitted starts
self.line_start = 0
# make the enter key call the self.enter function
self.ttyText.bind("<Return>",self.enter)
# a daemon to keep track of the threads so they can stop running
self.alive = True
# start the functions that get stdout and stderr in separate threads
Thread(target=self.readFromProccessOut).start()
Thread(target=self.readFromProccessErr).start()
# start the write loop in the main thread
self.writeLoop()
def destroy(self):
"This is the function that is automatically called when the widget is destroyed."
self.alive=False
# write exit() to the console in order to stop it running
self.p.stdin.write("exit()\n".encode())
self.p.stdin.flush()
# call the destroy methods to properly destroy widgets
self.ttyText.destroy()
tk.Frame.destroy(self)
def enter(self,e):
"The <Return> key press handler"
string = self.ttyText.get(1.0, tk.END)[self.line_start:]
self.line_start+=len(string)
self.p.stdin.write(string.encode())
self.p.stdin.flush()
def readFromProccessOut(self):
"To be executed in a separate thread to make read non-blocking"
while self.alive:
data = self.p.stdout.raw.read(1024).decode()
self.outQueue.put(data)
def readFromProccessErr(self):
"To be executed in a separate thread to make read non-blocking"
while self.alive:
data = self.p.stderr.raw.read(1024).decode()
self.errQueue.put(data)
def writeLoop(self):
"Used to write data from stdout and stderr to the Text widget"
# if there is anything to write from stdout or stderr, then write it
if not self.errQueue.empty():
self.write(self.errQueue.get())
if not self.outQueue.empty():
self.write(self.outQueue.get())
# run this method again after 10ms
if self.alive:
self.after(10,self.writeLoop)
def write(self,string):
self.ttyText.insert(tk.END, string)
self.ttyText.see(tk.END)
self.line_start+=len(string)
def createWidgets(self):
self.ttyText = tk.Text(self, wrap=tk.WORD)
self.ttyText.pack(fill=tk.BOTH,expand=True)
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root)
main_window.pack(fill=tk.BOTH,expand=True)
root.mainloop()
The reason that reading from stdout and stderr is in separate threads is because the read method is blocking, which causes the program to freeze until the console.py subprocess gives more output, unless these are in separate threads. The writeLoop method and the queues are needed to write to the Text widget since tkinter is not thread safe.
This certainly still has problems to be ironed out, such as the fact that any code on the Text widget is editable even once already submitted, but hopefully it answers your question.
EDIT: I've also neatened some of the tkinter such that the Console will behave more like a standard widget.
it isn't responding to commands
The reason it isn't responding to commands is because you haven't linked the Text widget (self.ttyText) into stdin. Currently when you type it adds text into the widget and nothing else. This linking can be done similarly to what you've already done with stdout and stderr.
When implementing this, you need to keep track of which part of the text in the widget is the text being entered by the user - this can be done using marks (as described here).
the thread doesn't stop when the user closes the window.
I don't think there is a "clean" way to solve this issue without a major code re-write, however a solution that seems to work well enough is it simply detect when the widget is destroyed and write the string "\n\nexit()" to the interpreter. This calls the exit function inside the interpreter, which causes the call to shell.interact to finish, which makes the thread finish.
So without further ado, here is the modified code:
import tkinter as tk
import sys
import code
from threading import Thread
import queue
class Console(tk.Frame):
def __init__(self, parent, _locals, exit_callback):
tk.Frame.__init__(self, parent)
self.parent = parent
self.exit_callback = exit_callback
self.destroyed = False
self.real_std_in_out = (sys.stdin, sys.stdout, sys.stderr)
sys.stdout = self
sys.stderr = self
sys.stdin = self
self.stdin_buffer = queue.Queue()
self.createWidgets()
self.consoleThread = Thread(target=lambda: self.run_interactive_console(_locals))
self.consoleThread.start()
def run_interactive_console(self, _locals):
try:
code.interact(local=_locals)
except SystemExit:
if not self.destroyed:
self.after(0, self.exit_callback)
def destroy(self):
self.stdin_buffer.put("\n\nexit()\n")
self.destroyed = True
sys.stdin, sys.stdout, sys.stderr = self.real_std_in_out
super().destroy()
def enter(self, event):
input_line = self.ttyText.get("input_start", "end")
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.mark_gravity("input_start", "left")
self.stdin_buffer.put(input_line)
def write(self, string):
self.ttyText.insert('end', string)
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.see('end')
def createWidgets(self):
self.ttyText = tk.Text(self.parent, wrap='word')
self.ttyText.grid(row=0, column=0, sticky=tk.N + tk.S + tk.E + tk.W)
self.ttyText.bind("<Return>", self.enter)
self.ttyText.mark_set("input_start", "end-1c")
self.ttyText.mark_gravity("input_start", "left")
def flush(self):
pass
def readline(self):
line = self.stdin_buffer.get()
return line
if __name__ == '__main__':
root = tk.Tk()
root.config(background="red")
main_window = Console(root, locals(), root.destroy)
main_window.mainloop()
This code has few changes other than those that solve the problems stated in the question.
The advantage of this code over my previous answer is that it works inside a single process, so can be created at any point in the application, giving the programmer more control.
I have also written a more complete version of this which also prevents the user from editing text which shouldn't be editable (e.g. the output of a print statement) and has some basic coloring: https://gist.github.com/olisolomons/e90d53191d162d48ac534bf7c02a50cd
I am using Python 2.6.6 for Windows (on Windows XP SP3) with pywin32-218.
In my Python application, I have a second thread (apart from the main thread) which spawns a subprocess to run another Windows executable.
My problem is that when the main process (python.exe) is killed (e.g. using taskkill), I want to terminate the subprocess (calc.exe) and perform some cleaning up.
I tried various methods (atexit, signal and win32api.handleConsoleCtrl), but none seem to be able to trap the taskkill signal.
My code as follows (test.py):
import sys
import os
import signal
import win32api
import atexit
import time
import threading
import subprocess
class SecondThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.secondProcess = None
def run(self):
secondCommand = ['C:\WINDOWS\system32\calc.exe']
self.secondProcess = subprocess.Popen(secondCommand)
print 'calc.exe running'
self.secondProcess.wait()
print 'calc.exe stopped'
# do cleanup here
def stop(self):
if self.secondProcess and self.secondProcess.returncode == None:
self.secondProcess.kill()
secondThread = SecondThread()
def main():
secondThread.start()
def cleanup():
print 'cleaning up'
secondThread.stop()
print 'cleaned up'
atexit.register(cleanup)
def handleSignal(signalNum, frame):
print 'handleSignal'
cleanup()
sys.exit(0)
for signalNum in (signal.SIGINT, signal.SIGILL, signal.SIGABRT, signal.SIGFPE, signal.SIGSEGV, signal.SIGTERM):
signal.signal(signalNum, handleSignal)
def handleConsoleCtrl(signalNum):
print ('handleConsoleCtrl')
cleanup()
win32api.SetConsoleCtrlHandler(handleConsoleCtrl, True)
main()
The application is launched using
python.exe test.py
The console then prints "calc.exe running", and the Calculator application runs, and using Process Explorer, I can see calc.exe as a sub-process of python.exe
Then I kill the main process using
taskkill /pid XXXX /f
(where XXXX is the PID for python.exe)
What happens after this is that the command prompt returns without further output (i.e. none of "cleaning up", "handleSignal" or "handleConsoleCtrl" gets printed), the Calculator application continues running, and from Process Explorer, python.exe is no longer running but calc.exe has re-parented itself.
Taskkill (normally) sends WM_CLOSE. If your application is console only and has no window, while you can get CTRL_CLOSE_EVENT via a handler set by SetConsoleCtrlHandler (which happens if your controlling terminal window is closed) you can't receive a bare WM_CLOSE message.
If you have to stick with taskkill (rather than using a different program to send a Ctrl-C) one solution is to set the aforementioned handler and ensure your application has its own terminal window (e.g. by usingstart.exe "" <yourprog> to invoke it). See https://stackoverflow.com/a/23197789/4513656 for details an alternatives.
I'm trying to do what sounds fairly simple but I keep running in to all sorts of problems.
I'm trying to create a GUI that can tail several files at the same time using PyQt.
I saw this answer on how to tail a file in pure Python
How can I tail a log file in Python?
I have tried using this code inside of a QThread.
The issues I'm having here are that the tail process never stops by itself; it needs to be killed.
It should be killed when the GUI is closed.
The other issues I'm getting with this specific solution below is
QThread: Destroyed while thread is still running
and
QWaitCondition::wakeAll(): mutex lock failure:
and
QThread: Destroyed while thread is still running
Traceback (most recent call last):
File "./tailer.py", line 27, in run
self.emit(SIGNAL('newline'), line.rstrip())
RuntimeError: underlying C/C++ object has been deleted
Other implementations I've tried have had the tail process complaining about a broken pipe but those stopped appearing once I did stderr=PIPE as well.
I'm worried now that I could be missing errors since I never read from stderr (since it would block and there shouldn't be any output).
To get the errors fire this up trying to tail 3 different file.
I wrote another script that loops and writes to those 3 files doing a sleep of 0.1 seconds.
I close the GUI and start it up over and over again. Sometimes I get errors sometimes I don't.
Please tell me what I'm doing wrong here.
#!/usr/bin/env python
from PyQt4.QtCore import *
from PyQt4.QtGui import *
import os
from subprocess import Popen, PIPE
class Tailer(QThread):
def __init__(self, fname, parent=None):
super(Tailer, self).__init__(parent)
self.fname = fname
self.connect(self, SIGNAL('finished()'), self.cleanup)
def cleanup(self):
print 'CLEANING UP'
self.p.kill()
print 'killed'
def run(self):
command = ["tail", "-f", self.fname]
print command
self.p = Popen(command, stdout=PIPE, stderr=PIPE)
while True:
line = self.p.stdout.readline()
self.emit(SIGNAL('newline'), line.rstrip())
if not line:
print 'BREAKING'
break
def foo(self):
self.p.kill()
class TailWidget(QWidget):
def __init__(self, fnames, parent=None):
super(TailWidget, self).__init__(parent)
layout = QGridLayout()
self.threads = {}
self.browsers = {}
for i, fname in enumerate(fnames):
if not os.path.exists(fname):
print fname, "doesn't exist; creating"
p = Popen(['touch', fname], stdout=PIPE, stderr=PIPE)
out, err = p.communicate()
ret = p.wait()
assert ret == 0
t = Tailer(fname, self)
self.threads[fname] = t
b = QTextBrowser()
self.browsers[fname] = b
layout.addWidget(QLabel('Tail on %s' % fname), 0, i)
layout.addWidget(b, 1, i)
self.connect(t, SIGNAL("newline"), b.append)
t.start()
self.setLayout(layout)
def closeEvent(self, event):
for fname, t in self.threads.items():
t.foo()
if __name__ == '__main__':
import sys
app = QApplication(sys.argv)
tw = TailWidget(sys.argv[1:])
tw.show()
sys.exit(app.exec_())
The problem is that the main thread isn't waiting on the background threads.
You tell them to stop here:
def closeEvent(self, event):
for fname, t in self.threads.items():
t.foo()
So, this kills all of the subprocesses, which will make all of the background threads quit eventually. But it won't make them stop immediately. That won't happen until the next time each one gets to its readline.
After killing the subprocesses, you return, letting Qt immediately close your window and destroy your widget. Any background threads that try to send a signal to that widget will then fail.
Imagine that thread 1 has done a readline, and is in the middle of its rstrip when thread 0 tries to close down. So, thread 0 kills thread 1's subprocess, then deletes the main widget. Thread 1 finishes its rstrip and calls emit, and it's now sending to a deleted widget.
I have a wrapper script, inside that there are many other test script. Inside of one of the test script I make a subprocess using Popen class. The problem is that I don't know how to terminate that child process and return to main process and continue with the next test script. My wrapper stops at the test script that has the child process and never continue. Can you give a hint? Thx.
P.S. kill() or terminate() or anyother function that I consider usefull, doesn't put me back to the main process. I want to terminate the subprocess and continue with the main process.
Keep a reference to the child in the main script. With that reference call the terminate()
from subprocess import Popen
class TestApp(object):
app = None
def start(self):
self.app = Popen(['your command'])
def stop(self):
self.app.terminate()
In the main script:
app1 = TestApp()
app1.start()
app2 = TestApp()
app2.start()
#do something here
app1.stop()
app2.stop()
#do more here