I worked with a bunch of examples I Googled, and looked at a bunch of questions about threading here on stackoverflow, but I still can't seem to get it.
Here is some sample code I am working with:
class Debugger(QTextBrowser):
def __init__(self, parent):
super(Debugger, self).__init__(parent)
sys.stdout = self
def write(self, text):
self.moveCursor(QTextCursor.End)
self.textCursor().insertText(text)
Basically, I am trying to catch any 'print' statements or other forms of writing to the stdoutstream. But I can't get it to print them one by one as they arrive. Instead, it waits until whatever process that prints the strings is finished, and then begins to log the information. So how do I thread this properly?
Edit: Reduced code to bare minimum.
I suspect you are running it all in one thread?
While the other code is running the Qt mainloop is not being runned, therefore it is not processing events (drawing on the screen, taking input, etc.). So while the textbox internally is being updated, it is not being painted on the screen, until your task completes, and then the window is redrawn.
The best solution is to run the process that is doing output (and taking some time) in a separate thread.
A quick-n-dirty solution is to manually iterate the Qt mainloop whenever you have written something to the textbox, see IPython iterate main loop manually?.
Related
I have a main process that does some stuff (e.g. analyzing data) and it runs alone just fine. I also want to make a simple GUI that displays the result of the main task using PyQT5. The idea is that the GUI should not interfere in the main process, that is, if I remove the GUI it shouldn't cause any issue to the main process.
The code of the main process is quite simple:
if __name__ == '__main__':
# initialize the object that performs the main task
tasker = Task()
# the graphical interface to visualize the result of tasker
gui = GUI(task) # GUI is a separate class that keeps a reference to tasker
# read the input data and do stuff on each new data instance
for f in listdir(inrepo):
data = read_new_data(f) # an utility function that reads new data from file
result = tasker.process(data) # tasker processes the new data and return some results
gui.update(data, result) # pass the data and result in the GUI to update it
The code of the GUI class is quite long so I only paste a few lines here, but the lines I skip are just to create widgets, nothing fancy (I didn't connect any event yet)
class GUI(QApplication):
def __init__(self, tasker):
"""Initialize the application"""
super().__init__([])
self.tasker = tasker
# define the main window
self.window = QWidget()
self.window.setWindowTitle('GUI')
... # layout and components etc.
# show stuff
self.window.show()
self.exec()
So I want the GUI to be completely independent from my main process. For example, if I don't need the GUI anymore, I can just comment out the 2 lines gui = GUI(task) and gui.update(data, result).
However, the problem is that starting the GUI blocks the entire process (I assume it's because of self.exec() in GUI.__init__, so my main process cannot proceed to loop over the data. Could you please show me how to make PyQT non-blocking? Is it even feasible?
Some options I have considered:
Threading: it seems more complicated than necessary for my use case and it may make referencing to the task instance difficult from a thread. All new updates to task should be reflected in the GUI. If I'm not mistaken, PyQT's application already runs on a thread. So multi-level threading may be troublesome.
Run the GUI from another Python process, communicated via shared folders: may incur high latency. Any new data and result from task should be immediately reflected in the GUI. Writing to file then reading from file then updating the GUI will cause some delay.
Perform the task in GUI: I could use some timeout event to read new data periodically and run task on them, but then everything depends heavily on the GUI, and I can't just comment it out if I don't need the GUI anymore.
Any suggestion is very much appreciated! Thank you very much!
Switching your program between GUI mode/console mode is often not as simple as commenting out some lines. PyQt in particular does not allow you to run a GUI loop from anything other than a main thread. Not all hope is lost, though - this simply means that you should decide as early as possible whether your program is going to run as a console application or as a GUI.
Instead of relying in commenting out code, you can create a "switch" of sorts inside your code that tells your code how to execute. One way to do this is to check the command-line arguments when you execute the code, e.g.:
import sys
if "--headless" in sys.argv[1:]: # checking the command-line arguments
run_code_without_gui()
else:
run_code_with_gui()
This way, executing your code as python mycode.py --headless will execute it without the GUI (through the run_code_without_gui function), while executing python mycode.py will run it as a GUI (through the run_code_with_gui function). (Although if you're actually going to parse command-line arguments, I recommend using the argparse library).
You can even keep the analysis code completely decoupled from the GUI code, so that you simply run a function from, say, analysis.py when executing without the GUI, and have the GUI call that exact same function when e.g. the user clicks on the "Analyze" button.
Another thing to note is that if your analysis code takes long to execute, it may inadvertently block the GUI. In this case, you should run the analysis code in a separate "worker" thread that spawns upon clicking the "Analyze" button, keeping the GUI responsive while it executes. This might be the way to go if you want the analysis to keep running alongside the GUI indefinitely - create the worker thread for the analysis at the same time that you display the GUI to the user.
I am building an app that, when the user hits a 'run' button, generates a table of buttons.
Because this process takes a while, I want to add a popup or progress bar to alert the user that the function is running and not frozen. To do this I decided to create a popup and call my function using threading so that the screen will be updated when the function starts (as opposed to once it is done).
mythread = threading.Thread(target=run_function)
mythread.start()
The trouble is that when I call my function from the above code it works very strangely: the columns of my table are the wrong width, some of my buttons are arbitrarily empty, and others have the wrong fill color. To fix this, all I need to do is to remove the threading operation and simply call run_function()
Any idea why this is happening?
I am new to Python, so it is likely some dumb mistake, but I have no idea. What is different between a process running as a thread and its default operation?
Disclaimer: I haven't worked with Kivy.
Not every framework works well with multithreading.
But most of the GUI frameworks have an event loop which is responsible for managing user events (mouse clicks, keyboard) and queued drawing operations (widgets).
In your case if don't want the UI to be freezed, you should regularly give control to your framework's event loop.
I guess kivy.base.EventLoopBase.dispatch_input is what you need to call to show an added widget or to handle user events.
Basically I have a program that starts a new thread, then tries to do stuff in this new thread. However, the new thread seems to get stuck until the main thread reaches the end of the program.
here is a working example of the problem I'm experiencing. it seems that gtk.main() has to be there for this problem to occur. if I use something like input() instead, the problem doesn't appear.
import threading,sys
class MyThread(threading.Thread):
def run(self):
import time
print('before')
time.sleep(3)
print('after')
MyThread().start()
from gi.repository import Gtk as gtk
class My_window:
def __init__(self):
self.window = gtk.Window()
self.window.connect("delete_event", gtk.main_quit)
self.button = gtk.Button("Hello World")
self.window.add(self.button)
self.button.show()
self.window.show()
My_window()
gtk.main()
What should happens: The window appears, the word before appears, and 3 seconds later the word after appears
What really happens: The window appears, the word before appears, then nothing happens. When you close the window, the word after appears. (As if somehow reaching the end of the program makes all the running threads that were frozen before, running again)
Things I've tried:
Replaced time.sleep with other functions, like a terminal command, or a busy loop. I get the same results
I tried recreating the problem without the window and other extra stuff, but couldn't. so I don't know where this problem resides, so I gave all the background info that I could.
I tried using sys.stdout.flush() to make sure that the text was actually showing up in the command line when it should. It was.
Does anyone have any suggestions as to what the problem might be?
I'm using python 3 (and gtk for the window), and would prefer my program to be compatible on all major os's
edit:
I tried putting a print('starting') right before gtk.main(). the output showed before then starting then after. I'm thinking that calling gtk.main freezes all threads, and when gtk.main ends, then the threads resume.
edit 2:
never-mind, in my original program, the thread was created while gtk.main is running, but there could still be something in gtk.main that is freezing the threads, that is being called every once in a while.
I managed to reproduce this, and spent ages trying to work out why it was failing, until I came across this FAQ entry, which suggested putting...
import gobject
gobject.threads_init()
...at the top of the code, which seems to fix the problem.
My guess is that after calling gtk.main(), GTK is holding on to Python's Global Interpreter Lock, which is a bit naughty for default behavior, but at least there's a way to prevent it.
Note that background threads are not able to manipulate GTK objects directly, so you'll have to have them pass messages to the main thread to do that instead.
if th is the thread object you created, you should th.setDaemon(1) before starting it, otherwise the main thread won't exit without the thread you started exit first.
I have a program that I'm just adding graphics to, but I'm having trouble running my main code along with the graphics. Basically I have something like this:
def mainFunction():
while True:
run code in here
root = Tk()
board = Canvas(root, height=710, width=1000)
board_image = PhotoImage(file="/path/example.jpg")
photo = board.create_image(0,0, anchor=NW, image=board_image)
board.pack()
mainFunction
root.mainloop()
I can only run either the mainFunction or the graphics because whichever one I make run first in the code is the only that runs. It doesn't stop to allow the next code to run. There has to be a simple way to get graphics and code to run together side by side. Thanks!
Use Tk.after_idle() to register a function that will do a piece of the work required. Keep doing piece after piece until all the work is done.
Generally speaking, you cannot put an infinite loop inside a Tkinter application. Why? It's already running an infinite loop: the event loop. Your code is the equivalent of this:
while <there are more events to service>:
while True:
<run code in here>
<get the next event>
<service the event>
See the problem? You're preventing the code from ever servicing events, and events are the life blood of a GUI.
Instead, you need to take advantage of the already-running infinite loop by adding code to be run inside the loop. You do this with after (and after_idle). This will put one even on the queue. If, during the processing of that event you again call after_idle, you've effectively set up an infinite loop that works within the event loop.
For example:
def do_one_iteration(self):
<run code in here>
self.after(100, self.do_one_iteration)
Then, somewhere in your main logic, or in response to a button, you call do_one_iteration. It will do one iteration of your previously-infinite-loop. When it is done it instructs Tkinter to call itself again 100 milliseconds later. When that time period elapses your code is run, it schedules another iteration in 100 milliseconds, etc. etc. You can change the interval to whatever you want; the smaller the interval the faster your code runs, but the greater the chance that you starve the GUI for events.
Note that this only works if <run code in here> runs relatively fast. While it is running your GUI will freeze. If it can complete one iteration in a couple hundred milliseconds then the user will never know. If it takes a second or more it will be noticeable.
Note: this example assumes your main application is an object that inherits from a Tkinter widget. If that's not the case it will still work, you just have to remove the self parameter. An even better solution is to refactor your GUI to use objects -- it's a much more flexible way of implementing GUIs.
in my python program to upload a file to the internet, im using a GTK progress bar to show the upload progress. But the problems that im facing is that the progress bar does not show any activity until the upload is complete, and then it abruptly indicates upload complete. im using pycurl to make the http requests...my question is -
do i need to have a multi-threaded application to upload the file and simultaneously update the gui? or is there some other mistake that im making?
Thanks in advance!
I'm going to quote the PyGTK FAQ:
You have created a progress bar inside a window, then you start running a loop that does some work:
while work_left:
...do something...
progressbar.set_fraction(...)
You will notice that the window doesn't even show up, or if it does the progress bar stays frozen until the end of the task. The explanation is simple: gtk is event driven, and you are stealing control away from the gtk main loop, thus preventing it from processing normal GUI update events.
The simplest solution consists on temporarily giving control back to gtk every time the progress is changed:
while work_left:
...do something...
progressbar.set_fraction(...)
while gtk.events_pending():
gtk.main_iteration()
Notice that with this solution, the user cannot quit the application (gtk.main_quit would not work because of new loop [gtk.main_iteration()]) until your heavy_work is done.
Another solution consists on using gtk idle functions, which are called by the gtk main loop whenever it has nothing to do. Therefore, gtk is in control, and the idle function has to do a bit of work. It should return True if there's more work to be done, otherwise False.
The best solution (it has no drawbacks) was pointed out by James Henstridge. It is taking advantage of python's generators as idle functions, to make python automatically preserve the state for us. It goes like this:
def my_task(data):
...some work...
while heavy_work_needed:
...do heavy work here...
progress_label.set_text(data) # here we update parts of UI
# there's more work, return True
yield True
# no more work, return False
yield False
def on_start_my_task_button_click(data):
task = my_task(data)
gobject.idle_add(task.next)
The 'while' above is just an example. The only rules are that it should yield True after doing a bit of work and there's more work to do, and it must yield False when the task is done.
More than likely the issue is that in your progress callback, which is where I presume you're updating the progress bar, you're not making a call to manually update the display i.e. run through the GUI's event loop. This is just speculation though, if you can provide more code, it might be easier to narrow it down further.
The reason you need to manually update the display is because your main thread is also performing the upload, which is where it's blocking.
In python 2.x integer operands result in integer division. Try this:
#Callback function invoked when download/upload has progress
def progress(download_t, download_d, upload_t, upload_d):
print 'in fileupload progress'
mainwin.mainw.prog_bar.set_fraction(float(upload_d) / upload_t)
Yes, you probably need concurrency, and yes threads are one approach, but if you do use threads, please use an method like this one: http://unpythonic.blogspot.com/2007/08/using-threads-in-pygtk.html which will abstract away the pain, and allow you to focus on the important aspects.
(I have not repeated everything in that blog post through laziness, hence community wiki).
One option, if you are not married to pycurl, is to use GObject's IO watchers.
http://pygtk.org/pygtk2reference/gobject-functions.html#function-gobject--io-add-watch
Using this you can interleave the file upload with the normal PyGTK event loop, and even do the set_progress call in your IO watch callback. If you are offloading all the work for uploading onto pycurl this is not really feasible, but if you're just uploading a file over HTTP, io_add_watch will make using a socket for this much less painful as well.