Develop Reference

Does a GUI-class argument violate thread-safety in tkinter?

I learned the hard way that tkinter is not thread-safe when starting independent threads with tkinter functionality from the main tkinter thread. I got error messages in a (for me) non-reproducible way, mostly "main thread is not in main loop" in connection to internal del calls after I stopped my application. Sometimes the kernel crashed during or after execution, often everything just ran smoothly.
These independent threads should run data acquisitions (DAQ) at a couple of instruments, with different GUIs depending on the type of instrument. Threading seems to be feasible as it is not known from start which instrument will be needed at some time, DAQ tasks should be queued up if an instrument is busy etc.
So, my idea now is to start the DAQ threads without any tkinter functionality from the main thread. The specific DAQ thread knows which specific GUI to use and puts this specific GUI class into a queue which is handled in the main GUI/tkinter thread. The instance of the GUI class will then be created in the GUI/tkinter thread.
Will this approach still violate thread-safety or is everything ok, as long as the GUI instances are created in the main tkinter thread?

As long as you only access tkinter widgets and functions from a single thread, it should work just fine. One exception, as far as I understand, is that it's safe to call the event_genereate method from other threads. You can push data on a queue and then generate an event, then the event can be handled in the main thread where data can be pulled off the queue and processed.

Start a function from a thread. (GUI thread and QThread errors)

I've made routine with Python's threading module to run a subprocess and wait for it to finish.
I do the threading with the following line :
t1=Thread(target=self.routineFunction, args=(self, "file1.txt", "file2.txt", self.nextFunction)).start()
Inside my function routineFunction(self,file1,file2,nextFunction) I call the next function to run once the subprocess has finished running.
Everything works fine until then.
But if I create new QObject items in my next function, I receive a lot of errors :
- "QPixmap: It is not safe to use pixmaps outside the GUI thread"
- "QObject::startTimer: QTimer can only be used with threads started with QThread"
My guess is that when I call nextFunction from the routine it is ran in the same thread as the routine, hence the errors.
Is there a way to call a function from the routine inside the "main" or "normal" thread ?.
Thank you for your help.

Generally, it is okay to call functions from other threads. But many GUI libraries (QT is among them) have some restrictions on this behavior.
For example, there are designated thread called 'GUI thread' which handles all graphical stuff, like dispatching messages from OS, redrawing windows, etc. And you also restricted to work with GUI withing this only thread. So, for example, you should not create QPixmap in other threads.
QTimer uses some QThread's internal data, so you should use such timers in threads only started with QThread, but not with plain Python thread module.
Returning to your question, if you want ot work with QT, you should spawn your threads using QThread and post events to GUI thread using postEvent() method. This will guarantee consistence of QT internal data structures.
So, you can ran your code in any QT thread, but if you want to work with GUI (your QObject uses QPixmap, so it is the case), you need to handle such calls only in GUI thread.

Ok so I think that werewindle solution is possible. Unfortunatly, I found it difficult to convert my current script from Thread to QThread (I am new to threading in general).
I found a workaround :
Instead of passing nextFunction as an argument, I pass a queue = Queue.Queue() and I use queue.put(True) if my subprocess in my new thread is a success. There is no more nextFunction, I simply wait for a value in the queue with queue.get().
By doing this, I can then continue in the GUI thread.

In wxPython what should the MainThread do while a separate thread is doing some work?

I have an OnClick function that is bound to a wx.Button object and when I click it, the function is run, it creates a thread and starts doing work behind the scenes. What should I do after this on the MainThread while the other thread finishes, to make the UI remain responsive? I'm currently doing this (in the main thread):
worker_thread = Thread(target = self.DoWork, name = "Solo Worker Thread")
worker_thread.start()
while worker_thread.is_alive():
self.Update() #self being the class that inherits wx.Panel
However the GUI is still unresponsive. Am I calling the wrong method?

Your example
while worker_thread.is_alive():
# do something
will block your GUI and eat up all available CPU cycles to ask the worker_tread if it is still working. wxPython seems to be not updateable if you block the main thread, even not if you force the update with wx.Yield or ….Update().
Therefore delete your while … statement. You can do the following to determine if the spun-off thread is still alive:
Signal the conclusion of the thread by calling back to the main-thread with the thread-save wx.CallAfter added to the end of the worker
Install a wx.Timer (e.g. every 1000 milliseconds) which polls the worker_thread if it is still alive

I suggest you to use wx.CallAfter(). You may find some useful examples here. Also you can use pubsub module to send messages to your GUI. Then your GUI won't block due to other threads.
Here is a nice blog that I read when I had a similar issue like yours.
You can also find some other questions on SO based that may assist you to understand this concept here, here, & here

You don't have to do anything. As long as the long running process is happening in a different thread, it shouldn't be blocking the GUI's main loop. See the following resources for examples:
http://wiki.wxpython.org/LongRunningTasks
http://www.blog.pythonlibrary.org/2010/05/22/wxpython-and-threads/
If you take a look at those, you will see that you spin up a thread and the thread does not block the UI.

Setting up idle thread/signalling thread

I'm using Python with wxPython for writing an app.
The method I'm considering to accomplish this may not be the best - if that's the case, let me know because I'm open to refactoring.
Right now, I have one GUI form. The main program start point instantiates an instance of the GUI form then runs wx.mainLoop(), which causes the app's main initial thread to block for the lifetime of the app.
We of course know that when events happen in the UI, the UI thread runs the code for them.
Now, I have another thread - a worker thread. This thread needs to sit idle, and then when something happens in the UI thread, e.g. a button is clicked, I want the worker thread to stop idling and do something else - run a function, say.
I can't envision this right now but I could see as the app gets more complex also having to signal the worker thread while it's actually busy doing something.
I have two questions about this setup:
How can I make my worker thread idle without using up CPU time? Doing something like while True: pass will suck CPU time, while something like while True: time.sleep(0.1) will not allow instantaneous reaction to events.
What's the best way to signal into the worker thread to do something? I don't want the UI thread to execute something, I want the worker thread to be signaled, by the UI thread, that it should change what it's doing. Ideally, I'd have some way for the worker thread to register a callback with the UI itself, so that when a button is clicked or any other UI Event happens, the worker thread is signalled to change what it's doing.
So, is this the best way to accomplish this? And what's the best way to do it?
Thanks!

First: Do you actually need a background thread to sit around idle in the first place?
On most platforms, starting a new thread is cheap. (Except on Windows and Linux, where it's supercheap.) So, why not just kick off a thread whenever you need it? (It's just as easy to keep around a list of threads as a single thread, right?)
Alternatively, why not just create a ThreadPoolExecutor, and just submit jobs to it, and let the executor worry about when they get run and on which thread. Any time you can just think in terms of "tasks that need to get run without blocking the main thread" instead of "worker threads that need to wait on work", you're making your life easier. Under the covers, there's still one or more worker threads waiting on a queue, or something equivalent, but that part's all been written (and debugged and optimized) for you. All you have to write are the tasks, which are just regular functions.
But, if you want to write explicit background threads, you can, so I'll explain that.
How can I make my worker thread idle without using up CPU time? … What's the best way to signal into the worker thread to do something?
The way to idle a thread until a value is ready is to wait on a synchronization object. On any modern OS, waiting on a synchronization object means the operating system stops giving you any CPU time until the object is ready for you.*
There are a variety of different options you can see in the Threading module docs, but the obvious one to use in most cases like this is a Condition. The way to signal the worker thread is then to notify the Condition.
However, often a Queue is a lot simpler. To wait on a Queue, just call its get method with block=True. To signal another thread to wake up, just put something on the Queue. (Under the covers, a Queue wraps up a list or deque or other collection, a Lock, and a Condition, so you just tell it what you want to do—check for a value, block until there's a value, add a value—instead of dealing with waiting and signaling and protecting the collection.)
See the answer to controlling UI elements in wxPython using threading for how to signal in both directions, from a worker thread to a UI thread and vice-versa.
I'd have some way for the worker thread to register a callback with the UI itself, so that when a button is clicked or any other UI Event happens, the worker thread is signalled to change what it's doing.
You can do it this way if you want. Just pass self.queue.put or def callback(value): self.value = value; self.condition.notify() or whatever as a callback, and the GUI thread doesn't even have to know that the callback is triggering another thread.
In fact, that's a pretty nice design that may make you very happy later, when you decide to move some code back and forth between inline and background-threaded, or move it off to a child process instead of a background thread, or whatever.
I can't envision this right now but I could see as the app gets more complex also having to signal the worker thread while it's actually busy doing something.
But what do you want to happen if it's busy?
If you just want to say "If you're idle, wake up and do this task; otherwise, hold onto it and do it whenever you're ready", that's exactly what a Queue, or an Executor, will do for you automatically.
If you want to say, "If you're idle, wake up, otherwise, don't worry about it", that's what a Condition or Event will do.
If you want to say, "If you're idle, wake up and do this, otherwise, cancel what you're doing and do this instead", that's a bit more complicated. You pretty much need to have the background thread periodically check an "interrupt_me" variable while it's busy (and put a Lock around it), and then you'll set that flag as well as notifying the Condition… although in some cases, you can merge the idle and busy cases into a single Condition or Event (by calling an infinite wait() when idle, and a quick-check wait(timeout=0) when busy).
* In some cases—e.g., a Linux futex or a Windows CriticalSection—it may actually spin off a little bit of CPU time in some cases, because that happens to be a good optimization. But the point is, you're not asking for any CPU time until you're ready to use it.

How do I tell my main GUI to wait on a worker thread?

I have successfully outsourced an expensive routine in my PyQT4 GUI to a worker QThread to prevent the GUI from going unresponsive. However, I would like the GUI to wait until the worker thread is finished processing to continue executing its own code.
The solution that immediately comes to my mind is to have the thread emit a signal when complete (as I understand, QThreads already do this), and then look for this signal in the main window before the rest of the code is executed. Is this hacked?
I know QThread provides the wait() function described here , but the usage is unclear to me. I think I want to call this on the main thread, but I'm not sure how to call that in my app...?

This is a really bad plan. Split up the 'before thread action' and 'after thread action'. The 'after thread action' should be a slot fired by a QueuedConnection that the thread can signal.
Do not wait in GUI event handlers!

If the GUI thread called the wait() function on the worker thread object, it would not return from it until the worker thread returns from its main function. This is not what you want to do.
The solution you describe using signal and slots seems to make plenty of sense to me. Or you could just use a boolean.