Develop Reference

What mechanisms does the Python interpreter use to be able to do event-driven programming?

I'm new to event-driven programming and I would really like to understand better what happens under the hood, when the CPython interpreter goes through the code line by line. So far I have only programmed sequentially, and I have a fairly good idea in mind how the interpreter converts my code into bytecode and then goes from one statement to the next and executes the commands.
But for event-driven programming, I'm totally confused how the interpreter works.
In particular, I'm confused
how the interpreter knows where to jump next in the source code
also how function handlers are called when certain events happen
how the refresh rate of the event loop is handled: Is actually all the code of the function handles run thousand of times per second, but not executed because some kind of "event-has-not-happened" flag say "don't execute this function now"?
To make this discussion more concrete, could you illustrate these points on the following example, taken from this site:
from Tkinter import *
ROOT = Tk()
def ask_for_userinput():
user_input = raw_input("Give me your command! Just type \"exit\" to close: ")
if user_input == "exit":
ROOT.quit()
else:
label = Label(ROOT, text=user_input)
label.pack()
ROOT.after(0, ask_for_userinput)
LABEL = Label(ROOT, text="Hello, world!")
LABEL.pack()
ROOT.after(0, ask_for_userinput)
ROOT.mainloop()
Ideally I'd like an explanation in a similar spirit to this article, where it is brilliantly explained from the point of view of how the CPython interpreter works, why some statements are thread-safe and some not, and how thread-safety is achieved.

All that an event loop does is call other functions when an event takes place. The graphical subsystem helps out here, by signalling to the event loop that events are waiting to be processed.
Events like keyboard input and mouse interactions (moving the pointer, clicking) are all handled by the graphical subsystem (GUI) and the operating system (OS). Keyboards and mice are hardware devices, and computers use interrupts to record their state for the GUI to pick up.
If you don't touch your keyboard or mouse, an event loop can just do nothing; the loop blocks, and the OS will execute other processes as the loop has signalled it is waiting for something to happen. The OS is in control at this point, the process is not given any CPU time and other processes run instead. Once something happens, then there are events in the queue and the OS can resume the process. Imagine a function call in the event loop that asks if there are more events, and that call won't return until there are.
Once the loop resumes, there are events in the queue to process ('mouse position is now x, y', 'the keyboard input queue contains the characters F, O, and O'). Each event can trigger code you wrote, and registered to be run on that event. For example, you can register a handler to be run when a button is clicked; the event framework has a registry that if the conditions are right ('mouse button click' event happened, cursor is at the right location on the screen, button is active and visible) so knows to call your custom event handler.
Such an event handler is entirely synchronous, if the handler takes a long time to complete you'll notice that your GUI 'freezes', does nothing else, as Python is too busy running that one handler. The usual work-around is to use threads in that case; your event handler quickly starts a separate thread to do the real work, and returns. That way the main thread (with the event loop) can handle the next event, while the OS switches between the work in the extra thread and the main thread.
As to the specific piece of code you posted, that's actually not a very good example to use. It actively ignores GUI input, using the raw_input() function to capture keyboard input from the console instead. The GUI is entirely blocked every time the function runs!
The ask_for_userinput() function is an event handler, it is registered as one with the after() method. after() uses a timer interrupt, (usually implemented with a SIGALRM interrupt) to be called after at least 0 seconds have passed (so as soon as possible, really). Each time it is called it adds a new label to the GUI (just a piece of text) and re-schedules itself. It is not very interesting!

How to integrate killable processes/thread in Python GUI?

Kind all, I'm really new to python and I'm facing a task which I can't completely grasp.
I've created an interface with Tkinter which should accomplish a couple of apparently easy feats.
By clicking a "Start" button two threads/processes will be started (each calling multiple subfunctions) which mainly read data from a serial port (one port per process, of course) and write them to file.
The I/O actions are looped within a while loop with a very high counter to allow them to go onward almost indefinitely.
The "Stop" button should stop the acquisition and essentially it should:
Kill the read/write Thread
Close the file
Close the serial port
Unfortunately I still do not understand how to accomplish point 1, i.e.: how to create killable threads without killing the whole GUI. Is there any way of doing this?
Thank you all!

First, you have to choose whether you are going to use threads or processes.
I will not go too much into differences, google it ;) Anyway, here are some things to consider: it is much easier to establish communication between threads than betweeween processes; in Python, all threads will run on the same CPU core (see Python GIL), but subprocesses may use multiple cores.
Processes
If you are using subprocesses, there are two ways: subprocess.Popen and multiprocessing.Process. With Popen you can run anything, whereas Process gives a simpler thread-like interface to running python code which is part of your project in a subprocess.
Both can be killed using terminate method.
See documentation for multiprocessing and subprocess
Of course, if you want a more graceful exit, you will want to send an "exit" message to the subprocess, rather than just terminate it, so that it gets a chance to do the clean-up. You could do that e.g. by writing to its stdin. The process should read from stdin and when it gets message "exit", it should do whatever you need before exiting.
Threads
For threads, you have to implement your own mechanism for stopping, rather than using something as violent as process.terminate().
Usually, a thread runs in a loop and in that loop you check for a flag which says stop. Then you break from the loop.
I usually have something like this:
class MyThread(Thread):
def __init__(self):
super(Thread, self).__init__()
self._stop_event = threading.Event()
def run(self):
while not self._stop_event.is_set():
# do something
self._stop_event.wait(SLEEP_TIME)
# clean-up before exit
def stop(self, timeout):
self._stop_event.set()
self.join(timeout)
Of course, you need some exception handling etc, but this is the basic idea.
EDIT: Answers to questions in comment
thread.start_new_thread(your_function) starts a new thread, that is correct. On the other hand, module threading gives you a higher-level API which is much nicer.
With threading module, you can do the same with:
t = threading.Thread(target=your_function)
t.start()
or you can make your own class which inherits from Thread and put your functionality in the run method, as in the example above. Then, when user clicks the start button, you do:
t = MyThread()
t.start()
You should store the t variable somewhere. Exactly where depends on how you designed the rest of your application. I would probably have some object which hold all active threads in a list.
When user clicks stop, you should:
t.stop(some_reasonable_time_in_which_the_thread_should_stop)
After that, you can remove the t from your list, it is not usable any more.

First you can use subprocess.Popen() to spawn child processes, then later you can use Popen.terminate() to terminate them.
Note that you could also do everything in a single Python thread, without subprocesses, if you want to. It's perfectly possible to "multiplex" reading from multiple ports in a single event loop.

Writing to a serial link continuously from a GUI program: need to use threads?

I've written a GUI program with PyQt4 that has to send a message string over a serial data link.
I have implemented a GUI interface with two button widgets. I need to send the data over the serial link continuously, once per second when the first button is clicked, and then stop when the second button is clicked.
My current program is able to send data only at the instant a button is clicked. This is the method I wrote to handle the button click signal:
def sendMessage(self):
while 1:
print "Hello........"
if checke == False:
break
Do I need to use threads to solve this problem?

It depends... if the send operation is fast, you can use the QTimer class. It integrates with the Qt event loop so you don't have to worry about threading issues. Serial communications can be slow, depending on how much data you are sending, so I can't say for sure if this is the right solution for you.

Yes. The key to GUI programming is never do any long operation on the main thread, because it'll block the whole program until that operation is complete.
If you want to continuously send data over network, do it in a background thread.
Some example code for you.
class MessageWorker(QtCore.QThread):
def __init__(self):
super(ParserWorker, self).__init__()
self.ok_to_send = False
self.terminated = True
def run(self):
while not self.terminated:
if self.ok_to_send:
self.send_message()
time.sleep(1)
def start_send():
self.ok_to_send = True
def pause_send():
self.ok_to_send = False
def terminated():
self.terminated = False
Then in the main program just call
worker = MessageWorker()
worker.start() # Start the background thread
worker.start_send() # Start sending message
worker.pause_send() # Pause sending message
worker.terminated() # Stop sending message permanently

Yes, you need to use threads. In any GUI-based program, any work that's going to take a non-trivial amount of time should always happen on a separate thread to avoid blocking the UI—whenever you see an "unresponsive" program, that's almost always due to the program failing to process window messages because its UI thread is blocked inside some long operation.
One easy way to startup a background thread is to use the threading module. Here's how you might use it to write data to the serial port once per second:
class MyClass:
# This method will run on a separate thread
def _serial_port_worker(self):
while self._run_worker:
self.send_data_to_serial_port()
time.sleep(1)
# Call this to start the worker thread
def start_worker_thread(self):
self._run_worker = True
worker_thread = threading.Thread(target=self._serial_port_worker,
args=(self,))
worker_thread.start()
# Call this to tell the worker thread to stop
def stop_worker_thread(self):
self._run_worker = False

Basically you have three options:
Use a second thread to do the serial comms. GUI toolkits aren't always thread-safe, so you should only make calls to them from the main thread. Additionally, there is a limitation to threading in Python; only one thread at a time can be executing Python bytecode.
Use the GUI toolkit's timeout function (might be called differently) to create an event every now and then. In the event handler do the serial comms. Make sure that you use non-blocking reads and writes (in pyserial, configure a timeout in the Serial object), otherwise your app might become unresponsive.
Do the serial communications from the second program using the multiprocessing module. Even if the second process blocks, it won't affect the GUI. You can use multiprocessing.Queue to communicate between the QUI and the other process.

QT/PySide processEvents call locks up

Why would a call to processEvents block doing nothing for up to 9 seconds?
I have an application with a PySide-based QT interface, where the UI code sits as a decoupled layer over the lower level actual application logic. When the user performs an action which executes lower level application logic that may run for a while, effectively what happens is:
Directly on the GUI thread a progress dialog is displayed.
Directly on the GUI thread, the lower level logic starts a worker thread.
Directly on the GUI thread, the lower level logic loops updating the progress dialog (indirectly/decoupled) and ticking the application event queue via QtGui.qApp.processEvents() (again indirectly/decoupled).
On the worker thread, QT functions are invoked (again indirectly/decoupled) in reaction to events, and these happen on the GUI thread via slots/signals, running when the GUI thread (as mentioned above) calls processEvents().
Directly on the GUI thread, before the loop exits, the last processEvents() call blocks for around 9 seconds. This is after all the logic on the worker thread is over and done with, and there are no more functions waiting to run on it via signal/slot calls. Nothing related to my application is happening in this call. What is it doing there? Why is it blocking? I've tried passing in a max processing time of 300 ms and seeing if it exits, but this makes no difference. The call locks up as long as it wants to.
The progress dialog closes and the user gets focus back.
This is all spread over a lot of files, implemented in a decoupled manner. I'll try and provide snippets to give a picture of the flow.
The decoupled lower level logic worker loop:
while not completed_event.wait(0.1) and not work_state.is_cancelled():
work_completeness, work_description = work_state.get_completeness(), work_state.get_description()
for client in self.clients:
if work_completeness != last_completeness or work_description != last_description:
client.event_prolonged_action_update(client is acting_client, work_description, step_count * work_completeness)
# THE LAST CALL TO THE NEXT LINE LOCKS UP FOR NO REASON
client.event_tick(client is acting_client)
last_completeness, last_description = work_completeness, work_description
The PySide/QT layer client event_tick function:
def event_tick(self, active_client):
# THIS IS WHERE THE LOCK UP HAPPENS
QtGui.qApp.processEvents()
Signal/slot usage in the PySide/QT layer to get worker thread calls happening on the GUI thread:
def event_pre_line_change(self, active_client, line0, line_count):
self.pre_line_change_signal.emit((line0, line_count))
def event_post_line_change(self, active_client, line0, line_count):
self.post_line_change_signal.emit((line0, line_count))
def event_uncertain_reference_modification(self, active_client, data_type_from, data_type_to, address, length):
self.uncertain_reference_modification_signal.emit((data_type_from, data_type_to, address, length))
The reason that I delegate the calls on the worker thread over to the GUI thread using signals/slots, is that this is what PySide/QT requires given they will update the UI.
Reproduction case if you want to dig deeper:
Download and get the code running according to the GitHub project readme text.
Download the test file "NEURO" (<200KB in size) from mega (apologies, it was easiest site to upload to).
Load NEURO in PeaSauce.
Go to offset 0x1A19E (CTRL-G)
Change the data type to code (Menu: Edit / Change address datatype / Code)
Observe progress dialog come and go.
Observe ~9 second lock up.

How do I make a button in my tk UI that makes the program do a whole ton of stuff when it is pressed not freeze the window?

I'm just getting started with Tk and using it in python. I set up a button that does a ton (like two minutes worth) of stuff behind the scenes when you press it. I also set up a status text to show what's going on while this is happening. I set it up like this:
status_text = Tkinter.StringVar()
ttk.Button(mainframe, text="Stats!", command=go).grid(column=1, row=4)
def go(*args):
status_text.set("Logging in...")
do_lots_of_stuff()
status_text.set("Doing stuff #1...")
do_even_more_stuff()
status_text.set("Success!")
The problem is that when you press that button the window freezes, and the status text never actually changes. It looks broken, and doesn't come out of this state until all the processing finishes 2-3 minutes later. How do I make it not freeze and update the status text?

It's time to learn multithreading!
What's happening is that the GUI (main thread) is waiting for the method to return so that it can continue updating the interface.
You'll want to cause the action of a button to spawn a threading.Thread instead of running the heavy code in the main thread. You'll also want to create a Queue to access the data from the other thread (since sending GUI requests should ONLY be made from the main thread).
import threading, Queue #queue in 3.x
my_queue = Queue.Queue()
def go(*args):
my_thread = threading.Thread(target=function_that_does_stuff)
def function_that_does_stuff():
my_queue.put("Logging in...")
do_lots_of_stuff()
my_queue.put("Doing stuff #1...")
do_even_more_stuff()
my_queue.put("Success!")
Then you'll need a function that is run when the update event happens.
def OnUpdate(*args):
try:
status_text.set(my_queue.get())
except Queue.Empty:
pass

If you have control of do_lots_of_stuff and you can break it into small chunks, you can place small jobs on the event queue to do each chunk.
For example, if you're do_lots_of_stuff is reading lines from a file, create a method that reads one line and then puts a job on the event queue to call itself after a ms or two. This way the event loop continues to run, and your code does a little processing on each iteration. This is a surprisingly effective technique, though it only works if you're able to break your 'lots of stuff' into atomic chunks.
If you can't do that you'll have to either use multiple threads or multiple processes. I personally prefer the latter -- multiprocessing code is (arguably) less difficult to write and debug.