Develop Reference

Keeping dcc.Intervals firing while browser tab is in background for a Plotly Dash application

Issue
I am prototyping a web-app using Dash which - among other things - performs some measurements through a serial peripheral periodically. As for now I am using a dcc.Interval component to periodically take a measurement from my sensors, and then plot and store them.
However, either using Firefox or Chrome, when the tab is not in the foreground, the performance throttling mechanisms of the web browsers greatly reduces the frequency at which the dcc.Interval components fires. At some points, the background timers even stop completely!
The issue is documented here and there.
I was able to create the following minimum working example which just counts in the console output:
import dash
from dash.dependencies import Input, Output
import dash_html_components as html
import dash_core_components as dcc
app = dash.Dash(__name__)
app.layout = html.Div([
html.Div("A div", id="my_div"),
dcc.Interval(id='my_interval', disabled=False, n_intervals=0),
])
#app.callback(Output('my_div', 'children'), [Input('my_interval', 'n_intervals')])
def update(n_intervals):
print(n_intervals)
return dash.no_update
if __name__ == '__main__':
app.run_server(debug=True, host='0.0.0.0', port=5000)
While this works fine when the tab is active, counting one by one each second (the default delay between two firings of the dcc.Interval component), it does not work anymore when the web browser switches for other tab. In particular, after a while, the interval between two firings increases.
With this very simple example it is not always the case depending on your browser and machine. But for a more complex app, the time between two callbacks while in background can reach several, up to several tens of seconds.
Question
Would you know a workaround, or a way to force the browser to consider the app as "important" thus not throttling it? The main objective would be that, at all time, the dcc.Interval components embedded in the app continue to fire at the same rate.
Workarounds
Up to now I tested three solutions, with little success:
Use dash_devices
Play an audio file to prevent the tab from going background
Place the dcc.Interval higher in the hierarchy
Using dash_devices
I stumbled on this thread and dash_devices seemed like a good idea since it uses websockets instead of HTTP requests for updates. I managed to make it work on a basic example but it needs to be tested at bigger scale.
Play an audio file
Playing audio (either at low volume or by disabling the tab sound if you do not want to listen to it) is a good solution to keep the tab in the foreground from the browser point of view.
However, if I was able to use the following snippet:
html.Audio(autoPlay=True, src='http://www.hochmuth.com/mp3/Haydn_Cello_Concerto_D-1.mp3', loop=True)
the same code using a path instead of an URL, i.e. either src='/path/to/my_audio_file.mp3' or src='file:///path/to/my_audio_file.mp3' does not seem to work (no audio is playing) and I don't know why...
I also tried something with base64 according to what I read here and there but only the first element works:
import dash
from dash.dependencies import Input, Output
import dash_html_components as html
import dash_core_components as dcc
import base64
haydn_path = "/Users/XXX/Haydn.mp3"
encoded_haydn = base64.b64encode(open(haydn_path, 'rb').read())
app = dash.Dash(__name__)
app.layout = html.Div([
html.Div("A div", id="my_div"),
dcc.Interval(id='my_interval', disabled=False, n_intervals=0),
html.Audio(autoPlay=True, src='http://www.hochmuth.com/mp3/Haydn_Cello_Concerto_D-1.mp3', loop=True),
html.Audio(autoPlay=True, src=haydn_path, loop=True),
html.Audio(autoPlay=True, src='file://' + haydn_path, loop=True),
html.Audio(autoPlay=True, src='data:audio/mp3;base64,{}'.format(encoded_haydn), loop=True),
])
#app.callback(Output('my_div', 'children'), [Input('my_interval', 'n_intervals')])
def update(n_intervals):
print(n_intervals)
return dash.no_update
if __name__ == '__main__':
app.run_server(debug=True)
Of the four audio players displayed, only the first one is able to play something (the other are greyed as if no audio file was specified).
Notes:
for the solution involving base64, I only adapted from what I saw, I may have made a mistake at some point...
I know this solution is a dirty fix, but at least it partially works.
Place the dcc.Interval higher in the hierarchy
Positioning the dcc.Interval component higher in the hierarchy seemed to work in certain cases, as reported here. However I tried it without success.
Conclusion and Outlooks
I would thus be very thankful to anyone who could help me with this. Either by:
finding a way to fire the dcc.Interval elements for background tabs,
finding a way to play locally stored audio file with an html.Audio component
finding another way round to have a Dash application on the one hand, and some periodically fired event on the other hand, while linking the two of them...
Originally asked 09/04/2021
Update as of 13/04/2021
According to emher's answer, I re-oriented toward a separate architecture with two different threads:
One thread performing periodic (using a while True / time.sleep() routine) pooling of my serial peripheral, written in pure Python, and storing the result of its pooling inside a global variable.
Another thread running the Dash application and periodically (using dcc.Interval) reading inside the afore-mentioned global variable.
Below is a minimal working example:
import dash
from dash.dependencies import Input, Output
import dash_html_components as html
import dash_core_components as dcc
import threading
import time
counter = 0
app = dash.Dash(__name__)
class DashThread(threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run(self):
global counter
global app
app.layout = html.Div([
dcc.Interval(id='my_interval', disabled=False, n_intervals=0),
html.Div("Counter :", style={"display": "inline-block"}),
html.Div(children=None, id="cnt_val", style={"display": "inline-block", "margin-left": "15px"}),
])
#app.callback(Output('cnt_val', 'children'), [Input('my_interval', 'n_intervals')])
def update(_):
return counter
app.run_server(dev_tools_silence_routes_logging=True) # , debug=True)
class CountingThread(threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run(self):
global counter
while True:
counter += 1
print(counter)
time.sleep(1)
a = DashThread("The Dash Application")
b = CountingThread("An Independent Thread")
b.start()
a.start()
a.join()
b.join()
Please note the commented-out debug=True argument at the end of line:
app.run_server(dev_tools_silence_routes_logging=True) # , debug=True)
This is because the way Dash calls Flask does not allow for debug mode to be enabled when the Dash application is launched from a thread which is not the main one. The complete issue is documented here.
This script just counts one every second, no matter if the Dash application is loaded in a browser, or in the foreground. While the script is still running, so will the counter, and opening the Dash application in a tab or bringing it to the foreground will only update the counter's display.
Disclaimer: contrary to the dcc.Interval method presented when the question was originally asked, the code above will progressively de-sync. Indeed, using dcc.Interval, the associated callback is called every 1.0 s, no matter whether the previously called callback finished running or not.
So if we run the program at t=0s and supposing that the tab is in foreground, we will see the following calls:
t=0s : callback()
t=1s : callback()
t=2s : callback()
t=3s : callback()
[...]
At the opposite, using the multithreaded approach, given the code that we want to run (above: counter += 1 ; print(counter)) takes an execution time dt, we will see the following calls:
t=0s : callback()
t=1s + dt : callback()
t=2s + 2*dt : callback()
t=3s + 3*dt : callback()
[...]
That is, the execution chain is progressively de-syncing from the expected "one callback per second" behavior. This can be tricky in certain situation, in which case please refer here for a workaround.

For this kind of use case, I would generally prefer an architecture where a separate process collects the data and inserts in into a cache (e.g. Redis) from which the UI (in your case Dash) reads the data. This solution is much more robust than fetching the data directly from the UI.
If you insist on fetching the data directly from UI, I would suggest using a Websocket component instead of the Interval component. While I haven't tested extensively, it is my impression that the connection will stay alive on most platforms.

How can I use async Python to allow Jupyter button widget clicks to process while running an async recursive function?

I set up a Jupyter Notebook where I'm running a recursive function that clears the output like so, creating an animated output:
from IPython.display import clear_output
active = True
def animate:
myOutput = ""
# do stuff
clear_output(wait=True)
print(myOutput)
sleep(0.2)
if active:
animate()
and that's working perfectly.
But now I want to add in one more step: A speed toggle. What I'm animating is a debugging visualization of a cursor moving through interpreted code as an interpreter I'm writing parses that code. I tried conditional slow-downs to have more time to read what's going on as the parsing continues, but what I really need is to be able to click a button to toggle the speed between fast and slow. Maybe I'll use a slider, but for now I just want a button for proof of concept.
This sounds simple enough. Note that I'm writing this statefully as a class because I need to read / write the state from within another imported class.
Jupyter block 1:
import ipywidgets as widgets
from IPython.display import display
out = widgets.Output()
class ToggleState():
def __init__(self):
self.button = widgets.Button(description="Toggle")
self.button.on_click(self.toggle)
display(self.button)
self.toggleState = False
print("Toggle State:", self.toggleState)
def toggle(self, arg): # arg has to be accepted here to meet on_click requirements
self.toggleState = not self.toggleState
print("Toggle State:", self.toggleState)
def read(self):
return self.toggleState
toggleState = ToggleState()
Then, in Jupyter block 2, note I decided to to do this in a separate block because the clear_output I'm doing with the animate func clears the button if it's in the same block, and therein lies the problem:
active = True
def animate:
myOutput = ""
# do stuff
clear_output(wait=True)
print(myOutput)
if toggleState.read():
sleep(5)
else:
sleep(0.2)
if active:
animate()
But the problem with this approach was that two blocks don't actually run at the same time (without using parallel kernels which is way more complexity than I care for) so that button can't keep receiving input in the previous block. Seems obvious now, but I didn't think about it.
How can I clear input in a way that doesn't delete my button too (so I can put the button in the animating block)?
Edit:
I thought I figured the solution, but only part of it:
Using the Output widget:
out = widgets.Output()
and
with out:
clear_output(wait=True) # clears only the logged output
# logging code
We can render to two separate stdouts within the same block. This works to an extent, as in the animation renders and the button isn't cleared. But still while the animation loop is running the button seems to be incapable of processing input. So it does seem like a synchronous code / event loop blocking problem. What's the issue here?
Do I need an alternative to sleep that frees up the event loop?
Edit 2:
After searching async code in Python, I learned about asyncio but I'm still struggling. Jupyter already runs the code via asyncio.run(), but the components obviously have to be defined as async for that to matter. I defined animate as async and tried using async sleeps, but the event loops still seems to be locked for the button.

Creating Toplevel widgets that are thread safe

I'm trying to learn how to use the thread module. I followed along with the instructions here: http://effbot.org/zone/tkinter-threads.htm
My hope is the test script will:
Print out the "count" every two seconds
Show a pop-up dialog window (also every 2 seconds)
The pop-ups should be allowed to accumulate (if I don't click "OK" for a while, there should be
multiple pop-ups)
However, when I run this script it will freeze the main window and after a while crash. I think I'm not implementing the thread module correctly.
Could someone please have a look and point out what I'm doing wrong?
Here is what I've tried so far:
from Tkinter import *
import thread
import Queue
import time
class TestApp:
def __init__(self, parent):
self.super_Parent = parent
self.main_container = Frame(parent)
self.main_container.pack()
self.top_frame = Frame(self.main_container)
self.top_frame.pack(side=TOP)
self.bottom_frame = Frame(self.main_container)
self.bottom_frame.pack(side=TOP)
self.text_box = Text(self.top_frame)
self.text_box.config(height=20, width=20)
self.text_box.pack()
self.queue = Queue.Queue()
self.update_me()
def show_popup(self):
self.my_popup = Toplevel(self.main_container)
self.my_popup.geometry('100x100')
self.popup_label = Label(self.my_popup, text="Hello!")
self.popup_label.pack(side=TOP)
self.pop_button = Button(self.my_popup, text="OK", command=self.my_popup.destroy)
self.pop_button.pack(side=TOP)
def write(self, line):
self.queue.put(line)
def update_me(self):
try:
while 1:
line = self.queue.get_nowait()
if line is None:
self.text_box.delete(1.0, END)
else:
self.text_box.insert(END, str(line))
self.text_box.see(END)
self.text_box.update_idletasks()
except Queue.Empty:
pass
self.text_box.after(100, self.update_me)
def pipeToWidget(input, widget):
widget.write(input)
def start_thread():
thread.start_new(start_test, (widget,))
def start_test(widget):
count = 0
while True:
pipeToWidget(str(count) + "\n", widget)
count += 1
time.sleep(2)
widget.show_popup()
root = Tk()
widget = TestApp(root)
start_button = Button(widget.bottom_frame, command=start_thread)
start_button.configure(text="Start Test")
start_button.pack(side=LEFT)
root.title("Testing Thread Module")
root.mainloop()

I can't reproduce your problem, but I can see why it would happen.
You're using the queue to pass messages from the background thread to the main thread for updating text_box, which is correct. But you're also calling widget.show_popup() from the background thread, which means it creates and displays a new Toplevel in the background thread. That's not correct.
All UI code must run in the same thread—not all UI code for each top-level window, all UI code period. On some platforms, you may get away with running each window in its own thread (or even free-threading everything), but that isn't supposed to work, and definitely will crash or do improper things on some platforms. (Also, that single UI thread has to be the initial thread on some platforms, but that isn't relevant here.)
So, to fix this, you need to do the same dance for creating the popups that you do for updating the textbox.
The obvious way to do that is to move the widget.show_popup() to the loop in update_me(). If you want it to happen 2 seconds after the textbox updates, just add self.top_frame.after(2000, self.show_popup) to the method.
But I'm guessing you're trying to teach yourself how to have multiple independent updating mechanisms, so telling you "just use a single update queue for everything" may not be a good answer. In that case, just create two queues, and a separate update method servicing each queue. Then, do your pipeToWidget, sleep 2 seconds, then pipeToPopup.
Another way around this is to use mtTkinter. It basically does exactly what you're doing, but makes it automatic, pushing each actual Tk GUI call onto a queue to be run later by the main loop. Of course your objects themselves have to be thread-safe, and this also means that you have to deal with the GUI calls from one thread getting interleaved with calls from another thread. But as long as neither of those is a problem (and they don't seem to be in your case), it's like magic.
If you want to know why this is freezing and/or crashing for you on Win7 and not for me on OS X 10.8… well, you really need to look into a mess of Tcl, C, and Python code, and also at how each thing is built. And, unless it's something simple (like your Tk build isn't free-threaded), it wouldn't tell you much anyway. The code isn't supposed to work, and if it seems to work for me… that probably just means it would work every time until the most important demo of my career, at which point it would fail.

How to stop a warning dialog from halting execution of a Python program that's controlling it?

Using Win32GUI and Watsup, I'm writing a bit of Python code to automate a search across a database that is accessed through a program that doesn't come with an interface for it. As such, I can take a string from a list and then input it into the search box and press 'lookup'.
However, when the search returns more than 1000 results, the program throws a warning dialog --which is simply a notification of the number of results--which halts the execution of the Python code. I can't get the code to progress past the line where it presses lookup.
At a guess, this would be because it doesn't expect a window or know how to handle a warning--but I don't either, other than manually accepting it. Below is the relevent sample of code, though it's probably not very enlightening. After "clickButton(LookupButton)", the execution halts.
LookupButtonlocation = elemstring.find("Lookup", AuthNameFieldlocation) - 15
#Use Regex search to find handles
number_regex = re.compile(';(\d+);')
AuthNameEdit = int(number_regex.search(elemstring[AuthNameFieldlocation:]).group(1))
LookupButton = int(number_regex.search(elemstring[LookupButtonlocation:]).group(1))
#Input new Author into Edit Field
setEditText(AuthNameEdit, "John Campbell")
#Click lookup button
clickButton(LookupButton)

I'm not a WATSUP user, but I do something very similar using pywinauto - in my case I'm running a number of automated tests that open various 3rd party programs that, in a similar way, throw up inconvenient warning dialogs. It's a bit difficult to deal with dialogs that you don't know about, however if you do know which dialogs appear, but not when they appear, you can start a thread to just deal with those pop-ups. The following is a simple example from what I'm doing, and uses pywinauto but you could adapt the approach for WATSUP:
import time
import threading
class ClearPopupThread(threading.Thread):
def __init__(self, window_name, button_name, quit_event):
threading.Thread.__init__(self)
self.quit_event = quit_event
self.window_name = window_name
self.button_name = button_name
def run(self):
from pywinauto import application, findwindows
while True:
try:
handles = findwindows.find_windows(title=self.window_name)
except findwindows.WindowNotFoundError:
pass #Just do nothing if the pop-up dialog was not found
else: #The window was found, so click the button
for hwnd in handles:
app = application.Application()
app.Connect(handle=hwnd)
popup = app[self.window_name]
button = getattr(popup, self.button_name)
button.Click()
if self.quit_event.is_set():
break
time.sleep(1) #should help reduce cpu load a little for this thread
Essentially this thread is just an infinite loop that looks for a pop-up window by name, and if it finds it, it clicks on a button to close the window. If you have many pop-up windows you can open one thread per popup (bug that's not overly efficient, though). Because it's an infinite loop, I have the thread looking to see if an event is set, to allow me to stop the thread from my main program. So, in the main program I do something like this:
#Start the thread
quit_event = threading.Event()
mythread = ClearPopupThread('Window Popup Title', 'Yes button', quit_event)
# ...
# My program does it's thing here
# ...
# When my program is done I need to end the thread
quit_event.set()
This is not necessarily the only way to deal with your issue, but is a way that's worked for me. Sorry I can't really help you much with dealing with WATSUP (I always found pywinauto a bit easier to use), but I noticed on the WATSUP homepage (http://www.tizmoi.net/watsup/intro.html), Example 2 does something similar without using threads, i.e., looks for a named window and clicks a specific button on that window.

ttk progress bar freezing

I want a progress bar that shows the user the download progress. When updating the GUI and downloading at the same time the progress bar freezes, and I understand why but I don't know how to solve it. I tried multithreading using this post:
Tkinter: How to use threads to preventing main event loop from “freezing” and using The Basics of Python Multithreading and Queues as a guid to help me fit it to my needs. The problem is that which way I try to achieve my goal, I always seem to make a mistake when changing it to do what I need it to do.
The most basic version of my code (without multithreading):
from Tkinter import *
import ttk
from urllib import URLopener # Downloading files
# Make frame to tell user what file is getting downloaded
self.Progressmsg = Label(self, text="TempValue")
self.Progressmsg.pack(pady=(10,0))
# Make progress bar to show user download progress
self.Progressbar = ttk.Progressbar(self, mode="determinate", orient='horizontal', lengt=280, maximum=len(self.AllClasses))
self.Progressbar.pack(padx=10, pady=10)
self.Progressbar["value"] = 0
def DownloadFile(Class):
# Update progress message
self.Progressmsg["text"] = "Downloading {0}.ics...".format(Class)
# Download each file from saxion website
CalFile = URLopener()
CalFile.retrieve("http://[school website]/ical/group/{0}.ics".format(Class), "Data/{0}.ics".format(Class))
# Update progress bar
self.Progressbar["value"] += 1
for Study in self.Parameters["Classes"]:
for Class in Study:
DownloadFile(Class)
Notes: In this code AllClasses is a list of different classes from which a calendar file has to be downloaded.
The code itself is part of a fairly large class which I didn't include. This is why I am using self.[variablename]
When this code runs the progressbar doesn't load or update, all the files download properly and when they are downloaded the progress bar updates everything at once. My question is: how do I solve this problem in my case?

Try this:
# Update progress bar
self.Progressbar["value"] += 1
self.Progressbar.update_idletasks()
If it does not work then use self.Progressbar.update() instead.
The GUI won't reflect your changes if there is something else to do (like downloading the next file) unless you call update_idletasks() or update().