I am writing an application in Python using Qt (currently PyQt) that receives telemetry (position data) over UDP and displays that data. It can handle several packet types each with several data fields each. I have a separate thread that handles receiving and decoding packets and they emits a signal to send that data to the Qt GUI for display. The data get displayed in several ways: a current value display, scrolling log display, and a 3D view display. All this runs great and in real time with each packet updating at up to about 30Hz.
I want to add another tab that shows a live plot. The user can select which data streams to plot and it will update in real time, showing a sliding window of the last 60sec of data (maybe configurable at some point). However my first approach is particularly slow. It is barely keeping up plotting only one line with our emulator that runs much slower than 30Hz.
I am using pyqtgraph to do the plotting. I have a PlotWidget instantiated in my .ui file and create a PlotDataItem for each data line that could be drawn on the plot. I am using deques to store the data to be plotted, both the value and the time. This way I can quickly add data as it comes in and remove it as it falls outside of the sliding window. I am storing all this in a dict for each packet type and field:
self.plotData[pktType][field] = {}
self.plotData[pktType][field]['curve'] = self.pwPlot.plot()
self.plotData[pktType][field]['starttime'] = time
self.plotData[pktType][field]['data'] = coll.deque()
self.plotData[pktType][field]['realtime'] = coll.deque()
self.plotData[pktType][field]['time'] = coll.deque()
'starttime' stores an initial datetime value for computing elapsed seconds. 'realtime' stores datetime values of when each packet was received (I am not currently using this, so I could drop it if it would save time). 'time' stores elapsed seconds from the 'starttime' for easier plotting and 'data' stores the values.
When a packet that comes in, I store data in the deques for each field I might want to parse. I then trim off any data outside the sliding window. Finally, the deque gets packaged in a numpy array and passed to the PlotDataItem setData method. Here's a simplified version of the code that runs for each received packet:
def updatePlot(self, time, pktData):
pktType = pktData['ptype']
keys = self.getPlottableFromType(pktType) # list of fields that can be plotted
if keys == None:
return
for k in keys:
self.plotData[pktType][k]['data'].append(pktData[k])
self.plotData[pktType][k]['realtime'].append(time)
runtime = (time - self.plotData[pktType][k]['starttime']).total_seconds()
if self.plotRate == 0:
self.plotData[pktType][k]['time'].append(runtime)
else:
if self.plotData[pktType][k]['time']: # has items
nexttime = self.plotData[pktType][k]['time'][-1] + 1. / self.plotRate
else:
nexttime = 0
self.plotData[pktType][k]['time'].append(nexttime)
while (self.plotData[pktType][k]['time'][-1] - self.plotData[pktType][k]['time'][0]) > self.plotRangeSec:
self.plotData[pktType][k]['data'].popleft()
self.plotData[pktType][k]['realtime'].popleft()
self.plotData[pktType][k]['time'].popleft()
self.drawPlot(pktType, k)
def drawPlot(self, pktType, k):
if self.plotIsEnabled(pktType, k) and self.plotData[pktType][k]['time']: # has items
npt = np.array(self.plotData[pktType][k]['time'])
npd = np.array(self.plotData[pktType][k]['data'])
self.plotData[pktType][k]['curve'].setData(npt, npd)
else:
self.plotData[pktType][k]['curve'].clear()
self.plotRate can be used to plot the data either using wall time or force the time axis to a fixed update rate. This is useful for using with the emulator since it runs slower than the real system.
First thing I should probably do is not call .clear() every time for plots that are not being plotted (just complicates the logic a little bit). Other than that, anyone have any tips for improving performance? Is the deque to numpy array a good strategy? Is there a better way to update the data since I am only changing a few values per line (adding a point and possibly removing a point or two)?
Related
I'd like to be able to change things about the slider (the value, the start/end values) programmatically.
So I take the standard slider.py demo, and just add this at the end:
for i in range(5):
amp_slider.value = amp_slider.value + 1
time.sleep(1)
That should move the value upwards every second for a few seconds. But the slider doesn't move. What am I doing wrong? Or similarly if I try to change the .end or .start value.
[I know sliders are supposed to be INPUT not OUTPUT devices. But nonetheless I'm trying to control its behavior.]
bokeh show() outputs the chart as html & javascript. Once it has done this it can no longer be modified (unless you wrote some javascript which was included to modify the page).
You need a library that renders in a 'dynamic' window (such as matplotlib to be able to replot a chart like this.
The only code inside your program that will be used again once the page is created is in the callback functions. If you adjust sliders.py so it reads:
def update_title(attrname, old, new):
amplitude.value += 1
Every time you update the text, the amplitude will increase.
Using Python to interface with Paraview, I want to get the "Points" data from an integrate variable filter.
I tried the GetArray("Points") but it can't find it even though you can clearly see it in the GUI if you go to spreadsheet view.
My code is below. With the GUI approach I get for Point ID = 0 the array "Points" has three values (0.54475, -1.27142e-18, 4.23808e-19) which makes sense because the default arrow is symmetric in y and z.
Is there any way to get the value 0.54475 inside python?
MWE
#Import Paraview Libraries
#import sys
#sys.path.append('Path\\To\\Paraview\\bin\\Lib\\site-packages')
from paraview.simple import *
#### disable automatic camera reset on 'Show'
paraview.simple._DisableFirstRenderCameraReset()
# create a new 'Arrow'
arrow1 = Arrow()
# create a new 'Integrate Variables'
integrateVariables1 = IntegrateVariables(Input=arrow1)
pdata = paraview.servermanager.Fetch(integrateVariables1).GetPointData()
print pdata.GetArray("Points") # prints None
You are very close. For all other arrays, you can access the value using the method you have written.
However VTK treats the point coordinates slightly differently, so the code you need for the point coordinates is:
arrow1 = Arrow()
integrateVariables1 = IntegrateVariables(Input=arrow1)
integrated_filter = paraview.servermanager.Fetch(integrateVariables1)
print integrated_filter.GetPoint(0)
This gives me: (0.5447500348091125, -1.2714243711743785e-18, 4.238081064918634e-19)
I would also suggest that you might want to do this in a Python Programmable Filter. Passing the filter from the server back to the client is not the best practice, and it is preferred to do all calculation on the server.
I have a class that uses IPython widgets to render rich output. This output changes over time. I manage this by adding a callback to the IOLoop that alters the output periodically.
class Foo(object):
def _ipython_display_(self, **kwargs):
html = HTML("<h2>hello</h2>")
def f():
html.value = ... # TODO: replace with weakrefs to avoid cycle
from tornado.ioloop import PeriodicCallback
html._pc = PeriodicCallback(f, 1000)
html._pc.start()
return html._ipython_display_(**kwargs)
However, I also want this PeriodicCallback to stop once html is no longer visible on the screen. Normally I would use __del__ or finalize or something, but it appears that the html object will likely be kept in the notebook's output history (variables like _10) and so my periodic callback survives forever.
Is there any way to get a signal when a displayable element is no longer displayed?
As you may know, widgets on the back end (the IPython kernel) are backed by a single object, while at the front end (the browser) it will have a model, and zero to N views.
Every time you display a widget, a new view is created. Since ipywidgets 7 it is possible to keep track of the number of views on the front end (the browser), by setting (on widget construction) the '_view_count' trait to 0, however, beware of what the help string says:
EXPERIMENTAL: The number of views of the model displayed in the frontend. This attribute is experimental and may change or be removed in the future. None signifies that views will not be tracked. Set this to 0 to start tracking view creation/deletion. The default is None, which will make it not track the view count (since for many widgets it will lead to a lot of traffic going around).
Assuming you are in the Jupyter notebook:
import ipywidgets
slider = ipywidgets.FloatSlider(value=2, _view_count=0)
assert slider._view_count == 0
For the next cell, you display it:
slider
And in the next cell you can check that the view count has gone up
assert slider._view_count == 1
Since this '_view_count' is a trait, you can also listen to changes
# use a output widget to lot lose output
output = ipywidgets.Output()
output
def view_count_changed(change):
with output:
print('View count has changes', change)
slider.observe(view_count_changed, '_view_count')
A more advanced example notebook that might be useful is here, where I make an 'ExpensivePlot' object, that will keep track of the 'dirty' state of the plot, and the _view_count to only update the plot when needed.
PS: This last example of course is not a really expensive plot, but a stripped down version of what I do in vaex when I use the bqplot for plotting, where it can take >1 second to update a plot).
I have a little app that allows me to change an input value with a tKinter scale widget and see how a graph reacts to different changes in inputs. Every time I move the scale, it's bound to an event that redoes the calculations for a list and replots. It's kind of slow.
Now, I'm replotting the entire thing, but it's stacking one axis on top of the other, hundreds after a few minutes of use.
deltaPlot = Figure(figsize=(4,3.5), dpi=75, frameon=False)
c = deltaPlot.add_subplot(111)
c.set_title('Delta')
deltaDataPlot = FigureCanvasTkAgg(deltaPlot, master=master)
deltaDataPlot.get_tk_widget().grid(row=0,rowspan=2)
and the main loop runs
c.cla()
c.plot(timeSpread,tdeltas,'g-')
deltaDataPlot.show()
It's clearing the initial plot, but like I said the axes are stacking (because it's redrawing one each time, corresponding to the slightly altered data points). Anyone know a fix?
To improve speed there are a couple of things you could do:
Either Run the remove method on the line produced by plot:
# inside the loop
line, = c.plot(timeSpread,tdeltas,'g-')
deltaDataPlot.show()
...
line.remove()
Or Re-use the line, updating its coordinates appropriately:
# outside the loop
line, = c.plot(timeSpread,tdeltas,'g-')
# inside the loop
deltaDataPlot.show()
line.set_data(timeSpread,tdeltas)
The documentation of Line2d can be found here.
You might also like to read the cookbook article on animation.
HTH
I'm attempting to do real-time plotting of data in matplotlib, in a "production" application that uses wxPython. I had been using Chaco for this purpose, but I'm trying to avoid Chaco in the future for many reasons, one of which is that since it's not well-documented I often must spend a long time reading the Chaco source code when I want to add even the smallest feature to one of my plots. One aspect where Chaco wins out over matplotlib is in speed, so I'm exploring ways to get acceptable performance from matplotlib.
One technique I've seen widely used for fast plots in matplotlib is to set animated to True for elements of the plot which you wish to update often, then draw the background (axes, tick marks, etc.) only once, and use the canvas.copy_from_bbox() method to save the background. Then, when drawing a new foreground (the plot trace, etc.), you use canvas.restore_region() to simply copy the pre-rendered background to the screen, then draw the new foreground with axis.draw_artist() and canvas.blit() it to the screen.
I wrote up a fairly simple example that embeds a FigureCanvasWxAgg in a wxPython Frame and tries to display a single trace of random data at 45 FPS. When the program is running with the Frame at the default size (hard-coded in my source), it achieves ~13-14 frames per second on my machine. When I maximize the window, the refresh drops to around 5.5 FPS. I don't think this will be fast enough for my application, especially once I start adding additional elements to be rendered in real-time.
My code is posted here: basic_fastplot.py
I wondered if this could be made faster, so I profiled the code and found that by far the largest consumers of processing time are the calls to canvas.blit() at lines 99 and 109. I dug a little further, instrumenting the matplotlib code itself to find that most of this time is spent in a specific call to MemoryDC.SelectObject(). There are several calls to SelectObject in surrounding code, but only the one marked below takes any appreciable amount of time.
From the matplotlib source, backend_wxagg.py:
class FigureCanvasWxAgg(FigureCanvasAgg, FigureCanvasWx):
# ...
def blit(self, bbox=None):
"""
Transfer the region of the agg buffer defined by bbox to the display.
If bbox is None, the entire buffer is transferred.
"""
if bbox is None:
self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None)
self.gui_repaint()
return
l, b, w, h = bbox.bounds
r = l + w
t = b + h
x = int(l)
y = int(self.bitmap.GetHeight() - t)
srcBmp = _convert_agg_to_wx_bitmap(self.get_renderer(), None)
srcDC = wx.MemoryDC()
srcDC.SelectObject(srcBmp) # <<<< Most time is spent here, 30milliseconds or more!
destDC = wx.MemoryDC()
destDC.SelectObject(self.bitmap)
destDC.BeginDrawing()
destDC.Blit(x, y, int(w), int(h), srcDC, x, y)
destDC.EndDrawing()
destDC.SelectObject(wx.NullBitmap)
srcDC.SelectObject(wx.NullBitmap)
self.gui_repaint()
My questions:
What could SelectObject() be doing that is taking so long? I had sort of assumed it would just be setting up pointers, etc., not doing much copying or computation.
Is there any way I might be able to speed this up (to get maybe 10 FPS at full-screen)?