In short, I'm trying to find a faster way to plot real time data coming through a serial input. The data looks like a coordinate (x,y) and about 40 are coming in each second. The stream will store the data in a array, using x as the index and setting y as the value for it. This portion is being threaded. While the stream can read in data immediatley, the pyqtgraph library isn't able to keep up with this speed.
Here's the portion of the code where I am plotting the data. The distances and theta variables are arrays with 6400 indexes. They have been transformed into polar values and plotted with each iteration. I added a delay there just to help keep it real-time, though it's only a temporary solution.
while True:
x = distances * np.cos(theta)
y = distances * np.sin(theta)
plot.plot(x, y, pen=None, symbol='o', clear=True)
pg.QtGui.QApplication.processEvents()
#sleep(0.025)
While it's going the way I expect it to, it's not able to plot the most recent data from the serial input. It's easily several seconds behind from the most recent reads, probably because it can not plot 6400 points every 1/40 of a second. I'm wondering if there's a way to only update 1 point rather than having to re-plot the entire scatter every time in pyqtgraph.
It may be possible to plot based on point, but if so, is there a way to keep track of each individual point? There should be no point that shares the same angle value and have different distances, and should essentially overwrite it.
I'm also wondering if there are other graphing animation libraries out there that may be a possible solution worth considering.
This is what it looks like, if you're wondering:
Threading allows you to always have data available to plot but the plot speed is bottlenecked due to the paintEvent latency for each plot iteration. From my understanding, there is no way to update 1 point per paint event with setData instead of having to replot the entire data set for each iteration. So if you have 6400, you must repaint all points even if you are updating the just data with 1 additional point.
Potential workarounds to this include downsampling your data or to only plot once every X amount of data points. Essentially, you are capped at the speed you can plot data to the screen but you can alter your data set to display the most relevant information with less screen refreshes.
Related
I have time series data consisting of x y z values of accelerometer readings. I want to be able to detect that the device was moved/rotated using these values in Python.
Following is a plot of such data -
plot of xyz values of acclerometer data
Using RMS is going to help you know if movement happened, but since you lose the individual axes values it becomes impossible to distinguish what kind of movement it is.
Movement in each direction is relatively simple as that will show up as an acceleration on that axis. For rotation, assuming you are near the earth, you will have a constant acceleration equal to -9.8m/s^2 toward the earth. As your sensor is rotated the distribution of that acceleration onto the different axes will change. That should become more clear if you look at something like an average across a period of time for each axis.
let me describe my problem, so I am creating a simple graphing calculator, the way I did it was that every y coordinate is calculated by putting it into a function f(x) then graphing the point (x, f(x)).
To make things simple for myself, whenever I wanted to shift the graph or zoom in I just adjust the dimensions of the current view and then recalculate all the new points on the screen. For example going from This to this by zooming in and shifting the screen would mean that every single point has been recalculated, for me to get the graph to look like it is formed by actual lines instead of just points I divide the width of the screen into about 1000 ~ 10000 points and plot it and if there are enough points it just looks like lines. These points are made by tuple pairs of floats.
As you could imagine there is a lot of overlap and recalculations that may be slowing down the program and so I am wondering what the best way to calculate a (x, f(x)) point, store it and anytime I change the view of the graph, if that x happens to be in view, be able to retrieve the f(x) and skip the calculation. The thing is there is going to be like thousands and thousands of these points and so I figured using list operations like "i in lst" is not efficient enough.
I am trying to make my graph as fast as possible so any suggestions would be helpful! Thanks.
I'm solving a set of coupled differential equations with odeint package from scipy.integrate.
For the integration time I have:
t=numpy.linspace(0,8e+9,5e+06)
where 5e+06 is the timestep.
I then plot the equations I have as such:
plt.xscale('symlog') #x axis logarithmic scale
plt.yscale('log',basey=2) #Y axis logarithmic scale
plt.gca().set_ylim(8, 100000) #Changing y axis ticks
ax = plt.gca()
ax.yaxis.set_major_formatter(matplotlib.ticker.ScalarFormatter())
ax.xaxis.set_major_formatter(matplotlib.ticker.ScalarFormatter())
plt.title("Example graph")
plt.xlabel("time (yr)")
plt.ylabel("quantity a")
plt.plot(t,a,"r-", label = 'Example graph')
plt.legend(loc='best')
where a is time dependent variable. (This is just one graph from many.)
However, the graphs look a bit jagged, rather than oscillatory and I obtain this error:
OverflowError: Exceeded cell block limit (set 'agg.path.chunksize' rcparam)
I'm not overly sure what this error means, I've looked at other answers but don't know how to implement the 'agg.path.chunksize'.
Also, the integration + plotting takes around 7 hours and that is with some CPU processing hacks, so I really do not want to implement anything that would increase the time.
How can I overcome this error?
I have attempted to reduce the timestep, however I obtain this error instead:
Excess work done on this call (perhaps wrong Dfun type).
Run with full_output = 1 to get quantitative information.
As the error message suggests, you may set the chunksize to a larger value.
plt.rcParams['agg.path.chunksize'] = 1000
However you may also critically reflect why this error occurs in the first place. It would only occur if you are trying to plot an unreasonably large amount of data on the graph. Meaning, if you try to plot 200000000 points, the renderer might have problems to keep them all in memory. But one should probably ask oneself, why is it necessary to plot so many points? A screen may display some 2000 points in lateral direction, a printed paper maybe 6000. Using more points does not make sense, generally speaking.
Now if the solution of your differential equations requires a large point density, it does not automatically mean that you need to plot them all.
E.g. one could just plot every 100th point,
plt.plot(x[::100], y[::100])
most probably without even affecting the visual plot appearance.
I've collected a sensor data every 5 minutes for a month (30 days).
That means, I have a timeseries data with 288*30 data points in total.
I'd like to scatterplot the data (x-axis: time, y-axis: sensor value).
the following code is for test.
import pandas as pd
from matplotlib import pyplot as plt
import numpy as np
# generate time series randomly (length: 1 month)
rng=pd.date_range("2015-11-11",periods=288*30,freq="5min")
ts=pd.Series(np.random.randn(len(rng)),rng)
nr=3
nc=1
fig=plt.figure(1)
fig.subplots_adjust(left=0.04,top=1,bottom=0.02,right=0.98,wspace=0.1,hspace=0.1)
for i in range(3):
ctr=i+1
ax=fig.add_subplot(nr,nc,ctr)
ax.scatter(ts.index,ts.values)
ax.set_xlim(ts.index.min(),ts.index.max())
plt.show()
I've generated random time series data having 288*30 observations and tried to draw it in scatter plot. However, as you can see, it is impossible to analyze the figure.
I want to redraw it satisfying the following conditions:
I want a zoomed-in version of the figure. In other words, a part of data points of some time range (e.g., 2~3 hours) is shown at once. Then, there should be enough space between adjacent points.
I want save the figure as png or pdf file. Then, if I open the file, the image (or pdf) viewer has a horizontal scroll bar which enables me to explore the whole figure.
Is there anyone who can solve it?
I do not think it will be not hard for a matplotlib expert, but quite hard for me, a beginner.
note to readers: answer changed significantly from v1 due to clarification of the question
I want a zoomed-in version of the figure. In other words, a part of data points of some time range (e.g., 2~3 hours) is shown at once. Then, there should be enough space between adjacent points.
Zooming in matplotlib is implemented with the x and y limits of the axis. So you can simply change the arguments to your call to ax.set_xlim such that the corresponding times differ by 2-3 hours or however long you want. Knowing that you have a sample every 5 minutes, since 2 hours/(5 min/sample) = 24, you could use
ax.set_xlim(ts.index.min(),ts.index.min() + 24)
to get a 2-hour range.
I want save the figure as png or pdf file. Then, if I open the file, the image (or pdf) viewer has a horizontal scroll bar which enables me to explore the whole figure.
Use savefig to save the figure to a file. Note that if you have set the axis limits using set_xlim or xlim or equivalent, this will save only the portion of the figure that is visible within the given limits. So to save the entire figure (with all data points visible), you will need to set the axis limits to the minimum and maximum values, respectively.
When you open the image/PDF file in a viewer, whether it displays a scroll bar (and how much of the figure is shown) is entirely up to the viewer. You cannot control this in Python. But you can give it some chance of showing up with a horizontal scroll bar by making the figure very large in the horizontal direction. To do so, you can pass the figsize=(width, height) keyword argument when creating the figure, or use the set_size_inches(width, height) method on an existing Figure object. The measurements are in inches in both cases. Pass a value for width that is much larger than that for height and you will get a very wide figure; for example, 40 for width and 4 for height. You'll have to experiment with these values to find which ones give your figure the proportions you want.
I am using mayavi.mlab to display 3D data extracted from images. The data is as follows:
3D camera parameters as 3 lines in the x, y, x direction around the camera center, usually for about 20 cameras using mlab.plot3d().
3D coloured points in space for about 4000 points using mlab.points3d().
For (1) I have a function to draw each line for each camera seperately. If I am correct, all these lines are added to the mayavi pipeline for the current scene. Upon mlab.show() the scene takes about 10 seconds to render all these lines.
For (2) I couldn't find a way to plot all the points at once with each point a different color, so at the moment I iterate with mlab.points3d(x,y,z, color = color). I have newer waited for this routine to finish as it takes to long. If I plot all the points at once with the same color, it takes about 2 seconds.
I already tried to start my script with fig.scene.disable_render = True and resetting fig.scene.disable_render = False before displaying the scene with mlab.show().
How can I display my data with mayavi within a reasonable waiting time?
The general principle is that vtk objects have a lot of overhead, and so you for rendering performance you want to pack as many things into one object as possible. When you call mlab convenience functions like points3d it creates a new vtk object to handle that data. Thus iterating and creating thousands of single points as vtk objects is a very bad idea.
The trick of temporarily disabling the rendering as in that other question -- the "right" way to do it is to have one VTK object that holds all of the different points.
To set the different points as different colors, give scalar values to the vtk object.
x,y,z=np.random.random((3,100))
some_data=mlab.points3d(x,y,z,colormap='cool')
some_data.mlab_source.dataset.point_data.scalars=np.random.random((100,))
This only works if you can adequately represent the color values you need in a colormap. This is easy if you need a small finite number of colors or a small finite number of simple colormaps, but very difficult if you need completely arbitrary colors.