I am making a live plotter to show the analog changes from an Arduino Sensor. The Arduino prints a value to the serial with a Baudrate of 9600. The Python code looks as following:
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import serial
import time
ser = serial.Serial("com3", 9600)
ser.readline()
optimal_frequency = 100
fig = plt.figure(figsize=(6, 6))
ax1 = fig.add_subplot(1, 1, 1)
# the following arrays must be initialized outside the loop
xar = []
yar = []
print(time.ctime())
def animate(i):
global b, xar, yar # otherwise a
for i in range(optimal_frequency):
a = str(ser.readline(), 'utf-8')
try:
b = float(a)
except ValueError:
ser.readline()
xar.append(str(time.time()))
yar.append(b)
ax1.clear()
ax1.plot(xar, yar)
ani = animation.FuncAnimation(fig, animate, interval=optimal_frequency)
plt.show()
A get an ok response time in the plot, but when I have been plotting over 20 minutes the reaction times increase to about 1 min. I.e. it takes 1 min forthe graph to get updated with the new values. I have also tried with PyQtGraph but that is delayed from a start.
Besides the delay for times over 20 minutes, I am getting some overshoots and undershoots in the plot.
Any help?
as mentioned in the comments, you are doing two things wrong:
you keep appending incoming data to your arrays, which get huge after a while
you clear your axes and create a new ax.plot() at every iteration.
what you want to do instead is:
in an initialization function, create an empty Line2D object
def init():
line, = ax.plot([], [], lw=2)
return line,
then in your update function (animate()), you use line.set_data() to update the coordinates of your points. However, to maintain performance, you have to keep the size of your arrays to a reasonable size, and so you will have to drop the older data as newer data comes in
def animate(i):
(...)
xar.append(str(time.time()))
yar.append(b)
line.set_data(xar, yar)
return line,
Check out some tutorials like this one. There are also plenty of questions on SO that already answers your question. For instance, this answer contains all the elements you need to get your code working.
As the comments refered to, reading from the arrays gets naturally slow due to their great size. With a baudrate of 9600 I get around 130 serial readings and time-stamps per second in the xar and yar variables; which after 20 min get a size of 156000. Reducing the baudrate to 2400 decreases the length of the variables to 36000, and that can be easily replotted.
Related
I am on python 3.7. I am trying to read data from a serial port, it would be 7 different bytes. Then I would like to plot each different byte on a different subplot. I want to read the serial port every 500ms and each time I read add the new data to the subplots. Every read is giving one more data to plot on every subplot. That's basically sensor reading.
Here is the code I have written:
from time import sleep
import serial
import matplotlib.pyplot as plt
f=plt.figure(1)
ax=[0 for x in range(7)]
for i in range(0,7):
ax[i]=f.add_subplot(4,2,1+i)
ser = serial.Serial('COM3', 115200) # Establish the connection on a specific port
counter = 0
byte=ser.readline() #first line not to be plotted
while True:
counter +=1
ser.write(b'9') # send a command to the arduino
byte=ser.read(7) #read 7 bytes back
for i in range(0,7):
ax[i].plot(counter, byte[i]) # Trying to plot the new values to each different subplots
plt.pause(0.01)
sleep(.5) # Delay for one half of a second
The figure is showing and the x axis and y axis are adapting to the value I want to plt but there is no data at all on the plot. If I use scatter instead of plot it works, but then it is less versatile and I can't draw te type of graph I want.
I also try to reproduce the problem without using a serial data but just displaying points of a list one after the other like that:
import matplotlib.pyplot as plt
from time import sleep
f=plt.figure()
series=[[4,3,2,1],[8,7,6,5],[12,11,10,9]]
counter=0
ax=[0 for x in range(7)]
for i in range(0,3):
ax[i]=f.add_subplot(4,2,1+i)
for j in range (0,4):
counter=counter+1
for i in range(0,3):
ax[i].plot(counter,series[i][j])
plt.pause(0.01)
sleep(1)
And it is doing exactly the same thing, the final image I have on the graph is that:
Which shows axis took what I wanted to plot but did not plot anything.
The point is I do not want to clear the full plot and redraw everything because for the data sensor I will have about 30days of data to display in continuous.
What am I doing wrong with the code I have written?
EDIT:
After comment of ImportanceOfBeingErnest I have tried implementing the answer given here. The code is then:
from time import sleep
import serial
import matplotlib.pyplot as plt
import numpy
plt.ion()
f=plt.figure()
ax=[0 for x in range(7)]
lines=[0 for x in range(7)]
for i in range(0,7):
ax[i]=f.add_subplot(4,2,1+i)
lines[i]=ax[0].plot([],[])
def update_line(hl, new_datax, new_datay):
hl.set_xdata(numpy.append(hl.get_xdata(), new_datax))
hl.set_ydata(numpy.append(hl.get_ydata(), new_datay))
plt.draw()
ser = serial.Serial('COM3', 115200) # Establish the connection on a specific port
counter = 0
byte=ser.readline() #first line not to be plotted
while True:
counter +=1
ser.write(b'9') # send a command to the arduino
byte=ser.read(7) #read 7 bytes back
for i in range(0,7):
update_line(lines[i][0], counter, byte[i]) # Trying to plot the new values to each different subplots
plt.pause(0.01)
sleep(.5) # Delay for one half of a second
But it still does not show anything. I kind of guess I am missing a plot and/or clear somewhere but after trying several options can't get it to work.
As someone who worked in an optics lab and struggled to get Matplotlib to perform real-time plotting, I feel your pain and I strongly suggest choosing something other than Matplotlib for this purpose (such as pyqtgraph).
That said, I've gotten Matplotlib to perform some real-time plotting from sensor data. I've found it to be buggy. Here are some thoughts as well as a solution that uses matplotlib:
Use dictionaries where possible.
Why? Because accessing dictionaries is fast, and I find that a dictionary key is easier to use than a list index for these purposes.
Use lists instead of NumPy arrays.
Why? Because every time you resize or append a NumPy array it must be completely rewritten as a new object in memory. This is very costly. Lists can be resized and appended for negligible cost.
The code below uses random data to simulate incoming sensor data and to make troubleshooting easier.
1. Imports
from time import sleep
import matplotlib.pyplot as plt
import numpy as np
#import serial
2. Setup your matplotlib objects and data containers
# specify how many points to show on the x-axis
xwidth = 10
# use real-time plotting
plt.ion()
# setup each of the subplots
ax = []
fig, ax[0:7] = plt.subplots(7, 1, sharex=False, sharey=False)
# set up each of the lines/curves to be plotted on their respective subplots
lines = {index: Axes_object.plot([],[])[0] for index, Axes_object in enumerate(ax)}
# cache background of each plot for fast re-drawing, AKA Blit
ax_bgs = {index: fig.canvas.copy_from_bbox(Axes_object.bbox)
for index, Axes_object in enumerate(ax)}
# initial drawing of the canvas
fig.canvas.draw()
# setup variable to contain incoming serial port data
y_data = {index:[0] for index in range(len(ax))}
x_data = [-1]
3. Write functions for update the plot and for updating your data containers
def update_data(new_byte, ):
x_data.append(x_data[-1] + 1)
for i, val in enumerate(new_byte):
y_data[i].append(val)
def update_graph():
for i in y_data.keys():
# update each line object
lines[i].set_data(x_data, y_data[i])
# try to set new axes limits
try:
ax[i].set_xlim([x_data[-1] - xwidth, x_data[-1]])
if max(y_data[i][-xwidth:]) > ax[i].get_ylim()[1]:
new_min = min(y_data[i][-xwidth:])
new_max = max(y_data[i][-xwidth:])
ax[i].set_ylim([new_min-abs(new_min)*0.2, new_max+abs(new_max)*0.2])
except:
continue
fig.canvas.draw()
4. Finally, run the loop
#ser = serial.Serial('COM3', 115200) # Establish the connection on a specific port
#byte=ser.readline() #first line not to be plotted
while x_data[-1] < 30:
# ser.write(b'9') # send a command to the arduino
# byte=ser.read(7) #read 7 bytes back
byte = np.random.rand(7)
update_data(byte)
update_graph()
sleep(.1) # Delay for an arbitrary amount of time
I hope that helps.
I'm using pyplot 2.0 with the FuncAnimation for streaming serial data to a figure where it is continously updated.
For the FuncAnimation to work you have a Figure, Func (function to call at each frame), and Frames (Source of data to pass func and each frame of the animation) fairly simple I thought and it does animate the way I would expect.
However the figure continues to chew up memory until it crashes the Raspberry Pi even though I have the blit flag set to optimize the drawing and only render the new data points.
Here is the code:
import matplotlib.pyplot as plt
import matplotlib.animation as animation
# Setup plotting, one figure and two subplots
fig, (ax1, ax2) = plt.subplots(2,1)#,sharex=True)
g_line = ax1.plot([],[],'g')
r_line = ax1.plot([],[],'r')
lines = []
# Add each line to o ur tuples of lines to be plotted
lines.extend(g_line)
lines.extend(r_line)
# Data plotting function, this is call from the animation
# Each data element is in the data list being returned below
def plot_data(data):
x1 = data[0]; y1 = data[1];
x2 = data[2]; y2 = data[3];
lines[0].set_data(x1,y1) # Audio - green
lines[1].set_data(x2,y2) # Audio thresh - red
return tuple(lines)
def data_gen():
my_list = []
my_list2 = []
# Main infinite loop for processing of serial data
while True:
# Grab input RX serial data
(my_list, my_list2) = readUART(serReader)
yield my_list, my_list2
#end while
ani = animation.FuncAnimation(fig, plot_data, data_gen, interval=1, blit=True)
plt.show()
I have tried to clear the figure after each iteration but that severly slows down the rendering.
Is there some memory i'm not aware of that is holding onto it each time it plots even when the plotting fills the figure completely and moves off the screen?
Note - I ran a memory profiler on the entire program minus the plotting and just did it on the while TRUE loop and the memory didn't move past 55mb the entire time I let it run.
I acquire some data in two arrays: one for the time, and one for the value. When I reach 1000 points, I trigger a signal and plot these points (x=time, y=value).
I need to keep on the same figure the previous plots, but only a reasonable number to avoid slowing down the process. For example, I would like to keep 10,000 points on my graph.
The matplotlib interactive plot works fine, but I don't know how to erase the first points and it slows my computer very quickly.
I looked into matplotlib.animation, but it only seems to repeat the same plot, and not really actualise it.
I'm really looking for a light solution, to avoid any slowing.
As I acquire for a very large amount of time, I erase the input data on every loop (the 1001st point is stored in the 1st row and so on).
Here is what I have for now, but it keeps all the points on the graph:
import matplotlib.pyplot as plt
def init_plot():
plt.ion()
plt.figure()
plt.title("Test d\'acqusition", fontsize=20)
plt.xlabel("Temps(s)", fontsize=20)
plt.ylabel("Tension (V)", fontsize=20)
plt.grid(True)
def continuous_plot(x, fx, x2, fx2):
plt.plot(x, fx, 'bo', markersize=1)
plt.plot(x2, fx2, 'ro', markersize=1)
plt.draw()
I call the init function once, and the continous_plot is in a process, called every time I have 1000 points in my array.
The lightest solution you may have is to replace the X and Y values of an existing plot. (Or the Y value only, if your X data does not change. A simple example:
import matplotlib.pyplot as plt
import numpy as np
import time
fig = plt.figure()
ax = fig.add_subplot(111)
# some X and Y data
x = np.arange(10000)
y = np.random.randn(10000)
li, = ax.plot(x, y)
# draw and show it
ax.relim()
ax.autoscale_view(True,True,True)
fig.canvas.draw()
plt.show(block=False)
# loop to update the data
while True:
try:
y[:-10] = y[10:]
y[-10:] = np.random.randn(10)
# set the new data
li.set_ydata(y)
fig.canvas.draw()
time.sleep(0.01)
except KeyboardInterrupt:
break
This solution is quite fast, as well. The maximum speed of the above code is 100 redraws per second (limited by the time.sleep), I get around 70-80, which means that one redraw takes around 4 ms. But YMMV depending on the backend, etc.
Use a fixed size array and plot that using matplot.
import collections
array = collections.deque([None] * 1000, maxlen=1000)
Whenver you append to the array it will remove the first element.
I know I'm late to answer this question, bt for your issue you could look into the "joystick" package. It is based on the line.set_data() and canvas.draw() methods, with optional axes re-scaling. It also allows for interactive text logging or image plotting (in addition to graph plotting).
No need to do your own loops in a separate thread, the package takes care of it, just give the update frequency you wish. Plus the console remains available for additional monitoring commands.
See http://www.github.com/ceyzeriat/joystick/ or https://pypi.python.org/pypi/joystick (use pip install joystick to install)
try:
import joystick as jk
import numpy as np
import time
class test(jk.Joystick):
# initialize the infinite loop decorator
_infinite_loop = jk.deco_infinite_loop()
def _init(self, *args, **kwargs):
"""
Function called at initialization, see the doc
"""
self._t0 = time.time() # initialize time
self.xdata = np.array([self._t0]) # time x-axis
self.ydata = np.array([0.0]) # fake data y-axis
# create a graph frame
self.mygraph = self.add_frame(jk.Graph(name="test", size=(500, 500), pos=(50, 50), fmt="go-", xnpts=10000, xnptsmax=10000, xylim=(None, None, 0, 1)))
#_infinite_loop(wait_time=0.2)
def _generate_data(self): # function looped every 0.2 second to read or produce data
"""
Loop starting with the simulation start, getting data and
pushing it to the graph every 0.2 seconds
"""
# concatenate data on the time x-axis
self.xdata = jk.core.add_datapoint(self.xdata, time.time(), xnptsmax=self.mygraph.xnptsmax)
# concatenate data on the fake data y-axis
self.ydata = jk.core.add_datapoint(self.ydata, np.random.random(), xnptsmax=self.mygraph.xnptsmax)
self.mygraph.set_xydata(t, self.ydata)
t = test()
t.start()
t.stop()
To be totally interactive, you could use Bokeh for this. Concretely you could use an update function that is called every X ms and stream the new data.
Here there is a fragment I use:
def update():
candle_data.stream(new_data, 300)
plot = figure(x_axis_type='datetime',x_range=(start_day, final_day), width=1500, height=900, title='Live Chart', sizing_mode='scale_both')
plot.segment(x0='time', y0='highest', x1='time', y1='lowest', color='black', source=candle_data)
plot.vbar(x='time', width = 0.5*60*60*50 ,bottom='open', top='close',fill_color='color', line_color='black', source = candle_data)
doc.add_root(column([plot]))
doc.add_periodic_callback(update, 20000)
doc.title = "Candle Data Live Rates"
This is a very general question.
I have a series of data with a quantity (y) versus time (x). It is a very long series and the data are sometimes pretty noisy, some times better.
I would like to write a python code that allows me to take a look at these data with a given x-range per time (just a snapshot, so to say), and then allow me to decide if I want to "store" the sequence or not. Then pass to the next sequence and do the same, and so on. So at the end I will have a stacked amount of sequences that I can analyze separately.
I need some suggestions about the graphical part: I don't have a clue of which modules I need.
Matplotlib is probably one of the best options for the graphical part. For example:
import numpy as np
import matplotlib.pyplot as plt
plt.ion()
# make some data of some size
nblocks = 10
block_size = 1000
size = block_size*nblocks
data = np.random.normal(0.,1.,size=size)
# create a matplotlib figure with some plotting axes
fig = plt.figure()
ax = fig.add_subplot(111)
# display the figure
plt.show()
# storage for blocks to keep
kept_blocks = []
for block in data.reshape(nblocks,block_size):
#plot the block
ax.plot(block)
#force matplotlib to rerender
plt.draw()
# ask user for some input
answer = raw_input("Keep block? [Y/n]")
if answer.lower() != "n":
kept_blocks.append(block)
#clear the plotting axes
ax.cla()
# turn the kept blocks into a 2D array
kept_blocks = np.vstack(kept_blocks)
#or a 1D array
#kept_blocks = np.hstack(kept_blocks)
Matplotlib is well supported and is the de facto plotting standard in python.
I'm trying to use blitting with pylab to animate my plots at a fast frame rate; the code below seems to work fine, but plots new data on top of the old rather than replotting, so that I end up with a figure filling up with lines rather than one animated line (in each subplot). Any suggestions to get a single animated line (in each subfigure) at as fast a frame rate as possible greatly appreciated.
import pylab
import time
import threading
import random
import math
time_series, cos_series, sin_series = [], [], []
MAX = 100
# This generates new data for the plot
def data_generator():
while True:
time.sleep(0.1)
ts = time.time()
time_series.append(ts)
cos_series.append(math.sin( ts ))
sin_series.append(math.cos( ts ))
if len(cos_series) > MAX:
cos_series.pop(0)
if len(sin_series) > MAX:
sin_series.pop(0)
if len(time_series) > MAX:
time_series.pop(0)
if __name__ == '__main__':
# Run the receiving function in a separate thread.
thread = threading.Thread(target=data_generator)
thread.daemon = True
thread.start()
fig = pylab.figure()
ax = fig.add_subplot(211)
bx = fig.add_subplot(212)
ax.grid(True)
bx.grid(True)
print(len(time_series),len(sin_series),len(cos_series))
fig.show()
fig.canvas.draw()
line1, = ax.plot(time_series, sin_series, '-.k', animated=True)
line2, = bx.plot(time_series, cos_series, 'r+-', animated=True)
ax.legend(['Sin(x)'])
bx.legend(['Cos(x)'])
ax.set_ylim([-1,1])
bx.set_ylim([-1,1])
background_a = [fig.canvas.copy_from_bbox(ax.bbox)]
background_b = [fig.canvas.copy_from_bbox(bx.bbox)]
timer = 0
t_start = time.time()
# Continuously update plot
while True:
time.sleep(0.1)
line1.set_data(time_series,sin_series)
ax.set_xlim([time_series[0],time_series[-1]])
line2.set_data(time_series,cos_series)
bx.set_xlim([time_series[0],time_series[-1]])
ax.draw_artist(line1)
bx.draw_artist(line2)
fig.canvas.restore_region(background_a)
fig.canvas.restore_region(background_b)
fig.canvas.blit(ax.bbox)
fig.canvas.blit(bx.bbox)
timer += 1
print('FPS = ',timer/(time.time() - t_start))
There are two problems with your code.
Firstly, when you do this:
background_a = [fig.canvas.copy_from_bbox(ax.bbox)]
background_b = [fig.canvas.copy_from_bbox(bx.bbox)]
you shouldn't put your buffer objects in a list - restore_region just takes the buffer objects directly, so you should just do this instead:
background_a = fig.canvas.copy_from_bbox(ax.bbox)
background_b = fig.canvas.copy_from_bbox(bx.bbox)
Secondly, in your rendering loop you need to restore the background before you draw any of your updated line artists on top, otherwise you'll always be drawing the background on top of your moving lines. Move those lines above your draw_artist calls, like this:
fig.canvas.restore_region(background_a)
fig.canvas.restore_region(background_b)
ax.draw_artist(line1)
bx.draw_artist(line2)
fig.canvas.blit(ax.bbox)
fig.canvas.blit(bx.bbox)
Now everything should work fine.
Update
If you want the x-axis to also be updated during the animation, things get a little bit more complicated. Firstly you'll need to set the x-axis to be animated for both sets of axes:
ax = fig.add_subplot(211)
bx = fig.add_subplot(212)
ax.xaxis.set_animated(True)
bx.xaxis.set_animated(True)
The axis bounding box (ax.bbox) doesn't contain the tick labels, so in order to get a large enough region to restore during the rendering loop you'll need to cache a larger region of the figure canvas, e.g. the whole figure bounding box:
figbackground = fig.canvas.copy_from_bbox(fig.bbox)
And to restore the background:
fig.canvas.restore_region(figbackground)
At each timepoint you need to force the x-axis to be re-drawn as well as the lines:
ax.draw_artist(line1)
bx.draw_artist(line2)
ax.xaxis.draw(fig.canvas.renderer)
bx.xaxis.draw(fig.canvas.renderer)
And finally, when you do the blitting you need to use the axes clipboxes, which contain the tick labels, rather than the bounding boxes, which do not:
fig.canvas.blit(ax.clipbox)
fig.canvas.blit(bx.clipbox)
With these changes the tick labels and the x-gridlines will get updated, but so far I haven't figured out how exactly to get the y-gridlines and the legend to be drawn correctly. Hopefully this gives you some idea of how to go about doing this.
Also tcaswell is right to suggest looking at the Animation class - for your case it might work out to be a lot simpler, although I think it's also good to have an understand of how blitting works under the hood.