Develop Reference

Real time data plotting from a high throughput source

I want to plot Real time in a way that updates fast.
The data I have:
arrives via serial port at 62.5 Hz
data corresponds to 32 sensors (so plot 32 lines vs time).
32points *62.5Hz = 2000 points/sec
The problem with my current plotting loop is that it runs slower than 62.5[Hz], meaning I miss some data coming in from serial port.
I am looking for any solution to this problem that allows for:
All data from serial port to be saved.
Plots the data (even skipping a few points/using averages/eliminating old points and only keeping the most recent)
Here is my code, I am using random data to simulate the serial port data.
import numpy as np
import time
import matplotlib.pyplot as plt
#extra plot debugging
hz_ = [] #list of speed
time_=[] #list for time vs Hz plot
#store all data generated
store_data = np.zeros((1, 33))
#only data to plot
to_plot = np.zeros((1, 33))
#color each line
colours = [f"C{i}" for i in range (1,33)]
fig,ax = plt.subplots(1,1, figsize=(10,8))
ax.set_xlabel('time(s)')
ax.set_ylabel('y')
ax.set_ylim([0, 300])
ax.set_xlim([0, 200])
start_time = time.time()
for i in range (100):
loop_time = time.time()
#make data with col0=time and col[1:11] = y values
data = np.random.randint(1,255,(1,32)).astype(float) #simulated data, usually comes in at 62.5 [Hz]
data = np.insert(data, 0, time.time()-start_time).reshape(1,33) #adding time for first column
store_data = np.append(store_data, data , axis=0)
to_plot = store_data[-100:,]
for i in range(1, to_plot.shape[1]):
ax.plot(to_plot[:,0], to_plot[:,i],c = colours[i-1], marker=(5, 2), linewidth=0, label=i)
#ax.lines = ax.lines[-33:] #This soluition speeds it up, to clear old code.
fig.canvas.draw()
fig.canvas.flush_events()
Hz = 1/(time.time()-loop_time)
#for time vs Hz plot
hz_.append(Hz)
time_.append( time.time()-start_time)
print(1/(time.time()-loop_time), "Hz - frequncy program loops at")
#extra fig showing how speed drops off vs time
fig,ax = plt.subplots(1,1, figsize=(10,8))
fig.suptitle('Decreasingn Speed vs Time', fontsize=20)
ax.set_xlabel('time(s)')
ax.set_ylabel('Hz')
ax.plot(time_, hz_)
fig.show()
I also tried while using
ax.lines = ax.lines[-33:]
to remove older points, and this speed up the plotting, but still slower than the speed i aquire data.
Any library/solution to make sure I collect all data and plot the general trendlines (so even not all points) is ok. Maybe something that runs acquiring data and plotting in parallel?

You could try to have two separate processes:
one for acquiring and storing the data
one for plotting the data
Below there are two basic scripts to get the idea.
You first run gen.py which starts to generate numbers and save them in a file.
Then, in the same directory, you can run plot.py which will read the last part of the file and will update the a Matplotlib plot.
Here is the gen.py script to generate data:
#!/usr/bin/env python3
import time
import random
LIMIT_TIME = 100 # s
DATA_FILENAME = "data.txt"
def gen_data(filename, limit_time):
start_time = time.time()
elapsed_time = time.time() - start_time
with open(filename, "w") as f:
while elapsed_time < limit_time:
f.write(f"{time.time():30.12f} {random.random():30.12f}\n") # produces 64 bytes
f.flush()
elapsed = time.time() - start_time
gen_data(DATA_FILENAME, LIMIT_TIME)
and here is the plot.py script to plot the data (reworked from this one):
#!/usr/bin/env python3
import io
import time
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.animation
BUFFER_LEN = 64
DATA_FILENAME = "data.txt"
PLOT_LIMIT = 20
ANIM_FILENAME = "video.gif"
fig, ax = plt.subplots(1, 1, figsize=(10,8))
ax.set_title("Plot of random numbers from `gen.py`")
ax.set_xlabel("time / s")
ax.set_ylabel("random number / #")
ax.set_ylim([0, 1])
def get_data(filename, buffer_len, delay=0.0):
with open(filename, "r") as f:
f.seek(0, io.SEEK_END)
data = f.read(buffer_len)
if delay:
time.sleep(delay)
return data
def animate(i, xs, ys, limit=PLOT_LIMIT, verbose=False):
# grab the data
try:
data = get_data(DATA_FILENAME, BUFFER_LEN)
if verbose:
print(data)
x, y = map(float, data.split())
if x > xs[-1]:
# Add x and y to lists
xs.append(x)
ys.append(y)
# Limit x and y lists to 10 items
xs = xs[-limit:]
ys = ys[-limit:]
else:
print(f"W: {time.time()} :: STALE!")
except ValueError:
print(f"W: {time.time()} :: EXCEPTION!")
else:
# Draw x and y lists
ax.clear()
ax.set_ylim([0, 1])
ax.plot(xs, ys)
# save video (only to attach here)
#anim = mpl.animation.FuncAnimation(fig, animate, fargs=([time.time()], [None]), interval=1, frames=3 * PLOT_LIMIT, repeat=False)
#anim.save(ANIM_FILENAME, writer='imagemagick', fps=10)
#print(f"I: Saved to `{ANIM_FILENAME}`")
# show interactively
anim = mpl.animation.FuncAnimation(fig, animate, fargs=([time.time()], [None]), interval=1)
plt.show()
plt.close()
Note that I have also included and commented out the portion of code that I used to generate the animated GIF above.
I believe this should be enough to get you going.

Updating a real time plot using Matplotlib

I'm trying to generate a real time plot using Matplotlib, with real time gathered data from an Arduino MKR1000.
Code:
import serial
import matplotlib.pyplot as plt
import numpy as np
ser = serial.Serial('COM7', 9600)
plt.close('all')
plt.figure(figsize=[10,8])
plt.ion()
data = np.array([])
while True:
a = ser.readline()
a = a.decode()
b = float(a[0:4])
data = np.append(data, b)
plt.plot(data)
plt.pause(0.1)
The output is a new plot created every 0.1 seconds, but I want to update the same real-time plot.
Anyone can help?
I'm using Python 3 on a Windows 10 system

speed up mplotlib when plotting live serial data

I am using an Arduino to create a 3d lidar scan of the area around it, and sending x,y,z coordinates over serial to a python script to visualize the data live. The issue I've run into is that mplotlib is much too slow to keep up with the stream. I have a feeling this is due to completely redrawing the plot, but I have not found a solution. I write s1000 over serial to start the lidar scan, and sending a x,y,z coordinate once per second.
import serial
import numpy
import matplotlib.pyplot as plt #import matplotlib library
from mpl_toolkits.mplot3d import Axes3D
from drawnow import *
from matplotlib import animation
import time
ser = serial.Serial('COM7',9600,timeout=5)
ser.flushInput()
time.sleep(5)
ser.write(bytes(b's1000'))
plt.ion()
fig = plt.figure(figsize=(16,12))
ax = fig.add_subplot(111, projection="3d")
ax.set_xlim3d(-255, 255)
ax.set_ylim3d(-255, 255)
ax.set_zlim3d(-255, 255)
x=list()
y=list()
z=list()
while True:
try:
ser_bytes = ser.readline()
data = str(ser_bytes[0:len(ser_bytes)-2].decode("utf-8"))
xyz = data.split(", ")
dx = float(xyz[0])
dy = float(xyz[1])
dz = float(xyz[2].replace(";",""))
x.append(dx);
y.append(dy);
z.append(dz);
ax.scatter(x,y,z, c='r',marker='o')
plt.draw()
plt.pause(0.0001)
except:
print("Keyboard Interrupt")
ser.close()
break

Graph figure not displayed when live plotting arduino data with matplotlib

I am trying to plot real-time lectures from Arduino UNO's analogue input with matplotlib.
My problem: The graph would not be shown. Only when I stop running the code (Ctrl+C), It will show the last values's graph.
when adding "print pData" line to the code in order to check whether the values are properly arriving to the computer, these are correctly displayed (shows the 25 values array each second) on the python terminal.
#!/usr/bin/python
from matplotlib import pyplot
import pyfirmata
from time import sleep
# Associate port and board with pyFirmata
port = '/dev/ttyACM0'
board = pyfirmata.Arduino(port)
# Using iterator thread to avoid buffer overflow
it = pyfirmata.util.Iterator(board)
it.start()
# Assign a role and variable to analog pin 0
a0 = board.get_pin('a:0:i')
pyplot.ion()
pData = [0.0] * 25
fig = pyplot.figure()
pyplot.title('Real-time Potentiometer reading')
ax1 = pyplot.axes()
l1, = pyplot.plot(pData)
pyplot.ylim([0, 1])
while True:
try:
sleep(1)
pData.append(float(a0.read()))
pyplot.ylim([0, 1])
del pData[0]
l1.set_xdata([i for i in xrange(25)])
l1.set_ydata(pData) # update the data
#print pData
pyplot.draw() # update the plot
except KeyboardInterrupt:
board.exit()
break

Here is a mock-up that uses matplotlib.animation for real-time plotting.
from matplotlib import pyplot
import matplotlib.animation as animation
import random
# Generate sample data
class Pin:
def read(self):
return random.random()
a0 = Pin()
n = 25
pData = [None] * n
fig, ax = pyplot.subplots()
pyplot.title('Real-time Potentiometer reading')
l1, = ax.plot(pData)
# Display past sampling times as negative, with 0 meaning "now"
l1.set_xdata(range(-n + 1, 1))
ax.set(ylim=(0, 1), xlim=(-n + 1, 0))
def update(data):
del pData[0]
pData.append(float(a0.read()))
l1.set_ydata(pData) # update the data
return l1,
ani = animation.FuncAnimation(fig, update, interval=1000, blit=True)
try:
pyplot.show()
finally:
pass
#board.exit()

Graph plots are slow than the real time values that are received

I am trying to get data from arduino and trying to then show the data in real time in python using subplot. The values that are coming from arduino uno board are fast and are displayed in the python console also at the same rate but when I am trying to plot the real time data in the graph it is very slowly plotted. It needs to as fast as the rate of values that are coming from the uno board.
Please help. Here is my code:
import serial
import numpy
import matplotlib.pyplot as plt
from drawnow import *
x = []
y = []
z = []
magnitude = []
arduinoData = serial.Serial('com4', 9600)
plt.ion()
count=0
fig = plt.figure()
def makeFig():
ax1 = fig.add_subplot(4,1,1)
ax1.plot(x, 'ro-', label='X axis')
ax2 = fig.add_subplot(4,1,2)
ax2.plot(y, 'b^-', label='Y axis')
ax3 = fig.add_subplot(4,1,3)
ax3.plot(z, 'gp-', label='Y axis')
ax4 = fig.add_subplot(4,1,4)
ax4.plot(magnitude, 'yo-', label='X axis')
while True:
while (arduinoData.inWaiting()==0):
pass
arduinoString = arduinoData.readline()
dataArray = arduinoString.split(',')
xaxis = float( dataArray[0])
yaxis = float( dataArray[1])
zaxis = float( dataArray[2])
mag =float( dataArray[3])
x.append(xaxis)
y.append(yaxis)
z.append(zaxis)
magnitude.append(mag)
drawnow(makeFig)
count = count + 1

nowThere are some things you must understand before you can find a good solution. How fast does the data arrive from arduino? How fast is the drawnow function? These timings are not under your control, so if the data arrives faster than the plotting routine can execute then the task as you have defined it is impossible. All Python versions have a time module, and the function time.time() returns the current time in seconds. This can be used to measure the speed of the drawnow function. You may need to cache a chunk of data before updating the plot. Updating the plot a few times a second will give the illusion of real time, and that may be good enough.
To see how fast the graph plots, use:
t = time.time()
drawnow()
print(time.time()-t) # time in seconds