I have a large dataset (~30GB) that I want to visualize by looking at it scrolling past. A great example is the top graph in this video.
My data is coming from CSV files.
What I have tried so far is importing the massive CSV files into a numpy array and using np.roll() to shift in a new column from the right side (like in the video) repeatedly until I hit the last column of the array (by calling np.roll() in the mpl.animation.FuncAnimation iterations.
This takes a large amount of CPU, and a much larger amount of memory.
Any suggestion on how to approach this? I couldn't find very many examples online that could help me with this.
here is some code from the mat plot lib tutorials.
import numpy as np
from matplotlib.lines import Line2D
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import math
class Scope(object):
def __init__(self, ax, maxt=2, dt=0.02):
self.ax = ax
self.dt = dt
self.maxt = maxt
self.tdata = [0]
self.ydata = [0]
self.line = Line2D(self.tdata, self.ydata)
self.ax.add_line(self.line)
self.ax.set_ylim(-.1, 1.1)
self.ax.set_xlim(0, self.maxt)
def update(self, y):
lastt = self.tdata[-1]
if lastt > self.tdata[0] + self.maxt: # reset the arrays
self.tdata = [self.tdata[-1]]
self.ydata = [self.ydata[-1]]
self.ax.set_xlim(self.tdata[0], self.tdata[0] + self.maxt)
self.ax.figure.canvas.draw()
t = self.tdata[-1] + self.dt
self.tdata.append(t)
self.ydata.append(y)
self.line.set_data(self.tdata, self.ydata)
return self.line,
def emitter(x=0):
'return a random value with probability p, else 0'
while True:
if x<361:
x = x + 1
yield math.sin(math.radians(x))
else:
x=0
x =x + 1
yield math.sin(math.radians(x))
# Fixing random state for reproducibility
np.random.seed(19680801)
fig, ax = plt.subplots()
scope = Scope(ax)
# pass a generator in "emitter" to produce data for the update func
ani = animation.FuncAnimation(fig, scope.update, emitter, interval=10,
blit=True)
plt.show()
my suggestion is to build a generator that yields the next data set you want to display every time it is called. this way you wont need to load the whole file into memory. more on that here. replace emitter function with generator that will pull from you file. disadvantage of this is I don't believe the full array will be available in the plot.
Related
I am trying to create a program that can visualize the change of a portfolio in real time. To do this, I update my data and create a new plot with it. When I run the code below in PyCharm, SciView stops displaying the plots after 30 iterations. Ideally, I would like to have it only show the most recent plot, but it would also be fine if it just truncated the history so that I at least always see the current plot. Is there any way to do this? I tried different ways to close the figures (e. g. using plt.close()), but did not achieve the desired result.
Code to reproduce:
import matplotlib.pyplot as plt
import numpy as np
import random
class RealTimeVisualizer:
def __init__(self, x, y):
self.x = x
self.y = y
def update_data(self, x_value, y_value):
"""
Appends values to the data arrays.
"""
self.x.append(x_value)
self.y.append(y_value)
def create_plot(self):
"""
Takes an x and a y (both 1D arrays and constructs a plot from it)
:return: a pyplot figure object
"""
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
# Draw x and y lists
ax.clear()
ax.plot(self.x, self.y)
# Format plot
plt.xticks(rotation=90)
plt.title('Portfolio')
plt.ylabel('Value')
plt.show()
plt.close('all')
if __name__ == '__main__':
portfolio_cash = 10000
tick = 0
real_time_visualizer = RealTimeVisualizer([tick], [portfolio_cash])
for i in np.arange(50):
tick += 1
portfolio_cash += random.randint(-50, 50)
real_time_visualizer.update_data(tick, portfolio_cash)
real_time_visualizer.create_plot()
Rather than creating a new plot and window every time, you can also update the current Matplotlib figure data in each iteration. You then need to view the plot in an interactive Matplotlib environment.
Live updating Matplotlib plots
You can use code similar to this to update the data inside the plot:
import matplotlib.pyplot as plt
import random
plt.ion() # Set pyplot to interactive mode
fig = plt.figure() # Create a figure
ax = fig.add_subplot(111) # Add a subplot to the figure
# Variables for our updating data
x = []
y = []
for i in range(50):
# Generate random data
x.append(i)
y.append(random.random())
# Update the plot with the new x, y data
ax.plot(x, y, 'ro-')
fig.canvas.draw()
fig.canvas.flush_events()
Allow for interactive Matplotlib mode when using SciView
Deactivate SciView or manually set your backend to another interactive GUI to see the updating plot.
This code snipped automatically chooses the correct backend (same list as in the Matplotlib code):
import matplotlib.pyplot as plt
candidates = ["macosx", "qt5agg", "gtk3agg", "tkagg", "wxagg"]
for candidate in candidates:
try:
plt.switch_backend(candidate)
print('Using backend: ' + candidate)
break
except (ImportError, ModuleNotFoundError):
pass
Applied to your code
Your code with suggested modifications would look like this:
import matplotlib.pyplot as plt
import numpy as np
import random
class RealTimeVisualizer:
def __init__(self, x, y):
self.x = x
self.y = y
def update_data(self, x_value, y_value):
"""
Appends values to the data arrays.
"""
self.x.append(x_value)
self.y.append(y_value)
def update_plot(self, fig, ax):
import _tkinter
try:
ax.plot(self.x, self.y, 'ro-')
fig.canvas.draw()
fig.canvas.flush_events()
# Capture an error in case the plotting window is being closed
except _tkinter.TclError:
pass
if __name__ == '__main__':
portfolio_cash = 10000
tick = 0
real_time_visualizer = RealTimeVisualizer([tick], [portfolio_cash])
# Choose the right backend
candidates = ["macosx", "qt5agg", "gtk3agg", "tkagg", "wxagg"]
for candidate in candidates:
try:
plt.switch_backend(candidate)
print('Using backend: ' + candidate)
break
except (ImportError, ModuleNotFoundError):
pass
# Create plot
plt.ion() # Set pyplot to interactive mode
fig = plt.figure() # Create a figure
ax = fig.add_subplot(111) # Add a subplot to the figure
for i in np.arange(50):
tick += 1
portfolio_cash += random.randint(-50, 50)
real_time_visualizer.update_data(tick, portfolio_cash)
real_time_visualizer.update_plot(fig, ax) # Update the plot the new data
Same issue here.
The workaround I found is to change the matplotlib backend to plot outside the PyCharm.
import matplotlib
matplotlib.use('qt5Agg')
matplotlib.pyplot.ioff()
Then you have to explicit open a new figure and show
for i in range(100):
plt.figure()
...
...
plt.show()
I'm trying to make an animated 3-D scatter plot with the ability to plot a dynamic number of classes as different colors. This is one of the attempts. I've included the whole code in case it is helpful, and marked the trouble spot with a row of stars:
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.animation as animation
from random import uniform
x_arr,y_arr,depth_arr,time_arr,ml_arr,cluster_arr = np.loadtxt(data, unpack=5, usecols=(0, 1, 2, 5, 6))
class Point:
def __init__(self,x,y,depth,time,cluster):
self.x=x
self.y=y
self.depth=depth
self.time=time
self.cluster=cluster
points = []
for i in range(0,len(x_arr)):
points.append(Point(x_arr[i],y_arr[i],depth_arr[i],time_arr[i],cluster_arr[i]))
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.set_xlim(min(x_arr), max(x_arr))
ax.set_ylim(min(y_arr), max(y_arr))
ax.set_zlim(min(depth_arr), max(depth_arr))
colors_1 = plt.cm.jet(np.linspace(0,max(cluster_arr),max(cluster_arr)+1))
colors = colors_1.reshape(-1,4)
def plot_points(time):
x = []
y = []
z = []
clust = []
points_cp = list(np.copy(points))
for i in range(0,(int(max(cluster_arr))+1)):
for event in points_cp:
if event.cluster == i:
if event.time < time:
points_cp.remove(event)
elif event.time <= time + 86400:
x.append(event.x)
y.append(event.y)
z.append(event.depth)
clust.append(event.cluster)
points_cp.remove(event)
# **************************************************************
color_ind = 0
first_ind = 0
last_ind = 0
for i in range(0,len(x)):
if clust[i] != color_ind:
last_ind = i
for i in range(0,len(x)):
ax.scatter(x[first_ind:last_ind],y[first_ind:last_ind],z[first_ind:last_ind],c=colors[int(color_ind)])
color_ind = clust[i]
first_ind = i
time = np.linspace(min(time_arr),max(time_arr),100)
ani = animation.FuncAnimation(fig,plot_points,time)
plt.show()
This gives me a plot with the correct colors, but once a point is plotted, it remains throughout the entire animation.
I have also tried set_x, set_color, etc., but this doesn't work with a loop (it is updated with each iteration, so that only the last class is actually plotted), and I need to use a for loop to accommodate a variable number of classes. I've tried using a colormap with a fixed extent, but have been unsuccessful, as colormapping doesn't work with the plot function, and I haven't been able to get the rest of the code to work with a scatter function.
Thanks in advance for your help, and my apologies if the code is a little wonky. I'm pretty new to this.
I want to plot data in matplotlib in real time. I want to open a figure once at the start of the programme, then update the figure when new data is acquired. Despite there being a few similar questions out there, none quite answer my specific question.
I want each set of data points new_data1 and new_data2 to be plotted on the same figure at the end of each while loop i.e. one line after the first while loop, two lines on the same figure after the second while loop etc. Currently they are all plotted together, but only right at the end of the programme, which is no use for real time data acquisition.
import matplotlib.pyplot as plt
import numpy
hl, = plt.plot([], [])
def update_line(hl, new_datax, new_datay):
hl.set_xdata(numpy.append(hl.get_xdata(), new_datax))
hl.set_ydata(numpy.append(hl.get_ydata(), new_datay))
plt.xlim(0, 50)
plt.ylim(0,200)
plt.draw()
x = 1
while x < 5:
new_data1 = []
new_data2 = []
for i in range(500):
new_data1.append(i * x)
new_data2.append(i ** 2 * x)
update_line(hl, new_data1, new_data2)
x += 1
else:
print("DONE")
This programme plots all 5 lines, but at the end of the programme. I want each line to be plotted after one another, after the while loop is completed. I have tried putting in plt.pause(0.001) in the function, but it has not worked.
This programme is different from the one that has been put forward - that programme only plots one graph and does not update with time.
If I correctly understood your specifications, you can modify just a bit your MWE as follows:
import matplotlib.pyplot as plt
import numpy
fig = plt.figure(figsize=(11.69,8.27))
ax = fig.gca()
ax.set_xlim(0, 50)
ax.set_ylim(0,200)
hl, = plt.plot([], [])
def update_line(hl, new_datax, new_datay):
# re initialize line object each time if your real xdata is not contiguous else comment next line
hl, = plt.plot([], [])
hl.set_xdata(numpy.append(hl.get_xdata(), new_datax))
hl.set_ydata(numpy.append(hl.get_ydata(), new_datay))
fig.canvas.draw_idle()
fig.canvas.flush_events()
x = 1
while x < 10:
new_data1 = []
new_data2 = []
for i in range(500):
new_data1.append(i * x)
new_data2.append(i ** 2 * x)
update_line(hl, new_data1, new_data2)
# adjust pause duration here
plt.pause(0.5)
x += 1
else:
print("DONE")
which displays :
Not sure, if I am reading the requirements right but below is a blueprint. Please change it to suit your requirements. You may want to change the function Redraw_Function and edit the frames (keyword parameter, which is np.arange(1,5,1) ) in the FuncAnimation call. Also interval=1000 means 1000 milliseconds of delay.
If you are using Jupyter then comment out the second last line (where it says plt.show()) and uncomment the last line. This will defeat your purpose of real time update but I am sorry I had trouble making it work real time in Jupyter. However if you are using python console or official IDLE please run the code as it is. It should work nicely.
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.animation import FuncAnimation
fig, ax = plt.subplots()
plot, = plt.plot([],[])
def init_function():
ax.set_xlim(0,50)
ax.set_ylim(0,250)
return plot,
def Redraw_Function(UpdatedVal):
new_x = np.arange(500)*UpdatedVal
new_y = np.arange(500)**2*UpdatedVal
plot.set_data(new_x,new_y)
return plot,
Animated_Figure = FuncAnimation(fig,Redraw_Function,init_func=init_function,frames=np.arange(1,5,1),interval=1000)
plt.show()
# Animated_Figure.save('MyAnimated.gif',writer='imagemagick')
When you run the code, you obtain the below result. I tried to keep very little code but I am sorry, if your requirement was totally different.
Best Wishes,
I have an algorithm which consists of two distinct parts which I want to visualize one after another (while possibly keeping the final state of animation 1 on the screen when animation 2 starts).
I can visualize both parts individually by calling animation.FuncAnimation and plt.show(). Since both parts have set number of frames and their very own behaviour, I would like to keep their implementations apart in two different classes and then do a wrapper around them which plays them in sequence.
However, is it possible to have two (or more) animation objects to be displayed one after another in the same figure?
Many thanks,
Matt
Thanks to the hint of ImportanceOfBeingErnest, I came up with a solution which updates only certain elements of my animator state depending on the current time step. Here is a small example illustrating this approach:
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from math import sin, radians
class AnimationHandler:
def __init__(self, ax):
self.ax = ax
self.lines = [self.ax.plot([], []), self.ax.plot([], [])]
self.colors = ['cyan', 'red']
self.n_steps = [360, 360]
self.step = 0
def init_animation(self):
for anim_idx in [0, 1]:
self.lines[anim_idx], = self.ax.plot([0, 10], [0, 0], c=self.colors[anim_idx], linewidth=2)
self.ax.set_ylim([-2, 2])
self.ax.axis('off')
return tuple(self.lines)
def update_slope(self, step, anim_idx):
self.lines[anim_idx].set_data([0, 10], [0, sin(radians(step))])
def animate(self, step):
# animation 1
if 0 < step < self.n_steps[0]:
self.update_slope(step, anim_idx=0)
# animation 2
if self.n_steps[0] < step < sum(self.n_steps):
self.update_slope(step - self.n_steps[0], anim_idx=1)
return tuple(self.lines)
if __name__ == '__main__':
fig, axes = plt.subplots()
animator = AnimationHandler(ax=axes)
my_animation = animation.FuncAnimation(fig,
animator.animate,
frames=sum(animator.n_steps),
interval=10,
blit=True,
init_func=animator.init_animation,
repeat=False)
Writer = animation.writers['ffmpeg']
writer = Writer(fps=24, metadata=dict(artist='Me'), bitrate=1800)
my_animation.save('./anim_test.mp4', writer=writer)
plt.show()
I used this approach to visualize/debug an algorithm which has different elements with varying runtimes. Approach is the same: You know the number of steps of each subsequence and adjust the state accordingly.
I can't figure how to plot a continuous function using Matplotlib. I got how to plot a scatterplot, but I would like a continuous plot.
Here is my code:
import matplotlib.pyplot as plt
from matplotlib.pyplot import autoscale
import matplotlib.animation as animation
import numpy
class MyPlot():
def __init__(self):
self.index = 0
self.setup()
def setup(self):
plt.ion()
self.fig, self.ax = plt.subplots()
self.line = self.ax.plot([],[])
autoscale()
plt.show()
def anim(self, i):
self.line.set_ydata(i) # update the data
return self.line,
def add(self, val):
print self.index, val
self.ax.plot(self.index, val)
animation.FuncAnimation(self.fig, self.anim, repeat=False)
plt.pause(0.05)
#if(self.index >= ntests):
self.index+=1
if __name__== "__main__":
import time
from random import random
p = MyPlot()
for i in range(100):
p.add(random())
time.sleep(0.5)
This works, but doesn't draw anything. The plot resizes itself, though.
You are only plotting a line with a single point at a time (which doesn't exist), so nothing shows up. If you replace self.ax.plot with self.ax.scatter, it plots correctly.
If you really want lines, you can just keep track of the last index and value and plot a line connecting the last index and value with the current index and value each time.
Add these two lines to add()
self.ax.plot([self.index-1, self.index], [self.lastval, val])
self.lastval = val
as well as a line initializing self.lastval to numpy.nan in setup()
You can actually append values to a line plot in matplotlib:
self.line.set_xdata(numpy.append(self.line.get_xdata(), self.index))
self.line.set_ydata(numpy.append(self.line.get_ydata(), val))
This way, you do not have to do any of the bookkeeping yourself.
More details can be found at https://stackoverflow.com/a/10944967/2988730