Develop Reference

matplotlib.animate in python using multiprocessing

I am trying to use a python process to animate a plot as shown below:
from multiprocessing import Process
import datetime as dt
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import numpy as np
process_enabled = 1;
print("Process enabled: ", process_enabled)
x = []
y = []
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
def start_animation():
# Set up plot to call animate() function periodically
ani = animation.FuncAnimation(fig, animate, fargs=(x, y), interval=1000)
print("Called animate function")
plt.show()
# This function is called periodically from FuncAnimation
def animate(i, xs, ys):
fx=[0.045,0.02,0.0,0.04,0.015,-0.01,0.015,0.045,0.035,0.01,
0.055,0.04,0.02,0.025,0.0,-0.005,-0.005,-0.02,-0.05,-0.03] # fx values
# Add x and y to lists
xs.append(dt.datetime.now().strftime('%H:%M:%S.%f'))
if(i<len(fx)):
ys.append(fx[i])
# Draw x and y lists
ax.clear()
if(i<len(fx)):
ys_stacked = np.stack((np.array(ys),0.1+np.array(ys)),axis=1)
ax.plot(xs, ys_stacked)
print("Animating")
# Format plot
if(i<len(fx)):
plt.xticks(rotation=45, ha='right')
plt.subplots_adjust(bottom=0.30)
plt.title('Force/Torque Sensor Data')
plt.ylabel('Fx (N)')
if(process_enabled):
p_graph = Process(name='Graph', target=start_animation)
print("Created graph process")
p_graph.start()
print("Started graph process")
else:
start_animation()
When I disable the process, the start_animation() function works fine and the plot is displayed and the animation begins. However, when the process is enabled, the process starts and then the code breaks at print("Called animate function"). There is no plot window and there are no error messages in the terminal).
I'm new to both multiprocessing in python and indeed matplotlib. Any direction would be much appreciated.
Cheers,
Tony

I'm trying to solve this same problem, but haven't quite figured it out completely. However, I think I can provide a few useful comments on your question.
To start, is there any reason why you want to handle the animation in a separate process? Your approach seems to work fine within a single process. There's a number of issues you'll need to address to do this. If you truly do require a separate process, then the following might be useful.
First, you won't be able to use your global variables in the 'graph' process, as that process doesn't share the same instances of those variables (see Globals variables and Python multiprocessing).
You can share state between processes, but this is difficult for complex objects that you'd want to share (i.e. plt.figure()). See the multiprocessing reference for more information (https://docs.python.org/3/library/multiprocessing.html#sharing-state-between-processes)
One final suggestion would be to do away with the pyplot interface. This is handy for straightforward scripts and interactive data analysis, but it obfuscates a lot of important things - like knowing which figure, axis etc you're dealing with when you call plt methods.
I've provided an alternative, object-oriented approach using a custom class, that can run your animation (without a separate process):
import sys
from multiprocessing import Process, Queue
import datetime as dt
from matplotlib.figure import Figure
from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg
from matplotlib.backends.qt_compat import QtWidgets
import matplotlib.animation as animation
class StripChart(FigureCanvasQTAgg):
def __init__(self):
self.fig = Figure(figsize=(8,5), dpi=100)
self.ax = self.fig.add_subplot(111)
# hold a copy of our torque data
self.fx = [0.045,0.02,0.0,0.04,0.015,-0.01,0.015,0.045,0.035,0.01,
0.055,0.04,0.02,0.025,0.0,-0.005,-0.005,-0.02,-0.05,-0.03]
super().__init__(self.fig)
# instantiate the data arrays
self.xs = []
self.ys = []
def start_animation(self):
print("starting animation")
# set up the animation
self.ani = animation.FuncAnimation(self.fig, self.animate, init_func=self.clear_frame,
frames=100, interval=500, blit=False)
def clear_frame(self):
self.ax.clear()
self.ax.plot([], [])
def animate(self, i):
print("animate frame")
# get the current time
t_now = dt.datetime.now()
# update trace values
self.xs.append(t_now.strftime("%H:%M:%S.%f"))
self.ys.append(self.fx[i % len(self.fx)])
# keep max len(self.fx) points
if len(self.xs) > len(self.fx):
self.xs.pop(0)
self.ys.pop(0)
self.ax.clear()
self.ax.plot(self.xs, self.ys)
# need to reapply format after clearing axes
self.fig.autofmt_xdate(rotation=45)
self.fig.subplots_adjust(bottom=0.30)
self.ax.set_title('Force/Torque Sensor Data')
self.ax.set_ylabel('Fx (N)')
if __name__=='__main__':
# start a new qapplication
qapp = QtWidgets.QApplication(sys.argv)
# create our figure in the main process
strip_chart = StripChart()
strip_chart.show()
strip_chart.start_animation()
# start qt main loop
qapp.exec()
Things of note in this example:
you'll need to have a backend installed in your environment (i.e. pip install pyqt5)
I've added an init_func to the animation, you don't really need this as you can call self.ax.clear() in the animate method.
If you need better performance for your animation, you can use blit=True but you'll need to modify the clear_frame and animate methods to return the artists that you want to update (see https://jakevdp.github.io/blog/2012/08/18/matplotlib-animation-tutorial/ for more info). One drawback is that you won't be able to update the axis labels with that approach.
I've set it up to run infinitely until you close the window
I'm assuming that the reason you want to run the animation in a separate process is that there is some time consuming/CPU intensive task that is involved in either updating the graph data, or drawing all the points. Perhaps you have this embedded in some other UI?
I've tried to execute the animation in a separate process, but you need to pass the instance of the figure that's displayed. As I mentioned this isn't straightforward, although there do appear to be ways to do it (https://stackoverflow.com/a/57793267/13752965). I'll update if I find a working solution.

python: How to monitor a variable ? Execute command when variable change

I have a class like this
class WaveData(object):
def __init__(self, data):
self.data = data
and create a data object, plot a figure
wave = WaveData([[1, 2, 3],
[7, 5, 6]])
import matplotlib.pyplot as plt
fig=plt.figure()
plot1, = fig.canvas.figure.subplots().plot(wave.data[0])
plot2, = fig.canvas.figure.subplots().plot(wave.data[1])
I hope when I change the wave value , plot will change synchronously
wave.data[1]=[5,6,7] # hope figure change together
I try to add method changedata for WaveData class, but:
it need use global variable fig , maybe not reasonale (I can put fig as self attribute , but in fact , the fig also link other class object which not written in here)
I cannot change fig by changing data directly: wave.data[1] =[5,6,7]
class WaveData(object):
def __init__(self, data):
self.data = data
def changedata(self,value,index):
self.data[index]=value
#-- change the plot index th plot data--#
global plot1,plot2,fig
plot1.set_ydata(self.data[1])
plot2.set_ydata(self.data[2])
fig.canvas.draw_idle()
#-- change the plot index th plot data--#
I want to create a watcher to monitor wave.data value . When detecte the value change , execute some action
How to do?

Right: plotting is not a dynamic or interactive process. You started the right way, with an access method to change the wave form. Now you have to re-plot and re-show the result ... which will possibly require closing the first plot manually, depending on the interface of your chosen drawing package (e.g. matplotlib).
To get a fully interactive experience, you may want to use an animation package, such as PyGame, where changes in the visual display are part of the package assumptions.

matplotlib animation: write to png files without third party module

The animation module in matplotlib usually requires third party modules like FFmpeg, mencoder or imagemagik to be able to save the animation to a file (e.g. here: https://stackoverflow.com/a/25143651/5082048).
Even the MovieWriter class in matplotlib seems to be build in a way that third party modules will be incorporated (starting and closing processes, cummunicating via pipe): http://matplotlib.org/api/animation_api.html#matplotlib.animation.MovieWriter.
I am looking for a way, how I can save a matplotlib.animation.FuncAnimation object frame to frame to png - directly, within python. Afterwards, I want to show the .png files as animation in an iPython notebook using this approach: https://github.com/PBrockmann/ipython_animation_javascript_tool/
Therefore my questions are:
How can I save an matplotlib.animation.FuncAnimation object directly to .png files without the need to use third party modules?
Is there a writer class implemented for this usecase?
How can I get figure objects frame by frame out of the FuncAnimation object (so that I could save them myself)?
Edit: The matplotlib.animation.FuncAnimation object is given, the task is to save it's frames using pure Python. Unfortunately, I cannot change the underlying animation function as ImportanceOfBeingErnest suggested.

Althoough this may seem a bit complicated, saving the frames of an animation may be easily done within the animation itself.
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.animation
import numpy as np
def animate(i):
line.set_ydata(np.sin(2*np.pi*i / 50)*np.sin(x))
#fig.canvas.draw() not needed see comment by #tacaswell
plt.savefig(str(i)+".png")
return line,
fig = plt.figure()
ax = fig.add_subplot(111)
ax.set_xlim(0, 2*np.pi)
ax.set_ylim(-1,1)
x = np.linspace(0, 2*np.pi, 200)
line, = ax.plot(x, np.zeros_like(x))
plt.draw()
ani = matplotlib.animation.FuncAnimation(fig, animate, frames=5, repeat=False)
plt.show()
Mind the repeat = False argument, which will prevent the animation to run continuously and repeat writing the same files to disk.
Note that if you're willing to loosen the restriction of "no external package" you may use imagemagick to save the pngs
ani.save("anim.png", writer="imagemagick")
which will save files anim-1.png, anim-2.png etc.
Finally note that there are of course easier methods to show an animation in a jupyter notebook.

You want to look at the FileMovieWriter sub-classes (See http://matplotlib.org/2.0.0rc2/api/animation_api.html#writer-classes) You probably want to sub-class FileMoveWriter, something like
import matplotlib.animation as ma
class BunchOFiles(ma.FileMovieWriter):
def setup(self, fig, dpi, frame_prefix):
super().setup(fig, dpi, frame_prefix, clear_temp=False)
def _run(self):
# Uses subprocess to call the program for assembling frames into a
# movie file. *args* returns the sequence of command line arguments
# from a few configuration options.
pass
def grab_frame(self, **savefig_kwargs):
'''
Grab the image information from the figure and save as a movie frame.
All keyword arguments in savefig_kwargs are passed on to the 'savefig'
command that saves the figure.
'''
# Tell the figure to save its data to the sink, using the
# frame format and dpi.
with self._frame_sink() as myframesink:
self.fig.savefig(myframesink, format=self.frame_format,
dpi=self.dpi, **savefig_kwargs)
def cleanup(self):
# explictily skip a step in the mro
ma.MovieWriter.cleanup(self)
(this is not tested, might be better to just implement a class that implements saving, grab_frame, finished, and setup)

I could not get tacaswell's answer to work without modification. So, here is my take at it.
from matplotlib.animation import FileMovieWriter
class BunchOFiles(FileMovieWriter):
supported_formats = ['png', 'jpeg', 'bmp', 'svg', 'pdf']
def __init__(self, *args, extra_args=None, **kwargs):
# extra_args aren't used but we need to stop None from being passed
super().__init__(*args, extra_args=(), **kwargs)
def setup(self, fig, dpi, frame_prefix):
super().setup(fig, dpi, frame_prefix, clear_temp=False)
self.fname_format_str = '%s%%d.%s'
self.temp_prefix, self.frame_format = self.outfile.split('.')
def grab_frame(self, **savefig_kwargs):
'''
Grab the image information from the figure and save as a movie frame.
All keyword arguments in savefig_kwargs are passed on to the 'savefig'
command that saves the figure.
'''
# Tell the figure to save its data to the sink, using the
# frame format and dpi.
with self._frame_sink() as myframesink:
self.fig.savefig(myframesink, format=self.frame_format,
dpi=self.dpi, **savefig_kwargs)
def finish(self):
self._frame_sink().close()
We can save a set of files with:
anim.save('filename.format', writer=BunchOFiles())
and it will save the files in the form 'filename{number}.format'.

How to deal with multiple objects listening to motion_notify_events on the same figure

I'm using matplotlib :
Let's say I have an object A, which has two attributes B and C, and a method that draws a figure. Both B and C have methods doing some stuff on the figure on events 'motion_notify_event'.
I've notice that both methods do not work at the same time, and there appears to be conflict.
How does one deal with such case ?
So I've written a code that shows the problem a little bit better than the above explanation.
import matplotlib.pyplot as plt
from matplotlib.widgets import MultiCursor
from matplotlib.patches import Circle
class Event1(object):
def __init__(self,axes):
self.fig = axes[0].figure
self.axes = axes
self.eid = self.fig.canvas.mpl_connect('motion_notify_event',self.onmove)
def onmove(self,event):
for ax in self.axes:
c = Circle((event.xdata,event.ydata),radius=0.05)
ax.add_patch(c)
ax.draw_artist(c)
self.fig.canvas.blit(self.fig.bbox)
class plotclass(object):
def __init__(self):
pass
def plotme(self):
self.fig = plt.figure()
self.ax1 = self.fig.add_subplot(211)
self.ax2 = self.fig.add_subplot(212)
for ax in (self.ax1,self.ax2):
ax.set_xlim((0,10))
ax.set_ylim((0,10))
# self.curs = MultiCursor(self.fig.canvas,(self.ax1,self.ax2))
self.ev1 = Event1((self.ax1,self.ax2))
self.fig.show()
def main():
pc = plotclass()
return pc
if __name__ == '__main__':
main()
Now in this code there are two stuff listening the motion_notify_event : the class Event1, which will draw circles at the cursor position, and the class 'plotclass', which creates the figure and draws cursors at the cursor position.
I have commented out the line self.curs = ..., and I see the circles as the mouse is moving, but if I uncomment it, I just see the cursors : why ? and how to see both?

Just to elaborate on my comment above, it's not due to the multiple event handling, it's due to different stages of blitting overwriting each other.
Blitting typically works by restoring a saved, fully-rendered state and then drawing on top of it.
In your current code, you're blitting but not restoring the saved state, so you get a "trail" of circles (presumably this is what you want).
However, MultiCursor calls fig.canvas.restore_region(...) before drawing itself (otherwise you'd have a "trail" of lines). Therefore, it restores a saved "blank" figure over what you've just drawn.
If you want use multiple passes of blitting, they'll need to coordinate with each other. There are a number of different ways to handle this, but they're overkill for most use cases. The quick fix is to pass in useblit=False to MultiCursor. This will slow your rendering down, however.
Can you elaborate a bit on what you're trying to do? Do you just want a cursor with a circle in the mouse position? (If so, just subclass MultiCursor.)

How to place several tabs into separate processes

I have been working on a problem for a while (I am a chemical engineer, so it takes me forever to understand how to code something) of how to have several tabs operating in their own process, but with each tab having it's own data to show in a matplotlib plot.
I have been running into many pickling errors and I wondered if anyone had any somewhat simple solutions. I believe the main reasons for the pickling errors is due to the object I am trying to pass as a property into the tab object. This object holds some data as well as many other objects which help fit the data it holds.
I feel like these objects are very nice and rather necessary, but I also realize that they are causing the problems with pickling.
Here is a very simplified version of my code:
(This will still compile if you want to copy/paste to test it out.)
import multiprocessing as mp
from PyQt4 import QtGui, QtCore
import numpy as np
import matplotlib
matplotlib.use('QtAgg')
from matplotlib.backends.backend_qt4agg import FigureCanvasQTAgg as FigureCanvas
from matplotlib import figure
import sys
import lmfit
# This object will just hold certain objects which will help create data objects ato be shown in matplotlib plots
# this could be a type of species with properties that could be quantized to a location on an axis (like number of teeth)
#, which special_object would hold another quantization of that property (like length of teeth)
class object_within_special_object:
def __init__(self, n, m):
self.n = n
self.m = m
def location(self, i):
location = i*self.m/self.n
return location
def NM(self):
return str(self.n) + str(self.m)
# This is what will hold a number of species and all of their properties,
# as well as some data to try and fit using the species and their properties
class special_object:
def __init__(self, name, X, Y):
self.name = name
self.X = X
self.Y = Y
self.params = lmfit.Parameters()
self.things = self.make_a_whole_bunch_of_things()
for thing in self.things:
self.params.add('something' + str(thing.NM()) + 's', value = 3)
def make_a_whole_bunch_of_things(self):
things = []
for n in range(0,20):
m=1
things.append(object_within_special_object(n,m))
return things
# a special type of tab which holds a (or a couple of) matplotlib plots and a special_object ( which holds the data to display in those plots)
class Special_Tab(QtGui.QTabWidget):
def __init__(self, parent, special_object):
QtGui.QTabWidget.__init__(self, parent)
self.special_object = special_object
self.grid = QtGui.QGridLayout(self)
# matplotlib figure put into tab
self.fig = figure.Figure()
self.plot = self.fig.add_subplot(111)
self.line, = self.plot.plot(self.special_object.X, self.special_object.Y, 'r-')
self.canvas = FigureCanvas(self.fig)
self.grid.addWidget(self.canvas)
self.canvas.show()
self.canvas.draw()
self.canvas_BBox = self.plot.figure.canvas.copy_from_bbox(self.plot.bbox)
ax1 = self.plot.figure.axes[0]
def process_on_special_object(self):
# do a long fitting process involving the properties of the special_object
return
def update_GUI(self):
# change the GUI to reflect changes made to special_object
self.line.set_data(special_object.X, special_object.Y)
self.plot.draw_artist(self.line)
self.plot.figure.canvas.blit(self.plot.bbox)
return
# This window just has a button to make all of the tabs in separate processes
class MainWindow(QtGui.QMainWindow):
def __init__(self, parent = None):
# This GUI stuff shouldn't be too important
QtGui.QMainWindow.__init__(self)
self.resize(int(app.desktop().screenGeometry().width()*.6), int(app.desktop().screenGeometry().height()*.6))
self.tabs_list = []
central_widget = QtGui.QWidget(self)
self.main_tab_widget = QtGui.QTabWidget()
self.layout = QtGui.QHBoxLayout(central_widget)
button = QtGui.QPushButton('Open Tabs')
self.layout.addWidget(button)
self.layout.addWidget(self.main_tab_widget)
QtCore.QObject.connect(button, QtCore.SIGNAL("clicked()"), self.open_tabs)
self.setCentralWidget(central_widget)
central_widget.setLayout(self.layout)
# Here we open several tabs and put them in different processes
def open_tabs(self):
for i in range(0, 10):
# this is just some random data for the objects
X = np.arange(1240.0/1350.0, 1240./200., 0.01)
Y = np.array(np.e**.2*X + np.sin(10*X)+np.cos(4*X))
# Here the special tab is created
new_tab = Special_Tab(self.main_tab_widget, special_object(str(i), X, Y))
self.main_tab_widget.addTab(new_tab, str(i))
# this part works fine without the .start() function
self.tabs_list.append(mp.Process(target=new_tab))
# this is where pickling errors occur
self.tabs_list[-1].start()
return
if __name__ == "__main__":
app = QtGui.QApplication([])
win = MainWindow()
win.show()
sys.exit(app.exec_())
I have noticed that the errors are coming from matplotlib axes (I'm not sure how?) and gives the error pickle.PicklingError: Can't pickle <class 'matplotlib.axes.AxesSubplot'>: it's not found as matplotlib.axes.AxesSubplot. In addition, I have noticed that commenting out the matplotlib plots will also give the pickling error pickle.PicklingError: Can't pickle <function <lambda> at 0x012A2B30>: it's not found as lmfit.parameter.<lambda>. I think this is because lambda functions cannot be pickled and I guess lmfit has a lambda somewhere in it's depths... but I don't really know what to do without some way around these errors.
Oddly enough, the error I see form the original code (not the simplified version shown here) is slightly different, but still basically the same in sentiment. The error I get in my other code is pickle.PicklingError: Can't pickle 'BufferRegion' object: <BufferRegion object at 0x037EBA04>
Does anyone have a better solution to this problem by moving the objects around in regards to where I pass them or any other ideas?
I very much appreciate your time and effort and any help on this problem.
EDIT: I tried unutbu's idea in a way, but with some alterations to the position of the process funciton.
The only problem with the proposed solution is that the do_long_fitting_process() function calls another function which iteratively updates the lines in the matplotlib plots. So the do_long_fitting_process() needs to have some access to the Special_Tab properties to change them and show the updates to the GUI.
I have tried doing this by pushing the do_long_fitting_process() function to just a global function and calling this:
[code]
def open_tabs(self):
for i in range(0, 10):
...
self.tabs_list.append(new_tab)
consumer, producer = mp.Pipe()
process = mp.Process(target=process_on_special_object, args=(producer,))
process.start()
while(True):
message = consumer.recv()
if message == 'done':
break
tab.update_GUI(message[0], message[1])
process_list[-1].join()
[/code]
Where I am passing the data to update_GUI() via a mp.Pipe(), but the window just goes to "Not Responding" as soon as I start the processes.

Separate the GUI code from the computation code. The GUI must run in a single process (though it may spawn multiple threads). Let the computation code be contained within in the special_object.
Let the Special_Tab call mp.Process when you want to perform a long-running computation:
class special_object:
def do_long_fitting_process(self):
pass
class Special_Tab(QtGui.QTabWidget):
def process_on_special_object(self):
# do a long fitting process involving the properties of the
# special_object
proc = mp.Process(target = self.special_object.do_long_fitting_process)
proc.start()
class MainWindow(QtGui.QMainWindow):
def open_tabs(self):
for i in range(0, 10):
...
self.tabs_list.append(new_tab)
new_tab.process_on_special_object()

The problem is that not all parts of an Axes object can be serialized, which is necessary to move data between the processes. I would suggest a slight re-organization of your code to push computation off to separate processes (that is any thing that will take more than a fraction of a second), but keep all of your plotting on the main process and only pass data back and forth.
Another option is to us QThread See time.sleep() required to keep QThread responsive? for two different ways of implementing it (one in the question, one in my answer).