Develop Reference

matplotlib.animate in python using multiprocessing

I am trying to use a python process to animate a plot as shown below:
from multiprocessing import Process
import datetime as dt
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import numpy as np
process_enabled = 1;
print("Process enabled: ", process_enabled)
x = []
y = []
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
def start_animation():
# Set up plot to call animate() function periodically
ani = animation.FuncAnimation(fig, animate, fargs=(x, y), interval=1000)
print("Called animate function")
plt.show()
# This function is called periodically from FuncAnimation
def animate(i, xs, ys):
fx=[0.045,0.02,0.0,0.04,0.015,-0.01,0.015,0.045,0.035,0.01,
0.055,0.04,0.02,0.025,0.0,-0.005,-0.005,-0.02,-0.05,-0.03] # fx values
# Add x and y to lists
xs.append(dt.datetime.now().strftime('%H:%M:%S.%f'))
if(i<len(fx)):
ys.append(fx[i])
# Draw x and y lists
ax.clear()
if(i<len(fx)):
ys_stacked = np.stack((np.array(ys),0.1+np.array(ys)),axis=1)
ax.plot(xs, ys_stacked)
print("Animating")
# Format plot
if(i<len(fx)):
plt.xticks(rotation=45, ha='right')
plt.subplots_adjust(bottom=0.30)
plt.title('Force/Torque Sensor Data')
plt.ylabel('Fx (N)')
if(process_enabled):
p_graph = Process(name='Graph', target=start_animation)
print("Created graph process")
p_graph.start()
print("Started graph process")
else:
start_animation()
When I disable the process, the start_animation() function works fine and the plot is displayed and the animation begins. However, when the process is enabled, the process starts and then the code breaks at print("Called animate function"). There is no plot window and there are no error messages in the terminal).
I'm new to both multiprocessing in python and indeed matplotlib. Any direction would be much appreciated.
Cheers,
Tony

I'm trying to solve this same problem, but haven't quite figured it out completely. However, I think I can provide a few useful comments on your question.
To start, is there any reason why you want to handle the animation in a separate process? Your approach seems to work fine within a single process. There's a number of issues you'll need to address to do this. If you truly do require a separate process, then the following might be useful.
First, you won't be able to use your global variables in the 'graph' process, as that process doesn't share the same instances of those variables (see Globals variables and Python multiprocessing).
You can share state between processes, but this is difficult for complex objects that you'd want to share (i.e. plt.figure()). See the multiprocessing reference for more information (https://docs.python.org/3/library/multiprocessing.html#sharing-state-between-processes)
One final suggestion would be to do away with the pyplot interface. This is handy for straightforward scripts and interactive data analysis, but it obfuscates a lot of important things - like knowing which figure, axis etc you're dealing with when you call plt methods.
I've provided an alternative, object-oriented approach using a custom class, that can run your animation (without a separate process):
import sys
from multiprocessing import Process, Queue
import datetime as dt
from matplotlib.figure import Figure
from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg
from matplotlib.backends.qt_compat import QtWidgets
import matplotlib.animation as animation
class StripChart(FigureCanvasQTAgg):
def __init__(self):
self.fig = Figure(figsize=(8,5), dpi=100)
self.ax = self.fig.add_subplot(111)
# hold a copy of our torque data
self.fx = [0.045,0.02,0.0,0.04,0.015,-0.01,0.015,0.045,0.035,0.01,
0.055,0.04,0.02,0.025,0.0,-0.005,-0.005,-0.02,-0.05,-0.03]
super().__init__(self.fig)
# instantiate the data arrays
self.xs = []
self.ys = []
def start_animation(self):
print("starting animation")
# set up the animation
self.ani = animation.FuncAnimation(self.fig, self.animate, init_func=self.clear_frame,
frames=100, interval=500, blit=False)
def clear_frame(self):
self.ax.clear()
self.ax.plot([], [])
def animate(self, i):
print("animate frame")
# get the current time
t_now = dt.datetime.now()
# update trace values
self.xs.append(t_now.strftime("%H:%M:%S.%f"))
self.ys.append(self.fx[i % len(self.fx)])
# keep max len(self.fx) points
if len(self.xs) > len(self.fx):
self.xs.pop(0)
self.ys.pop(0)
self.ax.clear()
self.ax.plot(self.xs, self.ys)
# need to reapply format after clearing axes
self.fig.autofmt_xdate(rotation=45)
self.fig.subplots_adjust(bottom=0.30)
self.ax.set_title('Force/Torque Sensor Data')
self.ax.set_ylabel('Fx (N)')
if __name__=='__main__':
# start a new qapplication
qapp = QtWidgets.QApplication(sys.argv)
# create our figure in the main process
strip_chart = StripChart()
strip_chart.show()
strip_chart.start_animation()
# start qt main loop
qapp.exec()
Things of note in this example:
you'll need to have a backend installed in your environment (i.e. pip install pyqt5)
I've added an init_func to the animation, you don't really need this as you can call self.ax.clear() in the animate method.
If you need better performance for your animation, you can use blit=True but you'll need to modify the clear_frame and animate methods to return the artists that you want to update (see https://jakevdp.github.io/blog/2012/08/18/matplotlib-animation-tutorial/ for more info). One drawback is that you won't be able to update the axis labels with that approach.
I've set it up to run infinitely until you close the window
I'm assuming that the reason you want to run the animation in a separate process is that there is some time consuming/CPU intensive task that is involved in either updating the graph data, or drawing all the points. Perhaps you have this embedded in some other UI?
I've tried to execute the animation in a separate process, but you need to pass the instance of the figure that's displayed. As I mentioned this isn't straightforward, although there do appear to be ways to do it (https://stackoverflow.com/a/57793267/13752965). I'll update if I find a working solution.

asynchronously plot data with pyqtgraph

I have a simulation code and I would like to add diagnostic plotting to it with pyqtgraph.
The simulation is implemented in a Python class. In pseudo-code,
class Simulation:
[...]
'''
Defines self.data, the stuff that needs plotting
and self.done, which is False and becomese True once the simulation
is over.
'''
def evolve(self):
''' takes care of updating self.data'''
[...]
self.done = True
def get_data(self):
return self.data
and I would like to keep it independent from the plotting infrastructure.
What I'm trying to do is to create a plotting class, which so far I tried to implement with python threading, that reads
import pyqtgraph as pg
import threading
from time import sleep
from pyqtgraph.Qt import QtGui
class Plot(threading.Thread):
def __init__(self, source, target_fps=10):
threading.Thread.__init__(self, daemon=True)
self.source = source
self.wait_time = 1. / target_fps
self.win = pg.GraphicsWindow()
self.plot = self.win.addPlot()
self.curve = self.plot.plot()
self.curve.setData(self.source.get_data())
def run(self):
while not self.source.done:
self.curve.setData(self.source.get_data())
sleep(self.wait_time)
whose job is to poll the source class once in a while and update the plot. Then my __main__ would be something along the lines of
if __name__ == "__main__":
sim = Simulation() # initialise simulation code
plotter = Plot(sim) # <- By commenting these three lines
plotter.start() # <- I fall back to a working simulation
sim.evolve() #
plotter.join() # <- without plotting diagnostics
Of course the previous __main__ doesn't work because it is missing the Qt event loop required by pyqtgraph, something of the like of pg.QtGui.QApplication.exec_(), which carries the inconvenient side effect of blocking the execution of the rest of the code after it. If I try to put the exec_() inside Plot.run() I get a warning that the event loop must be run from within the main thread, and plots don't show up.
Is there a workaround that I can use to make the plotting class work and not block the execution of the simulation? I would like not to touch the Simulation class.

how to send a value from one class to another using pyqt4 and python

i'm having some trouble in building my app, I have a GUI made in pyqt4 and Python, I use QThread to check the cpu load every 2 seconds and I want to diplay it on a progress bar. My GUI is in one class and my Qthread is in the other.
This is my code: pyqt classes, my code printscreen
I want to know how to pass my values collected in QThread to Qobjects in my other Class.
import sys,os,module,config_read,time,threading,datecs_print,mysql.connector as mariadb,commandList
import psutil,logging
from PyQt4 import QtGui, uic ,QtSql,QtCore
from PyQt4.QtCore import QThread, SIGNAL
import resources
import webbrowser
sys.stderr = open("errlog.txt", "w")
class systemValues(QThread):
def __init__(self):
QThread.__init__(self)
def __del__(self):
self.wait()
def cpuRunValue(self):
while (1):
for x in range(2):
p = psutil.cpu_percent(1,False)
return p
def cpuProgressBarUpdate(self):
while(1):
# MyWindow.setcpuBarValue(val=77)
MyWindow.cpuBar.setValue(value=77)
def run(self):
# self.cpuRunValue()
print(self.cpuRunValue())
# self.cpuProgressBarUpdate()
class MyWindow(QtGui.QMainWindow):
def __init__(self):
QtGui.QDialog.__init__(self)
super(MyWindow, self).__init__()
file_path = os.path.abspath("ui/sales_window.ui")
uic.loadUi(file_path, self)
self.myThread = systemValues()
self.myThread.start()
def setcpuBarValue(self):
threading.Thread(target=self.cpuBar.setValue(systemValues.cpuRunValue())).start()
This is my code, I get no error. I just cant transfer my value I get from cpuRunValue() to QprogressBar from MyWindow. I'm not very experienced with this.
PS: I eliminated lots of code that's not necessary, but please let me know if you need more info.
Thank You.

If you want to use threads in Qt then you must understand them well. For instance read this page first. Certainly don't mix the QThread with the threading package from the Python standard library.
I hope you're not offended if I think that your probably fairly new to programming. I advise you to not use threads (in general and in Qt) until you have more experience. Threads are hard to use correctly, even for seasoned programmers, and therefore should only be used if necessary. For this reason the Qt documentation on threads even includes a section about alternatives.
You want to execute a function every 2 seconds, so in your case the best solution is to use a QTimer and connect its timeout signal to a slot that updates the progress bar. Like so:
#!/usr/bin/env python
import sys
import psutil
from PyQt5 import QtCore, QtWidgets
class MyWidget(QtWidgets.QWidget):
def __init__(self, parent=None):
super().__init__(parent=parent)
self.layout = QtWidgets.QVBoxLayout()
self.setLayout(self.layout)
self.progressBar = QtWidgets.QProgressBar()
self.layout.addWidget(self.progressBar)
self.timer = QtCore.QTimer()
self.timer.timeout.connect(self.updateCpu)
self.timer.start(100) # Every 0.1 seconds
def updateCpu(self):
cpu = psutil.cpu_percent(None, False)
#print("cpu = {}%".format(cpu))
self.progressBar.setValue(cpu)
def main():
app = QtWidgets.QApplication(sys.argv)
win = MyWidget()
win.show()
win.raise_()
app.exec_()
if __name__ == "__main__":
main()
Note that I changed the first parameter of cpu_percent to None. This makes the call return immediately instead of lasting one second (and therefore you can update the progress bar more often if you want). Setting it to None causes the first call of cpu_percent to return a meaningless value, but in your cause that's not an issue I think. See the documention.
Finally, even though you removed a lot of unnecessary code, your code example was not yet minimal (e.g. the import datecs_print is not necessary to run it, and I don't have this module). I was also not complete (e.g. the "ui/sales_window.ui" file was missing so I couldn't run it). Please read the page on how to make an MCVE. You will get a lot more help next time if you include a small program that we can just copy-paste-execute.
Hope this helps

How to place several tabs into separate processes

I have been working on a problem for a while (I am a chemical engineer, so it takes me forever to understand how to code something) of how to have several tabs operating in their own process, but with each tab having it's own data to show in a matplotlib plot.
I have been running into many pickling errors and I wondered if anyone had any somewhat simple solutions. I believe the main reasons for the pickling errors is due to the object I am trying to pass as a property into the tab object. This object holds some data as well as many other objects which help fit the data it holds.
I feel like these objects are very nice and rather necessary, but I also realize that they are causing the problems with pickling.
Here is a very simplified version of my code:
(This will still compile if you want to copy/paste to test it out.)
import multiprocessing as mp
from PyQt4 import QtGui, QtCore
import numpy as np
import matplotlib
matplotlib.use('QtAgg')
from matplotlib.backends.backend_qt4agg import FigureCanvasQTAgg as FigureCanvas
from matplotlib import figure
import sys
import lmfit
# This object will just hold certain objects which will help create data objects ato be shown in matplotlib plots
# this could be a type of species with properties that could be quantized to a location on an axis (like number of teeth)
#, which special_object would hold another quantization of that property (like length of teeth)
class object_within_special_object:
def __init__(self, n, m):
self.n = n
self.m = m
def location(self, i):
location = i*self.m/self.n
return location
def NM(self):
return str(self.n) + str(self.m)
# This is what will hold a number of species and all of their properties,
# as well as some data to try and fit using the species and their properties
class special_object:
def __init__(self, name, X, Y):
self.name = name
self.X = X
self.Y = Y
self.params = lmfit.Parameters()
self.things = self.make_a_whole_bunch_of_things()
for thing in self.things:
self.params.add('something' + str(thing.NM()) + 's', value = 3)
def make_a_whole_bunch_of_things(self):
things = []
for n in range(0,20):
m=1
things.append(object_within_special_object(n,m))
return things
# a special type of tab which holds a (or a couple of) matplotlib plots and a special_object ( which holds the data to display in those plots)
class Special_Tab(QtGui.QTabWidget):
def __init__(self, parent, special_object):
QtGui.QTabWidget.__init__(self, parent)
self.special_object = special_object
self.grid = QtGui.QGridLayout(self)
# matplotlib figure put into tab
self.fig = figure.Figure()
self.plot = self.fig.add_subplot(111)
self.line, = self.plot.plot(self.special_object.X, self.special_object.Y, 'r-')
self.canvas = FigureCanvas(self.fig)
self.grid.addWidget(self.canvas)
self.canvas.show()
self.canvas.draw()
self.canvas_BBox = self.plot.figure.canvas.copy_from_bbox(self.plot.bbox)
ax1 = self.plot.figure.axes[0]
def process_on_special_object(self):
# do a long fitting process involving the properties of the special_object
return
def update_GUI(self):
# change the GUI to reflect changes made to special_object
self.line.set_data(special_object.X, special_object.Y)
self.plot.draw_artist(self.line)
self.plot.figure.canvas.blit(self.plot.bbox)
return
# This window just has a button to make all of the tabs in separate processes
class MainWindow(QtGui.QMainWindow):
def __init__(self, parent = None):
# This GUI stuff shouldn't be too important
QtGui.QMainWindow.__init__(self)
self.resize(int(app.desktop().screenGeometry().width()*.6), int(app.desktop().screenGeometry().height()*.6))
self.tabs_list = []
central_widget = QtGui.QWidget(self)
self.main_tab_widget = QtGui.QTabWidget()
self.layout = QtGui.QHBoxLayout(central_widget)
button = QtGui.QPushButton('Open Tabs')
self.layout.addWidget(button)
self.layout.addWidget(self.main_tab_widget)
QtCore.QObject.connect(button, QtCore.SIGNAL("clicked()"), self.open_tabs)
self.setCentralWidget(central_widget)
central_widget.setLayout(self.layout)
# Here we open several tabs and put them in different processes
def open_tabs(self):
for i in range(0, 10):
# this is just some random data for the objects
X = np.arange(1240.0/1350.0, 1240./200., 0.01)
Y = np.array(np.e**.2*X + np.sin(10*X)+np.cos(4*X))
# Here the special tab is created
new_tab = Special_Tab(self.main_tab_widget, special_object(str(i), X, Y))
self.main_tab_widget.addTab(new_tab, str(i))
# this part works fine without the .start() function
self.tabs_list.append(mp.Process(target=new_tab))
# this is where pickling errors occur
self.tabs_list[-1].start()
return
if __name__ == "__main__":
app = QtGui.QApplication([])
win = MainWindow()
win.show()
sys.exit(app.exec_())
I have noticed that the errors are coming from matplotlib axes (I'm not sure how?) and gives the error pickle.PicklingError: Can't pickle <class 'matplotlib.axes.AxesSubplot'>: it's not found as matplotlib.axes.AxesSubplot. In addition, I have noticed that commenting out the matplotlib plots will also give the pickling error pickle.PicklingError: Can't pickle <function <lambda> at 0x012A2B30>: it's not found as lmfit.parameter.<lambda>. I think this is because lambda functions cannot be pickled and I guess lmfit has a lambda somewhere in it's depths... but I don't really know what to do without some way around these errors.
Oddly enough, the error I see form the original code (not the simplified version shown here) is slightly different, but still basically the same in sentiment. The error I get in my other code is pickle.PicklingError: Can't pickle 'BufferRegion' object: <BufferRegion object at 0x037EBA04>
Does anyone have a better solution to this problem by moving the objects around in regards to where I pass them or any other ideas?
I very much appreciate your time and effort and any help on this problem.
EDIT: I tried unutbu's idea in a way, but with some alterations to the position of the process funciton.
The only problem with the proposed solution is that the do_long_fitting_process() function calls another function which iteratively updates the lines in the matplotlib plots. So the do_long_fitting_process() needs to have some access to the Special_Tab properties to change them and show the updates to the GUI.
I have tried doing this by pushing the do_long_fitting_process() function to just a global function and calling this:
[code]
def open_tabs(self):
for i in range(0, 10):
...
self.tabs_list.append(new_tab)
consumer, producer = mp.Pipe()
process = mp.Process(target=process_on_special_object, args=(producer,))
process.start()
while(True):
message = consumer.recv()
if message == 'done':
break
tab.update_GUI(message[0], message[1])
process_list[-1].join()
[/code]
Where I am passing the data to update_GUI() via a mp.Pipe(), but the window just goes to "Not Responding" as soon as I start the processes.

Separate the GUI code from the computation code. The GUI must run in a single process (though it may spawn multiple threads). Let the computation code be contained within in the special_object.
Let the Special_Tab call mp.Process when you want to perform a long-running computation:
class special_object:
def do_long_fitting_process(self):
pass
class Special_Tab(QtGui.QTabWidget):
def process_on_special_object(self):
# do a long fitting process involving the properties of the
# special_object
proc = mp.Process(target = self.special_object.do_long_fitting_process)
proc.start()
class MainWindow(QtGui.QMainWindow):
def open_tabs(self):
for i in range(0, 10):
...
self.tabs_list.append(new_tab)
new_tab.process_on_special_object()

The problem is that not all parts of an Axes object can be serialized, which is necessary to move data between the processes. I would suggest a slight re-organization of your code to push computation off to separate processes (that is any thing that will take more than a fraction of a second), but keep all of your plotting on the main process and only pass data back and forth.
Another option is to us QThread See time.sleep() required to keep QThread responsive? for two different ways of implementing it (one in the question, one in my answer).

How to combine Cmd and Matplotlib in python

I'd like to combine interactive plotting in Matplotlib and the Command line interface Cmd in python. How can I do this?
Can I use threading? I tried the following:
from cmd import Cmd
import matplotlib.pylab as plt
from threading import Thread
class MyCmd(Cmd):
def __init__(self):
Cmd.__init__(self)
self.fig = plt.figure()
self.ax = self.fig.add_subplot(1,1,1)
def do_foo(self, arg):
self.ax.plot(range(10))
self.fig.canvas.draw()
if __name__=='__main__':
c = MyCmd()
Thread(target=c.cmdloop).start()
plt.show()
It opens a figure window and I can type commands in the console that are actually executed. When the "foo" command is executed it draws in the figure window. So far everything is ok. When I reenter the console, however, the console seems to be stuck and there is now new command window. But when I click into the figure window the console outputs a new command prompt and I can enter a new command.
It seems the two loops are not really interleaved or something. Is there a better, more common way?

I found something that works, but is rather ugly
from cmd import Cmd
import matplotlib.pylab as plt
from threading import Thread
import time
class MyCmd(Cmd):
def __init__(self):
Cmd.__init__(self)
self.fig = plt.figure()
self.ax = self.fig.add_subplot(1,1,1)
def do_foo(self, arg):
self.ax.plot(range(10))
self.fig.canvas.draw()
if __name__=='__main__':
plt.ion()
c = MyCmd()
def loop():
while True:
c.fig.canvas.draw()
time.sleep(0.1)
Thread(target=loop).start()
c.cmdloop()
This simply calls the draw method of the figure periodically. If I don't do this, the figure is not redrawn, when it was occluded and comes to the front again.
But this seems ugly. Is there a better way?

iPython is an pretty popular. Take a look at Using matplotlib in a python shell.