I'm building a dashboard to display on a TV for production data using Django+RaspberryPI but the rendering is quite slow.
The dashboard contains 12 plotly indicator (number type) and 3 bar charts.
At my view, I'm reading a .csv file using pandas, about ~50 lines, then I start creating variables to generates the plotly graphs and pass as context, like below
Create plotly indicator:
df_36_ordered = df_detalhado_full.query('Maquina=="3.6"').sort_values(by='Data')
oee_36 = df_36_ordered['OEE'].iloc[-1:].iloc[0]
Create dataframe for bar plot:
df_36_historico = df_36_ordered[['dia_semana', 'OEE']].tail(6).iloc[:-1]
And then I generates the html for all of it:
oee_36 = plot(fig, output_type='div', include_plotlyjs=False, show_link=False, link_text='')
I'm looking for tips/alternatives on improving rendering the page.
From entering address to full load it's taking 20 seconds.
Thanks in advance!!
Dask Instead of Pandas, however, your dataset is only 50 lines, I don't know if it will improve processing time.
Try to implement multithreading to process your graphs simultaneously.
If you can only use plotly.graph_objects it's will probably make it faster, but harder for a similar result. Try to delete useless feature, plotly give you lot of built in feature when you plot somethings, those features will make your rendering slower
Related
I am trying to generate interactive plots using plotly. So far I have been using pio.renderers.default = 'browser' to create the plot as html and review it interactively. Nevertheless I have a task where I need to generate ~100 charts, and I would like to save them for later and be able to retrieve them for review them interactively without need of running the script again. Can this be possible with plotly?
I have a dataset with millions of Latitude/Longitude points that we are plotting at high resolution using plotly-dash with a Densitymapbox:
data = pandas.DataFrame()
# ...
go.Densitymapbox(
lat=data['Latitude'],
lon=data['Longitude'],
z=data['Count'],
hoverinfo='skip',
# ...
)
According to Mapbox, their library should support millions of points without issue as shown by their demo # https://demos.mapbox.com/100mpoints/
When I try to do this, it does appear that Mapbox is able to handle the requests. However in my implementation with plotly/dash, unlike the demo above, the browser gets underwater. The first load works fine (although does use a lot of memory), but on a reload of the data, Chrome crashes and Firefox reports an out of memory error to the console and does not update the heatmap.
The data set I am using is 1093737 points. Doing back-of-the-napkin math, this should only be < ~25 MB of data (1093737 * (8 + 8 + 8)) for 2 double precision floating point values and 1 (64bit) integer, and the amount of data sent to the browser does show this. However, the browser process balloons in memory to over 3.5GB and then on subsequent reloads, it appears the browser runs out of memory.
Are there any facilities in dash/plotly to prevent this from taking down the browser? I do not need to interact with the points of density plot, and have set the hoverinfo='skip' to indicate that, but would like to keep the interactivity of the heatmap recalculating the overlay when the map zoom changes. I am investigating other alternatives such as rasterizing the heatmap server side using datashader, but that would remove this interactivity which I would like to keep.
LensPy was created to solve this exact problem. It is built on top of Plotly Dash to allow you to plot very large datasets while maintaining fluid interactivity. Here is an example of how you can achieve this with a Mapbox.
import pandas as pd
import plotly.express as px
from lenspy import DynamicPlot
df = pd.read_csv(
'https://raw.githubusercontent.com/plotly/datasets/master/earthquakes-23k.csv')
fig = px.density_mapbox(df,
lat='Latitude', lon='Longitude',
z='Magnitude',
radius=10,
center=dict(lat=0, lon=180),
zoom=0,
mapbox_style="stamen-terrain")
plot = DynamicPlot(fig)
plot.show()
Disclaimer: I am the creator of LensPy.
I am trying to plot something with a huge number of data points (2mm-3mm) using plotly.
When I run
py.iplot(fig, filename='test plot')
I get the following error:
Woah there! Look at all those points! Due to browser limitations, the Plotly SVG drawing functions have a hard time graphing more than 500k data points for line charts, or 40k points for other types of charts. Here are some suggestions:
(1) Use the `plotly.graph_objs.Scattergl` trace object to generate a WebGl graph.
(2) Trying using the image API to return an image instead of a graph URL
(3) Use matplotlib
(4) See if you can create your visualization with fewer data points
If the visualization you're using aggregates points (e.g., box plot, histogram, etc.) you can disregard this warning.
So then I try to save it with this:
py.image.save_as(fig, 'my_plot.png')
But then I get this error:
PlotlyRequestError: Unknown Image Server Error
How do I do this properly? I don't care if it's a still image or an interactive display within my notebook.
Plotly really seems to be very bad in this. I am just trying to create a boxplot with 5 Million points, which is no problem in the simple R function "boxplot", but plotly is calculating endlessly for this.
It should be a major issue to improve this. Not all data has to be saved (and shown) in the plotly object. This is the main problem I guess.
one option would be down-sampling your data, not sure if you'd like that:
https://github.com/devoxi/lttb-py
I also have problems with plotly in the browser with large datasets - if anyone has solutions, please write!
Thank you!
You can try the render_mode argument. Example:
import plotly.express as px
import pandas as pd
import numpy as np
N = int(1e6) # Number of points
df = pd.DataFrame(dict(x=np.random.randn(N),
y=np.random.randn(N)))
fig = px.scatter(df, x="x", y="y", render_mode='webgl')
fig.update_traces(marker_line=dict(width=1, color='DarkSlateGray'))
fig.show()
In my computer N=1e6 takes about 5 seconds until the plot is visible, and the "interactiveness" is still very good. With N=10e6 it takes about 1 minute and the plot is not responsive anymore (i.e. it is really slow to zoom, pan or anything).
I'm writing a web interface for a database of genes values of some experiments with CGI in Python and I want to draw a graph for the data queried. I'm using matplotlib.pyplot, draw a graph, save it, and perform it on the web page. But usually there are many experiments queried hence there are a lot of values. Sometimes I want to know which experiment does one value belong to because it's a big value, whereas it's hard to identify because the picture is small in size. The names of the experiments are long strings so that it will mess the x axis if I put all the experiment names on the x axis.
So I wonder if there is a way to draw a graph that can interact with users, i.e. if I point my mouse to some part on the graph, there would be one small window appears and tells me the exact value and what is the experiment name here. And the most important is, I can use this function when I put the graph on the web page.
Thank you.
What you want is basically D3.js rendering of your plots. As far as I know, there are currently three great ways of achieving this, all under rapid development:
MPLD3 for creating graphs with Matplotlib and serving them as interactive web graphics (see examples in Jake's blog post).
Plotly where you can either generate the plots directly via Plotly or from Matplotlib figures (e.g. using matplotlylib) and have them served by Plotly.
Bokeh if you do not mind moving away from Matplotlib.
I have a large data set of tuples containing (time of event, latitude, longitude) that I need to visualize. I was hoping to generate a 'movie'-like xy-plot, but was wondering if anyone has a better idea or if there is an easy way to do this in Python?
Thanks in advance for the help,
--Leo
get matplotlib
The easiest option is matplotlib. Two particular solutions that might work for you are:
1) You can generate a series of plots, each a snapshot at a given time. These can either be displayed as a dynamic plot in matplotlib, where the axes stay the same and the data moves around; or you can save the series of plots to separate files and later combine them to make a movie (using a separate application). There a number of examples in the official examples for doing these things.
2) A simple scatter plot, where the colors of the circles changes with time might work well for your data. This is super easy. See this, for example, which produces this figure
alt text http://matplotlib.sourceforge.net/plot_directive/mpl_examples/pylab_examples/ellipse_collection.hires.png
I'd try rpy. All the power of R, from within python.
http://rpy.sourceforge.net/
rpy is awesome.
Check out the CRAN library for animations,
http://cran.r-project.org/web/packages/animation/index.html
Of course, you have to learn a bit about R to do this, but if you're planning to do this kind of thing routinely in future it will be well worth your while to learn.
If you are interested in scientific plotting using Python then have a look at Mlab: http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/mlab.html
It allows you to plot 2d / 3d and animate your data and the quality of the charts is really high.
Enthought's Chaco is designed for interactive/updating plots. the api and such takes a little while to get use to, but once you're there it's a fantastic framework to work with.
I have had reasonable success with Python applications generating SVG with animation features embedded, but this was with a smaller set of elements than what you probably have. For example, if your data is about a seismic event, show a circle that shows up when the event happened and grows in size matching the magnitude of the event. A moving indicator over a timeline is really simple to add.
Kaleidoscope (Opera, others maybe, Safari not) shows lots of pieces moving around and I found inspirational. Lots of other good SVG tutorial content on the site too.
You might want to look at PyQwt. It's a plotting library which works with Qt/PyQt.
Several of the PyQwt examples (in the qt4examples directory) show how to create "moving" / dynamically changing plots -- look at CPUplot.py, MapDemo.py, DataDemo.py.