Develop Reference

How to remove points from a dataframe based on a selected area on a plot

I have some experimental data that is often flawed with artifacts exemplified with something like this:
I need a quick way to manually select these random spikes and remove them from datasets.
I figured that any plotting library with a focus on interactive plots should have an easy way to do this but so far I keep struggling with finding a simple way to do what I want.
I'm a Matplotlib/Seaborn guy and this calls for interactive solution. I briefly checked Plotly, Bokeh and Altair and decided to go with the first one. My first attempt looks like this:
import pandas as pd
import plotly.graph_objects as go
from ipywidgets import interactive, HBox, VBox, Button
url='https://drive.google.com/file/d/1hCX8Bn_y30aXVN_TyHTTx015u44pO9yB/view?usp=sharing'
url='https://drive.google.com/uc?id=' + url.split('/')[-2]
df = pd.read_csv(url, index_col=0)
f = go.FigureWidget()
for col in df.columns[-1:]:
f.add_scatter(x = df.index, y=df[col], mode='markers+lines',
selected_marker=dict(size=5, color='red'),
marker=dict(size=1, color='lightgrey', line=dict(width=1, color='lightgrey')))
t = go.FigureWidget([go.Table(
header=dict(values=['selector range'],
fill = dict(color='#C2D4FF'),
align = ['left'] * 5),
cells=dict(values=['None selected' for col in ['ID']],
fill = dict(color='#F5F8FF'),
align = ['left'] * 5)
)])
def selection_fn(trace,points,selector):
t.data[0].cells.values = [selector.xrange]
def update_axes(dataset):
scatter = f.data[0]
scatter.x = df.index
scatter.y = df[dataset]
f.data[0].on_selection(selection_fn)
axis_dropdowns = interactive(update_axes, dataset = df.columns)
button1 = Button(description="Remove points")
button2 = Button(description="Reset")
button3 = Button(description="Fit data")
VBox((HBox((axis_dropdowns.children)), HBox((button1, button2, button3)), f,t))
Which gives:
So I managed to get Selector Box x coordinates (and temporarily print them inside the table widget). But what I couldn't figure out is how to easily bind a function to button1 that would take as an argument Box Selector coordinates and remove selected points from a dataframe and replot the data. So something like this:
def on_button_click_remove(scatter.selector.xrange):
mask = (df.index >= scatter.selector.xrange[0]) & (df.index <= scatter.selector.xrange[1])
clean_df = df.drop(df.index[mask])
scatter(data = clean_df...) #update scatter plot
button1 = Button(description="Remove points", on_click = on_button_click_remove)
I checked https://plotly.com/python/custom-buttons/ but I am still not sure how to use it for my purpose.

I suggest to use Holoviews and Panel.
They are high level visualization tools that facilitate the creation and control of low level bokeh, matplotlib or plotly figures.
Here are an example:
import panel as pn
import holoviews as hv
import pandas as pd
from bokeh.models import ColumnDataSource
# This example use bokeh as backend.
# You can try plotly or matplotlib with minor modification on the codes below.
# For example you can use on_selection callback from Plotly
# https://plotly.com/python/v3/selection-events/
hv.extension('bokeh')
display( pn.extension( ) ) # activate panel
df=pd.read_csv('spiked_data.csv',index_col=0).reset_index()
pt = hv.Points(
data=df, kdims=['index', 'A' ]
).options( marker='x', size=2,
tools=['hover', 'box_select', 'lasso_select', 'reset'],
height=250, width=600
)
fig = hv.render(pt)
source = fig.select({'type':ColumnDataSource})
bt = pn.widgets.Button(name='remove selected')
def rm_sel(evt):
i = df.iloc[source.selected.indices].index # get index to delete
df.drop(i, inplace=True, errors='ignore') # modify dataframe
source.data = df # update data source
source.selected.indices=[] # clear selection
pn.io.push_notebook(app) # update figure
bt.on_click(rm_sel)
app=pn.Column(fig,'Click to delete the selected points', bt)
display(app)
A related example can be found in this SO answer

slider for choropleth map plotly shows error - <ipython-input-17-c5022c9e0aab>:13: SettingWithCopyWarning:

I am trying to add a slider for my choropleth map using plotly. I have been unable to see the map in google colab but jupyter notebook has worked fine.
I have a dataset with values for each country of the world over a period of years. I want to make a slider underneath so you can scroll through the years and see the colours change. I have been able to get data from one year, but I cannot get the slider to appear under my map, and I get a large pink error message saying the value is trying to be set on a copy of a slice from a DataFrame.
Is there anything I can do to fix this? Please find attached my code
pip install chart-studio
import pandas as pd
import chart_studio.plotly as py
import plotly.offline as po
import matplotlib.pyplot as plt
%matplotlib inline
import plotly
import plotly.graph_objs as go
import plotly.offline as offline
from plotly.graph_objs import *
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, pilot
df=pd.read_csv("https://raw.githubusercontent.com/jamesjeffery77/jamesjeffery77.github.io/main/share-electricity-low-carbon_fullDataset.csv")
year=1985
### create empty list for data object:
data_slider = []
# Populate data object
for year in df.year.unique():
#select year
df2=df[(df['year']==year)]
for col in df2.columns:
df2[col] = df2[col].astype(str)
#dictionary with date for current year
data_one_year = dict(
type='choropleth',
locations=df2['code'],
z=df2['percentage'].astype(float),
text=df2['country'],)
#add data to next year
data_slider.append(data_one_year)
#steps for the slider
steps = []
for i in range(len(data_slider)):
step=dict(method='restyle',
args=['visible',[False] * len(data_slider)],
label='year {}'.format(i+1985))
step['args'][1][i] = True
steps.append(step)
## create the sliders object from the steps
sliders = [dict(active=0, pad={"t": 1}, steps=steps)]
#layout such that there is a global view
layout = dict(title = 'Global GDP - natural earth',
geo = dict( projection = {'type':'natural earth'},
showlakes = True,
lakecolor = 'rgb(0,191,255)'))
fig = dict(data=data_slider, layout=layout)
#plot graph
plotly.offline.iplot(fig)

You can get around the value is trying to be set on a copy of a slice from a DataFrame warning, you can set the columns as strings in one step - you don't need to loop through each column name separately.
# Populate data object
for year in df.year.unique():
#select year
df2=df[(df['year']==year)]
df2.columns = df2.columns.astype(str)
#dictionary with date for current year
data_one_year = dict(
type='choropleth',
locations=df2['code'],
z=df2['percentage'].astype(float),
text=df2['country'],)
#add data to next year
data_slider.append(data_one_year)
After this change, I ran your code in Google Colab, and I am able to display the map.

Interactive Choropleth using plotly & ipywidgets in python

I am trying to create an interactive choropleth using Plotly and ipywidgets for the dataframe which the head looks like the one shown below.
Data1.head()
The plot sums the values for all the available dates in the dataframe for each county.
Output for available FIPS (aggregated values)
I want to filter the values for a particular date using a dropdown menu and plot the values corresponding to this date.
import matplotlib.pyplot as plt
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from plotly import *
import plotly.figure_factory as ff
import plotly.graph_objects as go
date = widgets.Dropdown(
options=['All'] + Data1.date.unique().tolist(),
value='All',
description='Date:',
)
# date = Data1['date'][1] # If I use this instead of the widget defined above, the code works (at the cost of interactivity)
def func(date):
values = Data1[Data1['date']== date]['Infected'].tolist()
fips = Data1[Data1['date']== date]['Fips'].tolist()
colorscale = [
'rgb(193, 193, 193)',
'rgb(239,239,239)',
'rgb(195, 196, 222)',
'rgb(144,148,194)',
'rgb(101,104,168)',
'rgb(65, 53, 132)'
]
fig = ff.create_choropleth(
fips=fips, values=values, scope=['IL'],
binning_endpoints=[14348, 63983, 134827, 426762, 2081313], colorscale=colorscale,
county_outline={'color': 'rgb(255,255,255)', 'width': 0.5}, round_legend_values=True,
legend_title='Infected cases by County', title='Illinois State'
)
fig.layout.template = None
fig.show()
interactive(func, date= date)
I am able to get the dropdown menu using this code, but it does not print the plot.
Alternatively, I tried to hardcode the date (commented in code) and it did return the plot for the specific date, at the loss of interactivity.
Can someone help me to get the plot where I can let the user choose a date from the dropdown menu and plot the choropleth correspondingly?

How can I get all Plotly plots created inside a for loop display in single browser window?

I am trying to create several plots inside a for loop using plotly. Currently all the charts appear in separate tabs of browser. I want all the charts to appear in the same browser window.
In my data frame df, for each unique element in Tool_MeasurementSet column (unique elements saved as a list meas_set) has 16 data points for X-BAR and 16 for SIGMA. I was able to use subplot function to combine X-BAR and SIGMA plot for each element in meas_set. Currently the code is creating plots for each element in meas_set list in a separate tab of the browser. But I want to make all the plots appear in the same browser window with a vertical scroll bar instead of having to move from one tab to another to look at plots.
from plotly import tools
import plotly.plotly as py
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import plotly.offline as pyo
import plotly.graph_objs as go
df = pd.read_csv("C:\\DATA_REPORT_subset.csv")
meas_set = df['Tool_MeasurementSet'].unique()
## params are the column labels in the df dataframe
params = ['Data','UCL','LCL','CL']
for i in meas_set:
fig = tools.make_subplots(rows=2, cols=1,subplot_titles=('X-BAR Subplot','SIGMA Subplot'))
for j in range(0,len(params)):
y_xbar = df[(df['Tool_MeasurementSet']== i) & (df['Chart Type']== 'X-BAR')][params[j]]
x_xbar = df[(df['Tool_MeasurementSet']== i) & (df['Chart Type']== 'X-BAR')]['Date']
y_sigma = df[(df['Tool_MeasurementSet']== i) & (df['Chart Type']== 'SIGMA')][params[j]]
x_sigma = df[(df['Tool_MeasurementSet']== i) & (df['Chart Type']== 'SIGMA')]['Date']
trace1 = go.Scatter(x=x_xbar,y=y_xbar,mode='lines',name=params[j])
trace2 = go.Scatter(x=x_sigma,y=y_sigma,mode='lines',name=params[j])
fig.append_trace(trace1,1,1)
fig.append_trace(trace2,2,1)
fig['layout'].update(title= i)
pyo.plot(fig)
I want all the plots to appear in a single browser window with a scroll bar.

You could just move the point where you declare the figure outside of the loop and give it more rows or columns.
For example, make a figure with as many columns as there are datapoints. Then put the plots in the ith column. Something like:
# use len(meas_set) as number of columns
fig = tools.make_subplots(rows=2, cols=len(meas_set), subplot_titles=('X-BAR Subplot','SIGMA Subplot'))
for i in meas_set:
for j in range(0,len(params)):
# your logic here
trace1 = go.Scatter(x=x_xbar,y=y_xbar,mode='lines',name=params[j])
trace2 = go.Scatter(x=x_sigma,y=y_sigma,mode='lines',name=params[j])
# use i for column position
fig.append_trace(trace1,1,i)
fig.append_trace(trace2,2,i)

Create a stacked graph or bar graph using plotly in python

I have data like this :
[ ('2018-04-09', '10:18:11',['s1',10],['s2',15],['s3',5])
('2018-04-09', '10:20:11',['s4',8],['s2',20],['s1',10])
('2018-04-10', '10:30:11',['s4',10],['s5',6],['s6',3]) ]
I want to plot a stacked graph preferably of this data.
X-axis will be time,
it should be like this
I created this image in paint just to show.
X axis will show time like normal graph does( 10:00 ,April 3,2018).
I am stuck because the string value (like 's1',or 's2' ) will change in differnt bar graph.
Just to hard code and verify,I try this:
import plotly
import plotly.graph_objs as go
import matplotlib.pyplot as plt
import matplotlib
plotly.offline.init_notebook_mode()
def createPage():
graph_data = []
l1=[('com.p1',1),('com.p2',2)('com.p3',3)]
l2=[('com.p1',1),('com.p4',2)('com.p5',3)]
l3=[('com.p2',8),('com.p3',2)('com.p6',30)]
trace_temp = go.Bar(
x='2018-04-09 10:18:11',
y=l1[0],
name = 'top',
)
graph_data.append(trace_temp)
plotly.offline.plot(graph_data, filename='basic-scatter3.html')
createPage()
Error I am getting is Tuple Object is not callable.
So can someone please suggest some code for how I can plot such data.
If needed,I may store data in some other form which may be helpful in plotting.
Edit :
I used the approach suggested in accepted answer and succeed in plotting using plotly like this
fig=df.iplot(kin='bar',barmode='stack',asFigure=True)
plotly.offline.plt(fig,filename="stack1.html)
However I faced one error:
1.When Time intervals are very close,Data overlaps on graph.
Is there a way to overcome it.

You could use pandas stacked bar plot. The advantage is that you can create with pandas easily the table of column/value pairs you have to generate anyhow.
from matplotlib import pyplot as plt
import pandas as pd
all_data = [('2018-04-09', '10:18:11', ['s1',10],['s2',15],['s3',5]),
('2018-04-09', '10:20:11', ['s4',8], ['s2',20],['s1',10]),
('2018-04-10', '10:30:11', ['s4',10],['s5',6], ['s6',3]) ]
#load data into dataframe
df = pd.DataFrame(all_data, columns = list("ABCDE"))
#combine the two descriptors
df["day/time"] = df["A"] + "\n" + df["B"]
#assign each list to a new row with the appropriate day/time label
df = df.melt(id_vars = ["day/time"], value_vars = ["C", "D", "E"])
#split each list into category and value
df[["category", "val"]] = pd.DataFrame(df.value.values.tolist(), index = df.index)
#create a table with category-value pairs from all lists, missing values are set to NaN
df = df.pivot(index = "day/time", columns = "category", values = "val")
#plot a stacked bar chart
df.plot(kind = "bar", stacked = True)
#give tick labels the right orientation
plt.xticks(rotation = 0)
plt.show()
Output: