Having a DF of predicted (column) an actual (column) values. I would like to plot an overlaid histogram that have the same look as when using the hue property. I couldn't find a way of doing so without reconstructing my original data.
Here is an example of what I'm trying to do:
df = pd.DataFrame({'A':np.random.uniform(low=0.0, high=9.0, size=(150,)),'P':np.random.uniform(low=0.0, high=9.0, size=(150,))})
actual = df.A.to_frame()
predicted = df.P.to_frame()
print(df.head())
actual.columns = ['value']
actual['t'] = 'A'
predicted.columns = ['value']
predicted['t'] = 'P'
tmp = pd.concat([actual,predicted])
print(tmp.head())
sns.histplot(data=tmp,x='value' ,hue="t")
Output:
Original DF
A P
0 2.546046 2.503833
1 4.797077 2.306720
2 1.358222 4.839675
3 7.063206 8.828486
4 3.010978 7.406337
Manipulated DF
value t
0 2.546046 A
1 4.797077 A
2 1.358222 A
3 7.063206 A
4 3.010978 A
<matplotlib.axes._subplots.AxesSubplot at 0x7fd657112fd0>
Q: How can I get similar results without manipulating my original DF?
Unless I am misunderstanding what you're asking.
From the docs:
If neither x nor y is assigned, the dataset is treated as wide-form, and a histogram is drawn for each numeric column
df = pd.DataFrame(
{
"A": np.random.uniform(low=0.0, high=9.0, size=(150,)),
"P": np.random.uniform(low=0.0, high=9.0, size=(150,)),
}
)
ax = sns.histplot(df)
I tried to follow the Altair example of "Text over a Heatmap" but came across some problems.
My dataset consists of two indexes (N, Z) and a value column color.
I would like to set the origin at the bottom-left side.
How to display the heatmap with text in a script?
Is it possible to put labels (N, Z) in each pixel?
Below attached the part of the code.
def chart_altair(self):
import altair as alt
data = self.df.dropna().reset_index(name='color')
# Configure common options
base = alt.Chart(data).encode(
alt.X('N:O'),
alt.Y('Z:O'),
)
# Configure heatmap
heatmap = base.mark_rect().encode(
color=alt.Color(
'color:Q',
scale=alt.Scale(scheme='viridis'),
legend=alt.Legend()
)
)
text = base.mark_text(baseline='middle').encode(
text='color:Q'
)
I pasted below a few rows of my dataset which consists of two indexses Z, N, and a value column color (represents actually a atomic mass table). The "heatmap" should be similar to the chart of nuclei, with the neutron number increase to the right x-axis, and the proton number increases upwards in the y-axis. However, the proton number increases in the opposite direction (in the negative y-axis).
Z N color
0 1 8.07
1 0 7.29
1 1 13.14
1 2 14.95
2 1 14.93
1 3 24.62
2 2 2.42
3 3 14.09
4 2 18.38
2 5 26.07
3 4 14.91
4 3 15.77
5 2 27.68
Here is an example where I concatenate the two labels using transform_calculate. You could also do this by creating the label column in pandas instead.
import altair as alt
import numpy as np
import pandas as pd
# Compute x^2 + y^2 across a 2D grid
x, y = np.meshgrid(range(-5, 5), range(-5, 5))
z = x ** 2 + y ** 2
# Convert this grid to columnar data expected by Altair
source = pd.DataFrame({'x': x.ravel(),
'y': y.ravel(),
'z': z.ravel()})
heatmap = alt.Chart(source).mark_rect().encode(
x='x:O',
y='y:O',
color='z:Q'
)
heatmap + heatmap.mark_text().transform_calculate(label = '"" + datum.x + datum.y').encode(
text='label:N',
color=alt.value('black'))
I have a dataset with 7 columns - level,Time_30,Time_60,Time_90,Time_120,Time_150 and Time_180
My main goal is to do a time-series anomaly detection using cell count in a 30-minute interval.
I want to do the following data preparation steps:
(I) melt/reshape the df into the appropriate time-series format (from wide to long)- consolidate the columns time_30, time_60 ,....., time_180 into one column time with 6 levels (30,60,.....,180)
(II) since the result from (I) comes out as 30,60,.....180, I want to set the time column as the appropriate time or date format for time-series (something like this '%H:%M:%S')
(III) use a for-loop to plot the time-series plot for each level - A, B,...., F) for comparison purposes.
(IV) Anomaly detection
# generate/import dataset
import pandas as pd
df = pd.DataFrame({'level':[A,B,C,D,E,F],
'Time_30':[1993.05,1999.45, 2001.11, 2007.39, 2219.77],
'Time_60':[2123.15,2299.59, 2339.19, 2443.37, 2553.15],
'Time_90':[2323.56,2495.99,2499.13, 2548.71, 2656.0],
'Time_120':[2355.52,2491.19,2519.92,2611.81, 2753.11],
'Time_150':[2425.31,2599.51, 2539.9, 2713.77, 2893.58],
'Time_180':[2443.35,2609.92, 2632.49, 2774.03, 2901.25]} )
Desired outcome
# first series
level, time, count
A, 30, 1993.05
B, 60, 2123.15
C, 90, 2323.56
D, 120, 2355.52
E, 150, 2425.31
F, 180, 2443.35
# 2nd series
level,time,count
A,30,1999.45
B,60,2299.59
C,90,2495.99
D,120,2491.19
E,150,2599.51
F,180,2609.92
.
.
.
.
# up until the last series
See below for my attempt
# (I)
df1 = pd.melt(df,id_vars = ['level'],var_name = 'time',value_name = 'count') #
# (II)
df1['time'] = pd.to_datetime(df1['time'],format= '%H:%M:%S' ).dt.time
OR
df1['time'] = pd.to_timedelta(df1['time'], unit='m')
# (III)
plt.figure(figsize=(10,5))
plt.plot(df1)
for timex in range(30,180):
plt.axvline(datetime(timex,1,1), color='k', linestyle='--', alpha=0.3)
# Perform STL Decomp
stl = STL(df1)
result = stl.fit()
seasonal, trend, resid = result.seasonal, result.trend, result.resid
plt.figure(figsize=(8,6))
plt.subplot(4,1,1)
plt.plot(df1)
plt.title('Original Series', fontsize=16)
plt.subplot(4,1,2)
plt.plot(trend)
plt.title('Trend', fontsize=16)
plt.subplot(4,1,3)
plt.plot(seasonal)
plt.title('Seasonal', fontsize=16)
plt.subplot(4,1,4)
plt.plot(resid)
plt.title('Residual', fontsize=16)
plt.tight_layout()
estimated = trend + seasonal
plt.figure(figsize=(12,4))
plt.plot(df1)
plt.plot(estimated)
plt.figure(figsize=(10,4))
plt.plot(resid)
# Anomaly detection
resid_mu = resid.mean()
resid_dev = resid.std()
lower = resid_mu - 3*resid_dev
upper = resid_mu + 3*resid_dev
anomalies = df1[(resid < lower) | (resid > upper)] # returns the datapoints with the anomalies
anomalies
plt.plot(df1)
for timex in range(30,180):
plt.axvline(datetime(timex,1,1), color='k', linestyle='--', alpha=0.6)
plt.scatter(anomalies.index, anomalies.count, color='r', marker='D')
Please note: if you can only attempt I and/or II that would be much appreciated.
I made a few small edits to your sample dataframe based on my comment above:
import pandas as pd
df = pd.DataFrame({'level':['A','B','C','D','E'],
'Time_30':[1993.05,1999.45, 2001.11, 2007.39, 2219.77],
'Time_60':[2123.15,2299.59, 2339.19, 2443.37, 2553.15],
'Time_90':[2323.56,2495.99,2499.13, 2548.71, 2656.0],
'Time_120':[2355.52,2491.19,2519.92,2611.81, 2753.11],
'Time_150':[2425.31,2599.51, 2539.9, 2713.77, 2893.58],
'Time_180':[2443.35,2609.92, 2632.49, 2774.03, 2901.25]} )
First, manipulate the Time_* column names to be integer values:
timecols = [int(c.replace("Time_","")) for c in df.columns if c != 'level']
df.columns = ['level'] + timecols
After that you can pd.melt() like you were thinking, yielding a datarame with all those "series" you mentioned above concatenated together:
df1 = df.melt(id_vars=['level'], value_vars=timecols, var_name='time', value_name='count').sort_values(['level','time']).reset_index(drop=True)
print(df1.head(10))
level time count
0 A 30 1993.05
1 A 60 2123.15
2 A 90 2323.56
3 A 120 2355.52
4 A 150 2425.31
5 A 180 2443.35
6 B 30 1999.45
7 B 60 2299.59
8 B 90 2495.99
9 B 120 2491.19
If you want to loop over the levels, select them with:
for level in df1['level'].unique():
tmp = df1[df1['level']==level]
or
for level in df1['level'].unique():
tmp = df1[df1['level']==level].copy()
...if you intend to modify/add data to the tmp dataframe.
As for making timestamps, you could do:
df1['time'] = pd.to_timedelta(df1['time'], unit='min')
...like you were attempting, but it depends on how you're using it. If you just want strings that look like "00:30:00", etc, you can try something like:
df1['time'] = pd.to_timedelta(df1['time'], unit='min').apply(lambda x:str(x)[-8:])
Anyway, hope that gets you on track for what you need.
I am using python with matplotlib and need to visualize distribution percentage of sub-groups of an data set.
imagine this tree:
Data --- group1 (40%)
-
--- group2 (25%)
-
--- group3 (35%)
group1 --- A (25%)
-
--- B (25%)
-
--- c (50%)
and it can go on, each group can have several sub-groups and same for each sub group.
How can i plot a proper chart for this info?
I created a minimal reproducible example that I think fits your description, but please let me know if that is not what you need.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
data = pd.DataFrame()
n_rows = 100
data['group'] = np.random.choice(['1', '2', '3'], n_rows)
data['subgroup'] = np.random.choice(['A', 'B', 'C'], n_rows)
For instance, we could get the following counts for the subgroups.
In [1]: data.groupby(['group'])['subgroup'].value_counts()
Out[1]: group subgroup
1 A 17
C 16
B 5
2 A 23
C 10
B 7
3 C 8
A 7
B 7
Name: subgroup, dtype: int64
I created a function that computes the necessary counts given an ordering of the columns (e.g. ['group', 'subgroup']) and incrementally plots the bars with the corresponding percentages.
import matplotlib.pyplot as plt
import matplotlib.cm
def plot_tree(data, ordering, axis=False):
"""
Plots a sequence of bar plots reflecting how the data
is distributed at different levels. The order of the
levels is given by the ordering parameter.
Parameters
----------
data: pandas DataFrame
ordering: list
Names of the columns to be plotted.They should be
ordered top down, from the larger to the smaller group.
axis: boolean
Whether to plot the axis.
Returns
-------
fig: matplotlib figure object.
The final tree plot.
"""
# Frame set-up
fig, ax = plt.subplots(figsize=(9.2, 3*len(ordering)))
ax.set_xticks(np.arange(-1, len(ordering)) + 0.5)
ax.set_xticklabels(['All'] + ordering, fontsize=18)
if not axis:
plt.axis('off')
counts=[data.shape[0]]
# Get colormap
labels = ['All']
for o in reversed(ordering):
labels.extend(data[o].unique().tolist())
# Pastel is nice but has few colors. Change for a larger map if needed
cmap = matplotlib.cm.get_cmap('Pastel1', len(labels))
colors = dict(zip(labels, [cmap(i) for i in range(len(labels))]))
# Group the counts
counts = data.groupby(ordering).size().reset_index(name='c_' + ordering[-1])
for i, o in enumerate(ordering[:-1], 1):
if ordering[:i]:
counts['c_' + o]=counts.groupby(ordering[:i]).transform('sum')['c_' + ordering[-1]]
# Calculate percentages
counts['p_' + ordering[0]] = counts['c_' + ordering[0]]/data.shape[0]
for i, o in enumerate(ordering[1:], 1):
counts['p_' + o] = counts['c_' + o]/counts['c_' + ordering[i-1]]
# Plot first bar - all data
ax.bar(-1, data.shape[0], width=1, label='All', color=colors['All'], align="edge")
ax.annotate('All -- 100%', (-0.9, 0.5), fontsize=12)
comb = 1 # keeps track of the number of possible combinations at each level
for bar, col in enumerate(ordering):
labels = sorted(data[col].unique())*comb
comb *= len(data[col].unique())
# Get only the relevant counts at this level
local_counts = counts[ordering[:bar+1] +
['c_' + o for o in ordering[:bar+1]] +
['p_' + o for o in ordering[:bar+1]]].drop_duplicates()
sizes = local_counts['c_' + col]
percs = local_counts['p_' + col]
bottom = 0 # start at from 0
for size, perc, label in zip(sizes, percs, labels):
ax.bar(bar, size, width=1, bottom=bottom, label=label, color=colors[label], align="edge")
ax.annotate('{} -- {:.0%}'.format(label, perc), (bar+0.1, bottom+0.5), fontsize=12)
bottom += size # stack the bars
ax.legend(colors)
return fig
With the data shown above we would get the following.
fig = plot_tree(data, ['group', 'subgroup'], axis=True)
Have you tried stacked bar graph?
https://matplotlib.org/gallery/lines_bars_and_markers/bar_stacked.html#sphx-glr-gallery-lines-bars-and-markers-bar-stacked-py
Below is an example DataFrame.
joaquin manolo
xx 0 0.000000e+00 44.000000
1 1.570796e+00 52.250000
2 3.141593e+00 60.500000
3 4.712389e+00 68.750000
4 6.283185e+00 77.000000
yy 0 0.000000e+00 37.841896
1 2.078796e+00 39.560399
2 5.292179e-17 41.026434
3 -8.983291e-02 42.304767
4 -4.573916e-18 43.438054
As you can see, the row index has two levels, ['xx', 'yy'] and [0, 1, 2, 3, 4]. I want to call DataFrame.plot() in such a way that it will produce two subplots, one for joaquin and one for manolo, and where I can specify to use data.loc["xx", :] for the domain data and to use data.loc["yy", :] for the ordinate data. In addition, I want the option to supply the subplots on which the plots should be drawn, in a list (or array) of matplotlib.axes._subplots.AxesSubplot instances, such as those that can be returned by the DataFrame.hist() method. How can this be done?
Generating the data above
Just in case you're wondering, below is the code I used to generate the data. If there is an easier way to generate this data, I'd be very interested to know as a side-note.
joaquin_dict = {}
xx_joaquin = numpy.linspace(0, 2*numpy.pi, 5)
yy_joaquin = 10 * numpy.sin(xx_joaquin) * numpy.exp(-xx_joaquin)
for i in range(len(xx_joaquin)):
joaquin_dict[("xx", i)] = xx_joaquin[i]
joaquin_dict[("yy", i)] = yy_joaquin[i]
manolo_dict = {}
xx_manolo = numpy.linspace(44, 77, 5)
yy_manolo = 10 * numpy.log(xx_manolo)
for i in range(len(xx_manolo)):
manolo_dict[("xx", i)] = xx_manolo[i]
manolo_dict[("yy", i)] = yy_manolo[i]
data_dict = {"joaquin": joaquin_dict, "manolo": manolo_dict}
data = pandas.DataFrame.from_dict(data_dict)
Just use a for loop:
fig, axes = pl.subplots(1, 2)
for ax, col in zip(axes, data.columns):
data[col].unstack(0).plot(x="xx", y="yy", ax=ax, title=col)
output: