I have this kind of data :
ID x1 x2 x3 x4 x5 x6 x7 x8 x9 x10
1 -0.18 5 -0.40 -0.26 0.53 -0.66 0.10 2 -0.20 1
2 -0.58 5 -0.52 -1.66 0.65 -0.15 0.08 3 3.03 -2
3 -0.62 5 -0.09 -0.38 0.65 0.22 0.44 4 1.49 1
4 -0.22 -3 1.64 -1.38 0.08 0.42 1.24 5 -0.34 0
5 0.00 5 1.76 -1.16 0.78 0.46 0.32 5 -0.51 -2
what's the best method for visualizing this data, i'm using matplotlib to visualizing it, and read it from csv using pandas
thanks
Visualising data in a high-dimensional space is always a difficult problem. One solution that is commonly used (and is now available in pandas) is to inspect all of the 1D and 2D projections of the data. It doesn't give you all of the information about the data, but that's impossible to visualise unless you can see in 10D! Here's an example of how to do this with pandas (version 0.7.3 upwards):
import numpy as np
import pandas as pd
from pandas.plotting import scatter_matrix
#first make some fake data with same layout as yours
data = pd.DataFrame(np.random.randn(100, 10), columns=['x1', 'x2', 'x3',\
'x4','x5','x6','x7','x8','x9','x10'])
#now plot using pandas
scatter_matrix(data, alpha=0.2, figsize=(6, 6), diagonal='kde')
This generates a plot with all of the 2D projections as scatter plots, and KDE histograms of the 1D projections:
I also have a pure matplotlib approach to this on my github page, which produces a very similar type of plot (it is designed for MCMC output, but is also appropriate here). Here's how you'd use it here:
import corner_plot as cp
cp.corner_plot(data.as_matrix(),axis_labels=data.columns,nbins=10,\
figsize=(7,7),scatter=True,fontsize=10,tickfontsize=7)
You may change the plot over the time, for each instant you plot a different "dimension" of the dataframe.
Here an example on how you can do plots that change over the time, you may adjust it for your purposes
import matplotlib.pyplot as plt
import numpy as np
fig = plt.figure()
ax = fig.add_subplot(111)
plt.grid(True)
plt.hold(False)
x = np.arange(-3, 3, 0.01)
for n in range(15):
y = np.sin(np.pi*x*n) / (np.pi*x*n)
line, = ax.plot(x, y)
plt.draw()
plt.pause(0.5)
Related
I am trying to do a histogram like the one below but I am struggling with the bins. This is my code:
plt.subplots(figsize=(2, 1), dpi=400)
width = 0.005
plt.xticks(((density_1.index.unique()) | set(density_2.index.unique())), rotation=90, fontsize=1.5)
plt.yticks(list(set(density_1.unique()) | set(density_2.unique())), fontsize=2)
plt.hist(density_1.index, density_1, width, color='Green', label=condition_1,alpha=0.5)
plt.hist(density_2.index, density_2, width, color='Red', label=condition_2,alpha=0.5,bins=my_beans1)
plt.legend(loc="upper right", fontsize=2)
plt.show()
Those are my pandas:
1st Data sample:
Xticks Yticks
0.27 0.068182
0.58 0.045455
0.32 0.045455
0.47 0.045455
0.75 0.045455
0.17 0.045455
0.43 0.022727
0.66 0.022727
0.11 0.022727
0.68 0.022727
0.59 0.022727
2nd Data sample:
Xticks Yticks
0.94 0.058442
0.86 0.058442
0.74 0.045455
0.93 0.045455
0.99 0.045455
0.71 0.038961
0.63 0.019481
0.97 0.019481
0.87 0.019481
0.84 0.019481
0.75 0.019481
0.89 0.019481
0.80 0.012987
I did this picture by using plt.bar() but I need to do with plt.hist. That is for the full dataset, but I am providing a sample of my dataframe to make it shorter.
I saw some forums and webs to do hist and the use of bins but I always get errors.
I tried something like this:
my_bins1=density_2.unique()
my_bins2=10
Assuming that you want to use a histogram to count the frequency of y-ticks, something like this might be what you are looking for:
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import pandas as pd
data1 = pd.read_csv(r"file1", sep='\t')
data2 = pd.read_csv(r"file2", sep='\t')
data1 = data1.set_index('x_ticks')
data2 = data2.set_index('x_ticks')
plt.figure()
bins=np.linspace(0, 0.1, num=100)
n, bins, rectangles = plt.hist(data1, bins, color = 'green', alpha=0.5, label='dataset 1')
plt.hist(data2, bins, color = 'red', alpha=0.5, label='dataset 2')
plt.legend(loc='upper right')
plt.title('frequency of y-ticks')
plt.show()
Output looks like this:
I have a Pandas data frame with the following structure:
alpha beta gamma mse
0 0.00 0.00 0.00 0.000000
1 0.05 0.05 0.90 0.025411
2 0.05 0.10 0.85 0.025794
3 0.05 0.15 0.80 0.026289
4 0.05 0.20 0.75 0.025320
.. ... ... ... ...
148 0.75 0.05 0.20 0.026816
149 0.75 0.10 0.15 0.025817
150 0.75 0.15 0.10 0.025702
151 0.80 0.05 0.15 0.027104
152 0.80 0.10 0.10 0.025936
I would like to visualise the data frame with a heatmap where alpha is represented on the x-axis, beta is represented on the y-axis, and for each square of the lattice, the mean MSE over all gammas is computed. Is there an easy way to do this by using Seaborn?
Thanks in advance.
For what you showed, yes, you can do with:
sns.heatmap(df.pivot_table(index='beta', columns='alpha', values='mse'))
All the calculation should be done in your DataFrame.
Once you have the data, you could use pivoted DataFrame to build the heatmap
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
# Assuming that you have the df variable with your data
# pivot the data
pivoted = df.pivot('alpha', 'beta', 'mse')
# plot the heatmap
sns.heatmap(pivoted, annot=True)
plt.show()
More information in the official documentation: https://seaborn.pydata.org/generated/seaborn.heatmap.html
I am trying to interpolate a x/y series using matplotlib. The problem I am facing is that spline and interp1d fail because I have recurring values in both the x and y arrays.
I have tried using the spline and interp1d functions from scipy, but both fail because of the recurring values issue
x1 = [0.82 0.82 0.82 0.82 0.82 0.82 0.83 0.83 0.83 0.83 0.83 0.83 0.83]
y1 = [0.93 0.93 0.93 0.93 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94 0.94]
f = interp1d(x1, y1, kind='cubic') #this gives an error: Expect x to be a 1-D sorted array_like.
#another thing I tried
xnew = np.linspace(x1.min(),x1.max(),300)
splined = spline(x1,y1,xnew) #this gives an error: Matrix is singular
I am expecting the interpolated y value to gradually increase with the increase of x. So for example, the corresponding y value for x = 0.82 would be 0.931, 0.932, etc. My goal in the end is to get a smooth curve.
How about using a polynomial?
np.poly1d(np.polyfit(x1, y1, 2))(new_x) # 2 for second degree
I have a dataframe with 1000 rows like below
start_time val
0 15:16:25 0.01
1 15:17:51 0.02
2 15:26:16 0.03
3 15:27:28 0.04
4 15:32:08 0.05
5 15:32:35 0.06
6 15:33:02 0.07
7 15:33:46 0.08
8 15:33:49 0.09
9 15:34:04 0.10
10 15:34:23 0.11
11 15:34:32 0.12
12 15:34:32 0.13
13 15:35:53 0.14
14 15:37:31 0.15
15 15:38:11 0.16
16 15:38:17 0.17
17 15:38:29 0.18
18 15:40:07 0.19
19 15:40:32 0.20
20 15:40:53 0.21
... .... ..
I would like to plot it, with the the time on the x axis. I have used
plt.plot(df['start_time'].dt.total_seconds(),df['val'])
# generate a formatter, using the fields required
fmtr = mdates.DateFormatter("%H:%M")
# need a handle to the current axes to manipulate it
ax = plt.gca()
# set this formatter to the axis
ax.xaxis.set_major_formatter(fmtr)
And it works fine, but on the x axis I have labels which are not showing correct time, see below:
Any help? thank you in advance
You can convert timedeltas to seconds:
plt.plot(df['start_time'].dt.total_seconds(),df['val'])
Solution for converting timedeltas to strings from here, only necessary convert nanoseconds to seconds:
import datetime
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(df['start_time'], df['val'])
def timeTicks(x, pos):
seconds = x / 10**9
d = datetime.timedelta(seconds=seconds)
return str(d)
formatter = matplotlib.ticker.FuncFormatter(timeTicks)
ax.xaxis.set_major_formatter(formatter)
plt.xticks(rotation=90)
plt.show()
I have data as those ones
a b c d e
alpha 5.51 0.60 -0.12 26.90 76284.53
beta 3.39 0.94 -0.17 -0.20 -0.20
gamma 7.98 3.34 -1.41 7.74 28394.93
delta 2.29 1.24 0.40 0.29 0.28
I want to do a nice publishable histogram as this one
but with a break in the y axis so we can figure out the variation of a , b , c , d and e so that data will not be squashed by extreme values in e column as this one but using interlaced colorbar histogram:
I would like to do that in python (matplotlib, pandas, numpy/scipy) or in mathematica... or any other open and free high-level language (R, scilab, ...). Thanks for your help.
edit: using matplotlib through pandas allows to adjust the space between the two subgraph using option button at bottom left "hspace".
Have you seen this example? It's for a broken y-axis plot in matplotlib.
Hope this helps.
Combining with pandas this gives:
import pandas as pd
import matplotlib.pyplot as plt
from StringIO import StringIO
data = """\
a b c d e
alpha 5.51 0.60 -0.12 26.90 76284.53
beta 3.39 0.94 -0.17 -0.20 -0.20
gamma 7.98 3.34 -1.41 7.74 28394.93
delta 2.29 1.24 0.40 0.29 0.28
"""
df = pd.read_csv(StringIO(data), sep='\s+')
f, axis = plt.subplots(2, 1, sharex=True)
df.plot(kind='bar', ax=axis[0])
df.plot(kind='bar', ax=axis[1])
axis[0].set_ylim(20000, 80000)
axis[1].set_ylim(-2, 30)
axis[1].legend().set_visible(False)
axis[0].spines['bottom'].set_visible(False)
axis[1].spines['top'].set_visible(False)
axis[0].xaxis.tick_top()
axis[0].tick_params(labeltop='off')
axis[1].xaxis.tick_bottom()
d = .015
kwargs = dict(transform=axis[0].transAxes, color='k', clip_on=False)
axis[0].plot((-d,+d),(-d,+d), **kwargs)
axis[0].plot((1-d,1+d),(-d,+d), **kwargs)
kwargs.update(transform=axis[1].transAxes)
axis[1].plot((-d,+d),(1-d,1+d), **kwargs)
axis[1].plot((1-d,1+d),(1-d,1+d), **kwargs)
plt.show()