Develop Reference

How to graph semi-large datasets (~20k points) using pandas and matplotlib.pyplot? Are there better tools for graphing?

I'm trying to graph an imported csv file using pandas and matplotlib.pyplot. The csv file has 20k data points and for simplicity is linear. I have the following code:
import pandas as pd
import matplotlib.pyplot as plt
data = pd.read_csv(r'/Users/ephemeralhappiness/Desktop/Packet/20kData.csv')
df = pd.DataFrame(data, columns=['Displacement Into Surface', 'Load On Sample'])
x = df['Load On Sample']
y = df['Displacement Into Surface']
plt.scatter(x, y)
plt.xlabel('Load On Sample')
plt.ylabel('Displacement Into Surface')
plt.show()
When I run the program, I get the following graphical output:
The graph has black marks along the axes and the 20k points are not spaced out at all. How to fix this?

I don't think the tool is the issue here:
dot spacing: if you have a screen with 1920x1080 pixel resolution, the diagonal should have sqrt(1920^2 + 1080^2) ~= 2200 pixels.
So there is no way displaying ten times that many data points on a diagonal and them being distinctly recognisable.
What you can do is to initialise very large figures, and then plot small datapoint symbols. You can then magnify parts of the figure and see individual datapoints.
As for your code, when I run it with synthetic data (linear relationship between x and y, with 20000 datapoints), the axis labels work out nicely:
df = pd.DataFrame({'Load On Sample':np.arange(20000),
'Displacement Into Surface': 2*np.arange(20000)})
x = df['Load On Sample']
y = df['Displacement Into Surface']
plt.scatter(x, y, s=1)
plt.xlabel('Load On Sample')
plt.ylabel('Displacement Into Surface')
plt.show()
even if your code does not generate decent tickspacing for your dataset, you can adjust the xticks, yticks and the respective labels:
example for synthetic data:
plt.scatter(x, y, s=1)
plt.xlabel('Load On Sample')
plt.ylabel('Displacement Into Surface')
plt.gca().set_xticks([0,10000,20000])
plt.gca().set_yticks([10000,20000,30000,40000])
plt.show()

You can personalize your graph using xticks and yticks and numpy(you just have to adjust the value of STEP in th np.arange):
import numpy as np
fig = plt.figure()
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8]) # main axes
ax.scatter(x, y)
ax.set_xticks(np.arange(min(x), max(x)+1, 1.0))
ax.set_yticks(np.arange(min(y), max(y)+1, 1.0))

Hatch area using pcolormesh in Basemap

I try to hatch only the regions where I have statistically significant results. How can I do this using Basemap and pcolormesh?
plt.figure(figsize=(12,12))
lons = iris_cube.coord('longitude').points
lats = iris_cube.coord('latitude').points
m = Basemap(llcrnrlon=lons[0], llcrnrlat=lats[0], urcrnrlon=lons[-1], urcrnrlat=lats[-1], resolution='l')
lon, lat = np.meshgrid(lons, lats)
plt.subplot(111)
cs = m.pcolormesh(lon, lat, significant_data, cmap=cmap, norm=norm, hatch='/')

It seems pcolormesh does not support hatching (see https://github.com/matplotlib/matplotlib/issues/3058). Instead, the advice is to use pcolor, which starting from this example would look like,
import matplotlib.pyplot as plt
import numpy as np
dx, dy = 0.15, 0.05
y, x = np.mgrid[slice(-3, 3 + dy, dy),
slice(-3, 3 + dx, dx)]
z = (1 - x / 2. + x ** 5 + y ** 3) * np.exp(-x ** 2 - y ** 2)
z = z[:-1, :-1]
zm = np.ma.masked_less(z, 0.3)
cm = plt.pcolormesh(x, y, z)
plt.pcolor(x, y, zm, hatch='/', alpha=0.)
plt.colorbar(cm)
plt.show()
where a mask array is used to get the values of z greater than 0.3 and these are hatched using pcolor.
To avoid plotting another colour over the top (so you get only hatching) I've set alpha to 0. in pcolor which feels a bit like a hack. The alternative is to use patch and assign to the areas you want. See this example Python: Leave Numpy NaN values from matplotlib heatmap and its legend. This may be more tricky for basemaps, etc than just choosing areas with pcolor.

I have a simple solution for this problem, using only pcolormesh and not pcolor: Plot the color mesh, then hatch the entire plot, and then plot the original mesh again, this time by masking statistically significant cells, so that the only hatching visible is those on significant cells. Alternatively, you can put a marker on every cell (looks good too), instead of hatching the entire figure.
(I use cartopy instead of basemap, but this shouldn't matter.)
Step 1: Plot your field (z) normally, using pcolormesh.
mesh = plt.pcolormesh(x,y,z)
where x/y can be lons/lats.
Step 2: Hatch the entire plot. For this, use fill_between:
hatch = plt.fill_between([xmin,xmax],y1,y2,hatch='///////',color="none",edgecolor='black')
Check details of fill_between to set xmin, xmax, y1 and y2. You simply define two horizontal lines beyond the bounds of your plot, and hatch the area in between. Use more, or less /s to set hatch density.
To adjust hatch thickness, use below lines:
import matplotlib as mpl
mpl.rcParams['hatch.linewidth'] = 0.3
As an alternative to hatching everything, you can plot all your x-y points (or, lon-lat couples) as markers. A simple solution is putting a dot (x also looks good).
hatch = plt.plot(x,y,'.',color='black',markersize=1.5)
One of the above will be the basis of your 'hatch'. This is how it should look after Step 2:
Step 3: On top of these two, plot your color mesh once again with pcolormesh, this time masking cells containing statistically significant values. This way, the markers on your 'insignificant' cells become invisible again, while significant markers stay visible.
Assuming you have an identically sized array containing the t statistic for each cell (t_z), you can mask significant values using numpy's ma module.
z_masked = numpy.ma.masked_where(t_z >= your_threshold, z)
Then, plot the color mesh, using the masked array.
mesh_masked = plt.pcolormesh(x,y,z_masked)
Use zorder to make sure the layers are in correct order. This is how it should look after Step 3:

Scale colormap for contour and contourf

I'm trying to plot the contour map of a given function f(x,y), but since the functions output scales really fast, I'm losing a lot of information for lower values of x and y. I found on the forums to work that out using vmax=vmax, it actually worked, but only when plotted for a specific limit of x and y and levels of the colormap.
Say I have this plot:
import matplotlib.pyplot as plt
import numpy as np
fig = plt.figure()
u = np.linspace(-2,2,1000)
x,y = np.meshgrid(u,u)
z = (1-x)**2+100*(y-x**2)**2
cont = plt.contour(x,y,z,500,colors='black',linewidths=.3)
cont = plt.contourf(x,y,z,500,cmap="jet",vmax=100)
plt.colorbar(cont)
plt.show
I want to uncover whats beyond the axis limits keeping the same scale, but if I change de x and y limits to -3 and 3 I get:
See how I lost most of my levels since my max value for the function at these limits are much higher. A work around to this problem is to increase the levels to 1000, but that takes a lot of computational time.
Is there a way to plot only the contour levels that I need? That is, between 0 and 100.
An example of a desired output would be:
With the white space being the continuation of the plot without resizing the levels.
The code I'm using is the one given after the first image.

There are a few possible ideas here. The one I very much prefer is a logarithmic representation of the data. An example would be
from matplotlib import ticker
fig = plt.figure(1)
cont1 = plt.contourf(x,y,z,cmap="jet",locator=ticker.LogLocator(numticks=10))
plt.colorbar(cont1)
plt.show()
fig = plt.figure(2)
cont2 = plt.contourf(x,y,np.log10(z),100,cmap="jet")
plt.colorbar(cont2)
plt.show()
The first example uses matplotlibs LogLocator functions. The second one just directly computes the logarithm of the data and plots that normally.
The third example just caps all data above 100.
fig = plt.figure(3)
zcapped = z.copy()
zcapped[zcapped>100]=100
cont3 = plt.contourf(x,y,zcapped,100,cmap="jet")
cbar = plt.colorbar(cont3)
plt.show()

Python: Matplotlib Surface_plot

I'm trying to Plot a high resolution surface_plot, but I would also really like some nice grid lines on top of it. If i use the gridlines in the same argument
ax.plot_surface(x_itp, y_itp, z_itp, rstride=1, cstride=1, facecolors=facecolors, linewidth=0.1)
I get a LOT of grid lines. If I, on the other hand, set "rstride" and "cstride" to higher values, my sphere will become ugly.
I then tried to smash a
ax.plot_wireframe(x_itp, y_itp, z_itp, rstride=3, cstride=3)
in afterwards, but it just lies on top of the colored sphere.. meaning that I can see the backside of the wireframe and then the surface_plot behind it all.
Have anyone tried this?
Another option was to use "Basemap" which can create a nice grid, but then I will have to adapt my colored surface to that.?!
My plot looks like this:
If I add edges to the map with a higher "rstride" and "cstride" then it looks like this:
code :
norm = plt.Normalize()
facecolors = plt.cm.jet(norm(d_itp))
# surface plot
fig, ax = plt.subplots(1, 1, subplot_kw={'projection':'3d', 'aspect':'equal'})
ax.hold(True)
surf = ax.plot_surface(x_itp, y_itp, z_itp, rstride=4, cstride=4, facecolors=facecolors)
surf.set_edgecolors("black")
I want to show the \theta and \phi angles around the sphere.. maybe with 30 degrees apart.
Cheers!
Morten

It looks like you may need to use basemap. With plot_surface() you can either have high resolution plot or low resolution with good grid on top. But not both. I just made a simple basemap with contour plot. I think you can do easily apply pcolor on it. Just do not draw continent and country boundary. Then, you have a nice sphere which gives more control. After making your plot, you can easily add grid on it.
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import numpy as np
map = Basemap(projection='ortho',lat_0=45,lon_0=-150)
map.drawmapboundary(fill_color='aquamarine')
map.drawmeridians(np.arange(0,360,30)) # grid every 30 deg
map.drawparallels(np.arange(-90,90,30))
nlats = 73; nlons = 145; delta = 2.*np.pi/(nlons-1)
lats = (0.5*np.pi-delta*np.indices((nlats,nlons))[0,:,:])
lons = (delta*np.indices((nlats,nlons))[1,:,:])
wave = 0.6*(np.sin(2.*lats)**6*np.cos(4.*lons))
mean = 0.5*np.cos(2.*lats)*((np.sin(2.*lats))**2 + 2.)
x, y = map(lons*180./np.pi, lats*180./np.pi) # projection from lat, lon to sphere
cs = map.contour(x,y,wave+mean,15,linewidths=1.5) # contour data. You can use pcolor() for your project
plt.title('test1')
plt.show()

matplotlib pyplot 2 plots with different axes in same figure

I have a small issue with matplotlib.pyplot and I hope someone might have come across it before.
I have data that contain X,Y,e values that are the X, Y measurements of a variable and e are the errors of the measurements in Y. I need to plot them in a log log scale.
I use the plt.errorbars function to plot them and then set yscale and xscale to log and this works fine. But I need to also plot a line on the same graph that needs to be in linear scale.
I am able to have the plots done separately just fine but I would like to have them in the same image if possible. Do you have any ideas? I am posting what I have done for now.
Cheers,
Kimon
tdlist = np.array([0.01,0.02,0.05,0.1,0.2,0.3,0.4,0.5,0.8,1,2,5,10,15,20,25,30,40,60,80,100,150,200,250,300,400])
freqlist=np.array([30,40,50,60,70,80,90,100,110,120,140,160,180,200,220,250,300,350,400,450])
filename=opts.filename
data = reader(filename)
data2 = logconv(data)
#x,y,e the data. Calculating usefull sums
x = data2[0]
y = data2[1]
e = data2[2]
xoe2 = np.sum(x/e**2)
yoe2 = np.sum(y/e**2)
xyoe2 = np.sum(x*y/e**2)
oe2 = np.sum(1/e**2)
x2oe2 = np.sum(x**2/e**2)
aslope = (xoe2*yoe2-xyoe2*oe2)/(xoe2**2-x2oe2*oe2)
binter = (xyoe2-aslope*x2oe2)/xoe2
aerr = np.sqrt(oe2/(x2oe2*oe2-xoe2**2))
berr = np.sqrt(x2oe2/(x2oe2*oe2-xoe2**2))
print('slope is ',aslope,' +- ', aerr)
print('inter is ',binter,' +- ', berr)
fig = plt.figure()
ax1 = fig.add_subplot(1,1,1)
ax2 = fig.add_axes(ax1.get_position(), frameon=False)
ax1.errorbar(data[0],data[1],yerr=data[2],fmt='o')
ax1.set_xscale('log',basex=10)
ax1.set_yscale('log',basey=10)
ax1.set_yticks([])
ax1.set_xticks([])
ax2.plot(x,aslope*x+binter,'r')
ax2.plot(x,(aslope-aerr)*x+(binter+berr),'--')
ax2.plot(x,(aslope+aerr)*x+(binter-berr),'--')
ax2.set_xscale('linear')
ax2.set_yscale('linear')
plt.xticks(np.log10(freqlist),freqlist.astype('int'))
plt.yticks(np.log10(tdlist),tdlist.astype('float'))
plt.xlabel('Frequency (MHz)')
plt.ylabel('t_s (msec)')
fitndx1 = 'Fit slope '+"{0:.2f}".format(aslope)+u"\u00B1"+"{0:.2f}".format(aerr)
plt.legend(('Data',fitndx1))
plt.show()
Following Molly's suggestion I managed to get closer to my goal but still not there. I am adding a bit more info for what I am trying to do and it might clarify things a bit.
I am setting ax1 to the errobar plot that uses loglog scale. I need to use errorbar and not loglog plot so that I can display the errors with my points.
I am using ax2 to plot the linear fit in linealinear scale.
Moreover I do not want the x and y axes to display values that are 10,100,1000 powers of ten but my own axes labels that have the spacing I want therefore I am using the plt.xticks. I tried ax1.set_yticks and ax1.set_yticklabes but with no success. Below is the image I am getting.
I do not have enough reputation to post an image but here is the link of it uploaded
http://postimg.org/image/uojanigab/
The values of my points should be x range = 40 - 80 and y range = 5 -200 as the fit lines are now.

You can create two overlapping axes using the add_suplot method of figure. Here's an example:
from matplotlib import pyplot as plt
fig = plt.figure()
ax1 = fig.add_subplot(1,1,1)
ax2 = fig.add_axes(ax1.get_position(), frameon=False)
ax1.loglog([1,10,100,1000],[1000,1,100,10])
ax2.plot([5,10,11,13],'r')
plt.show()
You can then turn off the x and y ticks for the linear scale plot like this:
ax2.set_xticks([])
ax2.set_yticks([])

I was not able to get two sets of axis working with the errorbar function so I had to convert everything to log scale including my linear plot. Below is the code I use to get it might be useful to someone.
plt.errorbar(data[0],data[1],yerr=data[2],fmt='o')
plt.xscale('log',basex=10)
plt.yscale('log',basey=10)
plt.plot(data[0],data[0]**aslope*10**binter,'r')
plt.plot(data[0],data[0]**(aslope-aerr)*10**(binter+berr),'--')
plt.plot(data[0],data[0]**(aslope+aerr)*10**(binter-berr),'--')
plt.xticks(freqlist,freqlist.astype('int'))
plt.yticks(tdlist,tdlist.astype('float'))
plt.xlabel('Frequency (MHz)')
plt.ylabel('t_s (msec)')
fitndx1 = 'Fit slope '+"{0:.2f}".format(aslope)+u"\u00B1"+"{0:.2f}".format(aerr)
plt.legend(('Data',fitndx1))
plt.show()
And here is the link to the final image
http://postimg.org/image/bevj2k6nf/