Develop Reference

Python surface plotting

I have following table data(Please see the image)
for which I want to have a surface plot in python. Using surface plotting from matplotlib,
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import numpy as np
fig = plt.figure()
ax = fig.gca(projection='3d')
X=[2,3,5,8,20,30,50,80,100,150,175,200,250,300]
Y=[2,3,4,5,10,15,20,30,40,50,80,100,125,150,175,200]
Y,X=np.meshgrid(Y,X)
Z=np.array([
[0.2885307,0.269452,0.259193,0.2548041,0.2731868,0.4801551,0.7992361,1.7577641,3.2611327,5.428839,19.647976,37.59729,78.0871,152.21466,268.14572,0],
[0.2677955,0.2538363,0.2380033,0.2306999,0.4779794,0.9251045,1.5448972,3.508644,6.4968576,11.252151,0,0,0,0,0,0],
[0.2432982,0.2283371,0.2514196,0.3392502,0,0,0,0,0,0,0,0,0,0,0,0],
[0.2342575,0.3158406,0.4770729, 0.6795485,2.353042, 5.260077,9.78172,25.87004,59.52568, 0,0,0,0,0,0,0],
[0.6735384, 1.3873291,2.346506, 3.5654,0,0,0,0,0,0,0,0,0,0,0,0],
[1.3584715, 2.9405127,5.096819,8.155857,0,0,0,0,0,0,0,0,0,0,0,0],
[3.558062,8.216592,15.768077,27.386694,0,0,0,0,0,0,0,0,0,0,0,0],
[9.537899,25.202589,58.20041,0,0,0,0,0,0,0,0,0,0,0,0,0],
[16.083374,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[54.936775,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[89.185974,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]])
my_col = cm.jet(Z/np.amax(Z))
surf = ax.plot_surface(X, Y, Z,cmap=cm.coolwarm,linewidth=0, antialiased=False)
ax.set_zlim(0, 300)
ax.zaxis.set_major_locator(LinearLocator(10))
ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
I get a plot like this
which is correct but not very attractive or intuitive. How can I make the visualizations more smooth and clear? Please note that I have many blanks in my data. Should I use 'zero' for the blanks or 'nan'(not a number)? For the same data, excel shows a much better graph.
I appreciate your inputs in order to make python plot more visually attractive.

The difference between the matplotlib and excel plots is that matplotlib is plotting on a linear scale and excel is logarithmic (or something that looks deceptively like a log axis but actually isn't -- see below). Therefore, in the matplotlib the slopes look extremely steep, but in excel the slopes are dramatically stretched out by the log.
Unfortunately, matplotlib doesn't yet have log axes working well in 3D. I'm not sure why this is, but it is a serious shortcoming. You can see a plot similar to Excel though if you take the log10 of your X and Y data before you do the plots. You can also go further to DIY your own log axes, but I've just done a shorthand for that using a tick formatter.
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter, FuncFormatter
from mpl_toolkits.mplot3d import axes3d
import numpy as np
def format_log(x, pos=None):
x1 = 10**x
s = "%.3f" % x1
return s[:-4] if s[-3:]=="000" else " "
fig = plt.figure()
ax = fig.gca(projection='3d')
X=[2,3,5,8,20,30,50,80,100,150,175,200,250,300]
Y=[2,3,4,5,10,15,20,30,40,50,80,100,125,150,175,200]
X = np.log10(np.array(X))
Y = np.log10(np.array(Y))
Y,X=np.meshgrid(Y,X)
Z=np.array([
[0.2885307,0.269452,0.259193,0.2548041,0.2731868,0.4801551,0.7992361,1.7577641,3.2611327,5.428839,19.647976,37.59729,78.0871,152.21466,268.14572,0],
[0.2677955,0.2538363,0.2380033,0.2306999,0.4779794,0.9251045,1.5448972,3.508644,6.4968576,11.252151,0,0,0,0,0,0],
[0.2432982,0.2283371,0.2514196,0.3392502,0,0,0,0,0,0,0,0,0,0,0,0],
[0.2342575,0.3158406,0.4770729, 0.6795485,2.353042, 5.260077,9.78172,25.87004,59.52568, 0,0,0,0,0,0,0],
[0.6735384, 1.3873291,2.346506, 3.5654,0,0,0,0,0,0,0,0,0,0,0,0],
[1.3584715, 2.9405127,5.096819,8.155857,0,0,0,0,0,0,0,0,0,0,0,0],
[3.558062,8.216592,15.768077,27.386694,0,0,0,0,0,0,0,0,0,0,0,0],
[9.537899,25.202589,58.20041,0,0,0,0,0,0,0,0,0,0,0,0,0],
[16.083374,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[54.936775,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[89.185974,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]])
my_col = cm.jet(Z/np.amax(Z))
surf = ax.plot_surface(X, Y, Z,cmap=cm.coolwarm)
ax.set_zlim(0, 300)
ax.zaxis.set_major_locator(LinearLocator(10))
ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
ax.xaxis.set_major_formatter(FuncFormatter(format_log))
ax.yaxis.set_major_formatter(FuncFormatter(format_log))
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
Edit:
After coming back to this question, I realize that the Excel plot isn't actually showing a logarithmic axis, but is instead just plotting the X and Y values given with equal spacing along the axis even those values have no clear mathematical progression.
It's critical to note that this isn't a good representation of the data, since it gives the obvious impression that it's logarithmic (for the specific data presented), but it's actually not, although it requires unusually close inspection to see that. Here the gaps between adjacent numbers aren't even monotonic.
So I discourage this representation, but to reproduce that Excel plot, I'd suggest making equally spaced data, but labeling it with different numbers (and just this sentence alone should be enough to discourage this approach). But here's the code and approach:
fig = plt.figure()
ax = fig.gca(projection='3d')
x=[2,3,5,8,20,30,50,80,100,150,175,200,250,300]
y=[2,3,4,5,10,15,20,30,40,50,80,100,125,150,175,200]
Y,X=np.meshgrid(range(len(y)),range(len(x)))
Z=np.array([
[0.2885307,0.269452,0.259193,0.2548041,0.2731868,0.4801551,0.7992361,1.7577641,3.2611327,5.428839,19.647976,37.59729,78.0871,152.21466,268.14572,0],
[0.2677955,0.2538363,0.2380033,0.2306999,0.4779794,0.9251045,1.5448972,3.508644,6.4968576,11.252151,0,0,0,0,0,0],
[0.2432982,0.2283371,0.2514196,0.3392502,0,0,0,0,0,0,0,0,0,0,0,0],
[0.2342575,0.3158406,0.4770729, 0.6795485,2.353042, 5.260077,9.78172,25.87004,59.52568, 0,0,0,0,0,0,0],
[0.6735384, 1.3873291,2.346506, 3.5654,0,0,0,0,0,0,0,0,0,0,0,0],
[1.3584715, 2.9405127,5.096819,8.155857,0,0,0,0,0,0,0,0,0,0,0,0],
[3.558062,8.216592,15.768077,27.386694,0,0,0,0,0,0,0,0,0,0,0,0],
[9.537899,25.202589,58.20041,0,0,0,0,0,0,0,0,0,0,0,0,0],
[16.083374,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[54.936775,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[89.185974,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]])
my_col = cm.jet(Z/np.amax(Z))
surf = ax.plot_surface(X, Y, Z,cmap=cm.coolwarm)
ax.tick_params(axis='both', which='major', labelsize=6)
ax.set_zlim(0, 300)
ax.xaxis.set_major_locator(IndexLocator(1, 0))
ax.xaxis.set_major_formatter(FixedFormatter([repr(v) for v in x]))
ax.yaxis.set_major_locator(IndexLocator(1, 0))
ax.yaxis.set_major_formatter(FixedFormatter([repr(v) for v in y]))
fig.colorbar(surf, shrink=0.5, aspect=5)
If one wanted to show the specific numbers given for X and Y, one solution would be to plot with a logarithmic axis (since the numbers are spaced very approximately in a log way), and then plot the numbers specifically by lines on the axes, or alternatively, don't use these numbers instead of the usual regularly spaced numbers. (But to plot these as axes values, and space them visually at regular intervals, that is a problem.)

How to use a 3rd dataframe column as x axis ticks/labels in matplotlib scatter

I'm struggling to wrap my head around matplotlib with dataframes today. I see lots of solutions but I'm struggling to relate them to my needs. I think I may need to start over. Let's see what you think.
I have a dataframe (ephem) with 4 columns - Time, Date, Altitude & Azimuth.
I produce a scatter for alt & az using:
chart = plt.scatter(ephem.Azimuth, ephem.Altitude, marker='x', color='black', s=8)
What's the most efficient way to set the values in the Time column as the labels/ticks on the x axis?
So:
the scale/gridlines etc all remain the same
the chart still plots alt and az
the y axis ticks/labels remain as is
only the x axis ticks/labels are changed to the Time column.
Thanks

This isn't by any means the cleanest piece of code but the following works for me:
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.scatter(ephem.Azimuth, ephem.Altitude, marker='x', color='black', s=8)
labels = list(ephem.Time)
ax.set_xticklabels(labels)
plt.show()
Here you will explicitly force the set_xticklabels to the dataframe Time column which you have.

In other words, you want to change the x-axis tick labels using a list of values.
labels = ephem.Time.tolist()
# make your plot and before calling plt.show()
# insert the following two lines
ax = plt.gca()
ax.set_xticklabels(labels = labels)
plt.show()

matplotlib set_major_formatter creating 2 plots

This excerpt from my code changes the value of the y axis labels from exponential to millions. Problem is it creates 2 figures. The first one is an x and y axis with no plot (and the scale of the x axis is used for the y axis as well), and then the 2nd figure is exactly what I want. It is a double bar graph.
I am guessing it has something to do with using f.plot.bar instead of plt.bar but I am not sure. I just want to get rid of the first figure than all will be well.
from matplotlib.ticker import FuncFormatter
def millions(x, pos):
'The two args are the value and tick position'
return '%1.1fM' % (x*1e-6)
formatter = FuncFormatter(millions)
fig, ax = plt.subplots()
ax = tempg.plot.bar(y=['Republican2016Votes', 'Democrat2016Votes'], rot=0,
color = ['DarkRed','Blue'])
ax.yaxis.set_major_formatter(formatter)
plt.show()

formatting ticks on pyplot cmap loses the plot

Using the following code:-
fig=plt.figure()
#ax = fig.add_subplot(111)
ax=plt.axes()
font0 = FontProperties()
outgrid=[[x*y for x in range(testXaxis)] for y in range (levIndRange)]
vmin=0
vmax=testXaxis*levIndRange
height = [v*1000.5 for v in range (levMaxInd)]
colours='terrain'
cmap=plt.cm.get_cmap(colours)
norm=matplotlib.colors.Normalize(clip=False,vmin=vmin,vmax=vmax)
print 'vmax = ',vmax
m=plt.cm.ScalarMappable(cmap=cmap,norm=norm)
m.set_array(outgrid)
plt.imshow(np.flipud(outgrid),cmap=cmap, norm=norm, aspect=stretch)
#ax.imshow(np.flipud(outgrid),cmap=cmap, norm=norm, aspect=stretch)
ax.yaxis.set_major_formatter(FormatStrFormatter('%.0f'))
#plt.axis.YAxis.set_major_formatter(FormatStrFormatter('%.0f')) # 'module' object has no attribute 'set_major_formatter'
plt.yticks([s for s in range(0,levIndRange,levParInt)],[height[v] for v in range(levMinInd-1,levMaxInd-1,levParInt)])
plt.xticks([1,3,5,7,9,11,13,15,17,19])
#ax.xaxis.set_ticks([1,3,5,7,9,11,13,15,17,19])
#ax.yaxis.set_ticks([height[v] for v in range(levMinInd-1,levMaxInd-1,levParInt)]) # This one line makes the plot collapse
plt.ylabel(yLabel)
plt.xlabel(xLabel)
I get the following plot, which is fine, but I want to change the floating point precision on the y-axis:-
So, when I try to alter the precision on the y-axis using set_major_formatter and the following lines instead of plot.yticks :-
ax.yaxis.set_major_formatter(FormatStrFormatter('%.0f'))
ax.xaxis.set_ticks([1,3,5,7,9,11,13,15,17,19])
ax.yaxis.set_ticks([height[v] for v in range(levMinInd-1,levMaxInd-1,levParInt)]) # This one line makes the plot collapse
... the plot disappears:-
How can I alter the precision without losing the plot?
Any help gratefully received.
Thanks

I am answering rather than editing because I have solved the lost plot problem, and that might be a help to someone else. The solution is to move the plt.imshow line AFTER the ax.yaxis.set_ticks line. However, I am still not getting the expected Y axis ticks with no decimal point.
The first block of code below produces a plot with Y ticks: 0.0,1000.5,2001.0,3001.5,4002.0
import matplotlib
import matplotlib.pyplot as plt
outgrid=[[x*y for x in range(4)] for y in range (5)]
height = [v*1000.5 for v in range (5)]
cmap=plt.cm.get_cmap('terrain')
norm=matplotlib.colors.Normalize(clip=False)
m=plt.cm.ScalarMappable(cmap=cmap,norm=norm)
m.set_array(outgrid)
plt.imshow(outgrid,cmap=cmap, norm=norm, aspect=0.5)
plt.yticks([s for s in range(5)],[height[t] for t in range(5)])
plt.show()
This second block of code tries to format these ticks with no decimal point but just produces one Y tick of 0
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.ticker import FormatStrFormatter
ax=plt.axes()
outgrid=[[x*y for x in range(4)] for y in range (5)]
height = [v*1000.5 for v in range (5)]
cmap=plt.cm.get_cmap('terrain')
norm=matplotlib.colors.Normalize(clip=False)
m=plt.cm.ScalarMappable(cmap=cmap,norm=norm)
m.set_array(outgrid)
ax.yaxis.set_major_formatter(FormatStrFormatter('%.0f'))
ax.yaxis.set_ticks([height[t] for t in range(5)])
plt.imshow(outgrid,cmap=cmap, norm=norm, aspect=0.5)
plt.show()

Python matplotlib logarithmic autoscale

I need to get a x,y plot with a logarithmic scale on the x-axes. When I zoom in the x-axes should autoscale as the (not logarithmic) y-axes.
plt.xscale('log')
plt.grid(True)
plt.autoscale(True,True,True)
plt.legend(loc='best')
plt.show()
As you can see there is no working autoscale function on the x-axes.
How can I get this to display properly?

The solution by #hashcode55 does not work as it is what I was attempting before I found this thread.
It seems to me that there is simply a "bug" in that:
plt.yscale('log')
plt.autoscale(enable=True, axis='y')
are not compatible.
Here is my sample code:
import matplotlib.pyplot as plt
import matplotlib
import random
import numpy as np
# generate some random data and add it to the plot
x = np.array(range(1,100))
y = np.maximum(np.ones(99), np.random.randn(99))
plt.plot(x, y, markersize=4, marker='.', color='red')
# format
ax = plt.gca()
plt.ylabel('LOGARITHMIC SCALE')
plt.yscale('log')
plt.minorticks_on
ax.yaxis.set_major_formatter(matplotlib.ticker.ScalarFormatter())
ax.yaxis.set_minor_formatter(matplotlib.ticker.ScalarFormatter())
plt.autoscale(enable=True, axis='y')
#ax.set_ylim([np.min(y), np.max(y)])
#plot
plt.show()
which produces:
log scale, but clearly not autoscale
if I remove the comments from this line:
ax.set_ylim([np.min(y), np.max(y)])
Then it actually plots as would be expected with autoscale:
Nice, but what if I've lost reference to my y values on the plot?
while this solution/answer is a good "hack" to this sample problem, it its not a solid solution for my situation as my chart is a) live; continually updating every minute b) contains MANY plots c) is dropping off data older than past 24 hours; so such a solution would get really hacky if implemented every time something was added or removed from the plot in live session.
I would be interested in a true built-in "autoscale" solution, if such exists, that works with log y scale and I can auto update using plt.ion()
until then, what about this:
h/t #David Z
How to extract data from matplotlib plot
#if you do need to get data out of a plot, I think this should do it
gca().get_lines()[n].get_xydata()
#Alternatively you can get the x and y data sets separately:
line = gca().get_lines()[n]
xd = line.get_xdata()
yd = line.get_ydata()
implemented in our situation at hand (with an extra blue line to test multiple lines) as:
import matplotlib.pyplot as plt
import matplotlib
import random
import numpy as np
# generate some random data and add it to the plot
x = np.array(range(1,100))
y = np.maximum(np.ones(99), np.random.randn(99))
plt.plot(x, y, markersize=4, marker='.', color='red')
# test for compatibility with multilpes lines
x = np.array(range(1,100))
y = np.maximum(np.ones(99), 1.5*np.random.randn(99))
plt.plot(x, y, markersize=4, marker='.', color='blue')
# format
ax = plt.gca()
plt.ylabel('LOGARITHMIC SCALE')
plt.yscale('log')
plt.minorticks_on
ax.yaxis.set_major_formatter(matplotlib.ticker.ScalarFormatter())
ax.yaxis.set_minor_formatter(matplotlib.ticker.ScalarFormatter())
#plt.autoscale(enable=True, axis='y')
#####################################################
#force 'autoscale'
#####################################################
yd = [] #matrix of y values from all lines on plot
for n in range(len(plt.gca().get_lines())):
line = plt.gca().get_lines()[n]
yd.append(line.get_ydata())
ax.set_ylim([0.9*np.min(yd), 1.1*np.max(yd)])
#####################################################
#plot
plt.show()
which, in essence, is pulling all y data from all lines on the plot, finding the max and min; then implementing them via set_ylim; "forcing" autoscale
yields:
voila!
for my situation I had somewhat more complicated plots in the format:
plt.plot((x1,x2), (y1,y2))
creating a matrix in matrix situation producing a 'Value Error'
for that I had to flatten using:
yd = [item for sublist in yd for item in sublist]
h/t #Alex Martelli
Making a flat list out of list of lists in Python
and this was the final product:
#####################################################
#force 'autoscale'
#####################################################
yd = [] #matrix of y values from all lines on plot
for n in range(len(plt.gca().get_lines())):
line = plt.gca().get_lines()[n]
yd.append((line.get_ydata()).tolist())
yd = [item for sublist in yd for item in sublist]
ax.set_ylim([0.9*np.min(yd), 1.1*np.max(yd)])
#####################################################

If you look at the documentation the function is like-
matplotlib.pyplot.autoscale(enable=True, axis='both', tight=None)
What you are sending is an invalid argument...
just make it
plt.autoscale(True, axis = 'both')
And about tight -
If True, set view limits to data limits; if False, let the locator and
margins expand the view limits; if None, use tight scaling if the only
artist is an image, otherwise treat tight as False. The tight setting
is retained for future autoscaling until it is explicitly changed.

I had a similar problem and I was able to solve it by setting the 'log' scale before plotting. In this case, the autoscaling is working as expected.