Develop Reference

Make all data points of a matplotlib plot homogeneously transparent

I'd like to plot two scatter plots into the same Axes and turn the upper one's data points transparent such that the other plot shines through. However, I want the whole upper plot to have a homogeneous transparency level, such that superimposed markers of the upper plot do not add up their opacity as they would do if I simply set alpha=0.5.
In other words, I'd like both scatter plots to be rendered first and being set to one constant transparency level. Technically this should be possible for both raster and vector graphics (as SVG supports layer transparency, afaik), but either would be fine for me.
Here is some example code that displays what I do not want to achieve. ;)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(1, figsize=(6,4), dpi=160)
ax = fig.gca()
s1 = ax.scatter(np.random.randn(1000), np.random.randn(1000), color="b", edgecolors="none")
s2 = ax.scatter(np.random.randn(1000), np.random.randn(1000), color="g", edgecolors="none")
s2.set_alpha(0.5) # sadly the same as setting `alpha=0.5`
fig.show() # or display(fig)
I'd like the green markers around (2,2) to not be darker where they superimpose, for example. Is this possible with matplotlib?
Thanks for your time! :)

After searching some more, I found related questions and two solutions, of which at least one kind of works for me:
As I hoped one can render one layer and then afterwards blend them together like this:
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(1, figsize=(6,4), dpi=160)
ax1 = fig.gca()
s1 = ax1.scatter(np.random.randn(1000), np.random.randn(1000), color="#3355ff", edgecolors="none")
ax1.set_xlim(-3.5,3.5)
ax1.set_ylim(-3.5,3.5)
ax1.patch.set_facecolor("none")
ax1.patch.set_edgecolor("none")
fig.canvas.draw()
w, h = fig.canvas.get_width_height()
img1 = np.frombuffer(fig.canvas.buffer_rgba(), np.uint8).reshape(h, w, -1).copy()
ax1.clear()
s2 = ax1.scatter(np.random.randn(1000), np.random.randn(1000), color="#11aa44", edgecolors="none")
ax1.set_xlim(-3.5,3.5)
ax1.set_ylim(-3.5,3.5)
ax1.patch.set_facecolor("none")
ax1.patch.set_edgecolor("none")
fig.canvas.draw()
img2 = np.frombuffer(fig.canvas.buffer_rgba(), np.uint8).reshape(h, w, -1).copy()
fig.clf()
plt.imshow(np.minimum(img1,img2))
plt.subplots_adjust(0, 0, 1, 1)
plt.axis("off")
plt.show()
I may have to come up with better methods than just taking the np.minimum of both layers to keep more color options (and probably save the axes and labels to a third layer).
I did not try mplcairo as suggested in the other linked answer as it has too many dependencies for my use case (my solution should be portable).
I am still happy for additional input. :)

3D elevation on geographical map with python

I am trying to display elevation/topography in 3D on a geographical map
I am currently displaying elevation with a colormap using the scatter function of matplolib over a geographical map created with the basemap package. I would like to visualize it in 3D with a shady effect or something similar.
Bellow is a simple example using data created randomly. The only constrain is to keep the 'ortho' look shown bellow. Any python package could be used.
Input data could either be a 1D arrays or 2D arrays.
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
import numpy as np
size = 1000
# data to plot
data = np.arange(size)*0.5/size
# coordinates
lat = np.random.uniform(low=65, high=90, size=(size,))
lon = np.random.uniform(low=-180, high=180, size=(size,))
f1, ax = plt.subplots(1, 1,figsize=(9,8))
m = Basemap(projection='ortho',lat_0=70,lon_0=0,resolution='l',ax=ax)
m.drawcoastlines(linewidth=0.25, zorder=0)
m.drawparallels(np.arange(90,-90,-5), labels=[1,1,1,1],linewidth = 0.25, zorder=1)
m.drawmeridians(np.arange(-180.,180.,30.),labels=[1,1,1,1],latmax=85, linewidth = 0.25, zorder=1)
m.fillcontinents(color='dimgray',lake_color='grey', zorder=1)
x,y = m(lon,lat)
cmap='viridis'
m.scatter(x,y,c=data,s=10,cmap=cmap,vmin=0,vmax=0.5,zorder=3,alpha=1)
plt.show()
Thanks a lot,

Plot x,y,data in a heatmap with matplotlib

I'm having some trouble to plot data in a 2D heatmap using matplotlib.
The code is the following:
from ugtm import eGTM
import numpy as np
from sklearn import preprocessing
import matplotlib.pyplot as plt
from scipy.interpolate import griddata
X_train = np.random.randn(100, 50)
X_test = np.random.randn(50, 50)
gtm = eGTM().fit(X_train)
responsibilities = gtm.optimizedModel.matR
nodes_cumulated_responsibilities = preprocessing.MinMaxScaler().fit_transform(np.sum(gtm.optimizedModel.matR,0).reshape(-1,1))
nodes_coordinates = gtm.optimizedModel.matX
transformed = eGTM().fit(X_train).transform(X_test)
Z, xedges, yedges = np.histogram2d(nodes_coordinates[:,0], nodes_coordinates[:,1],normed=True,
bins=15,weights=nodes_cumulated_responsibilities[:,0])
plt.figure()
plt.pcolormesh(xedges, yedges,Z.T,cmap='bone_r')
plt.scatter(transformed[:,0],transformed[:,1],color='r')
plt.title('np.hist2d')
I have the "nodes_coordinates" array shape(256,2) which represents the coordinates x,y of the nodes. Then i have "nodes_cumulated_responsibilities" array shape(256,1) representing the value of each node in the respective x,y coordinate. I also have a "transformed" array shape(50,2) representing 50 cells that "X_test" hitted in the heatmap by the eGTM.
Resuming, i would like to make a heatmap using nodes_coordinates and nodes_cumulated_responsibilities as background, and then plot in this map a scatter plot of the "X_test" hitted nodes on the heatmap.
But i want the scatter plot to be in the center of the node for any X_train or any X_test.. This is whats i don't know how to do.
Some points for this code:
The size of the map is 16x16 by default for any size of X_test, but it can be set to any integer, thus the map can be any nxn.
The plt.pcolormesh needs the np.histogram2d to get a meshgrid with edges added to it. But i don't like to use np.histogram2d since i need to pass the 'bins' argument to adjust the heatmap correctly and i suspected it could lead me to some wrong maps or scatters.
The np.meshgrid gives me a meshgrid for the heatmap but without the edges, so i cant use this with pcolormesh.
The approach to use the plt.imshow to plot the heatmap using the np.meshgrid seems more trustable. But i don't know how to convert the 1D array "nodes_cumulated_responsibilities" to a 256x256 array.
A very similiar example of what i want to do is in: https://github.com/JustGlowing/minisom/blob/master/examples/BasicUsage.ipynb
Example of heatmap with scatter plot

retrieve leave colors from scipy dendrogram

I can not get the color leaves from the scipy dendrogram dictionary. As stated in the documentation and in this github issue, the color_list key in the dendrogram dictionary refers to the links, not the leaves. It would be nice to have another key referring to the leaves, sometimes you need this for coloring other types of graphics, such as this scatter plot in the example below.
import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import linkage, dendrogram
# DATA EXAMPLE
x = np.array([[ 5, 3],
[10,15],
[15,12],
[24,10],
[30,30],
[85,70],
[71,80]])
# DENDROGRAM
plt.figure()
plt.subplot(121)
z = linkage(x, 'single')
d = dendrogram(z)
# COLORED PLOT
# This is what I would like to achieve. Colors are assigned manually by looking
# at the dendrogram, because I failed to get it from d['color_list'] (it refers
# to links, not observations)
plt.subplot(122)
points = d['leaves']
colors = ['r','r','g','g','g','g','g']
for point, color in zip(points, colors):
plt.plot(x[point, 0], x[point, 1], 'o', color=color)
Manual color assignment seems easy in this example, but I'm dealing with huge datasets, so until we get this new feature in the dictionary (color leaves), I'm trying to infer it somehow with the current information contained in the dictionary but I'm out of ideas so far. Can anyone help me?
Thanks.

For scipy 1.7.1 the new functionality has been implemented and the dendogram function returns in the output dictionary also an entry 'leaves_color_list' that can be used to perform easily this task.
Here is a working code of the OP (see last line "NEW CODE")
import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import linkage, dendrogram
# DATA EXAMPLE
x = np.array([[ 5, 3],
[10,15],
[15,12],
[24,10],
[30,30],
[85,70],
[71,80]])
# DENDROGRAM
plt.figure()
plt.subplot(121)
z = linkage(x, 'single')
d = dendrogram(z)
# COLORED PLOT
# This is what I would like to achieve. Colors are assigned manually by looking
# at the dendrogram, because I failed to get it from d['color_list'] (it refers
# to links, not observations)
plt.subplot(122)
#NEW CODE
plt.scatter(x[d['leaves'],0],x[d['leaves'],1], color=d['leaves_color_list'])

The following approach seems to work. The dictionary returned by the dendogram contains 'color_list' with the colors of the linkages. And 'icoord' and 'dcoord' with the x, resp. y, plot coordinates of these linkages. These x-positions are 5, 15, 25, ... when the linkage starts at a point. So, testing these x-positions can bring us back from the linkage to the corresponding point. And allows to assign the color of the linkage to the point.
import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import linkage, dendrogram
# DATA EXAMPLE
x = np.random.uniform(0, 10, (20, 2))
# DENDROGRAM
plt.figure()
plt.subplot(121)
z = linkage(x, 'single')
d = dendrogram(z)
plt.yticks([])
# COLORED PLOT
plt.subplot(122)
points = d['leaves']
colors = ['none'] * len(points)
for xs, c in zip(d['icoord'], d['color_list']):
for xi in xs:
if xi % 10 == 5:
colors[(int(xi)-5) // 10] = c
for point, color in zip(points, colors):
plt.plot(x[point, 0], x[point, 1], 'o', color=color)
plt.text(x[point, 0], x[point, 1], f' {point}')
plt.show()
PS: This post about matching points with their clusters might also be relevant.

Plot a (polar) color wheel based on a colormap using Python/Matplotlib

I am trying to create a color wheel in Python, preferably using Matplotlib. The following works OK:
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
xval = np.arange(0, 2*pi, 0.01)
yval = np.ones_like(xval)
colormap = plt.get_cmap('hsv')
norm = mpl.colors.Normalize(0.0, 2*np.pi)
ax = plt.subplot(1, 1, 1, polar=True)
ax.scatter(xval, yval, c=xval, s=300, cmap=colormap, norm=norm, linewidths=0)
ax.set_yticks([])
However, this attempt has two serious drawbacks.
First, when saving the resulting figure as a vector (figure_1.svg), the color wheel consists (as expected) of 621 different shapes, corresponding to the different (x,y) values being plotted. Although the result looks like a circle, it isn't really. I would greatly prefer to use an actual circle, defined by a few path points and Bezier curves between them, as in e.g. matplotlib.patches.Circle. This seems to me the 'proper' way of doing it, and the result would look nicer (no banding, better gradient, better anti-aliasing).
Second (relatedly), the final plotted markers (the last few before 2*pi) overlap the first few. It's very hard to see in the pixel rendering, but if you zoom in on the vector-based rendering you can clearly see the last disc overlap the first few.
I tried using different markers (. or |), but none of them go around the second issue.
Bottom line: can I draw a circle in Python/Matplotlib which is defined in the proper vector/Bezier curve way, and which has an edge color defined according to a colormap (or, failing that, an arbitrary color gradient)?

One way I have found is to produce a colormap and then project it onto a polar axis. Here is a working example - it includes a nasty hack, though (clearly commented). I'm sure there's a way to either adjust limits or (harder) write your own Transform to get around it, but I haven't quite managed that yet. I thought the bounds on the call to Normalize would do that, but apparently not.
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import cm
import matplotlib as mpl
fig = plt.figure()
display_axes = fig.add_axes([0.1,0.1,0.8,0.8], projection='polar')
display_axes._direction = 2*np.pi ## This is a nasty hack - using the hidden field to
## multiply the values such that 1 become 2*pi
## this field is supposed to take values 1 or -1 only!!
norm = mpl.colors.Normalize(0.0, 2*np.pi)
# Plot the colorbar onto the polar axis
# note - use orientation horizontal so that the gradient goes around
# the wheel rather than centre out
quant_steps = 2056
cb = mpl.colorbar.ColorbarBase(display_axes, cmap=cm.get_cmap('hsv',quant_steps),
norm=norm,
orientation='horizontal')
# aesthetics - get rid of border and axis labels
cb.outline.set_visible(False)
display_axes.set_axis_off()
plt.show() # Replace with plt.savefig if you want to save a file
This produces
If you want a ring rather than a wheel, use this before plt.show() or plt.savefig
display_axes.set_rlim([-1,1])
This gives
As per #EelkeSpaak in comments - if you save the graphic as an SVG as per the OP, here is a tip for working with the resulting graphic: The little elements of the resulting SVG image are touching and non-overlapping. This leads to faint grey lines in some renderers (Inkscape, Adobe Reader, probably not in print). A simple solution to this is to apply a small (e.g. 120%) scaling to each of the individual gradient elements, using e.g. Inkscape or Illustrator. Note you'll have to apply the transform to each element separately (the mentioned software provides functionality to do this automatically), rather than to the whole drawing, otherwise it has no effect.

I just needed to make a color wheel and decided to update rsnape's solution to be compatible with matplotlib 2.1. Rather than place a colorbar object on an axis, you can instead plot a polar colored mesh on a polar plot.
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import cm
import matplotlib as mpl
# If displaying in a Jupyter notebook:
# %matplotlib inline
# Generate a figure with a polar projection
fg = plt.figure(figsize=(8,8))
ax = fg.add_axes([0.1,0.1,0.8,0.8], projection='polar')
# Define colormap normalization for 0 to 2*pi
norm = mpl.colors.Normalize(0, 2*np.pi)
# Plot a color mesh on the polar plot
# with the color set by the angle
n = 200 #the number of secants for the mesh
t = np.linspace(0,2*np.pi,n) #theta values
r = np.linspace(.6,1,2) #radius values change 0.6 to 0 for full circle
rg, tg = np.meshgrid(r,t) #create a r,theta meshgrid
c = tg #define color values as theta value
im = ax.pcolormesh(t, r, c.T,norm=norm) #plot the colormesh on axis with colormap
ax.set_yticklabels([]) #turn of radial tick labels (yticks)
ax.tick_params(pad=15,labelsize=24) #cosmetic changes to tick labels
ax.spines['polar'].set_visible(False) #turn off the axis spine.
It gives this: