Develop Reference

Plotting Sentinel-5P data in xarray

I'm trying to plot a grid of air pollution data from a netCDF files in python using xarray. However, I'm facing a couple roadblocks.
To start off, here is the data that can be used to reproduce my code:
Data
When you try to import this data using xarray.open_dataset, you end up with a file that has zero coordinates or variables, and lots of attributes:
FILE_NAME = "test2.nc". ##I changed the name to make it shorter
xr.open_dataset(FILE_NAME)
So I created variables of the data and tried to import those into xarray:
prd='PRODUCT'
metdata = "METADATA"
lat= ds.groups[prd].variables['latitude']
lon= ds.groups[prd].variables['longitude']
no2 = ds.groups[prd].variables['nitrogendioxide_tropospheric_column']
scanline = ds.groups[prd].variables['scanline']
time = ds.groups[prd].variables['time']
ground_pixel = ds.groups[prd].variables['ground_pixel']
ds = xr.DataArray(no2,
dims=["time","x","y"],
coords={
"lon":(["time","x", "y"], lon)
}
# coords=[("time", time), ("x", scanline),("y", ground_pixel)]
)
As you can see above, I tried multiple ways of creating the coordinates, but I'm still getting an error. The data in this netCDF file is on an irregular grid, and I just want to be able to plot that accurately and quickly using xarray.
Does someone know how I can do this?

Adding multiple images to a matplotlib subplot?

I am trying to make a matplottlib plot using some image data I have in numpy format, and was wondering if someone would be able to advise me on the best way to approach displaying multiples of these images within the boundaries of one subplot?
For example, using the following code...
n_samples = 10
sample_imgs, min_index = visualise_n_way(n_samples)
print(min_index)
print(sample_imgs.shape)
print(sample_imgs[0].shape)
print(x_train_w)
print(x_train_h)
img_matrix = []
for index in range(1, len(sample_imgs)):
img_matrix.append(np.reshape(sample_imgs[index], (x_train_w, x_train_h)))
img_matrix = np.asarray(img_matrix)
img_matrix = np.vstack(img_matrix)
f, ax = plt.subplots(1, 3, figsize = (10, 12))
f.tight_layout()
ax[0].imshow(np.reshape(sample_imgs[0], (x_train_w, x_train_h)),vmin=0, vmax=1,cmap='Greys')
ax[0].set_title("Test Image")
ax[1].imshow(img_matrix ,vmin=0, vmax=1,cmap='Greys')
ax[1].set_title("Support Set")
ax[2].imshow(np.reshape(sample_imgs[min_index], (x_train_w, x_train_h)),vmin=0, vmax=1,cmap='Greys')
ax[2].set_title("Image most similar to Test Image in Support Set")
I get the following image and output
1
(11, 784)
(784,)
28
28
Matplotlib Output
What I would like to do however is to have the second subplot, the one displaying img_matrix, to be the same size as the two either side of it, creating a grid of the images. Sort of like this
sketch.
I am at a loss as to how to do this however. I believe I may need to use something such as a gridspace, but I'm finding the documentation hard to follow for what I want to do.
Any help is greatly appreciated!

How to plot dotted lines from a shapefile in python?

I am not sure on how to plot a dotted line from a shapefile in Python. It appears that readshapefile() does not have any linestyle for me to set. Below I have a working code where I take a shapefile and plot it, but it only plots a solid line. Any ideas to set me in the right direction? Thanks!
The shapefile can be found here: http://www.natice.noaa.gov/products/daily_products.html, where the Start Date is Feb 15th, end date is Feb 17th, and the Date Types is Ice Edge. It should be the first link.
#!/awips2/python/bin/python
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
map = Basemap(llcrnrlon=-84.37,llcrnrlat=42.11,urcrnrlon=-20.93,urcrnrlat=66.48,
resolution='i', projection='tmerc', lat_0 = 55., lon_0 = -50.)
map.drawmapboundary(fill_color='aqua')
map.fillcontinents(color='#ddaa66',lake_color='aqua')
map.drawcoastlines(zorder = 3)
map.readshapefile('nic_autoc2018046n_pl_a', 'IceEdge', zorder = 2, color = 'blue')
plt.show()

From the Basemap documentation:
A tuple (num_shapes, type, min, max) containing shape file info is
returned. num_shapes is the number of shapes, type is the type code
(one of the SHPT* constants defined in the shapelib module, see
http://shapelib.maptools.org/shp_api.html) and min and max are
4-element lists with the minimum and maximum values of the vertices.
If drawbounds=True a matplotlib.patches.LineCollection object is
appended to the tuple.
drawbounds is True by default, so all you have to do is collect the return value of readshapefile and alter the linestyle of the returned LineCollection object, which can be done with LineCollection.set_linestyle(). So in principle you can change the linestyle of your plotted shape file with something like this:
result = m.readshapefile('shapefiles/nic_autoc2018046n_pl_a', 'IceEdge', zorder = 10, color = 'blue')#, drawbounds = False)
col = result[-1]
col.set_linestyle('dotted')
plt.show()
However, your shapefile contains 5429 separate line segments of different length and somehow matplotlib does not seem to be able to deal with this large amount of non-continuous lines. At least on my machine the plotting did not finish within one hour, so I interrupted the process. I played a bit with your file and it seems like many of the lines are broken into segments unnecessarily (I'm guessing this is because the ice sheet outlines are somehow determined on tiles and then pieced together afterwards, but only the providers will really know). Maybe it would help to piece together adjacent pieces, but I'm not sure.
I was also wondering whether the result would even look that great with a dotted line, because there are so many sharp bends. Below I show a picture where I only plot the 100 longest line segments (leaving out drawcoastlines and with thicker lines) using this code:
import numpy as np
result = m.readshapefile('shapefiles/nic_autoc2018046n_pl_a', 'IceEdge', zorder = 10, color = 'blue')#, drawbounds = False)
col = result[-1]
segments = col.get_segments()
seglens = [len(seg) for seg in col.get_segments()]
segments = np.array(segments)
seglens = np.array(seglens)
idx = np.argsort(seglens)
seglens = seglens[idx]
segments = segments[idx]
col.remove()
new_col = LineCollection(segments[-100:],linewidths = 2, linestyles='dotted', colors='b')
ax.add_collection(new_col)
plt.show()
And the result looks like this:

Python Basemap: Error using shadedrelief, bluemarble or etopo (false longitude format?)

I want to plot a map of the southern hemisphere centered on the pacific with some stuff drawn onto it with python matplotlib basemap.
Everything works fine unless I try to draw a background image with the basemap routines shadedrelief, bluemarble or etopo. The code (without the stuff i want to draw onto the map) looks like this:
import numpy as np
from mpl_toolkits.basemap import Basemap
from matplotlib.backends.backend_pdf import PdfPages
latmin = -72.5
latmax = 40.
lonmin = 60.
lonmax = 370.
pp = PdfPages('datamap.pdf')
m = Basemap(projection='merc', llcrnrlat=latmin, urcrnrlat=latmax, llcrnrlon=lonmin, urcrnrlon=lonmax, resolution="c")
m.drawcoastlines(linewidth=0.25)
#m.shadedrelief()
pp.savefig()
pp.close()
When I uncomment the m.shadedrelief() I get the following:
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
/xyz/datamap.py in <module>()
32
---> 33 m.shadedrelief()
34
/usr/local/lib/python2.7/site-packages/mpl_toolkits/basemap/__init__.pyc in shadedrelief(self, ax, scale, **kwargs)
3997 return self.warpimage(image='shadedrelief',ax=ax,scale=scale,**kwargs)
3998 else:
-> 3999 return self.warpimage(image='shadedrelief',scale=scale,**kwargs)
4000
4001 def etopo(self,ax=None,scale=None,**kwargs):
/usr/local/lib/python2.7/site-packages/mpl_toolkits/basemap/__init__.pyc in warpimage(self, image, scale, **kwargs)
4115 # any range of longitudes may be plotted on a world map.
4116 self._bm_lons = \
-> 4117 np.concatenate((self._bm_lons,self._bm_lons+360),1)
4118 self._bm_rgba = \
4119 np.concatenate((self._bm_rgba,self._bm_rgba),1)
IndexError: axis 1 out of bounds [0, 1)
When I choose (for test purposes) way smaller maps which also dont have longitude larger than 180 degree everything (including shadedrelief) works fine. This leads me to the assumption that something with the format of the longitude is not working here. I tried some things but i cant seem to find a way how to solve this while still plotting the same map section
Do you have any idea how i could draw a shadedrelief in the background of my map?
Best,
xilian

It seems there was a bug in the basemap/init code. In fact, if you look at the latest version on github [see https://github.com/matplotlib/basemap/blob/master/lib/mpl_toolkits/basemap/init.py#L4139 and how it is different from line 4117 of your error] you will find out that the bug has been solved already. This means you can either do:
# np.concatenate((self._bm_lons,self._bm_lons+360),1)
np.concatenate((self._bm_lons,self._bm_lons+360))
in the init.py code or get a newer version of basemap. Any of the two will solve your problem.

Curious (bad?) behavior creating all-sky projections with matplotlib

I am trying to make a density "all-sky" plot which is complete in RA (i.e 0 to 360 deg) but incomplete in DEC (let's say from -45 to 90 deg). If I plot this without any projection it is ok, but when I try to plot using the 'mollweide' projection I am not recovering the input, but if I do a little change in the code I do recover the expected behavior (however, I don't have a coherent explanation for this change as you'll see in the example).
Let's see a self-contained example with its outputs to be clearer:
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.backends.backend_agg
from math import pi
#array between 0 and 360 deg
RA = np.random.random(10000)*360
#array between -45 and 90 degrees. By construction!
DEC= np.random.random(10000)*135-45
fig = plt.Figure((10, 4.5))
ax = fig.add_subplot(111,projection='mollweide')
ax.grid(True)
ax.set_xlabel('RA')
ax.set_ylabel('DEC')
ax.set_xticklabels(np.arange(30,331,30))
hist,xedges,yedges = np.histogram2d(DEC,RA,bins=[90,180],range=[[-90,90],[0,360]])
#TO RECOVER THE EXPECTED BEHAVIOUR, I HAVE TO CHANGE -90 FOR -80 IN THE PREVIOUS LINE:
#hist,xedges,yedges = np.histogram2d(DEC,RA,bins=[90,180],range=[[-80,90],[0,360]])
#I DO NOT WHY!
extent = (-pi,pi,-pi/2.,pi/2.)
image = ax.imshow(hist,extent=extent,clip_on=False,aspect=0.5,origin='lower')
cb = fig.colorbar(image, orientation='horizontal')
canvas = matplotlib.backends.backend_agg.FigureCanvasAgg(fig)
fig.canvas.print_figure("image1.png")
And the output image is:
[As I am new here I am not allowed to post images, so I will post a link, if it does not work, please write me an email and I can share a Dropbox folder with the images ;)]
Output Image that I am getting
Where you can see clearly that the RA is OK so it ranges between 0 and 360, BUT the DEC ranges from -35 to 90 instead of -45 to 90. So far I do not understand why I am missing 10 deg.
However, if I do a little change in the code, replacing the line
hist,xedges,yedges = np.histogram2d(DEC,RA,bins=[90,180],range=[[-90,90],[0,360]]
for
hist,xedges,yedges = np.histogram2d(DEC,RA,bins=[90,180],range=[[-80,90],[0,360]]
I get what I think I should get, which is this plot:
Output Image 2
[Again, if the link does not work, let me know and I can share a Dropbox folder with you]
where DEC now ranges from -45 to 90 as expected because I created DEC in that way.
However the change of -90 for -80 doesn't make sense (I think).
So probably I am doing something wrong that I can't notice now, or I am misunderstanding something in the code or there is a curious bug in matplotlib??
Please any help/hint/correction would be greatly appreciate it
Eduardo

if you don't mind depending on an external package, you could do this with healpy, that provides a Mollweide projection for the Healpix sky pixellization:
https://github.com/healpy/healpy
See an example similar to your script here:
https://gist.github.com/1215159
More info about healpix:
http://healpix.jpl.nasa.gov/html/intro.htm
Ouput image:

If this is useful for someone else, this is the "corrected version" of my code, which gives as output this image. The main change is to use pcolormesh instead of imshow (as #Joe suggested):
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.backends.backend_agg
#array between 0 and 360 deg
#CAVEAT: it seems that is needed an array from -180 to 180, so is just a
#shift in the coordinates
RA = np.random.random(10000)*360-180
#array between -45 and 90 degrees
DEC= np.random.random(10000)*135-45
fig = plt.Figure((10, 5))
ax = fig.add_subplot(111,projection='mollweide')
ax.set_xlabel('RA')
ax.set_ylabel('DEC')
ax.set_xticklabels(np.arange(30,331,30))
hist,xedges,yedges = np.histogram2d(DEC,RA,bins=[60,40],range=[[-90,90],[-180,180]])
X,Y = np.meshgrid(np.radians(yedges),np.radians(xedges))
image = ax.pcolormesh(X,Y,hist)
ax.grid(True)
cb = fig.colorbar(image, orientation='horizontal')
canvas = matplotlib.backends.backend_agg.FigureCanvasAgg(fig)
fig.canvas.print_figure("image4.png")