I'm currently trying to plot data from an nc file containing IASI satellite's images of radiances of N2O gas in the atmosphere I work with netcdf4, numpy, matplotlib librairies.
Problem is , I get an indexError coming from my m.contour input variables : X and Y must be the same length as Z of the same array saize of Z and idk how to plot that according to my data : X,Y = longtitude, latitude = 1D arrays or lisst of 235586 elements and Z = 2d array 235586*14
Here is the code :
import netCDF4 as nc
import numpy as np
import matplotlib.colors as mcolors
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap, cm, shiftgrid, addcyclic # mapping
# Open nc file
filePath = 'C:\\Users\\Lucas\\Documents\\ProjetN2O\\iasi_ret_L2_n2o_ch4_2019_05_02.nc'
ds = nc.Dataset(filePath)
print(ds)
# lat,long,time,N20,avk,Surf_P : paramètres d'interêt
# Print metadata as python dictionary
#print(ds.__dict__)
#Print dimensions and var of the dataset
for dimension in ds.dimensions.values():
print(dimension)
for var in ds.variables.values():
print(var)
## Extraction of variables in numpy array
temps = np.array(ds.variables["time"])
latitude = np.array(ds.variables["lat"])
longitude = np.array(ds.variables["lon"])
n2O_r = np.array(ds.variables["n2o_retrieval"])
n2O_avk = np.array(ds.variables["n2o_AVK"])
n2O_ap = np.array(ds.variables["n2o_apriori"])
SurfP = np.array(ds.variables["Surf_P"])
SurfPid = np.array(ds.variables["Surf_P_id"])
#Select a portion of the rows index cause to many rows to plot in n2O_r
A = n2O_r[0:235587, 0:1] #All values in the 1st row
B = A.ravel()# resize to a 1D-array
# Plot variables
lat0 = 0; lon0 = 0 #lat et long 0
lon_max = longitude.max(); lon_min = longitude.min() #lat lon max
lat_max = latitude.max(); lat_min = latitude.min()
#Plot options
m = Basemap(projection='cyl',lat_0=lat0,lon_0=lon0,resolution='c',## Projection
llcrnrlat=lat_min,urcrnrlat=lat_max,\
llcrnrlon=lon_min,urcrnrlon=lon_max)
m.drawcoastlines(linewidth=1.2, linestyle='solid', color='k', antialiased=1,\
zorder=2)
m.drawcountries()
m.drawlsmask(land_color='none', ocean_color='aqua', zorder=1)
viridis =plt.get_cmap('viridis', 12)
cs = m.contour(latitude,longitude,B,1000,linewidths=1.5,cmap=viridis, \
colors='b', alpha=0.3)
I tried to reshape all the inputs for the plot into the same dimensions but still can't get rid of the error
error :
Traceback (most recent call last):
File "C:\Users\Lucas\Documents\Projet N2O\Plot.py", line 70, in
cs = m.contour(latitude,longitude,n2O_r,1000,linewidths=1.5,cmap=viridis,
File "C:\Users\Lucas\AppData\Local\Programs\Python\Python38\lib\site-packages\mpl_toolkits\basemap_init_.py", line 549, in with_transform
return plotfunc(self,x,y,data,*args,**kwargs)
File "C:\Users\Lucas\AppData\Local\Programs\Python\Python38\lib\site-packages\mpl_toolkits\basemap_init_.py", line 3570, in contour
xx = x[x.shape[0]//2,:]
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed
Variables :
enter image description here
Related
Edit: I'm starting to suspect the problems arising below are due to the metadata, because even after correcting the issues raised regarding units mpcalc.geostrophic_wind(z) still issues warnings about the coordinates and ordering. Maybe the function is unable to identify the coordinates from the file? Perhaps this is because WRF output data is non-CF compliant?
I would like to compute geostrophic and ageostrophic winds from WRF-ARW data using the MetPy function mpcalc.geostrophic_wind.
My attempt results in a bunch of errors and I don't know what I'm doing wrong. Can someone tell me how to modify my code to get rid of these errors?
Here is my attempt so far:
#
import numpy as np
from netCDF4 import Dataset
import metpy.calc as mpcalc
from wrf import getvar
# Open the NetCDF file
filename = "wrfout_d01_2016-10-04_12:00:00"
ncfile = Dataset(filename)
# Extract the geopotential height and wind variables
z = getvar(ncfile, "z", units="m")
ua = getvar(ncfile, "ua", units="m s-1")
va = getvar(ncfile, "va", units="m s-1")
# Smooth height data
z = mpcalc.smooth_gaussian(z, 3)
# Compute the geostrophic wind
geo_wind_u, geo_wind_v = mpcalc.geostrophic_wind(z)
# Calculate ageostrophic wind components
ageo_wind_u = ua - geo_wind_u
ageo_wind_v = va - geo_wind_v
#
The computation of the geostrophic wind throws several warnings:
>>> # Compute the geostrophic wind
>>> geo_wind_u, geo_wind_v = mpcalc.geostrophic_wind(z)
/mnt/.../.../metpy_en/lib/python3.9/site-packages/metpy/xarray.py:355: UserWarning: More than one time coordinate present for variable.
warnings.warn('More than one ' + axis + ' coordinate present for variable'
/mnt/.../.../lib/python3.9/site-packages/metpy/xarray.py:1459: UserWarning: Horizontal dimension numbers not found. Defaulting to (..., Y, X) order.
warnings.warn('Horizontal dimension numbers not found. Defaulting to '
/mnt/.../.../lib/python3.9/site-packages/metpy/xarray.py:355: UserWarning: More than one time coordinate present for variable "XLAT".
warnings.warn('More than one ' + axis + ' coordinate present for variable'
/mnt/.../.../lib/python3.9/site-packages/metpy/xarray.py:1393: UserWarning: y and x dimensions unable to be identified. Assuming [..., y, x] dimension order.
warnings.warn('y and x dimensions unable to be identified. Assuming [..., y, x] '
/mnt/.../.../lib/python3.9/site-packages/metpy/calc/basic.py:1274: UserWarning: Input over 1.5707963267948966 radians. Ensure proper units are given.
warnings.warn('Input over {} radians. '
Can anyone tell me why I'm getting these warnings?
And then trying to compute an ageostrophic wind component results in a bunch of errors:
>>> # Calculate ageostrophic wind components
>>> ageo_wind_u = ua - geo_wind_u
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/.../lib/python3.9/site-packages/xarray/core/_typed_ops.py", line 209, in __sub__
return self._binary_op(other, operator.sub)
File "/mnt/.../lib/python3.9/site-packages/xarray/core/dataarray.py", line 4357, in _binary_op f(self.variable, other_variable)
File "/mnt/.../lib/python3.9/site-packages/xarray/core/_typed_ops.py", line 399, in __sub__
return self._binary_op(other, operator.sub)
File "/mnt/.../lib/python3.9/site-packages/xarray/core/variable.py", line 2639, in _binary_op
f(self_data, other_data) if not reflexive else f(other_data, self_data)
File "/mnt/iusers01/fatpou01/sees01/w34926hb/.conda/envs/metpy_env/lib/python3.9/site-packages/pint/facets/numpy/quantity.py", line 61, in __array_ufunc__
return numpy_wrap("ufunc", ufunc, inputs, kwargs, types)
File "/mnt/.../lib/python3.9/site-packages/pint/facets/numpy/numpy_func.py", line 953, in numpy_wrap return handled[name](*args, **kwargs)
File "/mnt/.../lib/python3.9/site-packages/pint/facets/numpy/numpy_func.py", line 513, in _subtract (x1, x2), output_wrap = unwrap_and_wrap_consistent_units(x1, x2)
File "/mnt/.../lib/python3.9/site-packages/pint/facets/numpy/numpy_func.py", line 130, in unwrap_and_wrap_consistent_units args, _ = convert_to_consistent_units(*args, pre_calc_units=first_input_units)
File "/mnt/.../lib/python3.9/site-packages/pint/facets/numpy/numpy_func.py", line 111, in convert_to_consistent_units tuple(convert_arg(arg, pre_calc_units=pre_calc_units) for arg in args),
File "/mnt/.../lib/python3.9/site-packages/pint/facets/numpy/numpy_func.py", line 111, in <genexpr> tuple(convert_arg(arg, pre_calc_units=pre_calc_units) for arg in args),
File "/mnt/.../lib/python3.9/site-packages/pint/facets/numpy/numpy_func.py", line 93, in convert_arg raise DimensionalityError("dimensionless", pre_calc_units)
pint.errors.DimensionalityError: Cannot convert from 'dimensionless' to 'meter / second'
Any help would be appreciated.
(By the way, I looked at the script at https://github.com/Unidata/python-training/blob/master/pages/gallery/Ageostrophic_Wind_Example.ipynb and did not find it helpful because I'm not sure which of the data manipulations near the top I need to do for the WRF data.)
wrfpython's getvar function, while it takes units as a parameter, only uses this (as far as I can tell) to convert values in the arrays before returning them. To use this with MetPy you need to attach proper units. I would do this using a small helper function:
from metpy.units import units
def metpy_getvar(file, name, units_str):
return getvar(file, name, units=units_str) * units(units_str)
z = metpy_getvar(ncfile, "z", units="m")
ua = metpy_getvar(ncfile, "ua", units="m s-1")
va = metpy_getvar(ncfile, "va", units="m s-1")
That should eliminate the complaints about missing units.
EDIT: Fix name collision in hastily written function.
The data presented by raw WRF-ARW datasets and by variables extracted via wrf-python do not have metadata that interact well with MetPy's assumptions about unit attributes, coordinate variables, and grid projections (from the CF Conventions). Instead, I would recommend using xwrf, a recently released package for working with WRF data in a more CF-Conventions-friendly way. With xwrf, your example would look like:
import metpy.calc as mpcalc
import xarray as xr
import xwrf
# Open the NetCDF file
filename = "wrfout_d01_2016-10-04_12:00:00"
ds = xr.open_dataset(filename).xwrf.postprocess()
# Extract the geopotential height and wind variables
z = ds['geopotential_height']
ua = ds['wind_east']
va = ds['wind_north']
# Smooth height data
z = mpcalc.smooth_gaussian(z, 3)
# Compute the geostrophic wind
geo_wind_u, geo_wind_v = mpcalc.geostrophic_wind(z)
# Calculate ageostrophic wind components
ageo_wind_u = ua - geo_wind_u
ageo_wind_v = va - geo_wind_v
I've made some progress: an updated script and the resulting plot are included below. Part of the problem was that I needed to pass dx, dy, and lat into the function metpy.calc.geostrophic_wind, as they were seemingly not being read automatically from the numpy array.
There are still (at least) two problems:
I've passed x_dim=-2 and y_dim=-1 in an effort to set [X,Y] order. (The documentation here https://unidata.github.io/MetPy/latest/api/generated/metpy.calc.geostrophic_wind.html says the default is x_dim = -1 and y_dim=-2 for [...Y,X] order, but does not say what to set x_dim and y_dim to for [...X,Y] order, so I just guessed.) However, I am still getting ``UserWarning: Horizontal dimension numbers not found. Defaulting to (..., Y, X) order.''
Secondly, as you can see in the plot there is something weird going on with the geostrophic wind component at the coastlines.
u-component of geostrophic wind at 300 mb
Here is my current script:
import numpy as np
from netCDF4 import Dataset
import metpy.calc as mpcalc
from metpy.units import units
import matplotlib.pyplot as plt
from matplotlib.cm import get_cmap
from wrf import getvar, interplevel, to_np, get_basemap, latlon_coords
# Open the NetCDF file
filename = "wrfout_d01_2016-10-04_12:00:00"
ncfile = Dataset(filename)
z = getvar(ncfile, "z", units="m") * units.meter
# Smooth height data
z = mpcalc.smooth_gaussian(z, 3)
dx = 4000.0 * units.meter
dy = 4000.0 * units.meter
lat = getvar(ncfile, "lat") * units.degrees
geo_wind_u, geo_wind_v = mpcalc.geostrophic_wind(z,dx,dy,lat,x_dim=-2,y_dim=-1)
#####
p = getvar(ncfile, "pressure")
z = getvar(ncfile, "z", units="m")
ht_300 = interplevel(z, p, 300)
#geostrophic wind components on 300 mb level
geo_wind_u_300 = interplevel(geo_wind_u, p, 300)
geo_wind_v_300 = interplevel(geo_wind_v, p, 300)
# Get the lat/lon coordinates
lats, lons = latlon_coords(ht_300)
# Get the basemap object
bm = get_basemap(ht_300)
# Create the figure
fig = plt.figure(figsize=(12,12))
ax = plt.axes()
# Convert the lat/lon coordinates to x/y coordinates in the projection space
x, y = bm(to_np(lons), to_np(lats))
# Add the 300 mb height contours
levels = np.arange(8640., 9690., 40.)
contours = bm.contour(x, y, to_np(ht_300), levels=levels, colors="black")
plt.clabel(contours, inline=1, fontsize=10, fmt="%i")
# Add the wind contours
levels = np.arange(10, 70, 5)
geo_u_contours = bm.contourf(x, y, to_np(geo_wind_u_300), levels=levels, cmap=get_cmap("YlGnBu"))
plt.colorbar(geo_u_contours, ax=ax, orientation="horizontal", pad=.05, shrink=0.75)
# Add the geographic boundaries
bm.drawcoastlines(linewidth=0.25)
bm.drawstates(linewidth=0.25)
bm.drawcountries(linewidth=0.25)
plt.title("300 mb height (m) and u-component of geostrophic wind (m s-1) at 1200 UTC on 04-10-2016", fontsize=12)
plt.savefig('geo_u_300mb_04-10-2016_1200_smoothed.png', bbox_inches='tight')
I have spent countless hours looking through posts but can't seem to get my answers right.
-Subtract the mean values (mn) from each of the points (i.e. center the points around the origin and store these in a new matrix called A (This is the first step in PCA).
-Calculate the 3×3 Gram matrix
If anyone can please help me out id greatly appreciate it.
%matplotlib inline
import matplotlib.pylab as plt
import numpy as np
import sympy as sym
sym.init_printing(use_unicode=True)
#get the data file from the internet:
from urllib.request import urlopen, urlretrieve
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00229/Skin_NonSkin.txt'
file = "Skin_NonSkin.txt"
response = urlopen(url)
data = response.read() # a `bytes` object
text = data.decode('utf-8')
lines = text.split('\r\n')
data = []
#Read in file line by line
for line in lines:
try:
if line:
data.append(list(map(int, line.split('\t'))))
except:
print('invalid line of data:',line)
response.close()
#Convert the file to a list of points
P = np.matrix(data)
P.shape
#Mask out only face values and keep just the RGBs
mask = np.array(P[:,3]==1)
mask = mask.flatten()
points = P[mask,:]
## Change order to Red, Green, Blue
points = points[:,(2,1,0)]
# Plot the points in 3D using their actual color values
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(points[:,0], points[:,1], points[:,2], c=points/255)
ax.set_xlabel('Red');
ax.set_ylabel('Green');
ax.set_zlabel('Blue');
points = points[:,(2,1,0)]
red_mean = np.mean(points[:,0])
green_mean = np.mean(points[:,1])
blue_mean = np.mean(points[:,2])
mn = np.array((red_mean,green_mean,blue_mean), dtype=float)
You can calculate the mean of a matrix in a given axis, and subtract it directly.
mn = np.mean(points, axis=0)
C = points - mn;
The Gramm matrix G defined as the matrix whose entries is the inner product of the columns of C, to make the scale invariant to the size of C we divide the result by its number of rows
G = (C.T # C) / C.shape[0]
Using NASA's SRTM data, I've generated a global elevation heatmap.
The problem is, however, the continents tend to blend in with the ocean because of the range of elevation values. Is it possible to change the colorbar's scale so that the edges of the continents are more distinct from the ocean? I've tried different cmaps, but they all seem to suffer from the problem.
Here is my code. I'm initializing a giant array (with 0s) to hold global elevation data, and then populating it file by file from the SRTM dataset. Each file is 1 degree latitude by 1 degree longitude.
Another question I had was regarding the map itself. For some reason, the Appalachian Mountains seem to have disappeared entirely.
import os
import numpy as np
from .srtm_map import MapGenerator
from ..utils.hgt_parser import HGTParser
from tqdm import tqdm
import cv2
import matplotlib.pyplot as plt
import richdem as rd
class GlobalMapGenerator():
def __init__(self):
self.gen = MapGenerator()
self.base_dir = "data/elevation/"
self.hgt_files = os.listdir(self.base_dir)
self.global_elevation_data = None
def shrink(data, rows, cols):
return data.reshape(rows, data.shape[0]/rows, cols, data.shape[1]/cols).sum(axis=1).sum(axis=2)
def GenerateGlobalElevationMap(self, stride):
res = 1201//stride
max_N = 59
max_W = 180
max_S = 56
max_E = 179
# N59 --> N00
# S01 --> S56
# E000 --> E179
# W180 --> W001
# Initialize array global elevation
self.global_elevation_data = np.zeros(( res*(max_S+max_N+1), res*(max_E+max_W+1) ))
print("Output Image Shape:", self.global_elevation_data.shape)
for hgt_file in tqdm(self.hgt_files):
lat_letter = hgt_file[0]
lon_letter = hgt_file[3]
lat = int(hgt_file[1:3])
lon = int(hgt_file[4:7])
if lat_letter == "S":
# Shift south down by max_N, but south starts at S01 so we translate up by 1 too
lat_trans = max_N + lat - 1
else:
# Bigger N lat means further up. E.g. N59 is at index 0 and is higher than N00
lat_trans = max_N - lat
if lon_letter == "E":
# Shift east right by max_W
lon_trans = max_W + lon
else:
# Bigger W lon means further left. E.g. W180 is at index 0 and is more left than W001
lon_trans = max_W - lon
# load in data from file as resized
data = cv2.resize(HGTParser(os.path.join(self.base_dir, hgt_file)), (res, res))
# generate bounds (x/y --> lon.lat for data from this file for the giant array)
lat_bounds = [res*lat_trans, res*(lat_trans+1)]
lon_bounds = [res*lon_trans, res*(lon_trans+1)]
try:
self.global_elevation_data[ lat_bounds[0]:lat_bounds[1], lon_bounds[0]:lon_bounds[1] ] = data
except:
print("REFERENCE ERROR: " + hgt_file)
print("lat: ", lat_bounds)
print("lon: ", lon_bounds)
# generate figure
plt.figure(figsize=(20,20))
plt.imshow(self.global_elevation_data, cmap="rainbow")
plt.title("Global Elevation Heatmap")
plt.colorbar()
plt.show()
np.save("figures/GlobalElevationMap.npy", self.global_elevation_data)
plt.savefig("figures/GlobalElevationMap.png")
def GenerateGlobalSlopeMap(self, stride):
pass
Use a TwoSlopeNorm (docs) for your norm, like the example here.
From the example:
Sometimes we want to have a different colormap on either side of a conceptual center point, and we want those two colormaps to have different linear scales. An example is a topographic map where the land and ocean have a center at zero, but land typically has a greater elevation range than the water has depth range, and they are often represented by a different colormap.
If you set the midpoint at sea level (0), then you can have two very different scalings based on ocean elevation vs land elevation.
Example code (taken from the example linked above):
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as colors
import matplotlib.cbook as cbook
from matplotlib import cm
dem = cbook.get_sample_data('topobathy.npz', np_load=True)
topo = dem['topo']
longitude = dem['longitude']
latitude = dem['latitude']
fig, ax = plt.subplots()
# make a colormap that has land and ocean clearly delineated and of the
# same length (256 + 256)
colors_undersea = plt.cm.terrain(np.linspace(0, 0.17, 256))
colors_land = plt.cm.terrain(np.linspace(0.25, 1, 256))
all_colors = np.vstack((colors_undersea, colors_land))
terrain_map = colors.LinearSegmentedColormap.from_list(
'terrain_map', all_colors)
# make the norm: Note the center is offset so that the land has more
# dynamic range:
divnorm = colors.TwoSlopeNorm(vmin=-500., vcenter=0, vmax=4000)
pcm = ax.pcolormesh(longitude, latitude, topo, rasterized=True, norm=divnorm,
cmap=terrain_map, shading='auto')
# Simple geographic plot, set aspect ratio beecause distance between lines of
# longitude depends on latitude.
ax.set_aspect(1 / np.cos(np.deg2rad(49)))
ax.set_title('TwoSlopeNorm(x)')
cb = fig.colorbar(pcm, shrink=0.6)
cb.set_ticks([-500, 0, 1000, 2000, 3000, 4000])
plt.show()
See how it scales numbers with this simple usage (from docs):
>>> import matplotlib. Colors as mcolors
>>> offset = mcolors.TwoSlopeNorm(vmin=-4000., vcenter=0., vmax=10000)
>>> data = [-4000., -2000., 0., 2500., 5000., 7500., 10000.]
>>> offset(data)
array([0., 0.25, 0.5, 0.625, 0.75, 0.875, 1.0])
I try to make a climatic map in python which I am not used to use but want to try if it is more handy than plotting in R. I use the example by
http://joehamman.com/2013/10/12/plotting-netCDF-data-with-Python/ with my data.
from netCDF4 import Dataset
import numpy as np
myfil = "xxxxx"
fh = Dataset(myfil, mode='r')
lons = fh.variables['lon'][:]
lats = fh.variables['lat'][:]
tmean = fh.variables['Tmean_ANN'][:1]
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
# Get some parameters for the Stereographic Projection
lon0 = lons.mean()
lats0 = lats.mean()
m = Basemap(width=5000000,height=3500000,
resolution='l',projection='stere',
lat_ts=60,lat_0=lats0,lon_0=lon0)
lon, lat = np.meshgrid(lons, lats, sparse=True)
xi, yi = m(lon, lat)
# Plot Data
print(xi.shape)
print(yi.shape)
print(tmean.shape)
results
(1, 1142761)
(1142761, 1)
(1, 1069, 1069)
Trying to run this line
cs = m.contour(xi,yi, np.squeeze(tmean))
I got the error
cs = m.contour(xi,yi, np.squeeze(tmean))
Traceback (most recent call last):
File "<ipython-input-37-8be9f03a0e45>", line 1, in <module>
cs = m.contour(xi,yi, np.squeeze(tmean))
File "C:\ProgramData\Anaconda3\lib\site-packages\mpl_toolkits\basemap\__init__.py", line 546, in with_transform
return plotfunc(self,x,y,data,*args,**kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\mpl_toolkits\basemap\__init__.py", line 3566, in contour
np.logical_or(np.greater(x,self.xmax+epsx),np.greater(y,self.ymax+epsy))
MemoryError
Any help why do I got this. My hypothesis is that the dimension of xi and yi is not compatible with tmean thus I got the error. The np.sqeeze function works with tmean data outside the m.contour function But I could not solve that for a while.
I'm looking to compute poleward heat fluxes at a level in the atmosphere, i.e the mean of (u't') . I'm aware of the covariance function in NumPy, but cannot seem to implement it. Here is my code below.
from netCDF4 import Dataset
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
myfile = '/home/ubuntu/Fluxes_Test/out.nc'
Import = Dataset(myfile, mode='r')
lon = Import.variables['lon'][:] # Longitude
lat = Import.variables['lat'][:] # Latitude
time = Import.variables['time'][:] # Time
lev = Import.variables['lev'][:] # Level
wind = Import.variables['ua'][:]
temp = Import.variables['ta'][:]
lon = lon-180 # to shift co-ordinates to -180 to 180.
variable1 = np.squeeze(wind,temp, axis=0)
variable2 = np.cov(variable1)
m = Basemap(resolution='l')
lons, lats = np.meshgrid(lon,lat)
X, Y = m(lons, lats)
cs = m.pcolor(X,Y, variable2)
plt.show()
The shape of the variables wind and temp which I am trying to compute the flux of (the covariance) are both (3960,64,128), so 3960 pieces of data on a 64x128 grid (with co-ordinates).
I tried squeezing both variables to produce a array of (3960, 3960, 64,128) so cov could work on these first two series of data (the two 3960's) of wind and temp, but this didn't work.