Develop Reference

Issues with Scipy interpolate griddata

I have a netcdf file with a spatial resolution of 0.05º and I want to regrid it to a spatial resolution of 0.01º like this other netcdf. I tried using scipy.interpolate.griddata, but I am not really getting there, I think there is something that I am missing.
original_dataset = xr.open_dataset('to_regrid.nc')
target_dataset= xr.open_dataset('SSTA_L4_MED_0_1dg_2022-01-18.nc')
According to scipy.interpolate.griddata documentation, I need to construct my interpolation pipeline as following:
grid = griddata(points, values, (grid_x_new, grid_y_new),
method='nearest')
So in my case, I assume it would be as following:
#Saving in variables the old and new grids
grid_x_new = target_dataset['lon']
grid_y_new = target_dataset['lat']
grid_x_old = original_dataset ['lon']
grid_y_old = original_dataset ['lat']
points = (grid_x_old,grid_y_old)
values = original_dataset['analysed_sst'] #My variable in the netcdf is the sea surface temp.
Now, when I run griddata:
from scipy.interpolate import griddata
grid = griddata(points, values, (grid_x_new, grid_y_new),method='nearest')
I am getting the following error:
ValueError: shape mismatch: objects cannot be broadcast to a single
shape
I assume it has something to do with the lat/lon array shapes. I am quite new to netcdf field and don't really know what can be the issue here. Any help would be very appreciated!

In your original code the indices in grid_x_old and grid_y_old should correspond to each unique coordinate in the dataset. To get things working correctly something like the following will work:
import xarray as xr
from scipy.interpolate import griddata
original_dataset = xr.open_dataset('to_regrid.nc')
target_dataset= xr.open_dataset('SSTA_L4_MED_0_1dg_2022-01-18.nc')
#Saving in variables the old and new grids
grid_x_old = original_dataset.to_dataframe().reset_index().loc[:,["lat", "lon"]].lon
grid_y_old = original_dataset.to_dataframe().reset_index().loc[:,["lat", "lon"]].lat
grid_x_new = target_dataset.to_dataframe().reset_index().loc[:,["lat", "lon"]].lon
grid_y_new = target_dataset.to_dataframe().reset_index().loc[:,["lat", "lon"]].lat
values = original_dataset.to_dataframe().reset_index().loc[:,["lat", "lon", "analysed_sst"]].analysed_sst
points = (grid_x_old,grid_y_old)
grid = griddata(points, values, (grid_x_new, grid_y_new),method='nearest')

I recommend using xesm for regridding xarray datasets. The code below will regrid your dataset:
import xarray as xr
import xesmf as xe
original_dataset = xr.open_dataset('to_regrid.nc')
target_dataset= xr.open_dataset('SSTA_L4_MED_0_1dg_2022-01-18.nc')
regridder = xe.Regridder(original_dataset, target_dataset, "bilinear")
dr_out = regridder(original_dataset)

I am trying visualise railway track from shape shapes in python

I am trying to visualise railway tracks using Plotly lines on a map. I am a beginner who is working on gis data and visualise railway networks. i am not sure about the approach i am going about and i don't want to use arcgis or qgis.
Problem: I am not sure what should be added in the names attribute.
it would great i could get an overview of the code where i am going wrong
zipfile contains the .shp,.dbf.shx,.prj,.cpj
The error i am getting is 'GeoDataFrame' object has no attribute 'name'
this the code which i am working on.
import plotly.express as px
import geopandas as gpd
import shapely.geometry
import numpy as np
import wget
geo_df = gpd.read_file(r'railwaytrack.zip' )
lattiudes = [47.04691]
longitudes = [8.37467]
names=[ ]
for feature, line_geo in zip(geo_df.geometry, geo_df.name):
if isinstance(feature, shapely.geometry.linestring.LineString):
linestrings = [feature]
elif isinstance(feature, shapely.geometry.multilinestring.MultiLineString):
linestrings = feature.geoms
else:
continue
for linestring in linestrings:
x, y = linestring.xy
lattiudes = np.append(lattiudes, y)
longitudes = np.append(longitudes, x)
names = np.append(names, [name]*len(y))
lattiudes = np.append(lattiudes, None)
longitudes = np.append(longitudes, None)
names = np.append(names, None)
fig = px.line_geo(lat=lattiudes, lon=longitudes)
fig.show()

How to represent scalar variables over geographic map in Jupyter Notebook

I am representing some geographical data in a Jupyter notebook: temperature, ocean wave height, etc. I have numpy arrays that have the latitude, longitude, and value for those variables. I would like to display these variables over a geographical map, preferably using ipyleaflet (because that is what I am already using). I am trying to get a result similar to a heatmap.
I tried to use the ipyleaflet Heatmap, but it seems to me that it is designed to represent agregation of points and not scalar uniform arrays, because I can't get it to show the results properly. I think ipyleaflet may lack a function to represent this kind of data, but seems odd since it has a very nice Velocity funtion to represent vectorial variables.
The only way I can think of to make this would be to generate an image with matplotlib and then adding it to the map in an image layer, but I feel like that is not the proper way to do it.

For representing a heatmap I would recommend to use Cartopy in combination with Matplotlib.
Here a ready to use script I made for a world projection with a coastline:
import numpy as np
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.feature as cfeature
from cartopy.util import add_cyclic_point
# Set x y z variables
x = longitude_data
y = latitude_data
z = heat_map_data
# Set up figure and projection
z, x = add_cyclic_point(z, coord=x)
fig = plt.figure()
ax = fig.add_subplot(1,1,1, projection=ccrs.PlateCarree() )
# Set data range and colourmap
levels = np.arange(min,max,steps)
plt.contourf(x, y, z,levels = levels,transform=ccrs.PlateCarree(),cmap="rainbow")
# Set axes, extent (world) and labels
ax.set_xticks(np.linspace(-180,180,num=7), crs=ccrs.PlateCarree())
ax.set_yticks(np.linspace(-60,60,num=5), crs=ccrs.PlateCarree())
ax.add_feature(cfeature.COASTLINE) #Add coastline
ax.set_global()
ax.set_title('Heatmap')
ax.set_xlabel('Longitude')
ax.set_ylabel('Latitude')
# Add colorbar
plt.colorbar(ax=ax,shrink=0.7,orientation="vertical")
fig.show()
With the Cartopy and Matplotlib documentation you should now be able to create some maps.

As you say, you can generate an image and overlay it onto the map. This is the suggestion I was given when I asked about this on Github.
There's an example notebook here.
Not quite as easy as matplotlib, but you get all the nice interactivity of ipyleaflet!
Here's the current version of the notebook in case the link changes (converted to markdown via jupyter-nbconvert --to markdown Numpy.ipynb)
From NumPy to Leaflet
This notebook shows how to display some raster geographic data in IPyLeaflet. The data is a NumPy array, which means that you have all the power of the Python scientific stack at your disposal to process it.
The following libraries are needed:
* requests
* tqdm
* rasterio
* numpy
* scipy
* pillow
* matplotlib
* ipyleaflet
The recommended way is to try to conda install them first, and if they are not found then pip install.
import requests
import os
from tqdm import tqdm
import zipfile
import rasterio
from affine import Affine
import numpy as np
import scipy.ndimage
from rasterio.warp import reproject, Resampling
import PIL
import matplotlib.pyplot as plt
from base64 import b64encode
try:
from StringIO import StringIO
py3 = False
except ImportError:
from io import StringIO, BytesIO
py3 = True
from ipyleaflet import Map, ImageOverlay, basemap_to_tiles, basemaps
Download a raster file representing the flow accumulation for South America. This gives an idea of the river network.
url = 'https://edcintl.cr.usgs.gov/downloads/sciweb1/shared/hydrosheds/sa_30s_zip_grid/sa_acc_30s_grid.zip'
filename = os.path.basename(url)
name = filename[:filename.find('_grid')]
adffile = name + '/' + name + '/w001001.adf'
if not os.path.exists(adffile):
r = requests.get(url, stream=True)
with open(filename, 'wb') as f:
total_length = int(r.headers.get('content-length'))
for chunk in tqdm(r.iter_content(chunk_size=1024), total=(total_length/1024) + 1):
if chunk:
f.write(chunk)
f.flush()
zip = zipfile.ZipFile(filename)
zip.extractall('.')
We transform the data a bit so that rivers appear thicker.
dataset = rasterio.open(adffile)
acc_orig = dataset.read()[0]
acc = np.where(acc_orig<0, 0, acc_orig)
shrink = 1 # if you are out of RAM try increasing this number (should be a power of 2)
radius = 5 # you can play with this number to change the width of the rivers
circle = np.zeros((2*radius+1, 2*radius+1)).astype('uint8')
y, x = np.ogrid[-radius:radius+1,-radius:radius+1]
index = x**2 + y**2 <= radius**2
circle[index] = 1
acc = np.sqrt(acc)
acc = scipy.ndimage.maximum_filter(acc, footprint=circle)
acc[acc_orig<0] = np.nan
acc = acc[::shrink, ::shrink]
The original data is in the WGS 84 projection, but Leaflet uses Web Mercator, so we need to reproject.
# At this point if GDAL complains about not being able to open EPSG support file gcs.csv, try in the terminal:
# export GDAL_DATA=`gdal-config --datadir`
with rasterio.Env():
rows, cols = acc.shape
src_transform = list(dataset.transform)
src_transform[0] *= shrink
src_transform[4] *= shrink
src_transform = Affine(*src_transform[:6])
src_crs = {'init': 'EPSG:4326'}
source = acc
dst_crs = {'init': 'EPSG:3857'}
dst_transform, width, height = rasterio.warp.calculate_default_transform(src_crs, dst_crs, cols, rows, *dataset.bounds)
dst_shape = height, width
destination = np.zeros(dst_shape)
reproject(
source,
destination,
src_transform=src_transform,
src_crs=src_crs,
dst_transform=dst_transform,
dst_crs=dst_crs,
resampling=Resampling.nearest)
acc_web = destination
Let's convert our NumPy array to an image. For that we must specify a colormap (here plt.cm.jet).
acc_norm = acc_web - np.nanmin(acc_web)
acc_norm = acc_norm / np.nanmax(acc_norm)
acc_norm = np.where(np.isfinite(acc_web), acc_norm, 0)
acc_im = PIL.Image.fromarray(np.uint8(plt.cm.jet(acc_norm)*255))
acc_mask = np.where(np.isfinite(acc_web), 255, 0)
mask = PIL.Image.fromarray(np.uint8(acc_mask), mode='L')
im = PIL.Image.new('RGBA', acc_norm.shape[::-1], color=None)
im.paste(acc_im, mask=mask)
The image is embedded in the URL as a PNG file, so that it can be sent to the browser.
if py3:
f = BytesIO()
else:
f = StringIO()
im.save(f, 'png')
data = b64encode(f.getvalue())
if py3:
data = data.decode('ascii')
imgurl = 'data:image/png;base64,' + data
Not quite as easy as matplotlib, but you get all the nice interactivity of ipyleaflet!
Finally we can overlay our image and if everything went fine it should be exactly over South America.
b = dataset.bounds
bounds = [(b.bottom, b.left), (b.top, b.right)]
io = ImageOverlay(url=imgurl, bounds=bounds)
center = [-10, -60]
zoom = 2
m = Map(center=center, zoom=zoom, interpolation='nearest')
m
tile = basemap_to_tiles(basemaps.Esri.WorldStreetMap)
m.add_layer(tile)
You can play with the opacity slider and check that rivers from our data file match the rivers on OpenStreetMap.
m.add_layer(io)
io.interact(opacity=(0.0,1.0,0.01))

Converting VTK UnstructuredGrid to StructuredGrid

The background to my problem is that I have a 3D structure saved in a .vtk file that I need to manipulate (dilate, erode, etc.). The following code snippets are designed to be run sequentially, i.e. if you run them one after the other, there should be no problems (apart from those I mention!).
I'm very new to VTK, so apologies for any very basic mistakes!
Problem
My problem stems from a problem with SimpleITK, wherein it is unable to read UnstructuredGrid or PolyData:
In [1]: import SimpleITK as sitk
In [2]: img_vtk = sitk.ReadImage(file_vtk)
Traceback (most recent call last):
File "<ipython-input-52-435ce999db50>", line 1, in <module>
img_vtk = sitk.ReadImage(file_vtk)
File "/usr/local/lib/python3.5/dist-packages/SimpleITK/SimpleITK.py", line 8614, in ReadImage
return _SimpleITK.ReadImage(*args)
RuntimeError: Exception thrown in SimpleITK ReadImage: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:97:
sitk::ERROR: Unable to determine ImageIO reader for "/data/ROMPA_MRIandSeg/09S/Analysis/1_model/clip_dilate.vtk"
SimpleITK can, however, read StructuredGrid, so I tried to solve this by reading using VTK and converting.
import vtk
reader = vtk.vtkGenericDataObjectReader() # Using generic to allow it to match either Unstructured or PolyData
reader.SetFileName(file_vtk)
reader.Update()
output = reader.GetOutput()
However, from that point on, every method I've tried seems to have failed.
Proposed Solutions
Conversion to numpy, then conversion to sitk image
I attempted to convert it to a numpy array (), then interpolate a regular grid, with a dummy variable of 1 to specify the values on the structure.
from vtk.utils import numpy_support
import scipy.interpolate
import numpy as np
nparray = numpy_support.vtk_to_numpy(output.GetPointData().GetArray(0))
output_bounds = output.GetBounds()
x_grid = range(math.floor(output_bounds[0]),math.ceil(output_bounds[1]),1)
y_grid = range(math.floor(output_bounds[2]),math.ceil(output_bounds[3]),1)
z_grid = range(math.floor(output_bounds[4]),math.ceil(output_bounds[5]),1)
grid = list()
for x in x_grid:
for y in y_grid:
for z in z_grid:
grid.append((x,y,z))
dummy = np.array([1 for i in range(nparray.shape[0])])
npgrid = scipy.interpolate.griddata(nparray,dummy,grid,fill_value=0)
npgrid.reshape(len(x_grid),len(y_grid),len(z_grid))
img = sitk.GetImageFromArray(npgrid)
sitk.WriteImage(img,file_out)
However, when I load this in ParaView, a bounding box is displayed for the output, but a contour of the output is empty.
Using ShepardMethod
I attempted to interpolate using the built-in ShepardMethod, after converting the UnstructuredGrid to PolyData (as I'd mostly seen ShepardMethod being applied to PolyData):
bounds = output.GetBounds()
spacings = [1.0,1.0,1.0] # arbitrary spacing
dimensions = [0,0,0]
for i,spacing in enumerate(spacings):
dimensions[i] = int(math.ceil((bounds[i*2 + 1]-bounds[i*2])/spacing))
vtkPoints = vtk.vtkPoints()
for i in range(0,nparray.shape[0]):
x=nparray[i,0]
y=nparray[i,1]
z=nparray[i,2]
p=[x,y,z]
vtkPoints.InsertNextPoint(p)
poly = vtk.vtkPolyData()
poly.SetPoints(vtkPoints)
shepard = vtk.vtkShepardMethod()
shepard.SetInputData(poly)
shepard.SetSampleDimensions(dimensions)
shepard.SetModelBounds(output.GetBounds())
shepard.Update()
shepard_data = shepard.GetOutput().GetPointData().GetArray(0)
shepard_numpy = numpy_support.vtk_to_numpy(shepard_data)
shepard_numpy = shepard_numpy.reshape(dimensions[0],dimensions[1],dimensions[2])
shepard_img = sitk.GetImageFromArray(shepard_numpy)
sitk.WriteImage(shepard_img,file_out)
As with the numpy effort above, this provided a bounding box in ParaView. Applying a contour provided a structure of two triangles, i.e. next to nothing seems to have been successfully written. Alternatively, I attempted to write the output directly using VTK.
shepard_data = shepard.GetOutput()
shepard_grid = vtk.vtkImageToStructuredGrid()
shepard_grid.SetInputData(shepard_data)
shepard_grid.Update()
writer = vtk.vtkStructuredGridWriter()
writer.SetFileName(file_out)
writer.SetInputData(shepard_grid.GetOutput())
writer.Write()
This produced the same output as before.
Using ProbeFilter
I tried the above using ProbeFilter instead (with both conversion to numpy and writing directly). Unfortunately, the output was the same as above.
mesh = vtk.vtkStructuredGrid()
mesh.SetDimensions(dimensions)
probe = vtk.vtkProbeFilter()
probe.SetInputData(mesh)
probe.SetSourceData(output)
probe.Update()
probe_out = probe.GetOutput()
writer = vtk.vtkStructuredGridWriter()
writer.SetFileName(file_out)
writer.SetInputData(probe.GetOutput())
writer.Write()
probe_data = probe.GetOutput().GetPointData().GetArray(0)
probe_numpy = numpy_support.vtk_to_numpy(probe_data)
probe_numpy = probe_numpy.reshape(dimensions[0],dimensions[1],dimensions[2])
probe_img = sitk.GetImageFromArray(probe_numpy)
sitk.WriteImage(probe_img,file_out)
However, this seemed to produce no viable output (vtkStructuredGridWriter produced an empty file, and probe_numpy was empty).
Changing ParaView output
My original data comes from a structuredGrid .vtk file, that I open using ParaView, and then clip to remove structures that aren't required in the mesh. Saving the output saves an unstructuredGrid, and I have been unable to figure out whether I can change that, and avoid this mess in the first place!

Just use "Resample With Dataset" filter in ParaView.
Open ParaView
Open a StructuredGrid file file with the geometry you want it to have
Open your UnstructuredGrid file
Add a "Resample with dataset" filter
Select structured data as source input
Apply

(Python) Rtree intersection and fiona questions

I'm trying to create a function for an intersection where the input file is of some urban area, and a query box is used to create an output file that has the intersection containing just the buildings found in that query box.
import matplotlib.pyplot as plt
import matplotlib as mpl
from mpl_toolkits.basemap import Basemap
import fiona
import fiona.crs
import rtree
input_file = 'se_england_clean.shp'
out_file = 'se_england_out'
file_index = 'Rtree_index_east.idx'
query_box = [-0.0957870483,51.5134165224,-0.08664608,51.5192383994]
def write_clipped_file(name_file_in, out_file, file_index):
idx = rtree.index.Index(file_index)
idx.insert(0, (input_file))
list(idx.intersection((query_box)))[0]
count = 0
with fiona.open(input_file, 'w') as out_file : #?
for building in out_file: #?
No idea if my code is right so far, but I have two immediate problems:
First, I don't know how to open with Fiona the input shapefile and the new (clipped) shapefile that I want to produce in output. I want to cycle the list of indices, select the desired buildings, and write them in the new file 'out_file'. Second, I get an error:
RTreeError: Coordinates must be in the form (minx, miny, maxx, maxy)

idx.insert(0, (input_file))
You need to insert coordinates into the tree, not a file name.