Develop Reference

Problems with NetCDF regridding, python

Hello, everyone, this is my first question here ^^
I need to regrid some NetCDF files, so I've tried to do it with cdo and cf modules, but everything failed. Maybe someone knows some other modules for this purposes or can help me to get rid of my errors.
First, I've tried cf:
df = cf.read('my_file.nc')[0]
#define new grid
lat1 = cf.DimensionCoordinate(data=cf.Data(np.arange(-90, 90.01, 0.1), 'degreesN'))
lon1 = cf.DimensionCoordinate(data=cf.Data(np.arange(-180, 180.01, 0.1), 'degreesE'))
#regrid
g = df.regrids({'lat': lat1, 'lon': lon1}, method='linear')
But it gave me: "RuntimeError: Regridding methods will not work unless the ESMF library is installed". And I've got no idea how to install it to the remote server where I'm working on.
Then I've tried cdo:
#define input and output files
infile='my_file.nc'
outfile='newgrid.nc'
#file with needed grid parameters
gridfile='gridfile.txt'
#bilinear regrid
cdo.remapbil('gridfile.txt', input=infile, output=outfile)
And it gave me:
"...
STDERR:Error (cdf_load_bounds) : Allocation of 134369280000 bytes failed. [ line 2048 file stream_cdf_i.c ]
System error message : Cannot allocate memory"
My file is about 2 gb, how it comes 134 gb to be needed for this operation?
I've also tried xarray:
da_input = xarray.open_dataarray('my_file.nc') # the file the data will be loaded from
regrid_axis = np.arange(-90, 90, 0.1) # new coordinates
da_output = da_input.interp(latitude=regrid_axis) # specify calculation
da_output.to_netcdf('output.nc') # save direct to file
It worked, but I suppose that it does interpolation just on the cartesian plane and I need it to be on sphere.
Thank everyone in advance
Upd:
Thank you for all your answers. Turned out that the problem was with my file (resolution too high 300x300m). So I splitted it to some smaller files (for this operation cdo.sellonlatbox also needed 134 gb, but xarray.sel did it), then regridded with cdo and merged the result.

You could try solving this with nctoolkit, which uses CDO as a backend:
import nctoolkit as nc
ds = nc.open_data("my_file.nc")
ds.to_latlon(lon = [-180, 180], lat = [-90, 90], res = 0.1)
ds.to_nc("output.nc")
If your file is 2 GB, then presumably you have made a mistake with the grid file in your CDO example. It shouldn't be trying to allocate that much RAM. nctoolkit will generate the appropriate CDO grid file and commands for you.

GeoTiff can't be produced properly with python

I have a problem with exporting a GeoTIFF file by using python and gdal.
What I want to do is to convert a NumPy array into a GeoTIFF file.
There are reference GeoTIFF files, so I want to make sure that the produced GeoTIFF file has proper geometric coordinates.
The problem is that the tiff file seems to be produced, but the values it contains are not good.
I tried to view the file by using QGIS, but the appearance of the image was completely black. In addition to the problem with the appearance of the image, the values are also changed from the original NumPy array. For example, the maximum value of the NumPy array is 149, but QGIS says that there is no such value in the file.
What is the cause of this problem, and how can I fix it?
The codes are here.
#Check the metadata of the reference file.
with rio.open('/content/drive/My Drive/Colab Notebooks/satellite_data/MCD12Q1/MCD12Q1.A2019001.IGBP.Buryat.geotiff.tif') as filename : filename.bounds
filename.meta["transform"]
#Output=> {'count': 1,'crs': CRS.from_epsg(4326),'driver': 'GTiff','dtype': 'uint8','height': 1920, 'nodata': None, 'transform': Affine(0.00416666, 0.0, 97.99955519999997, 0.0, -0.00416666, 58.0000512),
'width': 4800}
srs = osr.SpatialReference()
srs.ImportFromEPSG(4326)
#Set some parameters
xsize=filename.meta["width"]
ysize=filename.meta["height"]
band=1
dtype = gdal.GDT_UInt16
output = gdal.GetDriverByName('GTiff').Create('/content/drive/My Drive/Colab Notebooks/outputs/output.tif', xsize, ysize, band, dtype)
output.SetGeoTransform((filename.meta["transform"][0],filename.meta["transform"][1],filename.meta["transform"][2],filename.meta["transform"][3],filename.meta["transform"][4],filename.meta["transform"][5]))
srs = osr.SpatialReference()
srs.ImportFromEPSG(4326)
output.SetProjection(srs.ExportToWkt())
output.GetRasterBand(1).WriteArray(NumOfFiresExpanded["2019"]) #The "NumOfFireExpanded["2019"] is the target numpy array.
output.FlushCache()
output = None

I have an example here which takes numpy arrays and builds a GeoTiff.
How do I write/create a GeoTIFF RGB image file in python?
Use gdalinfo -stats <geotiff> to view the GeoTiff and make sure the range is good. Viewing 16-bit imagery can be non-deterministic depending on the viewer. QGIS is a good test viewer.

How to transform .grib file into a GeoTIFF with correct projection using Python, GDAL, ArcPy

I am trying to transform a .grib file into a GeoTIFF to be used in a GIS (ArcGIS to be particular), but am having trouble getting the image to project properly. I have been able to create a GeoTIFF, using GDAL in Python, that shows the data but is not showing up in the correct location when brought into ArcGIS. The resulting image is below.
The data I am working with can be downloaded from: https://gimms.gsfc.nasa.gov/SMOS/SMAP/L05/
I am trying to project the data into WGS84 Web Mercator (Auxiliary Sphere), EPSG: 3857
Note: I have tried bringing in the data via ArcMap by creating a Raster Mosaic which should be able to work with .grib data, but I didn't have any luck.
Update: I have also tried using the Project Raster tool, but ArcGIS does not like the default projection that comes from the .grib file and gives an error.
The code I'm using:
import gdal
src_filename = r"C:\att\project\arcshare\public\disaster_response\nrt_products\smap\20150402_20150404_anom1.grib"
dst_filename = r"C:\att\project\arcshare\public\disaster_response\nrt_products\smap\smap_py_test1.tif"
#Open existing dataset
src_ds = gdal.Open(src_filename)
#Open output format driver, see gdal_translate --formats for list
format = "GTiff"
driver = gdal.GetDriverByName( format )
#Output to new format
dst_file = driver.CreateCopy( dst_filename, src_ds, 0 )
#Properly close the datasets to flush to disk
dst_ds = None
src_ds = None
I am not very well versed in using GDAL or GDAL in Python, so any help or tips would be greatly appreciated.

Try using gdal.Translate (in Python) or gdal_translate (from command line). Here are two examples of how I have used each approach in the past:
Option 1: Python approach
from osgeo import gdal
# Open existing dataset
src_ds = gdal.Open(src_filename)
# Ensure number of bands in GeoTiff will be same as in GRIB file.
bands = [] # Set up array for gdal.Translate().
if src_ds is not None:
bandNum = src_ds.RasterCount # Get band count
for i in range(bandNum+1): # Update array based on band count
if (i==0): #gdal starts band counts at 1, not 0 like the Python for loop does.
pass
else:
bands.append(i)
# Open output format driver
out_form= "GTiff"
# Output to new format using gdal.Translate. See https://gdal.org/python/ for osgeo.gdal.Translate options.
dst_ds = gdal.Translate(dst_filename, src_ds, format=out_form, bandList=bands)
# Properly close the datasets to flush to disk
dst_ds = None
src_ds = None
Option 2: Command line gdal_translate (called from Python) approach
import os
# Open output format driver, see gdal_translate --formats for list
out_form = "GTiff"
# Pull out specific band of interest
band=3
# Convert from GRIB to GeoTIFF using system gdal_translate
src_ds = src_filename
dst_ds = dst_filename
os.system("gdal_translate -b %s -of %s %s %s" %(str(band), out_form, src_ds, dst_ds))
I've had trouble in the past creating a multi-band GeoTiff using option 2, so I recommend using option 1 when possible.

Something like this should transform your native coordinates into your desired projection. This is not tested, yet. (Could by latitude instead of latitudes).
from cfgrib import xarray_store
from pyproj import Proj, transform
grib_data = xarray_store.open_dataset('your_grib_file.grib')
lat = grib_data.latitudes.value
lon = grib_data.longitudes.value
lon_transformed, lat_transformed = transform (Proj(init='init_projection'),
Proj(init='target_projection', lon, lat)

python: perform gdalwarp in memory with gdal bindings

I currently have a processing chain in R which downloads MODIS data and then calls gdalwarp from the system to reproject a specific subdataset (e.g. NDVI) into WGS1984. The resulting GeoTiffs are then collected into an HDF5 file for further processing.
Now I'm moving the processing chain to python, and I was wondering if there's a way to skip the step of writing GeoTiffs to disk with the functionalities of the gdal module.
To be clear, the question is:
Can i perform gdalwarp with using strictly the python bindings from the gdal module and without writing to disk?
I've been researching a bit and the closest answers to my questions were these posts:
How to project and resample a grid to match another grid with GDAL python?
Replicating result of gdalwarp using gdal Python bindings
The first method requires a template, so not really what I'm looking for.
The second method looks more promising, it's using the function AutoCreateWarpedVRT which seems to be quite what I want. Although, in contrary to the example in the answer, my result doesn't match the reference (independently of any error threshold).
In my previous implementation which calls gdalwarp directly, I've specified a target resolution in addition to the target reference system. So I assume that's something that could make the difference - but I haven't been able to set it within the gdal bindings in python.
Here's what I tried (sorry, not reproducible without the MODIS data):
import gdal
import osr
ds = gdal.Open('/data/MOD13A2.A2016305.h18v07.005.2016322013359.hdf')
t_srs = osr.SpatialReference()
t_srs.ImportFromEPSG(4326)
src_ds = gdal.Open(ds.GetSubDatasets()[0][0], gdal.GA_ReadOnly)
dst_wkt =t_srs.ExportToWkt()
error_threshold = 0.125
resampling=gdal.GRA_NearestNeighbour
tmp_ds = gdal.AutoCreateWarpedVRT( src_ds,
None, # src_wkt : left to default value --> will use the one from source
dst_wkt,
resampling,
error_threshold)
# create tiff
dst_ds = gdal.GetDriverByName('GTiff').CreateCopy('warp_test.tif', tmp_ds)
dst_ds = None
And this is for the reference:
gdalwarp -ot Int16 -tr 0.00892857142857143 0.00892857142857143 -t_srs EPSG:4326 "HDF4_EOS:EOS_GRID:MOD13A2.A2016305.h18v07.005.2016322013359.hdf:MODIS_Grid_16DAY_1km_VI:1 km 16 days NDVI" MOD13A2.A2016305.h18v07.005.2016322013359_MODIS_Grid_16DAY_1km_VI_1_km_16_days_NDVI.tif
The comparison:
i1 = gdal.Open('warp_test.tif')
i2 = gdal.Open('MOD13A2.A2016305.h18v07.005.2016322013359_MODIS_Grid_16DAY_1km_VI_1_km_16_days_NDVI.tif')
# test
print(i1.RasterXSize,i1.RasterYSize)
1267 1191
#reference
print(i2.RasterXSize,i2.RasterYSize)
1192 1120
i1.GetRasterBand(1).Checksum() == i2.GetRasterBand(1).Checksum()
False
So you can see, using the gdal.AutoCreateWarpedVRT function results in a dataset with different dimensions and resolution.

If you want to mimic your "reference" call to gdalwarp you can use:
import gdal
ds = gdal.Warp('warp_test.tif', infile, dstSRS='EPSG:4326',
outputType=gdal.GDT_Int16, xRes=0.00892857142857143, yRes=0.00892857142857143)
ds = None
If you dont want to output to a file on disk, you can warp to an in-memory VRT file, for example:
ds = gdal.Warp('', infile, dstSRS='EPSG:4326', format='VRT',
outputType=gdal.GDT_Int16, xRes=0.00892857142857143, yRes=0.00892857142857143)
You can of course warp to any format in memory, but for files other than VRT the warped result will actually be stored in-memory.

Spatial reference missing from rasterio write to GeoTiff

I am trying to use rasterio to load in an image, modify the ndarray, then write out using the same spatial reference system as the original image. The below function is my attempt to do this. But the spatial reference system is missing from the output geotiff. Any suggestions on what I am doing wrong?
I have checked the input geotiff crs is valid ('epsg:32611').
# Function to write out an ndarry as a GeoTIFF using the spatial references of a sample geotif file
def write_GeoTif_like(templet_tif_file, output_ndarry, output_tif_file):
import rasterio
orig = rasterio.open(templet_tif_file)
with rasterio.open(output_tif_file, 'w', driver='GTiff', height=output_ndarry.shape[0],
width=output_ndarry.shape[1], count=1, dtype=output_ndarry.dtype,
crs=orig.crs, transform=orig.transform, nodata=-9999) as dst:
dst.write(output_ndarry, 1)

Having been bitten by this issue before, I'd guess that your GDAL_DATA environment variable is not being set correctly (see https://github.com/conda/conda/issues/4050 for more detail). Without knowing more about your installation/OS, I cannot say for sure, but if gdal (and rasterio) are unable to find the location with metadata files such as those that support operations involving coordinate references systems, you'll lose the CRS in the output tif.