Develop Reference

Merging GOES17, EUMETSAT, GK-2A Meteorological satellite image

I want to generate global weather satellite image using GOES17, EUMETSAT, and GK-2A.
I want make it Plate carree coordinate. (GOES 17 netcdf file convert to Plate Carree)
First, using Satpy, I made plate carree image.
from satpy import Scene
from glob import glob
from pyresample import create_area_def
area_def = create_area_def("my_area_def", "+proj=eqc +datum=WGS84", resolution=2000)
goes17 = glob('./samplefile/*')
goes17_scene = Scene(reader="abi_l1b", filenames=goes17)
goes17_scene.load('[C13]')
new_scn = goes17_scene.resample(area_def)
# save to geotiffs
new_scn.save_datasets()
like this method, I want make other satellite image and merging to 1 image file. but is there any simple or easiest way to generate global weather image? My final goal is generate numpy array of global satellite image.
-- My entire code --
from satpy import Scene, MultiScene
from glob import glob
from pyresample import create_area_def
area_def = create_area_def("my_area_def", "+proj=eqc +datum=WGS84", resolution=2000,)
goes17 = glob('E:/Global/GOES_17/OR_ABI-L1b-RadF-M6C13_G17_s20212130000319_e20212130009396_c20212130009445.nc')
goes17_scene = Scene(reader="abi_l1b", filenames=goes17)
goes17_scene.load(['C13'])
gk2a = glob('E:/Global/GK-2A/gk2a_ami_le1b_ir105_fd020ge_202108010000.nc')
gk2a_scene = Scene(reader="ami_l1b", filenames=gk2a)
gk2a_scene.load(['IR105'])
eumetsat = glob('E:/Global/EUMETSAT/MSG4-SEVI-MSG15-0100-NA-20210801000010.306000000Z-20210801001259-4774254.nat')
eumetsat_scene = Scene(reader='seviri_l1b_native', filenames=eumetsat)
eumetsat_scene.load(['IR_108'])
from satpy import MultiScene, DataQuery
mscn = MultiScene([goes17_scene, gk2a_scene, eumetsat_scene])
groups = {DataQuery(name='IR_group', wavelength=(10.35, 10.35, 10.8)): ['C13', 'IR105', 'IR_108']}
mscn.group(groups)
from pyresample.geometry import AreaDefinition
resampled = mscn.resample(area_def, reduce_data=False)
resampled.load(['IR_group'])
blended = resampled.blend()
blended.show(['IR_group'])

There are some ways to do this inside Satpy, but typically people have specific ways they want the data joined together. That is a question you'll have to answer before you choose the code you want. First though you need to make a Scene for each separate satellite image you want in the final image and resample them to the same grid. A DynamicAreaDefinition (as you're using now) is not good for this overall process as each resampled Scene would be on a different final area (based on the satellite data being resampled that "froze" the DynamicAreaDefinition).
Your options for merging:
Satpy has a BackgroundCompositor where you can put one image on top of another. There is some documentation for creating a custom composite where you could make a composite like this. A series of these composites could be chained together to get the overall global composite you are looking for. You can put all the datasets in the same Scene to make things easier:
scn = Scene()
scn["C13"] = resampled_goes17_scene["C13"]
... and so on for the other sensors ...
scn.load(["custom_composite"])
Use the Satpy MultiScene, give it all of your resampled Scenes and run the "blend" method to join the images together. https://satpy.readthedocs.io/en/stable/multiscene.html#blending-scenes-in-multiscene
Use xarray and dask .where functions along with a custom mask array to say where each image should appear in the overall image. Some people do this type of thing with solar zenith angle have a nice blend between the images rather than just overlaying one on top of the other.
Create individual geotiffs for each resampled sensor and use GDAL's gdal_merge.py utility to join them into one geotiff.

Resample image rasterio/gdal, Python

How can I resample a single band GeoTIFF using Bilinear interpolation?
import os
import rasterio
from rasterio.enums import Resampling
from rasterio.plot import show,show_hist
import numpy as np
if __name__ == "__main__":
input_Dir = 'sample.tif'
#src = rasterio.open(input_Dir)
#show(src,cmap="magma")
upscale_factor = 2
with rasterio.open(input_Dir) as dataset:
# resample data to target shape
data = dataset.read(
out_shape=(
dataset.count,
int(dataset.height * upscale_factor),
int(dataset.width * upscale_factor)
),
resampling=Resampling.bilinear
)
# scale image transform
transform = dataset.transform * dataset.transform.scale(
(dataset.width / data.shape[-1]),
(dataset.height / data.shape[-2])
)
show(dataset,cmap="magma",transform=transform)
I have tried the following code and my output is as follows:
I am trying to achieve the following output:

One option would be to use the GDAL python bindings. Then you can perform the resample in memory (or you can save the image if you want). Assuming the old raster resolution was 0.25x0.25 and you're resampling to 0.10x0.10:
from osgeo import gdal
input_Dir = 'sample.tif'
ds = gdal.Translate('', input_Dir, xres=0.1, yres=0.1, resampleAlg="bilinear", format='vrt')
If you want to save the image put output filepath instead of the empty string for the first argument and change the format to 'tif'!

How to represent scalar variables over geographic map in Jupyter Notebook

I am representing some geographical data in a Jupyter notebook: temperature, ocean wave height, etc. I have numpy arrays that have the latitude, longitude, and value for those variables. I would like to display these variables over a geographical map, preferably using ipyleaflet (because that is what I am already using). I am trying to get a result similar to a heatmap.
I tried to use the ipyleaflet Heatmap, but it seems to me that it is designed to represent agregation of points and not scalar uniform arrays, because I can't get it to show the results properly. I think ipyleaflet may lack a function to represent this kind of data, but seems odd since it has a very nice Velocity funtion to represent vectorial variables.
The only way I can think of to make this would be to generate an image with matplotlib and then adding it to the map in an image layer, but I feel like that is not the proper way to do it.

For representing a heatmap I would recommend to use Cartopy in combination with Matplotlib.
Here a ready to use script I made for a world projection with a coastline:
import numpy as np
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.feature as cfeature
from cartopy.util import add_cyclic_point
# Set x y z variables
x = longitude_data
y = latitude_data
z = heat_map_data
# Set up figure and projection
z, x = add_cyclic_point(z, coord=x)
fig = plt.figure()
ax = fig.add_subplot(1,1,1, projection=ccrs.PlateCarree() )
# Set data range and colourmap
levels = np.arange(min,max,steps)
plt.contourf(x, y, z,levels = levels,transform=ccrs.PlateCarree(),cmap="rainbow")
# Set axes, extent (world) and labels
ax.set_xticks(np.linspace(-180,180,num=7), crs=ccrs.PlateCarree())
ax.set_yticks(np.linspace(-60,60,num=5), crs=ccrs.PlateCarree())
ax.add_feature(cfeature.COASTLINE) #Add coastline
ax.set_global()
ax.set_title('Heatmap')
ax.set_xlabel('Longitude')
ax.set_ylabel('Latitude')
# Add colorbar
plt.colorbar(ax=ax,shrink=0.7,orientation="vertical")
fig.show()
With the Cartopy and Matplotlib documentation you should now be able to create some maps.

As you say, you can generate an image and overlay it onto the map. This is the suggestion I was given when I asked about this on Github.
There's an example notebook here.
Not quite as easy as matplotlib, but you get all the nice interactivity of ipyleaflet!
Here's the current version of the notebook in case the link changes (converted to markdown via jupyter-nbconvert --to markdown Numpy.ipynb)
From NumPy to Leaflet
This notebook shows how to display some raster geographic data in IPyLeaflet. The data is a NumPy array, which means that you have all the power of the Python scientific stack at your disposal to process it.
The following libraries are needed:
* requests
* tqdm
* rasterio
* numpy
* scipy
* pillow
* matplotlib
* ipyleaflet
The recommended way is to try to conda install them first, and if they are not found then pip install.
import requests
import os
from tqdm import tqdm
import zipfile
import rasterio
from affine import Affine
import numpy as np
import scipy.ndimage
from rasterio.warp import reproject, Resampling
import PIL
import matplotlib.pyplot as plt
from base64 import b64encode
try:
from StringIO import StringIO
py3 = False
except ImportError:
from io import StringIO, BytesIO
py3 = True
from ipyleaflet import Map, ImageOverlay, basemap_to_tiles, basemaps
Download a raster file representing the flow accumulation for South America. This gives an idea of the river network.
url = 'https://edcintl.cr.usgs.gov/downloads/sciweb1/shared/hydrosheds/sa_30s_zip_grid/sa_acc_30s_grid.zip'
filename = os.path.basename(url)
name = filename[:filename.find('_grid')]
adffile = name + '/' + name + '/w001001.adf'
if not os.path.exists(adffile):
r = requests.get(url, stream=True)
with open(filename, 'wb') as f:
total_length = int(r.headers.get('content-length'))
for chunk in tqdm(r.iter_content(chunk_size=1024), total=(total_length/1024) + 1):
if chunk:
f.write(chunk)
f.flush()
zip = zipfile.ZipFile(filename)
zip.extractall('.')
We transform the data a bit so that rivers appear thicker.
dataset = rasterio.open(adffile)
acc_orig = dataset.read()[0]
acc = np.where(acc_orig<0, 0, acc_orig)
shrink = 1 # if you are out of RAM try increasing this number (should be a power of 2)
radius = 5 # you can play with this number to change the width of the rivers
circle = np.zeros((2*radius+1, 2*radius+1)).astype('uint8')
y, x = np.ogrid[-radius:radius+1,-radius:radius+1]
index = x**2 + y**2 <= radius**2
circle[index] = 1
acc = np.sqrt(acc)
acc = scipy.ndimage.maximum_filter(acc, footprint=circle)
acc[acc_orig<0] = np.nan
acc = acc[::shrink, ::shrink]
The original data is in the WGS 84 projection, but Leaflet uses Web Mercator, so we need to reproject.
# At this point if GDAL complains about not being able to open EPSG support file gcs.csv, try in the terminal:
# export GDAL_DATA=`gdal-config --datadir`
with rasterio.Env():
rows, cols = acc.shape
src_transform = list(dataset.transform)
src_transform[0] *= shrink
src_transform[4] *= shrink
src_transform = Affine(*src_transform[:6])
src_crs = {'init': 'EPSG:4326'}
source = acc
dst_crs = {'init': 'EPSG:3857'}
dst_transform, width, height = rasterio.warp.calculate_default_transform(src_crs, dst_crs, cols, rows, *dataset.bounds)
dst_shape = height, width
destination = np.zeros(dst_shape)
reproject(
source,
destination,
src_transform=src_transform,
src_crs=src_crs,
dst_transform=dst_transform,
dst_crs=dst_crs,
resampling=Resampling.nearest)
acc_web = destination
Let's convert our NumPy array to an image. For that we must specify a colormap (here plt.cm.jet).
acc_norm = acc_web - np.nanmin(acc_web)
acc_norm = acc_norm / np.nanmax(acc_norm)
acc_norm = np.where(np.isfinite(acc_web), acc_norm, 0)
acc_im = PIL.Image.fromarray(np.uint8(plt.cm.jet(acc_norm)*255))
acc_mask = np.where(np.isfinite(acc_web), 255, 0)
mask = PIL.Image.fromarray(np.uint8(acc_mask), mode='L')
im = PIL.Image.new('RGBA', acc_norm.shape[::-1], color=None)
im.paste(acc_im, mask=mask)
The image is embedded in the URL as a PNG file, so that it can be sent to the browser.
if py3:
f = BytesIO()
else:
f = StringIO()
im.save(f, 'png')
data = b64encode(f.getvalue())
if py3:
data = data.decode('ascii')
imgurl = 'data:image/png;base64,' + data
Not quite as easy as matplotlib, but you get all the nice interactivity of ipyleaflet!
Finally we can overlay our image and if everything went fine it should be exactly over South America.
b = dataset.bounds
bounds = [(b.bottom, b.left), (b.top, b.right)]
io = ImageOverlay(url=imgurl, bounds=bounds)
center = [-10, -60]
zoom = 2
m = Map(center=center, zoom=zoom, interpolation='nearest')
m
tile = basemap_to_tiles(basemaps.Esri.WorldStreetMap)
m.add_layer(tile)
You can play with the opacity slider and check that rivers from our data file match the rivers on OpenStreetMap.
m.add_layer(io)
io.interact(opacity=(0.0,1.0,0.01))

How to georeference an unreferenced aerial image using ground control points in python

I have a series of unreferenced aerial images that I would like to georeference using python. The images are identical spatially (they are actually frames extracted from a video), and I obtained ground control points for them by manually georeferencing one frame in ArcMap. I would like to apply the ground control points I obtained to all the subsequent images, and as a result obtain a geo-tiff or a jpeg file with a corresponding world file (.jgw) for each processed image. I know this is possible to do using arcpy, but I do not have access to arcpy, and would really like to use a free open source module if possible.
My coordinate system is NZGD2000 (epsg 2193), and here is the table of control points I wish to apply to my images:
176.412984, -310.977264, 1681255.524654, 6120217.357425
160.386905, -141.487145, 1681158.424227, 6120406.821253
433.204947, -310.547238, 1681556.948690, 6120335.658359
Here is an example image: https://imgur.com/a/9ThHtOz
I've read a lot of information on GDAL and rasterio, but I don't have any experience with them, and am failing to adapt bits of code I found to my particular situation.
Rasterio attempt:
import cv2
from rasterio.warp import reproject
from rasterio.control import GroundControlPoint
from fiona.crs import from_epsg
img = cv2.imread("Example_image.jpg")
# Creating ground control points (not sure if I got the order of variables right):
points = [(GroundControlPoint(176.412984, -310.977264, 1681255.524654, 6120217.357425)),
(GroundControlPoint(160.386905, -141.487145, 1681158.424227, 6120406.821253)),
(GroundControlPoint(433.204947, -310.547238, 1681556.948690, 6120335.658359))]
# The function requires a parameter "destination", but I'm not sure what to put there.
# I'm guessing this may not be the right function to use
reproject(img, destination, src_transform=None, gcps=points, src_crs=from_epsg(2193),
src_nodata=None, dst_transform=None, dst_crs=from_epsg(2193), dst_nodata=None,
src_alpha=0, dst_alpha=0, init_dest_nodata=True, warp_mem_limit=0)
GDAL attempt:
from osgeo import gdal
import osr
inputImage = "Example_image.jpg"
outputImage = "image_gdal.jpg"
dataset = gdal.Open(inputImage)
I = dataset.ReadAsArray(0,0,dataset.RasterXSize,dataset.RasterYSize)
outdataset = gdal.GetDriverByName('GTiff')
output_SRS = osr.SpatialReference()
output_SRS.ImportFromEPSG(2193)
outdataset = outdataset.Create(outputImage,dataset.RasterXSize,dataset.RasterYSize,I.shape[0])
for nb_band in range(I.shape[0]):
outdataset.GetRasterBand(nb_band+1).WriteArray(I[nb_band,:,:])
# Creating ground control points (not sure if I got the order of variables right):
gcp_list = []
gcp_list.append(gdal.GCP(176.412984, -310.977264, 1681255.524654, 6120217.357425))
gcp_list.append(gdal.GCP(160.386905, -141.487145, 1681158.424227, 6120406.821253))
gcp_list.append(gdal.GCP(433.204947, -310.547238, 1681556.948690, 6120335.658359))
outdataset.SetProjection(srs.ExportToWkt())
wkt = outdataset.GetProjection()
outdataset.SetGCPs(gcp_list,wkt)
outdataset = None
I don't quite know how to make the above code work, and I would really appreciate any help with this.

I ended up reading a book "Geoprocessing with Python" and finally found a solution that worked for me. Here is the code I adapted to my problem:
import shutil
from osgeo import gdal, osr
orig_fn = 'image.tif'
output_fn = 'output.tif'
# Create a copy of the original file and save it as the output filename:
shutil.copy(orig_fn, output_fn)
# Open the output file for writing for writing:
ds = gdal.Open(output_fn, gdal.GA_Update)
# Set spatial reference:
sr = osr.SpatialReference()
sr.ImportFromEPSG(2193) #2193 refers to the NZTM2000, but can use any desired projection
# Enter the GCPs
# Format: [map x-coordinate(longitude)], [map y-coordinate (latitude)], [elevation],
# [image column index(x)], [image row index (y)]
gcps = [gdal.GCP(1681255.524654, 6120217.357425, 0, 176.412984, 310.977264),
gdal.GCP(1681158.424227, 6120406.821253, 0, 160.386905, 141.487145),
gdal.GCP(1681556.948690, 6120335.658359, 0, 433.204947, 310.547238)]
# Apply the GCPs to the open output file:
ds.SetGCPs(gcps, sr.ExportToWkt())
# Close the output file in order to be able to work with it in other programs:
ds = None

For your gdal method, just using gdal.Warp with the outdataset should work, e.g.
outdataset.SetProjection(srs.ExportToWkt())
wkt = outdataset.GetProjection()
outdataset.SetGCPs(gcp_list,wkt)
gdal.Warp("output_name.tif", outdataset, dstSRS='EPSG:2193', format='gtiff')
This will create a new file, output_name.tif.

As an addition to #Kat's answer, to avoid quality loss of the original image file and set the nodata-value to 0, the following can be used.
#Load the original file
src_ds = gdal.Open(orig_fn)
#Create tmp dataset saved in memory
driver = gdal.GetDriverByName('MEM')
tmp_ds = driver.CreateCopy('', src_ds, strict=0)
#
# ... setting GCP....
#
# Setting no data for all bands
for i in range(1, tmp_ds.RasterCount + 1):
f = tmp_ds.GetRasterBand(i).SetNoDataValue(0)
# Saving as file
driver = gdal.GetDriverByName('GTiff')
ds = driver.CreateCopy(output_fn, tmp_ds, strict=0)

Splicing image array (FITS file) using coordinates from header

I am trying to splice a fits array based on the latitudes provided from the Header. However, I cannot seem to do so with my knowledge of Python and the documentation of astropy. The code I have is something like this:
from astropy.io import fits
import numpy as np
Wise1 = fits.open('Image1.fits')
im1 = Wise1[0].data
im1 = np.where(im1 > *latitude1, 0, im1)
newhdu = fits.PrimaryHDU(im1)
newhdulist = fits.HDUList([newhdu])
newhdulist.writeto('1b1_Bg_Removed_2.fits')
Here latitude1 would be a value in degrees, recognized after being called from the header. So there are two things I need to accomplish:
How to call the header to recognize Galactic Latitudes?
Splice the array in such a way that it only contains values for the range of latitudes, with everything else being 0.

I think by "splice" you mean "cut out" or "crop", based on the example you've shown.
astropy.nddata has a routine for world-coordinate-system-based (i.e., lat/lon or ra/dec) cutouts
However, in the simple case you're dealing with, you just need the coordinates of each pixel. Do this by making a WCS:
from astropy import wcs
w = wcs.WCS(Wise1[0].header)
xx,yy = np.indices(im.shape)
lon,lat = w.wcs_pix2world(xx,yy,0)
newim = im[lat > my_lowest_latitude]
But if you want to preserve the header information, you're much better off using the cutout tool, since you then do not have to manually manage this.
from astropy.nddata import Cutout2D
from astropy import coordinates
from astropy import units as u
# example coordinate - you'll have to figure one out that's in your map
center = coordinates.SkyCoord(mylon*u.deg, mylat*u.deg, frame='fk5')
# then make an array cutout
co = nddata.Cutout2D(im, center, size=[0.1,0.2]*u.arcmin, wcs=w)
# create a new FITS HDU
hdu = fits.PrimaryHDU(data=co.data, header=co.wcs.to_header())
# write to disk
hdu.writeto('cropped_file.fits')
An example use case is in the astropy documentation.