Related
I have a mosaic tif file (gdalinfo below) I made (with some additional info on the tiles here) and have looked extensively for a function that simply returns the elevation (the z value of this mosaic) for a given lat/long. The functions I've seen want me to input the coordinates in the coordinates of the mosaic, but I want to use lat/long, is there something about GetGeoTransform() that I'm missing to achieve this?
This example for instance here shown below:
from osgeo import gdal
import affine
import numpy as np
def retrieve_pixel_value(geo_coord, data_source):
"""Return floating-point value that corresponds to given point."""
x, y = geo_coord[0], geo_coord[1]
forward_transform = \
affine.Affine.from_gdal(*data_source.GetGeoTransform())
reverse_transform = ~forward_transform
px, py = reverse_transform * (x, y)
px, py = int(px + 0.5), int(py + 0.5)
pixel_coord = px, py
data_array = np.array(data_source.GetRasterBand(1).ReadAsArray())
return data_array[pixel_coord[0]][pixel_coord[1]]
This gives me an out of bounds error as it's likely expecting x/y coordinates (e.g. retrieve_pixel_value([153.023499,-27.468968],dataset). I've also tried the following from here:
import rasterio
dat = rasterio.open(fname)
z = dat.read()[0]
def getval(lon, lat):
idx = dat.index(lon, lat, precision=1E-6)
return dat.xy(*idx), z[idx]
Is there a simple adjustment I can make so my function can query the mosaic in lat/long coords?
Much appreciated.
Driver: GTiff/GeoTIFF
Files: mosaic.tif
Size is 25000, 29460
Coordinate System is:
PROJCRS["GDA94 / MGA zone 56",
BASEGEOGCRS["GDA94",
DATUM["Geocentric Datum of Australia 1994",
ELLIPSOID["GRS 1980",6378137,298.257222101004,
LENGTHUNIT["metre",1]],
ID["EPSG",6283]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433,
ID["EPSG",9122]]]],
CONVERSION["UTM zone 56S",
METHOD["Transverse Mercator",
ID["EPSG",9807]],
PARAMETER["Latitude of natural origin",0,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8801]],
PARAMETER["Longitude of natural origin",153,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8802]],
PARAMETER["Scale factor at natural origin",0.9996,
SCALEUNIT["unity",1],
ID["EPSG",8805]],
PARAMETER["False easting",500000,
LENGTHUNIT["metre",1],
ID["EPSG",8806]],
PARAMETER["False northing",10000000,
LENGTHUNIT["metre",1],
ID["EPSG",8807]],
ID["EPSG",17056]],
CS[Cartesian,2],
AXIS["easting",east,
ORDER[1],
LENGTHUNIT["metre",1,
ID["EPSG",9001]]],
AXIS["northing",north,
ORDER[2],
LENGTHUNIT["metre",1,
ID["EPSG",9001]]]]
Data axis to CRS axis mapping: 1,2
Origin = (491000.000000000000000,6977000.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 491000.000, 6977000.000) (152d54'32.48"E, 27d19'48.33"S)
Lower Left ( 491000.000, 6947540.000) (152d54'31.69"E, 27d35'45.80"S)
Upper Right ( 516000.000, 6977000.000) (153d 9'42.27"E, 27d19'48.10"S)
Lower Right ( 516000.000, 6947540.000) (153d 9'43.66"E, 27d35'45.57"S)
Center ( 503500.000, 6962270.000) (153d 2' 7.52"E, 27d27'47.16"S)
Band 1 Block=25000x1 Type=Float32, ColorInterp=Gray
NoData Value=-999
Update 1 - I tried the following:
tif = r"mosaic.tif"
dataset = rio.open(tif)
d = dataset.read()[0]
def get_xy_coords(latlng):
transformer = Transformer.from_crs("epsg:4326", dataset.crs)
coords = [transformer.transform(x, y) for x,y in latlng][0]
#idx = dataset.index(coords[1], coords[0])
return coords #.xy(*idx), z[idx]
longx,laty = 153.023499,-27.468968
coords = get_elevation([(laty,longx)])
print(coords[0],coords[1])
print(dataset.width,dataset.height)
(502321.11181384244, 6961618.891167777)
25000 29460
So something is still not right. Maybe I need to subtract the coordinates from the bottom left/right of image e.g.
coords[0]-dataset.bounds.left,coords[1]-dataset.bounds.bottom
where
In [78]: dataset.bounds
Out[78]: BoundingBox(left=491000.0, bottom=6947540.0, right=516000.0, top=6977000.0)
Update 2 - Indeed, subtracting the corners of my box seems to get closer.. though I'm sure there is a much nice way just using the tif metadata to get what I want.
longx,laty = 152.94646, -27.463175
coords = get_xy_coords([(laty,longx)])
elevation = d[int(coords[1]-dataset.bounds.bottom),int(coords[0]-dataset.bounds.left)]
fig,ax = plt.subplots(figsize=(12,12))
ax.imshow(d,vmin=0,vmax=400,cmap='terrain',extent=[dataset.bounds.left,dataset.bounds.right,dataset.bounds.bottom,dataset.bounds.top])
ax.plot(coords[0],coords[1],'ko')
plt.show()
You basically have two distinct steps:
Convert lon/lat coordinates to map coordinates, this is only necessary if your input raster is not already in lon/lat. Map coordinates are the coordinates in the projection that the raster itself uses
Convert the map coordinates to pixel coordinates.
There are all kinds of tool you might use, perhaps to make things simpler (like pyproj, rasterio etc). But for such a simple case it's probably nice to start with doing it all in GDAL, that probably also enhances your understanding of what steps are needed.
Inputs
from osgeo import gdal, osr
raster_file = r'D:\somefile.tif'
lon = 153.023499
lat = -27.468968
lon/lat to map coordinates
# fetch metadata required for transformation
ds = gdal.OpenEx(raster_file)
raster_proj = ds.GetProjection()
gt = ds.GetGeoTransform()
ds = None # close file, could also keep it open till after reading
# coordinate transformation (lon/lat to map)
# define source projection
# this definition ensures the order is always lon/lat compared
# to EPSG:4326 for which it depends on the GDAL version (2 vs 3)
source_srs = osr.SpatialReference()
source_srs.ImportFromWkt(osr.GetUserInputAsWKT("urn:ogc:def:crs:OGC:1.3:CRS84"))
# define target projection based on the file
target_srs = osr.SpatialReference()
target_srs.ImportFromWkt(raster_proj)
# convert
ct = osr.CoordinateTransformation(source_srs, target_srs)
mapx, mapy, *_ = ct.TransformPoint(lon, lat)
You could verify this intermediate result by for example adding it as Point WKT in something like QGIS (using the QuickWKT plugin, making sure the viewer has the same projection as the raster).
map coordinates to pixel
# apply affine transformation to get pixel coordinates
gt_inv = gdal.InvGeoTransform(gt) # invert for map -> pixel
px, py = gdal.ApplyGeoTransform(gt_inv, mapx, mapy)
# it wil return fractional pixel coordinates, so convert to int
# before using them to read. Round to nearest with +0.5
py = int(py + 0.5)
px = int(px + 0.5)
# read pixel data
ds = gdal.OpenEx(raster_file) # open file again
elevation_value = ds.ReadAsArray(px, py, 1, 1)
ds = None
The elevation_value variable should be the value you're after. I would definitelly verify the result independently, try a few points in QGIS or the gdallocationinfo utility:
gdallocationinfo -l_srs "urn:ogc:def:crs:OGC:1.3:CRS84" filename.tif 153.023499 -27.468968
# Report:
# Location: (4228P,4840L)
# Band 1:
# Value: 1804.51879882812
If you're reading a lot of points, there will be some threshold at which it would be faster to read a large chunk and extract the values from that array, compared to reading every point individually.
edit:
For applying the same workflow on multiple points at once a few things change.
So for example having the inputs:
lats = np.array([-27.468968, -27.468968, -27.468968])
lons = np.array([153.023499, 153.023499, 153.023499])
The coordinate transformation needs to use ct.TransformPoints instead of ct.TransformPoint which also requires the coordinates to be stacked in a single array of shape [n_points, 2]:
coords = np.stack([lons.ravel(), lats.ravel()], axis=1)
mapx, mapy, *_ = np.asarray(ct.TransformPoints(coords)).T
# reshape in case of non-1D inputs
mapx = mapx.reshape(lons.shape)
mapy = mapy.reshape(lons.shape)
Converting from map to pixel coordinates changes because the GDAL method for this only takes single point. But manually doing this on the arrays would be:
px = gt_inv[0] + mapx * gt_inv[1] + mapy * gt_inv[2]
py = gt_inv[3] + mapx * gt_inv[4] + mapy * gt_inv[5]
And rounding the arrays to integer changes to:
px = (px + 0.5).astype(np.int32)
py = (py + 0.5).astype(np.int32)
If the raster (easily) fits in memory, reading all points would become:
ds = gdal.OpenEx(raster_file)
all_elevation_data = ds.ReadAsArray()
ds = None
elevation_values = all_elevation_data[py, px]
That last step could be optimized by checking highest/lowest pixel coordinates in both dimensions and only read that subset for example, but it would require normalizing the coordinates again to be valid for that subset.
The py and px arrays might also need to be clipped (eg np.clip) if the input coordinates fall outside the raster. In that case the pixel coordinates will be < 0 or >= xsize/ysize.
I have two arrays loaded with complex numbers that represent a position in a cartesian coordinate (x,y).
sensors= np.array([-1.6-0.8j,-1.1-0.8j])
cameras= np.array([-3.7-0.8j,-1.6+0.9j,-1.6-0.9j])
Where the real part represents X and the imaginary part represents Y. These numbers represent in meters. So 1.5-0.5j = 1.5 meters +X and 0.5 meters -Y.
Using the isclose function has issues when the position of the sensors gets further from 0.0.
def close_to_sensors(sensors, observations):
tolerance = 0.6
observe_indices = np.zeros(observations.size, dtype=bool)
for sensor in sensors:
closeness = np.isclose(observations, np.ones(observations.size, dtype=np.complex128)*sensor, rtol=tolerance, atol=tolerance)
observe_indices = np.logical_or(observe_indices, closeness)
print("Closeness : ", closeness)
return np.argwhere(observe_indices).flatten()
This returns
Closeness : [False False True]
Likely Close: [2]
The isclose function is the wrong function to use. I need to return the indices of the cameras that are within 1 meter of the sensors. What would be the best way to do this?
To calculate distances from complex numbers, subtracting them and calculating the absolute value of the difference is a straight-forward approach to solve this problem:
import numpy as np
sensors = np.array([-1.6 - 0.8j, -1.1 - 0.8j])
cameras = np.array([-3.7 - 0.8j, -1.6 + 0.9j, -1.6 - 0.9j])
distance_limit = 1
# calculate difference of each sensor to each camera
# "None" is used to create a new axis, which enables broadcasting to a (sensors x cameras) matrix
complex_differences = sensors[:, None] - cameras
axis_sensor, axis_camera = (0,1)
distances = np.abs(complex_differences)
# check cameras which have any sensor within distance limit
within_range = distances < distance_limit
valid_cameras = np.any(within_range, axis=axis_sensor)
# show indices of valid cameras
print(np.where(valid_cameras)[0])
Thank you all for your responses but those resulted in undesired results. I eventually decided to change the complex number arrays to [real, imag] lists, then load the sensors list to a KDTree and searched the tree for observations that were close; where 1 = 1 meter. This provided the results I needed.
EDIT: Added code with data
import numpy as np
import scipy.spatial as spatial
def close_to_sensors(bifrost_sensors, observations):
sensors_x_y = []
observations_x_y = []
for i in range(bifrost_sensors.size):
sensors_x_y.append((bifrost_sensors[i].real, bifrost_sensors[i].imag))
for i in range(observations.size):
observations_x_y.append((observations[i].real, observations[i].imag))
observe_indices = np.zeros(observations.size, dtype=bool)
#KDTree the sensor list
sensor_tree = spatial.cKDTree(np.c_[sensors_x_y])
for i in range(len(observations_x_y)):
closeness = (sensor_tree.data[sensor_tree.query_ball_point(observations_x_y[i], 1)])
if closeness.size == 0:
observe_indices[i] = np.logical_or(observe_indices[i], 0)
else:
observe_indices[i] = np.logical_or(observe_indices[i], 1)
#Find the indices of array elements that are non-zero, grouped by element.
return np.argwhere(observe_indices).flatten()
#Excel copied data into arrays - 12 entries
sensors = np.array([-0.6-0.8j,-0.8-1.2j,-0.9-1.2j,-1.-0.9j,-1.1-1.j,1.1+1.j,-1.5-1.5j,-1.6-1.1j,-1.7-1.5j,1.1+1.j,1.8+0.8j,-2.-1.6j])
cameras = np.array([-4.03-1.1j,-4.15-1.14j,-1.5-1.16j,-4.05-1.14j,-4.05-1.14j,4.03+2.19j,-4.08-1.13j,-4.06-1.14j,-1.15-0.98j,3.21+1.92j,3.9+1.65j,-4.08-1.13j])
likely_bifrost = close_to_sensors(sensors, cameras)
print("Likely bifrost : ", likely_bifrost.size, " : ",likely_bifrost)
The healpy.query_disc() function takes an argument vec which is a three-component unit vector defining the center of the disc. What coordinate system is being used here - why is there a third dimension for a 2-d projection? What point is the "tail" of this vector llocated at?
Very good you found the solution yourself, for later reference here is a full working code example:
import healpy as hp
import numpy as np
# `lonlat=True` switches `ang2vec` from requiring colatitude $\theta$ and longitude $\phi$ in radians to longitude and latitude in degrees (notice that also the order changes)
# in degrees
lon = 60
lat = 30
vec = hp.ang2vec(lon, lat, lonlat=True)
nside = 256
large_disc = hp.query_disc(nside, vec, radius=np.radians(20))
small_disc = hp.query_disc(nside, vec, radius=np.radians(8))
tiny_disc = hp.query_disc(nside, vec, radius=np.radians(2))
# `query_disc` returns a list of pixels, by default in RING ordering, let's check their length:
list(map(len, [large_disc, small_disc, tiny_disc]))
# ## Create a map and plot it in Mollweide projection
m = np.zeros(hp.nside2npix(nside))
m[large_disc] = 1
m[small_disc] = 2
m[tiny_disc] = 3
hp.mollview(m)
hp.graticule()
See the notebook with plots here: https://zonca.dev/2020/10/example-healpy-query_disc.html
Solved. Used the output of ang2vec() to give the vector.
I need to generate a Healpyx map (using Healpy) from random $a_{\ell m}$, for a spin-2 function.
Schematically, this should look like that:
import healpy as hp
nside = 16 # for example
for el in range(1, L+1): #loop over ell mode
for m in range(-el,el): #for each ell mode loop over m
ind = hp.sphtfunc.Alm.getidx(nside, el, m)
if m == 0:
a_lm[ind] = np.random.randn()
else:
a_lm[ind] = np.random.randn() + 1j * np.random.randn()
a_tmp = hp.sphtfunc.alm2map(a_lm, nside, pol=True)
My two questions are:
1) how do I initialise a_lm ? Specifically, what would be its dimension, using
a_lm = np.zeros(???)
2) if I understood correctly, the output a_tmp is a 1 dimensional list. How do I reshape it into a two-dimensional list (the map) for plotting?
1) What properties do you want your alm to have? You could also just assume a certain power spectrum (C_ell) and use hp.synalm() or hp.synfast().
For the initialization, you've already implemented that m goes from -ell to +ell, so you have a one-dimensional array of length sum_0^ell [2ell+1]. Doing the math should give you the length you need.
2) For the plotting, you could just directly generate a random map and then use e.g. hp.mollview(), which takes the 1-dimensional HEALPix map.
Alternatively, you can use hp.alm2map() to convert your alm to a map.
I also suggest you check out the tutorial for the plotting.
Usually we can follow the following steps to get the length of a_lm.
import healpy as hp
inside = 16
# Get the maximum multipole with the current nside
lmax = 3*nside - 1 #This can vary according to the use. In cosmology, the common value is 2*nside
alm_len = hp.Alm.getsize(lmax)
a_lm = np.empty(alm_len)
I think the tutorial linked in #Daniel's answer is a good resource for plotting Healpix maps.
I want to get a list of indices (row,col) for all raster cells that fall within or are intersected by a polygon feature. Looking for a solution in python, ideally with gdal/ogr modules.
Other posts have suggested rasterizing the polygon, but I would rather have direct access to the cell indices if possible.
Since you don't provide a working example, it's bit unclear what your starting point is. I made a dataset with 1 polygon, if you have a dataset with multiple but only want to target a specific polygon you can add SQLStatement or where to the gdal.Rasterize call.
Sample polygon
geojson = """{"type":"FeatureCollection",
"name":"test",
"crs":{"type":"name","properties":{"name":"urn:ogc:def:crs:OGC:1.3:CRS84"}},
"features":[
{"type":"Feature","properties":{},"geometry":{"type":"MultiPolygon","coordinates":[[[[-110.254,44.915],[-114.176,37.644],[-105.729,36.41],[-105.05,43.318],[-110.254,44.915]]]]}}
]}"""
Rasterizing
Rasterizing can be done with gdal.Rasterize. You need to specify the properties of the target grid. If there is no predefined grid these could be extracted from the polygon itself
ds = gdal.Rasterize('/vsimem/tmpfile', geojson, xRes=1, yRes=-1, allTouched=True,
outputBounds=[-120, 30, -100, 50], burnValues=1,
outputType=gdal.GDT_Byte)
mask = ds.ReadAsArray()
ds = None
gdal.Unlink('/vsimem/tmpfile')
Converting to indices
Retrieving the indices from the rasterized polygon can be done with Numpy:
y_ind, x_ind = np.where(mask==1)
Clearly Rutger's solution above is the way to go with this, however I will leave my solution up. I developed a script that accomplished what I needed with the following:
Get the bounding box for each vector feature I want to check
Use the bounding box to limit the computational window (determine what portion of the raster could potentially have intersections)
Iterate over the cells within this part of the raster and construct a polygon geometry for each cell
Use ogr.Geometry.Intersects() to check if the cell intersects with the polygon feature
Note that I have only defined the methods, but I think implementation should be pretty clear -- just call match_cells with the appropriate arguments (ogr.Geometry object and geotransform matrix). Code below:
from osgeo import ogr
# Convert projected coordinates to raster cell indices
def parse_coords(x,y,gt):
row,col = None,None
if x:
col = int((x - gt[0]) // gt[1])
# If only x coordinate is provided, return column index
if not y:
return col
if y:
row = int((y - gt[3]) // gt[5])
# If only x coordinate is provided, return column index
if not x:
return row
return (row,col)
# Construct polygon geometry from raster cell
def build_cell((row,col),gt):
xres,yres = gt[1],gt[5]
x_0,y_0 = gt[0],gt[3]
top = (yres*row) + y_0
bottom = (yres*(row+1)) + y_0
right = (xres*col) + x_0
left = (xres*(col+1)) + x_0
# Create ring topology
ring = ogr.Geometry(ogr.wkbLinearRing)
ring.AddPoint(left,bottom)
ring.AddPoint(right,bottom)
ring.AddPoint(right,top)
ring.AddPoint(left,top)
ring.AddPoint(left,bottom)
# Create polygon
box = ogr.Geometry(ogr.wkbPolygon)
box.AddGeometry(ring)
return box
# Iterate over feature geometries & check for intersection
def match_cells(inputGeometry,gt):
matched_cells = []
for f,feature in enumerate(inputGeometry):
geom = feature.GetGeometryRef()
bbox = geom.GetEnvelope()
xmin,xmax = [parse_coords(x,None,gt) for x in bbox[:2]]
ymin,ymax = [parse_coords(None,y,gt) for y in bbox[2:]]
for cell_row in range(ymax,ymin+1):
for cell_col in range(xmin,xmax+1):
cell_box = build_cell((cell_row,cell_col),gt)
if cell_box.Intersects(geom):
matched_cells += [[(cell_row,cell_col)]]
return matched_cells
if you want to do this manually you'll need to test each cell for:
Square v Polygon intersection and
Square v Line intersection.
If you treat each square as a 2d point this becomes easier - it's now a Point v Polygon problem. Check in Game Dev forums for collision algorithms.
Good luck!