Develop Reference

Comparing two differently projected datasets

I have an array that has 14000 columns and 7000 rows of terrain data for the US that are equally spaced 500m apart. I also have the lower-left latitude and longitude:
ncols = 14000
nrows = 7000
xllcorner = -130
yllcorner = 20
cellsize = 0.05
I also have another dataset (polar --> Cartesian radar data) that is already in a projected coordinate system:
# radial data being converted to Cartesian
x = rangee * np.sin(np.deg2rad(az))[:,None]
y = rangee * np.cos(np.deg2rad(az))[:,None]
latitude = 35.9339
longitude = -80.0212
dataproj = Proj(f"+proj=stere +lat_0={latitude} +lat_ts={latitude} +lon_0={longitude} +ellps=WGS84 +units=m")
lons,lats = dataproj(x,y,inverse=True)
It should be noted that the terrain data spans throughout the US, whereas the radar data is located over North Carolina. Therefore, I have two separate gridded datasets where I would like to be able to match the terrain data as best as possible to the radar data. In other words, whether through interpolation and/or other methods, there should be one value of terrain for each [x,y] location of the radar data.
How could one achieve this?

I would recommend using rioxarray. It has a method called reproject_match to make the two grids align:
https://corteva.github.io/rioxarray/stable/examples/reproject_match.html

geopandas not recognizing point in polygon

I have two data frames. One has polygons of buildings (around 70K) and the other has points that may or not be inside the polygons (around 100K). I need to identify if a point is inside a polygon or not.
When I plot both dataframes (example below), the plot shows that some points are inside the polygons and other are not. However, when I use .within(), the outcome says none of the points are inside polygons.
I recreated the example creating one polygon and one point "by hand" rather than importing the data and in this case .within() does recognize that the point is in the polygon. Therefore, I assume I'm making a mistake but I don't know where.
Example: (I'll just post the part that corresponds to one point and one polygon for simplicity. In this case, each data frame contains either a single point or a single polygon)
1) Using the imported data. The data frame dmR has the points and the data frame dmf has the polygon
import pandas as pd
import geopandas as gpd
import numpy as np
import matplotlib.pyplot as plt
from shapely import wkt
from shapely.geometry import Point, Polygon
plt.style.use("seaborn")
# I'm skipping the data manipulation stage and
# going to the point where the data are used.
print(dmR)
geometry
35 POINT (-95.75207 29.76047)
print(dmf)
geometry
41964 POLYGON ((-95.75233 29.76061, -95.75194 29.760...
# Plot
fig, ax = plt.subplots(figsize=(5,5))
minx, miny, maxx, maxy = ([-95.7525, 29.7603, -95.7515, 29.761])
ax.set_xlim(minx, maxx)
ax.set_ylim(miny, maxy)
dmR.plot(ax=ax, c='Red')
dmf.plot(ax=ax, alpha=0.5)
plt.savefig('imported_data.png')
The outcome
shows that the point is inside the polygon. However,
print(dmR.within(dmf))
35 False
41964 False
dtype: bool
2) If I try to recreate this by hand, it would be as follows (there may be a better way to do this but I couldn't figure it out):
# Get the vertices of the polygon to create it by hand
poly1 = dmf['geometry']
g = [i for i in poly1]
x,y = g[0].exterior.coords.xy
x,y
(array('d', [-95.752332508564, -95.75193554162979, -95.75193151831627, -95.75232848525047, -95.752332508564]),
array('d', [29.760606530637265, 29.760607694859385, 29.76044470363038, 29.76044237518235, 29.760606530637265]))
# Create the polygon by hand using the corresponding vertices
coords = [(-95.752332508564, 29.760606530637265),
(-95.75193554162979, 29.760607694859385),
(-95.75193151831627, 29.7604447036303),
(-95.75232848525047, 29.76044237518235),
(-95.752332508564, 29.760606530637265)]
poly = Polygon(coords)
# Create point by hand (just copy the point from 1) above
p1 = Point(-95.75207, 29.76047)
# Create the GeoPandas data frames from the point and polygon
ex = gpd.GeoDataFrame()
ex['geometry']=[poly]
ex = ex.set_geometry('geometry')
ex_p = gpd.GeoDataFrame()
ex_p['geometry'] = [p1]
ex_p = ex_p.set_geometry('geometry')
# Plot and print
fig, ax = plt.subplots(figsize=(5,5))
ax.set_xlim(minx, maxx)
ax.set_ylim(miny, maxy)
ex_p.plot(ax=ax, c='Red')
ex.plot(ax = ax, alpha=0.5)
plt.savefig('by_hand.png')
In this case, the outcome also shows the point in the polygon. However,
ex_p.within(ex)
0 True
dtype: bool
which recognize that the point is in the polygon. All suggestions on what to do are appreciated! Thanks.

I don't know if this is the most efficient way to do it but I was able to do what I needed within Python and using Geopandas.
Instead of using point.within(polygon) approach, I did a spatial join (geopandas.sjoin(df_1, df_2, how = 'inner', op = 'contains')) This results in a new data frame that contains the points that are within polygons and excludes the ones that are not. More information on how to do this can be found here.

I assume something is fishy about your coordinate reference system (crs). I cannot tell about dmr as it is not provided but ex_p is a naive geometry as you generated it from points without specifying the crs. You can check the crs using:
dmr.crs
Let's assume it's in 4326, then it will return:
<Geographic 2D CRS: EPSG:4326>
Name: WGS 84
Axis Info [ellipsoidal]:
- Lat[north]: Geodetic latitude (degree)
- Lon[east]: Geodetic longitude (degree)
Area of Use:
- name: World
- bounds: (-180.0, -90.0, 180.0, 90.0)
Datum: World Geodetic System 1984
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich
In this case you would need to set a CRS for ex_p first using:
ex_p = ex_p.set_crs(epsg=4326)
If you want to inherit the crs of dmr dynamically you can also use:
ex_p = ex_p.set_crs(dmr.crs)
After you set a crs, you can re-project from one crs to another using:
ex_p = ex_p.to_crs(epsg=3395)
More on that topic:
https://geopandas.org/projections.html

Finding nearest geographic raster element

I have a large list of longitude latitude points and want to find the nearest rectangle (so which rectangle contains the point) in a given raster of geographic coordinates.
However, for the raster I only have the centroids of each rectangle (polygon) in the raster. I know though that the rectangles have a size of 250m x 250m.
Just checking for absolute distance or geographic distance to the centers does not work, as the rectangles are not necessarily aligned. I am happy to get ideas.

I think you could generate your raster of geographic coordinates that represent raster cells following this approach: https://gis.stackexchange.com/questions/177061/ascii-file-with-latitude-longitude-and-data-to-geotiff-using-python
And then if you created a shapefile of your latitute and longitude points you could get raster cell ID for each point using this approach:
def GetRasterValueAtPoints(rasterfile, shapefile, fieldname):
'''
__author__ = "Marc Weber <weber.marc#epa.gov>"
Original code attribution: https://gis.stackexchange.com/a/46898/2856
returns raster values at points in a point shapefile
assumes same projection in shapefile and raster file
Arguments
---------
rasterfile : a raster file with full pathname and extension
shapefile : a shapefile with full pathname and extension
fieldname : field name in the shapefile to identify values
'''
src_ds=gdal.Open(rasterfile)
no_data = src_ds.GetRasterBand(1).GetNoDataValue()
gt=src_ds.GetGeoTransform()
rb=src_ds.GetRasterBand(1)
df = pd.DataFrame(columns=(fieldname, "RasterVal"))
i = 0
ds=ogr.Open(shapefile)
lyr=ds.GetLayer()
for feat in lyr:
geom = feat.GetGeometryRef()
name = feat.GetField(fieldname)
mx,my=geom.GetX(), geom.GetY() #coord in map units
#Convert from map to pixel coordinates.
#Only works for geotransforms with no rotation.
px = int((mx - gt[0]) / gt[1]) #x pixel
py = int((my - gt[3]) / gt[5]) #y pixel
intval = rb.ReadAsArray(px,py,1,1)
if intval == no_data:
intval = -9999
df.set_value(i,fieldname,name)
df.set_value(i,"RasterVal",float(intval))
i+=1
return df

Why are Basemap south polar stereographic map projection coordinates not agreeing with those of data sets in the same projection?

Some satellite based earth observation products provide latitude/longitude information while others provide the X/Y coordinates within a given grid projection (and there are also some having both, see example).
My approach in the second case is to set up a Basemap map which has the same parameters (projection, ellipsoid, origin of map) as given by the data provider in a way that the given X/Y values equal the Basemap coordinates. However if I do so the geolocation does not agree with other data sets including the Basemap coastline.
I have experienced this with three different data sets from different trustworthy sources. For the minimal example I use Landsat data provided by the U.S. Geological Survey which includes both, X/Y coordinates of a South Polar Stereographic grid and the corresponding lat/lon coordinates for all four corners of the image.
From a Landsat metafile we get (ID: LC82171052016079LGN00):
CORNER_UL_LAT_PRODUCT = -66.61490 CORNER_UL_LON_PRODUCT = -61.31816
CORNER_UR_LAT_PRODUCT = -68.74325 CORNER_UR_LON_PRODUCT = -58.04533
CORNER_LL_LAT_PRODUCT = -67.68721 CORNER_LL_LON_PRODUCT = -67.01109
CORNER_LR_LAT_PRODUCT = -69.94052 CORNER_LR_LON_PRODUCT = -64.18581
CORNER_UL_PROJECTION_X_PRODUCT = -2259300.000
CORNER_UL_PROJECTION_Y_PRODUCT = 1236000.000
CORNER_UR_PROJECTION_X_PRODUCT = -1981500.000
CORNER_UR_PROJECTION_Y_PRODUCT = 1236000.000
CORNER_LL_PROJECTION_X_PRODUCT = -2259300.000
CORNER_LL_PROJECTION_Y_PRODUCT = 958500.000
CORNER_LR_PROJECTION_X_PRODUCT = -1981500.000
CORNER_LR_PROJECTION_Y_PRODUCT = 958500.000
...
GROUP = PROJECTION_PARAMETERS MAP_PROJECTION = "PS" DATUM = "WGS84"
ELLIPSOID = "WGS84" VERTICAL_LON_FROM_POLE = 0.00000 TRUE_SCALE_LAT =
-71.00000 FALSE_EASTING = 0 FALSE_NORTHING = 0 GRID_CELL_SIZE_PANCHROMATIC = 15.00 GRID_CELL_SIZE_REFLECTIVE = 30.00
GRID_CELL_SIZE_THERMAL = 30.00 ORIENTATION = "NORTH_UP"
RESAMPLING_OPTION = "CUBIC_CONVOLUTION" END_GROUP =
PROJECTION_PARAMETERS
By using Basemap with the right map projection we should be able to derive the corner lat/lon values from the X/Y values:
import numpy as np
from mpl_toolkits.basemap import Basemap
m=Basemap(resolution='h',projection='spstere', ellps='WGS84', boundinglat=-60,lon_0=180, lat_ts=-71)
x_crn=np.array([-2259300,-1981500,-2259300,-1981500])# upper left, upper right, lower left, lower right
y_crn=np.array([1236000, 1236000, 958500, 958500])# upper left, upper right, lower left, lower right
x0, y0= m(0, -90)
#Basemap coordinates at the south pole
#note that (0,0) of the Basemap is in a corner of the map,
#while other data sets use the south pole.
#This is easy to take into account:
lon_crn, lat_crn = m(x0-x_crn, y0-y_crn, inverse=True)
print 'lon_crn: '+str(lon_crn)
print 'lat_crn: '+str(lat_crn)
Which returns:
lon_crn: [-61.31816102 -58.04532791 -67.01108782 -64.1858106 ]
lat_crn: [-67.23548626 -69.3099076 -68.28071626 -70.47651326]
As you can see the longitudes agree to the given precision with those from the metafile, but the latitudes are to low.
I can approximate the latitudes by:
lat_crn=(lat_crn+90.)*1.0275-90.
But this is really not satisfying.
This is how the image is located if using the X/Y corner coordinates from the metafile (in red the Basemap drawcoastlines()):
and this is how it looks like using the corner lat/lon:
In this case I can simply use the lat/lon coordinates, but as mentioned before there are datasets (like this) which is provided by X/Y coordinates only, which makes it very important to rely on the Basemap projection. I know that there are other modules to re-project the data as a potential workaround, but it should work without other modules and a re-projection could introduce errors itself.
As this problem appears with different data sets I like to believe that it is a bug in the Basemap module, but I might also make the same mistake again and again or have wrong expectations.

I did some experimentation and it seems like changing lat_ts has no effect with projection='spstere'. In fact, it seems as if the projection latitude is implicitly assumed to be lat_ts=-90. regardless of what value you assign.
I had more success using projection='stere' instead, so that you would construct the Basemap in your example as follows:
m=Basemap(width=5400000., height=5400000., projection='stere',
ellps='WGS84', lon_0=180., lat_0=-90., lat_ts=-71.)
You may prefer to set the latitude and longitude of the corners instead of the width and height of the plot for your application.

Automatically center matplotlib basemap onto data

I would like a solution to automatically center a basemap plot on my coordinate data.
I've got things to automatically center, but the resulting area is much larger than the area actually used by my data. I would like the plot to be bounded by the plot coordinates, rather than an area drawn from the lat/lon boundaries.
I am using John Cook's code for calculating the distance between two points on (an assumed perfect) sphere.
First Try
Here is the script I started with. This was causing the width and height to bee small too small for the data area, and the center latitude (lat0) too far south.
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import numpy as np
import sys
import csv
import spheredistance as sd
print '\n'
if len(sys.argv) < 3:
print >>sys.stderr,'Usage:',sys.argv[0],'<datafile> <#rows to skip>'
sys.exit(1)
print '\n'
dataFile = sys.argv[1]
dataStream = open(dataFile, 'rb')
dataReader = csv.reader(dataStream,delimiter='\t')
numRows = sys.argv[2]
dataValues = []
dataLat = []
dataLon = []
print 'Plotting Data From: '+dataFile
dataReader.next()
for row in dataReader:
dataValues.append(row[0])
dataLat.append(float(row[1]))
dataLon.append(float(row[2]))
# center and set extent of map
earthRadius = 6378100 #meters
factor = 1.00
lat0new = ((max(dataLat)-min(dataLat))/2)+min(dataLat)
lon0new = ((max(dataLon)-min(dataLon))/2)+min(dataLon)
mapH = sd.distance_on_unit_sphere(max(dataLat),lon0new,
min(dataLat),lon0new)*earthRadius*factor
mapW = sd.distance_on_unit_sphere(lat0new,max(dataLon),
lat0new,min(dataLon))*earthRadius*factor
# setup stereographic basemap.
# lat_ts is latitude of true scale.
# lon_0,lat_0 is central point.
m = Basemap(width=mapW,height=mapH,
resolution='l',projection='stere',\
lat_0=lat0new,lon_0=lon0new)
#m.shadedrelief()
m.drawcoastlines(linewidth=0.2)
m.fillcontinents(color='white', lake_color='aqua')
#plot data points (omitted due to ownership)
#x, y = m(dataLon,dataLat)
#m.scatter(x,y,2,marker='o',color='k')
# draw parallels and meridians.
m.drawparallels(np.arange(-80.,81.,20.), labels=[1,0,0,0], fontsize=10)
m.drawmeridians(np.arange(-180.,181.,20.), labels=[0,0,0,1], fontsize=10)
m.drawmapboundary(fill_color='aqua')
plt.title("Example")
plt.show()

After generating some random data, it was obvious that the bounds that I chose did not work with this projection (red lines). Using map.drawgreatcircle(), I first visualized where I wanted the bounds while zoomed over the projection of random data.
I corrected the longitude by using the longitudinal difference at the southern most latitude (blue horizontal line).
I determined the latitudinal range using the Pythagorean theorem to solve for the vertical distance, knowing the distance between the northern most longitudinal bounds, and the central southernmost point (blue triangle).
def centerMap(lats,lons,scale):
#Assumes -90 < Lat < 90 and -180 < Lon < 180, and
# latitude and logitude are in decimal degrees
earthRadius = 6378100.0 #earth's radius in meters
northLat = max(lats)
southLat = min(lats)
westLon = max(lons)
eastLon = min(lons)
# average between max and min longitude
lon0 = ((westLon-eastLon)/2.0)+eastLon
# a = the height of the map
b = sd.spheredist(northLat,westLon,northLat,eastLon)*earthRadius/2
c = sd.spheredist(northLat,westLon,southLat,lon0)*earthRadius
# use pythagorean theorom to determine height of plot
mapH = pow(pow(c,2)-pow(b,2),1./2)
arcCenter = (mapH/2)/earthRadius
lat0 = sd.secondlat(southLat,arcCenter)
# distance between max E and W longitude at most souther latitude
mapW = sd.spheredist(southLat,westLon,southLat,eastLon)*earthRadius
return lat0,lon0,mapW*scale,mapH*scale
lat0center,lon0center,mapWidth,mapHeight = centerMap(dataLat,dataLon,1.1)
The lat0 (or latitudinal center) in this case is therefore the point half-way up the height of this triangle, which I solved using John Cooks method, but for solving for an unknown coordinate while knowing the first coordinate (the median longitude at the southern boundary) and the arc length (half that of the total height).
def secondlat(lat1, arc):
degrees_to_radians = math.pi/180.0
lat2 = (arc-((90-lat1)*degrees_to_radians))*(1./degrees_to_radians)+90
return lat2
Update:
The above function, as well as the distance between two coordinates can be achieved with higher accuracy using the pyproj Geod class methods geod.fwd() and geod.inv(). I found this in Erik Westra's Python for Geospatial Development, which is an excellent resource.
Update:
I have now verified that this also works for Lambert Conformal Conic (lcc) projections.