Develop Reference

SciKits BallTree method gives me incorrect "nearest neighbor"

I'm using code from the source given below to get the nearest "site".
Source: https://automating-gis-processes.github.io/site/notebooks/L3/nearest-neighbor-faster.html
My Code:
# Read data from a DB
test_df = pd.read_sql_query(sql, conn)
# Calculates distance between 2 points on a map using lat and long
# (Source: https://towardsdatascience.com/heres-how-to-calculate-distance-between-2-geolocations-in-python-93ecab5bbba4)
def haversine_distance(lat1, lon1, lat2, lon2):
r = 6371
phi1 = np.radians(float(lat1))
phi2 = np.radians(float(lat2))
delta_phi = np.radians(lat2 - lat1)
delta_lambda = np.radians(lon2- lon1)
a = np.sin(delta_phi / 2)**2 + np.cos(phi1) * np.cos(phi2) * np.sin(delta_lambda / 2)**2
res = r * (2 * np.arctan2(np.sqrt(a), np.sqrt(1 - a)))
return np.round(res, 2)
test_df["actualDistance (km)"] = test_df.apply(lambda row: haversine_distance(row['ClientLat'],row['ClientLong'],row['actual_SLa'],row['actual_SLo']), axis=1)
test_gdf = geopandas.GeoDataFrame(test_df, geometry=geopandas.points_from_xy(test_df.ClientLong, test_df.ClientLat))
site_gdf = geopandas.GeoDataFrame(site_df, geometry=geopandas.points_from_xy(site_df.SiteLong, site_df.SiteLat))
#-------Set up the functions as shown in the tutorial-------
def get_nearest(src_points, candidates, k_neighbors=1):
"""Find nearest neighbors for all source points from a set of candidate points"""
# Create tree from the candidate points
tree = BallTree(candidates, leaf_size=15, metric='haversine')
# Find closest points and distances
distances, indices = tree.query(src_points, k=k_neighbors)
# Transpose to get distances and indices into arrays
distances = distances.transpose()
indices = indices.transpose()
# Get closest indices and distances (i.e. array at index 0)
# note: for the second closest points, you would take index 1, etc.
closest = indices[0]
closest_dist = distances[0]
# Return indices and distances
return (closest, closest_dist)
def nearest_neighbor(left_gdf, right_gdf, return_dist=False):
"""
For each point in left_gdf, find closest point in right GeoDataFrame and return them.
NOTICE: Assumes that the input Points are in WGS84 projection (lat/lon).
"""
left_geom_col = left_gdf.geometry.name
right_geom_col = right_gdf.geometry.name
# Ensure that index in right gdf is formed of sequential numbers
right = right_gdf.copy().reset_index(drop=True)
# Parse coordinates from points and insert them into a numpy array as RADIANS
left_radians = np.array(left_gdf[left_geom_col].apply(lambda geom: (geom.x * np.pi / 180, geom.y * np.pi / 180)).to_list())
right_radians = np.array(right[right_geom_col].apply(lambda geom: (geom.x * np.pi / 180, geom.y * np.pi / 180)).to_list())
# Find the nearest points
# -----------------------
# closest ==> index in right_gdf that corresponds to the closest point
# dist ==> distance between the nearest neighbors (in meters)
closest, dist = get_nearest(src_points=left_radians, candidates=right_radians)
# Return points from right GeoDataFrame that are closest to points in left GeoDataFrame
closest_points = right.loc[closest]
# Ensure that the index corresponds the one in left_gdf
closest_points = closest_points.reset_index(drop=True)
# Add distance if requested
if return_dist:
# Convert to meters from radians
earth_radius = 6371000 # meters
closest_points['distance'] = dist * earth_radius
return closest_points
closest_sites = nearest_neighbor(test_gdf, site_gdf, return_dist=True)
# Rename the geometry of closest sites gdf so that we can easily identify it
closest_sites = closest_sites.rename(columns={'geometry': 'closest_site_geom'})
# Merge the datasets by index (for this, it is good to use '.join()' -function)
test_gdf = test_gdf.join(closest_sites)
#Extracted closest site latitude and longitude for data analysis
test_gdf['CS_lo'] = test_gdf.closest_site_geom.apply(lambda p: p.x)
test_gdf['CS_la'] = test_gdf.closest_site_geom.apply(lambda p: p.y)
The code is a replica of the tutorial link I provided. And based on their explanation it should've worked.
To verify this data I got some statistical data using .describe(), and it showed me that the tutorials method did indeed give me a mean distance that was much closer than the distance in the actual data (792 m vs the actual distance which was 1.80 km).
Closest Distance generated using the BallTree method
Actual Distance in the data
However when I plotted them out on a map using plotly I noticed that the BallTree method's outputs weren't closer than the "actual" distance.
This is generally what the plotted data looks like (Blue: predetermined site, Red: site predicted using the BallTree method
Could someone help me track down the discrepancy

I'm not sure why this works but it did. I decided to just write the code based on the docs instead of following the tutorial and this worked:
# Build BallTree with haversine distance metric, which expects (lat, lon) in radians and returns distances in radians
dist = DistanceMetric.get_metric('haversine')
tree = BallTree(np.radians(site_df[['SiteLat', 'SiteLong']]), metric=dist)
test_coords = np.radians(test_df[['ClientLat', 'ClientLong']])
dists, ilocs = tree.query(test_coords)

The problem is that the tutorial code provides coordinates in Longitude, Latitude format instead of the Latitude, Longitude format BallTree anticipates. So you're measuring distances between inverted points.
If you swap the order of geom.x and geom.y in the coordinate parsing code you will get correct measurements.
# Parse coordinates from points and insert them into a numpy array as RADIANS
left_radians = np.array(left_gdf[left_geom_col].apply(lambda geom: (geom.y * np.pi / 180, geom.x * np.pi / 180)).to_list())
right_radians = np.array(right[right_geom_col].apply(lambda geom: (geom.y * np.pi / 180, geom.x * np.pi / 180)).to_list())

calculating an intercept point between a straight line and an ellipse - python

Iv'e been trying lately to calculate a point an ellipse
The desired point is the green point , knowing the red dots
and the ellipse equation.
I've used numpy linspace to create an array on points
and iterate them using zip(x axis , y axis)
between the red points , and using the ellipse
equation figure which of the points is the closest to 1.
(which is the outcome of the ellipse equation ).
this concept works most of the time , but in some location
of the red outer dot , this method doesn't seem to give good outcome
long story short, any idea how to calculate the green dot in python?
p.s - ellipse might have angle, both of hes axis are known.

I end up using the ellipse equation from this answer:
and created an in_ellipse function
then Iv'e used the Intermediate value theorem , to get a good estimation
of the point
def in_ellipse(point, ellipse):
return true if point in ellipse
return false
dot_a = ellipse_center
dot_b = dot
for i in range(20):
center_point = ((dot_b.y - dot_a.y)/2, (dot_b.x - dot_a.x)/2)
if in_ellipse(center_point):
dot_a = center_point
else:
dot_b = center_point
return center_point
this system gives the point in 7 (2^20) digits resolution after decimal point
you can increase the range for better resolution.

Let ellipse center is (0,0) (otherwise just subtract center coordinates), semi-axes are a, b and rotation angle is theta. We can build affine tranformation to transform ellipse into circle and apply the same transform to point P.
1) Rotate by -theta
px1 = px * Cos(theta) + py * Sin(theta)
py1 = -px * Sin(theta) + py * Cos(theta)
2) Extend (or shrink) along OY axis by a/b times
px2 = px1
py2 = py1 * a / b
3) Find intersection point
plen = hypot(px2, py2) (length of p2 vector)
if (a > plen), then segment doesn't intersect ellipse - it fully lies inside
ix = a * px2 / plen
iy = a * py2 / plen
4) Make backward shrinking
ix2 = ix
iy2 = iy * b / a
5) Make backward rotation
ixfinal = ix2 * Cos(theta) - iy2 * Sin(theta)
iyfinal = ix2 * Sin(theta) + iy2 * Cos(theta)

Calculating geographic distance between a list of coordinates (lat, lng)

I'm writing a flask application, using some data extracted from a GPS sensor. I am able to draw the route on a Map and I want to calculate the distance the GPS sensor traveled. One way could be to just get the start and end coordinates, however due to the way the sensor travels this is quite inaccurate. Therefore I do sampling of each 50 sensor samples. If the real sensor sample size was 1000 I will now have 20 samples (by extracting each 50 sample).
Now I want to be able to put my list of samples through a function to calculate distance. So far I've been able to use the package geopy, but when I take large gps sample sets I do get "too many requests" errors, not to mention I will have extra processing time from processing the requests, which is not what I want.
Is there a better approach to calculating the cumulative distance of a list element containing latitude and longitude coordinates?
positions = [(lat_1, lng_1), (lat_2, lng_2), ..., (lat_n, lng_n)]
I found methods for lots of different mathematical ways of calculating distance using just 2 coordinates (lat1, lng1 and lat2 and lng2), but none supporting a list of coordinates.
Here's my current code using geopy:
from geopy.distance import vincenty
def calculate_distances(trips):
temp = {}
distance = 0
for trip in trips:
positions = trip['positions']
for i in range(1, len(positions)):
distance += ((vincenty(positions[i-1], positions[i]).meters) / 1000)
if i == len(positions):
temp = {'distance': distance}
trip.update(temp)
distance = 0
trips is a list element containing dictionaries of key-value pairs of information about a trip (duration, distance, start and stop coordinates and so forth) and the positions object inside trips is a list of tuple coordinates as visualized above.
trips = [{data_1}, {data_2}, ..., {data_n}]

Here's the solution I ended up using. It's called the Haversine (distance) function if you want to look up what it does for yourself.
I changed my approach a little as well. My input (positions) is a list of tuple coordinates:
def calculate_distance(positions):
results = []
for i in range(1, len(positions)):
loc1 = positions[i - 1]
loc2 = positions[i]
lat1 = loc1[0]
lng1 = loc1[1]
lat2 = loc2[0]
lng2 = loc2[1]
degreesToRadians = (math.pi / 180)
latrad1 = lat1 * degreesToRadians
latrad2 = lat2 * degreesToRadians
dlat = (lat2 - lat1) * degreesToRadians
dlng = (lng2 - lng1) * degreesToRadians
a = math.sin(dlat / 2) * math.sin(dlat / 2) + math.cos(latrad1) * \
math.cos(latrad2) * math.sin(dlng / 2) * math.sin(dlng / 2)
c = 2 * math.atan2(math.sqrt(a), math.sqrt(1 - a))
r = 6371000
results.append(r * c)
return (sum(results) / 1000) # Converting from m to km

I'd recommend transform your (x, y) coordinates into complex, as it is computational much easier to calculate distances. Thus, the following function should work:
def calculate_distances(trips):
for trip in trips:
positions = trip['positions']
c_pos = [complex(c[0],c[1]) for c in positions]
distance = 0
for i in range(1, len(c_pos)):
distance += abs(c_pos[i] - c_pos[i-1])
trip.update({'distance': distance})
What I'm doing is converting every (lat_1, lng_1) touple into a single complex number c1 = lat_1 + j*lng_1, and creates a list formed by [c1, c2, ... , cn].
A complex number is, all in all, a 2-dimensional number and, therefore, you can make this if you have 2D coordinates, which is perfect for geolocalization, but wouldn't be possible for 3D space coordinates, for instance.
Once you got this, you can easily compute the distance between two complex numbers c1 and c2 as dist12 = abs(c2 - c1). Doing this recursively you obtain the total distance.
Hope this helped!

Transform and remap an equirectangular image with a 90° roll

I have to transform and remap an equirectangular image to an other equirectangular image with a 90° roll.
I did it with Pano2VR.
The problem is that I have to do it programmatically from the server side. So I can't use a G.U.I. to do it.
First, I oriented my research to imagemagick. I tried the Fred ImageMagick scripts but could not find anyone to do what I want to do. Moreover, the processing time of an image appears very long compared to Pano2VR.
I directed my investigations to OpenCV and the libgnomonic. It's presently the most interesting way. This library allows te user to transform projections (equirectangular to rectilinear and vice versa) or make equirectangular mapping transformation. I played with Norama-suite wich contains some scripts to deal with the library. For example, I would to convert an rectilinear image to an equirectangular but the output was just a black background image (why ? I didn't find the answer).
However, this second link could resolve my problem. I have this image :
and I want to transform it to this image
Well, I'm not comfortable at all with C.
I think I should use this two files :
https://www.github.com/FoxelSA/libgnomonic/blob/master/src/gnomonic-transform.h
https://www.github.com/FoxelSA/libgnomonic/blob/master/src/gnomonic-transform.c
But I don't know how. And above all, I want to understand.
Am I on the right way ? What transformation is applied on the first image ? Is there a way to do it with python or a bash script ?
Well, thank you for your help.
**EDIT Transposition of C in python **
The following code didn't work and return and IndexError.
However I tried to catch and pass the exception and the first right part of the image did not seem changed.
import math
from PIL import Image
img = Image.open("img1.jpg")
img = img.convert('RGB')
pixel = img.load()
width, height = img.size
img2 = img.copy()
for y in xrange(height):
for x in xrange(width):
xx = 2*(y+0.5) / width - 1.0
yy = 2*(y+0.5)/ height - 1.0
lng = math.pi * xx
lat = 0.5 * math.pi * yy
# NOTE! These axes are transposed because that's what the question is about
Z = math.cos(lat) * math.cos(lng) # normally X
Y = math.cos(lat) * math.sin(lng) # normally Y
X = -math.sin(lat) # normally -Z
D = math.sqrt(X*X+Y*Y)
lat = math.atan2(Z, D) # ? normally lat = math.asin(Z)
lng = math.atan2(Y, X)
#ix and iy must be integers
ix = int((0.5 * lng / math.pi + 0.5) * width - 0.5)
iy = int((lat/math.pi + 0.5) * height - 0.5)
#not sure of this part to remap the image
newpixel = pixel[ix, iy]
img2.putpixel([(x+width/4) % width, y], newpixel)
#I tries as mentionned in the following code to invert x and y in the two previous lines but the index error out of range comes back
img2.show()

Your transformation has two steps. The first step is a transformation of the projection sphere, the second is the 90° roll.
A 90° roll of an equirectangular image is just a horizontal shift of a quarter of the image width. The first transformation is more complicated: You basically want to rotate the sphere so that the north pole is at latitude 0 and longitude 0 (somewhere in the Gulf of Guinea, if you take thze Earth as reference.)
You can go about this transformation with these steps;
Translate the x and y position of each pixel into a longitude, −π ≤ long ≤ π, and a latitude, −π/2 ≤ lat ≤ π/2. This is a linear transformation.
Create x, y, and z coordinates of the corresponding longitude and latitude on a unit sphere; positive z is the north pole.
Rotate these Cartesian coordinates. In your case, you just have to swap some dimensions, but this could be a general transformation with any transformation matrix.
Calculate the longituide and latitude of the rotated coordinates.
Transform the new longitude and latitude to a pixel position.
Here's C code that works. (I know that you have tagged the question with Python, but the code below is mostly formulas that work similarly in python. You have to take care: All of the number are floating point numbers except the pixel indixes x, y, ixand iy. I would have done this in Python, but I have no experience with the Python Image Library.)
for (y = 0; y < height; y++) {
for (x = 0; x < width; x++) {
double xx = 2 * (x + 0.5) / width - 1.0;
double yy = 2 * (y + 0.5) / height - 1.0;
double lng = pi * xx;
double lat = 0.5 * pi * yy;
double X, Y, Z;
double D;
int ix, iy;
Z = cos(lat) * cos(lng); // corresponds to original x
Y = cos(lat) * sin(lng); // corresponds to original y
X = -sin(lat); // corresponds to original z
D = sqrt(X*X + Y*Y); // distance in the XY plane
lat = atan2(Z, D);
lng = atan2(Y, X);
ix = (0.5 * lng / pi + 0.5) * width - 0.5;
iy = (lat / pi + 0.5) * height - 0.5;
dest[y][(x + width / 4) % width] = src[iy][ix];
// width/4 offset ist the 90° roll
// % width wraps the longitude
}
}
The quality of the resulting image is okay, but not as good as that of your reference image, especially near the poles. A better algorithm woul average and smoothze the colour values. The algorithm above just maps one destination pixel to a source pixel.

Python Work out area of a polygon on a spherical surface

I have a series of points, of right ascension and declination values.
These points correspond to the vertices of a polygon on the surface of a sphere.
What would be the best way to calculate the area enclosed by these points? I would assume that converting the points with an equal-area projection, and then carrying out typical polygonal area calculating on a flat surface would be an appropriate solution.
note: I cannot use custom python libraries. eg pyproj or shapely
Example code (works for latitude longitude, what modifications would be required to enure this works with sky coordinates?)
def reproject(latitude, longitude):
"""Returns the x & y coordinates in metres using a sinusoidal projection"""
from math import pi, cos, radians
earth_radius = 6371009
lat_dist = pi * earth_radius / 180.0
y = [lat * lat_dist for lat in latitude]
x = [long * lat_dist * cos(radians(lat))
for lat, long in zip(latitude, longitude)]
return x, y
def area_of_polygon(x, y):
"""Calculates the area of an arbitrary polygon given its vertices"""
area = 0.0
for i in xrange(-1, len(x)-1):
area += x[i] * (y[i+1] - y[i-1])
return abs(area) / 2.0
dec = [-15.,89.,89.,-15.,-15.]
ra = [105.,105.,285.,285.,105.]
x,y = reproject(dec, ra)
print area_of_polygon(x,y)

One of the ways is to perform a line integral based on Green's Theorem. See below an implementation, and this question for more details.
def polygon_area(lats, lons, algorithm = 0, radius = 6378137):
"""
Computes area of spherical polygon, assuming spherical Earth.
Returns result in ratio of the sphere's area if the radius is specified.
Otherwise, in the units of provided radius.
lats and lons are in degrees.
"""
from numpy import arctan2, cos, sin, sqrt, pi, power, append, diff, deg2rad
lats = np.deg2rad(lats)
lons = np.deg2rad(lons)
# Line integral based on Green's Theorem, assumes spherical Earth
#close polygon
if lats[0]!=lats[-1]:
lats = append(lats, lats[0])
lons = append(lons, lons[0])
#colatitudes relative to (0,0)
a = sin(lats/2)**2 + cos(lats)* sin(lons/2)**2
colat = 2*arctan2( sqrt(a), sqrt(1-a) )
#azimuths relative to (0,0)
az = arctan2(cos(lats) * sin(lons), sin(lats)) % (2*pi)
# Calculate diffs
# daz = diff(az) % (2*pi)
daz = diff(az)
daz = (daz + pi) % (2 * pi) - pi
deltas=diff(colat)/2
colat=colat[0:-1]+deltas
# Perform integral
integrands = (1-cos(colat)) * daz
# Integrate
area = abs(sum(integrands))/(4*pi)
area = min(area,1-area)
if radius is not None: #return in units of radius
return area * 4*pi*radius**2
else: #return in ratio of sphere total area
return area
Please find a somewhat more explicit version (and with many more references and TODOs...) here.

Looks like I can treat ra and dec like lat and long, work out the area on the Earth's surface in m^2, and use this value to convert into an area in sq degrees.
Please let me know if the solution I propose below is flawed:
def reproject(latitude, longitude):
"""Returns the x & y coordinates in metres using a sinusoidal projection"""
from math import pi, cos, radians
earth_radius = 6371009
lat_dist = pi * earth_radius / 180.0
y = [lat * lat_dist for lat in latitude]
x = [long * lat_dist * cos(radians(lat))
for lat, long in zip(latitude, longitude)]
return x, y
def area_of_polygon(x, y):
"""Calculates the area of an arbitrary polygon given its vertices"""
area = 0.0
for i in xrange(-1, len(x)-1):
area += x[i] * (y[i+1] - y[i-1])
return ((abs(area) / 2.0)/5.10100E14) * 41253
dec = [-15.,89.,89.,-15.,-15.]
ra = [105.,105.,285.,285.,105.]
x,y = reproject(dec, ra)
print area_of_polygon(x,y)