I want to generate a distance matrix 500X500 based on latitude and longitude of 500 locations, using Haversine formula.
Here is the sample data "coordinate.csv" for 10 locations:
Name,Latitude,Longitude
depot1,35.492807,139.6681689
depot2,33.6625572,130.4096027
depot3,35.6159881,139.7805445
customer1,35.622632,139.732631
customer2,35.857287,139.821461
customer3,35.955313,139.615387
customer4,35.16073,136.926239
customer5,36.118163,139.509548
customer6,35.937351,139.909783
customer7,35.949508,139.676462
After getting the distance matrix, I want to find the closest depot to each customer based on the distance matrix, and then save the output (Distance from each customer to the closet depot & Name of the closest depot) to Pandas DataFrame.
Expected outputs:
// Distance matrix
[ [..],[..],[..],[..],[..],[..],[..],[..],[..],[..] ]
// Closet depot to each customer (just an example)
Name,Latitude,Longitude,Distance_to_closest_depot,Closest_depot
depot1,35.492807,139.6681689,,
depot2,33.6625572,130.4096027,,
depot3,35.6159881,139.7805445,,
customer1,35.622632,139.732631,10,depot1
customer2,35.857287,139.821461,20,depot3
customer3,35.955313,139.615387,15,depot2
customer4,35.16073,136.926239,12,depot3
customer5,36.118163,139.509548,25,depot1
customer6,35.937351,139.909783,22,depot2
customer7,35.949508,139.676462,15,depot1
There are a couple of library functions that can help you with this:
cdist from scipy can be used to generate a distance matrix using whichever distance metric you like.
There is also a haversine function which you can pass to cdist.
After that it's just a case of finding the row-wise minimums from the distance matrix and adding them to your DataFrame. Full code below:
import pandas as pd
from scipy.spatial.distance import cdist
from haversine import haversine
df = pd.read_clipboard(sep=',')
df.set_index('Name', inplace=True)
customers = df[df.index.str.startswith('customer')]
depots = df[df.index.str.startswith('depot')]
dm = cdist(customers, depots, metric=haversine)
closest = dm.argmin(axis=1)
distances = dm.min(axis=1)
customers['Closest Depot'] = depots.index[closest]
customers['Distance'] = distances
Results:
Latitude Longitude Closest Depot Distance
Name
customer1 35.622632 139.732631 depot3 4.393506
customer2 35.857287 139.821461 depot3 27.084212
customer3 35.955313 139.615387 depot3 40.565820
customer4 35.160730 136.926239 depot1 251.466152
customer5 36.118163 139.509548 depot3 60.945377
customer6 35.937351 139.909783 depot3 37.587862
customer7 35.949508 139.676462 depot3 38.255776
As per comment, I have created an alternative solution which instead uses a square distance matrix. The original solution is better in my opinion, as the question stated that we only want to find the closest depot for each customer, so calculating distances between customers and between depots isn't necessary. However, if you need the square distance matrix for some other purpose, here is how you would create it:
import pandas as pd
import numpy as np
from scipy.spatial.distance import squareform, pdist
from haversine import haversine
df = pd.read_clipboard(sep=',')
df.set_index('Name', inplace=True)
dm = pd.DataFrame(squareform(pdist(df, metric=haversine)), index=df.index, columns=df.index)
np.fill_diagonal(dm.values, np.inf) # Makes it easier to find minimums
customers = df[df.index.str.startswith('customer')]
depots = df[df.index.str.startswith('depot')]
customers['Closest Depot'] = dm.loc[depots.index, customers.index].idxmin()
customers['Distance'] = dm.loc[depots.index, customers.index].min()
The final results are the same as before, except you now have a square distance matrix. You can put the 0s back on the diagonal after you have extracted the minimum values if you like:
np.fill_diagonal(dm.values, 0)
If you need a very big matrix and have access to a NVIDIA GPU with CUDA you can use this numba function:
from numba import cuda
import math
#cuda.jit
def haversine_gpu_distance_matrix(p, G):
i,j = cuda.grid(2)
if i < p.shape[0] == G.shape[0] and j < p.shape[0] == G.shape[1]:
if i == j:
G[i][j] = 0
else:
longit_a = math.radians(p[i][0])
latit_a = math.radians(p[i][1])
longit_b = math.radians(p[j][0])
latit_b = math.radians(p[j][1])
dist_longit_add = longit_b - longit_a
dist_latit_sub = latit_b - latit_a
dist_latit_add = latit_b + latit_a
pre_comp = math.sin(dist_latit_sub/2)**2
area = pre_comp + ((1 - pre_comp - math.sin(dist_latit_add/2)**2) * math.sin(dist_longit_add/2)**2)
central_angle = 2 * math.asin(math.sqrt(area))
radius = 3958
G[i][j] = math.fabs(central_angle * radius)
You can call this function using the following commands:
# 10k [lon, lat] elements, replace this with your [lon, lat] array
# if your data is in a Pandas DataFrame, please convert it to a numpy array
geo_array = np.ones((10000, 2))
# allocate an empty distance matrix to fill when the function is called
dm_global_mem = cuda.device_array((geo_array.shape[0], geo_array.shape[0]))
# move the data in geo_array onto the GPU
geo_array_global_mem = cuda.to_device(geo_array)
# specify kernel dimensions, this can/should be further optimized for your hardware
threadsperblock = (16, 16)
blockspergrid_x = math.ceil(geo_array.shape[0] / threadsperblock[0])
blockspergrid_y = math.ceil(geo_array.shape[1] / threadsperblock[1])
blockspergrid = (blockspergrid_x, blockspergrid_y)
# run the function, which will transform dm_global_mem inplace
haversine_gpu_distance_matrix[blockspergrid, threadsperblock](geo_array_global_mem, dm_global_mem)
Note that this can be further optimized for your hardware. The runtime on a g4dn.xlarge instance on 10k geographical coordinate pairs (i.e. 100M distance measurements) is less than 0.01 seconds after compiling. The radius value is set such that the distance matrix is in unit miles, you can change it to 6371 for meters.
Related
As seen in the picture I have an outlier and I would like to remove it(not the red one but the one above it in green, which is not aligned with other points) and hence I am trying to find the min distance and then try to eliminate it. But given the huge dataset it takes an eternity to execute. This is my code below. Appreciate any solution that helps, thanks! enter image description here
import math
#list of 11600 points
dataset = [[2478, 3534], [4217, 953],......,11600 points]
copy_dataset = dataset
Indices =[]
Min_Dists =[]
Distance = []
Copy_Dist=[]
for p1 in range(len(dataset)):
p1_x= dataset[p1][0]
p1_y= dataset[p1][1]
for p2 in range(len(copy_dataset)):
p2_x= copy_dataset[p2][0]
p2_y= copy_dataset[p2][1]
dist = math.sqrt((p1_x - p2_x) ** 2 + (p1_y - p2_y) ** 2)
Distance.append(dist)
Copy_Dist.append(dist)
min_dist_1= min(Distance)
Distance.remove(min_dist_1)
if(min_dist_1 !=0):
Min_Dists.append(min_dist_1)
ind_1 = Copy_Dist.index(min_dist_1)
Indices.append(ind_1)
min_dist_2=min(Distance)
Distance.remove(min_dist_2)
if(min_dist_2 !=0):
Min_Dists.append(min_dist_2)
ind_2 = Copy_Dist.index(min_dist_2)
Indices.append(ind_2)
To_Remove = copy_dataset.index([p1_x, p1_y])
copy_dataset.remove(copy_dataset[To_Remove])
Not sure how to solve this problem in general, but it's probably a lot faster to compute the distances in a vectorized fashion.
dataset_copy = dataset.copy()
dataset_copy = dataset_copy[:, np.newaxis]
distance = np.sqrt(np.sum(np.square(dataset - dataset_copy), axis=~0))
Thank you for the answers mates! I tried the below way to solve the issue it worked pretty quick.
from statistics import mean
from scipy.spatial import distance
D = distance.squareform(distance.pdist(dataset))
closest = np.argsort(D, axis=1)
d1 =[]
for i in range(len(dataset)):
d1.append(D[i][closest[i][1]])
avg_dist = int(mean(d1))
for i in range(len(dataset)):
d1= D[i][closest[i][1]]
d2= D[i][closest[i][2]]
if(abs(avg_dist-d1)>2):
if(abs(avg_dist-d2)>2):
print(dataset[i])
dataset.remove(dataset[i])
If you need all distances at once:
distances = scipy.spatial.distance_matrix(dataset, dataset)
If you need distances of one point to all others:
for pt in dataset:
distances = scipy.spatial.distance_matrix([pt], dataset)[0]
# distances.min() will be 0 because the point has 0 distance to itself
# the nearest neighbor will be the second element in sorted order
indices = np.argpartition(distances, 1) # or use argsort for a complete sort
nearest_neighbor = indices[1]
Documentation: distance_matrix, argpartition
I have two arrays loaded with complex numbers that represent a position in a cartesian coordinate (x,y).
sensors= np.array([-1.6-0.8j,-1.1-0.8j])
cameras= np.array([-3.7-0.8j,-1.6+0.9j,-1.6-0.9j])
Where the real part represents X and the imaginary part represents Y. These numbers represent in meters. So 1.5-0.5j = 1.5 meters +X and 0.5 meters -Y.
Using the isclose function has issues when the position of the sensors gets further from 0.0.
def close_to_sensors(sensors, observations):
tolerance = 0.6
observe_indices = np.zeros(observations.size, dtype=bool)
for sensor in sensors:
closeness = np.isclose(observations, np.ones(observations.size, dtype=np.complex128)*sensor, rtol=tolerance, atol=tolerance)
observe_indices = np.logical_or(observe_indices, closeness)
print("Closeness : ", closeness)
return np.argwhere(observe_indices).flatten()
This returns
Closeness : [False False True]
Likely Close: [2]
The isclose function is the wrong function to use. I need to return the indices of the cameras that are within 1 meter of the sensors. What would be the best way to do this?
To calculate distances from complex numbers, subtracting them and calculating the absolute value of the difference is a straight-forward approach to solve this problem:
import numpy as np
sensors = np.array([-1.6 - 0.8j, -1.1 - 0.8j])
cameras = np.array([-3.7 - 0.8j, -1.6 + 0.9j, -1.6 - 0.9j])
distance_limit = 1
# calculate difference of each sensor to each camera
# "None" is used to create a new axis, which enables broadcasting to a (sensors x cameras) matrix
complex_differences = sensors[:, None] - cameras
axis_sensor, axis_camera = (0,1)
distances = np.abs(complex_differences)
# check cameras which have any sensor within distance limit
within_range = distances < distance_limit
valid_cameras = np.any(within_range, axis=axis_sensor)
# show indices of valid cameras
print(np.where(valid_cameras)[0])
Thank you all for your responses but those resulted in undesired results. I eventually decided to change the complex number arrays to [real, imag] lists, then load the sensors list to a KDTree and searched the tree for observations that were close; where 1 = 1 meter. This provided the results I needed.
EDIT: Added code with data
import numpy as np
import scipy.spatial as spatial
def close_to_sensors(bifrost_sensors, observations):
sensors_x_y = []
observations_x_y = []
for i in range(bifrost_sensors.size):
sensors_x_y.append((bifrost_sensors[i].real, bifrost_sensors[i].imag))
for i in range(observations.size):
observations_x_y.append((observations[i].real, observations[i].imag))
observe_indices = np.zeros(observations.size, dtype=bool)
#KDTree the sensor list
sensor_tree = spatial.cKDTree(np.c_[sensors_x_y])
for i in range(len(observations_x_y)):
closeness = (sensor_tree.data[sensor_tree.query_ball_point(observations_x_y[i], 1)])
if closeness.size == 0:
observe_indices[i] = np.logical_or(observe_indices[i], 0)
else:
observe_indices[i] = np.logical_or(observe_indices[i], 1)
#Find the indices of array elements that are non-zero, grouped by element.
return np.argwhere(observe_indices).flatten()
#Excel copied data into arrays - 12 entries
sensors = np.array([-0.6-0.8j,-0.8-1.2j,-0.9-1.2j,-1.-0.9j,-1.1-1.j,1.1+1.j,-1.5-1.5j,-1.6-1.1j,-1.7-1.5j,1.1+1.j,1.8+0.8j,-2.-1.6j])
cameras = np.array([-4.03-1.1j,-4.15-1.14j,-1.5-1.16j,-4.05-1.14j,-4.05-1.14j,4.03+2.19j,-4.08-1.13j,-4.06-1.14j,-1.15-0.98j,3.21+1.92j,3.9+1.65j,-4.08-1.13j])
likely_bifrost = close_to_sensors(sensors, cameras)
print("Likely bifrost : ", likely_bifrost.size, " : ",likely_bifrost)
I am struggling to calculate the distance between multiple sets of latitude and longitude coordinates. In, short, I have found numerous tutorials that either use math or geopy. These tutorials work great when I just want to find the distance between ONE set of coordindates (or two unique locations). However, my objective is to scan a data set that has 400k combinations of origin and destination coordinates. One example of the code I have used is listed below, but it seems I am getting errors when my arrays are > 1 record. Any helpful tips would be much appreciated. Thank you.
# starting dataframe is df
lat1 = df.lat1.as_matrix()
long1 = df.long1.as_matrix()
lat2 = df.lat2.as_matrix()
long2 = df.df_long2.as_matrix()
from geopy.distance import vincenty
point1 = (lat1, long1)
point2 = (lat2, long2)
print(vincenty(point1, point2).miles)
Edit: here's a simple notebook example
A general approach, assuming that you have a DataFrame column containing points, and you want to calculate distances between all of them (If you have separate columns, first combine them into (lon, lat) tuples, for instance). Name the new column coords.
import pandas as pd
import numpy as np
from geopy.distance import vincenty
# assumes your DataFrame is named df, and its lon and lat columns are named lon and lat. Adjust as needed.
df['coords'] = zip(df.lat, df.lon)
# first, let's create a square DataFrame (think of it as a matrix if you like)
square = pd.DataFrame(
np.zeros(len(df) ** 2).reshape(len(df), len(df)),
index=df.index, columns=df.index)
This function looks up our 'end' coordinates from the df DataFrame using the input column name, then applies the geopy vincenty() function to each row in the input column, using the square.coords column as the first argument. This works because the function is applied column-wise from right to left.
def get_distance(col):
end = df.ix[col.name]['coords']
return df['coords'].apply(vincenty, args=(end,), ellipsoid='WGS-84')
Now we're ready to calculate all the distances.
We're transposing the DataFrame (.T), because the loc[] method we'll be using to retrieve distances refers to index label, row label. However, our inner apply function (see above) populates a column with retrieved values
distances = square.apply(get_distance, axis=1).T
Your geopy values are (IIRC) returned in kilometres, so you may need to convert these to whatever unit you want to use using .meters, .miles etc.
Something like the following should work:
def units(input_instance):
return input_instance.meters
distances_meters = distances.applymap(units)
You can now index into your distance matrix using e.g. loc[row_index, column_index].
You should be able to adapt the above fairly easily. You might have to adjust the apply call in the get_distance function to ensure you're passing the correct values to great_circle. The pandas apply docs might be useful, in particular with regard to passing positional arguments using args (you'll need a recent pandas version for this to work).
This code hasn't been profiled, and there are probably much faster ways to do it, but it should be fairly quick for 400k distance calculations.
Oh and also
I can't remember whether geopy expects coordinates as (lon, lat) or (lat, lon). I bet it's the latter (sigh).
Update
Here's a working script as of May 2021.
import geopy.distance
# geopy DOES use latlon configuration
df['latlon'] = list(zip(df['lat'], df['lon']))
square = pd.DataFrame(
np.zeros((df.shape[0], df.shape[0])),
index=df.index, columns=df.index
)
# replacing distance.vicenty with distance.distance
def get_distance(col):
end = df.loc[col.name, 'latlon']
return df['latlon'].apply(geopy.distance.distance,
args=(end,),
ellipsoid='WGS-84'
)
distances = square.apply(get_distance, axis=1).T
I recently had to do a similar job, I ended writing a solution I consider very easy to understand and tweak to your needs, but possibly not the best/fastest:
Solution
It is very similar to what urschrei posted: assuming you want the distance between every two consecutive coordinates from a Pandas DataFrame, we can write a function to process each pair of points as the start and finish of a path, compute the distance and then construct a new DataFrame to be the return:
import pandas as pd
from geopy import Point, distance
def get_distances(coords: pd.DataFrame,
col_lat='lat',
col_lon='lon',
point_obj=Point) -> pd.DataFrame:
traces = len(coords) -1
distances = [None] * (traces)
for i in range(traces):
start = point_obj((coords.iloc[i][col_lat], coords.iloc[i][col_lon]))
finish = point_obj((coords.iloc[i+1][col_lat], coords.iloc[i+1][col_lon]))
distances[i] = {
'start': start,
'finish': finish,
'path distance': distance.geodesic(start, finish),
}
return pd.DataFrame(distances)
Usage example
coords = pd.DataFrame({
'lat': [-26.244333, -26.238000, -26.233880, -26.260000, -26.263730],
'lon': [-48.640946, -48.644670, -48.648480, -48.669770, -48.660700],
})
print('-> coords DataFrame:\n', coords)
print('-'*79, end='\n\n')
distances = get_distances(coords)
distances['total distance'] = distances['path distance'].cumsum()
print('-> distances DataFrame:\n', distances)
print('-'*79, end='\n\n')
# Or if you want to use tuple for start/finish coordinates:
print('-> distances DataFrame using tuples:\n', get_distances(coords, point_obj=tuple))
print('-'*79, end='\n\n')
Output example
-> coords DataFrame:
lat lon
0 -26.244333 -48.640946
1 -26.238000 -48.644670
2 -26.233880 -48.648480
3 -26.260000 -48.669770
4 -26.263730 -48.660700
-------------------------------------------------------------------------------
-> distances DataFrame:
start finish \
0 26 14m 39.5988s S, 48 38m 27.4056s W 26 14m 16.8s S, 48 38m 40.812s W
1 26 14m 16.8s S, 48 38m 40.812s W 26 14m 1.968s S, 48 38m 54.528s W
2 26 14m 1.968s S, 48 38m 54.528s W 26 15m 36s S, 48 40m 11.172s W
3 26 15m 36s S, 48 40m 11.172s W 26 15m 49.428s S, 48 39m 38.52s W
path distance total distance
0 0.7941932910049856 km 0.7941932910049856 km
1 0.5943709651000332 km 1.3885642561050187 km
2 3.5914909016938505 km 4.980055157798869 km
3 0.9958396130609087 km 5.975894770859778 km
-------------------------------------------------------------------------------
-> distances DataFrame using tuples:
start finish path distance
0 (-26.244333, -48.640946) (-26.238, -48.64467) 0.7941932910049856 km
1 (-26.238, -48.64467) (-26.23388, -48.64848) 0.5943709651000332 km
2 (-26.23388, -48.64848) (-26.26, -48.66977) 3.5914909016938505 km
3 (-26.26, -48.66977) (-26.26373, -48.6607) 0.9958396130609087 km
-------------------------------------------------------------------------------
As of 19th May
For anyone working with multiple geolocation data, you can adapt the above code but modify a bit to read the CSV file in your data drive. the code will write the output distances in the marked folder.
import pandas as pd
from geopy import Point, distance
def get_distances(coords: pd.DataFrame,
col_lat='lat',
col_lon='lon',
point_obj=Point) -> pd.DataFrame:
traces = len(coords) -1
distances = [None] * (traces)
for i in range(traces):
start = point_obj((coords.iloc[i][col_lat], coords.iloc[i][col_lon]))
finish = point_obj((coords.iloc[i+1][col_lat], coords.iloc[i+1][col_lon]))
distances[i] = {
'start': start,
'finish': finish,
'path distance': distance.geodesic(start, finish),
}
output = pd.DataFrame(distances)
output.to_csv('geopy_output.csv')
return output
I used the same code and generated distance data for over 50,000 coordinates.
I am struggling to calculate the distance between multiple sets of latitude and longitude coordinates. In, short, I have found numerous tutorials that either use math or geopy. These tutorials work great when I just want to find the distance between ONE set of coordindates (or two unique locations). However, my objective is to scan a data set that has 400k combinations of origin and destination coordinates. One example of the code I have used is listed below, but it seems I am getting errors when my arrays are > 1 record. Any helpful tips would be much appreciated. Thank you.
# starting dataframe is df
lat1 = df.lat1.as_matrix()
long1 = df.long1.as_matrix()
lat2 = df.lat2.as_matrix()
long2 = df.df_long2.as_matrix()
from geopy.distance import vincenty
point1 = (lat1, long1)
point2 = (lat2, long2)
print(vincenty(point1, point2).miles)
Edit: here's a simple notebook example
A general approach, assuming that you have a DataFrame column containing points, and you want to calculate distances between all of them (If you have separate columns, first combine them into (lon, lat) tuples, for instance). Name the new column coords.
import pandas as pd
import numpy as np
from geopy.distance import vincenty
# assumes your DataFrame is named df, and its lon and lat columns are named lon and lat. Adjust as needed.
df['coords'] = zip(df.lat, df.lon)
# first, let's create a square DataFrame (think of it as a matrix if you like)
square = pd.DataFrame(
np.zeros(len(df) ** 2).reshape(len(df), len(df)),
index=df.index, columns=df.index)
This function looks up our 'end' coordinates from the df DataFrame using the input column name, then applies the geopy vincenty() function to each row in the input column, using the square.coords column as the first argument. This works because the function is applied column-wise from right to left.
def get_distance(col):
end = df.ix[col.name]['coords']
return df['coords'].apply(vincenty, args=(end,), ellipsoid='WGS-84')
Now we're ready to calculate all the distances.
We're transposing the DataFrame (.T), because the loc[] method we'll be using to retrieve distances refers to index label, row label. However, our inner apply function (see above) populates a column with retrieved values
distances = square.apply(get_distance, axis=1).T
Your geopy values are (IIRC) returned in kilometres, so you may need to convert these to whatever unit you want to use using .meters, .miles etc.
Something like the following should work:
def units(input_instance):
return input_instance.meters
distances_meters = distances.applymap(units)
You can now index into your distance matrix using e.g. loc[row_index, column_index].
You should be able to adapt the above fairly easily. You might have to adjust the apply call in the get_distance function to ensure you're passing the correct values to great_circle. The pandas apply docs might be useful, in particular with regard to passing positional arguments using args (you'll need a recent pandas version for this to work).
This code hasn't been profiled, and there are probably much faster ways to do it, but it should be fairly quick for 400k distance calculations.
Oh and also
I can't remember whether geopy expects coordinates as (lon, lat) or (lat, lon). I bet it's the latter (sigh).
Update
Here's a working script as of May 2021.
import geopy.distance
# geopy DOES use latlon configuration
df['latlon'] = list(zip(df['lat'], df['lon']))
square = pd.DataFrame(
np.zeros((df.shape[0], df.shape[0])),
index=df.index, columns=df.index
)
# replacing distance.vicenty with distance.distance
def get_distance(col):
end = df.loc[col.name, 'latlon']
return df['latlon'].apply(geopy.distance.distance,
args=(end,),
ellipsoid='WGS-84'
)
distances = square.apply(get_distance, axis=1).T
I recently had to do a similar job, I ended writing a solution I consider very easy to understand and tweak to your needs, but possibly not the best/fastest:
Solution
It is very similar to what urschrei posted: assuming you want the distance between every two consecutive coordinates from a Pandas DataFrame, we can write a function to process each pair of points as the start and finish of a path, compute the distance and then construct a new DataFrame to be the return:
import pandas as pd
from geopy import Point, distance
def get_distances(coords: pd.DataFrame,
col_lat='lat',
col_lon='lon',
point_obj=Point) -> pd.DataFrame:
traces = len(coords) -1
distances = [None] * (traces)
for i in range(traces):
start = point_obj((coords.iloc[i][col_lat], coords.iloc[i][col_lon]))
finish = point_obj((coords.iloc[i+1][col_lat], coords.iloc[i+1][col_lon]))
distances[i] = {
'start': start,
'finish': finish,
'path distance': distance.geodesic(start, finish),
}
return pd.DataFrame(distances)
Usage example
coords = pd.DataFrame({
'lat': [-26.244333, -26.238000, -26.233880, -26.260000, -26.263730],
'lon': [-48.640946, -48.644670, -48.648480, -48.669770, -48.660700],
})
print('-> coords DataFrame:\n', coords)
print('-'*79, end='\n\n')
distances = get_distances(coords)
distances['total distance'] = distances['path distance'].cumsum()
print('-> distances DataFrame:\n', distances)
print('-'*79, end='\n\n')
# Or if you want to use tuple for start/finish coordinates:
print('-> distances DataFrame using tuples:\n', get_distances(coords, point_obj=tuple))
print('-'*79, end='\n\n')
Output example
-> coords DataFrame:
lat lon
0 -26.244333 -48.640946
1 -26.238000 -48.644670
2 -26.233880 -48.648480
3 -26.260000 -48.669770
4 -26.263730 -48.660700
-------------------------------------------------------------------------------
-> distances DataFrame:
start finish \
0 26 14m 39.5988s S, 48 38m 27.4056s W 26 14m 16.8s S, 48 38m 40.812s W
1 26 14m 16.8s S, 48 38m 40.812s W 26 14m 1.968s S, 48 38m 54.528s W
2 26 14m 1.968s S, 48 38m 54.528s W 26 15m 36s S, 48 40m 11.172s W
3 26 15m 36s S, 48 40m 11.172s W 26 15m 49.428s S, 48 39m 38.52s W
path distance total distance
0 0.7941932910049856 km 0.7941932910049856 km
1 0.5943709651000332 km 1.3885642561050187 km
2 3.5914909016938505 km 4.980055157798869 km
3 0.9958396130609087 km 5.975894770859778 km
-------------------------------------------------------------------------------
-> distances DataFrame using tuples:
start finish path distance
0 (-26.244333, -48.640946) (-26.238, -48.64467) 0.7941932910049856 km
1 (-26.238, -48.64467) (-26.23388, -48.64848) 0.5943709651000332 km
2 (-26.23388, -48.64848) (-26.26, -48.66977) 3.5914909016938505 km
3 (-26.26, -48.66977) (-26.26373, -48.6607) 0.9958396130609087 km
-------------------------------------------------------------------------------
As of 19th May
For anyone working with multiple geolocation data, you can adapt the above code but modify a bit to read the CSV file in your data drive. the code will write the output distances in the marked folder.
import pandas as pd
from geopy import Point, distance
def get_distances(coords: pd.DataFrame,
col_lat='lat',
col_lon='lon',
point_obj=Point) -> pd.DataFrame:
traces = len(coords) -1
distances = [None] * (traces)
for i in range(traces):
start = point_obj((coords.iloc[i][col_lat], coords.iloc[i][col_lon]))
finish = point_obj((coords.iloc[i+1][col_lat], coords.iloc[i+1][col_lon]))
distances[i] = {
'start': start,
'finish': finish,
'path distance': distance.geodesic(start, finish),
}
output = pd.DataFrame(distances)
output.to_csv('geopy_output.csv')
return output
I used the same code and generated distance data for over 50,000 coordinates.
My data object is an instance of:
class data_instance:
def __init__(self, data, tlabel):
self.data = data # 1xd numpy array
self.true_label = tlabel # integer {1,-1}
So far in code, I have a list called data_history full with data_istance and a set of centers (numpy array with shape (k,d)).
For a given data_instance new_data, I want:
1/ Get the nearest center to new_data from centers (by euclidean distance) let it be called Nearest_center.
2/ Iterate trough data_history and:
2.1/ select elements where the nearest center is Nearest_center (result of 1/) into list called neighbors.
2.2/ Get labels of object in neighbors.
Bellow is my code which work but it steel slow and I am looking for something more efficient.
My Code
For 1/
def getNearestCenter(data,centers):
if centers.shape != (1,2):
dist_ = np.sqrt(np.sum(np.power(data-centers,2),axis=1)) # This compute distance between data and all centers
center = centers[np.argmin(dist_)] # this return center which have the minimum distance from data
else:
center=centers[0]
return center
For 2/ (To optimize)
def getLabel(dataPoint, C, history):
labels = []
cluster = getNearestCenter(dataPoint.data,C)
for x in history:
if np.all(getNearestCenter(x.data,C) == cluster):
labels.append(x.true_label)
return labels
You should rather use the optimized cdist from scipy.spatial which is more efficient than calculating it with numpy,
from scipy.spatial.distance import cdist
dist = cdist(data, C, metric='euclidean')
dist_idx = np.argmin(dist, axis=1)
An even more elegant solution is to use scipy.spatial.cKDTree (as pointed out by #Saullo Castro in comments), which could be faster for a large dataset,
from scipy.spatial import cKDTree
tr = cKDTree(C)
dist, dist_idx = tr.query(data, k=1)
Found it:
dist_ = np.argmin(np.sqrt(np.sum(np.power(data[:, None]-C,2),axis=2)),axis=1)
This should return the index of the nearest center in centers from each data point of data.