Develop Reference

Create line from multiple coorinates and get points from that line with a percentage

I have multiple GPS Coordinate Points I would like to create a line from in python. The points aren't in a straight line but are exact enough to connect them with straight lines.
I know how to connect one point to another, but not how I would connect multiple of these singular lines to a longer one and then get a point based on the percentage of the whole line.
I use this code to get the percentage of a singular line:
def pointAtPercent(p0, p1, percent):
if p0.x != p1.x:
x = p0.x + percent * (p1.x - p0.x)
else:
x = p0.x;
if p0.y != p1.y:
y = p0.y + percent * (p1.y - p0.y)
else:
y = p0.y
p = point()
p.x = x
p.y = y
return p;
Here is an example list:
[ 10.053417,
53.555737,
10.053206,
53.555748,
10.052497,
53.555763,
10.051125,
53.555757,
10.049193,
53.555756,
10.045511,
53.555762,
10.044863,
53.555767,
10.044319,
53.555763,
10.043685,
53.555769,
10.042765,
53.555759,
10.04201,
53.555756,
10.041919,
53.555757,
10.041904,
53.555766
]

You could create a list of x,y pairs and access the GPS point based on the length of the list:
points = [
10.053417, 53.555737, 10.053206, 53.555748, 10.052497, 53.555763, 10.051125,
53.555757, 10.049193, 53.555756, 10.045511, 53.555762, 10.044863, 53.555767,
10.044319, 53.555763, 10.043685, 53.555769, 10.042765, 53.555759, 10.04201,
53.555756, 10.041919, 53.555757, 10.041904, 53.555766
]
points = [(points[i], points[i + 1]) for i in range(0, len(points) - 1, 2)]
def pointAtPercent(points, percent):
lstIndex = int(
len(points) / 100. * percent
) # possibly creates some rounding issues !
print(points[lstIndex - 1])
pointAtPercent(points, 10)
pointAtPercent(points, 27.5)
pointAtPercent(points, 50)
pointAtPercent(points, 100)
Out:
(10.053417, 53.555737)
(10.052497, 53.555763)
(10.045511, 53.555762)
(10.041904, 53.555766)

The basic algorithm is this:
Determine the lengths of each segment. You are in spherical polar coordinates (assuming the Earth is a sphere, which it isn't (see WGS 84 if you need more precision)), so you can do this.
def great_circle_distance(lat_0, lon_0, lat_1, lon_1):
return math.acos(
math.sin(lat_0) * math.sin(lat_1)
+ math.cos(lat_0) * math.cos(lat_1) * math.cos(lon_1 - lon_0)
)
radian_points = [(math.radians(p.x), math.radians(p.y)) for p in points]
lengths = [
great_circle_distance(*p0, *p1) for p0, p1 in zip(radian_points, radian_points[1:])
]
path_length = sum(lengths)
Given a percentage, you can work out how far along the path it is.
distance_along = percentage * path_length
Find the index of the correct segment.
# Inefficient but easy (consider bisect.bisect with a stored list of sums)
index = max(
next(i for i in range(len(lengths) + 1) if sum(lengths[:i]) >= distance_along) - 1,
0,
)
Then use your original algorithm.
point = pointAtPercent(
points[index],
points[index + 1],
(distance_along - sum(lengths[:index])) / lengths[index],
)

Check if values are inside a specific area around a predefined linear function

My research on solving my issue was unfortunately unsuccessful and I hope you can help me. I have defined the following linear function for a straight line
x = [298358.3258395831, 298401.1779180078]
y = [5625243.628060675, 5625347.074197255]
m, b = np.polyfit(x, y, 1)
and I want to check, if values in an array are inside an area around this function. The area around the function could look like this:
I couldn't find a solution how to create an area around this straight line function and so I couldn't find a way how to check if the points in the array are inside or outside of the area.
Thanks in advance!

For a line given by the equation ax + by + c = 0, the distance from a point A = (x_a,y_a) to this line is given by the following formula :
dist = np.abs(a * x_a + b * y_a + c) / np.sqrt(a**2 + b**2)
Source here.
That way, if you have an array of points and a threshold above which you consider your points to be too far away from your line, you can simply do :
array_points = ... # Format : [[x_1,y_1], [x_2,y_2],...]
a, b, c = ... # Your line's parameters here
thresh = 1e-2 # For example
def is_close_line(array, threshold) :
array_dist = np.abs(a * array[:,0] + b * array[:,1] + c) / np.sqrt(a**2 + b**2)
return (array_dist < threshold)
is_close_line(array_points, thresh) will then output a boolean array, where the i-th item indicates wether or not the i-th element of array_points is close to your line.

A possible solution could be:
Take a distance and project it onto the x axis
Build two new lines by shifting your line according to the distance projection
Compare a new point with the so-built lines
Here a sample code (note that m=0 should be handled differently):
def near_line(point, dist, m, b):
# Data preparation
x, y = point
dist = abs(dist)
if m != 0:
# Case positive ramp
dist_projection = dist/np.sin(np.arctan(abs(m)))
return m*(x-dist_projection)+b < y < m*(x+dist_projection)+b
else:
# Case horizontal line
return b-dist < y < b+dist
print( near_line([298359, 5625244], dist=5, m=m, b=b) )
print( near_line([298400, 5625250], dist=5, m=m, b=b) )
Out:
True
False

My answer is based on this post where the concept of cross-product and norm are considered. This solution applies also to an infinite line like yours, where it's constructed starting from two points.
import numpy as np
def dist_array_line(a, l1, l2, threshold):
"""
a : numpy.array
Array of points of shape (M,2)
M is the total number of points to test
l1 : numpy.array
Array of shape (2,) indicating the first point standing on the line
l2 : numpy.array
Array of shape (2,) indicating the second point standing on the line
threshold : float
Maximum distance allowed between line and points
Returns
numpy.array
Array of shape (M,) with True/False indicating whether the points in `a`
are within/outside the rectangle around the line
"""
distances = np.abs(np.cross(a - l1, a - l2)) / np.linalg.norm(l1 - l2)
return (distances < threshold)
If you want to return the actual distances instead of a True/False array, just make the function return the distances object.
Example
# points on the line
p1 = np.array([298358.3258395831, 5625243.628060675])
p2 = np.array([298401.1779180078, 5625347.074197255])
# array of points to test
my_arr = np.array([
[298359.3258395831, 5625243.628060675],
[298368.3258395831, 5625243.628060675],
[(p1[0] + p2[0]) / 2, (p1[1] + p2[1]) / 2]
])
dist_array_line(my_arr, p1, p2, threshold=5.)
# array([ True, False, True])

Need to speed up very slow loop for image manipulation on Python

I am currently completing a program in Pyhton (3.6) as per internal requirement. As part of it, I am having to loop through a colour image (3 bytes per pixel, R, G & B) and distort the image pixel by pixel.
I have the same code in other languages (C++, C#), and non-optimized code executes in about two seconds, while optimized code executes in less than a second. By non-optimized code I mean that the matrix multiplication is performed by a 10 line function I implemented. The optimized version just uses external libraries for multiplication.
In Python, this code takes close to 300 seconds. I can´t think of a way to vectorize this logic or speed it up, as there are a couple of "if"s inside the nested loop. Any help would be greatly appreciated.
import numpy as np
#for test purposes:
#roi = rect.rect(0, 0, 1200, 1200)
#input = DCImage.DCImage(1200, 1200, 3)
#correctionImage = DCImage.DCImage(1200,1200,3)
#siteToImage= np.zeros((3,3), np.float32)
#worldToSite= np.zeros ((4, 4))
#r11 = r12 = r13 = r21 = r22 = r23 = r31 = r32 = r33 = 0.0
#xMean = yMean = zMean = 0
#tx = ty = tz = 0
#epsilon = np.finfo(float).eps
#fx = fy = cx = cy = k1 = k2 = p1 = p2 = 0
for i in range (roi.x, roi.x + roi.width):
for j in range (roi.y , roi.y + roi.height):
if ( (input.pixels [i] [j] == [255, 0, 0]).all()):
#Coordinates conversion
siteMat = np.matmul(siteToImage, [i, j, 1])
world =np.matmul(worldToSite, [siteMat[0], siteMat[1], 0.0, 1.0])
xLocal = world[0] - xMean
yLocal = world[1] - yMean
zLocal = z_ortho - zMean
#From World to camera
xCam = r11*xLocal + r12*yLocal + r13*zLocal + tx
yCam = r21*xLocal + r22*yLocal + r23*zLocal + ty
zCam = r31*xLocal + r32*yLocal + r33*zLocal + tz
if (zCam > epsilon or zCam < -epsilon):
xCam = xCam / zCam
yCam = yCam / zCam
#// DISTORTIONS
r2 = xCam*xCam + yCam*yCam
a1 = 2*xCam*yCam
a2 = r2 + 2*xCam*xCam
a3 = r2 + 2*yCam*yCam
cdist = 1 + k1*r2 + k2*r2*r2
u = int((xCam * cdist + p1 * a1 + p2 * a2) * fx + cx + 0.5)
v = int((yCam * cdist + p1 * a3 + p2 * a1) * fy + cy + 0.5)
if (u>=0 and u<correctionImage.width and v>=0 and v < correctionImage.height):
input.pixels [i] [j] = correctionImage.pixels [u][v]

You normally vectorize this kind of thing by making a displacement map.
Make a complex image where each pixel has the value of its own coordinate, apply the usual math operations to compute whatever transform you want, then apply the map to your source image.
For example, in pyvips you might write:
import sys
import pyvips
image = pyvips.Image.new_from_file(sys.argv[1])
# this makes an image where pixel (0, 0) (at the top-left) has value [0, 0],
# and pixel (image.width, image.height) at the bottom-right has value
# [image.width, image.height]
index = pyvips.Image.xyz(image.width, image.height)
# make a version with (0, 0) at the centre, negative values up and left,
# positive down and right
centre = index - [image.width / 2, image.height / 2]
# to polar space, so each pixel is now distance and angle in degrees
polar = centre.polar()
# scale sin(distance) by 1/distance to make a wavey pattern
d = 10000 * (polar[0] * 3).sin() / (1 + polar[0])
# and back to rectangular coordinates again to make a set of vectors we can
# apply to the original index image
distort = index + d.bandjoin(polar[1]).rect()
# distort the image
distorted = image.mapim(distort)
# pick pixels from either the distorted image or the original, depending on some
# condition
result = (d.abs() > 10 or image[2] > 100).ifthenelse(distorted, image)
result.write_to_file(sys.argv[2])
That's just a silly wobble pattern, but you can swap it for any distortion you want. Then run as:
$ /usr/bin/time -f %M:%e ./wobble.py ~/pics/horse1920x1080.jpg x.jpg
54572:0.31
300ms and 55MB of memory on this two-core, 2015 laptop to make:

After much testing, the only way to speed the function without writing it in C++ was dissassembling it and vectorizing it. The way to do it in this particular instance is to create an array with the valid indexes at the beginning of the funcion and use them as tuples to index the final solution.
subArray[roi.y:roi.y+roi.height,roi.x:roi.x+roi.width,] = input.pixels[roi.y:roi.y+roi.height,roi.x:roi.x+roi.width,]
#Calculate valid XY indexes
y_index, x_index = np.where(np.all(subArray== np.array([255,0,0]), axis=-1))
#....
#do stuff
#....
#Join result values with XY indexes
ij_xy = np.column_stack((i, j, y_index, x_index))
#Only keep valid ij values
valids_ij_xy = ij_xy [(ij_xy [:,0] >= 0) & (ij_xy [:,0] < correctionImage.height) & (ij_xy [:,1] >= 0) & (ij_xy [:,1] < correctionImage.width)]
#Assign values
input.pixels [tuple(np.array(valids_ij_xy [:,2:]).T)] = correctionImage.pixels[tuple(np.array(valids_ij_xy [:,:2]).T)]

Integrating 2D data over an irregular grid in python

So I have 2D function which is sampled irregularly over a domain, and I want to calculate the volume underneath the surface. The data is organised in terms of [x,y,z], taking a simple example:
def f(x,y):
return np.cos(10*x*y) * np.exp(-x**2 - y**2)
datrange1 = np.linspace(-5,5,1000)
datrange2 = np.linspace(-0.5,0.5,1000)
ar = []
for x in datrange1:
for y in datrange2:
ar += [[x,y, f(x,y)]]
for x in xrange2:
for y in yrange2:
ar += [[x,y, f(x,y)]]
val_arr1 = np.array(ar)
data = np.unique(val_arr1)
xlist, ylist, zlist = data.T
where np.unique sorts the data in the first column then the second. The data is arranged in this way as I need to sample more heavily around the origin as there is a sharp feature that must be resolved.
Now I wondered about constructing a 2D interpolating function using scipy.interpolate.interp2d, then integrating over this using dblquad. As it turns out, this is not only inelegant and slow, but also kicks out the error:
RuntimeWarning: No more knots can be added because the number of B-spline
coefficients already exceeds the number of data points m.
Is there a better way to integrate data arranged in this fashion or overcoming this error?

If you can sample the data with high enough resolution around the feature of interest, then more sparsely everywhere else, the problem definition then becomes how to define the area under each sample. This is easy with regular rectangular samples, and could likely be done stepwise in increments of resolution around the origin. The approach I went after is to generate the 2D Voronoi cells for each sample in order to determine their area. I pulled most of the code from this answer, as it had almost all the components needed already.
import numpy as np
from scipy.spatial import Voronoi
#taken from: # https://stackoverflow.com/questions/28665491/getting-a-bounded-polygon-coordinates-from-voronoi-cells
#computes voronoi regions bounded by a bounding box
def square_voronoi(xy, bbox): #bbox: (min_x, max_x, min_y, max_y)
# Select points inside the bounding box
points_center = xy[np.where((bbox[0] <= xy[:,0]) * (xy[:,0] <= bbox[1]) * (bbox[2] <= xy[:,1]) * (bbox[2] <= bbox[3]))]
# Mirror points
points_left = np.copy(points_center)
points_left[:, 0] = bbox[0] - (points_left[:, 0] - bbox[0])
points_right = np.copy(points_center)
points_right[:, 0] = bbox[1] + (bbox[1] - points_right[:, 0])
points_down = np.copy(points_center)
points_down[:, 1] = bbox[2] - (points_down[:, 1] - bbox[2])
points_up = np.copy(points_center)
points_up[:, 1] = bbox[3] + (bbox[3] - points_up[:, 1])
points = np.concatenate((points_center, points_left, points_right, points_down, points_up,), axis=0)
# Compute Voronoi
vor = Voronoi(points)
# Filter regions (center points should* be guaranteed to have a valid region)
# center points should come first and not change in size
regions = [vor.regions[vor.point_region[i]] for i in range(len(points_center))]
vor.filtered_points = points_center
vor.filtered_regions = regions
return vor
#also stolen from: https://stackoverflow.com/questions/28665491/getting-a-bounded-polygon-coordinates-from-voronoi-cells
def area_region(vertices):
# Polygon's signed area
A = 0
for i in range(0, len(vertices) - 1):
s = (vertices[i, 0] * vertices[i + 1, 1] - vertices[i + 1, 0] * vertices[i, 1])
A = A + s
return np.abs(0.5 * A)
def f(x,y):
return np.cos(10*x*y) * np.exp(-x**2 - y**2)
#sampling could easily be shaped to sample origin more heavily
sample_x = np.random.rand(1000) * 10 - 5 #same range as example linspace
sample_y = np.random.rand(1000) - .5
sample_xy = np.array([sample_x, sample_y]).T
vor = square_voronoi(sample_xy, (-5,5,-.5,.5)) #using bbox from samples
points = vor.filtered_points
sample_areas = np.array([area_region(vor.vertices[verts+[verts[0]],:]) for verts in vor.filtered_regions])
sample_z = np.array([f(p[0], p[1]) for p in points])
volume = np.sum(sample_z * sample_areas)
I haven't exactly tested this, but the principle should work, and the math checks out.

How to calculate the midpoint of several geolocations in python

Is there's a library or a way to calculate the center point for several geolocations points?
This is my list of geolocations based in New York and want to find the approximate midpoint geolocation
L = [
(-74.2813611,40.8752222),
(-73.4134167,40.7287778),
(-74.3145014,40.9475244),
(-74.2445833,40.6174444),
(-74.4148889,40.7993333),
(-73.7789256,40.6397511)
]

After the comments I received and comment from HERE
With coordinates that close to each other, you can treat the Earth as being locally flat and simply find the centroid as though they were planar coordinates. Then you would simply take the average of the latitudes and the average of the longitudes to find the latitude and longitude of the centroid.
lat = []
long = []
for l in L :
lat.append(l[0])
long.append(l[1])
sum(lat)/len(lat)
sum(long)/len(long)
-74.07461283333332, 40.76800886666667

Based on: https://gist.github.com/tlhunter/0ea604b77775b3e7d7d25ea0f70a23eb
Assume you have a pandas DataFrame with latitude and longitude columns, the next code will return a dictionary with the mean coordinates.
import math
x = 0.0
y = 0.0
z = 0.0
for i, coord in coords_df.iterrows():
latitude = math.radians(coord.latitude)
longitude = math.radians(coord.longitude)
x += math.cos(latitude) * math.cos(longitude)
y += math.cos(latitude) * math.sin(longitude)
z += math.sin(latitude)
total = len(coords_df)
x = x / total
y = y / total
z = z / total
central_longitude = math.atan2(y, x)
central_square_root = math.sqrt(x * x + y * y)
central_latitude = math.atan2(z, central_square_root)
mean_location = {
'latitude': math.degrees(central_latitude),
'longitude': math.degrees(central_longitude)
}

Considering that you are using signed degrees format (more), simple averaging of latitude and longitudes would create problems for even small regions near to antimeridian (i.e. + or - 180-degree longitude) due to discontinuity of longitude value at this line (sudden jump between -180 to 180).
Consider two locations whose longitudes are -179 and 179, their mean would be 0, which is wrong.

This link can be useful, first convert lat/lon into an n-vector, then find average. A first stab at converting the code into Python
is below
import numpy as np
import numpy.linalg as lin
E = np.array([[0, 0, 1],
[0, 1, 0],
[-1, 0, 0]])
def lat_long2n_E(latitude,longitude):
res = [np.sin(np.deg2rad(latitude)),
np.sin(np.deg2rad(longitude)) * np.cos(np.deg2rad(latitude)),
-np.cos(np.deg2rad(longitude)) * np.cos(np.deg2rad(latitude))]
return np.dot(E.T,np.array(res))
def n_E2lat_long(n_E):
n_E = np.dot(E, n_E)
longitude=np.arctan2(n_E[1],-n_E[2]);
equatorial_component = np.sqrt(n_E[1]**2 + n_E[2]**2 );
latitude=np.arctan2(n_E[0],equatorial_component);
return np.rad2deg(latitude), np.rad2deg(longitude)
def average(coords):
res = []
for lat,lon in coords:
res.append(lat_long2n_E(lat,lon))
res = np.array(res)
m = np.mean(res,axis=0)
m = m / lin.norm(m)
return n_E2lat_long(m)
n = lat_long2n_E(30,20)
print (n)
print (n_E2lat_long(np.array(n)))
# find middle of france and libya
coords = [[30,20],[47,3]]
m = average(coords)
print (m)

I would like to improve on the #BBSysDyn'S answer.
The average calculation can be biased if you are calculating the center of a polygon with extra vertices on one side. Therefore the average function can be replaced with centroid calculation explained here
def get_centroid(points):
x = points[:,0]
y = points[:,1]
# Solving for polygon signed area
A = 0
for i, value in enumerate(x):
if i + 1 == len(x):
A += (x[i]*y[0] - x[0]*y[i])
else:
A += (x[i]*y[i+1] - x[i+1]*y[i])
A = A/2
#solving x of centroid
Cx = 0
for i, value in enumerate(x):
if i + 1 == len(x):
Cx += (x[i]+x[0]) * ( (x[i]*y[0]) - (x[0]*y[i]) )
else:
Cx += (x[i]+x[i+1]) * ( (x[i]*y[i+1]) - (x[i+1]*y[i]) )
Cx = Cx/(6*A)
#solving y of centroid
Cy = 0
for i , value in enumerate(y):
if i+1 == len(x):
Cy += (y[i]+y[0]) * ( (x[i]*y[0]) - (x[0]*y[i]) )
else:
Cy += (y[i]+y[i+1]) * ( (x[i]*y[i+1]) - (x[i+1]*y[i]) )
Cy = Cy/(6*A)
return Cx, Cy
Note: If it is a polygon or more than 2 points, they must be listed in order that the polygon or shape would be drawn.