Develop Reference

OSMNX not choosing shorter route

I'm using OSMnx to do routing between two nodes in a graph and an obvious route is not being returned as one of the shortest.
The relevant code:
start = (42.73843806388065, -84.51908658324757)
distance_max = 3163.834987247283
G = ox.graph_from_point(start, network_type="drive", dist=distance_max, simplify=False)#, truncate_by_edge=True)
G = ox.utils_graph.get_undirected(G)
lng_orig = -84.51858078986406
lat_orig = 42.73524265064318
lng_dest = -84.51910484455361
lat_dest = 42.753847179060145
orig = ox.distance.nearest_nodes(G, X=lng_orig, Y=lat_orig)
dest = ox.distance.nearest_nodes(G, X=lng_dest, Y=lat_dest)
k_routes = ox.k_shortest_paths(G, orig, dest, 30, weight="distance")
fig, ax = ox.plot_graph_routes(G, list(k_routes), route_colors='r', route_linewidth=5, node_size=20)
plt.show()
The top 30 routes are shown in the image below, I expected a best route to be the road to the right of the long north-south road in the top half of the image. The nodes appear to be connected.
top 30 routes shown in red

Fix
You want to use a weight of length, not distance.
k_routes = ox.k_shortest_paths(G, orig, dest, 30, weight="length")
routes with length metric
Explanation
This problem is caused by a weird interaction between the OSMnx and networkx libraries.
If you read the docs for k_shortest_paths(), it never mentions that you can pass weight="distance". It turns out that you get the behavior you're seeing if you pass any invalid weight.
k_routes = ox.k_shortest_paths(G, orig, dest, 30, weight="blah")
If you do this, the weight value put on every graph edge becomes None. OSMnx then calls networkx.shortest_simple_paths(). The docs for that function say that if every weight is None, then the graph is treated as if every edge has unit weight:
If None all edges are considered to have unit weight.
The kind of distance it's optimizing for here is graph distance. In graph distance, all of the edges between each point have distance one. A road with a curve in it has a distance equal to the number of points used to represent the curve. A long straight road has a distance of one.

Connect nodes to neighbours via line of sight (straightline)

So I have a structure that is similar to a maze, but with much more open space. And for each node in the structure, I would like to find all it's 'neighbours' (nodes are neighbours if they are in line of sight, i.e no walls blocking the straight line between them).
Here is a little image to help explain what I mean.
Black lines are walls.
Red dots are nodes.
Blue lines are lines to join neighbours (note no blue line crosses a black line).
I am currently implimenting a very naive and extremely costly brute force approch. In which I check every combination of nodes for an intersect with any of the maze walls (walls stored in 'edges').
for n1 in nodes:
for n2 in nodes:
if not intersect(n1, n2, edges):
n1.neighbours.append(n2)
n2.neighbours.append(n1)
This works fine for small structures like the pexample above. But I would love for this to be scalable to much larger structures.
So my question is if there is any way to find all the neighbours of each node much faster/ more efficiently.
Cheers :)

You might want to read Monge's book on projective geometry :)
Let's use an occlusive screen around each node, a square is computationally easy, a circle needs more math. The screen is a collection of edges that hide space from the node. The screen.occlude() method takes one of your walls as input an calculates the projection on the screen, and then extends the occlusion by adding an edge or extending one.
The result is that there are (much?) less occlusion edges then walls.
Then we invert the loops over occlusion edges and nodes to gain time. Note that the method .remove_occluded_by() only loops over remaining candidate neighbours, which is a shrinking collection. I guess the gain is from O(n^2) to O(n*log(n))
You can also have on each side of the square 2 points that are the extremes of the occlusion in that direction, possibly the corners of the virtual square. Every node outside the 4 occlusion cones is visible. Not sure this will gain time.
for n1 in nodes:
n1.occlusion = a_1_by_1_square_occlusion( centre = n1 )
for e in edges:
n1.occlusion.occlude( e )
n1.neighbours = nodes - n1 # your choice
n1.neigbours.remove_connected_walls( n1 ) # your choice
for o in n1.occlusion.edges:
n1.neighbours.remove_occluded_by( o )

Gridless A* path finding algorithm. Problem with cost function

The problem I have been attempting to solve is the path finding from a given position to a given goal for a dubins car (no backwards motion, constant velocity) with obstacles. I've attempted to implement a gridless A* algorithm with some simple obstacle avoidance. I expected the generated path to head straight towards the goal, and make minor adjustments to drive around the obstacles that it found. However, as soon as obstacles are introduced to the map, the path instead seems to get stuck at local minimum points of the algorithm's cost function.
The cost function I've implemented is the following:
f(x) = c(x) + g(x)
where c(x) is total travel cost, namely the cumulative cost of moving from node i-1 to i.
Also, g(x) is the cost of the optimal path from the current node to the end goal, which becomes a straight line as it ignores obstacles.
The cost is used as a priority value in a min heap, where each iteration pops the minimum node and generates children nodes. As the children are generated, it is controlled that they are not out of bounds, have not already been visited and are not inside an obstacle. If these controls return false, then the child is added to the heap.
I've attempted introducing a weighting factor k * g(x) to the path cost, hoping that this would "incentivize" the algorithm to move towards the goal instead of getting stuck at a point. However, this merely shifted the minimum point to another location, but still resulted in getting stuck.
I will include my code implementation of the A* algorithm below:
# Description: Pathfinding algorithm, iteratively generates new neighbouring
# nodes and selects the cheapest of these through utilizing a min heap.
# In: Car class object, a node as starting point.
# Out: The finishing node, with attached parent pointers.
def Astar(car, current):
minHeap = [] #initialize heap as list
h.heappush(minHeap, current) #push initial node onto heap
heapCount = 1 #add upon pushes to heap, subtract upon pop
# Iterate through nodes in priority queue
while not ((goal(car, current)) or heapCount == 0):
current = h.heappop(minHeap)
heapCount -= 1
for phi in [-m.pi/4, 0, m.pi/4]: #Full turns or straight are optimal, according to Pontryagins maximum principle
#calculate new values for each phi (steering angle)
xn, yn, thetan = step(car, current.x, current.y, current.theta, phi)
#control feasibility of position
if validCheck(car, xn, yn, current):
#calculate costs for these directives
costC = current.travelled + m.hypot(current.x - xn, current.y - yn) #cost of travel from start position
costG = m.hypot(car.xt - xn, car.yt - yn) #current optimal distance to goal
totalCost = costC + costG
#renew time stamp
newTime = current.time + 0.01
#create child from new data
child = Node(xn, yn, thetan, phi, totalCost, costC, newTime, current)
#push child onto heap
h.heappush(minHeap, child)
heapCount += 1
return current
Note that car is a class which includes certain attributes:
x0 : float: initial x-position [m]
y0 : float: initial y-position [m]
xt : float: target x-position [m]
yt : float: target y-position [m]
xlb : float: minimum x-position [m]
xub : float: maximum x-position [m]
ylb : float: minimum y-position [m]
yub : float: maximum y-position [m]
obs : list: list of tuples for each obstacle obs[i]
It also includes a method step which can generate a new heading angle and position when given a steering angle and previous heading and position.
Any advice or help regarding this problem, why it is occurring and what I can do to improve the path finding would very much be appreciated.

I don't have a solution ready, but an explanation what's going on and and maybe a hint what you can do.
Analysis
The A* algorithm is for graph searching and, given a decent cost function, can greatly reduce the search space when compared with uninformed strategies like BFS. But still, the size of the problem graph matters.
In your code, I see a time increment of 0.01, and I read that as a hint that you are doing very small steps from parent to child nodes. That surely makes sense, to most closely approximating a smooth, non-quantized movement. But at the same time, it results in a huge growth of your problem graph.
Without obstacles, A* will still handle that huge graph gracefully. It will postpone all deviations from the straight line, as their cost will be higher than the node on the straight line. Your heap will grow (have some debug output show you its size...), but most nodes will never be explored further.
With obstacles, the game changes drastically. Let's say, there's an obstacle so that the resulting best path is 1.00 units longer than the straight line. Then A* will explore all nonsense paths, starting from somewhere on the line from start to obstacle, arbitrarily turning left or right until these paths reach an additional length of 1.00. There will be lots of these useless paths, and A* gets stuck in exploring nonsense.
Suggestion
I'd have the A* operate on a higher level.
I guess your obstacles are polygons. So the resulting total path will either ignore an obstacle or touch it at one of its corners. The elements between the touching points will start at a touching point with some heading direction, consist of an initial full-turn part, then a straight part, and then a final full-turn part, and then arrive at the next touching point with some (different) heading (to be honest, I'm not absolutely sure that this turn-straight-turn pattern will really cover all possible situations). Given start and end points and the desired end heading of such an element, you can compute the parts using some geometry, and by the way, check for collisions.
You can't know in advance the optimum heading when passing some touching point, so you'd have to check all possible headings.
I'd take these elements as the steps to be explored by A*.
So, that's how I'd apply A* to your task:
To compute the children of a node, for all other polygon corners and all headings at that corner, compute the element from the parent corner to the other corner, resulting in the given heading there. Check if such an element is geometrically possible and does not collide with some obstacle.
As the cost function, accumulate the length travelled so far, and then add the shortest obstacle-ignoring path to the target. This can either be the straight Pythagorean distance, or a more elaborate distance, taking into account the neccessary initial turn from the current heading to facing the target.

Is it possible to fit a coordinate to a street in OSMnx?

OSMnx provides solution to calculate the shortest path between two nodes, but I would like to the same with points on streets (I have GPS coordinates recorded from vehicles). I know there is also a method to get the closest node, but I have two question for this problem of mine.
i) When closest node computed is the street where the point is also taken into consideration? (I assume not)
ii) If I wanted to implement something like this, I like to know how a street (edge) is represented as a curve (Bézier curve maybe?). Is it possible to get the curve (or the equation of the curve) of an edge?
I asked this question here, because the guidelines for contributing of OSMnx asked it.

Streets and node in OSMnx are shapely.geometry.LineString, and shapely.geometry.Point objects, so there is no curve, only sequence of coordinates. The technical term for what you described is Map Matching. There are different ways of map matching, the simplest one being geometric map matching in which you find nearest geometry (node or edge) to the GPS point. point to point map matching can be easily achieved using built-in osmnx function ox.get_nearest_node(). If you have a luxury of dense GPS tracks, this approach could work reasonably good. For point to line map matching you have to use shapely functions. The problem with this approach is that it is very slow. you can speed up the algorithm using spatial index, but still, it will not be fast enough for most purposes. Note that geometric map matching are least accurate among all approaches. I wrote a function a few weeks ago that does simple point to line map matching using edge GeoDataFrame and node GeoDataFrame that you can get from OSMnx. I abandoned this idea and now I am working on a new algorithm (hopefully much faster), which I will publish on GitHub upon completion. Meanwhile, this may be of some help for you or someone else, so I post it here. This is an early version of abandoned code, not tested enough and not optimized. give it a try and let me know if it works for you.
def GeoMM(traj, gdfn, gdfe):
"""
performs map matching on a given sequence of points
Parameters
----------
Returns
-------
list of tuples each containing timestamp, projected point to the line, the edge to which GPS point has been projected, the geometry of the edge))
"""
traj = pd.DataFrame(traj, columns=['timestamp', 'xy'])
traj['geom'] = traj.apply(lambda row: Point(row.xy), axis=1)
traj = gpd.GeoDataFrame(traj, geometry=traj['geom'], crs=EPSG3740)
traj.drop('geom', axis=1, inplace=True)
n_sindex = gdfn.sindex
res = []
for gps in traj.itertuples():
tm = gps[1]
p = gps[3]
circle = p.buffer(150)
possible_matches_index = list(n_sindex.intersection(circle.bounds))
possible_matches = gdfn.iloc[possible_matches_index]
precise_matches = possible_matches[possible_matches.intersects(circle)]
candidate_nodes = list(precise_matches.index)
candidate_edges = []
for nid in candidate_nodes:
candidate_edges.append(G.in_edges(nid))
candidate_edges.append(G.out_edges(nid))
candidate_edges = [item for sublist in candidate_edges for item in sublist]
dist = []
for edge in candidate_edges:
# get the geometry
ls = gdfe[(gdfe.u == edge[0]) & (gdfe.v == edge[1])].geometry
dist.append([ls.distance(p), edge, ls])
dist.sort()
true_edge = dist[0][1]
true_edge_geom = dist[0][2].item()
pp = true_edge_geom.interpolate(true_edge_geom.project(p)) # projected point
res.append((tm, pp, true_edge, true_edge_geom))
return res

OSMnx was recently updated since there have been a couple of requests in this direction (see https://github.com/gboeing/osmnx/pull/234 and references therein). So in the last update, you'll find a function like this:
ox.get_nearest_edge(G, (lat, lon))
It will give you the ID of the nearest edge, which is much better than nearest nodes.
However, I think it is more useful to also get the actual distance of the nearest edge in order to check whether or not your data point is on the road or a few thousand meters apart...
To do this, I followed the implementation from https://github.com/gboeing/osmnx/pull/231/files
# Convert Graph to graph data frame
gdf = ox.graph_to_gdfs(G, nodes=False, fill_edge_geometry=True)
# extract roads and some properties
roads = gdf[["geometry", "u", "v","ref","name","highway","lanes"]].values.tolist()
# calculate and attach distance
roads_with_distances = [(road, ox.Point(tuple(reversed((lat,lon)))).distance(road[0])) for road in roads]
# sort by distance
roads_with_distances = sorted(roads_with_distances, key=lambda x: x[1])
# Select closest road
closest_road = roads_with_distances[0]
# Check whether you are actually "on" the road
if closest_road[1] < 0.0001: print('Hit the road, Jack!')
I have the impression that a distance on the order of $10^{-5}$ means that the coordinate is actually "on" the road.

Ellipsoid equation containing numerous points

I have a large quantity of pixel colors (96 thousands different colors):
And I want to get some kind of a mathematically-defined probability region like in this question:
The main obstacle I see right now – all methods on Google are mainly about visualisations and about two-dimensional spaces, yet there is no algorithm for finding coefficients of an equation like:
a1x2 + b1y2 + c1y2 + a2xy + b2xz + c2yz + a3x + b3y + c3z = 0
And this paper is too difficult for me to implement it in python. :(
Anyway, what I just want is to determine if some pixel is more-or-less lies within the diapason I have.
I tried making it using scikit clustering, but I failed due to having only one
set of data, probably. And creating an array 2563 elements
representing each pixel color seems a wrong way.
I wonder if there is an easy way to determine boundaries of this point cluster?
Or, maybe I'm just overthinking it and there is something like OpenCV
cv2.inRange() function?

this can be solved by optimization and fitting of the ellipsoid polynomial. However I would start with geometrical approach which is much faster:
find avg point position
that will be the center of your ellipsoid
p0 = sum (p[i]) / n // average
i = { 0,1,2,3,...,n-1 } // of all points
If your point density is not homogenuous then it is safer to use bounding box center instead. So find xmin,ymin,zmin,xmax,ymax,zmax and the middle between them is your center.
find most distant point to center
that will give you main semi axis
pa = p[j];
|p[j]-p0| >= |p[i]-p0| // max
i = { 0,1,2,3,...,n-1 } // of all points
find second semi-axises
so vector pa-p0 is normal to plane in which the other semi-axises should be. So find most distant point to p0 from that plane:
pb = p[j];
|p[j]-p0| >= |p[i]-p0| // max
dot(pa-p0,p[j]-p0) == 0 // but inly if inside plane
i = { 0,1,2,3,...,n-1 } // from all points
beware that the result of dot product may not be precisely zero so it is better to test against something like this:
|dot(pa-p0,p[j]-p0)| <= 1e-3
You can use any threshold you want (should be based on the ellipsoid size).
find last semi-axis
So we know that last semi-axis should be perpendicular to both
(pa-p0) AND (pb-p0)
So find point such that:
pc = p[j];
|p[j]-p0| >= |p[i]-p0| // max
dot(pa-p0,p[j]-p0) == 0 // but inly if inside plane
dot(pb-p0,p[j]-p0) == 0 // and perpendicular also to b semi-axis
i = { 0,1,2,3,...,n-1 } // from all points
Ellipsoid
Now you have all the parameters you need to form your ellipsoid. vectors
(pa-p0),(pb-p0),(pc-p0)
are the basis vectors of your ellipsoid (you can make them perpendicular by using cross product). Their size gives you the radiuses. And p0 is the center. You can also use this parametric equation:
a=pa-p0;
b=pb-p0;
c=pc-p0;
p(u,v) = p0 + a*cos(u)*cos(v)
+ b*cos(u)*sin(v)
+ c*sin(u);
u = < -0.5*PI , +0.5*PI >
v = < 0.0 , 2.0*PI >
This whole process is just O(n) and the results can be used as start point for both optimization and fitting to speed them up without the loss of accuracy. If you want to further improve accuracy See:
How approximation search works
The sub links shows you examples of fitting ...
You can also take a look at this:
Algorithms: Ellipse matching
which is basically similar to your task but only in 2D still may bring you some ideas.

Here is unstrict solution with fast and simple random search approach*. Best side - no heavy linear algebra library required**. Seems it worked fine for mesh collision detection.
Is assumes that ellipsoid center matches cloud center and then uses some sort of mirrored average to search for main axis.
Full working code is slightly bigger and placed on git, idea of main axis search is here:
np.random.shuffle(pts)
pts_len = len(pts)
pt_average = np.sum(pts, axis = 0) / pts_len
vec_major = pt_average * 0
minor_max, major_max = 0, 0
# may be improved with overlapped pass,
for pt_cur in pts:
vec_cur = pt_cur - pt_average
proj_len, rej_len = proj_length(vec_cur, vec_major)
if proj_len < 0:
vec_cur = -vec_cur
vec_major += (vec_cur - vec_major) / pts_len
major_max = max(major_max, abs(proj_len))
minor_max = max(minor_max, rej_len)
It can be improved/optimized even more at some points. Examples what it will produce:
And full experiment code with plots
*i.e. adjusting code lines randomly until they work
**was actually reason to figure out this solution