Develop Reference

find the least euclidean distance from the given points

I have a list of dictionaries. The key is item_cd and value is location_coordinates.
I would like to find the euclidian distance of these locations and create a separate similar list based on the proximity of their locations.
Steps
The initial distance will be calculated from the origin i.e. (0,0).
The closest item will be added to a separate list suppose it is {5036885955: [90.0, 61.73]} in this case.
Now I would like to find the closest order from the centroid of the previous orders. In this case, the centroid will be [90.0, 61.73] of order {5036885955: [90.0, 61.73]} as there is only one coordinate at the moment. The next closest order from the centroid of the previous order will be {5036885984: [86.0, 73.03]}
I would like to repeat the above steps until the list is empty
Input :
[{5036885850: [92.0, 88.73]}, {5036885955: [90.0, 61.73]}, {5036885984: [86.0, 73.03]}, {5036885998: [102.0, 77.54]}]
What I have tried :
def calculate_centroid(self, locations: list) -> list:
"""
get the centroid of the order
param: Location object
return: centroid of an order
"""
x, y = [p[0] for p in locations], [p[1] for p in locations]
centroid = [round(sum(x) / len(locations), 2), round(sum(y) / len(locations), 2)]
return centroid
def get_closest_order_to_origin(self, order_centroid_list: list) -> list:
"""
get the next closest order
"""
initial_order = [[0, 0]]
next_order = []
centroids = []
for order in order_centroid_list:
for shipping_req_key, centroid in order.items():
distance = math.sqrt(((initial_order[0][0] - centroid[0]) ** 2) + ((initial_order[0][1] - centroid[1]) ** 2))
I am not sure how should I proceed further. I would appreciate your feedback

You can use numpy.linalg.norm() to calculate the Euclidean distance between two numpy array. So you can declare those position arrays as a numpy array and apply this function to calculate distance.
Let's see this step by step.
Declared initial current_pos as origin (0,0). Extracted the points sequentially in the list named pos_list. Declared result_list as empty list to fetch results.
From the pos_list each positions can be taken and Euclidean distance with current_pos is calculated and stored in the dist_list.
The min(dist_list) gives the minimum distance min_dist. The corresponding index of min_dist can be fetched from dist_list and the relevant position and entry of the input data can be identified and processed (i.e. removing from data and appending to result_list)
The whole process continues until the data becomes empty.
The whole implementation:
import numpy as np
data = [{5036885850: [92.0, 88.73]}, {5036885955: [90.0, 61.73]}, {5036885984: [86.0, 73.03]}, {5036885998: [102.0, 77.54]}]
pos_list = [list(item.values())[0] for item in data]
current_pos = np.array([0,0])
result_list = []
while (len(data) != 0):
dist_list = [np.linalg.norm(current_pos - np.array(pos)) for pos in pos_list]
min_dist = min(dist_list)
item_index = dist_list.index(min_dist)
to_be_moved = data[item_index]
result_list.append(to_be_moved)
# current_pos = pos_list[item_index]
pos_list.remove(pos_list[item_index])
data.remove(to_be_moved)
print(result_list)
You're requirement was not clear in the question, so I'm handling two cases.
First case: Arrange the points in such an order such that the distances of each points from the origin is increasing. The above code works for this scenario and this matches your expected output.
Second case: Arrange the points a,b,c,d,... in such a way so that a is nearest to origin, b is nearest to a, c is nearest to d and so on. To see this case in action just revert the commented line.
NOTE: If any type of sort or positional modification is done, this solution will break.
UPDATE
According to your last added condition, you need to calculate the centroid of all the points in the resulting order, then find the order that is closest to the centroid.
You can pass the list of points that has been already added to the result find the centroid with the following function:
def get_centroid(lst):
arr = np.array(lst)
length = arr.shape[0]
if length == 0:
return 0,0
else:
sum_x = np.sum(arr[:, 0])
sum_y = np.sum(arr[:, 1])
return sum_x/float(length), sum_y/float(length)
to do so, keep a track of the point that is added to the final result and removed from the pos_list in another list called pos_removed_list:
pos_removed_list.append(pos_list[item_index])
and replace the commented line:
# current_pos = pos_list[item_index]
with
current_pos = np.array(get_centroid(pos_removed_list))
and you're ready to go!
**Full Code: **
import numpy as np
def get_centroid(lst):
arr = np.array(lst)
length = arr.shape[0]
if length == 0:
return 0,0
else:
sum_x = np.sum(arr[:, 0])
sum_y = np.sum(arr[:, 1])
return sum_x/float(length), sum_y/float(length)
data = [{5036885850: [92.0, 88.73]}, {5036885955: [90.0, 61.73]}, {5036885984: [86.0, 73.03]}, {5036885998: [102.0, 77.54]}]
pos_list = [list(item.values())[0] for item in data]
pos_removed_list = []
current_pos = np.array((0,0))
result_list = []
print('initial centroid: ' + str(current_pos))
while (len(data) != 0):
dist_list = [np.linalg.norm(current_pos - np.array(pos)) for pos in pos_list]
min_dist = min(dist_list)
item_index = dist_list.index(min_dist)
to_be_moved = data[item_index]
print('moved: ' + str(to_be_moved))
result_list.append(to_be_moved)
pos_removed_list.append(pos_list[item_index])
pos_list.remove(pos_list[item_index])
current_pos = np.array(get_centroid(pos_removed_list))
print('current centroid: ' + str(current_pos))
data.remove(to_be_moved)
print('\nfinal result: ' + str(result_list))
Your step by step output will be:
initial centroid: [0 0]
moved: {5036885955: [90.0, 61.73]}
current centroid: [90. 61.73]
moved: {5036885984: [86.0, 73.03]}
current centroid: [88. 67.38]
moved: {5036885998: [102.0, 77.54]}
current centroid: [92.66666667 70.76666667]
moved: {5036885850: [92.0, 88.73]}
current centroid: [92.5 75.2575]
final result: [{5036885955: [90.0, 61.73]}, {5036885984: [86.0, 73.03]}, {5036885998: [102.0, 77.54]}, {5036885850: [92.0, 88.73]}]
Let me know if this is what you were looking for

Using the key argument to the min function can help keep our code efficient and clean. In addition we can help keep track of everything with a few functions that use type aliases to annotate their inputs and outputs. We can calculate the new centroid in constant time from the old centroid as long as we keep track of how many points are involved in calculating the old centroid. Finally we can minimize distance squared to the centroid instead of distance and save ourselves calculating square roots.
from typing import Callable, cast, Dict, List, Tuple
Point = Tuple[float, float]
Order = Dict[int, List[float]]
orders: List[Order] = [
{5036885850: [92.0, 88.73]},
{5036885955: [90.0, 61.73]},
{5036885984: [86.0, 73.03]},
{5036885998: [102.0, 77.54]}
]
def new_centroid(new_point: Point, old_centroid: Point, old_centroid_weight: float) -> Point:
new_x, new_y = new_point
old_x, old_y = old_centroid
new_point_weight = 1 / (old_centroid_weight + 1)
return (
new_point_weight * new_x + (1 - new_point_weight) * old_x,
new_point_weight * new_y + (1 - new_point_weight) * old_y,
)
def distance_squared_to(point: Point) -> Callable[[Order], float]:
def distance_squared_to_point(order: dict) -> float:
order_point = get_point(order)
from_x, from_y = point
order_x, order_y = order_point
return (from_x - order_x) ** 2 + (from_y - order_y) ** 2
return distance_squared_to_point
def get_point(order: Order) -> Point:
return cast(Point, list(order.values()).pop())
sorted_orders = []
centroid = (0.0, 0.0)
centroid_weight = 0
while orders:
next_order = min(orders, key=distance_squared_to(centroid))
orders.remove(next_order)
sorted_orders.append(next_order)
next_point = get_point(next_order)
centroid = new_centroid(next_point, centroid, centroid_weight)
centroid_weight += 1
print(sorted_orders)
A couple notes here. Unfortunately, Python doesn't have a really nice idiom for removing the minimum element from a collection. We have to find the min (order n) and then call the remove on that element (order n again). We could use something like numpy.argmin but that's probably overkill and a pretty heavy dependency for this.
The second note is that our Orders are kinda hard to work with. To get the location, we have to cast the values as a list and pop an element (or just take the first). Given an order we can't index into it without introspecting to find the key. It might be easier if the orders were just a tuple where the first element was the order number and the second was the location. Any number of other forms could also be helpful (named tuple, data class, dictionary with keys item_cd and location_coordinates, ...). Location coordinates could also be just a tuple instead of a list, but that's more of a type hinting issue than anything else.

Distance map returned from shortest_distance function misses entries of certain vertices

I have a network present in a postgres database, where I can route with the pgrouting extension. I've read this into mem, and now want to calculate the distance of all nodes within 0.1 hours from a specific starting node:
dm = G.new_vp("double", np.inf)
gt.shortest_distance(G, source=nd[102481678], weights=wgts, dist_map = dm, max_dist=0.1)
where wgts is an EdgePropertyMap containing the weights per edge, and nd is a reverse mapping to get vertex index from the outside id.
In pgRouting this delivers 349 reachable nodes, using graph-tool only 328. The results are more or less the same (e.g. the furthest node is the same with the exact same cost, nodes present in both lists have same distance), but the graph-tool distance map just seems to miss certain nodes. The weird thing is that I found a cul-de-sac node labeled with a distance (second one from below), but the node connecting the cul-de-sac with the outside world is missing. Seems weird, because if the connecting node would not be reachable, the cul-de-sac would be unreachable as well.
I've compiled a MWE: https://gofile.io/d/YpgjSw
Below is the python code:
import graph_tool.all as gt
import numpy as np
import time
# construct list of source, target, edge-id (edge-id not really used in this example)
l = []
with open('nw.txt') as f:
rows = f.readlines()
for row in rows:
id = int(row.split('\t')[0])
source = int(row.split('\t')[1])
target = int(row.split('\t')[2])
l.append([source, target, id])
l.append([target, source, id])
print len(l)
# construct graph
G = gt.Graph(directed=True)
G.ep["edge_id"] = G.new_edge_property("int")
n = G.add_edge_list(l, hashed=True, eprops=G.ep["edge_id"])
# construct a dict for mapping outside node-id's to internal id's (node indexes)
nd = {}
i = 0
for x in n:
nd[x] = i
i = i + 1
# construct a dict for mapping (source, target) combis to a cost and reverse cost
db_wgts = {}
with open('costs.txt') as f:
rows = f.readlines()
for row in rows:
source = int(row.split('\t')[0])
target = int(row.split('\t')[1])
cost = float(row.split('\t')[2])
reverse_cost = float(row.split('\t')[3])
db_wgts[(source, target)] = cost
db_wgts[(target, source)] = reverse_cost
# construct an edge property and fill it according to previous dict
wgts = G.new_edge_property("double")
i = 0
for e in G.edges():
i = i + 1
print i
print e
s = n[int(e.source())]
t = n[int(e.target())]
try:
wgts[e] = db_wgts[(s, t)]
except KeyError:
# this was necessary
wgts[e] = 1000000
# calculate shortest distance to all nodes within 0.1 total cost from source-node with outside-id of 102481678
dm = G.new_vp("double", np.inf)
gt.shortest_distance(G, source=nd[102481678], weights=wgts, dist_map = dm, max_dist=0.1)
# some mumbo-jumbo for getting the result in a nice node-id: cost format
ar = dm.get_array()
idxs = np.where(dm.get_array() < 0.1)
vals = ar[ar < 0.1]
final_res = [(i, v) for (i,v) in zip(list(idxs[0]), list(vals))]
final_res.sort(key=lambda tup: tup[1])
for x in final_res:
print n[x[0]], x[1]
# output saved in result_missing_nodes.txt
# 328 records, should be 349
To illustrate (one of the) missing nodes:
>>> dm[nd[63447311]]
0.0696234786274957
>>> dm[nd[106448775]]
0.06165528930577409
>>> dm[nd[127601733]]
inf
>>> dm[nd[100428293]]
0.0819900275163846
>>>
This doesn't seem possible because this is the local layout of the network, labels are the id's referenced above:

This is a numerical precision problem. You have very low edge weights (1e-6) combined with very large values (1000000), which cause differences to be lost to finite precision. If you replace all values 1000000 (which I assume mean infinite weight) by numpy.inf, you actually get a more stable calculation, and no missing nodes in your example.
An even better alternative is to actually remove the "infinite weight"
edges using an edge filter:
u = GraphView(G, efilt=wgts.fa < 1000000)
and compute the distances on that.

Python - generating weight map in C4D

Hello Python and Stack community in general.
First, let me say I'm a 3d guy and not much of a code guy. So thanks for the understanding...
The exact problem:
I have to generate a gradient map based on a distance from points to multiple objects
Here is a simple preview of what my problem looks like
Preview_C4D
My base code is the following:
import c4d
#Welcome to the world of Python
warray = [0.0]
def main():
global warray
wtag = op[c4d.ID_USERDATA,1] #drag vertex map from user data panel
obj = wtag.GetObject() #the object of the vertex map
pts = obj.GetAllPoints()
cnt = len(pts)
null = op.GetObject()
nullpos = null.GetMg().off #vector magnitude from matrix
minDistance = op[c4d.ID_USERDATA,4] #drag slider map from user data panel
maxDistance = op[c4d.ID_USERDATA,5] #drag slider map from user data panel
if len(warray) != cnt:
diff = cnt - len(warray)
warray.extend([0.0]*diff)
for x in xrange(cnt): #remapping in the range 0-1
point = pts[x]
distance = (nullpos - point).GetLength()
warray[x] = c4d.utils.RangeMap(distance,minDistance,maxDistance,1,0,False)
if warray[x] > 1:
warray[x] = 1.0
elif warray[x] < 0:
warray[x] = 0.0
wtag.SetAllHighlevelData(warray) #bake the new vertex map
Now lets say I have a list with multiple objects:
parent = doc.SearchObject('parent')
list1 = parent.GetChildren() # list of cubes under parent
count = len(list1)
for a in range(list):
obj = list1[a]
distance = ?
So I'm stuck here, because I can't figure it out how to merge
the new values with the old.
In few words I need a loop that evaluates the values for the points
for each object in the list and then adds them toghether.
I just can't dial it up right.
So I'll be very grateful if anybody can help.
Regards

Calculating the number of graphs created and the number of vertices in each graph from a list of edges

Given a list of edges such as, edges = [[1,2],[2,3],[3,1],[4,5]]
I need to find how many graphs are created, by this I mean how many groups of components are created by these edges. Then get the number of vertices in the group of components.
However, I am required to be able to handle 10^5 edges, and i am currently having trouble completing the task for large number of edges.
My algorithm is currently getting the list of edges= [[1,2],[2,3],[3,1],[4,5]] and merging each list as set if they have a intersection, this will output a new list that now contains group components such as , graphs = [[1,2,3],[4,5]]
There are two connected components : [1,2,3] are connected and [4,5] are connected as well.
I would like to know if there is a much better way of doing this task.
def mergeList(edges):
sets = [set(x) for x in edges if x]
m = 1
while m:
m = 0
res = []
while sets:
common, r = sets[0], sets[1:]
sets = []
for x in r:
if x.isdisjoint(common):
sets.append(x)
else:
m = 1
common |= x
res.append(common)
sets = res
return sets
I would like to try doing this in a dictionary or something efficient, because this is toooo slow.

A basic iterative graph traversal in Python isn't too bad.
import collections
def connected_components(edges):
# build the graph
neighbors = collections.defaultdict(set)
for u, v in edges:
neighbors[u].add(v)
neighbors[v].add(u)
# traverse the graph
sizes = []
visited = set()
for u in neighbors.keys():
if u in visited:
continue
# visit the component that includes u
size = 0
agenda = {u}
while agenda:
v = agenda.pop()
visited.add(v)
size += 1
agenda.update(neighbors[v] - visited)
sizes.append(size)
return sizes

Do you need to write your own algorithm? networkx already has algorithms for this.
To get the length of each component try
import networkx as nx
G = nx.Graph()
G.add_edges_from([[1,2],[2,3],[3,1],[4,5]])
components = []
for graph in nx.connected_components(G):
components.append([graph, len(graph)])
components
# [[set([1, 2, 3]), 3], [set([4, 5]), 2]]

You could use Disjoint-set data structure:
edges = [[1,2],[2,3],[3,1],[4,5]]
parents = {}
size = {}
def get_ancestor(parents, item):
# Returns ancestor for a given item and compresses path
# Recursion would be easier but might blow stack
stack = []
while True:
parent = parents.setdefault(item, item)
if parent == item:
break
stack.append(item)
item = parent
for item in stack:
parents[item] = parent
return parent
for x, y in edges:
x = get_ancestor(parents, x)
y = get_ancestor(parents, y)
size_x = size.setdefault(x, 1)
size_y = size.setdefault(y, 1)
if size_x < size_y:
parents[x] = y
size[y] += size_x
else:
parents[y] = x
size[x] += size_y
print(sum(1 for k, v in parents.items() if k == v)) # 2
In above parents is a dict where vertices are keys and ancestors are values. If given vertex doesn't have a parent then the value is the vertex itself. For every edge in the list the ancestor of both vertices is set the same. Note that when current ancestor is queried the path is compressed so following queries can be done in O(1) time. This allows the whole algorithm to have O(n) time complexity.
Update
In case components are required instead of just number of them the resulting dict can be iterated to produce it:
from collections import defaultdict
components = defaultdict(list)
for k, v in parents.items():
components[v].append(k)
print(components)
Output:
defaultdict(<type 'list'>, {3: [1, 2, 3], 5: [4, 5]})

Python - speed up pathfinding

This is my pathfinding function:
def get_distance(x1,y1,x2,y2):
neighbors = [(-1,0),(1,0),(0,-1),(0,1)]
old_nodes = [(square_pos[x1,y1],0)]
new_nodes = []
for i in range(50):
for node in old_nodes:
if node[0].x == x2 and node[0].y == y2:
return node[1]
for neighbor in neighbors:
try:
square = square_pos[node[0].x+neighbor[0],node[0].y+neighbor[1]]
if square.lightcycle == None:
new_nodes.append((square,node[1]))
except KeyError:
pass
old_nodes = []
old_nodes = list(new_nodes)
new_nodes = []
nodes = []
return 50
The problem is that the AI takes to long to respond( response time <= 100ms)
This is just a python way of doing https://en.wikipedia.org/wiki/Pathfinding#Sample_algorithm

You should replace your algorithm with A*-search with the Manhattan distance as a heuristic.

One reasonably fast solution is to implement the Dijkstra algorithm (that I have already implemented in that question):
Build the original map. It's a masked array where the walker cannot walk on masked element:
%pylab inline
map_size = (20,20)
MAP = np.ma.masked_array(np.zeros(map_size), np.random.choice([0,1], size=map_size))
matshow(MAP)
Below is the Dijkstra algorithm:
def dijkstra(V):
mask = V.mask
visit_mask = mask.copy() # mask visited cells
m = numpy.ones_like(V) * numpy.inf
connectivity = [(i,j) for i in [-1, 0, 1] for j in [-1, 0, 1] if (not (i == j == 0))]
cc = unravel_index(V.argmin(), m.shape) # current_cell
m[cc] = 0
P = {} # dictionary of predecessors
#while (~visit_mask).sum() > 0:
for _ in range(V.size):
#print cc
neighbors = [tuple(e) for e in asarray(cc) - connectivity
if e[0] > 0 and e[1] > 0 and e[0] < V.shape[0] and e[1] < V.shape[1]]
neighbors = [ e for e in neighbors if not visit_mask[e] ]
tentative_distance = [(V[e]-V[cc])**2 for e in neighbors]
for i,e in enumerate(neighbors):
d = tentative_distance[i] + m[cc]
if d < m[e]:
m[e] = d
P[e] = cc
visit_mask[cc] = True
m_mask = ma.masked_array(m, visit_mask)
cc = unravel_index(m_mask.argmin(), m.shape)
return m, P
def shortestPath(start, end, P):
Path = []
step = end
while 1:
Path.append(step)
if step == start: break
if P.has_key(step):
step = P[step]
else:
break
Path.reverse()
return asarray(Path)
And the result:
start = (2,8)
stop = (17,19)
D, P = dijkstra(MAP)
path = shortestPath(start, stop, P)
imshow(MAP, interpolation='nearest')
plot(path[:,1], path[:,0], 'ro-', linewidth=2.5)
Below some timing statistics:
%timeit dijkstra(MAP)
#10 loops, best of 3: 32.6 ms per loop

The biggest issue with your code is that you don't do anything to avoid the same coordinates being visited multiple times. This means that the number of nodes you visit is guaranteed to grow exponentially, since it can keep going back and forth over the first few nodes many times.
The best way to avoid duplication is to maintain a set of the coordinates we've added to the queue (though if your node values are hashable, you might be able to add them directly to the set instead of coordinate tuples). Since we're doing a breadth-first search, we'll always reach a given coordinate by (one of) the shortest path(s), so we never need to worry about finding a better route later on.
Try something like this:
def get_distance(x1,y1,x2,y2):
neighbors = [(-1,0),(1,0),(0,-1),(0,1)]
nodes = [(square_pos[x1,y1],0)]
seen = set([(x1, y1)])
for node, path_length in nodes:
if path_length == 50:
break
if node.x == x2 and node.y == y2:
return path_length
for nx, ny in neighbors:
try:
square = square_pos[node.x + nx, node.y + ny]
if square.lightcycle == None and (square.x, square.y) not in seen:
nodes.append((square, path_length + 1))
seen.add((square.x, square.y))
except KeyError:
pass
return 50
I've also simplified the loop a bit. Rather than switching out the list after each depth, you can just use one loop and add to its end as you're iterating over the earlier values. I still abort if a path hasn't been found with fewer than 50 steps (using the distance stored in the 2-tuple, rather than the number of passes of the outer loop). A further improvement might be to use a collections.dequeue for the queue, since you could efficiently pop from one end while appending to the other end. It probably won't make a huge difference, but might avoid a little bit of memory usage.
I also avoided most of the indexing by one and zero in favor of unpacking into separate variable names in the for loops. I think this is much easier to read, and it avoids confusion since the two different kinds of 2-tuples had had different meanings (one is a node, distance tuple, the other is x, y).