Develop Reference

How to append and pair coordinate values in nested for loop

I am finding the distance between two pairs of random points, I am then duplicating the points in a 3 x 3 pattern so that the same points are seen after a certain distance, which is done with a nested for loop. I am trying to find the distance between the newly created points from the a for loop.
I tried using append within the loop to store the points, which gives me the distances, but it is only giving me 24 distances when there should be a lot more between 9 copies of 4 points.
Am I not implementing append correcting to account for additional distances?
Code
import numpy as np
import matplotlib.pyplot as plt
import random
import math
dist = []
#scale of the plot
scalevalue = 10
x = [random.uniform(1, 10) for n in range(4)]
y = [random.uniform(1, 10) for n in range(4)]
tiles = np.linspace(-scalevalue, scalevalue, 3)
for i in tiles:
for j in tiles:
bg_tile = plt.scatter(x + i,y + j, c="black", s=3)
dist.append(i)
dist.append(j)
pairs = list(zip(x + i,y + j))
plt.show()
def distance(x, y):
return math.sqrt((x[0]-x[1])**2 + (y[0]-y[1])**2)
for i in range(len(pairs)):
for j in range(i+1,len(pairs)):
dist.append(distance(pairs[i],pairs[j]))
print(dist)

Run your code:
x (and y) is a list of numbers (4):
In [553]: x
Out[553]: [8.699962201099193, 3.1643082386096975, 5.245385542599207, 3.0412506367299033]
tiles is an array:
In [554]: tiles
Out[554]: array([-10., 0., 10.])
And the first iteration - without the plot, and doing one (i,j) append, rather than the sequential. This better separates the i values from the j ones:
In [558]: dist=[]
...: for i in tiles:
...: for j in tiles:
...: dist.append((i,j))
...: pairs = list(zip(x + i,y + j))
In [559]: dist
Out[559]:
[(-10.0, -10.0), # that just reflects how you iterate on tiles
(-10.0, 0.0),
(-10.0, 10.0),
(0.0, -10.0),
(0.0, 0.0),
(0.0, 10.0),
(10.0, -10.0),
(10.0, 0.0),
(10.0, 10.0)]
That flat list you show in the comment confuses those values. Why are you doing this?
pairs ends up with the last i,j values; earlier iterations are thrown away.
In [560]: pairs
Out[560]:
[(18.699962201099193, 18.63063210113664),
(13.164308238609697, 12.329695190243902),
(15.245385542599207, 16.685778921185936),
(13.041250636729902, 15.89730196643608)]
So the first column is:
In [561]: i
Out[561]: 10.0
In [562]: x+i
Out[562]: array([18.6999622 , 13.16430824, 15.24538554, 13.04125064])
x is a list, but i is np.float64, so the addition is array addition (list 'addition' is join).
pairs
With that last pairs:
In [567]: alist = []
...: for i in range(len(pairs)):
...: for j in range(i+1,len(pairs)):
...: alist.append(distance(pairs[i],pairs[j]))
...:
In [568]: alist
Out[568]:
[0.8374876734992962,
1.442060937629651,
2.8568926932380996,
1.664725810930718,
2.9755013255616056,
3.1987125977481807]
What the iteration is doing is get the 6 combinations of these 4 pairs
In [574]: distance(pairs[0],pairs[1])
Out[574]: 0.8374876734992962
Those 6 values (different in my case because of different random numbers) have nothing to do with the tile values that you previously accumulated in dist.
If I make a 2d array from pairs:
In [575]: arr = np.array(pairs); arr
Out[575]:
array([[18.6999622 , 18.6306321 ],
[13.16430824, 12.32969519],
[15.24538554, 16.68577892],
[13.04125064, 15.89730197]])
I can replicate the distance with:
In [576]: (arr[:,1]-arr[:,0])**2
Out[576]: array([4.80666276e-03, 6.96578941e-01, 2.07473309e+00, 8.15702920e+00])
In [577]: np.sqrt(np.sum(_[:2]))
Out[577]: 0.8374876734992962
I don't know what's the significance of this. pairs is just the x,y values with an added 10:
In [579]: np.column_stack((x,y))+10
Out[579]:
array([[18.6999622 , 18.6306321 ],
[13.16430824, 12.32969519],
[15.24538554, 16.68577892],
[13.04125064, 15.89730197]])

Numpy/Torch insert smallest value in case of collision

I have an empty numpy array, a list of indices, and list of values associated with the indices. The issue is that there may be duplicates in the indices. In all these "collision" cases, I'd like the smallest value to be picked. Just wondering what is the best way to go about it.
Eg:
array = [0,0,0,0,0,0,0]
indices = [0, 0, 2, 3, 2, 4]
values = [1.0, 3.0, 3.5, 1.5, 2.5, 8.0]
Result:
out = [1.0, 0, 2.5, 1.5, 8.0, 0.0, 0.0]

You can always implement something manually like:
import numpy as np
def index_reduce(arr, indices, out, reducer=min):
touched = np.zeros_like(out, dtype=np.bool_)
for i, x in enumerate(indices):
if not touched[x]:
out[x] = arr[i]
touched[x] = True
else:
out[x] = reducer(out[x], arr[i])
return out
which essentially loops through the indices and assign the values of arr to out if not already touched (keeping track of this with the touched array) and reducing the output with the specified reducer.
NOTE: The reducer function needs to be such that the final result can only depend on the current and previous value.
The usage of this would be:
indices = [0, 0, 2, 3, 2, 4]
values = [1.0, 3.0, 3.5, 1.5, 2.5, 8.0]
array = np.zeros(7)
index_reduce(values, indices, array)
# array([1. , 0. , 2.5, 1.5, 8. , 0. , 0. ])
If performances are of concern, you can also accelerate the above code with Numba with a simple decoration provided that also the values and indices inputs are NumPy arrays:
import numba as nb
index_reduce_nb = nb.njit(index_reduce)
indices = np.array([0, 0, 2, 3, 2, 4])
values = np.array([1.0, 3.0, 3.5, 1.5, 2.5, 8.0])
array = np.zeros(7)
index_reduce_nb(values, indices, array)
# array([1. , 0. , 2.5, 1.5, 8. , 0. , 0. ])
Benchmarks
The above solutions can be compared to a Torch-based solution (reworked from #Shai's answer):
import torch
def index_reduce_torch(arr, indices, out, reduce_="amin"):
arr = torch.from_numpy(arr)
indices = torch.from_numpy(indices)
out = torch.from_numpy(out)
return out.index_reduce_(dim=0, index=indices, source=arr, reduce=reduce_, include_self=False).numpy()
or, with additional skipping of Torch gradients:
index_reduce_torch_ng = torch.no_grad()(index_reduce_torch)
index_reduce_torch_ng.__name__ = "index_reduce_torch_ng"
and a Pandas-based solution (reworked from #bpfrd's answer):
import pandas as pd
def index_reduce_pd(arr, indices, out, reducer=min):
df = pd.DataFrame(data=zip(indices, arr))
df1 = df.groupby(0, as_index=False).agg(reducer)
out[df1[0]] = df1[1]
return out
using the following code:
funcs = index_reduce, index_reduce_nb, index_reduce_pd, index_reduce_torch, index_reduce_torch_ng
timings = {}
for i in range(4, 18):
n = 2 ** i
print(f"n = {n}, i = {i}")
extrema = 0, 2 * n
indices = np.random.randint(*extrema, n)
values = np.random.random(n)
out = np.zeros(extrema[1] + 1)
timings[n] = []
base = funcs[0](values, indices, out)
for func in funcs:
res = func(values, indices, out)
is_good = np.allclose(base, res)
timed = %timeit -r 16 -n 16 -q -o func(values, indices, out)
timing = timed.best * 1e6
timings[n].append(timing if is_good else None)
print(f"{func.__name__:>24} {is_good} {timing:10.3f} µs")
to produce with the additional lines:
import matplotlib.pyplot as plt
df = pd.DataFrame(data=timings, index=[func.__name__ for func in funcs]).transpose()
df.plot(marker='o', xlabel='Input size / #', ylabel='Best timing / µs', figsize=(6, 4))
df.plot(marker='o', xlabel='Input size / #', ylabel='Best timing / µs', ylim=[0, 500], figsize=(6, 4))
fig = plt.gcf()
fig.patch.set_facecolor('white')
these plots:
(the second is a zoomed-in version of the first).
These indicate that the Numba accelerated solution could be the fastest, closely followed by the Torch-based solution while the Pandas approach could be the slowest, even slower than the explicit solution without acceleration.

You are looking for index_reduce_, which was introduced in PyTorch 1.12.
import torch
array = torch.zeros(7)
indices = torch.tensor([0, 0, 2, 3, 2, 4])
values = torch.tensor([1.0, 3.0, 3.5, 1.5, 2.5, 8.0])
out = array.index_reduce_(dim=0, index=indices, source=values, reduce='amin', include_self=False)
You'll get your desired output:
tensor([1.0000, 0.0000, 2.5000, 1.5000, 8.0000, 0.0000, 0.0000])
Note that this method is in "beta" and its API may change in future PyTorch versions.

You can use pandas groupby agg as the following:
indices = [0, 0, 2, 3, 2, 4]
values = [1.0, 3.0, 3.5, 1.5, 2.5, 8.0]
array = [0,0,0,0,0,0,0]
df = pd.DataFrame(zip(indices, values), columns=['indices','values'])
df1 = df.groupby('indices', as_index=False).agg(values=('values', min))
for i,j in zip(df1['indices'].tolist(), df1['values'].tolist()):
array[i] = j
output:
array
>[1.0, 0, 2.5, 1.5, 8.0, 0, 0]

Python is adding wrong numbers to the list

I am writing a Sierpinski Triangle generator, with this code:
import numpy as np
import matplotlib.pyplot as mpl
from random import choice
a=np.array([[-1,0],[1,0],[0,np.sqrt(3)]])
listofpoints=[]
for i in a:
listofpoints.append(list(i))
startpoint=[0,0.5]
listofpoints.append(startpoint)
newpoint=startpoint.copy()
for i in range(2):
randomcorner=choice(a)
print(i,"randomcorner",randomcorner)
print(i,"newpoint",newpoint)
halfdistance_x=(randomcorner[0]-newpoint[0])/2
newpoint[0]+=halfdistance_x
halfdistance_y=(randomcorner[1]-newpoint[1])/2
newpoint[1]+=halfdistance_y
print(i,"newpointadded",newpoint)
listofpoints.append(newpoint)
print(listofpoints)
x=[]
y=[]
for i in range(len(listofpoints)):
x.append(listofpoints[i][0])
y.append(listofpoints[i][1])
mpl.plot(x,y,'o')
I encountered the issue, while I am trying to set next point, it counts them propely, however, only the last iteration value is being added this many times, as the range() says (in this example 2).
Values that I keep getting:
0 randomcorner [1. 0.]
0 newpoint [0, 0.5]
0 newpointadded [0.5, 0.25]
1 randomcorner [1. 0.]
1 newpoint [0.5, 0.25]
1 newpointadded [0.75, 0.125]
[[-1.0, 0.0], [1.0, 0.0], [0.0, 1.7320508075688772], [0, 0.5], [0.75, 0.125], [0.75, 0.125]]
The list in the solution should be:
[[-1.0, 0.0], [1.0, 0.0], [0.0, 1.7320508075688772], [0, 0.5], **[0.5, 0.25]**, [0.75, 0.125]]
How to solve this problem?

Code is OK.
Problem is how Python keeps newpoint on list. It doesn't duplicate it but it keeps references to newpoint. But after adding newpoint to list you use newpoint to calculate next point - so finally it modifies all points on list when you modify newpoint.
You have to duplicate it.
newpoint = newpoint.copy()
Or you should reassign value to variable.
# use `+` instead of `-` to get directly middle point
new_x = (randomcorner[0] + newpoint[0]) / 2
new_y = (randomcorner[1] + newpoint[1]) / 2
newpoint = [new_x, new_y]
Your version with newpoint = newpoint.copy() and range(1000)
And as startpoint you should use some value from list, not [0,0.5] because it is point in wrong place.
import numpy as np
import matplotlib.pyplot as mpl
from random import choice
a = np.array([[-1,0],[1,0],[0,np.sqrt(3)]])
listofpoints = []
for i in a:
listofpoints.append(list(i))
startpoint = a[0] # <--- HERE
listofpoints.append(startpoint)
newpoint = startpoint.copy()
for i in range(1000):
randomcorner = choice(a)
print(i,"randomcorner",randomcorner)
print(i,"newpoint",newpoint)
halfdistance_x=(randomcorner[0]-newpoint[0])/2
newpoint[0] += halfdistance_x
halfdistance_y=(randomcorner[1]-newpoint[1])/2
newpoint[1] += halfdistance_y
print(i, "newpointadded", newpoint)
listofpoints.append(newpoint)
newpoint = newpoint.copy() # <--- HERE
print(listofpoints)
x=[]
y=[]
for i in range(len(listofpoints)):
x.append(listofpoints[i][0])
y.append(listofpoints[i][1])
mpl.plot(x,y,'o')
mpl.show()
Result:
My version with some modifications.
See also: PEP 8 -- Style Guide for Python Code
import matplotlib.pyplot as mpl
from random import choice
a = [[-1, 0], [1,0], [0, 3**0.5]]
listofpoints = list(a)
newpoint = a[0]
for i in range(1000):
print(i, "newpoint", newpoint)
randomcorner = choice(a)
print(i, "randomcorner", randomcorner)
new_x = (randomcorner[0] + newpoint[0]) / 2
new_y = (randomcorner[1] + newpoint[1]) / 2
newpoint = [new_x, new_y]
print(i, "newpoint added", newpoint)
listofpoints.append(newpoint)
print('---')
print('--- listofpoints ---')
print(listofpoints)
x = []
y = []
for point in listofpoints:
x.append(point[0])
y.append(point[1])
# OR
#for px, py in listofpoints:
# x.append(px)
# y.append(py)
mpl.plot(x, y, '.')
mpl.show()

Dividing circumference into equal parts and returning coordinates

I have created several circles with different origins using Python and I am trying to implement a function that will divide each circle into n number of equal parts along the circumference. I am trying to populate an array that contains the starting [x,y] coordinate for each part on the circumference.
My code is as follows:
def fnCalculateArcCoordinates(self,intButtonCount,radius,center):
lstButtonCoord = []
#for degrees in range(0,360,intAngle):
for arc in range(1,intButtonCount + 1):
degrees = arc * 360 / intButtonCount
xDegreesCoord = int(center[0] + radius * math.cos(math.radians(degrees)))
yDegreesCoord = int(center[1] + radius * math.sin(math.radians(degrees)))
lstButtonCoord.append([xDegreesCoord,yDegreesCoord])
return lstButtonCoord
When I run the code for 3 parts, an example of the set of coordinates that are returned are:
[[157, 214], [157, 85], [270, 149]]
This means the segments are of different sizes. Could someone please help me identify where my error is?

The exact results of such trigonometric calculations are rarely exact integers. By flooring them to int, you lose some precision, of course. The approximate (Pythagorean) distance checks suggest that your math is correct:
(270-157)**2 + (149-85)**2
# 16865
(270-157)**2 + (214-149)**2
# 16994
(157-157)**2 + (214-85)**2
# 16641
Furthermore, you can use the built-in complex number type and the cmath module. In particular cmath.rect converts polar coordinates (a radius and an angle) into rectangular coordinates:
import cmath
def calc(count, radius, center):
x, y = center
for i in range(count):
r = cmath.rect(radius, (2*cmath.pi)*(i/count))
yield [round(x+r.real, 2), round(y+r.imag, 2)]
list(calc(4, 2, [0, 0]))
# [[2.0, 0.0], [0.0, 2.0], [-2.0, 0.0], [-0.0, -2.0]]
list(calc(6, 1, [0, 0]))
# [[1.0, 0.0], [0.5, 0.87], [-0.5, 0.87], [-1.0, 0.0], [-0.5, -0.87], [0.5, -0.87]]
You want to change rounding as you see fit.

Cutting geological boreholes (csv data file) to extract some value using python

It's my first question/post so first of all I would like to say THANKS for all great ideas and solutions that I have found in this place.
I have a problem with pretty simple task: I've got a csv file with geophysical measurements of soil/rock electrical resistivity in some grup of boreholes. I have to find rho value at some cutoff level e.g. 5 meters. I have measurement number (m_nr), which is also a layer number, x and y coordinates, ordinate ("o" as meters above sea level), resistivity (rho), layer depth (h) and layer thickness (d). The value of rho which I'm looking for is in the first row of different borehole which meet the condition h >= cutoff. I'm using python 3.6 and that's how my code looks:
file = open('measurement.csv', newline='')
file = csv.reader(file, delimiter=';', quotechar='|')
measurements = list(file)
result = []
cutoff=5
for m_nr, x, y, ordinate, rho, h, d in measurements:
m_nr = int(m_nr)
x = int(x)
y = int(y)
o = float(ordinate)
rho = float(rho)
h = float(h)
d = float(d)
if h >= cutoff:
result.append([x, y, m_nr, o-cutoff, rho, h, d])
and some output:
[[20456, 10234, 4, 90.0, 2356.0, 7.0, 2.25],
[20456, 10234, 5, 90.0, 24563.0, 15.0, 8.0],
[20456, 10234, 6, 90.0, 250.0, 21.0, 6.0],
[10122, 15678, 3, 108.0, 245.0, 6.0, 2.0],
[10122, 15678, 4, 108.0, 2356.0, 7.0, 1.0],
[10122, 15678, 5, 108.0, 24563.0, 15.0, 8.0],
[30111, 34444, 2, 75.0, 4686.0, 12.0, 11.0],
[30111, 34444, 3, 75.0, 245.0, 16.0, 4.0],
[30111, 34444, 4, 75.0, 2356.0, 28.0, 12.0]]
That's just a test file and I expect that in some near future I will have houndrets of boreholes so effectivity of code matters... For each borehole (different set of x,y) only the first row in the list is the one that I need. I don't know how to extract it from my results and that's where I'm asking for your help.
Regards,
Matsu

I'll just go over several things.
It's cleaner to open the file using a with statement so you don't have to worry about closing it
You can use the DictReader class to make the data accessible more easily.
Don't do list(file), just iterate over the reader directly. That way you don't have to load the whole thing into memory.
You can keep track of the x, y values and skip the rest after you find a match.
Result:
with open('measurement.csv', newline='') as file:
fieldnames = ['m_nr', 'x', 'y', 'ord', 'rho', 'h', 'd']
reader = csv.DictReader(file, fieldnames=fieldnames)
result = []
last_xy = None
cutoff=5
for line in reader:
xy = int(line['x']), int(line['y'])
if xy == last_xy:
continue # skip processing since we already have a match
h = float(line['h'])
if h >= cutoff:
result.append(line)
last_xy = xy # if we find a match, save the xy
Finally, if the goal is to put the result into a new CSV file, I'd just have an output file open for writing at the same time and write out the results instead of appending them to a list. That way you never need to have more than a few lines in memory at a time.