Related
When running the ODR algorithm on some experiment data, I've been asked to run it with the following model:
It is clear that this fitting function is containing a redundant degree of freedom.
When I run the fitting on my experiment data I get enormous relative errors of beta, starting from 8000% relative error.
When I try to run the fitting again but with a fitting function that doesn't have a redundant degree of freedom, such as:
I don't get this kind of problem.
Why is this happening? Why the ODR algorithm is so sensitive for redundant degrees of freedom? I wasn't able to answer these questions to my supervisors. An answer will be much appreciated.
Reproducing code example:
from scipy.odr import RealData, Model, ODR
def func1(a, x):
return a[0] * (x + a[1]) / (a[3] * (x + a[1]) + a[1] * x) + a[2]
def func2(a, x):
return a[0] / (x + a[1]) + a[2]
# fmt: off
zx = [
1911.125, 2216.95, 2707.71, 3010.225, 3410.612, 3906.015, 4575.105, 5517.548,
6918.481,
]
dx = [
0.291112577, 0.321695254, 0.370771197, 0.401026507, 0.441068641, 0.490601621,
0.557573268, 0.651755155, 0.79184836,
]
zy = [
0.000998056, 0.000905647, 0.000800098, 0.000751041, 0.000699982, 0.000650532,
0.000600444, 0.000550005, 0.000500201,
]
dy = [
5.49029e-07, 5.02824e-07, 4.5005e-07, 4.25532e-07, 3.99991e-07, 3.75266e-07,
3.50222e-07, 3.25003e-07, 3.00101e-07,
]
# fmt: on
data = RealData(x=zx, y=zy, sx=dx, sy=dy)
print("Func 1")
print("======")
beta01 = [
1.46,
4775.4,
0.01,
1000,
]
model1 = Model(func1)
odr1 = ODR(data, model1, beta0=beta01)
result1 = odr1.run()
print("beta", result1.beta)
print("sd beta", result1.sd_beta)
print("relative", result1.sd_beta / result1.beta * 100)
print()
print()
print("Func 2")
print("======")
beta02 = [
1,
1,
1,
]
model2 = Model(func2)
odr2 = ODR(data, model2, beta0=beta02)
result2 = odr2.run()
print("beta", result2.beta)
print("sd beta", result2.sd_beta)
print("relative", result2.sd_beta / result2.beta * 100)
This prints out:
Func 1
======
beta [ 1.30884537e+00 -2.82585952e+03 7.79755196e-04 9.47943376e+01]
sd beta [1.16144608e+02 3.73765816e+06 6.12613738e-01 4.20775596e+03]
relative [ 8873.82193523 -132266.24068473 78564.88054498 4438.82627453]
Func 2
======
beta [1.40128121e+00 9.80844274e+01 3.00511669e-04]
sd beta [2.73990552e-03 3.22344713e+00 3.74538794e-07]
relative [0.1955286 3.28640051 0.12463369]
Scipy/Numpy/Python version information:
Versions are:
Scipy - 1.4.1
Numpy - 1.18.2
Python - 3.7.2
The problem is not with the degrees of freedom.
The degrees of freedom is the difference between the number of data points and the number of fitting parameters.
The problem has the same number of degrees of freedom for the two formulae, as they have the same number of parameters.
It also looks like that you do not have free degrees of freedom, which is good news, it means that it can potentially be fitted.
However, you are right that first expression has some problem: the parameters you are trying to fit are not independent.
This is probably better understood with some simpler example.
Consider the following expression:
y = x + b + c
which you try to fit, given n data for x and y with n >> 2.
The question is: what are the optimal value for b and c? This cannot be answered. All you can say from x and y data is about the combination. Therefore, if b + c is 0, the fit cannot tell us if b = 1000, c = -1000 or b = 1, c= -1, but at least we can say that given b we can determine c.
What is the error on a given b? Potentially infinite. That is the reason for the fitting to give you that large relative error.
I want to save time on expensive linear solves for my optimization by using the previous linear solutions as initial guesses for the subsequent iteration in an optimization. I'm looking through OpenMDAO's example for the cache_linear_solution feature which seems to have been developed for this purpose (here) and code shown below:
from distutils.version import LooseVersion
import numpy as np
import scipy
from scipy.sparse.linalg import gmres
import openmdao.api as om
class QuadraticComp(om.ImplicitComponent):
"""
A Simple Implicit Component representing a Quadratic Equation.
R(a, b, c, x) = ax^2 + bx + c
Solution via Quadratic Formula:
x = (-b + sqrt(b^2 - 4ac)) / 2a
"""
def setup(self):
self.add_input('a', val=1.)
self.add_input('b', val=1.)
self.add_input('c', val=1.)
self.add_output('states', val=[0,0])
self.declare_partials(of='*', wrt='*')
def apply_nonlinear(self, inputs, outputs, residuals):
a = inputs['a']
b = inputs['b']
c = inputs['c']
x = outputs['states'][0]
y = outputs['states'][1]
residuals['states'][0] = a * x ** 2 + b * x + c
residuals['states'][1] = a * y + b
def solve_nonlinear(self, inputs, outputs):
a = inputs['a']
b = inputs['b']
c = inputs['c']
outputs['states'][0] = (-b + (b ** 2 - 4 * a * c) ** 0.5) / (2 * a)
outputs['states'][1] = -b/a
def linearize(self, inputs, outputs, partials):
a = inputs['a'][0]
b = inputs['b'][0]
c = inputs['c'][0]
x = outputs['states'][0]
y = outputs['states'][1]
partials['states', 'a'] = [[x**2],[y]]
partials['states', 'b'] = [[x],[1]]
partials['states', 'c'] = [[1.0],[0]]
partials['states', 'states'] = [[2*a*x+b, 0],[0, a]]
self.state_jac = np.array([[2*a*x+b, 0],[0, a]])
def solve_linear(self, d_outputs, d_residuals, mode):
if mode == 'fwd':
print("incoming initial guess", d_outputs['states'])
if LooseVersion(scipy.__version__) < LooseVersion("1.1"):
d_outputs['states'] = gmres(self.state_jac, d_residuals['states'], x0=d_outputs['states'])[0]
else:
d_outputs['states'] = gmres(self.state_jac, d_residuals['states'], x0=d_outputs['states'], atol='legacy')[0]
elif mode == 'rev':
if LooseVersion(scipy.__version__) < LooseVersion("1.1"):
d_residuals['states'] = gmres(self.state_jac, d_outputs['states'], x0=d_residuals['states'])[0]
else:
d_residuals['states'] = gmres(self.state_jac, d_outputs['states'], x0=d_residuals['states'], atol='legacy')[0]
p = om.Problem()
indeps = p.model.add_subsystem('indeps', om.IndepVarComp(), promotes_outputs=['a', 'b', 'c'])
indeps.add_output('a', 1.)
indeps.add_output('b', 4.)
indeps.add_output('c', 1.)
p.model.add_subsystem('quad', QuadraticComp(), promotes_inputs=['a', 'b', 'c'], promotes_outputs=['states'])
p.model.add_design_var('a', cache_linear_solution=True)
p.model.add_constraint('states', upper=10)
p.setup(mode='fwd')
p.run_model()
print(p['states'])
derivs = p.compute_totals(of=['states'], wrt=['a'])
print(derivs['states', 'a'])
p['a'] = 4
derivs = p.compute_totals(of=['states'], wrt=['a'])
print(derivs['states', 'a'])
The above code gives the following print out:
[-0.26794919 -4. ]
incoming initial guess [0. 0.]
[[-0.02072594]
[ 4. ]]
incoming initial guess [0. 0.]
[[-0.02072594]
[ 4. ]]
From the print out of this example it doesn't look like the initial guess for the linear guess is actually being updated. Am I missing something? I've also tried running the code with the cache_linear_solution set to False and the result seems to be the same.
Currently, the caching of linear solutions only happens when the total derivatives are computed during the run of a driver, so if you want to check to make sure it's happening during your optimization (in the run_driver call), change
derivs = p.compute_totals(of=['states'], wrt=['a'])
to
derivs = p.driver._compute_totals(of=['states'], wrt=['a'], global_names=False)
When I do that with your code, I get the following output:
[-0.26794919 -4. ]
incoming initial guess [0. 0.]
[[-0.02072594]
[ 4. ]]
incoming initial guess [-0.02072594 4. ]
[[-0.02072594]
[ 4. ]]
Note that the global_names=False arg is only needed if you use promoted names for your of and wrt variables.
I will update our example code to reflect the correct way to do this.
I have an array of scalars of m rows and n columns. I have a Variable(m) and a Variable(n) that I would like to find solutions for.
The two variables represent values that need to be broadcast over the columns and rows respectively.
I was naively thinking of writing the variables as Variable((m, 1)) and Variable((1, n)), and adding them together as if they're ndarrays. However, that doesn't work, as broadcasting is not allowed.
import cvxpy as cp
import numpy as np
# Problem data.
m = 3
n = 4
np.random.seed(1)
data = np.random.randn(m, n)
# Construct the problem.
x = cp.Variable((m, 1))
y = cp.Variable((1, n))
objective = cp.Minimize(cp.sum(cp.abs(x + y + data)))
# or:
#objective = cp.Minimize(cp.sum_squares(x + y + data))
prob = cp.Problem(objective)
result = prob.solve()
print(x.value)
print(y.value)
This fails on the x + y expression: ValueError: Cannot broadcast dimensions (3, 1) (1, 4).
Now I'm wondering two things:
Is my problem indeed solvable using convex optimization?
If yes, how can I express it in a way that cvxpy understands?
I'm very new to the concept of convex optimization, as well as cvxpy, and I hope I described my problem well enough.
I offered to show you how to represent this as a linear program, so here it goes. I'm using Pyomo, since I'm more familiar with that, but you could do something similar in PuLP.
To run this, you will need to first install Pyomo and a linear program solver like glpk. glpk should work for reasonable-sized problems, but if you are finding it's taking too long to solve, you could try a (much faster) commercial solver like CPLEX or Gurobi.
You can install Pyomo via pip install pyomo or conda install -c conda-forge pyomo. You can install glpk from https://www.gnu.org/software/glpk/ or via conda install glpk. (I think PuLP comes with a version of glpk built-in, so that might save you a step.)
Here's the script. Note that this calculates absolute error as a linear expression by defining one variable for the positive component of the error and another for the negative part. Then it seeks to minimize the sum of both. In this case, the solver will always set one to zero since that's an easy way to reduce the error, and then the other will be equal to the absolute error.
import random
import pyomo.environ as po
random.seed(1)
# ~50% sparse data set, big enough to populate every row and column
m = 10 # number of rows
n = 10 # number of cols
data = {
(r, c): random.random()
for r in range(m)
for c in range(n)
if random.random() >= 0.5
}
# define a linear program to find vectors
# x in R^m, y in R^n, such that x[r] + y[c] is close to data[r, c]
# create an optimization model object
model = po.ConcreteModel()
# create indexes for the rows and columns
model.ROWS = po.Set(initialize=range(m))
model.COLS = po.Set(initialize=range(n))
# create indexes for the dataset
model.DATAPOINTS = po.Set(dimen=2, initialize=data.keys())
# data values
model.data = po.Param(model.DATAPOINTS, initialize=data)
# create the x and y vectors
model.X = po.Var(model.ROWS, within=po.NonNegativeReals)
model.Y = po.Var(model.COLS, within=po.NonNegativeReals)
# create dummy variables to represent errors
model.ErrUp = po.Var(model.DATAPOINTS, within=po.NonNegativeReals)
model.ErrDown = po.Var(model.DATAPOINTS, within=po.NonNegativeReals)
# Force the error variables to match the error
def Calculate_Error_rule(model, r, c):
pred = model.X[r] + model.Y[c]
err = model.ErrUp[r, c] - model.ErrDown[r, c]
return (model.data[r, c] + err == pred)
model.Calculate_Error = po.Constraint(
model.DATAPOINTS, rule=Calculate_Error_rule
)
# Minimize the total error
def ClosestMatch_rule(model):
return sum(
model.ErrUp[r, c] + model.ErrDown[r, c]
for (r, c) in model.DATAPOINTS
)
model.ClosestMatch = po.Objective(
rule=ClosestMatch_rule, sense=po.minimize
)
# Solve the model
# get a solver object
opt = po.SolverFactory("glpk")
# solve the model
# turn off "tee" if you want less verbose output
results = opt.solve(model, tee=True)
# show solution status
print(results)
# show verbose description of the model
model.pprint()
# show X and Y values in the solution
for r in model.ROWS:
print('X[{}]: {}'.format(r, po.value(model.X[r])))
for c in model.COLS:
print('Y[{}]: {}'.format(c, po.value(model.Y[c])))
Just to complete the story, here's a solution that's closer to your original example. It uses cvxpy, but with the sparse data approach from my solution.
I don't know the "official" way to do elementwise calculations with cvxpy, but it seems to work OK to just use the standard Python sum function with a lot of individual cp.abs(...) calculations.
This gives a solution that is very slightly worse than the linear program, but you may be able to fix that by adjusting the solution tolerance.
import cvxpy as cp
import random
random.seed(1)
# Problem data.
# ~50% sparse data set
m = 10 # number of rows
n = 10 # number of cols
data = {
(i, j): random.random()
for i in range(m)
for j in range(n)
if random.random() >= 0.5
}
# Construct the problem.
x = cp.Variable(m)
y = cp.Variable(n)
objective = cp.Minimize(
sum(
cp.abs(x[i] + y[j] + data[i, j])
for (i, j) in data.keys()
)
)
prob = cp.Problem(objective)
result = prob.solve()
print(x.value)
print(y.value)
I did not get the idea, but just some hacky stuff based on the assumption:
you want some cvxpy-equivalent to numpy's broadcasting-rules behaviour on arrays (m, 1) + (1, n)
So numpy-wise:
m = 3
n = 4
np.random.seed(1)
a = np.random.randn(m, 1)
b = np.random.randn(1, n)
a
array([[ 1.62434536],
[-0.61175641],
[-0.52817175]])
b
array([[-1.07296862, 0.86540763, -2.3015387 , 1.74481176]])
a + b
array([[ 0.55137674, 2.48975299, -0.67719333, 3.36915713],
[-1.68472504, 0.25365122, -2.91329511, 1.13305535],
[-1.60114037, 0.33723588, -2.82971045, 1.21664001]])
Let's mimic this with np.kron, which has a cvxpy-equivalent:
aLifted = np.kron(np.ones((1,n)), a)
bLifted = np.kron(np.ones((m,1)), b)
aLifted
array([[ 1.62434536, 1.62434536, 1.62434536, 1.62434536],
[-0.61175641, -0.61175641, -0.61175641, -0.61175641],
[-0.52817175, -0.52817175, -0.52817175, -0.52817175]])
bLifted
array([[-1.07296862, 0.86540763, -2.3015387 , 1.74481176],
[-1.07296862, 0.86540763, -2.3015387 , 1.74481176],
[-1.07296862, 0.86540763, -2.3015387 , 1.74481176]])
aLifted + bLifted
array([[ 0.55137674, 2.48975299, -0.67719333, 3.36915713],
[-1.68472504, 0.25365122, -2.91329511, 1.13305535],
[-1.60114037, 0.33723588, -2.82971045, 1.21664001]])
Let's check cvxpy semi-blindly (we only dimensions; too lazy to setup a problem and fix variable to check the output :-D):
import cvxpy as cp
x = cp.Variable((m, 1))
y = cp.Variable((1, n))
cp.kron(np.ones((1,n)), x) + cp.kron(np.ones((m, 1)), y)
# Expression(AFFINE, UNKNOWN, (3, 4))
# looks good!
Now some caveats:
i don't know how efficient cvxpy can reason about this matrix-form internally
unclear if more efficient as a simple list-comprehension based form using cp.vstack and co (it probably is)
this operation itself kills all sparsity
(if both vectors are dense; your matrix is dense)
cvxpy and more or less all convex-optimization solvers are based on some sparsity assumption
scaling this problem up to machine-learning dimensions will not make you happy
there is probably a much more concise mathematical theory for your problem then to use (sparsity-assuming) (pretty) general (DCP implemented in cvxpy is a subset) convex-optimization
I wanted a very simple spring system written in numpy. The system would be defined as a simple network of knots, linked by links. I'm not interested in evaluating the system over time, but instead I want to go from an initial state, change a variable (usually move a knot to a new position) and solve the system until it reaches a stable state (last applied force is below a given threshold). The knots have no mass, there's no gravity, the forces are all derived from each link's current lengths/init lengths. And the only "special" variable is that each knot can bet set as "anchored" (doesn't move).
So I wrote this simple solver below, and included a simple example. Jump to the very end for my question.
import numpy as np
from numpy.core.umath_tests import inner1d
np.set_printoptions(precision=4)
np.set_printoptions(suppress=True)
np.set_printoptions(linewidth =150)
np.set_printoptions(threshold=10)
def solver(kPos, kAnchor, link0, link1, w0, cycles=1000, precision=0.001, dampening=0.1, debug=False):
"""
kPos : vector array - knot position
kAnchor : float array - knot's anchor state, 0 = moves freely, 1 = anchored (not moving)
link0 : int array - array of links connecting each knot. each index corresponds to a knot
link1 : int array - array of links connecting each knot. each index corresponds to a knot
w0 : float array - initial link length
cycles : int - eval stops when n cycles reached
precision : float - eval stops when highest applied force is below this value
dampening : float - keeps system stable during each iteration
"""
kPos = np.asarray(kPos)
pos = np.array(kPos) # copy of kPos
kAnchor = 1-np.clip(np.asarray(kAnchor).astype(float),0,1)[:,None]
link0 = np.asarray(link0).astype(int)
link1 = np.asarray(link1).astype(int)
w0 = np.asarray(w0).astype(float)
F = np.zeros(pos.shape)
i = 0
for i in xrange(cycles):
# Init force applied per knot
F = np.zeros(pos.shape)
# Calculate forces
AB = pos[link1] - pos[link0] # get link vectors between knots
w1 = np.sqrt(inner1d(AB,AB)) # get link lengths
AB/=w1[:,None] # normalize link vectors
f = (w1 - w0) # calculate force vectors
f = f[:,None] * AB
# Apply force vectors on each knot
np.add.at(F, link0, f)
np.subtract.at(F, link1, f)
# Update point positions
pos += F * dampening * kAnchor
# If the maximum force applied is below our precision criteria, exit
if np.amax(F) < precision:
break
# Debug info
if debug:
print 'Iterations: %s'%i
print 'Max Force: %s'%np.amax(F)
return pos
Here's some test data to show how it works. In this case i'm using a grid, but in reality this can be any type of network, like a string with many knots, or a mess of polygons...:
import cProfile
# Create a 5x5 3D knot grid
z = np.linspace(-0.5, 0.5, 5)
x = np.linspace(-0.5, 0.5, 5)[::-1]
x,z = np.meshgrid(x,z)
kPos = np.array([np.array(thing) for thing in zip(x.flatten(), z.flatten())])
kPos = np.insert(kPos, 1, 0, axis=1)
'''
array([[-0.5 , 0. , 0.5 ],
[-0.25, 0. , 0.5 ],
[ 0. , 0. , 0.5 ],
...,
[ 0. , 0. , -0.5 ],
[ 0.25, 0. , -0.5 ],
[ 0.5 , 0. , -0.5 ]])
'''
# Define the links connecting each knots
link0 = [0,1,2,3,5,6,7,8,10,11,12,13,15,16,17,18,20,21,22,23,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
link1 = [1,2,3,4,6,7,8,9,11,12,13,14,16,17,18,19,21,22,23,24,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24]
AB = kPos[link0]-kPos[link1]
w0 = np.sqrt(inner1d(AB,AB)) # this is a square grid, each link's initial length will be 0.25
# Set the anchor states
kAnchor = np.zeros(len(kPos)) # All knots will be free floating
kAnchor[12] = 1 # Middle knot will be anchored
This is what the grid looks like:
If we run my code using this data, nothing will happen since the links aren't pushing or stretching:
print np.allclose(kPos,solver(kPos, kAnchor, link0, link1, w0, debug=True))
# Returns True
# Iterations: 0
# Max Force: 0.0
Now lets move that middle anchored knot up a bit and solve the system:
# Move the center knot up a little
kPos[12] = np.array([0,0.3,0])
# eval the system
new = solver(kPos, kAnchor, link0, link1, w0, debug=True) # positions will have moved
#Iterations: 102
#Max Force: 0.000976603249133
# Rerun with cProfile to see how fast it runs
cProfile.run('solver(kPos, kAnchor, link0, link1, w0)')
# 520 function calls in 0.008 seconds
And here's what the grid looks like after being pulled by that single anchored knot:
Question:
My actual use cases are a little more complex than this example and solve a little too slow for my taste: (100-200 knots with a network anywhere between 200-300 links, solves in a few seconds).
How can i make my solver function run faster? I'd consider Cython but i have zero experience with C. Any help would be greatly appreciated.
Your method, at a cursory glance, appears to be an explicit under-relaxation type of method. Calculate the residual force at each knot, apply a factor of that force as a displacement, repeat until convergence. It's the repeating until convergence that takes the time. The more points you have, the longer each iteration takes, but you also need more iterations for the constraints at one end of the mesh to propagate to the other.
Have you considered an implicit method? Write the equation for the residual force at each non-constrained node, assemble them into a large matrix, and solve in one step. Information now propagates across the entire problem in a single step. As an additional benefit, the matrix you construct should be sparse, which scipy has a module for.
Wikipedia: explicit and implicit methods
EDIT Example of an implicit method matching (roughly) your problem. This solution is linear, so it doesn't take into account the effect of the calculated displacement on the force. You would need to iterate (or use non-linear techniques) to calculate this. Hope it helps.
#!/usr/bin/python3
import matplotlib.pyplot as pp
from mpl_toolkits.mplot3d import Axes3D
import numpy as np
import scipy as sp
import scipy.sparse
import scipy.sparse.linalg
#------------------------------------------------------------------------------#
# Generate a grid of knots
nX = 10
nY = 10
x = np.linspace(-0.5, 0.5, nX)
y = np.linspace(-0.5, 0.5, nY)
x, y = np.meshgrid(x, y)
knots = list(zip(x.flatten(), y.flatten()))
# Create links between the knots
links = []
# Horizontal links
for i in range(0, nY):
for j in range(0, nX - 1):
links.append((i*nX + j, i*nX + j + 1))
# Vertical links
for i in range(0, nY - 1):
for j in range(0, nX):
links.append((i*nX + j, (i + 1)*nX + j))
# Create constraints. This dict takes a knot index as a key and returns the
# fixed z-displacement associated with that knot.
constraints = {
0 : 0.0,
nX - 1 : 0.0,
nX*(nY - 1): 0.0,
nX*nY - 1 : 1.0,
2*nX + 4 : 1.0,
}
#------------------------------------------------------------------------------#
# Matrix i-coordinate, j-coordinate and value
Ai = []
Aj = []
Ax = []
# Right hand side array
B = np.zeros(len(knots))
# Loop over the links
for link in links:
# Link geometry
displacement = np.array([ knots[1][i] - knots[0][i] for i in range(2) ])
distance = np.sqrt(displacement.dot(displacement))
# For each node
for i in range(2):
# If it is not a constraint, add the force associated with the link to
# the equation of the knot
if link[i] not in constraints:
Ai.append(link[i])
Aj.append(link[i])
Ax.append(-1/distance)
Ai.append(link[i])
Aj.append(link[not i])
Ax.append(+1/distance)
# If it is a constraint add a diagonal and a value
else:
Ai.append(link[i])
Aj.append(link[i])
Ax.append(+1.0)
B[link[i]] += constraints[link[i]]
# Create the matrix and solve
A = sp.sparse.coo_matrix((Ax, (Ai, Aj))).tocsr()
X = sp.sparse.linalg.lsqr(A, B)[0]
#------------------------------------------------------------------------------#
# Plot the links
fg = pp.figure()
ax = fg.add_subplot(111, projection='3d')
for link in links:
x = [ knots[i][0] for i in link ]
y = [ knots[i][1] for i in link ]
z = [ X[i] for i in link ]
ax.plot(x, y, z)
pp.show()
so I've got a function:
def connection(n,m,r):
is_connected = ((x[n]-x[m])**2 + (y[n]-y[m])**2)**0.5
if is_connected < 2*r:
return n + " " + "connects with" + " " + m
else:
return "no connection"
This basically sees whether two circles (with coordinates that correspond to the indices n and m) connect. The n and m parameters refer to the indices in the data sets x and y, which come from a numpy.random array:
array([[ 0.31730234, 0.73662906],
[ 0.54488759, 0.09462212],
[ 0.07500703, 0.36148366],
[ 0.33200281, 0.04550565],
[ 0.3420866 , 0.9425797 ],
[ 0.36115391, 0.16670599],
[ 0.95586938, 0.52599398],
[ 0.13707665, 0.6574444 ],
[ 0.77766138, 0.56875582],
[ 0.79618595, 0.7139309 ]])
Since the array is basically 10 sets of coordinates, I have produced two lists out of them, x and y (x is the first column of the array, y is the second). m and n are indices in these lists. Therefore, n and m correspond to indices in the array, but I'm not sure how?
What I've been doing now is manually inputting the indices to see whether any two circles in this array connect - is there a -for loop that can do this in a more efficient way?
You should be doing things differently anyway. Unfortunatly the cKDTree which is much faster does not have the necessary features, but even the other KDTree should give you a vast speed increase (and solve it much more elegantly)
from scipy.spatial import KDTree
from itertools import chain
tree = KDTree(circles)
# unfortunatly only a list of lists, because there may be a different amount
# also the point itself is included every time.
connections = tree.query_ball_tree(tree, 2*r)
# if all you want is a list of lists of what connects with what
# connections is already what you need. The rest creates a connectivity matrix:
repeats = [len(l) for l in connections]
x_point = np.arange(len(circles)).repeat(repeats)
y_point = np.fromiter(chain(*connections), dtype=np.intp)
# or construct a sparse matrix here instead, scipy.sparse has some graph tools
# maybe it even has a better thing to do this.
connected = np.zeros((len(circles),) * 2, dtype=bool)
connected[x_point, y_point] = True
While it doesn't use cKDTree unfortunatly, this still saves you the O(N^2) complexity... Of course if len(circles) is small, that does not matter, but then you can just use broadcasting, (or also distance_matrix from scipy.spatial):
distances = np.sqrt(((circles[:,None,:] - circles)**2).sum(-1))
connected = distances < (2 * r)
# if you need the list of lists/arrays here you can do:
connections = [np.flatnonzero(c) for c in connected]
But note that the second method is a memory hungry monster and only any good if circles is small.
EDIT: Just realized what follows is just an expanded version of seberg's last method...
If your data sets are small, as in (very) few thousands of elements, you can brute force things with numpy:
import numpy as np
n = 10 # the number of circles
circles = np.random.rand(n, 2) # the array of centers
distances = circles.reshape(n, 1, 2) - circles.reshape(1, n, 2)
# distances now has shape (n, n, 2)
distances = np.sqrt(np.sum(distances**2, axis=2))
# distances now has shape (n, n)
# distances[i, j] holds the distance between the i-th and j-th circle centers
When you want to check which circles of radius r overlap, you can do something like this:
r = 0.1
overlap = distances < 2 * r
# overlap[i, j] is True if the i-th and j-th circle overlap, False if not
These last 2 lines you can reuse for any values of r you want, without having to do the more calculation intensive previous step.
It uses a lot of unnecessary memory, so it will break down for (moderately) large data sets, but since all loops are being run under hood by numpy, it should be fast.
you can use map
to do so, simply change connection to accept a circle as its parameter, and have the r as part of the circle
def connection(circle):
n, m, r = circle
is_connected = ((x[n]-x[m])**2 + (y[n]-y[m])**2)**0.5
if is_connected < 2*r:
return n + " " + "connects with" + " " + m
else:
return "no connection"
and have your list be a list of circles AND their radius.
circles = array([
[ 0.31730234, 0.73662906, r],
[ 0.54488759, 0.09462212, r],
[ 0.07500703, 0.36148366, r],
[ 0.33200281, 0.04550565, r],
[ 0.3420866 , 0.9425797 , r],
[ 0.36115391, 0.16670599, r],
[ 0.95586938, 0.52599398, r],
[ 0.13707665, 0.6574444 , r],
[ 0.77766138, 0.56875582, r],
[ 0.79618595, 0.7139309 , r]])
then simply do this:
map(connection, circles)
if the r is external, or a member then you want to use this:
def connection(coord):
n, m = coord
is_connected = ((x[n]-x[m])**2 + (y[n]-y[m])**2)**0.5
if is_connected < 2*r:
return n + " " + "connects with" + " " + m
else:
return "no connection"
coords = array([
[ 0.31730234, 0.73662906],
[ 0.54488759, 0.09462212],
[ 0.07500703, 0.36148366],
[ 0.33200281, 0.04550565],
[ 0.3420866 , 0.9425797 ],
[ 0.36115391, 0.16670599],
[ 0.95586938, 0.52599398],
[ 0.13707665, 0.6574444 ],
[ 0.77766138, 0.56875582],
[ 0.79618595, 0.7139309 ]])
map(connection, coords)
or you can keep your current format, and do a slightly uglier implementation. and i still dont know where you get the r from.
for item in circles:
connection(item[0], item[1], r)
A straightforward example would be:
# m is your array
rows = m.shape[0]
for x in range(rows):
for y in range(rows)[x+1:]:
# x, y correspond to indices of two circles
conn = connection(x, y, r)
After reviewing some of your responses, I have written you a new answer which changes your code entirely.
def check_overlap(circleA, circleB, r):
# your code.
# in each circle you have [x-coord, y-coord]
circles = [
[0.34, 0.74], [0.27, 0.19],
[0.24. 0.94], [0.64, 1.42]]
for a, b in ((a,b) for a in circles for b in circles):
if a != b:
check_overlap(a, b, r)
this makes it clear what you are doing in your code, python is all about readability.
note: the last function is the same as using itertools.product but without importing it.