Develop Reference

SGD on a Piecewise non-differentiable function

Setup for the problem:
I have a canvas which represents a city, each second I add a new resident to the city. The each resident has a job with a location that is randomly sampled from a distribution. Each resident also has a custom cost function that helps them decide where they want to live which they do by minimizing this cost function with respect to two variables x and y. So the function for example looks something like:
cost(x,y) = distance_to_job(x,y) + distance_to_center_of_city(x,y) + population_density(x,y)
where population_density(x,y) is just the population density at point (x,y). Naturally population_density(x,y) (without any transformations) is a piecewise non-differentiable function as one has to define a grid of blocks in the city and keep track of how many people per grid unit there is (think of a population density map of the world, each country has a distinct value that isn't necessarily the same as its neighbor, so if you were to map this on a 3-D plot, the function that you map would not be smooth).
Let me know if this setup is confusing, I'll try to make it a bit more clear.
The Question:
One could define a transformation where between each grid cell you designate a steep but smooth transition between the values of the piecewise function but as of now my population density function is not smooth and not differentiable at each boundary between grid cells. At first I did not think that SGD optimization in tensorflow would not work as I don't have a differentiable cost function but it seems to run fine. I am confused about what exactly is going on here and would love any clarification about how SGD optimization works and if my code is doing what I want it to.
Relevant Code:
def concentrationLookup(self, x, y):
r_index = int(x // (self.city.total_w / self.city.rows))
c_index = int(y // (self.city.total_h / self.city.cols))
return self.city.grid[r_index, c_index]
tf_jobCost = lambda x,y: (0.1/travelCost) * (tf.pow(x - self.jobx, 2) + tf.pow(y - self.joby, 2))
tf_cityCost = lambda x,y: 0.01 * (tf.pow(x - self.city.centerX, 2)) + 0.01*(tf.pow(y - self.city.centerY, 2))
xVar = tf.Variable(locX)
yVar = tf.Variable(locY)
self.costfn = lambda: tf_jobCost(xVar, yVar) + tf_cityCost(xVar, yVar) + self.concentrationLookup(xVar, yVar)
opt = tf.keras.optimizers.SGD(learning_rate = 3.0)
for _ in range(100):
opt.minimize(self.costfn, var_list = [xVar, yVar])
self.x = xVar.numpy()
self.y = yVar.numpy()

I believe it's treating population_density(x,y) as a constant function of x,y. In other words, it doesn't contribute to the gradient, and doesn't contribute to the solution.
You can also verify this by zeroing out other components of the loss, and verifying that opt.minimize() fails with something like ValueError: No gradients provided for any variable....
I think the solution should be to forget that the function is piecewise constant and non-differentiable, and instead to treat it as piecewise linear instead. In that case, concentrationLookup(x,y) can be written as returning a bilinearly-weighted sum of points at the 4 neighboring pixels.
Something like this:
def concentrationLookup(x, y):
r = x / (total_w / rows) # no quantizing
c = y / (total_h / cols) # no quantizing
r1, c1 = int(r), int(c) # lower bounds
r2, c2 = r1 + 1, c1 + 1 # upper bounds
w_r2, w_c2 = r - r1, c - c1
w_r1, w_c1 = 1.0 - w_r2, 1.0 - w_c2
# Assume constant boundary conditions.
c2 = tf.clip_by_value(c2, 0, grid.shape[1]-1)
c1 = tf.clip_by_value(c1, 0, grid.shape[1]-1)
r2 = tf.clip_by_value(r2, 0, grid.shape[0]-1)
r1 = tf.clip_by_value(r1, 0, grid.shape[0]-1)
return w_r1*w_c1*grid[r1, c1] + w_r2*w_c2*grid[r2,c2] + w_r1*w_c2*grid[r1,c2] + w_r2*w_c1*grid[r2, c1]
In this case, the gradient seems to be well defined.

Given a unit vector find the two angles of rotation that aligns that vector with an axis

So I'm not that well versed in linear algebra so I'm struggling with this.
I have a unit vectors v. I want to find two angles(angle 1, rotation around x-axis, and angle 2, rotation around z-axis) such that when I rotate v by them it aligns the vector v with the y-axis. From this question I have a function that can find the angle between arbitrary vectors and returns a rotation. But this function returns 3 angles. Essentially there is an infinite number of 3d rotation that aligns v with the y-axis so I want the two unique angles.
This the code I have now, it requires numpy and scipy:
import numpy as np
import random
from scipy.spatial.transform import Rotation as R
def rotation_from_unit_vectors(a, b):
v = np.cross(a, b)
c = np.dot(a, b)
s = np.linalg.norm(v)
kmat = np.array([[0, -v[2], v[1]], [v[2], 0, -v[0]], [-v[1], v[0], 0]])
rotation_matrix = np.eye(3) + kmat + kmat.dot(kmat) * ((1 - c) / (s ** 2))
return R.from_matrix(rotation_matrix)
y_axis = np.asarray([0.0, 1.0, 0.0])
alpha = random.uniform(0, 10)
beta = random.uniform(0, 10)
gamma = random.uniform(0, 10)
v = np.asarray([alpha, beta, gamma])
v = v / np.linalg.norm(v)
r = rotation_from_unit_vectors(v, y_axis)
print(r.as_euler('xyz', degrees = True))
print(r.apply(v))

Taking advantage of the fixed target alignment, this can be done in a straightforward manner with just trigonometry:
import math
def to_y(x,y,z):
rx=-math.atan2(z,y) # or +math.atan2(z,-y)
y2=y*math.cos(rx)-z*math.sin(rx) # -> (x,y2,0)
return rx,math.atan2(x,y2)
The rotations are defined as counterclockwise when looking at the origin from +x or +z (the right-hand rule); the rotation direction is always that with the smaller magnitude, but it may be possible to find a physically smaller rotation as indicated in the comment. Note that the input need not be normalized, and NaN is never produced (unless it appears in the input).

Hum, non-standard problem, required thinking a little.
Given v1 and v2 you want to rotate_z(rotate_x(v1, alpha), beta) to be on the same direction as v2.
We know that the aligned vector can be obtained by simply scaling scaling v2, this will gives x1,y3,z3 = v3 = v2 * |v1| / |v2|. Since rotation around z-axis, does not affect the z coordinate, we can determine alpha such that the z coordinate of rotate_x(v1, alpha) equals z3. After that we determine the angle beta to align place the X and Y coordinates properly
import numpy as np
def alignment_angles(v1, v2):
x1,y1,z1 = v1 # initial vector
x2,y2,z2 = v2 # reference vector
# magnitude of the two vectors
r1 = np.sqrt(x1**2 + y1**2 + z1**2)
r2 = np.sqrt(x2**2 + y2**2 + z2**2)
# this will be the result when aligning v1 to v2
# it has the magnitude of v1 and the direction of v2
x3,y3,z3 = x2*r1/r2, y2*r1/r2, z2*r1/r2
# the rotation in x must set the z coordinate to the
# final value, since the rotation over the z axis will
# not affect the z coordinate (this have two solutions)
rho1 = np.sqrt(z1**2 + y1**2)
if(abs(z3 / rho1) > 1):
raise ValueError('Cannot align these vectors')
alpha = np.arcsin(z3 / rho1) - np.arctan2(z1, y1);
# apply the rotation to make easier to calcualte the next stage
y1, z1 = (y1 * np.cos(alpha) - z1 * np.sin(alpha),
y1 * np.sin(alpha) + z1 * np.cos(alpha))
np.allclose(rho1, np.sqrt(z1**2 + y1**2))
#assert(np.allclose(z1, z3))
# now it is just a matter of aligning (x1, y1) to (x3, y3)
beta = np.arctan2(y3, x3) - np.arctan2(y1, x1)
x1, y1 = (x1 * np.cos(beta) - y1 * np.sin(beta),
x1 * np.sin(beta) + y1 * np.cos(beta))
# ensure the fotated v1 was correctly aligned
assert(np.allclose([x1, y1, z1], [x3, y3, z3]))
return alpha, beta
Then you just call
alignment_angles((1,2,3), (3,4,5))
or you can also use numpy arrays with 3 rows.
Initially I thought it would be an application of spherical coordinates, that would be the case if the axis for the second rotation was the z-axis rotated accordingly to the first rotation.
Edit
There are some vectors that cannot be aligned with a rotation on x and a rotation on y.
Suppose you want to align the vector v1 = (1, 0, 0) to the vector v2 = (0, 0, 1) the rotation in x will not affect v1, it will always point in the direction x, then when you rotate around the z axis it will always be on the XY plan.
The example you gave was really giving the wrong answer because asin is not injective.
I changed the function to raise a value error when you cannot align the given vectors.

Generating solid spheres in python [duplicate]

This question already has an answer here:
How to generate uniform random points inside d-dimension ball / sphere?
(1 answer)
Closed 1 year ago.
I'd like to generate random uniform samples from n-dimensional solid spheres.
My current method is this
def sample_sphere(d, npoints):
points = np.zeros((npoints, d))
for i in range(npoints):
r = np.random.rand() #random radius
v = np.random.uniform(low= -1, high=1, size=d) #random direction
v = v / np.linalg.norm(v)
points[i] = r * v
return points
But unsurpringly, this method gives a higher concentration of points near 0, since the sampling density is not proportional to the volume:
How can I sample uniformly?

There are two basic interpretations of a spherical distribution:
1. Bounded solid sphere
The probability of getting a point at a given radius is given by the volume of a shell with thickness dr at that radius: p(r) ~ r^D up to a constant. The radial CDF is therefore (r / R)^(D+1), where R is the outer radius and D is the dimensionality. The sphere has to be bounded unless you want to select from all of space.
A standard way to generate samples given the CDF is to invert it:
random_radius = R * np.pow(np.random.uniform(0, 1), 1 / D)
By the way, a common way of picking properly uniform directions is to use the Gaussian distribution (see here):
random_direction = np.random.normal(size=D)
random_direction /= np.linalg.norm(random_direction)
You can, and should, vectorize your function. There is no need for looping:
def sample_sphere(r, d, n):
radii = r * np.pow(1 - np.random.uniform(0, 1, size=(n, 1)), 1 / d)
directions = np.random.normal(size=(n, d))
directions *= radii / np.linalg.norm(directions, axis=1, keepdims=True)
return directions
Notice the 1 - np.random.uniform(...). This is because the range of the function is [0, 1), while we want (0, 1]. The subtraction inverts the range without affecting the uniformity.
2. Unbounded decaying distribution
In this case you're looking for something like a Gaussian with spherical symmetry. The radial CDF of such a distribution scales with R^(D-1). For the Gaussian, this is called the Rayleigh distribution in 2D, Maxwell in 3D, and chi in the general case. These are available directly through the scipy.stats.rayleigh, scipy.stats.maxwell and scipy.stats.chi objects, respectively.
In this case, radius has less meaning, but it can be interpreted as the standards deviation of your distribution.
The unbounded version of your function (also vectorized), then becomes:
def sample_sphere(s, d, n):
radii = scipy.stats.chi.rvs(df=d, scale=s, size=(n, 1))
directions = np.random.normal(size=(n, d))
directions *= radii / np.linalg.norm(directions, axis=1, keepdims=True)
return directions

Tiling fixed-size rectangles to cover a given set of points

The problem - given a list of planar points [p_1, ..., p_n] and the dimensions of some rectangle w, h, find the minimal set of rectangles w, h that cover all points (edit - the rectangles are not rotated).
My inital solution was:
find the bounding-box of all points
divide the width and height of the bounding-box by the w, h of the given rectangle and round the number up to get the number of instances of the rectangle in x and y
to further optimize, go through all rectangles and delete the ones that have zero points inside them.
An example in Python:
def tile_rect(points, rect):
w, h = rect
xs = [p.x for p in points]
ys = [p.y for p in points]
bbox_w = abs(max(xs) - min(xs))
bbox_h = abs(max(ys) - min(ys))
n_x, n_y = ceil(bbox_w / w), ceil(bbox_h / h)
rect_xs = [(min(xs) + n * w for n in range(n_x)]
rect_ys = [(min(ys) + n * h for n in range(n_y)]
rects = remove_empty(rect_xs, rect_ys)
return rects
How can I do better? What algorithm can I use to decrease the number of rectangles?

To discretize the problem for integer programming, observe that given a rectangle we can slide it in the +x and +y directions without decreasing the coverage until the min x and the min y lines both have a point on them. Thus the integer program is just the standard min cover:
minimize sum_R x_R
subject to
for every point p, sum_{R contains p} x_R >= 1
x_R in {0, 1}
where R ranges over all rectangles whose min x is the x of some point and whose min y is the y of some point (not necessarily the same point).
Demo Python:
import random
from ortools.linear_solver import pywraplp
w = 0.1
h = 0.1
points = [(random.random(), random.random()) for _ in range(100)]
rectangles = [(x, y) for (x, _) in points for (_, y) in points]
solver = pywraplp.Solver.CreateSolver("min cover", "SCIP")
objective = solver.Objective()
constraints = [solver.RowConstraint(1, pywraplp.inf, str(p)) for p in points]
variables = [solver.BoolVar(str(r)) for r in rectangles]
for (x, y), var in zip(rectangles, variables):
objective.SetCoefficient(var, 1)
for (px, py), con in zip(points, constraints):
if x <= px <= x + w and y <= py <= y + h:
con.SetCoefficient(var, 1)
solver.Objective().SetMinimization()
solver.Solve()
scale = 6 * 72
margin = 72
print(
'<svg width="{}" height="{}">'.format(
margin + scale + margin, margin + scale + margin
)
)
print(
'<text x="{}" y="{}">{} rectangles</text>'.format(
margin // 2, margin // 2, round(objective.Value())
)
)
for x, y in points:
print(
'<circle cx="{}" cy="{}" r="3" fill="none" stroke="black"/>'.format(
margin + x * scale, margin + y * scale
)
)
for (x, y), var in zip(rectangles, variables):
if var.solution_value():
print(
'<rect x="{}" y="{}" width="{}" height="{}" fill="none" stroke="rgb({},{},{})"/>'.format(
margin + x * scale,
margin + y * scale,
w * scale,
h * scale,
random.randrange(192),
random.randrange(192),
random.randrange(192),
)
)
print("</svg>")
Example output:

Assuming an approximate, rather than optimal solution is acceptable, how about a routine generally like:
Until no points are left:
(1) Find the convex hull of the remaining points.
(2) Cover each point/s on the hull so the
covering rectangles extend "inward."
(Perhaps examine neighbouring hull points
to see if a rectangle can cover more than one.)
(3) Remove the newly covered points.
Clearly, the orientation of the covering rectangles has an effect on the procedure and result. I think there is a way to combine (1) and (3), or possibly rely on a nested convex hull, but I don't have too much experience with those.

This is can be transformed into a mostly standard set cover problem. The general steps are as follows, given n points in the plane.
First, generate all possible maximally inclusive rectangles, of which there are at most n^2, named R. The key insight is that given a point p1 with coordinates (x1, y1), use x1 as the leftmost bound for a set of rectangles. For all other points p2 with (x2,y2) where x1 <= x2 <= x1+w and where y1-h <= y2 <= y1+h, generate a rectangle ((x1, y2), (x1+w, y2+h)).
For each rectangle r generated, count the points included in that rectangle cover(r).
Choose a subset of the rectangles R, s, such that all points are in Union(r in s) cover(r)
Now, the tricky part is that last step. Fortunately, it is a standard problem and there are many algorithms suggested in the literature. For example, combinatorial optimization solvers (such as SAT solvers, MIP solvers, and Constraint programming solvers) can be used.
Note that the above re-formulation only works if it is ok for rectangles to cover each other. It might be the case that the generated set of rectangles is not enough to find the least set of rectangles that do not overlap.

Calculating MSE between numpy arrays

The Scientific Question:
I have lots of 3D volumes all with a cylinder in them orientated with the cylinder 'upright' on the z axis. The volumes containing the cylinder are incredibly noisy, like super noisy you can't see the cylinder in them as a human. If I average together 1000s of these volumes I can see the cylinder. Each volume contains a copy of the cylinder but in a few cases the cylinder may not be orientated correctly so I want a way of figuring this out.
The Solution I have come up with:
I have taken the averaged volume and projected it down the z and x axis (just projecting the numpy array) so that I get a nice circle in one direction and a rectangle in the other. I then take each 3D volume and project every single one down the Z axis. The SNR is still so bad that I cannot see a circle but if I average the 2D slices I can begin to see a circle after averaging a few hundred and it is easy to see after the first 1000 are averaged. To calculate a score of how each volume I figured calculating the MSE of the 3D volumes projected down z against three other arrays, the first would be the average projected down Z, then the average projected down y or x, and finally an array with a normal distribution of noise in it.
Currently I have the following where RawParticle is the 3D data and Ave is the average:
def normalise(array):
min = np.amin(array)
max = np.amax(array)
normarray = (array - min) / (max - min)
return normarray
def Noise(mag):
NoiseArray = np.random.normal(0, mag, size=(200,200,200))
return NoiseArray
#3D volume (normally use a for loop to iterate through al particles but for this example just showing one)
RawParticleProjected = np.sum(RawParticle, 0)
RawParticleProjectedNorm = normalise(RawParticleProjected)
#Average
AveProjected = np.sum(Ave, 0)
AveProjectedNorm = normalise(AveProjected)
#Noise Array
NoiseArray = Noise(0.5)
NoiseNorm = normalise(NoiseArray)
#Mean squared error
MSE = (np.square(np.subtract(RawParticleProjectedNorm, AveProjectedNorm))).mean()
I then repeat this with the Ave summed down axis 1 and then again compared the Raw particle to the Noise array.
However my output from this gives highest MSE when I am comparing the projections that should both be circles as shown below:
My understanding of MSE is that the other two populations should have high MSE and my populations that agree should have low MSE. Perhaps my data is too noisy for this type of analysis? but if that is true then I don't really know how to do what I am doing.
If anyone could glance at my code or enlighten my understanding of MSE I would be super appreciative.
Thank you for taking the time to look and read.

If I understood your question correctly you want to figure how close your different samples are to the average.
And by comparing the samples you want to find the outliers which contain a disoriented cylinder.
This fits pretty well to the definition of the L2 norm, so MSE should work here.
I would calculate the average 3D image of all samples and than compute the distance of each sample to this average. Then I would just compare those values.
The idea of comparing the samples to an image of artificial noise is not bad, but I am not sure if a normal distribution and your normalization work out as you planned. I could be apple and oranges.
And I don't think it is a good idea to look at projections along different axis,
just compare the 3D images.
I made some small tests with a circle in 2D with a parameter alpha which indicates how much noise and how much circle there is in a picture.
(alpha=0 means only noise, alpha=1 means only circle`)
import numpy as np
import matplotlib.pyplot as plt
grid_size = 20
radius = 5
mag = 1
def get_circle_stencil(radius):
xx, yy = np.meshgrid(np.linspace(-grid_size/2+1/2, grid_size/2-1/2, grid_size),
np.linspace(-grid_size/2+1/2, grid_size/2-1/2, grid_size))
dist = np.sqrt(xx**2 + yy**2)
inner = dist < (radius - 1/2)
return inner.astype(float)
def create_noise(mag, n_dim=2):
# return np.random.normal(0, mag, size=(grid_size,)*n_dim)
return np.random.uniform(0, mag, size=(grid_size,)*n_dim)
def create_noisy_sample(alpha, n_dim=2):
return (np.random.uniform(0, 1-alpha, size=(grid_size,)*n_dim) +
alpha*get_circle_stencil(radius))
fig = plt.figure()
ax = fig.subplots(nrows=3, ncols=3)
np.unravel_index(3, shape=(3, 3))
alpha_list = np.arange(9) / 10
for i, alpha in enumerate(alpha_list):
r, c = np.unravel_index(i, shape=(3, 3))
ax[r][c].imshow(*norm(create_noisy_sample(alpha=alpha)), cmap='Greys')
ax[r][c].set_title(f"alpha={alpha}")
ax[r][c].xaxis.set_ticklabels([])
ax[r][c].yaxis.set_ticklabels([])
Than I tried some metrics (mse, cosine similarity and binary cross entropy
and looked how they behave for different values of alpha.
def normalize(*args):
return [a / np.linalg.norm(a) for a in args]
def cosim(a, b):
return np.sum(a * b)
def mse(a, b):
return np.sqrt(np.sum((a-b)**2))
def bce(a, b):
# binary cross entropy implemented from tensorflow / keras
eps = 1e-7
res = a * np.log(b + eps)
res += (1 - a) * np.log(1 - b + eps)
return np.mean(-res)
I compared NoiseA-NoiseB, Circle-Circle, Circle-Noise, Noise-Sample, Circle-Sample
alpha = 0.1
noise = create_noise(mag=1, grid_size=grid_size)
noise_b = create_noise(mag=1, grid_size=grid_size)
circle_reference = get_circle_stencil(radius=radius, grid_size=grid_size)
sample = create_noise(mag=1, grid_size=grid_size) + alpha * circle_reference
print('NoiseA-NoiseB:', mse(*norm(noise, noise_b))) # 0.718
print('Circle-Circle:', mse(*norm(circle, circle))) # 0.000
print('Circle-Noise:', mse(*norm(circle, noise))) # 1.168
print('Noise-Sample:', mse(*norm(noise, sample))) # 0.697
print('Circle-Sample:', mse(*norm(circle, sample))) # 1.100
print('NoiseA-NoiseB:', cosim(*norm(noise, noise_b))) # 0.741
print('Circle-Circle:', cosim(*norm(circle, circle))) # 1.000
print('Circle-Noise:', cosim(*norm(circle, noise))) # 0.317
print('Noise-Sample:', cosim(*norm(noise, sample))) # 0.757
print('Circle-Sample:', cosim(*norm(circle, sample))) # 0.393
print('NoiseA-NoiseB:', bce(*norm(noise, noise_b))) # 0.194
print('Circle-Circle:', bce(*norm(circle, circle))) # 0.057
print('Circle-Noise:', bce(*norm(circle, noise))) # 0.111
print('Noise-Circle:', bce(*norm(noise, circle))) # 0.636
print('Noise-Sample:', bce(*norm(noise, sample))) # 0.192
print('Circle-Sample:', bce(*norm(circle, sample))) # 0.104
n = 1000
ns = np.zeros(n)
cs = np.zeros(n)
for i, alpha in enumerate(np.linspace(0, 1, n)):
sample = create_noisy_sample(alpha=alpha)
ns[i] = mse(*norm(noise, sample))
cs[i] = mse(*norm(circle, sample))
fig, ax = plt.subplots()
ax.plot(np.linspace(0, 1, n), ns, c='b', label='noise-sample')
ax.plot(np.linspace(0, 1, n), cs, c='r', label='circle-sample')
ax.set_xlabel('alpha')
ax.set_ylabel('mse')
ax.legend()
For your problem I would just look at the comparison circle-sample (red).
Different samples will behave as if they have different alpha values and you can group them accordingly. And you should be able to detect outliers because they should have an higher mse.
You said you have to combine 100-1000 pictures to see the cylinder, which indictas you have an really small alpha value in your problem, but in average mse
should work.