Develop Reference

Identify the grid particles belong to

A square box of size 10,000*10,000 has 10,00,000 particles distributed uniformly. The box is divided into grids, each of size 100*100. There are 10,000 grids in total. At every time-step (for a total of 2016 steps), I would like to identify the grid to which a particle belongs. Is there an efficient way to implement this in python? My implementation is as below and currently takes approximately 83s for one run.
import numpy as np
import time
start=time.time()
# Size of the layout
Layout = np.array([0,10000])
# Total Number of particles
Population = 1000000
# Array to hold the cell number
cell_number = np.zeros((Population),dtype=np.int32)
# Limits of each cell
boundaries = np.arange(0,10100,step=100)
cell_boundaries = np.dstack((boundaries[0:100],boundaries[1:101]))
# Position of Particles
points = np.random.uniform(0,Layout[1],size = (Population,2))
# Generating a list with the x,y boundaries of each cell in the grid
x = []
limit_list = cell_boundaries
for i in range(0,Layout[1]//100):
for j in range(0,Layout[1]//100):
x.append([limit_list[0][i,0],limit_list[0][i,1],limit_list[0][j,0],limit_list[0][j,1]])
# Identifying the cell to which the particles belong
i=0
for y in (x):
cell_number[(points[:,1]>y[0])&(points[:,1]<y[1])&(points[:,0]>y[2])&(points[:,0]<y[3])]=i
i+=1
print(time.time()-start)

I am not sure about your code. You seem to be accumulating the i variable globally. While it should be accumulated on a per cell basis, correct? Something like cell_number[???] += 1, maybe?
Anyhow, the way I see is from a different perspective. You could start by assigning each point a cell id. Then inverse the resulting array with a kind of counter function. I have implemented the following in PyTorch, you will most likely find equivalent utilities in Numpy.
The conversion from 2-point coordinates to cell ids corresponds to applying floor on the coordinates then unfolding them according to your grid's width.
>>> p = torch.from_numpy(points).floor()
>>> p_unfold = p[:, 0]*10000 + p[:, 1]
Then you can "inverse" the statistics, i.e. find out how many particles there are in each respective cell based on the cell ids. This can be done using PyTorch histogram's counter torch.histc:
>>> torch.histc(p_unfold, bins=Population)

Packing hard spheres in a box

I am trying to pack hard-spheres in a unit cubical box, such that these spheres cannot overlap on each other. This is being done in Python.
I am given some packing fraction f, and the number of spheres in the system is N.
So, I say that the diameter of each sphere will be
d = (p*6/(math.pi*N)**)1/3).
My box has periodic boundary conditions - which means that there is a recurring image of my box in all direction. If there is a particle who is at the edge of the box and has a portion of it going beyond the wall, it will stick out at the other side.
My attempt:
Create a numpy N-by-3 array box which holds the position vector of each particle [x,y,z]
The first particle is fine as it is.
The next particle in the array is checked with all the previous particles. If the distance between them is more than d, move on to the next particle. If they overlap, randomly change the position vector of the particle in question. If the new position does not overlap with the previous atoms, accept it.
Repeat steps 2-3 for the next particle.
I am trying to populate my box with these hard spheres, in the following manner:
for i in range(1,N):
mybool=True
print("particles in box: " + str(i))
while (mybool): #the deal with this while loop is that if we place a bad particle, we need to change its position, and restart the process of checking
for j in range(0,i):
displacement=box[j,:]-box[i,:]
for k in range(3):
if abs(displacement[k])>L/2:
displacement[k] -= L*np.sign(displacement[k])
distance = np.linalg.norm(displacement,2) #check distance between ith particle and the trailing j particles
if distance<diameter:
box[i,:] = np.random.uniform(0,1,(1,3)) #change the position of the ith particle randomly, restart the process
break
if j==i-1 and distance>diameter:
mybool = False
break
The problem with this code is that if p=0.45, it is taking a really, really long time to converge. Is there a better method to solve this problem, more efficiently?

I think what you are looking for is either the hexagonal closed-packed (HCP or sometime called face-centered cubic, FCC) lattice or the cubic closed-packed one (CCP). See e.g. Wikipedia on Close-packing of equal spheres.
Since your space has periodic conditions, I believe it doesn't matter which one you chose (hcp or ccp), and they both achieve the same density of ~74.04%, which was proved by Gauss to be the highest density by lattice packing.
Update:
For the follow-up question on how to generate efficiently one such lattice, let's take as an example the HCP lattice. First, let's create a bunch of (i, j, k) indices [(0,0,0), (1,0,0), (2,0,0), ..., (0,1,0), ...]. Then, get xyz coordinates from those indices and return a DataFrame with them:
def hcp(n):
dim = 3
k, j, i = [v.flatten()
for v in np.meshgrid(*([range(n)] * dim), indexing='ij')]
df = pd.DataFrame({
'x': 2 * i + (j + k) % 2,
'y': np.sqrt(3) * (j + 1/3 * (k % 2)),
'z': 2 * np.sqrt(6) / 3 * k,
})
return df
We can plot the result as scatter3d using plotly for interactive exploration:
import plotly.graph_objects as go
df = hcp(12)
fig = go.Figure(data=go.Scatter3d(
x=df.x, y=df.y, z=df.z, mode='markers',
marker=dict(size=df.x*0 + 30, symbol="circle", color=-df.z, opacity=1),
))
fig.show()
Note: plotly's scatter3d is not a very good rendering of spheres: the marker sizes are constant (so when you zoom in and out, the "spheres" will appear to change relative size), and there is no shading, limited z-ordering faithfulness, etc., but it's convenient to interact with the plot.
Resize and clip to the unit box:
Here, a strict clipping (each sphere needs to be completely inside the unit box). Your "periodic boundary condition" is something you will need to address separately (see further below for ideas).
def hcp_unitbox(r):
n = int(np.ceil(1 / (np.sqrt(3) * r)))
df = hcp(n) * r
df += r
df = df[(df <= 1 - r).all(axis=1)]
return df
With this, you find that a radius of 0.06 gives you 608 fully enclosed spheres:
hcp_unitbox(.06).shape # (608, 3)
Where you would go next:
You may dig deeper into the effect of your so-called "periodic boundary conditions", and perhaps play with some rotations (and small translations).
To do so, you may try to generate an HCP-lattice that is large enough that any rotation will still fully enclose your unit cube. For example:
r = 0.2 # example
n = int(np.ceil(2 / r))
df = hcp(n) * r - 1
Then rotate it (by any amount) and translate it (by up to 1 radius in any direction) as you wish for your research, and clip. The "periodic boundary conditions", as you call them, present a bit of extra challenge, as the clipping becomes trickier. First, clip any sphere whose center is outside your box. Then select spheres close enough to the boundaries, or even partition the regions of interest into overlapping regions along the walls of your cube, then check for collisions among the spheres (as per your periodic boundary conditions) that fall in each such region.

Return searchlight vectors for a given numpy array

Consider a 3D numpy array D of dimension, say, (30 x 40 x 50). For each voxel D[x,y,z] I want to store a vector that contains neighboring voxels within a certain radius (including the D[x,y,z] itself).
(As an example here is a picture of such a sphere of radius 2: https://puu.sh/wwIYW/e3bd63ceae.png)
Is there a simple and fast way to code this?
I have written a function for it, but it is painfully slow and IDLE eventually crashes because the data structure I store the vectors in becomes too large.
Current code:
def searchlight(M_in):
radius = 4
[m,n,k] = M_in.shape
M_out = np.zeros([m,n,k],dtype=object)
count = 0
for i in range(m):
for j in range(n):
for z in range(k):
i_interval = list(range((i-4),(i+5)))
j_interval = list(range((j-4),(j+5)))
z_interval = list(range((z-4),(z+5)))
coordinates = list(itertools.product(i_interval,j_interval,z_interval))
coordinates = [pair for pair in coordinates if ((abs(pair[0]-i)+abs(pair[1]-j)+abs(pair[2]-z))<=radius)]
coordinates = [pair for pair in coordinates if ((pair[0]>=0) and (pair[1]>=0) and pair[2]>=0) and (pair[0]<m) and (pair[1]<n) and (pair[2]<k)]
out = []
for pair in coordinates:
out.append(M_in[pair[0],pair[1],pair[2]])
M_out[i,j,z] = out
count = count +1
return M_out

Here a way to do that. For efficiency, you need therefore to use ndarrays : This only take in account complete voxels. Edges must be managed "by hand".
from pylab import *
a=rand(100,100,100) # the data
r=4
ra=range(-r,r+1)
sphere=array([[x,y,z] for x in ra for y in ra for z in ra if np.abs((x,y,z)).sum()<=r])
# the unit "sphere"
indcenters=array(meshgrid(*(range(r,n-r) for n in a.shape),indexing='ij'))
# indexes of the centers of the voxels. edges are cut.
all_inds=(indcenters[newaxis].T+sphere.T).T
#all the indexes.
voxels=np.stack([a[tuple(inds)] for inds in all_inds],-1)
# the voxels.
#voxels.shape is (92, 92, 92, 129)
All the costly operations are vectorized. Comprehension lists are prefered for clarity in external loop.
You can now perform vectorized operations on voxels. for exemple the brightest voxel :
light=voxels.sum(-1)
print(np.unravel_index(light.argmax(),light.shape))
#(33,72,64)
All of this is of course extensive in memory. you must split your space for
big data or voxels.

Since you say the data structure is too large, you'll likely have to compute the vector on the fly for a given voxel. You can do this pretty quickly though:
class SearchLight(object):
def __init__(self, M_in, radius):
self.M_in = M_in
m, n, k = self.M_in.shape
# compute the sphere coordinates centered at (0,0,0)
# just like in your sample code
i_interval = list(range(-radius,radius+1))
j_interval = list(range(-radius,radius+1))
z_interval = list(range(-radius,radius+1))
coordinates = list(itertools.product(i_interval,j_interval,z_interval))
coordinates = [pair for pair in coordinates if ((abs(pair[0])+abs(pair[1])+abs(pair[2]))<=radius)]
# store those indices as a template
self.sphere_indices = np.array(coordinates)
def get_vector(self, i, j, k):
# offset sphere coordinates by the requested centre.
coordinates = self.sphere_indices + [i,j,k]
# filter out of bounds coordinates
coordinates = coordinates[(coordinates >= 0).all(1)]
coordinates = coordinates[(coordinates < self.M_in.shape).all(1)]
# use those coordinates to index the initial array.
return self.M_in[coordinates[:,0], coordinates[:,1], coordinates[:,2]]
To use the object on a given array you can simply do:
sl = SearchLight(M_in, 4)
# get vector of values for voxel i,j,k
vector = sl.get_vector(i,j,k)
This should give you the same vector you would get from
M_out[i,j,k]
in your sample code, without storing all the results at once in memory.
This can also probably be further optimized, particularly in terms of the coordinate filtering, but it may not be necessary. Hope that helps.

Find irregular region in 4D numpy array of gridded data (lat/lon)

I have a large 4-dimensional dataset of Temperatures [time,pressure,lat,lon].
I need to find all grid points within a region defined by lat/lon indices and calculate an average over the region to leave me with a 2-dimensional array.
I know how to do this if my region is a rectangle (or square) but how can this be done with an irregular polygon?
Below is an image showing the regions I need to average together and the lat/lon grid the data is gridded to in the array

I believe this should solve your problem.
The code below generates all cells in a polygon defined by a list of vertices.
It "scans" the polygon row by row keeping track of the transition columns where you (re)-enter or exit the polygon.
def row(x, transitions):
""" generator spitting all cells in a row given a list of transition (in/out) columns."""
i = 1
in_poly = True
y = transitions[0]
while i < len(transitions):
if in_poly:
while y < transitions[i]:
yield (x,y)
y += 1
in_poly = False
else:
in_poly = True
y = transitions[i]
i += 1
def get_same_row_vert(i, vertices):
""" find all vertex columns in the same row as vertices[i], and return next vertex index as well."""
vert = []
x = vertices[i][0]
while i < len(vertices) and vertices[i][0] == x:
vert.append(vertices[i][1])
i += 1
return vert, i
def update_transitions(old, new):
""" update old transition columns for a row given new vertices.
That is: merge both lists and remove duplicate values (2 transitions at the same column cancel each other)"""
if old == []:
return new
if new == []:
return old
o0 = old[0]
n0 = new[0]
if o0 == n0:
return update_transitions(old[1:], new[1:])
if o0 < n0:
return [o0] + update_transitions(old[1:], new)
return [n0] + update_transitions(old, new[1:])
def polygon(vertices):
""" generator spitting all cells in the polygon defined by given vertices."""
vertices.sort()
x = vertices[0][0]
transitions, i = get_same_row_vert(0, vertices)
while i < len(vertices):
while x < vertices[i][0]:
for cell in row(x, transitions):
yield cell
x += 1
vert, i = get_same_row_vert(i, vertices)
transitions = update_transitions(transitions, vert)
# define a "strange" polygon (hook shaped)
vertices = [(0,0),(0,3),(4,3),(4,0),(3,0),(3,2),(1,2),(1,1),(2,1),(2,0)]
for cell in polygon(vertices):
print cell
# or do whatever you need to do

The general class of problems is called "Point in Polygon", where the (fairly) standard algorithm is based on drawing a test line through the point under consideration and counting the number of times it crosses polygon boundaries (its really cool/weird that it works so simply, I think). This is a really good overview which includes implementation information.
For your problem in particular, since each of your regions are defined based on a small number of square cells - I think a more brute-force approach might be better. Perhaps something like:
For each region, form a list of all of the (lat/lon) squares which define it. Depending on how your regions are defined, this may be trivial, or annoying...
For each point you are examining, figure out which square it lives in. Since the squares are so well behaves, you can do this manually using opposite corners of each square, or using a method like numpy.digitize.
Test whether the square the point lives in, is in one of the regions.
If you're still having trouble, please provide some more details about your problem (specifically, how your regions are defined) --- that will make it easier to offer advice.

Converting G Force data to rotation?

I've been playing around with an accelerometer with 3 axis: X, Y and Z. It says on the supplier's site that it measures gravitational force.
I'm sending this data to the blender games engine where I am rotating a cube in real time depending on the data values coming from the accelerometer. However the values coming through don't seem to match up.
On each axis the accelerometer spits out values from -700 to 700 on each axis and I need to convert these values to something I can use in Blender. My maths knowledge is not up to scratch so I don't know where to start with this one.
If anybody could shed some light on this, that would be great.
Many thanks
Will
EDIT
Currently I'm using a bit of python code to convert the rotation values to a matrix:
def reorient(alpha, beta, gamma):
a = math.cos(alpha)
b = math.sin(alpha)
c = math.cos(beta)
d = math.sin(beta)
e = math.cos(gamma)
f = math.sin(gamma)
ad = a*d
bd = b*d
matrix = [[c*e, -a*f+b*d*e, b*f+a*d*e], [c*f, a*e+b*d*f, -b*e+a*d*f], [-d, b*c, a*c]]
return matrix
I am then using setOrientation(matrix) to affect the rotation of the cube. However I am currently throwing the wrong values into the matrix reorient() function

I guess you are using the measured acceleration to find the direction of gravitational pull (ie, down). If you are moving the accelerometer, apart from just turning it, there will be some additional force; think of the accelerometer having a pendulum weight handing from it, as you move it the pendulum sways (although in this case it would be a very short, fast-reacting pendulum?). You could try doing some sort of movement compensation, but it might be simpler to just try to keep the sensor in a fixed location.
Edit: ok, it looks like I totally misread the question - you want to know how to do the rotation in a script?
It looks like each Blender object has three properties (.RotX, .RotY, .RotZ) which contain the current values (in radians) and a method (.rot(new_rotx, new_roty, new_rotz)) which performs a rotation (see documentation at http://www.blender.org/documentation/249PythonDoc/Object.Object-class.html). I am currently looking at how the rotations are applied; more shortly.
Edit2: it looks like the angles are specified as Euler angles (http://en.wikipedia.org/wiki/Euler_angles); they give some conversion matrices. It also looks like your accelerometer data is underconstrained (you need one more constraint, specifying rotation about the 'down' direction - maybe some sort of inertial 'least distance from previous position' calculation?)
Edit3: there is a sample script which may be helpful; on my machine it is at C:\Users\Me\AppData\Roaming\Blender Foundation\Blender.blender\scripts\object_random_loc_sz_rot.py It shows how to get the currently selected object and tweak its rotation. Hope that helps!
Edit4: for sake of discussion, here is some sample code; it may be a bit redundant (I haven't worked in Blender before) and it doesn't solve the problem, but it will at least give us a common basis for further discussion ;-)
#!BPY
"""
Name: 'Set rotation by accelerometer'
Blender: 249
Group: 'Object'
Tooltip: 'Set the selected objects rotation by accelerometer'
"""
__bpydoc__=\
'''
This script sets the selected objects rotation by accelerometer.
'''
from Blender import Draw, Scene
import math
def reorient(alpha, beta, gamma):
a = math.cos(alpha)
b = math.sin(alpha)
c = math.cos(beta)
d = math.sin(beta)
e = math.cos(gamma)
f = math.sin(gamma)
ad = a*d
bd = b*d
return = [
[c*e, -a*f+b*d*e, b*f+a*d*e],
[c*f, a*e+b*d*f, -b*e+a*d*f],
[-d, b*c, a*c ]
]
def getAccel():
# test stub -
# need to get actual values from accelerometer here
dx = -700
dy = 100
dz = 250
return (dx,dy,dz)
def normalize(vec):
"Return scaled unit vector"
x,y,z = vec
mag = (x*x + y*y + z*z)**0.5
return (x/mag, y/mag, z/mag)
def main():
scn = Scene.GetCurrent()
try:
obj = scn.objects.context
euler = (obj.RotX, obj.RotY, obj.RotZ)
except AttributeError:
return
down = normalize(getAccel())
matrix = None
# do something here to find new rotation-matrix
# based on euler and down
# then
if matrix:
obj.setOrientation(matrix)
else:
# test value:
# if reorient() is working properly, the
# object's rotation should not change!
obj.setOrientation(reorient(*euler))
if __name__=="__main__":
main()

Let's assume that you can use the accelerometers to correctly determine which way is 'up'. We can call that vector N. It seems to me that you want the 'up' direction of your cube to align with N. As already mentioned, that leaves the cube free to spin, but you can still find a rotation matrix that might accomplish what you're going for, but you need to account for two separate cases or else you'll have a singularity in the solution. I'll assume that 'Z' is 'up' on the cube.
If you treat the three accelerometer values as a vector and normalize it (to get N), you've got the new 'Z' axis portion of your rotation matrix so that a vector pointed in the z direction will now align with the 'up' vector.
| a d N.x | |0| |N.x|
| b e N.y | * |0| = |N.y|
| c f N.z | |1| |N.z|
So we need to decide what to do with a-f. One common thing to do is this: if N is pointing mostly along the original 'Z' axis, then make the new 'Y' axis portion of the matrix be M = N cross X:
d = 0
e = N.z
f = -N.y
Normalize M and then find the 'X' axis portion of the matrix: L = M cross N. Normalize L.
If N is not pointing mostly along the 'Z' axis (N.z < .707), then you find the new 'Y' axis portion as M = N cross Z. Normalize M and find L = M cross N and, finally, normalize L.
Edit:
So we have our three accelerometer values: A.x, A.y, A.z. First step is to normalize them:
a = sqrt(A.x*A.x + A.y*Ay + A.z*A.z); and then
N.x = A.x/a; N.y = A.y/a; N.z = A.z/a;
We assume that if N == [0, 0, 1] then the correct rotation matrix is the identity matrix. If N doesn't point directly along the z-axis, then we want to form a matrix that will rotate the z-axis of the cube so that it lines up with N.