Develop Reference

How to split a 3D array of positions into subvolumes

Not sure if this question has been asked before–I looked through similar examples and they weren't exactly what I need to do.
I have an array of positions (shape = (8855470, 3)) in a cube with physical coordinates in between 0 and 787.5. These positions represent point masses in some space. Here's a look at the first three entries of this array:
array([[224.90635586, 720.494766 , 19.40263367],
[491.25279546, 41.26026654, 7.35436416],
[407.70436788, 340.32618713, 328.88192913]])
I want to split this giant cube into a number of smaller cubes. For example, if I wanted to split it on each side into 10 cubes, making 1,000 subcubes total, then each subcube would contain only the points that have positions within that subcube. I have been experimenting with np.meshgrid to create the 3D grid necessary to conditionally apportion the appropriate entries of the positions array to subcubes:
split = np.arange(0.,(787.5+787.5/10.),step=787.5/10.)
xg,yg,zg = np.meshgrid(split,split,split,indexing='ij')
But I'm not sure if this is the way to go about this.
Let me know if this question is too vague or if you need any additional information.

For sake of problem I will work with toy data. I think you're near with the meshgrid. Here's a propossal
Create grid but with points until 757.5 not included, with values as you did in arange.
Reshape then to have a 1d_array. for in arrays zip to get masks with the cube shape.
create a list to save all subcube points.
import numpy as np
data = np.random.randint(0,787,( 10000,3))
start = 0
end = 787.5
step = (end-start)/10
split = np.arange(start,end,step)
xg,yg,zg = np.meshgrid(split,split,split,indexing='ij')
xg = xg.reshape(-1)
yg = yg.reshape(-1)
zg = zg.reshape(-1)
subcube_data = []
for x,y,z in zip(xg,yg,zg):
mask_x = (x<= data[:,0] ) * ( data[:,0] < x+step) #data_x between start and end for this subcube
mask_y = (y<= data[:,1] ) * ( data[:,1] < y+step) #data_y between start and end for this subcube
mask_z = (z<= data[:,2] ) * ( data[:,2] < z+step) #data_z between start and end for this subcube
mask = mask_x * mask_y * mask_z
subcube_data.append(data[mask])
Now you will have a list with 1000 elements where each element is a sub_cube containing an Nx3 point list. If you want to recover the 3d index corresponding to every sub_cube[i] you just could do [xg[i],yg[i],zg[i]].
Last you can plot to see some of the sub_cubes and the rest of data
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
#plot data as 3d scatter border black
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
#plot subcubes 0 1 2 3 4 in colors
for i in range(5):
ax.scatter(subcube_data[i][:,0],
subcube_data[i][:,1],
subcube_data[i][:,2], marker='o', s=2)
for i in range(5,len(subcube_data)):
ax.scatter(subcube_data[i][:,0],
subcube_data[i][:,1],
subcube_data[i][:,2],marker='o', s=1, color='black')

wrong number of arguments when using x, y arrays as pixel positions and z as colour value in matplotlib.pcolor()

I have been trying to make a coloured image with a colourbar scale. Each pixel should correspond to colour bar values from the Z array, and x,y should be pixel position arguments (in mm) similar to this:
https://ars.els-cdn.com/content/image/1-s2.0-S0010218014001989-gr6_lrg.jpg
Data is imported across many text files. However, a small example section of this input data can be recreated from the following code file to pandas dataframe code, (where the x axis range is -30 - 30mm (increment by 1) and y axis range is 6 - 15 mm (increment by 3):
import numpy as np
import pandas as pd
x = np.linspace(-30,30,61)
y = np.linspace(6, 15, 4)
z = 1.35 * np.random.rand(61*4, 1)
for i in range(0,3,1):
x = np.append(x,x[:61])
y = np.repeat(y,61)
df = pd.DataFrame()
df['X [mm]'] = x
df ['Y [mm]'] = y
df['LDA1-Mean [m/s]'] = z
print(df)
Now running the following code:
import matplotlib.pyplot as plt
Z = df['LDA1-Mean [m/s]'].to_list()
positions = np.array(list(zip(df['X [mm]'], df['Y [mm]'])))
plt.pcolor(positions, Z)
plt.show()
plt.savefig('solutions/graphs/test.png', dpi=300, bbox_inches="tight")
Produces the following error:
TypeError: pcolor() takes 1 or 3 positional arguments but 2 were given
Is there a better way to do this with imshow() or contourf(), I'm open to suggestions.
As the order of X and Y data cannot as consistent as, I'd prefer to take always use the X and Y position data rather than reordering Z data (and using only 1 argument).
Thank you in advance for the help - I am still new to programming. Please feel free to ask questions if there is something I have not explained.

Python: Get values of array which correspond to contour lines

Is there a way to extract the data from an array, which corresponds to a line of a contourplot in python? I.e. I have the following code:
n = 100
x, y = np.mgrid[0:1:n*1j, 0:1:n*1j]
plt.contour(x,y,values)
where values is a 2d array with data (I stored the data in a file but it seems not to be possible to upload it here). The picture below shows the corresponding contourplot. My question is, if it is possible to get exactly the data from values, which corresponds e.g. to the left contourline in the plot?

Worth noting here, since this post was the top hit when I had the same question, that this can be done with scikit-image much more simply than with matplotlib. I'd encourage you to check out skimage.measure.find_contours. A snippet of their example:
from skimage import measure
x, y = np.ogrid[-np.pi:np.pi:100j, -np.pi:np.pi:100j]
r = np.sin(np.exp((np.sin(x)**3 + np.cos(y)**2)))
contours = measure.find_contours(r, 0.8)
which can then be plotted/manipulated as you need. I like this more because you don't have to get into the deep weeds of matplotlib.

plt.contour returns a QuadContourSet. From that, we can access the individual lines using:
cs.collections[0].get_paths()
This returns all the individual paths. To access the actual x, y locations, we need to look at the vertices attribute of each path. The first contour drawn should be accessible using:
X, Y = cs.collections[0].get_paths()[0].vertices.T
See the example below to see how to access any of the given lines. In the example I only access the first one:
import matplotlib.pyplot as plt
import numpy as np
n = 100
x, y = np.mgrid[0:1:n*1j, 0:1:n*1j]
values = x**0.5 * y**0.5
fig1, ax1 = plt.subplots(1)
cs = plt.contour(x, y, values)
lines = []
for line in cs.collections[0].get_paths():
lines.append(line.vertices)
fig1.savefig('contours1.png')
fig2, ax2 = plt.subplots(1)
ax2.plot(lines[0][:, 0], lines[0][:, 1])
fig2.savefig('contours2.png')
contours1.png:
contours2.png:

plt.contour returns a QuadContourSet which holds the data you're after.
See Get coordinates from the contour in matplotlib? (which this question is probably a duplicate of...)

ZeroDivisionError: float division by zero in a code for Surface plot

I have got this code to generate a surface plot. But it gives a zero division error. I am not able to figure out what is wrong. Thank you.
import pylab, csv
import numpy
from mayavi.mlab import *
def getData(fileName):
try:
data = csv.reader(open(fileName,'rb'))
except:
print 'File not found'
else:
data = [[float(row[0]), float(row[1]),float(row[2])] for row in data]
x = [row[0] for row in data]
y = [row[1] for row in data]
z = [row[2] for row in data]
return (x, y, z)
def plotData(fileName):
xVals, yVals, zVals = getData(fileName)
xVals = pylab.array(xVals)
yVals = pylab.array(yVals)
zVals = (pylab.array(zVals)*10**3)
x, y = numpy.mgrid[-0.5:0.5:0.001, -0.5:0.5:0.001]
s = surf(x, y, zVals)
return s
plotData('data')

If I have understood the code correctly, there is a problem with zVals in mayavi.mlab.surf.
According to the documentation of the function, s is the elevation matrix, a 2D array, where indices along the first array axis represent x locations, and indices along the second array axis represent y locations. Your file reader seems to return a 1D vector instead of an array.
However, this may not be the most difficult problem. Your file seems to contain triplets of x, y, and z coordinates. You can use mayavi.mlab.surf only if your x and y coordinates in the file form a regular square grid. If this is the case, then you just have to recover that grid and form nice 2D arrays of all three parts. If the points are in the file in a known order, it is easy, otherwise it is rather tricky.
Maybe you would want to start with mayavi.mlab.points3d(xVals, yVals, zVals). That will give you an overall impression of your data. (Or if already know more about your data, you might give us a hint by editing your question and adding more information!)
Just to give you an idea of probably slightly pythonic style of writing this, your code is rewritten (and surf replaced) in the following:
import mayavi.mlab as ml
import numpy
def plot_data(filename):
data = numpy.loadtxt(filename)
xvals = data[:,0]
yvals = data[:,1]
zvals = data[:,2] * 1000.
return ml.points3d(x, y, z)
plot_data('data')
(Essential changes: the use of numpy.loadtxt, get rid of pylab namespace here, no import *, no CamelCase variable or function names. For more information, see PEP 8.)
If you only need to see the shape of the surface, and the data in the file is ordered row-by-row and with the same number of data points in each row (i.e. fixed number of columns), then you may use:
import mayavi.mlab as ml
import numpy
importt matplotlib.pyplot as plt
# whatever you have as the number of points per row
columns = 13
data = numpy.loadtxt(filename)
# draw the data points into a XY plane to check that they really for a rectangular grid:
plt.plot(data[:,0], data[:,1])
# draw the surface
zvals = data[:,2].reshape(-1,columns)
ml.surf(zvals, warp_scale='auto')
As you can see, this code allows you to check that your values really are in the right kind of grid. It does not check that they are in the correct order, but at least you can see they form a nice grid. Also, you have to input the number of columns manually. The keyword warp_scale takes care of the surface scaling so that it should look reasonable.

turn scatter data into binned data with errors bars equal to standard deviation

I have a bunch of data scattered x, y. If I want to bin these according to x and put error bars equal to the standard deviation on them, how would I go about doing that?
The only I know of in python is to loop over the data in x and group them according to bins (max(X)-min(X)/nbins) then loop over those blocks to find the std. I'm sure there are faster ways of doing this with numpy.
I want it to look similar to "vert symmetric" in: http://matplotlib.org/examples/pylab_examples/errorbar_demo.html

You can bin your data with np.histogram. I'm reusing code from this other answer to calculate the mean and standard deviation of the binned y:
import numpy as np
import matplotlib.pyplot as plt
x = np.random.rand(100)
y = np.sin(2*np.pi*x) + 2 * x * (np.random.rand(100)-0.5)
nbins = 10
n, _ = np.histogram(x, bins=nbins)
sy, _ = np.histogram(x, bins=nbins, weights=y)
sy2, _ = np.histogram(x, bins=nbins, weights=y*y)
mean = sy / n
std = np.sqrt(sy2/n - mean*mean)
plt.plot(x, y, 'bo')
plt.errorbar((_[1:] + _[:-1])/2, mean, yerr=std, fmt='r-')
plt.show()

No loop ! Python allows you to avoid looping as much as possible.
I am not sure to get everything, you have the same x vector for all data and many y vectors corresponding to different measurement no ? And you want to plot your data as the "vert symmetric" with the mean value of y for each x and a standard deviation for each x as an errorbar ?
Then it is easy. I assume you have a M-long x vector and a N*M array of your N sets of y data already loaded in variable names x and y.
import numpy as np
import pyplot as pl
error = np.std(y,axis=1)
ymean = np.mean(y,axis=1)
pl.errorbar(x,ymean,error)
pl.show()
I hope it helps. Let me know if you have any question or if it is not clear.