Develop Reference

Splitting data into subsamples

I have a huge dataset which contains coordinates of particles. In order to split the data into test and training set I want to divide the space into many subspaces; I did this with a for-loop in every direction (x,y,z) but when running the code it takes very long and is not efficient enough especially for large datasets:
particle_boxes = []
init = 0
final = 50
number_box = 5
for i in range(number_box):
for j in range(number_box):
for k in range(number_box):
index_particle = df_particles['X'].between(init+i*final, final+final*i)&df_particles['Y'].between(init+j*final, final+final*j)&df_particles['Z'].between(init+k*final, final+final*k)
particle_boxes.append(df_particles[index_particle])
where init and final define the box size, df_particles contains every particle coordinate (x,y,z).
After running this particle_boxes contains 125 (number_box^3) equal spaced subboxes.
Is there any way to write this code more efficiently?

Note on efficiency
I conducted a number of tests using other tricks and nothing changed substantially. This is roughly as good as any other technique I used.
I'm curious to see if anyone else comes up with something order of magnitude faster.
Sample data
np.random.seed([3, 1415])
df_particles = pd.DataFrame(
np.random.randint(250, size=(1000, 3)),
columns=['X', 'Y', 'Z']
)
Solution
Construct an array a that represents your boundaries
a = np.array([50, 100, 150, 200, 250])
Then use searchsorted to create the individual dimensional bins
x_bin = a.searchsorted(df_particles['X'].to_numpy())
y_bin = a.searchsorted(df_particles['Y'].to_numpy())
z_bin = a.searchsorted(df_particles['Z'].to_numpy())
Use groupby on the three bins. I used trickery to get that into a dict
g = dict((*df_particles.groupby([x_bin, y_bin, z_bin]),))
We can see the first zone
g[(0, 0, 0)]
X Y Z
30 2 36 47
194 0 34 45
276 46 37 34
364 10 16 21
378 4 15 4
429 12 34 13
645 36 17 5
743 18 36 13
876 46 11 34
and the last
g[(4, 4, 4)]
X Y Z
87 223 236 213
125 206 241 249
174 218 247 221
234 222 204 237
298 208 211 225
461 234 204 238
596 209 229 241
731 210 220 242
761 225 215 231
762 206 241 240
840 211 241 238
846 212 242 241
899 249 203 228
970 214 217 232
981 236 216 248

Instead of multiple nested for loops, consider one loop using itertools.product. But of course avoid any loops if possible as #piRSquared shows:
from itertools import product
particle_boxes = []
for i, j, k in product(range(number_box), range(number_box), range(number_box)):
index_particle = (df_particles['X'].between(init+i*final, final+final*i) &
df_particles['Y'].between(init+j*final, final+final*j) &
df_particles['Z'].between(init+k*final, final+final*k))
particle_boxes.append(df_particles[index_particle])
Alternatively, with list comprehension:
def sub_df(i, j, k)
index_particle = (df_particles['X'].between(init+i*final, final+final*i) &
df_particles['Y'].between(init+j*final, final+final*j) &
df_particles['Z'].between(init+k*final, final+final*k))
return df_particles[index_particle]
particle_boxes = [sub_df(i, j, k) for product(range(number_box), range(number_box), range(number_box))]

Have a look at train_test_split function available in the scikit-learn lib.
I think it is almost the kind of functionality that you need.
The code is consultable on Github.

Why is Python automatically removing negative values from image array?

I am trying to apply the below convolve method below on the cameraman image. The kernel applied to the image is a 3x3 filter populated with -1/9. I print the values of the cameraman image before applying the convolve method, and all I get are positive values. Next, when I apply the 3x3 negative kernel on the Image, I still get positive values when I print the values of the cameraman image after convolution.
The convolving function:
def convolve2d(image, kernel):
# This function which takes an image and a kernel
# and returns the convolution of them
# Args:
# image: a numpy array of size [image_height, image_width].
# kernel: a numpy array of size [kernel_height, kernel_width].
# Returns:
# a numpy array of size [image_height, image_width] (convolution output).
output = np.zeros_like(image) # convolution output
# Add zero padding to the input image
padding = int(len(kernel)/2)
image_padded=np.pad(image,((padding,padding),(padding,padding)),'constant')
for x in range(image.shape[1]): # Loop over every pixel of the image
for y in range(image.shape[0]):
# element-wise multiplication of the kernel and the image
output[y,x]=(kernel*image_padded[y:y+3,x:x+3]).sum()
return output
And here is the filter I am applying to the image:
filter2= [[-1/9,-1/9,-1/9],[-1/9,-1/9,-1/9],[-1/9,-1/9,-1/9]]
Finally, these are the intial values of the images, and the values after convolution respectively:
[[156 159 158 ... 151 152 152]
[160 154 157 ... 154 155 153]
[156 159 158 ... 151 152 152]
...
[114 132 123 ... 135 137 114]
[121 126 130 ... 133 130 113]
[121 126 130 ... 133 130 113]]
After convolution:
[[187 152 152 ... 154 155 188]
[152 99 99 ... 104 104 155]
[152 99 100 ... 103 103 154]
...
[175 133 131 ... 127 130 174]
[174 132 124 ... 125 130 175]
[202 173 164 ... 172 173 202]]
This is how I call the convolve2d method:
convolved_camManImage= convolve2d(camManImage,filter2)

This might be caused by how numpy dtypes work. As numpy.zeros_like's help says:
Return an array of zeros with the same shape and type as a given
array.
Thus your output might be dtype uint8, which use modulo arithmetics. To check if this is case add print(output.dtype) immediately after output = np.zeros_like(image) line

Bad display with pyplot, image too dark

I am working on an image processing problem.
I create a function that applies a salt and pepper noise to an image.
Here is the function:
def sp_noise(image,prob):
res = np.zeros(image.shape,np.uint8)
for i in range(image.shape[0]):
for j in range(image.shape[1]):
rdn = random.random()
if rdn < prob:
rdn2 = random.random()
if rdn2 < 0.5:
res[i][j] = 0
else:
res[i][j] = 255
else:
res[i][j] = image[i][j]
return res
Problems happen when I want to display the result.
wood = loadPNGFile('wood.jpeg',rgb=False)
woodSP = sp_noise(bois,0.01)
plt.subplot(1,2,1)
plt.imshow(bois,'gray')
plt.title("Wood")
plt.subplot(1,2,2)
plt.imshow(woodSP,'gray')
plt.title("Wood SP")
I can not post the image directly but here is the link:
The picture is darker. But when I display the value of the pixels
But when I display the value of the pixels between the 2 images the values ​​are the same:
[[ 99 97 96 ... 118 90 70]
[110 110 103 ... 116 115 101]
[ 79 73 65 ... 96 121 121]
...
[ 79 62 46 ... 105 124 113]
[ 86 98 100 ... 114 119 99]
[ 96 95 95 ... 116 111 90]]
[[255 97 96 ... 118 90 70]
[110 110 103 ... 116 115 101]
[ 79 73 65 ... 96 121 121]
...
[ 79 62 46 ... 105 124 113]
[ 86 98 100 ... 114 119 99]
[ 96 95 95 ... 116 111 90]]
I also check the mean value:
117.79877369007804
117.81332616658703
Apparently the problem comes from the display plt.imshow, but I can not find a solution

Looking at the documentation of imshow, there are 2 optional parameters, vmin, vmax which:
When using scalar data and no explicit norm, vmin and vmax define the
data range that the colormap covers. By default, the colormap covers
the complete value range of the supplied data. vmin, vmax are ignored
if the norm parameter is used.
Therefore, if no values are specified for these parameters, the range of luminosity is based on the actual data values, with the minimum value being set to black and the maximum value being set to white. This is useful in visualization, but not in comparisons, as you found out. Therefore, just set vmin and vmax to appropriate values (probably 0 and 255).

Speeding up array iteration time in python

Currently I am iterating over one array and for each value in this array I am looking for the closest value at the corresponding point in another array that is within a region surrounding the corresponding point.
In summary: For any point in an array, how far away from a corresponding point in another array do you need to go to get the same value.
The code seems to work well for small arrays, however I am working now with 1024x768 arrays, leading me to wait a long time for each run....
Any help or advice would be greatly appreciated as I have been on this for a while!!
Example matrix in format Im using: np.array[[1,2],[3,4]]
#Distance to agreement
#Used later to define a region of pixels around a corresponding point
#to iterate over:
DTA = 26
#To account for noise in pixels - doesnt have to find the exact value,
#just one within +/-130 of it.
limit = 130
#Containers for all pixel value matches and also the smallest distance
#to pixel match
Dist = []
Dist_min = []
#Continer matrix for gamma pass/fail values
Dist_to_agree = np.zeros((i_size,j_size))
#i,j indexes the reference matrix (x), ii,jj indexes the measured
#matrix(y). Finds a match within the limits,
#appends the distance to the match into Dist.
#Then find the minimum distance to a match for that pixel and append it
#to dist_min
for i, k in enumerate(x):
for j, l in enumerate(k):
#added 10 packing to y matrix, so need to shift it by 10 in i&j
for ii in range((i+10)-DTA,(i+10)+DTA):
for jj in range((j+10)-DTA,(j+10)+DTA):
#If the pixel value is within a range to account for noise,
#let it be "found"
if (y[ii,jj]-limit) <= x[i,j] <= (y[ii,jj]+limit):
#Calculating distance
dist_eu = sqrt(((i)-(ii))**2 + ((j) - (jj))**2)
Dist.append(dist_eu)
#If a value cannot be found within the noise range,
#append 10 = instant fail.
else:
Dist.append(10)
try:
Dist_min.append(min(Dist))
Dist_to_agree[i,j] = min(Dist)
except ValueError:
pass
#Need to reset container or previous values will also be
#accounted for when finding minimum
Dist = []
print Dist_to_agree

First, you are getting the elements of x in k and l, but then throwing that away and indexing x again. So in place of x[i,j], you could just use l, which would be much faster (although l isn't a very meaningful name, something like xi and xij might be better).
Second, you are recomputing y[ii,jj]-limit and y[ii,jj]+limitevery time. If you have enough memory, you can-precomputer these:ym = y-limitandyp = y+limit`.
Third, appending to a list is slower than creating an array and setting the values for long lists vs. long arrays. You can also skip the entire else clause by pre-setting the default value.
Fourth, you are computing min(dist) twice, and further may be using the python version rather than the numpy version, the latter being faster for arrays (which is another reason to make dist and array).
However, the biggest speedup would be to vectorize the inner two loops. Here is my tests, with x=np.random.random((10,10)) and y=np.random.random((100,100)):
Your version takes 623 ms.
Here is my version, which takes 7.6 ms:
dta = 26
limit = 130
dist_to_agree = np.zeros_like(x)
dist_min = []
ym = y-limit
yp = y+limit
for i, xi in enumerate(x):
irange = (i-np.arange(i+10-dta, i+10+dta))**2
if not irange.size:
continue
ymi = ym[i+10-dta:i+10+dta, :]
ypi = yp[i+10-dta:i+10+dta, :]
for j, xij in enumerate(xi):
jrange = (j-np.arange(j+10-dta, j+10+dta))**2
if not jrange.size:
continue
ymij = ymi[:, j+10-dta:j+10+dta]
ypij = ypi[:, j+10-dta:j+10+dta]
imesh, jmesh = np.meshgrid(irange, jrange, indexing='ij')
dist = np.sqrt(imesh+jmesh)
dist[ymij > xij or xij < ypij] = 10
mindist = dist.min()
dist_min.append(mindist)
dist_to_agree[i,j] = mindist
print(dist_to_agree)

#Ciaran
Meshgrid is kinda a vectorized equivalent of two nested loops. Below are two equivalent ways of calculating the dist. One with loops and one with meshgrid+numpy vector operations. The second one is six times faster.
DTA=5
i=100
j=200
def func1():
dist1=np.zeros((DTA*2,DTA*2))
for ii in range((i+10)-DTA,(i+10)+DTA):
for jj in range((j+10)-DTA,(j+10)+DTA):
dist1[ii-((i+10)-DTA),jj-((j+10)-DTA)] =sqrt(((i)-(ii))**2 + ((j) - (jj))**2)
return dist1
def func2():
dist2=np.zeros((DTA*2,DTA*2))
ii, jj = meshgrid(np.arange((i+10)-DTA,(i+10)+DTA),
np.arange((j+10)-DTA,(j+10)+DTA))
dist2=np.sqrt((i-ii)**2+(j-jj)**2)
return dist2
This is how ii and jj matrices look after meshgrid operation
ii=
[[105 106 107 108 109 110 111 112 113 114]
[105 106 107 108 109 110 111 112 113 114]
[105 106 107 108 109 110 111 112 113 114]
[105 106 107 108 109 110 111 112 113 114]
[105 106 107 108 109 110 111 112 113 114]
[105 106 107 108 109 110 111 112 113 114]
[105 106 107 108 109 110 111 112 113 114]
[105 106 107 108 109 110 111 112 113 114]
[105 106 107 108 109 110 111 112 113 114]
[105 106 107 108 109 110 111 112 113 114]]
jj=
[[205 205 205 205 205 205 205 205 205 205]
[206 206 206 206 206 206 206 206 206 206]
[207 207 207 207 207 207 207 207 207 207]
[208 208 208 208 208 208 208 208 208 208]
[209 209 209 209 209 209 209 209 209 209]
[210 210 210 210 210 210 210 210 210 210]
[211 211 211 211 211 211 211 211 211 211]
[212 212 212 212 212 212 212 212 212 212]
[213 213 213 213 213 213 213 213 213 213]
[214 214 214 214 214 214 214 214 214 214]]

for loops are very slow in pure python and you have four nested loops which will be very slow. Cython does wonders to the for loop speed. You can also try vectorization. While I'm not sure I fully understand what you are trying to do, you may try to vectorize at last some of the operations. Especially the last two loops.
So instead of two ii,jj cycles over
y[ii,jj]-limit) <= x[i,j] <= (y[ii,jj]+limit)
you can do something like
ii, jj = meshgrid(np.arange((i+10)-DTA,(i+10)+DTA), np.arange((j+10)-DTA,(j+10)+DTA))
t=(y[(i+10)-DTA,(i+10)+DTA]-limit>=x[i,j]) & (y[(i+10)-DTA,(i+10)+DTA]+limit<=x[i,j])
Dist=np.sqrt((i-ii)**2)+(j-jj)**2))
np.min(Dist[t]) will have your minimum distance for element i,j

The numbapro compiler offers gpu Acceleration. Unfortunately it isn't free.
http://docs.continuum.io/numbapro/

Speed up while loop matching pattern in array

I have the following data array, with 2 million entries:
[20965 1239 296 231 -1 -1 20976 1239 299 314 147 337
255 348 -1 -1 20978 1239 136 103 241 154 27 293
-1 -1 20984 1239 39 161 180 184 -1 -1 20990 1239
291 31 405 50 569 357 -1 -1 20997 1239 502 25
176 215 360 281 -1 -1 21004 1239 -1 -1 21010 1239
286 104 248 252 -1 -1 21017 1239 162 38 331 240
368 363 321 412 -1 -1 21024 1239 428 323 -1 -1
21030 1239 -1 -1 21037 1239 325 28 353 102 477 189
366 251 143 452 ... ect
This array contains x,y coordinates of photons on a CCD chip, I want to go through the array and add up all these photon events in a matrix with dimensions equal to the CCD chip.
The formatting is as follows: number number x0 y0 x1 y1 -1 -1. The two number entries I don't care too much about, the x0 y0 ect. is what I want to get out. The -1 entries is a delimiter indicating a new frame, after these there is always the 2 'number' entries again.
I have made this code, which does work:
i = 2
pixels = np.int32(data_height)*np.int32(data_width)
data = np.zeros(pixels).reshape(data_height, data_width)
while i < len(rdata):
x = rdata[i]
y = rdata[i+1]
if x != -1 and y != -1:
data[y,x] = data[y,x] + 1
i = i + 2
elif x == -1 and y == -1:
i = i + 4
else:
print "something is wrong"
print i
print x
print y
rdata is my orignal array. data is the resulting matrix which starts out with only zeroes. The while loop starts at the first x coord, at index 2 and then if it finds two consecutive -1 entries it will skip four entries.
The script works fine, but it does take 7 seconds to run. How can I speed up this script? I am a beginner with python, and from the hardest way to learn python I know that while loops should be avoided, but rewriting to a for loop is even slower!
for i in range(2, len(rdata), 2):
x = rdata[i]
y = rdata[i+1]
if x != -1 and y != -1:
px = rdata[i-2]
py = rdata[i-1]
if px != -1 and py != -1:
data[y,x] = data[y,x] + 1
Maybe someone can think of a faster method, something along the lines of np.argwhere(rdata == -1) and use this output to extract the locations of the x and y coordinates?
Update: Thanks for all answers!
I used askewchan's method to conserve frame information, however, as my data file is 300000 frames long I get a memory error when I try to generate a numpy array with dimensions (300000, 640, 480). I could get around this by making a generator object:
def bindata(splits, h, w, data):
f0=0
for i,f in enumerate(splits):
flat_rdata = np.ravel_multi_index(tuple(data[f0:f].T)[::-1], (h, w))
dataslice = np.zeros((w,h), dtype='h')
dataslice = np.bincount(flat_rdata, minlength=pixels).reshape(h, w)
f0 = f
yield dataslice
I then make a tif from the array using a modified version of Gohlke's tifffile.py to generate a tiff file from the data. It works fine, but I need to figure out a way to compress the data as the tiff file is >4gb (at this point the script crashes). I have very sparse arrays, 640*480 all zeros with some dozen ones per frame, the original data file is 4MB so some compression should be possible.

Sounds like all you want is to do some boolean indexing magic to get rid of the invalid frame stuff, and then of course add the pixels up.
rdata = rdata.reshape(-1, 2)
mask = (rdata != -1).all(1)
# remove every x, y pair that is after a pair with a -1.
mask[1:][mask[:-1] == False] = False
# remove first x, y pair
mask[0] = False
rdata = rdata[mask]
# Now need to use bincount, [::-1], since you use data[y,x]:
flat_rdata = np.ravel_multi_index(tuple(rdata.T)[::-1], (data_height, data_width))
res = np.bincount(flat_rdata, minlength=data_height * data_width)
res = res.reshape(data_height, data_width)

Use this to remove the -1s and numbers:
rdata = np.array("20965 1239 296 231 -1 -1 20976 1239 299 314 147 337 255 348 -1 -1 20978 1239 136 103 241 154 27 293 -1 -1 20984 1239 39 161 180 184 -1 -1 20990 1239 291 31 405 50 569 357 -1 -1 20997 1239 502 25 176 215 360 281 -1 -1 21004 1239 -1 -1 21010 1239 286 104 248 252 -1 -1 21017 1239 162 38 331 240 368 363 321 412 -1 -1 21024 1239 428 323 -1 -1 21030 1239 -1 -1 21037 1239 325 28 353 102 477 189 366 251 143 452".split(), dtype=int)
rdata = rdata.reshape(-1,2)
splits = np.where(np.all(rdata==-1, axis=1))[0]
nonxy = np.hstack((splits,splits+1))
data = np.delete(rdata, nonxy, axis=0)[1:]
Now, using part of #seberg's method to convert the x-y lists into arrays, you can make a 3D array where each 'layer' is a frame:
nf = splits.size + 1 # number of frames
splits -= 1 + 2*np.arange(nf-1) # account for missing `-1`s and `number`s
datastack = np.zeros((nf,h,w))
f0 = 0 # f0 = start of the frame
for i,f in enumerate(splits): # f = end of the frame
flat_data = np.ravel_multi_index(tuple(data[f0:f].T)[::-1], (h, w))
datastack[i] = np.bincount(flat_rdata, minlength=h*w).reshape(h, w)
f0 = f
Now, datastack[i] is a 2D array showing the ith frame of your data.

if x0, y0, x1, y1 != -1 could you not do something like filter(lambda a: a != -1, rdata) and then not bother with the ifs? that could speed your code up.