I have a 16x16x4 array in Numpy.
Dimension 1: Horizontal position [0,15]
Dimension 2: Vertical position [0,15]
Dimension 3: An RGB value 0-255 [0,3]
Replace 16x16 with 2048x1285 and:
for x in range(0,15):
for y in range(0,15):
Doesn't cut it (upwards of 7 minutes to do this and a flood fill at each interesting point). Iterating over a PIL image is plenty fast, but a numpy array drags (i.e. 7+ minutes).
numpy.where(bitmap == [red, green, blue, alpha])
doesn't seem like it's what I'm looking for. What's a reasonably fast way to go about this?
Edit:
bitmap == [red, green, blue, alpha]
is actually almost useful. How do I go from a 16x16x4 array to a 16x16x1 array where array[x,y] is 1 if z = [True,True,True,True] and 0 otherwise?
I can't reproduce your speeds -- even a brute-force iteration on my now-ancient notebook is about 14 times faster -- and I'm not sure where works the way you think it does, so I suspect that most of your time is spent elsewhere (say in your fill). Anyway:
How do I go from a 16x16x4 array to a 16x16x1 array where array[x,y]
is 1 if z = [True,True,True,True] and 0 otherwise?
I would:
In [169]: m = numpy.random.randint(0, 16, size=(2048, 1285, 4))
In [170]: b = [4,5,6,7]
In [171]: matches = (m == b).all(axis=2)*1
In [172]: matches.shape
Out[172]: (2048, 1285)
and it's pretty fast:
In [173]: timeit matches = (m == b).all(axis=2)*1
10 loops, best of 3: 137 ms per loop
It turns out that what I described is achieved by
zip(*numpy.where(numpy.logical_and.reduce(image == [255,255,255])))
Which was not clear according to any documentation, but there you have it. (Edit: lack of alpha channel is not significant.)
The test I'm interested in isn't actually equality to a point, but Euclidian distance to that point being within a threshold.
numpy.apply_along_axis(distance_from_white ...
where distance_from_white is a function returning Euclidian distance from white, works at 16x16 but takes minutes at 2048x1245. scipy.spatial.distance.pdist (or cdist?) might be the answer there, but I can't figure out how to make it compute distance against a single point instead of distance between all points in 2 arrays (this works at 16x16, but it's so much wasted computation I'm hesitant to even try it at actual size).
Iterating with for loops on a ndarray isn't very efficient, as you have noticed. If you want to find the indices of the entries satisfying your condition, you should indeed use
indices = np.where(bitmap == [R, G, B, A])
This will return a 3-element tuple giving the indices of the solution along the 1st, 2nd and third axis. As you're only interested in the first two dimensions, you can drop the third item. And if you want a series of indices like (x,y), you just have to use something like
zip(*indices[:2])
A second possibility is to view your (N,M,4) standard integer ndarray into a (N,M) structured array with dtype=[[('',int)]*4] (don't bother for the field names, they'll be automatically expended to 'f0', 'f1', ... :
alt_bitmap = bitmap.view([('',int)'*4).squeeze()
(The squeeze is introduced to collapse the (N,M,1) array into a (N,M) array)
You can then use the np.where function, but the values you're testing must be a np.array too:
indices = np.where(bitmap==np.array((R, G, B, A), dtype=bitmap.dtype))
This time, indices will only be a 2-tuple, and you can get the (x,y) couples with the zip(*indices) presented earlier.
Related
I am handling a set of data recorded by a 2D detector. Therefore, the data are represented by three arrays: x and y labelling the coordinate of a pixel and intensity storing the measured signal.
For example, a 6x6 grid will give a set of data:
xraw = np.array([0,1,2,3,4,5,0,1,2,3,4,5,...])
yraw = np.array([0,0,0,0,0,0,1,1,1,1,1,1,...])
intensity = np.array([i_00,i_01,i_02,i_03,i_04,i_05,i_10,i_11,...])
Due to various reasons, such as pixel defects, some of the data points are discarded in the raw data. Therefore, xraw, yraw, intensity have a size smaller than 36 (if that's a 6x6 grid), with, say, the point at (2,3) missing.
The intensity data needs further treatment by an element-wise multiplication with another array. This treatment array is from theoretical calculation and so it has a size of nxn (6x6 in this case). However, as some of the points in the true data are missing, the two arrays have different sizes.
I can use a loop to check for the missing points and eliminate the corresponding element in the treatment array. I wonder if there are some methods in numpy that take care of such operations. Thanks
First, construct the indices of available and all possible pixel positions by
avail_ind = yraw * h + xraw
all_ind = np.arange(0, h * w)
where h and w is the image's height and width in pixels.
Then, find the indices of the missing pixels by
missing_ind = all_ind[~np.in1d(all_ind, avail_ind)]
Once having the missing indices, use np.delete to construct a copy of the treatment_array with elements at the indices removed, then simply multiply that with your intensity array.
result = intensity * np.delete(treatment_array, missing_ind)
I am learning numpy , have a question in my mind not able to clearly visualise from where this 1 as come in shape
import numpy as np
a = np.array([ [[1],[56]] , [[8],[98]] ,[[89],[62]] ])
np.shape(a)
The output is printed as : (3 ,2 , 1)
Will be appreciated if you could represent in diagrammatic / image format
What actually the 1 means in output
Basically, that last 1 is because every number in a has brackets around it.
Formally, it's the length of your "last" or "innermost" dimension. You can take your first two dimensions and arrange a as you would a normal matrix, but note that each element itself has brackets around it - each element is itself an array:
[[ [1] [56]]
[ [8] [98]]
[[89] [62]]]
If you add an element to each innermost-array, making that third shape number get larger, it's like stacking more arrays behind this top one in 3d, where now the corresponding elements in the "behind" array are in the same innermost array as the "front" array.
Equivalently, instead of considering the first two indices to denote the regular flat matrices, you can think of the back two making the flat matrices. This is how numpy does it: try printing out an array like this: x = np.random.randint(10, size = (3,3,3)). Along the first dimension, x[0], x[1], and x[2] are printed after each other, and each one individually is formatted like a 3x3 matrix. Then the second index corresponds to the rows of each individual matrix, and the third index corresponds to the columns. Note that when you print a, there's only one column displayed - its third dimension has size 1. You can play with the definition of x to see more what's going on (change the numbers in the size argument).
An alright example of visualizing a 3d array this way is this image, found on the Wikipedia page for the Levi-Civita symbol:
Don't worry too much about what the Levi-Civita symbol actually is - just note that here, if it were a numpy array it would have shape (3,3,3) (like the x I defined above). You use three indices to specify each element, i, j, and k. i tells you the depth (blue, red, or green), j tells you the row, and k tells you the column. When numpy prints, it just lists out blue, red, then green in order.
I am looking for the fastest way that I can read an image in, specify a pixel, and find the nearest pixel to that pixel given my conditionals.
I originally had a nested loop that would go through each pixel in my 2D image array, check for conditional, than add it to a new array. Than go through the new array and do the distance calculations on each member.
That was terribly slow, and has a time complexity of n^2 I believe.
I am now doing the distance calculations, and sorting the array by that, which is quite fast when using numpy. But then I still have to go through that 2D sorted array with a nested loop to check for conditionals, which again is a time complexity of n^2. It does save some time though because I can normally find the pixel I am looking for sooner in the array.
img2=cv2.imread(filename)
distances = numpy.sqrt((img2[:,:] - currR) ** 2 + (img2[:,:] - currC) ** 2)
nearest = numpy.sort(distances)
for row in range(nearest.shape[0]):
for col in range(nearest.shape[1]):
if pixelInLine[row*imgCol + col] == 0 and colorCheck(row,col) and numpy.any(img2[row, col] != 0):
#do my calculations on the specified pixel. and break the loop
I am unsure how I can optimize this further, and potentially lower the time complexity from n^2 to something more reasonable.
Go spiraling away from the target pixel and stop as soon as you meet the "conditionals". For convenience, you can use a square spiral (in fact nested squares).
If the distance is Euclidean, the cost of the search will be between 2d² and 4d² where d is the distance to the hit.
I have calculated the boundaries in which I want to sample points.
For example one dataset looks like:
Now I want to find point in the red area, which I do in the following way:
The plot consist of 10 lines, so I reshape to get the region limits per value of x.
limits = data.reshape(data.shape + (5, 2))
Now for a particular value of x data looks like:
limits[20] = array([[ 5.65624197, 6.70331962],
[ 13.68248989, 14.77227669],
[ 15.50973796, 16.61491606],
[ 24.03948128, 25.14907398],
[ 26.41541777, 27.53475798]])
I thought to make a mesh and mask the area as following
X, Y = np.meshgrid(xs, ys)
bool_array = np.zeros(Y.shape)
for j, y in enumerate(limits):
for min_y, max_y in y:
inds = np.where(np.logical_and(ys >= min_y, ys <= max_y))[0]
bool_array[inds, j] = True
plt.imshow(bool_array[::-1])
(don't know why the graph need to be plotted inverted)
results in
which is indeed the data I'm looking for , now I could use the True values to take points with a different function.
The problem is that this code is very slow, and my datasets will get much bigger.
I would like to find a more efficient way of finding this "mask".
I tried several things and ended up with the following result which worked for my simple cases
low_bound = limits[:,:,0]
upp_bound = limits[:,:,1]
mask = np.any((low_bound[:,None,:] <= Y.T[:,:,None]) & ( Y.T[:,:,None] <= upp_bound[:,None,:]),axis=-1).T
I know it looks ugly. What I do is introducing an additional dimension in which I subsequently check conditions whether it lies between two ending points. At the end I collapse the additional dimension by using np.any.
I don't know how much faster it is compared to your code. However, given that I don't use a single for loop there should be a performance boost.
Check the code with your data and tell me if something goes wrong.
Edit:
plt.imshow plots (0,0) in the lower left edge when you use
plt.imshow(mask,origin='lower')
So this is a little follow up question to my earlier question: Generate coordinates inside Polygon and my answer https://stackoverflow.com/a/15243767/1740928
In fact, I want to bin polygon data to a regular grid. Therefore, I calculate a couple of coordinates within the polygon and translate their lat/lon combination to their respective column/row combo of the grid.
Currently, the row/column information is stored in a numpy array with its number of rows corresponding to the number of data polygons and its number of columns corresponding to the coordinates in the polygon.
The whole code takes less then a second, but this code is the bottleneck at the moment (with ~7sec):
for ii in np.arange(len(data)):
for cc in np.arange(data_lats.shape[1]):
final_grid[ row[ii,cc], col[ii,cc] ] += data[ii]
final_grid_counts[ row[ii,cc], col[ii,cc] ] += 1
The array "data" simply contains the data values for each polygon (80000,). The arrays "row" and "col" contain the row and column number of a coordinate in the polygon (shape: (80000,16)).
As you can see, I am summing up all data values within each grid cell and count the number of matches. Thus, I know the average for each grid cell in case different polygons intersect it.
Still, how can these two for loops take around 7 seconds? Can you think of a faster way?
I think numpy should add an nd-bincount function, I had one lying around from a project I was working on some time ago.
import numpy as np
def two_d_bincount(row, col, weights=None, shape=None):
if shape is None:
shape = (row.max() + 1, col.max() + 1)
row = np.asarray(row, 'int')
col = np.asarray(col, 'int')
x = np.ravel_multi_index([row, col], shape)
out = np.bincount(x, weights, minlength=np.prod(shape))
return out.reshape(shape)
weights = np.column_stack([data] * row.shape[1])
final_grid = two_d_bincount(row.ravel(), col.ravel(), weights.ravel())
final_grid_counts = two_d_bincount(row.ravel(), col.ravel())
I hope this helps.
I might not fully understand the shapes of your different grids, but you can maybe eliminate the cc loop using something like this:
final_grid = np.empty((nrows,ncols))
for ii in xrange(len(data)):
final_grid[row[ii,:],col[ii,:]] = data[ii]
This of course assumes that final_grid is starting with no other info (that the count you're incrementing starts at zero). And I'm not sure how to test if it works not understanding how your row and col arrays work.