I am learning numpy , have a question in my mind not able to clearly visualise from where this 1 as come in shape
import numpy as np
a = np.array([ [[1],[56]] , [[8],[98]] ,[[89],[62]] ])
np.shape(a)
The output is printed as : (3 ,2 , 1)
Will be appreciated if you could represent in diagrammatic / image format
What actually the 1 means in output
Basically, that last 1 is because every number in a has brackets around it.
Formally, it's the length of your "last" or "innermost" dimension. You can take your first two dimensions and arrange a as you would a normal matrix, but note that each element itself has brackets around it - each element is itself an array:
[[ [1] [56]]
[ [8] [98]]
[[89] [62]]]
If you add an element to each innermost-array, making that third shape number get larger, it's like stacking more arrays behind this top one in 3d, where now the corresponding elements in the "behind" array are in the same innermost array as the "front" array.
Equivalently, instead of considering the first two indices to denote the regular flat matrices, you can think of the back two making the flat matrices. This is how numpy does it: try printing out an array like this: x = np.random.randint(10, size = (3,3,3)). Along the first dimension, x[0], x[1], and x[2] are printed after each other, and each one individually is formatted like a 3x3 matrix. Then the second index corresponds to the rows of each individual matrix, and the third index corresponds to the columns. Note that when you print a, there's only one column displayed - its third dimension has size 1. You can play with the definition of x to see more what's going on (change the numbers in the size argument).
An alright example of visualizing a 3d array this way is this image, found on the Wikipedia page for the Levi-Civita symbol:
Don't worry too much about what the Levi-Civita symbol actually is - just note that here, if it were a numpy array it would have shape (3,3,3) (like the x I defined above). You use three indices to specify each element, i, j, and k. i tells you the depth (blue, red, or green), j tells you the row, and k tells you the column. When numpy prints, it just lists out blue, red, then green in order.
Related
I get confused by this example.
A = np.random.random((6, 4, 5))
A
A.min(axis=0)
A.min(axis=1)
A.min(axis=2)
What mins are we really computing here?
I know I can think of this array as a 6x5x4 Parallelepiped in 3D space and I know A.min(axis=0) means we go along the 0-th axis. OK, but as we go along that 0-th axis all we get is 6 "layers" which are basically rectangles of size 4x5 filled with numbers. So what min am I computing when saying A.min(axis=0) for example?!?! I am just trying to visualize it in my head.
From A.min(axis=0) I get back a 4x5 2D matrix. Why? Shouldn't I get just 6 values in a 1D array. I am walking along the 0-th axis so shouldn't I get 6 values back - one value for each of these 4x5 rectangles?
I always find this notation confusing and just don't get it, sorry.
You calculate the min across one particular axis when you are interested in maintaining the structure of the remainder axes.
The gif below may help to understand.
In this example, your result will have shape (3, 2).
That's because you are getting the smallest value along axis 0, which squeezes that dimension into only 1 value, so we don't need the dimension anymore.
How to change the shape of array from ixMxNx3 to (M*N)xix3?
I have a ixMxNx3 array L. You can think of L as an array containing i images, each image has height=M, width=N, and in each pixel it has a three-dimensional vector (or rgb). Let P = M*N. I can change its shape to ixPx3 by L.reshape(i,P,3). (I hope it is really changing it to the shape I want). How do I change its shape to Pxix3? i.e. an array that contains P points, each point has i images, each image of that point has a three-dimensional vector.
How can this change of shape be accomplished?
numpy.rollaxis can shift the position of an axis in a NumPy array:
L = L.reshape([i, P, 3])
L = numpy.rollaxis(L, 1)
It takes 3 arguments, one optional. The first is the array, the second is the axis to move, and the third is confusingly documented as "The axis is rolled until it lies before this position". Basically, if you want to move the ith axis to the jth position and j<i, the third argument should be j. If j>i, the third argument should be j+1. I don't know why it works that way. The third argument defaults to 0.
A 3D label map is matrix in which every pixel (voxel) has an integer label. These values are expected to be contiguous, meaning that a segment with label k will not be fragmented.
Given such label map (segmentation), what is the fastest way to obtain the coordinates of a minimum bounding box around each segment, in Python?
I have tried the following:
Iterate through the matrix using multiindex iterator (from numpy.nditer) and construct a reverse index dictionary. This means that for every label you get the 3 coordinates of every voxel where the label is present.
For every label get the max and min of each coordinate.
The good thing is that you get all the location information in one O(N) pass. The bad thing is that I dont need this detailed information. I just need the extremities, so there might be a faster way to do this, using some numpy functions which are faster than so many list appends. Any suggestions?
The one pass through the matrix takes about 8 seconds on my machine, so it would be great to get rid of it. To give an idea of the data, there are a few hundred labels in a label map. Sizes of the label map can be 700x300x30 or 300x300x200 or something similar.
Edit: Now storing only updated max and min per coordinate for every label. This removes the need to maintain and store all these large lists (append).
If I understood your problem correctly, you have groups of voxels, and you would like to have the extremes of a group in each axis.
Let'd define:
arr: 3D array of integer labels
labels: list of labels (integers 0..labmax)
The code:
import numpy as np
# number of highest label:
labmax = np.max(labels)
# maximum and minimum positions along each axis (initialized to very low and high values)
b_first = np.iinfo('int32').min * np.ones((3, labmax + 1), dtype='int32')
b_last = np.iinfo('int32').max * np.ones((3, labmax + 1), dtype='int32')
# run through all of the dimensions making 2D slices and marking all existing labels to b
for dim in range(3):
# create a generic slice object to make the slices
sl = [slice(None), slice(None), slice(None)]
bf = b_first[dim]
bl = b_last[dim]
# go through all slices in this dimension
for k in range(arr.shape[dim]):
# create the slice object
sl[dim] = k
# update the last "seen" vector
bl[arr[sl].flatten()] = k
# if we have smaller values in "last" than in "first", update
bf[:] = np.clip(bf, None, bl)
After this operation we have six vectors giving the smallest and largest indices for each axis. For example, the bounding values along second axis of label 13 are b_first[1][13] and b_last[1][13]. If some label is missing, all corresponding b_first and b_last will be the maximum int32 value.
I tried this with my computer, and for a (300,300,200) array it takes approximately 1 sec to find the values.
So this is a little follow up question to my earlier question: Generate coordinates inside Polygon and my answer https://stackoverflow.com/a/15243767/1740928
In fact, I want to bin polygon data to a regular grid. Therefore, I calculate a couple of coordinates within the polygon and translate their lat/lon combination to their respective column/row combo of the grid.
Currently, the row/column information is stored in a numpy array with its number of rows corresponding to the number of data polygons and its number of columns corresponding to the coordinates in the polygon.
The whole code takes less then a second, but this code is the bottleneck at the moment (with ~7sec):
for ii in np.arange(len(data)):
for cc in np.arange(data_lats.shape[1]):
final_grid[ row[ii,cc], col[ii,cc] ] += data[ii]
final_grid_counts[ row[ii,cc], col[ii,cc] ] += 1
The array "data" simply contains the data values for each polygon (80000,). The arrays "row" and "col" contain the row and column number of a coordinate in the polygon (shape: (80000,16)).
As you can see, I am summing up all data values within each grid cell and count the number of matches. Thus, I know the average for each grid cell in case different polygons intersect it.
Still, how can these two for loops take around 7 seconds? Can you think of a faster way?
I think numpy should add an nd-bincount function, I had one lying around from a project I was working on some time ago.
import numpy as np
def two_d_bincount(row, col, weights=None, shape=None):
if shape is None:
shape = (row.max() + 1, col.max() + 1)
row = np.asarray(row, 'int')
col = np.asarray(col, 'int')
x = np.ravel_multi_index([row, col], shape)
out = np.bincount(x, weights, minlength=np.prod(shape))
return out.reshape(shape)
weights = np.column_stack([data] * row.shape[1])
final_grid = two_d_bincount(row.ravel(), col.ravel(), weights.ravel())
final_grid_counts = two_d_bincount(row.ravel(), col.ravel())
I hope this helps.
I might not fully understand the shapes of your different grids, but you can maybe eliminate the cc loop using something like this:
final_grid = np.empty((nrows,ncols))
for ii in xrange(len(data)):
final_grid[row[ii,:],col[ii,:]] = data[ii]
This of course assumes that final_grid is starting with no other info (that the count you're incrementing starts at zero). And I'm not sure how to test if it works not understanding how your row and col arrays work.
I have a 16x16x4 array in Numpy.
Dimension 1: Horizontal position [0,15]
Dimension 2: Vertical position [0,15]
Dimension 3: An RGB value 0-255 [0,3]
Replace 16x16 with 2048x1285 and:
for x in range(0,15):
for y in range(0,15):
Doesn't cut it (upwards of 7 minutes to do this and a flood fill at each interesting point). Iterating over a PIL image is plenty fast, but a numpy array drags (i.e. 7+ minutes).
numpy.where(bitmap == [red, green, blue, alpha])
doesn't seem like it's what I'm looking for. What's a reasonably fast way to go about this?
Edit:
bitmap == [red, green, blue, alpha]
is actually almost useful. How do I go from a 16x16x4 array to a 16x16x1 array where array[x,y] is 1 if z = [True,True,True,True] and 0 otherwise?
I can't reproduce your speeds -- even a brute-force iteration on my now-ancient notebook is about 14 times faster -- and I'm not sure where works the way you think it does, so I suspect that most of your time is spent elsewhere (say in your fill). Anyway:
How do I go from a 16x16x4 array to a 16x16x1 array where array[x,y]
is 1 if z = [True,True,True,True] and 0 otherwise?
I would:
In [169]: m = numpy.random.randint(0, 16, size=(2048, 1285, 4))
In [170]: b = [4,5,6,7]
In [171]: matches = (m == b).all(axis=2)*1
In [172]: matches.shape
Out[172]: (2048, 1285)
and it's pretty fast:
In [173]: timeit matches = (m == b).all(axis=2)*1
10 loops, best of 3: 137 ms per loop
It turns out that what I described is achieved by
zip(*numpy.where(numpy.logical_and.reduce(image == [255,255,255])))
Which was not clear according to any documentation, but there you have it. (Edit: lack of alpha channel is not significant.)
The test I'm interested in isn't actually equality to a point, but Euclidian distance to that point being within a threshold.
numpy.apply_along_axis(distance_from_white ...
where distance_from_white is a function returning Euclidian distance from white, works at 16x16 but takes minutes at 2048x1245. scipy.spatial.distance.pdist (or cdist?) might be the answer there, but I can't figure out how to make it compute distance against a single point instead of distance between all points in 2 arrays (this works at 16x16, but it's so much wasted computation I'm hesitant to even try it at actual size).
Iterating with for loops on a ndarray isn't very efficient, as you have noticed. If you want to find the indices of the entries satisfying your condition, you should indeed use
indices = np.where(bitmap == [R, G, B, A])
This will return a 3-element tuple giving the indices of the solution along the 1st, 2nd and third axis. As you're only interested in the first two dimensions, you can drop the third item. And if you want a series of indices like (x,y), you just have to use something like
zip(*indices[:2])
A second possibility is to view your (N,M,4) standard integer ndarray into a (N,M) structured array with dtype=[[('',int)]*4] (don't bother for the field names, they'll be automatically expended to 'f0', 'f1', ... :
alt_bitmap = bitmap.view([('',int)'*4).squeeze()
(The squeeze is introduced to collapse the (N,M,1) array into a (N,M) array)
You can then use the np.where function, but the values you're testing must be a np.array too:
indices = np.where(bitmap==np.array((R, G, B, A), dtype=bitmap.dtype))
This time, indices will only be a 2-tuple, and you can get the (x,y) couples with the zip(*indices) presented earlier.