I am a new user to numpy and I was using numpy delete, where it mention that to delete horizontal row we should use axis=0 but in other documentation of numpy glossary, it says horizontal axis is 1. It would be great if someone can let me know what is wrong in my understanding.
An array is a systematic way of structuring numbers in grids of any dimensionality. The grid directions have labels, and these labels come from a convention of how new dimensions are added to a grid.
Here's the convention:
The simplest such grid is a 0-dimensional (0D) array, which has no axes and can only hold a scalar. This is a 0D array:
42
If we start putting scalars into a list we get a 1D array. This new grid only has one axis, and if we want to label that axis with a number, we better start with something simple - like axis=0! A 1D array could be:
# ----0--->
[42, π, √2]
Now we want to create an array of 1D arrays, which will give us a 2D array. The horizontal axis will still be 0, but the new vertical axis will get the next lowest number we know, axis=1. Here's what it could look like:
# ----0---->
[[42, π, √2], # |
[1, 2, 3], # 1
[10, 20, 30]] # V
The true beauty is that this generalizes to infinity. If we need a box of numbers we'd create a 3D array by stacking 2D arrays, and the direction that traces the depth of the box would naturally have to be axis=2. If we wanted a 4D array, we would just make a list of boxes (3D arrays), and call every box using an index along axis=3. This can go on forever.
In NumPy:
Any function/method that takes an axis-argument uses this convention. For a 2D array this means that doing something like np.delete(X, [1, 2, 3], axis=0) will iterate over arrays extruded along the 0'th axis, to return X without rows 1, 2 and 3. The same logic applies for getting values from an array.
X[rows_along_0th_axis, columns_along_1st_axis, ..., vectors_along_nth_axis]
Taking from the links that you provided, here the excerpts from numpy delete and glossary that probably caused you some confusions and the clarification in the following.
Excerpt
>>> arr = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
>>> arr
array([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12]])
>>> np.delete(arr, 1, 0)
array([[ 1, 2, 3, 4],
[ 9, 10, 11, 12]])
Excerpt
the first running vertically downwards across rows (axis 0), and the
second running horizontally across columns (axis 1)
I think the confusion derives from the words vertically and horizontally in the second excerpt.
What the second excerpt means is that by setting axis it is possible to decide over which dimension to move. For example, in a 2d matrix, axis=0 corresponds to iterating over the rows (thus moving vertically over the array), while axis=1 corresponds
to iterating over columns (so moving horizontally over the array). It does not say that axis=1 corresponds to the horizontal axis as the OP understood.
The delete function follows the above description, as indeed, by using np.delete(arr, 1, axis=0), the function iterates over the rows, and deletes the row with index 1. If, instead, columns should be deleted, then axis=1. For example, on the same array arr
>>> np.delete(arr, [0,1,4], axis=1)
array([[ 3, 4],
[ 7, 8],
[11, 12]])
in which delete iterates over the columns, and the columns with indices 0, 1 are deleted, and nothing else is deleted as column with index 4 does not exist.
Related
I have a numpy array of shape [batch_size, timesteps_per_samples, width, height], where width and height refer to a 2D grid. The values in this array can be interpreted as an elevation at a certain location that changes over time.
I want to know the elevation over time for various paths within this array. Therefore i have a second array of shape [batch_size, paths_per_batch_sample, timesteps_per_path, coordinates] (coordinates = 2, for x and y in the 2D plane).
The resulting array should be of shape [batch_size, paths_per_batch_sample, timesteps_per_path] containing the elevation over time for each sample within the batch.
The following two examples work. The first one is very slow and just serves for understanding what I am trying to do. I think the second one does what I want but I have no idea why this works nor if it may crash under certain circumstances.
Code for the problem setup:
import numpy as np
batch_size=32
paths_per_batch_sample=10
timesteps_per_path=4
width=64
height=64
elevation = np.arange(0, batch_size*timesteps_per_path*width*height, 1)
elevation = elevation.reshape(batch_size, timesteps_per_path, width, height)
paths = np.random.randint(0, high=width-1, size=(batch_size, paths_per_batch_sample, timesteps_per_path, 2))
range_batch = range(batch_size)
range_paths = range(paths_per_batch_sample)
range_timesteps = range(timesteps_per_path)
The following code works but is very slow:
elevation_per_time = np.zeros((batch_size, paths_per_batch_sample, timesteps_per_path))
for s in range_batch:
for k in range_paths:
for t in range_timesteps:
x_co, y_co = paths[s,k,t,:].astype(int)
elevation_per_time[s,k,t] = elevation[s,t,x_co,y_co]
The following code works (even fast) but I can't understand why and how o.0
elevation_per_time_fast = elevation[
:,
range_timesteps,
paths[:, :, range_timesteps, 0].astype(int),
paths[:, :, range_timesteps, 1].astype(int),
][range_batch, range_batch, :, :]
Prove that the results are equal
check = (elevation_per_time == elevation_per_time_fast)
print(np.all(check))
Can somebody explain how I can slice an nd-array by multiple other arrays?
Especially, I don't understand how the numpy knows that 'range_timesteps' has to run in step (for the index in axis 1,2,3).
Thanks in advance!
Lets take a quick look at slicing numpy array first:
a = np.arange(0,9,1).reshape([3,3])
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
Numpy has 2 ways of slicing array, full sections start:stop and by index from a list [index1, index2 ...]. The output will still be an array with the shape of your slice:
a[0:2,:]
array([[0, 1, 2],
[3, 4, 5]])
a[:,[0,2]]
array([[0, 2],
[3, 5],
[6, 8]])
The second part is that since you get a returned array with the same amount of dimensions you can easily stack any number of slices as long as you dont try to directly access an index outside of the array.
a[:][:][:][:][:][:][:][[0,2]][:,[0,2]]
array([[0, 2],
[6, 8]])
This relates to this earlier post: Numpy random choice of tuples
I have a 2D numpy array and want to choose from it using a 2D probability array. The only way I could think to do this was to flatten and then use the modulo and remainder to convert the result back to a 2D index
import numpy as np
# dummy data
x=np.arange(100).reshape(10,10)
# dummy probability array
p=np.zeros([10,10])
p[4:7,1:4]=1.0/9
xy=np.random.choice(x.flatten(),1,p=p.flatten())
index=[int(xy/10),(xy%10)[0]] # convert back to index
print(index)
which gives
[5, 2]
but is there a cleaner way that avoids flattening and the modulo? i.e. I could pass a list of coordinate tuples as x, but how can I then handle the weights?
I don't think it's possible to directly specify a 2D shaped array of probabilities. So raveling should be fine. However to get the corresponding 2D shaped indices from the flat index you can use np.unravel_index
index= np.unravel_index(xy.item(), x.shape)
# (4, 2)
For multiple indices, you can just stack the result:
xy=np.random.choice(x.flatten(),3,p=p.flatten())
indices = np.unravel_index(xy, x.shape)
# (array([4, 4, 5], dtype=int64), array([1, 2, 3], dtype=int64))
np.c_[indices]
array([[4, 1],
[4, 2],
[5, 3]], dtype=int64)
where np.c_ stacks along the right hand axis and gives the same result as
np.column_stack(indices)
You could use numpy.random.randint to generate an index, for example:
# assumes p is a square array
ij = np.random.randint(p.shape[0], size=p.ndim) # size p.ndim = 2 generates 2 coords
# need to convert to tuple to index correctly
p[tuple(i for i in ij))]
>>> 0.0
You can also index multiple random values at once:
ij = np.random.randint(p.shape[0], size=(p.ndim, 5)) # get 5 values
p[tuple(i for i in ij))]
>>> array([0. , 0. , 0. , 0.11111111, 0. ])
While I have already found the documentation on scipy.ndimage.convolve function and I "practically know what it does", when I try to calculate the resulting arrays I can't follow the mathematical formula. Let's take for example:
a = np.array([[1, 2, 0, 0],`
[5, 3, 0, 4],
[0, 0, 0, 7],
[9, 3, 0, 0]])
k = np.array([[1,1,1],[1,1,0],[1,0,0]])
from scipy import ndimage
ndimage.convolve(a, k, mode='constant', cval=0.0)
# Why is the result like this ?
array([[11, 10, 7, 4],
[10, 3, 11, 11],
[15, 12, 14, 7],
[12, 3, 7, 0]])
I would appreciate a step by step calculation.
Details on NDImage.convolve
I stumbled on this NDImage convolution eventhough I know the basic np.convolve, and the document is not much self explanatory, so I took the effort to crunch through and supplement the earlier explanatory post:
A. Basics:
Reference: refers to following if your concept on convolution is not so well grounded
https://en.wikipedia.org/wiki/Kernel_(image_processing),
https://en.wikipedia.org/wiki/Convolution
Essentially NDimage.convolve has 4 modes, this post focused on the Constant mode, for which you use the value as specified by cval=0 or whatever and add padded rows and columns as needed (will explain in a little bit)
The convolution essentially slides the kernel from left and right and then step down again and from left to right again until the needed (same number) number of convolved elements are achieved
The function will calculate the padded rows/columns needed. In this case the filter K is 3 x 3 matrix, and the source image is matrix a is 4 x 4, so you need two padded rows at top and bottom and two padded rows at left and right (4 + 2 = 6, and the number of rows or columns needed is 3 + 1 + 1 + 1 = 6, each slide will need the extra one row or column)
B. Operations:
Add a row and column of zeros to the top and left of Array a (to convolve a 3 x 3 to 4 x 4 evenly,
you need extra padded row/column at the 1st and 4th sliding window) and also one row/column of padded zeros to the bottom and right
Flip the kernel K as Kflip: [[0,0,1], [0,1,1], [1,1,1]]
you can use numpy np.flip (why it need to be flipped basically relates to the concept of convolution vs correlation which are like twins in opposite direction)
Slide the flipped K matrix onto this size 6 x 6 expanded matrix [[0,0,0,0,0], [0,1,2,0,0], [0,5,3,0,4,0], [0,0,0,0,7], [0,9,3,0,0,0], [0,0,0,0,0]]
For the first step of sliding window (note the first row of column of the kernel will convolved with the padded zeros), you get:
Flipped K dot sum [[0,0,0], [0,1,2], [0,5,3]] = 11 (1*1+1*2+1*5+1*3, others are zeros)
(dot sum refers to sum of the inner dot element-wise multiplication, basically just multiply the corresponding elements in the same positions for the two given matrices)
Slide K one step to the right, you will have 10 (first row all zeros due to padded zeros, second row: 1*2+, third row 1*3 + 1*4, fourth row all zeros due to [0,0,0,0,7])
likewise you slide to the right for another two steps to get all four elements for the convolved matrix (note for the 4th of this row, again we partially convolved on expanded padded row/columns)(
Then you slide the K filter one row down and reset to the far left of the "expanded /padded matrix"
You will have again the same 10 (first row: 1*2+, second row 1*3 + 1*4), so on and so forth
Just to warm up consider
k = np.array([[1,0,0],[0,1,0],[0,0,0]])
instead of your k, then if you
ndimage.convolve(a, k, mode='constant', cval=0.0)
you get
array([[4, 2, 4, 0],
[5, 3, 7, 4],
[3, 0, 0, 7],
[9, 3, 0, 0]])
and note that any element is the sum of it's own position (due to the 2nd 1 in k) and the one below and to the right (due to the 1st 1 in k), ie the 4 in the top corner is from the original 1 in the top corner plus the 3 diagonally down from it.
The (possibly) confusing part is that the effect of the k is opposite of what you might expect, ie for the k above you might expect the first 1 to add the value above and to the left, instead of down and to the right.
Now back to yours: the 12 (3 down and 2 across) is the sum of 9+3+0+0+0+0.
Note that anything outside the matrix is assumed to be 0.
I have two numpy arrays, like
A: = array([[0, 1],
[2, 3],
[4, 5]])
B = array([[ 6, 7],
[ 8, 9],
[10, 11]])
For each row of A and B, say Ra and Rb respectively, I want to calculate transpose(Ra)*Rb. So for given value of A and B, i want following answer:
array([[[ 0, 0],
[ 6, 7]],
[[ 16, 18],
[ 24, 27]],
[[ 40, 44],
[ 50, 55]]])
I have written the following code to do so:
x = np.outer(np.transpose(A[0]), B[0])
for i in range(1,len(A)):
x = np.append(x,np.outer(np.transpose(A[i]), B[i]),axis=0)
Is there any better way to do this task.
You can use extend dimensions of A and B with np.newaxis/None to bring in broadcasting for a vectorized solution like so -
A[...,None]*B[:,None,:]
Explanation : np.outer(np.transpose(A[i]), B[i]) basically does elementwise multiplications between a columnar version of A[i] and B[i]. You are repeating this for all rows in A against corresoinding rows in B. Please note that the np.transpose() doesn't seem to make any impact as np.outer takes care of the intended elementwise multiplications.
I would describe these steps in a vectorized language and thus implement, like so -
Extend dimensions of A and B to form 3D shapes for both of them such that we keep axis=0 aligned and keep as axis=0 in both of those extended versions too. Thus, we are left with deciding the last two axes.
To bring in the elementwise multiplications, push axis=1 of A in its original 2D version to axis=1 in its 3D version, thus creating a singleton dimension at axis=2 for extended version of A.
This last singleton dimension of 3D version of A has to align with the elements from axis=1 in original 2D version of B to let broadcasting happen. Thus, extended version of B would have the elements from axis=1 in its 2D version being pushed to axis=2 in its 3D version, thereby creating a singleton dimension for axis=1.
Finally, the extended versions would be : A[...,None] & B[:,None,:], multiplying whom would give us the desired output.
I have many measurements of several quantities in an array, like this:
m = array([[2, 1, 3, 2, 1, 4, 2], # measurements for quantity A
[8, 7, 6, 7, 5, 6, 8], # measurements for quantity B
[0, 1, 2, 0, 3, 2, 1], # measurements for quantity C
[5, 6, 7, 5, 6, 5, 7]] # measurements for quantity D
)
The quantities are correlated and I need to plot various contour plots. Like "contours of B vs. D x A".
It is true that in the general case the functions might be not well defined -- for example in the above data, columns 0 and 3 show that for the same (D=5,A=2) point there are two distinct values for B (B=8 and B=7). But still, for some combinations I know there is a functional dependence, which I need plotted.
The contour() function from MatPlotLib expects three arrays: X and Y can be 1D arrays, and Z has to be a 2D array with corresponding values. How should I prepare/extract these arrays from m?
You will probably want to use something like scipy.interpolate.griddata to prepare your Z arrays. This will interpolate your data to a regularly spaced 2D array, given your input X and Y, and a set of sorted, regularly spaced X and Y arrays which you will need for eventual plotting. For example, if X and Y contain data points between 1 and 10, then you need to construct a set of new X and Y with a step size that makes sense for your data, e.g.
Xout = numpy.linspace(1,10,10)
Yout = numpy.linspace(1,10,10)
To turn your Xout and Yout arrays into 2D arrays you can use numpy.meshgrid, e.g.
Xout_2d, Yout_2d = numpy.meshgrid(Xout,Yout)
Then you can use those new regularly spaced arrays to construct your interpolated Z array that you can use for plotting, e.g.
Zout = scipy.interpolate.griddata((X,Y),Z,(Xout_2d,Yout_2d))
This interpolated 2D Zout should be usable for a contour plot with Xout_2d and Yout_2d.
Extracting your arrays from m is simple, you just do something like this:
A, B, C, D = (row for row in m)