I am somewhat confused about selecting a column of an NumPy array, because the result is different from Matlab and even from NumPy matrix. Please see the following cases.
In Matlab, we use the following command to select a column vector out of a matrix.
x = [0, 1; 2 3]
out = x(:, 1)
Then out becomes [0; 2], which is a column vector.
To do the same thing with a NumPy Matrix
x = np.matrix([[0, 1], [2, 3]])
out = x[:, 0]
Then the output is np.matrix([[0], [2]]) which is expected, and it is a column vector.
However, in case of NumPy array
x = np.array([[0, 1], [2, 3]])
out = x[:, 0]
The output is np.array([0, 2]) which is 1 dimensional, so it is not a column vector. My expectation is it should have been np.array([[0], [2]]).
I have two questions.
1. Why is the output from the NumPy array case different form the NumPy matrix case? This is causing a lot of confusion to me, but I think there might be some reason for this.
2. To get a column vector from a 2-Dim NumPy Array, then should I do additional things like expand_dims
x = np.array([[0, 1], [2, 3]])
out = np.expand_dims(x[:, 0], axis = 1)
In MATLAB everything has atleast 2 dimensions. In older MATLABs, 2d was it, now they can have more. np.matrix is modeled on that old MATLAB.
What does MATLAB do when you index a 3d matrix?
np.array is more general. It can have 0, 1, 2 or more dimensions.
x[:, 0]
x[0, :]
both select one column or row, and return an array with one less dimension.
x[:, [0]]
x[[0], :]
would return 2d arrays, with a singleton dimension.
In Octave (MATLAB clone) indexing produces inconsistent results, depending on which side of matrix I select:
octave:7> x=ones(2,3,4);
octave:8> size(x)
ans =
2 3 4
octave:9> size(x(1,:,:))
ans =
1 3 4
octave:10> size(x(:,:,1))
ans =
2 3
MATLAB/Octave adds dimensions at the end, and apparently readily squeezes them down on that side as well.
numpy orders the dimensions in the other direction, and can add dimensions at the start as needed. But it is consistent in squeezing out singleton dimensions when indexing.
The fact that numpy can have any number of dimensions, while MATLAB has a minimum of 2 is a crucial difference that often trips up MATLAB users. But one isn't any more logical than the other. MATLAB's practice is more a more matter of history than general principals.
Related
Let's say that we have the following 2d numpy array:
arr = np.array([[1,1,0,1,1],
[0,0,0,1,0],
[1,0,0,0,0],
[0,0,1,0,0],
[0,1,0,0,0]])
and the following indices for rows and columns:
rows = np.array([0,2,4])
cols = np.array([1,2])
The objective is to slice arr using rows and cols to take the following expected result:
arr_sliced = np.array([[1,0],
[0,0],
[1,0]])
Using directly the arrays as indices like arr[rows, cols] leads to:
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (3,) (2,)
So what is the straightforward way to achieve this kind of slicing?
Update: useful information about the solution
So the solution was simple enough and it demands a basic comprehension about numpy's broadcasting. Someone could read these nice but not so representative examples from numpy. Also, the general broadcasting rules explains why there is no shape mismatch in:
arr[rows[:, np.newaxis], cols]
# rows[:, np.newaxis].shape == (3,1)
# cols.shape == (2,)
You can use:
arr[rows[:,None], cols[None]]
Output:
array([[1, 0],
[0, 0],
[1, 0]])
It looks like it is much quicker than indexing for large arrays.
arr[np.ix_([0,2,4],[1,2])]
array([[1, 0],
[0, 0],
[1, 0]])
document: https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.ix_.html
This function takes N 1-D sequences and returns N outputs with N dimensions each, such that the shape is 1 in all but one dimension and the dimension with the non-unit shape value cycles through all N dimensions.
I am trying to get the dotproduct of two arrays in python using the numpy package. I get as output an array of size (n,). It says that my array has no column while I do see the results when I print it. Why does my array have no column and how do I fix this?
My goal is to calculate y - np.dot(x,b). The issue is that y is (124, 1) while np.dot(x,b) is (124,)
Thanks
It seems that you are trying to subtract two arrays of a different shape. Fortunately, it is off by a single additional axis, so there are two ways of handling it.
(1) You slice the y array to match the shape of the dot(x,b) array:
y = y[:,0]
print(y-np.dot(x,b))
(2) You add an additional axis on the np.dot(x,b) array:
dot = np.dot(x,b)
dot = dot[:,None]
print(y-dot)
Hope this helps
it may depends on the dimension of your array
For example :
a = [1, 0]
b = [[4, 1], [2, 2]]
c = np.dot(a,b)
gives
array([4, 1])
and its shape is (2,)
but if you change a like :
a = [[1, 0],[1,1]]
then result is :
array([[4, 1],
[6, 3]])
and its shape is (2,2)
I have a numpy array of shape [batch_size, timesteps_per_samples, width, height], where width and height refer to a 2D grid. The values in this array can be interpreted as an elevation at a certain location that changes over time.
I want to know the elevation over time for various paths within this array. Therefore i have a second array of shape [batch_size, paths_per_batch_sample, timesteps_per_path, coordinates] (coordinates = 2, for x and y in the 2D plane).
The resulting array should be of shape [batch_size, paths_per_batch_sample, timesteps_per_path] containing the elevation over time for each sample within the batch.
The following two examples work. The first one is very slow and just serves for understanding what I am trying to do. I think the second one does what I want but I have no idea why this works nor if it may crash under certain circumstances.
Code for the problem setup:
import numpy as np
batch_size=32
paths_per_batch_sample=10
timesteps_per_path=4
width=64
height=64
elevation = np.arange(0, batch_size*timesteps_per_path*width*height, 1)
elevation = elevation.reshape(batch_size, timesteps_per_path, width, height)
paths = np.random.randint(0, high=width-1, size=(batch_size, paths_per_batch_sample, timesteps_per_path, 2))
range_batch = range(batch_size)
range_paths = range(paths_per_batch_sample)
range_timesteps = range(timesteps_per_path)
The following code works but is very slow:
elevation_per_time = np.zeros((batch_size, paths_per_batch_sample, timesteps_per_path))
for s in range_batch:
for k in range_paths:
for t in range_timesteps:
x_co, y_co = paths[s,k,t,:].astype(int)
elevation_per_time[s,k,t] = elevation[s,t,x_co,y_co]
The following code works (even fast) but I can't understand why and how o.0
elevation_per_time_fast = elevation[
:,
range_timesteps,
paths[:, :, range_timesteps, 0].astype(int),
paths[:, :, range_timesteps, 1].astype(int),
][range_batch, range_batch, :, :]
Prove that the results are equal
check = (elevation_per_time == elevation_per_time_fast)
print(np.all(check))
Can somebody explain how I can slice an nd-array by multiple other arrays?
Especially, I don't understand how the numpy knows that 'range_timesteps' has to run in step (for the index in axis 1,2,3).
Thanks in advance!
Lets take a quick look at slicing numpy array first:
a = np.arange(0,9,1).reshape([3,3])
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
Numpy has 2 ways of slicing array, full sections start:stop and by index from a list [index1, index2 ...]. The output will still be an array with the shape of your slice:
a[0:2,:]
array([[0, 1, 2],
[3, 4, 5]])
a[:,[0,2]]
array([[0, 2],
[3, 5],
[6, 8]])
The second part is that since you get a returned array with the same amount of dimensions you can easily stack any number of slices as long as you dont try to directly access an index outside of the array.
a[:][:][:][:][:][:][:][[0,2]][:,[0,2]]
array([[0, 2],
[6, 8]])
I am running the following code:
import numpy as np
a = np.array([1, 2])
b = np.array([[1, 2]])
a = a + b
print(a)
[[2 , 4 ]]
As you can see. dimension of a is 1, and b is 2.
Mathematically, it is not possible to add between different dimensional arrays
how can it work under the numpy? and what does that mean [ [ 2, 4 ] ]?
a.shape is (2, )
b.shape is (1 ,2)
(a+b).shape is (1 ,2)
However the following code yields an error:
import numpy as np
a = np.array([1, 2])
b = np.array([[1, 2]])
a += b
Why doesn't it work? What makes different result?
As already implied in the comments, it always helps to check the documentation; quoting:
When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward. Two dimensions are compatible when
they are equal,
or one of them is 1
In your case both of the arrays share a dimension that has 1 element (1 column in A and 1 row in B).
Because of that it adds them in a way that would not make sense mathematically.
If you changed this and had arrays that share a dimension which is 1, then you would get an error.
How do I use Numpy matrix operations to calculate over multiple vector samples at once?
Please see below the code I came up with, 'd' is the outcome I'm trying to get. But this is only one sample. How do I calculate the output without doing something like repeat the code for every sample OR looping through every sample?
a = np.array([[1, 2, 3]])
b = np.array([[1, 2, 3]])
c = np.array([[1, 2, 3]])
d = ((a.T * b).flatten() * c.T)
a1 = np.array([[2, 3, 4]])
b1 = np.array([[2, 3, 4]])
c1 = np.array([[2, 3, 4]])
d1 = ((a1.T * b1).flatten() * c1.T)
a2 = np.array([[3, 4, 5]])
b2 = np.array([[3, 4, 5]])
c2 = np.array([[3, 4, 5]])
d2 = ((a2.T * b2).flatten() * c2.T)
The way broadcasting works is to repeat your data along an axis of size one as many times as necessary to make your element-wise operation work. That is what is happening to axis 1 of a.T and axis 0 of b. Similar for the product of the result. My recommendation would be to concatenate all your inputs along another dimension, to allow broadcasting to happen along the existing two.
Before showing how to do that, let me just mention that you would be much better off using ravel instead of flatten in your example. flatten makes a copy of the data, while ravel only makes a view. Since a.T * b is a temporary matrix anyway, there is really no reason to make the copy.
The easiest way to combine some arrays along a new dimension is np.stack. I would recommend combining along the first dimension for a couple of reasons. It's the default for stack and your result can be indexed more easily: d[0] will be d, d[1] will be d1, etc. If you ever add matrix multiplication into your pipeline, np.dot will work out of the box since it operates on the last two dimensions.
a = np.stack((a0, a1, a2, ..., aN))
b = np.stack((b0, b1, b2, ..., bN))
c = np.stack((c0, c1, c2, ..., cN))
Now a, b and c are all 3D arrays the first dimension is the measurement index. The second and third correspond to the two dimensions of the original arrays.
With this structure, what you called transpose before is just swapping the last two dimensions (since one of them is 1), and raveling/flattening is just multiplying out the last two dimensions, e.g. with reshape:
d = (a.reshape(N, -1, 1) * b).reshape(N, 1, -1) * c.reshape(N, -1, 1)
If you set one of the dimensions to have size -1 in the reshape, it will absorb the remaining size. In this case, all your arrays have 3 elements, so the -1 will be equivalent to 3.
You have to be a little careful when you convert the ravel operation to 3D. In 2D, x.ravel() * c.T implicitly transforms x into a 1xN array before broadcasting. In 3D, x.reshape(3, -1) creates a 2D 3x27 array, which you multiply by c.reshape(3, -1, 1), which is 3x3x1. Broadcasting rules state that you are effectively multiplying a 1x3x27 array by a 3x3x1, but you really want to multiply a 3x1x27 array by the 3x3x1, so you need to specify all three axes for the 3D "ravel" explicitly.
Here is an IDEOne link with your sample data for you to play with: https://ideone.com/p8vTlx