Related
Is it possible to divide multiple numpy array columns by another 1D column (row wise division)?
Example:
a1 = np.array([[1,2,3],[4,5,6],[7,8,9]])
array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
a2 = np.array([11,12,13])
array([11, 12, 13])
# that is divide all rows, but only columns 1 and 2 by a2 array
a1[:,:2] / a2
ValueError: operands could not be broadcast together with shapes (3,2) (3,)
I did try this, but this does not look elegant
(a1[:,:2].T / a2).T
array([[0.09090909, 0.18181818],
[0.33333333, 0.41666667],
[0.53846154, 0.61538462]])
Your a1 array is 2D and a2 is 1D, try expanding the dimension of a2 array to 2D before performing division:
>>> a1[:,:2]/np.expand_dims(a2, 1)
array([[0.09090909, 0.18181818],
[0.33333333, 0.41666667],
[0.53846154, 0.61538462]])
Apparently, you can just use a2[:, None] instead of calling expand_dims function for even cleaner code.
The issue
I have a very simple function which uses numpy.where() for a very simple calculation.
If the input is a scalar, the output is a numpy array of size ().
If I multiply it by one, it becomes a numpy int32 of the same size.
My questions are:
what is the difference between 1 and 2?
Are they both scalars? Does such a thing as a numpy scalar even exist?
Why does multiplying it by one change the type? Is this a known, documented feature/bug?
why do other numpy functions, e.g. np.arange(5,6) return an array of size (1,) instead?
I doubt I am the first one to come across this but I haven't found much online.
I have found questions on the difference between an array of shape (n,) and one of shape (n,1), but that's a different matter.
Toy example:
import numpy as np
def my_find(a):
return np.where(a == 0 , 1, 0)
out_scalar = my_find(5)
out_scalar_times_1 = 1 * out_scalar
print("a scalar input return an output of type:")
print(type(out_scalar))
print("and of shape")
print(out_scalar.shape)
print("")
print("Multiplying it by 1 returns a:")
print(type(out_scalar_times_1))
out_array = my_find(np.arange(0,5))
Spyder screenshot
Yes, there is such a thing as a numpy scalar
https://numpy.org/doc/stable/reference/arrays.scalars.html
A numpy array can have 0,1,2 or more dimensions. There's a lot of overlap between
np.int64(3) # numpy int
np.array(3) # 0d array
np.array([3]) # 1d array with 1 element
np.int(3) # python int
3 # python int
The first 3 have array attributes like shape and dtype. The differences between the first two are minor.
In a function like where, numpy first converts the arguments to array, e.g. np.array(5), np.array(1)
In [161]: np.where(5, 1, 0)
Out[161]: array(1)
In [162]: _.shape
Out[162]: ()
In [163]: np.array(5)
Out[163]: array(5)
But math like addition with a scalar may return a numpy scalar:
In [164]: np.array(5) + 1
Out[164]: 6
In [165]: type(_)
Out[165]: numpy.int64
In [166]: np.array(5) * 1
Out[166]: 5
In [167]: type(_)
Out[167]: numpy.int64
Indexing an array can also produce such a scalar:
In [182]: np.arange(3)[1]
Out[182]: 1
In [183]: type(_)
Out[183]: numpy.int64
where 'broadcasts' the arguments, so the resulting shape is, in the broadcasted sense, the "largest":
In [168]: np.where(np.arange(5),1,0)
Out[168]: array([0, 1, 1, 1, 1])
In [173]: np.where(5, [1],0)
Out[173]: array([1])
In [174]: np.where(0, [1],0)
Out[174]: array([0])
In [175]: np.where([[0]], [1],0)
Out[175]: array([[0]])
If spyder has tab completion like ipython, you can get a list of all the methods attached to an object. The methods for an np.int64(3) will look a lot like the those for np.array(3). But very different from 3.
There are also arrays with 0 elements - if one of the dimensions is 0
Out[184]: array([], dtype=int64)
In [185]: _.shape
Out[185]: (0,)
In [186]: np.arange(1)
Out[186]: array([0])
In [187]: _.shape
Out[187]: (1,)
Obviously a 0d can't have 0 elements, because it doesn't have any 0 dimensions.
Indexing a 0d array (or numpy scalar) is a bit tricker (but still logical):
In [189]: np.array(3)[()] # 0 element indexing tuple
Out[189]: 3
In [190]: type(_)
Out[190]: numpy.int64
In [191]: np.array(3).item()
Out[191]: 3
In [192]: type(_)
Out[192]: int
In [193]: np.array(3)[()][()]
Out[193]: 3
The return of addition might be explained by 'array_priority'
dtype is not preserved in operations like this. Add a float to an int, and get a float.
In [203]: type(np.array(3, np.int16) + 3)
Out[203]: numpy.int64
In [204]: type(np.array(3, np.int16) + 3.0)
Out[204]: numpy.float64
ufunc casting
+ is actually a call to np.add ufunc. ufunc take key words like casting that give finer control over what results can be:
In [214]: np.add(np.array(3, np.int16), 3)
Out[214]: 6
In [215]: np.add(np.array(3, np.int16), 3, casting='no')
Traceback (most recent call last):
File "<ipython-input-215-631cb3a3b303>", line 1, in <module>
np.add(np.array(3, np.int16), 3, casting='no')
UFuncTypeError: Cannot cast ufunc 'add' input 0 from dtype('int16') to dtype('int64') with casting rule 'no'
In [217]: np.add(np.array(3, np.int16), 3, casting='safe')
Out[217]: 6
https://numpy.org/doc/stable/reference/ufuncs.html#output-type-determination
I was speculating that __array_priority__ played a role in returning a np.int64, but priorities go the wrong way.
In [194]: np.array(3).__array_priority__
Out[194]: 0.0
In [195]: np.int64(3).__array_priority__
Out[195]: -1000000.0
In [196]: np.array(3) + np.int64(3)
Out[196]: 6
In [197]: type(_)
Out[197]: numpy.int64
I don't know where it's documented, but often an operation will return a numpy scalar rather than a 0d array.
I just remembered/discovered one difference between 0d and numpy scalar - mutability
In [222]: x
Out[222]: array(3)
In [223]: x[...] = 4
In [224]: x
Out[224]: array(4)
In [225]: x = np.int64(3)
In [226]: x[...] = 4
Traceback (most recent call last):
File "<ipython-input-226-f7dca2cc5565>", line 1, in <module>
x[...] = 4
TypeError: 'numpy.int64' object does not support item assignment
Python classes can share a lot of behaviors/methods, but differ in others.
I'm trying to use individual 1-dimensional boolean arrays to slice a multi-dimension array. For some reason, this code doesn't work:
>>> a = np.ones((100, 200, 300, 2))
>>> a.shape
(100, 200, 300, 2)
>>> m1 = np.asarray([True]*200)
>>> m2 = np.asarray([True]*300)
>>> m2[-1] = False
>>> a[:,m1,m2,:]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (200,) (299,)
>>> m2 = np.asarray([True]*300) # try again with all 300 dimensions True
>>> a[:,m1,m2,:]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (200,) (300,)
But this works just fine:
>>> a = np.asarray([[[1, 2], [3, 4], [5, 6]], [[11, 12], [13, 14], [15, 16]]])
>>> a.shape
(2, 3, 2)
>>> m1 = np.asarray([True, False, True])
>>> m2 = np.asarray([True, False])
>>> a[:,m1,m2]
array([[ 1, 5],
[11, 15]])
Any idea of what I might be doing wrong in the first example?
Short answer: The number of True elements in m1 and m2 must match, unless one of them has only one True term.
Also distinguish between 'diagonal' indexing and 'rectangular' indexing. This is about indexing, not slicing. The dimensions with : are just along for the ride.
Initial ideas
I can get your first case working with:
In [137]: a=np.ones((100,200,300,2))
In [138]: m1=np.ones((200,),bool)
In [139]: m2=np.ones((300,),bool)
In [140]: m2[-1]=False
In [141]: I,J=np.ix_(m1,m2)
In [142]: a[:,I,J,:].shape
Out[142]: (100, 200, 299, 2)
np.ix_ turns the 2 boolean arrays into broadcastable index arrays
In [143]: I.shape
Out[143]: (200, 1)
In [144]: J.shape
Out[144]: (1, 299)
Note that this picks 200 'rows' in one dimension, and 299 in the other.
I'm not sure why this kind of reworking of the arrays is needed in this case, but not in the 2nd
In [154]: b=np.arange(2*3*2).reshape((2,3,2))
In [155]: n1=np.array([True,False,True])
In [156]: n2=np.array([True,False])
In [157]: b[:,n1,n2]
Out[157]:
array([[ 0, 4], # shape (2,2)
[ 6, 10]])
Taking the same ix_ strategy produces the same values but a different shape:
In [164]: b[np.ix_(np.arange(b.shape[0]),n1,n2)]
# or I,J=np.ix_(n1,n2);b[:,I,J]
Out[164]:
array([[[ 0],
[ 4]],
[[ 6],
[10]]])
In [165]: _.shape
Out[165]: (2, 2, 1)
Both cases use all rows of the 1st dimension. The ix one picks 2 'rows' of the 2nd dim, and 1 column of the last, resulting the (2,2,1) shape. The other picks b[:,0,0] and b[0,2,0] terms, resulting (2,2) shape.
(see my addenda as to why both are simply broadcasting).
These are all cases of advanced indexing, with boolean and numeric indexes. One can study the docs, or one can play around. Sometimes it's more fun to do the later. :)
(I knew that ix_ was good for adding the necessary np.newaxis to arrays so can be broadcast together, but didn't realize that worked with boolean arrays as well - it uses np.nonzero() to convert boolean to indices.)
Resolution
Underlying this is, I think, a confusion over 2 modes of indexing. which might called 'diagonal' and 'rectangular' (or element-by-element selection versus block selection). To illustrate look at a small 2d array
In [73]: M=np.arange(6).reshape(2,3)
In [74]: M
Out[74]:
array([[0, 1, 2],
[3, 4, 5]])
and 2 simple numeric indexes
In [75]: m1=np.arange(2); m2=np.arange(2)
They can be used 2 ways:
In [76]: M[m1,m2]
Out[76]: array([0, 4])
and
In [77]: M[m1[:,None],m2]
Out[77]:
array([[0, 1],
[3, 4]])
The 1st picks 2 points, the M[0,0] and M[1,1]. This kind of indexing lets us pick out the diagonals of an array.
The 2nd picks 2 rows and from that 2 columns. This is the kind of indexing the np.ix_ produces. The 1st picks 2 points, the M[0,0] and M[1,1]. This a 'rectangular' form of indexing.
Change m2 to 3 values:
In [78]: m2=np.arange(3)
In [79]: M[m1[:,None],m2] # returns a 2x3
Out[79]:
array([[0, 1, 2],
[3, 4, 5]])
In [80]: M[m1,m2] # produces an error
...
ValueError: shape mismatch: objects cannot be broadcast to a single shape
But if m2 has just one element, we don't get the broadcast error - because the size 1 dimension can be expanded during broadcasting:
In [81]: m2=np.arange(1)
In [82]: M[m1,m2]
Out[82]: array([0, 3])
Now change the index arrays to boolean, each matching the length of the respective dimensions, 2 and 3.
In [91]: m1=np.ones(2,bool); m2=np.ones(3,bool)
In [92]: M[m1,m2]
...
ValueError: shape mismatch: objects cannot be broadcast to a single shape
In [93]: m2[2]=False # m1 and m2 each have 2 True elements
In [94]: M[m1,m2]
Out[94]: array([0, 4])
In [95]: m2[0]=False # m2 has 1 True element
In [96]: M[m1,m2]
Out[96]: array([1, 4])
With 2 and 3 True terms we get an error, but with 2 and 2 or 2 and 1 it runs - just as though we'd used the indices of the True elements: np.nonzero(m2).
To apply this to your examples. In the first, m1 and m2 have 200 and 299 True elements. a[:,m1,m2,:] fails because of a mismatch in the number of True terms.
In the 2nd, they have 2 and 1 True terms, with nonzero indices of [0,2] and [0], which can be broadcast to [0,0]. So it runs.
http://docs.scipy.org/doc/numpy-1.10.0/reference/arrays.indexing.html
explains boolean array indexing in terms of nonzero and ix_.
Combining multiple Boolean indexing arrays or a Boolean with an integer indexing array can best be understood with the obj.nonzero() analogy. The function ix_ also supports boolean arrays and will work without any surprises.
Addenda
On further thought the distinction between 'diagonal' and 'block/rectangular' indexing might be more my mental construct that numpys. Underlying both is the concept of broadcasting.
Take the n1 and n2 booleans, and get their nonzero equivalents:
In [107]: n1
Out[107]: array([ True, False, True], dtype=bool)
In [108]: np.nonzero(n1)
Out[108]: (array([0, 2], dtype=int32),)
In [109]: n2
Out[109]: array([ True, False], dtype=bool)
In [110]: np.nonzero(n2)
Out[110]: (array([0], dtype=int32),)
Now try broadcasting in the 'diagonal' and 'rectangular' modes:
In [105]: np.broadcast_arrays(np.array([0,2]),np.array([0]))
Out[105]: [array([0, 2]),
array([0, 0])]
In [106]: np.broadcast_arrays(np.array([0,2])[:,None],np.array([0]))
Out[106]:
[array([[0],
[2]]),
array([[0],
[0]])]
One produces (2,) arrays, the other (2,1).
This might be a simple workaround:
a[:,m1,:,:][:,:,m2,:]
Starting with a 2D Numpy array I would like to create a 1D array in which each value corresponds to the minimum value of each row in the 2D array.
For example if
dog=[[1,2],[4,3],[6,7]]
then I would like to create an array from
'dog':[1,3,6]
This seems like it should be easy to do, but I'm not getting it so far.
In [54]: dog=[[1,2],[4,3],[6,7]]
In [55]: np.min(dog, axis=1)
Out[55]: array([1, 3, 6])
or, if dog is a NumPy array, you could call its min method:
In [57]: dog = np.array([[1,2],[4,3],[6,7]])
In [58]: dog.min(axis=1)
Out[58]: array([1, 3, 6])
Since dog.shape is (3,2), (for 3 rows, 2 columns), the axis=1 refers to the second dimension in the shape -- the one with 2 elements. Putting axis=1 in the call to dog.min tells NumPy to take the min over the axis=1 direction, thus eliminating the axis of length 2. The result is thus of shape (3,).
Without numpy:
dog=[[1,2],[4,3],[6,7]]
mins = [min(x) for x in dog]
I'm new to python and numpy in general. I read several tutorials and still so confused between the differences in dim, ranks, shape, aixes and dimensions. My mind seems to be stuck at the matrix representation. So if you say that A is a matrix that looks like this:
A =
1 2 3
4 5 6
then all I can think of is a 2x3 matrix (two rows and three columns). Here I understand that the shape is 2x3. But I really I am unable to go out side the thinking of a 2D matrices. I don't understand for example the dot() documentation when it says "For N dimensions it is a sum product over the last axis of a and the second-to-last of b". I'm so confused and unable to understand this. I don't understand like if V is a N:1 vector and M is N:N matrix, how dot(V,M) or dot(M,V) work and the difference between them.
Can anyone then please explain to me what is a N dimensional array, what's a shape, what's an axis and how does it relate to the documentation of the dot() function? It would be great if the explanation visualizes the ideas.
Dimensionality of NumPy arrays must be understood in the data structures sense, not the mathematical sense, i.e. it's the number of scalar indices you need to obtain a scalar value.(*)
E.g., this is a 3-d array:
>>> X = np.arange(24).reshape(2, 3, 4)
>>> X
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
Indexing once gives a 2-d array (matrix):
>>> X[0]
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
Indexing twice gives a 1-d array (vector), and indexing three times gives a scalar.
The rank of X is its number of dimensions:
>>> X.ndim
3
>>> np.rank(X)
3
Axis is roughly synonymous with dimension; it's used in broadcasting operations:
>>> X.sum(axis=0)
array([[12, 14, 16, 18],
[20, 22, 24, 26],
[28, 30, 32, 34]])
>>> X.sum(axis=1)
array([[12, 15, 18, 21],
[48, 51, 54, 57]])
>>> X.sum(axis=2)
array([[ 6, 22, 38],
[54, 70, 86]])
To be honest, I find this definition of "rank" confusing since it matches neither the name of the attribute ndim nor the linear algebra definition of rank.
Now regarding np.dot, what you have to understand is that there are three ways to represent a vector in NumPy: 1-d array, a column vector of shape (n, 1) or a row vector of shape (1, n). (Actually, there are more ways, e.g. as a (1, n, 1)-shaped array, but these are quite rare.) np.dot performs vector multiplication when both arguments are 1-d, matrix-vector multiplication when one argument is 1-d and the other is 2-d, and otherwise it performs a (generalized) matrix multiplication:
>>> A = np.random.randn(2, 3)
>>> v1d = np.random.randn(2)
>>> np.dot(v1d, A)
array([-0.29269547, -0.52215117, 0.478753 ])
>>> vrow = np.atleast_2d(v1d)
>>> np.dot(vrow, A)
array([[-0.29269547, -0.52215117, 0.478753 ]])
>>> vcol = vrow.T
>>> np.dot(vcol, A)
Traceback (most recent call last):
File "<ipython-input-36-98949c6de990>", line 1, in <module>
np.dot(vcol, A)
ValueError: matrices are not aligned
The rule "sum product over the last axis of a and the second-to-last of b" matches and generalizes the common definition of matrix multiplication.
(*) Arrays of dtype=object are a bit of an exception, since they treat any Python object as a scalar.
np.dot is a generalization of matrix multiplication.
In regular matrix multiplication, an (N,M)-shape matrix multiplied with a (M,P)-shaped matrix results in a (N,P)-shaped matrix. The resultant shape can be thought of as being formed by squashing the two shapes together ((N,M,M,P)) and then removing the middle numbers, M (to produce (N,P)). This is the property that np.dot preserves while generalizing to arrays of higher dimension.
When the docs say,
"For N dimensions it is a sum product over the last axis of a and the
second-to-last of b".
it is speaking to this point. An array of shape (u,v,M) dotted with an array of shape (w,x,y,M,z) would result in an array of shape (u,v,w,x,y,z).
Let's see how this rule looks when applied to
In [25]: V = np.arange(2); V
Out[25]: array([0, 1])
In [26]: M = np.arange(4).reshape(2,2); M
Out[26]:
array([[0, 1],
[2, 3]])
First, the easy part:
In [27]: np.dot(M, V)
Out[27]: array([1, 3])
There is no surprise here; this is just matrix-vector multiplication.
Now consider
In [28]: np.dot(V, M)
Out[28]: array([2, 3])
Look at the shape of V and M:
In [29]: V.shape
Out[29]: (2,)
In [30]: M.shape
Out[30]: (2, 2)
So np.dot(V,M) is like matrix multiplication of a (2,)-shaped matrix with a (2,2)-shaped matrix, which should result in a (2,)-shaped matrix.
The last (and only) axis of V and the second-to-last axis of M (aka the first axis of M) are multiplied and summed over, leaving only the last axis of M.
If you want to visualize this: np.dot(V, M) looks as though V has 1 row and 2 columns:
[[0, 1]] * [[0, 1],
[2, 3]]
and so, when V is multiplied by M, np.dot(V, M) equals
[[0*0 + 1*2], [2,
[0*1 + 1*3]] = 3]
However, I don't really recommend trying to visualize NumPy arrays this way -- at least I never do. I focus almost exclusively on the shape.
(2,) * (2,2)
\ /
\ /
(2,)
You just think about the "middle" axes being dotted, and disappearing from the resultant shape.
np.sum(arr, axis=0) tells NumPy to sum the elements in arr eliminating the 0th axis. If arr is 2-dimensional, the 0th axis are the rows. So for example, if arr looks like this:
In [1]: arr = np.arange(6).reshape(2,3); arr
Out[1]:
array([[0, 1, 2],
[3, 4, 5]])
then np.sum(arr, axis=0) will sum along the columns, thus eliminating the 0th axis (i.e. the rows).
In [2]: np.sum(arr, axis=0)
Out[2]: array([3, 5, 7])
The 3 is the result of 0+3, the 5 equals 1+4, the 7 equals 2+5.
Notice arr had shape (2,3), and after summing, the 0th axis is removed so the result is of shape (3,). The 0th axis had length 2, and each sum is composed of adding those 2 elements. The shape (2,3) "becomes" (3,). You can know the resultant shape in advance! This can help guide your thinking.
To test your understanding, consider np.sum(arr, axis=1). Now the 1-axis is removed. So the resultant shape will be (2,), and element in the result will be the sum of 3 values.
In [3]: np.sum(arr, axis=1)
Out[3]: array([ 3, 12])
The 3 equals 0+1+2, and the 12 equals 3+4+5.
So we see that summing an axis eliminates that axis from the result. This has bearing on np.dot, since the calculation performed by np.dot is a sum of products. Since np.dot performs a summing operation along certain axes, that axis is removed from the result. That is why applying np.dot to arrays of shape (2,) and (2,2) results in an array of shape (2,). The first 2 in both arrays is summed over, eliminating both, leaving only the second 2 in the second array.
In your case,
A is a 2D array, namely a matrix, with its shape being (2, 3). From docstring of numpy.matrix:
A matrix is a specialized 2-D array that retains its 2-D nature through operations.
numpy.rank return the number of dimensions of an array, which is quite different from the concept of rank in linear algebra, e.g. A is an array of dimension/rank 2.
np.dot(V, M), or V.dot(M) multiplies matrix V with M. Note that numpy.dot do the multiplication as far as possible. If V is N:1 and M is N:N, V.dot(M) would raise an ValueError.
e.g.:
In [125]: a
Out[125]:
array([[1],
[2]])
In [126]: b
Out[126]:
array([[2, 3],
[1, 2]])
In [127]: a.dot(b)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-127-9a1f5761fa9d> in <module>()
----> 1 a.dot(b)
ValueError: objects are not aligned
EDIT:
I don't understand the difference between Shape of (N,) and (N,1) and it relates to the dot() documentation.
V of shape (N,) implies an 1D array of length N, whilst shape (N, 1) implies a 2D array with N rows, 1 column:
In [2]: V = np.arange(2)
In [3]: V.shape
Out[3]: (2,)
In [4]: Q = V[:, np.newaxis]
In [5]: Q.shape
Out[5]: (2, 1)
In [6]: Q
Out[6]:
array([[0],
[1]])
As the docstring of np.dot says:
For 2-D arrays it is equivalent to matrix multiplication, and for 1-D
arrays to inner product of vectors (without complex conjugation).
It also performs vector-matrix multiplication if one of the parameters is a vector. Say V.shape==(2,); M.shape==(2,2):
In [17]: V
Out[17]: array([0, 1])
In [18]: M
Out[18]:
array([[2, 3],
[4, 5]])
In [19]: np.dot(V, M) #treats V as a 1*N 2D array
Out[19]: array([4, 5]) #note the result is a 1D array of shape (2,), not (1, 2)
In [20]: np.dot(M, V) #treats V as a N*1 2D array
Out[20]: array([3, 5]) #result is still a 1D array of shape (2,), not (2, 1)
In [21]: Q #a 2D array of shape (2, 1)
Out[21]:
array([[0],
[1]])
In [22]: np.dot(M, Q) #matrix multiplication
Out[22]:
array([[3], #gets a result of shape (2, 1)
[5]])