There is a simple function, which intends to accept a scalar parameter, but also works for a numpy matrix. Why does the function fun works for a matrix?
>>> import numpy as np
>>> def fun(a):
return 1.0 / a
>>> b = 2
>>> c = np.mat([1,2,3])
>>> c
matrix([[1, 2, 3]])
>>> fun(b)
0.5
>>> fun(c)
matrix([[ 1. , 0.5 , 0.33333333]])
>>> v_fun = np.vectorize(fun)
>>> v_fun(b)
array(0.5)
>>> v_fun(c)
matrix([[ 1. , 0.5 , 0.33333333]])
It seems like fun is vectorized somehow, because the explictly vectorized function v_fun behaves same on matrix c. But their get different outputs on scalar b. Could anybody explain it? Thanks.
What happens in the case of fun is called broadcasting.
General Broadcasting Rules
When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward. Two dimensions are compatible when
they are equal, or
one of them is 1
If these conditions are not met, a ValueError: frames are not aligned exception is thrown, indicating that the arrays have incompatible shapes. The size of the resulting array is the maximum size along each dimension of the input arrays.
fun already works for both scalars and arrays - because elementwise division is defined for both (their own methods). fun(b) does not involve numpy at all, that just a Python operation.
np.vectorize is meant to take a function that only works with scalars, and feed it elements from an array. In your example it first converts b into an array, np.array(b). For both c and this modified b, the result is an array of matching size. c is a 2d np.matrix, and result is the same. Notice that fun(b) is type array, not matrix.
This not a good example of using np.vectorize, nor an example of broadcasting. np.vectorize is a rather 'simple minded' function and doesn't handle scalars in a special way.
1/c or even b/c works because c, an array 'knows' about division. Similarly array multiplication and addition are defined: 1+c or 2*c.
I'm tempted to mark this as a duplicate of
Python function that handles scalar or arrays
Related
I'm having some trouble understanding the rules for array broadcasting in Numpy.
Obviously, if you perform element-wise multiplication on two arrays of the same dimensions and shape, everything is fine. Also, if you multiply a multi-dimensional array by a scalar it works. This I understand.
But if you have two N-dimensional arrays of different shapes, it's unclear to me exactly what the broadcasting rules are. This documentation/tutorial explains that: In order to broadcast, the size of the trailing axes for both arrays in an operation must either be the same size or one of them must be one.
Okay, so I assume by trailing axis they are referring to the N in a M x N array. So, that means if I attempt to multiply two 2D arrays (matrices) with equal number of columns, it should work? Except it doesn't...
>>> from numpy import *
>>> A = array([[1,2],[3,4]])
>>> B = array([[2,3],[4,6],[6,9],[8,12]])
>>> print(A)
[[1 2]
[3 4]]
>>> print(B)
[[ 2 3]
[ 4 6]
[ 6 9]
[ 8 12]]
>>>
>>> A * B
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: shape mismatch: objects cannot be broadcast to a single shape
Since both A and B have two columns, I would have thought this would work. So, I'm probably misunderstanding something here about the term "trailing axis", and how it applies to N-dimensional arrays.
Can someone explain why my example doesn't work, and what is meant by "trailing axis"?
Well, the meaning of trailing axes is explained on the linked documentation page.
If you have two arrays with different dimensions number, say one 1x2x3 and other 2x3, then you compare only the trailing common dimensions, in this case 2x3. But if both your arrays are two-dimensional, then their corresponding sizes have to be either equal or one of them has to be 1. Dimensions along which the array has size 1 are called singular, and the array can be broadcasted along them.
In your case you have a 2x2 and 4x2 and 4 != 2 and neither 4 or 2 equals 1, so this doesn't work.
From http://cs231n.github.io/python-numpy-tutorial/#numpy-broadcasting:
Broadcasting two arrays together follows these rules:
If the arrays do not have the same rank, prepend the shape of the lower rank array with 1s until both shapes have the same length.
The two arrays are said to be compatible in a dimension if they have the same size in the dimension, or if one of the arrays has size 1 in that dimension.
The arrays can be broadcast together if they are compatible in all dimensions.
After broadcasting, each array behaves as if it had shape equal to the elementwise maximum of shapes of the two input arrays.
In any dimension where one array had size 1 and the other array had size greater than 1, the first array behaves as if it were copied along that dimension
If this explanation does not make sense, try reading the explanation from the documentation or this explanation.
we should consider two points about broadcasting. first: what is possible. second: how much of the possible things is done by numpy.
I know it might look a bit confusing, but I will make it clear by some example.
lets start from the zero level.
suppose we have two matrices. first matrix has three dimensions (named A) and the second has five (named B). numpy tries to match last/trailing dimensions. so numpy does not care about the first two dimensions of B. then numpy compares those trailing dimensions with each other. and if and only if they be equal or one of them be 1, numpy says "O.K. you two match". and if it these conditions don't satisfy, numpy would "sorry...its not my job!".
But I know that you may say comparison was better to be done in way that can handle when they are devisable(4 and 2 / 9 and 3). you might say it could be replicated/broadcasted by a whole number(2/3 in out example). and i am agree with you. and this is the reason I started my discussion with a distinction between what is possible and what is the capability of numpy.
I often end up implementing routines that operate on 1-D arrays, and then generalize them so that, if they are called on matrices, they treat each row independently. For example, suppose we want a function that subtracts the mean of a vector from it, and then does that on every row of a matrix, if its input has more than one dimensions. This can be implemented as follows:
import numpy as np
def _sub_mean(a):
""" Subtract the mean from vector elements """
assert isinstance(a, np.ndarray) and a.ndim == 1
return a - np.mean(a)
def sub_mean(a):
""" Subtract the mean
If `a` is a 1D array it returns a 1D array (obtained by subtracting the
mean of `a` from each of its elements). If `a` has more than one dimension
each row is treated separately
"""
a = np.asanyarray(a)
if a.ndim <= 1:
a = np.atleast_1d(a)
return np.squeeze(_sub_mean(a))
retval = np.empty(a.shape)
for idx in np.ndindex(*a.shape[:-1]):
retval[idx] = _sub_mean(a[idx])
return retval
Here are a few examples of the output of sub_mean:
>>> sub_mean(5)
array(0.)
>>> sub_mean([1, 2])
array([-0.5, 0.5])
>>> sub_mean([[1, 2], [4, 6]])
array([[-0.5, 0.5],
[-1. , 1. ]])
>>>
Note that the 'core' computation is taking place in the private function _sub_mean. Indeed the same code in sub_mean can be used to generalize any function that operates on 1D arrays to one that operates on arbitrary number of dimensions, by replacing _sub_mean. Also one can think of further generalization, for example adding an axis argument that specifies on which axis the function operates and/or possibility of operating on the flattened input array.
I was wondering if NumPy already provides a decorator to generalize functions that operate on vectors to those that operate on more-than-one-dimensional arrays? I.e., if I can replace the above code by:
import numpy as np
#np.some_decorator
def sub_mean(a):
""" Subtract the mean
If `a` is a 1D array it returns a 1D array (obtained by subtracting the
mean of `a` from each of its elements). If `a` has more than one dimension
each row is treated separately """
assert isinstance(a, np.ndarray) and a.ndim == 1
return a - np.mean(a)
(and, obviously, get the same outputs.)
UPDATE: I ended up writing the following decorator:
class _Expand:
def __init__(self, func1d):
functools.update_wrapper(self, func1d)
self._func1d = func1d
def __call__(self, arr, *args, **kwargs):
arr = np.asanyarray(arr)
axis = kwargs.pop('axis', -1)
return np.apply_along_axis(self._func1d, axis, arr, *args, **kwargs)
which would then enable me to write sub_mean (or any other complex function that operates on 1D arrays) as:
#_Expand
def sub_mean(a):
""" Subtract the mean
If `a` is a 1D array it returns a 1D array (obtained by subtracting the
mean of `a` from each of its elements). If `a` has more than one dimension
each row is treated separately """
assert isinstance(a, np.ndarray) and a.ndim == 1
return a - np.mean(a)
(Note that _Expand allows to pick the axis along which the operation is performed --- that's slightly more generic than what I needed.)
Yet, I would still be curious to know if such a decorator is already implemented in NumPy?
This is a fairly common question, but has no perfect answer. The real answer is that you must end up tailoring each function individually.
Assuming you want to stick to pure numpy (and vanilla python) rather than using numba or cython, here are some considerations to keep in mind:
np.apply_along_axis is implemented mostly as a python loop. It doesn't offer any real advantage besides a more compact notation.
Almost all numpy functions accept an axis argument. As of numpy v1.15, you can supply a tuple multiple simultaneous axes to ufuncs, and to many of the functions built on top of them.
Ufuncs additionally have methods that let you apply them at particular locations and combinations.
Reduction functions, such as np.mean often have a keepdims argument. This allows you to combine reduction operations with the original array. Functions that don't support keepdims can be mitigated with manual axis insertion using, for example, np.expand_dims.
There are functions that perform operations specifically on indices; most of their names start with arg*: np.nonzero, np.argmin, np.argmax, np.argpartition, np.argsort, etc.
Taken together, you can vectorize almost any function that can be written for a single dimension. Sometimes the result requires extra cleverness, and sometimes it is outright impossible, but generally it is fairly simple. One example I can think of that is non-trivial is anything involving run-length encoding. The issue there is that you end up with a ragged array in some of the steps, requiring tools beyond numpy.
Your concrete example can be written as follows:
def sub_mean(a, axis):
a = np.array(a, copy=False, subok=True)
return a - np.mean(a, axis=axis, keepdims=True)
I need it for 2d arrays but would also be interested in higher dimensions.
If I have two numpy 2DArrays lets say A and B and a function f which takes 2 vectors and puts out a scalar. Performance is important and numpy array operations are really fast so how do I get the matrix:
(f(a1,b1),f(a1, b2)...)
(f(a2,b1), f(a2,b2)...)
(... )
a(i) and b(i) being 1d arrays/vectors
By applying something like f(A,B) and without using any loops.
With some methods I can use the axis=1 but how do I define my own function like that or is there an other way?
The easiest, and most efficient, solution is to use a for loop. np.vectorize won't help you here, because it passes scalars to a function.
Assuming your arrays have two columns:
for a,b in zip(A,B):
print(f(a[0],b[0]))
print(f(a[1],b[1]))
would print the values of f that you would want to return.
Is there an easier way to get the sum of all values (assuming they are all numbers) in an ndarray :
import numpy as np
m = np.array([[1,2],[3,4]])
result = 0
(dim0,dim1) = m.shape
for i in range(dim0):
for j in range(dim1):
result += m[i,j]
print result
The above code seems somewhat verbose for a straightforward mathematical operation.
Thanks!
Just use numpy.sum():
result = np.sum(matrix)
or equivalently, the .sum() method of the array:
result = matrix.sum()
By default this sums over all elements in the array - if you want to sum over a particular axis, you should pass the axis argument as well, e.g. matrix.sum(0) to sum over the first axis.
As a side note your "matrix" is actually a numpy.ndarray, not a numpy.matrix - they are different classes that behave slightly differently, so it's best to avoid confusing the two.
Yes, just use the sum method:
result = m.sum()
For example,
In [17]: m = np.array([[1,2],[3,4]])
In [18]: m.sum()
Out[18]: 10
By the way, NumPy has a matrix class which is different than "regular" numpy arrays. So calling a regular ndarray matrix causes some cognitive dissonance. To help others understand your code, you may want to change the name matrix to something else.
Given the following code, I expect the last two lines to behave the same, however the don't.
import numpy as np
C = np.matrix(np.zeros((4,4)))
C[0, 0:2] = np.matrix([[1, 2]]) # Works as expected.
C[0, [0,1]] = np.matrix([[1, 2]]) # Throws an "array is not broadcastable to correct shape" error.
When using an ndarray instead, things work as expected (adjusting the right-hand-side of the assignment to a one-dimensional ndarray):
D = np.zeros((4,4))
D[0, 0:2] = np.array([1, 2]) # Works as expected.
D[0, [0,1]] = np.array([1, 2]) # Works too.
And to make things even weirder, if one is only indexing the matrix C (as opposed to assigning to it), it seems using slice indices or a list just return the same:
C[0, 0:2] # => matrix([[ 1., 2.]])
C[0, [0, 1]] # => matrix([[ 1., 2.]])
The question is, why is the behavior of the two approaches in assignment different? What am I missing?
(Edit: typo)
It appears to be a bug in numpy: http://projects.scipy.org/numpy/ticket/803 . The solution is to assign an ordinary list or numpy array instead of assigning a matrix to the selected elements.
Edit: Had to realize that while what I write is true, the fact that D[0,0:2] = ... is different from D[0,[0,1]] = ... (so for arrays) is maybe a real inconsistency (and related).
Maybe an explenation why this happens as far as I see. Check this:
D[0,[0,1]] = np.array([[1,2]])
Gives the same error. The thing is that internally the slicing operation takes place before the matrix shape is "fixed" to 2D again, which, since matrix is a subclass occurs whenver a new view is created, but here no view is created as its unnecessary normally!
This means that when you are setting elements like this, it always behaves like:
C.A[0,[0,1]] = matrix([[1,2]]) # Note the C.A giving normal array view of C.
Which fails, because the matrix is 2D, but C.A[0,[0,1]] is 1D (since it is not "fixed" to be at least 2D by the matrix object), in this case one could say that since its just removing a 1 dimension axis from the right hand side numpy could maybe tolerate it, but as long as it doesn't it would require the matrix object to make a full custom sets of in place/assignment operators which would not be very elegent as well probably.
But maybe the use of C.A, etc. can help getting around this inconvenience. On a general note however, in numpy it is better to always use base class arrays unless you are doing a lot of matrix multiplications, etc. (in which case if it is limited to one part of the program, its likely better to just view your arrays as matrixes before it but work with arrays in the rest)