I have these numpy objects:
>>> x = np.matrix([[1],[2],[3]])
>>> i = ([1,2],0)
>>> y = np.matrix([[4],[5]])
When I do x[i] I get as I expect:
>>> x[i]
matrix([[2],
[3]])
However, when I try to assign to x[i], I get funky behavior:
>>> x[i] = y
ValueError: array is not broadcastable to correct shape
>>> y.shape
(2, 1)
>>> x[i].shape
(2, 1)
I have figured out some workarounds, but they're workarounds and not what I want:
>>> x[1:,0] = y
>>> x
matrix([[1],
[4],
[5]])
>>> x = np.array([[1],[2],[3]]); y = np.array(y)
>>> x[i] = y[:,0]
>>> x
array([[1],
[4],
[5]])
The second workaround is not acceptable because y could have a more general shape than a column vector.
Don't use numpy.matrix. It's terrible. It causes so many weird incompatibilities and has so many inconsistencies, including this one.
Use numpy.array. With arrays, x[i] is one-dimensional, and assigning a one-dimensional y of equal shape to x[i] will work fine.
import numpy
x = numpy.array([[1], [2], [3]])
y = numpy.array([4, 5])
i = ([1, 2], 0)
print(x[i].shape)
print(y.shape)
x[i] = y
print(repr(x))
Output:
(2,)
(2,)
array([[1],
[4],
[5]])
If you want to do matrix multiplication, use the # operator, or the dot method if you're on too old a Python version or too old a NumPy version to have #.
Indexing with a simpe [1,2] works:
In [71]: x
Out[71]:
matrix([[1],
[2],
[3]])
In [72]: y
Out[72]:
matrix([[4],
[5]])
In [73]: x[[1,2],:]
Out[73]:
matrix([[2],
[3]])
In [74]: x[[1,2],:] = y
Trying to use a tuple has problems:
In [78]: i = ([1,2],0)
In [79]: x[i] = y
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-79-699d63749224> in <module>()
----> 1 x[i] = y
ValueError: shape mismatch: value array of shape (2,1) could not be broadcast to indexing result of shape (2,)
Changing i to i = ([1,2],slice(None)) matches my first case.
Using 0 instead of :' is also a problem:
In [82]: x[[1,2],0] = y
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-82-8b683eb93954> in <module>()
----> 1 x[[1,2],0] = y
ValueError: shape mismatch: value array of shape (2,1) could not be broadcast to indexing result of shape (2,)
I suspect it is trying to do the assgnment to a (2,) as a numpy array, before converting the results back to matrix.
It's trying to the equivalent of:
In [83]: x.A[[1,2],0] = y
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-83-b2ccc78af912> in <module>()
----> 1 x.A[[1,2],0] = y
ValueError: shape mismatch: value array of shape (2,1) could not be broadcast to indexing result of shape (2,)
where as it needs a flattened version of y:
In [84]: x.A[[1,2],0] = y.A1 # max `y` a 1d array
x[[1,2],0] = y.A1 # also works
Earlier today there was a similar problem: Copy NumPy matrix into NumPy array
The convenience of using * for matrix multiplication rarely outweighs the clumkiness of restricting dimensions to 2. np.dot works just as well. And arrays also provide tensordot, einsum and matmul (and the # operator).
np.matrix is a subclass of np.ndarray. It inherits methods. Often it performs the action with the ndarray method, and cleans up the dimesions, either forcing 1d results to 2d, or scalars.
Here's the code for the * matrix multiplication:
Signature: x.__mul__(other)
Source:
def __mul__(self, other):
if isinstance(other, (N.ndarray, list, tuple)) :
# This promotes 1-D vectors to row vectors
return N.dot(self, asmatrix(other))
if isscalar(other) or not hasattr(other, '__rmul__') :
return N.dot(self, other)
return NotImplemented
Related
This has happened to me several times with different equations, and right now, I struggle with it again. I am sure others have the same problem and look for a memorable strategy to shape arrays and vectors so that non-trivial NumPy computations go through without fiddling and poking.
This is the equation I want to implement: , where k is a scalar, and n, a, R and R0 are 3D vectors. R0 is a NumPy array holding four 3D vectors.
Now I get stuck in my workflow because I try to find a way to pierce together the NumPy calculation. I want to use NumPy's awesome parallelization for math and not break up the four 3D vectors and do a for-loop. I am not good enough at NumPy to "see" the shapes of intermediate result arrays. And so I end up with try-and-error, which gives ugly results and is unsatisfying.
Please don't just give a working solution but explain why you did what you did.
Here is my current code - which does not work as it should (the transposing of random arrays should not be there).
import numpy as np
def main():
r_0 = np.array([[.265, 0, .382],
[0, .712, .764],
[0, .712, 0],
[.752, .712, .382]])
b_measured = np.array([[1.64712096, 4.87716371, -0.77060129],
[1.55980676, 4.93977942, -0.7133636],
[1.40883668, 4.96624651, -0.71742247],
[1.6531721, 5.02004066, -0.72243437]])
n = np.array([0, 0, 1])
a = np.array([.2, 0, 0])
r = np.array([0, 2, 0])
r_relativ = (r - r_0)
r_magnitude = np.linalg.norm(r_relativ, axis=1)
error = -np.cross(a - 2 * (r_relativ.T * np.matmul(r_relativ, a)).T, n,
axisa=1) / r_magnitude ** 4 - b_measured
print(error)
if '__name__' == '__main__':
main()
This is a casting problem from numpy. In this line:
r_magnitude = np.linalg.norm(r_relativ, axis=1)
you are getting a (4,) shape, which is a vector, not an array. You can check that by adding a
print(r_magnitude.shape)
This generates difficulties with numpy casting, as numpy doesn't quite know how to cast this second undefined dimension. That is why you probably get the error
ValueError: operands could not be broadcast together with shapes (4,3) (4,)
Which means, you are trying to broadcast your vector r_magnitude (shape = (4,)) into a (4,3) shaped numpy array. To understand why this happens, you can check this link to Tutorialspoint. Towards the end they explain it.
What I believe is the standard, is to use the keepdims kwarg for basically every numpy method.
r_magnitude = np.linalg.norm(r_relativ, axis=1, keepdims=True)
This will make the operation keep the same number of dimentions as the original array (2 in this case), that is, r_magnitude's shape is now (4,1), making the broadcasting possible
Keeping track of shapes is a large part of numpy coding (and before that MATLAB).
First, I like to have a clear idea of the shapes of the inputs:
In [1]: r_0 = np.array([[.265, 0, .382],
...: [0, .712, .764],
...: [0, .712, 0],
...: [.752, .712, .382]])
...:
...: b_measured = np.array([[1.64712096, 4.87716371, -0.77060129],
...: [1.55980676, 4.93977942, -0.7133636],
...: [1.40883668, 4.96624651, -0.71742247],
...: [1.6531721, 5.02004066, -0.72243437]])
...:
...: n = np.array([0, 0, 1])
...: a = np.array([.2, 0, 0])
...: r = np.array([0, 2, 0])
...:
In [2]: r_0.shape
Out[2]: (4, 3)
In [3]: b_measured.shape
Out[3]: (4, 3)
The rest are (3,)
While one can use the debugger, I prefer to test code in an interactive session like ipython.
In [4]: r_relativ = (r - r_0) # this was a (4,3) broadcasting with (3,)
In [5]: r_relativ.shape
Out[5]: (4, 3)
norm with axis, is like sum, mean etc, a reduction function, removing a dimension:
In [6]: r_magnitude = np.linalg.norm(r_relativ, axis=1)
In [7]: r_magnitude.shape
Out[7]: (4,)
though it is possible retain that dimension as a broadcastable size 1:
In [8]: np.linalg.norm(r_relativ, axis=1, keepdims=True).shape
Out[8]: (4, 1)
Your next line is complex, and the error doesn't identify the operator:
In [9]: error = -np.cross(a - 2 * (r_relativ.T * np.matmul(r_relativ, a)).T, n,
...: axisa=1) / r_magnitude ** 4 - b_measured
Traceback (most recent call last):
File "<ipython-input-9-8d7646046c43>", line 1, in <module>
error = -np.cross(a - 2 * (r_relativ.T * np.matmul(r_relativ, a)).T, n,
ValueError: operands could not be broadcast together with shapes (4,3) (4,)
Looking at the individual pieces:
In [10]: np.matmul(r_relativ, a).shape
Out[10]: (4,)
This is a (4,3) with (3,) producing a (4,). (Also matmul issues different error messages).
In [11]: (r_relativ.T * np.matmul(r_relativ, a)).T.shape
Out[11]: (4, 3)
The cross argument looks ok. cross itself runs. I don't use it enough to remember the exact axis arguments, though the basic operation is for a pair of (N,3) arrays
In [13]: np.cross(a - 2 * (r_relativ.T * np.matmul(r_relativ, a)).T, n,
...: axisa=1).shape
Out[13]: (4, 3)
Finally, here's the error:
In [15]: np.cross(a - 2 * (r_relativ.T * np.matmul(r_relativ, a)).T, n,
...: axisa=1)/r_magnitude**4
Traceback (most recent call last):
File "<ipython-input-15-c4bed1fd9807>", line 1, in <module>
np.cross(a - 2 * (r_relativ.T * np.matmul(r_relativ, a)).T, n,
ValueError: operands could not be broadcast together with shapes (4,3) (4,)
Why? As noted the cross produces (4,3). But r_magnitude is (4,). It should be (4,1) to work with the (4,3).
In sum, I keep the 2 broadcasting rules upper most - it can add leading dimensions, and size 1 dimensions can be scaled to match.
The other issue is keeping track of how reductions like norm change the dimensions. keepdims can be a big help here, though it's always possible to use None to add trailing dimensions.
I don't think there's any short cut to being pedantic about the dimensions. Details matter.
With the (4,) correction to (4,1), error runs:
In [16]: r_magnitude = np.linalg.norm(r_relativ, axis=1, keepdims=True)
In [17]: error = -np.cross(a - 2 * (r_relativ.T * np.matmul(r_relativ, a)).T, n,
...:
...: axisa=1) / r_magnitude ** 4 - b_measured
In [18]: error.shape
Out[18]: (4, 3)
focusing on the transposes.
a - 2 * (r_relativ.T * np.matmul(r_relativ, a)).T
the matmul produces (4,). r_relativ is (4,3); change that to (3,4), and it multiplies with the (4,). Transpose back to (4,3) so it can add to (3,), and work with axis1=1 (default).
I think these should work as well (not tested)
a - 2 * r_relativ * np.matmul(r_relativ,a)[:,None]
(3,) (4,3) * (4,1) => (4,3)
a - 2 * r_relativ * np.matmul(r_relativ, a[:,None])
(4,3) # (3,1)=>(4,1)
One can use numpy.where for selecting values from two arrays depending on a condition:
import numpy
a = numpy.random.rand(5)
b = numpy.random.rand(5)
c = numpy.where(a > 0.5, a, b) # okay
If the array has more dimensions, however, this does not work anymore:
import numpy
a = numpy.random.rand(5, 2)
b = numpy.random.rand(5, 2)
c = numpy.where(a[:, 0] > 0.5, a, b) # !
Traceback (most recent call last):
File "p.py", line 10, in <module>
c = numpy.where(a[:, 0] > 0.5, a, b) # okay
File "<__array_function__ internals>", line 6, in where
ValueError: operands could not be broadcast together with shapes (5,) (5,2) (5,2)
I would have expected a numpy array of shape (5,2).
What's the issue here? How to work around it?
Remember that broadcasting in numpy only works from the right, so while (5,) shaped arrays can broadcast with (2,5) shaped arrays they can't broadcast with (5,2) shaped arrays. to broadcast with a (5,2) shaped array you need to maintain the second dimension so that the shape is (5,1) (anything can broadcast with 1)
Thus, you need to maintain the second dimension when indexing it (otherwise it removes the indexed dimension when only one value exists). You can do this by putting the index in a one-element list:
a = numpy.random.rand(5, 2)
b = numpy.random.rand(5, 2)
c = numpy.where(a[:, [0]] > 0.5, a, b) # works
You can use c = numpy.where(a > 0.5, a, b)
however if you want to use only the first column of a then you need to consider the shape of the output.
let's first see what is the shape of this operation
(a[:, 0] > 0.5).shape # outputs (5,)
it's one dimensional
while the shape of a and b is (5, 2)
it's two dimensional and hence you can't broadcast this
the solution is to reshape the mask operation to be of shape (5, 1)
your code should look like this
a = numpy.random.rand(5, 2)
b = numpy.random.rand(5, 2)
c = numpy.where((a[:, 0] > 0.5).reshape(-1, 1), a, b) # !
You can try:
import numpy
a = numpy.random.rand(5, 2)
b = numpy.random.rand(5, 2)
c = numpy.where(a > 0.5, a, b)
instead of: c = np.where(a>0.5,a,b)
you can use: c = np.array([a,b])[a>0.5]
which works for multidimensional arrays if a and b have the same shape.
We all know that dot product between vectors must return a scalar:
import numpy as np
a = np.array([1,2,3])
b = np.array([3,4,5])
print(a.shape) # (3,)
print(b.shape) # (3,)
a.dot(b) # 26
b.dot(a) # 26
perfect. BUT WHY if we use a "real" (take a look at Difference between numpy.array shape (R, 1) and (R,)) row vector or column vector the numpy dot product returns error on dimension ?
arow = np.array([[1,2,3]])
brow = np.array([[3,4,5]])
print(arow.shape) # (1,3)
print(brow.shape) # (1,3)
arow.dot(brow) # ERROR
brow.dot(arow) # ERROR
acol = np.array([[1,2,3]]).reshape(3,1)
bcol = np.array([[3,4,5]]).reshape(3,1)
print(acol.shape) # (3,1)
print(bcol.shape) # (3,1)
acol.dot(bcol) # ERROR
bcol.dot(acol) # ERROR
Because by explicitly adding a second dimension, you are no longer working with vectors but with two dimensional matrices. When taking the dot product of matrices, the inner dimensions of the product must match.
You therefore need to transpose one of your matrices. Which one you transpose will determine the meaning and shape of the result.
A 1x3 times a 3x1 matrix will result in a 1x1 matrix (i.e., a scalar). This is the inner product. A 3x1 times a 1x3 matrix will result in a 3x3 outer product.
You can also use the # operator, which is actually matrix multiplication.
In this case, as well as in dot product, you need to be aware to the matrices sizes (ndarray should always be dim compatible ), but it's more readable:
>>> a = np.array([1,2,3])
>>> a.shape
(3,)
>>> b= np.array([[1,2,3]])
>>> b.shape
(1, 3)
>>> a#b
Traceback (most recent call last):
File "<input>", line 1, in <module>
ValueError: shapes (3,) and (1,3) not aligned: 3 (dim 0) != 1 (dim 0)
>>> a#b.T
array([14])
You can also do like this
import numpy as npy
Vector1 = npy.array([0,2,3])
Vector2 = npy.array([3,5,1])
print("Dot Product of", Vector1, "and", Vector2,)
def DotProduct(a,b):
NetValue = 0
for i in range(len(a)):
NetValue += a[i]*b[i]
return NetValue
ans = DotProduct(Vector1,Vector2)
print("The answer is =",ans)
In Python, I have a list and a numpy array.
I would like to multiply the array by the list in such a way that I get an array where the 3rd dimension represents the input array multiplied by each element of the list. Therefore:
in_list = [2,4,6]
in_array = np.random.rand(5,5)
result = ...
np.shape(result) ---> (3,5,5)
where (0,:,:) is the input array multiplied by the first element of the list (2);
(1,:,:) is the input array multiplied by the second element of the list (4), etc.
I have a feeling this question will be answered by broadcasting, but I'm not sure how to go around doing this.
You want np.multiply.outer. The outer method is defined for any NumPy "ufunc", including multiplication. Here's a demonstration:
In [1]: import numpy as np
In [2]: in_list = [2, 4, 6]
In [3]: in_array = np.random.rand(5, 5)
In [4]: result = np.multiply.outer(in_list, in_array)
In [5]: result.shape
Out[5]: (3, 5, 5)
In [6]: (result[1, :, :] == in_list[1] * in_array).all()
Out[6]: True
As you suggest, broadcasting gives an alternative solution: if you convert in_list to a 1d NumPy array of length 3, you can then reshape to an array of shape (3, 1, 1), and then a multiplication with in_array will broadcast appropriately:
In [9]: result2 = np.array(in_list)[:, None, None] * in_array
In [10]: result2.shape
Out[10]: (3, 5, 5)
In [11]: (result2[1, :, :] == in_list[1] * in_array).all()
Out[11]: True
I have a question regarding the conversion between (N,) dimension arrays and (N,1) dimension arrays. For example, y is (2,) dimension.
A=np.array([[1,2],[3,4]])
x=np.array([1,2])
y=np.dot(A,x)
y.shape
Out[6]: (2,)
But the following will show y2 to be (2,1) dimension.
x2=x[:,np.newaxis]
y2=np.dot(A,x2)
y2.shape
Out[14]: (2, 1)
What would be the most efficient way of converting y2 back to y without copying?
Thanks,
Tom
reshape works for this
a = np.arange(3) # a.shape = (3,)
b = a.reshape((3,1)) # b.shape = (3,1)
b2 = a.reshape((-1,1)) # b2.shape = (3,1)
c = b.reshape((3,)) # c.shape = (3,)
c2 = b.reshape((-1,)) # c2.shape = (3,)
note also that reshape doesn't copy the data unless it needs to for the new shape (which it doesn't need to do here):
a.__array_interface__['data'] # (22356720, False)
b.__array_interface__['data'] # (22356720, False)
c.__array_interface__['data'] # (22356720, False)
Use numpy.squeeze:
>>> x = np.array([[[0], [1], [2]]])
>>> x.shape
(1, 3, 1)
>>> np.squeeze(x).shape
(3,)
>>> np.squeeze(x, axis=(2,)).shape
(1, 3)
Slice along the dimension you want, as in the example below. To go in the reverse direction, you can use None as the slice for any dimension that should be treated as a singleton dimension, but which is needed to make shapes work.
In [786]: yy = np.asarray([[11],[7]])
In [787]: yy
Out[787]:
array([[11],
[7]])
In [788]: yy.shape
Out[788]: (2, 1)
In [789]: yy[:,0]
Out[789]: array([11, 7])
In [790]: yy[:,0].shape
Out[790]: (2,)
In [791]: y1 = yy[:,0]
In [792]: y1.shape
Out[792]: (2,)
In [793]: y1[:,None]
Out[793]:
array([[11],
[7]])
In [794]: y1[:,None].shape
Out[794]: (2, 1)
Alternatively, you can use reshape:
In [795]: yy.reshape((2,))
Out[795]: array([11, 7])
the opposite translation can be made by:
np.atleast_2d(y).T
Another option in your toolbox could be ravel:
>>> y2.shape
(2, 1)
>>> y_ = y2.ravel()
>>> y_.shape
(2,)
Again, a copy is made only if needed, but this is not the case:
>>> y2.__array_interface__["data"]
(2700295136768, False)
>>> y_.__array_interface__["data"]
(2700295136768, False)
For further details, you can take a look at this answer.