In my code, I multiply two matrices:
c = b*a
Where a outputs as
array([array([-0.08358731, 0.07145386, 0.1052811 , -0.05362566]),
array([-0.05335939, -0.03136824, -0.01260714, 0.11532605]),
array([-0.09164538, 0.02280118, -0.00290509, 0.09415849])], dtype=object)
and b outputs as
array([ 0.60660017, 0.54703557, 0.69928535, 0.70157223])
...That should work right (where values of b is multiplied by each value of each row in a)?
Instead, I get
ValueError: operands could not be broadcast together with shapes (3) (4)
But then when I try it in a separate python console, it works great.
(bare in mind I've set array = np.array)
>>> aa = array([array([-0.12799382, 0.07758469, -0.02968546, -0.01811048]),
array([-0.00465869, -0.00483031, -0.00591955, -0.00386022]),
array([-0.02036786, 0.0078658 , 0.09493727, -0.01790333])], dtype=object)
>>> bb = array([ 0.16650179, 0.74140229, 0.60859776, 0.37505098])
>>> aa * bb
array([[-0.021311200138937801, 0.057521466834940096, -0.0180665044605696,
-0.0067923532722703999],
[-0.00077568022405510005, -0.0035812028954099002,
-0.0036026248702079999, -0.0014477792940156],
[-0.0033912851484694004, 0.0058317221326819992, 0.0577786098625152,
-0.0067146614617633995]], dtype=object)
The fact it works here really confuses me...
Your first array has only 1 dimension and 3 "object" elements while your second array has 1 dimension and 4 float elements. numpy uses element-wise arithmetic operations and there is just no way it can do that with one 3-item array and a 4-item array therefore the Exception.
>>> x = np.empty(3, dtype=object)
>>> x[0] = np.array([-0.08358731, 0.07145386, 0.1052811 , -0.05362566])
>>> x[1] = np.array([-0.05335939, -0.03136824, -0.01260714, 0.11532605])
>>> x[2] = np.array([-0.09164538, 0.02280118, -0.00290509, 0.09415849])
>>> x.shape
(3, )
The example above is an awful way of creating a numpy.array and should be avoided!
The difference to your second example is that it doesn't have numpy-arrays inside an array, it creates a multidimensional (3x4) array:
>>> x_new = np.array(list(x))
>>> x_new # no nested arrays!
array([[-0.12799382, 0.07758469, -0.02968546, -0.01811048],
[-0.00465869, -0.00483031, -0.00591955, -0.00386022],
[-0.02036786, 0.0078658, 0.09493727, -0.01790333]], dtype=object)
>>> x_new.shape
(3, 4)
That the multiplication operation works with the new array (x_new or your aa) is because numpy broadcasts the arrays. Here every row will be multiplied by one of your items in the second array.
Your original a and the copy aa have different shapes. Do a a.shape and aa.shape. The problem is with how object arrays are created. np.array tries to create as high a dimensional object as it can.
a is (3,) array, a 1d array containing 3 arrays.
aa is (3,4) array, a 2d array contain numbers as objects (not floats).
To construct a I have to take a convoluted route:
In [659]: a=np.empty((3,), object)
In [660]: a[0]=np.array([-0.08358731, 0.07145386, 0.1052811 , -0.05362566])
...: a[1]=np.array([-0.05335939, -0.03136824, -0.01260714, 0.11532605])
...: a[2]=np.array([-0.09164538, 0.02280118, -0.00290509, 0.09415849])
...:
In [661]: a
Out[661]:
array([array([-0.08358731, 0.07145386, 0.1052811 , -0.05362566]),
array([-0.05335939, -0.03136824, -0.01260714, 0.11532605]),
array([-0.09164538, 0.02280118, -0.00290509, 0.09415849])], dtype=object)
In [662]: a.shape
Out[662]: (3,)
I can multiply those 3 elements with another 3 element array (that doesn't always work with object arrays, but here the elements implement *.)
In [663]: a*np.array([0,1,2])
Out[663]:
array([array([-0., 0., 0., -0.]),
array([-0.05335939, -0.03136824, -0.01260714, 0.11532605]),
array([-0.18329076, 0.04560236, -0.00581018, 0.18831698])], dtype=object)
But if I copy-n-paste as you did I get
In [665]: aa = array([array([-0.12799382, 0.07758469, -0.02968546, -0.01811048]
...: ),
...: array([-0.00465869, -0.00483031, -0.00591955, -0.00386022])
...: ,
...: array([-0.02036786, 0.0078658 , 0.09493727, -0.01790333])
...: ], dtype=object)
In [666]: aa.shape
Out[666]: (3, 4)
Now that (3,4) can multiply a (4,) array.
vstack can convert the (3,) object array into a (3,4) array of floats:
In [667]: a3=np.vstack(a)
In [668]: a3.shape
Out[668]: (3, 4)
In [669]: a3.dtype
Out[669]: dtype('float64')
In [670]: a3
Out[670]:
array([[-0.08358731, 0.07145386, 0.1052811 , -0.05362566],
[-0.05335939, -0.03136824, -0.01260714, 0.11532605],
[-0.09164538, 0.02280118, -0.00290509, 0.09415849]])
==============
You can multiply a by a b that matches it in shape and type:
In [681]: b=np.empty((3,),object)
In [682]: for i in range(3):
...: b[i]=np.arange(i,i+4)
...:
In [683]: b # 3 arrays of length 4 each
Out[683]: array([array([0, 1, 2, 3]), array([1, 2, 3, 4]), array([2, 3, 4, 5])], dtype=object)
In [684]: a*b
Out[684]:
array([array([-0. , 0.07145386, 0.2105622 , -0.16087698]),
array([-0.05335939, -0.06273648, -0.03782142, 0.4613042 ]),
array([-0.18329076, 0.06840354, -0.01162036, 0.47079245])], dtype=object)
Basically it is doing: for i in range(3): res[i]=a[i]*b[i]
Related
I have an 'empty' 2D array in numpy as
arr = np.array([[[], [], []], [[], [], []]]).
When I do np.transpose(arr), I get the result: [], instead of the expected:
[[[],[]],[[],[]],[[],[]]].
You get an empty array [] with the right shape. Mind that also arr is an empty array [].
arr = np.array([[[], [], []], [[], [], []]])
print(arr, arr.shape)
t = arr.T
print(t, t.shape)
[] (2, 3, 0)
[] (0, 3, 2)
Look at what your expression produces:
In [41]: arr = np.array([[[], [], []], [[], [], []]])
In [42]: arr
Out[42]: array([], shape=(2, 3, 0), dtype=float64)
In [43]: print(arr)
[]
In [44]: print(repr(arr))
array([], shape=(2, 3, 0), dtype=float64)
The print shows the str display, while repr is a fuller one that tells us shape and dtype. np.array has followed the [] all the way down, making a 3d array that has float elements. But since the lowest level is created from [], it has size 0 dimension, and overall the array has 0 elements.
What you want, based on the comment, is a (2,3) array with object dtype. This can hold objects such as lists. But making that with np.array is tricky. A more general tool is to make one with the right shape and dtype.
I like to use empty for this, since is fills the object array with None elements. (In a numeric dtype np.empty has other problems, but for object it's nice.)
In [45]: arr = np.empty((2,3), dtype=object)
In [46]: arr
Out[46]:
array([[None, None, None],
[None, None, None]], dtype=object)
But trying to assign a list to elements of such an array can be tricky:
In [47]: arr[:]=[]
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-47-b5ed8d639464> in <module>
----> 1 arr[:]=[]
ValueError: could not broadcast input array from shape (0) into shape (2,3)
In [48]: np.full((2,3),[])
...
ValueError: could not broadcast input array from shape (0) into shape (2,3)
full has the same problem. In addition assignment like this, if it worked, would put the same list in each slot, the equivalent of making a list with [[]]*3. We want a new [] in each slot.
We can do this in the 2d arr, but the iteration is simpler with a 1d (which can be reshaped later):
In [49]: arr = np.empty(6, object)
In [50]: arr
Out[50]: array([None, None, None, None, None, None], dtype=object)
In [51]: for i in range(6): arr[i]=[]
In [52]: arr
Out[52]:
array([list([]), list([]), list([]), list([]), list([]), list([])],
dtype=object)
In [53]: arr = np.reshape(arr, (2,3))
In [54]: arr
Out[54]:
array([[list([]), list([]), list([])],
[list([]), list([]), list([])]], dtype=object)
Obvious we could transpose that, but could just as well use (3,2) in the reshape.
Note that this display of arr clearly shows that it contains list objects.
But do you really need such an array? Is it worth the extra work?
I got an array and reshaped it to the following dimentions: (-1,1,1,1) and (-1,1):
Array A:
[-0.888788523827 0.11842529285 0.319928774626 0.319928774626 0.378755429421 1.225877519716 3.830653798838]
A.reshape(-1,1,1,1):
[[[[-0.888788523827]]]
[[[ 0.11842529285 ]]]
[[[ 0.319928774626]]]
[[[ 0.319928774626]]]
[[[ 0.378755429421]]]
[[[ 1.225877519716]]]
[[[ 3.830653798838]]]]
A.reshape(-1,1):
[[-0.888788523827]
[ 0.11842529285 ]
[ 0.319928774626]
[ 0.319928774626]
[ 0.378755429421]
[ 1.225877519716]
[ 3.830653798838]]
Then I have done substractig and broadcasting came in, so my resulting matrix is 7x1x7x1.
I have a hard time to visualize the intermediate step what broadcasting does. I mean I cannot imagine what elements of arrays are repeated and what they look like while broadcasting.
Could somebody shed some light on this problem,please?
In [5]: arr = np.arange(4)
In [6]: A = arr.reshape(-1,1,1,1)
In [7]: B = arr.reshape(-1,1)
In [8]: C = A + B
In [9]: C.shape
Out[9]: (4, 1, 4, 1)
In [10]: A.shape
Out[10]: (4, 1, 1, 1)
In [11]: B.shape
Out[11]: (4, 1)
There are 2 basic broadcasting rules:
expand the dimensions to match - by adding size 1 dimensions at the start
adjust all size 1 dimensions to match
So in this example:
(4,1,1,1) + (4,1)
(4,1,1,1) + (1,1,4,1) # add 2 size 1's to B
(4,1,4,1) + (4,1,4,1) # adjust 2 of the 1's to 4
(4,1,4,1)
The first step is, perhaps, the most confusing. The (4,1) is expanded to (1,1,4,1), not (4,1,1,1). The rule is intended to avoid ambiguity - by expanding in a consistent manner, not necessarily what a human might intuitively want.
Imagine the case where both arrays need expansion to match, and it could add a dimension in either direction:
(4,) and (3,)
(1,4) and (3,1) or (4,1) and (1,3)
(3,4) or (4,3)
confusion
The rule requires that the programmer choose which one expands to the right (4,1) or (3,1). numpy can then unambiguously add the other.
For a simpler example:
In [22]: A=np.arange(3).reshape(-1,1)
In [23]: B=np.arange(3)
In [24]: C = A+B (3,1)+(3,) => (3,1)+(1,3) => (3,3)
In [25]: C
Out[25]:
array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4]])
In [26]: C.shape
Out[26]: (3, 3)
The [0,2,4] are present, but on the diagonal of C.
When broadcasting like this, the result is a kind of outer sum:
In [27]: np.add.outer(B,B)
Out[27]:
array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4]])
I have an array of shape (5,2) which each row consist of an array of shape (4,3,2) and a float number.
After I slice that array[:,0], I get an array of shape (5,) which each element has shape of (4,3,2), instead of an array of shape (5,4,3,2) (even if I'd use np.array()).
Why?
Edited
Example:
a1 = np.arange(50).reshape(5, 5, 2)
a2 = np.arange(50).reshape(5, 5, 2)
b1 = 15.0
b2 = 25.0
h = []
h.append(np.array([a1, b1]))
h.append(np.array([a2, b2]))
h = np.array(h)[:,0]
np.shape(h) # (2,)
np.shape(h[0]) # (5, 5, 2)
np.shape(h[1]) # (5, 5, 2)
h = np.array(h)
np.shape(h) # (2,) Why not (2, 5, 5, 2)?
You have an array of objects; You can use np.stack to convert it to the shape you need if you are sure all the sub elements have the same shape:
np.stack(a[:,0])
a = np.array([[np.arange(24).reshape(4,3,2), 1.]]*5)
a.shape
# (5, 2)
a[:,0].shape
# (5,)
a[:,0][0].shape
# (4, 3, 2)
np.stack(a[:,0]).shape
# (5, 4, 3, 2)
In [121]: a1.dtype, a1.shape
Out[121]: (dtype('int32'), (5, 5, 2))
In [122]: c1 = np.array([a1,b1])
In [123]: c1.dtype, c1.shape
Out[123]: (dtype('O'), (2,))
Because a1 and b1 are different shaped objects (b1 isn't even an array), an array made from them will have dtype object. And the h made from several continues to be object dtype.
In [124]: h = np.array(h)
In [125]: h.dtype, h.shape
Out[125]: (dtype('O'), (2, 2))
In [126]: h[:,1]
Out[126]: array([15.0, 25.0], dtype=object)
In [127]: h[:,0].dtype
Out[127]: dtype('O')
After the appends, h (as an array) is object dtype. The 2nd column is the b1 and b2 values, the 1st column the a1 and a2.
Some form of concatenate is required to combine those a1 a2 arrays into one. stack does it on a new axis.
In [128]: h[0,0].shape
Out[128]: (5, 5, 2)
In [129]: np.array(h[:,0]).shape # np.array doesn't cross the object boundary
Out[129]: (2,)
In [130]: np.stack(h[:,0]).shape
Out[130]: (2, 5, 5, 2)
In [131]: np.concatenate(h[:,0],0).shape
Out[131]: (10, 5, 2)
Turning the (2,) array into a list, does allow np.array to recombine the elements into a higher dimensional array, just as np.stack does:
In [133]: np.array(list(h[:,0])).shape
Out[133]: (2, 5, 5, 2)
You appear to believe that Numpy can magically divine your intent. As #Barmar explains in the comments, when you slice a shape(5,2) array with [:, 0] you get all rows of the first column of that array. Each element of that slice is a shape(4,3,2) array. Numpy is giving you exactly what you asked for.
If you want to convert that into a shape(5,4,3,2) array you'll need to perform further processing to extract the elements from the shape(4,3,2) arrays.
I'm trying to use individual 1-dimensional boolean arrays to slice a multi-dimension array. For some reason, this code doesn't work:
>>> a = np.ones((100, 200, 300, 2))
>>> a.shape
(100, 200, 300, 2)
>>> m1 = np.asarray([True]*200)
>>> m2 = np.asarray([True]*300)
>>> m2[-1] = False
>>> a[:,m1,m2,:]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (200,) (299,)
>>> m2 = np.asarray([True]*300) # try again with all 300 dimensions True
>>> a[:,m1,m2,:]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (200,) (300,)
But this works just fine:
>>> a = np.asarray([[[1, 2], [3, 4], [5, 6]], [[11, 12], [13, 14], [15, 16]]])
>>> a.shape
(2, 3, 2)
>>> m1 = np.asarray([True, False, True])
>>> m2 = np.asarray([True, False])
>>> a[:,m1,m2]
array([[ 1, 5],
[11, 15]])
Any idea of what I might be doing wrong in the first example?
Short answer: The number of True elements in m1 and m2 must match, unless one of them has only one True term.
Also distinguish between 'diagonal' indexing and 'rectangular' indexing. This is about indexing, not slicing. The dimensions with : are just along for the ride.
Initial ideas
I can get your first case working with:
In [137]: a=np.ones((100,200,300,2))
In [138]: m1=np.ones((200,),bool)
In [139]: m2=np.ones((300,),bool)
In [140]: m2[-1]=False
In [141]: I,J=np.ix_(m1,m2)
In [142]: a[:,I,J,:].shape
Out[142]: (100, 200, 299, 2)
np.ix_ turns the 2 boolean arrays into broadcastable index arrays
In [143]: I.shape
Out[143]: (200, 1)
In [144]: J.shape
Out[144]: (1, 299)
Note that this picks 200 'rows' in one dimension, and 299 in the other.
I'm not sure why this kind of reworking of the arrays is needed in this case, but not in the 2nd
In [154]: b=np.arange(2*3*2).reshape((2,3,2))
In [155]: n1=np.array([True,False,True])
In [156]: n2=np.array([True,False])
In [157]: b[:,n1,n2]
Out[157]:
array([[ 0, 4], # shape (2,2)
[ 6, 10]])
Taking the same ix_ strategy produces the same values but a different shape:
In [164]: b[np.ix_(np.arange(b.shape[0]),n1,n2)]
# or I,J=np.ix_(n1,n2);b[:,I,J]
Out[164]:
array([[[ 0],
[ 4]],
[[ 6],
[10]]])
In [165]: _.shape
Out[165]: (2, 2, 1)
Both cases use all rows of the 1st dimension. The ix one picks 2 'rows' of the 2nd dim, and 1 column of the last, resulting the (2,2,1) shape. The other picks b[:,0,0] and b[0,2,0] terms, resulting (2,2) shape.
(see my addenda as to why both are simply broadcasting).
These are all cases of advanced indexing, with boolean and numeric indexes. One can study the docs, or one can play around. Sometimes it's more fun to do the later. :)
(I knew that ix_ was good for adding the necessary np.newaxis to arrays so can be broadcast together, but didn't realize that worked with boolean arrays as well - it uses np.nonzero() to convert boolean to indices.)
Resolution
Underlying this is, I think, a confusion over 2 modes of indexing. which might called 'diagonal' and 'rectangular' (or element-by-element selection versus block selection). To illustrate look at a small 2d array
In [73]: M=np.arange(6).reshape(2,3)
In [74]: M
Out[74]:
array([[0, 1, 2],
[3, 4, 5]])
and 2 simple numeric indexes
In [75]: m1=np.arange(2); m2=np.arange(2)
They can be used 2 ways:
In [76]: M[m1,m2]
Out[76]: array([0, 4])
and
In [77]: M[m1[:,None],m2]
Out[77]:
array([[0, 1],
[3, 4]])
The 1st picks 2 points, the M[0,0] and M[1,1]. This kind of indexing lets us pick out the diagonals of an array.
The 2nd picks 2 rows and from that 2 columns. This is the kind of indexing the np.ix_ produces. The 1st picks 2 points, the M[0,0] and M[1,1]. This a 'rectangular' form of indexing.
Change m2 to 3 values:
In [78]: m2=np.arange(3)
In [79]: M[m1[:,None],m2] # returns a 2x3
Out[79]:
array([[0, 1, 2],
[3, 4, 5]])
In [80]: M[m1,m2] # produces an error
...
ValueError: shape mismatch: objects cannot be broadcast to a single shape
But if m2 has just one element, we don't get the broadcast error - because the size 1 dimension can be expanded during broadcasting:
In [81]: m2=np.arange(1)
In [82]: M[m1,m2]
Out[82]: array([0, 3])
Now change the index arrays to boolean, each matching the length of the respective dimensions, 2 and 3.
In [91]: m1=np.ones(2,bool); m2=np.ones(3,bool)
In [92]: M[m1,m2]
...
ValueError: shape mismatch: objects cannot be broadcast to a single shape
In [93]: m2[2]=False # m1 and m2 each have 2 True elements
In [94]: M[m1,m2]
Out[94]: array([0, 4])
In [95]: m2[0]=False # m2 has 1 True element
In [96]: M[m1,m2]
Out[96]: array([1, 4])
With 2 and 3 True terms we get an error, but with 2 and 2 or 2 and 1 it runs - just as though we'd used the indices of the True elements: np.nonzero(m2).
To apply this to your examples. In the first, m1 and m2 have 200 and 299 True elements. a[:,m1,m2,:] fails because of a mismatch in the number of True terms.
In the 2nd, they have 2 and 1 True terms, with nonzero indices of [0,2] and [0], which can be broadcast to [0,0]. So it runs.
http://docs.scipy.org/doc/numpy-1.10.0/reference/arrays.indexing.html
explains boolean array indexing in terms of nonzero and ix_.
Combining multiple Boolean indexing arrays or a Boolean with an integer indexing array can best be understood with the obj.nonzero() analogy. The function ix_ also supports boolean arrays and will work without any surprises.
Addenda
On further thought the distinction between 'diagonal' and 'block/rectangular' indexing might be more my mental construct that numpys. Underlying both is the concept of broadcasting.
Take the n1 and n2 booleans, and get their nonzero equivalents:
In [107]: n1
Out[107]: array([ True, False, True], dtype=bool)
In [108]: np.nonzero(n1)
Out[108]: (array([0, 2], dtype=int32),)
In [109]: n2
Out[109]: array([ True, False], dtype=bool)
In [110]: np.nonzero(n2)
Out[110]: (array([0], dtype=int32),)
Now try broadcasting in the 'diagonal' and 'rectangular' modes:
In [105]: np.broadcast_arrays(np.array([0,2]),np.array([0]))
Out[105]: [array([0, 2]),
array([0, 0])]
In [106]: np.broadcast_arrays(np.array([0,2])[:,None],np.array([0]))
Out[106]:
[array([[0],
[2]]),
array([[0],
[0]])]
One produces (2,) arrays, the other (2,1).
This might be a simple workaround:
a[:,m1,:,:][:,:,m2,:]
I am new to Python. I have a trouble doing matrix multiplication. I have two lists:
A =[3.0,3.0]
# 1 by 2 matrix
B =[[ 50.33112583, -49.66887417],
[-49.66887417, 50.33112583]]
# 2 by 2 matrix
Result should be :
# 1 by 2 matrix
c = [1.9867549668874176, 1.986754966887446]
Right now I am doing:
>> A = numpy.matrix(A)
>> B = numpy.matrix(B)
>> C =A * B
>> C
matrix([[ 1.98675497, 1.98675497]])
>>C.tolist()
[[1.9867549668874176, 1.986754966887446]]
If I do dot product then,
>>> B =numpy.array(B)
>>> B
array([[ 50.33112583, -49.66887417],
[-49.66887417, 50.33112583]])
>>> A = [ 3., 3.]
>>> A =numpy.array(A)
>>> A
array([ 3., 3.])
>>> C = numpy.dot(A,B)
>>> C
array([ 1.98675497, 1.98675497])
>>> C.tolist()
[1.9867549668874176, 1.986754966887446]
Why I am getting two brackets when I use matrix multiplication?? Whether dot product and matrix multiplication are same here? Can some one explain me this??
When you use np.matrix() it is by definition a 2-D container and the operations must be performed between 2-D entities and will return 2-D entities:
np.matrix([[1,2,3], [4,5,6]])*[[1], [2], [3]]
#matrix([[14],
# [32]])
np.matrix([[1,2,3], [4,5,6]])*[1, 2, 3]
#ValueError
When you use a np.array() in tha case of dot() between two 2-D arrays the result is a 2-D array; while between a 2-D array and a 1-D array the result is a 1-D array:
np.array([[1,2,3], [4,5,6]]).dot([[1], [2], [3]])
#array([[14],
# [32]])
np.array([[1,2,3], [4,5,6]]).dot([1, 2, 3])
#array([14, 32])
More complex and flexible broadcasting rules for arrays are available when element-wise operations are desired. Here is how each row can be multiplied by a different scalar:
np.array([[1,2,3], [4,5,6]])*[[1], [2]]
#array([[ 1, 2, 3],
# [ 8, 10, 12]])
and how each column can be multiplied by a different scalar:
np.array([[1,2,3], [4,5,6]])*[1, 2, 3]
#array([[ 1, 4, 9],
# [ 4, 10, 18]])