Is there a means to manipulate matrices in NumPy without excessive use of flatten(), ravel(), creating tuples for creating every matrix, etc.?
I do understand it is not matlab but writing 40 chars instead of 4 doesn't seem very efficient.
For example:
A = ones(2,2) # doesn't work
A = ones((2,2)) # works with tuple
v = np.matlib.rand(2)
dot(v, A#v) # doesn't work: shapes are not aligned
vdot(v,A#v) # works
Now I want to update matrix column:
A[:,0]=A#v # nope! shapes are not aligned
# beautiful solution:
c = v.reshape(2,1)
c = A#c
c = c.flatten()
A[:,0]=c
I am assuming the initialization of A is with ones from numpy.ones. We could have a one-liner, like so -
A[:,[0]] = A#v.T
LHS : A[:,[0]] keeps the number of dimensions intact as 2D, as compared to A[:,0], which would have a dimension reduced and thus allows us to assign A#v.T, which is also 2D.
RHS : A#v.T takes care of the first two lines of codes :
c = v.reshape(2,1)
c = A#c
We don't need that third step of c = c.flatten(), because for LHS, we are using a 2D view with A[:,[0]] as explained earlier.
Thus, we are left with a modified fourth step, which is the solution itself listed as the very first code in this post.
Another way
A[:,0] would be a (2,) array, whereas A#v.T would be a (2,1) array. So, (A#v.T).T would be a (1,2) array, which is broadcastable against A[:,0]. So, that gives us another way -
A[:,0] = (A#v.T).T
The argument signature for ones is:
ones(shape, dtype=None, order='C')
shape is one argument, not an open ended *args.
ones(2,2) passes 2 as shape, and 2 asdtype`; so it does not work.
ones((2,2)) passes the tuple (2,2) as shape.
Sometimes functions are written to accept either a tuple or an expanded tuple, e.g. foo((1,2)), foo(*(1,2)), foo(1,2). But that requires extra checking inside the function. Try writing such a function to see for yourself.
Also tuples do not add computational costs. Python creates and uses tuples all the time; Simply using a comma in an expression can create a tuple (if it isn't part of making a list).
Simply defining a function to take an open ended 'list' of arguments creates a tuple:
def foo(*args):
print(type(args))
return args
In [634]: foo(1)
<class 'tuple'>
Out[634]: (1,)
In [635]: foo(1,2)
<class 'tuple'>
Out[635]: (1, 2)
In [636]: foo((1,2))
<class 'tuple'>
Out[636]: ((1, 2),)
In [637]: foo(*(1,2))
<class 'tuple'>
Out[637]: (1, 2)
v = np.matlib.rand(2) doesn't work for me. What is v (shape, dtype)? matlab has a minimum 2d dimensionality; so I suspect v is 2d, maybe even a np.matrix class array.
vdot says it flattens input arguments to 1-D vectors first
Ok, with a special import I get matlib (an old compatibility package):
In [644]: from numpy import matlib
In [645]: matlib.rand(2)
Out[645]: matrix([[ 0.32975512, 0.3491822 ]])
In [646]: _.shape
Out[646]: (1, 2)
Let's try the double dot:
In [647]: v=matlib.rand(2)
In [648]: A=np.ones((2,2))
In [649]: A#v
...
ValueError: shapes (2,2) and (1,2) not aligned: 2 (dim 1) != 1 (dim 0)
Why does it work for you? For 2d arrays we can work directly with dot. # sometimes works as an operator, but adds some of its own quirks.
(edit - later you use A#c where c is a reshaped v, the equivalent of v.T (transpose).)
In [650]: np.dot(A,v.T) # (2,2) dot (2,1) => (2,1)
Out[650]:
matrix([[ 0.63976046],
[ 0.63976046]])
In [651]: np.dot(v,np.dot(A,v.T)) # (1,2) dot with (2,1) -> (1,1)
Out[651]: matrix([[ 0.40929344]])
Come to think of it, since v is np.matrix, this also works: v * A * v.T
We don't need to use matlib to make a 2d array of random floats:
In [662]: v1 = np.random.rand(1,2)
In [663]: v1.shape
Out[663]: (1, 2)
In [668]: np.dot(A,v1.T)
Out[668]:
array([[ 1.63412808],
[ 1.63412808]])
In [669]: np.dot(v1,np.dot(A,v1.T))
Out[669]: array([[ 2.67037459]])
Or if we skip the 2d, making v1 1d
In [670]: v1 = np.random.rand(2)
In [671]: np.dot(A,v1)
Out[671]: array([ 0.8922862, 0.8922862])
In [672]: np.dot(v1, np.dot(A,v1))
Out[672]: 0.79617465579446423
Notice in this last case, we get a scalar, not (1,1) array (or matrix).
np.random.rand is one those functions that accepts *args, the expanded 'tuple'.
In your last example you have to use flat because the A[:,0] slot is (2,) (if A were np.matrix it would still be (2,1)), while the # produces a (2,1), which has to be flattened to fit in (2,)
In [675]: A#v.T
Out[675]:
matrix([[ 0.63976046],
[ 0.63976046]])
In [676]: A[:,0].shape
Out[676]: (2,)
With my 1d v1, A[:,0] = np.dot(A,v1) works without further reshaping.
In general, matlib and np.matrix functions add confusion. There were created to make like easier for wayward MATLAB coders.
And the simplest way to calculate this:
In [681]: np.einsum('i,ij,j',v1,A,v1)
Out[681]: 0.77649708535481299
but with the (1,2) version we can do:
In [683]: v2 = v1[None,:]
In [684]: v2
Out[684]: array([[ 0.20473681, 0.68754938]])
In [685]: v2 # A # v2.T
Out[685]: array([[ 0.77649709]])
Related
What is the best way to perform normal vector addition, where one of the operands is an n x 1 matrix?
Why do I care? Sometimes, a function that should return a vector returns an n x 1 matrix (because the function would equivalently work element-wise on a matrix). When I want to further work with the returned "vector", I always have to reshape - there must be a better way.
For example:
v = np.zeros(shape=(2,1))
w = np.array([1,1])
print('{}, {}'.format(v.shape,w.shape))
Prints: (2, 1), (2,)
print(v+w)
[[1. 1.]
[1. 1.]]
print(v+w.reshape((2,1)))
[[1.]
[1.]] (the desired output!)
Sounds a bit like you are coming from MATLAB where everything is 2d (scalars size is (1,1)) and the trailing dimension is outermost. Or a linear algebra that treats 'vectors' as single column matrices.
In numpy, 0 and 1d arrays are just a normal as 2d. A shape like (n,) is common. By the rules of broadcasting adding a leading dimension is automatic (1,n), but adding a trailing dimension requires user action. That a[:,None] is most idiomatic, though not the only option.
The v+w broadcasting logic is
(2,1) + (2,) => (2,1) + (1,2) => (2,2)
The auto-leading logic avoids ambiguity (what should happen if you try to add a (2,) to a (3,)?). And since leading dimensions are 'outer-most' it makes most sense to expand in that direction. MATLAB on the other hand 'naturally' expends and contracts the trailing dimensions.
So to some degree or other a (n,1) shape is more awkward in numpy, though it is still relatively easy to create.
Another example of an auto leading dimension:
In [129]: np.atleast_2d(np.arange(3)).shape
Out[129]: (1, 3)
On the other hand expand_dims lets us add dimensions all over the place
In [132]: np.expand_dims(np.arange(3),(0,2,3)).shape
Out[132]: (1, 3, 1, 1)
If w has the desired shape of the result (e.g. (2,)), and v has the same size (e.g. (2,1), or (2,)), this is safe and easy:
w + v.reshape(w.shape)
Less generally, if all you want is to get rid of the last dimension, knowing it is of length 1, you can do:
w + v[..., 0]
I am porting some Matlab code to python and I have the following statement in Matlab:
cross([pt1,1]-[pp,0],[pt2,1]-[pp,0]);
pt1, pt2 and pp are 2D points.
So, my corresponding python code looks as follows:
np.cross(np.c_[pt1 - pp, 1], np.c_[pt2 - pp, 1])
The points are defined as:
pt1 = np.asarray((440.0, 59.0))
pt2 = np.asarray((-2546.23, 591.03))
pp = np.asarray([563., 456.5])
When I execute the statement with the cross product, I get the following error:
ValueError: all the input array dimensions except for the concatenation axis must match exactly
So looking at some other posts, here I thought I would try np.column_stack but I get the same error:
np.cross(np.column_stack((pt1 - pp, 1)), np.column_stack((pt2 - pp, 1)))
This might be what you are looking for:
np.cross(np.append(pt1-pp, 1), np.append(pt2-pp, 1))
If you use np.r_ instead it works:
In [40]: np.cross(np.r_[pt1 - pp, 1], np.r_[pt2 - pp, 1])
Out[40]: array([-5.32030000e+02, -2.98623000e+03, -1.25246611e+06])
Your pt1 and pp are (2,) arrays. To add a 1 to them you need to use a 1d concatenate, np.r_ for 'row', as opposed to columns.
There are lots of ways of constructing a 3 element array:
In [43]: np.r_[pt1 - pp, 1]
Out[43]: array([-123. , -397.5, 1. ])
In [44]: np.append(pt1 - pp, 1)
Out[44]: array([-123. , -397.5, 1. ])
In [45]: np.concatenate((pt1 - pp, [1]))
Out[45]: array([-123. , -397.5, 1. ])
concatenate is the base operation. The others tweak the 1 to produce a 1d array that can be joined with the (2,) shape array to make a (3,).
Concatenate turns all of its inputs into arrays, if they aren't already: np.concatenate((pt1 - pp, np.array([1]))).
Note that np.c_ docs say it is the equivalent of
np.r_['-1,2,0', index expression]
That initial string expression is a bit complicated. The key point is it tries to concatenate 2d arrays (whereas your pt1 is 1d).
It is like column_stack, joiningn(2,1)arrays to make a(2,n)` array.
In [48]: np.c_[pt1, pt2]
Out[48]:
array([[ 440. , -2546.23],
[ 59. , 591.03]])
In [50]: np.column_stack((pt1, pt2))
Out[50]:
array([[ 440. , -2546.23],
[ 59. , 591.03]])
In MATLAB everything has at least 2 dimensions, and because it is Fortran based, the outer dimensions are last. So in a sense its most natural 'vector' shape is n x 1, a column matrix. numpy is built on Python, with a natural interface to its scalars and nested lists. Order is c based; the initial dimensions are outer most. So numpy code can have true scalars (Python numbers without shape or size), or arrays with 0 or more dimensions. A 'vector' most naturally has shape (n,) (a 1 element tuple). It can easily be reshaped to (1,n) or (n,1) if needed.
If you want a (3,1) array (instead of (3,) shaped), you'd need to use some sort of 'vertical' concatenation, joining a (2,1) array with a (1,1):
In [51]: np.r_['0,2,0', pt1-pp, 1]
Out[51]:
array([[-123. ],
[-397.5],
[ 1. ]])
In [53]: np.vstack([(pt1-pp)[:,None], 1])
Out[53]:
array([[-123. ],
[-397.5],
[ 1. ]])
(But np.cross wants (n,3) or (3,) arrays, not (3,1)!)
In [58]: np.cross(np.r_['0,2,0', pt1-pp, 1], np.r_['0,2,0', pt2-pp, 1])
...
ValueError: incompatible dimensions for cross product
(dimension must be 2 or 3)
To get around this specify an axis:
In [59]: np.cross(np.r_['0,2,0', pt1-pp, 1], np.r_['0,2,0', pt2-pp, 1], axis=0)
Out[59]:
array([[-5.32030000e+02],
[-2.98623000e+03],
[-1.25246611e+06]])
Study np.cross if you want an example of manipulating dimensions. In this axis=0 case it transposes the arrays so they are (1,3) and then does the calculation.
I have a Numpy array of shape (5,5,3,2). I want to take the element (1,4) of that matrix, which is also a matrix of shape (3,2), and add an element to it -so it becomes a (4,2) array.
The code I'm using is the following:
import numpy as np
a = np.random.rand(5,5,3,2)
a = np.array(a, dtype = object) #So I can have different size sub-matrices
a[2][3] = np.append(a[2][3],[[1.0,1.0]],axis=0) #a[2][3] shape = (3,2)
I'm always obtaining the error:
ValueError: could not broadcast input array from shape (4,2) into shape (3,2)
I understand that the shape returned by the np.append function is not the same as the a[2][3] sub-array, but I thought that the dtype=object would solve my problem. However, I need to do this. Is there any way to go around this limitation?
I also tried to use the insert function but I don't know how could I add the element in the place I want.
Make sure you understand what you have produced. That requires checking the shape and dtype, and possibly looking at the values
In [29]: a = np.random.rand(5,5,3,2)
In [30]: b=np.array(a, dtype=object)
In [31]: a.shape
Out[31]: (5, 5, 3, 2) # a is a 4d array
In [32]: a.dtype
Out[32]: dtype('float64')
In [33]: b.shape
Out[33]: (5, 5, 3, 2) # so is b
In [34]: b.dtype
Out[34]: dtype('O')
In [35]: b[2,3].shape
Out[35]: (3, 2)
In [36]: c=np.append(b[2,3],[[1,1]],axis=0)
In [37]: c.shape
Out[37]: (4, 2)
In [38]: c.dtype
Out[38]: dtype('O')
b[2][3] is also an array. b[2,3] is the proper numpy way of indexing 2 dimensions.
I suspect you wanted b to be a (5,5) array containing arrays (as objects), and you think that you you can simply replace one of those with a (4,2) array. But the b constructor simply changes the floats of a to objects, without changing the shape (or 4d nature) of b.
I could construct a (5,5) object array, and fill it with values from a. And then replace one of those values with a (4,2) array:
In [39]: B=np.empty((5,5),dtype=object)
In [40]: for i in range(5):
...: for j in range(5):
...: B[i,j]=a[i,j,:,:]
...:
In [41]: B.shape
Out[41]: (5, 5)
In [42]: B.dtype
Out[42]: dtype('O')
In [43]: B[2,3]
Out[43]:
array([[ 0.03827568, 0.63411023],
[ 0.28938383, 0.7951006 ],
[ 0.12217603, 0.304537 ]])
In [44]: B[2,3]=c
In [46]: B[2,3].shape
Out[46]: (4, 2)
This constructor for B is a bit crude. I've answered other questions about creating/filling object arrays, but I'm not going to take the time here to streamline this case. It's for illustration purposes only.
In an array of object, any element can be indeed an array (or any kind of object).
import numpy as np
a = np.random.rand(5,5,3,2)
a = np.array(a, dtype=object)
# Assign an 1D array to the array element ``a[2][3][0][0]``:
a[2][3][0][0] = np.arange(10)
a[2][3][0][0][9] # 9
However a[2][3] is not an array element, it is a whole array.
a[2][3].ndim # 2
Therefore when you do a[2][3] = (something) you are using broadcasting instead of assigning an element: numpy tries to replace the content of the subarray a[2][3] and fails because of shape mismatch. The memory layout of numpy arrays does not allow to change the shape of subarrays.
Edit: Instead of using numpy arrays you could use nested lists. These nested lists can have arbitrary sizes. Note that the memory is higher and that the access time is higher compared to numpy array.
import numpy as np
a = np.random.rand(5,5,3,2)
a = np.array(a, dtype=object)
b = np.append(a[2][3], [[1.0,1.0]],axis=0)
a_list = a.tolist()
a_list[2][3] = b.tolist()
The problem here, is that you try to assign to a[2][3]
Make a new array instead.
new_array = np.append(a[2][3],np.array([[1.0,1.0]]),axis=0)
I'm currently wondering how the numpy array behaves. I feel like the dimensions are not consistent from vectors (Nx1 dimensional) to 'real arrays' (NxN dimensional).
I dont get, why this isn't working:
a = array(([1,2],[3,4],[5,6]))
concatenate((a[:,0],a[:,1:]), axis = 1)
# ValueError: all the input arrays must have same number of dimensions
It seems like the : (at 1:]) makes the difference, but (:0 is not working)
Thanks in advance!
Detailled Version: So I would expect that shape(b)[0] references the vertical direction in (Nx1 arrays), like in an 2D (NxN) array. But it seems like dimension [0] is the horizontal direction in arrays (Nx1 arrays)?
from numpy import *
a = array(([1,2],[3,4],[5,6]))
b = a[:,0]
print shape(a) # (3L, 2L), [0] is vertical
print a # [1,2],[3,4],[5,6]
print shape(b) # (3L, ), [0] is horizontal
print b # [1 3 5]
c = b * ones((shape(b)[0],1))
print shape(c) # (3L, 3L), I'd expect (3L, 1L)
print c # [[ 1. 3. 5.], [ 1. 3. 5.], [ 1. 3. 5.]]
What did I get wrong? Is there a nicer way than
d = b * ones((1, shape(b)[0]))
d = transpose(d)
print shape(d) # (3L, 1L)
print d # [[ 1.], [ 3.], [ 5.]]
to get the (Nx1) vector that I expect or want?
There are two overall issues here. First, b is not an (N, 1) shaped array, it is an (N,) shaped array. In numpy, 1D and 2D arrays are different things. 1D arrays simply have no direction. Vertical vs. horizontal, rows vs. columns, these are 2D concepts.
The second has to do with something called "broadcasting". In numpy arrays, you are able to broadcast lower-dimensional arrays to higher-dimensional ones, and the lower-dimensional part is applied elementwise to the higher-dimensional one.
The broadcasting rules are pretty simple:
When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward. Two dimensions are compatible when
they are equal, or
one of them is 1
In your case, it starts with the last dimension of ones((shape(b)[0],1)), which is 1. This meets the second criteria. So it multiplies the array b elementwise for each element of ones((shape(b)[0],1)), resulting in a 3D array.
So it is roughly equivalent to:
c = np.array([x*b for x in ones(shape(b))])
Edit:
To answer your original question, what you want to do is to keep both the first and second arrays as 2D arrays.
numpy has a very simple rule for this: indexing reduces the number of dimensions, slicing doesn't. So all you need is to have a length-1 slice. So in your example, just change a[:,0] to a[:,:1]. This means 'get every column up to the second one'. Of course that only includes the first column, but it is still considered a slice operation rather than getting an element, so it still preservers the number of dimensions:
>>> print(a[:, 0])
[1 3 5]
>>> print(a[:, 0].shape)
(3,)
>>> print(a[:, :1])
[[1]
[3]
[5]]
>>> print(a[:, :1].shape)
(3, 1)
>>> print(concatenate((a[:,:1],a[:,1:]), axis = 1))
[[1 2]
[3 4]
[5 6]]
In numpy, I have two "arrays", X is (m,n) and y is a vector (n,1)
using
X*y
I am getting the error
ValueError: operands could not be broadcast together with shapes (97,2) (2,1)
When (97,2)x(2,1) is clearly a legal matrix operation and should give me a (97,1) vector
EDIT:
I have corrected this using X.dot(y) but the original question still remains.
dot is matrix multiplication, but * does something else.
We have two arrays:
X, shape (97,2)
y, shape (2,1)
With Numpy arrays, the operation
X * y
is done element-wise, but one or both of the values can be expanded in one or more dimensions to make them compatible. This operation is called broadcasting. Dimensions, where size is 1 or which are missing, can be used in broadcasting.
In the example above the dimensions are incompatible, because:
97 2
2 1
Here there are conflicting numbers in the first dimension (97 and 2). That is what the ValueError above is complaining about. The second dimension would be ok, as number 1 does not conflict with anything.
For more information on broadcasting rules: http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html
(Please note that if X and y are of type numpy.matrix, then asterisk can be used as matrix multiplication. My recommendation is to keep away from numpy.matrix, it tends to complicate more than simplifying things.)
Your arrays should be fine with numpy.dot; if you get an error on numpy.dot, you must have some other bug. If the shapes are wrong for numpy.dot, you get a different exception:
ValueError: matrices are not aligned
If you still get this error, please post a minimal example of the problem. An example multiplication with arrays shaped like yours succeeds:
In [1]: import numpy
In [2]: numpy.dot(numpy.ones([97, 2]), numpy.ones([2, 1])).shape
Out[2]: (97, 1)
Per numpy docs:
When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward. Two dimensions are compatible when:
they are equal, or
one of them is 1
In other words, if you are trying to multiply two matrices (in the linear algebra sense) then you want X.dot(y) but if you are trying to broadcast scalars from matrix y onto X then you need to perform X * y.T.
Example:
>>> import numpy as np
>>>
>>> X = np.arange(8).reshape(4, 2)
>>> y = np.arange(2).reshape(1, 2) # create a 1x2 matrix
>>> X * y
array([[0,1],
[0,3],
[0,5],
[0,7]])
You are looking for np.matmul(X, y). In Python 3.5+ you can use X # y.
It's possible that the error didn't occur in the dot product, but after.
For example try this
a = np.random.randn(12,1)
b = np.random.randn(1,5)
c = np.random.randn(5,12)
d = np.dot(a,b) * c
np.dot(a,b) will be fine; however np.dot(a, b) * c is clearly wrong (12x1 X 1x5 = 12x5 which cannot element-wise multiply 5x12) but numpy will give you
ValueError: operands could not be broadcast together with shapes (12,1) (1,5)
The error is misleading; however there is an issue on that line.
Use np.mat(x) * np.mat(y), that'll work.
We might confuse ourselves that a * b is a dot product.
But in fact, it is broadcast.
Dot Product :
a.dot(b)
Broadcast:
The term broadcasting refers to how numpy treats arrays with different
dimensions during arithmetic operations which lead to certain
constraints, the smaller array is broadcast across the larger array so
that they have compatible shapes.
(m,n) +-/* (1,n) → (m,n) : the operation will be applied to m rows
Convert the arrays to matrices, and then perform the multiplication.
X = np.matrix(X)
y = np.matrix(y)
X*y
we should consider two points about broadcasting.
first: what is possible.
second: how much of the possible things is done by numpy.
I know it might look a bit confusing, but I will make it clear by some example.
lets start from the zero level.
suppose we have two matrices. first matrix has three dimensions (named A) and the second has five (named B).
numpy tries to match last/trailing dimensions. so numpy does not care about the first two dimensions of B.
then numpy compares those trailing dimensions with each other. and if and only if they be equal or one of them be 1, numpy says "O.K. you two match". and if it these conditions don't satisfy, numpy would "sorry...its not my job!".
But I know that you may say comparison was better to be done in way that can handle when they are devisable(4 and 2 / 9 and 3). you might say it could be replicated/broadcasted by a whole number(2/3 in out example). and i am agree with you. and this is the reason I started my discussion with a distinction between what is possible and what is the capability of numpy.
This is because X and y are not the same types. for example X is a numpy matrix and y is a numpy array!
Error: operands could not be broadcast together with shapes (2,3) (2,3,3)
This kind of error occur when the two array does not have the same shape.
to correct this you need reshape one array to match the other.
see example below
a1 = array([1, 2, 3]), shape = (2,3)
a3 =array([[[1., 2., 3.],
[2., 3., 2.],
[2., 4., 5.]],
[[1., 0., 3.],
[2., 3., 7.],
[2., 4., 6.]]])
with shape = (2,3,3)
IF i try to run np.multiply(a2,a3) it will return the error below
Error: operands could not be broadcast together with shapes (2,3) (2,3,3)
to solve this check out the broadcating rules
which state hat Two dimensions are compatible when:
#1.they are equal, or
#2.one of them is 1`
Therefore lets reshape a2.
reshaped = a2.reshape(2,3,1)
Now try to run np.multiply(reshaped,a3)
the multiplication will run SUCCESSFUL!!
ValueError: operands could not be broadcast together with shapes (x ,y) (a ,b)
where x ,y are variables
Basically this error occurred when value of y (no. of columns) doesn't equal to the number of elements in another multidimensional array.
Now let's go through by ex=>
coding apart
import numpy as np
arr1= np.arange(12).reshape(3,
output of arr1
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
arr2= np.arange(4).reshape(1,4)
or (both are same 1 rows and 4 columns)
arr2= np.arange(4)
ouput of arr2=>
array([0, 1, 2, 3])
no of elements in arr2 is equal no of no. of the columns in arr1 it will be excute.
for x,y in np.nditer([a,b]):
print(x,y)
output =>
0 0
1 1
2 2
3 3
4 0
5 1
6 2
7 3
8 0
9 1
10 2
11 3