I'm new to numpy and am having trouble understanding how shapes of arrays are decided.
An array of the form
[[[5, 10, 15], [20, 25, 30], [35, 40, 45]], [1,2,4,3]]
has a shape of (2,) while one of the form
[[[5, 10, 15], [20, 25, 30], [35, 40, 45]], [1,2,4]]
has a shape of (2,3). Moreover,
[[[5, 10, 15], [20, 25, 30], [35, 40, 45]], [[1,2,4], [3,4,2]]]
has a shape of (2,) but adding another vector as
[[[5, 10, 15], [20, 25, 30], [35, 40, 45]], [[1,2,4], [3,4,2], [1,2,4]]]
changes the shape to (2,3,3). Intuitively, I feel that all the arrays should be 3 - dimensional. Could anyone help me understand what's happening exactly?
The underlying idea is that np.array tries to create as high a dimensional array as it can. When the sublists have matching numbers of elements the result is easy to see. When they mix lists of differing lengths the result can be confusing.
In your first case you have 2 sublists, one of length 3, the other length 4. So it makes a 2 element object array, and doesn't try to parse the sublists of the 1st sublist
In [1]: arr = np.array([[[5, 10, 15], [20, 25, 30], [35, 40, 45]], [1,2,4,3]])
In [2]: arr
Out[2]: array([[[5, 10, 15], [20, 25, 30], [35, 40, 45]],
[1, 2, 4, 3]
], dtype=object) # adjusted format
In [3]: arr.dtype
Out[3]: dtype('O')
In [4]: arr.shape
Out[4]: (2,)
In [5]: arr[0]
Out[5]: [[5, 10, 15], [20, 25, 30], [35, 40, 45]] # 3 element list of lists
In [6]: arr[1]
Out[6]: [1, 2, 4, 3] # 4 element list of numbers
In the 2nd case you have two sublists, both of length 3. So it makes a 2x3 array. But one sublist contains lists, the other numbers - so the result is again object array:
In [7]: arr = np.array([[[5, 10, 15], [20, 25, 30], [35, 40, 45]], [1,2,4]] )
In [8]: arr
Out[8]:
array([[[5, 10, 15], [20, 25, 30], [35, 40, 45]],
[1, 2, 4]
], dtype=object)
In [9]: arr.shape
Out[9]: (2, 3)
In [10]: arr[0,0]
Out[10]: [5, 10, 15]
Finally, 2 lists, each with 3 elements, each of which is also 3 element lists - a 3d array.
In [11]: arr = np.array([[[5, 10, 15], [20, 25, 30], [35, 40, 45]], [[1,2,4], [3,4,2], [1,2,4]]] )
In [12]: arr
Out[12]:
array([[[ 5, 10, 15],
[20, 25, 30],
[35, 40, 45]],
[[ 1, 2, 4],
[ 3, 4, 2],
[ 1, 2, 4]]])
In [13]: arr.shape
Out[13]: (2, 3, 3)
There are also mixes of sublist lengths that can raise an error.
In general don't mix sublists of differing size and content type casually. np.array behaves most predictably when given lists that will produce a nice multidimensional array. Mixing list lengths leads to confusion.
Updated numpy:
In [1]: np.__version__
Out[1]: '1.13.1'
In [2]: np.array([[[5, 10, 15], [20, 25, 30], [35, 40, 45]], [1,2,4,3]])
Out[2]: array([list([[5, 10, 15], [20, 25, 30], [35, 40, 45]]), list([1, 2, 4, 3])], dtype=object)
In [3]: np.array([[[5, 10, 15], [20, 25, 30], [35, 40, 45]], [1,2,4]] )
Out[3]:
array([[list([5, 10, 15]), list([20, 25, 30]), list([35, 40, 45])],
[1, 2, 4]], dtype=object)
It now identifies the list elements
This last example is still (2,3) object array. As such each of those 6 elements could be a different Python type, e.g.:
In [11]: np.array([[[5, 10, 15], np.array([20, 25, 30]), (35, 40, 45)], [None,2,'astr']] )
Out[11]:
array([[list([5, 10, 15]), array([20, 25, 30]), (35, 40, 45)],
[None, 2, 'astr']], dtype=object)
In [12]: [type(x) for x in _.flat]
Out[12]: [list, numpy.ndarray, tuple, NoneType, int, str]
Related
I have to np arrays
a = np.array[[1,2]
[2,3]
[3,4]
[5,6]]
b = np.array [[2,4]
[6,8]
[10,11]
I want to multiple each row of a against each element in array b so that array c is created with dimensions of a-rows x b rows (as columns)
c = np.array[[2,8],[6,16],[10,22]
[4,12],[12,21],[20,33]
....]
There are other options for doing this, but I would really like to leverage the speed of numpy's ufuncs...if possible.
any and all help is appreciated.
Does this do what you want?
>>> a
array([[1, 2],
[2, 3],
[3, 4],
[5, 6]])
>>> b
array([[ 2, 4],
[ 6, 8],
[10, 11]])
>>> a[:,None,:]*b
array([[[ 2, 8],
[ 6, 16],
[10, 22]],
[[ 4, 12],
[12, 24],
[20, 33]],
[[ 6, 16],
[18, 32],
[30, 44]],
[[10, 24],
[30, 48],
[50, 66]]])
>>> _.shape
(4, 3, 2)
Or if that doesn't have the right shape, you can reshape it:
>>> (a[:,None,:]*b).reshape((a.shape[0]*b.shape[0], 2))
array([[ 2, 8],
[ 6, 16],
[10, 22],
[ 4, 12],
[12, 24],
[20, 33],
[ 6, 16],
[18, 32],
[30, 44],
[10, 24],
[30, 48],
[50, 66]])
I have two matrices (numpy arrays), mu and nu. From these I would like to create a third array as follows:
new_array_{j, k, l} = mu_{l, k} nu_{j, k}
I can do it naively using list comprehensions:
[[[mu[l, k] * nu[j, k] for k in np.arange(N)] for l in np.arange(N)] for j in np.arange(N)]
but it quickly becomes slow.
How can I create new_array using numpy functions which should be faster?
Two quick solutions (without my usual proofs and explanations):
res = np.einsum('lk,jk->jkl', mu, nu)
res = mu.T[None,:,:] * nu[:,:,None] # axes in same order as result
#!/usr/bin/env python
import numpy as np
# example data
mu = np.arange(10).reshape(2,5)
nu = np.arange(15).reshape(3,5) + 20
# get array sizes
nl, nk = mu.shape
nj, nk_ = nu.shape
assert(nk == nk_)
# get arrays with dimensions (nj, nk, nl)
# in the case of mu3d, we need to add a slowest varying dimension
# so (after transposing) this can be done by cycling through the data
# nj times along the slowest existing axis and then reshaping
mu3d = np.concatenate((mu.transpose(),) * nj).reshape(nj, nk, nl)
# in the case of nu3d, we need to add a new fastest varying dimension
# so this can be done by repeating each element nl times, and again it
# needs reshaping
nu3d = nu.repeat(nl).reshape(nj, nk, nl)
# now just multiple element by element
new_array = mu3d * nu3d
print(new_array)
Gives:
>>> mu
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
>>> nu
array([[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34]])
>>> nj, nk, nl
(3, 5, 2)
>>> mu3d
array([[[0, 5],
[1, 6],
[2, 7],
[3, 8],
[4, 9]],
[[0, 5],
[1, 6],
[2, 7],
[3, 8],
[4, 9]],
[[0, 5],
[1, 6],
[2, 7],
[3, 8],
[4, 9]]])
>>> nu3d
array([[[20, 20],
[21, 21],
[22, 22],
[23, 23],
[24, 24]],
[[25, 25],
[26, 26],
[27, 27],
[28, 28],
[29, 29]],
[[30, 30],
[31, 31],
[32, 32],
[33, 33],
[34, 34]]])
>>> new_array
array([[[ 0, 100],
[ 21, 126],
[ 44, 154],
[ 69, 184],
[ 96, 216]],
[[ 0, 125],
[ 26, 156],
[ 54, 189],
[ 84, 224],
[116, 261]],
[[ 0, 150],
[ 31, 186],
[ 64, 224],
[ 99, 264],
[136, 306]]])
I am building a neural network. where I have to flatten my training dataset.
I have two options.
1 is:
train_x_flatten = train_x_orig.reshape(train_x_orig.shape[0], -1).T
and 2nd one is:
train_x_flatten = train_x_orig.reshape(train_x_orig.shape[1]*train_x_orig.shape[2]*train_x_orig.shape[3], 209)
both gave the same shape but I found difference while computing cost?
why is that? thank you
Your original tensor is of at least rank 4 based on the second example. The first example pulls each element, ordered by increasing the right-most index, and inserts the elements into rows the length of the zeroth shape. Then transposes.
The second example again pull elements from by incrementing from the right-most index, i.e.:
element = train_x_orig[0, 0, 0, 0]
new_row.append(element)
element = train_x_orig[0, 0, 0, 1]
new_row.append(element)
but the size of the row is different. It is now the dimension of everything else in the tensor.
Here is an example to illustrate.
First we create an ordered array and reshape it to rank 4.
import numpy as np
x = np.arange(36).reshape(3,2,3,2)
x
# returns:
array([[[[ 0, 1],
[ 2, 3],
[ 4, 5]],
[[ 6, 7],
[ 8, 9],
[10, 11]]],
[[[12, 13],
[14, 15],
[16, 17]],
[[18, 19],
[20, 21],
[22, 23]]],
[[[24, 25],
[26, 27],
[28, 29]],
[[30, 31],
[32, 33],
[34, 35]]]])
Here is the output of the first example
x.reshape(x.shape[0], -1).T
# returns:
array([[ 0, 12, 24],
[ 1, 13, 25],
[ 2, 14, 26],
[ 3, 15, 27],
[ 4, 16, 28],
[ 5, 17, 29],
[ 6, 18, 30],
[ 7, 19, 31],
[ 8, 20, 32],
[ 9, 21, 33],
[10, 22, 34],
[11, 23, 35]])
And here is the second example
x.reshape(x.shape[1]*x.shape[2]*x.shape[3], -1)
# returns:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23],
[24, 25, 26],
[27, 28, 29],
[30, 31, 32],
[33, 34, 35]])
How the elements get reordered is fundamentally different.
In python, I have an array of shape n*2 ( where n is a positive integer ). Essentially, this is an array of pairs. I wish to remove all mirror pairs in this array. For example, the following array A is of shape 10*2. The pairs [0,55] and [55, 0] would constitute one such mirror pair in A, and I wish to keep one out of those two.
A = np.array([[ 0, 55], [ 5, 25], [12, 62], [27, 32], [25, 73],
[55, 0], [25, 5], [62, 12], [32, 27], [99, 95]])
For the aforementioned example, I would want the result array to look like:
B = np.array([[ 0, 55], [ 5, 25], [12, 62], [27, 32], [25, 73], [99,95])
since there are 6 unique pairs (after 4 mirror pairs are excluded).
I realize that I can achieve this using two nested for loops, but I would want to achieve this using the fastest possible method, since for the actual problem at hand, I will be dealing with huge arrays. I will be thankful to have some help.
A cryptic one-liner:
In [301]: A
Out[301]:
array([[ 0, 55],
[ 5, 25],
[12, 62],
[27, 32],
[25, 73],
[55, 0],
[25, 5],
[62, 12],
[32, 27],
[99, 95]])
In [302]: np.unique(np.sort(A, axis=1).view(','.join([A.dtype.char]*2))).view(A.dtype).reshape(-1, 2)
Out[302]:
array([[ 0, 55],
[ 5, 25],
[12, 62],
[25, 73],
[27, 32],
[95, 99]])
Break it down into steps...
First, create a copy that is sorted along the second axis. In the sorted array, we want to remove duplicate rows.
In [303]: a = np.sort(A, axis=1)
In [304]: a
Out[304]:
array([[ 0, 55],
[ 5, 25],
[12, 62],
[27, 32],
[25, 73],
[ 0, 55],
[ 5, 25],
[12, 62],
[27, 32],
[95, 99]])
numpy.unique() can be used to find the unique elements of an array, but it only works with one-dimensional data. So we'll create a one-dimensional view of b in which each row becomes a single structure with two fields. One way to define the new data type that we want is as a string:
In [305]: dt = ','.join([A.dtype.char]*2)
In [306]: dt
Out[306]: 'l,l'
b is a structured array; it is the one-dimensional view of a:
In [307]: b = a.view(dt)
In [308]: b
Out[308]:
array([[( 0, 55)],
[( 5, 25)],
[(12, 62)],
[(27, 32)],
[(25, 73)],
[( 0, 55)],
[( 5, 25)],
[(12, 62)],
[(27, 32)],
[(95, 99)]],
dtype=[('f0', '<i8'), ('f1', '<i8')])
Now we use numpy.unique() to find the unique elements of b:
In [309]: u = np.unique(b)
In [310]: u
Out[310]:
array([( 0, 55), ( 5, 25), (12, 62), (25, 73), (27, 32), (95, 99)],
dtype=[('f0', '<i8'), ('f1', '<i8')])
Next, create a view of u, using the data type of the original array A. This will be one-dimensional:
In [311]: v = u.view(A.dtype)
In [312]: v
Out[312]: array([ 0, 55, 5, 25, 12, 62, 25, 73, 27, 32, 95, 99])
Finally, reshape v to restore the two-dimensional array:
In [313]: w = v.reshape(-1, 2)
In [314]: w
Out[314]:
array([[ 0, 55],
[ 5, 25],
[12, 62],
[25, 73],
[27, 32],
[95, 99]])
If you are using a pure python list, try the following code.
>>> list(set([tuple(i) for i in map(sorted, b)]))
[(27, 32), (5, 25), (12, 62), (95, 99), (25, 73), (0, 55)]
I'm assuming the order of the pairs doesn't matter (for example: [1,2] = [2,1]). If this is the case, you can flip all pairs so that the first number is always smaller than the second number.
[[1,2], [4,3], [1,7], [10,2]]
becomes
[[1,2], [3,4], [1,7], [2,10]]
Then you could sort all the pairs by the first, then second number:
[[1,2], [1,7], [2,10], [3,4]]
Finally, you could loop through the list and remove any duplicate pairs.
If you use an efficient sorting algorithm, like a mergesort, this entire process will have O(n*log(n)) work, which is much better than O(n^2) work (what you get with nested for loops.
I will show u my way(because it has one my favourites tricks to convert 1D list to nD),even if there are probably easier ways:
A = [[ 0, 55], [ 5, 25], [12, 62], [27, 32], [25, 73],
[55, 0], [25, 5], [62, 12], [32, 27], [99, 95]]
B=[]
long = int(len(A)/2)
for i in range(long):
if A[i][0] == A[i+long][1] and A[i][1] == A[i+long][0]:
B.append(A[i][0])
B.append(A[i][1])
else:
B.append(A[i][0])
B.append(A[i][1])
B.append(A[i+long][0])
B.append(A[i+long][1])
#Now we created an 1D list and then we convert it to 2D!
B=[B[i:i+2] for i in range(0,len(B),2)]
I have many 3*2 matrices(A1,A2,A3..), and each of the 3*2 is a draw. In the case two draws, we have a 3*4 ( we horizontally stack each draw of A1,A2). Clearly, it is easier for me to draw the 3*4 matrix (A) as a larger matrices once instead of draw a 3*2 over and over again.
But I need to perform a matrix multiplication for each draw(each A1,A2...) to a matrix B. Say A1*B, and A2*B ...AN*B
#each draw of the 3*2 matrix
A1 = np.array([[ 0, 1],
[ 4, 5],
[ 8, 9]])
A2 = np.array([[ 2, 3],
[ 6, 7],
[ 10, 11]])
# A is [A1,A2]
# Easier to draw A once for all (the larger matrix)
A = np.array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
b = np.array([[ 0, 1],
[ 4, 5]
])
desired output
array([[ 4, 5, 12, 17],
[20, 29, 28, 41],
[36, 53, 44, 65]])
You can reshape matrix A to 2 columns so that it is conformable to b, do the matrix multiplication, and then reshape it back:
np.dot(A.reshape(-1, 2), b).reshape(3, -1)
#array([[ 4, 5, 12, 17],
# [20, 29, 28, 41],
# [36, 53, 44, 65]])
If you are unsure about how to store/stack the incoming arrays, one way would be stacking those as a 3D array, such that the each of those incoming arrays are index-able by its first axis -
a = np.array((A1,A2))
Sample run -
In [143]: a = np.array((A1,A2))
In [144]: a.shape
Out[144]: (2, 3, 2)
|-----------------> axis of stacking
Then, to get the equivalent output of matrix-multiplications of each incoming array with b, we could use np.tensordot on the 3D stacked array a with b, thus losing the last axis from a and first from b in the sum-reduction, like so -
out = np.tensordot(a,b,axes=((2),(0)))
Let's have a look at the output values and compare against each matrix-multiplication with A1, A2, etc. -
In [138]: out[0]
Out[138]:
array([[ 4, 5],
[20, 29],
[36, 53]])
In [139]: out[1]
Out[139]:
array([[12, 17],
[28, 41],
[44, 65]])
In [140]: A1.dot(b)
Out[140]:
array([[ 4, 5],
[20, 29],
[36, 53]])
In [141]: A2.dot(b)
Out[141]:
array([[12, 17],
[28, 41],
[44, 65]])
Thus, essentially with this stacking operation and later on tensordot we have :
out[0], out[1], .... = A1.dot(b), A2.dot(b), ....
Alternative to np.tensordot -
We could use a simpler version with np.matmul, to get the same output as with tensordot -
out = np.matmul(a,b)
On Python 3.5, there's an even simpler version that replaces np.matmul, the # operator -
out = a # b
Even if not needed for the calculation einsum can help us think through the problem:
In [584]: np.einsum('ij,jk->ik', A1,b)
Out[584]:
array([[ 4, 5],
[20, 29],
[36, 53]])
In [585]: np.einsum('ij,jk->ik', A2,b)
Out[585]:
array([[12, 17],
[28, 41],
[44, 65]])
A is (3,4), which won't work with the (2,2) b. Think of it as trying work with a doubled j dimension: 'i(2j),jk->i?k'. But what if we inserted an axis? 'imk,jk->imk'? Or added the extra dimension to i?
In [587]: np.einsum('imj,jk->imk', A.reshape(3,2,2),b)
Out[587]:
array([[[ 4, 5],
[12, 17]],
[[20, 29],
[28, 41]],
[[36, 53],
[44, 65]]])
The numbers are there, just the shape is (3,2,2).
In [590]: np.einsum('imj,jk->imk', A.reshape(3,2,2),b).reshape(3,4)
Out[590]:
array([[ 4, 5, 12, 17],
[20, 29, 28, 41],
[36, 53, 44, 65]])
Or you could build A from the start so that mij,jk->mik works (#Divaker)
#Psidom:
np.einsum('ij,jk->ik', A.reshape(3,2,2).reshape(-1,2) ,b).reshape(3,-1)
`#piRSquared':
'kj,jI->kI`
Shift you perspective. You are locking yourself into 3 x 2 unnecessarily.
You can think of A1 and A2 as 2x3 instead, then A would be
array([[ 0, 4, 8, 2, 6, 10],
[ 1, 5, 9, 3, 7, 11]])
Then take the transpose of b = b.T
array([[0, 4],
[1, 5]])
So that you can do you operation
b # A
array([[ 4, 20, 36, 12, 28, 44],
[ 5, 29, 53, 17, 41, 65]])
Let your "draws" look like this
A = np.random.randint(10, size=(2, 9))
A
array([[7, 2, 1, 0, 9, 9, 1, 0, 2],
[8, 6, 1, 6, 6, 2, 4, 2, 9]])
b # A
array([[32, 24, 4, 24, 24, 8, 16, 8, 36],
[47, 32, 6, 30, 39, 19, 21, 10, 47]])