Related
I have a n x n dimensional numpy array of eigenvectors as columns, and want to return the last v of them as another array. However, they are currently in ascending order, and I wish to return them in descending order.
Currently, I'm attempting to index as follows
eigenvector_array[:,-1:-v]
But this doesn't seem to be working. Is there a more efficient way to do this?
Given a 2d array:
In [44]: x = np.arange(15).reshape(3,5);x
Out[44]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14]])
the last 3 columns:
In [45]: x[:,-3:]
Out[45]:
array([[ 2, 3, 4],
[ 7, 8, 9],
[12, 13, 14]])
and reversing their order:
In [46]: x[:,-3:][:,::-1]
Out[46]:
array([[ 4, 3, 2],
[ 9, 8, 7],
[14, 13, 12]])
Or we could reverse the order, and take the first n. x[:,::-1][:,:3]
I tried combining the selection and order, but getting the end points is trickier. Separating the operations is easier.
x[:,:-4:-1]
Lets re-write this to make it a little less confusing.
eigenvector_array[:,-1:-v]
to:
eigenvector_array[:][-1:-v]
Now remember how slicing works in python:
[start:stop:step]
If you set step. to -1 it will return them in reverse, so:
eigenvector_array[:,-1:-v:-1] should be your answer.
Suppose I have an array a of shape (2, 2, 2):
a = np.array([[[7, 9],
[19, 18]],
[[24, 5],
[18, 11]]])
and an array b that is the max of a: b=a.max(-1) (row-wise):
b = np.array([[9, 19],
[24, 18]])
I'd like to obtain the index of elements in b using index in flattened a, i.e. a.reshape(-1):
array([ 7, 9, 19, 18, 24, 5, 18, 11])
The result should be an array that is the same shape with b with indices of b in flattened a:
array([[1, 2],
[4, 6]])
Basically this is the result of maxpool2d when return_indices= True in pytorch, but I'm looking for an implementation in numpy. I've used where but it seems doesn't work, also is it possible to combine finding max and indices in one go, to be more efficient? Thanks for any help!
I have a solution similar to that of Andras based on np.argmax and np.arange. Instead of "indexing the index" I propose to add a piecewise offset to the result of np.argmax:
import numpy as np
a = np.array([[[7, 9],
[19, 18]],
[[24, 5],
[18, 11]]])
off = np.arange(0, a.size, a.shape[2]).reshape(a.shape[0], a.shape[1])
>>> off
array([[0, 2],
[4, 6]])
This results in:
>>> a.argmax(-1) + off
array([[1, 2],
[4, 6]])
Or as a one-liner:
>>> a.argmax(-1) + np.arange(0, a.size, a.shape[2]).reshape(a.shape[0], a.shape[1])
array([[1, 2],
[4, 6]])
The only solution I could think of right now is generating a 2d (or 3d, see below) range that indexes your flat array, and indexing into that with the maximum indices that define b (i.e. a.argmax(-1)):
import numpy as np
a = np.array([[[ 7, 9],
[19, 18]],
[[24, 5],
[18, 11]]])
multi_inds = a.argmax(-1)
b_shape = a.shape[:-1]
b_size = np.prod(b_shape)
flat_inds = np.arange(a.size).reshape(b_size, -1)
flat_max_inds = flat_inds[range(b_size), multi_inds.ravel()]
max_inds = flat_max_inds.reshape(b_shape)
I separated the steps with some meaningful variable names, which should hopefully explain what's going on.
multi_inds tells you which "column" to choose in each "row" in a to get the maximum:
>>> multi_inds
array([[1, 0],
[0, 0]])
flat_inds is a list of indices, from which one value is to be chosen in each row:
>>> flat_inds
array([[0, 1],
[2, 3],
[4, 5],
[6, 7]])
This is indexed into exactly according to the maximum indices in each row. flat_max_inds are the values you're looking for, but in a flat array:
>>> flat_max_inds
array([1, 2, 4, 6])
So we need to reshape that back to match b.shape:
>>> max_inds
array([[1, 2],
[4, 6]])
A slightly more obscure but also more elegant solution is to use a 3d index array and use broadcasted indexing into it:
import numpy as np
a = np.array([[[ 7, 9],
[19, 18]],
[[24, 5],
[18, 11]]])
multi_inds = a.argmax(-1)
i, j = np.indices(a.shape[:-1])
max_inds = np.arange(a.size).reshape(a.shape)[i, j, multi_inds]
This does the same thing without an intermediate flattening into 2d.
The last part is also how you can get b from multi_inds, i.e. without having to call a *max function a second time:
b = a[i, j, multi_inds]
This is a long one-liner
new = np.array([np.where(a.reshape(-1)==x)[0][0] for x in a.max(-1).reshape(-1)]).reshape(2,2)
print(new)
array([[1, 2],
[4, 3]])
However number = 18 is repeated twice; So which index is the target.
z = np.arange(15).reshape(3,5)
indexx = [0,2]
indexy = [1,2,3,4]
zz = []
for i in indexx:
for j in indexy:
zz.append(z[i][j])
Output:
zz >> [1, 2, 3, 4, 11, 12, 13, 14]
This essentially flattens the array but only keeping the elements that have indicies present in the two indices list.
This works, but it is very slow for larger arrays/list of indicies. Is there a way to speed this up using numpy?
Thanks.
Edited to show desired output.
A list of integers can be used to access the entries of interest for numpy arrays.
z[indexx][:,indexy].flatten()
x = {"apple", "banana", "cherry"}
y = {"google", "microsoft", "apple"}
z = x.intersection(y)
print(z)
z => apples
If I understand you correctly, just use Python set. And then cast it to list.
Indexing in several dimensions at once requires broadcasting the indices against each other. np.ix_ is a handy tool for doing this:
In [127]: z
Out[127]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14]])
In [128]: z[np.ix_(indexx, indexy)]
Out[128]:
array([[ 1, 2, 3, 4],
[11, 12, 13, 14]])
Converting that to 1d is a trivial ravel() task.
Look at the ix_ produces, here it's a (2,1) and (1,4) array. You can construct such arrays 'from-scratch':
In [129]: np.ix_(indexx, indexy)
Out[129]:
(array([[0],
[2]]),
array([[1, 2, 3, 4]]))
I am using Python and looking to iterate through each row of an Nx9 array and extract certain values from the row to form another matrix with them. The N value can change depending on the file I am reading but I have used N=3 in my example. I only require the 0th, 1st, 3rd and 4th values of each row to form into an array which I need to store. E.g:
result = np.array([[1, 2, 3, 4, 5, 6, 7, 8, 9],
[11, 12, 13, 14, 15, 16, 17, 18, 19],
[21, 22, 23, 24, 25, 26, 27, 28, 29]])
#Output matrix of first row should be: ([[1,2],[4,5]])
#Output matrix of second row should be: ([[11,12],[14,15]])
#Output matrix of third row should be: ([[21,22],[24,25]])
I should then end up with N number of matrices formed with the extracted values - a 2D matrix for each row. However, the matrices formed appear 3D so when transposed and subtracted I receive the error ValueError: operands could not be broadcast together with shapes (2,2,3) (3,2,2). I am aware that a (3,2,2) matrix cannot be subtracted from a (2,2,3) so how do I obtain a 2D matrix N number of times? Would a loop be better suited? Any suggestions?
import numpy as np
result = np.array([[1, 2, 3, 4, 5, 6, 7, 8, 9],
[11, 12, 13, 14, 15, 16, 17, 18, 19],
[21, 22, 23, 24, 25, 26, 27, 28, 29]])
a = result[:, 0]
b = result[:, 1]
c = result[:, 2]
d = result[:, 3]
e = result[:, 4]
f = result[:, 5]
g = result[:, 6]
h = result[:, 7]
i = result[:, 8]
output = [[a, b], [d, e]]
output = np.array(output)
output_transpose = output.transpose()
result = 0.5 * (output - output_transpose)
In [276]: result = np.array(
...: [
...: [1, 2, 3, 4, 5, 6, 7, 8, 9],
...: [11, 12, 13, 14, 15, 16, 17, 18, 19],
...: [21, 22, 23, 24, 25, 26, 27, 28, 29],
...: ]
...: )
...:
...: a = result[:, 0]
...
...: i = result[:, 8]
...: output = [[a, b], [d, e]]
In [277]: output
Out[277]:
[[array([ 1, 11, 21]), array([ 2, 12, 22])],
[array([ 4, 14, 24]), array([ 5, 15, 25])]]
In [278]: arr = np.array(output)
In [279]: arr
Out[279]:
array([[[ 1, 11, 21],
[ 2, 12, 22]],
[[ 4, 14, 24],
[ 5, 15, 25]]])
In [280]: arr.shape
Out[280]: (2, 2, 3)
In [281]: arr.T.shape
Out[281]: (3, 2, 2)
transpose exchanges the 1st and last dimensions.
A cleaner way to make a (N,2,2) array from selected columns is:
In [282]: arr = result[:,[0,1,3,4]].reshape(3,2,2)
In [283]: arr.shape
Out[283]: (3, 2, 2)
In [284]: arr
Out[284]:
array([[[ 1, 2],
[ 4, 5]],
[[11, 12],
[14, 15]],
[[21, 22],
[24, 25]]])
Since the last 2 dimensions are 2, you could transpose them, and take the difference:
In [285]: arr-arr.transpose(0,2,1)
Out[285]:
array([[[ 0, -2],
[ 2, 0]],
[[ 0, -2],
[ 2, 0]],
[[ 0, -2],
[ 2, 0]]])
Another way to get the (N,2,2) array is with a matrix index:
In [286]: result[:,[[0,1],[3,4]]]
Out[286]:
array([[[ 1, 2],
[ 4, 5]],
[[11, 12],
[14, 15]],
[[21, 22],
[24, 25]]])
Ok, this is not a coding problem, but a math problem. I wrote some code for you, since it's pretty obvious you're a beginner, so there will be some unfamiliar syntax in there that you should look into so you can avoid problems like this in the future. You might not use them all that often, but it's good to know how to do it, because it expands your understanding of python syntax in general.
First up, the complete code for easy copy and pasting:
import numpy as np
result=np.array([[1,2,3,4,5,6,7,8,9],
[11,12,13,14,15,16,17,18,19],
[21,22,23,24,25,26,27,28,29]])
output = np.array(tuple(result[:,i] for i in (0,1,3)))
def Matrix_Operation(Matrix,Coefficient):
if (Matrix.shape == Matrix.shape[::-1]
and isinstance(Matrix,np.ndarray)
and isinstance(Coefficient,float)):
return Coefficient*(Matrix-Matrix.transpose())
else:
print('The shape of you Matrix is not palindromic')
print('You cannot substitute matrices of unequal shape')
print('Your shape: %s'%str(Matrix.shape))
print(Matrix_Operation(output,0.5))
Now let's talk about a step by step explanation of what's happening here:
import numpy as np
result=np.array([[1,2,3,4,5,6,7,8,9],
[11,12,13,14,15,16,17,18,19],
[21,22,23,24,25,26,27,28,29]])
Python uses indentation (alignment of whitespaces) as an integral part of it's syntax. However, if you provide brackets, a lot of the time you don't need aligning indentations in order for the interpreter to understand your code. If you provide a large array of values manually, it is usually adviseable to start new lines at the commas (here, the commas separating the sublists). It's just more readable and that way your data isn't off screen in your coding program.
output = np.array(tuple(result[:,i] for i in (0,1,3)))
List comprehensions are a big deal in python and really handy for dirty one liners. As far as I know, no other language gives you this option. That's one of the reasons why python is so great. I basically created a list of lists, where each sublist is result[:,i] for every i in (0,1,3). This is cast as a tuple (yes, list comprehensions can also be done with tuples, not just lists). Finally I wrapped it in the np.array function, since this is the type required for our mathematical operations later on.
def Matrix_Operation(Matrix,Coefficient):
if (Matrix.shape == Matrix.shape[::-1]
and isinstance(Matrix,np.ndarray)
and isinstance(Coefficient,(float,int))):
return Coefficient*(Matrix-Matrix.transpose())
else:
print('The shape of you Matrix is not palindromic')
print('You cannot substitute matrices of unequal shape')
print('Your shape: %s'%str(Matrix.shape))
print(Matrix_Operation(output,0.5))
If you're gonna create a complex formula in python code, why not wrap it inside an abstractable function? You can incorporate a lot of "quality control" into a function as well, to check if it is given the correct input for the task it is supposed to do.
Your code failed, because you were trying to subtract a (2,2,3) shaped matrix from a (3,2,2) matrix. So we'll need a code snippet to check, if our provided matrix has a palindromic shape. You can reverse the order of items in a container by doing Container[::-1] and so we ask, if Matrix.shape == Matrix.shape[::-1]. Further, it is necessary, that our Matrix is a np.ndarray and if our coefficient is a number. That's what I'm doing with the isinstance() function. You can check for multiple types at once, which is why the isinstance(Coefficient,(float,int)) contains a tuple with both int and float in it.
Now that we have ensured that our input makes sense, we can preform our Matrix_Operation.
So in conclusion: Check if your math is solid before asking SO for help, because people here can get pretty annoyed at that sort of thing. You probably noticed by now that someone has already downvoted your question. Personally, I believe it's necessary to let newbies ask a couple stupid questions before they get into the groove, but that's what the voting button is for, I guess.
Ok, I know how to transpose a matrix, with for instance:
A = np.arange(25).reshape(5, 5)
print A
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24]])
A.T
array([[ 0, 5, 10, 15, 20],
[ 1, 6, 11, 16, 21],
[ 2, 7, 12, 17, 22],
[ 3, 8, 13, 18, 23],
[ 4, 9, 14, 19, 24]])
In the case of unidimensional arrays, it's not possible to use this ".T" tool (I don't know why, honestly) so to transpose a vector you have to change the paradigm and use, for instance:
B = np.arange(5)
print B
array([0, 1, 2, 3, 4])
and because B.T would give the same result, we, applying this change of paradigm, use:
B[ :, np.newaxis]
array([[0],
[1],
[2],
[3],
[4]])
and I find this change of paradigm a little bit antiesthetic because a 1-D vector is in no way a different entity to a 2-D vector (a matrix), in the sense that mathematically speaking they come from the same family and share many things.
My question is: is it possible to do this tranposition with the (sometimes called) jewel of the crown of numpy that is einsum, in a more compact and unifying way for every kind of tensor? I know that for a matrix you do
np.einsum('ij->ji', A)
and you get, as previosuly with A.T:
array([[ 0, 5, 10, 15, 20],
[ 1, 6, 11, 16, 21],
[ 2, 7, 12, 17, 22],
[ 3, 8, 13, 18, 23],
[ 4, 9, 14, 19, 24]])
is it possible to do it with 1-D arrays?
Thank you in advance.
Yes, you can transpose a 1D array using einsum
In [17]: B = np.arange(5)
In [35]: np.einsum('i,j->ji', np.ones(1), B)
Out[35]:
array([[ 0.],
[ 1.],
[ 2.],
[ 3.],
[ 4.]])
but that isn't really what einsum is for, since einsum is computing a sum of products. As you might expect, it is slower than simply adding a new axis.
In [36]: %timeit np.einsum('i,j->ji', np.ones(1), B)
100000 loops, best of 3: 5.43 µs per loop
In [37]: %timeit B[:, None]
1000000 loops, best of 3: 230 ns per loop
If you are looking for a single syntax for transposing 1D or 2D arrays here are two options:
Use np.atleast_2d(b).T:
In [39]: np.atleast_2d(b).T
Out[39]:
array([[0],
[1],
[2],
[3],
[4]])
In [40]: A = np.arange(25).reshape(5,5)
In [41]: np.atleast_2d(A).T
Out[41]:
array([[ 0, 5, 10, 15, 20],
[ 1, 6, 11, 16, 21],
[ 2, 7, 12, 17, 22],
[ 3, 8, 13, 18, 23],
[ 4, 9, 14, 19, 24]])
Use np.matrix:
In [44]: np.matrix(B).T
Out[44]:
matrix([[0],
[1],
[2],
[3],
[4]])
In [45]: np.matrix(A).T
Out[45]:
matrix([[ 0, 5, 10, 15, 20],
[ 1, 6, 11, 16, 21],
[ 2, 7, 12, 17, 22],
[ 3, 8, 13, 18, 23],
[ 4, 9, 14, 19, 24]])
A matrix is a subclass of ndarray. It is a specialized class which provides nice syntax for dealing with matrices and vectors. All matrix objects (both matrices and vectors) are 2-dimensional -- a vector is implemented as a 2D matrix with either a single column or a single row:
In [47]: np.matrix(B).shape # one row
Out[47]: (1, 5)
In [48]: np.matrix(B).T.shape # one column
Out[48]: (5, 1)
There are other differences between matrixs and ndarrayss. The *
operator computes matrix multiplication for matrixs, but performs
element-wise multiplication for ndarrays. Be sure to study the
differences if you use
np.matrix.
By the way, there is a certain beauty to the way NumPy defines transpose for ndarrays.
Remember that the nd in ndarray alludes to the fact that these objects can represent N-dimensional arrays. So whatever definition these objects use for .T must apply in N dimensions.
In particular, .T reverses the order of the axes.
In 2 dimensions, reversing the order of the axes coincides with matrix
transposition. In 1 dimension, the transpose does nothing -- reversing the
order of a single axis returns the same axis. The beautiful part is that this
definition works in N-dimensions.
The basic action of einsum is to iterate on all dimensions performing some sum of products. It takes short cuts in a few cases, even returning views if it can.
But I think it will complain about 'i->j' or 'i->ij'. You can't have indices on right that aren't already present on left.
einsum('i,j->ji',A,[1]) or some variant, might work. But it will be much slower.
In [19]: np.einsum('i,j->ij',[1,2,3],[1])
Out[19]:
array([[1],
[2],
[3]])
In [30]: %%timeit x=np.arange(1000)
...: y=x[:,None]
1000000 loops, best of 3: 439 ns per loop
In [31]: %%timeit x=np.arange(1000)
...: np.einsum('i,j->ij',x,[1])
100000 loops, best of 3: 15.3 µs per loop