I have an array like:
a = array([[1,2,3],[3,4,5],[4,5,6]])
What's the most efficient way to slice out a 1x2 array out of this that has only the first two columns of "a"?
i.e.
array([[2,3],[4,5],[5,6]]) in this case.
Two dimensional numpy arrays are indexed using a[i,j] (not a[i][j]), but you can use the same slicing notation with numpy arrays and matrices as you can with ordinary matrices in python (just put them in a single []):
>>> from numpy import array
>>> a = array([[1,2,3],[3,4,5],[4,5,6]])
>>> a[:,1:]
array([[2, 3],
[4, 5],
[5, 6]])
Is this what you're looking for?
a[:,1:]
To quote documentation, the basic slice syntax is i:j:k where i is the starting index, j is the stopping index, and k is the step (when k > 0).
Now if i is not given, it defaults to 0 if k > 0. Otherwise i defaults to n - 1 for k < 0 (where n is the length of the array).
If j is not given, it defaults to n (length of array).
That's for a one dimensional array.
Now a two dimensional array is a different beast. The slicing syntax for that is a[rowrange, columnrange].
So if you want all the rows, but just the last two columns, like in your case, you do:
a[0:3, 1:3]
Here, "[0:3]" means all the rows from 0 to 3. and "[1:3]" means all columns from column 1 to column 3.
Now as you may be wondering, even though you have only 3 columns and the numbering starts from 1, it must return 3 columns right? i.e: column 1, column 2, column 3
That is the tricky part of this syntax. The first column is actually column 0. So when you say "[1:3]", you are actually saying give me column 1 and column 2. Which are the last two columns you want. (There actually is no column 3.)
Now if you don't know how long your matrix is or if you want all the rows, you can just leave that part empty.
i.e.
a[:, 1:3]
Same goes for columns also. i.e if you wanted say, all the columns but just the first row, you would write
a[0:1,:]
Now, how the above answer a[:,1:] works is because when you say "[1:]" for columns, it means give me everything except for column 0, and till the end of all the columns. i.e empty means 'till the end'.
By now you must realize that anything on either side of the comma is all a subset of the one dimensional case I first mentioned above. i.e if you want to specify your rows using step sizes you can write
a[::2,1]
Which in your case would return
array([[2, 3],
[5, 6]])
i.e. a[::2,1] elucidates as: give me every other row, starting with the top most, and give me only the 2nd column.
This took me some time to figure out. So pasting it here, just in case it helps someone.
Related
Code:
import numpy as np
ray = [1,22,33,42,51], [61,71,812,92,103], [113,121,132,143,151], [16,172,183,19,201]
ray = np.asarray(ray)
type(ray)
ray[np.ix_([-2:],[3:4])]
I'd like to use index slicing and get a subarray consisting of the last two rows and the 3rd/4th columns. My current code produces an error:
I'd also like to sum each column. What am I doing wrong? I cannot post a picture because I need at least 10 reputation points.
So you want to make a slice of an array. The most straightforward way to do it is... slicing:
slice = ray[-2:,3:]
or if you want it explicitly
slice = ray[-2:,3:5]
See it explained in Understanding slicing
But if you do want to use np.ix_ for some reason, you need
slice = ray[np.ix_([-2,-1],[3,4])]
You can't use : here, because [] here don't make a slice, they construct lists and you should specify explicitly every row number and every column number you want in the result. If there are too many consecutive indices, you may use range:
slice = ray[np.ix_(range(-2, 0),range(3, 5))]
And to sum each column:
slice.sum(0)
0 means you want to reduce the 0th dimension (rows) by summation and keep other dimensions (columns in this case).
I have the following numpy array matrix:
example = np.array([[1.5525672727035909, 0.9550488599348534, 0.04495114006514658, -4757.845003575899, -4747.172432255857, 1],
[1.3050643768242065, 0.962214983713355, 0.03778501628664495, -5024.418466943938, -5013.745895623896, 2],
[1.3950687447554788, 0.9596091205211726, 0.040390879478827364, -4922.047207088476, -4911.374635768434, 3],
[1.2375603195101852, 0.9641693811074918, 0.035830618892508145, -5105.942048800849, -5095.269477480807, 4],
[1.2375603195101852, 0.9641693811074918, 0.035830618892508145, -5105.942048800849, -5095.269477480807, 5],
[1.2375597985998075, 0.9641693811074918, 0.035830618892508145, -5105.942048800849, -5095.269477480807, 6],
[1.2375597985998072, 0.9641693811074918, 0.035830618892508145, -5105.942048800849, -5095.269477480807, 7],
[1.215059487982556, 0.9648208469055375, 0.03517915309446254, -5134.107976656531, -5123.435405336489, 8],
[1.1250535573201497, 0.9674267100977199, 0.03257328990228013, -5252.243174800487, -5241.570603480445, 9],
[1.1250551200512835, 0.9674267100977199, 0.03257328990228013, -5252.243174800487, -5241.570603480445, 10]])
As you can see it consists in an array of different parameters in 10 different rows, (the last column is an index of the rows). What I am trying to do is get the row index of the one that meets better different criteria from different columns.
For instance, the closer to zero from column 1, the closer to 100 from column 2, the further from zero from column 3 and 4. In a way, I would like an optimisation that gets the index of the row that meets different criteria the best.
So far I have only got the select the rows for the columns based on the individual criteria, but I have not get to the point where it takes all the considerations together.
Thanks in advance.
As mentioned in the comment you need to be more specific about your cost function.
In general a quadratic error function is a good idea for reasons I won't go into here. In that case it's simply a case of doing the following:
goal = np.array([0,100, 0 ,0,0,0])
weight = np.array([1, 1, -1 , -1,0,0])
costs = np.sum(((example - goal)**2)*weight, axis = 1)
print(example[np.argmin(costs),-1])
goal defines the points that are the targets.
weight represents if we want to be close to them or far from them.
So to compute the cost you substract the goal from the example and you are left with distances. Next you square them (this gets rid of negative signs, alternatively you can take the absolute value) and multiply them by the weights. Finally you need to sum up the individual column costs to get the overall row cost to select the "cheapest" one. You can then pick out the row index (the last column)
An example is shown as follows:
>>> import numpy as np
>>> a=np.zeros((288,512))
>>> x1,x2,y1,y2=0,16,0,16
>>> p=a[x1:x2][y1:y2]
>>> p.shape
(16, 512)
>>> p=a[x1:x2,y1:y2]
>>> p.shape
I try to query a patch from an array, ranging from columns 0 to 16, and rows 0 to 16. I index the array in two ways and get very different result. a[x1:x2][y1:y2] gives me the wrong result.
Why?
Thx for helping me!!!
When you do a[x1:x2][y1:y2], you are slicing by rows twice. That is, a[x1:x2] will give you a shape (16,512). The second slice operation in a[x1:x2][y1:y2] is slicing the result of the first operation and will give you the same result.
In the second case, when you do a[x1:x2,y1:y2], you are slicing by the two dimensions of your 2-dimensional array.
Important note: If you have a 2-dimensional array and you slice like this:
a = np.zeros((10,15))
a[1:3].shape
Output:
(2, 15)
you will slice only by rows. Your resulting array will have 2 rows and the total number of columns (15 columns). If you want to slice by rows and columns, you will have to use a[1:3, 1:3].
The two methods of indexing you tried are not equivalent. In the first one (a[x1:x2][y1:y2]), you are essentially indexing the first axis twice. In the second, you are indexing the first and second axes.
a[x1:x2][y1:y2] can be rewritten as
p = a[x1:x2] # result still has two dimensions
p = p[y1:y2]
You are first indexing 0:16 in the first dimension. Then you index 0:16 in the first dimension of the result of the previous operation (which will simply return the same as a[x1:x2] because x1==y1 and x2==y2).
In the second method, you index the first and second dimensions directly. I would not write it this way, but one could write it like this to contrast it with the first method:
a[x1:x2][:, y1:y2]
Given an array defined below as:
a = np.arange(30).reshape((3, 10)
col_index = [[1,2,3,5], [3,4,5,7]]
row_index = [2,1]
Is it possible to index a[row_index, col_index], so I can do something like
a[row_index, col_index] =1, so then a becomes
[[0,1,2,3,4,5,6,7,8,9], [10,11,12,1,1,1,16,1,18,19], [20,1,1,1,24,1,26,27,28,29]]
So to clarify, in row 2, column 1,2,3, and 5 are set to one, and in row 1, column 3,4,5,7 is also set to 1.
Or (if you don't like typing)
a[np.c_[row_index], col_index] = 1
or even shorter but Python 2 only
a[zip(row_index), col_index] = 1
What all these solutions do is to make row and col indices broadcastable to each other. np.c_ is the column concatenation convenience object. It makes columns out of 1D objects.
zip used to do essentially the same. Only, since Python 3 it returns an iterator instead of a list and numpy can't handle those. (One could do list(zip(row_index)) but that's not short.)
I have a NumPy matrix that contains mostly non-zero values, but occasionally will contain a zero value. I need to be able to:
Count the non-zero values in each row and put that count into a variable that I can use in subsequent operations, perhaps by iterating through row indices and performing the calculations during the iterative process.
Count the non-zero values in each column and put that count into a variable that I can use in subsequent operations, perhaps by iterating through column indices and performing the calculations during the iterative process.
For example, one thing I need to do is to sum each row and then divide each row sum by the number of non-zero values in each row, reporting a separate result for each row index. And then I need to sum each column and then divide the column sum by the number of non-zero values in the column, also reporting a separate result for each column index. I need to do other things as well, but they should be easy after I figure out how to do the things that I am listing here.
The code I am working with is below. You can see that I am creating an array of zeros and then populating it from a csv file. Some of the rows will contain values for all the columns, but other rows will still have some zeros remaining in some of the last columns, thus creating the problem described above.
The last five lines of the code below are from another posting on this forum. These last five lines of code return a printed list of row/column indices for the zeros. However, I do not know how to use that resulting information to create the non-zero row counts and non-zero column counts described above.
ANOVAInputMatrixValuesArray=zeros([len(TestIDs),9],float)
j=0
for j in range(0,len(TestIDs)):
TestID=str(TestIDs[j])
ReadOrWrite='Read'
fileName=inputFileName
directory=GetCurrentDirectory(arguments that return correct directory)
inputfile=open(directory,'r')
reader=csv.reader(inputfile)
m=0
for row in reader:
if m<9:
if row[0]!='TestID':
ANOVAInputMatrixValuesArray[(j-1),m]=row[2]
m+=1
inputfile.close()
IndicesOfZeros = indices(ANOVAInputMatrixValuesArray.shape)
locs = IndicesOfZeros[:,ANOVAInputMatrixValuesArray == 0]
pts = hsplit(locs, len(locs[0]))
for pt in pts:
print(', '.join(str(p[0]) for p in pt))
Can anyone help me with this?
import numpy as np
a = np.array([[1, 0, 1],
[2, 3, 4],
[0, 0, 7]])
columns = (a != 0).sum(0)
rows = (a != 0).sum(1)
The variable (a != 0) is an array of the same shape as original a and it contains True for all non-zero elements.
The .sum(x) function sums the elements over the axis x. Sum of True/False elements is the number of True elements.
The variables columns and rows contain the number of non-zero (element != 0) values in each column/row of your original array:
columns = np.array([2, 1, 3])
rows = np.array([2, 3, 1])
EDIT: The whole code could look like this (with a few simplifications in your original code):
ANOVAInputMatrixValuesArray = zeros([len(TestIDs), 9], float)
for j, TestID in enumerate(TestIDs):
ReadOrWrite = 'Read'
fileName = inputFileName
directory = GetCurrentDirectory(arguments that return correct directory)
# use directory or filename to get the CSV file?
with open(directory, 'r') as csvfile:
ANOVAInputMatrixValuesArray[j,:] = loadtxt(csvfile, comments='TestId', delimiter=';', usecols=(2,))[:9]
nonZeroCols = (ANOVAInputMatrixValuesArray != 0).sum(0)
nonZeroRows = (ANOVAInputMatrixValuesArray != 0).sum(1)
EDIT 2:
To get the mean value of all columns/rows, use the following:
colMean = a.sum(0) / (a != 0).sum(0)
rowMean = a.sum(1) / (a != 0).sum(1)
What do you want to do if there are no non-zero elements in a column/row? Then we can adapt the code to solve such a problem.
A fast way to count nonzero elements per row in a scipy sparse matrix m is:
np.diff(m.tocsr().indptr)
The indptr attribute of a CSR matrix indicates the indices within the data corresponding to the boundaries between rows. So calculating the difference between each entry will provide the number of non-zero elements in each row.
Similarly, for the number of nonzero elements in each column, use:
np.diff(m.tocsc().indptr)
If the data is already in the appropriate form, these will run in O(m.shape[0]) and O(m.shape[1]) respectively, rather than O(m.getnnz()) in Marat and Finn's solutions.
If you need both row and column nozero counts, and, say, m is already a CSR, you might use:
row_nonzeros = np.diff(m.indptr)
col_nonzeros = np.bincount(m.indices)
which is not asymptotically faster than first converting to CSC (which is O(m.getnnz())) to get col_nonzeros, but is faster because of implementation details.
The faster way is to clone your matrix with ones instead of real values. Then just sum up by rows or columns:
X_clone = X.tocsc()
X_clone.data = np.ones( X_clone.data.shape )
NumNonZeroElementsByColumn = X_clone.sum(0)
NumNonZeroElementsByRow = X_clone.sum(1)
That worked 50 times faster for me than Finn Årup Nielsen's solution (1 second against 53)
edit:
Perhaps you will need to translate NumNonZeroElementsByColumn into 1-dimensional array by
np.array(NumNonZeroElementsByColumn)[0]
For sparse matrices, use the getnnz() function supported by CSR/CSC matrix.
E.g.
a = scipy.sparse.csr_matrix([[0, 1, 1], [0, 1, 0]])
a.getnnz(axis=0)
array([0, 2, 1])
(a != 0) does not work for sparse matrices (scipy.sparse.lil_matrix) in my present version of scipy.
For sparse matrices I did:
(i,j) = X.nonzero()
column_sums = np.zeros(X.shape[1])
for n in np.asarray(j).ravel():
column_sums[n] += 1.
I wonder if there is a more elegant way.