Suppose I have an array a = array([a0, a1,..., an]). For every element in the array, I need to find all cumulative sums in the direction from left to right. Thus, for n=2 I have to find
a0, a0+a1, a0+a1+a2, a1, a1+a2, a2.
The obvious approach is the following:
list_of_sums = [np.cumsum(a[i:]) for i in xrange(n+1)]
I wonder if there exists a more efficient way to do this. The issue is that n may be quite large and in addition I want to broadcast this for many different arrays a of the same length. Unfortunately, I did not succeed to create a vectorized function for this.
Another problem is how to store all these sums. Instead of using list, we can represent all the data as 2D array A, where in the first column A[:,0] I have all the sums with a0 as the first term, in the second column A[:,1] I have all the sums with a1 as the first term, and so on. This gives us (n+1)x(n+1) matrix, half elements of which are zeros (the right lower triangle). On the other hand, this requires in 2 times more memory than list use, or force us to use sparse matrices that may be overkill.
Related
Given a two-dimensional array T of size NxN, filled with various natural numbers (They do not have to be sorted in any way as in the example below.). My task is to write a program that transforms the array in such a way that all elements lying above the main diagonal are larger than each element lying on the diagonal and all elements lying below the main diagonal are to be smaller than each element on the diagonal.
For example:
T looks like this:
[2,3,5][7,11,13][17,19,23] and one of the possible solutions is:
[13,19,23][3,7,17][5,2,11]
I have no clue how to do this. Would anyone have an idea what algorithm should be used here?
Let's say the matrix is NxN.
Put all N² values inside an array.
Sort the array with whatever method you prefer (ascending order).
In your final array, the (N²-N)/2 first values go below the diagonal, the following N values go to the diagonal, and the final (N²-N)/2 values go above the diagonal.
The following pseudo-code should do the job:
mat <- array[N][N] // To be initialized.
vec <- array[N*N]
for i : 0 to (N-1)
for j : 0 to (N-1)
vec[i*N+j]=mat[i][j]
next j
next i
sort(vec)
p_below <- 0
p_diag <- (N*N-N)/2
p_above <- (N*N+N)/2
for i : 0 to (N-1)
for j : 0 to (N-1)
if (i>j)
mat[i][j] = vec[p_above]
p_above <- p_above + 1
endif
if (i<j)
mat[i][j] = vec[p_below]
p_below <- p_below + 1
endif
if (i=j)
mat[i][j] = vec[p_diag]
p_diag <- p_diag + 1
endif
next j
next i
Code can be heavily optimized by sorting directly the matrix, using a (quite complex) custom sort operator, so it can be sorted "in place". Technically, you'll do a bijection between the matrix indices to a partitioned set of indices representing "below diagonal", "diagonal" and "above diagonal" indices.
But I'm unsure that it can be considered as an algorithm in itself, because it will be highly dependent on the language used AND on how you stored, internally, your matrix (and how iterators/indices are used). I could write one in C++, but I lack knownledge to give you such an operator in Python.
Obviously, if you can't use a standard sorting function (because it can't work on anything else but an array), then you can write your own with the tricky comparison builtin the algorithm.
For such small matrixes, even a bubble-sort can work properly, but obviously implementing at least a quicksort would be better.
Elements about optimizing:
First, we speak about the trivial bijection from matrix coordinate [x][y] to [i]: i=x+y*N. The invert is obviously x=floor(i/N) & y=i mod N. Then, you can parse the matrix as a vector.
This is already what I do in the first part initializing vec, BTW.
With matrix coordinates, it's easy:
Diagonal is all cells where x=y.
The "below" partition is everywhere x<y.
The "above" partition is everywhere x>y.
Look at coordinates in the below 3x3 matrix, it's quite evident when you know it.
0,0 1,0 2,0
0,1 1,1 2,1
0,2 1,2 2,2
We already know that the ordered vector will be composed of three parts: first the "below" partition, then the "diagonal" partition, then the "above" partition.
The next bijection is way more tricky, since it requires either a piecewise linear function OR a look-up table. The first requires no additional memory but will use more CPU power, the second use as much memory as the matrix but will require less CPU power.
As always, optimization for speed often cost memory. If memory is scarse because you use huge matrixes, then you'll prefer a function.
In order to shorten a bit, I'll explain only for "below" partition. In the vector, the (N-1) first elements will be the ones belonging to the first column. Then, we'll have (N-2) elements for the 2nd column, (N-3) for the third, until we had only 1 element for the (N-1)th column. You see the scheme, sum of the number of elements and the column (zero-based index) is always (N-1).
I won't write the function, because it's quite complex and, honestly, it won't help so much to understand. Simply know that converting from matrix indices to vector is "quite easy".
The opposite is more tricky and CPU-intensive, and it SHOULD use a (N-1) element vector to store where each column starts within the vector to GREATLY speed up the process. Thanks, this vector can also be used (from end to begin) for the "above" partition, so it won't burn too much memory.
Now, you can sort your "vector" normally, simply by chaining the two bijection together with the vector index, and you'll get a matrix cell instead. As long as the sorting algorithm is stable (that's usually the case), it will works and will sort your matrix "in place", at the expense of a lot of mathematical computing to "route" the linear indexes to matrix indexes.
Please note that, despite we speak about bijections, we need ONLY the "vector to matrix" formulas. The "matrix to vector" are important - it MUST be a bijection! - but you won't use them, since you'll sort directly the (virtual) vector from 0 to N²-1.
This problem is resulting from the spatial analysis of unstructured grids in 3D.
I have 2 2D arrays to compare, each with 3 columns for xyz coordinates.
One of the array is a reference, the other is evaluated against it (it is the result of CKde tree query against the reference array). In the end I want the number of matching row of the reference.
I have tried to find an array concatenation solution but I am lost in the different dimensions
reference=np.array([[0,1,33],[0,33,36],[0,2,36],[1, 33, 34]])
query= np.array([[0,1,33],[0,1,33],[1, 33, 34],[0,33,36],[0,33,36],[0,1,33],[0,33,36]])
Something in the style is where I am heading
filter=reference[:,:,None]==query.all(axis=0)
result = filter.sum(axis=1)
but I cannot find the right way of broadcasting to be able to compare the rows of the 2 arrays.
The result should be:
np.array([3,3,0,1])
You need to broadcast the two arrays. Since you cannot compare the 1D array directly, you first need to do a reduction using all on the last dimension. Then you can count the matched rows with sum sum. Here is the resulting code:
(reference[None,:,:] == query[:,None,:]).all(axis=2).sum(axis=0)
That being said, this solution is not the most efficient for bigger arrays. Indeed for m rows for size n in reference and k rows in query, the complexity of the solution is O(n m k) while the optimal solution is O(n m + n k). This can be achieved using hash maps (aka dict). The idea is to put rows of reference array in a hash map with associated values set to 0 and then for each value of query increase the value of the hash map with the key set to the row of query. One just need to iterate over the hash map to get the final array. Hash map accesses are done in (amortized) constant time. Unfortunately, Python dict does not support array as key since array cannot be hashed, but tuples can be. Here is an example:
counts = {tuple(row):0 for row in reference}
for row in query:
key = tuple(row)
if key in counts:
counts[key] += 1
print(list(counts.values()))
Which results in printing: [3, 3, 0, 1].
Note that the order is often not conserved in hash maps, but it should be ok for Python dict. Alternatively, one can use another hash map to rebuild the final array.
The resulting solution may be slower for small arrays, but it should be better for huge ones.
I have two lists of indices. I would like to generate the relevant permutation matrix. The two lists have equal size n and have all integers from 0 up to n-1.
Simple Example:
Given initial and final indices (as per the two-line convention https://en.wikipedia.org/wiki/Permutation_matrix):
initial_index = [3,0,2,1] and final_index = [0,1,3,2]
In other words, the last entry (3) has got to go to the first (0), the first (0) has got to go to the second (1) etc. You could also imagine zipping these two lists in order to obtain the permutation rules: [(3,0),(0,1),(2,3),(1,2)], read this as (3 -> 0),(0 -> 1) and so forth. This is a right-shift for a list, or a down-shift for a column vector. The resulting permutation matrix should be the following:
M = [[0,0,0,1],
[1,0,0,0],
[0,1,0,0],
[0,0,1,0]]
Multiplying this matrix by a column vector indeed shifts the entries down by 1, as required.
Are there any relevant operations that could achieve this efficiently?
You want an n-by-n matrix where, for every i from 0 to n-1, the cell at row final_index[i] and column initial_index[i] is set to 1, and every other cell is set to 0.
NumPy advanced indexing can be used to set those cells easily:
permutation_matrix = numpy.zeros((n, n), dtype=int)
permutation_matrix[final_index, initial_index] = 1
Alternatively to the good answer of #user2357112, you can use sparse matrices to be efficient in memory:
from scipy.sparse import csr_matrix
permutation_matrix = csr_matrix((np.ones(n, dtype=int), (final_index, initial_index)), shape=(n,n))
# Use permutation_matrix.todense() to convert the matrix if needed
The complexity of building this sparse matrix is O(n) in both time and space while for dense matrices it is O(n^2). So they are much better for big vectors (>1000).
I have a matrix NxM.
N is big enough N >> 10000.
I wonder if there is an algorithm to mix all the lines of a matrix to get a 100 matrix for example. My matrices C must not be identical.
Thoughts?
So, do you want to keep the shape of the matrix and just shuffle the rows or do you want to get subsets of the matrix?
For the first case I think the permutation algorithm from numpy could be your choice. Just create a permutation of a index list, like Souin propose.
For the second case just use the numpy choice funtion (also from the random module) without replacement if I understood your needs correctly.
For alpha and k fixed integers with i < k also fixed, I am trying to encode a sum of the form
where all the x and y variables are known beforehand. (this is essentially the alpha coordinate of a big iterated matrix-vector multiplication)
For a normal sum varying over one index I usually create a 1d array A and set A[i] equal to the i indexed entry of the sum then use sum(A), but in the above instance the entries of the innermost sum depend on the indices in the previous sum, which in turn depend on the indices in the sum before that, all the way back out to the first sum which prevents me using this tact in a straightforward manner.
I tried making a 2D array B of appropriate length and width and setting the 0 row to be the entries in the innermost sum, then the 1 row as the entries in the next sum times sum(np.transpose(B),0) and so on, but the value of the first sum (of row 0) needs to vary with each entry in row 1 since that sum still has indices dependent on our position in row 1, so on and so forth all the way up to sum k-i.
A sum which allows for a 'variable' filled in by each position of the array it's summing through would thusly do the trick, but I can't find anything along these lines in numpy and my attempts to hack one together have thus far failed -- my intuition says there is a solution that involves summing along the axes of a k-i dimensional array, but I haven't been able to make this precise yet. Any assistance is greatly appreciated.
One simple attempt to hard-code something like this would be:
for j0 in range(0,n0):
for j1 in range(0,n1):
....
Edit: (a vectorized version)
You could do something like this: (I didn't test it)
temp = np.ones(n[k-i])
for j in range(0,k-i):
temp = x[:n[k-i-1-j],:n[k-i-j]].T#(y[:n[k-i-j]]*temp)
result = x[alpha,:n[0]]#(y[:n[0]]*temp)
The basic idea is that you try to press it into a matrix-vector form. (note that this is python3 syntax)
Edit: You should note that you need to change the "k-1" to where the innermost sum is (I just did it for all sums up to index k-i)
This is 95% identical to #sehigle's answer, but includes a generic N vector:
def nested_sum(XX, Y, N, alpha):
intermediate = np.ones(N[-1], dtype=XX.dtype)
for n1, n2 in zip(N[-2::-1], N[:0:-1]):
intermediate = np.sum(XX[:n1, :n2] * Y[:n2] * intermediate, axis=1)
return np.sum(XX[alpha, :N[0]] * Y[:N[0]] * intermediate)
Similarly, I have no knowledge of the expression, so I'm not sure how to build appropriate tests. But it runs :\