This problem is resulting from the spatial analysis of unstructured grids in 3D.
I have 2 2D arrays to compare, each with 3 columns for xyz coordinates.
One of the array is a reference, the other is evaluated against it (it is the result of CKde tree query against the reference array). In the end I want the number of matching row of the reference.
I have tried to find an array concatenation solution but I am lost in the different dimensions
reference=np.array([[0,1,33],[0,33,36],[0,2,36],[1, 33, 34]])
query= np.array([[0,1,33],[0,1,33],[1, 33, 34],[0,33,36],[0,33,36],[0,1,33],[0,33,36]])
Something in the style is where I am heading
filter=reference[:,:,None]==query.all(axis=0)
result = filter.sum(axis=1)
but I cannot find the right way of broadcasting to be able to compare the rows of the 2 arrays.
The result should be:
np.array([3,3,0,1])
You need to broadcast the two arrays. Since you cannot compare the 1D array directly, you first need to do a reduction using all on the last dimension. Then you can count the matched rows with sum sum. Here is the resulting code:
(reference[None,:,:] == query[:,None,:]).all(axis=2).sum(axis=0)
That being said, this solution is not the most efficient for bigger arrays. Indeed for m rows for size n in reference and k rows in query, the complexity of the solution is O(n m k) while the optimal solution is O(n m + n k). This can be achieved using hash maps (aka dict). The idea is to put rows of reference array in a hash map with associated values set to 0 and then for each value of query increase the value of the hash map with the key set to the row of query. One just need to iterate over the hash map to get the final array. Hash map accesses are done in (amortized) constant time. Unfortunately, Python dict does not support array as key since array cannot be hashed, but tuples can be. Here is an example:
counts = {tuple(row):0 for row in reference}
for row in query:
key = tuple(row)
if key in counts:
counts[key] += 1
print(list(counts.values()))
Which results in printing: [3, 3, 0, 1].
Note that the order is often not conserved in hash maps, but it should be ok for Python dict. Alternatively, one can use another hash map to rebuild the final array.
The resulting solution may be slower for small arrays, but it should be better for huge ones.
Related
Any efficient way to merge one tensor to another in Pytorch, but on specific indexes.
Here is my full problem.
I have a list of indexes of a tensor in below code xy is the original tensor.
I need to preserve the rows (those rows who are in indexes list) of xy and apply some function on elements other than those indexes (For simplicity let say the function is 'multiply them with two),
xy = torch.rand(100,4)
indexes=[1,2,55,44,66,99,3,65,47,88,99,0]
Then merge them back into the original tensor.
This is what I have done so far:
I create a mask tensor
indexes=[1,2,55,44,66,99,3,65,47,88,99,0]
xy = torch.rand(100,4)
mask=[]
for i in range(0,xy.shape[0]):
if i in indexes:
mask.append(False)
else:
mask.append(True)
print(mask)
import numpy as np
target_mask = torch.from_numpy(np.array(mask, dtype=bool))
print(target_mask.sum()) #output is 89 as these are element other than preserved.
Apply the function on masked rows
zy = xy[target_mask]
print(zy)
zy=zy*2
print(zy)
Code above is working fine and posted here to clarify the problem
Now I want to merge tensor zy into xy on specified index saved in the list indexes.
Here is the pseudocode I made, as one can see it is too complex and need 3 for loops to complete the task. and it will be too much resources wastage.
# pseudocode
for masked_row in indexes:
for xy_rows_index in xy:
if xy_rows_index= masked_row
pass
else:
take zy tensor row and replace here #another loop to read zy.
But I am not sure what is an efficient way to merge them, as I don't want to use NumPy or for loop etc. It will make the process slow, as the original tensor is too big and I am going to use GPU.
Any efficient way in Pytorch for this?
Once you have your mask you can assign updated values in place.
zy = 2 * xy[target_mask]
xy[target_mask] = zy
As for acquiring the mask I don't see a problem necessarily with your approach, though using the built-in set operations would probably be more efficient. This also gives an index tensor instead of a mask, which, depending on the number of indices being updated, may be more efficient.
i = list(set(range(len(xy)))-set(indexes))
zy = 2 * xy[i]
xy[i] = zy
Edit:
To address the comment, specifically to find the complement of indices of i we can do
i_complement = list(set(range(len(xy)))-set(i))
However, assuming indexes contains only values between 0 and len(xy)-1 then we could equivalently use i_complement = len(set(indexes)), which just removes the repeated values in indexes.
I have a dictionary of 2D arrays and I would like to normalize each row of each 2D array by its mean.
I have:
for key, value in sorted(baseline.items()):
for i in baseline[str(key)]:
i = i / np.mean(i)
Where:
baseline is a dict
baseline[str(key)] is a 2D numpy array
i is a 1D array
print(i) results in the appropriately updated values, however the individual rows across baseline.items() do not get updated.
What am I missing?
First of all, here is a solution:
for i in baseline.values():
i /= i.mean(axis=1, keepdims=True)
Now as to why. The loop for i in baseline[key]: binds a view into the row of a 2D array to the name i at each iteration. You don't need str(key) because the outer loop ensures that the keys are correct. In fact, avoid transforming the keys unnecessarily to avoid surprises, like if you accidentally get an integer key.
The line i = i / np.mean(i) does not do in-place division of the array by its mean. It computes the array i / np.mean(i), then rebinds the name i to the new array. The new array is then discarded on the next iteration.
You can fix this by re-assigning into the slice that i represents:
i[:] = i / np.mean(i)
Alternatively, you can perform the division in-place using the correct operator:
i /= np.mean(i)
As you can see in my solution, there is no need to iterate over the rows at all. np.mean is a vectorized function that can operate along any axis of an array. By setting keepdims=True, you ensure that the result has the right shape to be broadcasted right back over the original when you divide them.
A less flexible alternative to i.mean(axis=1, keepdims=True) specific for 2D arrays is
i.mean(axis=1)[:, None]
I have 1-dimensional numpy array and want to store sparse updates of it.
Say I have array of length 500000 and want to do 100 updates of 100 elements. Updates are either adds or just changing the values (I do not think it matters).
What is the best way to do it using numpy?
I wanted to just store two arrays: indices, values_to_add and therefore have two objects: one stores dense matrix and other just keeps indices and values to add, and I can just do something like this with the dense matrix:
dense_matrix[indices] += values_to_add
And if I have multiple updates, I just concat them.
But this numpy syntax doesn't work fine with repeated elements: they are just ignored.
Updating pair when we have an update that repeats index is O(n). I thought of using dict instead of array to store updates, which looks fine from the point of view of complexity, but it doesn't look good numpy style.
What is the most expressive way to achieve this? I know about scipy sparse objects, but (1) I want pure numpy because (2) I want to understand the most efficient way to implement it.
If you have repeated indices you could use at, from the documentation:
Performs unbuffered in place operation on operand ‘a’ for elements
specified by ‘indices’. For addition ufunc, this method is equivalent
to a[indices] += b, except that results are accumulated for elements
that are indexed more than once.
Code
a = np.arange(10)
indices = [0, 2, 2]
np.add.at(a, indices, [-44, -55, -55])
print(a)
Output
[ -44 1 -108 3 4 5 6 7 8 9]
Suppose I have an array a = array([a0, a1,..., an]). For every element in the array, I need to find all cumulative sums in the direction from left to right. Thus, for n=2 I have to find
a0, a0+a1, a0+a1+a2, a1, a1+a2, a2.
The obvious approach is the following:
list_of_sums = [np.cumsum(a[i:]) for i in xrange(n+1)]
I wonder if there exists a more efficient way to do this. The issue is that n may be quite large and in addition I want to broadcast this for many different arrays a of the same length. Unfortunately, I did not succeed to create a vectorized function for this.
Another problem is how to store all these sums. Instead of using list, we can represent all the data as 2D array A, where in the first column A[:,0] I have all the sums with a0 as the first term, in the second column A[:,1] I have all the sums with a1 as the first term, and so on. This gives us (n+1)x(n+1) matrix, half elements of which are zeros (the right lower triangle). On the other hand, this requires in 2 times more memory than list use, or force us to use sparse matrices that may be overkill.
I have a master array of length n of id numbers that apply to other analogous arrays with corresponding data for elements in my simulation that belong to those id numbers (e.g. data[id]). Were I to generate a list of id numbers of length m separately and need the information in the data array for those ids, what is the best method of getting a list of indices idx of the original array of ids in order to extract data[idx]? That is, given:
a=numpy.array([1,3,4,5,6]) # master array
b=numpy.array([3,4,3,6,4,1,5]) # secondary array
I would like to generate
idx=numpy.array([1,2,1,4,2,0,3])
The array a is typically in sequential order but it's not a requirement. Also, array b will most definitely have repeats and will not be in any order.
My current method of doing this is:
idx=numpy.array([numpy.where(a==bi)[0][0] for bi in b])
I timed it using the following test:
a=(numpy.random.uniform(100,size=100)).astype('int')
b=numpy.repeat(a,100)
timeit method1(a,b)
10 loops, best of 3: 53.1 ms per loop
Is there a better way of doing this?
The current way you are doing it with where searching through the whole array of a each time. You can make this look-up O(1) instead of O(N) using a dict. For instance, I used the following method:
def method2(a,b):
tmpdict = dict(zip(a,range(len(a))))
idx = numpy.array([tmpdict[bi] for bi in b])
and got a very large speed-up which will be even better for larger arrays. For the sizes that you had in your example code, I got a speed-up of 15x. The only problem with my code is that if there are repeated elements in a, then the dict will currently point to the last instance of the element while with your method it will point to the first instance. However, that can remedied if there are to be repeated elements in the actual usage of the code.
I'm not sure if there is a way to do this automatically in python, but you're probably best off sorting the two arrays and then generating your output in one pass through b. The complexity of that operation should be O(|a|*log|a|)+O(|b|*log|b|)+O(|b|) = O(|b|*log|b|) (assuming |b| > |a|). I believe your original try has complexity O(|a|*|b|), so this should provide a noticeable improvement for a sufficiently large b.