This is a simple example.
Assume that I have an input tensor M. Now I have a tensor of indices of M with size 2 x 3 such as [[0, 1], [2,2], [0,1]] and a new array of values which is corresponding with the index tensor is [1, 2, 3]. I want to assign these values to the input M satisfying that the value is assigned to the element of M at index [0,1] will be the min value (1 in this example).
It means M[0,1] = 1 and M[2,2] = 2.
Can I do that by using some available functions in Pytorch without a loop?
It can be done without loops, but I am generally not sure whether it is such a great idea, due to significantly increased runtime.
The basic idea is relatively simple: Since tensor assignments always assign the last element, it is sufficient to sort your tuples in M in descending order, according to the respective values stored in the value list (let's call it v).
To do this in pytorch, let us consider the following example:
import torch as t
X = t.randn([3, 3]) # Random matrix of size 3x3
v = t.tensor([1, 2, 3])
M = t.tensor([[0, 2, 0],
[1, 2, 1]]) # accessing the elements described above
# Showcase pytorch's result with "naive" tensor assignment:
X[tuple(M)] = v # This would assign the value 3 to position (0, 1)
# To correct behavior, sort v in decreasing order.
v_desc = v.sort(decreasing=True)
# v now contains both the values and the indices of original position
print(v_desc)
# torch.return_types.sort(
# values=tensor([3, 2, 1]),
# indices=tensor([2, 1, 0]))
# Access M in the correct order:
M_desc = M[:, v_desc.indices]
# Finally assign correct order:
X[tuple(M_desc)] = v_desc
Again, this is relatively complicated, because it involves sorting the values, and "re-shuffling" of the tensors. You can surely save at least some memory if you perform operations in-place, which is something I disregarded for the sake of legibility.
As an answer whether this can also be achieved without sorting, I am fairly certain that the answer will be "no"; tensor assignments could only be done on fairly simple conditionals, but not something more complicated like your inter-dependent conditionals would require.
Related
I often need to convert a multi-column (or 2D) numpy array into an indicator vector in a stable (i.e., order preserved) manner.
For example, I have the following numpy array:
import numpy as np
arr = np.array([
[2, 20, 1],
[1, 10, 3],
[2, 20, 2],
[2, 20, 1],
[1, 20, 3],
[2, 20, 2],
])
The output I like to have is:
indicator = [0, 1, 2, 0, 3, 2]
How can I do this (preferably using numpy only)?
Notes:
I am looking for a high performance (vectorized) approach as the arr (see the example above) has millions of rows in a real application.
I am aware of the following auxiliary solutions, but none is ideal. It would be nice to hear expert's opinion.
My thoughts so far:
1. Numpy's unique: This would not work, as it is not stable:
arr_unq, indicator = np.unique(arr, axis=0, return_inverse=True)
print(arr_unq)
# output 1:
# [[ 1 10 3]
# [ 1 20 3]
# [ 2 20 1]
# [ 2 20 2]]
print(indicator)
# output 2:
# [2 0 3 2 1 3]
Notice how the indicator starts from 2. This is because unique function returns a "sorted" array (see output 1). However, I would like it to start from 0.
Of course I can use LabelEncoder from sklearn to convert the items in a manner that they start from 0 but I feel that there is a simple numpy trick that I can use and therefore avoid adding sklearn dependency to my program.
Or I can resolve this by a dictionary mapping like below, but I can imagine that there is a better or more elegant solution:
dct = {}
for idx, item in enumerate(indicator):
if item not in dct:
dct[item] = len(dct)
indicator[idx] = dct[item]
print(indicator)
# outputs:
# [0 1 2 0 3 2]
2. Stabilizing numpy's unique output: This solution is already posted in stackoverflow and correctly returns an stable unique array. But I do not know how to convert the returned indicator vector (returned when return_inverse=True) to represent the values in an stable order starting from 0.
3. Pandas's get_dummies: function. But it returns a "hot encoding" (matrix of indicator values). In contrast, I would like to have an indicator vector. It is indeed possible to convert the "hot encoding" to the indicator vector by few lines of code and data manipulation. But again that approach is not going to be highly efficient.
In addition to return_inverse, you can add the return_index option. This will tell you the first occurrence of each sorted item:
unq, idx, inv = np.unique(arr, axis=0, return_index=True, return_inverse=True)
Now you can use the fact that np.argsort is its own inverse to fix the order. Note that idx.argsort() places unq into sorted order. The corrected result is therefore
indicator = idx.argsort().argsort()[inv]
And of course the byproduct
unq = unq[idx.argsort()]
Of course there's nothing special about these operations to 2D.
A Note on the Intuition
Let's say you have an array x:
x = np.array([7, 3, 0, 1, 4])
x.argsort() is the index that tells you what elements of x are placed at each of the locations in the sorted array. So
i = x.argsort() # 2, 3, 1, 4, 0
But how would you get from np.sort(x) back to x (which is the problem you express in #2)?
Well, it happens that i tells you the original position of each element in the sorted array: the first (smallest) element was originally at index 2, the second at 3, ..., the last (largest) element was at index 0. This means that to place np.sort(x) back into its original order, you need the index that puts i into sorted order. That means that you can write x as
np.sort(x)[i.argsort()]
Which is equivalent to
x[i][i.argsort()]
OR
x[x.argsort()][x.argsort().argsort()]
So, as you can see, np.argsort is effectively its own inverse: argsorting something twice gives you the index to put it back in the original order.
I combined the lists like this allSpeed = np.concatenate((smallspeed, bigspeed)). I then sorted it and took the 100 smallest values. I now need to identify how many of each came from the original lists. Is this possible?
Yes, this is possible.
If you use argsort, you will get the indices of the sort order, e.g.
import numpy as np
a = np.array([1, 4, 0, 2])
print(np.argsort(a)) # => [2, 0, 3, 1]
Depending on whether you need the actual sorted values or just to know how many came from each array, you can work directly from the argsort result:
smallspeed = np.array([1, 3, 2, 5])
bigspeed = np.array([4, 0, 6]
all_speed = np.concatenate((smallspeed, bigspeed))
sort_order = np.argsort(all_speed)
lowest_4 = sort_order[:4] # or 100 for your case
small_count = np.count_nonzero(lowest_4 < len(smallspeed))
big_count = 4 - small_count
# or
big_count = np.count_nonzero(lowest_4 >= len(smallspeed))
Note that you will need to decide what it means if there are values that appear in both arrays and that value happens to be at the 100 value cutoff. This approach will give you an answer according to where each value came from, but that won't necessarily be meaningful. You will need to consider which sort algorithm you wish to use and which order you concatenate your arrays. For example, if you want to preferentially count "small speeds" in the lowest 100 when there are duplicates, then concatenate small + big (as you currently have), and use a stable sorting algorithm.
I want to find an index of each group duplicate value like this:
s = [2,6,2,88,6,...]
The results must return the index from original s: [[0,2],[1,4],..] or the result can show another way.
I find many solutions so I find the fastest way to get duplicate group:
s = np.sort(a, axis=None)
s[:-1][s[1:] == s[:-1]]
But after sort I got wrong index from original s.
In my case, I have ~ 200mil value on the list and I want to find the fastest way to do that. I use an array to store value because I want to use GPU to make it faster.
Using hash structures like dict helps.
For example:
import numpy as np
from collections import defaultdict
a=np.array([2,4,2,88,15,4])
table=defaultdict(list)
for ind,num in enumerate(a):
table[num]+=[ind]
Outputs:
{2: [0, 2], 4: [1, 5], 88: [3], 15: [4]}
If you want to show duplicated elements in the order from small to large:
for k,v in sorted(table.items()):
if len(v)>1:
print(k,":",v)
Outputs:
2 : [0, 2]
4 : [1, 5]
The speed is determined by how many different values in the number list.
See if this meets your performance requirements (here, s is your input array):
counts = np.bincount(s)
cum_counts = np.add.accumulate(counts)
sorted_inds = np.argsort(s)
result = np.split(sorted_inds, cum_counts[:-1])
Notes:
The result would be a list of arrays.
Each of these arrays would contain indices of a repeated value in s. Eg, if the value 13 is repeated 7 times in s, there would be an array with 7 indices among the arrays of result
If you want to ignore singleton values of s (values that occur only once in s), you can pass minlength=2 to np.bincount()
(This is a variation of my other answer. Here, instead of splitting the large array sorted_inds, we take slices from it, so it's likely to have a different kind of performance characteristic)
If s is the input array:
counts = np.bincount(s)
cum_counts = np.add.accumulate(counts)
sorted_inds = np.argsort(s)
result = [sorted_inds[:cum_counts[0]]] + [sorted_inds[cum_counts[i]:cum_counts[i+1]] for i in range(cum_counts.size-1)]
I learned on my web search that numpy.arange take less space than python range function. but i tried
using below it gives me different result.
import sys
x = range(1,10000)
print(sys.getsizeof(x)) # --> Output is 48
a = np.arange(1,10000,1,dtype=np.int8)
print(sys.getsizeof(a)) # --> OutPut is 10095
Could anyone please explain?
In PY3, range is an object that can generate a sequence of numbers; it is not the actual sequence. You may need to brush up on some basic Python reading, paying attention to things like lists and generators, and their differences.
In [359]: x = range(3)
In [360]: x
Out[360]: range(0, 3)
We have use something like list or a list comprehension to actually create those numbers:
In [361]: list(x)
Out[361]: [0, 1, 2]
In [362]: [i for i in x]
Out[362]: [0, 1, 2]
A range is often used in a for i in range(3): print(i) kind of loop.
arange is a numpy function that produces a numpy array:
In [363]: arr = np.arange(3)
In [364]: arr
Out[364]: array([0, 1, 2])
We can iterate on such an array, but it is slower than [362]:
In [365]: [i for i in arr]
Out[365]: [0, 1, 2]
But for doing things math, the array is much better:
In [366]: arr * 10
Out[366]: array([ 0, 10, 20])
The array can also be created from the list [361] (and for compatibility with earlier Py2 usage from the range itself):
In [376]: np.array(list(x)) # np.array(x)
Out[376]: array([0, 1, 2])
But this is slower than using arange directly (that's an implementation detail).
Despite the similarity in names, these shouldn't be seen as simple alternatives. Use range in basic Python constructs such as for loop and comprehension. Use arange when you need an array.
An important innovation in Python (compared to earlier languages) is that we could iterate directly on a list. We didn't have to step through indices. And if we needed indices along with with values we could use enumerate:
In [378]: alist = ['a','b','c']
In [379]: for i in range(3): print(alist[i]) # index iteration
a
b
c
In [380]: for v in alist: print(v) # iterate on list directly
a
b
c
In [381]: for i,v in enumerate(alist): print(i,v) # index and values
0 a
1 b
2 c
Thus you might not see range used that much in basic Python code.
the range type constructor creates range objects, which represent sequences of integers with a start, stop, and step in a space efficient manner, calculating the values on the fly.
np.arange function returns a numpy.ndarray object, which is essentially a wrapper around a primitive array. This is a fast and relatively compact representation, compared to if you created a python list, so list(range(N)), but range objects are more space efficient, and indeed, take constant space, so for all practical purposes, range(a) is the same size as range(b) for any integers a, b
As an aside, you should take care interpreting the results of sys.getsizeof, you must understand what it is doing. So do not naively compare the size of Python lists and numpy.ndarray, for example.
Perhaps whatever you read was referring to Python 2, where range returned a list. List objects do require more space than numpy.ndarray objects, generally.
arange store each individual value of the array while range store only 3 values (start, stop and step). That's the reason arange is taking more space compared to range.
As the question is about the size, this will be the answer.
But there are many advantages of using numpy array and arange than python lists for speed, space and efficiency perspective.
Suppose I have a matrix M and an indexing set idx=[(0,1),(2,3),(3,2)] and I want to create two sets of tuples, idx_leq1 consisting of those tuples whose first and second elements are both less than or equal to 1 and idx_geq2 consisting of those tuples whose first and second elements are both greater than or equal to 2.
I want to access the elements M[idx_leq1] and M[idx_geq2] cleanly. I have tried idx_leq1 = tuple([e for e in idx if e[0]<=1 and e[1]<=1]), but this returns idx_leq1 = ((0,1),) which I can't use to index M. On the other hand, idx_geq2 = tuple([e for e in idx if e[0]>=2 and e[1]>=2]) = ((2,3),(3,2)) works.
How can I solve this for the case where my first index set consists of only one coordinate pair? I don't want to do M[idx_leq1[0]].
I can do: list(chain(*[(e,) for e in idx if e[0]<=1 and e[1]<=1])) and list(chain(*[(e,) for e in idx if e[0]>=2 and e[1]>=2])), but then I still have to grab the first element for idx_leq1 whereas I can pass idx_geq2 to M and grab the appropriate elements.
Thanks!
[Tested with numpy.mat]
[M[0, 1]] should be fetched as in M[[0], [1]]. When indexing matrix, Multidimensional list-of-locations indexing requires k lists of index each working with one dimension.
For example, in order to fetch M[0, 3], M[1, 4], M[2, 5], one should use M[[0, 1, 2], [3, 4, 5]]. In other word, the index that you give to M should not be considered lists of coordinates. Rather, they are lists of "coordinates on each dimension".
In your case, a M[[0, 1]] (or its equivalent in tuple type) fetches M[0], M[1] as [0, 1] is considered to work on the first dimension, and the second dimension is broadcasted.
Ref: http://scipy-cookbook.readthedocs.io/items/Indexing.html#Multidimensional-list-of-locations-indexing
This reference believes that the reason to use "list of dims" rather than "list of coordinates" is to save number of instances, as unpacking many tuples might be expensive.