Related
What is the equivalent of tf.math.segment_sum in numpy?
So basically I like to rewrite the exact same code in tf to np where I am using segment sum to group together certain elements using a segment_ids array and sum those segments. What is the equivalent code in numpy? I have an array and the segment_ids array and I like to perform segment_sum but in numpy.
You can create something pretty close to tf.math.segment_sum with the method numpy.add.at, which is the at method of the add ufunc:
def segment_sum(data, segment_ids):
data = np.asarray(data)
s = np.zeros((np.max(segment_ids)+1,) + data.shape[1:], dtype=data.dtype)
np.add.at(s, segment_ids, data)
return s
For example,
In [53]: c = np.array([[1, 2, 3, 4], [4, 3, 2, 1], [5, 6, 7, 8]])
In [54]: ids = [0, 0, 1]
In [55]: segment_sum(c, ids)
Out[55]:
array([[5, 5, 5, 5],
[5, 6, 7, 8]])
In [56]: x = [10, 20, 20, 30, 10, 0, 1, 2]
In [57]: xids = [1, 1, 0, 0, 2, 2, 2, 3]
In [58]: segment_sum(x, xids)
Out[58]: array([50, 30, 11, 2])
In [59]: w = np.arange(72).reshape(6, 2, 6) % 5
In [60]: w
Out[60]:
array([[[0, 1, 2, 3, 4, 0],
[1, 2, 3, 4, 0, 1]],
[[2, 3, 4, 0, 1, 2],
[3, 4, 0, 1, 2, 3]],
[[4, 0, 1, 2, 3, 4],
[0, 1, 2, 3, 4, 0]],
[[1, 2, 3, 4, 0, 1],
[2, 3, 4, 0, 1, 2]],
[[3, 4, 0, 1, 2, 3],
[4, 0, 1, 2, 3, 4]],
[[0, 1, 2, 3, 4, 0],
[1, 2, 3, 4, 0, 1]]])
In [61]: wids = [0, 0, 1, 2, 2, 2]
In [62]: segment_sum(w, wids)
Out[62]:
array([[[2, 4, 6, 3, 5, 2],
[4, 6, 3, 5, 2, 4]],
[[4, 0, 1, 2, 3, 4],
[0, 1, 2, 3, 4, 0]],
[[4, 7, 5, 8, 6, 4],
[7, 5, 8, 6, 4, 7]]])
I'm trying to mark the value and indices of max values in a 3D array, getting the max in the third axis.
Now this would have been obvious in a lower dimension:
argmaxes=np.argmax(array)
maximums=array[argmaxes]
but NumPy doesn't understand the second syntax properly for higher than 1D.
Let's say my 3D array has shape (8,8,250). argmaxes=np.argmax(array,axis=-1)would return a (8,8) array with numbers between 0 to 250. Now my expected output is an (8,8) array containing the maximum number in the 3rd dimension. I can achieve this with maxes=np.max(array,axis=-1) but that's repeating the same calculation twice (because I need both values and indices for later calculations)
I can also just do a crude nested loop:
for i in range(8):
for j in range(8):
maxes[i,j]=array[i,j,argmaxes[i,j]]
But is there a nicer way to do this?
You can use advanced indexing. This is a simpler case when shape is (8,8,3):
arr = np.random.randint(99, size=(8,8,3))
x, y = np.indices(arr.shape[:-1])
arr[x, y, np.argmax(array,axis=-1)]
Sample run:
>>> x
array([[0, 0, 0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3, 3, 3],
[4, 4, 4, 4, 4, 4, 4, 4],
[5, 5, 5, 5, 5, 5, 5, 5],
[6, 6, 6, 6, 6, 6, 6, 6],
[7, 7, 7, 7, 7, 7, 7, 7]])
>>> y
array([[0, 1, 2, 3, 4, 5, 6, 7],
[0, 1, 2, 3, 4, 5, 6, 7],
[0, 1, 2, 3, 4, 5, 6, 7],
[0, 1, 2, 3, 4, 5, 6, 7],
[0, 1, 2, 3, 4, 5, 6, 7],
[0, 1, 2, 3, 4, 5, 6, 7],
[0, 1, 2, 3, 4, 5, 6, 7],
[0, 1, 2, 3, 4, 5, 6, 7]])
>>> np.argmax(arr,axis=-1)
array([[2, 1, 1, 2, 0, 0, 0, 1],
[2, 2, 2, 1, 0, 0, 1, 0],
[1, 2, 0, 1, 1, 1, 2, 0],
[1, 0, 0, 0, 2, 1, 1, 0],
[2, 0, 1, 2, 2, 2, 1, 0],
[2, 2, 0, 1, 1, 0, 2, 2],
[1, 1, 0, 1, 1, 2, 1, 0],
[2, 1, 1, 1, 0, 0, 2, 1]], dtype=int64)
This is a visual example of array to help to understand it better:
I want to go from one array A of 10 elements to the array B of 100 elements.
Each element of B from 0 to 9 is equal to the element 0 of A
Each element of B from 10 to 19 is equal to the element 1 of A
....
Each element of B from 90 to 99 is equal to the element 9 of A
I did the following code but it does not work
a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
A = np.asarray(a)
b = []
for i in range(len(A)*10):
b.append(0)
B = np.asarray(b)
for i in range(len(A)):
for j in range(9):
B[j]=A[i]
Expected result:
B [ 0,0,0,0,0,0,0,0,0,0,
1,1,1,1,1,1,1,1,1,1,
2,2,2,2,2,2,2,2,2,2
...,
9,9,9,9,9,9,9,9,9,9 ]
You are saving values only in first 9 list elements. You have to 'scale' it by adding i*10 to index.
import numpy as np
a=[0, 1, 2, 3, 4, 5, 6, 7]
A = np.asarray(a)
b = []
for i in range(len(A)**2):
b.append(0)
B = np.asarray(b)
for i in range(len(A)):
for j in range(len(A)):
B[j + i*len(A)]=A[i]
print(B)
This works for me:
>>> a = [1,2,3]
>>> [ x for i in a for x in [i]*3]
[1, 1, 1, 2, 2, 2, 3, 3, 3]
>>>
You may replace 3 with 10 or whatever you like.
Answering the question from Jacob:
>>> [[a]*10 for a in A]
[[1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [2, 2, 2, 2, 2, 2, 2, 2, 2, 2], [3, 3, 3, 3, 3, 3, 3, 3, 3, 3]]
You should avoid loops with numpy whenever possible. It kind of defeats the point. Here you can just use repeat():
import numpy as np
a=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
A = np.asarray(a)
B = A.repeat(10)
B:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8,
8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9])
If want the a nested list, just reshape:
B = A.repeat(10).reshape(-1, 10)
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3],
[4, 4, 4, 4, 4, 4, 4, 4, 4, 4],
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5],
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6],
[7, 7, 7, 7, 7, 7, 7, 7, 7, 7],
[8, 8, 8, 8, 8, 8, 8, 8, 8, 8],
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]])
You can use numpy and specify how many iterations of each element you want:
import numpy as np
A = [1,2,3,4]
B = [np.full(10, a) for a in A]
print(B)
Or if you prefer to not use numpy, instead use:
A = [1,2,3,4]
B = [[a]*10 for a in A]
print(B)
Giving you the wanted list B
Try this:
a = [*range(10)]
b = []
for i in range(10):
b.extend([a[i]* 10])
B = np.asarray(b)
a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
b = []
for x in a:
b += [x] * 10
print b
This answer is better, idea from lenik
a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
b = [x for x in a for i in range(10)]
print b
Answer in a single line: print([item for sublist in [[i]*10 for i in range(1,10)] for item in sublist])
If a were a generic list and not an ordered sequence
In [20]: a = [1, 'a', 3.14159, False, {1:2, 3:4}]
you could do as follows
In [21]: [_ for _ in (zip(*(a for _ in a))) for _ in _]
Out[21]:
[1,
1,
1,
1,
1,
'a',
'a',
'a',
'a',
'a',
3.14159,
3.14159,
3.14159,
3.14159,
3.14159,
False,
False,
False,
False,
False,
{1: 2, 3: 4},
{1: 2, 3: 4},
{1: 2, 3: 4},
{1: 2, 3: 4},
{1: 2, 3: 4}]
list1=[]
list2=[]
for i in range (0,10,1):
list1.append(i)
print(list1)
for i in range (0,10,1):
for j in range (0,10,1):
j=i
list2.append(j)
print(list2)
A and B are Numpy arrays of common shape [n1,n2,n3]. The values of B are all integers in [0,n3). I want A to "invert" B in the sense that each value of A satisfies A[i,j,B[i,j,k]]=k for all i,j,k in the appropriate ranges. While it's obvious how to do this with for loops, I suspect that there is a clever one-liner using fancy indexing. Does anyone see it?
Here are two methods.
The first method is a one-liner: A = B.argsort(axis=-1)
Here's an example. B has shape (3, 5, 7) and for each fixed i and j, B[i,j,:] is a permutation of range(B.shape[2]).
In [386]: B
Out[386]:
array([[[1, 5, 4, 6, 2, 3, 0],
[6, 5, 3, 4, 2, 1, 0],
[4, 5, 0, 3, 1, 2, 6],
[0, 5, 6, 3, 2, 1, 4],
[4, 1, 5, 2, 6, 3, 0]],
[[2, 6, 0, 1, 5, 4, 3],
[3, 2, 4, 0, 1, 5, 6],
[3, 4, 6, 5, 1, 2, 0],
[4, 6, 3, 0, 2, 5, 1],
[0, 3, 1, 6, 4, 5, 2]],
[[0, 3, 6, 2, 1, 5, 4],
[3, 1, 2, 4, 6, 0, 5],
[1, 3, 5, 6, 4, 0, 2],
[4, 1, 6, 0, 2, 3, 5],
[6, 4, 5, 1, 0, 3, 2]]])
In [387]: A = B.argsort(axis=-1)
In [388]: A
Out[388]:
array([[[6, 0, 4, 5, 2, 1, 3],
[6, 5, 4, 2, 3, 1, 0],
[2, 4, 5, 3, 0, 1, 6],
[0, 5, 4, 3, 6, 1, 2],
[6, 1, 3, 5, 0, 2, 4]],
[[2, 3, 0, 6, 5, 4, 1],
[3, 4, 1, 0, 2, 5, 6],
[6, 4, 5, 0, 1, 3, 2],
[3, 6, 4, 2, 0, 5, 1],
[0, 2, 6, 1, 4, 5, 3]],
[[0, 4, 3, 1, 6, 5, 2],
[5, 1, 2, 0, 3, 6, 4],
[5, 0, 6, 1, 4, 2, 3],
[3, 1, 4, 5, 0, 6, 2],
[4, 3, 6, 5, 1, 2, 0]]])
Verify the desired property by sampling a few values.
In [389]: A[0, 0, B[0, 0, 0]]
Out[389]: 0
In [390]: A[0, 0, B[0, 0, 1]]
Out[390]: 1
In [391]: A[0, 0, B[0, 0, :]]
Out[391]: array([0, 1, 2, 3, 4, 5, 6])
In [392]: A[2, 3, B[2, 3, :]]
Out[392]: array([0, 1, 2, 3, 4, 5, 6])
The second method has a lower time complexity than using argsort, but it is a three-liner rather than a one-liner. I'll use the same B as above.
Create A, but with no values assigned yet.
In [393]: A = np.empty_like(B)
Create index arrays for each dimension of B.
In [394]: i, j, k = np.ogrid[[slice(n) for n in B.shape]] # or np.ix_(*[range(n) for n in B.shape])
This is the cool part. Do the assignment exactly as you wrote it in the question.
In [395]: A[i, j, B[i, j, k]] = k
Verify that we have the same A as above.
In [396]: A
Out[396]:
array([[[6, 0, 4, 5, 2, 1, 3],
[6, 5, 4, 2, 3, 1, 0],
[2, 4, 5, 3, 0, 1, 6],
[0, 5, 4, 3, 6, 1, 2],
[6, 1, 3, 5, 0, 2, 4]],
[[2, 3, 0, 6, 5, 4, 1],
[3, 4, 1, 0, 2, 5, 6],
[6, 4, 5, 0, 1, 3, 2],
[3, 6, 4, 2, 0, 5, 1],
[0, 2, 6, 1, 4, 5, 3]],
[[0, 4, 3, 1, 6, 5, 2],
[5, 1, 2, 0, 3, 6, 4],
[5, 0, 6, 1, 4, 2, 3],
[3, 1, 4, 5, 0, 6, 2],
[4, 3, 6, 5, 1, 2, 0]]])
After poking around some more on SO, I see that both these methods appear in answers to the question "How to invert a permutation array in numpy". The only thing really new here is doing the inversion along one axis of a three-dimensional array.
I have a 2D numpy array that represents a monochrome image from a CCD that has been binned 3x3 (that is, each value in the array represents 9 pixels (3x3) on the physical CCD).
I want to rescale it to match the original CCD layout (so I can easily overlay it with a non-binned image from the same CCD).
I saw Resampling a numpy array representing an image, but that doesn't seem to do what I want.
Suppose I have an array g:
import numpy as np
import scipy.ndimage
g = np.array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
When I try to scale it by a factor of 2:
o = scipy.ndimage.zoom(g, 2, order=0)
I get exactly what I expect - each value is now 2x2 identical values:
array([[0, 0, 1, 1, 2, 2],
[0, 0, 1, 1, 2, 2],
[3, 3, 4, 4, 5, 5],
[3, 3, 4, 4, 5, 5],
[6, 6, 7, 7, 8, 8],
[6, 6, 7, 7, 8, 8]])
But when I try to scale by a factor of 3, I get this:
o = scipy.ndimage.zoom(g, 3, order=0)
Gives me:
array([[0, 0, 1, 1, 1, 1, 2, 2, 2],
[0, 0, 1, 1, 1, 1, 2, 2, 2],
[3, 3, 4, 4, 4, 4, 5, 5, 5],
[3, 3, 4, 4, 4, 4, 5, 5, 5],
[3, 3, 4, 4, 4, 4, 5, 5, 5],
[3, 3, 4, 4, 4, 4, 5, 5, 5],
[6, 6, 7, 7, 7, 7, 8, 8, 8],
[6, 6, 7, 7, 7, 7, 8, 8, 8],
[6, 6, 7, 7, 7, 7, 8, 8, 8]])
I wanted each value in the original array to become a set of 3x3 values...that's not what I get.
How can I do it? (And why do I get this unintuitive result?)
You can use np.kron:
In [16]: g
Out[16]:
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
In [17]: np.kron(g, np.ones((3,3), dtype=int))
Out[17]:
array([[0, 0, 0, 1, 1, 1, 2, 2, 2],
[0, 0, 0, 1, 1, 1, 2, 2, 2],
[0, 0, 0, 1, 1, 1, 2, 2, 2],
[3, 3, 3, 4, 4, 4, 5, 5, 5],
[3, 3, 3, 4, 4, 4, 5, 5, 5],
[3, 3, 3, 4, 4, 4, 5, 5, 5],
[6, 6, 6, 7, 7, 7, 8, 8, 8],
[6, 6, 6, 7, 7, 7, 8, 8, 8],
[6, 6, 6, 7, 7, 7, 8, 8, 8]])
The output of zoom(g, 3, order=0) is a bit surprising. Consider the first row: [0, 0, 1, 1, 1, 1, 2, 2, 2]. Why are there four 1s?
When order=0 zoom (in effect) computes np.linspace(0, 2, 9), which looks like
In [80]: np.linspace(0, 2, 9)
Out[80]: array([ 0. , 0.25, 0.5 , 0.75, 1. , 1.25, 1.5 , 1.75, 2. ])
and then rounds the values. If you use np.round(), you get:
In [71]: np.round(np.linspace(0, 2, 9)).astype(int)
Out[71]: array([0, 0, 0, 1, 1, 1, 2, 2, 2])
Note that np.round(0.5) gives 0, but np.round(1.5) gives 2. np.round() uses the "round half to even" tie-breaking rule. Apparently the rounding done in the zoom code uses the "round half down" rule: it rounds 0.5 to 0 and 1.5 to 1, as in the following
In [81]: [int(round(x)) for x in np.linspace(0, 2, 9)]
Out[81]: [0, 0, 1, 1, 1, 1, 2, 2, 2]
and that's why there are four 1s in there.
And why do I get this unintuitive result?
Because zoom is a spline interpolation function. In other words, it draws a cubic spline from the midpoint of that 1 to the midpoint of that 0, and the values in between get the values of the spline at the appropriate location.
If you want nearest, linear or quadratic interpolation instead of cubic, you can use the order=0 or order=1 or order=2 argument. But if you don't want interpolation at all—which you don't—don't use an interpolation function. This is like asking why using [int(i*2.3) for i in range(10)] to get even numbers from 0 to 20 gives you some odd numbers. It's not a function to get even numbers from 0 to 20, so it doesn't do that, but it does exactly what you asked it to.
How can I do it?
Again, if you want non-interpolated scaling, don't use an interpolation function. The simplest way is probably to use np.kron, to Kroenecker-multiply your array with np.ones((scale, scale)).