Forgive me for a vague title. I honestly don't know which title will suit this question. If you have a better title, let's change it so that it will be apt for the problem at hand.
The problem.
Let's say result is a 2D array and values is a 1D array. values holds some values associated with each element in result. The mapping of an element in values to result is stored in x_mapping and y_mapping. A position in result can be associated with different values. Now, I have to find the sum of the values grouped by associations.
An example for better clarification.
result array:
[[0, 0],
[0, 0],
[0, 0],
[0, 0]]
values array:
[ 1., 2., 3., 4., 5., 6., 7., 8.]
Note: Here result and values have the same number of elements. But it might not be the case. There is no relation between the sizes at all.
x_mapping and y_mapping have mappings from 1D values to 2D result. The sizes of x_mapping, y_mapping and values will be the same.
x_mapping - [0, 1, 0, 0, 0, 0, 0, 0]
y_mapping - [0, 3, 2, 2, 0, 3, 2, 1]
Here, 1st value(values[0]) have x as 0 and y as 0(x_mapping[0] and y_mappping[0]) and hence associated with result[0, 0]. If we are counting the number of associations, then element value at result[0,0] will be 2 as 1st value and 5th value are associated with result[0, 0]. If we are taking the sum, the result[0, 0] = value[0] + value[4] which is 6.
Current solution
# Initialisation. No connection with the solution.
result = np.zeros([4,2], dtype=np.int16)
values = np.linspace(start=1, stop=8, num=8)
y_mapping = np.random.randint(low=0, high=values.shape[0], size=values.shape[0])
x_mapping = np.random.randint(low=0, high=values.shape[1], size=values.shape[0])
# Summing the values associated with x,y (current solution.)
for i in range(values.size):
x = x_mapping[i]
y = y_mapping[i]
result[-y, x] = result[-y, x] + values[i]
The result,
[[6, 0],
[ 6, 2],
[14, 0],
[ 8, 0]]
Failed solution; But why?
test_result = np.zeros_like(result)
test_result[-y_mapping, x_mapping] = test_result[-y_mapping, x_mapping] + values # solution
To my surprise elements are overwritten in test_result. Values at test_result,
[[5, 0],
[6, 2],
[7, 0],
[8, 0]]
Question
1. Why, in the second solution, every element is overwritten?
As #Divakar has pointed out in the comment in his answer -
NumPy doesn't assign accumulated/summed values when the indices are repeated in test_result[-y_mapping, x_mapping] =. It randomly assigns from one of the instances.
2. Is there any Numpy way to do this? That is without looping? I'm looking for some speed optimization.
Approach #2 in #Divakar's answer gives me good results. For 23315 associations, for loop took 50 ms while Approach #1 took 1.85 ms. Beating all these, Approach #2 took 668 µs.
Side note
I'm using Numpy version 1.14.3 with Python 3.5.2 on an i7 processor.
Approach #1
Most intutive one would be with np.add.at for those repeated indices -
np.add.at(result, [-y_mapping, x_mapping], values)
Approach #2
We need to perform binned summations owing to the possible repeated nature of x,y indices. Hence, another way could be to use NumPy's binned summation func : np.bincount and have an implementation like so -
# Get linear index equivalents off the x and y indices into result array
m,n = result.shape
out_dtype = result.dtype
lidx = ((-y_mapping)%m)*n + x_mapping
# Get binned summations off values based on linear index as bins
binned_sums = np.bincount(lidx, values, minlength=m*n)
# Finally add into result array
result += binned_sums.astype(result.dtype).reshape(m,n)
If you are always starting off with a zeros array for result, the last step could be made more performant with -
result = binned_sums.astype(out_dtype).reshape(m,n)
I guess you were to write
y_mapping = np.random.randint(low=0, high=result.shape[0], size=values.shape[0])
x_mapping = np.random.randint(low=0, high=result.shape[1], size=values.shape[0])
With that correction, the code works for me as expected.
Related
I have a function in which I do some operations and want to speed it up with numba. In my code changing the values in an array with advanced indexing is not working. I think they do say that in the numba documents. But what is a workaround for like numpy.put()?
Here a short example what I want to do:
#example array
array([[ 0, 1, 2],
[ 0, 2, -1],
[ 0, 3, -1]])
changeing the values at given indexes with any method working in numba...to get:
changed values at:[0,0], [1,2], [2,1]
#changed example array by given indexes with one given value (10)
array([[ 10, 1, 2],
[ 0, 2, 10],
[ 0, 10, -1]])
Here what I did in python, but not working with numba:
indexList is a Tuple, which works with numpy.take()
This is the working example python code and the values in the array change to 100.
x = np.zeros((151,151))
print(x.ndim)
indexList=np.array([[0,1,3],[0,1,2]])
indexList=tuple(indexList)
def change(xx,filter_list):
xx[filter_list] = 100
return xx
Z = change(x,indexList)
Now using #jit on the function:
#jit
def change(xx,filter_list):
xx[filter_list] = 100
return xx
Z = change(x,indexList)
Compilation is falling back to object mode WITH looplifting enabled because Function "change" failed type inference due to: No implementation of function Function() found for signature: setitem(array(float64, 2d, C), UniTuple(array(int32, 1d, C) x 2), Literalint)
This error comes up. So I need a workaround for this. numpy.put() is not supported by numba.
I would be greatful for any ideas.
Thankyou
If it's not a problem for your to keep the indexList as an array you can use it in conjunction with for loops in the change function to make it compatible with numba:
indexList = np.array([[0,1,3],[0,1,2]]).T
#njit()
def change(xx, filter_list):
for y, x in filter_list:
xx[y, x] = 100
return xx
change(x, indexList)
Note that the indexList has to be transposed in order to have the y, x coordinates along the 1st axis. In other words, it has to have a shape of (n, 2) rather than (2, n) for the n points to be change. Effectively it's now a list of coordinates: [[0, 0],[1, 1],[3, 2]]
#mandulaj posted the way to go. Here a little different way I went before mandulaj gave his answer.
With this function I get a deprecation warning...so best way to go with #mandulaj and dont forget to transpose the indexList.
#jit
def change_arr(arr,idx,val): # change values in array by np index array to value
for i,item in enumerate(idx):
arr[item[0],item[1]]= val
return arr
Let's say I have 2D numpy array with 0 and 1 as values. I want to randomly pick an index that contains 1. Is there efficient way to do this using numpy?
I achieved it in pure python, but it's too slow.
Example input:
[[0, 1], [1, 0]]
output:
(0, 1)
EDIT:
For clarification: I want my function to get 2D numpy array with values belonging to {0, 1}. I want the output to be a tuple (2D index) of randomly (uniformly) picked value from the given array that is equal to 1.
EDIT2:
Using Paul H's suggestion, I came up with this:
nonzero = np.nonzero(a)
return random.choice(list(zip(nonzero)))
But it doesn't work with numpy's random choice, only with python's. Is there a way to optimise it better?
It's easier to get all the non-zero coordinates and sample from there:
xs,ys = np.where([[0, 1], [1, 0]])
# randomly pick a number:
idx = np.random.choice(np.arange(len(xs)) )
# output:
out = xs[idx], ys[idx]
You may try argwhere and permutation
a = np.array([[0, 1], [1, 0]])
b = np.argwhere(a)
tuple(np.random.permutation(b)[0])
I'm writing a code that compares fluxes of pixels on an astronomical map with the corresponding area on another one. Both maps are numpy arrays of data.
In order to do that, I need to transform pixel indexes on the first map (Av) to their equivalent on sky coordinates, then transform those sky coordinates to their pixel indexes equivalent on the second map (CO). Then, I scale the fluxes of the second map to match the values of the first map. After that, I have to keep handling the data.
The problem is that with thousands of pixels on the first map, the code is taking a really long time to finish doing what it's supposed to do, which is a hassle for troubleshooting. I've figured out that the slowest thing on this part of the code is the for loop.
Is there any way I can iterate through a numpy array, being able to work with the indexes and calculate data from every pixel, faster than a for loop? Is there a better way to do this at all?
In pseudocode, my code is something like this:
for pixel i,j in 1st map:
sky_x1,sky_y1 = pixel_2_skycoord(i,j)
i2,j2 = skycoord_2_pixel(sky_x1,sky_y1)
Avmap.append(Avflux[i,j])
COmap.append(COflux[i2,j2]*scale)
The actual code is:
for i in xrange(0,sAv_y-1):
for j in xrange(0,sAv_x-1):
if not np.isnan(Avdata[i,j]):
y,x=wcs.utils.skycoord_to_pixel(wcs.utils.pixel_to_skycoord(i,j,wAv,0),wcs=wCO)
x=x.astype(int)+0 #the zero is because i don't understand the problem with numpy but it fixes it anyway
y=y.astype(int)+0 #i couldn't get the number from an array with 1 value but adding zero resolves it somehow
COflux=COdata[x,y]
ylist.append(Avdata[i,j])
xlist.append(COflux*(AvArea/COArea))
The culprit here is the two for loops. Numpy has many functions that prevent the use of for loops to allow fast compiled code. The trick is to vectorize your code.
You can look into numpy's meshgrid function to convert this data into a vectorized form that you can then use something like this SO question to apply an arbitrary function to that vector.
Something along the lines of:
x_width = 15
y_width = 10
x, y = np.meshgrid(range(x_width), range(y_width))
def translate(x, y, x_o, y_o):
x_new = x + x_o
y_new = y + y_o
return x_new, y_new
x_new, y_new = translate(x, y, 3, 3)
x_new[4,5], y[4,5]
(8, 4)
You must avoid loops, and do the heavy computation in the underlying C code, in Numpy or in Astropy for the sky/pixel conversion. There are several options to do this with astropy.wcs.
The first one is with SkyCoord. Let's first create a grid of value for your pixel indices:
In [30]: xx, yy = np.mgrid[:5, :5]
...: xx, yy
Out[30]:
(array([[0, 0, 0, 0, 0],
[1, 1, 1, 1, 1],
[2, 2, 2, 2, 2],
[3, 3, 3, 3, 3],
[4, 4, 4, 4, 4]]), array([[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4]]))
Now we can create the SkyCoord object (which is a Numpy array subclass), from the pixels indices, and using the wcs:
In [33]: from astropy.coordinates import SkyCoord
...: sky = SkyCoord.from_pixel(xx, yy, wcs)
...: sky
Out[33]:
<SkyCoord (FK5: equinox=2000.0): (ra, dec) in deg
[[(53.17127889, -27.78771333), (53.17127889, -27.78765778),
(53.17127889, -27.78760222), (53.17127889, -27.78754667),
(53.17127889, -27.78749111)],
....
Note that this is using wcs.utils.skycoord_to_pixel. This object also has a method to project to pixel with a wcs. I will the same here for practical purpose:
In [34]: sky.to_pixel(wcs)
Out[34]:
(array([[ 0.00000000e+00, -1.11022302e-16, -2.22044605e-16,
-3.33066907e-16, 1.13149046e-10],
...
[ 4.00000000e+00, 4.00000000e+00, 4.00000000e+00,
4.00000000e+00, 4.00000000e+00]]),
array([[-6.31503738e-11, 1.00000000e+00, 2.00000000e+00,
3.00000000e+00, 4.00000000e+00],
...
[-1.11457732e-10, 1.00000000e+00, 2.00000000e+00,
3.00000000e+00, 4.00000000e+00]]))
We get a tuple of float values for the new x and y indices. So you will need to round these values and convert to int to use that as array indices.
The second option is to use the lower level functions, e.g. wcs.pixel_to_world_values and wcs.world_to_pixel_values, which takes Nx2 arrays and return this as well:
In [37]: wcs.pixel_to_world_values(np.array([xx.ravel(), yy.ravel()]).T)
Out[37]:
array([[ 53.17127889, -27.78771333],
[ 53.17127889, -27.78765778],
[ 53.17127889, -27.78760222],
[ 53.17127889, -27.78754667],
...
I have a problem vectorizing some code in pytorch.
A numpy solution would also help, but a pytorch solution would be better.
I'm going to use array and Tensor interchangeably.
The problem I am facing is this:
Given an 2D float array X of size (n, x), and a boolean 2D array A of size (n, n), compute the mean over rows in X indexed by rows in A.
The problem is that the rows in A contain a variable number of True indices.
Example (numpy):
import numpy as np
A = np.array([[0, 1, 0, 0, 0, 0],
[1, 0, 1, 0, 0, 0],
[0, 1, 0, 0, 0, 0],
[0, 0, 0, 0, 1, 0],
[0, 0, 0, 1, 0, 0],
[0, 1, 1, 1, 0, 0]])
X = np.arange(6 * 3, dtype=np.float32).reshape(6, 3)
# Compute the mean in numpy with a for loop
means_np = np.array([X[A.astype(np.bool)[i]].mean(axis=0) for i in np.arange(len(A)])
So this examples works, but this formulation has 3 problems:
The for loop is slow for larger A and X. I need to loop over a few 10 thousand indices.
It can happen that A[i] contains no True indices. This results in np.mean(np.array([])), which is NaN. I want this to be 0 instead.
Implementing it this way in pytorch results in SIGFPE (Floating point error) during the backwards pass of backpropagation through this function. The cause is when nothing being selected.
The workaround that I am using now is (also see code below):
Set the diagonal elements of A to True so that there is always at least one element to select
sum of all selected elements, subtract the values in X from that sum (the diagonal is guaranteed to be False in the beginning), and divide by the number of True elements - 1 clamped to at least 1 in each row.
This works, is differentiable in pytorch and does not produce NaN, but I still need a loop over all indices.
How can I get rid of this loop?
This is my current pytorch code:
import torch
A = torch.from_numpy(A).bytes()
X = torch.from_numpy(X)
A[np.diag_indices(len(A)] = 1 # Set the diagonal to 1
means = [(X[A[i]].sum(dim=0) - X[i]) / torch.clamp(A[i].sum() - 1, min=1.) # Compute the mean safely
for i in range(len(A))] # Get rid of the loop somehow
means = torch.stack(means)
I don't mind if your version looks completely different, as long as it is differentiable and produces the same result.
We can leverage matrix-multiplication -
c = A.sum(1,keepdims=True)
means_np = np.where(c==0,0,A.dot(X)/c)
We can optimize it further by converting A to float32 dtype if it's not already so and if the loss of precision is okay there, as shown below -
In [57]: np.random.seed(0)
In [58]: A = np.random.randint(0,2,(1000,1000))
In [59]: X = np.random.rand(1000,1000).astype(np.float32)
In [60]: %timeit A.dot(X)
10 loops, best of 3: 27 ms per loop
In [61]: %timeit A.astype(np.float32).dot(X)
100 loops, best of 3: 10.2 ms per loop
In [62]: np.allclose(A.dot(X), A.astype(np.float32).dot(X))
Out[62]: True
Thus, use A.astype(np.float32).dot(X) to replace A.dot(X).
Alternatively, to solve for the case where the row-sum is zero, and that requires us to use np.where, we could assign any non-zero value, say 1 into c and then simply divide by it, like so -
c = A.sum(1,keepdims=True)
c[c==0] = 1
means_np = A.dot(X)/c
This would also avoid the warning that we would otherwise get from np.where in those zero row sum cases.
I am working on a genetic algorithm code. I am fairly new to python.
My code snippet is as follows:
import numpy as np
pop_size = 10 # Population size
noi = 2 # Number of Iterations
M = 2 # Number of Phases in the Data
alpha = [np.random.randint(0, 64, size = pop_size)]* M
phi = [np.random.randint(0, 64, size = pop_size)]* M
reduced_tensor = [np.zeros((pop_size,3,3))]* M
for n_i in range(noi):
alpha_en = [(2*np.pi*alpha/63.00) for alpha in alpha]
phi_en = [(phi/63.00) for phi in phi]
for i in range(M):
for j in range(pop_size):
reduced_tensor[i][j] = [[1, 0, 0],
[0, phi_en[i][j], 0],
[0, 0, 0]]
Here I have a list of numpy arrays. The variable 'alpha' is a list containing two numpy arrays. How do I use list comprehension in this case? I want to create a similar list 'alpha_en' which operates on every element of alpha. How do I do that? I know my current code is wrong, it was just trial and error.
What does 'for alpha in alpha' mean (line 11)? This line doesn't give any error, but also doesn't give the desired output. It changes the dimension and value of alpha.
The variable 'reduced_tensor' is a list of an array of 3x3 matrix, i.e., four dimensions in total. How do I differentiate between the indexing of a list comprehension and a numpy array? I want to perform various operations on a list of matrices, in this case, assign the values of phi_en to one of the elements of the matrix reduced_tensor (as shown in the code). How should I do it efficiently? I think my current code is wrong, if not just confusing.
There some questionable programming in these 2 lines
alpha = [np.random.randint(0, 64, size = pop_size)]* M
...
alpha_en = [(2*np.pi*alpha/63.00) for alpha in alpha]
The first makes an array, and then makes a list with M pointers to the same thing. Note, M copies of the random array. If I were to change one element of alpha, I'd change them all. I don't see the point to this type of construction.
The [... for alpha in alpha] works because the 2 uses of alpha are different. At least in newer Pythons the i in [i*3 for i in range(3)] does not 'leak out' of the comprehension. That said, I would not approve of that variable naming. At the very least is it confusing to readers.
The arrays in alpha_en are separate. Values are derived from the array in alpha, but they are new.
for a in alphas:
a *= 2
would modify each array in alphas; how ever due to how alphas is constructed this ends up multiplying the array many times.
reduced_tensor = [np.zeros((pop_size,3,3))]* M
has the same problem; it's a list of M references to the same 3d array.
reduced_tensor[i][j]
references the i reference in that list, and the j 'row' of that array. I like to use
reduced_tensor[i][j,:,:]
to make it clearer to me and my reader the expected dimensions of the result.
The iteration over M does nothing for you; it just repeats the same assignment M times.
At the root of your problems is that use of list replication.
In [30]: x=[np.arange(3)]*3
In [31]: x
Out[31]: [array([0, 1, 2]), array([0, 1, 2]), array([0, 1, 2])]
In [32]: [id(i) for i in x]
Out[32]: [3036895536, 3036895536, 3036895536]
In [33]: x[0] *= 10
In [34]: x
Out[34]: [array([ 0, 10, 20]), array([ 0, 10, 20]), array([ 0, 10, 20])]