Related
There are two arrays and I want to get distance between two arrays based on known individual elements distance.
dist = {(4,3): 0.25, (4,1):0.75, (0,0):0, (3,3):0, (2,1):0.25, (1,0): 0.25}
a = np.array([[4, 4, 0], [3, 2, 1]])
b = np.array([[3, 1, 0]])
a
array([[4, 4, 0],
[3, 2, 1]])
b
array([[3, 1, 0]])
expected output based on dictionary dist:
array([[0.25, 0.75, 0. ],
[0. , 0.25, 0.25]])
So, if we need which elements are different we can do a!=b. Similarly, instead of !=, I want to apply the below function -
def get_distance(a, b):
return dist[(a, b)]
to get the expected output above.
I tried np.vectorize(get_distance)(a, b) and it works. But I am not sure if it is the best way to do the above in vectorized way. So, for two numpy arrays, what is the best way to apply custom function/operator?
Instead of storing your distance mapping as a dict, use a np.array for lookup (or possibly a sparse matrix if size becomes an issue).
d = np.zeros((5, 4))
for (x, y), z in dist.items():
d[x, y] = z
Then, simply index.
>>> d[a, b]
array([[0.25, 0.75, 0. ],
[0. , 0.25, 0.25]])
For a sparse solution (code is almost identical):
In [14]: from scipy import sparse
In [15]: d = sparse.dok_matrix((5, 4))
In [16]: for (x, y), z in dist.items():
...: d[x, y] = z
...:
In [17]: d[a, b].A
Out[17]:
array([[0.25, 0.75, 0. ],
[0. , 0.25, 0.25]])
I was running a simple experiment when I noticed a difference between R's and Python's FFT.
First, Python:
import numpy as np
from pyfftw.interfaces.numpy_fft import fft
a = np.array([1, 2, 3])
fft(a)
>> array([ 6. +0.j , -1.5+0.8660254j, -1.5-0.8660254j])
b = np.array([[1, 2, 3], [4, 5, 6]])
fft(b)
>> array([[ 6. +0.j , -1.5+0.8660254j, -1.5-0.8660254j],
[15. +0.j , -1.5+0.8660254j, -1.5-0.8660254j]])```
Now R:
> a = c(1, 2, 3)
> fft(a)
[1] 6.0+0.000000i -1.5+0.866025i -1.5-0.866025i
> b = rbind(c(1, 2, 3), c(4, 5, 6))
> fft(b)
[,1] [,2] [,3]
[1,] 21+0i -3+1.732051i -3-1.732051i
[2,] -9+0i 0+0.000000i 0+0.000000i
I notice that the first row of the R result corresponds to the element-wise sum of the first and second row of the Python result, whereas the second row of the R result corresponds to the subtraction.
What am I doing wrong? I run the same experiment using np.matrix and R matrix, but got the same results. Which one should be the correct result when applying the FFT to a matrix or multidimensional array?
Following the suggestion in the comments, I did the following:
from pyfftw.interfaces.numpy_fft import fftn
b = np.array([[1, 2, 3], [4, 5, 6]])
fftn(b)
>> array([[21.+0.j , -3.+1.73205081j, -3.-1.73205081j],
[-9.+0.j , 0.+0.j , 0.+0.j ]])
which works with fft2 too.
Indeed, Python FFT is 1D (over each row) unless fft2 or fftn are used.
I want to find the two find the pair of values and their index number in a meshgrid that a closets to another pair of values. Suppose I have two vectors a= np.array([0.01,0.5,0.9]) and b = np.array([0,3,6,10]) and two meshgrids X,Y = np.meshgrid(a,b). For illustration, they look as follows:
X= array([[ 0.1, 0.5, 0.9],
[ 0.1, 0.5, 0.9],
[ 0.1, 0.5, 0.9],
[ 0.1, 0.5, 0.9]])
Y =array([[ 0, 0, 0],
[ 3, 3, 3],
[ 6, 6, 6],
[10, 10, 10]])
Now, I have another array called c of dimension (2 x N). For illustration suppose c contains the following entries:
c = array([[ 0.07268017, 0.08816632, 0.11084398, 0.13352165, 0.1490078 ],
[ 0.00091219, 0.00091219, 0.00091219, 0.00091219, 0.00091219]])
Denote a column vector of c by x. For each vector x I want to find
To complicate matters a bit, I am in fact not only looking for the index with the smallest distance (i,j) but also the second smallest distance (i',j').
All my approaches so far turned out to be extremely complicated and involved a lot of side routes. Does someone have an idea for how to tackle the problem efficiently?
If X, Y always come from meshgrid(), your minimization is separable in X and Y. Just find the closest elements of X to c[0,] and the closest elements of Y to c[1,] ---
you don't need to calculate the 2-dimensional metric.
If either a or b have uniform steps, you can save yourself even more time if you scale the corresponding values of c onto the indexes. In your example, all(a == 0.1+0.4*arange(3)), so you can find the x values by inverting: x = (c[0,] - 0.1)/0.4. If you have an invertible (possibly non-linear) function that maps integers onto b, you can similarly find y values directly by applying the inverse function to c[1,].
This is more a comment than an answer but i like to [... lots of stuff mercifully deleted, that you can still see using the revision history ...]
Complete Revision
As a followup of my own comment, please look at the following
Setup
In [25]: from numpy import *
In [26]: from scipy.spatial import KDTree
In [27]: X= array([[ 0.1, 0.5, 0.9],
[ 0.1, 0.5, 0.9],
[ 0.1, 0.5, 0.9],
[ 0.1, 0.5, 0.9]])
In [28]: Y =array([[ 0, 0, 0],
[ 3, 3, 3],
[ 6, 6, 6],
[10, 10, 10]])
In [29]: c = array([[ 0.07268017, 0.08816632, 0.11084398, 0.13352165, 0.1490078 ],
[ 0.00091219, 0.00091219, 0.00091219, 0.00091219, 0.00091219]])
Solution
Two lines of code, please notice that you have to pass the transpose of your c array.
In [30]: tree = KDTree(zip(X.ravel(), Y.ravel()))
In [31]: tree.query(c.T,k=2)
Out[31]:
(array([[ 0.02733505, 0.4273208 ],
[ 0.01186879, 0.41183469],
[ 0.01088228, 0.38915709],
[ 0.03353406, 0.36647949],
[ 0.04901629, 0.35099339]]), array([[0, 1],
[0, 1],
[0, 1],
[0, 1],
[0, 1]]))
Comment
To interpret the result, the excellent scipy docs inform you that tree.query() gives you back two arrays, containing respectively for each point in c
a scalar or an array of length k>=2 giving you the distances
from the point to the closest point on grid, the second closest, etc,
a scalar or an array of length k>=2 giving you the indices
pointing to the grid point(s) closest (next close etc).
To access the grid point, KDTree maintains a copy of the grid data, e.g
In [32]: tree.data[[0,1]]
Out[32]:
array([[ 0.1, 0. ],
[ 0.5, 0. ]])
where [0,1] is the first element of the second output array.
Should you need the indices of the closest(s) point in the mesh matrices, it simply a matter of using divmod.
I have come across the numpy.apply_along_axis function in some code. And I don't understand the documentation about it.
This is an example of the documentation:
>>> def new_func(a):
... """Divide elements of a by 2."""
... return a * 0.5
>>> b = np.array([[1,2,3], [4,5,6], [7,8,9]])
>>> np.apply_along_axis(new_func, 0, b)
array([[ 0.5, 1. , 1.5],
[ 2. , 2.5, 3. ],
[ 3.5, 4. , 4.5]])
As far I as thought I understood the documentation, I would have expected:
array([[ 0.5, 1. , 1.5],
[ 4 , 5 , 6 ],
[ 7 , 8 , 9 ]])
i.e. having applied the function along the axis [1,2,3] which is axis 0 in [[1,2,3], [4,5,6], [7,8,9]]
Obviously I am wrong. Could you correct me ?
apply_along_axis applies the supplied function along 1D slices of the input array, with the slices taken along the axis you specify. So in your example, new_func is applied over each slice of the array along the first axis. It becomes clearer if you use a vector valued function, rather than a scalar, like this:
In [20]: b = np.array([[1,2,3], [4,5,6], [7,8,9]])
In [21]: np.apply_along_axis(np.diff,0,b)
Out[21]:
array([[3, 3, 3],
[3, 3, 3]])
In [22]: np.apply_along_axis(np.diff,1,b)
Out[22]:
array([[1, 1],
[1, 1],
[1, 1]])
Here, numpy.diff (i.e. the arithmetic difference of adjacent array elements) is applied along each slice of either the first or second axis (dimension) of the input array.
The function is performed on 1-d arrays along axis=0. You can specify another axis using the "axis" argument. A usage of this paradigm is:
np.apply_along_axis(np.cumsum, 0, b)
The function was performed on each subarray along dimension 0. So, it is meant for 1-D functions and returns a 1D array for each 1-D input.
Another example is :
np.apply_along_axis(np.sum, 0, b)
Provides a scalar output for a 1-D array.
Of course you could just set the axis parameter in cumsum or sum to do the above, but the point here is that it can be used for any 1-D function you write.
Im working with two arrays, trying to work with them like a 2 dimensional array. I'm using a lot of vectorized calculations with NumPy. Any idea how I would populate an array like this:
X = [1, 2, 3, 1, 2, 3, 1, 2, 3]
or:
X = [0.2, 0.4, 0.6, 0.8, 0.2, 0.4, 0.6, 0.8, 0.2, 0.4, 0.6, 0.8, 0.2, 0.4, 0.6, 0.8]
Ignore the first part of the message.
I had to populate two arrays in a form of a grid. But the grid dimensions varied from the users, thats why I needed a general form. I worked on it all this morning and finally got what I wanted.
I apologize if I caused any confusion earlier. English is not my tongue language, and sometimes it is hard for me to explain things.
This is the code that did the job for me:
myIter = linspace(1, N, N)
for x in myIter:
for y in myIter:
index = ((x - 1)*N + y) - 1
X[index] = x / (N+1)
Y[index] = y / (N+1)
The user inputs N.
And the length of X, Y is N*N.
You can use the function tile. From the examples:
>>> a = np.array([0, 1, 2])
>>> np.tile(a, 2)
array([0, 1, 2, 0, 1, 2])
With this function, you can also reshape your array at once like they do in the other answers with reshape (by defining the 'repeats' is more dimensions):
>>> np.tile(a, (2, 1))
array([[0, 1, 2],
[0, 1, 2]])
Addition: and a little comparison of the difference in speed between the built in function tile and the multiplication:
In [3]: %timeit numpy.array([1, 2, 3]* 3)
100000 loops, best of 3: 16.3 us per loop
In [4]: %timeit numpy.tile(numpy.array([1, 2, 3]), 3)
10000 loops, best of 3: 37 us per loop
In [5]: %timeit numpy.array([1, 2, 3]* 1000)
1000 loops, best of 3: 1.85 ms per loop
In [6]: %timeit numpy.tile(numpy.array([1, 2, 3]), 1000)
10000 loops, best of 3: 122 us per loop
EDIT
The output of the code you gave in your question can also be achieved as following:
arr = myIter / (N + 1)
X = numpy.repeat(arr, N)
Y = numpy.tile(arr, N)
This way you can avoid looping the arrays (which is one of the great advantages of using numpy). The resulting code is simpler (if you know the functions of course, see the documentation for repeat and tile) and faster.
print numpy.array(range(1, 4) * 3)
print numpy.array(range(1, 5) * 4).astype(float) * 2 / 10
If you want to create lists of repeating values, you could use list/tuple multiplication...
>>> import numpy
>>> numpy.array((1, 2, 3) * 3)
array([1, 2, 3, 1, 2, 3, 1, 2, 3])
>>> numpy.array((0.2, 0.4, 0.6, 0.8) * 3).reshape((3, 4))
array([[ 0.2, 0.4, 0.6, 0.8],
[ 0.2, 0.4, 0.6, 0.8],
[ 0.2, 0.4, 0.6, 0.8]])
Thanks for updating your question -- it's much clearer now. Though I think joris's answer is the best one in this case (because it is more readable), I'll point out that the new code you posted could also be generalized like so:
>>> arr = numpy.arange(1, N + 1) / (N + 1.0)
>>> X = arr[numpy.indices((N, N))[0]].flatten()
>>> Y = arr[numpy.indices((N, N))[1]].flatten()
In many cases, when using numpy, one avoids while loops by using numpy's powerful indexing system. In general, when you use array I to index array A, the result is an array J of the same shape as I. For each index i in I, the value A[i] is assigned to the corresponding position in J. For example, say you have arr = numpy.arange(0, 9) / (9.0) and you want the values at indices 3, 5, and 8. All you have to do is use numpy.array([3, 5, 8]) as the index to arr:
>>> arr
array([ 0. , 0.11111111, 0.22222222, 0.33333333, 0.44444444,
0.55555556, 0.66666667, 0.77777778, 0.88888889])
>>> arr[numpy.array([3, 5, 8])]
array([ 0.33333333, 0.55555556, 0.88888889])
What if you want a 2-d array? Just pass in a 2-d index:
>>> arr[numpy.array([[1,1,1],[2,2,2],[3,3,3]])]
array([[ 0.11111111, 0.11111111, 0.11111111],
[ 0.22222222, 0.22222222, 0.22222222],
[ 0.33333333, 0.33333333, 0.33333333]])
>>> arr[numpy.array([[1,2,3],[1,2,3],[1,2,3]])]
array([[ 0.11111111, 0.22222222, 0.33333333],
[ 0.11111111, 0.22222222, 0.33333333],
[ 0.11111111, 0.22222222, 0.33333333]])
Since you don't want to have to type indices like that out all the time, you can generate them automatically -- with numpy.indices:
>>> numpy.indices((3, 3))
array([[[0, 0, 0],
[1, 1, 1],
[2, 2, 2]],
[[0, 1, 2],
[0, 1, 2],
[0, 1, 2]]])
In a nutshell, that's how the above code works. (Also check out numpy.mgrid and numpy.ogrid -- which provide slightly more flexible index-generators.)
Since many numpy operations are vectorized (i.e. they are applied to each element in an array) you just have to find the right indices for the job -- no loops required.
import numpy as np
X = range(1,4)*3
X = list(np.arange(.2,.8,.2))*4
these will make your two lists, respectively. Hope thats what you were asking
I'm not exactly sure what you are trying to do, but as a guess: if you have a 1D array and you need to make it 2D you can use the array classes reshape method.
>>> import numpy
>>> a = numpy.array([1,2,3,1,2,3])
>>> a.reshape((2,3))
array([[1, 2, 3],
[1, 2, 3]])