Develop Reference

apply_along_axis with various variables

I have a cooccurrence matrix, a symmetric matrix (Numpy Array) in which each cell indicates the frequency of two co-occurring words.
In this matrix, I want to calculate the association strength. Which is defined as the number of times word i and j co-occur, divided by the product of i- and j's total frequency:
def calculate_association_strength(self, cooc, i, j, word_occurrences):
return cooc/(word_occurrences[i]*word_occurrences[j])
Here:
cooc = the cooccurrence of word i and j, with size vocabulary_size x vocabulary_size.
word_occurences = a list of length vocabulary_size, showing at each index the frequency of word i.
i and j = integers, indicating the word indices.
I am looping through the cooccurrence matrix to calculate the association strength per cell. However, this approach is very slow. I am familiar with the apply_along_axis method. However, it is unclear how to use it for this method. Is this possible? And if so, how can I do this?

IIUC you want to row- and columnwise divide each element of coor by word_occurrences. This can be done by simple elementwise division and broadcasting:
import numpy as np
cooc = np.array([[0, 1, 1, 0], [1, 0, 2, 1], [1, 2, 0, 1], [0, 1, 1, 0]])
word_occurrences = [1, 2, 3, 4]
cooc / word_occurrences / np.array(word_occurrences)[:, np.newaxis]
Result:
array([[0. , 0.5 , 0.33333333, 0. ],
[0.5 , 0. , 0.33333333, 0.125 ],
[0.33333333, 0.33333333, 0. , 0.08333333],
[0. , 0.125 , 0.08333333, 0. ]])
Is this what you are looking for?

How to do an absolute sorting of columns in a matrix (numpy)

So i have a matrix:
a = np.array([[7,-1,0,5],
[2,5.2,4,2],
[3,-2,1,4]])
which i would like to sort by absolute value ascending column . I used np.sort(abs(a)) and sorted(a,key=abs), sorted is probably the right one but do not know how to use it for columns. I wish to get
a = np.array([[2,-1,0,2],
[3,-2,1,4],
[7,5.2,4,5]])

Try argsort on axis=0 then take_along_axis to apply the order to a:
import numpy as np
a = np.array([[7, -1, 0, 5],
[2, 5.2, 4, 2],
[3, -2, 1, 4]])
s = np.argsort(abs(a), axis=0)
a = np.take_along_axis(a, s, axis=0)
print(a)
a:
[[ 2. -1. 0. 2. ]
[ 3. -2. 1. 4. ]
[ 7. 5.2 4. 5. ]]

Interpolate 2D matrix along columns using Python

I am trying to interpolate a 2D numpy matrix with the dimensions (5, 3) to a matrix with the dimensions (7, 3) along the axis 1 (columns). Obviously, the wrong approach would be to randomly insert rows anywhere between the original matrix, see the following example:
Source:
[[0, 1, 1]
[0, 2, 0]
[0, 3, 1]
[0, 4, 0]
[0, 5, 1]]
Target (terrible interpolation -> not wanted!):
[[0, 1, 1]
[0, 1.5, 0.5]
[0, 2, 0]
[0, 3, 1]
[0, 3.5, 0.5]
[0, 4, 0]
[0, 5, 1]]
The correct approach would be to take every row into account and interpolate between all of them to expand the source matrix to a (7, 3) matrix. I am aware of the scipy.interpolate.interp1d or scipy.interpolate.interp2d methods, but could not get it to work with other Stack Overflow posts or websites. I hope to receive any type of tips or tricks.
Update #1: The expected values should be equally spaced.
Update #2:
What I want to do is basically use the separate columns of the original matrix, expand the length of the column to 7 and interpolate between the values of the original column. See the following example:
Source:
[[0, 1, 1]
[0, 2, 0]
[0, 3, 1]
[0, 4, 0]
[0, 5, 1]]
Split into 3 separate Columns:
[0 [1 [1
0 2 0
0 3 1
0 4 0
0] 5] 1]
Expand length to 7 and interpolate between them, example for second column:
[1
1.66
2.33
3
3.66
4.33
5]

It seems like each column can be treated completely independently, but for each column you need to define essentially an "x" coordinate so that you can fit some function "f(x)" from which you generate your output matrix.
Unless the rows in your matrix are associated with some other datastructure (e.g. a vector of timestamps), an obvious set of x values is just the row-number:
x = numpy.arange(0, Source.shape[0])
You can then construct an interpolating function:
fit = scipy.interpolate.interp1d(x, Source, axis=0)
and use that to construct your output matrix:
Target = fit(numpy.linspace(0, Source.shape[0]-1, 7)
which produces:
array([[ 0. , 1. , 1. ],
[ 0. , 1.66666667, 0.33333333],
[ 0. , 2.33333333, 0.33333333],
[ 0. , 3. , 1. ],
[ 0. , 3.66666667, 0.33333333],
[ 0. , 4.33333333, 0.33333333],
[ 0. , 5. , 1. ]])
By default, scipy.interpolate.interp1d uses piecewise-linear interpolation. There are many more exotic options within scipy.interpolate, based on higher order polynomials, etc. Interpolation is a big topic in itself, and unless the rows of your matrix have some particular properties (e.g. being regular samples of a signal with a known frequency range), there may be no "truly correct" way of interpolating. So, to some extent, the choice of interpolation scheme will be somewhat arbitrary.

You can do this as follows:
from scipy.interpolate import interp1d
import numpy as np
a = np.array([[0, 1, 1],
[0, 2, 0],
[0, 3, 1],
[0, 4, 0],
[0, 5, 1]])
x = np.array(range(a.shape[0]))
# define new x range, we need 7 equally spaced values
xnew = np.linspace(x.min(), x.max(), 7)
# apply the interpolation to each column
f = interp1d(x, a, axis=0)
# get final result
print(f(xnew))
This will print
[[ 0. 1. 1. ]
[ 0. 1.66666667 0.33333333]
[ 0. 2.33333333 0.33333333]
[ 0. 3. 1. ]
[ 0. 3.66666667 0.33333333]
[ 0. 4.33333333 0.33333333]
[ 0. 5. 1. ]]

Set numpy array elements to zero for each row's smallest 2 elements [duplicate]

This question already has an answer here:
Fill a matrix from a matrix of indices
(1 answer)
Closed 5 years ago.
For example
E =
array([[ 10. , 2.38761596, 7.00090613, 4.51495754],
[ 2.38761596, 10. , 2.80035826, 1. ],
[ 7.00090613, 2.80035826, 10. , 5.95109207],
[ 4.51495754, 1. , 5.95109207, 10. ]])
The indices for smallest 2 for each row can be get from argsort :
IndexSortE = np.argsort(E)
smallest2 = IndexSortE[:,0:2]
smallest2
array([[1, 3],
[3, 0],
[1, 3],
[1, 0]])
Now how do I get E0 like this ?? :
E0 =
array([[ 10. , 0.00000000, 7.00090613, 0.00000000],
[ 0.00000000, 10. , 2.80035826, 0.00000000],
[ 7.00090613, 0.00000000, 10. , 0.00000000],
[ 0.00000000, 0.00000000, 5.95109207, 10. ]])
Thanks

You can create another array of row indices; then take advantage of advanced indexing to modify the corresponding values:
E[np.arange(E.shape[0])[:,None], smallest2] = 0
E
#array([[ 10. , 0. , 7.00090613, 0. ],
# [ 0. , 10. , 2.80035826, 0. ],
# [ 7.00090613, 0. , 10. , 0. ],
# [ 0. , 0. , 5.95109207, 10. ]])
To add some explanations, use np.broadcast_arrays to see how these indices are broadcasted:
np.broadcast_arrays(np.arange(E.shape[0])[:,None], smallest2)
# [array([[0, 0],
# [1, 1],
# [2, 2],
# [3, 3]]), array([[1, 3],
# [3, 0],
# [1, 3],
# [1, 0]])]
gives a length two list, the first one gives row indices while the second one gives column indices. Now according to advanced indexing rules, this pair will position elements at
(0, 1), (0, 3),
(1, 3), (1, 0),
...
etc.

Elegant grid search in python/numpy

I have a function that has a bunch of parameters. Rather than setting all of the parameters manually, I want to perform a grid search. I have a list of possible values for each parameter. For every possible combination of parameters, I want to run my function which reports the performance of my algorithm on those parameters. I want to store the results of this in a many-dimensional matrix, so that afterwords I can just find the index of the maximum performance, which would in turn give me the best parameters. Here is how the code is written now:
param1_list = [p11, p12, p13,...]
param2_list = [p21, p22, p23,...] # not necessarily the same number of values
...
results_size = (len(param1_list), len(param2_list),...)
results = np.zeros(results_size, dtype = np.float)
for param1_idx in range(len(param1_list)):
for param2_idx in range(len(param2_list)):
...
param1 = param1_list[param1_idx]
param2 = param2_list[param2_idx]
...
results[param1_idx, param2_idx, ...] = my_func(param1, param2, ...)
max_index = np.argmax(results) # indices of best parameters!
I want to keep the first part, where I define the lists as-is, since I want to easily be able to manipulate the values over which I search.
I also want to end up with the results matrix as is, since I will be visualizing how changing different parameters affects the performance of the algorithm.
The bit in the middle, though, is quite repetitive and bulky (especially because I have lots of parameters, and I might want to add or remove parameters), and I feel like there should be a more succinct/elegant way to initialize the results matrix, iterate over all of the indices, and set the appropriate parameters.
So, is there?

You can use the ParameterGrid from the sklearn module
http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.ParameterGrid.html
Example
from sklearn.grid_search import ParameterGrid
param_grid = {'param1': [value1, value2, value3], 'paramN' : [value1, value2, valueM]}
grid = ParameterGrid(param_grid)
for params in grid:
your_function(params['param1'], params['param2'])

I think scipy.optimize.brute is what you're after.
>>> from scipy.optimize import brute
>>> a,f,g,j = brute(my_func,[param1_list,param2_list,...],full_output = True)
Note that if the full_output argument is True, the evaluation grid will be returned.

The solutions from John Vinyard and Sibelius Seraphini are good built-in options, but but if you're looking for more flexibility, you could use broadcasting + vectorize. Use ix_ to produce a broadcastable set of parameters, and then pass those to a vectorized version of the function (but see caveat below):
a, b, c = range(3), range(3), range(3)
def my_func(x, y, z):
return (x + y + z) / 3.0, x * y * z, max(x, y, z)
grids = numpy.vectorize(my_func)(*numpy.ix_(a, b, c))
mean_grid, product_grid, max_grid = grids
With the following results for mean_grid:
array([[[ 0. , 0.33333333, 0.66666667],
[ 0.33333333, 0.66666667, 1. ],
[ 0.66666667, 1. , 1.33333333]],
[[ 0.33333333, 0.66666667, 1. ],
[ 0.66666667, 1. , 1.33333333],
[ 1. , 1.33333333, 1.66666667]],
[[ 0.66666667, 1. , 1.33333333],
[ 1. , 1.33333333, 1.66666667],
[ 1.33333333, 1.66666667, 2. ]]])
product grid:
array([[[0, 0, 0],
[0, 0, 0],
[0, 0, 0]],
[[0, 0, 0],
[0, 1, 2],
[0, 2, 4]],
[[0, 0, 0],
[0, 2, 4],
[0, 4, 8]]])
and max grid:
array([[[0, 1, 2],
[1, 1, 2],
[2, 2, 2]],
[[1, 1, 2],
[1, 1, 2],
[2, 2, 2]],
[[2, 2, 2],
[2, 2, 2],
[2, 2, 2]]])
Note that this may not be the fastest approach. vectorize is handy, but it's limited by the speed of the function passed to it, and python functions are slow. If you could rewrite my_func to use numpy ufuncs, you could get your grids faster, if you cared to. Something like this:
>>> def mean(a, b, c):
... return (a + b + c) / 3.0
...
>>> mean(*numpy.ix_(a, b, c))
array([[[ 0. , 0.33333333, 0.66666667],
[ 0.33333333, 0.66666667, 1. ],
[ 0.66666667, 1. , 1.33333333]],
[[ 0.33333333, 0.66666667, 1. ],
[ 0.66666667, 1. , 1.33333333],
[ 1. , 1.33333333, 1.66666667]],
[[ 0.66666667, 1. , 1.33333333],
[ 1. , 1.33333333, 1.66666667],
[ 1.33333333, 1.66666667, 2. ]]])

You may use numpy meshgrid for this:
import numpy as np
x = range(1, 5)
y = range(10)
xx, yy = np.meshgrid(x, y)
results = my_func(xx, yy)
note that your function must be able to work with numpy.arrays.