Develop Reference

python how to get proper distance value out of scipy condensed distance matrix

I am using python 2.7 with scipy to calculate a distance matrix for an array.
I don't get how to find the wanted distance values in the returned condensed matrix.
See example
from scipy.spatial.distance import pdist
import numpy as np
a = np.array([[1],[4],[0],[5]])
print a
print pdist(a)
will print
[ 3. 1. 4. 4. 1. 5.]
I found here that the ij entry in the condensed matrix should store the distance between the i and j entries where ithread wondering if they mean ij as i*j or str.join(i,j) e.g 1,2 -> 2 or 12.
I can't find a consistent way to know the wanted index.
see my example, you should expect that all of the distances from entry 0 to anywhere else will be stored in entry 0 if the first option is valid.
can anyone shed some light on how can i extract my wanted distance from entry x to entry y? which index am i looking for?
Thanks!

This vector is in condensed form. It enumerates all pairs of indices in a natural order (in your example 0,1 0,2 0,3 0,4 1,2 1,3 1,4 2,3 2,4 ) and yields the distance between the elements at these array entries.
There is also the squareform function, which transforms the condensed form into a square matrix form (and vice versa). The square matrix form is exactly what you expect, i.e. at entry ij (row i, column j), it stores the distance between the i-th and j-th entry. For example, if you add print squareform(d) at the end of you code, the output will be:
array([[ 0., 3., 1., 4.],
[ 3., 0., 4., 1.],
[ 1., 4., 0., 5.],
[ 4., 1., 5., 0.]])

Theano: Operate on nonzero elements of sparse matrix

I'm trying to take the exp of nonzero elements in a sparse theano variable. I have the current code:
A = T.matrix("Some matrix with many zeros")
A_sparse = theano.sparse.csc_from_dense(A)
I'm trying to do something that's equivalent to the following numpy syntax:
mask = (A_sparse != 0)
A_sparse[mask] = np.exp(A_sparse[mask])
but Theano doesn't support != masks yet. (And (A_sparse > 0) | (A_sparse < 0) doesn't seem to work either.)
How can I achieve this?

The support for sparse matrices in Theano is incomplete, so some things are tricky to achieve. You can use theano.sparse.structured_exp(A_sparse) in that particular case, but I try to answer your question more generally below.
Comparison
In Theano one would normally use the comparison operators described here: http://deeplearning.net/software/theano/library/tensor/basic.html
For example, instead of A != 0, one would write T.neq(A, 0). With sparse matrices one has to use the comparison operators in theano.sparse. Both operators have to be sparse matrices, and the result is also a sparse matrix:
mask = theano.sparse.neq(A_sparse, theano.sparse.sp_zeros_like(A_sparse))
Modifying a Subtensor
In order to modify part of a matrix, one can use theano.tensor.set_subtensor. With dense matrices this would work:
indices = mask.nonzero()
A = T.set_subtensor(A[indices], T.exp(A[indices]))
Notice that Theano doesn't have a separated boolean type—the mask is zeros and ones—so nonzero() has to be called first to take the indices of the nonzero elements. Furthermore, this is not implemented for sparse matrices.
Operating on Nonzero Sparse Elements
Theano provides sparse operations that are said to be structured and operate only on the nonzero elements. See:
http://deeplearning.net/software/theano/tutorial/sparse.html#structured-operation
More precisely, they operate on the data attribute of a sparse matrix, independent of the indices of the elements. Such operations are straightforward to implement. Note that the structured operations will operate on all the values in the data array, also those that are explicitly set to zero.

Here's a way of doing this with the scipy.sparse module. I don't know how theano implements its sparse. It's likely to be based on similar ideas (since it uses name like csc)
In [224]: A=sparse.csc_matrix([[1.,0,0,2,0],[0,0,3,0,0],[0,1,1,2,0]])
In [225]: A.A
Out[225]:
array([[ 1., 0., 0., 2., 0.],
[ 0., 0., 3., 0., 0.],
[ 0., 1., 1., 2., 0.]])
In [226]: A.data
Out[226]: array([ 1., 1., 3., 1., 2., 2.])
In [227]: A.data[:]=np.exp(A.data)
In [228]: A.A
Out[228]:
array([[ 2.71828183, 0. , 0. , 7.3890561 , 0. ],
[ 0. , 0. , 20.08553692, 0. , 0. ],
[ 0. , 2.71828183, 2.71828183, 7.3890561 , 0. ]])
The main attributes of the csc format at data, indices, indptr. It's possible that data has some 0 values if you fiddle with them after creation, but a freshly created matrix shouldn't.
The matrix also has a nonzero method modeled on the numpy one. In practice it converts the matrix to coo format, filters out any zero values, and returns the row and col attributes:
In [229]: A.nonzero()
Out[229]: (array([0, 0, 1, 2, 2, 2]), array([0, 3, 2, 1, 2, 3]))
And the csc format allows indexing just as a dense numpy array:
In [230]: A[A.nonzero()]
Out[230]:
matrix([[ 2.71828183, 7.3890561 , 20.08553692, 2.71828183,
2.71828183, 7.3890561 ]])

T.where works.
A_sparse = T.where(A_sparse == 0, 0, T.exp(A_sparse))
#Seppo Envari's answer seems faster though. So I'll accept his answer.

Removing length 1 dimensions in arrays

I'm a real beginner with Python, and I have a recurrent problem with my ndarrays.
I'm very confused with the brackets (is there any schematic synthesis of the use of brackets in Python anywhere?). I always end up having arrays with many dimensions.
Right now I have this one:
>>> values
Out[1]:
array([[[ array([[ 4.23156519, -0.93539198],
[ 3.50074853, -1.67043386],
[ 4.64192393, -1.03918172],
[ 4.52056725, 0.2561267 ],
[ 3.36400016, 0.26435125],
[ 3.82025672, 1.16503286]])]]], dtype=object)
From here, how can I reduce the dimensions? I just wanted a 6x2 array. I tried np.reshape but since the current shape of values is (1,1,1) I can't directly reshape the array in a 6x2 one.
I'm sorry for the silly question, I'm seeking a general and schematic answer that would explain me how to pass from a higher dimension to a lower one and vice versa.
Here is the way I created the array. values is clustered_points
indices=[] # initialize indices
clustered_points=[] # initialize array containing points in different sub-arrays=clusters
for k in range(len(mu)):
a=r[:,k]
index=[t for t in range(len(a)) if a[t] == 1]
indices.append(index)
clustered_points.append(data[indices[k]])
clustered_points=np.reshape(clustered_points,(len(clustered_points),1,1))

To make an array that matches your initial display, I have to take special care to embed one array within another:
In [402]: x=np.array([[ 4.23156519, -0.93539198],
[ 3.50074853, -1.67043386],
[ 4.64192393, -1.03918172],
[ 4.52056725, 0.2561267 ],
[ 3.36400016, 0.26435125],
[ 3.82025672, 1.16503286]])
In [403]: a=array([[[None]]],dtype=object)
In [404]: a[0,0,0]=x
In [405]: a
Out[405]:
array([[[ array([[ 4.23156519, -0.93539198],
[ 3.50074853, -1.67043386],
[ 4.64192393, -1.03918172],
[ 4.52056725, 0.2561267 ],
[ 3.36400016, 0.26435125],
[ 3.82025672, 1.16503286]])]]], dtype=object)
In [406]: a.shape
Out[406]: (1, 1, 1)
In [407]: a[0,0,0].shape
Out[407]: (6, 2)
Simply doing a cut-n-paste from the display produces a different array with shape (1,1,1,6,2). That does not have the inner array marking. Either way a[0,0,0] gives the inner (6,2) array.
reshape and squeeze work on a (1,1,1,6,2) array, but not on a (6,2) nested inside a (1,1,1). You need to understand the difference.
(edit)
To run your 'how I did it' clip, I have to make some guesses as to the inputs (that almost merits a downvote).
I'll guess at some inputs:
In [420]: mu=np.arange(3); r=np.ones((4,3));data=np.ones(5)
In [421]: %paste
indices=[] # initialize indices
clustered_points=[] # initialize array containing points in different sub-arrays=clusters
for k in range(len(mu)):
a=r[:,k]
index=[t for t in range(len(a)) if a[t] == 1]
indices.append(index)
clustered_points.append(data[indices[k]])
## -- End pasted text --
In [422]: clustered_points
Out[422]:
[array([ 1., 1., 1., 1.]),
array([ 1., 1., 1., 1.]),
array([ 1., 1., 1., 1.])]
cluster_points is a list with several 1d arrays.
I can do
np.reshape(clustered_points,(12,1,1))
np.reshape(clustered_points,(3,4,1,1))
though it would be better, I think, to do np.array(clustered_points) first, and may be even check its shape.
Since
np.reshape(clustered_points,(len(clustered_points),1,1))
supposedly works then clustered_points must be a list of n single element arrays. But this reshape should produce a (n,1,1) array, not your (1,1,1,...) array.
So that edit doesn't help.
=========================
I'm seeking a general and schematic answer that would explain me how to pass from a higher dimension to a lower one and vice versa.
The first step is be clear, to yourself and others, what is the structure of your array. That includes knowing shape and dtype. And if the dtype is anything other than simple numerics, pay attention to the structure of the elements (e.g. objects within the array).
Singular dimensions (value 1) can be removed with indexing, [0], or squeeze. reshape also removes demensions (or adds them), but you have to pay attention to the total number of elements. If the old shape had 12 elements, the new has to have 12 as well. But reshape does not operate across dtype boundaries.

I think you're looking for numpy.squeeze:
#!/usr/bin/env python
import numpy
a = [[[[[ 4.23156519, -0.93539198],
[ 3.50074853, -1.67043386],
[ 4.64192393, -1.03918172],
[ 4.52056725, 0.2561267 ],
[ 3.36400016, 0.26435125],
[ 3.82025672, 1.16503286]]]]]
a = numpy.array(a)
print("a.shape=%s" % str(a.shape))
b = numpy.squeeze(a)
print("b.shape=%s" % str(b.shape))
gives
a.shape=(1, 1, 1, 6, 2)
b.shape=(6, 2)

The underlying issue is probably, how did you create this array?
A python list object cannot be manipulated in the same ways that np.ndarrays can be. In general, once your final list is created, you can cast it into a numpy array:
values = []
# fill values with values.append(...)
# ...
values = np.asarray(values)
To find the shape of an array, you can use either A.shape or np.shape(A). To remove dimensions of length one, the best approach is to use the squeeze method either as A.squeeze() or np.squeeze(A), i.e:
>>> values.squeeze()
array([[4.23156519, -0.93539198],
[3.50074853, -1.67043386],
[4.64192393, -1.03918172],
[4.52056725, 0.2561267],
[3.36400016, 0.26435125],
[3.82025672, 1.16503286]], dtype=object)
If your values array is really what you've said, then it should also be fine to use reshape
>>> values.reshape(6,2)
array([[4.23156519, -0.93539198],
[3.50074853, -1.67043386],
[4.64192393, -1.03918172],
[4.52056725, 0.2561267],
[3.36400016, 0.26435125],
[3.82025672, 1.16503286]], dtype=object)
If you're getting an error trying to reshape values, is it possible it is actually a list instead of an array?

If you want to create 6x2 array then just do this:
A = array([[4.23156519, -0.93539198],
[3.50074853, -1.67043386],
[4.64192393, -1.03918172],
[4.52056725, 0.2561267],
[3.36400016, 0.26435125],
[3.82025672, 1.16503286]], dtype=object)
If you want to reduce your array:
A = array([[[ array([[ 4.23156519, -0.93539198],
[ 3.50074853, -1.67043386],
[ 4.64192393, -1.03918172],
[ 4.52056725, 0.2561267 ],
[ 3.36400016, 0.26435125],
[ 3.82025672, 1.16503286]])]]], dtype=object)
it is actually 1x1x1x6x2 array, you can get 6x2 by doing A[0][0][0]

Access data points scikit KNNR

Question
After fitting the data with neigh.fit() I would like to access these data-points, how do I do this?
Details
>>> samples = [[0., 0., 0.], [0., .5, 0.], [1., 1., .5]]
>>> samplesy = [80, 60, 40]
>>> from sklearn import neighbors
>>> neigh = neighbors.KNeighborsRegressor(n_neighbors=1)
>>> neigh.fit(samples, samplesy)
>>> print(neigh.kneighbors([1., 1., 1.]))
(array([[ 0.5]]), array([[2]]))
So from this I learned that the closest data-point is 'samples[2]'.
However in the case I don't have access anymore to the variable 'samples', is there a way to access the data-point in 'neigh'? Maybe something like 'neigh[2]'? Because the data-points have to be saved somewhere in the model of 'neigh' right?
Why
I would like to access the 5 closest neighbors data-points and calculate a cluster-center of these data-points. Then I want to calculate the distance of this cluster-center to the new data point to get an idea of how far this new data-point is from the original data.

The data used to fit the model are stored in neigh._fit_X:
>>> neigh._fit_X
array([[ 0. , 0. , 0. ],
[ 0. , 0.5, 0. ],
[ 1. , 1. , 0.5]])
However: The leading underscore of the variable name should be a signal to you that this is supposed to be somewhat of a private attribute. You shouldn't expect for this data to behave in any particular way, or even to exist in future versions of the library. Use it at your own risk.
A better way might be to just keep track of the input data on your own.

weighted moving average with numpy.convolve

I'm writing a moving average function that uses the convolve function in numpy, which should be equivalent to a (weighted moving average). When my weights are all equal (as in a simple arithmatic average), it works fine:
data = numpy.arange(1,11)
numdays = 5
w = [1.0/numdays]*numdays
numpy.convolve(data,w,'valid')
gives
array([ 3., 4., 5., 6., 7., 8.])
However, when I try to use a weighted average
w = numpy.cumsum(numpy.ones(numdays,dtype=float),axis=0); w = w/numpy.sum(w)
instead of the (for the same data) 3.667,4.667,5.667,6.667,... I expect, I get
array([ 2.33333333, 3.33333333, 4.33333333, 5.33333333, 6.33333333,
7.33333333])
If I remove the 'valid' flag, I don't even see the correct values. I would really like to use convolve for the WMA as well as MA as it makes the code cleaner (same code, different weights) and otherwise I think I'll have to loop through all the data and take slices.
Any ideas about this behavior?

What you want is np.correlate in a convolution the second argument is inverted basically, so that your expected result would be with np.convolve(data, w[::-1], 'valid').