Develop Reference

Different results using affinity propagation "precomputed" distance matrix

I am working with two-dimensional data
X = array([[5.40310335, 0. ],
[6.86136114, 6.56225717],
[0. , 0. ],
...,
[5.88838732, 0. ],
[6.0003473 , 0. ],
[6.25971331, 0. ]])
looking for clusters, using euclidean distance, i run affinity propagation from scikit learn with this raw data as follows
af = AffinityPropagation(damping=.9, max_iter=300, random_state=0).fit(X)
obtaining as a result 9 clusters.
I understand that when you want to use another distance you have to enter the negative distance matrix, and use affintity = 'precomputed' as it follows
af_c = AffinityPropagation(damping=.9, max_iter=300,
affinity='precomputed', random_state=0).fit(distM)
if as distM I use the Euclidean distance matrix calculated as follows
distM_E = -distance_matrix(X,X)
np.fill_diagonal(distM, np.median(distM))
completing the diagonal with the median since it is a predefined preference value also in the method.
Using this I am getting 34 clusters as a result and I would expect to have 9 as if working with the default distance. I don't know if I'm interpreting the way of entering the distance matrix correctly or if the library does something different when one uses everything predefined.
I would appreciate any help.

Cosine similarity of list of values with each other

I am trying to find the cosine similarity of a list of strings. I used sklearn tfidf vector to convert the text into a numerical vector first and then used the pairwise cosine_similarity api to find the score for each string pair.
The strings seem similar, but I am getting a weird answer. The first and third value in the string array are similar except the word TRENTON, but the cosine similarity is 0. Similarly, the 1st,3rd and 4th string are the same, except for a space between GREEN and CHILLI and the cosine similarity is zero. Isn't that strange?
My code:
from sklearn.metrics import pairwise_kernels
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectorizer=TfidfVectorizer()
values =['GREENCHILLI TRENTON'
,'GREENCHILLI'
,'GREEN CHILLI'
,'GREEN CHILLI']
X_train_counts = tfidf_vectorizer.fit_transform(values)
similarities = cosine_similarity(X_train_counts)
print(similarities)
Output
[[1. 0.6191303 0. 0. ]
[0.6191303 1. 0. 0. ]
[0. 0. 1. 1. ]
[0. 0. 1. 1. ]]

coma (,) missing between last two GREEN CHILLI so tfidf is treating them as only 3 records not 4.
If you correct it you should see below cosine similarity
[[1. 0.6191303 0. 0. ]
[0.6191303 1. 0. 0. ]
[0. 0. 1. 1. ]
[0. 0. 1. 1. ]]
How to interpret the above matrix: The value in the nth row are cosine similarities of that tfidf vector with all other vectors (in sequential order). So all the diagonal will be 1 because every vector is similar to itself.

The first and third value in the string array values is similar except the word Trenton but cosine similarity is 0.
Similarly, 1st,3rd and 4th strings are same only space between GREEN and CHILLI and the cosine similarity is zero. isn't it strange?
It is not as strange as you might think. You will only get a non-zero cosine similarity if you have exact word matches between the strings that you compare. I will try to explain what happens:
When the TF-IDF vectorizer creates vectors from your list of strings, it starts by making a list of all words that occur.
So in your case, the list would look like this:
GREENCHILLI
TRENTON
GREEN
CHILLI
Now, every word becomes an axis in a coordinate system that the algorithm uses. All axes are perpendicular to each other.
So when you compare 'GREENCHILLI TRENTON' with 'GREEN CHILLI', the algorithm makes two vectors. One from 'GREENCHILLI TRENTON' that has a component parallel to 'GREENCHILLI' and a component parallel to 'TRENTON'. The vector from the string 'GREEN CHILI' has components in 'GREEN' and 'CHILLI' direction of your coordinate system. When you calculate the dot product between the two you will get a zero. So the cosine similarity is zero as well.
So the gap in 'GREEN CHILLI' makes all the difference, when you compare it to 'GREENCHILLI'. The letters don't matter anymore, once the vectorizer made its coordinate system based on all the words it found in your list, because it identifies 'GREENCHILLI', 'GREEN' and 'CHILLI' as different words and makes them into perpendicular axes in its reference coordinate system.
Hope that makes it more clear. I suggest reading the following article series for a more in-depth understanding of whats going on:
http://blog.christianperone.com/2011/09/machine-learning-text-feature-extraction-tf-idf-part-i/

How to plot a weighted graph of k-Neighbors in Python

I have created a weighted graph of k-Neighbors using scikit-learn, I'm wondering if there is any way to plot it as a graph.
Here is the result of computation in form of array which I want to plot:
array([[0. , 2.08243189, 0. , 3.42661108],
[2.08243189, 0. , 3.27141008, 0. ],
[0. , 3.27141008, 0. , 1.57294787],
[0. , 3.29779083, 1.57294787, 0. ]])
I just need to get some visualization of data, that's all I need.
More details about the array:
Each row represents a node and each column represents the weight of connectivity of that node with the other nodes.
For example: second column of first row (2.08243189) is the weight of connectivity from first node to second node.
Another example: second row, second column (0): the weight of connectivity from node 2 to itself.
The numbers represents euclidean distance.

Are you talking about something simple like this where the size of the point gives a visual indication of the relative weight compared to the other values? Assume the array is named ar:
for i in range(len(ar)):
for j in range(len(ar)):
v = ar[i,j]
plt.scatter(i+1,j+1,lw=0,s=10**v)
plt.grid(True)
plt.xlabel('Row')
plt.ylabel('Column')
ticks = list(range(1,1+len(ar)))
plt.xticks(ticks)
plt.yticks(ticks)

Statsmodels - Wald Test for significance of trend in coefficients in Linear Regression Model (OLS)

I have used Statsmodels to generate a OLS linear regression model to predict a dependent variable based on about 10 independent variables. The independent variables are all categorical.
I am interested in looking closer at the significance of the coefficients for one of the independent variables. There are 4 categories, so 3 coefficients -- each of which are highly significant. I would also like to look at the significance of the trend across all 3 categories. From my (limited) understanding, this is often done using a Wald Test and comparing all of the coefficients to 0.
How exactly is this done using Statsmodels? I see there is a Wald Test method for the OLS function. It seems you have to pass in values for all of the coefficients when using this method.
My approach was the following...
First, here are all of the coefficients:
np.array(lm.params) = array([ 0.21538725, 0.05675108, 0.05020252, 0.08112228, 0.00074715,
0.03886747, 0.00981819, 0.19907263, 0.13962354, 0.0491201 ,
-0.00531318, 0.00242845, -0.0097336 , -0.00143791, -0.01939182,
-0.02676771, 0.01649944, 0.01240742, -0.00245309, 0.00757727,
0.00655152, -0.02895381, -0.02027537, 0.02621716, 0.00783884,
0.05065323, 0.04264466, -0.13068456, -0.15694931, -0.25518566,
-0.0308599 , -0.00558183, 0.02990139, 0.02433505, -0.01582824,
-0.00027538, 0.03170669, 0.01130944, 0.02631403])
I am only interested in params 2-4 (which are the 3 coefficients of interest).
coeffs = np.zeros_like(lm.params)
coeffs = coeffs[1:4] = [0.05675108, 0.05020252, 0.08112228]
Checking to make sure this worked:
array([ 0. , 0.05675108, 0.05020252, 0.08112228, 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. ])
Looks good, now to run in the test!
lm.wald_test(coeffs) =
<class 'statsmodels.stats.contrast.ContrastResults'>
<F test: F=array([[ 13.11493673]]), p=0.000304699208434, df_denom=1248, df_num=1>
Is this the correct approach? I could really use some help!

A linear hypothesis has the form R params = q where R is the matrix that defines the linear combination of parameters and q is the hypothesized value.
In the simple case where we want to test whether some parameters are zero, the R matrix has a 1 in the column corresponding to the position of the parameter and zeros everywhere else, and q is zero, which is the default. Each row specifies a linear combination of parameters, which defines a hypothesis as part of the overall or joint hypothesis.
In this case, the simplest way to get the restriction matrix is by using the corresponding rows of an identity matrix
R = np.eye(len(results.params))[1:4]
Then, lm.wald_test(R) will provide the test for the joint hypothesis that the 3 parameters are zero.
A simpler way to specify the restriction is by using the names of the parameters and defining the restrictions by a list of strings.
The model result classes also have a new method wald_test_terms which automatically generates the wald tests for terms in the design matrix where the hypothesis includes several parameters or columns, as in the case of categorical explanatory variables or of polynomial explanatory variables. This is available in statsmodels master and will be in the upcoming 0.7 release.

Rotate transformation matrix to match reflection vector

There are a couple of ball-bounce related questions on stackoverflow that i've looked through, however none of them seem to get me past my predicament. I have a turtle cursor defined by a transformation matrix that intersects a line in 3d space. What I want is to rotate the cursor, that is, the transformation matrix, at the point of intersection so that it's new direction matches the reflection vector. I have functions that will get both the reflection vector R from the incident vector V and the normal of the reflecting line N. I normalize each before evaluating:
N,V=unit_vector(N),unit_vector(V)
R = -2*(np.dot(V,N))*N - V
R=unit_vector(R)
My transformation matrix, T is in a numpy array:
array([[ -0.84923515, -0.6 , 0. , 3.65341878],
[ 0.52801483, -0.84923515, 0. , 25.12882224],
[ 0. , 0. , 1. , 0. ],
[ 0. , 0. , 0. , 1. ]])
How can I transform T by R to get the correct direction vector? I've found and used the R2_vect function from here to get a rotation matrix from one vector to another but only a few of the resulting reflections appear correct when i send them to vtk to render. I'm asking about this here because I seem to be reaching the limit of what I can remember from my already shaky linear algebra. Thanks for any information.

A little extra research clarified things: the first 3 columns of the transformation matrix represent 3 orthonormal vectors ( x1, x2, x3 ) and the 4th column represents the coordinates in space of the cursor at given time interval. the final row contains no data, it's just there to keep the matrix square. rotating the vectors was just a matter of removing the last row of T, taking the 3x3 rotation matrix from my listed function R and rotating each vector: R.dot(x1), R.dot(x2), R.dot(x3) Then I just had to composite the values back into a 4x4 matrix.