Forgive my terminology, I'm not an ML pro. I might use the wrong terms below.
I'm trying to perform multivariable linear regression. Let's say I'm trying to work out user gender by analysing page views on a web site.
For each user whose gender I know, I have a feature matrix where each row represents a web site section, and the second element whether they visited it, e.g.:
male1 = [
[1, 1], # visited section 1
[2, 0], # didn't visit section 2
[3, 1], # visited section 3, etc
[4, 0]
]
So in scikit, I am building xs and ys. I'm representing a male as 1, and female as 0.
The above would be represented as:
features = male1
gender = 1
Now, I'm obviously not just training a model for a single user, but instead I have tens of thousands of users whose data I'm using for training.
I would have thought I should create my xs and ys as follows:
xs = [
[ # user1
[1, 1],
[2, 0],
[3, 1],
[4, 0]
],
[ # user2
[1, 0],
[2, 1],
[3, 1],
[4, 0]
],
...
]
ys = [1, 0, ...]
scikit doesn't like this:
from sklearn import linear_model
clf = linear_model.LinearRegression()
clf.fit(xs, ys)
It complains:
ValueError: Found array with dim 3. Estimator expected <= 2.
How am I supposed to supply a feature matrix to the linear regression algorithm in scikit-learn?
You need to create xs in a different way. According to the docs:
fit(X, y, sample_weight=None)
Parameters:
X : numpy array or sparse matrix of shape [n_samples, n_features]
Training data
y : numpy array of shape [n_samples, n_targets]
Target values
sample_weight : numpy array of shape [n_samples]
Individual weights for each sample
Hence xs should be a 2D array with as many rows as users and as many columns as web site sections. You defined xs as a 3D array though. In order to reduce the number of dimensions by one you could get rid of the section numbers through a list comprehension:
xs = [[visit for section, visit in user] for user in xs]
If you do so, the data you provided as an example gets transformed into:
xs = [[1, 0, 1, 0], # user1
[0, 1, 1, 0], # user2
...
]
and clf.fit(xs, ys) should work as expected.
A more efficient approach to dimension reduction would be that of slicing a NumPy array:
import numpy as np
xs = np.asarray(xs)[:,:,1]
Related
I give TSNE a list of vectors, some of these vectors are exactly the same. But the output of fit() function can be different for each!
IS this expected behavior? How can i assure each input vector will be mapped to same output vector?
Exclamation,
I cannot tell for sure, but I even noticed that the first entry in the input list of vectors always gets different unexpected value.
Consider the following simple example.
Notice that the first three vectors are the same, the random_state is static but the first three 2D vectors in the output can be different from each others.
from sklearn import manifold
import numpy as np
X= np.array([ [2, 1, 3, 5],
[2, 1, 3, 5],
[2, 1, 3, 5],
[2, 1, 3, 5],
[12, 1, 3, 5],
[87, 22, 3, 5],
[3, 23, 9, 5],
[43, 87, 3, 5],
[121, 65, 3, 5]])
m = manifold.TSNE(
n_components=2,
perplexity=0.666666,
verbose=0,
random_state=42,
angle=.99,
init='pca',
metric='cosine',
n_iter=1000)
X_emedded = m.fit_transform(X)
# The following might fail
assert( sum(X_emedded[1] - X_emedded[2] ) == 0)
assert( sum(X_emedded[0] - X_emedded[1] ) == 0)
Update....
sklearn.version is '1.2.0'
t-SNE, as presenter by van der Maaten and Hinton 2008 is a technique to "visualizes high-dimensional data by giving each
datapoint a location in a two or three-dimensional map".
There is no guarantee that two identical points are mapped to the same low dimensional point. As a matter of fact it almost never happens as one can see with Algorithm 1 in (Maaten and Hinton 2008). The points in the low dimensional space are obtained with a gradient descent minimizing a cost function after a random initialisation.
I have some code which calculates the nearest neighbors amongst some vectors (values).
However, the values of these vectors are dependent on weights. Each column of the vectors has a different weight at every iteration.
Just for the sake of the example, at the code below I try to find everytime the nearest neighbor of the last vector (vector[3]).
That's a very simplified version of my code:
from sklearn.neighbors import NearestNeighbors
knn = NearestNeighbors(n_neighbors=1)
values = [
[2, 5, 1],
[4, 2, 3],
[1, 5, 2],
[4, 5, 4]
]
weights = [
[1, 3, 1],
[0.5, 2, 1],
[3, 1, 2]
]
# weights set No1
new_values = []
for line in values:
new_values.append([a*b for a,b in zip(line,weights[0])])
knn.fit(new_values)
print(knn.kneighbors(new_values[3]))
# weights set No2
new_values = []
for line in values:
new_values.append([a*b for a,b in zip(line,weights[1])])
knn.fit(new_values)
print(knn.kneighbors(new_values[3]))
# weights set No3
new_values = []
for line in values:
new_values.append([a*b for a,b in zip(line,weights[2])])
knn.fit(new_values)
print(knn.kneighbors(new_values[3]))
(Obviously I could have a for loop for the different weights sets but I just wanted to point the repetition of the matter)
My question is, is there any way that I can avoid using the KNN 3 times but just use it once at the beginning to do the initial similarity ranking/sorting and then just do some re-calculations?
In different words, is there any way to reduce the computation complexity of this code in terms of calling the KNN fewer times?
PS
I know that there are KNN implementations which are much faster than the ScikitLearn one but that's not really the point; the point is more on using KNN just once instead of N=3 times or something like that.
assuming calling the KNN fewer times means the number of times the KNN is fit, yes it's possible. if calling the KNN means the number of times kneighbors is invoked, that might be difficult due to how relative distances aren't preserved under affine transformations.
This solution runs in O(wk log n) time compared to the original O(wn) time with w being the number of weights.
what you're doing is
taking the input points
scaling its dimensions (projecting the input points into a new coordinate space)
building a knn model from the scaled inputs
classifying the target based on the scaled input.
However, consider
taking the input points
building a knn model from the scaled inputs
inverse scaling the target point (projecting the target into the original coordinate space)
classifying the inverse scaled target based on the input
the result of this process would be that steps 1 and 2 could be reused for each target point. weights with value 0 will require special handling.
this would look would be something like:
from sklearn.neighbors import NearestNeighbors
knn = NearestNeighbors(n_neighbors=1, algorithm="kd_tree")
values = [
[2, 5, 1],
[4, 2, 3],
[1, 5, 2],
[4, 5, 4]
]
weights = [
[1, 3, 1],
[0.5, 2, 1],
[3, 1, 2]
]
targets = [
[4, 15, 4], # values[3] * weights[0]
[2.0, 10, 4], # values[3] * weights[1]
[12, 5, 8] # values[3] * weights[2]
]
knn.fit(values)
# weights set No1
print(knn.kneighbors([[a/b for a, b in zip(targets[0], weights[0])]]))
# weights set No2
print(knn.kneighbors([[a/b for a, b in zip(targets[1], weights[1])]]))
# weights set No3
print(knn.kneighbors([[a/b for a, b in zip(targets[2], weights[2])]]))
The preprocessing module further provides a utility class
StandardScaler that implements the Transformer API to compute the mean
and standard deviation on a training set so as to be able to later
reapply the same transformation on the testing set.
http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler.fit_transform
When transforming the dataset you run an algorithm on, how do you link the results back to the original dataset?
E.g.
data = [[0, 0], [0, 0], [1, 1], [1, 1]]
print(data);
-->[[0, 0], [0, 0], [1, 1], [1, 1]]
myData = StandardScaler().fit_transform(data)
print(myData);
-->[[-1. -1.]
[-1. -1.]
[ 1. 1.]
[ 1. 1.]]
When running an algorithm on myData (unsupervised), how can you interpret results on that dataset when it's changed before running? E.g. when you run a clustering algorithm on myData, you are not clustering the original data.
Apply the inverse_transform to get back to the original data:
from sklearn.preprocessing import StandardScaler
import numpy as np
data = [[0, 0], [0, 1], [1, 0], [1, 1]]
scaler = StandardScaler()
myData = scaler.fit_transform(data)
restored = scaler.inverse_transform(myData)
assert np.allclose(restored, data) # check we got the original data back
Note how an instance of StandardScaler is stored in a variable for later use. After fitting, this instance contains all the information required to repeat or undo the transformation.
Now, if you performed clustering on myData you can pass the cluster prototypes (centers, or whatever you get from the clustering algorithm) to scaler.inverse_transform to get the clusters in the original data space.
This question already has answers here:
Preprocessing in scikit learn - single sample - Depreciation warning
(8 answers)
Closed 5 years ago.
I wrote a very simple scikit-learn decision tree to implement XOR:
from sklearn import tree
X = [[0, 0], [1, 1], [0, 1], [1, 0]]
Y = [0, 0, 1, 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(X, Y)
print(clf.predict([0,1]))
print(clf.predict([0,0]))
print(clf.predict([1,1]))
print(clf.predict([1,0]))
predict part generates some warning like this:
DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17
and will raise ValueError in 0.19. Reshape your data either using
X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1)
if it contains a single sample.
I don't have a clear idea what needs to change and why? Please enlighten me!
Thank you in advance!
The input to clf.predict should be a 2D array. Thus, instead of writing
print(clf.predict([0,1]))
you need to write
print(clf.predict([[0,1]]))
The method operates on matrices (2D arrays), rather than vectors (1D arrays). As a convenience, the older code accepted a vector as a 1xN matrix. This led to usage errors as some users forgot which way a vector was oriented (1xN vs Nx1).
The suggestion tells you how to reshape your vector to the proper matrix shape. For constant vectors, just write them as matrices:
clf.predict( [ [0, 1] ] )
The "other direction" (wrong for this application) would be
clf.predict( [ [0], [1] ] )
As the warning message pointed out, you have single sample to test. Thus you could use reshape or fix as followings,
from sklearn import tree
X = [[0, 0], [1, 1], [0, 1], [1, 0]]
Y = [0, 0, 1, 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(X, Y)
print (clf.predict([[0,1]]))
print (clf.predict([[0,0]]))
print (clf.predict([[1,1]]))
print (clf.predict([[1,0]]))
I try to use t-SNE algorithm in the scikit-learn:
import numpy as np
from sklearn.manifold import TSNE
X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])
model = TSNE(n_components=2, random_state=0)
np.set_printoptions(suppress=True)
model.fit_transform(X)
Output:
array([[ 0.00017599, 0.00003993], #1
[ 0.00009891, 0.00021913],
[ 0.00018554, -0.00009357],
[ 0.00009528, -0.00001407]]) #2
After that I try to add some points with the coordinates exactly like in the first array X to the existing model:
Y = np.array([[0, 0, 0], [1, 1, 1]])
model.fit_transform(Y)
Output:
array([[ 0.00017882, 0.00004002], #1
[ 0.00009546, 0.00022409]]) #2
But coords in the second array not equal to the first and last coords from the first array.
I understand that this is the right behaviour, but how can I add new coords to the model and get the same coords in the output array for the same coords in the input array?
Also I still need to get closest points even after appending new points.
Quoting the author of t-SNE from here: https://lvdmaaten.github.io/tsne/
Once I have a t-SNE map, how can I embed incoming test points in that map?
t-SNE learns a non-parametric mapping, which means that it does not learn an explicit function that maps data from the input space to the map. Therefore, it is not possible to embed test points in an existing map (although you could re-run t-SNE on the full dataset). A potential approach to deal with this would be to train a multivariate regressor to predict the map location from the input data. Alternatively, you could also make such a regressor minimize the t-SNE loss directly, which is what I did in this paper.
Also, this answer on stats.stackexchange.com contains ideas and a link to
a very nice and very fast recent Python implementation of t-SNE https://github.com/pavlin-policar/openTSNE that allows embedding of new points out of the box
and a link to https://github.com/berenslab/rna-seq-tsne/.