How to I get the nearst items (neighbors) with knn? - python

I would like to provide suggestions as to which items are similar to each other. I'm using here for k-nearst-neighbors. Now my question is how do I get the nearest neighbors with the probability?
I would like to have something [[item, probability] ..., [item n, probability n]].
How do I get such a list?
import pandas as pd
from scipy.sparse import csr_matrix
from sklearn.neighbors import NearestNeighbors
list = [[0, 3, 8, 0], [0, 8, 7, 0], [0, 2, 9, 0], [1, 10, 10, 1], [2, 3, 8, 2], [2, 10, 10, 2], [3, 4, 12, 3], [3, 12, 4, 3], [3, 3, 8, 3], [4, 12, 4, 4], [4, 3, 8, 4], [4, 4, 12, 4], [5, 8, 7, 5], [5, 6, 13, 5], [5, 3, 8, 5], [6, 0, 3, 6], [6, 5, 11, 6], [6, 12, 4, 6], [7, 9, 6, 7], [7, 9, 6, 7], [8, 13, 5, 8], [9, 1, 0, 9], [9, 7, 2, 9], [9, 11, 1, 9], [9, 11, 1, 9]]
# Note: location isn't relevant
df = DataFrame (list,columns=['buyerid','itemid', 'group', 'location'])
sparse_item_user = sparse.csr_matrix((df['group'].astype(float), (df['itemid'], df['buyerid'])))
sparse_user_item = sparse.csr_matrix((df['group'].astype(float), (df['buyerid'], df['itemid'])))
model_knn= NearestNeighbors(metric='cosine', algorithm='brute', n_neighbors=20)
model_knn.fit(sparse_item_user)
desired_item = 8
model_knn.kneighbors(....) # Now get the nearst items


From the sklearn docs, the following is the signature of kneighbors:
kneighbors(X=None, n_neighbors=None, return_distance=True)
Thus, to get the nearest neighbor of some point x, you do kneighbors(x, return_distance=True). In this case, n_neighbors was already specified in your constructor to be 20, so we need not give it here.

Related

Why does modifying a copy of a list also modify other copies of the same list?

grid = np.empty((10,0)).tolist()
coords = grid.copy()
distances = grid.copy()
print(id(grid), id(coords), id(distances))
for i in range(10):
for j in range(10):
distances[i].append(i+j)
coords[i].append([i, j])
print(distances)
print(coords)
I created a nested list for turning into a 2D grid. I want to create two different 2D grid, one containing the coordinate of each element, the other containing the distance of each coordinate from (0,0).
I used the .copy() method twice, and confirmed that coords and distances are in fact different locations in memory. So why does my code output both as the same incorrect list?
Output:
[[0, [0, 0], 1, [0, 1], 2, [0, 2], 3, [0, 3], 4, [0, 4], 5, [0, 5], 6, [0, 6], 7, [0, 7], 8, [0, 8], 9, [0, 9]], [1, [1, 0], 2, [1, 1], 3, [1, 2], 4, [1, 3], 5, [1, 4], 6, [1, 5], 7, [1, 6], 8, [1, 7], 9, [1, 8], 10, [1, 9]], [2, [2, 0], 3, [2, 1], 4, [2, 2], 5, [2, 3], 6, [2, 4], 7, [2, 5], 8, [2, 6], 9, [2, 7], 10, [2, 8], 11, [2, 9]], [3, [3, 0], 4, [3, 1], 5, [3, 2], 6, [3, 3], 7, [3, 4], 8, [3, 5], 9, [3, 6], 10, [3, 7], 11, [3, 8], 12, [3, 9]], [4, [4, 0], 5, [4, 1], 6, [4, 2], 7, [4, 3], 8, [4, 4], 9, [4, 5], 10, [4, 6], 11, [4, 7], 12, [4, 8], 13, [4, 9]], [5, [5, 0], 6, [5, 1], 7, [5, 2], 8, [5, 3], 9, [5, 4], 10, [5, 5], 11, [5, 6], 12, [5, 7], 13, [5, 8], 14, [5, 9]], [6, [6, 0], 7, [6, 1], 8, [6, 2], 9, [6, 3], 10, [6, 4], 11, [6, 5], 12, [6, 6], 13, [6, 7], 14, [6, 8], 15, [6, 9]], [7, [7, 0], 8, [7, 1], 9, [7, 2], 10, [7, 3], 11, [7, 4], 12, [7, 5], 13, [7, 6], 14, [7, 7], 15, [7, 8], 16, [7, 9]], [8, [8, 0], 9, [8, 1], 10, [8, 2], 11, [8, 3], 12, [8, 4], 13, [8, 5], 14, [8, 6], 15, [8, 7], 16, [8, 8], 17, [8, 9]], [9, [9, 0], 10, [9, 1], 11, [9, 2], 12, [9, 3], 13, [9, 4], 14, [9, 5], 15, [9, 6], 16, [9, 7], 17, [9, 8], 18, [9, 9]]]

How to transfer List of nested lists into a NumPy array?

I have the following sample object:
list_sample = [['N', [], None], ['B:maj', [3, 6, 11], 11], ['C#:maj', [1, 5, 8], 1], ['Bb:min', [1, 5, 10], 10], ['Eb:min/b3', [3, 6, 6, 10], 6], ['B:maj', [3, 6, 11], 11], ['C#:maj', [1, 5, 8], 1], ['F#:maj', [1, 6, 10], 6], ['F#:maj', [1, 6, 10], 6], ['B:maj', [3, 6, 11], 11], ['C#:maj', [1, 5, 8], 1], ['Bb:min', [1, 5, 10], 10], ['Eb:min', [3, 6, 10], 3], ['B:maj', [3, 6, 11], 11]]
I would like to transfer this list into a ndarray, so I tried this method suggested in this postDebugging Numpy VisibleDeprecationWarning (ndarray from ragged nested sequences):
np.array(formatted_chords,dtype = object)
But it gives me undesired result like this:
[['N' list([]) None]
['B:maj' list([3, 6, 11]) 11]
['C#:maj' list([1, 5, 8]) 1]
['Bb:min' list([1, 5, 10]) 10]
['Eb:min/b3' list([3, 6, 6, 10]) 6]
['B:maj' list([3, 6, 11]) 11]]
My target output would a ndarray which looks like this:
[['N',[],None]
['B:maj',[3, 6, 11]),11]
['C#:maj',[1, 5, 8]),1]
['Bb:min',[1, 5, 10]),10]
['Eb:min/b3',[3, 6, 6, 10],6]
['B:maj',[3, 6, 11],11]]
Can anybody shares how to do this transformation?

Is there a better method to create such a numpy array?

I want a numpy array like this:
b = np.array([[1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3],
[4, 4, 4, 4, 4, 4],
[5, 5, 5, 5, 5, 5],
[6, 6, 6, 6, 6, 6],
[7, 7, 7, 7, 7, 7],
[8, 8, 8, 8, 8, 8],
[9, 9, 9, 9, 9, 9]])
Is there a faster way to create a NumPy array like this instead of typing them manually?

You can do something like this:
>>> np.repeat(np.arange(1, 10).reshape(-1,1), 6, axis=1)
array([[1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3],
[4, 4, 4, 4, 4, 4],
[5, 5, 5, 5, 5, 5],
[6, 6, 6, 6, 6, 6],
[7, 7, 7, 7, 7, 7],
[8, 8, 8, 8, 8, 8],
[9, 9, 9, 9, 9, 9]])
Explanation:
np.arange(1, 10).reshape(-1,1) creates an array
array([[1],
[2],
[3],
[4],
[5],
[6],
[7],
[8],
[9]])
np.repeat(_, 6, axis=1) repeats this 6 times on the first (or second in human words) axis.

Yes. There are plenty of methods. This is one:
np.repeat(np.arange(1,10),6,axis=0).reshape(9,6)

Another method is to use broadcasting:
>>> np.arange(1,10)[:,None] * np.ones(6, dtype=int)
array([[1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3],
[4, 4, 4, 4, 4, 4],
[5, 5, 5, 5, 5, 5],
[6, 6, 6, 6, 6, 6],
[7, 7, 7, 7, 7, 7],
[8, 8, 8, 8, 8, 8],
[9, 9, 9, 9, 9, 9]])

For any w*l size, convert a list of lists into an np.array like so:
w = 6
l = 9
[np.array([[1+i]*w for i in range(d)])
array([[1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3],
[4, 4, 4, 4, 4, 4],
[5, 5, 5, 5, 5, 5],
[6, 6, 6, 6, 6, 6],
[7, 7, 7, 7, 7, 7],
[8, 8, 8, 8, 8, 8],
[9, 9, 9, 9, 9, 9]])

np.transpose(np.array(([np.arange(1,10)] * 6)))
np.arange(1,10) creates an numpy array from 1 to 9.
[] puts the array into a list.
*6 augments the array 6 times.
np.array() converts the resulting structure (list of arrays) to a numpy array
np.transpose() rotates the orientation of the numpy array to get vertical one.

numpy: broadcast array by rolling along new axis with variable shift given in 2nd array

I know that numpy.roll can shift an array along one or more existing axes. How would I create a new axis on array x along which I want to have views or copies of itself rolled by array shift?
Example:
x = np.arange(10)
shift = np.array([2, 4])
#input
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
#output
array(
[[8, 6],
[9, 7],
[0, 8],
[1, 9],
[2, 0],
[3, 1],
[4, 2],
[5, 3],
[6, 4],
[7, 5]])
Edit: I'm looking for a general solution (ideally without looping) that can also be applied on higher dimensional arrays. Another example:
x = np.arange(20).reshape(2, 10)
shift = np.array([2, 4])
#input
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]])
#output
array(
[[[ 8, 6],
[ 9, 7],
[ 0, 8],
[ 1, 9],
[ 2, 0],
[ 3, 1],
[ 4, 2],
[ 5, 3],
[ 6, 4],
[ 7, 5]],
[[18, 16],
[19, 17],
[10, 18],
[11, 19],
[12, 10],
[13, 11],
[14, 12],
[15, 13],
[16, 14],
[17, 15]]])

Here's a vectorized solution leveraging broadcasting that covers generic n-dim array cases -
np.take(x,(-shift + np.arange(x.shape[-1])[:,None]),axis=-1)
Sample runs
1) x as 1D -
In [114]: x = np.arange(10)
...: shift = np.array([2, 4])
In [115]: np.take(x,(-shift + np.arange(x.shape[-1])[:,None]),axis=-1)
Out[115]:
array([[8, 6],
[9, 7],
[0, 8],
[1, 9],
[2, 0],
[3, 1],
[4, 2],
[5, 3],
[6, 4],
[7, 5]])
2) x as 2D -
In [116]: x = np.arange(20).reshape(2, 10)
...: shift = np.array([2, 4])
In [117]: np.take(x,(-shift + np.arange(x.shape[-1])[:,None]),axis=-1)
Out[117]:
array([[[ 8, 6],
[ 9, 7],
[ 0, 8],
[ 1, 9],
[ 2, 0],
[ 3, 1],
[ 4, 2],
[ 5, 3],
[ 6, 4],
[ 7, 5]],
[[18, 16],
[19, 17],
[10, 18],
[11, 19],
[12, 10],
[13, 11],
[14, 12],
[15, 13],
[16, 14],
[17, 15]]])

I almost hate to provide this alternative because I think #BenT's answer is simple and logical
np.array([np.roll(x,sh) for sh in shift]).T
np.stack([np.roll(x,sh) for sh in shift], axis=1) # may be easier to generalize
but I can do the original x=np.arange(10) case with as_strided:
Perform all shifts:
In [352]: arr = np.lib.stride_tricks.as_strided(np.hstack((x,x)),shape=(10,10), strides=(8,8))
In [353]: arr
Out[353]:
array([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[1, 2, 3, 4, 5, 6, 7, 8, 9, 0],
[2, 3, 4, 5, 6, 7, 8, 9, 0, 1],
[3, 4, 5, 6, 7, 8, 9, 0, 1, 2],
[4, 5, 6, 7, 8, 9, 0, 1, 2, 3],
[5, 6, 7, 8, 9, 0, 1, 2, 3, 4],
[6, 7, 8, 9, 0, 1, 2, 3, 4, 5],
[7, 8, 9, 0, 1, 2, 3, 4, 5, 6],
[8, 9, 0, 1, 2, 3, 4, 5, 6, 7],
[9, 0, 1, 2, 3, 4, 5, 6, 7, 8]])
Then select the ones you want:
In [358]: arr[::-1][shift-1]
Out[358]:
array([[8, 9, 0, 1, 2, 3, 4, 5, 6, 7],
[6, 7, 8, 9, 0, 1, 2, 3, 4, 5]])
I wrote and tested the stack version with one try, but had to try several things to get the as_strided right.
I'd also prefer generalizing the list comprehension to higher dimensions.
For your 2d x:
np.stack([np.roll(x,sh, axis=1) for sh in shift],2)

numpy optimize way(remove loops)

I want to use this code on very huge array. this code take long time to execute and it is not efficient.
is there any way to remove loop and convert this code to optimum way?
>>> import numpy as np
>>> x=np.random.randint(10, size=(4,5,3))
>>> x
array([[[3, 2, 6],
[4, 6, 6],
[3, 7, 9],
[6, 4, 2],
[9, 0, 1]],
[[9, 0, 4],
[1, 8, 9],
[6, 8, 1],
[9, 4, 5],
[1, 5, 2]],
[[6, 1, 6],
[1, 8, 8],
[3, 8, 3],
[7, 1, 0],
[7, 7, 0]],
[[5, 6, 6],
[8, 3, 1],
[0, 5, 4],
[6, 1, 2],
[5, 6, 1]]])
>>> y=[]
>>> for i in range(x.shape[1]):
for j in range(x.shape[2]):
y.append(x[:, i, j].tolist())
>>> y
[[3, 9, 6, 5], [2, 0, 1, 6], [6, 4, 6, 6], [4, 1, 1, 8], [6, 8, 8, 3], [6, 9, 8, 1], [3, 6, 3, 0], [7, 8, 8, 5], [9, 1, 3, 4], [6, 9, 7, 6], [4, 4, 1, 1], [2, 5, 0, 2], [9, 1, 7, 5], [0, 5, 7, 6], [1, 2, 0, 1]]

You could permute axes with np.transpose and then reshape to 2D -
y = x.transpose(1,2,0).reshape(-1,x.shape[0])
Append with .tolist() for list output.

yes, either use np.reshape(x, shape) or try it with np.ndarray.flatten(x, order='F') (F for Fortran style, column first, according to your example).
read the documentation to find out which parameters fit the best. IMHO, I think ndarray.flatten is the better and more elegant option for you here. However, depending on your exact wanted solution, you might have to reshape the array first.

Categories

Resources