Hi I have a lemmatized text in the format as shown by lemma. I want to get TfIdf score for each word this is the function that I wrote:
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
lemma=["'Ah", 'yes', u'say', 'softly', 'Harry',
'Potter', 'Our', 'new', 'celebrity', 'You',
'learn', 'subtle', 'science', 'exact', 'art',
'potion-making', u'begin', 'He', u'speak', 'barely',
'whisper', 'caught', 'every', 'word', 'like',
'Professor', 'McGonagall', 'Snape', 'gift',
u'keep', 'class', 'silent', 'without', 'effort',
'As', 'little', 'foolish', 'wand-waving', 'many',
'hardly', 'believe', 'magic', 'I', 'dont', 'expect', 'really',
'understand', 'beauty']
def Tfidf_Vectorize(lemmas_name):
vect = TfidfVectorizer(stop_words='english',ngram_range=(1,2))
vect_transform = vect.fit_transform(lemmas_name)
# First approach of creating a dataframe of weight & feature names
vect_score = np.asarray(vect_transform.mean(axis=0)).ravel().tolist()
vect_array = pd.DataFrame({'term': vect.get_feature_names(), 'weight': vect_score})
vect_array.sort_values(by='weight',ascending=False,inplace=True)
# Second approach of getting the feature names
vect_fn = np.array(vect.get_feature_names())
sorted_tfidf_index = vect_transform.max(0).toarray()[0].argsort()
print('Largest Tfidf:\n{}\n'.format(vect_fn[sorted_tfidf_index[:-11:-1]]))
return vect_array
tf_dataframe=Tfidf_Vectorize(lemma)
print(tf_dataframe.iloc[:5,:])
The output I am getting by:
print('Largest Tfidf:\n{}\n'.format(vect_fn[sorted_tfidf_index[:-11:-1]]))
is
Largest Tfidf:
[u'yes' u'fools' u'fury' u'gale' u'ghosts' u'gift' u'glory' u'glow' u'good'
u'granger']
The result of tf_dataframe
term weight
261 snape 0.027875
238 say 0.022648
211 potter 0.013937
181 mind 0.010453
123 harry 0.010453
60 dark 0.006969
75 dumbledore 0.006969
311 voice 0.005226
125 head 0.005226
231 ron 0.005226
Shouldn't both approaches lead to the same result of top features? I just want to calculate the tfidf scores and get the top 5 features/weight. What am i doing wrong?
I am not sure what I am looking at here but I have the feeling that you're using TfidfVectorizer incorrectly. However, please correct me in case I got the wrong idea of what you're trying.
So.. what you need is a list of documents which you feed to fit_transform(). From that you can construct a matrix where, for example, each column represents a document and each row a word. One cell in that matrix is the tf-idf score of the word i in document j.
Here's an example:
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
documents = [
"This is a document.",
"This is another document with slightly more text.",
"Whereas this is yet another document with even more text than the other ones.",
"This document is awesome and also rather long.",
"The car he drove was red."
]
document_names = ['Doc {:d}'.format(i) for i in range(len(documents))]
def get_tfidf(docs, ngram_range=(1,1), index=None):
vect = TfidfVectorizer(stop_words='english', ngram_range=ngram_range)
tfidf = vect.fit_transform(documents).todense()
return pd.DataFrame(tfidf, columns=vect.get_feature_names(), index=index).T
print(get_tfidf(documents, ngram_range=(1,2), index=document_names))
Which will give you:
Doc 0 Doc 1 Doc 2 Doc 3 Doc 4
awesome 0.0 0.000000 0.000000 0.481270 0.000000
awesome long 0.0 0.000000 0.000000 0.481270 0.000000
car 0.0 0.000000 0.000000 0.000000 0.447214
car drove 0.0 0.000000 0.000000 0.000000 0.447214
document 1.0 0.282814 0.282814 0.271139 0.000000
document awesome 0.0 0.000000 0.000000 0.481270 0.000000
document slightly 0.0 0.501992 0.000000 0.000000 0.000000
document text 0.0 0.000000 0.501992 0.000000 0.000000
drove 0.0 0.000000 0.000000 0.000000 0.447214
drove red 0.0 0.000000 0.000000 0.000000 0.447214
long 0.0 0.000000 0.000000 0.481270 0.000000
ones 0.0 0.000000 0.501992 0.000000 0.000000
red 0.0 0.000000 0.000000 0.000000 0.447214
slightly 0.0 0.501992 0.000000 0.000000 0.000000
slightly text 0.0 0.501992 0.000000 0.000000 0.000000
text 0.0 0.405004 0.405004 0.000000 0.000000
text ones 0.0 0.000000 0.501992 0.000000 0.000000
The two methods you show to get to words and their respective scores calculate the mean over all documents and fetch the max score of each word respectively.
So let's do this and compare the two methods:
df = get_tfidf(documents, ngram_range=(1,2), index=index)
print(pd.DataFrame([df.mean(1), df.max(1)], index=['score_mean', 'score_max']).T)
We can see that the scores are of course different.
score_mean score_max
awesome 0.096254 0.481270
awesome long 0.096254 0.481270
car 0.089443 0.447214
car drove 0.089443 0.447214
document 0.367353 1.000000
document awesome 0.096254 0.481270
document slightly 0.100398 0.501992
document text 0.100398 0.501992
drove 0.089443 0.447214
drove red 0.089443 0.447214
long 0.096254 0.481270
ones 0.100398 0.501992
red 0.089443 0.447214
slightly 0.100398 0.501992
slightly text 0.100398 0.501992
text 0.162002 0.405004
text ones 0.100398 0.501992
Note:
You can convince yourself that this does the same as calling min/max on the TfidfVectorizer:
vect = TfidfVectorizer(stop_words='english', ngram_range=(1,2))
tfidf = vect.fit_transform(documents)
print(tfidf.max(0))
print(tfidf.mean(0))
Related
I would like to extract unique combinations of letters within words using the scikit-learn TF-IDF vectorizer for an NLP problem. However, I'm not interested in individual letters, but letter combinations, so that, e.g. "the" should produce "th" and "he" but not "t", "h" or "e". My understanding is I should be able to use ngram_range. However, using ngram_range=(2,3) is still returning unigrams.
Example:
from sklearn.feature_extraction.text import TfidfVectorizer
examples = ['The cat on the mat',
'Fast and bulbous']
tfidf = TfidfVectorizer(max_features=None,
analyzer='char_wb',
ngram_range=(2, 3))
data=tfidf.fit_transform(examples)
print(pd.DataFrame(data=data.todense(),
index=examples,
columns = tfidf.get_feature_names_out()))
gives me the 2- and 3-gram results as expected but also unigrams (i.e. I don't want "a", "b", etc.):
a an b bu c \
The cat on the mat 0.000000 0.000000 0.000000 0.000000 0.139994
Fast and bulbous 0.181053 0.181053 0.181053 0.181053 0.000000
ca f fa m ma ... \
The cat on the mat 0.139994 0.000000 0.000000 0.139994 0.139994 ...
Fast and bulbous 0.000000 0.181053 0.181053 0.000000 0.000000 ...
s st st t th \
The cat on the mat 0.000000 0.000000 0.000000 0.199213 0.279987
Fast and bulbous 0.181053 0.181053 0.181053 0.128821 0.000000
the ul ulb us us
The cat on the mat 0.279987 0.000000 0.000000 0.000000 0.000000
Fast and bulbous 0.000000 0.181053 0.181053 0.181053 0.181053
[2 rows x 53 columns]
I would've expected this output with ngram_range=(1,3) but not ngram_range=(2,3).
Edit:
I just noticed that "a" is extracted from "Fast and bulbous", presumably as it occurs as " a", i.e. with a space before the "a", but not in "The cat on the mat" as the "a" in "cat" is surrounded by "c" and "t". Likewise, "u" is not extracted as there is no space surrounding it in either text.
It seems like TfidfVectorizer is extracting bigrams including spaces. Is there a way to turn this off? (I though using analyzer='char_wb' searched within words rather than across words).
I constructed a callable to pass to analyzer. It is stolen from based on the function from the source code of TfidfVectorizer used when analyzer is set to 'char_wb':
def char_wb_ngrams(text_document, ngram_range):
"""Callable for TfidfVectorizer analyzer, based on _char_wb_ngrams from TfidfVectorizer source code at
https://github.com/scikit-learn/scikit-learn/blob/f3f51f9b6/sklearn/feature_extraction/text.py"""
ngrams = []
min_n, max_n = ngram_range
for w in text_document.lower().split():
# This line in _char_wb_ngrams pads words with spaces and needs to be removed:
#w = " " + w + " "
w_len = len(w)
for n in range(min_n, max_n + 1):
offset = 0
ngrams.append(w[offset : offset + n])
while offset + n < w_len:
offset += 1
ngrams.append(w[offset : offset + n])
if offset == 0: # count a short word (w_len < n) only once
break
return ngrams
This works on the example data from above:
from functools import partial
tfidf_no_space = TfidfVectorizer(max_features=None,
analyzer=partial(char_wb_ngrams, ngram_range=(2,3)),
ngram_range=(2, 3))
data=tfidf_no_space.fit_transform(examples)
print(pd.DataFrame(data=data.todense(),
index=examples,
columns = tfidf_no_space.get_feature_names_out()))
which yields
an and as ast at \
The cat on the mat 0.000000 0.000000 0.000000 0.000000 0.436436
Fast and bulbous 0.229416 0.229416 0.229416 0.229416 0.000000
bo bou bu bul ca ... \
The cat on the mat 0.000000 0.000000 0.000000 0.000000 0.218218 ...
Fast and bulbous 0.229416 0.229416 0.229416 0.229416 0.000000 ...
nd on ou ous st \
The cat on the mat 0.000000 0.218218 0.000000 0.000000 0.000000
Fast and bulbous 0.229416 0.000000 0.229416 0.229416 0.229416
th the ul ulb us
The cat on the mat 0.436436 0.436436 0.000000 0.000000 0.000000
Fast and bulbous 0.000000 0.000000 0.229416 0.229416 0.229416
[2 rows x 28 columns]
I'm not sure this would work with punctuation, though. It would be good to have a version that strips punctuation and also doesn't require the call to partial (which fixes ngram_range in the function).
I have to similarity between columns within the one data frame. The expected result is the same as the correlation matrix output, but the calculation function is different(I wrote my self calc function). So the calc function should get calc_func(column1, column2). The idea is to get similarities between columns. row size is not important. as an Output I expect (937,937) matrix.
Sample data
0011 0012 0013 0014 0015 0019 0111 0112 0121 0122 0123 0125 0129 0161 0168 0172 0174 0175 0176 0221 0222 0223 0224 0230 0241 0242 0243 0249 0251 0252 0341 0342 0344 0345 0351 0352 0353 0361 0362 0363 0371 0372 0411 0412 0421 0422 0423 0430 0441 0449 0452 0453 0459 0461 0471 0472 0481 0482 0483 0484 0485 0541 0542 0544 0545 0546 0547 0548 0561 0564 0566 0567 0571 0572 0573 0574 0575 0576 0577 0579 0581 0583 \
Reporter ISO
AFG 0.149474 0.699753 0.000000e+00 0.000000 0.000000 6.084805 0.000000e+00 0.013655 0.123035 0.000000e+00 0.000000 0.011263 0.000000 0.000000e+00 0.000000 0.000000e+00 0.000000 0.000000 0.000000 0.000000 0.000000 0.040835 0.009775 0.000000 0.000000 0.000000 1.902343e-04 0.003110 0.000000 0.000000 9.900480e-04 0.000382 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.002613 0.002373 0.000184 0.000000 0.000000e+00 5.570409e-04 0.000000 0.001367 0.023009 1.074305 0.000000 4.309246e-04 2.267049 0.135528 6.845710 0.000172 4.785010e-02 5.620574e-04 0.015391 0.000000 0.000000 0.008071 0.000000 0.602458 56.772035 4.902713e+01 11.542497 0.175537 0.000000 8.314311 6.899700e-01 0.009341 0.000000e+00 0.118446 0.465433 0.634222 0.008141 4.406345e+01 1.806608e+02 1457.266474 37.572639 16.111153 0.278868 5.828552e-01
AGO 0.000233 0.000000 2.169950e-05 0.000436 0.000021 0.206904 1.850937e-05 0.001081 0.000054 4.163925e-04 0.000437 0.000348 0.000059 1.287730e-04 0.000289 9.425705e-04 0.000140 0.002698 0.000444 0.000116 0.002252 0.000614 0.000295 0.000481 0.000003 0.000008 0.000000e+00 0.000142 0.000742 0.002136 3.700936e-01 1.594887 0.024370 0.000002 0.039695 0.146148 0.000020 0.267286 0.866269 0.036852 0.000384 0.046401 4.496454e-06 0.000000e+00 0.000834 0.016216 0.001110 0.000000 0.000065 7.354149e-04 0.000061 0.004332 0.000039 0.239055 2.501597e-01 3.830196e-04 0.000546 0.008450 0.015806 0.001086 0.002724 0.009187 0.005919 3.321169e-04 0.002146 0.000693 0.001050 0.006553 2.621612e-03 0.017289 8.732829e-05 0.000309 0.000343 0.000303 0.053893 5.683467e-04 6.637084e-05 0.000005 0.000036 0.001527 0.000022 1.886017e-07
ALB 0.004472 0.093826 0.000000e+00 0.000000 0.000000 4.959089 5.096002e-02 0.000000 0.000000 2.111634e-03 0.000487 0.003162 17.984117 1.681137e-02 0.000287 3.287117e-02 0.001309 0.001702 0.000093 0.005981 0.000027 0.004139 0.007258 0.000442 0.000000 0.122049 0.000000e+00 0.028040 1.376963 0.109314 2.201071e+00 0.646953 0.427123 0.055488 37.156633 24.666195 0.000416 0.452249 0.423161 1.855032 9.630443 16.673592 2.321445e-03 5.822343e-03 0.264946 0.000000 0.001616 0.000000 0.067036 1.468721e-03 0.000867 0.000000 0.000000 0.051276 3.280251e-02 1.379145e-02 0.026767 0.000000 0.315969 0.634852 0.004309 0.343613 0.302088 2.262782e+01 4.408535 0.013666 1.185906 1.818876 1.082149e+00 0.031302 5.695562e-03 3.008238 1.286605 0.064267 0.004062 1.028946e+00 5.242426e-02 2.020501 0.595951 1.282575 0.059749 8.487325e-01
Using exactly the example in the docs for df.corr():
def histogram_intersection(a, b):
v = np.minimum(a, b).sum().round(decimals=1)
return v
df = pd.DataFrame([(.2, .3), (.0, .6), (.6, .0), (.2, .1)],
columns=['dogs', 'cats'])
df.corr(method=histogram_intersection)
# output:
dogs cats
dogs 1.0 0.3
cats 0.3 1.0
So just pass your function in as the method parameter:
df.corr(method=calc_func)
when I build a matrix using the last row of my dataframe:
x = w.iloc[-1, :]
a = np.mat(x).T
it goes:
ValueError: ndarray is not contiguous
`print the x shows(I have 61 columns in my dataframe):
print(x)
cdl2crows 0.000000
cdl3blackcrows 0.000000
cdl3inside 0.000000
cdl3linestrike 0.000000
cdl3outside 0.191465
cdl3starsinsouth 0.000000
cdl3whitesoldiers_x 0.000000
cdl3whitesoldiers_y 0.000000
cdladvanceblock 0.000000
cdlhighwave 0.233690
cdlhikkake 0.218209
cdlhikkakemod 0.000000
...
cdlidentical3crows 0.000000
cdlinneck 0.000000
cdlinvertedhammer 0.351235
cdlkicking 0.000000
cdlkickingbylength 0.000000
cdlladderbottom 0.002259
cdllongleggeddoji 0.629053
cdllongline 0.588480
cdlmarubozu 0.065362
cdlmatchinglow 0.032838
cdlmathold 0.000000
cdlmorningdojistar 0.000000
cdlmorningstar 0.327749
cdlonneck 0.000000
cdlpiercing 0.251690
cdlrickshawman 0.471466
cdlrisefall3methods 0.000000
Name: 2010-01-04, Length: 61, dtype: float64
how to solve it? so many thanks
np.mat expects array form of input.
refer to the doc
doc
So your code should be
x = w.iloc[-1, :].values
a = np.mat(x).T
.values will give numpy array format of dataframe values, so np.mat will work.
Use np.array instead of np.mat:
a = np.array(x).T
I need to calculate the tfidf matrix for few sentences. sentence include both numbers and words.
I am using below code to do so
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
data1=['1/8 wire','4 tube','1-1/4 brush']
dataset=pd.DataFrame(data1, columns=['des'])
vectorizer1 = TfidfVectorizer(lowercase=False)
tf_idf_matrix = pd.DataFrame(vectorizer1.fit_transform(dataset['des']).toarray(),columns=vectorizer1.get_feature_names())
Tfidf function is considering only words as its vocabulary i.e
Out[3]: ['brush', 'tube', 'wire']
but i need numbers to be part of tokens
expected
Out[3]: ['brush', 'tube', 'wire','1/8','4','1-1/4']
After reading TfidfVectorizer documentation, I came to know have to change token_pattern and tokenizer parameters. But I am not getting how to change it to consider numbers and punctuation.
can anyone please tell me how to change the parameters.
You're right, token_pattern requires a custom regex pattern, pass a regex that treats any one or more characters that don't contain whitespace characters as a single token.
tfidf = TfidfVectorizer(lowercase=False, token_pattern=r'\S+')
tf_idf_matrix = pd.DataFrame(
tfidf.fit_transform(dataset['des']).toarray(),
columns=tfidf.get_feature_names()
)
print(tf_idf_matrix)
1-1/4 1/8 4 brush tube wire
0 0.000000 0.707107 0.000000 0.000000 0.000000 0.707107
1 0.000000 0.000000 0.707107 0.000000 0.707107 0.000000
2 0.707107 0.000000 0.000000 0.707107 0.000000 0.000000
you can explicitly point out in token_pattern parameter the symbols you would like to parse:
token_pattern_ = r'([a-zA-Z0-9-/]{1,})'
where {1,} indicates the minimum number of symbols the word should contain. End then you pass this as a parameter to token_pattern:
tfidf = TfidfVectorizer(token_pattern = token_pattern_)
I've a list of raw document, already filtered and removed english stopwords:
rawDocument = ['sport british english sports american english includes forms competitive physical activity games casual organised ...', 'disaster serious disruption occurring relatively short time functioning community society involving ...', 'government system group people governing organized community often state case broad associative definition ...', 'technology science craft greek τέχνη techne art skill cunning hand λογία logia collection techniques ...']
and I've used
from sklearn.feature_extraction.text import TfidfVectorizer
sklearn_tfidf = TfidfVectorizer(norm='l2', min_df=0, use_idf=True, smooth_idf=False, sublinear_tf=False)
sklearn_representation = sklearn_tfidf.fit_transform(rawDocuments)
But I got a
<4x50 sparse matrix of type '<class 'numpy.float64'>'
with 51 stored elements in Compressed Sparse Row format>
and I cant interpret the result. So, am I using the right tool or have I to change the way?
My goal is to get the relevant word in each document, in order to perform a cosine similarity with other words in a query document.
Thank you in advance.
Very often Pandas module can be used to better visualize your data:
Demo:
import pandas as pd
df = pd.SparseDataFrame(sklearn_tfidf.fit_transform(rawDocument),
columns=sklearn_tfidf.get_feature_names(),
default_fill_value=0)
Result:
In [85]: df
Out[85]:
activity american art associative british ... system techne techniques technology time
0 0.25 0.25 0.000000 0.000000 0.25 ... 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.00 0.00 0.000000 0.000000 0.00 ... 0.000000 0.000000 0.000000 0.000000 0.308556
2 0.00 0.00 0.000000 0.282804 0.00 ... 0.282804 0.000000 0.000000 0.000000 0.000000
3 0.00 0.00 0.288675 0.000000 0.00 ... 0.000000 0.288675 0.288675 0.288675 0.000000
[4 rows x 48 columns]