Develop Reference

Cannot load Doc2vec object using gensim

I am trying to load a pre-trained Doc2vec model using gensim and use it to map a paragraph to a vector. I am referring to https://github.com/jhlau/doc2vec and the pre-trained model I downloaded is the English Wikipedia DBOW, which is also in the same link. However, when I load the Doc2vec model on wikipedia and infer vectors using the following code:
import gensim.models as g
import codecs
model="wiki_sg/word2vec.bin"
test_docs="test_docs.txt"
output_file="test_vectors.txt"
#inference hyper-parameters
start_alpha=0.01
infer_epoch=1000
#load model
test_docs = [x.strip().split() for x in codecs.open(test_docs, "r", "utf-8").readlines()]
m = g.Doc2Vec.load(model)
#infer test vectors
output = open(output_file, "w")
for d in test_docs:
output.write(" ".join([str(x) for x in m.infer_vector(d, alpha=start_alpha, steps=infer_epoch)]) + "\n")
output.flush()
output.close()
I get an error:
/Users/zhangji/Desktop/CSE547/Project/NLP/venv/lib/python2.7/site-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
'See the migration notes for details: %s' % _MIGRATION_NOTES_URL
Traceback (most recent call last):
File "/Users/zhangji/Desktop/CSE547/Project/NLP/AbstractMapping.py", line 19, in <module>
output.write(" ".join([str(x) for x in m.infer_vector(d, alpha=start_alpha, steps=infer_epoch)]) + "\n")
AttributeError: 'Word2Vec' object has no attribute 'infer_vector'
I know there are couple of threads regarding the infer_vector issue on stack overflow, but none of them resolved my problem. I downloaded the gensim package using
pip install git+https://github.com/jhlau/gensim
In addition, after I looked at the source code in gensim package, I found that when I use Doc2vec.load(), the Doc2vec class doesn't really have a load() function by itself, but since it is a subclass of Word2vec, it calls the super method of load() in Word2vec and then make the model m a Word2vec object. However, the infer_vector() function is unique to Doc2vec and does not exist in Word2vec, and that's why it is causing the error. I also tried casting the model m to a Doc2vec, but I got this error:
>>> g.Doc2Vec(m)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/zhangji/Library/Python/2.7/lib/python/site-packages/gensim/models/doc2vec.py", line 599, in __init__
self.build_vocab(documents, trim_rule=trim_rule)
File "/Users/zhangji/Library/Python/2.7/lib/python/site-packages/gensim/models/word2vec.py", line 513, in build_vocab
self.scan_vocab(sentences, trim_rule=trim_rule) # initial survey
File "/Users/zhangji/Library/Python/2.7/lib/python/site-packages/gensim/models/doc2vec.py", line 635, in scan_vocab
for document_no, document in enumerate(documents):
File "/Users/zhangji/Library/Python/2.7/lib/python/site-packages/gensim/models/word2vec.py", line 1367, in __getitem__
return vstack([self.syn0[self.vocab[word].index] for word in words])
TypeError: 'int' object is not iterable
In fact, all I want with gensim for now is to convert a paragraph to a vector using a pre-trained model that works well on academic articles. For some reasons I don't want to train the models on my own. I would be really grateful if someone can help me resolve the issue.
Btw, I am using python2.7, and the current gensim version is 0.12.4.
Thanks!

I would avoid using either the 4-year-old nonstandard gensim fork at https://github.com/jhlau/doc2vec, or any 4-year-old saved models that only load with such code.
The Wikipedia DBOW model there is also suspiciously small at 1.4GB. Wikipedia had well over 4 million articles even 4 years ago, and a 300-dimensional Doc2Vec model trained to have doc-vectors for the 4 million articles would be at least 4000000 articles * 300 dimensions * 4 bytes/dimension = 4.8GB in size, not even counting other parts of the model. (So, that download is clearly not the 4.3M doc, 300-dimensional model mentioned in the associated paper – but something that's been truncated in other unclear ways.)
The current gensim version is 3.8.3, released a few weeks ago.
It'd likely take a bit of tinkering, and an overnight or more runtime, to build your own Doc2Vec model using current code and a current Wikipedia dump - but then you'd be on modern supported code, with a modern model that better understands words coming into use in the last 4 years. (And, if you trained a model on a corpus of the exact kind of documents of interest to you – such as academic articles – the vocabulary, word-senses, and match to your own text-preprocessing to be used on later inferred documents will all be better.)
There's a Jupyter notebook example of building a Doc2Vec model from Wikipedia that either functional or very-close-to-functional inside the gensim source tree at:
https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-wikipedia.ipynb

Error when Converting Node Type Slice with onnx_coreml

I have converted my darknet YOLOv3-SPP model into a PyTorch .pt model. I then converted the .pt to a .onnx. My end goal is to get to a CoreML model. I tried to use this GitHub repository. However when converting my model I am getting an error like this...
...
145/229: Converting Node Type LeakyRelu
146/229: Converting Node Type Conv
147/229: Converting Node Type Reshape
148/229: Converting Node Type Transpose
149/229: Converting Node Type Reshape
150/229: Converting Node Type Slice
Traceback (most recent call last):
File "convert2.py", line 11, in <module>
coreml_model = convert(model_proto, image_input_names=['inputImage'], image_output_names=['outputImage'], minimum_ios_deployment_target='13')
File "/usr/local/lib/python3.6/dist-packages/onnx_coreml/converter.py", line 626, in convert
_convert_node_nd(builder, node, graph, err)
File "/usr/local/lib/python3.6/dist-packages/onnx_coreml/_operators_nd.py", line 2387, in _convert_node_nd
return converter_fn(builder, node, graph, err)
File "/usr/local/lib/python3.6/dist-packages/onnx_coreml/_operators_nd.py", line 2011, in _convert_slice
end_masks=end_masks
File "/usr/local/lib/python3.6/dist-packages/coremltools/models/neural_network/builder.py", line 4220, in add_slice_static
assert len(strides) == rank
AssertionError
The script I am using is this...
import sys
from onnx import onnx_pb
from onnx_coreml import convert
model_in = sys.argv[1]
model_out = sys.argv[2]
model_file = open(model_in, 'rb')
model_proto = onnx_pb.ModelProto()
model_proto.ParseFromString(model_file.read())
coreml_model = convert(model_proto, image_input_names=['inputImage'], image_output_names=['outputImage'], minimum_ios_deployment_target='13')
coreml_model.save(model_out)
This simple python script should work, but I don't know why I am getting this error. I am very new to Machine Learning, so I do not understand how I can even begin to try to solve this issue. What should I do in order to convert my .onnx model to CoreML successfully?

Looks like rank mis-match between input tensor rank and slice parameter
Could you please file bug at onnx-coreml
As #matthijs-hollemans commented, try installing latest onnx-coreml
pip install onnx-coreml==1.2
Few other concerns:
What is the version of onnx model you are working with? with Operator-9 slice behavior is changed and that could be potential failure point from converter.
Could you please attach ONNX model as well?

Facing issue while loading the pre-trained model

I've trained my model using google colab and saved it as model.pkl. When I try to load the model in my laptop it is throwing the below error:
Traceback (most recent call last):
File "app.py", line 8, in <module>
model = pickle.load(open('model.pkl', 'rb'))
File "sklearn\tree\_tree.pyx", line 606, in sklearn.tree._tree.Tree.__cinit__
ValueError: Buffer dtype mismatch, expected 'SIZE_t' but got 'long long'
I've done some research on the above error and got to know that the random forest code uses different types for indices on 32-bit and 64-bit machines. I've seen similar question on this platform but NOT satisfied with the accepted answer because the answer suggesting to train the model again which is not suitable in case since there are lot of thing to re-do and i don't want to put load on the server again.
Any suggestions or solutions ?

Not sure about '.pkl' format,but can you try saving it as
model.save('modelweight.h5') and then load as model.load ('modelweight.h5').
This shall work.
Thanks.

try to use cpickle instead of pickle
try:
import cPickle as pickle
except:
import pickle
f = open('model.pkl','w+')
pickle.dump(model, f)#to save the model into file
f = open('model.pkl','r')
model = pickle.load(f)

How to fix 'Error(s) in loading state_dict for AWD_LSTM' when using fast-ai

I am using fast-ai library in order to train a sample of the IMDB reviews dataset. My goal is to achieve sentiment analysis and I just wanted to start with a small dataset (this one contains 1000 IMDB reviews). I have trained the model in a VM by using this tutorial.
I saved the data_lm and data_clas model, then the encoder ft_enc and after that I saved the classifier learner sentiment_model. I, then, got those 4 files from the VM and put them in my machine and wanted to use those pretrained models in order to classify sentiment.
This is what I did:
# Use the IMDB_SAMPLE file
path = untar_data(URLs.IMDB_SAMPLE)
# Language model data
data_lm = TextLMDataBunch.from_csv(path, 'texts.csv')
# Sentiment classifier model data
data_clas = TextClasDataBunch.from_csv(path, 'texts.csv',
vocab=data_lm.train_ds.vocab, bs=32)
# Build a classifier using the tuned encoder (tuned in the VM)
learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)
learn.load_encoder('ft_enc')
# Load the trained model
learn.load('sentiment_model')
After that, I wanted to use that model in order to predict the sentiment of a sentence. When executing this code, I ran into the following error:
RuntimeError: Error(s) in loading state_dict for AWD_LSTM:
size mismatch for encoder.weight: copying a param with shape torch.Size([8731, 400]) from checkpoint, the shape in current model is torch.Size([8888, 400]).
size mismatch for encoder_dp.emb.weight: copying a param with shape torch.Size([8731, 400]) from checkpoint, the shape in current model is torch.Size([8888, 400]).
And the Traceback is:
Traceback (most recent call last):
File "C:/Users/user/PycharmProjects/SentAn/mainApp.py", line 51, in <module>
learn = load_models()
File "C:/Users/user/PycharmProjects/SentAn/mainApp.py", line 32, in load_models
learn.load_encoder('ft_enc')
File "C:\Users\user\Desktop\py_code\env\lib\site-packages\fastai\text\learner.py", line 68, in load_encoder
encoder.load_state_dict(torch.load(self.path/self.model_dir/f'{name}.pth'))
File "C:\Users\user\Desktop\py_code\env\lib\site-packages\torch\nn\modules\module.py", line 769, in load_state_dict
self.__class__.__name__, "\n\t".join(error_msgs)))
So the error occurs when loading the encoder. But, I also tried to remove the load_encoder line but the same error occurred at the next line learn.load('sentiment_model').
I searched through the fast-ai forum and noticed that others also had this issue but found no solution. In this post the user says that this might have to do with different preprocessing, though I couldn't understand why this would happen.
Does anyone have an idea about what I am doing wrong?

It seems vocabulary size of data_clas and data_lm are different. I guess the problem is caused by different preprocessing used in data_clas and data_lm. To check my guess I simply used
data_clas.vocab.itos = data_lm.vocab.itos
Before the following line
learn_c = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.3)
This has fixed the error.

Easy.py (libsvm) crashing before creating model

I've been trying to run the easy.py script provided by libsvm-3.17 however it is crashing before creating the model. It does generate the range, scale, output file and gnuplot image but not model, and I believe I do need this to classify the test data. Any input is greatly appreciated :) Thanks.
The error is :
Traceback (most recent call last):
File "tools\easy.py", line 61, in
c,g,rate = map(float,last_line.split())
ValueError: could not convert string to float: b'[0x7FFFB2243810]'
I've tried several data sets, this error pops up everytime.