Glove6b50d parsing: could not convert string to float: '-' - python
I am trying to parse the Glove6b50d data from Kaggle in via Google Colab, then run it through the word2vec process (apologies for the huge URL - it's the fastest link I've found). However, I'm hitting a bug where '-' tokens are not parsed correctly, resulting in the above error.
I have attempted to handle this in a few ways. I've also looked into the load_word2vec_format method itself and tried to ignore errors, however it doesn't seem to make a difference. I've tried a map method on line two, following combinations of advice from these links: [a] and [b]. This hasn't fixed or changed the error message received (i.e. removing it changes nothing in the text).
gloveFile = pd.read_fwf("https://storage.googleapis.com/kagglesdsdata/datasets/652874/1154868/glove.6B.50d.txt?GoogleAccessId=web-data#kaggle-161607.iam.gserviceaccount.com&Expires=1589683535&Signature=kaS%2FTkSmvp7lhqwLJ%2B1lyuvP76PcDpwK1dnsCZEO0AiVXqQm7jsBc1r5g9af%2BuVkOSvMgqUDXYL4O%2BN43pnL5RLs7ns%2B3w%2BEtCYDTfJz6q1O0zfPz4%2BTcD3GV7UAGgVjVNIvncC9fHWcd2YuKwiZaTvKL%2BGRnMkf9b%2BYnOweYeXEeA1sX005krj%2FLMBbVTXmDTwOtN4HwVNb3%2BrbezkWkoEC6sxLPnGcsEKaBe%2Biv%2FuVSQG5FsQlwvRgsSU%2FMgk0c4bi%2FHxF04lrQW0E0s767TIXwHeodRHYpk5KQeKmyd91uKD2Zb8v8xQcf2%2BkmSNGQHbX0mDz8HBwYEmOdV7aMQ%3D%3D&response-content-disposition=attachment%3B+filename%3Dglove.6B.50d.txt",
delimiter="\n\t\s+", header=None)
map(lambda gloveFile: gloveFile.replace(r'[^\x00-\x7F]+' , '-'), gloveFile[0])
numpy.savetxt(r'/usr/local/lib/python3.6/dist-packages/gensim/test/test_data/glove6b50d.txt', gloveFile.values, fmt="%s")
from gensim.models import KeyedVectors
from gensim.test.utils import datapath, get_tmpfile
from gensim.scripts.glove2word2vec import glove2word2vec
glove_file = datapath('glove6b50d.txt')
glove2word2vec(glove_file, "glove6b50d_word2vec.txt")
model = KeyedVectors.load_word2vec_format("glove6b50d_word2vec.txt", binary=False)
Per the comment below, the exact error I'm getting is as follows:
/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:253: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
'See the migration notes for details: %s' % _MIGRATION_NOTES_URL
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-132-6ad5a51f4fb3> in <module>()
9 glove2word2vec(glove_file, "glove6b50d_word2vec.txt")
10
---> 11 model = KeyedVectors.load_word2vec_format("glove6b50d_word2vec.txt", binary=False)
12
2 frames
/usr/local/lib/python3.6/dist-packages/gensim/models/utils_any2vec.py in <listcomp>(.0)
220 if len(parts) != vector_size + 1:
221 raise ValueError("invalid vector on line %s (is this really the text format?)" % line_no)
--> 222 word, weights = parts[0], [datatype(x) for x in parts[1:]]
223 add_word(word, weights)
224 if result.vectors.shape[0] != len(result.vocab):
ValueError: could not convert string to float: '-'
The system works fine using a text file containing only: "test -1.0 1.526 -2.55" or "- -1.0 1.526 -2.55". Additionally, searching the source text file (glove.6B.50d.txt) for occurrences of " - " comes up with no results. I'm on Windows, so I have done so by executing:
findstr /C:" - " glove.6B.50d.txt
Calling print(gloveFile) both pre- and post-map call provide the following output. Note that I've kept the mapping call in for completeness of my efforts, not for its effect.
0 the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.0...
1 , 0.013441 0.23682 -0.16899 0.40951 0.63812 0....
2 . 0.15164 0.30177 -0.16763 0.17684 0.31719 0.3...
3 of 0.70853 0.57088 -0.4716 0.18048 0.54449 0.7...
4 to 0.68047 -0.039263 0.30186 -0.17792 0.42962 ...
... ...
399995 chanty 0.23204 0.025672 -0.70699 -0.045465 0.1...
399996 kronik -0.60921 -0.67218 0.23521 -0.11195 -0.4...
399997 rolonda -0.51181 0.058706 1.0913 -0.55163 -0.1...
399998 zsombor -0.75898 -0.47426 0.4737 0.7725 -0.780...
399999 andberger 0.072617 -0.51393 0.4728 -0.52202 -0...
If I print the first ten lines of the glove6b50d_word2vec.txt file, I get the following text, which matches the word2vec format. Additionally, if I count the occurrences of the string " - " in the document, I find none.
['400000 50\n', 'the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.044457 -0.49688 -0.17862 -0.00066023 -0.6566 0.27843 -0.14767 -0.55677 0.14658 -0.0095095 0.011658 0.10204 -0.12792 -0.8443 -0.12181 -0.016801 -0.33279 -0.1552 -0.23131 -0.19181 -1.8823 -0.76746 0.099051 -0.42125 -0.19526 4.0071 -0.18594 -0.52287 -0.31681 0.00059213 0.0074449 0.17778 -0.15897 0.012041 -0.054223 -0.29871 -0.15749 -0.34758 -0.045637 -0.44251 0.18785 0.0027849 -0.18411 -0.11514 -0.78581\n', ', 0.013441 0.23682 -0.16899 0.40951 0.63812 0.47709 -0.42852 -0.55641 -0.364 -0.23938 0.13001 -0.063734 -0.39575 -0.48162 0.23291 0.090201 -0.13324 0.078639 -0.41634 -0.15428 0.10068 0.48891 0.31226 -0.1252 -0.037512 -1.5179 0.12612 -0.02442 -0.042961 -0.28351 3.5416 -0.11956 -0.014533 -0.1499 0.21864 -0.33412 -0.13872 0.31806 0.70358 0.44858 -0.080262 0.63003 0.32111 -0.46765 0.22786 0.36034 -0.37818 -0.56657 0.044691 0.30392\n', '. 0.15164 0.30177 -0.16763 0.17684 0.31719 0.33973 -0.43478 -0.31086 -0.44999 -0.29486 0.16608 0.11963 -0.41328 -0.42353 0.59868 0.28825 -0.11547 -0.041848 -0.67989 -0.25063 0.18472 0.086876 0.46582 0.015035 0.043474 -1.4671 -0.30384 -0.023441 0.30589 -0.21785 3.746 0.0042284 -0.18436 -0.46209 0.098329 -0.11907 0.23919 0.1161 0.41705 0.056763 -6.3681e-05 0.068987 0.087939 -0.10285 -0.13931 0.22314 -0.080803 -0.35652 0.016413 0.10216\n', 'of 0.70853 0.57088 -0.4716 0.18048 0.54449 0.72603 0.18157 -0.52393 0.10381 -0.17566 0.078852 -0.36216 -0.11829 -0.83336 0.11917 -0.16605 0.061555 -0.012719 -0.56623 0.013616 0.22851 -0.14396 -0.067549 -0.38157 -0.23698 -1.7037 -0.86692 -0.26704 -0.2589 0.1767 3.8676 -0.1613 -0.13273 -0.68881 0.18444 0.0052464 -0.33874 -0.078956 0.24185 0.36576 -0.34727 0.28483 0.075693 -0.062178 -0.38988 0.22902 -0.21617 -0.22562 -0.093918 -0.80375\n', 'to 0.68047 -0.039263 0.30186 -0.17792 0.42962 0.032246 -0.41376 0.13228 -0.29847 -0.085253 0.17118 0.22419 -0.10046 -0.43653 0.33418 0.67846 0.057204 -0.34448 -0.42785 -0.43275 0.55963 0.10032 0.18677 -0.26854 0.037334 -2.0932 0.22171 -0.39868 0.20912 -0.55725 3.8826 0.47466 -0.95658 -0.37788 0.20869 -0.32752 0.12751 0.088359 0.16351 -0.21634 -0.094375 0.018324 0.21048 -0.03088 -0.19722 0.082279 -0.09434 -0.073297 -0.064699 -0.26044\n', 'and 0.26818 0.14346 -0.27877 0.016257 0.11384 0.69923 -0.51332 -0.47368 -0.33075 -0.13834 0.2702 0.30938 -0.45012 -0.4127 -0.09932 0.038085 0.029749 0.10076 -0.25058 -0.51818 0.34558 0.44922 0.48791 -0.080866 -0.10121 -1.3777 -0.10866 -0.23201 0.012839 -0.46508 3.8463 0.31362 0.13643 -0.52244 0.3302 0.33707 -0.35601 0.32431 0.12041 0.3512 -0.069043 0.36885 0.25168 -0.24517 0.25381 0.1367 -0.31178 -0.6321 -0.25028 -0.38097\n', 'in 0.33042 0.24995 -0.60874 0.10923 0.036372 0.151 -0.55083 -0.074239 -0.092307 -0.32821 0.09598 -0.82269 -0.36717 -0.67009 0.42909 0.016496 -0.23573 0.12864 -1.0953 0.43334 0.57067 -0.1036 0.20422 0.078308 -0.42795 -1.7984 -0.27865 0.11954 -0.12689 0.031744 3.8631 -0.17786 -0.082434 -0.62698 0.26497 -0.057185 -0.073521 0.46103 0.30862 0.12498 -0.48609 -0.0080272 0.031184 -0.36576 -0.42699 0.42164 -0.11666 -0.50703 -0.027273 -0.53285\n', 'a 0.21705 0.46515 -0.46757 0.10082 1.0135 0.74845 -0.53104 -0.26256 0.16812 0.13182 -0.24909 -0.44185 -0.21739 0.51004 0.13448 -0.43141 -0.03123 0.20674 -0.78138 -0.20148 -0.097401 0.16088 -0.61836 -0.18504 -0.12461 -2.2526 -0.22321 0.5043 0.32257 0.15313 3.9636 -0.71365 -0.67012 0.28388 0.21738 0.14433 0.25926 0.23434 0.4274 -0.44451 0.13813 0.36973 -0.64289 0.024142 -0.039315 -0.26037 0.12017 -0.043782 0.41013 0.1796\n', '" 0.25769 0.45629 -0.76974 -0.37679 0.59272 -0.063527 0.20545 -0.57385 -0.29009 -0.13662 0.32728 1.4719 -0.73681 -0.12036 0.71354 -0.46098 0.65248 0.48887 -0.51558 0.039951 -0.34307 -0.014087 0.86488 0.3546 0.7999 -1.4995 -1.8153 0.41128 0.23921 -0.43139 3.6623 -0.79834 -0.54538 0.16943 -0.82017 -0.3461 0.69495 -1.2256 -0.17992 -0.057474 0.030498 -0.39543 -0.38515 -1.0002 0.087599 -0.31009 -0.34677 -0.31438 0.75004 0.97065\n']
My search methods are evidently thusfar ineffective. Would really appreciate some help.
In can't reproduce the problem running the following code (on a linux machine, Python 3.6):
In [1]: from gensim.models import KeyedVectors
In [2]: from gensim.scripts.glove2word2vec import glove2word2vec
In [3]: glove2word2vec('glove.6B.50d.txt', 'glove.68.50d.w2v.txt')
Out[3]: (400000, 50)
In [4]: model = KeyedVectors.load_word2vec_format('glove.68.50d.w2v.txt')
In [5]: len(model)
Out[5]: 400000
In [6]: model['the']
Out[7]:
array([ 4.1800e-01, 2.4968e-01, -4.1242e-01, 1.2170e-01, 3.4527e-01,
-4.4457e-02, -4.9688e-01, -1.7862e-01, -6.6023e-04, -6.5660e-01,
2.7843e-01, -1.4767e-01, -5.5677e-01, 1.4658e-01, -9.5095e-03,
1.1658e-02, 1.0204e-01, -1.2792e-01, -8.4430e-01, -1.2181e-01,
-1.6801e-02, -3.3279e-01, -1.5520e-01, -2.3131e-01, -1.9181e-01,
-1.8823e+00, -7.6746e-01, 9.9051e-02, -4.2125e-01, -1.9526e-01,
4.0071e+00, -1.8594e-01, -5.2287e-01, -3.1681e-01, 5.9213e-04,
7.4449e-03, 1.7778e-01, -1.5897e-01, 1.2041e-02, -5.4223e-02,
-2.9871e-01, -1.5749e-01, -3.4758e-01, -4.5637e-02, -4.4251e-01,
1.8785e-01, 2.7849e-03, -1.8411e-01, -1.1514e-01, -7.8581e-01],
dtype=float32)
Do these exact lines trigger the exact same error as originally reported for you? (If you still get an error, but the error is even the slightest bit different, can you add the updated error to your question?)
My best guess if you're still having a problem is some Windows-specific default-encoding mangling during one of the steps, or if the file was opened/saved in some other editor.
Related
huggingface fine tuning distilbert-base-uncased and then using pipeline throws error KeyError: 'logits'
I am referring to the model. I am fine tuning that model. Before fine tuning I could use pipeline as below
from transformers import AutoTokenizer
model_check = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_check)
from transformers import pipeline
from transformers import AutoModelForSequenceClassification
modelz = (AutoModelForSequenceClassification.from_pretrained(model_check))
classifier = pipeline("text-classification", model=modelz,tokenizer=tokenizer)
custom_tweet = "I saw a movie today and it was really good."
preds = classifier(custom_tweet, return_all_scores=True)
preds
[[{'label': 'LABEL_0', 'score': 0.5338158011436462},
{'label': 'LABEL_1', 'score': 0.46618416905403137}]]
type(modelz)
transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification
then I am using from transformers import Trainer to train the model and saving it. But after that I havent been able to use pipeline :(
saved_model_path='/zyz'
trainer.save_model(saved_model_path)
model_saved=AutoModel.from_pretrained(saved_model_path)
tokenizer_saved=AutoTokenizer.from_pretrained(saved_model_path)
type(model_saved)
#transformers.models.distilbert.modeling_distilbert.DistilBertModel
from transformers import pipeline
classifier = pipeline("text-classification", model=model_saved,tokenizer=tokenizer_saved)
custom_tweet = "I saw a movie today and it was really good."
preds = classifier(custom_tweet)#, return_all_scores=True)
The model 'DistilBertModel' is not supported for text-classification. Supported models are ['FNetForSequenceClassification', 'GPTJForSequenceClassification', 'LayoutLMv2ForSequenceClassification', 'RemBertForSequenceClassification', 'CanineForSequenceClassification', 'RoFormerForSequenceClassification', 'BigBirdPegasusForSequenceClassification', 'BigBirdForSequenceClassification', 'ConvBertForSequenceClassification', 'LEDForSequenceClassification', 'DistilBertForSequenceClassification', 'AlbertForSequenceClassification', 'CamembertForSequenceClassification', 'XLMRobertaForSequenceClassification', 'MBartForSequenceClassification', 'BartForSequenceClassification', 'LongformerForSequenceClassification', 'RobertaForSequenceClassification', 'SqueezeBertForSequenceClassification', 'LayoutLMForSequenceClassification', 'BertForSequenceClassification', 'XLNetForSequenceClassification', 'MegatronBertForSequenceClassification', 'MobileBertForSequenceClassification', 'FlaubertForSequenceClassification', 'XLMForSequenceClassification', 'ElectraForSequenceClassification', 'FunnelForSequenceClassification', 'DebertaForSequenceClassification', 'DebertaV2ForSequenceClassification', 'GPT2ForSequenceClassification', 'GPTNeoForSequenceClassification', 'OpenAIGPTForSequenceClassification', 'ReformerForSequenceClassification', 'CTRLForSequenceClassification', 'TransfoXLForSequenceClassification', 'MPNetForSequenceClassification', 'TapasForSequenceClassification', 'IBertForSequenceClassification'].
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-92-cf10c8e85589> in <module>()
4
5 custom_tweet = "I saw a movie today and it was really good."
----> 6 preds = classifier(custom_tweet)#, return_all_scores=True)
5 frames
/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py in __getitem__(self, k)
1952 if isinstance(k, str):
1953 inner_dict = {k: v for (k, v) in self.items()}
-> 1954 return inner_dict[k]
1955 else:
1956 return self.to_tuple()[k]
KeyError: 'logits'
I am confused about 2 things -
why does it say The model 'DistilBertModel' is not supported for text-classification. Supported models are in the second part of the code when the first part runs without any problems. I tried type(modelz) and type(model_saved) and I see that the trainer is being saved differently. Why is that? How should I save the trianer so that it's class doesnt change?
Why does it throw error KeyError: 'logits'
update 1================================found answer
I fixed it by changing as below
model_saved=AutoModelForSequenceClassification.from_pretrained(saved_model_path)
print word vectors simply instead of obtaining them as array
A very simple task but I don't seem to do it. I want to obtain my vectors like this:
the -0.038194 -0.24487 0.72812 -0.39961 0.083172 0.043953 -0.39141 0.3344 -0.57545 0.087459 0.28787 -0.06731 0.30906 -0.26384 -0.13231 -0.20757 0.33395 -0.33848 -0.31743 -0.48336 0.1464 -0.37304 0.34577 0.052041 0.44946 -0.46971 0.02628 -0.54155 -0.15518 -0.14107 -0.039722 0.28277 0.14393 0.23464 -0.31021 0.086173 0.20397 0.52624 0.17164 -0.082378 -0.71787 -0.41531 0.20335 -0.12763 0.41367 0.55187 0.57908 -0.33477 -0.36559 -0.54857 -0.062892 0.26584 0.30205 0.99775 -0.80481 -3.0243 0.01254 -0.36942 2.2167 0.72201 -0.24978 0.92136 0.034514 0.46745 1.1079 -0.19358 -0.074575 0.23353 -0.052062 -0.22044 0.057162 -0.15806 -0.30798 -0.41625 0.37972 0.15006 -0.53212 -0.2055 -1.2526 0.071624 0.70565 0.49744 -0.42063 0.26148 -1.538 -0.30223 -0.073438 -0.28312 0.37104 -0.25217 0.016215 -0.017099 -0.38984 0.87424 -0.72569 -0.51058 -0.52028 -0.1459 0.8278 0.27062
, -0.10767 0.11053 0.59812 -0.54361 0.67396 0.10663 0.038867 0.35481 0.06351 -0.094189 0.15786 -0.81665 0.14172 0.21939 0.58505 -0.52158 0.22783 -0.16642 -0.68228 0.3587 0.42568 0.19021 0.91963 0.57555 0.46185 0.42363 -0.095399 -0.42749 -0.16567 -0.056842 -0.29595 0.26037 -0.26606 -0.070404 -0.27662 0.15821 0.69825 0.43081 0.27952 -0.45437 -0.33801 -0.58184 0.22364 -0.5778 -0.26862 -0.20425 0.56394 -0.58524 -0.14365 -0.64218 0.0054697 -0.35248 0.16162 1.1796 -0.47674 -2.7553 -0.1321 -0.047729 1.0655 1.1034 -0.2208 0.18669 0.13177 0.15117 0.7131 -0.35215 0.91348 0.61783 0.70992 0.23955 -0.14571 -0.37859 -0.045959 -0.47368 0.2385 0.20536 -0.18996 0.32507 -1.1112 -0.36341 0.98679 -0.084776 -0.54008 0.11726 -1.0194 -0.24424 0.12771 0.013884 0.080374 -0.35414 0.34951 -0.7226 0.37549 0.4441 -0.99059 0.61214 -0.35111 -0.83155 0.45293 0.082577
. -0.33979 0.20941 0.46348 -0.64792 -0.38377 0.038034 0.17127 0.15978 0.46619 -0.019169 0.41479 -0.34349 0.26872 0.04464 0.42131 -0.41032 0.15459 0.022239 -0.64653 0.25256 0.043136 -0.19445 0.46516 0.45651 0.68588 0.091295 0.21875 -0.70351 0.16785 -0.35079 -0.12634 0.66384 -0.2582 0.036542 -0.13605 0.40253 0.14289 0.38132 -0.12283 -0.45886 -0.25282 -0.30432 -0.11215 -0.26182 -0.22482 -0.44554 0.2991 -0.85612 -0.14503 -0.49086 0.0082973 -0.17491 0.27524 1.4401 -0.21239 -2.8435 -0.27958 -0.45722 1.6386 0.78808 -0.55262 0.65 0.086426 0.39012 1.0632 -0.35379 0.48328 0.346 0.84174 0.098707 -0.24213 -0.27053 0.045287 -0.40147 0.11395 0.0062226 0.036673 0.018518 -1.0213 -0.20806 0.64072 -0.068763 -0.58635 0.33476 -1.1432 -0.1148 -0.25091 -0.45907 -0.096819 -0.17946 -0.063351 -0.67412 -0.068895 0.53604 -0.87773 0.31802 -0.39242 -0.23394 0.47298 -0.028803
of -0.1529 -0.24279 0.89837 0.16996 0.53516 0.48784 -0.58826 -0.17982 -1.3581 0.42541 0.15377 0.24215 0.13474 0.41193 0.67043 -0.56418 0.42985 -0.012183 -0.11677 0.31781 0.054177 -0.054273 0.35516 -0.30241 0.31434 -0.33846 0.71715 -0.26855 -0.15837 -0.47467 0.051581 -0.33252 0.15003 -0.1299 -0.54617 -0.37843 0.64261 0.82187 -0.080006 0.078479 -0.96976 -0.57741 0.56491 -0.39873 -0.057099 0.19743 0.065706 -0.48092 -0.20125 -0.40834 0.39456 -0.02642 -0.11838 1.012 -0.53171 -2.7474 -0.042981 -0.74849 1.7574 0.59085 0.04885 0.78267 0.38497 0.42097 0.67882 0.10337 0.6328 -0.026595 0.58647 -0.44332 0.33057 -0.12022 -0.55645 0.073611 0.20915 0.43395 -0.012761 0.089874 -1.7991 0.084808 0.77112 0.63105 -0.90685 0.60326 -1.7515 0.18596 -0.50687 -0.70203 0.66578 -0.81304 0.18712 -0.018488 -0.26757 0.727 -0.59363 -0.34839 -0.56094 -0.591 1.0039 0.20664
to -0.1897 0.050024 0.19084 -0.049184 -0.089737 0.21006 -0.54952 0.098377 -0.20135 0.34241 -0.092677 0.161 -0.13268 -0.2816 0.18737 -0.42959 0.96039 0.13972 -1.0781 0.40518 0.50539 -0.55064 0.4844 0.38044 -0.0029055 -0.34942 -0.099696 -0.78368 1.0363 -0.2314 -0.47121 0.57126 -0.21454 0.35958 -0.48319 1.0875 0.28524 0.12447 -0.039248 -0.076732 -0.76343 -0.32409 -0.5749 -1.0893 -0.41811 0.4512 0.12112 -0.51367 -0.13349 -1.1378 -0.28768 0.16774 0.55804 1.5387 0.018859 -2.9721 -0.24216 -0.92495 2.1992 0.28234 -0.3478 0.51621 -0.43387 0.36852 0.74573 0.072102 0.27931 0.92569 -0.050336 -0.85856 -0.1358 -0.92551 -0.33991 -1.0394 -0.067203 -0.21379 -0.4769 0.21377 -0.84008 0.052536 0.59298 0.29604 -0.67644 0.13916 -1.5504 -0.20765 0.7222 0.52056 -0.076221 -0.15194 -0.13134 0.058617 -0.31869 -0.61419 -0.62393 -0.41548 -0.038175 -0.39804 0.47647 -0.15983
and -0.071953 0.23127 0.023731 -0.50638 0.33923 0.1959 -0.32943 0.18364 -0.18057 0.28963 0.20448 -0.5496 0.27399 0.58327 0.20468 -0.49228 0.19974 -0.070237 -0.88049 0.29485 0.14071 -0.1009 0.99449 0.36973 0.44554 0.28998 -0.1376 -0.56365 -0.029365 -0.4122 -0.25269 0.63181 -0.44767 0.24363 -0.10813 0.25164 0.46967 0.3755 -0.23613 -0.14129 -0.44537 -0.65737 -0.042421 -0.28636 -0.28811 0.063766 0.20281 -0.53542 0.41307 -0.59722 -0.38614 0.19389 -0.17809 1.6618 -0.011819 -2.3737 0.058427 -0.2698 1.2823 0.81925 -0.22322 0.72932 -0.053211 0.43507 0.85011 -0.42935 0.92664 0.39051 1.0585 -0.24561 -0.18265 -0.5328 0.059518 -0.66019 0.18991 0.28836 -0.2434 0.52784 -0.65762 -0.14081 1.0491 0.5134 -0.23816 0.69895 -1.4813 -0.2487 -0.17936 -0.059137 -0.08056 -0.48782 0.014487 -0.6259 -0.32367 0.41862 -1.0807 0.46742 -0.49931 -0.71895 0.86894 0.19539
in 0.085703 -0.22201 0.16569 0.13373 0.38239 0.35401 0.01287 0.22461 -0.43817 0.50164 -0.35874 -0.34983 0.055156 0.69648 -0.17958 0.067926 0.39101 0.16039 -0.26635 -0.21138 0.53698 0.49379 0.9366 0.66902 0.21793 -0.46642 0.22383 -0.36204 -0.17656 0.1748 -0.20367 0.13931 0.019832 -0.10413 -0.20244 0.55003 -0.1546 0.98655 -0.26863 -0.2909 -0.32866 -0.34188 -0.16943 -0.42001 -0.046727 -0.16327 0.70824 -0.74911 -0.091559 -0.96178 -0.19747 0.10282 0.55221 1.3816 -0.65636 -3.2502 -0.31556 -1.2055 1.7709 0.4026 -0.79827 1.1597 -0.33042 0.31382 0.77386 0.22595 0.52471 -0.034053 0.32048 0.079948 0.17752 -0.49426 -0.70045 -0.44569 0.17244 0.20278 0.023292 -0.20677 -1.0158 0.18325 0.56752 0.31821 -0.65011 0.68277 -0.86585 -0.059392 -0.29264 -0.55668 -0.34705 -0.32895 0.40215 -0.12746 -0.20228 0.87368 -0.545 0.79205 -0.20695 -0.074273 0.75808 -0.34243
a -0.27086 0.044006 -0.02026 -0.17395 0.6444 0.71213 0.3551 0.47138 -0.29637 0.54427 -0.72294 -0.0047612 0.040611 0.043236 0.29729 0.10725 0.40156 -0.53662 0.033382 0.067396 0.64556 -0.085523 0.14103 0.094539 0.74947 -0.194 -0.68739 -0.41741 -0.22807 0.12 -0.48999 0.80945 0.045138 -0.11898 0.20161 0.39276 -0.20121 0.31354 0.75304 0.25907 -0.11566 -0.029319 0.93499 -0.36067 0.5242 0.23706 0.52715 0.22869 -0.51958 -0.79349 -0.20368 -0.50187 0.18748 0.94282 -0.44834 -3.6792 0.044183 -0.26751 2.1997 0.241 -0.033425 0.69553 -0.64472 -0.0072277 0.89575 0.20015 0.46493 0.61933 -0.1066 0.08691 -0.4623 0.18262 -0.15849 0.020791 0.19373 0.063426 -0.31673 -0.48177 -1.3848 0.13669 0.96859 0.049965 -0.2738 -0.035686 -1.0577 -0.24467 0.90366 -0.12442 0.080776 -0.83401 0.57201 0.088945 -0.42532 -0.018253 -0.079995 -0.28581 -0.01089 -0.4923 0.63687 0.23642
" -0.30457 -0.23645 0.17576 -0.72854 -0.28343 -0.2564 0.26587 0.025309 -0.074775 -0.3766 -0.057774 0.12159 0.34384 0.41928 -0.23236 -0.31547 0.60939 0.25117 -0.68667 0.70873 1.2162 -0.1824 -0.48442 -0.33445 0.30343 1.086 0.49992 -0.20198 0.27959 0.68352 -0.33566 -0.12405 0.059656 0.33617 0.37501 0.56552 0.44867 0.11284 -0.16196 -0.94346 -0.67961 0.18581 0.060653 0.43776 0.13834 -0.48207 -0.56141 -0.25422 -0.52445 0.097003 -0.48925 0.19077 0.21481 1.4969 -0.86665 -3.2846 0.56854 0.41971 1.2294 0.78522 -0.29369 0.63803 -1.5926 -0.20437 1.5306 0.13548 0.50722 0.18742 0.48552 -0.28995 0.19573 0.0046515 0.092879 -0.42444 0.64987 0.52839 0.077908 0.8263 -1.2208 -0.34955 0.49855 -0.64155 -0.72308 0.26566 -1.3643 -0.46364 -0.52048 -1.0525 0.22895 -0.3456 -0.658 -0.16735 0.35158 0.74337 0.26074 0.061104 -0.39079 -0.84557 -0.035432 0.17036
's 0.58854 -0.2025 0.73479 -0.68338 -0.19675 -0.1802 -0.39177 0.34172 -0.60561 0.63816 -0.26695 0.36486 -0.40379 -0.1134 -0.58718 0.2838 0.8025 -0.35303 0.30083 0.078935 0.44416 -0.45906 0.79294 0.50365 0.32805 0.28027 -0.4933 -0.38482 -0.039284 -0.2483 -0.1988 1.1469 0.13228 0.91691 -0.36739 0.89425 0.5426 0.61738 -0.62205 -0.31132 -0.50933 0.23335 1.0826 -0.044637 -0.12767 0.27628 -0.032617 -0.27397 0.77764 -0.50861 0.038307 -0.33679 0.42344 1.2271 -0.53826 -3.2411 0.42626 0.025189 1.3948 0.65085 0.03325 0.37141 0.4044 0.35558 0.98265 -0.61724 0.53901 0.76219 0.30689 0.33065 0.30956 -0.15161 -0.11313 -0.81281 0.6145 -0.44341 -0.19163 -0.089551 -1.5927 0.37405 0.85857 0.54613 -0.31928 0.52598 -1.4802 -0.97931 -0.2939 -0.14724 0.25803 -0.1817 1.0149 0.77649 0.12598 0.54779 -1.0316 0.064599 -0.37523 -0.94475 0.61802 0.39591
for -0.14401 0.32554 0.14257 -0.099227 0.72536 0.19321 -0.24188 0.20223 -0.89599 0.15215 0.035963 -0.59513 -0.051635 -0.014428 0.35475 -0.31859 0.76984 -0.087369 -0.24762 0.65059 -0.15138 -0.42703 0.18813 0.091562 0.15192 0.11303 -0.15222 -0.62786 -0.23923 0.096009 -0.46147 0.41526 -0.30475 0.1371 0.16758 0.53301 -0.043658 0.85924 -0.41192 -0.21394 -0.51228 -0.31945 0.12662 -0.3151 0.0031429 0.27129 0.17328 -1.3159 -0.42414 -0.69126 0.019017 -0.13375 -0.096057 1.7069 -0.65291 -2.6111 0.26518 -0.61178 2.095 0.38148 -0.55823 0.2036 -0.33704 0.37354 0.6951 -0.001637 0.81885 0.51793 0.27746 -0.37177 -0.43345 -0.42732 -0.54912 -0.30715 0.18101 0.2709 -0.29266 0.30834 -1.4624 -0.18999 0.92277 -0.099217 -0.25165 0.49197 -1.525 0.15326 0.2827 0.12102 -0.36766 -0.61275 -0.18884 0.10907 0.12315 0.090066 -0.65447 -0.17252 2.6336e-05 0.25398 1.1078 -0.073074
Here's the full link for the text file so u can see the format completely: https://www.kaggle.com/terenceliu4444/glove6b100dtxt
And Here's my code:
with codecs.open('data/{}.tsv'.format(lcode), 'w', 'utf-8') as fout:
for i, word in enumerate(model.index2word):
fout.write(u"{}\t{}\t{}\n".format(str(i), word.encode('utf8').decode('utf8'),
np.array_str(model[word])
))
and my output is like this:
the [ 0.28177965 -1.3835016 -0.85463244 0.5744817 -0.42041674 0.4850773
-0.18238722 0.9088641 1.6516979 -0.24690722 0.5303408 -0.8106607
0.27385864 0.6186187 -2.061754 1.2491482 0.44255176 -0.25498274
-0.11942661 -0.1751283 0.2187617 -1.2942451 -0.79252934 1.8655167
-1.4975996 -0.02266688 0.26935738 -0.36034968 -1.5055205 0.0860498
1.0129709 1.1270534 -1.3774556 -0.02182451 -0.52671534 -0.7581365
-0.16326018 -0.2763609 0.5690212 -1.355627 0.43560007 2.4623177
-0.46482357 0.85816354 -0.5735287 -0.99033487 0.646639 -0.18928614
-0.6105273 -0.94887084 -0.39465773 0.38946334 -0.5338978 -0.0211645
-0.06462063 1.1689087 -0.88438195 -0.60245454 1.0320075 0.75902534
-1.9108475 -0.8921491 0.57644296 1.8618042 -0.5125161 -1.4219466
0.45342374 0.25558227 1.0577608 0.48511812 0.76758397 -1.0726306
1.5792096 0.01924564 1.8321682 -0.4707404 -0.41836467 0.07758982
0.50893927 -0.71105474 -0.33766833 1.4899743 0.60877067 -0.09521568
0.6654671 -0.0286361 -0.17863822 0.8811929 -1.330545 -1.104361
0.51000476 0.2639544 1.2233694 -0.10699744 -1.1367066 0.6225027
0.5847332 -0.03609625 2.3312287 0.1025821 ]
a [ 1.0829129 0.84877855 -1.1785074 0.13858096 2.008711 0.44480678
0.41152284 -0.9221507 1.5342509 0.8937895 -0.12867515 1.2286083
-1.6460084 0.96246207 0.11606621 -0.7079361 0.7204446 -2.17121
0.21708168 -1.029137 -0.53540015 0.40489924 -0.52271795 1.7237337
-1.3921518 -1.4322941 1.392808 0.7498414 -1.4813395 1.655896
1.0292306 -0.10302904 -0.09161732 0.9659639 0.13209064 -0.5149641
-0.11515223 -1.6309028 -1.1918032 0.34248984 0.6209429 1.0181456
-0.65688735 0.80660087 -0.6315423 -0.68773484 -0.44171524 -0.8294182
0.62340856 -1.0040073 0.40221572 -0.30175862 0.02053229 0.31205446
-0.16386059 0.18476132 0.18067418 -0.28932625 1.0893115 0.11680666
0.1104597 -0.30494598 -0.06541535 0.75524884 3.3038845 -0.5918715
1.0957772 -0.51271206 1.3486993 0.6190552 -1.365369 -2.9811475
1.3973937 1.4510086 0.45045042 0.61286205 1.7809817 -0.639005
-0.22986257 -1.4068168 0.34073976 0.38807136 -0.10908178 -0.9710727
-0.2207968 0.66323316 2.2619925 1.8806032 -0.06102083 0.86097974
-0.07785034 0.3742449 1.800688 -0.92509884 0.1773087 0.38380435
0.44551063 0.5976865 1.8766458 0.23904268]
I tried to remove array from my code and still couldn't print those word vectors in that format. Im using Genism by the way to obtain vectors.
That looks like roughly the format (minus one leading line) that's already written-out by the .save_word2vec_format() method on gensim word-vector classes. You should try using that, perhaps with the write_first_line parameter as False, or simply editing the file afterwards. For example: model.save_word2vec_format(your_filename, binary=False, write_first_line=False) (I'd also note: your example format, and this method, will only use single spaces between fields, **not* tabs, so your existing file-suffic of .tsv for 'tab-separated-values' would be misleading.)
how to fix - error: bad escape \u at position 0
Hello I'm trying to export a gmap html using ipywidgets in jupyter notebook but am encountering the following error: - error: bad escape \u at position 0.
I'm new to programing and could use help fixing whatever is causing this error to occur. If there is any easier way to go about exporting the html file I'm happy to change approaches.
Thanks
Here is a snippet of the code: I can add the entire thing if its helpful.
import pandas as pd
import gmaps
from ipywidgets.embed import embed_minimal_html
from ipywidgets import IntSlider
gmaps.configure(api_key='XXXX')
pd.options.mode.chained_assignment = None # default='warn'
file2 = '005 lat:long.csv'
state2 = pd.read_csv(file2)
state2 = state2.rename(columns={'Address1': 'address', 'City':'city',
'State':'state', 'Zip': 'zip'})
storenumbs = state2['Store'].str.split('#', expand=True)
state2 = state2.join(storenumbs)
state2 = state2.drop(['Store', 0], axis=1)
state2 = state2.rename(columns={1: 'store_#'})
state2['store_#'] = state2['store_#'].astype(int)
fig = gmaps.figure(center=(42.5, -71.4), map_type='TERRAIN', zoom_level=9.8)
scale = 4
one_layer = (gmaps.symbol_layer(low_points_lat_long, fill_color='red', stroke_color='red', scale= scale))
two_layer = (gmaps.symbol_layer(low_med_points_lat_long, fill_color='red', stroke_color='yellow', scale= scale))
three_layer = (gmaps.symbol_layer(med_high_points_lat_long, fill_color='yellow', stroke_color='green', scale= scale))
four_layer = (gmaps.symbol_layer(high_points_lat_long, fill_color='green', stroke_color='green', scale= scale))
fig.add_layer(one_layer)
fig.add_layer(two_layer)
fig.add_layer(three_layer)
fig.add_layer(four_layer)
fig
embed_minimal_html('export.html', views=[fig]
Long Form Error Bellow
)
KeyError Traceback (most recent call last)
~/miniconda3/lib/python3.7/sre_parse.py in parse_template(source, pattern)
1020 try:
-> 1021 this = chr(ESCAPES[this][1])
1022 except KeyError:
KeyError: '\\u'
During handling of the above exception, another exception occurred:
error Traceback (most recent call last)
<ipython-input-7-c096ac365396> in <module>
20
21 slider = IntSlider(value=40)
---> 22 embed_minimal_html('export.html', views=[slider], title='Widgets export')
~/miniconda3/lib/python3.7/site-packages/ipywidgets/embed.py in embed_minimal_html(fp, views, title, template, **kwargs)
300 {embed_kwargs}
301 """
--> 302 snippet = embed_snippet(views, **kwargs)
303
304 values = {
~/miniconda3/lib/python3.7/site-packages/ipywidgets/embed.py in embed_snippet(views, drop_defaults, state, indent, embed_url, requirejs, cors)
266 widget_views = u'\n'.join(
267 widget_view_template.format(view_spec=escape_script(json.dumps(view_spec)))
--> 268 for view_spec in data['view_specs']
269 )
270
~/miniconda3/lib/python3.7/site-packages/ipywidgets/embed.py in <genexpr>(.0)
266 widget_views = u'\n'.join(
267 widget_view_template.format(view_spec=escape_script(json.dumps(view_spec)))
--> 268 for view_spec in data['view_specs']
269 )
270
~/miniconda3/lib/python3.7/site-packages/ipywidgets/embed.py in escape_script(s)
239 involving `<` is readable.
240 """
--> 241 return script_escape_re.sub(r'\u003c\1', s)
242
243 #doc_subst(_doc_snippets)
~/miniconda3/lib/python3.7/re.py in _subx(pattern, template)
307 def _subx(pattern, template):
308 # internal: Pattern.sub/subn implementation helper
--> 309 template = _compile_repl(template, pattern)
310 if not template[0] and len(template[1]) == 1:
311 # literal replacement
~/miniconda3/lib/python3.7/re.py in _compile_repl(repl, pattern)
298 def _compile_repl(repl, pattern):
299 # internal: compile replacement pattern
--> 300 return sre_parse.parse_template(repl, pattern)
301
302 def _expand(pattern, match, template):
~/miniconda3/lib/python3.7/sre_parse.py in parse_template(source, pattern)
1022 except KeyError:
1023 if c in ASCIILETTERS:
-> 1024 raise s.error('bad escape %s' % this, len(this))
1025 lappend(this)
1026 else:
error: bad escape \u at position 0
This is an error in Python 3.7, and an issue with Python 3.6 (but it is OK with Python 2.7).
If you use a raw string (prefixed by "r") for the replacement in re.sub function, then the \u is escaped. For instance, r'\u003c\1' is like '\\u003c\\1': this is a string '\u', followed by '003c' and \1.
The solution is to write:
return script_escape_re.sub('\u003c\\1', s)
Quoting the documentation:
Changed in version 3.7: Unknown escapes in repl consisting of '\' and an ASCII letter now are errors.
I was facing a similar issue while trying to escape Unicode characters that have the pattern \uXXXX. Let's take a string containing Unicode characters:
>>> text = "The \u201c\u3010\u3011\u201d in this template are used to mark the variables"
>>> text
'The “【】” in this template are used to mark the variables'
Escape the Unicode characters:
>>> text = text.encode('unicode_escape').decode('ascii')
>>> text
'The \\u201c\\u3010\\u3011\\u201d in this template are used to mark the variables'
And then replace them using re.sub(r'\\u(.){4}', '', text):
>>> import re
>>> re.sub(r'\\u(.){4}', '', text)
'The in this template are used to mark the variables'
I have had the same issue during
[m.start() for m in re.finditer('Valuation Date")', 'dummytext')]
*** sre_constants.error: unbalanced parenthesis at position 15
But it was solved with re.escape help
[m.start() for m in re.finditer(re.escape('Valuation Date")'), 'dummytext')]
Enjoy.
Pandas Series.apply doesn't work consist of strings
It's seems possible to relate with Japanese Language problem,
So I asked in Japanese StackOverflow also.
When I use string just object, it works fine.
I tried to encode but I couldn't find the reason of this error.
Could you please give me advice?
MeCab is an open source text segmentation library for use with text written in the Japanese language originally developed by the Nara Institute of Science and Technology and currently maintained by Taku Kudou (工藤拓) as part of his work on the Google Japanese Input project.
https://en.wikipedia.org/wiki/MeCab
sample.csv
0,今日も夜まで働きました。
1,オフィスには誰もいませんが、エラーと格闘中
2,デバッグばかりしていますが、どうにもなりません。
This is Pandas Python3 code
import pandas as pd
import MeCab
# https://en.wikipedia.org/wiki/MeCab
from tqdm import tqdm_notebook as tqdm
# This is working...
df = pd.read_csv('sample.csv', encoding='utf-8')
m = MeCab.Tagger ("-Ochasen")
text = "りんごを食べました、そして、みかんも食べました"
a = m.parse(text)
print(a)# working!
# But I want to use Pandas's Series
def extractKeyword(text):
"""Morphological analysis of text and returning a list of only nouns"""
tagger = MeCab.Tagger('-Ochasen')
node = tagger.parseToNode(text)
keywords = []
while node:
if node.feature.split(",")[0] == u"名詞": # this means noun
keywords.append(node.surface)
node = node.next
return keywords
aa = extractKeyword(text) #working!!
me = df.apply(lambda x: extractKeyword(x))
#TypeError: ("in method 'Tagger_parseToNode', argument 2 of type 'char const *'", 'occurred at index 0')
This is the trace error
りんご リンゴ りんご 名詞-一般
を ヲ を 助詞-格助詞-一般
食べ タベ 食べる 動詞-自立 一段 連用形
まし マシ ます 助動詞 特殊・マス 連用形
た タ た 助動詞 特殊・タ 基本形
、 、 、 記号-読点
そして ソシテ そして 接続詞
、 、 、 記号-読点
みかん ミカン みかん 名詞-一般
も モ も 助詞-係助詞
食べ タベ 食べる 動詞-自立 一段 連用形
まし マシ ます 助動詞 特殊・マス 連用形
た タ た 助動詞 特殊・タ 基本形
EOS
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-174-81a0d5d62dc4> in <module>()
32 aa = extractKeyword(text) #working!!
33
---> 34 me = df.apply(lambda x: extractKeyword(x))
~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in apply(self, func, axis, broadcast, raw, reduce, args, **kwds)
4260 f, axis,
4261 reduce=reduce,
-> 4262 ignore_failures=ignore_failures)
4263 else:
4264 return self._apply_broadcast(f, axis)
~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in _apply_standard(self, func, axis, ignore_failures, reduce)
4356 try:
4357 for i, v in enumerate(series_gen):
-> 4358 results[i] = func(v)
4359 keys.append(v.name)
4360 except Exception as e:
<ipython-input-174-81a0d5d62dc4> in <lambda>(x)
32 aa = extractKeyword(text) #working!!
33
---> 34 me = df.apply(lambda x: extractKeyword(x))
<ipython-input-174-81a0d5d62dc4> in extractKeyword(text)
20 """Morphological analysis of text and returning a list of only nouns"""
21 tagger = MeCab.Tagger('-Ochasen')
---> 22 node = tagger.parseToNode(text)
23 keywords = []
24 while node:
~/anaconda3/lib/python3.6/site-packages/MeCab.py in parseToNode(self, *args)
280 __repr__ = _swig_repr
281 def parse(self, *args): return _MeCab.Tagger_parse(self, *args)
--> 282 def parseToNode(self, *args): return _MeCab.Tagger_parseToNode(self, *args)
283 def parseNBest(self, *args): return _MeCab.Tagger_parseNBest(self, *args)
284 def parseNBestInit(self, *args): return _MeCab.Tagger_parseNBestInit(self, *args)
TypeError: ("in method 'Tagger_parseToNode', argument 2 of type 'char const *'", 'occurred at index 0')w
I see you got some help on the Japanese StackOverflow, but here's an answer in English:
The first thing to fix is that read_csv was treating the first line of your example.csv as the header. To fix that, use the names argument in read_csv.
Next, df.apply will by default apply the function on columns of the dataframe. You need to do something like df.apply(lambda x: extractKeyword(x['String']), axis=1), but this won't work because each sentence will have a different number of nouns and Pandas will complain it cannot stack a 1x2 array on top of a 1x5 array. The simplest way is to apply on the Series of String.
The final problem is, there's a bug in the MeCab Python3 bindings: see https://github.com/SamuraiT/mecab-python3/issues/3 You found a workaround by running parseToNode twice, you can also call parse before parseToNode.
Putting all these three things together:
import pandas as pd
import MeCab
df = pd.read_csv('sample.csv', encoding='utf-8', names=['Number', 'String'])
def extractKeyword(text):
"""Morphological analysis of text and returning a list of only nouns"""
tagger = MeCab.Tagger('-Ochasen')
tagger.parse(text)
node = tagger.parseToNode(text)
keywords = []
while node:
if node.feature.split(",")[0] == u"名詞": # this means noun
keywords.append(node.surface)
node = node.next
return keywords
me = df['String'].apply(extractKeyword)
print(me)
When you run this script, with the example.csv you provide:
➜ python3 demo.py
0 [今日, 夜]
1 [オフィス, 誰, エラー, 格闘, 中]
2 [デバッグ]
Name: String, dtype: object
parseToNode fail everytime ,
so needed to put this code
tagger.parseToNode('dummy')
before
node = tagger.parseToNode(text)
and It's worked!
But I don't know the reason, maybe parseToNode method has bug..
def extractKeyword(text):
"""Morphological analysis of text and returning a list of only nouns"""
tagger = MeCab.Tagger('-Ochasen')
tagger.parseToNode('ダミー')
node = tagger.parseToNode(text)
keywords = []
while node:
if node.feature.split(",")[0] == u"名詞": # this means noun
keywords.append(node.surface)
node = node.next
return keywords
R DTW multivariate series with asymmetric step fails to compute alignment
I'm using the DTW implementation found in R along with the python bindings in order to verify the effects of changing different parameters(like local constraint, local distance function and others) for my data. The data represents feature vectors that an audio processing frontend outputs(MFCC). Because of this I am dealing with multivariate time series, each feature vector has a size of 8. The problem I'm facing is when I try to use certain local constraints ( or step patterns ) I get the following error:
Error in if (is.na(gcm$distance)) { : argument is of length zero
Traceback (most recent call last):
File "r_dtw_simplified.py", line 32, in <module>
alignment = R.dtw(canDist, rNull, "Euclidean", stepPattern, "none", True, Fa
lse, True, False )
File "D:\Python27\lib\site-packages\rpy2\robjects\functions.py", line 86, in _
_call__
return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
File "D:\Python27\lib\site-packages\rpy2\robjects\functions.py", line 35, in _
_call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in if (is.na(gcm$distance)) { : argument is
of length zero
Because the process of generating and adapting the input data is complicated I only made a simplified script to ilustrate the error i'm receiving.
#data works
#reference = [[-0.126678, -1.541763, 0.29985, 1.719757, 0.755798, -3.594681, -1.492798, 3.493042], [-0.110596, -1.638184, 0.128174, 1.638947, 0.721085, -3.247696, -0.920013, 3.763977], [-0.022415, -1.643539, -0.130692, 1.441742, 1.022064, -2.882172, -0.952225, 3.662842], [0.071259, -2.030411, -0.531891, 0.835114, 1.320419, -2.432281, -0.469116, 3.871094], [0.070526, -2.056702, -0.688293, 0.530396, 1.962128, -1.681915, -0.368973, 4.542419], [0.047745, -2.005127, -0.798203, 0.616028, 2.146988, -1.895874, 0.371597, 4.090881], [0.013962, -2.162796, -1.008545, 0.363495, 2.062866, -0.856613, 0.543884, 4.043335], [0.066757, -2.152969, -1.087097, 0.257263, 2.592697, -0.422424, -0.280533, 3.327576], [0.123123, -2.061035, -1.012863, 0.389282, 2.50206, 0.078186, -0.887711, 2.828247], [0.157455, -2.060425, -0.790344, 0.210419, 2.542114, 0.016983, -0.959274, 1.916504], [0.029648, -2.128204, -1.047318, 0.116547, 2.44899, 0.166534, -0.677551, 2.49231], [0.158554, -1.821365, -1.045044, 0.374207, 2.426712, 0.406952, -1.055084, 2.543762], [0.077026, -1.863235, -1.14827, 0.277069, 2.669067, 0.362549, -1.294342, 1.66748], [0.101822, -1.800293, -1.126801, 0.364594, 2.503815, 0.294846, -0.881302, 1.281616], [0.166138, -1.627762, -0.866013, 0.494476, 2.450668, 0.569, -1.392868, 0.651184], [0.225006, -1.596069, -1.07634, 0.550049, 2.167435, 0.554123, -1.432983, 1.166931], [0.114777, -1.462769, -0.793167, 0.565704, 2.183792, 0.345978, -1.410919, 0.708679], [0.144028, -1.444458, -0.831985, 0.536652, 2.222366, 0.330368, -0.715149, 0.517212], [0.147888, -1.450577, -0.809372, 0.479584, 2.271378, 0.250763, -0.540359, -0.036072], [0.090714, -1.485474, -0.888153, 0.268768, 2.001221, 0.412537, -0.698868, 0.17157], [0.11972, -1.382767, -0.890457, 0.218414, 1.666519, 0.659592, -0.069641, 0.914307], [0.189774, -1.18428, -0.785797, 0.106659, 1.429977, 0.195236, 0.627029, 0.503296], [0.194702, -1.098068, -0.956818, 0.020386, 1.369247, 0.10437, 0.641724, 0.410767], [0.215134, -1.069092, -1.11644, 0.283234, 1.313507, 0.110962, 0.600861, 0.752869], [0.216766, -1.065338, -1.047974, 0.080231, 1.500702, -0.113388, 0.712646, 0.914307], [0.259933, -0.964386, -0.981369, 0.092224, 1.480667, -0.00238, 0.896255, 0.665344], [0.265991, -0.935257, -0.93779, 0.214966, 1.235275, 0.104782, 1.33754, 0.599487], [0.266098, -0.62619, -0.905792, 0.131409, 0.402908, 0.103363, 1.352814, 1.554688], [0.273468, -0.354691, -0.709579, 0.228027, 0.315125, -0.15564, 0.942123, 1.024292], [0.246429, -0.272522, -0.609924, 0.318604, -0.007355, -0.165756, 1.07019, 1.087708], [0.248596, -0.232468, -0.524887, 0.53009, -0.476334, -0.184479, 1.088089, 0.667358], [0.074478, -0.200455, -0.058411, 0.662811, -0.111923, -0.686462, 1.205154, 1.271912], [0.063065, -0.080765, 0.065552, 0.79071, -0.569946, -0.899506, 0.875687, 0.095215], [0.117706, -0.270584, -0.021027, 0.723694, -0.200073, -0.365158, 0.892624, -0.152466], [0.00148, -0.075348, 0.017761, 0.757507, 0.719299, -0.355362, 0.749329, 0.315247], [0.035034, -0.110794, 0.038559, 0.949677, 0.478699, 0.005951, 0.097305, -0.388245], [-0.101944, -0.392487, 0.401886, 1.154938, 0.199127, 0.117371, -0.070007, -0.562439], [-0.083282, -0.388657, 0.449066, 1.505951, 0.46405, -0.566208, 0.216293, -0.528076], [-0.152054, -0.100113, 0.833054, 1.746857, 0.085861, -1.314102, 0.294632, -0.470947], [-0.166672, -0.183777, 0.988373, 1.925262, -0.202057, -0.961441, 0.15242, 0.594421], [-0.234573, -0.227707, 1.102112, 1.802002, -0.382492, -1.153336, 0.29335, 0.074036], [-0.336426, 0.042435, 1.255096, 1.804535, -0.610153, -0.810745, 1.308441, 0.599854], [-0.359344, 0.007248, 1.344543, 1.441559, -0.758286, -0.800079, 1.0233, 0.668213], [-0.321823, 0.027618, 1.1521, 1.509827, -0.708267, -0.668152, 1.05722, 0.710571], [-0.265335, 0.012344, 1.491501, 1.844971, -0.584137, -1.042419, -0.449188, 0.5354], [-0.302399, 0.049698, 1.440643, 1.674866, -0.626633, -1.158554, -0.906937, 0.405579], [-0.330276, 0.466675, 1.444153, 0.855499, -0.645447, -0.352158, 0.730423, 0.429932], [-0.354721, 0.540207, 1.570786, 0.626648, -0.897446, -0.007416, 0.174042, 0.100525], [-0.239609, 0.669983, 0.978851, 0.85321, -0.156784, 0.107986, 0.915054, 0.114197], [-0.189346, 0.930756, 0.824295, 0.516083, -0.339767, -0.206314, 0.744049, -0.36377]]
#query = [[0.387268, -1.21701, -0.432266, -1.394104, -0.458984, -1.469788, 0.12764, 2.310059], [0.418091, -1.389526, -0.150146, -0.759155, -0.578003, -2.123199, 0.276001, 3.022339], [0.264694, -1.526886, -0.238907, -0.511108, -0.90683, -2.699249, 0.692032, 2.849854], [0.246628, -1.675171, -0.533432, 0.070007, -0.392151, -1.739227, 0.534485, 2.744019], [0.099335, -1.983826, -0.985291, 0.428833, 0.26535, -1.285583, -0.234451, 2.4729], [0.055893, -2.108063, -0.401825, 0.860413, 0.724106, -1.959137, -1.360458, 2.350708], [-0.131592, -1.928314, -0.056213, 0.577698, 0.859146, -1.812286, -1.21669, 2.2052], [-0.162796, -2.149933, 0.467239, 0.524231, 0.74913, -1.829498, -0.741913, 1.616577], [-0.282745, -1.971008, 0.837616, 0.56427, 0.198288, -1.826935, -0.118027, 1.599731], [-0.497223, -1.578705, 1.277298, 0.682983, 0.055084, -2.032562, 0.64151, 1.719238], [-0.634232, -1.433258, 1.760513, 0.550415, -0.053787, -2.188568, 1.666687, 1.611938], [-0.607498, -1.302826, 1.960556, 1.331726, 0.417633, -2.271973, 2.095001, 0.9823], [-0.952957, -0.222076, 0.772064, 2.062256, -0.295258, -1.255371, 3.450974, -0.047607], [-1.210587, 1.00061, 0.036392, 1.952209, 0.470123, 0.231628, 2.670502, -0.608276], [-1.213287, 0.927002, -0.414825, 2.104065, 1.160126, 0.088898, 1.32959, -0.018311], [-1.081558, 1.007751, -0.337509, 1.7146, 0.653687, 0.297089, 1.916733, -0.772461], [-1.064804, 1.284302, -0.393585, 2.150635, 0.132294, 0.443298, 1.967575, 0.775513], [-0.972366, 1.039734, -0.588135, 1.413818, 0.423813, 0.781494, 1.977509, -0.556274], [-0.556381, 0.591309, -0.678314, 1.025635, 1.094284, 2.234711, 1.504013, -1.71875], [-0.063477, 0.626129, 0.360489, 0.149902, 0.92804, 0.936493, 1.203018, 0.264282], [0.162003, 0.577698, 0.956863, -0.477051, 1.081161, 0.817749, 0.660843, -0.428711], [-0.049515, 0.423615, 0.82489, 0.446228, 1.323853, 0.562775, -0.144196, 1.145386], [-0.146851, 0.171906, 0.304871, 0.320435, 1.378937, 0.673004, 0.188416, 0.208618], [0.33992, -2.072418, -0.447968, 0.526794, -0.175858, -1.400299, -0.452454, 1.396606], [0.226089, -2.183441, -0.301071, -0.475159, 0.834961, -2.191864, -1.092361, 2.434814], [0.279556, -2.073181, -0.517639, -0.766479, 0.974808, -2.070374, -2.003891, 2.706421], [0.237961, -1.9245, -0.708435, -0.582153, 1.285934, -1.75882, -2.146164, 2.369995], [0.149658, -1.703705, -0.539749, -0.215332, 1.369705, -1.484802, -1.506256, 1.04126], [0.078735, -1.719543, 0.157013, 0.382385, 1.100998, -0.223755, 0.021683, -0.545654], [0.106003, -1.404358, 0.372345, 1.881165, -0.292511, -0.263855, 1.579529, -1.426025], [0.047729, -1.198608, 0.600769, 1.901123, -1.106949, 0.128815, 1.293701, -1.364258], [0.110748, -0.894348, 0.712601, 1.728699, -1.250381, 0.674377, 0.812302, -1.428833], [0.085754, -0.662903, 0.794312, 1.102844, -1.234283, 1.084442, 0.986938, -1.10022], [0.140823, -0.300323, 0.673508, 0.669983, -0.551453, 1.213074, 1.449326, -1.567261], [0.03743, 0.550293, 0.400909, -0.174622, 0.355301, 1.325867, 0.875854, 0.126953], [-0.084885, 1.128906, 0.292099, -0.248779, 0.722961, 0.873871, -0.409515, 0.470581], [0.019684, 0.947754, 0.19931, -0.306274, 0.176849, 1.431702, 1.091507, 0.701416], [-0.094162, 0.895203, 0.687378, -0.229065, 0.549088, 1.376953, 0.892303, -0.642334], [-0.727692, 0.626495, 0.848877, 0.521362, 1.521912, -0.443481, 1.247238, 0.197388], [-0.82048, 0.117279, 0.975174, 1.487244, 1.085281, -0.567993, 0.776093, -0.381592], [-0.009827, -0.553009, -0.213135, 0.837341, 0.482712, -0.939423, 0.140884, 0.330566], [-0.018127, -1.362335, -0.199265, 1.260742, 0.005188, -1.445068, -1.159653, 1.220825], [0.186172, -1.727814, -0.246552, 1.544128, 0.285416, 0.081848, -1.634003, -0.47522], [0.193649, -1.144043, -0.334854, 1.220276, 1.241302, 1.554382, 0.57048, -1.334961], [0.344604, -1.647461, -0.720749, 0.993774, 0.585709, 0.953522, -0.493042, -1.845703], [0.37471, -1.989471, -0.518555, 0.555908, -0.025787, 0.148132, -1.463425, -0.844849], [0.34523, -1.821625, -0.809418, 0.59137, -0.577927, 0.037903, -2.067764, -0.519531], [0.413193, -1.503876, -0.752243, 0.280396, -0.236206, 0.429932, -1.684097, -0.724731], [0.331299, -1.349243, -0.890121, -0.178589, -0.285721, 0.809875, -2.012329, -0.157227], [0.278946, -1.090057, -0.670441, -0.477539, -0.267105, 0.446045, -1.95668, 0.501343], [0.127304, -0.977112, -0.660324, -1.011658, -0.547409, 0.349182, -1.357574, 1.045654], [0.217728, -0.793182, -0.496262, -1.259949, -0.128937, 0.38855, -1.513306, 1.863647], [0.240143, -0.891541, -0.619995, -1.478577, -0.361481, 0.258362, -1.630585, 1.841064], [0.241547, -0.758453, -0.515442, -1.370605, -0.428238, 0.23996, -1.469406, 1.307617], [0.289948, -0.714661, -0.533798, -1.574036, 0.017929, -0.368317, -1.290283, 0.851563], [0.304916, -0.783752, -0.459915, -1.523621, -0.107651, -0.027649, -1.089905, 0.969238], [0.27179, -0.795593, -0.352432, -1.597656, -0.001678, -0.06189, -1.072495, 0.637329], [0.301956, -0.823578, -0.152115, -1.637634, 0.2034, -0.214508, -1.315491, 0.773071], [0.282486, -0.853271, -0.162094, -1.561096, 0.15686, -0.289307, -1.076874, 0.673706], [0.299881, -0.97052, -0.051086, -1.431152, -0.074692, -0.32428, -1.385452, 0.684326], [0.220886, -1.072266, -0.269531, -1.038269, 0.140533, -0.711273, -1.7453, 1.090332], [0.177628, -1.229126, -0.274292, -0.943481, 0.483246, -1.214447, -2.026321, 0.719971], [0.176987, -1.137543, -0.007645, -0.794861, 0.965118, -1.084717, -2.37677, 0.598267], [0.135727, -1.36795, 0.09462, -0.776367, 0.946655, -1.157959, -2.794403, 0.226074], [0.067337, -1.648987, 0.535721, -0.665833, 1.506119, -1.348755, -3.092728, 0.281616], [-0.038101, -1.437347, 0.983917, -0.280762, 1.880722, -1.351318, -3.002258, -0.599609], [-0.152573, -1.146027, 0.717545, -0.60321, 2.126541, -0.59198, -2.282028, -1.048584], [-0.113525, -0.629669, 0.925323, 0.465393, 2.368698, -0.352661, -1.969391, -0.915161], [-0.140121, -0.311951, 0.884262, 0.809021, 1.557693, -0.552429, -1.776062, -0.925537], [-0.189423, -0.117767, 0.975174, 1.595032, 1.284485, -0.698639, -2.007202, -1.307251], [-0.048874, -0.176941, 0.820679, 1.306519, 0.584259, -0.913147, -0.658066, -0.630981], [-0.127594, 0.33313, 0.791336, 1.400696, 0.685577, -1.500275, -0.657959, -0.207642], [-0.044128, 0.653351, 0.615326, 0.476685, 1.099625, -0.902893, -0.154449, 0.325073], [-0.150223, 1.059845, 1.208405, -0.038635, 0.758667, 0.458038, -0.178909, -0.998657], [-0.099854, 1.127197, 0.789871, -0.013611, 0.452805, 0.736176, 0.948273, -0.236328], [-0.250275, 1.188568, 0.935989, 0.34314, 0.130463, 0.879913, 1.669037, 0.12793], [-0.122818, 1.441223, 0.670029, 0.389526, -0.15274, 1.293549, 1.22908, -1.132568]]
#this one doesn't
reference = [[-0.453598, -2.439209, 0.973587, 1.362091, -0.073654, -1.755112, 1.090057, 4.246765], [-0.448502, -2.621201, 0.723282, 1.257324, 0.26619, -1.375351, 1.328735, 4.46991], [-0.481247, -2.29718, 0.612854, 1.078033, 0.309708, -2.037506, 1.056305, 3.181702], [-0.42482, -2.306702, 0.436157, 1.529907, 0.50708, -1.930069, 0.653198, 3.561768], [-0.39032, -2.361343, 0.589294, 1.965607, 0.611801, -2.417084, 0.035675, 3.381104], [-0.233444, -2.281525, 0.703171, 2.17868, 0.519257, -2.474442, -0.502808, 3.569153], [-0.174652, -1.924591, 0.180267, 2.127075, 0.250626, -2.208527, -0.396591, 2.565552], [-0.121078, -1.53801, 0.234344, 2.221039, 0.845367, -1.516205, -0.174149, 1.298645], [-0.18631, -1.047806, 0.629654, 2.073303, 0.775024, -1.931076, 0.382706, 2.278442], [-0.160477, -0.78743, 0.694214, 1.917572, 0.834885, -1.574707, 0.780045, 2.370422], [-0.203659, -0.427246, 0.726486, 1.548767, 0.465698, -1.185379, 0.555206, 2.619629], [-0.208298, -0.393707, 0.771881, 1.646484, 0.612946, -0.996277, 0.658539, 2.499146], [-0.180679, -0.166656, 0.689209, 1.205994, 0.3918, -1.051483, 0.771072, 1.854553], [-0.1978, 0.082764, 0.723541, 1.019104, 0.165405, -0.127533, 1.0522, 0.552368], [-0.171127, 0.168533, 0.529541, 0.584839, 0.702011, -0.36525, 0.711792, 1.029114], [-0.224243, 0.38765, 0.916031, 0.45108, 0.708923, -0.059326, 1.016312, 0.437561], [-0.217072, -0.981766, 1.67363, 1.864014, 0.050812, -2.572815, -0.22937, 0.757996], [-0.284714, -0.784927, 1.720383, 1.782379, -0.093414, -2.492111, 0.623398, 0.629028], [-0.261169, -0.427979, 1.680038, 1.585358, 0.067093, -1.8181, 1.276291, 0.838989], [-0.183075, -0.08197, 1.094147, 1.120392, -0.117752, -0.86142, 1.94194, 0.966858], [-0.188919, 0.121521, 1.277664, 0.90979, 0.114288, -0.880875, 1.920517, 0.95752], [-0.226868, 0.338455, 0.78067, 0.803009, 0.347092, -0.387955, 0.641296, 0.374634], [-0.206329, 0.768158, 0.759537, 0.264099, 0.15979, 0.152618, 0.911636, -0.011597], [-0.230453, 0.495941, 0.547165, 0.137604, 0.36377, 0.594406, 1.168839, 0.125916], [0.340851, -0.382736, -1.060455, -0.267792, 1.1306, 0.595047, -1.544922, -1.6828], [0.341492, -0.325836, -1.07164, -0.215607, 0.895645, 0.400177, -0.773956, -1.827515], [0.392075, -0.305389, -0.885422, -0.293427, 0.993225, 0.66655, -1.061218, -1.730713], [0.30191, -0.339005, -0.877853, 0.153992, 0.986588, 0.711823, -1.100525, -1.648376], [0.303574, -0.491241, -1.000183, 0.075378, 0.686295, 0.752792, -1.192123, -1.744568], [0.315781, -0.629456, -0.996063, 0.224731, 1.074173, 0.757736, -1.170807, -2.08313], [0.313675, -0.804688, -1.00325, 0.431641, 0.685883, 0.538879, -0.988373, -2.421326], [0.267181, -0.790329, -0.726974, 0.853027, 1.369629, -0.213638, -1.708023, -1.977844], [0.304459, -0.935257, -0.778061, 1.042633, 1.391861, -0.296768, -1.562164, -2.014099], [0.169754, -0.792953, -0.481842, 1.404236, 0.766983, -0.29805, -1.587265, -1.25531], [0.15918, -0.9814, -0.197662, 1.748718, 0.888367, -0.880234, -1.64949, -1.359802], [0.028244, -0.772934, -0.186172, 1.594238, 0.863571, -1.224701, -1.153183, -0.292664], [-0.020401, -0.461578, 0.368088, 1.000366, 1.079636, -0.389603, -0.144409, 0.651733], [0.018555, -0.725418, 0.632599, 1.707336, 0.535049, -1.783859, -0.916122, 1.557007], [-0.038971, -0.797668, 0.820419, 1.483093, 0.350494, -1.465073, -0.786453, 1.370361], [-0.244888, -0.469513, 1.067978, 1.028809, 0.4879, -1.796585, -0.77887, 1.888977], [-0.260193, -0.226593, 1.141754, 1.21228, 0.214005, -1.200943, -0.441177, 0.532715], [-0.165283, 0.016129, 1.263016, 0.745514, -0.211288, -0.802368, 0.215698, 0.316406], [-0.353134, 0.053787, 1.544189, 0.21106, -0.469086, -0.485367, 0.767761, 0.849548], [-0.330215, 0.162704, 1.570053, 0.304718, -0.561172, -0.410294, 0.895126, 0.858093], [-0.333847, 0.173904, 1.56958, 0.075531, -0.5569, -0.259552, 1.276764, 0.749084], [-0.347107, 0.206665, 1.389832, 0.50473, -0.721664, -0.56955, 1.542618, 0.817444], [-0.299057, 0.140244, 1.402924, 0.215363, -0.62767, -0.550461, 1.60788, 0.506958], [-0.292084, 0.052063, 1.463348, 0.290497, -0.462875, -0.497452, 1.280609, 0.261841], [-0.279877, 0.183548, 1.308609, 0.305756, -0.6483, -0.374771, 1.647781, 0.161865], [-0.28389, 0.27916, 1.148636, 0.466736, -0.724442, -0.21991, 1.819901, -0.218872], [-0.275528, 0.309753, 1.192856, 0.398163, -0.828781, -0.268066, 1.763672, 0.116089], [-0.275284, 0.160019, 1.200623, 0.718628, -0.925552, -0.026596, 1.367447, 0.174866], [-0.302795, 0.383438, 1.10556, 0.441833, -0.968323, -0.137375, 1.851791, 0.357971], [-0.317078, 0.22876, 1.272217, 0.462219, -0.855789, -0.294296, 1.593994, 0.127502], [-0.304932, 0.207718, 1.156189, 0.481506, -0.866776, -0.340027, 1.670105, 0.657837], [-0.257217, 0.155655, 1.041428, 0.717926, -0.761597, -0.17244, 1.114151, 0.653503], [-0.321426, 0.292358, 0.73848, 0.422607, -0.850754, -0.057907, 1.462357, 0.697754], [-0.34642, 0.361526, 0.69722, 0.585175, -0.464508, -0.26651, 1.860596, 0.106201], [-0.339844, 0.584229, 0.542603, 0.184937, -0.341263, 0.085648, 1.837311, 0.160461], [-0.32338, 0.661224, 0.512833, 0.319702, -0.195572, 0.004028, 1.046799, 0.233704], [-0.346329, 0.572388, 0.385986, 0.118988, 0.057556, 0.039001, 1.255081, -0.18573], [-0.383392, 0.558395, 0.553391, -0.358612, 0.443573, -0.086014, 0.652878, 0.829956], [-0.420395, 0.668991, 0.64856, -0.021271, 0.511475, 0.639221, 0.860474, 0.463196], [-0.359039, 0.748672, 0.522964, -0.308899, 0.717194, 0.218811, 0.681396, 0.606812], [-0.323914, 0.942627, 0.249069, -0.418365, 0.673599, 0.797974, 0.162674, 0.120361], [-0.411301, 0.92775, 0.493332, -0.286346, 0.165054, 0.63446, 1.085571, 0.120789], [-0.346191, 0.632309, 0.635056, -0.402496, 0.143814, 0.785614, 0.952164, 0.482727], [-0.203812, 0.789261, 0.240433, -0.47699, -0.12912, 0.91832, 1.145493, 0.052002], [-0.048203, 0.632095, 0.009583, -0.53833, 0.232727, 1.293045, 0.308151, 0.188904], [-0.062393, 0.732315, 0.06694, -0.697144, 0.126221, 0.864578, 0.581635, -0.088379]]
query = [[-0.113144, -3.316223, -1.101563, -2.128418, 1.853867, 3.61972, 1.218185, 1.71228], [-0.128952, -3.37915, -1.152237, -2.033081, 1.860199, 4.008179, 0.445938, 1.665894], [-0.0392, -2.976654, -0.888245, -1.613953, 1.638641, 3.849518, 0.034073, 0.768188], [-0.146042, -2.980713, -1.044113, -1.44397, 0.954514, 3.20929, -0.232422, 1.050781], [-0.155029, -2.997192, -1.064438, -1.369873, 0.67688, 2.570709, -0.855347, 1.523438], [-0.102341, -2.686401, -1.029648, -1.00531, 0.950089, 1.933228, -0.526367, 1.598633], [-0.060272, -2.538727, -1.278259, -0.65332, 0.630875, 1.459717, -0.264038, 1.872925], [0.064087, -2.592682, -1.112823, -0.775024, 0.848618, 0.810883, 0.298965, 2.312134], [0.111557, -2.815277, -1.203506, -1.173584, 0.54863, 0.46756, -0.023071, 3.029053], [0.266068, -2.624786, -1.089066, -0.864136, 0.055389, 0.619446, -0.160965, 2.928589], [0.181488, -2.31073, -1.307785, -0.720276, 0.001297, 0.534668, 0.495499, 2.989502], [0.216202, -2.25354, -1.288193, -0.902039, -0.152283, -0.060791, 0.566315, 2.911621], [0.430084, -2.0289, -1.099594, -1.091736, -0.302505, -0.087799, 0.955963, 2.677002], [0.484253, -1.412842, -0.881882, -1.087158, -1.064072, -0.145935, 1.437683, 2.606567], [0.339081, -1.277222, -1.24498, -1.048279, -0.219498, 0.448517, 1.168625, 0.563843], [0.105728, 0.138275, -1.01413, -0.489868, 1.319275, 1.604645, 1.634003, -0.94812], [-0.209061, 1.025665, 0.180405, 0.955566, 1.527405, 0.91745, 1.951233, -0.40686], [-0.136993, 1.332275, 0.639862, 1.277832, 1.277313, 0.361267, 0.390717, -0.728394], [-0.217758, 1.416718, 1.080002, 0.816101, 0.343933, -0.154175, 1.10347, -0.568848]]
reference = np.array( reference )
query = np.array( query )
rpy2.robjects.numpy2ri.activate()
# Set up our R namespaces
R = rpy2.robjects.r
rNull = R("NULL")
rprint = rpy2.robjects.globalenv.get("print")
rplot = rpy2.robjects.r('plot')
distConstr = rpy2.robjects.r('proxy::dist')
DTW = importr('dtw')
stepName = "asymmetricP05"
stepPattern = rpy2.robjects.r( stepName )
canDist = distConstr( reference, query, "Euclidean" ) #
alignment = R.dtw(canDist, rNull, "Euclidean", stepPattern, "none", True, False, True, False )
For some series the script doesn't generate the error but there are some which do. See the commented lines for examples. It is worth noting that for the classic constraint this error does not appear. I am thinking that perhaps I have not set-up something correct but I am no expert in python nor in R so that is why I was hoping that others who have used the R DTW can help me on this. I am sorry for the long lines for reference and query (the data is from outputting the MFCC's of a 2 second wav file).
One of the two series is too short to be compatible with the fancy step pattern you chose. Use the common symmetric2 pattern, which does not restrict slopes, before the more exotic ones.