Related
This question already has answers here:
Saving dictionary of numpy arrays
(3 answers)
Closed 1 year ago.
I need to save a python dictionary of the format {key : [numpy array]} to a file (doesnt matter if the file is human readable or not, I just need to be able to retrieve it back into the same format later). I'd prefer to keep the numpy array as a numpy array as it is very large. Also the numpy array is 300dimensional so iterating through them would be impractical. I haven't seen any other questions asking this, because it doesn't look like I can use the numpy save methods as I am saving a dictionary. JSON does not work as the dictionary contains a numpy array. Does anyone know how I can do this?
Thanks :)
The pickle module can handle numpy arrays. It is used almost exactly like the json module.
>>> import pickle
>>> import numpy as np
>>> a = np.arange(200).reshape((20,10))
>>> pickle.dump( a, open('xxx.bin','wb') )
... exit and reload ...
>>> import pickle
>>> pickle.load(open('xxx.bin','rb'))
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[ 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[ 30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[ 40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
[ 50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
[ 60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
[ 70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
[ 80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
[ 90, 91, 92, 93, 94, 95, 96, 97, 98, 99],
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109],
[110, 111, 112, 113, 114, 115, 116, 117, 118, 119],
[120, 121, 122, 123, 124, 125, 126, 127, 128, 129],
[130, 131, 132, 133, 134, 135, 136, 137, 138, 139],
[140, 141, 142, 143, 144, 145, 146, 147, 148, 149],
[150, 151, 152, 153, 154, 155, 156, 157, 158, 159],
[160, 161, 162, 163, 164, 165, 166, 167, 168, 169],
[170, 171, 172, 173, 174, 175, 176, 177, 178, 179],
[180, 181, 182, 183, 184, 185, 186, 187, 188, 189],
[190, 191, 192, 193, 194, 195, 196, 197, 198, 199]])
>>>
I have a the following tensor lets call it lookup_table:
tensor([266, 103, 84, 12, 32, 34, 1, 523, 22, 136, 268, 432, 53, 63,
201, 51, 164, 69, 31, 42, 122, 131, 119, 36, 245, 60, 28, 81,
9, 114, 105, 3, 41, 86, 150, 79, 104, 120, 74, 420, 39, 427,
40, 59, 24, 126, 202, 222, 145, 429, 43, 30, 38, 55, 10, 141,
85, 121, 203, 240, 96, 7, 64, 89, 127, 236, 117, 99, 54, 90,
57, 11, 21, 62, 82, 25, 267, 75, 111, 518, 76, 56, 20, 2,
61, 516, 80, 78, 555, 246, 133, 497, 33, 421, 58, 107, 92, 68,
13, 113, 235, 875, 35, 98, 102, 27, 14, 15, 72, 37, 16, 50,
517, 134, 223, 163, 91, 44, 17, 412, 18, 48, 23, 4, 29, 77,
6, 110, 67, 45, 161, 254, 112, 8, 106, 19, 498, 101, 5, 157,
83, 350, 154, 238, 115, 26, 142, 143])
And I have another tensor lets call it data, which looks like this:
tensor([[517, 235, 236, 76, 81, 25, 110, 59, 245, 39],
[523, 114, 350, 246, 30, 222, 39, 517, 106, 2],
[ 35, 235, 120, 99, 266, 63, 236, 133, 412, 38],
[134, 2, 497, 21, 78, 60, 142, 498, 24, 89],
[ 60, 111, 120, 145, 91, 141, 164, 81, 350, 55]])
Now I want something which looks similar to this:
tensor([112, 100, ..., 40],
[7, 29, ..., 2],
..., ])
I want to use my data tensor to get the index of the lookup table.
Basically I want to vectorize this:
(lookup_table == data).nonzero()
So that this works for multidimensional arrays.
I have read this, but they are not working for my case:
How Pytorch Tensor get the index of specific value
How Pytorch Tensor get the index of elements?
Pytorch tensor - How to get the indexes by a specific tensor
EDIT:
I am basically searching for an optimized/vectorized version of this:
x_data = torch.stack([(lookuptable == data[0][i]).nonzero(as_tuple=False) for i in range(len(data[0]))]).flatten().unsqueeze(0)
print(x_data.size())
for o in range(1, len(data)):
x_data = torch.cat((x_data, torch.stack([(lookuptable == data[o][i]).nonzero(as_tuple=False) for i in range(len(data[o]))]).flatten().unsqueeze(0)), dim=0)
EDIT 2 Minimal example:
We have the data tensor:
data = torch.Tensor([
[523, 114, 350, 246, 30, 222, 39, 517, 106, 2],
[ 35, 235, 120, 99, 266, 63, 236, 133, 412, 38],
[555, 104, 14, 81, 55, 497, 222, 64, 57, 131]
])
And we have the lookup_table tensor, see above.
If we apply this code to the 2 tensors:
# convert champion keys into index notation
x_data = torch.stack([(lookuptable == x[0][i]).nonzero(as_tuple=False) for i in range(len(x[0]))]).flatten().unsqueeze(0)
for o in range(1, len(data) - 1):
x_data = torch.cat((x_data, torch.stack([(lookuptable == x[o][i]).nonzero(as_tuple=False) for i in range(len(x[o]))]).flatten().unsqueeze(0)), dim=0)
We get an output of this:
tensor([[ 7, 29, 141, 89, 51, 47, 40, 112, 134, 83],
[102, 100, 37, 67, 0, 13, 65, 90, 119, 52],
[ 88, 36, 106, 27, 53, 91, 47, 62, 70, 21]
])
This output is what I want, and like I said above its the index of where each value of the tensor data lies on the tensor lookuptable.
The problem is that this is not vectorized.
And I have no Idea how to vectorize it.
Using searchsorted:
Scanning the whole lookup_table array for each input element is quite inefficient. How about sorting the lookup table first (this only needs to be done once)
sorted_lookup_table, indexes = torch.sort(lookup_table)
and then using searchsorted
index_into_sorted = torch.searchsorted(sorted_lookup_table, data)
If you need an index into the original lookup_table, you can get it with
index_into_lookup_table = indexes[index_into_sorted]
Another, faster, approach that assumes that all values have a limited range, and are int64 (Here, I also assume that they are non-negative, but this limitation can be worked around):
Prep work:
sorted_lookup_table, indexes = torch.sort(lookup_table)
lut = torch.zeros(size=(sorted_lookup_table[-1]+1,), dtype=torch.int64)
lut[:] = -1 # "not found"
lut[sorted_lookup_table] = indexes
Data processing:
index_into_lookup_table = lut[data]
I have a dataset as below:
data = [[92, 155],
[56, 186, 117, 210, 224],
[247, 202, 189, 210, 65, 3, 270, 224],
[20, 14, 157, 224],
[17, 89, 158, 224],
[263, 283, 68, 224],
[182, 166, 224],
[176, 37, 100, 224],
[33, 102, 41, 269, 177, 224],
[0, 260, 49, 207, 278, 217, 35],
[119],
[118],
[142, 185, 7, 246, 224],
[104, 22, 101, 224],
[84, 205, 224],
[225, 93, 54, 224],
[98, 32, 78, 224],
[159, 217, 212, 198, 224],
[178, 94, 187, 224],
[211, 149, 193, 149, 66, 139, 67, 28, 106, 224],
[133, 151],
[259, 109, 29, 224],
[215, 241, 73, 255, 77, 144, 224],
[36, 254, 19, 268, 183, 224],
[47, 234, 203, 111, 231, 141, 30],
[127, 275, 220, 161],
[214, 267, 22, 90, 224],
[46, 217, 103],
[17, 89, 128, 224],
[225, 22, 101, 224],
[285, 265, 151],
[215, 206, 264, 43, 224],
[244, 21, 224],
[82, 122, 240, 5, 224],
[259, 136, 162, 194, 224],
[176, 208, 112, 224],
[172, 19, 146, 276, 31, 246, 51, 224],
[45, 10],
[229, 24, 224],
[143, 108, 239, 224],
[225, 282, 83, 224],
[110, 267, 171],
[176, 245, 95, 123, 270, 224],
[248, 195, 139, 261, 173, 281, 232, 80, 18, 224],
[61, 60, 233],
[211, 120, 1, 23],
[225, 267, 249, 224],
[247, 202, 86, 196, 224],
[15, 127, 222, 224],
[247, 202, 186, 226, 145, 224],
[174, 242, 196, 224],
[259, 152, 71, 224],
[235, 44, 230, 224],
[69, 96, 50, 99, 116],
[259, 279, 224],
[228, 70],
[39, 139, 201, 190, 224],
[132, 40, 219, 81, 224],
[159, 221, 224],
[267, 16, 6, 62],
[143, 59, 175, 129, 48, 224],
[280, 140, 224],
[284, 124, 167, 150, 274],
[113, 265, 184],
[179, 4, 257, 145, 224],
[247, 202, 72, 11, 224],
[64],
[192, 125, 105],
[174, 134, 224],
[58, 139, 85, 160, 209, 224],
[130, 169, 137, 256, 224],
[215, 163, 265, 185, 26],
[176, 147, 74, 224],
[0, 266],
[143, 34, 153, 188, 224],
[121],
[243, 75, 135],
[38, 218, 199, 253, 224],
[178, 271, 224],
[154, 164, 180, 27, 270, 224],
[176, 189, 148, 139, 277, 224],
[57, 62],
[91, 168, 251, 224],
[172, 19, 146, 276, 53, 97, 200, 224],
[64],
[8, 237, 224],
[138, 107, 224],
[176, 238, 224],
[204, 217, 63, 165, 224],
[215, 216, 272, 62, 170, 2, 55, 224],
[247, 273, 202, 223, 9, 148, 224],
[258, 267, 181, 224],
[262, 76, 126, 12, 224],
[36, 254, 19, 268, 250, 213, 48, 224],
[227, 42],
[79, 197, 52, 87, 224],
[143, 131, 224],
[156, 88, 115, 236, 224],
[259, 13, 252, 224],
[114, 25, 191, 224]]
target = [9,
31,
20,
9,
3,
26,
16,
11,
28,
0,
9,
9,
9,
9,
7,
1,
33,
9,
13,
15,
9,
21,
9,
34,
9,
9,
9,
9,
3,
1,
9,
27,
14,
22,
21,
11,
17,
9,
6,
8,
1,
9,
11,
9,
9,
9,
1,
20,
29,
20,
23,
21,
9,
9,
21,
9,
18,
9,
9,
30,
8,
9,
9,
9,
9,
20,
9,
32,
23,
9,
24,
9,
11,
9,
8,
9,
9,
9,
13,
10,
11,
9,
12,
17,
9,
5,
9,
11,
9,
2,
20,
9,
25,
34,
9,
9,
8,
4,
21,
19]
I wish to do classification by using naive bayes or any other BEST algorithms available. However, I received error when using naive bayes as below:
from sklearn.naive_bayes import MultinomialNB
mnb = MultinomialNB(class_prior=[.25,.75])
mnb.fit(data, target)
Errros:
ValueError: Expected 2D array, got 1D array instead:
array=[list([92, 155]) list([56, 186, 117, 210, 224])
list([247, 202, 189, 210, 65, 3, 270, 224]) list([20, 14, 157, 224])
list([17, 89, 158, 224]) list([263, 283, 68, 224]) list([182, 166, 224])
list([176, 37, 100, 224]) list([33, 102, 41, 269, 177, 224])
list([0, 260, 49, 207, 278, 217, 35]) list([119]) list([118])
list([142, 185, 7, 246, 224]) list([104, 22, 101, 224])
list([84, 205, 224]) list([225, 93, 54, 224]) list([98, 32, 78, 224])
list([159, 217, 212, 198, 224]) list([178, 94, 187, 224])
list([211, 149, 193, 149, 66, 139, 67, 28, 106, 224]) list([133, 151])
list([259, 109, 29, 224]) list([215, 241, 73, 255, 77, 144, 224])
list([36, 254, 19, 268, 183, 224])
list([47, 234, 203, 111, 231, 141, 30]) list([127, 275, 220, 161])
list([214, 267, 22, 90, 224]) list([46, 217, 103])
list([17, 89, 128, 224]) list([225, 22, 101, 224]) list([285, 265, 151])
list([215, 206, 264, 43, 224]) list([244, 21, 224])
list([82, 122, 240, 5, 224]) list([259, 136, 162, 194, 224])
list([176, 208, 112, 224]) list([172, 19, 146, 276, 31, 246, 51, 224])
list([45, 10]) list([229, 24, 224]) list([143, 108, 239, 224])
list([225, 282, 83, 224]) list([110, 267, 171])
list([176, 245, 95, 123, 270, 224])
list([248, 195, 139, 261, 173, 281, 232, 80, 18, 224])
list([61, 60, 233]) list([211, 120, 1, 23]) list([225, 267, 249, 224])
list([247, 202, 86, 196, 224]) list([15, 127, 222, 224])
list([247, 202, 186, 226, 145, 224]) list([174, 242, 196, 224])
list([259, 152, 71, 224]) list([235, 44, 230, 224])
list([69, 96, 50, 99, 116]) list([259, 279, 224]) list([228, 70])
list([39, 139, 201, 190, 224]) list([132, 40, 219, 81, 224])
list([159, 221, 224]) list([267, 16, 6, 62])
list([143, 59, 175, 129, 48, 224]) list([280, 140, 224])
list([284, 124, 167, 150, 274]) list([113, 265, 184])
list([179, 4, 257, 145, 224]) list([247, 202, 72, 11, 224]) list([64])
list([192, 125, 105]) list([174, 134, 224])
list([58, 139, 85, 160, 209, 224]) list([130, 169, 137, 256, 224])
list([215, 163, 265, 185, 26]) list([176, 147, 74, 224]) list([0, 266])
list([143, 34, 153, 188, 224]) list([121]) list([243, 75, 135])
list([38, 218, 199, 253, 224]) list([178, 271, 224])
list([154, 164, 180, 27, 270, 224]) list([176, 189, 148, 139, 277, 224])
list([57, 62]) list([91, 168, 251, 224])
list([172, 19, 146, 276, 53, 97, 200, 224]) list([64])
list([8, 237, 224]) list([138, 107, 224]) list([176, 238, 224])
list([204, 217, 63, 165, 224]) list([215, 216, 272, 62, 170, 2, 55, 224])
list([247, 273, 202, 223, 9, 148, 224]) list([258, 267, 181, 224])
list([262, 76, 126, 12, 224]) list([36, 254, 19, 268, 250, 213, 48, 224])
list([227, 42]) list([79, 197, 52, 87, 224]) list([143, 131, 224])
list([156, 88, 115, 236, 224]) list([259, 13, 252, 224])
list([114, 25, 191, 224])].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Please can anyone please help me with this? Or can someone show me some other best examples of algorithms to use for machine learning like decision tree, svm or anything.
So the direct answer to why you're getting an error is that you're passing an array of lists as an argument. Sklearn thus think you're passing in a 1D array of lists. It is not possible to transform your data to a 2D matrix because the number of values in your list in inconsistent.
From my understanding (which could be wrong), each row of your input feature matrix need to have the same amount of numbers. Given this is satisfied, then you should be able to feed your data into MultinomialNB no problem.
Consider padding with zeros:
data1 = np.zeros((len(data), 10))
for i in range(len(data)):
data1[i, :len(data[i])] = data[i]
lets say i have an image presented as this numpy array:
array([[ 55, 229, 185, 21, 128, 50, 109, 121, 251],
[138, 0, 143, 153, 22, 244, 102, 6, 63],
[250, 235, 57, 28, 220, 15, 217, 147, 70],
[121, 164, 128, 224, 56, 206, 104, 87, 154],
[232, 51, 20, 235, 8, 200, 119, 234, 180],
[182, 79, 79, 22, 221, 233, 54, 11, 209],
[249, 64, 92, 70, 167, 151, 214, 188, 213]], dtype=uint8)
this is 7X9 matrix and i want to double the width of the image to 7x18.
i know what to do when you want to compress an image, but im not sure what i supposed to do if i want to increase the size.
thanks!
`
Put your array in a, then
np.repeat(a, 2, axis=1)
gives
array([[ 55, 55, 229, 229, 185, 185, 21, 21, 128, 128, 50, 50, 109,
109, 121, 121, 251, 251],
[138, 138, 0, 0, 143, 143, 153, 153, 22, 22, 244, 244, 102,
102, 6, 6, 63, 63],
[250, 250, 235, 235, 57, 57, 28, 28, 220, 220, 15, 15, 217,
217, 147, 147, 70, 70],
[121, 121, 164, 164, 128, 128, 224, 224, 56, 56, 206, 206, 104,
104, 87, 87, 154, 154],
[232, 232, 51, 51, 20, 20, 235, 235, 8, 8, 200, 200, 119,
119, 234, 234, 180, 180],
[182, 182, 79, 79, 79, 79, 22, 22, 221, 221, 233, 233, 54,
54, 11, 11, 209, 209],
[249, 249, 64, 64, 92, 92, 70, 70, 167, 167, 151, 151, 214,
214, 188, 188, 213, 213]])
Which has shape 7x18.
This question already has answers here:
Understanding slicing
(38 answers)
Closed 7 years ago.
I have a python list of 300+ elements. I am trying to limit the print of the elements from the list, in order to show only 10 at a time.
Here is an example with only 5 elements inside a list, and I want to show just 2:
items = ["one", "two", "three", "four", "five"]
max_num = 2
for item in items:
# here I am not sure how I restrict the number elements from items
# do something
There are multiple ways of doing it.
1. (Pythonic way)
As #aa333 suggested:
for item in items[:max_num]:
print(item)
2. Probably a bit faster:
for i in xrange(max_num):
print(items[i])
3. Using a while loop:
counter = 0
while counter < max_num:
print(items[i])
counter += 1
Try this.
a = range(300)
for i in range(len(a)):
limit = 10;
i=(i*limit)
print a[i:(i+limit)]
if(i>(len(a)-limit-1)):
break;
This is the basic answer. I am sure better logic is available than this.
Output is like:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
[21, 22, 23, 24, 25, 26, 27, 28, 29, 30]......
Hope this is the desired output.
Simple way:
>>> m = 300
>>> k=range(m)
>>> s = 0
>>> e = 10
>>> while m > 0:
... print k[s:e]
... s+=10
... e+=10
... m-=10
...
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39]
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
[60, 61, 62, 63, 64, 65, 66, 67, 68, 69]
[70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
[80, 81, 82, 83, 84, 85, 86, 87, 88, 89]
[90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109]
[110, 111, 112, 113, 114, 115, 116, 117, 118, 119]
[120, 121, 122, 123, 124, 125, 126, 127, 128, 129]
[130, 131, 132, 133, 134, 135, 136, 137, 138, 139]
[140, 141, 142, 143, 144, 145, 146, 147, 148, 149]
[150, 151, 152, 153, 154, 155, 156, 157, 158, 159]
[160, 161, 162, 163, 164, 165, 166, 167, 168, 169]
[170, 171, 172, 173, 174, 175, 176, 177, 178, 179]
[180, 181, 182, 183, 184, 185, 186, 187, 188, 189]
[190, 191, 192, 193, 194, 195, 196, 197, 198, 199]
[200, 201, 202, 203, 204, 205, 206, 207, 208, 209]
[210, 211, 212, 213, 214, 215, 216, 217, 218, 219]
[220, 221, 222, 223, 224, 225, 226, 227, 228, 229]
[230, 231, 232, 233, 234, 235, 236, 237, 238, 239]
[240, 241, 242, 243, 244, 245, 246, 247, 248, 249]
[250, 251, 252, 253, 254, 255, 256, 257, 258, 259]
[260, 261, 262, 263, 264, 265, 266, 267, 268, 269]
[270, 271, 272, 273, 274, 275, 276, 277, 278, 279]
[280, 281, 282, 283, 284, 285, 286, 287, 288, 289]
[290, 291, 292, 293, 294, 295, 296, 297, 298, 299]
If I understood correctly the question, it could be done in the following way:
from math import floor
from numpy import arange
max_num=10.
lis=arange(0,404)
len_num=int(floor(float(len(lis))/max_num))
for i in range(len_num):
print lis[i*int(max_num):(i+1)*int(max_num)]
wait=input()
print(lis[(i+1)*int(max_num):])
This way, your list is shown slice by slice and the next slice isn't shown until the user presses one key