Related
I am trying to train a LSTM with a dataset in which both the input and the output are a sequence of numbers of different lenght. Each number in the input represents a timestep. Example of input and output:
Input:
ent
229 [3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, ...
511 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 3.0, ...
110 [2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, ...
243 [2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, ...
334 [3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, ...
Output:
sal
229 [6.0, 7.0, 3.0, 0.0, 1.0, 4.0, 5.0, 2.0]
511 [0.0, 1.0, 6.0, 7.0, 2.0, 4.0, 5.0, 6.0, 7.0]
110 [3.0, 5.0, 0.0, 1.0, 5.0, 6.0, 7.0, 3.0]
243 [3.0, 6.0, 7.0, 4.0, 6.0, 7.0, 0.0, 1.0, 4.0]
334 [6.0, 7.0, 3.0, 4.0, 3.0, 5.0, 4.0]
When executing the train of the model always appears this error:
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type numpy.ndarray).
model = keras.Sequential()
model.add(layers.Input(shape=(None, 200)))
model.add(layers.LSTM(20))
Should I select a different NN or include padding?
I have also tried changing the dimension to:
ent
229 [[3.0], [3.0], [3.0], [3.0], [3.0], [3.0], [3....
Do you know what could I do?
Traceback (most recent call last):
at block 8, line 8
at /opt/python/envs/default/lib/python3.8/site-packages/keras/utils/traceback_utils.pyline 67, in error_handler(*args, **kwargs)
at /opt/python/envs/default/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.pyline 106, in convert_to_eager_tensor(value, ctx, dtype)
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type numpy.ndarray).
I'd like to perform a convolution in a Lambda layer, but I can't get it to work any way.
kernel = [1.0,2.0,1.0] # weighted moving average
x = [ # history_size=5, num_features=10
[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0],
[2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0],
[3.0,3.0,3.0,3.0,3.0,3.0,3.0,3.0,3.0,3.0],
[4.0,4.0,4.0,4.0,4.0,4.0,4.0,4.0,4.0,4.0],
[5.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0],
]
k = tf.constant(kernel, dtype=tf.float32)
y = tf.nn.conv1d(x, k, stride=1, padding='SAME')
I realize dimensions are not correct in the above example, but that's my data's actual format. The training samples have a shape of (history_size, num_features) and the kernel has to convolve along history_size, each feature separately. Any help would be appreciated. I cannot find an example on how to perform tf.nn.conv1d manually.
You could use numpy.convolve() for this.
import numpy as np
kernel = [1.0,2.0,1.0] # weighted moving average
x = [ # history_size=5, num_features=10
[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0],
[2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0,2.0],
[3.0,3.0,3.0,3.0,3.0,3.0,3.0,3.0,3.0,3.0],
[4.0,4.0,4.0,4.0,4.0,4.0,4.0,4.0,4.0,4.0],
[5.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0,5.0],
]
output = []
for i in range(len(x)):
output.append(list(np.convolve(x[i], kernel, mode = 'same')))
output
'''
[[3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0],
[6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0],
[9.0, 12.0, 12.0, 12.0, 12.0, 12.0, 12.0, 12.0, 12.0, 9.0],
[12.0, 16.0, 16.0, 16.0, 16.0, 16.0, 16.0, 16.0, 16.0, 12.0],
[15.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 15.0]]
'''
You could try changing the mode whichever fits best to you according to the documentation.
I have an numpy array called expected which is a list of a list of a list.
expected = [[[45.0, 10.0, 10.0], [110.0, 10.0, 8.0], [60.0, 10.0, 5.0], [170.0, 10.0, 4.0]], [[-80.0, 20.0, 10.0], [97.0, 15.0, 12.0], [5.0, 20.0, 8.0], [93.0, 10.0, 8.0], [12.0, 5.0, 15.0], [-88.0, 10.0, 10.0], [176.0, 10.0, 8.0]]]
I want to put it through a loop without having to hardcode so its applicable to different lengths of list.
When the loop runs for the first time i want it to solve this:
horizontal_exp = expected[0][0][1]*expected[0][0][2]
*np.cos(np.deg2rad(expected[0][0][0]))
Then the next loop to be like this:
horizontal_exp = expected[1][1][1]*expected[1][1][2]
*np.cos(np.deg2rad(expected[1][1][0]))
And the following loop to be like this:
horizontal_exp = expected[2][2][1]*expected[2][2][2]
*np.cos(np.deg2rad(expected[2][2][0]))
and so on until it finished the different sections of rows.
I don't understand why the 'i' never worked??
In the end I want horizontal expected to be a list of a list
e.g.
expected = [ [12,21,23,34], [12,32,54,65,76,87,65] ] # These are not the values I'm just giving an example
where the [12,21,23,24] corresponds to the [[45.0, 10.0, 10.0], [110.0, 10.0, 8.0], [60.0, 10.0, 5.0], [170.0, 10.0, 4.0]]
and the [12,32,54,65,76,87,65] corresponds to the [[-80.0, 20.0, 10.0], [97.0, 15.0, 12.0], [5.0, 20.0, 8.0], [93.0, 10.0, 8.0], [12.0, 5.0, 15.0], [-88.0, 10.0, 10.0], [176.0, 10.0, 8.0]]
I'm unsure how to do this, I know you have to append it with a for loop but how do you separate it into a list of a list??
horizontal_expected = []
for i in list(range(len(expected[i]))):
horizontal_exp = expected[i][i][1]*expected[i][i][2]
*np.cos(np.deg2rad(expected[i][i][0]))
horizontal_expected.append(horizontal_exp)
print(horizontal_expected)
The reason why you don't see the desired output is that, even though you have nested list expected, you are iterating only through the nested lists. You first need to iterate through the outer lists and then iterate through the nested lists internally:
import numpy as np
expected = [ [[45.0, 10.0, 10.0], [110.0, 10.0, 8.0], [60.0, 10.0, 5.0], [170.0, 10.0, 4.0]], [[-80.0, 20.0, 10.0], [97.0, 15.0, 12.0], [5.0, 20.0, 8.0], [93.0, 10.0, 8.0], [12.0, 5.0, 15.0], [-88.0, 10.0, 10.0], [176.0, 10.0, 8.0]] ]
horizontal_expected = []
for i in range(len(expected)):
tmp_list = []
for j in range(len(expected[i])):
horizontal_exp = expected[i][i][1]*expected[i][i][2]*np.cos(np.deg2rad(expected[i][i][0]))
tmp_list.append(horizontal_exp)
horizontal_expected.append(tmp_list)
print(horizontal_expected)
The output of that is a list of lists:
>>> print(horizontal_expected)
[[70.71067811865476, 70.71067811865476, 70.71067811865476, 70.71067811865476], [-21.936481812926527, -21.936481812926527, -21.936481812926527, -21.936481812926527, -21.936481812926527, -21.936481812926527, -21.936481812926527]]
As you can see, it holds a value for each of the lists in the input, but the value is the same. This is due to the way that your equation was set up.
You want the indices to be updated based on the level of the loop:
horizontal_exp = expected[i][j][1]*expected[i][j][2]*np.cos(np.deg2rad(expected[i][j][0]))
The full working code would look like this:
import numpy as np
expected = [ [[45.0, 10.0, 10.0], [110.0, 10.0, 8.0], [60.0, 10.0, 5.0], [170.0, 10.0, 4.0]], [[-80.0, 20.0, 10.0], [97.0, 15.0, 12.0], [5.0, 20.0, 8.0], [93.0, 10.0, 8.0], [12.0, 5.0, 15.0], [-88.0, 10.0, 10.0], [176.0, 10.0, 8.0]] ]
horizontal_expected = []
for i in range(len(expected)):
tmp_list = []
for j in range(len(expected[i])):
horizontal_exp = expected[i][j][1]*expected[i][j][2]*np.cos(np.deg2rad(expected[i][j][0]))
tmp_list.append(horizontal_exp)
horizontal_expected.append(tmp_list)
print(horizontal_expected)
And the output:
>>> print(horizontal_expected)
[[70.71067811865476, -27.361611466053496, 25.000000000000007, -39.39231012048832], [34.72963553338608, -21.936481812926527, 159.39115169467928, -4.186876499435507, 73.36107005503543, 3.489949670250108, -79.80512402078594]]
I have the following Python dictionary:
b = {'SP:1': 1.0,
'SP:2': 2.0,
'SP:3': 3.0,
'SP:4': 4.0,
'SP:5': 5.0,
'SP:6': 6.0,
'SP:7': 40.0,
'SP:8': 7.0,
'SP:9': 8.0}
I want to take this list and iterate over it to create 9 lists, each successive list being a superset of its predecessor. So:
[1.0]
[1.0,2.0]
[1.0,2.0,3.0]
[1.0,2.0,3.0,4.0]
...
[1.0,2.0,3.0,4.0,5.0,6.0,40.0,7.0,8.0]
There is probably a really easy way of doing this with a list comprehension, but I cant work it out!
You can do the following:
>>> vals = [v for k, v in sorted(b.items())]
# or shorter, but less explicit:
# vals = [b[k] for k in sorted(b)]
>>> [vals[:i+1] for i in range(len(vals))]
[[1.0],
[1.0, 2.0],
[1.0, 2.0, 3.0],
[1.0, 2.0, 3.0, 4.0],
[1.0, 2.0, 3.0, 4.0, 5.0],
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0],
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 40.0],
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 40.0, 7.0],
[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 40.0, 7.0, 8.0]]
The first comprehension gives you the values sorted by key as the initial dict is inherently unordered. The second gives you all of the desired slices of that list of values.
Dictionaries are not meant to be used in this form, and should never considered to be ordered. However, since the keys are basically indicies, we can use them like that:
[[b['SP:'+str(j+1)] for j in range(i+1)] for i in range(len(b))]
I have a list of lists:
a = [[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0],
[3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0],
[1.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0],
[5.0, 5.0, 5.0],
[1.0]
]
What I need to do is remove all the duplicates in the list of lists and keep the previous sequence. Such as
a = [[1.0],
[2.0, 3.0, 4.0],
[3.0, 5.0],
[1.0, 4.0, 5.0],
[5.0],
[1.0]
]
If order is important, you can just compare to the set of items seen so far:
a = [[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0],
[3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0],
[1.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0],
[5.0, 5.0, 5.0],
[1.0]]
for index, lst in enumerate(a):
seen = set()
a[index] = [i for i in lst if i not in seen and seen.add(i) is None]
Here i is added to seen as a side-effect, using Python's lazy and evaluation; seen.add(i) is only called where the first check (i not in seen) evaluates True.
Attribution: I saw this technique yesterday from #timgeb.
If you have access to the OrderedDict (in Python 2.7 on), abusing it a good way to do this:
import collections
import pprint
a = [[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0],
[3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0],
[1.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0],
[5.0, 5.0, 5.0],
[1.0]
]
b = [list(collections.OrderedDict.fromkeys(i)) for i in a]
pprint.pprint(b, width = 40)
Outputs:
[[1.0],
[2.0, 3.0, 4.0],
[3.0, 5.0],
[1.0, 4.0, 5.0],
[5.0],
[1.0]]
This will help you.
a = [[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0],
[3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0],
[1.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0],
[5.0, 5.0, 5.0],
[1.0]
]
for _ in range(len(a)):
a[_] = sorted(list(set(a[_])))
print a
OUTPUT:
[[1.0], [2.0, 3.0, 4.0], [3.0, 5.0], [1.0, 4.0, 5.0], [5.0], [1.0]]
Inspired by DOSHI, here's another way, probably best way for a small number of possible elements (i.e. a small number of index lookups for sorted) otherwise a way that remembers insertion order may be better:
b = [sorted(set(i), key=i.index) for i in a]
So just to compare the methods, a seen set versus sorting a set by an original index lookup:
>>> setup = 'l = [1,2,3,4,1,2,3,4,1,2,3,4]*100'
>>> timeit.repeat('sorted(set(l), key=l.index)', setup)
[23.231241687943111, 23.302754517266294, 23.29650511717773]
>>> timeit.repeat('seen = set(); [i for i in l if i not in seen and seen.add(i) is None]', setup)
[49.855933579601697, 50.171151882997947, 51.024657420945005]
Here we see that for a larger case where, the contain test that Jon uses for every element becomes relatively very costly, and since insertion order is quickly determined by index in this case, this method is much more efficient.
However, by appending more elements to the end of the list, we see that Jon's method does not bear much increased cost, whereas mine does:
>>> setup = 'l = [1,2,3,4,1,2,3,4,1,2,3,4]*100 + [8,7,6,5]'
>>> timeit.repeat('sorted(set(l), key=l.index)', setup)
[93.221347206941573, 93.013769266020972, 92.64512197257136]
>>> timeit.repeat('seen = set(); [i for i in l if i not in seen and seen.add(i) is None]', setup)
[51.042504915545578, 51.059295348750311, 50.979311841569142]
I think I'd prefer Jon's method with a seen set, given the bad lookup times for the index.