Let's say I have a tensor of shape [3000,20,10], I'll call it input... I have another tensor of shape [3000] that contains only indices for the 3rd dimension of the first tensor, I'll call it "indices"
Basically, for every i in 0:2999, I want input[i,20,indices[i]]... So my final output will be of shape [3000,20]
I should be able to easily do this with a for loop, but I would like to find how to do this without a for loop... I tried looking at tf.gather but was honestly too confused to have any success in that direction
Any help would be greatly appreciated
Try this code:
from tensorflow.keras.layers import Conv2D, Input, BatchNormalization, Activation, Add, MaxPooling2D, LSTM, Dense
import tensorflow as tf
input = tf.range(3000*20*10)
input = tf.reshape(input, (3000,20,10))
indices = tf.random.uniform((3000,), 0, 10, dtype=tf.int32)
inds = indices[:, tf.newaxis, tf.newaxis]
inds = tf.repeat(inds, 20, 1)
output = tf.gather(input, inds, batch_dims=2, axis=2)
output = tf.squeeze(output)
import random
for _ in range(100):
i = random.randint(0, 2999)
j = random.randint(0, 19)
print(input[i][j][[indices[i]]]==output[i][j])
Related
I struggle with understanding the effect channels vs. width in a Conv2D layer of tensorflow. I am trying to use it to process multivariant time series data, so the meaningfulness of my input in shape (t, ch, 1) and (t, 1, ch) is pretty much the same.
My question is if the two code examples below produce the mathematical operation to the same output when used to build a tensorflow model:
# Code A:
import tensorflow as tf
from keras.layers import Input, Reshape, Conv2D
x = Input((t, ch, 1))
x = Reshape((t, 1, ch))(x)
x = Conv2D(n_filters,
kernel_size=(k,1),
strides=(1,1),
padding='valid')(x)
x = Reshape((t, n_filters))(x)
...
# Code B:
import tensorflow as tf
from keras.layers import Input, Conv2D
x = Input((t, ch, 1))
x = Conv2D(n_filters,
kernel_size=(k,ch),
strides=(1,1),
padding='valid')(x)
Given that the contend of the kernels form Code A spread over the different channels are the same as the kernels in Code B spread over the width of the kernel. Meaning kernelA[k, 1, ch]== kernel_B[k, ch, 1].
I am not sure how I would go about testing this. Maybe some can help my understanding just by explaining or propose an experiment.
Tensor("flatten_3/Identity:0", shape=(None, 100), dtype=float32)
Hi I have tried to print tensor as numpy array as follows:
import tensorflow as tf
import numpy as np
from keras.layers import Input
print(tf.executing_eagerly())
x = Input(shape=(32,))
print(x.numpy())
To be honest I'm sure there is a cleaner way to visualize your input tensor, but here's a hacky one for what it's worth:
import tensorflow as tf
def tf_print(x):
tf.print("my tensor:")
tf.print(x)
return x
iput = tf.keras.layers.Input(shape=(1,), dtype='int32')
printt = tf.keras.layers.Lambda(tf_print)(iput) # branch that prints your tensor
oput = tf.keras.layers.Flatten()(iput) # branch that is the rest of your model
model = tf.keras.Model(inputs=[iput], outputs=[oput, printt])
model(4)
I'm making a model whose output I want to be of dims (A,B).
So I'm making a list of denses (A elements, B outputs each) and I wanted my output to be (No_samples, A,B).
It is a list of A elements with (No_samples,B). The method of having one dense with output AxB does not help because for every row I want to softmax accross only that
I've attempted to use tf.concatenate, tf.reshape, but there is always either an error or the same undesirable output. My difficulty is that in order to proceed I have to do some really weird reshaping and I wish to avoid that by
for i in range(0, A):
outputs.append(Dense(B, activation="softmax")(out))
And I've tried everything below (separately):
outputs = tf.stack(outputs)
outputs = Reshape(self.output_shape)(outputs)
outputs = tf.convert_to_tensor(outputs)
The expected outcome is that the output has shape of (A,?,B) instead of (?, A, B). Is there another method that I can have multiple denses in parralel with the above behaviour?
Simple example with A=3, B=1.
from keras import backend as K
from keras.layers import Concatenate, Dense, Input, Lambda
from keras.models import Model
import numpy as np
def expand_dims(x):
return K.expand_dims(x, axis=-2) #expand (None, 1) to (None, 1, 1)
x = Input((2,))
A = 3
B = 1
y = Lambda(expand_dims)(Dense(B, activation="softmax")(x))
for i in range(0, A-1):
# Concatenate on the newly added dimension
y = Concatenate(axis=-2)([y,Lambda(expand_dims)(Dense(B, activation="softmax")(x))])
model = Model(x, y)
print(model.predict(np.ones((4,2))).shape)
(4, 3, 1) # Output shape is (No_samples, A,B)
I would like to extract single values from a tensor and manipulate it while retaining backpropagation. My current implementation:
import keras
from keras import backend as K
from keras.models import Model
from keras.layers import Dense, Activation, Input
import tensorflow as tf
input = Input(shape=(100,1), dtype='float32')
x = Dense(100)(input)
x = Activation('relu')(x)
x = Dense(5)(x)
x = Activation('tanh')(x)
start_pad = 40.0 + 5.0 * x[0] # important line
# ...
zs = K.arange(0.0, 1000, step=1.0)
zs = K.relu( zs - start_pad )
# ...
out = zs # + ...
out = Reshape( (trace_length,1) )(out)
model = Model(inputs = input, outputs = out)
However, start_pad seems to be a tensor with dimensions of x. Running code above gives error:
ValueError: Dimensions must be equal, but are 1000 and 5 for 'sub' (op: 'Sub') with input shapes: [1000], [100,5].
where start_pad object is <tf.Tensor 'add_1:0' shape=(100, 5) dtype=float32>.
I would like to have scalar like value for start_pad and subtract from zs with broadcasting. How do I achive this with Tensorflow/Keras?
Ok, the solution i found is
x = tf.unstack(x, axis=1)
which returns a list of tf tensors
Summary
I am having trouble being able to provide the proper dimensions for my input data into a prediction after the model has already been properly generated.
I'm receiving the following error:
ValueError: Error when checking : expected lstm_13_input to have shape (None, 40, 39) but got array with shape (1, 39, 39)
Background
Using Anaconda as my virtual environment
Keras version 2.0.6
TensorFlow version 1.1.0
I am creating an example fairly closely following this tutorial.
Code
And here are the relevant snippets of code:
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.layers import LSTM,TimeDistributed,SimpleRNN
from keras.utils.data_utils import get_file
import numpy as np
from time import sleep
import random
import sys
...
X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)
y = np.zeros((len(sentences),maxlen, len(chars)), dtype=np.bool) # y is also a sequence , or a seq of 1 hot vectors
for i, sentence in enumerate(sentences):
for t, char in enumerate(sentence):
X[i, t, char_indices[char]] = 1
for i, sentence in enumerate(next_chars):
for t, char in enumerate(sentence):
y[i, t, char_indices[char]] = 1
...
model = Sequential()
model.add(LSTM(512, return_sequences=True, input_shape=(maxlen, len(chars)))) # original one
model.add(LSTM(512, return_sequences=True)) #- original
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(len(chars))))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
...
history=model.fit(X, y, batch_size=128, nb_epoch=1,verbose=0)
...
seed_string="brutus:\nbeing so moved, he will not spare"
x=np.zeros((1, len(seed_string), len(chars)))
for t, char in enumerate(seed_string):
x[0, t, char_indices[char]] = 1.
preds = model.predict(x, verbose=0)[0]
Error
At this last line, it errors out, with the following error:
ValueError: Error when checking : expected lstm_13_input to have shape (None, 40, 39) but got array with shape (1, 39, 39)
Efforts to Resolve
I have played with the dimensionality of the seed_string and the x generated from it, but no matter how I try to adjust them, I have some sort of mis-match, always due to this requirement of None (I think). To be clear, I had either added or taken away characters from the seed string, so that it is 40 characters. However, when I set it to 40, the error says I actually have 41, and when I set it to 39, it says I have 39, as shown above. There is something else - which I don't understand - going on here.
I looked at Reshape's code and an example of how to use it, but since Keras' Reshape is meant for model layers, I don't even understand how I could use it to reshape the input to the prediction, and Numpy has no way to reshape creating a None dimension (at least not to my knowledge).
The length of your seed_string needs to match maxlen. You can handle both the scenario of your string being shorter or longer than maxlen with pad_sequences. In your case your string is too long.
from keras.preprocessing.sequence import pad_sequences
seed_string = "brutus:\nbeing so moved, he will not spare"
x = np.zeros((1, len(seed_string), len(chars)))
for t, char in enumerate(seed_string):
x[0, t, char_indices[char]] = 1.
x = pad_sequences(x, maxlen=maxlen)
preds = model.predict(x, verbose=0)[0]