I have the same question for tf.contrib.rnn.LSTMBlockCell and tf.contrib.cudnn_rnn.CudnnCompatibleLSTMCell:
How do I initialize the LSTM weights from numpy arrays correctly? The following code-snipped executes, but does not seem
to do what I am looking for:
train_data = np.load('mnist_train_data.npy').reshape(-1,28,28)
train_label = np.load('mnist_train_label.npy')
params = [np.random.randn(28+128, 4*128), np.zeros(4*128)]
X = tf.placeholder(tf.float32, shape=[54999, 28, 28])
y = tf.placeholder(tf.int64, None)
state = LSTMStateTuple(*(tf.zeros((54999, 128), dtype=tf.float32) for _ in range(2)))
cell = tf.contrib.rnn.LSTMBlockCell(128)
cell.build(tf.TensorShape((None, 28)))
cell.set_weights(params)
initial_weights = cell.get_weights()
print(np.array_equal(params[0], initial_weights[0]))
w1 = tf.Variable(np.random.randn(128, 10), dtype=tf.float32)
b1 = tf.Variable(np.zeros(10), dtype=tf.float32)
full_seq, current_state = tf.nn.dynamic_rnn(cell, X, initial_state=state, dtype=tf.float32)
output = tf.matmul(current_state[1], w1)
output += b1
loss = tf.losses.softmax_cross_entropy(y, output)
train_step = tf.train.AdamOptimizer(0.01).minimize(loss)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(1):
feed_dict = {X: train_data, y: train_label}
sess.run(train_step, feed_dict=feed_dict)
final_weights = cell.get_weights()
print(np.array_equal(initial_weights[0], final_weights[0]))
This prints out False in the first print-statement, so the numpy arrays do not actually seem to be used as weights.
Moreover, after the training session, this prints out True thus implying, that these weights are not actually updated during training.
Thanks in advance for any help on the subject.
Lets say that my input data, x, has the shape (2000, 2) where 2000 is the number of samples and 2 is the number of features.
So for this input data, I can setup a place holder like this:
x = tf.placeholder(tf.float32, shape=[None, 2], name='features')
My question is, if I transpose my input data, x, so that the shape is now (2, 2000) where 2000 is still the number of samples, how would I change the "shape" parameter in tf.placeholder?
I have tried setting shape=[2, None], but I just get an error. Does the 1st element in the "shape" parameter always have to be "None"?
Here the error I get: "ValueError: The last dimension of the inputs to Dense should be defined. Found None."
import tensorflow as tf
# Binary Classifier Implementation
# Training data
x_train = np.transpose(X) #shape=(2, 2000)
y_train = np.hstack((np.zeros((1, 1000)),np.zeros((1, 1000)) + 1)) #shape=(1, 2000)
# Variables
x = tf.placeholder(tf.float32, shape=[2, None], name='features')
y_ = tf.placeholder(tf.int64, shape=[1, None], name='labels')
h1 = tf.layers.dense(inputs=x, units=50, activation=tf.nn.relu) #one hidden layer with 50 neurons
y = tf.layers.dense(inputs=h1, units=1, activation=tf.nn.sigmoid) #one output layer with 1 neuron
# Functions
#loss
cross_entropy = tf.losses.sigmoid_cross_entropy(multi_class_labels=y_, logits=y)
train_step = tf.train.GradientDescentOptimizer(0.1).minimize(cross_entropy)
# Initializer
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
for i in range(1000):
sess.run([cross_entropy], feed_dict={x: x_train, y_: y_train})
I think you might be having trouble with shape inconsistency, not with transposing or having an undefined dimension size somewhere other than the first dim.
The following things work fine for me:
import tensorflow as tf
x = tf.placeholder(tf.float32, shape=[None, 2])
x_t = tf.transpose(x)
y = tf.placeholder(tf.float32, shape=[2, None])
print(x_t.shape)
>> TensorShape([Dimension(2), Dimension(None)])
print(y.shape)
>> TensorShape([Dimension(2), Dimension(None)])
I'm assuming the first dimension might be your batch-size? Are you maybe not being consistent with this or feeding the wrong shaped data?
Or maybe you're trying to manually change the shape of the tensor? (If so then you don't need to. tf takes care of that as apparent in my example).
That's the most one can help without your giving an idea of the error you're getting, a code snippet, or anything more than a vague and general description, but I hope it helps nonetheless.
Good luck!
Update due to updated question:
The error you're getting is telling you exactly what the problem is. You can transpose whatever you want, but the dense layer specifically cannot accept a tensor for which the last dimension is None. This makes sense; you have cannot create a fully connected layer without knowing the sizes of the weight matrices for instance.
If you're transposing x, then you're saying you have 2000 features (and 2 data-points) and the NN needs to know this in order to create the parameters you're going to train.
If you still consider yourself to have 2 features and however-many examples, then you shouldn't be working with a shape of (2, 2000) in the first place!
Try the following:
# Training data
x_train = X #shape=(2000, 2)
y_train = np.hstack((np.zeros((1000, 1)),np.zeros((1000, 1)) + 1)) #shape=(2000, 1)
# Variables
x = tf.placeholder(tf.float32, shape=[None, 2], name='features')
y_ = tf.placeholder(tf.int64, shape=[None, 1], name='labels')
h1 = tf.layers.dense(inputs=x, units=50, activation=tf.nn.relu) #one hidden layer with 50 neurons
y = tf.layers.dense(inputs=h1, units=1, activation=tf.nn.sigmoid) #one output layer with 1 neuron
# Functions
#loss
cross_entropy = tf.losses.sigmoid_cross_entropy(multi_class_labels=y_, logits=y)
train_step = tf.train.GradientDescentOptimizer(0.1).minimize(cross_entropy)
# Initializer
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
for i in range(1000):
sess.run([cross_entropy], feed_dict={x: x_train, y_: y_train})
Hope this helps.
On a completely different and unrelated note: I hope you know what you're doing when embedding 2 features in a 50 dimensional space that way.
I'm completely new to Python and Tensorflow. I want to implement a simple kind of CNN, and this is what I have done till now:
import tensorflow as tf
import numpy as np
from libs import utils
import cv2
import glob
from tensorflow.python.framework.ops import reset_default_graph
reset_default_graph()
# We first get the graph that we used to compute the network
g = tf.get_default_graph()
# And can inspect everything inside of it
[op.name for op in g.get_operations()]
X = tf.placeholder(tf.float32, [None,720000])
Y = tf.placeholder(tf.int32, [None])
X_data = []
files = glob.glob ("C:/Users/Maede/Desktop/Master Thesis/imlearning/*.jpg")
for myFile in files:
print(myFile)
image = cv2.imread (myFile)
X_data.append (image)
print('X_data shape:', np.array(X_data).shape)
data=np.array(X_data)
data=np.reshape(data,(30,720000))
label=np.array([(0,1),(1,0),(0,1),(1,0),(0,1),(1,0),(0,1),(1,0),(0,1),(1,0),
(0,1),(1,0),(0,1),(1,0),(0,1),(1,0),(0,1),(1,0),(0,1),(1,0),
(0,1),(1,0),(0,1),(1,0),(0,1),(1,0),(0,1),(1,0),(0,1),(1,0)])
###########################################################
train_batch_size = 2
def random_batch():
num_images = 30
idx = np.random.choice(num_images,
size=train_batch_size,
replace=False)
x_batch = data[idx,:]
y_batch = label[idx, :]
return x_batch, y_batch
######################
#
X_tensor = tf.reshape(X, [-1, 400,600,3])
filter_size = 5
n_filters_in = 3
n_filters_out = 32
W_1 = tf.get_variable(
name='W',
shape=[filter_size, filter_size, n_filters_in, n_filters_out],
initializer=tf.random_normal_initializer())
b_1 = tf.get_variable(
name='b',
shape=[n_filters_out],
initializer=tf.constant_initializer())
h_1 = tf.nn.relu(
tf.nn.bias_add(
tf.nn.conv2d(input=X_tensor,
filter=W_1,
strides=[1, 2, 2, 1],
padding='SAME'),
b_1))
n_filters_in = 32
n_filters_out = 64
n_output = 2
W_2 = tf.get_variable(
name='W2',
shape=[filter_size, filter_size, n_filters_in, n_filters_out],
initializer=tf.random_normal_initializer())
b_2 = tf.get_variable(
name='b2',
shape=[n_filters_out],
initializer=tf.constant_initializer())
h_2 = tf.nn.relu(
tf.nn.bias_add(
tf.nn.conv2d(input=h_1,
filter=W_2,
strides=[1, 2, 2, 1],
padding='SAME'),
b_2))
# We'll now reshape so we can connect to a fully-connected/linear layer:
h_2_flat = tf.reshape(h_2, [-1, 100*150* n_filters_out])
# NOTE: This uses a slightly different version of the linear function than the lecture!
h_3, W = utils.linear(h_2_flat, 400, activation=tf.nn.relu, name='fc_1')
# NOTE: This uses a slightly different version of the linear function than the lecture!
Y_pred, W = utils.linear(h_3, n_output, activation=tf.nn.softmax, name='fc_2')
y_one_hot = tf.one_hot( Y , 2 )
cross_entropy = -tf.reduce_sum(y_one_hot * tf.log(Y_pred + 1e-12))
optimizer = tf.train.AdamOptimizer().minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(Y_pred, 1), tf.argmax(y_one_hot, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))
sess = tf.Session()
sess.run(tf.global_variables_initializer())
batch_size = 2
n_epochs = 5
for epoch_i in range(n_epochs):
for batch_xs, batch_ys in random_batch():
sess.run(optimizer, feed_dict={
X: np.array(batch_xs).reshape([1,720000]),
Y: batch_ys
})
valid = data #### DATA haie validation
print(sess.run(accuracy,
feed_dict={
X: data,
Y: label
}))
The input is 30 image with 400*600*3 dimension and i want to classify them into two class. The problem is when i'm using this command:
X: np.array(batch_xs).reshape([1,720000]),
The error is like below:
ValueError: cannot reshape array of size 2 into shape (1,720000)
and when I'm using:
X: batch_xs
The error is:
ValueError: Cannot feed value of shape (720000,) for Tensor 'Placeholder:0', which has shape '(?, 720000)'
I'm totally confused what is batch_xs dimension and why does it change in different situation.
np.array(batch_xs) is not the same size as your image.
for batch_xs, batch_ys in random_batch() is also a slightly strange way to run the code and I guess this also causes your problem. You usually use for to iterate over a some iterable.
In your case the iterable is just what your function returns, a tuple with batch_xs, batch_ys. But in the same step you are unpacking the first (!) value of the tuple into two variables batch_xs and batch_ys.
The replace=False does not do anything in your case, because you are calling the function random_batch() only once. In the next iteration it will have the complete dataset again.
Here is an simple example on your case:
import numpy as np
# I removed a dimension from the arrays
data = np.array([[1.0, 1.0, 1.0],
[2.0, 2.0, 2.0],
[3.0, 3.0, 3.0]])
label = np.array([[10.0, 10.0, 10.0],
[20.0, 20.0, 20.0],
[30.0, 30.0, 30.0]])
def random_batch():
idx = np.random.choice(3, size=2)
x_batch = data[idx,:]
y_batch = label[idx, :]
return x_batch, y_batch
# the outer variable names x_batch and y_batch are not related at all to the ones
# inside random_batch()
# iterate over whatever random_batch() returns
# for x_batch, y_batch in random_batch() is equivalent to
# for (x_batch, y_batch) in random_batch()
# in the first iteration the iterable is `x_batch`, in the second one`y_batch`.
# and each of the iterable is "unpacked", basically in the first iteration
# your are assigning
# (x_batch, y_batch) = x_batch
# in the second iteration
# (x_batch, y_batch) = y_batch
# When unpacking you are splitting the two elements created by `size=2`
# in `random_batch()`
for (x_batch, y_batch) in random_batch():
print(x_batch)
print(y_batch)
This is fundamental Python basics, to get familiar with it look for tuple unpacking, iterable and for loops.
Replace the inner for-loop with this, it should work. It might not be what you expected, but it is what you code is supposed to do.
batch_xs, batch_ys = random_batch()
sess.run(optimizer, feed_dict={
X: np.array(batch_xs).reshape([1,720000]),
Y: batch_ys
})
If if you want to train with 100 batches do something like this
for k in range(100):
batch_xs, batch_ys = random_batch()
sess.run(optimizer, feed_dict={
X: np.array(batch_xs).reshape([1,720000]),
Y: batch_ys
})
Usually you try to remove as much of the code that is not related to the problem to make it easier to find the problem. Find as less code as possible that still shows your problem.
Your problem is not related to tensorflow, so you could remove everything related to tensorflow to make it easier to find. Your problem is related to numpy and array shapes.
I'm working on adapting one of the MNIST tensorflow tutorials, and I receive this TypeError. According to this question you have to use a placeholder in the dictionary key because numpy arrays are mutable. I believe I'm doing that, but I'm still receiving this error.
# Network Parameters
n_input = 44100 # length of FFT
n_classes = 6 # 6 instrument classes
dropout = 0.75 # Dropout, probability to keep units
# TF Graph input
x = tf.placeholder(tf.float32, [None, n_input])
y = tf.placeholder(tf.float32, [None, n_classes])
keep_prob = tf.placeholder(tf.float32)
I fill up my batches and then pass them to the session.
for file_name in os.listdir('./Input_FFTs'):
if file_name.endswith('.txt'):
path = './Input_FFTs/' + file_name
y, x = getData(path)
batch_ys[count] = y
batch_xs[count] = x
count += 1
sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys,
keep_prob: dropout})
When I print and check the sizes of batch_xs and batch_ys, they are [batch_size, 44100] and [batch_size, 6] with the correct data. These match the expected sizes of the x and y placeholders.
Can anyone tell me what the problem may be?
Thank you!
Be very careful about your variable names!
I was replacing my placeholders x, y with arrays x, and y in my loops to fill my train and test patches.
I just started using tensorflow and I followed the tutorial example on MNIST dataset. It went well, I got like around 90% accuracy.
But after I replace the next_batch with my own version, the result was way worse than it used to be, usually 50%.
Instead of using the data Tensorflow downloaded and parsed, I download the dataset from this website. Using numpy to get what I want.
df = pd.read_csv('mnist_train.csv', header=None)
X = df.drop(0,1)
Y = df[0]
temp = np.zeros((Y.size, Y.max()+1))
temp[np.arange(Y.size),Y] = 1
np.save('X',X)
np.save('Y',temp)
do the same thing to the test data, then following the tutorial, nothing is changed
x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])
X = np.load('X.npy')
Y = np.load('Y.npy')
X_test = np.load('X_test.npy')
Y_test = np.load('Y_test.npy')
BATCHES = 1000
W = tf.Variable(tf.truncated_normal([784,10], stddev=0.1))
# W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
sess = tf.InteractiveSession()
tf.global_variables_initializer().run()
right here is my own get_mini_batch, I shuffle the original data's index, then every time I get 100 data out of it, which seems to be like the exact same thing example code does. The only difference is data I throw away some of the data in the tail.
pos = 0
idx = np.arange(X.shape[0])
np.random.shuffle(idx)
for _ in range(1000):
batch_xs, batch_ys = X[idx[range(pos,pos+BATCHES)],:], Y[idx[range(pos,pos+BATCHES)],]
if pos+BATCHES >= X.shape[0]:
pos = 0
idx = np.arange(X.shape[0])
np.random.shuffle(idx)
pos += BATCHES
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
print(sess.run(accuracy, feed_dict={x: X_test, y_: Y_test}))
It confuses me why my version is way worse than the tutorial one.
Like lejilot said, we should normalize the data before we push it into the neural network.
See this post