Develop Reference

Keras seems slower than tensorflow while feeding training data

I am currently converting a project from tensorflow to keras.
Everything seems fine and I am very impressed by how easy it is to build models with keras. However, training is much slower with Keras, where my GPU is significantly less utilized.
I am using a Tensorflow generator Dataset to load my training data. Luckily keras seems to accept that with no problems.
The problem
However while using tensorflow to train on the dataset I archieve an average GPU utilization of ~70%. When I am training the same network with the same dataset generator using Keras I only archieve ~35% GPU utilization
The problem seems to be that I have a very simple network, hence I need to feed data to the GPU as fast as possible since much time is spent here, compared to actually doing backpropagation.
Using tensorflow the key here seemed to be to not use feed-dicts but instead use the tensor from my dataset as input to the graph. Basically, this can be reduced to
x, y = iterator.get_next() # Get the dataset tensors
loss = tf.reduce_sum(tf.square(y - model_out)) # Use the y tensor directly for loss
# Use x as the input layer in my model <- Implememntation omitted
I would like to achieve the same thing with keras, hence I did something like this, where i set x as the input and y as the target tensor. (Can I somehow get rid of having to put y in a list for the target tensor?)
x, y = iterator.get_next() # Get the dataset tensors
model_input = keras.Input(tensor=x)
# Build model with model_input as input layer and something as output layer. <- Implememntation omitted
model = tf.keras.Model(inputs=model_input, outputs=something) # Insert the dataset tensor directly as input
model.compile(loss='mean_squared_error',
optimizer=#something,
metrics=['accuracy'],
target_tensors=[y]) # Input the dataset y tensor directly for use in the loss calculation
Basically that should set x as the input tensor and y as the tensor used directly for loss, just like in the tensorflow version. I can now cal model.fit without providing x and y arguments explicitly since they are used directly in the graph
model.fit(validation_data=validation_iterator,
steps_per_epoch=5000,
validation_steps=1)
To me it seems like I am doing the same thing now with keras and tensorflow, however, keras is way slower with about half the GPU utilization of the pure tensorflow implementation
Am I doing something wrong here, or should i just accept this slowdown if I want to use keras?

I experienced the same issue on TensorFlow 1.13 and solved it by upgrading to TensorFlow 1.14 / 2.0.0.
For a sanity check, I wrapped the TensorFlow graph (as is) as a Keras model and trained the model using model.fit(). When using TensorFlow 1.13, I got a slowdown in throughput of 50% relative to the throughput of training the pure TensorFlow implementation. In both cases, I used the same tf.data.dataset input pipeline.
Using TensorFlow version 1.14 solved the issue (now I get the ~same throughput for both cases mentioned above). Later I migrated to TensorFlow 2.0.0 (alpha) and also got the same throughput for both cases.

How to determine activation, loss, optimizer in keras while making artificial neural network

This is my dataframe
https://drive.google.com/file/d/1qAnyOkp_YayqzZ4i0CwqCTDiYTIOmv6I/view?usp=sharing
I need to find the value of ra, last column of that dataset via the ANN
I have used keras library to make that, here is my code
https://gist.github.com/anonymous/9955247ad7341e5bc119556dead9fc71
But the y_pred variable has set of 0s in output. Am I doing anything wrong with activation function?
I need to predict the ra values with training dataset
P.S: I am a newbie to datascience and just I have started learning it via udemy

You can remove the second hidden layer as simple Ann is enough for this and also we don’t have to use activator at the output layer as it is regression problem.
Please see the sample code https://github.com/naveenkambham/MachineLearningModels/blob/master/NeuralNetwork.py . This is similar to your requirement.

How to obtain the Tensorflow code version of a NN built in Keras?

I have been working with Keras for a week or so. I know that Keras can use either TensorFlow or Theano as a backend. In my case, I am using TensorFlow.
So I'm wondering: is there a way to write a NN in Keras, and then print out the equivalent version in TensorFlow?
MVE
For instance suppose I write
#create seq model
model = Sequential()
# add layers
model.add(Dense(100, input_dim = (10,), activation = 'relu'))
model.add(Dense(1, activation = 'linear'))
# compile model
model.compile(optimizer = 'adam', loss = 'mse')
# fit
model.fit(Xtrain, ytrain, epochs = 100, batch_size = 32)
# predict
ypred = model.predict(Xtest, batch_size = 32)
# evaluate
result = model.evaluate(Xtest)
This code might be wrong, since I just started, but I think you get the idea.
What I want to do is write down this code, run it (or not even, maybe!) and then have a function or something that will produce the TensorFlow code that Keras has written to do all these calculations.

First, let's clarify some of the language in the question. TensorFlow (and Theano) use computational graphs to perform tensor computations. So, when you ask if there is a way to "print out the equivalent version" in Tensorflow, or "produce TensorFlow code," what you're really asking is, how do you export a TensorFlow graph from a Keras model?
As the Keras author states in this thread,
When you are using the TensorFlow backend, your Keras code is actually building a TF graph. You can just grab this graph.
Keras only uses one graph and one session.
However, he links to a tutorial whose details are now outdated. But the basic concept has not changed.
We just need to:
Get the TensorFlow session
Export the computation graph from the TensorFlow session
Do it with Keras
The keras_to_tensorflow repository contains a short example of how to export a model from Keras for use in TensorFlow in an iPython notebook. This is basically using TensorFlow. It isn't a clearly-written example, but throwing it out there as a resource.
Do it with TensorFlow
It turns out we can actually get the TensorFlow session that Keras is using from TensorFlow itself, using the tf.contrib.keras.backend.get_session() function. It's pretty simple to do - just import and call. This returns the TensorFlow session.
Once you have the TensorFlow session variable, you can use the SavedModelBuilder to save your computational graph (guide + example to using SavedModelBuilder in the TensorFlow docs). If you're wondering how the SavedModelBuilder works and what it actually gives you, the SavedModelBuilder Readme in the Github repo is a good guide.
P.S. - If you are planning on heavy usage of TensorFlow + Keras in combination, have a look at the other modules available in tf.contrib.keras

So you want to use instead of WX+b a different function for your neurons. Well in tensorflow you explicitly calculate this product, so for example you do
y_ = tf.matmul(X, W)
you simply have to write your formula and let the network learn. It should not be difficult to implement.
In addition what you are trying to do (according to the paper you link) is called batch normalization and is relatively standard. The idea being you normalize your intermediate steps (in the different layers). Check for example https://www.google.ch/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0ahUKEwikh-HM7PnWAhXDXRQKHZJhD9EQFggyMAE&url=https%3A%2F%2Farxiv.org%2Fabs%2F1502.03167&usg=AOvVaw1nGzrGnhPhNGEczNwcn6WK or https://www.google.ch/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&ved=0ahUKEwikh-HM7PnWAhXDXRQKHZJhD9EQFghCMAM&url=https%3A%2F%2Fbcourses.berkeley.edu%2Ffiles%2F66022277%2Fdownload%3Fdownload_frd%3D1%26verifier%3DoaU8pqXDDwZ1zidoDBTgLzR8CPSkWe6MCBKUYan7&usg=AOvVaw0AHLwD_0pUr1BSsiiRoIFc
Hope that helps,
Umberto

Multi Column Deep Neural Network with TFLearn and Tensorflow

I am trying to build a multi column deep neural network (MDNN) with tflearn and tensorflow. The MDNN is explained in this paper. The part I am struggling with is how I can add two or more inputs together to be fed to tensorflow.
For a single column I have:
network = tflearn.input_data(shape=[None, image_shape, image_shape, 3])
and
model.fit(X_input, y_train, n_epoch=50, shuffle=True,
validation_set=(X_test_norm, y_test),
show_metric=True, batch_size=240, run_id='traffic_cnn2')
where X_input is of shape (31367, 32, 32, 3). I am pretty new to numpy, tensorflow and tflearn. The difficulty for now really lays in how to specify multiple inputs to tflearn.
Any help is greatly appreciated.

The MDNN explained in the paper individually trains several models using random (but bounded) distortions on the data. Once all models are trained, they produce predictions using an ensemble classifier by averaging the output of all the models on different versions of the data.
As far as I understand, the columns are not jointly but independently trained. So you must create different models and call fit on each on them. I recommend you start training a single model and once you have a training setting getting good results, replicate it. To generate predictions, you must compute the average of the predicted probabilities from the predict function and take the most probable class.
One way to a generate data from your inputs is to use data augmentation. However, instead of generating new data you must replace it by the modified versions.

Output of neurons in Multiple Layer Perceprton Classifier in scikit-learn

I am currently working on MLPClassifier of neural_network package of sklearn.
I trained the classifier and it is predicting/running fine. Now I need the output values of neurons(nodes) in each layers when it predicts class for a particular input after training, for visualisation purposes.
I read the api and there is an attribute - coefs_ which returns the weight matrix of the network but couldn't find any method or attribute which would return the output of neurons.
So being not mentioned in the documentation, I suppose it is not possible to get it directly. Is there any way/tweaking available to get these output of neurons at each layer OR alternatively any direct method of visualisation of the MLPClassifier.
Note - MLPClassifier is currently not available in stable version of scikit-learn and is 0.18 dev version only.
I am using Python 2.7 and scikit-learn 0.18 dev version.

Output of neurons only makes sense if you have inputs. It's not an intrinsic part of the model. You take an inner product of the inputs with the weights to get the "outputs".