This should be very simple Keras program. It works until the last line of code. But I have called the predict method. Granted I used the same input data as the training data, but that should not matter.
from keras.models import Sequential
import pandas as pd
url = 'https://raw.githubusercontent.com/werowe/logisticRegressionBestModel/master/KidCreative.csv'
data = pd.read_csv(url, delimiter=',')
labels=data['Buy']
features = data.iloc[:,2:16]
model = Sequential()
model.compile(optimizer='rmsprop' ,loss='binary_crossentropy',metrics=['accuracy'])
model.fit(data, labels, epochs=10, batch_size=32)
model.evaluate(labels, features, batch_size=128)
model.predict(labels)
model.summary()
But I get this error:
model.summary()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/walker/tf3/lib/python3.4/site-packages/keras/engine/network.py", line 1263, in summary
'This model has never been called, this its weights '
ValueError: This model has never been called, this its weights have not yet been created, so no summary can be displayed. Build the model first (e.g. by calling it on some test data).
The first step should be to actually build a model, which you don't do; i.e. after model = Sequential(), there should be some model.add statements, in order to build the model, before compiling it, fitting it, and use it for evaluation or getting its summary.
Instead of getting guidance from some repo of ambiguous quality, it's better to start from the official examples & tutorials - look for example at the Keras MNIST CNN example; replicating here only the model-related parts, we get:
model = Sequential()
# no model built, as in your case
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
This should already give an error:
TypeError: Sequential model cannot be built: model is empty. Add some layers first.
which I am surprised you don't report in your case.
Here is what we should do instead (again, refer to the link for the full details):
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
model.summary()
# works OK
As #desertnaut said, it is necessary to have layers in a model before calling mode.summary().
Edit: I want to add that, Even if your model has layers model.summary() may not work if it doesn't have "input_shape" in the first layer.
Because in Sequential model dimensions of input of each layer (thus the number of parameters) are calculated using output dimension of the preceding layer but to start this chain input dimension for the first layer is needed. So it is not possible to generate a summary of the model without the input dimension.
Note: If you don't explicitly specify input dimension than keras will figure it out when you call fit() for the first time. So if you don't specify input_shape, then you need to call model.fit() before the model.summary()
Thanks Matias Valdenegro, I added some input layers and changed the loss function. Now this works:
import tensorflow as tf
from keras.models import Sequential
import pandas as pd
from keras.layers import Dense
url = 'https://raw.githubusercontent.com/werowe/logisticRegressionBestModel/master/KidCreative.csv'
data = pd.read_csv(url, delimiter=',')
labels=data['Buy']
features = data.iloc[:,2:16]
model = Sequential()
model.add(Dense(units=64, activation='relu', input_dim=1))
model.add(Dense(units=14, activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer='sgd',
metrics=['accuracy'])
model.fit(labels, features,
batch_size=12,
epochs=10,
verbose=1,
validation_data=(labels, features))
model.evaluate(labels, features, verbose=0)
model.summary()
Related
I was wondering whether creating the model by passing activity_regularizer='l1_l2' as an argument to Conv2D()
model = keras.Sequential()
model.add(Conv2D(filters=16, kernel_size=(6, 6), strides=3, padding='valid', activation='relu',
activity_regularizer='l1_l2', input_shape=X_train[0].shape))
model.add(Dropout(0.2))
model.add(MaxPooling2D(pool_size=(3, 1), strides=3, padding='valid'))
model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(10, activation='softmax'))
model.compile(optimizer=Adam(learning_rate = 0.001), loss = 'sparse_categorical_crossentropy', metrics = ['accuracy'])
model.summary()
history = model.fit(X_train, y_train, epochs = 10, validation_data = (X_val, y_val), verbose=0)
will mathematically make a difference to creating the model by adding model.add(ActivityRegularization(l1=..., l2=...)) seperately?
model = keras.Sequential()
model.add(Conv2D(filters=16, kernel_size=(6, 6), strides=3, padding='valid', activation='relu',
input_shape=X_train[0].shape))
model.add(Dropout(0.2))
model.add(ActivityRegularization(l1=some_number, l2=some_number))
model.add(MaxPooling2D(pool_size=(3, 1), strides=3, padding='valid'))
model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(10, activation='softmax'))
model.compile(optimizer=Adam(learning_rate = 0.001), loss = 'sparse_categorical_crossentropy', metrics = ['accuracy'])
model.summary()
history = model.fit(X_train, y_train, epochs = 10, validation_data = (X_val, y_val), verbose=0)
For me, it is hard to tell, as training always involves some randomness. But the results seem similar.
One additional question I have is: I accidentally passed the activity_regularizer='l1_l2' argument to the MaxPooling2D() layer before, and the code ran. How can that be, considering that activity_regularizer is not given as a possible argument for MaxPooling2D() in tensorflow?
Technically, if you are not applying any other constraint on the layer output, applying the activity regularizer inside the layer as well as outside the convolution layer is same. However, applying it outside the convolution layer gives the user more flexibility. For instance, the user might want to regularize the output units after the skip connections are set up instead of after the convolution. It is just like to have an activation function inside the convolution layer or using keras.activations to use the activations after he convolution layer. Sometimes this is done after batch normalization.
For your second question, the MaxPool2D layer takes the activity regularizer constraint. Even though this is not mentioned in their documentation, it kind of makes sense intuitionally, since the user might want to regularize the outputs after max-pooling. You can check that activity_regularizer does not only work with the MaxPool2D layer but also with other layers such as the BatchNormalization layer for the same reason.
tf.keras.layers.Conv2D(64 , 2 , padding='same', activity_regularizer='l1_l2')
and this code,
tf.keras.layers.Conv2D(64 , 2 , padding='same')
tf.keras.layers.ActivityRegularization()
They both do the same job, actually doing inside or outside has the same impact. Moreover, Tensorflow on the backend makes a graph of it which will first apply the ConvLayer then it will apply the Activity-Regularization, in both cases, the computation shall be done in the same way with no difference...
I am studying deep learning and as an assignment, I am doing a classification project, which has 17k records with 14 features and a target variable that have 11 classes.
I tried to train a simple neural network
# define the keras model
model1 = keras.Sequential()
model1.add(keras.layers.Dense(64, input_dim=14, activation='relu'))
model1.add(keras.layers.Dense(128, activation='relu'))
model1.add(keras.layers.Dense(64, activation='relu'))
model1.add(keras.layers.Dense(1, activation='softmax'))
# compile the keras model
model1.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# fit the keras model on the dataset
performance1 = model1.fit(x_train, y_train, epochs=100, validation_split=0.2)
But the problem here is I am getting the same accuracy for each epoch, it doesn't seem that model is even learning.
I tried to research this problem and found some similar problems on StackOverflow like this question and tried following things
Applied StandardScaler
Increased/Decreased the hidden layer and neurons
Added dropout layer
Changed the optimizers, loss, and activation function
I also tried to batch_size
But none of them worked, of course, the accuracy was different in the different trials (but has the same problem).
Few of trials are as follows:
# define the keras model
model1 = keras.Sequential()
model1.add(keras.layers.Dense(64, input_dim=14, activation='sigmoid'))
model1.add(keras.layers.Dense(128, activation='sigmoid'))
model1.add(keras.layers.Dense(64, activation='sigmoid'))
model1.add(keras.layers.Dense(1, activation='softmax'))
sgd = keras.optimizers.SGD(lr=0.01)
# compile the keras model
model1.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
# define the keras model
model1 = keras.Sequential()
model1.add(keras.layers.Dense(64, input_dim=14, activation='relu'))
model1.add(keras.layers.Dense(128, activation='relu'))
model1.add(keras.layers.Dropout(0.2))
model1.add(keras.layers.Dense(64, activation='relu'))
model1.add(keras.layers.Dropout(0.2))
model1.add(keras.layers.Dense(1, activation='softmax'))
# compile the keras model
model1.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
I don't know what's the problem here. Please let me know if you require more details to process this. And please don't close this question I know this question stands a chance to marked as a duplicate question but trust me I tried many things which I can understand as a beginner.
The problem is that the softmax should be applied on an output array to get probabilities and that output array from the model should represent the logits for each target class. hence you would have to change this line
model1.add(keras.layers.Dense(1, activation='softmax'))
# TO
model1.add(keras.layers.Dense(df['Class'].nunique(), activation='softmax'))
EDIT:
# Let's say you have 11 unique values in your class then you last layer will become
model1.add(keras.layers.Dense(11, activation='softmax'))
# Now your loss will be
model1.compile(loss=tf.keras.loss.SparseCategoricalCrossentropy(), optimizer='adam', metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
I have been working on a Machine Learning project in python. After getting a basic neural net running well, I am trying to set up a grid search to optimize the parameters using the GridSearchCV function from sklearn. The grid.fit(X,Y) function is throwing this error: TypeError: only size-1 arrays can be converted to Python scalars. My interpretation is that the fit function doesn't like the format of the X and Y I gave it. This is confusing to me because the net was running fine without the grid search and I didn't mess with the network or the data at all. Can anybody explain what is happening here and how I could fix it?
This code creates the network and the grid search:
#Creating the neural network
def create_model():
model=Sequential()
model.add(Dense(512, activation='relu',input_shape=(2606,)))
model.add(Dense(256, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(1, activation='relu'))
opt=optimizers.Adam(lr=learn_rate)
model.compile(optimizer=opt, loss='mean_squared_error', metrics=['accuracy'])
#I commented this out because I believe it is delegated to the grid.fit() fn later on.
#model.fit(X_train, Y_train, batch_size=30, epochs=6000, verbose=1)
return model
#Now setting up the grid search
model=KerasClassifier(build_fn=create_model())
learn_rate=np.arange(.00001,.001,.00002).tolist()
batch_size=np.arange(10,2606,2).tolist()
epochs=np.arange(1000,10000,100).tolist()
param_grid=dict(learn_rate=learn_rate, batch_size=batch_size, epochs=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3)
grid_results=grid.fit(X_train,Y_train) #This is the line referenced in the error message.
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
Any advice would be greatly appreciated!
Edit:
The X_train data has shape (167,2606). Each of the 167 elements are an array of length 2606. This is why the input_shape for the network is (2606,). The Y_train has shape (167,).
So, the issue is that GridSearchCV creates a new model with the new parameters for every combination thereof. You are passing an already created model, and a list of parameters. I believe that is the source of the array vs scalar error. Below, I've altered your code (with some garbage sample data) that will run.
The primary changes to take note of are that I altered the signature of your create_model to accept the parameters values you pass into the GridSearch. I also removed your assignment of the KerasClassifier instance to the variable model and instead put that call as the estimator in GridSearchCV.
import numpy as np
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras import optimizers
from sklearn.model_selection import GridSearchCV
#Creating the neural network
def create_model(learn_rate, batch_size, epochs):
model=Sequential()
model.add(Dense(512, activation='relu',input_shape=(2606,)))
model.add(Dense(256, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(1, activation='relu'))
opt=optimizers.Adam(lr=learn_rate)
model.compile(optimizer=opt, loss='mean_squared_error', metrics=['accuracy'])
#I commented this out because I believe it is delegated to the grid.fit() fn later on.
#model.fit(X_train, Y_train, batch_size=30, epochs=6000, verbose=1)
return model
#Now setting up the grid search
X_train = np.empty((167,2606), dtype=float, order='C')
Y_train = np.empty((167,), dtype=float, order='C')
learn_rate=np.arange(.00001,.001,.00002).tolist()
batch_size=np.arange(10,2606,2).tolist()
epochs=np.arange(1000,10000,100).tolist()
param_grid=dict(learn_rate=learn_rate, batch_size=batch_size, epochs=epochs)
grid = GridSearchCV(estimator=KerasClassifier(build_fn=create_model),
param_grid=param_grid, n_jobs=-1, cv=3)
grid_results=grid.fit(X_train,Y_train) #This is the line referenced in the error message.
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
How do I train 1 model multiple times and combine them at the output layer?
For example:
model_one = Sequential() #model 1
model_one.add(Convolution2D(32, 3, 3, activation='relu', input_shape=(1,28,28)))
model_one.add(Flatten())
model_one.add(Dense(128, activation='relu'))
model_two = Sequential() #model 2
model_two.add(Dense(128, activation='relu', input_shape=(784)))
model_two.add(Dense(128, activation='relu'))
model_???.add(Dense(10, activation='softmax')) #combine them here
model.compile(loss='categorical_crossentropy', #continu together
optimizer='adam',
metrics=['accuracy'])
model.fit(X_train, Y_train, #continu together somehow, even though this would never work because X_train and Y_train have wrong formats
batch_size=32, nb_epoch=10, verbose=1)
I've heard I can do this through a graph model but I can't find any documentation on it.
EDIT: in reply to the suggestion below:
A1 = Conv2D(20,kernel_size=(5,5),activation='relu',input_shape=( 28, 28, 1))
---> B1 = MaxPooling2D(pool_size=(2,2))(A1)
throws this error:
AttributeError: 'Conv2D' object has no attribute 'get_shape'
Graph notation would do it for you. Essentially you give every layer a unique handle then link back to the previous layer using the handle in brackets at the end:
layer_handle = Layer(params)(prev_layer_handle)
Note that the first layer must be an Input(shape=(x,y)) with no prior connection.
Then when you make your model you need to tell it that it expects multiple inputs with a list:
model = Model(inputs=[in_layer1, in_layer2, ..], outputs=[out_layer1, out_layer2, ..])
Finally when you train it you also need to provide a list of input and output data that corresponds with your definition:
model.fit([x_train1, x_train2, ..], [y_train1, y_train2, ..])
Meanwhile everything else is the same so you just need to combine together the above to give you the network layout that you want:
from keras.models import Model
from keras.layers import Input, Convolution2D, Flatten, Dense, Concatenate
# Note Keras 2.02, channel last dimension ordering
# Model 1
in1 = Input(shape=(28,28,1))
model_one_conv_1 = Convolution2D(32, (3, 3), activation='relu')(in1)
model_one_flat_1 = Flatten()(model_one_conv_1)
model_one_dense_1 = Dense(128, activation='relu')(model_one_flat_1)
# Model 2
in2 = Input(shape=(784, ))
model_two_dense_1 = Dense(128, activation='relu')(in2)
model_two_dense_2 = Dense(128, activation='relu')(model_two_dense_1)
# Model Final
model_final_concat = Concatenate(axis=-1)([model_one_dense_1, model_two_dense_2])
model_final_dense_1 = Dense(10, activation='softmax')(model_final_concat)
model = Model(inputs=[in1, in2], outputs=model_final_dense_1)
model.compile(loss='categorical_crossentropy', #continu together
optimizer='adam',
metrics=['accuracy'])
model.fit([X_train_one, X_train_two], Y_train,
batch_size=32, nb_epoch=10, verbose=1)
Documentation can be found in the Functional Model API. I'd recommend reading around other questions or checking out Keras' repo as well since the documentation currently doesn't have many examples.
I originally tried the same approach and ran into the same error as this SO questioner. However, using the accepted (and only) answer there gave me another error: "input_dim is not a legal parameter."
I then tried to use the solution ("add an input_dim keyarg to the KerasClassifier constructor") on the original question, and got the same error again. Am I doing something wrong, or is there a new way to pass a first layer's input_dim through the sklearn wrapper KerasClassifier now?
Minimal code example below:
from keras.models import Sequential
from keras.layers import Dense
from sklearn import datasets
from keras.wrappers.scikit_learn import KerasClassifier
import numpy as np
def create_model():
# create model
model = Sequential()
model.add(Dense(12, input_dim=4, init='uniform', activation='relu'))
model.add(Dense(6, init='uniform', activation='relu'))
model.add(Dense(1, init='uniform', activation='sigmoid'))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
return model
#Error thrown here:
model = KerasClassifier(build_fn=create_model, input_dim=5, nb_epoch=150, batch_size=10, verbose=0)
ValueError: input_dim is not a legal parameter
You need to pass the input_dim as one of the parameters to create_model()
def create_model(input_dim):
# create model
model = Sequential()
# model.add(Dense(12, input_dim=4, init='uniform', activation='relu'))
model.add(Dense(12, input_dim=input_dim, init='uniform', activation='relu'))
Though you need not use input_dim inside create_model for the sake of just removing the error.