I am running the test script from the Keras website for Multilayer Perceptron (MLP) for multi-class softmax classification. Running in the jupyter notebook I get the error "name 'keras' is not defined". This may be a simple python syntax problem that I am not keen to, however this code comes straight from keras so I expect it should work as is. I have run other neural nets using keras, so I am pretty sure that I have installed everything (installed keras using anaconda). Can anyone help? I have included both the code and the error at the bottom. Thanks!
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.optimizers import SGD
# Generate dummy data
import numpy as np
x_train = np.random.random((1000, 20))
y_train = keras.utils.to_categorical(np.random.randint(10, size=(1000, 1)), num_classes=10)
x_test = np.random.random((100, 20))
y_test = keras.utils.to_categorical(np.random.randint(10, size=(100, 1)), num_classes=10)
model = Sequential()
# Dense(64) is a fully-connected layer with 64 hidden units.
# in the first layer, you must specify the expected input data shape:
# here, 20-dimensional vectors.
model.add(Dense(64, activation='relu', input_dim=20))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
optimizer=sgd,
metrics=['accuracy'])
model.fit(x_train, y_train,
epochs=20,
batch_size=128)
score = model.evaluate(x_test, y_test, batch_size=128)
This is the error message:
NameError Traceback (most recent call last)
<ipython-input-1-6d8174e3cf2a> in <module>()
6 import numpy as np
7 x_train = np.random.random((1000, 20))
----> 8 y_train = keras.utils.to_categorical(np.random.randint(10, size=(1000, 1)), num_classes=10)
9 x_test = np.random.random((100, 20))
10 y_test = keras.utils.to_categorical(np.random.randint(10, size=(100, 1)), num_classes=10)
NameError: name 'keras' is not defined
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.optimizers import SGD
From above, you only imported following submodules in keras
keras.models
keras.layers
keras.optimizers
But this does not automatically import the outer module like keras
or other submodules keras.utils
So, you can do either one
import keras
import keras.utils
from keras import utils as np_utils
but from keras import utils as np_utils is the most widely used.
Especially import keras is not a good practice because importing the higher module does not necessarily import its submodules (though it works in Keras)
For example,
import urllib
does not necessarily import urllib.request because if there are so many big submodules, it's inefficient to import all of its submodules every time.
EDIT:
With the introduction of Tensorflow 2, keras submodules such as keras.utils should now be imported as
from tensorflow.keras import utils as np_utils
General way:
from keras.utils import to_categorical
Y_train = to_categorical(y_train, num_classes)
Concrete way:
from keras.utils import to_categorical
print(to_categorical(1, 2))
print(to_categorical(0, 2))
Will output
[0. 1.]
[1. 0.]
Although this is an old question but yet updating the latest approach to access to_categorical function.
This function has now been packed in np_utils.
The correct way to access it is:
from keras.utils.np_utils import to_categorical
This works for me:
import tensorflow as tf
from keras import utils as np_utils
y_train = tf.keras.utils.to_categorical(y_train, num_classes)
y_test = tf.keras.utils.to_categorical(y_test, num_classes)
Related
I am relatively new to the neural network, so I was trying to use it for unsupervised clustering. My data is in dataframe with 5 different columns (features), I wanted to get like 4 classes from this, see the full model below
from sklearn import preprocessing as pp
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import log_loss
from sklearn.metrics import precision_recall_curve, average_precision_score
from sklearn.metrics import roc_curve, auc, roc_auc_score
import keras
from keras import backend as K
from keras.models import Sequential, Model
from keras.layers import Activation, Dense, Dropout , Flatten
from keras.layers import BatchNormalization, Input, Lambda
from keras import regularizers
from keras.losses import mse, categorical_crossentropy
model = Sequential()
model.add(Dense(32, activation='relu',input_shape=[5]))
model.add(Flatten())
model.add(Dense(units=32, activation='relu'))
model.add(Dense(units=16, activation='relu'))
model.add(Dense(units=4, activation='relu'))
model.add(Dense(4, activation = "softmax"))
model.compile(optimizer='adam',loss="categorical_crossentropy",metrics=['accuracy'])
when I give the option of generating 4 classes I get the error message:
ValueError: Shapes (None, 5) and (None, 4) are incompatible
I don't know what I am doing wrong. I have tried to use a different loss function, same error.
i get the error when i input my data,
out_class = model.fit(x=pd_pca_std,
y=pd_pca_std,
epochs=num_epochs,
batch_size=batch_size,
shuffle=True,
validation_data=(pd_pca_std, pd_pca_std),
verbose=1)
the values are
batch_size = 33
epochs = 20
num_classes = 4
input_shape = (990000, 5)
output_shape = (990000, 4)
I would suggest using 5 classes or something relative to the 5 classes instead. I'll explain.
So in Neural Networks and machine learning in general, there are certain matrix operations that happen in the background in TensorFlow. So say I create the following:
import numpy as np
x = np.random.random((3, 4))
y = np.random.random((3, 3))
np.dot(x, y) # if I try multiplying 2 incompatible matrices, the program will fail :(
So what's happening here is that the matrices are incompatible for simple matrix arithmetic, because they need to be certain shapes for them to be compatible. So what I recommend doing is either changing the shapes of the matrices/arrays in question, or play with different shapes in the program to see which will succeed...
You could also learn some linear algebra which has the rules for matrix manipulation and arithmetic, but I won't go into that right now. However, what I will do is leave a link for you to check out regarding this subject so you know what to do in the future...
Here it is:
https://www.mathlynx.com/online/LinAlg_Matrices_rules
Hopefully this helps...
Have a nice day :)
TL;DR
Your output units should match the number of classes you are testing on.
This is the skeleton of how I replicated you problem and had it working
from sklearn import preprocessing as pp
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import log_loss
from sklearn.metrics import precision_recall_curve, average_precision_score
from sklearn.metrics import roc_curve, auc, roc_auc_score
import numpy as np
import keras
from keras import backend as K
from keras.models import Sequential, Model
from keras.layers import Activation, Dense, Dropout , Flatten
from keras.layers import BatchNormalization, Input, Lambda
from keras import regularizers
from keras.losses import mse, categorical_crossentropy
X = '''input data here as an array''' # I used X = np.zeros((990000, 5))
y = '''output data here as an array'''#I used y = np.ones((990000, 4))
batch_size = 33
num_epochs = 20
num_classes = 4
model = Sequential()
model.add(Dense(32, activation='relu',input_shape=X.shape[1:])) #Input shape = 5
model.add(Flatten())
model.add(Dense(units=32, activation='relu'))
model.add(Dense(units=16, activation='relu'))
model.add(Dense(units=4, activation='relu'))
model.add(Dense(y.shape[1], activation = "softmax")) #Output = y.shape[1] = 4
model.compile(optimizer='adam',loss="categorical_crossentropy",metrics=['accuracy'])
model.summary() #Will show you a summary of the model
model.fit(x=X, y=y,epochs=num_epochs, batch_size=batch_size, shuffle=True,validation_data=(X,y),verbose=1) #You may want to use different variables in your validation.
I've been trying to create a model that recognizes different singing techniques. I have got good results but I want to do different tests with different optimizers, layers, etc. However, I can't get reproducible results. By running twice this model training:
num_epochs = 100
batch_size = 128
history = modelo.fit(X_train_f, Y_train, validation_data=(X_test_f,Y_test), epochs=num_epochs, batch_size=batch_size, verbose=2)
I can get 25% accuracy the first run and then 34% the second. Then if I change the optimizer from "sgd" to "adam", I would get a 99%. If I come back to the previous "sgd" optimizer that got me 34% the second run, I would get 100% or something crazy like that. I don't understand why.
I've tried many things I've read in similar questions. The following lines show how I am trying to make my code to be reproducible, and these are actually the first lines of my whole code:
import numpy as np
import tensorflow as tf
import random as rn
import os
#https://stackoverflow.com/questions/57305909/tensorflow-keras-reproducibility-problem-on-google-colab
os.environ['PYTHONHASHSEED']=str(5)
np.random.seed(5)
rn.seed(12345)
session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,
inter_op_parallelism_threads=1)
tf.compat.v1.set_random_seed(1234)
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
tf.compat.v1.keras.backend.set_session(sess)
Question is, what am I doing wrong with the code above that is not working (as I mentioned)?
Here's where I create the training sets:
from keras.datasets import mnist
from keras.utils import np_utils
from keras.models import Sequential
from keras.layers.convolutional import Conv1D, MaxPooling1D
from keras.layers.core import Dense, Flatten
from keras.layers import BatchNormalization,Activation
from keras.optimizers import SGD, Adam
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size=0.2, random_state=2)
My model:
from tensorflow.keras import layers
from tensorflow.keras import initializers
input_dim = X_train_f.shape[1]
output_dim = Y_train.shape[1]
modelo = Sequential()
modelo.add(Conv1D(filters=6, kernel_initializer=initializers.glorot_uniform(seed=5), kernel_size=5, activation='relu', input_shape=(40, 1))) # 6
modelo.add(MaxPooling1D(pool_size=2))
modelo.add(Conv1D(filters=16, kernel_initializer=initializers.glorot_uniform(seed=5), kernel_size=5, activation='relu')) # 16
modelo.add(MaxPooling1D(pool_size=2))
modelo.add(Flatten())
modelo.add(Dense(120, kernel_initializer=initializers.glorot_uniform(seed=5), activation='relu')) # 120
modelo.add(Dense(84, kernel_initializer=initializers.glorot_uniform(seed=5), activation='relu')) # 84
modelo.add(Dense(nclases, kernel_initializer=initializers.glorot_uniform(seed=5), activation='softmax'))
sgd = SGD(lr=0.1)
#modelo.compile(loss='categorical_crossentropy',
# optimizer='adam',
# metrics=['accuracy'])
modelo.compile(loss='categorical_crossentropy',
optimizer=sgd,
metrics=['accuracy'])
modelo.summary()
modelo.input_shape
It is a normal situation. Adam optimizer is much more powerful comparing to SGD. Adam implicitly performs coordinate-wise gradient clipping and can hence, unlike SGD, tackle heavy-tailed noise.
firstly im quiet new with this library. My code is simple there is input x and there is y(x*2). it should learsn simple int*2 but it cant. I think maybe paramaters are wrong but how can i determine true parameters?
from tensorflow import keras
import numpy as np
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
import keras
from keras.layers import Input, Dense
import numpy as np
import pandas as pd
xs = np.array([[1,3,4,5] , [9,2,3,4]]).reshape(1,2,4)
ys = np.array([[2,6,8,10] , [18,4,6,8]]).reshape(1,2,4)
# model = tf.keras.Sequential([layers.Dense(units=1, input_shape=[2,4])])
model = Sequential()
model.add(Dense(8,input_shape=[2,4]))
model.add(Activation('relu'))
model.add(Dense(6))
model.add(Activation('relu'))
model.add(Dense(4))
model.add(Activation('softmax'))
model.compile(optimizer='Adadelta', loss='mean_squared_error')
model.fit(xs, ys, epochs=5, batch_size=1)
p = np.array([[1,3,4,5] , [9,2,3,4]]).reshape(1,2,4)
print(model.predict(p))
My second try is more hard than *2 but i created 100 piece input and output value but still bad performance, how can i make it MORE accurate?
from tensorflow import keras
import numpy as np
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
import keras
from keras.layers import Input, Dense
import numpy as np
import pandas as pd
xs = np.array([[1,3,4,5,9,2,3,4]]).reshape(1,1,8)
xg=np.ones((100,8)).reshape(100,8)
yg=np.ones((100,8)).reshape(100,8)
for i in range(100):
xg[i-1]=xs*np.random.randint(500)
yg[i-1]=xg[i-1]*np.sin(20)
xs=xg
ys=yg
# model = tf.keras.Sequential([layers.Dense(units=1, input_shape=[2,4])])
model = Sequential()
model.add(Dense(8,input_shape=[100,8]))
model.add(Activation('relu'))
model.add(Dense(8))
model.add(Activation('relu'))
model.add(Dense(8))
# opt = keras.optimizers.sgd(learning_rate=0.2)
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(xs, ys, epochs=8000, batch_size=100)
model.summary()
p = np.array([[1,340,4,512,9,2,3,4]])
print(model.predict(p))
print(np.sin(p))
Your problem come from 2 places. First, you accidentally use softmax as output activation function, you can solve that by just comment it out (this is your root problem). Softmax function use for classification problem but your problem is regression problem there is no need for activation in the last layer. Second(after taking softmax out), it come from low number of epochs, you need to train the model for larger number of epochs for better performance.
I want to build a non linear regression model using keras to predict a +ve continuous variable.
For the below model how do I select the following hyperparameters?
Number of Hidden layers and Neurons
Dropout ratio
Use BatchNormalization or not
Activation function out of linear, relu, tanh, sigmoid
Best optimizer to use among adam, rmsprog, sgd
Code
def dnn_reg():
model = Sequential()
#layer 1
model.add(Dense(40, input_dim=13, kernel_initializer='normal'))
model.add(Activation('tanh'))
model.add(Dropout(0.2))
#layer 2
model.add(Dense(30, kernel_initializer='normal'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.4))
#layer 3
model.add(Dense(5, kernel_initializer='normal'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.4))
model.add(Dense(1, kernel_initializer='normal'))
model.add(Activation('relu'))
# Compile model
model.compile(loss='mean_squared_error', optimizer='adam')
return model
I have considered random gridsearch but instead want to use hyperopt which I believe will be faster. I initially implemented the tuning using https://github.com/maxpumperla/hyperas. Hyperas is not working with latest version of keras. I suspect that keras is evolving fast and it's difficult for the maintainer to make it compatible. So I think using hyperopt directly will be a better option.
PS: I am new to bayesian optimization for hyper parameter tuning and hyperopt.
I've had a lot of success with Hyperas. The following are the things I've learned to make it work.
1) Run it as a python script from the terminal (not from an Ipython notebook)
2) Make sure that you do not have any comments in your code (Hyperas doesn't like comments!)
3) Encapsulate your data and model in a function as described in the hyperas readme.
Below is an example of a Hyperas script that worked for me (following the instructions above).
from __future__ import print_function
from hyperopt import Trials, STATUS_OK, tpe
from keras.datasets import mnist
from keras.layers.core import Dense, Dropout, Activation
from keras.models import Sequential
from keras.utils import np_utils
import numpy as np
from hyperas import optim
from keras.models import model_from_json
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.convolutional import Convolution2D, MaxPooling2D
from keras.optimizers import SGD , Adam
import tensorflow as tf
from hyperas.distributions import choice, uniform, conditional
__author__ = 'JOnathan Hilgart'
def data():
"""
Data providing function:
This function is separated from model() so that hyperopt
won't reload data for each evaluation run.
"""
import numpy as np
x = np.load('training_x.npy')
y = np.load('training_y.npy')
x_train = x[:15000,:]
y_train = y[:15000,:]
x_test = x[15000:,:]
y_test = y[15000:,:]
return x_train, y_train, x_test, y_test
def model(x_train, y_train, x_test, y_test):
"""
Model providing function:
Create Keras model with double curly brackets dropped-in as needed.
Return value has to be a valid python dictionary with two customary keys:
- loss: Specify a numeric evaluation metric to be minimized
- status: Just use STATUS_OK and see hyperopt documentation if not feasible
The last one is optional, though recommended, namely:
- model: specify the model just created so that we can later use it again.
"""
model_mlp = Sequential()
model_mlp.add(Dense({{choice([32, 64,126, 256, 512, 1024])}},
activation='relu', input_shape= (2,)))
model_mlp.add(Dropout({{uniform(0, .5)}}))
model_mlp.add(Dense({{choice([32, 64, 126, 256, 512, 1024])}}))
model_mlp.add(Activation({{choice(['relu', 'sigmoid'])}}))
model_mlp.add(Dropout({{uniform(0, .5)}}))
model_mlp.add(Dense({{choice([32, 64, 126, 256, 512, 1024])}}))
model_mlp.add(Activation({{choice(['relu', 'sigmoid'])}}))
model_mlp.add(Dropout({{uniform(0, .5)}}))
model_mlp.add(Dense({{choice([32, 64, 126, 256, 512, 1024])}}))
model_mlp.add(Activation({{choice(['relu', 'sigmoid'])}}))
model_mlp.add(Dropout({{uniform(0, .5)}}))
model_mlp.add(Dense(9))
model_mlp.add(Activation({{choice(['softmax','linear'])}}))
model_mlp.compile(loss={{choice(['categorical_crossentropy','mse'])}}, metrics=['accuracy'],
optimizer={{choice(['rmsprop', 'adam', 'sgd'])}})
model_mlp.fit(x_train, y_train,
batch_size={{choice([16, 32, 64, 128])}},
epochs=50,
verbose=2,
validation_data=(x_test, y_test))
score, acc = model_mlp.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', acc)
return {'loss': -acc, 'status': STATUS_OK, 'model': model_mlp}
enter code here
if __name__ == '__main__':
import gc; gc.collect()
with K.get_session(): ## TF session
best_run, best_model = optim.minimize(model=model,
data=data,
algo=tpe.suggest,
max_evals=2,
trials=Trials())
X_train, Y_train, X_test, Y_test = data()
print("Evalutation of best performing model:")
print(best_model.evaluate(X_test, Y_test))
print("Best performing model chosen hyper-parameters:")
print(best_run)
it induced by different gc sequence, if python collect session first , the program will exit successfully, if python collect swig memory(tf_session) first, the program exit with failure.
you can force python to del session by:
del session
or if you are using keras, you cant get the session instance, you can run following code at end of your code:
import gc; gc.collect()
This can be also another approach:
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from sklearn.metrics import roc_auc_score
import sys
X = []
y = []
X_val = []
y_val = []
space = {'choice': hp.choice('num_layers',
[ {'layers':'two', },
{'layers':'three',
'units3': hp.uniform('units3', 64,1024),
'dropout3': hp.uniform('dropout3', .25,.75)}
]),
'units1': hp.uniform('units1', 64,1024),
'units2': hp.uniform('units2', 64,1024),
'dropout1': hp.uniform('dropout1', .25,.75),
'dropout2': hp.uniform('dropout2', .25,.75),
'batch_size' : hp.uniform('batch_size', 28,128),
'nb_epochs' : 100,
'optimizer': hp.choice('optimizer',['adadelta','adam','rmsprop']),
'activation': 'relu'
}
def f_nn(params):
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import Adadelta, Adam, rmsprop
print ('Params testing: ', params)
model = Sequential()
model.add(Dense(output_dim=params['units1'], input_dim = X.shape[1]))
model.add(Activation(params['activation']))
model.add(Dropout(params['dropout1']))
model.add(Dense(output_dim=params['units2'], init = "glorot_uniform"))
model.add(Activation(params['activation']))
model.add(Dropout(params['dropout2']))
if params['choice']['layers']== 'three':
model.add(Dense(output_dim=params['choice']['units3'], init = "glorot_uniform"))
model.add(Activation(params['activation']))
model.add(Dropout(params['choice']['dropout3']))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy', optimizer=params['optimizer'])
model.fit(X, y, nb_epoch=params['nb_epochs'], batch_size=params['batch_size'], verbose = 0)
pred_auc =model.predict_proba(X_val, batch_size = 128, verbose = 0)
acc = roc_auc_score(y_val, pred_auc)
print('AUC:', acc)
sys.stdout.flush()
return {'loss': -acc, 'status': STATUS_OK}
trials = Trials()
best = fmin(f_nn, space, algo=tpe.suggest, max_evals=50, trials=trials)
print('best: ', best)
Source
I'm using a combination of sklearn and Keras running with Theano as its back-end. I'm using the following code-
import numpy as np
import pandas as pd
from pandas import Series, DataFrame
import keras
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.constraints import maxnorm
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import SGD
from keras.wrappers.scikit_learn import KerasClassifier
from keras.constraints import maxnorm
from keras.utils.np_utils import to_categorical
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from datetime import datetime
import time
from datetime import timedelta
from __future__ import division
seed = 7
np.random.seed(seed)
Y = data['Genre']
del data['Genre']
X = data
encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)
X = X.as_matrix().astype("float")
calls=[EarlyStopping(monitor='acc', patience=10), ModelCheckpoint('C:/Users/1383921/Documents/NNs/model', monitor='acc', save_best_only=True, mode='auto', period=1)]
def create_baseline():
# create model
model = Sequential()
model.add(Dense(18, input_dim=9, init='normal', activation='relu'))
model.add(Dense(9, init='normal', activation='relu'))
model.add(Dense(12, init='normal', activation='softmax'))
# Compile model
sgd = SGD(lr=0.01, momentum=0.8, decay=0.0, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
return model
np.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasClassifier(build_fn=create_baseline, nb_epoch=300, batch_size=16, verbose=2)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
results = cross_val_score(pipeline, X, encoded_Y, cv=kfold, fit_params={'mlp__callbacks':calls})
print("Baseline: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
The result when I start running this last part is-
Epoch 1/10
...
Epoch 2/10
etc.
It's supposed to be Epoch 1/300 and it works just fine when I run it on a different notebook.
What do you guys think is happening? np_epoch=300...
What Keras version is this? If its greater than 2.0, then nb_epoch was changed to just epochs. Else it defaults to 10.
In Keras 2.0 the nb_epoch parameter was renamed to epochs so when you set epochs=300 it runs 300 epochs. If you use nb_epoch=300 it will default to 10 instead.
Another solution to your problem: Forget about nb_epoch (doesn't work). Pass epochs inside fit_params:
results = cross_val_score(pipeline, X, encoded_Y, cv=kfold,
fit_params={'epochs':300,'mlp__callbacks':calls})
And that would work. fit_params goes straight into the Fit method and it will get the right epochs.
The parameter name in your function should be epochs instead of nb_epochs.
Be very careful though. For example, I trained my ANN with the old fashioned way of declaring the parameters (nb_epochs = number), and it worked (the iPython console only showed me some warnings), but when I plugged the same parameter names in the cross_val_score function, it did not work.
I think that what sklearn calls "Epoch" is one step of your crossvalidation. So it does 300 epochs of training 10 times :-) is that possible? Try with verbose=1