How to test a RASA model? - python

I'm trying to write my own chatbot with the RASA framework.
Right now I'm just playing around with it and I have the following piece of code for training purposes.
from rasa.nlu.training_data import load_data
from rasa.nlu.config import RasaNLUModelConfig
from rasa.nlu.model import Trainer
from rasa.nlu import config
training_data = load_data("./data/nlu.md")
trainer = Trainer(config.load("config.yml"))
interpreter = trainer.train(training_data)
model_directory = trainer.persist("./models/nlu",fixed_model_name="current")
Now, I read that if I wanted to test it I should do something like this.
from rasa.nlu.evaluate import run_evaluation
run_evaluation("nlu.md", model_directory)
But this code is not available anymore in rasa.nlu.evaluate nor in rasa.nlu.test!
What's the way, then, of testing a RASA model?

The module was renamed.
Please import
from rasa.nlu.test import run_evaluation
Alternatively you now also do
from rasa.nlu import test
test_result = test(path_to_test_data, unpacked_model)
intent_evaluation_report = test_result["intent_evaluation"]["report"]
print(intent_evaluation_report)

Related

tqdm.notebook progress bar is not running

I am running a model using tqdm.notebook to check the progress using python3.8. However, the progress bar is not running though the generation works okay.
It just shows this on and on.
Here is my following code, and the model I'm running.
import numpy as np
import tensorflow as tf
from midi_ddsp.utils.midi_synthesis_utils import synthesize_mono_midi, conditioning_df_to_audio
from midi_ddsp.utils.inference_utils import get_process_group
from midi_ddsp.midi_ddsp_synthesize import load_pretrained_model
from midi_ddsp.data_handling.instrument_name_utils import INST_NAME_TO_ID_DICT
from tqdm.notebook import tqdm
# -----MIDI Synthesis-----
midi_file = '/Users/midi-ddsp/midi_example/ode_to_joy.mid'
# Load pre-trained model
synthesis_generator, expression_generator = load_pretrained_model()
# Synthesize with violin:
instrument_name = 'violin'
instrument_id = INST_NAME_TO_ID_DICT[instrument_name]
# Run model prediction
midi_audio, midi_control_params, midi_synth_params, conditioning_df = synthesize_mono_midi(synthesis_generator,
expression_generator,
midi_file, instrument_id,
output_dir=None)
synthesized_audio = midi_audio # The synthesized audio
conditioning_df_changed = conditioning_df.copy()
idk what's the problem. Hope someone can tell me. I appreciate it!

pm4py Error: cannot import name 'factory' from 'pm4py.algo.discovery.alpha'

I am trying to run the following code:
from pm4py.algo.discovery.alpha import factorial as alpha_miner
from pm4py.objects.log.importer.xes import factory as xes_importer
event_log = xes_importer.import_log(os.path.join("tests","input_data","running-example.xes"))
net, initial_marking, final_marking = alpha_miner.apply(event_log)
gviz = pn_vis_factory.apply(net, initial_marking, final_marking)
pn_vis_factory.view(gviz)
However, when I run the alpha miner, I get an error message that factory cannot be imported.
What could be the reason or does anyone know a soulution for this?
Many thanks for the answer
from pm4py.algo.discovery.alpha import algorithm as alpha_miner
Find all process discoveries and its information at:
https://pm4py.fit.fraunhofer.de/documentation#discovery
Try this:
import os
# Alpha Miner
from pm4py.algo.discovery.alpha import algorithm as alpha_miner
# XES Reader
from pm4py.objects.log.importer.xes import importer as xes_importer
# Visualize
from pm4py.visualization.petri_net import visualizer as pn_visualizer
log = xes_importer.apply(os.path.join("tests","input_data","running-example.xes"))
net, initial_marking, final_marking = alpha_miner.apply(log)
gviz = pn_visualizer.apply(net, initial_marking, final_marking)
pn_visualizer.view(gviz)

How to load an experiment in azureml?

I have many experiment, like:
and now, i want load an experiment
#%% sumonando os pacotes e verificando azureml.core
import azureml.core
import pandas as pd
import numpy as np
import logging
print("AzureML SDK Version: ", azureml.core.VERSION)
#%% Conectando ao azure e crinado o exparimento
from azureml.core import Workspace, Experiment
ws = Workspace.from_config()
print(Experiment.list(ws))
#%%
Experiment = Experiment.from_directory('teste2-Monitor-Runs') `
but
"error": {
"message": "No cache found for current project, try providing resource group and workspace
arguments"
}`
Content: azureml.core.Experiment class - Azure Machine Learning Python
Following AzureML's Experiment class documentation, they define an Experiment as a container of trials or runs. If you wish to access the container or the Experiment as a whole, you can use just access it by name (as long as you have the right workspace configured). See snippet below:
from azureml.core import Experiment, Workspace
ws = Workspace.from_config()
my_exp = Experiment(ws, name="teste2-Monitor-Runs")
I believe it is that way.
from azureml.core import Experiment, Workspace
Experiment = ws.experiments["teste2-Monitor-Runs"]

Couldn't create model.tar.gz file while training scikit learn model in AWS Sagemaker

I want to create an endpoint for scikit logistic regression in AWS Sagemaker. I have a train.py file which contains training code for scikit sagemaker.
import subprocess as sb
import pandas as pd
import numpy as np
import pickle,json
import sys
def install(package):
sb.call([sys.executable, "-m", "pip", "install", package])
install('s3fs')
import argparse
import os
if __name__ =='__main__':
parser = argparse.ArgumentParser()
# hyperparameters sent by the client are passed as command-line arguments to the script.
parser.add_argument('--solver', type=str, default='liblinear')
# Data, model, and output directories
parser.add_argument('--output_data_dir', type=str, default=os.environ.get('SM_OUTPUT_DIR'))
parser.add_argument('--model_dir', type=str, default=os.environ.get('SM_MODEL_DIR'))
parser.add_argument('--train', type=str, default=os.environ.get('SM_CHANNEL_TRAIN'))
args, _ = parser.parse_known_args()
# ... load from args.train and args.test, train a model, write model to args.model_dir.
input_files = [ os.path.join(args.train, file) for file in os.listdir(args.train) ]
if len(input_files) == 0:
raise ValueError(('There are no files in {}.\n' +
'This usually indicates that the channel ({}) was incorrectly specified,\n' +
'the data specification in S3 was incorrectly specified or the role specified\n' +
'does not have permission to access the data.').format(args.train, "train"))
raw_data = [ pd.read_csv(file, header=None, engine="python") for file in input_files ]
df = pd.concat(raw_data)
y = df.iloc[:,0]
X = df.iloc[:,1:]
solver = args.solver
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(solver=solver).fit(X, y)
from sklearn.externals import joblib
def model_fn(model_dir):
lr = joblib.dump(lr, "model.joblib")
return lr
In my sagemaker notebook I ran the following code
import os
import boto3
import re
import copy
import time
from time import gmtime, strftime
from sagemaker import get_execution_role
import sagemaker
role = get_execution_role()
region = boto3.Session().region_name
bucket=<bucket> # Replace with your s3 bucket name
prefix = <prefix>
output_path = 's3://{}/{}/{}'.format(bucket, prefix,'output_data_dir')
train_data = 's3://{}/{}/{}'.format(bucket, prefix, 'train')
train_channel = sagemaker.session.s3_input(train_data, content_type='text/csv')
from sagemaker.sklearn.estimator import SKLearn
sklearn = SKLearn(
entry_point='train.py',
train_instance_type="ml.m4.xlarge",
role=role,output_path = output_path,
sagemaker_session=sagemaker.Session(),
hyperparameters={'solver':'liblinear'})
I'm fitting my model here
sklearn.fit({'train': train_channel})
Now, for creating endpoint,
from sagemaker.predictor import csv_serializer
predictor = sklearn.deploy(1, 'ml.m4.xlarge')
While trying to create endpoint, it is throwing
ClientError: An error occurred (ValidationException) when calling the CreateModel operation: Could not find model data at s3://<bucket>/<prefix>/output_data_dir/sagemaker-scikit-learn-x-y-z-000/output/model.tar.gz.
I checked my S3 bucket. Inside my output_data_dir there is sagemaker-scikit-learn-x-y-z-000 dir which has debug-output\training_job_end.ts file. An additional directory got created outside my <prefix> folder with name sagemaker-scikit-learn-x-y-z-000 that has source\sourcedir.tar.gz file. Generally whenever I trained my models with sagemaker built-in algorithms, output_data_dir\sagemaker-scikit-learn-x-y-z-000\output\model.tar.gz kind of files get created. Can someone please tell me where my scikit model got stored, how to push source\sourcedir.tar.gz inside my prefix code without having doing it manually and how to see contents of sourcedir.tar.gz?
Edit: I elaborated the question regarding prefix. Whenever I run sklearn.fit(), two files with same name sagemaker-scikit-learn-x-y-z-000 are getting created in my S3 bucket. One created inside my <bucket>/<prefix>/output_data_dir/sagemaker-scikit-learn-x-y-z-000/debug-output/training_job_end.ts and other file is created in <bucket>/sagemaker-scikit-learn-x-y-z-000/source/sourcedir.tar.gz. Why is the second file not created inside my <prefix> like the first one? What is contained in sourcedir.tar.gz file?
I am not sure if your model is really stored, if you can't find it in S3. While you define a function with the call of joblib.dump in your entry point script, I am having the call at the end of the main. For example:
# persist model
path = os.path.join(args.model_dir, "model.joblib")
joblib.dump(myestimator, path)
print('model persisted at ' + path)
Then the file can be found in ..\output\model.tar.gz just as in your other cases. In order to double-check that is created you maybe want to have a print statement that can be found in the protocol of the training.
You must dump the model as the last step of your training code. Currently you are doing it in the wrong place, as model_fn goal is to load the model for inference, not for training.
Add the dump after training:
lr = LogisticRegression(solver=solver).fit(X, y)
lr = joblib.dump(lr, args.model_dir)
Change model_fn() to load the model instead of dumping it.
See more here.
This post here explains it well:
https://towardsdatascience.com/deploying-a-pre-trained-sklearn-model-on-amazon-sagemaker-826a2b5ac0b6
In short, the tar.gz gets created by tar-gz-ing the model.joblib binary which was first created joblib.dump. To quote the article:
#Build tar file with model data + inference code
bashCommand = "tar -cvpzf model.tar.gz model.joblib inference.py"
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
The inference.py is probably optional.

Using Rasa NLU model with python API instead of HTTP server

Is there a way to use https://nlu.rasa.com model without the HTTP server ? I want to use it as a python library/module.
Yes, and this is documented in there docs at nlu.rasa.com specifically this section.
As of version 0.12.3:
Training
from rasa_nlu.training_data import load_data
from rasa_nlu.config import RasaNLUModelConfig
from rasa_nlu.model import Trainer
from rasa_nlu import config
training_data = load_data('data/examples/rasa/demo-rasa.json')
trainer = Trainer(config.load("sample_configs/config_spacy.yml"))
trainer.train(training_data)
model_directory = trainer.persist('./projects/default/') # Returns the directory the model is stored in
Parsing
from rasa_nlu.model import Metadata, Interpreter
# where `model_directory points to the folder the model is persisted in
interpreter = Interpreter.load(model_directory)
interpreter.parse(u"The text I want to understand")

Categories

Resources