I have done a lot of research on how to create chat bots (the responding part) however I can't find a way to make it more advanced. For example, I keep seeing NLTK reflections but I want to know if there are more advanced methods in NLTK (or other modules) that allow me to create a learning bot, smart bot or even an AI but I am struggling in finding modules, tutorials or documentation that help with getting started and proceeding that way. Reflections don't always work well like responding in context unless you have many lines of code pre-written for content which is inefficient and may not always be accurate. Note: I don't want to be spoon fed, I just want to be pointed in the right direction of stuff that I can do and look at.
a solution would be
e.g. user asks: "who is your favourite actor?"
bot replies with: "Brad Pitt"
(only though of Brad because of the ad astra advertisements xD)
Below is the code that I am trying to stay away from.
pairs = [
[
r"my name is (.*)",
["Hello %1, How are you today ?",]
],
[
r"what is your name ?",
["My name is Chatty and I'm a chatbot ?",]
],
[
r"how are you ?",
["I'm doing good\nHow about You ?",]
],
[
r"sorry (.*)",
["Its alright","Its OK, never mind",]
],
[
r"i'm (.*) doing good",
["Nice to hear that","Alright :)",]
]```
There are two main styles of conversational agents: retrieval and generative. The regex code you show can be thought of as a very simple retrieval model. More complicated retrieval models classify user input with a classifier (at this point, almost always a neural network). Generative models treat the input to output mapping as a machine translation problem, and use techniques from NMT, neural machine translation.
Some resources:
The new verrsion of Speech and Language processing, by Dan Jurafsky and James H. Martin has three chapters on bots/question-answering
If you are trying to build something practical, then you should use a mature library. IMO, Rasa has been leading the pack for a few years at this point
The code you want to stay away from used to be the very beginning of cahtbots (Eliza: https://blog.infermedica.com/introduction-to-chatbots-in-healthcare/). A good starting point is a full dialogue system. You could use for example the trindikit for python, which is basically a dialogue manager. Furthermore, you need to implement some sort of common sense reasoning database (e.g. compare Erik T Mueller: Commonsense Reasoning - An event Calculus based approach). Normally, most chatbots are focussed on a specific domain (product questions, recommender etc.), so you need to fix exactly what intentions may provoke which speech acts, classify and model them accordingly (LSTM for calssification, Bayes for production).
Upon all this you either have to build a surface realisation system or use canned text as templates, which is growing work when your domain expands more and more.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I want to create an intelligent chatbot-type app for iOS that uses the Cornell movie database and some custom training data I create to train a machine learning model that can talk to the user, preferably keeping track of the conversation and performing a few simple tasks.
I'll go more in depth.
Recently I came across a way to convert TensorFlow models to CoreML models so I tried working with TensorFlow and realized that the conversion method is limited and working with TensorFlow would require me to learn a lot more than I already knew and I simply do not have the time to do that. I am an iOS developer and am therefore familiar with Swift and Objective-C, so I was wondering if there was a way to do the same thing using them.
I would like to be able to train a model with the Cornell movie database I tried using when training TensorFlow, plus whatever contributions I make to it. What I mean by that is I might want my chatbot to do some specific tasks or reply to certain abnormal questions in a specific way. I think I could do this by adding my own "conversations" to the Cornell database that contain my specific needs. If there is some other way I could inject my own data into this, please let me know.
I also want the chatbot to be able to do specific tasks. Therefore I believe I need to have a more complicated model that can recognize when the chatbot is told to do something such as turn on a light here or send a text to someone (very small things that would just need a trigger and some parameters).
It would also be great to have the model remember certain aspects of the conversation. For instance, if the chatbot asks for my name, and I reply, it should remember what my name is later if I ask it what my name is. This would also apply to more complicated things such as a scenario where I ask it what the weather is in Toronto tomorrow, it replies, and then I ask "What about next Tuesday?", and it replies with the weather in Toronto for next Tuesday.
Overall, I would like to use CoreML (and possibly its seemingly very good Natural Language processing capabilities) to create a complex model for a chatbot for iOS applications.
I don't know if this is at all possible but I would really like to try to get the most advanced and conversational chatbot for iOS as possible.
If you’re fine with making calls to an API (which would make it useless without WiFi) and paying for it, Google’s Dialogue Flow might be a good tool for you, as it already has great NLP. You’d still have to do a lot of the information storing by yourself, and write your own functionality for other parts (like actually sending requests to a weather API) but it’s very good for understanding the intent behind someone’s speech.
If you don’t want to pay or need it to work without WiFi, There is a 40+ hour long course called Natural Language Processing A-Z on Udemy which is very useful for learning NLP with tensorflow. Takes more work but google doesn’t take a cut and you might be able to work without WiFi.
I am working on automating task flow of application using text based Natural Language Processing.
It is something like chatting application where the user can type in the text area. At same time python code interprets what user wants and it performs the corresponding action.
Application has commands/actions like:
Create Task
Give Name to as t1
Add time to task
Connect t1 to t2
The users can type in chat (natural language). It will be like a general English conversation, for example:
Can you create a task with name t1 and assign time to it. Also, connect t1 to t2
I could write a rule drive parser, but it would be limited to few rules only.
Which approach or algorithm can I use to solve this task?
How can I map general English to command or action?
I think the best solution would be to use an external service like API.ai or wit.ai. You can create a free account and then you can map certain texts to so-called 'intents'.
These intents define the main actions of your system. You can also define 'entities' that would capture, for instance, the name of the task. Please have a look at these tools. I'm sure they can handle your use case.
I think your issue is related to Rule-based system (Wiki).
You need to two basic components in core of project like this:
1- Role base:
list of your roles.
2- Inference engine:
infers information or takes action based on the interaction of input and the rule base.
spacy is python approach that I think it will help you. (More information).
You may want to try nltk. This is an excellent library for NLP and comes with a handy book to get you started. I think you may find chapter 8 helpful for finding sentence structure, and chapter 7 useful for figuring out what your user is requesting the bot to do. I would recommend you read the entire thing if you have more than a passing interest in NLP, as most of it is quite general and can be applied outside of NLTK.
What you are describing is a general problem with quite a few possible solutions. Your business requirements, which we do not know, are going to heavily influence the correct approach.
For example, you will need to tokenize the natural language input. Should you use a rules-based approach, or a machine learning one? Maybe both? Let's consider your input string:
Can you create a task with name t1 and assign time to it. Also, connect t1 to t2
Our system might tokenize this input in the following manner:
Can you [create a task] with [name] [t1] and [assign] [time] to it. Also, [connect] [t1] to [t2]
The brackets indicate semantic information, entirely without structure. Does the structure matter? Do you need to know that connect t1 is related to t2 in the text itself, or can we assume that it is because all inputs are going to follow this structure?
If the input will always follow this structure, and will always contain these kinds of semantics, you might be able to get away with parsing this using regular expressions and feeding prebuilt methods.
If the input is instead going to be true natural language (ie, you are building a siri or alexa competitor) then this is going to be wildly more complex, and you aren't going to get a useful answer in a SO post like this. You would instead have a few thousand SO posts ahead of you, assuming you have sufficient familiarity with both linguistics and computer science to allow you to approach the problem systematically.
Lets say text is "Please order a pizza for me" or "May I have a cab booking from uber"
Use a good library like nltk and parse these sentences. As social English is generally grammatically incorrect, you might have to train your parser with your custom broken English corpora. Next, These are the steps you have to follow to get an idea about what a user wants.
Find out the full stop's in a paragraph, keeping in mind the abbreviations, lingos like ...., ??? etc.
Next find all the verbs and noun phrases in individual sentences can be done through POS(part of speech tagging) by different libraries.
After that the real work starts, My approach would be to create a graph of verbs where similar verbs are close to each other and dissimilar verbs are very far off.
Lets say you have words like arrange, instruction , command, directive, dictate which are closer to order. So if your user writes any one of the above verbs in their text , your algorithm will identify that user really means to imply order. you can also use edges of that graph to specify the context in which the verb was used.
Now, you have to assign action to this verb "order" based on the noun phrase which were parsed in the original sentence.
This is just a high level explanation of this algorithm, it has many problems which needs serious considerations, some of them are listed below.
Finding similarity index between root_verb and the given verb in very short time.
New words who doesn't have an entry in the graph. A possible approach is to update your graph by searching google for this word, find a context from the pages on which it was mentioned and find an appropriate place for this new word in the graph.
Similarity indexes of misspelled words with proper verbs or nouns.
If you want to build a more sophisticated model, you can construct graph for every part of speech and can select appropriate words from each graph to form sentences in response to the queries. Above mentioned graph is meant for Verb Part of speech.
Although, #whrrgarbl is right. It seems like you do not want to train a bot.
So, then to handle language input variations(lexical, semantic..) you would need a pre-trained bot which you can customize(or may be just add rules according to your need).
The easiest business oriented solution is Amazon Lex. There is a free preview program too.
Another option would be to use Google's Parsey McParseface(a pre-trained English parser, there is support for 40 languages) and integrate it with a chat-framework. Here is a link to a python repo, where the author claims to have made the installation and training process convenient.
Lastly, this provides a comparison of various chatbot platforms.
As a learning experience, I would like to build a site which is structured in this (simplified) fashion:
Registered users can log in to their accounts. Each user has his own "profile"
Once logged in, they can join certain "networks" or groups (or whatever you'd like to call them.)
These users can only view the content of these networks if they have joined them.
Networks have admins, each with varying, customizable permissions (x, y, and z). These admins can edit the network's content.
How can I go about achieving this (in basic terms) with Pyramid's __acl__, RootFactory, and authorization/authentication features? How would each network have its own permissions relative to the basic log-in/out feature for user accounts?
I've read the documentation multiple times, and I am starting to grasp how Pyramid security works; however, I feel as if I need some more specific clarification. For instance, in order to create the most simple of log-in features, do I need to specify an __acl__ in my RootFactory, or is this simply defining various groups, regardless of who can log in? Basic things, but also with enough "unusual" circumstance that I can see how it all fits together...
Note: I'm not looking for someone to write me code. I'm just looking for some basic specification in a situation which I do not understand.
Thanks for the help.
I recently added login/authentication/authorization to an URL Dispatch based pyramid application. I used a chapter from one of the official tutorials (http://docs.pylonsproject.org/projects/pyramid/en/latest/tutorials/wiki2/authorization.html). I prefered this concise chapter over the large security chapter of the narrative documentation . If you want to use Traversal, choose same chapter from official tutorial creating a Wiki in ZODB.
At least this chapter will answer your question regarding basic login/logout, create users/groups, create and apply basic ACL to RootFactory, protect views with permissions.
To address your site structure I would recommend Michael Merickels Tutorial (http://michael.merickel.org/projects/pyramid_auth_demo/index.html). It applies the concepts from the documentation about resources/security to several use cases. I really liked it, may be you get inspired too. If you come back to ask more specific questions regarding these use cases, answers could be more easy and concise than the one to your current question.
Vocabulary that I am using:
nounphrase -- A short phrase that refers to a specific person, place, or idea. Examples of different nounphrases include "Barack Obama", "Obama", "Water Bottle", "Yellowstone National Park", "Google Chrome web browser", etc.
category -- The semantic concept defining which nounphrases belong to it and which ones do not. Examples of categories include, "Politician", "Household items", "Food", "People", "Sports teams", etc. So, we would have that "Barack Obama" belongs to "Politician" and "People" but does not belong to "Food" or "Sports teams".
I have a very lage unlabeled NLP dataset consisting of millions of nounphrases. I would like to use Freebase to label these nounphrases. I have a mapping of Freebase types to my own categories. What I need to do is download every single examples for every single Freebase type that I have.
The problem that I face is that need to figure out how to structure this type of query. At a high level, the query should ask Freebase "what are all of the examples of topic XX?" and Freebase should respond with "here's a list of all examples of topic XX." I would be very grateful if someone could give me the syntax of this query. If it can be done in Python, that would be awesome :)
The basic form of the query (for a person, for example) is
[{
"type":"/people/person",
"name":None,
"/common/topic/alias":[],
"limit":100
}]
There's documentation available at http://wiki.freebase.com/wiki/MQL_Manual
Using freebase.mqlreaditer() from the Python library http://code.google.com/p/freebase-python/ is the easiest way to cycle through all of these. In this case, the "limit" clause determines the chunk size used for querying, but you'll get each result individually at the API level.
BTW, how do you plan to disambiguate Jack Kennedy the president, from the hurler, from the football player, from the book, etc, etc http://www.freebase.com/search?limit=30&start=0&query=jack+kennedy You may want to consider capturing additional information from Freebase (birth & death dates, book authors, other types assigned, etc) if you'll have enough context to be able to use it to disambiguate.
Past a certain point, it may be easier and/or more efficient to work from the bulk data dumps rather than the API http://wiki.freebase.com/wiki/Data_dumps
Edit - here's a working Python program which assumes you've got a list of type IDs in a file called 'types.txt':
import freebase
f = file('types.txt')
for t in f:
t=t.strip()
q = [{'type':t,
'mid':None,
'name':None,
'/common/topic/alias':[],
'limit':500,
}]
for r in freebase.mqlreaditer(q):
print '\t'.join([t,r['mid'],r['name']]+r['/common/topic/alias'])
f.close()
If you make the query much more complex, you'll probably want to lower the limit to keep from running into timeouts, but for a simple query like this, boosting the limit above the default of 100 will make it more efficient by querying in bigger chunks.
The general problem described here is called Entity Linking in natural language processing.
Unabashed self plug:
See our book chapter on the topic for an introduction and an approach to perform large scale entity linking.
http://cs.jhu.edu/~delip/entity_linking.pdf
#deliprao
We want to implement a "News feed" where a user can see messages
broadcasted by her friends, sorted with newest message first. But the
feed should reflect changes in her friends list. (If she adds new
friends, messages from those should be included in the feed, and if
she removes friends their messages should not be included.) If we use
the pubsub-test example and attach a recipient list to each message
this means a lot of manipulation of the message recipients lists when users
connect and disconnect friends.
We first modeled publish-subscribe "fan out" using conventional RDBMS
thinking. It seemed to work at first, but then, since the IN operator
works the way it does, we quickly realized we couldn't continue on
that path. We found Brett Slatkin's presentation from last years
Google I/O and we have now watched it a few times but it isn't clear to
us how to do it with "dynamic" recipient lists.
What we need are some hints on how to "think" when modeling this.
Pasting the answer I got for this question in the Google Group for Google App Engine http://groups.google.com/group/google-appengine/browse_thread/thread/09a05c5f41163b4d# By Ikai L (Google)
A couple of thoughts here:
is removing of friends a common event? similarly, is adding of
friends a common event? (All relative,
relative to "reads" of the news feed)
From what I remember, the only way to make heavy reads scale is to write
the data multiple times in peoples'
streams. Twitter does this, from what
I remember, using a "eventually
consistent" model. This is why your
feed will not update for several
minutes when they are under heavy
load. The general consensus, though,
is that a relational, normalized
model simply will not work.
the Jaiku engine is open source for your study:
http://code.google.com/p/jaikuengine.
This runs on App Engine Hope these
help when you're considering a design.