How can I search effectively for information in a database, being able to handle both spelling errors and longer texts, unlike icontains and trigram similarity?
Info
Language: Python 3.9.7
Platform: Django 3.2.8
Database: PostgreSQL 14.1
Goal
Searching effectively for information in a database by being able to handle both spelling errors and longer texts.
There are many ways I can query for information in a database, mine being PostgreSQL, such as __contains, __icontains, unnaccent__icontains, __unaccent__lower__trigram_similar, and so on. Though these are very handy, none of these helped my really reach the goal I have in mind: searching for the most similar entry. Trigram similarity got pretty close, but it no longer worked for longer strings. Is there any operation that can get the closest value, also working for longer strings?
Here is the problem I'm facing:
In my website, I want a person to be able to search for books. I'm using trigram similarity. Searching for lord and lord of yields no results, while lord of the, lord of the rings, and lord of the rings: fellowship of the ring gives the correct results.
I don't like this! People who might not want to type all of that text wouldn't do it, they'd probably just type lord or something like that. I could just use __contains or __icontains, but I want to handle spelling mistakes.
How do I handle both of them?
Thoughts & Ideas
I've used SpaCy before, but that was only for my machine learning projects. Should I use that?
If there is no available answer, I think I'm going to have to make one on my own.
Related
This request is more about some research materials...
I want to build an application to control medicine dosage and to do this I think about making sample database with like 1000 medicines with info most importantly about a dosage (4 times a day, every 4 hours...) and composition.
Does anyone know about some webpage that could easily provide me with this kind of information? I think about webscraping it with python. Or maybe somewhere there is already created sample database like this.
Also, I think about making feature about what elements in medicine composition can be dangerous together. Something like: That you can't take some 3 medicines together. Maybe someone know and could recommend some webpage, article, maybe algorithm that could provide me more knowledge about this subject.
I am working on automating task flow of application using text based Natural Language Processing.
It is something like chatting application where the user can type in the text area. At same time python code interprets what user wants and it performs the corresponding action.
Application has commands/actions like:
Create Task
Give Name to as t1
Add time to task
Connect t1 to t2
The users can type in chat (natural language). It will be like a general English conversation, for example:
Can you create a task with name t1 and assign time to it. Also, connect t1 to t2
I could write a rule drive parser, but it would be limited to few rules only.
Which approach or algorithm can I use to solve this task?
How can I map general English to command or action?
I think the best solution would be to use an external service like API.ai or wit.ai. You can create a free account and then you can map certain texts to so-called 'intents'.
These intents define the main actions of your system. You can also define 'entities' that would capture, for instance, the name of the task. Please have a look at these tools. I'm sure they can handle your use case.
I think your issue is related to Rule-based system (Wiki).
You need to two basic components in core of project like this:
1- Role base:
list of your roles.
2- Inference engine:
infers information or takes action based on the interaction of input and the rule base.
spacy is python approach that I think it will help you. (More information).
You may want to try nltk. This is an excellent library for NLP and comes with a handy book to get you started. I think you may find chapter 8 helpful for finding sentence structure, and chapter 7 useful for figuring out what your user is requesting the bot to do. I would recommend you read the entire thing if you have more than a passing interest in NLP, as most of it is quite general and can be applied outside of NLTK.
What you are describing is a general problem with quite a few possible solutions. Your business requirements, which we do not know, are going to heavily influence the correct approach.
For example, you will need to tokenize the natural language input. Should you use a rules-based approach, or a machine learning one? Maybe both? Let's consider your input string:
Can you create a task with name t1 and assign time to it. Also, connect t1 to t2
Our system might tokenize this input in the following manner:
Can you [create a task] with [name] [t1] and [assign] [time] to it. Also, [connect] [t1] to [t2]
The brackets indicate semantic information, entirely without structure. Does the structure matter? Do you need to know that connect t1 is related to t2 in the text itself, or can we assume that it is because all inputs are going to follow this structure?
If the input will always follow this structure, and will always contain these kinds of semantics, you might be able to get away with parsing this using regular expressions and feeding prebuilt methods.
If the input is instead going to be true natural language (ie, you are building a siri or alexa competitor) then this is going to be wildly more complex, and you aren't going to get a useful answer in a SO post like this. You would instead have a few thousand SO posts ahead of you, assuming you have sufficient familiarity with both linguistics and computer science to allow you to approach the problem systematically.
Lets say text is "Please order a pizza for me" or "May I have a cab booking from uber"
Use a good library like nltk and parse these sentences. As social English is generally grammatically incorrect, you might have to train your parser with your custom broken English corpora. Next, These are the steps you have to follow to get an idea about what a user wants.
Find out the full stop's in a paragraph, keeping in mind the abbreviations, lingos like ...., ??? etc.
Next find all the verbs and noun phrases in individual sentences can be done through POS(part of speech tagging) by different libraries.
After that the real work starts, My approach would be to create a graph of verbs where similar verbs are close to each other and dissimilar verbs are very far off.
Lets say you have words like arrange, instruction , command, directive, dictate which are closer to order. So if your user writes any one of the above verbs in their text , your algorithm will identify that user really means to imply order. you can also use edges of that graph to specify the context in which the verb was used.
Now, you have to assign action to this verb "order" based on the noun phrase which were parsed in the original sentence.
This is just a high level explanation of this algorithm, it has many problems which needs serious considerations, some of them are listed below.
Finding similarity index between root_verb and the given verb in very short time.
New words who doesn't have an entry in the graph. A possible approach is to update your graph by searching google for this word, find a context from the pages on which it was mentioned and find an appropriate place for this new word in the graph.
Similarity indexes of misspelled words with proper verbs or nouns.
If you want to build a more sophisticated model, you can construct graph for every part of speech and can select appropriate words from each graph to form sentences in response to the queries. Above mentioned graph is meant for Verb Part of speech.
Although, #whrrgarbl is right. It seems like you do not want to train a bot.
So, then to handle language input variations(lexical, semantic..) you would need a pre-trained bot which you can customize(or may be just add rules according to your need).
The easiest business oriented solution is Amazon Lex. There is a free preview program too.
Another option would be to use Google's Parsey McParseface(a pre-trained English parser, there is support for 40 languages) and integrate it with a chat-framework. Here is a link to a python repo, where the author claims to have made the installation and training process convenient.
Lastly, this provides a comparison of various chatbot platforms.
I have a MySQL database, working with Python, with some entries for products such as Samsung Television, Samsung Galaxy Phone etc. Now, if a user searches for Samsung T.V or just T.V, is there any way to return the entry for Samsung Television?
Do full text search libraries like Solr or Haystack support such features? If not, then how do I actually proceed with this?
Thanks.
yes, Solr will surely allow you to do this and much more.You can start Here
and SolrCloud is a really good way to provide for High Availabilty to end users.
You should have a look at the SynonymFilterFactory for your analyzer. When reading the documentation you will find this section that rather sounds like the scenario you describe.
Even when you aren't worried about multi-word synonyms, idf differences still make index time synonyms a good idea. Consider the following scenario:
An index with a "text" field, which at query time uses the SynonymFilter with the synonym TV, Televesion and expand="true"
Many thousands of documents containing the term "text:TV"
A few hundred documents containing the term "text:Television"
You should keep in mind to have separate analyzers for index and query time, as described in this SO question How to make solr synonyms work.
I have developed a search engine for restaurants. I have a social network wherein users can add friends and form groups and recommend restaurants to each other. Each restaurant may serve multiple cuisines. All of this in Python.
So based on what restaurants a user has recommended, we can zero in on the kind of cuisines a user might like more. At the same time, we will know which price tier the user is more likely to explore(high-end, fast food, cafe, lounges etc)
His friends will recommend some places which will carry more weightage. There are similar non-friend users who have the recommended some the restaurants the user has recommended and some more which the user hasn't.
The end problem is to recommend restaurants to the user based on:
1) What he has recommended(Other restaurants with similar cuisines) - 50% weightage
2) What his friends have recommended(filtering restaurants which serve the cuisines the user likes the most) - 25% weightage
3) Public recommendation by 'similar' non-friend users - 25% weightage.
I am spending a lot of time reading up on Neo4j, and I think Neo4j looks promising. Apart from that I tried pysuggest, but it didn't suit the above problem. I also tried reco4j but it is a Java based solution, whereas I'm looking for a Python based solution. There is also no activity on the Reco4j community, and it is still under development.
Although I've researched quite a lot, I might be missing out on something.
I'd like to know how would you go about implementing the above solution? Could you give any use cases for the same?
I think you won't find any out-of-the box solution for you problem, as it is quite specific. What you could do with Neo4j is to store all your data that you use for building recommendations (users, friendships links, users' restaurants recommendations and reviews etc) and then build you recommendation engine on this data. From my experience it is quite straightforward to do once you get all your data in Neo4j (either with Cypher or Gremlin).
I have the following search:
titles = Title.objects.filter(title__icontains=search)
If this is a search to find:
Thomas: Splish, Splash, Splosh
I can type in something like "Thomas" or "Thomas: Splish, Splash, Splosh" and it will work.
However, if I type in something like "Thomas Splash", it will not work. How would I improve the search to do something like that (also note that if we split on words, the comma and other non-alphanumerics should be ignored -- for example, the split words should not be "Thomas:", "Splish," but rather "Thomas", "Splish", etc.
This kind of search is starting to push the boundaries of django and the ORM. Once it gets to this level of complexity I always switch over to a system that is built entirely for search. I dig lucene, so I usually go for ElasticSearch or Solr
Keep in mind that full text searching is a subsystem all unto itself, but can really add a lot of value to your site.
As Django models are using database queries there is not much magic you can do.
You could split your search by non-alphanumeric chars and search objects containing all words but this will not be smart and efficient.
If you want something really smart maybe you should check out haystack:
http://haystacksearch.org/