Natural Language Search with Elastic Search

An Elastic Search chatbot is able to index and search a textual database. We use it in our chatbots to increase the performance of the FAQ

The power of Elastic Search for
chatbots FAQ

When you work in the chatbot world, you are always on the lookout for new technologies in terms of natural language understanding.

We initially focused on machine learning algorithms capable of detecting intentions and entities. These algorithms are state of the art and work perfectly. But they reach their limit when you want to add a large number of FAQ topics in a chatbot (too long to train, lack of understanding on related topics and on single keywords). We could talk about it for hours.

So we started to test a less new technology (well, it was created in 2004, it’s not prehistoric either) to see if we could get something from it.

Our approach was quite unique in the chatbot world. Hang on, we’ll tell you everything!

All about Elastic Search and chatbots

Find our user documentation that explains how and why to set up Elastic Search to optimize the NLU of your chatbot.

How elastic search works

Unlike intent detection algorithms, Elastic Search does not use machine learning and therefore does not need training phrases to work.

It is therefore very practical for FAQ type uses in which the chatbot has to understand a lot of subjects (sometimes several thousands). In concrete terms, this saves you a lot of time on the configuration. It is also easier to administer.

This algorithm works as follows:

  1. The chatbot will clean the user’s sentence: it will remove all unnecessary words (“from”, “to”, “and”, “if”, “that”, etc…) and take the root of the other main words (e.g.: “downloads” becomes “download”)
  2. It will then compare this cleaned sentence with the content present in the chatbot, and especially with the associated keywords,
  3. If it finds a match it returns the associated resource

And that’s it.

It is notably thanks to this technology that we can import subjects in mass via Excel.

Natural language understanding + Elastic Search: winning duo

Once our tests were completed and concluded on FAQ skills only, we encountered another problem.

Our clients’ chatbots are primarily action-oriented. That is, their primary purpose is to perform actions: send an email, reboot a password, etc…

These skills require an understanding in NLU mode. That’s what it’s for.

So we needed to have chatbots with NLU and Elastic Search.

So we did.

And the results are pretty crazy. The optimal structure for understanding is as follows:

  1. The user types a phrase
  2. The chatbot first activates the NLU and finds an intention
    1. If it is a intention action then it detects the entities and refers to the associated action skill
    2. If it is a intention FAQ then it refers to the associated FAQ skill and then searches in the said skill to find the closest resource

So you end up with a chatbot that can perform complete actions AND answer simple questions in a powerful way!