Complete On-Premise and Fully Customisable Chat Bot - Part 4 - Integrating the Natural Language Processor NLP

Welcome back Folks !!

In our part 3 we have seen how to communicate to the bot that we have built. Each and every people would talk the same thing in the different format and on his style. For each response if we start writing the code, Imagine how hard its gonna be.

What can we do for that ?

Natural Language Processor makes it easy to do. NLP is a component that identifies the user conversation and identifies the meaningful context from that.

What NLP is available ?

There are few NLP services like,
1. Dialog flow
2. Luie
3. IBM Voice etc.

But all these are the cloud services and we need to go internet for this. As we speaking about the on premise chat bot we will go for the RASA solution.

What is RASA ?

RASA is a Chatbot server and RASA NLU is the NLP server that comes handy and is built on the python tech stack. Its is a open source and backed by a strong community.

Lets get started with RASA ...

1. Installing the python environment : 

Since we know that the RASA is built on the python tech stack, we need to install the python. To install the python follow the below instructions,

Step 1: Open the respective page for the Operating system of your interest and follow the instruction and install the anaconda with python 3.x (contains most of the packages needed for the RASA)

• Windows - https://conda.io/docs/user-guide/install/windows.html
• MacOs - https://conda.io/docs/user-guide/install/macos.html
• Linux - https://conda.io/docs/user-guide/install/linux.html

Step 2 : Test you Anaconda installation by using "conda list" in the command window.

Step 3 : Test your python using "python -V" command.

Step 4 : let's start installing the RASA NLU server. Open the command prompt and key in the following command 

                 

pip install rasa_nlu

view raw command hosted with ❤ by GitHub

Step 5 :  RASA NLU used the SpaCy, MITE , SkLearn combinations to server the natural language processing. Most of the cases the combination of the SpaCy and Sklearn will work out. So to install the spacy.

                

	pip install rasa_nlu[spacy]
	python -m spacy download en_core_web_md
	python -m spacy link en_core_web_md en

view raw command hosted with ❤ by GitHub

Step 6 : Create a folder called "NLU" inside your bot project. < Revisit this post to see the directory structure >

Step 7: Our NLP processor is a model that learns the experiences from the past data and improves on the accuracy. So we need to provide the data that the user might have spoken for a particular intent of interest. So create the folder inside the NLU folder as data.

Step 8 : Following are the contents of the data folder, rasa_data.json file. The main components of the data file are as follows

	{
	"text": "hey", // The test that the user may speak
	"intent": "greet", // The intent that the text get classified
	"entities": [] // Any data that can be obtained from the text
	}
	// To clearly show how entities are marked
	{
	"text": "show me chinese restaurants",
	"intent": "restaurant_search",
	"entities": [
	{ // This entitiy shows where the value is there and what value is there and what is the name of the value
	"start": 8,
	"end": 15,
	"value": "chinese",
	"entity": "cuisine",
	"confidence": 0.875
	}
	]
	}

view raw components hosted with ❤ by GitHub

Below one is the entire file created for a chinese restaurant use case

	{
	"rasa_nlu_data": {
	"regex_features": [
	{
	"name": "zipcode",
	"pattern": "[0-9]{5}"
	},
	{
	"name": "greet",
	"pattern": "hey[^\\s]*"
	}
	],
	"entity_synonyms": [
	{
	"value": "chinese",
	"synonyms": ["Chinese", "Chines", "chines"]
	},
	{
	"value": "vegetarian",
	"synonyms": ["veggie", "vegg"]
	}
	],
	"common_examples": [
	{
	"text": "hey",
	"intent": "greet",
	"entities": []
	},
	{
	"text": "howdy",
	"intent": "greet",
	"entities": []
	},
	{
	"text": "hey there",
	"intent": "greet",
	"entities": []
	},
	{
	"text": "hello",
	"intent": "greet",
	"entities": []
	},
	{
	"text": "hi",
	"intent": "greet",
	"entities": []
	},
	{
	"text": "good morning",
	"intent": "greet",
	"entities": []
	},
	{
	"text": "good evening",
	"intent": "greet",
	"entities": []
	},
	{
	"text": "dear sir",
	"intent": "greet",
	"entities": []
	},
	{
	"text": "yes",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "yep",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "yeah",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "indeed",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "that's right",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "ok",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "great",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "right, thank you",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "correct",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "great choice",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "sounds really good",
	"intent": "affirm",
	"entities": []
	},
	{
	"text": "i'm looking for a place to eat",
	"intent": "restaurant_search",
	"entities": []
	},
	{
	"text": "I want to grab lunch",
	"intent": "restaurant_search",
	"entities": []
	},
	{
	"text": "I am searching for a dinner spot",
	"intent": "restaurant_search",
	"entities": []
	},
	{
	"text": "i'm looking for a place in the north of town",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 31,
	"end": 36,
	"value": "north",
	"entity": "location"
	}
	]
	},
	{
	"text": "show me chinese restaurants",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 8,
	"end": 15,
	"value": "chinese",
	"entity": "cuisine"
	}
	]
	},
	{
	"text": "show me chines restaurants in the north",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 8,
	"end": 14,
	"value": "chinese",
	"entity": "cuisine"
	},
	{
	"start": 34,
	"end": 39,
	"value": "north",
	"entity": "location"
	}
	]
	},
	{
	"text": "show me a mexican place in the centre",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 31,
	"end": 37,
	"value": "centre",
	"entity": "location"
	},
	{
	"start": 10,
	"end": 17,
	"value": "mexican",
	"entity": "cuisine"
	}
	]
	},
	{
	"text": "i am looking for an indian spot called olaolaolaolaolaola",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 20,
	"end": 26,
	"value": "indian",
	"entity": "cuisine"
	}
	]
	}, {
	"text": "search for restaurants",
	"intent": "restaurant_search",
	"entities": []
	},
	{
	"text": "anywhere in the west",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 16,
	"end": 20,
	"value": "west",
	"entity": "location"
	}
	]
	},
	{
	"text": "anywhere near 18328",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 14,
	"end": 19,
	"value": "18328",
	"entity": "location"
	}
	]
	},
	{
	"text": "I am looking for asian fusion food",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 17,
	"end": 29,
	"value": "asian fusion",
	"entity": "cuisine"
	}
	]
	},
	{
	"text": "I am looking a restaurant in 29432",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 29,
	"end": 34,
	"value": "29432",
	"entity": "location"
	}
	]
	},
	{
	"text": "I am looking for mexican indian fusion",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 17,
	"end": 38,
	"value": "mexican indian fusion",
	"entity": "cuisine"
	}
	]
	},
	{
	"text": "central indian restaurant",
	"intent": "restaurant_search",
	"entities": [
	{
	"start": 0,
	"end": 7,
	"value": "central",
	"entity": "location"
	},
	{
	"start": 8,
	"end": 14,
	"value": "indian",
	"entity": "cuisine"
	}
	]
	},
	{
	"text": "bye",
	"intent": "goodbye",
	"entities": []
	},
	{
	"text": "goodbye",
	"intent": "goodbye",
	"entities": []
	},
	{
	"text": "good bye",
	"intent": "goodbye",
	"entities": []
	},
	{
	"text": "stop",
	"intent": "goodbye",
	"entities": []
	},
	{
	"text": "end",
	"intent": "goodbye",
	"entities": []
	},
	{
	"text": "farewell",
	"intent": "goodbye",
	"entities": []
	},
	{
	"text": "Bye bye",
	"intent": "goodbye",
	"entities": []
	},
	{
	"text": "have a good one",
	"intent": "goodbye",
	"entities": []
	}
	]
	}
	}

view raw rasa_data.json hosted with ❤ by GitHub

If you feel this process of editing the intents and entities are difficult. There is a handy tool at

• online version : https://rasahq.github.io/rasa-nlu-trainer/
• offline tool : https://github.com/RasaHQ/rasa-nlu-trainer

Just upload the file to the online / Offline version and add the intents and entities as shown in the picture below

Step 9: Now your are in the data directory of the RASA project. Come back a step (means to the root directory). Create the .json file and place the below contents.

	{
	"pipeline": "spacy_sklearn",
	"path" : "./projects",
	"data" : "./data/demo-ra.json",
	"emulate" : "luis",
	"cors": true,
	"cors_origins": ["localhost","*","localhost:3000"]
	}
	/**Do not copy this
	pipeline : RASA-NLU to use which pipeline either spacy_sklearn or tensorflow pipeline
	path: Path where the training should create the project and model
	data : path to the data file we created in step 8
	cors : Allow / Disallow CORS for the python NLU server
	"cors_origins": ["localhost","*","localhost:3000"] : Allowed Domains list
	**/

view raw config_spacy.json hosted with ❤ by GitHub

Step 10:  Now it time to train the model with the data we created. Open the command window in the current directory and run this command "python -m rasa_nlu.train -c config_spacy.json"

Step 11: Once if the command completes successfully, you can see the projects folder getting created. At this point, invoke another command to run the HTTP server "python -m rasa_nlu.server -c config_spacy.json --server_model_dirs=<newly Created folder path>"

Tadaaaa !! the server is up and running. Using the postman hit the below url and see the magic.
http://localhost:3000/parse?q="Hello this is the user response".

The Nlu server should understand this and give you back the JSON object that have the predicted Intent Name. Use this in the chatbot to respond back to the user.