In the previous 4 installments of our Building a Voicebot IVR with Dialogflow series, we validated we could make a decent phone-based voicebot interface for your average retail business. As we documented, setting up this system is no trivial matter. It is certainly not something a typical small business would do, so we set out on researching ways to make this technology more widely accessible. We devised a system for using Dialogflow to provide individual voicebots for different businesses using a templating approach.

In this post we will walk through how we arrived at our architecture, how we created a templating system for providing customized fulfillment, and we share a lot of our node.js code for reference. We’ll start by covering our approach and then dig into the implementation.

Architecture

Hierarchies for handling multiple businesses

We wanted to allow a non-technical business owner to set up their own voicebot IVR in minutes. There are many different kinds of businesses with a near infinite number of things a caller could ask. To reduce the scope of possibilities, we started with a few key assumptions. First, we assumed businesses in a similar category would have similar intents - i.e. restaurants will get asked about “menus”. Second, we assumed business in a given sub-category will have very similar same intents with individual fulfillment that is basically the same except for varying parameter values - i.e. “how big is your large pizza” will vary by pizza restaurant - say “10 inch” for Joe’s Pizza” and “12 inch for Alice’s Pizza”. This means we could start with a few business categories and setup a reasonable number of generic intents for each. These intents could be grouped into templates by business or sub-business type and we could template the responses.

We also wanted to make sure every business has its own unique phone number. As covered in the previous posts, we had a system for that, so we just needed to figure how to tie that phone number to what appears to be a unique agent, applying the unique responses for each business at the right time. For that, we considered 2 options:

1. One agent per category
2. One agent per business

One agent per category

Dialogflow is not really designed to handle multiple applications within the same bot, but as long as your intents are the same you can use different fulfillment to provide unique responses for each individual business. This approach would let us use one Dialogflow agent per business category - i.e pizza shop, hairdresser, dentist, etc..

Using the contexts discussed in the previous post with VoxImplant’s VoxEngine API’s, we were able to pass the called phone number as a unique identifier for each business through to Dialogflow’s fulfillment. Our fulfillment service would then use the phone number to look up the appropriate responses for that business.

One Agent per Business

The other approach is to use Dialogflow more for how it was intended - one agent per bot with each having its own intents and fulfillment. In this case, we do not need to rely so heavily on fulfillment, since much of each business's individual responses could be statically stored with the intent. Fulfillment still needs to be used in some cases where a static response is not practical, but this is far less than doing it for every intent like we had to in the the one agent per template approach. Responses are faster when you can define them statically in the bot since you can skip the fulfillment step.

Adding flexibility, at a cost

We chose the one agent per business approach, but it wasn’t without some cost. The one agent per template approach works ok if you have a simple bot without much variation between businesses. However, we came up with many examples where we might want to have business-specific intents. While it is possible to use contexts to filter these on a per-business basis, this would lead to a bloated bot with many intents. The Dialogflow GUI is set up for the one agent per business model, so using that allows for easy tweaking without always having to involve communicating with a database in our back-end.

Programmatically creating agents

Since we chose to use one agent per business, we wanted a way to programmatically create the agent from a template. This was not possible when we originally started putting together this summary. It was only possible to create an agent manually through the Dialogflow web console. Our solution was just to manually create a bunch of Dialogflow agents and keep track of them as available or not available in our database.

Whenever you create a new Agent, a new project is created in the Google Cloud (GCP) account you used to authenticate against Dialogflow. Google forces you to create one project per agent, so there is no sharing of keys. This forced us to create extra logic to track the Dialogflow API keys for each agent. Manually copying over this key information was not pleasant 🙁.

Of course, Google added an API to create an agent right after we finished our research. We did not try this, but it would appear to address much of the bot creation pains mentioned above.

Implementation

Static vs. dynamic responses

Imagine you are a business owner. If someone asks:

When are you open?

Without context, you could respond with an all-encompassing response like:

“We are open from 5 PM to midnight, Monday to Thursday. On Friday and Saturday we open at 10 am and close at 2 am. On Sunday we are open from 10 am to midnight”

This is actually easy to fit into a static response, built into the intent. However, if someone asks something more specific like:

How late are you open tomorrow night?

You could respond with the long phrase above, but that’s a long response with way more information than was requested - not a great experience. Assuming it was Saturday, it would be better to respond with just:

tomorrow we are open to midnight

In this case we can’t just use a static intent without sounding robotic. Fulfilment is needed here to:

1. Figure out what day it was an add 1 to it for “tomorrow”
2. Look up the hours and return the appropriate response

It is easy to get carried away by getting more specific, so we sought to find a balance between intents that were too broad and too specific. In this example, we could simplify the intent to cover any questions about “hours for tomorrow”, we could probably get away with broadening that to:

On Sunday we are open from 10 am to midnight

Using a methodology like this we mapped intents to a set of fixed and dynamic responses. The next step was to stick this all in a database that could be used for our agent templates (for static responses) and fulfillment (for dynamic ones).

Data Model

This is just an example that shows our journey in this topic. We used node.js with MongoDB using mongoose, lodash and Express but you could use any similar stack you prefer. Note intermediate to advanced node skills are assumed for most of the rest of this post and we will skip a lot of detail to keep focus on the Dialoflow specifics. You can see the full code in our repo.

Business Information

Before we start digging into some code, lets review the restaurant data models we setup. Let’s start with our restaurant model that keeps track of all the business specific details. Below is the database entry:

This is also gives you an idea of our data structure. Pay special attention to the agent and openHours fields, we will be using those moving forward: openHours will be used as an example on how to respond back to Dialogflow’s fulfilment request. agent tells us the Dialogflow agent that is tied to this business.

Dialogflow agent tracking

This is an agent entry example:

You can see here how we keep track of the agent credentials so we can individually update them (you should have them in a separate config file but we are trying to show everything as straightforward as possible), the project name, and if the agent is already assigned to a business or not. When a new user finishes filling the data for their business, you just need to go through this collection checking for which agents are available, mark them as available = false and assign them to the corresponding restaurant document for the user (agent field on the restaurant model).

Agent template

The last database entry we would like to show is what we called agent template. This document keeps the format of each answer we need to create in order to reply back to a fulfillment request. Let’s just focus on hoursOfOperation as we mentioned above:

Ok, now that you have a rough idea of the contents of the DB, let's get to the the Dialogflow setup and fulfillment code.

Dialogflow Setup

First you need to get into Dialogflow and create an agent.

Make your Intent

After that we will include a sample intent for an intent named Hours Of Operation.

Make sure you properly set the entity type you will be getting out of each question in the Action and parameters section.

As shown above in the agent template model, each intent property has a nested structure. At the first child level you will see date and date-time. This corresponds to the parameter we get in the Dialogflow fulfillment request body according to how the user asked the question.

Intent JSON responses

Our code needs to understand what exactly does “now” or “tomorrow” mean. If you set the parameters above Dialogflow should automatically extract values for these. The JSON below shows how Dialogflow flow codes “Are you open now?” into tangible information we can use to compute the answer:

{  
   "responseId":"9eb99604-0303-4d64-b757-8d4d125e11ce-2dd8e723",
   "queryResult":{  
      "queryText":"are you open now?",
      "parameters":{  
         "date-time":{  
            "date_time":"2019-06-09T23:42:17-03:00"
         },
         "date":""
      },
      "allRequiredParamsPresent":true,
      "intent":{  
         "name":"projects/my-demo-agent/agent/intents/30fe78ed-9d2d-4156-acfb-10c174db0417",
         "displayName":"Hours of operation"
      }
   },
   "session":"projects/my-demo-agent/agent/sessions/f2698a2d-c8ca-1029-9dc0-c6a41a3a5352"
}

Here we got a date-time parameter. If the question was “are you open tomorrow” we get a date parameter instead since no specific hour is provided:

{  
   "responseId":"2a1e9fdd-e5f4-4597-be68-73b7e16b1d29-2dd8e723",
   "queryResult":{  
      "queryText":"are you open tomorrow?",
      "parameters":{  
         "date":"2019-06-10T12:00:00-03:00",
         "date-time":""
      },
      "allRequiredParamsPresent":true,
  }
}

Fulfillment

Lets see how we can set up an application that can provide an answer to each of these requests using the data we gathered from our customer. First of all, make sure you set up your intent to use fulfillment. As you can see from this image we are tunneling our application using ngrok*, so Dialogflow can hit our HTTP server:

*If you never heard about ngrok, it's a great tool for making your local web servers accessible over the Internet that you will definitely love.

Fulfillment code

For our example, let’s try to make our bot answer the question:

Are you open tomorrow?

This is the piece of code that will handle the incoming request from Dialogflow.
Then, at a high level we just build our answer and reply back.

const {WebhookClient} = require('dialogflow-fulfillment');

app.post('/fulfillment', (req, res) => {
  const agent = new WebhookClient({ request: req, response: res });

  buildIntentAnswer(agent)
  	.then(respondToFulfilment(agent));
});

const buildIntentAnswer = data => getFormattedData(data).then(getAnswer);

const respondToFulfilment = _.curry((agent, response) => {
	let intentMap = new Map();
  	intentMap.set(agent.intent, (agent) => agent.add(response));
  	return agent.handleRequest(intentMap);
});

If you want to know more about the WebhookClient and handleRequest functions, I strongly suggest you to check out Dialogflow’s docs on those.

The most important function of this chain is buildIntentAnswer which we will cover deeply up next.

getFormattedData - gathering restaurant information

To get an answer for our question, the first thing we need to do is to gather some relevant business information. We need to know if the restaurant is actually open or closed at the moment the user asked. In order to do that, we need to query our database and find the restaurant hours of operation.

const getProjectName = data => data.session.split('/')[1];

const getFormattedData = (data) => {
	const agent = getProjectName(data);
	const agentType = 'restaurant';

	return new Promise((resolve, reject) => {
		getRelevantBusinessInformation(agent, agentType, data.intent).then((businessInfoForIntent) => {
			const fData = hoursOfOperation.getFormattedData(data, businessInfoForIntent);
			fData.agent = getProjectName(data);
			fData.agentType = agentType;
			resolve(fData);
		});
	});
}

const getRelevantBusinessInformation = (agent, agentType, intentName) => {
	return new Promise((resolve, reject) => {
		Agent.findOne({'name': agent}).then((agent) => {
			models[agentType].findOne({agent: agent._id}).then((restaurant) => {
				if (intentName === 'Hours of operation')
					return resolve(restaurant.operation.openHours.toObject());
				else
					throw new Error('Intent not yet supported');

			});
		});
	});
}

getFormattedData cares about obtaining that business information and shaping it in a way we can use to compute our answer.

getRelevantBusinessInformation matches the agent in use (we got the name in the incoming request object), with the particular restaurant that is assigned to it. Once we have a matching restaurant it just returns the relevant information for our intent, in this case openHours.

Once we got the openHours object, we need to use that information to create meaningful string values that will be used in the answer. A tangible example is that you need to translate the openingHour time from 12 in the database to “twelve o ‘ clock” (hourToText function).

For that, we include an hoursOfOperation intent module, which encapsulates all of this logic. Think that each intent will need a complete different set of rules. These individual per-intent modules serve that purpose.

hoursOfOperation.js

const DAYS_OF_THE_WEEK = ['sunday', 'monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday'];
const converter = require('number-to-words');

hourToText = (hour) => {
	const strHour = hour.toString();
	const lastTwo = strHour.slice(-2);
	const remaining = strHour.substring(0, strHour.length - 2);
	const txtMins = lastTwo === '00' ? 'o clock' : converter.toWords(lastTwo);
	const txthours = converter.toWords(remaining);
	return `${txthours} ${txtMins}`
};

const dateHandler = (data, operationData) => {
	let date = data.parameters.date;
	date = new Date(date);
	const day = DAYS_OF_THE_WEEK[date.getDay()];
	const dayInfo = operationData[day];
	dayInfo.day = day;
	
	return {
		intent: data.intent,
		businessData: dayInfo,
		processedParams: {
			type: 'date',
			day
		},
		txtVars: {
			day,
			openHour: hourToText(dayInfo.openHour),
			closeHour: hourToText(dayInfo.closeHour)
		}
	};
}

const getHandler = (paramType) => {
	const mapping = {
		'date': dateHandler,
		'date-time': dateTimeHandler,
		'time': timeHandler
	};
	return mapping[paramType] ? mapping[paramType] : defaultHandler;
}

function getRelevantParam(params) {
	for (let param in params) {
		if (params[param]) return param;
	}
	throw new Error('No relevant param found');
}

exports.getFormattedData = (data, operationData) => {
	const handler = getHandler(getRelevantParam(data.parameters));
	return handler(data, operationData);
};

exports.getState = (data) => {
	if (data.processedParams.type === 'date-time') {
		if (!data.businessData.open) return 'closed';

		const openHour = data.businessData.openHour;
		const closeHour = data.businessData.closeHour;
		const hour = data.processedParams.hour;

		//The restaurant closes before the day ends
		if (openHour < closeHour) {
			if (hour > openHour && hour < closeHour) return 'open';
			else return 'closed';
		} else { //The restaurant closes the day after
			if (hour > openHour) return 'open';
			else return 'closed'
		}
	}

	if (data.processedParams.type === 'date') {
		return data.businessData.open ? 'open' : 'closed';
	}
};

getAnswer - building the answer

We just finished gathering all the information we need, now we need to build the answer. We need the actual explicit sentence we want to play over the phone. We will make use of getState in our hoursOfOperation module, which will let us know if our restaurant is open or closed, comparing the information provided in the question and the actual restaurant opening hours. Note that this module contains logic to handle different types of parameters. We just included a dateHandler in this example, but you may need one for each type of parameter you could potentially get.

const getAnswer = (relevantData) => {
	return new Promise((resolve, reject) => {
		const answerVariant = computeAnswerVariant(relevantData);
		AgentTemplate.findOne({type: relevantData.agentType}).then((template) => {
			const possibleAnswers = template.getAnswers(relevantData.intent, answerVariant, relevantData.txtVars);
			const answer = _.sample(possibleAnswers);
			return resolve(answer);
		});
	});
}


const computeAnswerVariant = (data) => {
	return {
		state: hoursOfOperation.getState(data),
		paramType: data.processedParams.type
	};	
}

The template model gets all possible answers (if we don't want to repeat the same sentence over and over again we provide different variations of the same answer). Lodash's _.sample method simply picks a random one of the array of answers.

This is a piece of the AgentTemplate mongoose schema that produces the text sentences:

const getTemplatePropertyFromIntentName = (intentName) => {
  const mapping = {
    'Hours of operation': 'HoursOfOperation',
    'Location': 'Location'
  };

  return mapping[intentName];
}

agentTemplateSchema.methods.getAnswers = (intentName, answerVariant, params) => {
  const replaceVars = (text, params) => {
    for (let prop in params) text = text.replace(`{{${prop}}}`, params[prop]);
    return text;
  }

  const templateProp = getTemplatePropertyFromIntentName(intentName);

  let answers;

  try {
    answers = this.intents[templateProp][answerVariant.paramType][answerVariant.state];
  } catch(e) {
    answers = this.intents[templateProp][answerVariant];
  }
  
  return answers.map(ans => replaceVars(ans, params));
};

const AgentTemplate = mongoose.model('AgentTemplate', agentTemplateSchema);

module.exports = AgentTemplate;

As you can see, getAnswers method in the model just accesses the proper child in the document (date in this case) and then, replaces all the values surrounded by curly braces with the proper value.

That covers our brief code review. We encourage you to clone the repo and run it locally to see it working.

Everything that has a beginning has an end

This is the last post in our Building a Voicebot IVR with Dialogflow series. Our plan to summarize our research in a single post early in May morphed into a series covering methods for adding phone connectivity to Dialogflow, reviews of the Dialogflow connectors provided by SignalWire and VoxImplant, how to add SMS to your IVR and this last one showing how we pulled it all together. Next week Chad will be giving a presentation summarizing this series during his Kill Your IVR with a Voicebot talk at ClueCon in Chicago on August 6th.

Now it’s your turn! We are always happy to share notes on projects and talk about our work. Make sure to leave some comments below so we can connect.

About the Authors

Chad Hart is an analyst and consultant with cwh.consulting, a product management, marketing, and strategy advisory helping to advance the communications industry. In addition, he recently co-authored a study on AI in RTC and helped to organize an event / YouTube series covering that topic.

Emiliano Pelliccioni is a computer engineer working at webRTC.ventures and specializes in developing real time communication applications for clients around the globe. One of his projects includes developing bot integrations for a major CPaaS provider.

Remember to subscribe for new post notifications and follow @cogintai.