Since I started Pand.ai almost a year ago with my co-founder to build “smart” chatbots for financial institutions, one of the most commonly asked questions we get from clients is: How good is your chatbot?

Until recently, I’ve tried to answer that question by explaining how we are using deep learning for natural language process (deep NLP) to extract the semantic of a user’s input, which allows our chatbot to “understand” a question better, and thus provide a more accurate answer.

While we had some successes in signing up big enterprises as clients, it took me until recently to realise the focus of my whole pitch was wrong. In reality, none of my clients paid any attention to the kind of technology we used. In fact, many of them outright didn’t believe we had a more superior technology than some of the more established players (which we actually did, and still do) — this is according to one C-suite executive from our client company. So why are they doing business with us? “Because you offer us a “full package” that extends beyond technology,” he said.

Here, I’ll discuss the thought process we subconsciously used to build smart chatbots for our satisfied clients. I’m now consolidating them into a product framework that form our core design principle – hopefully, it will be useful to anyone who is looking to build an AI-powered chatbot, or looking to buy one.

This framework is based on 4 letters: U.S.E.D. We believe a good chatbot must possess the following attributes:

1. It Understands me

This is by far the most important attribute, because without understanding, your chatbot would not be able to respond in the way that you intended it to be (although technically, chatbots are not “understanding” user’s input, but that’s another story).

Most chatbots that could be built over a weekend relies on a technique called “pattern-matching”, which is extremely useful to “understand” common short phrases like “how are you”, “what’s your name”, etc., but virtually useless when the input gets longer or become less common.

For most serious chatbots, a NLP (Natural Language Processing) engine is usually required. Here, there are a few big players that offer generic NLP engines, often as a cloud service, which could be easily integrated into any chatbot.

Alternatively, there are niche players (like Pand.ai) that offers a more specialized NLP engine for specific verticals or markets where local nuances, jargons or slangs are important.

If you are a developer without too much data science background, it is probably easier to just use one of those generic, off-the-shelf NLP engines to build your chatbot than building one yourself. If you are a buyer, you might want to work with a partner who has their own proprietary NLP engine, for maximum flexibility in customising your chatbot. (Most vendors who do not have their own proprietary NLP engine will find it difficult, if not impossible, to make modification to the core NLP algorithm to meet your needs.)

Also read: Here are 5 myths about chatbots that people are getting completely wrong

2. It Serves me

A good chatbot needs to perform a certain function or functions to serve its users. Let’s say you are deploying a chatbot to your sales team to help them understand your products better, then you have to make sure you have all the relevant product-related content ready, ideally in a structured format (this would not only make it easier for the chatbot to find the right answers quickly and accurately, but also for you to make sure you are covering the right areas).

And beyond the basic Q&A, you may want to consider adding a quiz component to your chatbot, so that it helps your users (i.e. the salespeople) to refresh their understanding of the products. If you want your salespeople to be able to check the application status for their mortgage customers, for example, then you need to connect the chatbot to your internal mortgage approval system, while making sure you comply with your local PDPA (Personal Data Protection Act) requirement.

3. It Engages me

At the initial stage, getting users to interact with your chatbot is likely very challenging, no matter how good your NLP engine is or how much quality content you have. After all, we are talking about getting users to overcome their inertia and to change their behaviours. It is therefore imperative that you have a “push content” strategy in place to help your users get used to using a different channel/tool.

Of course, “push content” doesn’t mean you have to prepare original long articles and be in the news all the time; a simple Father’s Day greeting message, for example, delivered to only the fathers in the group (if you know who they are), will be equally engaging and arguably more heartwarming.

4. It Delights me

Most chatbots today tend to be quite quirky and “cute”. This is because everybody enjoys a good laugh (human’s nature, nobody likes to read boring PR gibberish). However, as more and more chatbots start to flood the market, it is no longer realistic to expect being quirky/cute/humorous will carry the day for you.

It is also just painfully awkward (if not downright ridiculous) for, say, a banking bot, to use a language that is quirky and cute, when the overall brand promise is one that is serious, cautious, and reliable. Thankfully, language is not the only weapon in your chatbot arsenal that can use to delight your users. For example, try burying an Easter egg in one of your answers – you will find that element of delightful surprise trumps any quirky language.

Also read: 9 principles for using chatbots, messaging and speech-based assistants

The golden standard

This “U.S.E.D.” framework also leads to what we believe to be the “North Star Metric” of every enterprise-grade chatbot, i.e. usage. While there are many KPIs one could use to measure how good your chatbot is, such as response time and accuracy, the only one that really matters is whether it helps you achieve your business objective(s). Whether you are looking to use chatbots to deflect customer service volume, increase sales productivity or to generate leads, in order for it to help you achieved your business objectives, it needs to be “used”. Some of the “useful” (no pun intended) KPI we find include:

  • No. of conversations (not messages);
  • Percentage of monthly active users (MAU, also possible are DAU and WAU);
  • No. of conversations per active users.

While you could certainly keep track of as many KPIs as you wish, we recommend that you focus on just ONE to help you in making future product decisions to improve your bot. As you progress through your chatbot lifecycle, you might find it necessary to switch your primary KPI, and that’s completely normal. However, I strongly recommend that you to not deviate from a usage-based KPI, especially if your chatbot also has a machine learning component.

Developing a chatbot, even an AI-powered chatbot is relatively straightforward; building a good, smart chatbot that is heavily “USED”, less so.

—-

The author, Shin Wee, is a second time entrepreneur, co-founder/ CEO of Pand.ai, a fintech veteran. He is currently leading a team of finance and technology experts in his latest venture, a fintech startup that specialises in AI-powered chatbots for the financial industry.

Editor’s note: e27 publishes relevant guest contributions from the community. Share your honest opinions and expert knowledge by submitting your content here.

Featured Image Copyright: 123RF