Cyara Solutions:
Chatbot Testing
Don’t Let Your Chatbots Fail.
Take Them from Good to Great.
The use of chatbots is on the rise. Every industry, including ecommerce retailers, banks, and healthcare providers, are deploying chatbots for self-service to improve their customers’ experiences while keeping productivity up and costs down. Market size projections are off the charts with a projected $1.34 billion by 2024, and a 31% CAGR. 1
While companies are excited to usher in chatbots as the new customer service channel, consumers are far less enthusiastic. According to Forrester, 54% of consumers expect a chatbot to negatively affect their lives. 2
Why are so many chatbots failing? Some of the primary reasons are that most chatbots do not understand customer intent, and many lack sufficient training and testing. And to exacerbate the situation, when chatbots cannot understand a customer, many fail to provide an easy escalation to a human.
For chatbots to be successful, they need to be helpful to customers. To achieve that status, enterprise chatbot development teams should focus their attention on the capabilities below.
Ensure Chatbots Understand Customer Intent
Customer intent is what the customer wants to get done, or the ultimate outcome they’re seeking when they engage with your company.
Examples of customer intent are returning a purchased item, receiving flight status, and replenishing funds for a debit card. Similar to Interactive Voice Response (IVRs), chatbots are challenged by understanding customer intent. For chatbots, there are many ways that humans express what they need in written format — that’s called customer input. Natural Language Understanding (NLU), a branch of AI, is often used by chatbots to interpret typed customer inputs.
For example, Andrea wants to return a t-shirt she purchased. Here is a small sampling of how she might type the request to a chatbot. Note that none of the requests uses the word “return,” yet the chatbot must understand that the customer intent is to return an item.
- t shirt too big and want $ back
- hey, how do i send this back to u? can u pick?
- Need a refund on this tee
Understanding customer intent comes naturally to humans, but is extremely difficult to imbue in chatbots, particularly with the nuanced ways humans express themselves through customer input. Chatbots are only as good as the knowledge that powers them so development teams must collect large amounts of data that relate to customer intent.
Perform Chatbot Training
Chatbots need training that provides data on how to understand customer intent. Because of the complexity and diversity of ways in which humans convey their intent, it takes large amounts of data to represent the broad range of inputs that chatbots must be able to interpret. This training data is a set of examples expressing each intent that a chatbot uses to turn into a mathematical model for recognizing each intent.
Training data emphasizes target use cases, enabling chatbots to successfully recognize and handle each use case. Obtain training data from domain experts (conversational and customer service expertise), chat logs based on customers talking to chat agents, website requests, SMS, email, and any other documentation on how customers type requests.
In Andrea’s example above, to make sure chatbots are prepared to help her return the t-shirt that she purchased, development teams should collect the many ways that she might type her request, and include these in chatbot training data, like:
- t shirt too big and want $ back
- hey, how do i send this back to u? can u pick?
- Need refund on this t
- can i exchange for a small
- give me credit for t shirt
- hey amazon, take this back
Continuously Test Chatbot Technology
Why? Because continuously testing chatbots assures quality at scale. Similar to other customer experience (CX) channels, chatbots require continuous testing and continuous improvement. In the previous example on Andrea’s request to return a purchase, the training data would be used to create testing data and test cases. There are increasingly new ways that humans express what they need, and automating the testing of your chatbot technology is the only efficient way to continually learn and adapt to meet customers’ needs.
The DevOps methodology of shifting left directly relates to chatbot development. When teams adopt a shift-left mentality, it means they’re able to focus on problem prevention, rather than detection. As CX software is designed, testing is designed in parallel. With shift-left development, teams create test cases directly from designs, and can typically execute testing through the design phase.
To apply CX software best practices to chatbot testing, follow these critical steps:
- Create testing data
- Test target use cases
- Test non-target use cases
- Test chatbots in the context of the whole omnichannel customer journey
- Test chatbot escalations to a live chat agent
- Test chatbot and live chat agent performance under peak load conditions
- Monitor the chatbot in production
Chatbots are on the rise, but so are chatbot failures. Teach your chatbots how to understand customer intent, train your chatbot, and continuously test your chatbot technology to ensure that, if issues occur, you can catch and resolve them before your customers experience them.
Utilizing chatbots to cost-effectively and efficiently deliver a flawless customer experience is within reach. Make your chatbot an example of great customer experience. Learn how Cyara’s Automated CX Assurance Platform and the Cyara Botium product enable flawless CX journeys across voice and digital channels.
Learn More
See how the Cyara Automated Customer Experience Platform can help you achieve your CX Assurance goals, and delight your customers.
1 New Chatbot Market 2018-2024, Market Study Report, December 2018.
2 Ian Jacobs, Julie A. Ask, Andrew Hogan, Forrester Infographic: Customer Service Chatbots Fail Consumers Today, Forrester, January 30, 2019.