This article was originally published on Botium’s blog on February 2, 2021, prior to Cyara’s acquisition of Botium. Learn more about Cyara + Botium
“What is the value of testing chatbots?” — A question that hits right in the heart of every passionate tester. Seeing it again in my inbox, it’s time to shed some light on this topic.
First – Chatbots are Software
“Thank you, Captain Obvious” is what you might think. But in fact, that’s not that clear for everyone. Conversational AI, meaning chatbots and virtual assistants are Software programs. And Software needs to be tested. This is common sense in the industry right now but it took us decades of time and billions of error fixing costs in production to learn it.
According to the National Institute of Standards and Technology (NIST), software bugs cost the U.S. economy $60 billion every year.
Second – The value of testing
The value of testing is derived from quality. And quality means the satisfaction a user can feel every time he talks to a chatbot or virtual assistant. This is why we call the whole discipline “Quality Assurance”. And of course confidence. A high confidence level to ship products into production environments is of utmost importance in agile projects.
Automation is the key enabler. Test automation leads to the early identification of defects and unwanted behavior. Decreasing the time to market as a result and the costs of quality while amplifying the end-user satisfaction level at the same time.
Third – The holistic test approach of Botium
Everything here at Botium started with an open-source test automation framework for bots called Botium Core. This pretty soon became the industry standard for testing conversation flows but in fact, it was missing some key features. Therefore we introduced Botium (formerly Botium Box) to offer a holistic test approach for chatbots.
The following image represents the BDLC (Bot Development Life Cycle). Milestones in every phase are shown underneath the gray line. Starting from the left with analyzing the user needs, deciding on your bot channels, followed by the NLP selection, and so on. Above the gray line, you can see the holistic approach of Botium by adding value in every phase.
PLAN: Data sets for different domains help to identify actual user needs. Provider benchmark tests done with Botium Coach are the foundation to choose the right NLP engine for your bot.
DESIGN: Based on the conversation flow designed, Botium Box will automatically generate data sets for testing and training in just a few seconds. This is a very good starting point to automate your chatbot.
DEVELOP: The Botium Crawler will crawl through the entire conversation tree of your chatbot and generate data sets based on the data collected. And if you add speech support to your bot, Botium has built-in voice-based testing.
TRAIN: The natural language processing and its NLP score are continuously challenged using Botium Coach.
TEST: Conversation flow testing, NLP score testing, real End-to-End testing, voice-based testing, testing IVR systems, performance testing, security, and GDPR testing just to name a few superpowers.
DEPLOY: Constantly keep an eye on your chatbot in production using Botium Monitoring.
Wrap-Up
Chatbots are Software and Software needs to be tested. The value is derived from quality and the end-user satisfaction. To keep both at the highest level possible, Botium offers a holistic test approach for conversational AI with many superpowers.
