This article was originally published on Botium’s blog on April 20, 2021, prior to Cyara’s acquisition of Botium. Learn more about Cyara + Botium
One question we hear a lot is “What’s the best way to start testing a chatbot?”
The answer is surprisingly simple: Build up knowledge by educating yourself, and you will be able to answer this question yourself. This post will walk you through the major milestones that test automation engineers have to take for building this knowledge. We will suggest readings and other material that you might find helpful as well.
Step One: Build a First Sample Chatbot
The best way to get going is to actually build yourself a sample chatbot. The reason is that some things like natural language processing (NLP) might appear like magic if you are new to the topic. They aren’t. I suggest taking one of these tutorials — don’t worry, all of them have free plans available:
- Quickstart: Build an agent (Google Dialogflow)
- Getting started with Watson Assistant (IBM Watson Assistant)
Go the whole way from starting the project to publishing a chatbot on a preview website — everything you need is included in those cloud platforms.
I do not recommend to start with a free, self-hosted stack like Rasa, as it might distract from the journey to become a chatbot tester by tearing your focus more towards infrastructure topics and actual chatbot development.
Step Two: Understand Machine Learning Concepts and Metrics
When dealing with non-trivial chatbots, there is always a machine learning component for natural language processing (NLP) involved. You do not have to get an expert in machine learning to efficiently test them, but you definitely have to understand the taxonomy used in machine learning projects. The central concepts to learn are:
- Intents, Entities, Utterances
- Accuracy, Sensitivity, Specificity, Precision, Recall, F1-Score
- Confidence and Confidence Threshold
- Confusion Matrix
Here are some links to introductory articles:
- Upskilling for test automation engineers — Testing Conversational AI
- Quality Metrics for NLU/Chatbot Training Data
- Upskilling Test Engineers for Chatbot Projects
Step Three: Get Comfortable with BotiumScript
You learned how to test websites with Selenium. You learned how to test smartphone apps with Appium. You might be tempted to think A chatbot is basically a website answering questions, why not use Selenium or Appium for testing a chatbot? Chatbots are a new kind of app that require new kinds of tools to perform effective and high-coverage testing — a goal that cannot be reached with slow, real-time, flaky end-2-end tests.
You can read more in one of my previous blog posts
So invest some days to educate yourself with BotiumScript, an easy-to-learn scripting language for describing chatbot test cases. You can start at the official Botium Docs or watch Shama Ugale in one of her awesome talks.
Step Four: Get Your Toolstack in Place
In the kitchen, one of the most important phrases is mise en place — it means to put everything into place before getting started with cooking. Now that you have a good understanding of the basics, it is time to get your tool stack in place you can use for your chatbot testing projects.
- Subscribe to a free-forever Botium Box Mini instance to optimize your learning curve — Botium Box includes everything to get you started, all you need is a web browser
- For command line aficionados, install Botium CLI, the swiss-army-knife for chatbot testing
- If you are used to test runners like Mocha, Jest or Jasmine, you will love Botium Bindings
To get started in Botium Box, read through the Botium Box Wiki with tutorials, how-tos, technical background and reference docs.
Step Five: Understand the Challenges of Chatbot Testing
There are some important differences when testing a chatbot as compared to websites or smartphone apps. What will strike experienced test automation engineers most is that a user can ask a chatbot literally anything (and the users will do it as soon as the chatbot is live, believe me). Dealing with non-determinism is a critical topic in software testing. Due to the nature of human language, it is impossible for software tests to cover all possible situations.
You can read more in one of my previous blog posts.
Because of this, it is even more important to understand the required test levels to meet at least the minimum quality standards in software projects.
- Regression Testing on API level— Identify flaws in the conversation flow before going to production
- NLP Testing — Improve your chatbot understanding
- E2E Testing — Verifying the end-user experience
- Voice Testing — Understand your users on voice channels
- Performance Testing — Ensure your chatbot is responsive under high load
- Security Testing — Making your chatbot secure
- Monitoring — Get notified when problems arise
Read about some important pitfalls here: 4 DO’s and 3 DON’Ts for Chatbot Testing Strategies
The good news is that you can shine and partially re-use your test automation skills gathered in your previous projects in chatbot testing as well.
Up and Beyond
Wherever you are in your journey, from building your first sample chatbot to running a full-blown continuous chatbot testing pipeline, we’re delighted to have you as part of our Botium community. Please feel free to ask questions on our Botium forum. We look forward to helping you in your journey to become a chatbot testing expert!
