This article was originally published on Botium’s blog on February 3, 2021, prior to Cyara’s acquisition of Botium. Learn more about Cyara + Botium
Automated voice app testing is the process of using software tools to simulate user interactions with voice-enabled applications to validate conversation flows, audio quality, and system availability without manual intervention. This guide suggests best practices, infrastructure, and tools to ensure your voice app continues to deliver outstanding user experience.
Key takeaways
- Voice apps face unique testing challenges including limitless user inputs, audio transcription instability, and environmental variables like background noise and microphone quality.
- Essential infrastructure includes a text-to-speech (TTS) engine, audio effects simulation capabilities, and a continuous monitoring setup.
- Core testing practices involve recording or synthesizing test cases, applying homophone mappings to reduce flakiness, and simulating real-life audio conditions.
Questions when testing voice apps
Application of the suggested practices helps answer the questions:
- Is my voice app following the designed conversation flow?
Validate this by creating test cases that compare expected responses against actual voice app outputs at each conversation step.
- How does my voice app work under real-life conditions?
Test with simulated background noise, varied audio quality, and slow network connections to ensure robust performance.
- Is my voice app available 24×/7?
Use continuous monitoring with scheduled smoke tests to detect service interruptions immediately.
What are the unique challenges of testing voice apps?
Testing voice apps presents distinct challenges compared to graphical user interfaces, such as limitless user inputs, audio transcription instability, and environmental flakiness.
While a graphical user interface restricts the possible user interactions by the controls it offers, with natural language, the number of possible user inputs is limitless. Additionally, when using voice as user input there are more variables to take into account: the individual nuances in voices, the quality of the microphone, the background noises surrounding the speaker, and more — when testing a graphical user interface, a button click is always perceived the same by the application, regardless of who actually clicked it.
The platforms behind powerful voice applications are still evolving and are subject to constant improvements — which means that developers have to rely on components that they do not own and the possible influence is limited.
Glossary of Key Terms:
- Flakiness: Inconsistent test results caused by variable factors like transcription accuracy, leading to false failures.
- Homophone mappings: Configuration rules that treat similar-sounding words as equivalent to reduce transcription-related test failures.
- Text-To-Speech (TTS): Technology that converts written text into synthesized spoken audio for test automation.
- Smoke test: A basic test that checks fundamental functionality, such as sending a simple greeting and verifying a response.
Testing the voice conversation flow
Cyara Botium provides you with all the tools required for implementing a comprehensive, holistic test strategy for your voice apps. You can read about Botium and the background on testing conversation flow in the official Botium documentation.
We will use Bring! Shopping List as an example of a voice app to test. It is published as Alexa Skill, and we can use the Botium Connector for Amazon Alexa with AVS for simulating voice input and output with Botium.
Record test cases
The quickest way to get started is to use the Live Chat in Botium Box to record your own voice with your microphone. You can immediately see and listen to the response of your voice app.
Depending on the technology of your voice app, both text and audio response are shown or either of them.
You can save the conversation as a test case and make some changes afterward.
- Refining input and output text and audio
- Using wildcard matching or utterance lists instead of full text
- Add additional test steps or asserters
Synthesize test cases with text-to-speech
Instead of recording your own voice for the test cases, you may decide to instead (or additionally) use synthesized voice samples. Botium has its own text-to-speech and speech-to-text platform based on the best open source and cloud engines available, Botium Speech Processing.
Test cases are showing plain text now instead of audio input:
Eliminating flakiness: homophone mappings
A typical problem when testing voice apps is that audio transcriptions, especially for low-quality audio, can be rather unstable. In test automation, we usually rely on hard facts (fixed text assertions), and this will lead to increased flakiness of the test results.
In this example, you can see that instead of okay milch ist auf deiner liste the transcription says okay milch is auf s einer liste — this one character difference will make a test case fail:
Botium provides the option to specify homophone mappings to deal with audio snippets that are often misinterpreted by the Speech-To-Text engine.
Test cases use these mappings to qualify transcription results as success or failed.
Testing real-life scenarios
Using your own microphone in front of your laptop might be a good starting point, but in real-life voice apps are used in another way, such as with smartphones, with home automation or entertainment devices like Alexa or Google Home, in a car. To come up with meaningful end-to-end test cases for these scenarios you will have to make your test data similar to those scenarios.
- Add background noise on various levels.
- Pitch volume up or down.
- Simulate various levels of distance.
- Simulate technical restrictions like GSM phone line or low bandwidth.
- Simulate otherwise bad audio quality like interruptions or various levels of silence.
- … you name it …
In Botium, you can apply various effects for simulating real-life usage scenarios to your own clean recordings or synthesized audio samples.
Continuous monitoring
The recipe for ensuring the availability of your voice app is actually rather simple, all you need is:
- A smoke test for checking basic behavior (for instance, just sending a simple hello to the voice app and listening for a response)
- A scheduler to run the smoke test every few minutes (e.g., every 5–10 minutes for critical applications)
- A notification mechanism to inform you in case of failures
With Botium, everything you need is coming out of the box.
Best practices for automated voice app testing
- Use synthesized voice samples for consistent test repeatability.
- Implement homophone mappings to handle transcription variations and reduce flakiness.
- Apply audio effects (background noise, volume changes, bandwidth limitations) to simulate real-life conditions.
- Set up continuous monitoring with scheduled smoke tests to verify 24/7 availability.
- Record and refine test cases using wildcard matching or utterance lists for flexible assertions.
Summary
Now you know what is needed for automated testing of your voice app, you may give Botium a try.
- Record your own voice or use a synthesized voice
- Apply audio effects for real-life simulation
- Conversation flow testing with Botium
Frequently Asked Questions
Automated voice app testing uses software to simulate user interactions with your voice application, validating conversation flows, voice prompts, and backend integrations without manual effort. It helps ensure your app performs as designed across a wide range of real-world scenarios.
Unlike graphical user interfaces that restrict possible inputs, voice apps face a limitless number of possible user inputs, plus additional variables like microphone quality, background noise, and individual voice nuances. Manual testing cannot scale to cover this complexity consistently.
The quickest way to start is by using a live chat tool to record your own voice and save the conversation as a test case. From there, you can refine inputs and outputs, add assertions, and expand coverage using synthesized voice samples.
Homophone mapping lets you define common misinterpretations made by speech-to-text engines so that small transcription variations do not cause test cases to fail. This reduces flakiness in test results, especially when testing low-quality audio.
You can apply audio effects to your recordings or synthesized samples to simulate scenarios like background noise, varying volume levels, distance, GSM phone lines, low bandwidth, and audio interruptions. This helps ensure your app remains reliable in the environments where users actually interact with it.
Continuous monitoring combines a smoke test that checks basic app behavior, a scheduler that runs the test every few minutes, and a notification mechanism that alerts you immediately if a failure is detected. This approach helps you catch and resolve service interruptions before they affect your customers.
Yes, you can use text-to-speech synthesis to generate voice samples for your test cases instead of or in addition to recording your own voice. This provides broader coverage across different voice characteristics and scenarios.
