Automated Testing and Monitoring for Alexa Skills with Botium

This article was originally published on Botium’s blog on June 28, 2020, prior to Cyara’s acquisition of Botium. Learn more about Cyara + Botium

TL;DR; Botium and the Alexa Voice Service (AVS) connector can perform automated testing for Alexa skills – no speaking out loud necessary.

By connecting Botium to Alexa Voice Service (AVS) all Botium Stack tools are immediately available for testing your Alexa skill:

• Use the Record&Replay in Botium to, well, record and replay conversations with your Alexa skill
• Run hundreds or thousands of automated test cases against your Alexa skill, every day, every hour to make sure it behaves as expected
• Integrate automated testing for the Alexa skill in your DevOps pipeline
• Monitor your Alexa skill for functional flaws in production
• Use Botium Coach to continuously analyze and improve the understanding of your Alexa skill
• And more

Sidenote: Botium includes another connector using the Skills Management API (SMAPI), which is also used for testing Alexa skills. Both of them should be part of a complete test strategy:

• Use the SMAPI technology for regression tests and NLP performance tests
• Use the AVS technology for E2E tests

See this blog post for an overview of the available test levels with Botium.

Setting up Botium to work with Alexa Voice Service requires two steps:

1. Register Botium as a Virtual Alexa device with the Alexa Voice Service
2. Connect Botium to a Speech-To-Text- and Text-To-Speech-engine

Why do you need a Speech-To-Text and Text-To-Speech engine?

Surprising answer: Because Alexa operates mainly with voice input and output. When Botium is talking to your Alexa skill this works basically like this:

1. Botium turns the test case text into audio using text-to-speech (TTS)
2. Botium sends the audio to the Alexa Voice Service
3. Botium receives the audio reply
4. Botium turns the audio back into text for doing the test case assertions and analytics

There are several options available:

• Use Botium Speech Processing as a free and open-source engine
  • You will have to host it yourself (or use one of the Botium paid plans)
• Use Google Cloud Text-To-Speech and Speech-To-Text
  • Costs are involved!
• Use Amazon Polly and Amazon Transcribe
  • Costs are involved!
  • Amazon Transcribe is rather slow and not recommended for this purpose

Register Botium as a Virtual Alexa device

1. Open the Alexa Voice Service console and click on Products to manage your AVS products
2. Click on Add New Product to register Botium as a new virtual device category
   • As product type, choose Application with Alexa built-in
   • Note that it doesn’t really matter what values you select in the other fields of the form. Fill out the required fields and it will be fine
3. Create a new Security Profile
   • Again, doesn’t really matter what values you enter
   • Finally, choose Other devices and platforms, enter a Client ID name, then click Generate ID
   • No need to download the file
4. From the main product screen, you will need these values:
   1. Client ID
   2. Client secret
   3. Product ID

Now continue in Botium Box:

1. Open Chatbots section and click Register New Chatbot
2. Enter any chatbot name and select Alexa Voice Service in Connector/Chatbot Technology
3. Enter Product ID, Client ID and Client secret and click the 2.1 Authorize Device button
   • You should see a Device Authorized message
4. Select the Language of your Alexa account
5. Click the now enabled 2.2 Register Your Device button
   • This will open a separate browser window to enter a device authorization code
   • The device authorization code is in your clipboard – you can insert it there by doing Ctrl-V
   • Accept the conditions and wait for the registration to finish
6. Now click the 2.3 Acquire Refresh Token button, wait for the token to appear and click on Save

Connect Botium to a Speech-To-Text- and Text-To-Speech-engine

As an example, we will now continue setup with Google Cloud services.

1. Select or create a Cloud Platform project
2. Enable billing for your project (free tier available)
3. Enable the Google Cloud Text-to-Speech and Speech-to-Text API
4. Set up authentication with a service account so you can access the API from your local workstation
   1. Make sure to select Owner or Cloud Speech Service Agent as role
5. Save the JSON credentials file

Now header over to the chatbot view and open the Connection Settings in the Settings tabs.

1. Open the Import Config Files tab
2. Drag&Drop the JSON credentials file from above to the right drop zone
3. In the Text-ToSpeech and Speech-To-Text sections, select the language of your Alexa account
4. Save

Finally, click the Say Hello button to do a quick check if the configuration is working. It will send a simple Hello to the Alexa Voice Service and show the response.

You can even listen to the audio sent to Alexa and received from Alexa by opening the Botium Code view with the <> button.

Botium and Alexa Voice Service setup is ready now. Let’s have some test automation fun.

Recording Alexa Skill Sessions

Click on the Live Chat button to start a recording session. Open a conversation by telling Alexa something like Alexa, open my skill and …

Click on Save Test Case to make Botium save the conversation as a test case in a new test set. By activating the New Test Project option, Botium will create a new Test Project as well, which is required for CI/CD integration.

The Test script is ready now, and you can open it in the Test Script designer, or you can also see the generated BotiumScript if you want to go deeper into Botium.

You can now record more sessions with the Live Chat, or create the test scripts step by step in the test case designer.

Enable Monitoring

Open the new Test Project and switch to the Settings tab.

Here you can enter a good-old-style cron expression for scheduled test execution, as well as an email to receive notifications on test failures.

It’s as easy as that.

Next Steps

These were only the first steps in voice app testing with Botium. Next things to discover:

• Integrate your Alexa skill test project with a DevOps pipeline like Jenkins or Bamboo
• Give Botium Coach a try to improve your understanding of your Alexa skill
• Dive into the Botium Wiki for another boost to your test automation infrastructure

Learn more about Cyara Botium.

Read more about: Alexa, Automated testing, Cyara Botium, Test Automation