How to Automate Testing of Your WhatsApp Chatbot

This article was originally published on Botium’s blog on January 28, 2021, prior to Cyara’s acquisition of Botium. Learn more about Cyara + Botium

When it comes to testing WhatsApp chatbots up to now there have been mainly two approaches:

• Testing manually on a smartphone
• Testing backend functionality with API Testing

Automated vs Manual Testing

Both approaches are valid and no enterprise-level test strategy should miss any of them. But there are two obvious flaws:

1. Manual testing is not scalable to the extent that is required to guarantee constant high quality for production-level chatbots
2. Backend testing does not consider the end-user experience

Botium fills this gap with a new connector for testing WhatsApp chatbots on real and virtual smartphone devices.

Building Test Infrastructure

Here is what you need:

• Some smartphones with SIM cards and USB cables
• Or Docker + Docker-Compose to launch virtual devices

Setting Up Real Devices

1. Go and purchase some smartphones (or use your own one) with SIM cards
2. Install and register WhatsApp on them
3. Enable Android developer mode — ask Google how to do it for your device model
4. Attach it to your workstation with the USB data cable
5. Install Android SDK
6. Install and launch Appium

Setting Up Virtual Devices

1. Install Docker and Docker-Compose
2. Launch a virtual device with the help of this project

In short, you can create a file docker-compose.yml and launch it with docker-compose up -d to get a single virtual machine with a single virtual Samsung device up and running. You can see it in action by browsing http://localhost:6080

version: '3'
services: null
samsung_galaxy_S8: null
image: budtmo/docker-android-x86-11.0
privileged: true
ports:
- '6080:6080'
- '4723:4723'
- '5555:5555'
environment:
- DEVICE=Samsung Galaxy S8
- APPIUM=true
- MOBILE_WEB_TEST=false
- AUTO_RECORD=false

Setup and Configure Whatsapp

Now you have to install Whatsapp on this virtual device. There are several options for this, here is the way I did it:

• Download the Whatsapp APK here
• Install the Android SDK on your local machine, you will need adb
• Connect to the virtual device: adb connect localhost:5555
• Upload Whatsapp APK to the virtual device: adb push /path-to/WhatsApp.apk /data/local/tmp/wa.apk
• Install the Whatsapp APK: adb shell pm install -r /data/local/tmp/wa.apk
• Launch Whatsapp: adb shell monkey -p com.whatsapp 1

Now you have to register WhatsApp with a phone number of your choice — purchase a SIM card or use a service like Twilio.

In one of the next Botium releases, the registration step will be automated by Botium Box as well.

Additionally, you have to register the Whatsapp Chatbot you want to test as a Whatsapp contact. The name of the contact is required when configuring Botium later.

Do any tasks regarding registration, terms&conditions etc required by your Whatsapp Chatbot upfront, as this things typically are not part of the test cases.

Now that your devices are up and running and WhatsApp is available, let’s see what else to prepare.

Connect Botium to WhatsApp

You need an installation of Botium Box for this purpose. Experienced DevOps engineers can give Botium Core a try, the free and open-source automation library powering Botium Box and other Botium products.

Get your copy of Botium Core here

Connect Device Lab To Botium

In the Botium Box settings, register a new device provider.

Tell Botium about the Device

While for most device cloud providers the available devices can be listed automatically (by calling the device cloud listing APIs), this is not possible for your local Appium installation. Edit the file LOCALSELENIUM.json in the resources folder of Botium Box to tell Botium about the available devices:

[
{
"name":"Samsung Galaxy S8 Emulator",
"value":{
"type":"MOBILEAPP",
"capabilities":{
"appium:platformName":"Android"
}
}
}
]

Compose a Device Set

Botium Box groups the devices you want to run your tests on in device sets. Create a new device set for your Appium endpoint and select the Samsung Galaxy S8 Emulator (and maybe other devices as well if you connected them).

Configure Botium Connector for Appium

Register a new chatbot in Botium Box

• As Connector/Chatbot Technology use WebdriverIO (Selenium or Appium)
• As Automation Technology select Use Appium
• As Webdriver Script choose Whatsapp

As an experienced Appium developer you might ask Where do I enter the Selenium CSS Selectors ? — with Botium you don’t have to do this as this is part of the Webdriver Script Whatsapp

On the last step of the Quickstart Wizard, make sure to select your device set at the very bottom to start your tests.

Write Test Cases

You can now use the full power of BotiumScript to write your test cases. An easy one could look like this:

hi#me
hi#bot
Welcome to the World Health Organization

Watch Botium Automating Whatsapp

When running a test case, Botium will now

• open the WhatsApp app
• select the contact representing the chatbot
• cleanup the message history
• send test case input from WhatsApp
• receive WhatsApp output and compare with the test case
• repeat the last two steps until the test case is ready

Wrap-Up

With Botium Box and Appium, it is now possible to run automated end-2-end tests of the conversational flow of your WhatsApp chatbot.

Read more about: Chatbot Testing, Chatbots, Conversational AI, Cyara Botium, Tutorial / How-to Guide