I recently caught up with Scott Anderson, SVP, Senior Team Lead at Bank of America, and asked him how his team makes test automation so successful. Scott is an expert in IVR quality assurance and testing and has been at the forefront of customer experience assurance.
Q: Which IVRs do you work with at Bank of America?
A: I lead automation for a QA team that handles the regression testing for three very large IVRs: consumer home loan, credit card and deposits, and wealth management. These three IVRs handle over a million calls each day. The IVRs are continually being updated and changes need to be pushed to production almost every month.
Our focus has been on IVR regression testing. I work with QA teams dedicated to the automation of the regression testing. We maintain and execute a regression suite of thousands of scripts that need to be run on a regular basis.
Q: Where did you start in test automation?
A: Our initial focus was to use automation to increase the breadth, depth and speed of regression testing. IVR calls are slow to test when done manually because each call needs to be dialed and data needs to be input for each step. A single test might have more than 60 steps. So as you can imagine, manually testing thousands of IVR tests requires many well-trained testers and takes a long time. Before automation, we were lucky to have the time and resources to execute hundreds of regression test cases, with automation, we execute thousands of test cases each code release.
Q: How has the migration to an Agile methodology impacted your testing and use of automation?
A: As we moved to an Agile methodology, I was asked to see if automation could be used, not just for regression, but to help speed up new functionality testing. My first thought was how different automated regression scripts are compared to Software Integration Test (SIT) new functionality test cases.
Regression: Purpose – Make sure we didn’t break code that wasn’t touched.
The script must be precisely tuned. No one will be checking scripts that Cyara passes, so if not tuned well, a defect could be missed. Also, if not tuned well, scripts could fail that aren’t defects, causing triage and additional tuning, which is costly and inefficient. Regression scripts must stand the test of time, they are run before every release/install for years.
Agile New Functionality: Purpose – Test all aspects of a code change.
Many of the scripts are only needed for a single sprint (usually only 2 weeks). Although, some will be kept for regression and/or rolled into future sprints. Because this is a new code change, all steps related to the change must be checked by the tester. We can’t rely on Cyara to pass the script. The normal rigor we put into tuning an automated regression script will slow us down and negate the benefits of automated execution.
It was clear that we couldn’t use our same regression script tuning approach for Agile new functionality testing. Instead, we focused on only one thing: have Cyara place the call and record it, then the QA testers only have to review the recorded steps to pass the scripts.
This simple approach also helped us with training. Most of the Agile QA testers have been executing manual test cases for months or years. Intense, robust script tuning would have a longer/steeper learning curve. We came up with a few rules that allowed them to quickly and efficiently automate most of their Agile new functionality scripts.
Q: What are some of the Agile automation best practices (for humans) you developed to reap the full benefit of test automation?
A: We have a simple goal, to make Agile new functionality testing faster and easier. This goal requires a “speed play” which is a mindset change. The best practices are focused on bringing together the best of both worlds, technology and humans.
1. Don’t place manual calls
Manual testing IVR calls is a laborious and time-consuming process. For each test script, you have to place a call, listen to the Pre-IVR steps to help set up the environment, call flow, greeting, identification and authorization, and many other call flows that have not changed.
We brought in the Cyara CX Assurance Platform to eliminate manual testing. Let Cyara do what it does best, which is to rigorously place the calls and run through all the steps in each test script.
2. Don’t watch the scripts run
This seems obvious, but manual testers are used to executing each script, one by one, and determining pass/fail. I received feedback that the automation, ‘really wasn’t much faster than manual testing’. After reviewing what they were doing, we realized that some of the testers were executing a script in Cyara and watching it execute! So we added a simple rule:
Once you set up and launch a test script or campaign, move on to another task.
The time savings in test automation is lost if you sit and watch Cyara run the scripts. What is important is that Cyara executes the whole script and captures the recordings. You can verify after the scripts run that the testing was completed. Yes, it is amazing how quickly Cyara runs the scripts, but you don’t have time to watch it run.
3. Only review the steps in the IVR test that are related to the code change
You only need to review the steps in the test calls that are important to you. A test script for an IVR call might have more than 50 steps. Some steps might be related to the testing process (e.g., test case ID, DNIS, etc.), some steps might be related to identification and authorization (e.g. name or PIN), and other steps might not have changed. If there are 20 scripts with 25 steps each, then there are 500 steps. Let Cyara run through all the steps, then selectively review only the steps that are important, for example, the steps where there are code changes.
We have a group review process and the call recording feature saves everyone’s time. The group does not have to listen to the whole call. We jump right to the steps that are important to us and listen closely to those recordings. That helps us focus and do a better job.
4. It is not always necessary to tune the test scripts
This may sound strange, but you do not need to tune the test scripts in order to pass the new functionality. You just need to know if the new functionality works correctly. Even if a script fails in Cyara, you only need to listen to the parts of the call that relate to the new functionality. The test script must have the correct data and steps, to ensure that Cyara will execute the entire flow as expected, but we don’t care about confidence scores, etc., being perfect. Cyara passing or failing the script doesn’t matter since the tester must listen to the recordings for all steps involved with the change. It’s the manual review that determines whether the script has passed or failed.
The QA tester may wish to fully tune a script so that it looks good during a script demo with the customer/business owner; or they might want to fully tune the script before handing it off to the regression team, but tuning isn’t required to pass the script in the current Agile sprint.
As a side note, Agile testers enjoy having steps recorded when working with the IVR development team. It makes it very clear why they feel there is a defect and what the customer experience is. Our business owners also love having Cyara’s recordings to ensure the customer experience meets their needs.
5. Use a platform to manage test data
Carefully create and manage test data. Having a set of test data gives us control of the testing to ensure that data does not change while the tests are run. Years ago, we built a test database ourselves and populated it. Today, Cyara has its CX Model Editor which provides an environment for the design and documentation of customer experience features. It also has the capability to manage reusable test data and capture test details. Teams should not need to build their own database for test data, nor should they need to invent new test data for each Agile sprint.
Q: How do you deal with frequent design changes during the short timeline of an Agile sprint?
A: Adjusting to change was already a priority for us. With thousands of automated regression scripts and continual changes to the IVR code, we had to find a quick and easy way to keep our regression scripts up to date. The Cyara team helped us with this initially by pushing us towards the ‘prompt is a block’ approach. Which means that every prompt in the IVR is a block. If the business changes a prompt, we can update our block and all scripts using that block are instantly updated. That is fantastic for prompt updates, but — What if the call flow changes? For example, adding a prompt, deleting a prompt, inserting extra prompts and replies, etc. The change could literally impact thousands of scripts. To deal with this issue, we built custom code using the Cyara APIs that allows us to make mass changes to our test scripts.
For example, one of my automation resources, encountered a set of new prompts and a reply that would need to be inserted in over 300 scripts, with the new feature he was able to implement the change in about 10 minutes.
This same technique has been critical for our Agile testers. One of our first testers came to me with an issue, she just finished writing 32 scripts for execution that day, but at the daily stand up, the design was changed. Using our new feature, she was able to implement the change in all 32 scripts in just a few minutes. Having Cyara execute the scripts saves time in execution, having this mass update capability allows us to deal with the constant change that is Agile.
Q: What have been the results of using test automation in your Agile environment?
A: When we first started using Cyara, we knew there was tremendous potential to test faster. As I mentioned, this is a speed play for us. We are still learning, growing, and adapting to automation in the Agile environment, but we have seen plenty of success.
We now have 6 Agile teams using the Cyara platform and we are significantly faster in executing in-cycle new functionality tests. It has become core to our Agile practices and what we do. Whenever we have QA testers saying that they are not getting the expected speed improvements, we take a look and develop new best practices to keep us moving fast.
Watch Scott Anderson talk about the benefits of automated testing. And to learn more about how Cyara can help you with your Agile/DevOps transformation, contact us or get in touch with your Account Executive.
