“How do you know how many ports you need to run a regression testing campaign?” A couple of guys on our pre-sales team were asked to weigh in on this question recently regarding Cyara Replay, our regression testing solution, for a proposal to a prospective customer. [UPDATE: Please read about Cyara Velocity and our regression testing solutions.] So, if you do regression testing, you may be interested in the process and the information we require to provide an accurate answer.
Let’s start with why you need these ports. You need the ports because regression testing is a big part of your life, if your job is to keep your IVR up and running. Ports are used to run test cases through your IVR to test your call flows. You change things in your IVR, right? Well, after a change, you have to test to make sure you didn’t break anything and to be sure that the changes you made work as expected. Regression testing ensures that you don’t expose your customer to errors that lead to poor customer experiences. Regression testing can also save you money by identifying errors early, when they can be fixed more easily, instead of in production when they may have caused a significant and expensive issue such as causing several hundred calls to fall out of the IVR unexpectedly.
Planning Your Testing
Now, think about all of the different ways a customer can interact with your IVR. You want to be sure all of your call flows work, so you will need to run test cases to cover all of the scenarios. That’s why it’s not unusual for a company to have thousands of test cases for regression testing.
For testing, you can choose to break these test cases up into smaller batches across your development cycle to make them more manageable. A typical scenario for a large IVR would be 4,000 to 6,000 test cases run over a period of maybe every two weeks, or even every month. Depending on how much rework you are anticipating for your code changes, these tests could occur multiple times during a release cycle.
From a technical perspective the number of ports needed is based on your average call duration, the number of test cases you want to execute per test cycle, and your test cycle duration. Once you have a feel for these three parameters, theoretically it’s simple arithmetic to calculate the number of ports required for regression testing.
There are a number of websites that have free Erlang calculators that you can use to estimate call center traffic. There’s absolutely no need, however, to overthink this calculation. It’s not as critical as calculating the number of seats needed for a call center to keep queue times low or the number of ports needed in an IVR to prevent busy signals. First of all, a regression testing campaign by design limits both the rate of calls coming into the IVR and the total number of concurrent calls. Second, a regression testing campaign won’t create congestion in your IVR. That’s why, even a large campaign may require no more than 20 ports.
Getting Started
To get started, you need to collect a couple of key data points. This is arithmetic, but it’s pretty simple.
What is the average length of your test case calls?
As you create your test cases, you will probably run a validation to determine how long the call would be. For a simple example, we can look at 5 test cases:
- Test Case A = 123 seconds
- Test Case B = 55 seconds
- Test Case C = 195 seconds
- Test Case D = 34 seconds
- Test Case E = 185 seconds
If we add up the estimated time for each of our test cases, we get 592 seconds. The average (when we divide by the number of test cases) is 118 seconds. Next, we should add to this the time needed for a port to cycle (the time needed for a port to become ready for the next call once a call ends). Let’s assume a port cycle time of 4 seconds. So our average call length per test case is 122 seconds.
What is the ideal time for running a regression testing campaign?
The ideal time to run your campaign depends on your company’s schedule and peak times. Some companies will start a testing campaign near the end of the workday and run it overnight, and they may choose to break up their testing over multiple days.
One of the largest telecommunications providers in the world has migrated to continuous integration with rapid Agile sprints. They literally push out new contact center code twice a week and use the REST API to import test cases and maximize their efficiency. This company has cut their testing resources to a third of what was needed previously and has reduced the time needed for full regression testing from 2 days to 90 minutes.
How Much Should You Test?
This may be the most important question you need to answer. Some companies place a higher priority on testing than others do. So, to a certain extent, the amount of testing you do may be determined by your company’s testing practices. Companies that deliver “best in class” customer experience strive for 100% test coverage and realize that the only way they can get to 100% is with automated testing. Why is 100% coverage important? That’s an easy answer. Any errors your customers find may just be in the test cases you decided not to run.
How Many Ports do You Need?
This is where things get a little more complex; with the information above including the coverage you require and upon understanding your specific scenario our team will work with you to calculate the optimum number of ports you require.
Read more about the Cyara Platform, Cyara Velocity, and our regression testing solutions, and contact us with your questions.
