This article was originally published on QBox’s blog, prior to Cyara’s acquisition of QBox. Learn more about Cyara + QBox.
Cross-validation testing evaluates how well your chatbot performs against data it wasn’t trained on. Use it after your automated test scores reach acceptable levels to detect overfitting and identify gaps in your training data before deployment.
You’ve worked hard to make improvements to your chatbot model and it’s now scoring very well for correctness (and hopefully for confidence and clarity, if applicable) in automated tests. But your work hasn’t finished just because overall model correctness has reached 80% or more (a commonly used internal benchmark). The next step in your chatbot improvement journey now is to start cross-validation testing.
Cyara’s automated chatbot testing solutions allow you to assure quality at every stage of development.
Why perform cross-validation testing?
Cross-validation testing is a method used to evaluate how well a chatbot model generalizes to new, unseen data by testing it against variations not included in the original training set.
Cross-validation testing is recommended for several key reasons:
- Identifies gaps in your training data.
- Detects overfitting before deployment.
- Validates real-world performance against data the model hasn’t seen.
What is an overfit model?
An overfit model is a chatbot model that is very finetuned to its existing training dataset but performs poorly when faced with new data, even if it’s just a small digression from the training set.
It is important to look out for an overfit model because they can be deceptive and will lull the chatbot builder/trainer into a false sense of security. On the face of it, the model looks like it’s a success because the overall scores are high in the automated testing. But really, its predictive power is feeble, as the model has not gained much learning value from the existing training set to be able to apply its learned knowledge in the real world successfully.
Signs your chatbot model may be overfit
- Similar utterances repeated across training data expressing the same concept
- Limited vocabulary diversity
- Lack of synonyms (e.g., only “change” but not “update” or “amend”)
- Missing tense variations (e.g., “change” but not “changed”)
How to conduct cross-validation testing
Follow these steps to perform effective cross-validation testing:
- Prepare a separate labeled dataset with utterances not included in the original training data.
- Confirm the test data is created by someone not involved in the original build.
- Run the cross-validation test against your chatbot model.
- Compare cross-validation scores to your automated test results.
- Investigate any intents where scores differ significantly.
Interpreting your results
When you start running cross-validation tests, you should expect your overall correctness score to be a little lower than the automated correctness score, and anything up to 10% lower is considered acceptable (and natural, it’s simply not possible to think up every single permutation that your customers will use to express themselves within your chatbot model!). BUT…if you find your cross-validation test is more than 10% lower for overall correctness than your latest automated test, it could mean your model is overfit.
We often see client models that are overfitting, and sometimes you can tell by looking at the existing training data. But usually, it is not immediately obvious the model is overfitting until a cross-validation test is done, which is why this is a crucial next step in the chatbot improvement journey.
Frequently Asked Questions
Cross-validation testing is a method that evaluates how well your chatbot model performs against new utterances it wasn’t trained on, helping ensure it will work effectively with real users.
Cross-validation test after your automated test scores reach acceptable levels (typically 80% or higher correctness) and before deploying to production.
A drop greater than 10% compared to automated test scores typically indicates your model is overfit and may struggle with real-world user inputs that differ from the training data.
Cross-validation testing uses a separate dataset the model has never seen to check for gaps in training data and identify whether a chatbot model is overfit. It is a crucial next step after automated testing, especially once overall model correctness reaches 80% or more.
An overfit model is finely tuned to its existing training data but performs poorly when faced with new or slightly different user inputs. Despite appearing successful in automated tests with high scores, its real-world predictive power is feeble.
A drop of up to 10% in correctness compared to automated tests is considered acceptable and natural. A drop of more than 10% is a strong signal that your model may be overfit.
Overfitting can sometimes be spotted by looking at training data that contains many very similar utterances or limited vocabulary, such as using “change” repeatedly without including synonyms like “update” or “amend.” However, overfitting is often not obvious until a cross-validation test is run.
No, using your training data for cross-validation will produce inaccurate results. The cross-validation dataset must be separate from the data used to train the model.
To avoid model bias, the dataset should ideally be created by someone not directly involved in the initial chatbot build. Involving colleagues or others to express intents in their own words helps keep the data objective and diverse.
Aim for at least one test utterance for every training utterance per intent. For complex or high-frequency intents, increasing this to two or three times the training data volume helps ensure more robust coverage.
Cross-validation testing should be a regular part of your development cycle, particularly after major updates or when new intents are added. Consistent testing helps catch regressions and ensures that changes do not negatively impact the broader model.
