Skip to content
Outrich Data

Glossary

Evaluation data

Evaluation data is a set of examples used to test a model, not to train it. Each example has an input and a known correct outcome, so the model's answers can be scored. Its value depends on the model never having seen it, which is why unpublished company data is sought after.

A model's developers need to know whether it actually improved. They measure this by running it on tasks with known answers and comparing results. If the test material was part of the model's training, the score reflects memory, not ability. Public benchmarks degrade for this reason, as their contents spread across the web and into training sets.

Private company records make strong evaluation data because they were never public. A set of real cases with recorded outcomes, such as tickets and their resolutions or claims and their decisions, lets a lab test whether a model reaches the result that a trained professional reached. Evaluation sets are usually much smaller than training sets, but they need more precise documentation: what the task is, what counts as correct and where the data has been. A license for evaluation typically forbids using the data for training, and buyers often want exclusivity so that no other model is trained on the same material.

A seller can raise the value of an evaluation set by documenting the outcome of each case clearly and by keeping a written record showing the data has not been shared elsewhere.

Find out if your data qualifies.

Eleven questions, about five minutes, no call required. You get an answer on screen and a written follow-up.

esc