Your eval suite is the product

Every AI project starts with the same demo: a document goes in, a surprisingly good answer comes out, the room nods. Six weeks later the same system is in production and someone asks why it has started inventing policy numbers. The demo never had an answer for that. The eval suite did.

We treat model behaviour as software behaviour. That means a versioned test set, explicit graders, a threshold that blocks release, and a dashboard that shows whether today is better than yesterday. It is not glamorous work. It is the difference between a product and a magic trick.

Start with the failures you can name

Before writing a single prompt, list the ways the system must not fail. For a clinical intake agent: never invent a diagnosis, never omit a payer requirement, never present a low-confidence extraction as certain. Those sentences become test cases. A useful first suite is thirty examples, not three thousand.

Grade the trail, not just the answer

A correct answer produced by an unsafe path is a latent incident. Log the retrieved passages, the tools called and the arguments used. Then assert on the trajectory: did it check authorisation before querying the record? Did it cite a source that exists? Did it stop when confidence dropped?

Make the threshold a business decision

The right pass rate is not 100%. It is the point where the cost of a false positive is lower than the cost of a human doing the task. Write that number down with the person who owns the risk, and let it change the architecture — a 99.5% requirement usually means a human review step, not a better prompt.

The teams that win with AI are not the ones with the most impressive launch demo. They are the ones who can answer, on any given Tuesday, whether the system is still working.