The question with an unsatisfying answer
Earlier this week I sat down to write about evaluation suites and started with the question I would put to any team shipping AI features: how do you know the output is correct? Then I made the mistake of pointing the question at my own platform — a product whose output informs decisions about people. My honest answer is that a person looks at it.
That answer used to sound responsible, and it is a real control. I have argued before that a human owns every merge, and on this project that line holds: nothing AI-assisted reaches a customer without human judgment on it. But somewhere between writing that sentence and living with it, I noticed what it quietly assumes. Human review is a quality bar. It is not a specification. For ordinary code we never confuse the two — we write tests that state what correct means and then let the machine check it, thousands of times, for free. For the non-deterministic parts of our systems, the parts where the same input can produce a different answer on Tuesday, most of us have nothing that plays that role.
Evals are that missing piece: the unit tests of non-deterministic software. We do not have them yet. This piece is what I now know about the gap, written from inside it.
The demo is a biased sample
The reason the gap survives is that the demo keeps telling everyone it isn’t there. A demo samples the happy path — the prompt someone rehearsed, the document that parses cleanly, the question the feature was built around — and the model performs beautifully on it, because plausibility is the one thing these systems produce reliably. The defects that matter never show up in a demo, because a demo selects for exactly the quality that makes them dangerous. The expensive defects are the plausible ones.
The industry numbers suggest this is the norm, not the exception. In LangChain’s State of Agent Engineering survey of 1,340 practitioners, 89 percent had observability wired into their agents while 52 percent ran evals. We watch our AI more than we test it. Observability is valuable, but it tells you something went wrong after a user was there to see it. A test tells you before. If a team shipped ordinary software on those terms — monitoring in production, no test suite — we would call it what it is: untested code in front of customers.
A minimal suite is small and boring
The word “evals” sounds like a research program, which is part of the problem, because what a first eval suite actually looks like is almost embarrassing. Five to ten golden examples — real inputs, and what a correct response must and must not contain — are enough to start. Fifty to a hundred catch the obvious failures. Rechat, the case in Hamel Husain’s essay that made much of the industry take this seriously, runs hundreds of assertions as plain unit tests: one of them is a regex checking that internal UUIDs never leak into a customer-facing answer. That is the level of sophistication required to begin. It is unglamorous work, which is usually the sign it is the right work.
The habit that turns the toy suite into a real one is a single rule: every failure that reaches production becomes a test case. Your suite stops being a sample of what you imagined going wrong and becomes a record of what actually did. Wire it into CI — the tooling is already a commodity; Microsoft ships a GitHub Action that runs eval suites with significance testing so a noisy model run doesn’t fail your build — and a model upgrade or prompt edit that quietly breaks last month’s fix never leaves the pull request. Without that gate, every prompt change is an unpriced bet, and a team shipping AI features makes several of those a week.
The judge needs auditing too
At some point assertions run out — summaries, advice, anything where correctness has a texture no regex can hold — and the tempting shortcut is to have a model grade the model. It works, within limits, and the limits deserve to be named, because an LLM judge is a measurably biased instrument. Swap the order of two answers and the verdict flips in 10 to 30 percent of comparisons. Let a model judge output written by itself and it inflates the score by around ten percent. These are known defects with known mitigations — randomize the order, calibrate the judge against a set of examples that humans actually labeled — but they only get mitigated when someone decides the judge is itself a system under test.
The leadership translation is short. An unaudited judge is a review you outsourced without deciding to, and whoever left it unaudited made a quality decision nobody signed off on.
Writing the eval is writing the spec
Here is what building the golden set actually forces, and it is the real reason to start even if you never automate a single check. To write down ten inputs and their correct outputs, someone has to say what correct means for this feature — which cases must always work, which failure would be embarrassing, which would be harmful. Sit with a team and try it. In my experience the model is rarely the obstacle; the obstacle is discovering the organization never agreed on the answer. That is not a machine-learning problem. That is the oldest problem I write about — intent that was never made clear enough to work from — now with a probabilistic system amplifying the ambiguity.
So that is the plan I am holding myself to on this project: one feature, a first golden set this quarter, and human review staying exactly where it is, as the backstop. For a platform whose output touches decisions about people, I want the spec to exist before the speed does. The order matters more than the tooling.
Nobody in your organization would accept “it compiles” as a quality bar for software. “It looked good in the demo” is the same sentence wearing better clothes — and most of us, my own team included, have been accepting it since the day these features shipped. The teams that stop accepting it will not look like they slowed down. They will look like the only ones who know what their systems are doing.