The regression your telemetry will not catch
In April 2025, OpenAI shipped an update to GPT-4o and rolled it back three days later, because the model had turned into a flatterer — agreeing with users, validating doubts, endorsing whatever was put in front of it. What makes the incident worth studying is not the failure but the postmortem: by their own account, the offline evals were not broad enough to catch it, the A/B tests did not surface it, and no deployment check was watching for it. The best-resourced AI lab in the world learned about the regression the same way everyone else did — from users.
I read that postmortem with the discomfort of recognition. On a platform I am responsible for, the ordinary telemetry is thorough: request rates, latency percentiles, error budgets, alerts with names attached. Then there are the AI-shaped questions — what context did the model actually see, why did it answer the way it did, what did that answer cost — and the answers there are thinner. If a model update quietly changed the character of our output next Tuesday, who would catch it here? I know which of those two dashboards I would rather defend in front of a customer.
Drift fails politely
The reason this gap survives in otherwise well-instrumented organizations is that model drift does not fail the way software fails. When a service breaks, something pages: an exception, a spike, a flat line where traffic should be. When a model degrades, every request still returns 200. Latency is fine. The error rate does not move, because nothing in the pipeline knows what an error is. The output just gets slightly worse — and it gets worse in the specific way I keep warning about: plausible, confident, subtly wrong.
This is not hypothetical. Researchers at Stanford and Berkeley measured GPT-4’s accuracy on a simple task — identifying whether a number is prime — and watched it fall from 84 percent to 51 percent between the March and June versions of the same API. Same endpoint, same integration, no announcement your monitoring could subscribe to. A dependency changed underneath every product built on it, and for teams without their own quality signal, the first detector was a confused user. Half of your systems’ behavior can now change without a single line of your code being touched. That is a new operational fact, and most org charts have not absorbed it.
The trace is the only memory the system has
What should a team actually be recording? I have argued that an AI agent is a new hire with no memory — it rebuilds its understanding from the context window every session, then forgets. The operational consequence follows directly: the trace of what the model saw is the only record of why it acted. There is no mental model to interview after the incident, no author to ask what they were thinking. If you did not capture the context, the answer, and the cost, the explanation does not exist anywhere.
The tooling for this is genuinely commodity now. OpenTelemetry has GenAI conventions in active development — spans for model calls, token usage, tool invocations. Langfuse wires tracing into an existing stack in an afternoon; on Azure, AI Foundry does the same into Application Insights. In LangChain’s survey of 1,340 practitioners, 89 percent already have some form of observability wired into their agents. The industry has, by and large, bought the dashboards.
So the problem I am pointing at is not the missing tooling. Only 52 percent of those same teams run offline evals — we watch our systems far more than we test them, a gap I have written about separately. But there is a quieter failure sitting inside the 89 percent: the dashboard exists, and nobody owns it.
A dashboard nobody owns is a decision nobody made
I have made this argument before about escalations — most of them are ownership gaps wearing a technical costume — and AI observability is the same pattern on a new surface. Ask a team who gets paged when the checkout service is down and you get a name in one second. Ask who gets paged when the AI feature’s output quality drifts and you get a pause, then something about the platform team, then an honest admission that nothing pages, because no one ever defined the threshold that would.
That unowned dashboard is a bill nobody opens. The amount is printed right there — quality trending down, cost per request creeping up — and it still goes unpaid, because opening it is not anyone’s job. The organization has observability in the tooling sense and none in the sense that matters: a person who is accountable for noticing.
Closing that gap is not a platform project. It is four decisions, and a director can make them in a week.
- Name an owner for each AI feature’s quality. A person, not a guild or a rotation of everyone. The same rule that applies to every system worth having applies here: one accountable name.
- Put model quality in the pager rotation. Pick the two or three signals that would have caught your last bad output — a judge score on sampled traffic, a complaint rate, a cost ceiling — and set thresholds a human chose and wrote down. An alert nobody calibrated is an alert nobody trusts.
- Treat the prompt as a production artifact. It is versioned, it has an owner, and editing it is a deploy — reviewed, traceable, revertible. On most teams today, the prompt is the only load-bearing code that can be changed without either.
- Decide in advance who may turn it off. When quality drifts at a customer-facing feature, the expensive minutes are the ones spent discovering nobody knows whether they are allowed to roll it back. Write the answer down while nothing is on fire.
None of this requires new budget. All of it requires the thing the tooling vendors cannot sell: someone with authority deciding that model quality is a named person’s job.
The org chart question
The instinct, when AI observability comes up, is to evaluate tracing platforms — and the instinct is wrong in an instructive way, because the platform was never the decision. Wiring the spans is an afternoon. The leadership work is everything the afternoon does not include: which signals mean degrading, what threshold means act, whose name is on the page, who may pull the feature. Those are decision rights, and they are exactly the surface I keep finding most neglected in AI adoption.
You cannot manage what your agents will not tell you. But your agents only tell you what someone decided to ask — and that decision is sitting, right now, in an org chart near you, unmade.