Talks & conversations

From Vibes to Metrics: Why Evals Are Hard — AI Native DevCon London 2026

From Vibes to Metrics: Why Evals Are Hard — AI Native DevCon London 2026

At AI Native DevCon London 2026, Rob Willoughby and I explained why output-only evals miss how AI agents work—and how trajectory evals and instrumentation make performance measurable.

At AI Native DevCon London 2026, Rob Willoughby and I explained why output-only evals miss how AI agents work—and how trajectory evals and instrumentation make performance measurable.

On 2 June 2026, Rob Willoughby and I presented at AI Native DevCon in London on how teams can measure what AI coding agents actually do—not only whether they produce a plausible final answer. Our central argument was that output-only evaluations miss the agent’s journey. A result may look correct while the agent ignores required instructions, skips an intended skill, or follows an unsafe path. Trajectory evaluations and instrumentation make those behaviors visible.


I connected the problem to large-scale evidence on AI adoption and software engineering productivity. Rob showed how task-level metrics—such as skill activation, instruction compliance, and coverage—can turn agent evaluation into an engineering discipline.

On 2 June 2026, Rob Willoughby and I presented at AI Native DevCon in London on how teams can measure what AI coding agents actually do—not only whether they produce a plausible final answer. Our central argument was that output-only evaluations miss the agent’s journey. A result may look correct while the agent ignores required instructions, skips an intended skill, or follows an unsafe path. Trajectory evaluations and instrumentation make those behaviors visible.


I connected the problem to large-scale evidence on AI adoption and software engineering productivity. Rob showed how task-level metrics—such as skill activation, instruction compliance, and coverage—can turn agent evaluation into an engineering discipline.