From Vibes to Metrics: Why Evals Are Hard — AI Native DevCon London 2026

On June 2, 2026, Simon Obstbaum and Rob Willoughby presented at AI Native DevCon in London on how teams can measure what AI coding agents actually do—not only whether they produce a plausible final answer. The central argument is that output-only evaluations miss the agent’s journey. A result may look correct while the agent ignores required instructions, skips an intended skill, or follows an unsafe path. Trajectory evaluations and instrumentation make those behaviors visible.

Simon connected the problem to large-scale evidence on AI adoption and software engineering productivity. Rob showed how task-level metrics—such as skill activation, instruction compliance, and coverage—can turn agent evaluation into an engineering discipline.