On 2 June 2026, Rob Willoughby and I presented at AI Native DevCon in London on how teams can measure what AI coding agents actually do—not only whether they produce a plausible final answer. Our central argument was that output-only evaluations miss the agent’s journey. A result may look correct while the agent ignores required instructions, skips an intended skill, or follows an unsafe path. Trajectory evaluations and instrumentation make those behaviors visible.
I connected the problem to large-scale evidence on AI adoption and software engineering productivity. Rob showed how task-level metrics—such as skill activation, instruction compliance, and coverage—can turn agent evaluation into an engineering discipline.
On 2 June 2026, Rob Willoughby and I presented at AI Native DevCon in London on how teams can measure what AI coding agents actually do—not only whether they produce a plausible final answer. Our central argument was that output-only evaluations miss the agent’s journey. A result may look correct while the agent ignores required instructions, skips an intended skill, or follows an unsafe path. Trajectory evaluations and instrumentation make those behaviors visible.
I connected the problem to large-scale evidence on AI adoption and software engineering productivity. Rob showed how task-level metrics—such as skill activation, instruction compliance, and coverage—can turn agent evaluation into an engineering discipline.
