METR Chart Shows 2.6x Runtime Gap Between Top AI Models

⬤ A fresh METR efficiency chart is making waves by revealing just how differently top AI models perform when speed actually matters. The chart tracks how much ground each model covers relative to the time it burns through during complex evaluation tasks. What jumps out immediately: Anthropic's Claude models are dominating the efficiency game, particularly in coding and reasoning workloads where runtime can make or break real-world applications.

⬤ Claude 4 Opus, Claude 4.5 Opus, Claude 4.1 Opus, and Claude 4 Sonnet all cluster at the top of the efficiency rankings, managing to tackle longer task horizons without dragging out benchmark completion times. Meanwhile, other heavyweight models aren't keeping pace despite their raw power. The standout stat: GPT-5.1 Codex Max takes roughly 2.6 times longer to finish the full METR evaluation compared to Claude's top performers, a gap that's hard to ignore when you're thinking about production deployments.

⬤ The chart doesn't just spotlight Anthropic. It also tracks DeepSeek V3, Gemini 2.5 Pro Preview, Grok-4, Qwen 2.5, and Kimi k2-thinking. Some of these show modest gains over earlier versions, but there's a clear pattern emerging: throwing more tokens or inference budget at a problem doesn't automatically buy you better time efficiency. The bigger takeaway? Pure runtime or total cost numbers miss the point when benchmark tasks themselves vary wildly in complexity and duration.

⬤ What's missing from the picture is the kind of granular data that would really settle the efficiency debate. To make truly meaningful comparisons, you'd need per-task breakdowns showing inference time, cost, and success rates measured against human performance baselines. METR supposedly has this level of detail, but it hasn't been made public in a format that allows for that kind of analysis. The chart makes one thing clear though: as AI models continue scaling up, measuring real-world efficiency, not just theoretical capability, is becoming the critical question no one can afford to ignore.