MIT Technology Review is reporting on a new findings from METR, an AI evaluation nonprofit, that updates the organization's tracking of AI capability progression over time. The report finds that certain AI capabilities continue to advance along an exponential curve, while also revealing a meaningful gap between how well current models handle coding tasks versus tasks that demand genuine judgment.
To understand why this matters, it helps to know what METR is and why its work occupies unusual territory in the AI landscape. METR, which stands for Model Evaluation and Threat Research, is one of a small number of organizations dedicated to rigorously benchmarking what AI systems can actually do, as opposed to what their developers claim they can do. The organization has been building out what it calls a "tasks" framework — essentially a set of real-world challenges that measure autonomous AI performance over extended time horizons. This is distinct from most popular benchmarks, which test narrow knowledge retrieval or short-horizon problem-solving. METR's approach attempts to measure something closer to genuine capability: can a model pursue a complex goal, adapt when things go wrong, and reach a useful conclusion without constant human guidance?
The graph referenced in the headline has become something of a flashpoint in AI discourse precisely because it is easy to misread. When METR published earlier iterations of this trajectory data, some observers treated it as a simple validation of the most optimistic AI scaling narratives — the idea that capability improvements are reliable, predictable, and headed somewhere dramatic in the near term. Others dismissed it as hype dressed up in empirical clothing. Both readings missed the nuance. The graph tracks performance on a specific set of agentic tasks, meaning tasks that require a model to take sequences of actions over time, and the exponential trend it shows is real within that defined domain. It says nothing, without careful interpretation, about general intelligence or about domains the benchmark does not cover.
That interpretive gap is exactly what the new update addresses, and it does so in a way that complicates both the optimistic and the dismissive readings. The finding that models continue to improve exponentially on certain capability measures is significant. Exponential curves are easy to underestimate precisely because the gains look small in early stages and then become startling. If METR's trajectory holds, the practical implications for software development, research assistance, and automated reasoning pipelines could be substantial within a relatively short window.
But the accompanying finding — that models underperform on tasks requiring judgment relative to their coding performance — is arguably the more important signal for anyone thinking seriously about deployment. Coding is, in a meaningful sense, a constrained domain. There are syntax rules, test suites, and objectively verifiable outputs. Judgment tasks, by contrast, involve navigating ambiguity, weighing competing considerations, and reaching defensible conclusions in the absence of a ground truth. The gap METR is identifying suggests that the capabilities driving the exponential curve may be more brittle than headline benchmark numbers imply. A model that writes excellent code but struggles to decide which code to write, or when to stop, or whether the goal it has been given is the right one, is a more limited tool than it first appears.
This has real consequences for the people and organizations making decisions about how to integrate AI into consequential workflows. The enterprise technology sector in particular has been moving quickly to deploy AI agents in settings where autonomous judgment matters — legal analysis, financial modeling, medical triage support, strategic planning. The likely reading of METR's findings is that these applications are absorbing a capability that is less mature than the coding-focused benchmarks used to sell them might suggest. That is not an argument against deployment, but it is an argument for calibrated expectations and robust human oversight in exactly the domains where organizations may be most tempted to let models run unsupervised.
For AI developers, the report presents a different kind of pressure. The major labs have invested heavily in the narrative that scaling continues to produce reliable capability gains. METR's data partially supports that story, which the labs will emphasize. But the judgment gap gives critics and regulators a concrete, empirically grounded place to push back — and it gives safety researchers a sharper target for future evaluation work.
The things worth watching in the months ahead are several. Whether other evaluation organizations replicate or challenge METR's trajectory findings will matter considerably — a single organization's methodology, however rigorous, benefits from independent verification. How the major labs respond to the judgment gap specifically, whether through training approaches, architectural changes, or simply improved prompting guidance, will reveal something about how seriously they take evaluation as a check on their claims. And perhaps most consequentially, whether the policy community in Washington and Brussels begins to treat capability evaluation reports from organizations like METR as inputs to regulatory decisions, rather than technical curiosities, may determine how much real-world weight this kind of work ultimately carries.




