Meta's Latest Model Accused of Benchmark Gaming at the Cost of Real-World Usefulness
François Chollet, creator of the ARC-AGI benchmark and one of the field's most credible voices on model evaluation, publicly called Meta's newest model a disappointment, arguing it is overoptimized for public benchmark performance at the expense of genuine utility.
8. Meta's Latest Model Accused of Benchmark Gaming at the Cost of Real-World Usefulness
François Chollet, creator of the ARC-AGI benchmark and one of the field's most credible voices on model evaluation, publicly called Meta's newest model a disappointment, arguing it is overoptimized for public benchmark performance at the expense of genuine utility. The critique lands with particular weight given Chollet's specific expertise: his work on ARC-AGI was built precisely to resist the kind of surface-level pattern matching that inflates benchmark scores without reflecting actual reasoning capability. Chollet framed sound evaluation methodology as a "core competency" that separates credible AI labs from pretenders.
The consequences here extend beyond a single model release. Meta has been positioning its Llama family as a serious rival to OpenAI and Google in the open-weights space, and a credibility hit on evaluation integrity undermines that positioning at a structural level. If Llama's strong benchmark numbers don't translate to practitioner value, enterprise developers and researchers who adopted it based on those numbers will recalibrate their trust, potentially shifting workloads toward alternatives like Mistral, Qwen, or Google's Gemma line. The real losers are Meta's developer relations efforts and the downstream startups that built product bets on the assumption that reported benchmark gains meant capability gains.
This is part of a widening fracture in AI evaluation credibility. Labs across the board face increasing pressure to post competitive numbers on shared leaderboards, creating systematic incentives to overfit training toward those specific benchmarks. Chollet's public call-out signals that the informal peer-review function of the research community is activating, and that labs releasing models without credible third-party evaluation pipelines will face growing reputational risk from exactly the kind of named, expert criticism that sticks in developer memory.
Source: https://twitter.com/fchollet/status/2042004767585751284