What if your favorite AI leaderboard… was lying to you?
That’s exactly what a fascinating new report from OpenTools suggests. Using an unlikely but brilliant experiment—playing classic Pokémon games—researchers showed just how flawed popular AI benchmarks like MMLU and GSM8K really are when it comes to comparing language models. And the results are more than just surprising—they’re a wake-up call.
At the heart of the issue? We’re comparing apples and oranges without realizing it.
The Pokémon Problem
Researchers created an environment where different AI models had to play Pokémon games using natural language commands. The goal wasn’t just nostalgia—it was to test how models reason, adapt, and interact in real-world-like tasks. And here’s the twist: the models that scored highest on traditional academic benchmarks often didn’t perform best in the game.
For example, Claude 2 was the lowest scorer in MMLU but topped the Pokémon gameplay charts. Meanwhile, GPT-4 and Gemini, darlings of traditional benchmarks, didn’t shine in this practical setting. Suddenly, the kingpins of the leaderboard didn’t look so mighty.
As the researchers put it:
“Benchmarks may tell you how smart a model is in the classroom. But Pokémon tells you how it handles life outside.”
Why This Matters
Most people—even many developers—assume that a higher benchmark score means a better, smarter, more useful AI model. But what this experiment shows is that those tests are often narrow and don’t reflect real-world intelligence or problem-solving.
It’s the equivalent of hiring someone because they aced a standardized test—only to realize they can’t solve a real customer issue or adapt on the fly.
This study highlights the urgent need for more practical, scenario-based evaluation methods. We need benchmarks that reflect how models think, not just what they know.
Q&A: What People Are Asking
Q: Why did Claude 2 outperform GPT-4 in the Pokémon test?
A: Claude 2 seems better at following multi-step instructions, adapting to unexpected changes, and staying focused on objectives—skills that traditional benchmarks don’t test. This made it better at real-time problem-solving in the game environment.
Q: Should developers stop trusting AI benchmarks like MMLU?
A: Not entirely, but with caution. These benchmarks are useful for some types of evaluation but shouldn’t be the only measure of a model’s usefulness. Real-world tasks—like the Pokémon test—give a much clearer picture of performance.
Love this kind of AI insight? Sign up for our AI newsletter to stay ahead of the curve with stories that challenge assumptions, spark debate, and make you think differently about technology.
Got thoughts on which AI model you would bet on in a Pokémon battle? Drop a comment below. Let’s hear your take.
And if you’re working on AI evaluation, model deployment, or want help understanding what tools are actually right for your business—click here for consulting expertise tailored to your goals.
