Benchmark and eval methodology
“Benchmark and eval results depend heavily on setup choices and can be misleading across models”
10 people · 7 agree · 2 no clear side · 2 unclear · 13 views · 2026-09-22 to 2026-10-08
Who stands where on the statement above, from their own words. A person's side is shown only where two separate readings of the quote agree; otherwise they are under "Side unclear". This page does not say who is right.
Agree 7 people
- ✓artificialanalysis.ai
While model intelligence generally translates across use cases, specific evaluations may be more relevant for certain use cases.
While model intelligence generally translates across use cases, specific evaluations may be more relevant for certain use cases.
- ✓John Platt
People themselves will act like these LLMs and try to reward hack. It goes back to Goodhart’s law: any metric that becomes a target is no longer good as a metric.
- ✓Simon Willison
Comparing different model vendors by how well they draw a pelican riding a bicycle may not make much sense now (if it ever did)
- ✓nrehiew_
FrontierCode penalizes unnecessary changes, and higher effort produces scope creep
- ✓theo
@theo questioned using max reasoning for writing evals.
- ✓Thom_Wolf
this more likely reflects models recognizing cheating tests than a real behavior change
- ✓Piotr Migdał
I encourage you to use the same prompt with different models, or different harnesses.
No clear side 2 people
- ~Simon Willison
I'm still finding value in using them for comparisons of the same model families at different reasoning levels.
- ~Zvi Mowshowitz
You can obviously get around AI issues by doing oral exams, if you want it enough.
Side unclear 3 people
- ·Reddit commenters
speculated that the behavior may be a byproduct of reinforcement training or benchmark/test-centric fine-tuning
- ·Nate B Jones
Three prompts that turn your own recurring work into a test you can rerun every time a new model ships.
- ·Simon Willison
which results in a much less visually appealing heatmap since each square is either 100% or 0%
Week by week
- · 2: Zvi Mowshowitz ~, Piotr Migdał ✓
- · 5: Thom_Wolf ✓, Reddit commenters ·, Nate B Jones ·, artificialanalysis.ai ✓, Simon Willison ·
- · 5: artificialanalysis.ai ✓, John Platt ✓, Simon Willison ✓, nrehiew_ ✓, theo ✓
Built 2026-11-21 06:50 UTC from the news knowledge
graph by code. Every quote was checked against the source's own words; which side a quote takes is a model's reading,
made twice.