There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items

When you look at a leaderboard that ranks large language models, the numbers often feel objective. Yet the way those scores are computed can shift dramatically depending on tiny choices in the evaluation setup—like the order of answer options or how the prompt is phrased. A recent study treats these choices as an experimental variable and shows that the instability is not random noise; it concentrates on specific test items that can flip a model’s relative standing.

What You Need to Know

The paper introduces a tool called the fragility grid. Researchers ran 12 open‑weight, instruction‑tuned LLMs from four families through the same 3,679 multiple‑choice items taken from four widely used benchmarks: ARC, HellaSwag, MMLU, and TruthfulQA. For each item they varied three aspects of the evaluation harness: the permutation of answer options, the exact wording of the system prompt, and whether the model’s answer was extracted from raw generation or from per‑option likelihood scores.

By measuring how each model’s score changed across these harness variations, they built a matrix that flags which items are most sensitive to harness changes. Rather than reporting a single aggregate variance, the fragility grid isolates the subset of questions where swapping option order or tweaking the prompt can cause a model to move from correct to incorrect (or vice‑versa). This reveals that a non‑trivial portion of the benchmark’s discriminative power hinges on arbitrary presentation details.

Importantly, the sensitivity is not evenly distributed. Certain items act as “config‑fragile” points: they consistently swing the outcome for multiple models, and they are often the very questions that separate one model from another in the published leaderboards. The authors show that removing or stabilizing these items reduces the observed variance in overall scores by up to 40 %.

Why It Matters

Leaderboard rankings are frequently used to guide research funding, model selection, and public perception of AI progress. If a model’s position can change simply because the answer choices were shuffled, then those rankings may reflect evaluation artifacts rather than genuine differences in capability. Recognizing which items are fragile helps developers design more robust benchmarks and avoid over‑interpreting narrow score differences.

Beyond benchmark design, the findings suggest that reporting a single aggregate score without describing the evaluation harness is insufficient. Transparent evaluation should include details about option ordering, prompt templates, and answer‑extraction method, or better yet, present performance averaged over multiple harness configurations. This practice would make comparisons more reliable and reduce the chance that a model appears superior solely due to a fortunate test configuration.

Key Details

• 12 LLMs from families including LLaMA, Mistral, Phi, and OLMo were evaluated.
• Each model answered 3,679 items across ARC, HellaSwag, MMLU, and TruthfulQA.
• Three harness variables were permuted: option order, prompt wording, and answer extraction mode.
• The fragility grid identified ~18 % of items as highly sensitive to at least one harness change.
• Stabilizing the top‑quartile of fragile items cut overall score variance by up to 40 %.

What’s Next

Future work could extend the fragility analysis to open‑ended generation tasks, where answer extraction is less straightforward but prompt sensitivity remains a concern. Additionally, benchmark creators might adopt a protocol that reports both the mean performance across a set of harness configurations and the worst‑case drop, giving users a clearer picture of a model’s reliability under realistic variability.

📌 Source: Arxiv Ai