Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation

When evaluating language‑model agents, researchers often turn to a second language model as an automatic judge because running the true environment reward is too costly or unavailable at scale. These proxy judges need a scoring rubric that reflects actual task success, yet many existing approaches either hand‑craft the rubric or fine‑tune the judge’s weights. Both tendencies can reward fluent language even when the agent fails to achieve the goal, inflating performance estimates. The paper “Inducing Reward‑Free Judging Rubrics that Reduce Over‑Crediting in Agent Evaluation” (arXiv:2608.13564v1) proposes a method that learns the rubric itself from a small set of labeled trajectories, anchoring the judge’s scores in observable outcomes.

What You Need to Know

The core contribution is RubricForge, an algorithm that evolves a textual judging rubric through reflective evolution. Starting from an initial prompt that asks the judge to score an agent trajectory, RubricForge iteratively mutates and selects rubric variants based on how well their scores match a handful of ground‑truth labeled examples. The mutation operators edit the rubric’s language—adding, removing, or rephrasing criteria—while the selection step uses a simple correlation metric between the judge’s scores and the true binary success labels.

Because the rubric remains a natural‑language description, the resulting judge is still a language model that reads the rubric and the trajectory, then outputs a score. The process does not require gradient updates to the judge’s weights; instead, it searches the space of possible rubrics. Experiments on several benchmark tasks show that RubricForge‑induced rubrics reduce the tendency to over‑credit fluent but unsuccessful trajectories, yielding scores that correlate more closely with actual environment rewards than hand‑written or fine‑tuned baselines.

Why It Matters

Reliable automatic evaluation is essential when scaling agent development, especially for tasks where running the true reward function is slow, expensive, or impossible (e.g., real‑world robotics, interactive simulations, or costly human‑in‑the‑loop studies). Over‑crediting fluent failures can mislead researchers into believing an agent is more capable than it is, causing misallocation of effort toward superficial improvements. By grounding the judge’s rubric in observed outcomes, RubricForge offers a practical way to keep evaluation honest without needing massive labeled datasets or costly reward engineering.

Moreover, the method is lightweight: only a few dozen labeled trajectories are needed to start the evolutionary search, and the judge itself remains a black‑box language model that can be swapped out for newer models as they become available. This makes the approach attractive for teams that want a plug‑and‑play evaluation component that adapts to new tasks with minimal overhead.

Key Details

• RubricForge treats the judging rubric as a mutable string prompt.
• Evolutionary loop: generate rubric variants → score a small validation set → keep variants with highest rank‑order correlation to true labels.
• Mutation operations include synonym replacement, criterion insertion/deletion, and clause reordering.
• No gradient‑based fine‑tuning of the judge model is required; the judge remains a zero‑shot scorer guided by the evolved rubric.
• Evaluation on tasks such as ALFWorld, WebShop, and a synthetic navigation benchmark shows a 10‑15% reduction in over‑crediting compared to G‑Eval and fine‑tuned judges.
• The method is robust to different judge model sizes; improvements persist from 125M to 7B parameter models.

What’s Next

Future work could explore combining RubricForge with active learning to select the most informative trajectories for labeling, further reducing the annotation budget. Extending the evolutionary objective to optimize for calibration or robustness across distribution shifts would also make the induced rubrics more reliable in realistic deployment settings where the agent’s behavior may drift over time.

📌 Source: Arxiv Ai