Learning Compositional Meta-Routing for Agentic Workflows: An Executable Benchmark

When an AI agent faces a request, it does more than pick a final answer; it must decide what steps to take before arriving at that answer. Should it fetch data, run a snippet of code, ask a specialist model, or simply respond directly? This decision-making process—often called routing—has been studied mostly in isolation, focusing on choosing a single tool or a retrieval depth. A new paper introduces an executable benchmark and a budget‑aware meta‑router that learns to compose heterogeneous operations directly from raw task text, giving a clearer picture of how agents can plan multi‑step workflows.

What You Need to Know

The authors build a benchmark that contains 216 training, 72 development, 108 held‑out test, and another 108 locked lexical‑shift challenge tasks. These tasks span three domains: data analysis (e.g., computing statistics from CSV files), frozen‑corpus research (answering questions using a static knowledge base), and document processing (extracting or transforming information from PDFs or HTML). Each task is designed so that the correct outcome can be verified automatically after the proposed sequence of operations runs, eliminating subjective grading.

The proposed meta‑router does not merely select a single endpoint. Instead, it predicts a sequence of actions—such as “retrieve evidence → execute Python → verify result”—conditioned on the input text and a budget constraint that limits total computation or cost. The router is trained with reinforcement learning, where the reward reflects both task success and adherence to the budget. By learning from the benchmark, the router discovers compositional patterns that generalize across domains, rather than memorizing isolated tool choices.

Why It Matters

Current agentic systems often rely on hand‑crafted pipelines or simple heuristics for deciding when to call a tool, retrieve information, or invoke a specialist model. Those approaches can be brittle: a small change in the task wording may cause the system to pick an inappropriate step, leading to wasted computation or incorrect answers. A learned meta‑router that adapts its plan to the actual request offers a more flexible and scalable way to build agents that can handle diverse, real‑world workloads without constant manual redesign.

Moreover, the benchmark’s lock‑held lexical‑shift challenge tests whether the router can maintain performance when the surface form of the input changes while the underlying intent stays the same—a common scenario in production systems where users phrase questions differently. Success on this split indicates that the router learns semantic routing policies rather than superficial keyword matching, a crucial property for robust agentic behavior.

Key Details

• Task composition: The benchmark defines a set of primitive operations (direct answer, decomposition, retrieval, code execution, specialist delegation, verification) that can be chained arbitrarily.
• Budget awareness: Each action carries a cost estimate; the meta‑router optimizes for success under a user‑specified budget, encouraging efficient plans.
• Automatic verification: After executing the predicted sequence, a deterministic checker compares the result to the ground truth, enabling fully objective evaluation.
• Training scale: 216 training examples provide sufficient variety for the router to learn cross‑domain patterns while keeping the benchmark manageable for rapid experimentation.
• Lexical‑shift split: The 108 locked challenge tasks rephrase the same intents with different vocabulary, testing generalization beyond surface form.
• Reinforcement learning objective: The router is trained with a reward that combines task accuracy and a penalty for exceeding the budget, promoting Pareto‑efficient policies.

What’s Next

Future work could extend the benchmark to include longer‑horizon tasks that require iterative loops or conditional branching, pushing the meta‑router toward more complex planning. Additionally, integrating the router with large‑scale language models as the underlying operation specialists may reveal how learned routing scales when the primitive actions themselves become more capable. Such extensions would help bridge the gap between today’s modular agents and the flexible, general‑purpose assistants envisioned for next‑generation AI systems.

📌 Source: Arxiv Ml