SchemaRouter: Field-Aware Tool Routing for Efficient Heterogeneous Agentic RAG

When developers build retrieval‑augmented generation systems that pull from many data sources, they often face a trade‑off: expose every possible tool to the language model and risk sending too much data, or rely on vague similarity scores and miss needed fields. Both choices can inflate latency, increase token consumption, and leave gaps in the answer. The SchemaRouter paper proposes a thin routing layer that decides, before any call is made, exactly which tools and which specific fields are required to satisfy a user query.

What You Need to Know

SchemaRouter treats every API endpoint, database table, vector store collection, and graph node as a node in a schema graph. Edges capture relationships such as parameter‑to‑field mappings, unit conversions, provenance tags, and licensing constraints. This graph is constructed once from the available tools and stored as a lightweight lookup structure, not as a massive prompt.

When a query arrives, a small language model (typically a few hundred million parameters) reads the query and the schema graph to produce a tool plan. The plan lists the exact endpoints to invoke and, for each, the subset of response fields that must be returned. Because the plan is field‑aware, the system never asks for columns or properties that are irrelevant to the answer, keeping payloads small.

The routing step adds only a few milliseconds of overhead. Experiments in the paper show that, compared with naïve vector‑similarity tool selection or full‑tool exposure, SchemaRouter reduces average token usage by 30‑45 % and cuts latency by a comparable margin, while maintaining or improving answer accuracy on benchmark tasks that require heterogeneous data.

Why It Matters

Many production RAG pipelines already orchestrate dozens of services—internal CRM APIs, public datasets, embeddings stores, and knowledge graphs. Sending unnecessary data not only wastes compute budget but can also breach privacy or licensing rules if fields containing sensitive information are fetched unintentionally. A routing mechanism that knows exactly what is needed helps teams stay within cost limits and compliance boundaries without manually curating tool lists for every new query.

Beyond cost savings, field‑aware routing improves the reliability of generated answers. When the model receives only the pertinent columns, it is less likely to be distracted by irrelevant noise, which can lead to hallucinations or incorrect inferences. For applications where answer precision is critical—such as medical decision support or financial reporting—this precision gain translates directly into trustworthiness.

Key Details

• The schema graph is built from OpenAPI specifications, database schemas, and store metadata, then serialized as a compact adjacency list.
• A 2‑layer transformer with ~80 M parameters serves as the planner; it is fine‑tuned on a synthetic dataset of query‑to‑plan pairs generated from the schema.
• During inference, the planner outputs a structured JSON plan that includes endpoint URLs, HTTP methods, parameter values, and a field mask for each response.
• Field masks are enforced by a lightweight wrapper that strips unwanted keys before the data reaches the main LLM, ensuring no extra tokens are processed.
• Ablation studies show that removing the unit‑conversion edges increases token waste by ~12 %, while dropping provenance tags leads to occasional license‑policy violations in simulated audits.

What’s Next

The authors plan to release the schema‑graph builder as an open‑source library and explore integrating license‑aware reasoning directly into the planner, so that the system can automatically refuse to call endpoints whose usage would violate a user’s data‑usage policy. They also intend to test SchemaRouter on larger, cross‑domain agentic frameworks where tools number in the hundreds, to verify that the routing layer scales linearly with graph size.

📌 Source: Arxiv Ai