Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory

When an agentic language model needs to call a tool, it must first embed the tool’s schema into its prompt. As the number of available tools grows, this prefill step becomes quadratic in sequence length and quickly dominates the time‑to‑first‑token, turning a promising agent into a bottleneck. Nexus tackles this problem by separating the decision of which tool to use from the costly schema encoding step, allowing the model to route calls efficiently even when the tool registry swells to hundreds of entries.

What You Need to Know

The core idea of Nexus is a two‑stage pipeline. First, an INT8 semantic look‑aside buffer (SLB) stores compact embeddings of every tool’s description. A calibrated cross‑encoder margin gate compares the user query against these embeddings and selects the most relevant tool purely by retrieval. Because the SLB is fixed‑size and quantized to 8‑bit integers, the lookup cost is essentially constant regardless of how many tools are registered.

Second, once a tool is chosen, Nexus does not feed the full schema into the model’s KV‑cache. Instead, it generates the tool’s arguments over a compressed textual signature—a median length of just 19 tokens that captures the essential signature of the tool’s interface. This signature replaces the traditional splice‑all‑schemas KV cache, making the argument generation step depth‑independent: the model’s compute does not grow with the number of tools or with the depth of the model’s layers.

Why It Matters

In agentic workflows, the time‑to‑first‑token often determines whether the system feels responsive. When the tool registry exceeds a few dozen entries, the naïve approach of concatenating every schema into the prompt can overflow the model’s context window, forcing developers to prune tools or accept latency spikes. Nexus shows that routing accuracy stays near 89 % even as the registry scales to 250 tools, while the baseline concatenation method fails entirely beyond roughly 80 tools due to context‑window overflow.

Beyond latency, the method reduces memory pressure on the accelerator. By keeping the SLB in INT8 format and avoiding large KV‑cache slices for each tool, the overall footprint drops significantly, which can translate to lower power consumption and the ability to run larger models on the same hardware. This makes Nexus especially relevant for edge or latency‑sensitive deployments where every millisecond counts.

Key Details

• SLB stores INT8 embeddings of tool descriptions; lookup is O(1) with respect to registry size.
• Cross‑encoder margin gate calibrates similarity scores to maintain a stable retrieval threshold.
• Tool argument generation uses a compressed textual signature (median 19 tokens) instead of splicing full KV‑cache.
• Routing accuracy remains ~89 % when the registry grows from 10 to 250 tools.
• Baseline “concatenate‑all‑schemas” approach exceeds the model’s context window at ≈80 tools.
• Depth‑independent design ensures that argument generation cost does not increase with model depth.

What’s Next

The authors suggest extending the SLB to support dynamic tool addition without retraining, exploring hybrid routing that falls back to a lightweight classifier for low‑confidence cases, and evaluating Nexus on multimodal agents where tool signatures may include visual or audio components. Future work could also examine how the compressed signature length trades off against argument generation quality across different domains.

📌 Source: Arxiv Ai