ThinkReset: Learnable Intermediate Interface Construction for Bounded-Context Long-Horizon Reasoning

When a language model tries to solve a problem that needs many steps of reasoning, it often runs out of space in its context window. The early thoughts get pushed out, and the model either repeats information, gets stuck on mistakes, or guesses the answer just to finish before the window closes. This limits how well models can handle long, multi‑stage tasks such as complex math proofs or detailed code debugging.

What You Need to Know

The paper argues that the main obstacle is not how to compress the existing chain of thought or how to steer the model at test time, but the lack of a reusable “interface” that can stand in for the discarded history. ThinkReset introduces a learnable text‑based slot that lives alongside the reasoning trace. When the context window fills, the oldest part of the chain is replaced by this slot, which carries forward the essential information needed to continue solving the problem.

ThinkReset is trained with a reinforcement‑learning signal that rewards correct final answers while also encouraging the model to keep the intermediate slot informative. The authors show that, unlike standard outcome‑reward RL which pushes the model to guess when the window is nearly full, ThinkReset lets the model keep reasoning even after several resets, reducing premature guesses and improving accuracy on long‑horizon benchmarks.

Why It Matters

Many real‑world applications—such as automated theorem proving, multi‑step planning agents, or interactive coding assistants—require models to maintain a coherent line of reasoning over dozens or hundreds of tokens. Current approaches either truncate the chain, leading to loss of critical steps, or rely on costly external memory systems. A lightweight, learned interface that fits inside the existing transformer architecture offers a practical way to extend reasoning depth without redesigning the model or adding heavyweight modules.

By addressing the specific failure mode where reward shaping encourages early guessing, ThinkReset also improves the reliability of long‑chain RL training. This can make agents more robust in settings where the cost of an incorrect early answer is high, such as safety‑critical planning or scientific hypothesis generation.

Key Details

• ThinkReset adds a single learnable text token (or short span) that is updated at each reasoning step.
• When the context window reaches its limit, the oldest reasoning tokens are dropped and replaced by the current ThinkReset token.
• The token is trained via a hybrid loss: standard cross‑entropy on the final answer plus an auxiliary prediction loss that encourages the token to encode useful summary information.
• Experiments on GSM‑8K, MATH, and a synthetic long‑horizon puzzle show a 7‑12% absolute gain in accuracy over baseline chain‑of‑thought with truncation.
• Ablation studies reveal that removing the auxiliary loss leads to a drop similar to standard outcome‑reward RL, confirming its role in preventing premature guessing.

What’s Next

Future work could explore multiple ThinkReset slots to capture different types of intermediate information (e.g., algebraic symbols vs. procedural steps) and investigate how the interface interacts with retrieval‑augmented generation or external tool use. Scaling the approach to larger models and longer horizons will help determine whether a simple learned text interface can become a standard component for bounded‑context, long‑horizon reasoning.

📌 Source: Arxiv Ai