ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understanding

Understanding long videos requires an agent to keep track of who or what appears, where they are, and how they change over time. Most current methods compress frames or rely on similarity‑based retrieval, which can mix up identities and propagate mistakes. ViSAGE introduces a memory system that binds entity cues across modalities and corrects itself as new information arrives.

What You Need to Know

ViSAGE builds an entity‑centric memory by first extracting visual, auditory, and textual features from each segment of a video. It then creates cross‑modal bindings that tie a specific identity—such as a person’s face, voice, and name—to a persistent memory slot. These bindings are updated continuously, allowing the system to retain fine‑grained details even when the video spans hours.

The self‑correcting mechanism works by comparing incoming evidence with the stored binding. If a mismatch is detected—say, a new face clip does not align with the stored voice‑face pair—the system flags the conflict and re‑estimates the entity’s representation using a weighted consensus of all available modalities. This reduces the chance that unrelated but semantically similar snippets overwrite the correct identity.

Because the memory is organized around entities rather than raw frames, retrieval is guided by identity keys instead of vague similarity scores. The authors show that this approach lowers entity confusion errors by roughly 30 % on benchmark long‑form video QA datasets, while keeping computational overhead comparable to existing segment‑wise methods.

Why It Matters

Long‑horizon tasks such as video summarization, surveillance analysis, or interactive storytelling depend on accurate, temporally grounded reasoning about individuals and objects. When an agent confuses one person for another, downstream answers become unreliable, leading to hallucinations or faulty decisions. ViSAGE directly addresses this failure mode by anchoring memory to verifiable, cross‑modal identity cues.

Beyond improving answer correctness, the framework offers a clearer audit trail: each memory slot can be inspected to see which modalities contributed to the current belief and how it has been corrected over time. This transparency is valuable for applications where accountability is required, such as medical video review or legal evidence processing.

Key Details

• Cross‑modal binding uses a contrastive loss to align face, voice, and text embeddings for the same entity.
• Memory slots are updated via an exponential moving average, weighted by confidence scores from each modality.
• A conflict detection module computes a modality‑wise disagreement score; if it exceeds a threshold, the slot triggers a re‑encoding step.
• The system maintains a short‑term buffer for recent segments and a long‑term archive for stable entity states.
• Experiments were conducted on the ActivityNet‑Long and Ego4D benchmarks, showing improved entity‑consistent QA scores without increased latency.
• Ablation studies reveal that removing either the cross‑modal binding or the self‑correction step drops performance back to baseline levels.

What’s Next

The authors plan to extend ViSAGE to handle dynamic entity creation and deletion, allowing the memory to grow or shrink as new characters appear or disappear in a video. They also aim to integrate the framework with end‑to‑end trainable agents, testing whether the self‑correcting memory can improve performance in reinforcement‑learning‑based video navigation tasks.

📌 Source: Arxiv Ai