Voice Memory for Agentic Speech Recognition

Analysis

Why This Paper Matters

This paper addresses a critical pain point in modern speech recognition: the tendency of generative error correction (GER) to over-correct, often breaking correct tokens. Voice Memory introduces a simple yet effective inference-only scheme that leverages a frozen corrector and a score-gated optimizer to reduce over-correction while improving accuracy. The approach is particularly significant because it requires no weight changes, making it auditable and portable across different corrector families. This is a practical solution for AI practitioners who need to deploy ASR systems in diverse domains without costly retraining.

The paper's focus on "restraint" as an operative skill is a refreshing departure from the trend of ever-larger models. By coupling a listener and thinker only through a memory file, Voice Memory maintains the learned skill in a transparent, editable form. This aligns with the growing demand for interpretable and controllable AI systems, especially in enterprise settings where compliance and debugging are paramount.

Technical Contributions

• Listener-Thinker Architecture: Decouples ASR (listener) from correction (thinker) via a shared memory file, enabling zero-weight-change adaptation.
• Score-Gated Optimizer: Asynchronously revises the memory file through bounded edits, accepting only edits that strictly improve a held-out score, preventing degradation.
• Inference-Only Correction: The frozen corrector reads the memory per utterance and decides whether to act on the hypothesis or abstain, keeping the 1-best as fallback.
• Over-Correction Reduction: Reduces the rate of broken correct tokens from 64% to 35% on financial news, demonstrating the value of restraint.
• Zero-Parameter Addition: The memory adds no parameters to the inference path, ensuring minimal overhead.

Results

Voice Memory achieves a weighted word error rate (WER) reduction from 8.36% to 7.52% across ten HyPoradise domains, with further improvement to 7.47% when three in-context examples are added. Notably, no dataset regresses below its 1-best baseline. Gains are most pronounced where recoverable headroom is largest: air-travel commands improve from 8.40% to 3.40% (a 59.5% relative reduction), and CHiME-4 noisy far-field speech improves from 12.69% to 10.46% (a 17.6% relative reduction). The memory also transfers across corrector families, demonstrating robustness.

Significance

Voice Memory offers a practical, auditable, and portable method for improving ASR accuracy without retraining. Its zero-parameter inference path and ability to transfer across corrector families make it highly suitable for production systems that require domain adaptation and reliability. The paper's emphasis on restraint over unconstrained correction provides a valuable lesson for the broader AI field: sometimes less is more. This work could influence future designs of agentic systems where memory-based coupling replaces weight-based fine-tuning, enabling safer and more transparent AI deployments.