Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations

The authors propose VoxPolyMem, an interaction-aware multimodal memory framework designed for multi-party spoken conversations, combining incremental speaker identification with a hierarchical memory structure consisting of interaction, fact, and participant memories. Retrieval is formulated as sequential decision-making where an agent rewrites queries and selects tools based on accumulated evidence. To optimize training, they introduce Evidence-Gain GRPO (EG-GRPO) utilizing round-wise credit assignment. Evaluated on VoxPolyBench, VoxPolyMem achieves an overall score of 85.0, outperforming the strongest baseline by 23.6 points, alongside high scores on Mem-Gallery and H2HMem-Multi.

VoxPolyMem achieves an overall score of 85.0 on VoxPolyBench, surpassing the strongest evaluated baseline by 23.6 points.

On Mem-Gallery and H2HMem-Multi, it scores 89.6 and 74.4, respectively.

It exceeds the strongest evaluated public memory baselines by over 8 points each.