HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understanding

Comprehending extended audiovisual experiences remains a fundamental challenge for computational systems. Current approaches struggle with temporal integration and cross-modal associations that humans accomplish effortlessly through hippocampal-cortical networks. We introduce HippoMM, a biologically-inspired architecture that transforms hippocampal mechanisms into computational advantages for multimodal understanding. HippoMM implements three key innovations: (i) hippocampus-inspired pattern separation and completion specifically engineered for continuous audiovisual streams, (ii) explicit short-to-long term memory consolidation that transforms perceptual details into semantic abstractions, and (iii) novel cross-modal associative retrieval pathways enabling modality-crossing queries. Unlike existing retrieval systems with rigid indexes, HippoMM dynamically forms integrated episodic representations through adaptive temporal segmentation and dual-process memory encoding. Evaluations on our challenging HippoVlog benchmark demonstrate that HippoMM significantly outperforms state-of-the-art approaches (78.2% vs. 64.2% accuracy) while providing substantially faster response times (20.4s vs. 112.5s). Our results demonstrate that translating hippocampal principles offers a promising pathway for robust and efficient long-form audiovisual AI.

HippoMM introduces a novel architecture inspired by the human hippocampus to handle long audiovisual events. The system consists of three main components:

1. Pattern Separation and Completion: Specialized neural networks process continuous audiovisual streams, separating distinct events while maintaining temporal coherence.
2. Memory Consolidation: A hierarchical memory system that transforms raw perceptual inputs into semantic representations through progressive abstraction.
3. Cross-modal Retrieval: Novel attention mechanisms enable efficient querying across modalities while maintaining temporal context.

The architecture dynamically segments input streams using adaptive temporal windows and implements a dual-process encoding strategy that balances detailed recall with efficient retrieval.