MIRIX: Revolutionizing Long-Term Memory in AI Agents with Modular Multi-Agent Architecture

Addressing the Memory Gap in LLM Agents

Recent advances in Large Language Model (LLM) agents have enhanced their ability to perform complex tasks. However, their capacity to remember and reason over user-specific information across time remains limited. Most LLM-based agents are stateless, meaning they cannot maintain context beyond a single prompt, which restricts their effectiveness in real-world applications that demand consistency and personalization.

Introducing MIRIX: A Modular Multi-Agent Memory System

MIRIX AI presents MIRIX, a modular system designed to provide robust long-term memory for LLM-based agents. Unlike traditional text-only memory systems, MIRIX incorporates multiple structured memory types across different modalities, including visual data. It operates on a coordinated multi-agent architecture that manages various memory components.

Core Architecture and Memory Components

MIRIX consists of six specialized memory components, each managed by a dedicated Memory Manager:

Core Memory: Stores persistent information about the agent and user, divided into ‘persona’ (agent's profile, tone, behavior) and ‘human’ (user details such as name and preferences).
Episodic Memory: Records time-stamped events and user interactions, including event type, summary, details, participants, and timestamps.
Semantic Memory: Contains abstract concepts, knowledge graphs, and named entities organized by type, summary, details, and source.
Procedural Memory: Holds structured workflows and task sequences, typically formatted in JSON for easy manipulation.
Resource Memory: Keeps references to external resources like documents, images, and audio with titles, summaries, types, and links.
Knowledge Vault: Secures sensitive data such as credentials and API keys with strict access controls and labeling.

A Meta Memory Manager oversees these components, enabling smart message routing, hierarchical storage, and efficient retrieval. Additional agents handle chat and interface functions within this system.

Active Retrieval and Interaction

MIRIX features an Active Retrieval mechanism that autonomously infers the topic from user input, fetches relevant data from all memory components, and tags this data for inclusion in system prompts. This reduces reliance on outdated model knowledge and strengthens response accuracy.

Multiple retrieval strategies such as embedding_match, bm25_match, and string_match ensure precise, context-aware access to memories. The architecture is flexible for future retrieval tool integration.

Implementation and Usage

MIRIX is implemented as a cross-platform assistant using React-Electron for the UI and Uvicorn for backend APIs. It captures screenshots every 1.5 seconds, filtering out redundant images, and updates memory in batches after collecting about 20 unique screenshots.

Uploads to the Gemini API are streamed, allowing visual data processing with latency under 5 seconds. Users interact via a chat interface that dynamically utilizes memory components to provide personalized, contextually relevant answers. Semantic and procedural memories are displayed as expandable trees or lists for transparency.

Performance on Benchmarks

MIRIX excels on both multimodal and conversational benchmarks:

ScreenshotVQA: Outperforms retrieval-augmented generation baselines by 35% in accuracy while reducing storage needs by 99.9% compared to text-heavy methods.
LOCOMO: Achieves 85.38% average accuracy on long-form conversation memory, surpassing strong open-source systems by over 8 points.

Its modular design supports high performance across different inference scenarios.

Future Applications and Memory Marketplace

MIRIX supports lightweight AI wearables such as smart glasses through its efficient architecture, enabling hybrid on-device and cloud memory handling. Use cases include meeting summarization, location and context recall, and dynamic user habit modeling.

A novel Memory Marketplace allows secure memory sharing, monetization, and collaborative personalization with fine-grained privacy controls, encryption, and decentralized storage to maintain user data sovereignty.

Frequently Asked Questions

What sets MIRIX apart from other memory systems? MIRIX offers compositional multi-component memory with multimodal support and a multi-agent retrieval structure for scalable, accurate long-term memory.

How does MIRIX manage low-latency visual memory updates? Streaming uploads with Gemini APIs enable under 5-second latency during active sessions.

Is MIRIX compatible with closed-source LLMs like GPT-4? Yes, MIRIX functions externally and can augment any LLM regardless of architecture or license.

For more details, check out the Paper, GitHub, and Project pages.