How RAG Memory Works

Memo's memory system uses Retrieval-Augmented Generation (RAG) to automatically recall relevant context for every conversation. This page explains the technical pipeline from message to memory retrieval.

The Pipeline at a Glance

User message
    │
    ▼
┌─────────────────────┐
│ 1. Embedding        │  → Convert text to vector (nomic-embed-text-v1.5)
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ 2. Semantic Search  │  → Vector similarity (cosine) + FTS5 keyword search
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ 3. RRF Merge        │  → Reciprocal Rank Fusion — combine vector + FTS results
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ 4. Context Packing  │  → Inject top N memories into the system prompt
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ 5. LLM Response     │  → Model generates with memory context
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ 6. Persist          │  → Store the new interaction for future retrieval
└─────────────────────┘

Step 1: Embedding

Every message is converted into a dense vector (embedding) using nomic-embed-text-v1.5, a 137M parameter model specialized for semantic similarity. This model runs locally via the bundled llama.cpp server.

Message Chunking

Long messages are split into overlapping 300-word chunks with a 50-word overlap window:

// internal/memory/chunker.go
type Chunk struct {
    Text    string
    Index   int
    WordLen int
}

func ChunkMessage(text string) []Chunk {
    // Splits on word boundaries, 300 words per chunk, 50-word overlap
}

Each chunk is embedded and stored independently. Previously, long messages were stored as one block — information buried in the middle was often missed.

Combined Embedding

When storing, the embedding is computed from both the user message and the assistant reply concatenated. This means queries like "what did you suggest about pricing?" find the right memory even when the user message alone doesn't contain the answer.

Step 2: Hybrid Search

Memory retrieval uses two search strategies simultaneously:

Vector Search (Semantic)

Uses sqlite-vec with the vec0 virtual table — an approximate nearest neighbor (ANN) index:

SELECT rowid, distance
FROM memory_vec
WHERE embedding MATCH ?
ORDER BY distance ASC
LIMIT ?

Distance is computed using cosine similarity. Results are semantic matches — conceptually related content even when exact words differ.

Full-Text Search (Keyword)

Uses SQLite FTS5 with the unicode61 tokenizer:

SELECT rowid, rank
FROM memory_fts
WHERE content MATCH ?
ORDER BY rank
LIMIT ?

Each word is matched independently (not as a phrase). This catches exact terms, names, dates, and technical jargon.

Dynamic Candidate Pool

The retrieval candidate pool is dynamically sized:

candidatePool := min(max(topK * 5, 20), 100)

Previously fixed at 15 — far too small for large memory bases beyond a few thousand entries.

Conversation-Aware Retrieval

The last 3 user messages are appended as additional context. Short follow-ups like "remember what we said about that?" now find relevant memories instead of returning nothing.

Multi-Query Retrieval

For queries longer than 7 words, a second embedding is computed from the first 5 words as a topic anchor. Both embeddings are searched separately and merged via RRF, improving recall when the core topic is buried mid-sentence.

Step 3: Reciprocal Rank Fusion (RRF)

Results from vector search and FTS are merged using Reciprocal Rank Fusion:

RRF_score(d) = Σ 1 / (k + rank_i(d))

Where k = 60 (a damping constant that smooths rank differences). Memories appearing in both result sets are promoted to the top. A quality filter discards low-confidence matches below a threshold.

Step 4: Context Construction

Top N matching memories (typically 5–10, configurable) are formatted and injected into the system prompt:

[Relevant Memories]
- 2024-06-15: User decided on usage-based pricing with a free tier.
  (match: hybrid, importance: 5)
- 2024-06-10: User flagged that seat pricing feels wrong for a local-first tool.
  (match: vector, importance: 4)

Each memory includes its timestamp, match type ("vector", "fts", or "hybrid"), and importance score.

Step 5: Response Generation

The LLM receives the full prompt — system prompt + relevant memories + conversation history + user message — and generates a response informed by past context.

Step 6: Persistence

After the conversation turn, the exchange (user message + assistant reply) is:

1. Chunked (if long)
2. Embedded via the local model
3. Stored in data/memory/memory.db in three tables:
   • memories — raw text, metadata, importance score
   • memory_vec — vector embeddings (vec0 virtual table)
   • memory_fts — full-text search index (FTS5 virtual table)

All three tables are written atomically in a single transaction.

Data Storage

SQLite Schema

-- Text storage
CREATE TABLE memories (
    id TEXT PRIMARY KEY,
    content TEXT NOT NULL,
    timestamp TEXT NOT NULL,
    importance INTEGER DEFAULT 1,
    retrieval_count INTEGER DEFAULT 0,
    pinned INTEGER DEFAULT 0,
    soft_deleted INTEGER DEFAULT 0,
    deleted_at TEXT
);

-- Vector index (sqlite-vec vec0)
CREATE VIRTUAL TABLE memory_vec USING vec0(
    embedding FLOAT[768]
);

-- Full-text index (FTS5)
CREATE VIRTUAL TABLE memory_fts USING fts5(
    content,
    tokenize='unicode61'
);

Importance & Decay

Every memory has an importance score (1–5):

• Explicitly pinned memories (/remember) → importance 5, never decayed
• Frequently retrieved memories → promoted (+1 per retrieval, capped at 5)
• Old and unused memories → gradually downgraded
• Importance 0 for 7 days → soft-deleted

The decay cycle runs as a daily background job.

Memory Consolidation

A daily background process prevents near-duplicate accumulation:

1. Loads the 200 most recent memories with embeddings
2. Computes pairwise cosine similarity across all pairs
3. Identifies pairs with similarity ≥ 0.92
4. Sends each pair to the AI provider for merging into one consolidated entry
5. Soft-deletes the two originals after the merged entry is confirmed written

This runs after the decay cycle and never blocks chat responses.

Debugging & Inspection

Settings → Memory → Debug Search: Type any query to see the top retrieved memories with scores, timestamps, and match type badges.

Memory Analytics Panel shows:

• Total memories, pinned count, this week's additions, expiring soft-deletes
• Top 5 most-accessed memories by retrieval count

Commands

Command	Action
/remember <text>	Save a pinned memory with max importance
/forget <pattern>	Delete memories matching the pattern

Performance

Operation	Typical Latency
Embedding (300-word chunk)	~50–100ms
Vector search (10K memories)	~5–15ms
FTS5 search (10K memories)	~1–3ms
RRF merge	~1ms
Full pipeline (query → results)	~60–120ms

Performance scales logarithmically with the number of stored memories due to the vec0 ANN index. A memory base of 100K entries adds roughly 2× the search latency of 10K.