Memory Search: Embeddings and Hybrid Retrieval

memory_search pulls relevant notes out of your memory files, even when the phrasing in the query doesn't match the original text. It splits memory into small segments and searches those with embeddings, keywords, or a combination of the two.

Quick start

OpenAI embeddings are the default choice for OpenClaw. If you want a different provider, you must specify it explicitly:

{
  memory: {
    search: {
      provider: "openai", // or "gemini", "voyage", "mistral", "bedrock", "local", "ollama", "lmstudio", "github-copilot", "openai-compatible"
    },
  },
}

A custom models.providers.<id> entry can also be referenced by provider (such as ollama-5080), provided that entry defines api as "ollama" or another provider id that has a memory embedding adapter.

For local embeddings that don't need an API key, install and set up the official llama.cpp provider, then configure provider: "local":

openclaw plugins install @openclaw/llama-cpp-provider

Pick llama.cpp once during interactive setup. OpenClaw then installs a verified llama-server, pulls down the embedding GGUF, and writes out the managed service configuration.

Certain OpenAI-compatible embedding endpoints expect asymmetric input_type labels, for instance "query" when searching and "document"/"passage" for indexed chunks. Configure these through queryInputType and documentInputType; the Memory configuration reference has more details.

Supported providers

How search works

OpenClaw executes two retrieval paths simultaneously and then combines the outcomes:

flowchart LR
    Q["Query"] --> E["Embedding"]
    Q --> T["Tokenize"]
    E --> VS["Vector search"]
    T --> BM["BM25 search"]
    VS --> M["Weighted merge"]
    BM --> M
    M --> D["Recency and importance"]
    D --> R["MMR diversity"]
    R --> O["Top results"]

• Vector search catches semantic similarity, so "gateway host" can match "the machine running OpenClaw".
• BM25 keyword search catches exact terms like IDs, error strings, and config keys.
• Filename search keeps paths in a separate index from note bodies. Exact full paths, basenames, and filename stems are ranked above partial path matches, while snippet and body keyword scores still derive from note content.

When only one path is usable, the other runs by itself.

After that, the builtin engine applies a deterministic ranking:

hybrid relevance × recency decay × importance multiplier

Importance gets scored once at write time by a memory workflow that already has a model in the loop. A missing importance value is treated as neutral, so existing indexes keep their prior relevance signal. Dated daily notes lose weight with a 30-day half-life; curated files like MEMORY.md and USER.md never decay. This mirrors the relevance, recency, and importance scoring in Generative Agents (arXiv:2304.03442) without making a query-time model call.

MMR then reorders the scored hybrid candidate set to cut down on redundant snippets. It leaves scores, threshold eligibility, and provider calls untouched.

Deterministic trigger recall

On eligible interactive turns, the builtin engine also checks the inbound message against short trigger phrases stored on indexed entries. Strong matches can insert up to three compact entries into hidden context before the reply is generated. The prefilter relies on the existing keyword and vector retrieval paths and never invokes a recall model.

Automatic injection is intentionally more restrictive than memory_search: only promoted, trusted entries qualify. Until indexed provenance exists, that means entries from root MEMORY.md and USER.md only. Daily notes, imported transcripts, and session transcripts stay reachable through explicit memory tools or Active Memory escalation, but they are never injected automatically.

FTS-only mode. Setting provider: "none" deliberately turns off embeddings and searches with keywords alone. Leaving provider unset or set to "auto" falls back to keyword-only ranking when embedding setup or a request fails, and so does provider: "local" (the GGUF/llama.cpp provider). Creation-time fallback still indexes text for keyword search, and memory_search puts the redacted embedding-bootstrap reason into debug.embeddingBootstrap even when no matches exist.

Explicit provider unavailable. If you explicitly name another provider (like openai, ollama, or gemini) and it goes down at request time (bad auth, network failure), memory_search reports memory as unavailable instead of quietly degrading to FTS-only results. This keeps a broken configured provider visible. Set provider: "none" for deliberate FTS-only recall, or repair the provider/auth configuration to bring back semantic ranking.

Improving search quality

Two deterministic ranking passes are on by default for hybrid search.

Recency decay

Older notes gradually lose ranking weight so recent information shows up first. With the default 30-day half-life, a note from last month carries 50% of its original weight. MEMORY.md and other non-dated files under memory/ are evergreen and never decay; only dated memory/YYYY-MM-DD.md files decay.

MMR (diversity)

Redundant results are trimmed down. When five notes all reference the same router configuration, MMR picks a similarly relevant result with different content rather than echoing nearly identical snippets. The fixed relevance-biased mode relies on lambda 0.7 combined with Jaccard overlap across snippet tokens. Its local workload is O(k²): standard defaults ask for 24 candidates per retrieval leg, capping at 48 unique non-exact candidates before overlap; broader project and identifier searches keep their own separate limits.

Tip

Nothing needs to be set up. FTS-only and vector-only fallback routes skip the hybrid MMR pass entirely.

Multimodal memory

With gemini-embedding-2-preview, images and audio can be indexed alongside Markdown. This applies solely to files located under memory.search.extraPaths; default memory roots (MEMORY.md, memory/*.md) remain restricted to Markdown. Queries still use text, but they match against visual and audio material. For configuration details, see Memory configuration reference.

Session memory search

For precise full-text recall from session transcripts, run sessions_search and then open a result with sessions_history. Session-memory search stays the semantic, experimental counterpart.

Session transcripts can optionally be indexed so memory_search recalls earlier conversations. This is opt-in: set experimental.sessionMemory: true and add "sessions" to sources (default sources is ["memory"]).

Session hits follow tools.sessions.visibility: the default "tree" reveals the current session and any sessions it created. When the caller is the canonical main session, tree spans all same-agent sessions. With session.dmScope: "main", a multi-user DM setup shares that main session and its recall scope. For DM isolation, use a per-peer dmScope, or choose visibility "self" for strict current-session recall. Non-main callers still require "agent" visibility for unrelated same-agent sessions.

Troubleshooting

No results? Run openclaw memory status to inspect the index. If it comes back empty, run openclaw memory index --force.

Only keyword matches? Your embedding provider might not be configured. Verify openclaw memory status --deep.

Local embeddings time out? ollama, lmstudio, and local rely on longer provider-owned batch deadlines. Run openclaw memory status --deep to review the managed server endpoints before rebuilding the index.

CJK text not found? Rebuild the FTS index using openclaw memory index --force.

Related

• Memory overview
• Active memory
• Builtin memory engine
• Memory configuration reference