Dreaming: Background Memory Consolidation in memory-core

Dreaming serves as the background memory consolidation mechanism in memory-core. It transfers robust short-term signals into long-term storage while preserving explainability and auditability throughout the process.

Note

This feature operates by default. To turn it off, configure plugins.entries.memory-core.config.dreaming.enabled: false.

What dreaming writes

• Machine state held in memory/.dreams/ (recall store, phase signals, ingestion checkpoints, locks).
• Rewrite preimages persisted in SQLite-backed plugin state prior to an accepted MEMORY.md rewrite.
• Human-readable output located in DREAMS.md (or an existing dreams.md) and optional phase report files under memory/dreaming/<phase>/YYYY-MM-DD.md.

Only MEMORY.md receives writes during long-term promotion. Every newly promoted entry includes trailing recall metadata derived from the candidate: up to three concept tags in <!-- trigger: phrase one, phrase two --> and a constrained <!-- importance: N --> value ranging from 1 to 10. Consolidation leaves existing annotated entries untouched unless an explicit merge or supersession occurs.

Phase model

Each sweep executes three cooperative phases sequentially: light -> REM -> deep. These are internal implementation stages, not user-selectable modes.

Light phase

• Pulls recent short-term recall state, daily memory files, and redacted session transcripts when accessible.
• Removes duplicate signals and stages candidate lines.
• Produces a managed ## Light Sleep block when inline output is part of storage.
• Logs reinforcement signals for subsequent deep ranking.
• Refrains from writing to MEMORY.md.

REM phase

• Constructs theme and reflection summaries from recent short-term traces.
• Produces a managed ## REM Sleep block when inline output is part of storage.
• Logs REM reinforcement signals that deep ranking relies on.
• Refrains from writing to MEMORY.md.

Deep phase

• Evaluates candidates using weighted scoring and threshold gates (minScore, minRecallCount, minUniqueQueries must all pass).
• Refreshes snippets from live daily files before writing, skipping stale or deleted ones.
• Forwards gated owner and agent-derived candidates to a consolidation subagent with the current MEMORY.md.
• Modifies MEMORY.md only when the outcome retains enough prior entries, includes candidate source references, and respects the bootstrap budget.
• Defaults to the prior append-only promotion path when the model is unavailable or the rewrite fails validation.
• Records a ## Deep Sleep summary in DREAMS.md and optionally memory/dreaming/deep/YYYY-MM-DD.md.

Session transcript ingestion

Redacted session transcripts can be fed into the dreaming corpus. Only interactive sessions qualify. Cron, heartbeat, subagent, and unknown sessions are excluded from durable candidate ingestion. Personal and sensitive content gets redacted before ingestion, and runtime-marked recalled context is stripped so recalled snippets cannot be re-learned as new memory.

Consolidation safety

The deterministic score, recall-count, and query-diversity thresholds still act as the candidate gate. Consolidation proceeds only after those gates pass.

Before the consolidation prompt is assembled, memory-core removes candidates whose indexed provenance is untrusted or system. This acts as a structural taint gate, not a score penalty. Eligible candidates carry their origin, session kind, observation time, optional supersession key, and daily-note source reference.

An accepted rewrite must:

• keep prior entries within phases.deep.maxPriorEntryLossFraction
• include every promoted candidate's Source: path#Lx-Ly reference
• respect the MEMORY.md bootstrap-safe file budget
• parse as the expected structured response

Before the file changes, the previous MEMORY.md is saved in SQLite-backed plugin state. DREAMS.md receives added, merged, and superseded counts plus short diff-style highlights. This keeps each rewrite reviewable without turning the Dream Diary into a promotion source.

Background consolidation draws on sleep-time compute (arXiv:2504.13171). The provenance and reflection boundary follows the durable memory framing in the Generative Agents research.

Dream Diary

A narrative Dream Diary is maintained in DREAMS.md. Once each phase has sufficient material, memory-core executes a best-effort background subagent turn and appends a short diary entry, using the default runtime model unless dreaming.model is configured. If the configured model is unavailable, the diary run retries once with the session default model; trust or allowlist failures are not retried and remain visible in logs instead of silently falling back to a generic diary entry.

Note

The diary is intended for human reading in the Dreams UI, not a promotion source. Diary/report artifacts are excluded from short-term promotion; only grounded memory snippets are eligible to promote into MEMORY.md.

A grounded historical backfill lane also exists for review and recovery work:

Backfill commands

• memory rem-harness --path ... --grounded previews grounded diary output from historical YYYY-MM-DD.md notes.
• memory rem-backfill --path ... writes reversible grounded diary entries into DREAMS.md.
• memory rem-backfill --path ... --stage-short-term stages grounded durable candidates into the same short-term evidence store the normal deep phase uses.
• memory rem-backfill --rollback and --rollback-short-term remove those staged backfill artifacts without touching ordinary diary entries or live short-term recall.
• memory session-backfill --agent <id> previews trusted candidates from the agent's retained session history, oldest unprocessed day first.
• memory session-backfill --agent <id> --apply stages those candidates through the normal short-term store and writes reversible diary blocks without changing MEMORY.md or USER.md.
• memory session-backfill --agent <id> --rem writes a deterministic grounded preview per day to DREAMS.md without staging candidates or calling a model.
• memory session-backfill --agent <id> --rollback clears the shared grounded backfill candidates and diary blocks, including artifacts created by rem-backfill.

Session backfill relies on canonical retained transcript identities, including sessions preserved across rotation. Messages are bucketed in the configured dreaming timezone and share live ingestion's tracked message hashes and signal caps. Apply drains bounded batches to completion in one command. Rollback removes generated artifacts plus the hashes and cursor progress owned by those batches, allowing the same candidates to be staged again. Foreign files supplied with --archive-files are treated conservatively. Their embedded ownership fields are caller-controlled and therefore remain untrusted; without an authenticated provenance contract, they cannot enter short-term staging. Tool output, web content, and non-owner turns are excluded from the canonical session path as well.

The Control UI offers the same diary backfill and reset flow on the agent's Memory tab, located on the Agents page, letting you review dream scene results before deciding whether grounded candidates deserve promotion. A separate grounded Scene lane distinguishes staged short-term entries originating from historical replay, identifies which promoted items were grounded-led, and allows clearing only grounded-only staged entries without altering live short-term state.

Deep ranking signals

Deep ranking combines six weighted base signals with phase reinforcement:

Hits during light and REM phases add a modest boost, decayed by recency, sourced from memory/.dreams/phase-signals.json.

Scheduling

When turned on, memory-core automatically handles a single cron job for a complete dreaming sweep, with deduplication applied across the primary runtime workspace and any configured agent workspaces. This ensures subagent workspace fan-out does not exclude the main agent's DREAMS.md and memory state.

Quick start

Enable dreaming

{
  "plugins": {
    "entries": {
      "memory-core": {
        "config": {
          "dreaming": {
            "enabled": true
          }
        }
      }
    }
  }
}

Custom sweep cadence

{
  "plugins": {
    "entries": {
      "memory-core": {
        "config": {
          "dreaming": {
            "enabled": true,
            "timezone": "America/Los_Angeles",
            "frequency": "0 */6 * * *"
          }
        }
      }
    }
  }
}

Slash command

/dreaming status
/dreaming on
/dreaming off
/dreaming help

Owner status is a prerequisite for /dreaming on and /dreaming off when invoked by channel callers, while Gateway clients need operator.admin instead. Both /dreaming status and /dreaming help operate in read-only mode.

CLI workflow

Promotion preview / apply

openclaw memory promote
openclaw memory promote --apply
openclaw memory promote --limit 5
openclaw memory status --deep

Unless overridden through CLI flags, manual memory promote relies on deep-phase thresholds by default.

Explain promotion

Get an explanation for why a given candidate would or would not be promoted:

openclaw memory promote-explain "router vlan"
openclaw memory promote-explain "router vlan" --json

REM harness preview

See REM reflections, candidate truths, and deep promotion output in advance, with nothing written:

openclaw memory rem-harness
openclaw memory rem-harness --json

Key defaults

All settings reside under plugins.entries.memory-core.config.dreaming.

• enabled (boolean, default: true), Turns the dreaming sweep on or off.

• phases.deep.maxPriorEntryLossFraction (number, default: 0.25), Discards a consolidation rewrite when it removes more than this proportion of prior entries.

• frequency (string, default: 0 3 * * *), Cron cadence governing the full dreaming sweep.

• model (string), Optional model override for the Dream Diary subagent. When also setting a subagent allowedModels allowlist, use a canonical provider/model value.

• phases.deep.maxPromotedSnippetTokens (number, default: 160), Maximum estimated token count preserved from each short-term recall snippet promoted into MEMORY.md. Ranking provenance stays visible.

Warning

plugins.entries.memory-core.subagent.allowModelOverride: true is mandatory for dreaming.model. To limit it, configure plugins.entries.memory-core.subagent.allowedModels as well. Automatic retries handle only model-unavailable errors; trust or allowlist failures remain visible in logs rather than silently falling back.

Note

Much of the phase policy, thresholds, and storage behavior are internal implementation details. The full key list appears in the Memory configuration reference.

Dreams UI

With it enabled, the Gateway Dreams tab displays:

• the current dreaming enabled state
• phase-level status and whether a managed sweep is present
• counts for short-term, grounded, signal, and promoted-today entries
• the timing of the next scheduled run
• a separate grounded Scene lane for staged historical replay entries
• an expandable Dream Diary reader powered by doctor.memory.dreamDiary

When the bundled memory-wiki plugin is active, the Diary view gains two extra sub-tabs beside Dreams:

• Imported Insights: clustered insights surfaced by external-history imports (such as openclaw wiki chatgpt import), available for review before anything graduates into durable memory
• Memory Wiki: the compiled wiki the memory system can search and reason over, covering synthesis, entity, and concept pages (plus sources and reports carrying claims, open questions, or contradictions), with per-page counts, a full-vault breakdown, and inline page previews

If memory-wiki is off, both sub-tabs instead show an enable hint.

Related

• Memory
• Memory CLI
• Memory configuration reference
• Memory search