Personal Agent Benchmark Pack for QA Scenario Testing
A compact QA scenario pack for local personal assistant workflows, using existing private QA stacks and synthetic channels. Ideal for developers verifying reminder, thread, and memory preference features.
Read this when
- Running local personal agent reliability checks
- Extending the repo-backed QA scenario catalog
- Verifying reminder, reply, memory, redaction, safe tool followthrough, task status, share-safe diagnostics, proof-backed completion claims, and failure recovery
The Personal Agent Benchmark Pack is a compact, repository-backed QA scenario pack designed for local personal assistant workflows. This is not a general-purpose model benchmark, and it does not require a separate runner. Instead, it relies on the existing private QA stack (QA overview), the synthetic QA channel, and the current qa/scenarios YAML catalog.
Scenarios
Ten scenarios are defined in qa/scenarios/personal/*.yaml:
| Scenario id | Checks |
|---|---|
personal-reminder-roundtrip | Fake personal reminders through local cron delivery |
personal-channel-thread-reply | Fake DM and thread reply routing through qa-channel |
personal-memory-preference-recall | Fake preference recall from the temporary QA workspace memory files |
personal-redaction-no-secret-leak | Fake secret no-echo checks |
personal-tool-safety-followthrough | Safe read-backed tool followthrough after a short approval-style turn |
personal-approval-denial-stop | Approval denial stop behavior for a sensitive local read request |
personal-task-followthrough-status | Proof-backed task status reporting that keeps pending, blocked, and done separate |
personal-share-safe-diagnostics-artifact | Share-safe diagnostics artifacts that keep useful status while omitting raw personal content |
personal-no-fake-progress | Proof-backed completion claims that avoid fake progress before local evidence exists |
personal-failure-recovery | Failure recovery that reports partial status and keeps retry boundaries clear |
The machine-readable personal-agent profile is located in the root taxonomy.yaml as semantic coverage IDs. Every primary owner is resolved by QA Lab directly from the catalog, so there is no secondary scenario-ID list. To execute it, use:
OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm openclaw qa run \
--qa-profile personal-agent \
--provider-mode mock-openai \
--concurrency 1
Repeat the --scenario flags to narrow the profile. Neither the scenario file order nor the taxonomy order influences membership or the sequence of execution.
The pack is aimed at qa-channel with mock-openai or another local QA provider lane. Avoid pointing it at live chat services or real personal accounts.
Privacy Model
Scenarios exclusively employ fake users, fake preferences, fake secrets, and the temporary QA gateway workspace that the suite creates. They must never read or write actual OpenClaw user memory, sessions, credentials, launch agents, global configs, or live gateway state.
Artifacts remain within the existing QA suite artifact directory and are handled like test output. Redaction checks rely on fake markers, so any failures are safe to inspect and file in issues.
Extending the pack
Add new .yaml cases under qa/scenarios/personal/, declare the exact primary coverage ID they demonstrate, and include that semantic ID in the taxonomy profile when it fits this benchmark. Keep each case small, local, deterministic in mock-openai, and focused on a single personal assistant behavior.
Suitable follow-up candidates: redacted trajectory export checks, local-only plugin workflow checks.
Do not introduce a new runner, plugin, dependency, live transport, or model judge until the scenario catalog contains enough stable cases to justify such a surface.