MongoDB

User preferences and memory live in ~/Clawic/data/mongodb/ (see setup.md on first use, memory-template.md for the file format). If you have data at an old location (~/mongodb/ or ~/clawic/mongodb/), move it to ~/Clawic/data/mongodb/.

When To Use

• Designing or reviewing document schemas: embed vs reference, growing arrays, multi-tenant layout, versioned shapes
• A query, pipeline, or dashboard is slow: reading explain plans, designing compound indexes, killing a scan
• Building or debugging aggregation pipelines, $lookup joins, materialized views, window functions
• Connecting an application: connection strings, pooling, timeouts, retries, ODM behavior, serverless clients
• Operating production: replica sets, sharding, write and read concerns, backups, upgrades, security, Atlas
• A live incident: no primary, connection storm, disk full, runaway operation, cache stall, blown oplog window
• Not for SQL or relational databases — normalization instincts from relational design actively mislead here (see Related Skills)

Quick Reference

Situation	Play
Query is slow, cause unknown	explain("executionStats") before touching anything; triage order in slow-queries.md
explain shows COLLSCAN	No usable index, or the index exists in the wrong field order — check ESR before creating another (→ indexes.md)
totalDocsExamined far above nReturned	Index is not selective enough for this shape; the ratio target is Core Rule 1
Embed or reference this relationship?	Three questions in order: read together, bounded, independently updated (→ schema.md)
An array grows once per user action	Never embed it — child collection or time-series collection (Core Rule 3)
Pipeline aborts "exceeded memory limit"	A blocking stage hit its 100MB budget: index the sort or reshape upstream (→ aggregation.md)
find().sort() fails on memory	Different limit: 32MB in-memory sort for find, not the pipeline's 100MB (→ slow-queries.md)
$lookup inside a hot query	A schema signal, not a tuning problem: extended reference or embed (→ schema.md)
Deep pagination (skip in the tens of thousands)	Range on the sort key: {_id: {$gt: lastSeenId}} — constant cost per page
"not writable primary", server selection timed out	Topology or failover, not your query (→ errors.md, then connections.md)
Latency spikes right after a deploy	Pool math: pods × maxPoolSize (Core Rule 8) or Mongoose autoIndex rebuilding on boot (→ connections.md)
Cursor died mid-iteration (code 43)	10-minute idle cursor timeout — re-query by range per batch (→ connections.md)
Duplicate key error on an optional field	Plain unique index counts two missing fields as duplicate nulls — partial unique index (→ indexes.md)
Secondary lagging, oplog window shrinking	→ replication.md; flow control and the resync threshold live there
Majority writes hang after losing one node	PSA topology trap (→ replication.md)
One shard absorbs all the writes	Monotonic shard key (→ sharding.md)
Full-text, fuzzy, faceted, or vector search	The built-in text index is the wrong tool past basic keyword match (→ search.md)
React to writes: CDC, cache invalidation, outbox	Change streams, resume tokens, oplog window (→ change-streams.md)
Changing a field's shape under live traffic	Expand → dual-write → backfill in batches → contract (→ migrations.md)
Metrics, events, or sensor readings by timestamp	Time-series collection, not a normal collection with a date index (→ time-series.md)
Multi-document invariant across collections	Try to co-locate it first; transactions with a retry loop if you cannot (→ transactions.md)
Database reachable from the internet	Auth, bindIp, and TLS audit before anything else (→ security.md)
An error code or message to decode	Error Codes below; full catalog with fixes in errors.md
Anything else	Reproduce with the smallest find() that still shows it, explain() before and after every change; if production is stuck, read db.currentOp() before you change a thing

Depth on demand, by phase:

• Diagnose — slow-queries.md explain plans and triage · errors.md code to cause · incidents.md no primary, disk full, connection storm, cache stall · monitoring.md what to watch and alert on · mongosh.md the shell toolkit
• Design — schema.md embed vs reference and the pattern catalog · indexes.md ESR, index types, lifecycle · aggregation.md pipeline craft · time-series.md measurements over time · search.md text and vector search
• Change — migrations.md online schema change, backfills, bulk loading · transactions.md sessions and retry loops · change-streams.md reacting to writes
• Operate — connections.md drivers, pools, timeouts, ODMs · replication.md replica sets, concerns, oplog, upgrades · sharding.md shard keys and the balancer · tuning.md WiredTiger cache and host settings · backups.md dumps, snapshots, PITR, drills · security.md auth, roles, TLS, injection, encryption · atlas.md the managed platform

Core Rules

1. Read explain("executionStats"), never guess. Healthy ratio: totalDocsExamined / nReturned ≈ 1. Worked example: 1,240,000 examined for 25 returned = 49,600:1 — missing or wrong index (Atlas's Query Targeting alert fires at 1000:1 by default). A ratio near 1 with a slow query is a different bug: look at nReturned itself, then at the sort.
2. 16MB is a ceiling, not a budget. WiredTiger rewrites the whole document on every update: a 5MB document taking a 20-byte $inc still costs a 5MB rewrite in cache. Keep working documents in the KB range; a document you routinely read in full should fit in a single-digit number of 4KB pages.
3. Unbounded arrays are the #1 schema failure. Anything that grows per event goes to its own collection or a time-series collection (MongoDB >=5.0). A multikey index adds one entry per element: a 10,000-element array = 10,000 index entries for one document, paid on every write to that document.
4. Set write and read concern explicitly in the connection string. Implicit defaults changed across versions (→ replication.md); code relying on them silently changes durability semantics on upgrade. Baseline URI: ?retryWrites=true&w=majority.
5. One compound index per query shape, ordered Equality → Sort → Range. Prefix rule: {a: 1, b: 1, c: 1} also serves queries on {a} and {a, b} — delete those redundant single-field indexes. Index intersection exists but the planner rarely picks it; never design for it.
6. Single-document atomicity is the concurrency primitive. Model invariants inside one document before reaching for transactions; when you do use them, stay under 1,000 modified documents and well inside the 60s default transaction lifetime (→ transactions.md).
7. Read-your-own-writes requires primary reads or a causal-consistency session. Secondary lag is usually sub-second but unbounded under load — never assume freshness on a secondary; bound it with maxStalenessSeconds when you read one deliberately.
8. One MongoClient per process, sized from concurrency, not from the default. Total server connections = processes × maxPoolSize (driver default 100): 40 pods × 100 = 4,000 connections ≈ 4GB of mongod RAM before a single document is served. Creating a client per request or per serverless invocation is the same bug at higher speed (→ connections.md).

Query Semantics That Bite

• {tags: "red"} matches docs where tags IS "red" or an array CONTAINING "red" — implicit array traversal, both directions, and you cannot turn it off.
• {qty: {$gte: 5, $lte: 10}} on an array field matches [3, 12]: different elements satisfy each bound. One element must satisfy all conditions → $elemMatch.
• {field: null} matches explicit null AND missing field. Distinguish with {field: {$type: "null"}} vs {field: {$exists: false}}.
• $ne, $nin, $not technically use the index but scan most of it — near-collection-scan cost; restructure with a positive predicate or a status field.
• skip(100000).limit(20) walks 100,020 index entries. Paginate by range on the sort key instead: {_id: {$gt: lastSeenId}} — constant cost per page.
• Anchored case-sensitive regex /^abc/ uses the index; /^abc/i cannot. Case-insensitive lookups need a collation index (→ indexes.md).
• Comparisons cross BSON types in a fixed order rather than erroring: the string "5" is never $gt: 4. Mixed types in one field silently split every range query — enforce the type with a $jsonSchema validator.
• $in with a large array is a range, not an equality, for index-order purposes; a short $in gets exploded into parallel index scans and keeps the sort order (→ indexes.md).
• Dotted paths into arrays are positional-ambiguous: {"items.0.sku": x} targets a position, {"items.sku": x} targets any element. $ and $[] update operators follow the same split.

Consistency Model

• Acknowledged ≠ durable: a w: 1 write vanishes into a rollback file if the primary fails before replication — someone must reconcile those by hand, and nothing in the application will ever know.
• Failovers happen; retryable writes (default on in modern drivers) hide most of them — but updateMany and deleteMany are NOT retryable writes. Wrap multi-doc mutations in idempotent logic or a transaction.
• Causal sessions give read-your-own-writes even across secondaries; use them instead of forcing primary everywhere.
• Read concern local can return writes that later roll back; majority cannot. available on a sharded cluster skips orphan filtering and can return documents twice — never use it for correctness-sensitive reads.
• Arbiters are a durability trap, not a cheap third node: the PSA topology stalls majority writes when one data node is down (→ replication.md).

ObjectId and _id

• 12 bytes: 4-byte unix timestamp + 5-byte random + 3-byte counter. ObjectId.getTimestamp() recovers creation time; sorting by _id approximates insertion order (per-process counter, not a global clock).
• Predictable enough to enumerate — never use as a security token or unguessable URL.
• Monotonic growth makes _id a hotspotting shard key (→ sharding.md).
• _id is immutable: changing it means insert-new + delete-old, which is a migration, not an update (error 66 in errors.md).
• Custom _id is legitimate and free of a second index when the natural key is stable and short ({_id: "<tenant>:<sku>"}). Random UUIDv4 as _id costs B-tree locality on inserts; UUIDv7 or a prefixed key keeps writes clustered.

Error Codes

Codes are stable; message text is not. Match on the code. Full catalog with recovery steps in errors.md.

Code	Name	First move
11000	DuplicateKey	Genuine duplicate, a race, or a unique index counting missing fields as null (→ indexes.md)
112	WriteConflict	Two writers hit the same document; inside a transaction, retry the whole transaction (→ transactions.md)
50	MaxTimeMSExpired	Your maxTimeMS fired as designed — the query is slow, not broken (→ slow-queries.md)
43	CursorNotFound	Cursor idled past 10 minutes; batch and re-query by range (→ connections.md)
292	QueryExceededMemoryLimitNoDiskUseAllowed	Blocking stage over 100MB with disk use off (→ aggregation.md)
10107	NotWritablePrimary	You are talking to a secondary or a stepped-down node — connect with the full replica set URI
189	PrimarySteppedDown	Failover in progress; retryable writes cover single-document writes only
133	FailedToSatisfyReadPreference	No member matches the read preference or maxStalenessSeconds (→ replication.md)
251	NoSuchTransaction	The transaction expired (60s default) or the session was lost (→ transactions.md)
121	DocumentValidationFailure	A $jsonSchema validator rejected the write — read errInfo for the failing path
18 / 13	AuthenticationFailed / Unauthorized	Credentials vs privileges: 18 is who you are, 13 is what you may do (→ security.md)
66	ImmutableField	Something tried to modify _id or a shard key field the wrong way

Configuration

User-dependent variables. Defaults apply until the user states a preference; store them in ~/Clawic/data/mongodb/config.yaml.

Variable	Type	Default	Effect
server_version	number (4.4-8.0)	7.0	Which version-gated advice applies (feature >=X lines) when the live server version is unknown; also gates upgrade recommendations
deployment	atlas | self-hosted | docker | documentdb | cosmos-mongo	atlas	Switches between Atlas console recipes and mongod.conf/setParameter, and suppresses features the emulation layers lack (→ atlas.md)
driver	mongosh | node | python | java | go | csharp	mongosh	Language of every emitted code example and connection string
odm	none | mongoose | prisma | beanie | spring-data	none	Whether examples are raw driver or ODM-shaped, and which ODM traps get surfaced (→ connections.md)
id_style	objectid | uuid | natural	objectid	_id type in generated schemas, migrations, and shard-key advice
field_naming	camelCase | snake_case	camelCase	Field names in generated documents, index specs, and pipelines
write_concern_default	majority | w1	majority	Write concern written into emitted URIs and examples (Core Rule 4)
slow_ms	number (ms, 20-500)	100	Profiler threshold in monitoring.md recipes and what counts as "slow" when reporting
backfill_batch	number (docs, 100-10000)	1000	Batch size in every backfill and migration loop (→ migrations.md)
destructive_confirm	bool	true	drop, dropDatabase, dropIndex, unfiltered deleteMany/updateMany, and killOp are emitted for review instead of run

Preference areas — customizable dimensions; a stated preference is recorded in config.yaml and applied from then on:

• Tooling — shell vs Compass vs driver REPL, migration framework (plain scripts, migrate-mongo, Mongock), diagram and schema-analysis tooling
• Thresholds — profiler slowms, the examined:returned ratio worth reporting, pool size, replication-lag alarm, index-size budget per collection
• Conventions — collection naming (plural/singular), timestamp field names, soft-delete policy, schema_version usage, index naming, enum-as-string vs code
• Platform — server version, deployment target, storage class, instance memory and core count — affects every number in tuning.md
• Risk posture — whether to run mutations directly or hand back reviewed scripts, whether secondary reads are allowed, how aggressive index changes may be on a live collection
• Output format — snippets only vs snippets plus an explain walkthrough, how much plan detail to narrate, whether to include the rollback path by default
• Integrations — monitoring stack (Atlas metrics, Prometheus exporter, FTDC), backup tooling, search backend (Atlas Search vs external engine)
• Restrictions — compliance regimes that mandate field-level encryption or auditing, collections that must never be touched online, features the platform forbids
• Cadence — restore-drill frequency, index-usage review cycle, oplog-window review, upgrade windows

Output Gates

Before emitting a schema, an index, a pipeline, or a migration:

• Is every array in this schema bounded by something other than optimism?
• Does each new index correspond to a real query shape, in ESR order, with its redundant single-field prefixes removed?
• Was the query run through explain("executionStats"), with examined:returned checked, and not just read?
• Does the pipeline's first stage reach an index, and does no blocking stage sit ahead of the filter?
• Is the write concern explicit, and does it match the data class (telemetry vs money)?
• Is the backfill batched, resumable, and safe to run twice?
• Does the destructive step (drop, unfiltered deleteMany, index drop) ship separately from the code change, after the code that stopped using it?
• For anything user-supplied reaching a query object: is it type-checked so {$ne: null} cannot arrive where a string was expected (→ security.md)?

Traps

Trap	Why it fails	Do instead
Treating "schemaless" as "no schema"	Schema moved into app code, unenforced; shapes drift per deploy	$jsonSchema validator + schema_version field (→ schema.md)
$push growing an array forever	Full-document rewrite per push, then the 16MB wall	$slice cap or a child collection
countDocuments({}) for dashboard totals	Runs a scan-backed count on every load	estimatedDocumentCount() — metadata, O(1); accepts orphan drift on sharded clusters
One collection per tenant or per day	Each collection and index is a WiredTiger file; thousands degrade checkpoints and startup	tenant_id field + compound indexes (→ schema.md)
$where / $function in hot paths	JavaScript per document, no index use — and an injection surface	Rewrite with native operators (→ security.md)
Ignoring write concern on "unimportant" writes	Data appears written, lost on failover	Pick a concern per data class (→ replication.md)
Building an index on a live collection at peak	Non-blocking since 4.2, but it still burns IO and cache on every member	Schedule off-peak, or roll it member by member (→ indexes.md)
A new MongoClient per request or per Lambda invocation	Every invocation opens a fresh pool; the server hits its connection cap while the app looks idle	One client per process, cached outside the handler (Core Rule 8)
Iterating a huge cursor while doing slow work per document	The cursor idles out at 10 minutes mid-loop, code 43	Page by range key, one query per page (→ connections.md)
find() in a loop instead of one $in or $lookup	N round trips at network latency each; the database is not the bottleneck, the trips are	Batch the keys, or fix the shape so one read serves the page
Trusting retryWrites to cover everything	updateMany/deleteMany and multi-statement work are not retryable	Idempotent logic or a transaction (→ transactions.md)
Deleting millions of documents with one deleteMany	Long-running write, oplog flood, replication lag, no resume point	Batched deletes by range, or a TTL index that does it continuously

Where Experts Disagree

• Embed-first vs reference-first. MongoDB's own guidance is embed-first; teams from relational or microservice backgrounds reference-first for independent lifecycles. Boundary: embed when the child is always read with the parent AND bounded; reference otherwise.
• Transactions. One school treats a multi-document transaction as a schema-design smell (redesign so the invariant fits one document); the other uses them freely since 4.0. Boundary: cross-entity invariants you cannot co-locate (ledger + balance) justify them; convenience joins do not.
• Secondary reads for scale. Often called a myth — every secondary applies every write anyway, so they add read capacity only. Still legitimate for analytics isolation and geo-local latency with a bounded maxStalenessSeconds.
• Schema validators. One camp enforces $jsonSchema at the database because application-only validation always drifts; the other keeps validation in code because a validator change is a live DDL-shaped operation. Testable boundary: more than one writer (services, scripts, humans in the shell) means the database has to hold the rule.
• Aggregation in the database vs the application. Pushing analytics into pipelines competes with the operational workload for the same WiredTiger cache. Boundary: sub-second, index-backed, user-facing aggregations belong in MongoDB; anything scanning history belongs in a materialized collection, an analytics node, or a warehouse (→ aggregation.md).

Related Skills

More Clawic skills, get them at https://clawic.com/skills/mongodb (install if the user confirms):

• pg — PostgreSQL; jump there when the model is genuinely relational or you need cross-row constraints and SQL
• db — general database operations, reliability, and scaling patterns
• database-indexing — index theory beyond MongoDB specifics: write-cost budgets, structures, composite ordering
• vector-databases — comparing Atlas Vector Search against dedicated vector stores by scale and filter needs
• elasticsearch — when text search outgrows Atlas Search: analyzers, relevance tuning, aggregations

Part of Clawic, the verified skill library. Get this skill: https://clawic.com/skills/mongodb.