@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/mongodb/SKILL.md

@@ -0,0 +1,342 @@
+---
+name: mongodb
+description: "Use when modeling MongoDB documents (embed versus reference, the 16MB cap, bucket and subset patterns), choosing or fixing indexes (compound order by the ESR rule, partial, TTL, multikey, reading explain), writing aggregation pipelines that stay index-eligible, running multi-document transactions with retry, or operating and securing a deployment (replica set, read/write concern, Atlas tiers, Vector Search, Queryable Encryption). MongoDB 8.2, driver-agnostic. NOT relational schema, SQL or EXPLAIN ANALYZE (that is `postgresdb`)."
+tags: [mongodb, nosql, database, aggregation, indexing]
+recommends: [secure-coding, harness]
+profiles: []
+origin: ai-hub
+---
+
+# MongoDB — modeling, indexing, aggregation, transactions, ops
+
+Engine-level MongoDB 8.2 guidance: model documents for the queries you actually run, pick the
+index the planner will use, write aggregation pipelines that stay index-eligible, run
+multi-document transactions with correct retry, and operate/secure a deployment. Driver-agnostic —
+every example is `mongosh` shell syntax that maps 1:1 to the official drivers (Node, Python, Go,
+Java, Rust). This skill owns the **server's query and the index it picks**, not any ODM's API.
+
+## When to use / When NOT to use
+
+**When to use:**
+
+- Document modeling: embed vs reference, the 16 MB cap, one-to-many/many-to-many, the
+  subset/extended-reference/bucket/computed/outlier patterns, taming unbounded array growth.
+- Index decisions: single-field, compound (the ESR ordering rule), multikey, partial, TTL, text,
+  wildcard, `2dsphere`; and when an index is NOT worth it.
+- Any query that is slow or scans too much; reading `explain("executionStats")`.
+- Aggregation pipelines: stage order so `$match`/`$sort` hit an index, `$lookup` cost, `$unwind`
+  explosion, `$group`/`$sort` memory limits and `allowDiskUse`, `$merge`/`$out`, faceting.
+- Multi-document transactions: sessions, `withTransaction` retry semantics, read/write concern.
+- Operating/securing: replica set, read preference, write concern, Atlas tier choice, Atlas Search
+  & Vector Search, Queryable Encryption, role-based access, connection-pool knobs.
+
+**When NOT to use:**
+
+- Relational schema / SQL / `EXPLAIN ANALYZE` → [`postgresdb`](../postgresdb/SKILL.md). Different
+  engine, planner, and concurrency model.
+- ODM/driver API ergonomics (Mongoose pre-save hooks, the Node driver's `bulkWrite` return shape,
+  `updateMany`'s result object) → that tool's own docs. This skill owns the server query and the
+  index the server picks, not the JS object the driver hands back.
+- App-layer caching as a product (Redis-in-front-of-reads).
+- Cloud-console click-paths — we give the shell command / connection string, not the Atlas UI tour.
+- Picking a vector store *across engines* (Pinecone vs Weaviate). Atlas Vector Search *inside* Mongo
+  is in scope; cross-engine selection is not.
+
+Deep dives: [data-modeling](references/data-modeling.md) (embed/reference tree, all six patterns,
+16 MB math, polymorphic & schema versioning) · [aggregation](references/aggregation.md) (per-stage
+index eligibility, `$lookup` variants, `$facet`, window fns, `$merge`/`$out`, reading pipeline
+`explain`) · [transactions-and-ops](references/transactions-and-ops.md) (retry wrappers, concern
+semantics, Atlas tiers, Vector Search, Queryable Encryption, RBAC, pooling, change streams).
+
+## Non-negotiables
+
+1. **Design for the queries you run, not the shape of your data.** The schema is the set of
+   documents that make your common reads single-document and index-eligible.
+2. **Never let an array grow unbounded inside a document.** It walks toward the 16 MB cap, bloats
+   every read of the parent, and kills update performance — reference or bucket it.
+3. **The hard ceiling is 16 MB per document.** If a one-to-many can exceed it, you reference; there
+   is no `TOAST`-style overflow here.
+4. **A single-document write is already atomic.** Reach for a multi-document transaction *only* when
+   two or more documents must change together — otherwise you are paying for nothing.
+5. **Every transaction retries on the `TransientTransactionError` label** (and commit retries on
+   `UnknownTransactionCommitResult`). `withTransaction` does both for you; a hand-rolled loop must.
+6. **Index by ESR: Equality fields, then the Sort field, then Range fields.** This order lets one
+   compound index serve the filter, the sort, and the range without an in-memory sort.
+7. **Read `explain("executionStats")` before and after adding an index** — confirm `IXSCAN`, not
+   `COLLSCAN`, and `totalKeysExamined ≈ nReturned`. Or it didn't happen.
+8. **`w:"majority"` for money and state transitions**, read concern `"majority"`/`"snapshot"` when a
+   read must reflect a durable write. `w:1` can be rolled back on a primary failover.
+9. **Money is `Decimal128` (`NumberDecimal("...")`), never a JS `double`.** Binary floats drift;
+   `0.1 + 0.2 !== 0.3` in your ledger.
+10. **Never store a secret in plaintext.** Use Queryable Encryption / client-side field-level
+    encryption; never commit a `mongodb://user:pass@` literal.
+
+## Decision rules
+
+### Embed or reference
+
+| Relationship | Choose | Why |
+| --- | --- | --- |
+| Read together, small, bounded (address on a user) | embed | one read, no `$lookup`, atomic update |
+| One-to-few, bounded (≤ a few dozen, won't grow) | embed | stays well under 16 MB |
+| One-to-many, growth not bounded (comments on a post) | reference | array would chase the 16 MB cap |
+| Many-to-many (students↔courses) | reference (array of ids on the lighter side) | shared, independently mutated |
+| Child shared across parents | reference | one source of truth, no duplication drift |
+| Child independently and frequently mutated | reference | avoid rewriting a big parent per child edit |
+| High-cardinality / huge child set | reference (+ optional subset embed) | keep the hot read small |
+
+### Which schema pattern
+
+| Symptom | Pattern | What it does |
+| --- | --- | --- |
+| List view reads 3 fields of a heavy doc | **subset** | embed only the hot fields, reference the rest |
+| `$lookup` on every read just to show a name/price | **extended reference** | copy the few joined fields you display |
+| Unbounded time-ordered events (readings, logs) | **bucket** | group N events per doc by time window |
+| Same `count`/`sum` recomputed on every read | **computed** | store the rollup, update it on write |
+| 1% of docs break the shape (a few mega-children) | **outlier** | flag them, overflow into linked docs |
+| One collection holds several entity shapes | **polymorphic** | a `type` discriminator + shared `_id` space |
+
+Full Bad→Good documents for each in [data-modeling](references/data-modeling.md).
+
+### Which index type
+
+| Access pattern | Index | Note |
+| --- | --- | --- |
+| `=` on one field | single-field | also covers the field's sort |
+| filter + sort + range together | compound, ordered **ESR** | one index serves all three |
+| query into an array field | multikey (automatic on an array key) | one multikey field per compound index |
+| query only a subset of docs (`status:"active"`) | partial (`partialFilterExpression`) | smaller, cheaper to maintain |
+| auto-expire docs after a time | TTL (`expireAfterSeconds` on a Date) | single-field only; deletes in background |
+| language-aware text search | text **or** Atlas Search | Atlas Search is far richer; text is legacy |
+| unpredictable / many query shapes on subdocs | wildcard (`"$**"`) | last resort; never beats a targeted index |
+| geospatial proximity / within | `2dsphere` | GeoJSON `Point`/`Polygon` |
+| vector similarity (8.2, Community+) | Atlas/Vector Search index | see transactions-and-ops ref |
+
+### When NOT to add an index
+
+- Low-cardinality field (a boolean, a 3-value status) — the planner skips it; `COLLSCAN` wins.
+- Tiny collection — a collection scan reads one or two pages; the index is pure write tax.
+- A field already the **left prefix** of an existing compound index — redundant.
+- Write-heavy field rarely filtered — every index is paid on every insert/update.
+- "Just in case" indexes — an unused index costs writes and RAM, returns nothing.
+
+## Copy-paste patterns
+
+Every fence is `mongosh` syntax.
+
+### Model the document for the read (Bad → Good)
+
+```javascript
+// BAD: comments embedded in the post — array grows without bound toward 16 MB,
+// every post read drags the entire comment history, money is a float.
+db.posts.insertOne({
+  _id: ObjectId(),
+  title: "Indexing 101",
+  authorId: ObjectId(),
+  price: 9.99,                       // double — drifts in arithmetic
+  comments: [ /* ...unbounded... */ ] // chases the 16 MB cap
+})
+
+// GOOD: post stays small; comments referenced; money is Decimal128;
+// the few fields the feed needs are duplicated (extended reference).
+db.posts.insertOne({
+  _id: ObjectId(),
+  title: "Indexing 101",
+  author: { _id: ObjectId(), name: "Ada" }, // extended ref: name shown without a $lookup
+  price: NumberDecimal("9.99"),
+  commentCount: 0,                            // computed rollup, bumped on write
+  createdAt: new Date()
+})
+db.comments.insertOne({ _id: ObjectId(), postId: ObjectId(), body: "…", createdAt: new Date() })
+```
+
+### Compound index in ESR order + the query that uses it
+
+```javascript
+// Feed query: filter by author (equality), sort by date (sort), bound by a date (range).
+// ESR => author first, then the sort/range key.
+db.posts.createIndex({ "author._id": 1, createdAt: -1 })
+
+db.posts.find({ "author._id": authorId, createdAt: { $gte: since } })
+        .sort({ createdAt: -1 })
+        .limit(20)
+// Confirm the plan: IXSCAN on the index above, no in-memory SORT stage.
+```
+
+### Partial + TTL indexes
+
+```javascript
+// Partial: index only the rows you actually query (active orders), not the archive.
+db.orders.createIndex(
+  { customerId: 1, createdAt: -1 },
+  { partialFilterExpression: { status: "active" } }
+)
+
+// TTL: expire sessions 30 minutes after lastSeen. Field MUST be a Date.
+db.sessions.createIndex({ lastSeen: 1 }, { expireAfterSeconds: 1800 })
+```
+
+### Aggregation: `$match` first, `$lookup`, `$group` with `allowDiskUse`
+
+```javascript
+db.orders.aggregate([
+  // $match FIRST so it uses the compound index and shrinks the working set early.
+  { $match: { status: "paid", createdAt: { $gte: since } } },
+  { $sort:  { createdAt: -1 } },                  // index-eligible here, before any $group/$project
+  { $lookup: {
+      from: "customers",
+      localField: "customerId",
+      foreignField: "_id",
+      as: "customer",
+      pipeline: [ { $project: { name: 1 } } ]     // project inside $lookup: pull only what you need
+  }},
+  { $group: { _id: "$customerId", total: { $sum: "$amount" } } }
+], { allowDiskUse: true })  // $group/$sort spill past 100 MB/stage; this lets large groups complete,
+                            // it is NOT a substitute for a missing $match index — see anti-patterns.
+```
+
+### Read `explain("executionStats")` — the four numbers
+
+```javascript
+db.posts.find({ "author._id": authorId }).sort({ createdAt: -1 })
+        .explain("executionStats")
+```
+
+Read these before declaring a fix:
+
+1. **`winningPlan.stage`** — must be `IXSCAN` (or `FETCH`→`IXSCAN`), not `COLLSCAN`.
+2. **`totalKeysExamined` vs `nReturned`** — close means the index is selective; a huge ratio means
+   the index scans far more than it returns (wrong key order, low selectivity).
+3. **A `SORT` stage** — an in-memory sort the index should have satisfied; reorder by ESR to remove it.
+4. **`rejectedPlans`** — what the planner considered and dropped; a near-miss hints at a better index.
+
+### Multi-document transaction with full retry
+
+```javascript
+// Use withTransaction — it retries the body on TransientTransactionError and retries the
+// commit on UnknownTransactionCommitResult for you. Requires a replica set / sharded cluster.
+const session = db.getMongo().startSession();
+try {
+  session.withTransaction(() => {
+    const orders  = session.getDatabase("shop").orders;
+    const ledger  = session.getDatabase("shop").ledger;
+    orders.updateOne({ _id: orderId, status: "pending" }, { $set: { status: "paid" } }, { session });
+    ledger.insertOne({ orderId, amount: NumberDecimal("9.99"), at: new Date() }, { session });
+  }, { readConcern: { level: "snapshot" }, writeConcern: { w: "majority" } });
+} finally {
+  session.endSession();
+}
+// If both writes target ONE document, drop the transaction — that write is already atomic.
+```
+
+### `bulkWrite` upsert
+
+```javascript
+db.inventory.bulkWrite([
+  { updateOne: {
+      filter: { sku: "ABC-1" },
+      update: { $inc: { qty: 5 }, $setOnInsert: { createdAt: new Date() } },
+      upsert: true
+  }}
+], { ordered: false })  // ordered:false keeps going past one failed op and parallelizes
+```
+
+### Change stream (resumable tail)
+
+```javascript
+// Watch only the events you care about; persist resumeToken to restart without gaps.
+const cs = db.orders.watch([{ $match: { operationType: { $in: ["insert", "update"] } } }]);
+while (cs.hasNext()) { const change = cs.next(); /* process; save change._id as resume token */ }
+```
+
+More variants (`$facet`, window functions, `$merge`/`$out`, vector search) live in the references.
+
+## Anti-patterns / rationalizations → STOP
+
+| Rationalization | Reality → STOP |
+| --- | --- |
+| "Embed all the comments, it's one read" | Array grows unbounded toward 16 MB and bloats every post read. Reference or bucket. |
+| "`$lookup` is just a JOIN, use it everywhere" | Mongo is not relational; per-document `$lookup` is expensive. Prefer modeling (extended reference) so the read needs no join. |
+| "Wrap this single-document update in a transaction to be safe" | A single-doc write is already atomic. The transaction adds latency and a replica-set requirement for zero gain. |
+| "`COLLSCAN` is fine, it's fast on my 100 docs" | It is O(n); at 4M docs it is a full table read. Add the index now and prove `IXSCAN`. |
+| "Set `allowDiskUse:true` and the slow pipeline is fixed" | That masks a missing `$match` index by spilling to disk. Fix stage order / add the index first. |
+| "Store the price as a number, round on display" | JS doubles drift across `$sum`/`$inc`. Use `NumberDecimal` (`Decimal128`). |
+| "Group the whole collection, no `$match`" | A blocking `$group` over everything blows the 100 MB/stage limit. `$match` first to shrink it. |
+| "One collection for users, orders, logs — fewer to manage" | Mixed shapes kill index selectivity and balloon working set. Split by access pattern. |
+| "`$where` lets me run a quick JS predicate" | Runs JS per document, no index, a server-side injection surface. Use query operators / `$expr`. |
+| "Index every field just in case" | Each index is a write tax and RAM cost; unused indexes return nothing. Index for real query shapes only. |
+
+## Quick reference
+
+### Read/write concern matrix
+
+| Need | Write concern | Read concern | Note |
+| --- | --- | --- | --- |
+| Money / state transition | `w:"majority"` | `"majority"` | survives a primary failover |
+| Read your own durable write | `w:"majority"` | `"majority"` (+ causal session) | no rollback window |
+| Transaction default | `w:"majority"` | `"snapshot"` | consistent point-in-time |
+| Logs / fire-and-forget | `w:1` | `"local"` | fast, may be rolled back |
+
+### Atlas tier chooser
+
+| Tier | Use it for | Limits |
+| --- | --- | --- |
+| **M0** | learning, tiny prototypes | free forever, up to 5 GB, shared, no SLA |
+| **Flex** (GA Feb 2025) | small prod / variable load | $8 base capped at $30/mo, 100 ops/sec (burst 500), 5 GB; supports Atlas Search, Vector Search, Change Streams, Triggers |
+| **M10+** (dedicated) | production, isolation, scale-up | from ~$0.08/hr (~$57/mo); dedicated resources, full features |
+
+M0 does **not** run Vector Search well for real workloads — move to Flex or dedicated. Legacy
+Serverless / M2 / M5 were auto-migrated to Flex.
+
+### Aggregation memory
+
+Each blocking stage (`$group`, `$sort` without an index, `$bucket`) is capped at **100 MB**. Past it
+the stage errors unless `allowDiskUse:true` lets it spill. Spilling is a correctness fallback for
+genuinely large groups, not a performance fix for a missing index.
+
+## Verify
+
+Run `scripts/verify.sh` from your project root. It is read-only, never connects to a database, and
+never writes. It scans discovered `.js`/`.mongodb.js` files and flags foot-guns: a committed
+plaintext `mongodb://user:pass@` credential (the **only** hard failure), `createIndex` calls with no
+options, redundant compound-index prefixes, `$where` predicates, unbounded `$lookup`,
+`allowDiskUse:true` that may be masking a missing index, and money stored as a JS number in seed
+scripts. If `node` is present it runs `node --check` for a syntax pass; otherwise that step is
+`[skip]`. Everything except a committed credential is advisory `[warn]`/`[skip]`. It runs on stock
+macOS bash 3.2 and exits 0 on a clean or empty target.
+
+## Project grounding (02-DOCS + CLAUDE.md)
+
+When this skill runs in a project with a `02-DOCS/` layer (the [`harness`](../harness/SKILL.md)
+Karpathy wiki), record this project's MongoDB decisions there and index them from the root
+`CLAUDE.md`, so the next agent inherits the conventions instead of re-deriving them.
+
+1. **Find the article** `02-DOCS/wiki/stack/mongodb.md`, indexed in `02-DOCS/wiki/index.md` (the
+   Knowledge map index; root `CLAUDE.md` points to it).
+2. **If missing or stale**, create/update it with the project's real choices — collection layout and
+   embed/reference decisions, the index set and its ESR rationale, read/write concern policy, the
+   Atlas tier, and any encryption/RBAC setup — then index it in `02-DOCS/wiki/index.md` (the
+   Knowledge map; root `CLAUDE.md` keeps only a short pointer to it).
+3. **Read it first on every use** and stay consistent; when a convention changes, update the article
+   (bump its `Updated` date) in the same change.
+
+No `02-DOCS/` layer? Skip silently (optionally suggest `harness`). Technical conventions are
+*recorded, not gated* — never block the task on this.
+
+## See Also
+
+- [references/data-modeling.md](references/data-modeling.md) — embed/reference tree, the six
+  patterns with worked documents, 16 MB math, polymorphic & schema versioning.
+- [references/aggregation.md](references/aggregation.md) — per-stage index eligibility, `$lookup`
+  variants, `$facet`, window functions, `$merge`/`$out`, hybrid `$scoreFusion`, reading pipeline
+  `explain`.
+- [references/transactions-and-ops.md](references/transactions-and-ops.md) — retry wrappers, concern
+  semantics, replica-set requirement, Atlas tiers, Search/Vector Search, Queryable Encryption, RBAC,
+  pooling, change streams.
+- Sibling skills: [`harness`](../harness/SKILL.md) (scaffolds the `01-TOOLS/MONGODB` operational
+  tool) and [`secure-coding`](../secure-coding/SKILL.md) (auth, encryption, least-privilege).
+- For relational work — SQL, foreign keys, `EXPLAIN ANALYZE`, MVCC — use
+  [`postgresdb`](../postgresdb/SKILL.md), not this skill. Different engine and planner.
+- Out of scope here — external tools with their own docs: ODM/driver API surface (Mongoose hooks,
+  the Node driver's `bulkWrite`/`updateMany` return shapes) and cross-engine vector-store selection.
+  This skill owns the server query and the index the server picks.

package/skills/shared/mongodb/evals/README.md

@@ -0,0 +1,29 @@
+# Eval harness — `mongodb`
+
+These cases check two things: **triggering** (does the skill fire on real MongoDB problems and stay
+quiet on near-misses?) and **capability** (does loading it measurably improve the answer?). Cases
+live in `cases.yaml` and are **LLM-graded against the rubric — no live MongoDB is needed**; run them
+through the repo's agent eval harness, not a shell script.
+
+## Triggering
+
+For each prompt, start a fresh agent session with only the `mongodb` skill discoverable (its
+`SKILL.md` description in scope, no siblings loaded), paste the prompt verbatim, and run 3–5 trials
+(the trigger decision is stochastic):
+
+- `should_trigger`: PASS if the skill fires in the majority of trials. Several prompts deliberately
+  avoid the words "index" / "MongoDB" (the COLLSCAN complaint, the Spanish modeling request) to test
+  intent recognition over keyword matching.
+- `should_not_trigger`: PASS if the skill does **not** fire. Each names a `route_to` sibling
+  (`postgresdb`, `nodejs`, `secure-coding`, `redis`, `vector-db`); a wrong-but-not-`mongodb` route
+  still passes this skill's gate.
+
+Pass bar: ≥ 90% trigger accuracy across all trigger cases.
+
+## Capability
+
+Run the `capability` scenario twice — once WITHOUT the skill (clean agent) and once WITH it — and
+grade each transcript against the `must_include` checklist, one point per bullet that is correctly
+and specifically covered (not merely name-dropped). Pass bar: WITH the skill covers ≥ 80% of bullets
+and clearly beats WITHOUT (target ≥ +30 points). Record trial counts and per-bullet verdicts so a
+reviewer can audit; don't report a bare pass/fail.

package/skills/shared/mongodb/evals/cases.yaml

@@ -0,0 +1,41 @@
+skill: mongodb
+
+should_trigger:
+  - prompt: "This find() takes forever and the logs say it's scanning the whole collection — 4 million docs. How do I make it fast?"
+    why: "A full collection scan (COLLSCAN) on a large collection is core scope; the skill owns reading explain and choosing the index, even though the user never says 'index' or 'COLLSCAN'."
+  - prompt: "Should I put the comments inside the post document or keep them in their own collection?"
+    why: "Embed vs reference is a named modeling decision with an explicit rule and the 16 MB rationale."
+  - prompt: "My aggregation with a $lookup and a $group is slow and sometimes errors out on memory. What's wrong?"
+    why: "Pipeline stage order, $lookup cost, and the 100 MB blocking-stage limit / allowDiskUse are all in scope."
+  - prompt: "My multi-document transaction keeps throwing TransientTransactionError and I don't know how to handle it."
+    why: "Non-obvious: the fix is the mandatory retry-on-the-transient-label (withTransaction), a core transactions topic the user wouldn't phrase as 'retry'."
+  - prompt: "Necesito modelar una colección de pedidos con muchísimos items que crece sin límite, cómo lo hago?"
+    why: "Spanish; unbounded array growth → reference or bucket pattern, a named modeling case."
+  - prompt: "Which Atlas tier do I actually need to run Vector Search — is the free M0 enough?"
+    why: "Non-obvious tier boundary: M0 won't run Vector Search for real workloads; Atlas tier choice + Vector Search are explicitly in scope."
+
+should_not_trigger:
+  - prompt: "Design a normalized Postgres schema with foreign keys and help me read this EXPLAIN ANALYZE output."
+    route_to: "postgresdb"
+    why: "Relational engine, SQL, and EXPLAIN ANALYZE — a different planner and concurrency model; the named boundary sibling."
+  - prompt: "My Mongoose pre-save hook isn't firing and I don't understand what updateMany returns."
+    route_to: "nodejs"
+    why: "ODM/driver API ergonomics, not engine behavior; nearest catalog skill is the JS runtime, the ODM's own docs own this."
+  - prompt: "Audit my repo for hardcoded credentials and unsafe authentication patterns."
+    route_to: "secure-coding"
+    why: "A security audit of the codebase, not MongoDB modeling/indexing; secure-coding owns app-layer security review."
+  - prompt: "Set up a Redis cache in front of my read queries to cut database load."
+    route_to: "redis"
+    why: "App-layer caching with a different store; explicitly out of scope for this engine skill."
+  - prompt: "Help me choose between Pinecone and Weaviate for our vector store."
+    route_to: "vector-db"
+    why: "Cross-engine vector-DB selection is out of scope; Atlas Vector Search inside Mongo would be in scope, but choosing between external products is not."
+
+capability:
+  - scenario: "A posts/comments app where the number of comments per post is unbounded, and a feed query filters by author and sorts by date. The user asks how to model the collections and which index to add."
+    must_include:
+      - "Chooses to reference comments in a separate collection (not an unbounded embedded array), citing the 16 MB document cap — or proposes a bucket pattern for the comment stream"
+      - "Designs a compound index ordered by ESR: author equality first, then createdAt as the sort key (e.g. { 'author._id': 1, createdAt: -1 })"
+      - "Shows explain('executionStats') and reads it: confirms IXSCAN (not COLLSCAN) and totalKeysExamined close to nReturned, with no in-memory SORT stage"
+      - "States that inserting a single comment or post is already atomic, so no multi-document transaction is needed for it"
+      - "Uses mongosh syntax (createIndex/find/explain), money as Decimal128 if money appears, not relational/SQL framing"

package/skills/shared/mongodb/references/aggregation.md

@@ -0,0 +1,115 @@
+# Aggregation pipelines (MongoDB 8.2)
+
+A pipeline is stages run in order; the order decides whether an index is used and whether a stage
+blows the memory limit. The single most important rule: **put `$match` and `$sort` as early as
+possible**, because once a `$group`, `$project`, `$unwind`, or `$lookup` has reshaped the stream, the
+original index no longer applies.
+
+## Per-stage index eligibility
+
+| Stage | Uses an index? | Notes |
+| --- | --- | --- |
+| `$match` (first stage) | yes | behaves like a `find` filter; ESR rules apply |
+| `$sort` (before any reshape) | yes, if a compound index covers filter+sort | after `$group`/`$project` it is an in-memory SORT |
+| `$match` after `$group`/`$project` | no | runs on computed/in-memory docs |
+| `$lookup` | uses the *foreign* collection's index on `foreignField` | index the join target |
+| `$group` | no (blocking) | 100 MB/stage; `allowDiskUse` to spill |
+| `$unwind` | no | can multiply document count — `$match` before it |
+| `$limit` / `$skip` | benefits from a preceding indexed `$sort` | push `$limit` up where correctness allows |
+
+## `$lookup` variants
+
+```javascript
+// Simple equality join — index customers._id (it is the _id, already indexed).
+{ $lookup: { from: "customers", localField: "customerId", foreignField: "_id", as: "customer" } }
+
+// Subpipeline join — project/filter the foreign side so you pull only what you display.
+{ $lookup: {
+    from: "customers",
+    let: { cid: "$customerId" },
+    pipeline: [
+      { $match: { $expr: { $eq: ["$_id", "$$cid"] } } },
+      { $project: { name: 1, tier: 1 } }
+    ],
+    as: "customer"
+}}
+```
+
+`$lookup` runs the foreign lookup per input document — it is not a hash join. If you do it on every
+read, model it away with the **extended reference** pattern (see data-modeling) instead.
+
+## `$facet` — multiple aggregations in one pass
+
+```javascript
+db.products.aggregate([
+  { $match: { active: true } },
+  { $facet: {
+      byCategory: [ { $group: { _id: "$category", n: { $sum: 1 } } } ],
+      priceStats: [ { $group: { _id: null, avg: { $avg: "$price" }, max: { $max: "$price" } } } ],
+      page:       [ { $sort: { createdAt: -1 } }, { $limit: 20 } ]
+  }}
+])
+// Note: the input to a $facet is materialized; the per-facet sub-pipelines do NOT use indexes.
+// Keep the upstream $match selective so the materialized set is small.
+```
+
+## Window functions (`$setWindowFields`)
+
+```javascript
+// Running total of amount per customer, ordered by date — no self-join.
+db.orders.aggregate([
+  { $setWindowFields: {
+      partitionBy: "$customerId",
+      sortBy: { createdAt: 1 },
+      output: { runningTotal: { $sum: "$amount", window: { documents: ["unbounded", "current"] } } }
+  }}
+])
+```
+
+## `$merge` / `$out` — materialized output
+
+```javascript
+// $merge: upsert results into a target collection (incremental rollups, materialized views).
+{ $merge: { into: "daily_rollup", on: "_id", whenMatched: "merge", whenNotMatched: "insert" } }
+
+// $out: replace the target collection wholesale. Destructive — overwrites. Prefer $merge for
+// incremental jobs; reach for $out only for a full rebuild.
+{ $out: "report_snapshot" }
+```
+
+## Hybrid search (8.2)
+
+MongoDB 8.2 adds the `$scoreFusion` stage, which combines the ranked results of multiple subqueries
+(for example an Atlas Search full-text query and a Vector Search semantic query) into one fused
+ranking — the building block for hybrid search without doing the score blending in application code.
+Pair it with a `$vectorSearch`/`$search` stage per input ranking. (Atlas/Search-index setup lives in
+transactions-and-ops.)
+
+## Memory limit and `allowDiskUse`
+
+Each blocking stage (`$group`, `$sort` without a supporting index, `$bucket`,
+`$setWindowFields`) is capped at **100 MB**. Past it the stage errors unless you pass
+`allowDiskUse: true`, which lets it spill to temporary files on disk.
+
+Spilling is the right tool for a genuinely large aggregation (an analytics rollup over the whole
+collection). It is the **wrong** tool to "fix" a slow pipeline whose real problem is a `$match` that
+isn't using an index — that just makes the unindexed scan spill instead of fixing it. Always check
+the explain first.
+
+## Reading a pipeline's explain
+
+```javascript
+db.orders.explain("executionStats").aggregate([
+  { $match: { status: "paid", createdAt: { $gte: since } } },
+  { $sort:  { createdAt: -1 } },
+  { $group: { _id: "$customerId", total: { $sum: "$amount" } } }
+])
+```
+
+Look for:
+
+- The first stage shows `IXSCAN`, not `COLLSCAN` — the leading `$match` used an index.
+- No surprise in-memory `SORT` stage where a compound index (ESR) should have served the order.
+- `totalDocsExamined` close to the count the `$match` should return — a big gap means the index
+  isn't selective enough or the wrong one was chosen.
+- For `$lookup`, the sub-explain shows an `IXSCAN` on the foreign `foreignField`.

package/skills/shared/mongodb/references/data-modeling.md

@@ -0,0 +1,135 @@
+# Data modeling (MongoDB 8.2)
+
+The rule that overrides every other: **model for the queries you run, not the shape of your data.**
+A document should make your common read a single, index-eligible fetch. Everything below is a tool
+for getting there.
+
+## The embed-vs-reference decision tree
+
+Walk it top to bottom; the first match wins.
+
+1. **Is the child read together with the parent, almost always?** No → reference.
+2. **Is the child set bounded and small (will it stay under a few dozen, forever)?** No → reference.
+3. **Is the child shared across multiple parents, or independently/frequently mutated?** Yes →
+   reference (embedding would duplicate and drift, or rewrite a big parent per child edit).
+4. **Could the embedded array ever push the document toward 16 MB?** Yes → reference or bucket.
+5. **Otherwise → embed.** One read, atomic update, no `$lookup`.
+
+The 16 MB cap is the hard backstop. There is no overflow storage; a write that would exceed it
+fails. So any one-to-many whose "many" is unbounded (comments, events, line items on a long-lived
+cart) is referenced or bucketed, never embedded as a growing array.
+
+### 16 MB math (sanity-check before you embed)
+
+Estimate average child size × expected count, with headroom. A 300-byte comment subdocument hits
+16 MB at roughly 52,000 comments — but you are in trouble long before that, because every read of the
+parent deserializes the whole array. Treat a few hundred unbounded children as the practical ceiling
+for embedding, and reference past it.
+
+## The six patterns, with worked Bad → Good
+
+### 1. Subset
+
+Embed only the hot fields the common view needs; reference the cold rest.
+
+```javascript
+// BAD: product carries its full 200-review history; the catalog card read drags all of it.
+db.products.insertOne({ _id: ObjectId(), name: "Lamp", reviews: [ /* 200 full reviews */ ] })
+
+// GOOD: embed the 3 most-recent reviews for the card; full set lives in its own collection.
+db.products.insertOne({
+  _id: ObjectId(), name: "Lamp",
+  recentReviews: [ { author: "Ada", stars: 5, text: "…" } /* top 3 only */ ],
+  reviewCount: 200
+})
+db.reviews.insertOne({ _id: ObjectId(), productId: ObjectId(), author: "Ada", stars: 5, text: "…" })
+```
+
+### 2. Extended reference
+
+Copy the few joined fields you display so the read needs no `$lookup`.
+
+```javascript
+// BAD: every order render does a $lookup into customers just to show a name.
+db.orders.insertOne({ _id: ObjectId(), customerId: ObjectId(), amount: NumberDecimal("9.99") })
+
+// GOOD: duplicate the small, slow-changing fields you actually display.
+db.orders.insertOne({
+  _id: ObjectId(),
+  customer: { _id: ObjectId(), name: "Ada", tier: "gold" }, // the display fields, copied
+  amount: NumberDecimal("9.99")
+})
+// Trade-off: when the source name changes, refresh the copies (a background job or change stream).
+```
+
+### 3. Bucket
+
+Group N time-ordered events per document instead of one document per event — fewer documents,
+smaller indexes, and the array stays bounded by the bucket size.
+
+```javascript
+// BAD: one document per sensor reading — millions of tiny docs, huge index.
+db.readings.insertOne({ sensorId: ObjectId(), ts: new Date(), value: 21.4 })
+
+// GOOD: one bucket per sensor per hour, capped at N readings; roll to a new bucket past the cap.
+db.readings.updateOne(
+  { sensorId: ObjectId(), hour: ISODate("2026-06-02T10:00:00Z"), count: { $lt: 200 } },
+  { $push: { samples: { ts: new Date(), value: 21.4 } }, $inc: { count: 1 } },
+  { upsert: true }
+)
+// For pure metrics, also consider a native time-series collection (timeField/metaField).
+```
+
+### 4. Computed
+
+Store the rollup; update it on write rather than recomputing on every read.
+
+```javascript
+// GOOD: keep commentCount on the post; bump it when a comment is inserted.
+db.posts.updateOne({ _id: postId }, { $inc: { commentCount: 1 } })
+// Read is O(1); you trade a tiny write cost for avoiding a count aggregation per page load.
+```
+
+### 5. Outlier
+
+When ~1% of documents break the shape (a celebrity with millions of followers), flag them and
+overflow into linked documents so the common case stays small.
+
+```javascript
+db.users.insertOne({
+  _id: ObjectId(), name: "Ada",
+  followerIds: [ /* up to N */ ],
+  hasExtras: false            // flip to true and spill into a followers_overflow collection past N
+})
+```
+
+### 6. Polymorphic
+
+One collection holding several entity shapes, distinguished by a `type` field, sharing an `_id`
+space and the indexes they have in common.
+
+```javascript
+db.assets.insertMany([
+  { _id: ObjectId(), type: "image", url: "…", width: 800, height: 600 },
+  { _id: ObjectId(), type: "video", url: "…", durationSec: 42 }
+])
+// Index the shared fields (type, createdAt); query with the discriminator in the filter.
+```
+
+## Schema versioning
+
+Documents in one collection can have different shapes over time. Stamp a `schemaVersion` and migrate
+lazily on read/write — no big-bang migration, no downtime.
+
+```javascript
+db.users.insertOne({ _id: ObjectId(), schemaVersion: 2, name: "Ada", emails: ["ada@x.io"] })
+// App reads: if doc.schemaVersion < 2, upconvert in code (and optionally write the upgraded shape back).
+```
+
+## `_id` discipline
+
+- Default `ObjectId` is fine: 12 bytes, roughly time-ordered, generated client- or server-side.
+- A natural unique key (a slug, an external id) is a valid `_id` — but it is immutable, so only use
+  one that never changes.
+- Don't add a second unique field as your "real" key and ignore `_id`; make the meaningful key the
+  `_id` or give it its own unique index.