vybekiit 0.4.0 → 0.6.0

package/dist/global-skills/mongodb-schema-design/references/pattern-archive.md

@@ -0,0 +1,143 @@
+---
+title: Use Archive Pattern for Historical Data
+impact: MEDIUM
+impactDescription: "Reduces active collection size, improves query performance, lowers storage costs"
+tags: schema, patterns, archive, data-lifecycle, merge, ttl, online-archive
+---
+
+## Use Archive Pattern for Historical Data
+
+**Storing old data alongside recent data degrades performance.** As collections grow with historical data that's rarely accessed, queries slow down, indexes bloat, and working set exceeds RAM. The archive pattern moves old data to separate storage while keeping your active collection fast.
+
+**Incorrect (all data in one collection):**
+
+A sales collection with 5 years of data (50M documents) where only the recent 6 months are actively queried suffers from: indexes covering the full 50M documents when only ~1M are relevant, working set including old data pages, backups including rarely-accessed history, and hot-tier storage costs for data that could be cold.
+
+**Correct (archive old data separately):**
+
+```javascript
+// Step 1: Define archive threshold (older than 6 months)
+const sixMonthsAgo = new Date()
+sixMonthsAgo.setMonth(sixMonthsAgo.getMonth() - 6)
+
+// Step 2: Move old data to archive collection using $merge
+db.sales.aggregate([
+  { $match: { date: { $lt: sixMonthsAgo } } },
+  { $merge: {
+      into: "sales_archive",
+      on: "_id",
+      whenMatched: "keepExisting",  // Don't overwrite if re-run
+      whenNotMatched: "insert"
+    }
+  }
+])
+
+// Step 3: Delete archived data from active collection
+db.sales.deleteMany({ date: { $lt: sixMonthsAgo } })
+
+// Result:
+// - sales: Recent data, fast queries, small indexes
+// - sales_archive: Historical data, rarely queried
+```
+
+**Archive storage options (best to worst for cost/performance):**
+
+1. **External file storage (S3, cloud object storage)** — Best for compliance and long-term retention at lowest cost. Export to JSON/BSON, store in S3, query via Atlas Data Federation when needed.
+2. **Separate, cheaper cluster** — Best for occasional historical queries. Replicate to a lower-tier Atlas cluster at reduced cost.
+3. **Separate collection on same cluster** — Best for simple implementation with frequent historical access. As shown above with `sales_archive`, but still uses the same storage tier.
+4. **Atlas Online Archive (Atlas only)** — MongoDB manages automatic movement to cloud object storage; query via Federated Database Instance.
+
+**Design tips for archivable schemas:**
+
+```javascript
+// TIP 1: Use embedded data model for archives
+// Archived data must be self-contained
+
+// BAD: References that may be deleted
+{
+  _id: "order123",
+  customerId: "cust456",  // Customer may be deleted
+  productIds: ["prod1", "prod2"]  // Products may change
+}
+
+// GOOD: Embedded snapshot of related data
+{
+  _id: "order123",
+  customer: {
+    _id: "cust456",
+    name: "Jane Doe",
+    email: "jane@example.com"
+  },
+  products: [
+    { _id: "prod1", name: "Widget", price: 29.99 },
+    { _id: "prod2", name: "Gadget", price: 49.99 }
+  ],
+  date: ISODate("2020-01-15")
+}
+
+// TIP 2: Store age in a single, indexable field
+// Makes archive queries efficient
+{
+  date: ISODate("2020-01-15"),  // Single field for age
+  // NOT: { year: 2020, month: 1, day: 15 }
+}
+
+// TIP 3: Handle "never expire" documents
+{
+  date: ISODate("2025-01-15"),
+  retentionPolicy: "permanent"  // Or use far-future date
+}
+
+// Archive query excludes permanent records:
+db.sales.aggregate([
+  { $match: {
+      date: { $lt: fiveYearsAgo },
+      retentionPolicy: { $ne: "permanent" }
+    }
+  },
+  { $merge: { into: "sales_archive" } }
+])
+```
+
+**Automated archival with scheduling:**
+
+Create a script (run via cron, Atlas Triggers, or an application scheduler) that:
+
+1. Counts documents older than the cutoff date (excluding those with `retentionPolicy: "permanent"`).
+2. Processes in batches (e.g. 10,000 IDs at a time) to avoid long-running operations: fetch a batch of `_id` values, pipe them through an aggregation with `$match` and `$merge` into the archive collection, then `deleteMany` the batch from the active collection.
+3. Logs progress after each batch.
+
+This reuses the same `$merge`-based archival shown above but throttles work to avoid overloading the cluster.
+
+**Atlas Online Archive (Atlas only):**
+
+Atlas Online Archive automatically tiers data to MongoDB-managed cloud object storage based on a date-field rule (e.g. archive after 365 days). Archived data is queried transparently via a Federated Database Instance — slightly slower but much cheaper. No application code changes are required.
+
+**When NOT to use archive pattern:**
+
+- **Small datasets**: If total data fits comfortably in RAM, archiving adds complexity without benefit.
+- **Uniform access patterns**: If old and new data are queried equally.
+- **Compliance requires instant access**: If regulations require sub-second queries on all historical data.
+- **Already using TTL**: If data should be deleted, not archived, use TTL indexes.
+
+## Verify with
+
+```javascript
+// Analyze archive candidates
+const cutoff = new Date()
+cutoff.setFullYear(cutoff.getFullYear() - 5)
+
+db.sales.aggregate([
+  { $facet: {
+      total: [{ $count: "count" }],
+      old: [
+        { $match: { date: { $lt: cutoff } } },
+        { $count: "count" }
+      ]
+    }
+  }
+])
+// If old documents are >30% of total, archiving can improve performance
+```
+
+Reference: [Archive Pattern](https://mongodb.com/docs/manual/data-modeling/design-patterns/archive/)

package/dist/global-skills/mongodb-schema-design/references/pattern-attribute.md

@@ -0,0 +1,77 @@
+---
+title: Use Attribute Pattern for Sparse or Variable Fields
+impact: MEDIUM
+impactDescription: "Reduces sparse indexes and enables efficient search across many optional fields"
+tags: schema, patterns, attribute, sparse-fields, indexing, flexible-schema
+---
+
+## Use Attribute Pattern for Sparse or Variable Fields
+
+**If documents have many optional fields, move them into a key-value array.** This avoids dozens of sparse indexes and lets you query across attributes with a single multikey index.
+
+**Incorrect (separate field and index per optional attribute):**
+
+```javascript
+// Many optional fields - most are missing on any given document
+{
+  _id: 1,
+  name: "Bottle",
+  color: "red",
+  size: "M",
+  material: "glass",
+  // 20+ other optional fields, varying per document
+}
+
+// One partial index per optional field — correct use of partialFilterExpression,
+// but you end up maintaining dozens of indexes as attributes grow
+db.items.createIndex({ color: 1 }, { partialFilterExpression: { color: { $exists: true } } })
+db.items.createIndex({ size: 1 }, { partialFilterExpression: { size: { $exists: true } } })
+db.items.createIndex({ material: 1 }, { partialFilterExpression: { material: { $exists: true } } })
+// … repeated for every new attribute
+```
+
+**Correct (attribute pattern):**
+
+```javascript
+// Store optional fields as key-value pairs
+{
+  _id: 1,
+  name: "Bottle",
+  attributes: [
+    { k: "color", v: "red" },
+    { k: "size", v: "M" },
+    { k: "material", v: "glass" }
+  ]
+}
+
+// Single multikey index for all attributes
+
+db.items.createIndex({ "attributes.k": 1, "attributes.v": 1 })
+
+// Query for color = red
+
+db.items.find({
+  attributes: { $elemMatch: { k: "color", v: "red" } }
+})
+```
+
+**When NOT to use this pattern:**
+
+- **Fixed schema**: If fields are stable and always present.
+- **Type-specific validation**: If each field needs strict schema rules.
+- **Single-field queries only**: A normal field may be simpler and faster.
+- **Atlas Search workloads**: The `{ k, v }` key-value structure cannot be mapped as
+  named fields in Atlas Search indexes. If you need full-text search on attribute
+  values by key name, use static named fields instead.
+
+## Verify with
+
+```javascript
+// Ensure queries use the multikey index
+
+db.items.find({
+  attributes: { $elemMatch: { k: "material", v: "glass" } }
+}).explain("executionStats")
+```
+
+Reference: [Attribute Pattern](https://mongodb.com/docs/manual/data-modeling/design-patterns/group-data/attribute-pattern/)

package/dist/global-skills/mongodb-schema-design/references/pattern-bucket.md

@@ -0,0 +1,102 @@
+---
+title: Use Bucket Pattern to Group Related Data
+impact: MEDIUM
+impactDescription: "Reduces document count and can align storage with application access patterns like pagination"
+tags: schema, patterns, bucket, grouping, pagination, arrays
+---
+
+## Use Bucket Pattern to Group Related Data
+
+**Group a series of related items into bounded arrays within a single document.** The bucket pattern separates long series of data into distinct objects, reducing document count and aligning storage with how data is actually consumed. This is especially useful when an application accesses data in fixed-size groups (e.g. pages).
+
+> **For time-series data**, prefer [Time Series Collections](https://www.mongodb.com/docs/manual/core/timeseries-collections/), which apply bucketing automatically with built-in compression and indexing optimizations.
+
+**Incorrect (one document per event):**
+
+Storing one document per stock trade (e.g. `{ ticker, customerId, type, quantity, date }`) means the application pages through trades using skip/limit, which degrades as offset grows. Each trade is a separate document and index entry.
+
+**Correct (bucket pattern - group by customer, bounded per page):**
+
+Each document holds up to N trades for one customer (e.g. 10 trades = one page). The `_id` encodes customer ID and the first trade’s epoch seconds (e.g. `"123_1698349623"`), with a `count` field and a `history` array of trade objects. One bucket equals one page of data — a regex on `_id` uses the default `_id` index with no extra index needed, and document count drops by up to the bucket-size factor.
+
+**Insert with atomic upsert:**
+
+```javascript
+// Insert a new trade into the correct bucket
+db.trades.findOneAndUpdate(
+  {
+    "_id": /^123_/,            // Match buckets for this customer
+    "count": { $lt: 10 }       // Only if bucket isn't full
+  },
+  {
+    $push: {
+      history: {
+        type: "buy",
+        ticker: "MSFT",
+        qty: 42,
+        date: ISODate("2023-11-02T11:43:10Z")
+      }
+    },
+    $inc: { count: 1 },
+    $setOnInsert: {
+      _id: "123_1698939791",   // New bucket ID if upsert fires
+      customerId: 123
+    }
+  },
+  { upsert: true, sort: { _id: -1 } }
+)
+// If a bucket with room exists, the trade is pushed into it
+// Otherwise a new bucket document is created
+// Array is bounded — never exceeds 10 elements
+```
+
+**Query patterns:**
+
+```javascript
+// Page 1 of trades for customer 123
+db.trades.find({ _id: /^123_/ }).sort({ _id: 1 }).limit(1)
+
+// Page N (e.g. page 10)
+db.trades.find({ _id: /^123_/ }).sort({ _id: 1 }).skip(9).limit(1)
+
+// Each returned document IS a page — no per-trade skip/limit needed
+```
+
+**Choosing bucket boundaries:**
+
+| Bucketing Strategy | Good For | Example |
+|-------------------|----------|---------|
+| Fixed count (N items) | Pagination, evenly-sized pages | 10 trades per bucket |
+| Time window | Log/event grouping (when not using Time Series Collections) | 1 hour of events per bucket |
+| Logical grouping | Domain-driven partitioning | All line items in one order |
+
+**When NOT to use this pattern:**
+
+- **Time-series workloads**: Use [Time Series Collections](https://www.mongodb.com/docs/manual/core/timeseries-collections/) instead — they handle bucketing, compression, and indexing automatically.
+- **Random single-item access**: If you frequently query individual items by their own ID, buckets add unnecessary indirection.
+- **Low volume**: If the total series per entity is small, the added complexity isn't worth it.
+- **Highly variable item sizes**: Bucketing works best when items are roughly uniform in size so bucket documents stay predictable.
+
+## Verify with
+
+```javascript
+// Check that bucket size matches expectations
+db.trades.aggregate([
+  { $group: {
+    _id: null,
+    avgCount: { $avg: "$count" },
+    maxCount: { $max: "$count" },
+    totalBuckets: { $sum: 1 }
+  }}
+])
+// avgCount should approach your target bucket size
+// maxCount should not exceed it
+
+// Check average document size
+db.trades.aggregate([
+  { $project: { size: { $bsonSize: "$$ROOT" } } },
+  { $group: { _id: null, avgSize: { $avg: "$size" } } }
+])
+```
+
+Reference: [Group Data with the Bucket Pattern](https://www.mongodb.com/docs/manual/data-modeling/design-patterns/group-data/bucket-pattern/)

package/dist/global-skills/mongodb-schema-design/references/pattern-computed.md

@@ -0,0 +1,162 @@
+---
+title: Use Computed Pattern for Expensive Calculations
+impact: MEDIUM
+impactDescription: "Improves read latency by pre-computing frequently-requested aggregations"
+tags: schema, patterns, computed, aggregation, performance, denormalization
+---
+
+## Use Computed Pattern for Expensive Calculations
+
+**Pre-calculate and store frequently-accessed computed values.** If you're running the same aggregation on every page load, you're wasting CPU cycles. Store the result in the document and update it on write or via background job—trades write complexity for read speed.
+
+**Incorrect (calculate on every read):**
+
+```javascript
+// Movie with all screenings in separate collection
+{ _id: "movie1", title: "The Matrix" }
+
+// Screenings collection - thousands of records
+{ movieId: "movie1", date: ISODate("..."), viewers: 344, revenue: 3440 }
+{ movieId: "movie1", date: ISODate("..."), viewers: 256, revenue: 2560 }
+// ... 10,000 screenings
+
+// Movie page aggregates every time
+db.screenings.aggregate([
+  { $match: { movieId: "movie1" } },
+  { $group: {
+    _id: "$movieId",
+    totalViewers: { $sum: "$viewers" },
+    totalRevenue: { $sum: "$revenue" },
+    screeningCount: { $sum: 1 }
+  }}
+])
+// Repeated scans can add substantial read latency and CPU overhead
+// 1M page views/day = 1M expensive aggregations
+```
+
+**Correct (pre-computed values):**
+
+Store computed stats directly in the movie document: `stats.totalViewers`, `stats.totalRevenue`, `stats.screeningCount`, `stats.avgViewersPerScreening`, and `stats.computedAt`. The movie page reads a single document with no aggregation needed on the hot path.
+
+**Update strategies:**
+
+```javascript
+// Strategy 1: Update on write (low write volume)
+// When new screening is added
+db.screenings.insertOne({
+  movieId: "movie1",
+  viewers: 400,
+  revenue: 4000
+})
+
+// Immediately update computed values
+db.movies.updateOne(
+  { _id: "movie1" },
+  {
+    $inc: {
+      "stats.totalViewers": 400,
+      "stats.totalRevenue": 4000,
+      "stats.screeningCount": 1
+    },
+    $set: { "stats.computedAt": new Date() }
+  }
+)
+
+// Strategy 2: Background job (high write volume)
+// Run hourly/daily aggregation job
+db.screenings.aggregate([
+  { $group: {
+    _id: "$movieId",
+    totalViewers: { $sum: "$viewers" },
+    totalRevenue: { $sum: "$revenue" },
+    count: { $sum: 1 }
+  }},
+  { $merge: {
+    into: "movies",
+    on: "_id",
+    whenMatched: [{
+      $set: {
+        "stats.totalViewers": "$$new.totalViewers",
+        "stats.totalRevenue": "$$new.totalRevenue",
+        "stats.screeningCount": "$$new.count",
+        "stats.computedAt": "$$NOW"
+      }
+    }]
+  }}
+])
+```
+
+**Common computed values:**
+
+| Source Data | Computed Value | Update Strategy |
+|-------------|----------------|-----------------|
+| Order line items | Order total | On write (single doc) |
+| Product reviews | Avg rating, review count | Background job |
+| User activity | Engagement score | Background job |
+| Transaction history | Account balance | On write |
+| Page views | View count, trending score | Batched updates |
+
+**Handling staleness:**
+
+Include a `computedAt` timestamp alongside the stats. Application code compares this timestamp against a freshness threshold (e.g. one hour) and triggers a refresh if the values are stale. Alternatively, surface the timestamp to users (e.g. “1,840,000 viewers — updated 1 hour ago”).
+
+**Windowed computations:**
+
+```javascript
+// Compute for time windows (rolling 30 days)
+{
+  _id: "movie1",
+  stats: {
+    allTime: { viewers: 1840000, revenue: 25880000 },
+    last30Days: { viewers: 45000, revenue: 630000 },
+    last7Days: { viewers: 12000, revenue: 168000 }
+  }
+}
+
+// Background job updates rolling windows
+db.screenings.aggregate([
+  { $match: {
+    movieId: "movie1",
+    date: { $gte: thirtyDaysAgo }
+  }},
+  { $group: {
+    _id: null,
+    viewers: { $sum: "$viewers" },
+    revenue: { $sum: "$revenue" }
+  }}
+])
+// Then update movie.stats.last30Days
+```
+
+**Consider on-demand materialized views:**
+
+When the computed results are best stored in a separate collection rather than embedded in the source documents, MongoDB's [on-demand materialized views](https://www.mongodb.com/docs/manual/core/materialized-views/) formalize this approach. An on-demand materialized view is an aggregation pipeline whose output is written to a separate collection using `$merge` or `$out`—the same mechanism shown in Strategy 2 above. The difference is conceptual: instead of updating a field on existing documents, you maintain a dedicated read-optimized collection that can be independently indexed. This is especially useful when:
+
+- The computed data has a different shape or granularity than the source (e.g. monthly summaries from daily records).
+- Multiple consumers need the pre-aggregated data, and a shared collection is cleaner than duplicating fields across documents.
+- You want to index the computed results independently of the source collection.
+
+On-demand materialized views are not automatically refreshed—you control when to re-run the pipeline, which gives you the same staleness trade-offs described above.
+
+**When NOT to use this pattern:**
+
+- **Rarely accessed calculations**: If stat is viewed once/day, compute on demand.
+- **High write frequency**: If source data changes every second, update overhead may exceed read savings.
+- **Complex multi-collection joins**: Some computations are too complex to maintain incrementally.
+- **Strong consistency required**: Computed values may be slightly stale.
+
+## Verify with
+
+### Find expensive aggregations that should be pre-computed
+
+For Atlas M10+ use slow query logs to find the slowest aggregations. See [Slow query logs](references/source-slow-query-logs.md). 
+Use codebase if available, ask the user.
+
+### Check if same aggregation runs repeatedly
+
+For Atlas M10+ use $queryStats. See [Query Stats](references/source-query-stats.md)
+Use codebase if available, ask the user.
+
+High count + high avgMs on an aggregation that computes a result = candidate for computed pattern
+
+Reference: [Computed Schema Pattern](https://mongodb.com/docs/manual/data-modeling/design-patterns/computed-values/computed-schema-pattern/)

package/dist/global-skills/mongodb-schema-design/references/pattern-document-versioning.md

@@ -0,0 +1,166 @@
+---
+title: "Document Versioning Pattern"
+impact: MEDIUM
+impactDescription: "Enables reproducing exact historical document state for audit, compliance, and rollback"
+tags: schema, patterns, versioning, audit, compliance
+---
+
+## Document Versioning Pattern
+
+**Store full document history in a separate `revisions` collection to enable reproducing historical state.** This is different from schema versioning (which handles field migration)—document versioning stores complete snapshots of each change. Use it for insurance policies, legal documents, compliance audit trails, and any data where you must reproduce exact historical state.
+
+**Incorrect (overwrite history with no trail):**
+
+```javascript
+// Policy document — only current state exists
+{
+  _id: "POL-001",
+  holder: "Jane Smith",
+  premium: 450,
+  coverage: "comprehensive",
+  updatedAt: ISODate("2024-06-01")
+}
+
+// When premium changes, old value is lost forever
+db.policies.updateOne(
+  { _id: "POL-001" },
+  { $set: { premium: 475, updatedAt: new Date() } }
+)
+// Previous premium of 450 is gone — no audit trail
+// Cannot reproduce what the policy looked like on 2024-03-15
+// Compliance audit fails: "show me the policy as of Q1"
+```
+
+**Correct (current collection + revisions collection):**
+
+```javascript
+// currentPolicies collection — current state only (fast reads)
+{
+  _id: "POL-001",
+  holder: "Jane Smith",
+  premium: 450,
+  coverage: "comprehensive",
+  v: 3,
+  updatedAt: ISODate("2024-06-01")
+}
+
+// policyRevisions collection — full history snapshots
+{
+  policyId: "POL-001",
+  v: 2,
+  snapshot: {
+    holder: "Jane Smith",
+    premium: 425,
+    coverage: "basic",
+    v: 2
+  },
+  changedAt: ISODate("2024-03-15")
+}
+```
+
+**Implementation:**
+
+```javascript
+async function updatePolicy(policyId, newData, session) {
+  const current = await db.currentPolicies.findOne({ _id: policyId }, { session })
+
+  await db.policyRevisions.insertOne({
+    policyId: current._id,
+    v: current.v,
+    snapshot: { ...current },
+    changedAt: new Date()
+  }, { session })
+
+  await db.currentPolicies.updateOne(
+    { _id: policyId },
+    { $set: { ...newData, v: current.v + 1, updatedAt: new Date() } },
+    { session }
+  )
+}
+
+async function getPolicyAtVersion(policyId, version) {
+  if (version === 'current') {
+    return db.currentPolicies.findOne({ _id: policyId })
+  }
+  const rev = await db.policyRevisions.findOne({ policyId, v: version })
+  return rev?.snapshot
+}
+```
+
+**Using Transactions for Atomicity:**
+
+The `updatePolicy` function writes to two collections (inserting a revision **and** updating the current document). It may or may not be prudent to wrap the call in a [multi-document transaction](https://mongodb.com/docs/manual/core/transactions/) to guarantee both writes succeed or fail together, depending on the use case:
+
+```javascript
+const session = client.startSession()
+try {
+  await session.withTransaction(async () => {
+    await updatePolicy("POL-001", { premium: 475, coverage: "premium" }, session)
+  })
+} finally {
+  await session.endSession()
+}
+```
+
+**Indexes:**
+
+```javascript
+db.policyRevisions.createIndex({ policyId: 1, v: -1 })
+// Optional TTL for retention (e.g., 7 years)
+db.policyRevisions.createIndex({ changedAt: 1 }, { expireAfterSeconds: 220752000 })
+```
+
+**Difference from Schema Versioning:**
+
+| Pattern | Purpose | Stores |
+|---------|---------|--------|
+| Schema Versioning | Handle field structure migration | `schemaVersion` field on each doc |
+| Document Versioning | Reproduce complete historical state | Full snapshots in revisions collection |
+
+**When NOT to use this pattern:**
+
+- **High-frequency updates**: If documents change many times per second, use event sourcing instead.
+- **Approximate history is sufficient**: If you only need to know "what changed" but not reproduce exact state.
+- **Unbounded revision growth without retention**: Ensure you have a TTL or archival policy for the revisions collection.
+
+## Verify with
+
+```javascript
+// Check revision collection growth
+db.policyRevisions.aggregate([
+  { $group: {
+    _id: "$policyId",
+    revisionCount: { $sum: 1 },
+    oldestRevision: { $min: "$changedAt" },
+    newestRevision: { $max: "$changedAt" }
+  }},
+  { $sort: { revisionCount: -1 } },
+  { $limit: 10 }
+])
+// Monitor for documents with unexpectedly high revision counts
+
+// Verify current docs have version field
+db.currentPolicies.countDocuments({ v: { $exists: false } })
+// Should be 0 — all documents need version tracking
+
+// Check that revisions are consistent with current version
+db.currentPolicies.aggregate([
+  { $lookup: {
+    from: "policyRevisions",
+    localField: "_id",
+    foreignField: "policyId",
+    as: "revisions"
+  }},
+  { $project: {
+    currentVersion: "$v",
+    revisionCount: { $size: "$revisions" },
+    maxRevisionVersion: { $max: "$revisions.v" }
+  }},
+  { $match: {
+    $expr: { $ne: [{ $subtract: ["$currentVersion", 1] }, "$maxRevisionVersion"] }
+  }}
+])
+// Finds documents where revision history has gaps
+```
+
+Reference: [Keep a History of Document Versions](https://mongodb.com/docs/manual/data-modeling/design-patterns/data-versioning/document-versioning/)