vybekiit 0.4.0 → 0.6.0

package/dist/global-skills/mongodb-schema-design/references/pattern-extended-reference.md

@@ -0,0 +1,72 @@
+---
+title: Use Extended Reference Pattern
+impact: MEDIUM
+impactDescription: "Reduces repeated `$lookup` on hot paths by caching selected referenced fields"
+tags: schema, patterns, extended-reference, denormalization, caching
+---
+
+## Use Extended Reference Pattern
+
+**Copy frequently-accessed fields from referenced documents into the parent.** If you always display author name with articles, embed it. This eliminates $lookup for common queries while keeping the full data normalized—best of both worlds.
+
+**Incorrect (always $lookup for display data):**
+
+```javascript
+// Order references customer by ID only
+{
+  _id: "order123",
+  customerId: "cust456",  // Customer reference by ID only
+  items: [...],
+  total: 299.99
+}
+
+// Every order list/display requires $lookup
+db.orders.aggregate([
+  { $match: { status: "pending" } },
+  { $lookup: {
+    from: "customers",
+    localField: "customerId",
+    foreignField: "_id",
+    as: "customer"
+  }},
+  { $unwind: "$customer" }
+])
+// Repeated joins add avoidable work for a common list view
+```
+
+**Correct (extended reference):**
+
+Embed frequently-needed customer fields directly in the order document: include a `customer` subdocument with `_id` (kept as a reference for full lookups), `name`, and `email`. The order list query returns customer display data without `$lookup`. Full customer data is still available via a targeted read to the `customers` collection when needed.
+
+**Keeping cached data in sync:**
+
+When the source field changes (e.g. customer name), update the source collection first, then update cached copies in the orders collection using `updateMany` on the embedded reference `_id`. This can be done synchronously or asynchronously via Change Streams / background jobs. For data that changes more often, add a `cachedAt` timestamp to the embedded subdocument so the application can refresh on read when the cache exceeds a staleness threshold.
+
+**What to cache (extend):**
+
+| Cache | Don't Cache |
+|-------|-------------|
+| Display name, avatar | Full bio, description |
+| Status, type | Sensitive PII |
+| Slowly-changing data | Real-time values (balance, inventory) |
+| Fields used in sorting/filtering | Large binary data |
+
+**Alternative: Hybrid pattern with cache expiry:**
+
+Keep both a bare reference (`customerId`) and an optional cache subdocument (`customerCache`) with `name`, `email`, and `cachedAt`. On read, if the cache is missing or older than a threshold (e.g. one day), refresh it from the `customers` collection and write the updated cache back to the order.
+
+**When NOT to use this pattern:**
+
+- **Frequently-changing data**: If customer name changes daily, update overhead exceeds $lookup cost.
+- **Large cached payloads**: Don't embed 50KB of author bio in every article.
+- **Sensitive data segregation**: Don't copy PII into collections with different access controls.
+- **Writes >> Reads**: If writes greatly outnumber reads, caching adds overhead.
+
+## Verify with
+
+Find lookup-heavy aggregations. See how often lookups hit the same collection. High count = candidate for extended reference
+
+For Atlas M10+ use $queryStats. See [Query Stats](references/source-query-stats.md) and [Slow query logs](references/source-slow-query-logs.md)
+Use codebase if available, ask the user.
+
+Reference: [Reduce $lookup Operations](https://mongodb.com/docs/manual/data-modeling/design-antipatterns/reduce-lookup-operations/)

package/dist/global-skills/mongodb-schema-design/references/pattern-outlier.md

@@ -0,0 +1,159 @@
+---
+title: Use Outlier Pattern for Exceptional Documents
+impact: MEDIUM
+impactDescription: "Isolates unusually large documents so hot-path queries stay optimized for typical cases"
+tags: schema, patterns, outlier, arrays, performance, edge-cases
+---
+
+## Use Outlier Pattern for Exceptional Documents
+
+**Isolate atypical documents with large arrays to prevent them from degrading performance for typical queries.** When a small subset of documents is much larger than the rest, those outliers can dominate memory, index, and query costs. Split overflow data into a separate collection and flag the document.
+
+**Problem scenario:**
+
+A typical book might have 50 customers in an embedded array, while a bestseller like Harry Potter accumulates 50,000 (~2.5MB). Queries return the full document, so the outlier dominates memory and network cost. A multikey index on that array produces 50,000 entries for a single document.
+
+**Correct (outlier pattern):**
+
+Typical documents keep their full embedded array and set `hasExtras: false`. Outlier documents cap the embedded array at a threshold (e.g. 50), set `hasExtras: true`, store a denormalized `customerCount`, and overflow remaining items into a separate collection in batched documents (e.g. `{ bookId, customers: [...], batch: 1, count: 950 }`). Application code checks the `hasExtras` flag to decide whether to load overflow batches.
+
+**Implementation with threshold (example; tune per workload):**
+
+```javascript
+const CUSTOMER_THRESHOLD = 50
+
+async function addCustomer(bookId, customerId) {
+  // Try the normal case first: atomically add to the embedded array only if
+  // the current customerCount is below the threshold (treat missing/null as 0).
+  const result = await db.books.updateOne(
+    {
+      _id: bookId,
+      $or: [
+        { customerCount: { $lt: CUSTOMER_THRESHOLD } },
+        { customerCount: { $exists: false } },
+        { customerCount: null }
+      ]
+    },
+    {
+      $push: { customers: customerId },
+      $inc: { customerCount: 1 }
+    }
+  )
+
+  if (result.matchedCount > 0) {
+    // Normal case succeeded - customer added to embedded array
+    return
+  }
+
+  // Outlier case - add to overflow collection
+  const lastBatchDoc = await db.book_customers_extra
+    .find({ bookId: bookId })
+    .sort({ batch: -1 })
+    .limit(1)
+    .next()
+
+  const nextBatch = lastBatchDoc ? lastBatchDoc.batch + 1 : 1
+  const targetBatch =
+    lastBatchDoc && lastBatchDoc.count < 1000
+      ? lastBatchDoc.batch
+      : nextBatch
+
+  // First, try to append to the intended batch, enforcing the 1000-item cap under concurrency.
+  const overflowFilter = { bookId: bookId, batch: targetBatch }
+  if (targetBatch !== nextBatch) {
+    // Only enforce the count cap when targeting an existing batch.
+    overflowFilter.count = { $lt: 1000 }
+  }
+
+  const overflowResult = await db.book_customers_extra.updateOne(
+    overflowFilter, // Write to the intended batch, respecting the count cap when reusing a batch
+    {
+      $push: { customers: customerId },
+      $inc: { count: 1 },
+      $setOnInsert: { bookId: bookId, batch: targetBatch }
+    },
+    { upsert: targetBatch === nextBatch }
+  )
+
+  // If we failed to match when trying to reuse the previous batch (it filled concurrently),
+  // fall back to writing into the next batch.
+  if (overflowResult.matchedCount === 0 && targetBatch !== nextBatch) {
+    await db.book_customers_extra.updateOne(
+      { bookId: bookId, batch: nextBatch },
+      {
+        $push: { customers: customerId },
+        $inc: { count: 1 },
+        $setOnInsert: { bookId: bookId, batch: nextBatch }
+      },
+      { upsert: true }
+    )
+  }
+
+  await db.books.updateOne(
+    { _id: bookId },
+    {
+      $set: { hasExtras: true },
+      $inc: { customerCount: 1 }
+    }
+  )
+}
+```
+
+**Index strategy:**
+
+```javascript
+// Index on main collection - only 50 entries per outlier doc
+db.books.createIndex({ "customers": 1 })
+
+// Index on overflow collection
+db.book_customers_extra.createIndex({ bookId: 1 })
+db.book_customers_extra.createIndex({ customers: 1 })
+```
+
+**When to use outlier pattern:**
+
+| Scenario | What to measure | Example |
+|----------|-----------------|---------|
+| Book customers | Array-size distribution and long tail | Bestsellers vs. typical books |
+| Social followers | Growth rate and read-path impact | Celebrities vs. regular users |
+| Product reviews | Index fan-out and read locality | Viral products vs. typical |
+| Event attendees | Outlier frequency vs. implementation complexity | Major events vs. small meetups |
+
+**When NOT to use this pattern:**
+
+- **Uniform distribution**: If all documents have similar array sizes, no outliers to isolate.
+- **Always need full data**: If you always display all 50,000 customers, pattern doesn't help.
+- **Write-heavy outliers**: Complex update logic may not be worth the read optimization.
+- **Small outliers**: If outliers are 200 vs typical 50, just use larger threshold.
+
+## Verify with
+
+```javascript
+// Find outlier documents
+db.books.aggregate([
+  { $project: {
+    title: 1,
+    customerCount: { $size: { $ifNull: ["$customers", []] } }
+  }},
+  { $sort: { customerCount: -1 } },
+  { $limit: 20 }
+])
+
+// Calculate distribution
+db.books.aggregate([
+  { $project: { count: { $size: { $ifNull: ["$customers", []] } } } },
+  { $bucket: {
+    groupBy: "$count",
+    boundaries: [0, 50, 100, 500, 1000, 10000, 100000],
+    default: "100000+",
+    output: { count: { $sum: 1 } }
+  }}
+])
+// Look for a long-tail distribution where a small subset is far above median/p95
+
+// Check index sizes
+db.books.stats().indexSizes
+// Large multikey index suggests outliers are bloating it
+```
+
+Reference: [Outlier Pattern](https://mongodb.com/docs/manual/data-modeling/design-patterns/group-data/outlier-pattern/)

package/dist/global-skills/mongodb-schema-design/references/pattern-polymorphic.md

@@ -0,0 +1,167 @@
+---
+title: Use Polymorphic Pattern for Heterogeneous Documents
+impact: MEDIUM
+impactDescription: "Keeps related entities in one collection while preserving type-specific fields"
+tags: schema, patterns, polymorphic, discriminator, flexible-schema, indexing, single-collection
+---
+
+## Use Polymorphic Pattern for Heterogeneous Documents
+
+**Store related but different document shapes in one collection with a type discriminator.** This keeps shared queries and indexes simple while allowing type-specific fields. Common use cases: product catalogs with different product types, content management systems, event stores, and any domain with inheritance.
+
+**Incorrect (separate collections per subtype):**
+
+Using a separate collection per product type (e.g. `products_books`, `products_electronics`, `products_clothing`) means querying across all products requires multiple calls or `$unionWith`, shared indexes must be duplicated, adding new types requires new collections, and application code must branch on collection names.
+
+**Correct (single collection using optional fields):**
+
+Store all product types in one `products` collection. All documents share common fields (`name`, `price`, `inStock`); each type adds its own specific fields (books: `author`, `isbn`, `pages`; electronics: `brand`, `wattage`, `batteryHours`, `warranty`; clothing: `size`, `color`, `material`). If the categories are always fully disjoint, use a `type` discriminator field (e.g. `"book"`, `"electronics"`, `"clothing"`). Cross-type queries use shared fields; type-specific queries filter by `type` plus type-specific fields. If there is potential overlap (e.g. between different categories of users), you can omit this field and rely entirely on optional fields.
+
+**Index strategies for polymorphic collections:**
+
+```javascript
+// Strategy 1: Compound index with type first
+// Best for: Queries that always filter by type
+db.products.createIndex({ type: 1, price: 1 })
+db.products.createIndex({ type: 1, name: 1 })
+
+// Query uses index efficiently:
+db.products.find({ type: "book", price: { $lt: 50 } })
+
+// Strategy 2: Compound index with type second
+// Best for: Queries that rarely filter by type
+db.products.createIndex({ price: 1, type: 1 })
+
+// Query across all types uses index:
+db.products.find({ price: { $lt: 50 } })
+
+// Strategy 3: Partial indexes for type-specific fields
+// Best for: Fields that only exist on some types
+db.products.createIndex(
+  { author: 1 },
+  { partialFilterExpression: { type: "book" } }
+)
+
+db.products.createIndex(
+  { brand: 1, wattage: 1 },
+  { partialFilterExpression: { type: "electronics" } }
+)
+
+// Strategy 4: Wildcard index for varying fields
+// Best for: Many type-specific fields, ad-hoc queries
+db.products.createIndex({ "specs.$**": 1 })
+
+// Documents store type-specific data in specs:
+{ type: "book", specs: { author: "...", isbn: "..." } }
+{ type: "electronics", specs: { brand: "...", wattage: 20 } }
+```
+
+**Query patterns across types:**
+
+```javascript
+// Pattern 1: Query all types with shared fields
+db.products.find({ price: { $lt: 100 }, inStock: true })
+  .sort({ price: 1 })
+
+// Pattern 2: Query specific type with type-specific fields
+db.products.find({
+  type: "book",
+  pages: { $gt: 300 },
+  author: /bradshaw/i
+})
+
+// Pattern 3: Aggregation across types with type-specific handling
+db.products.aggregate([
+  { $match: { inStock: true } },
+  { $group: {
+      _id: "$type",
+      count: { $sum: 1 },
+      avgPrice: { $avg: "$price" }
+    }
+  }
+])
+
+// Pattern 4: Faceted search with type breakdown
+db.products.aggregate([
+  { $match: { price: { $lt: 100 } } },
+  { $facet: {
+      byType: [{ $group: { _id: "$type", count: { $sum: 1 } } }],
+      priceRanges: [
+        { $bucket: {
+            groupBy: "$price",
+            boundaries: [0, 25, 50, 100],
+            default: "100+"
+          }
+        }
+      ]
+    }
+  }
+])
+```
+
+**Validation per type:**
+
+```javascript
+// Use JSON Schema with discriminator-based validation
+db.runCommand({
+  collMod: "products",
+  validator: {
+    $jsonSchema: {
+      bsonType: "object",
+      required: ["type", "name", "price"],
+      properties: {
+        type: { enum: ["book", "electronics", "clothing"] },
+        name: { bsonType: "string" },
+        price: { bsonType: "number", minimum: 0 }
+      },
+      oneOf: [
+        {
+          properties: { type: { enum: ["book"] } },
+          required: ["author", "isbn"]
+        },
+        {
+          properties: { type: { enum: ["electronics"] } },
+          required: ["brand"]
+        },
+        {
+          properties: { type: { enum: ["clothing"] } },
+          required: ["size", "color"]
+        }
+      ]
+    }
+  },
+  validationLevel: "moderate"
+})
+```
+
+**Adding new types:**
+
+The polymorphic pattern makes adding types straightforward — no schema migration needed. Insert documents with the new `type` value and any type-specific fields. Add partial indexes for type-specific queries as needed, and update schema validation to include the new type if using strict validation.
+
+**When NOT to use polymorphic pattern:**
+
+- **Completely different access patterns**: If each type is queried independently with no cross-type queries, separate collections may be cleaner.
+- **Conflicting index requirements**: If types need many different indexes, the index overhead may outweigh benefits.
+- **Strict type separation required**: Regulatory or security requirements may mandate separate collections.
+- **Vastly different document sizes**: If one type has 100-byte docs and another has 100KB docs, working set suffers.
+- **Type-specific sharding needs**: Different types may need different shard keys.
+
+## Verify with
+
+```javascript
+// Get type distribution
+db.products.aggregate([
+  { $group: {
+      _id: "$type",
+      count: { $sum: 1 },
+      avgSize: { $avg: { $bsonSize: "$$ROOT" } }
+    }
+  },
+  { $sort: { count: -1 } }
+])
+
+// Check for missing type field
+db.products.countDocuments({ type: { $exists: false } })
+```
+
+Reference: [Polymorphic Schema Pattern](https://mongodb.com/docs/manual/data-modeling/design-patterns/polymorphic-data/polymorphic-schema-pattern/)

package/dist/global-skills/mongodb-schema-design/references/pattern-schema-versioning.md

@@ -0,0 +1,273 @@
+---
+title: Schema Evolution and Preventing Drift
+impact: CRITICAL
+impactDescription: "Prevents application errors from inconsistent schemas and enables safe online migrations"
+tags: schema, patterns, versioning, migration, evolution, backward-compatibility, backfill, anti-pattern, validation, consistency, data-quality, atlas-suggestion
+---
+
+## Schema Evolution and Preventing Drift
+
+**Schema changes are inevitable, but uncontrolled changes cause schema drift** — documents in the same collection with inconsistent structures, leading to application errors and query failures. Use `schemaVersion` fields for safe migration and schema validation to prevent unexpected drift.
+
+### The problem: schema drift
+
+MongoDB's flexibility is a feature, but undisciplined field additions lead to code that must handle many document shapes.
+
+**Incorrect (uncontrolled drift over time):**
+
+```javascript
+// Over time, different versions of "user" documents accumulate
+{ _id: 1, name: "Alice", email: "alice@ex.com" }                  // 2021
+{ _id: 3, firstName: "Carol", lastName: "Smith", email: "carol@ex.com" }  // 2022 - restructured name
+{ _id: 4, firstName: "Dave", lastName: "Jones", emails: ["dave@ex.com"] } // 2023 - email → emails
+
+// Application code becomes defensive nightmare
+function getUserEmail(user) {
+  if (user.email) return user.email
+  if (user.emails) return user.emails[0]
+  throw new Error("No email found")
+}
+
+// Queries fail silently
+db.users.find({ email: "test@ex.com" })  // Misses users with emails[] array
+```
+
+### Solution: versioned documents with migration path
+
+Add a `schemaVersion` field to every document. Application code checks version and handles both formats. This allows old and new documents to coexist, new code to deploy before data migration, gradual migration during low-traffic periods, and easy rollback.
+
+**Correct (versioned with validation):**
+
+```javascript
+// Define and enforce consistent schema
+db.createCollection("users", {
+  validator: {
+    $jsonSchema: {
+      bsonType: "object",
+      required: ["emails", "profile", "schemaVersion"],
+      properties: {
+        emails: {
+          bsonType: "array",
+          items: {
+            bsonType: "string",
+            pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
+          }
+        },
+        profile: {
+          bsonType: "object",
+          required: ["firstName", "lastName"],
+          properties: {
+            firstName: { bsonType: "string", minLength: 1 },
+            lastName: { bsonType: "string", minLength: 1 }
+          }
+        },
+        schemaVersion: {
+          bsonType: "int",
+          enum: [1, 2]  // Accept both during migration
+        }
+      }
+    }
+  },
+  validationLevel: "strict",
+  validationAction: "error"
+})
+```
+
+### Online migration strategies
+
+```javascript
+// Strategy 1: Background batch migration
+// Best for: Large collections, can tolerate mixed versions temporarily
+
+function migrateToV2(batchSize = 1000) {
+  let migrated = 0
+  let cursor = db.users.find({ schemaVersion: { $lt: 2 } }).limit(batchSize)
+
+  for (const doc of cursor) {
+    const [firstName, ...rest] = (doc.name || "").split(" ")
+    const lastName = rest.join(" ") || "Unknown"
+
+    db.users.updateOne(
+      { _id: doc._id, schemaVersion: { $lt: 2 } },  // Prevent double-migration
+      {
+        $set: {
+          schemaVersion: 2,
+          profile: { firstName, lastName },
+          emails: doc.emails || (doc.email ? [doc.email] : []),
+        },
+        $unset: { name: "", email: "" }
+      }
+    )
+    migrated++
+  }
+  return migrated
+}
+
+// Run in batches during off-peak hours
+while (migrateToV2(1000) > 0) {
+  sleep(100)  // Throttle to reduce load
+}
+
+
+// Strategy 2: Aggregation pipeline update (MongoDB 4.2+)
+// Best for: Simple transformations, moderate collection sizes
+
+db.users.updateMany(
+  { schemaVersion: { $lt: 2 } },
+  [
+    {
+      $set: {
+        schemaVersion: 2,
+        profile: {
+          $cond: {
+            if: { $eq: [{ $type: "$name" }, "string"] },
+            then: {
+              firstName: { $arrayElemAt: [{ $split: ["$name", " "] }, 0] },
+              lastName: { $ifNull: [
+                { $arrayElemAt: [{ $split: ["$name", " "] }, 1] },
+                "Unknown"
+              ]}
+            },
+            else: "$profile"
+          }
+        },
+        emails: {
+          $cond: {
+            if: { $eq: [{ $type: "$email" }, "string"] },
+            then: ["$email"],
+            else: { $ifNull: ["$emails", []] }
+          }
+        },
+      }
+    },
+    { $unset: ["name", "email"] }
+  ]
+)
+
+
+// Strategy 3: Read-time migration (lazy migration)
+// Best for: Low-traffic documents, immediate consistency needed
+
+function getUser(userId) {
+  const user = db.users.findOne({ _id: userId })
+
+  if (user && user.schemaVersion < 2) {
+    const migrated = migrateUserToV2(user)
+    db.users.replaceOne({ _id: userId }, migrated)
+    return migrated
+  }
+
+  return user
+}
+```
+
+### Handling multiple version jumps
+
+```javascript
+// v1 → v2 → v3: define transformation functions for each step
+const migrations = {
+  1: (doc) => ({
+    ...doc,
+    schemaVersion: 2,
+    profile: {
+      firstName: doc.name.split(" ")[0],
+      lastName: doc.name.split(" ").slice(1).join(" ") || "Unknown"
+    },
+    emails: doc.email ? [doc.email] : []
+  }),
+  2: (doc) => ({
+    ...doc,
+    schemaVersion: 3,
+    profile: {
+      ...doc.profile,
+      displayName: `${doc.profile.firstName} ${doc.profile.lastName}`
+    }
+  })
+}
+
+function migrateToLatest(doc, targetVersion = 3) {
+  let current = doc
+  while (current.schemaVersion < targetVersion) {
+    const migrator = migrations[current.schemaVersion]
+    if (!migrator) throw new Error(`No migration from v${current.schemaVersion}`)
+    current = migrator(current)
+  }
+  return current
+}
+```
+
+### When a version bump is (and isn't) needed
+
+**No version bump needed (backward-compatible):**
+- Adding new optional fields (old code ignores them)
+- Adding new indexes (transparent to application)
+- Relaxing validation (making a required field optional)
+
+**Version bump required (breaking):**
+- Renaming fields (`address` → `shippingAddress`)
+- Changing field types (`price: "19.99"` → `price: 19.99`)
+- Restructuring (flat `firstName`/`lastName` → nested `name: { first, last }`)
+- Removing fields that old code reads
+
+### Detecting existing schema drift
+
+```javascript
+// Find all unique field combinations
+db.users.aggregate([
+  { $project: { fields: { $objectToArray: "$$ROOT" } } },
+  { $project: { keys: "$fields.k" } },
+  { $group: { _id: "$keys", count: { $sum: 1 } } },
+  { $sort: { count: -1 } }
+])
+// Multiple distinct key-sets = schema drift exists
+
+// Find documents missing required fields
+db.users.find({
+  $or: [
+    { emails: { $exists: false } },
+    { profile: { $exists: false } },
+    { "profile.firstName": { $exists: false } }
+  ]
+})
+
+// Find documents with wrong field types
+db.users.find({
+  emails: { $not: { $type: "array" } }
+})
+```
+
+### When NOT to strictly enforce schema or use versioning
+
+- **Truly polymorphic data**: Event logs with different event types may need flexible schemas — use `pattern-polymorphic` instead.
+- **Early prototyping**: Skip validation during exploration, add before production.
+- **User-defined fields**: Some applications allow custom metadata fields.
+- **Small datasets with downtime window**: If you can migrate all data in minutes during maintenance.
+- **Additive-only changes**: If you only add optional fields, versioning is overkill.
+
+## Verify with
+
+```javascript
+// Track version distribution
+db.users.aggregate([
+  { $group: { _id: "$schemaVersion", count: { $sum: 1 } } },
+  { $sort: { _id: 1 } }
+])
+
+// Check for missing version field (implicit v1 documents)
+db.users.countDocuments({ schemaVersion: { $exists: false } })
+
+// Check if validation exists on the collection
+const collInfo = db.getCollectionInfos({ name: "users" })[0]
+const validator = collInfo?.options?.validator
+// Missing validator = higher schema drift risk
+
+// Find documents that don't match current validator
+if (validator) {
+  db.users.find({ $nor: [validator] }).limit(20)
+  db.users.countDocuments({ $nor: [validator] })
+}
+```
+
+References:
+- [Schema Versioning Pattern](https://mongodb.com/docs/manual/data-modeling/design-patterns/data-versioning/schema-versioning/)
+- [Schema Validation](https://mongodb.com/docs/manual/core/schema-validation/)