vybekiit 0.4.0 → 0.6.0

package/dist/global-skills/mongodb-schema-design/references/pattern-time-series-collections.md

@@ -0,0 +1,190 @@
+---
+title: Use Time Series Collections for Time Series Data
+impact: MEDIUM
+impactDescription: "10-100× lower storage and index overhead with automatic bucketing and compression"
+tags: schema, patterns, time-series, collections, bucketing, ttl, granularity, compression
+---
+
+## Use Time Series Collections for Time Series Data
+
+**Time series collections are purpose-built for append-only measurements.** MongoDB automatically buckets, compresses, and indexes time series data so you get high ingest rates with far less storage and index overhead than a standard collection. Use them for IoT sensor data, application metrics, financial data, and event logs.
+
+**MongoDB 8.0 Performance:** Block processing introduced in MongoDB 8.0 can significantly improve eligible analytical pipelines (for example, `$match` + `$sort` on the time field + `$group`). In some cases, throughput improves by more than 200%. This is automatic for eligible queries.
+
+**Incorrect (regular collection for measurements):**
+
+```javascript
+// Regular collection: one document per reading
+// Creates huge collections and indexes at scale
+{
+  sensorId: "temp-01",
+  ts: ISODate("2025-01-15T10:00:00Z"),
+  value: 22.5
+}
+
+// Problems:
+// 1. Each measurement is a separate document
+// 2. Index overhead per document
+// 3. No automatic compression
+// 4. Working set grows linearly
+
+// Standard index (large and grows fast)
+db.sensor_data.createIndex({ sensorId: 1, ts: 1 })
+```
+
+**Correct (time series collection with optimized settings):**
+
+```javascript
+// Create time series collection with careful configuration
+db.createCollection("sensor_data", {
+  timeseries: {
+    timeField: "ts",           // Required: timestamp field
+    metaField: "metadata",     // Recommended: grouping field
+    granularity: "minutes"     // Match your data rate
+  },
+  expireAfterSeconds: 60 * 60 * 24 * 90  // 90-day retention
+})
+
+// Insert documents - MongoDB buckets automatically
+db.sensor_data.insertOne({
+  metadata: { sensorId: "temp-01", location: "building-A" },
+  ts: new Date(),
+  value: 22.5,
+  unit: "celsius"
+})
+
+// Benefits:
+// - Automatic bucketing (many measurements per internal doc)
+// - Column compression (40-60% disk reduction)
+// - MongoDB 6.3+: auto-created compound index on metaField + timeField for new collections
+// - Optimized for time-range queries
+```
+
+**Choose the right metaField:**
+
+```javascript
+// metaField groups measurements into buckets
+// Choose fields that:
+// 1. Are queried together with time ranges
+// 2. Have moderate cardinality (not too unique, not too few)
+// 3. Don't change for a given time series
+
+// GOOD: Sensor/device identifier as metaField
+{
+  metadata: { sensorId: "temp-01", region: "us-east" },
+  ts: new Date(),
+  value: 22.5
+}
+// Queries like: "All readings from temp-01 in last hour"
+
+// BAD: High-cardinality field as metaField
+{
+  metadata: { requestId: "uuid-123..." },  // Unique per doc!
+  ts: new Date()
+}
+// Creates one bucket per requestId - no compression benefit
+
+// BAD: Frequently changing field in metaField
+{
+  metadata: { sensorId: "temp-01", currentValue: 22.5 },  // Changes!
+  ts: new Date()
+}
+// metaField should be static for the time series
+```
+
+**Select appropriate granularity:**
+
+```javascript
+// Granularity determines bucket time span
+// Match it to your data ingestion rate
+
+// "seconds" - DEFAULT. High-frequency ingestion. Bucket spans ~1 hour.
+db.createCollection("high_freq_metrics", {
+  timeseries: { timeField: "ts", metaField: "host", granularity: "seconds" }
+})
+
+// "minutes" - Data every few seconds to minutes. Bucket spans ~24 hours.
+db.createCollection("app_metrics", {
+  timeseries: { timeField: "ts", metaField: "service", granularity: "minutes" }
+})
+
+// "hours"   - Data every few hours. Bucket spans ~30 days.
+db.createCollection("daily_reports", {
+  timeseries: { timeField: "ts", metaField: "reportType", granularity: "hours" }
+})
+
+// Custom bucketing (MongoDB 6.3+) for precise control
+db.createCollection("custom_metrics", {
+  timeseries: {
+    timeField: "ts",
+    metaField: "device",
+    bucketMaxSpanSeconds: 3600,      // Max 1 hour per bucket
+    bucketRoundingSeconds: 3600      // Align to hour boundaries
+  }
+})
+```
+
+**Optimize insert performance:**
+
+```javascript
+// Batch inserts with insertMany
+// Group documents with same metaField value together
+const batch = [
+  { metadata: { sensorId: "temp-01" }, ts: new Date(), value: 22.5 },
+  { metadata: { sensorId: "temp-01" }, ts: new Date(), value: 22.6 },
+  { metadata: { sensorId: "temp-02" }, ts: new Date(), value: 19.2 },
+]
+
+db.sensor_data.insertMany(batch, { ordered: false })
+// ordered: false allows parallel processing
+// Use consistent field order and omit empty values for better compression
+```
+
+**Secondary indexes on time series:**
+
+```javascript
+// MongoDB 6.3+: time series auto-creates index on { metaField, timeField } for new collections
+// Add secondary indexes for other query patterns
+
+// Index on measurement values for threshold queries
+db.sensor_data.createIndex({ "value": 1 })
+// Query: "All readings where value > 100"
+
+// Compound index for filtered time queries
+db.sensor_data.createIndex({ "metadata.location": 1, "ts": 1 })
+// Query: "Readings from building-A in last hour"
+
+// Partial index for specific conditions
+db.sensor_data.createIndex(
+  { "metadata.alertLevel": 1 },
+  { partialFilterExpression: { "metadata.alertLevel": { $exists: true } } }
+)
+```
+
+**When NOT to use time series collections:**
+
+- **Not time-based data**: Primary access isn't time range queries.
+- **Frequent updates/deletes**: Time series optimized for append-only; updates to old data are slow.
+- **Very low volume**: A few hundred events don't benefit from bucketing.
+- **Need transactional writes**: Time series collections don't support writes in transactions (reads are supported).
+- **Complex queries on measurements**: If you mostly query by non-time fields, regular collections may be better.
+
+## Verify with
+
+```javascript
+// Get collection info
+const info = db.getCollectionInfos({ name: "sensor_data" })[0]
+const ts = info?.options?.timeseries
+// Check timeField, metaField, granularity, expireAfterSeconds
+
+// Check bucket efficiency (via system.buckets)
+const bucketColl = `system.buckets.sensor_data`
+const bucketCount = db.getCollection(bucketColl).countDocuments({})
+const stats = db.sensor_data.stats()
+if (bucketCount > 0 && stats.count) {
+  const docsPerBucket = stats.count / bucketCount
+  // Low docs/bucket suggests adjusting granularity or metaField
+}
+```
+
+Reference: [Time Series Collections](https://mongodb.com/docs/manual/core/timeseries-collections/)

package/dist/global-skills/mongodb-schema-design/references/source-query-stats.md

@@ -0,0 +1,123 @@
+# Query Stats
+
+## When to use
+
+Analyzes query access patterns with minimal performance overhead. Use for identifying co-accessed fields, collection relationships, and query frequencies. Only supports `find`, `aggregate`, and `distinct` operations.
+
+## Requirements
+
+Atlas M10+ tier.
+
+## How to use
+
+Aggregate on the admin database.
+
+With mcp-server, use the `mcp__mongodb__aggregateDB` tool with database set to `admin`.
+
+```javascript
+db.getSiblingDB("admin").aggregate([{ $queryStats: {} }])
+```
+
+**Example 1: Find collections frequently queried together with others (embedding candidates)**
+```javascript
+db.aggregate([
+  { $queryStats: {} },
+  {
+    $match: {
+      "key.queryShape.cmdNs.db": "databaseName",
+      "key.queryShape.command": "aggregate",
+      "key.queryShape.pipeline.$lookup": { $exists: true }
+    }
+  },
+  { $unwind: "$key.queryShape.pipeline" },
+  { 
+    $match: { "key.queryShape.pipeline.$lookup": { $exists: true } } 
+  },
+  { 
+    $set: { 
+      stageKeyValue: { 
+        $first: { $objectToArray: "$key.queryShape.pipeline" } 
+      } 
+    }
+  },
+  { 
+    $group: {
+      _id: { 
+        source: "$key.queryShape.cmdNs.coll",
+        target: "$stageKeyValue.v.from"
+      },
+      totalLookupHits: { $sum: "$metrics.execCount" },
+      avgPipelineMs: {
+        $avg: { $divide: [
+          { $divide: ["$metrics.totalExecMicros.sum", 1000] },
+          "$metrics.execCount"
+        ]}
+      }
+    }
+  },
+  { $sort: { totalLookupHits: -1 } }
+])
+
+// High totalLookupHits = frequently joined 
+// High avgPipelineMS = lookup is part of slow queries (does not automatically mean that the $lookup is slow, could be the whole pipeline - see the full query shapes)
+// High scores on both - consider embedding to avoid $lookup 
+```
+
+**Example 1.1: Find query shapes that use $lookup on specific collections**
+```javascript
+db.aggregate([
+  { $queryStats: {} },
+  {
+    $match: {
+      "key.queryShape.cmdNs.db": "databaseName",
+      "key.queryShape.command": "aggregate",
+      "key.queryShape.cmdNs.coll": "sourceCollectionName",
+      "key.queryShape.pipeline.$lookup.from": "targetCollectionName"
+    }
+  },
+  {
+    $project: {
+      database: "$key.queryShape.cmdNs.db",
+      collection: "$key.queryShape.cmdNs.coll",
+      pipeline: "$key.queryShape.pipeline",
+      execCount: "$metrics.execCount",
+      avgMs: {
+          $divide: [
+            { $divide: ["$metrics.totalExecMicros.sum", 1000] },
+            "$metrics.execCount"
+          ]
+      }
+    },
+  },
+  { $sort: { execCount: -1 } },
+  { $limit: 10 }
+])
+```
+
+**Example 2: Find top most frequent query shapes (optimize hot paths)**
+```javascript
+db.getSiblingDB("admin").aggregate([
+  { $queryStats: {} },
+  { $sort: { "metrics.execCount": -1 } },
+  { $limit: 10 },
+  {
+    $project: {
+      command: "$key.queryShape.command",
+      database: "$key.queryShape.cmdNs.db",
+      collection: "$key.queryShape.cmdNs.coll",
+      queryShape: "$key.queryShape",
+      execCount: "$metrics.execCount",
+      avgMs: {
+        $divide: [
+          { $divide: ["$metrics.totalExecMicros.sum", 1000] },
+          "$metrics.execCount"
+        ]
+      }
+    }
+  }
+])
+
+// High execCount = hot path → design your schema for these queries first
+// Cross reference with avgMS or [slow query logs](references/source-slow-query-logs.md) to find queries that are both frequent and slow
+// Note: Query stats do not include write patterns (update, insert)
+```

package/dist/global-skills/mongodb-schema-design/references/source-slow-query-logs.md

@@ -0,0 +1,63 @@
+# Atlas Slow Query Logs
+
+## When to use
+
+Retrieves log lines for slow queries as determined by the Performance Advisor. Use to identify slow queries and performance bottlenecks. Provides actual query examples (not shapes) with execution times. Captures all operation types including writes, unlike Query Stats which currently only covers find/aggregate/distinct.
+
+## Requirements
+
+- Atlas M10+ cluster
+- Atlas API credentials configured
+- Performance Advisor enabled (enabled by default on M10+)
+
+If the API call returns auth or access errors, see the [Performance Advisor docs](https://www.mongodb.com/docs/atlas/performance-advisor/).
+
+## How to use
+
+Atlas Admin API endpoint ([query parameters reference](https://www.mongodb.com/docs/ops-manager/current/reference/api/performance-advisor/get-slow-queries/#request-query-parameters)):
+```
+GET /groups/{PROJECT-ID}/hosts/{HOST-ID}/performanceAdvisor/slowQueryLogs
+```
+
+With MongoDB MCP server:
+```javascript
+mcp__plugin_mongodb_mongodb__atlas-get-performance-advisor({
+  projectId: "507f1f77bcf86cd799439011",
+  clusterName: "MyCluster",
+  operations: ["slowQueryLogs"]
+})
+```
+
+Performance Advisor analyzes up to 200,000 of the cluster's most recent log lines.
+
+**Example response structure:**
+```javascript
+{
+  "slowQueries": [
+    {
+      "line": "2026-05-06T10:23:45.447+0000 I COMMAND [conn10614] command mydb.orders appName: \"MongoDB Shell\" command: find { find: \"orders\", filter: { status: \"pending\", customerId: 12345 }, sort: { createdAt: -1 } } planSummary: COLLSCAN keysExamined:0 docsExamined:50000 nreturned:100 executionTimeMillis:1247 ...",
+      "namespace": "mydb.orders"
+    }
+  ]
+}
+```
+
+The response contains raw log lines. Parse the log line to extract:
+- Timestamp (beginning of line)
+- Operation type (command: find, aggregate, update, etc.)
+- Query details (filter, pipeline, etc.)
+- Execution metrics (executionTimeMillis, docsExamined, planSummary, etc.)
+
+## What to Look For
+
+When analyzing slow query logs, focus on:
+
+**Slow $lookup operations:**
+- Look for `$lookup` in the log line
+- Consider embedding to reduce slow $lookup operations
+- Cross-reference with Query Stats to identify frequent lookups
+- High executionTimeMillis + high frequency = urgent schema redesign
+
+**Other slow aggregations:**
+- Consider the Computed Pattern to avoid slow aggregations
+

package/dist/global-skills/mongodb-search-and-ai/SKILL.md

@@ -0,0 +1,142 @@
+---
+name: mongodb-search-and-ai
+description: |
+  Guides MongoDB users through implementing and optimizing Atlas Search (full-text), Vector Search (semantic), and Hybrid Search solutions. Use this skill when users need to build search functionality for text-based queries (autocomplete, fuzzy matching, faceted search), semantic similarity (embeddings, RAG applications), or combined approaches. Also use when users need text containment, substring matching ('contains', 'includes', 'appears in'), case-insensitive or multi-field text search, or filtering across many fields with variable combinations. Provides workflows for selecting the right search type, creating indexes, constructing queries, and optimizing performance using the MongoDB MCP server.
+license: Apache-2.0
+metadata:
+  version: "1.0.0"
+---
+
+# MongoDB Search and AI Recommendations Skill
+
+You are helping MongoDB users implement, optimize, and troubleshoot Atlas Search (lexical), Vector Search (semantic), and Hybrid Search (combined) solutions. Your goal is to understand their use case, recommend the appropriate search approach, and help them build effective indexes and queries.
+
+## Core Principles
+
+1. **Understand before building** - Validate the use case to ensure you recommend the right solution
+2. **Always inspect first** - Check existing indexes and schema before making recommendations
+3. **Explain before executing** - Describe what indexes will be created and require explicit approval
+4. **Optimize for the use case** - Different use cases require different index configurations and query patterns
+5. **Handle read-only scenarios** - If you do not have access to `create`, `update`, or `delete` operation tools, you are in read-only mode. Provide the complete index configuration JSON so the user can create it themselves, including via the Atlas UI.
+
+## Workflow
+
+### 1. Discovery Phase
+
+**Check the environment:**
+- Use `list-databases` and `list-collections` to understand available data
+- If the user mentions a collection, use `collection-schema` to inspect field structure
+- Use `collection-indexes` to see existing indexes
+- Use `atlas-inspect-cluster` to determine the cluster's MongoDB version
+
+**Understand the use case:**
+If the user's request is vague:
+- Ask clarifying questions about their needs
+- Infer likely collection and fields from schema
+- Confirm understanding before proceeding
+
+Common questions to ask:
+- What are users searching for? (products, movies, documents, etc.)
+- What fields contain the searchable content?
+- Do they need exact matching, fuzzy matching, or semantic similarity?
+- Do they need filters (price ranges, categories, dates)?
+- Do they need autocomplete/typeahead functionality?
+
+### 2. Determine Search Type
+
+**Atlas Search (Lexical/Full-Text):**
+Use when users need:
+- Keyword matching with relevance scoring
+- Fuzzy matching for typo tolerance
+- Autocomplete/typeahead
+- Faceted search with filters
+- Language-specific text analysis
+- Token-based search
+- Lexical search with views
+
+**Vector Search (Semantic):**
+Use when users need:
+- Semantic similarity ("find movies about coming of age stories")
+- Natural language understanding
+- RAG (Retrieval Augmented Generation) applications
+- Finding conceptually similar items
+- Cross-modal search
+- Vector search with views
+
+**Hybrid Search:**
+Use when users need:
+- Combining multiple search approaches (e.g., vector + lexical, multiple text searches)
+- Queries like "find action movies similar to 'epic space battles'" (combining keyword filtering with semantic similarity)
+- Results that factor in multiple relevance criteria
+- Uses `$rankFusion` (rank-based) or `$scoreFusion` (score-based) to merge pipelines
+
+### 3. Version Check (Hybrid Search only)
+
+If the search type is **Hybrid using `$rankFusion` or `$scoreFusion`**, verify the cluster version before proceeding:
+- `$rankFusion` requires MongoDB 8.0+
+- `$scoreFusion` requires MongoDB 8.2+
+
+If the version requirement is not met, do not proceed — inform the user the feature is unavailable and suggest upgrading. Do not consult `references/hybrid-search.md`.
+
+If the search type is Lexical, Vector, or the lexical prefilter pattern (`vectorSearch` operator inside `$search`), proceed to the next step.
+
+### 4. Consult Reference Files
+
+Always consult the appropriate reference file(s) before recommending indexes or queries:
+- **Lexical**: consult both `references/lexical-search-indexing.md` (index) and `references/lexical-search-querying.md` (query)
+- **Vector**: consult `references/vector-search.md`
+- **Hybrid**: consult `references/hybrid-search.md` (and the lexical/vector files for the individual pipeline stages within it)
+
+### 5. Execution and Validation
+
+**Creating indexes:**
+1. Explain the index configuration in plain language
+2. Show the JSON structure
+3. Ask what the user wants to name the index
+4. Get explicit approval: "Should I create this index?"
+5. Use MCP's `create-index` tool after approval
+6. In read-only mode, provide the complete index JSON for creation via the Atlas UI
+
+**Running queries:**
+1. Show the aggregation pipeline
+2. Execute using MCP's `aggregate` tool
+3. Present results clearly
+
+**Refining existing queries:**
+1. Ask the user to share their current query
+2. Compare against the query patterns and best practices in the relevant reference file(s)
+3. Propose specific improvements with before/after examples
+4. Run the revised query with `aggregate` to validate the results
+
+## Anti-Patterns to Avoid
+
+**NEVER recommend $regex or $text for search use cases:**
+- **$regex**: Not designed for full-text search. Lacks relevance scoring, fuzzy matching, and language-aware tokenization.
+- **$text**: Legacy operator that doesn't scale well for search workloads.
+
+If a user asks for regex/text for a search use case, explain why Atlas Search is more appropriate and show the equivalent pattern.
+
+## Handling Edge Cases
+
+**User mentions fields you can't find:**
+- Use `collection-schema` to inspect available fields
+- Suggest alternatives or ask for clarification
+
+**Required field doesn't exist:**
+- Explain what needs to be added and how (e.g., embedding field for vector search)
+
+**Query fails or index missing:**
+- Use `collection-indexes` to verify index exists
+- If missing, explain index needs to be created first
+
+**Multiple collections are relevant:**
+- List options and ask which one they mean
+- If context makes it obvious, confirm your assumption
+
+## Remember
+
+- Always check existing indexes before recommending new ones
+- Explain technical concepts in accessible language
+- Require approval before creating indexes
+- Map user's business requirements to technical implementations
+- Use the appropriate search type for the use case