vybekiit 0.4.0 → 0.6.0

package/dist/global-skills/mongodb-atlas-stream-processing/SKILL.md

@@ -0,0 +1,284 @@
+---
+name: mongodb-atlas-stream-processing
+description: "Manages MongoDB Atlas Stream Processing (ASP) workflows. Handles workspace provisioning, data source/sink connections, processor lifecycle operations, debugging diagnostics, and tier sizing. Supports Kafka, Atlas clusters, S3, HTTPS, and Lambda integrations for streaming data workloads and event processing. NOT for general MongoDB queries or Atlas cluster management. Requires MongoDB MCP Server with Atlas API credentials."
+license: Apache-2.0
+metadata:
+  version: "1.0.0"
+  user-invocable: "true"
+---
+
+# MongoDB Atlas Streams
+
+Build, operate, and debug Atlas Stream Processing (ASP) pipelines using four MCP tools from the MongoDB MCP Server.
+
+## Prerequisites
+
+This skill requires the **MongoDB MCP Server** connected with:
+- Atlas API credentials (`apiClientId` and `apiClientSecret`)
+
+The 4 tools: `atlas-streams-discover`, `atlas-streams-build`, `atlas-streams-manage`, `atlas-streams-teardown`.
+
+**All operations require an Atlas project ID.** If unknown, call `atlas-list-projects` first to find your project ID.
+
+## If MCP tools are unavailable
+
+If the MongoDB MCP Server is not connected or the streams tools are missing, see [references/mcp-troubleshooting.md](references/mcp-troubleshooting.md) for diagnostic steps and fallback options.
+
+## Tool Selection Matrix
+
+### atlas-streams-discover — ALL read operations
+| Action | Use when |
+|--------|----------|
+| `list-workspaces` | See all workspaces in a project |
+| `inspect-workspace` | Review workspace config, state, region |
+| `list-connections` | See all connections in a workspace |
+| `inspect-connection` | Check connection state, config, health |
+| `list-processors` | See all processors in a workspace |
+| `inspect-processor` | Check processor state, pipeline, config |
+| `diagnose-processor` | Full health report: state, stats, errors |
+| `get-networking` | PrivateLink and VPC peering details. Optional: `cloudProvider` + `region` to get Atlas account details for PrivateLink setup |
+
+**Pagination** (all list actions): `limit` (1-100, default 20), `pageNum` (default 1).
+**Response format**: `responseFormat` — `"concise"` (default for list actions) or `"detailed"` (default for inspect/diagnose).
+
+### atlas-streams-build — ALL create operations
+| Resource | Key parameters |
+|----------|---------------|
+| `workspace` | `cloudProvider`, `region`, `tier` (default SP10), `includeSampleData` |
+| `connection` | `connectionName`, `connectionType` (Kafka/Cluster/S3/Https/Kinesis/Lambda/SchemaRegistry/Sample), `connectionConfig` |
+| `processor` | `processorName`, `pipeline` (must start with `$source`, end with `$merge`/`$emit`), `dlq`, `autoStart` |
+| `privatelink` | `privateLinkConfig` (project-level, not tied to a specific workspace) |
+
+**Field mapping — only fill fields for the selected resource type:**
+
+- **resource = "workspace":** Fill: `projectId`, `workspaceName`, `cloudProvider`, `region`, `tier`, `includeSampleData`. Leave empty: all connection and processor fields.
+- **resource = "connection":** Fill: `projectId`, `workspaceName`, `connectionName`, `connectionType`, `connectionConfig`. Leave empty: all workspace and processor fields. (See [references/connection-configs.md](references/connection-configs.md) for type-specific schemas.)
+- **resource = "processor":** Fill: `projectId`, `workspaceName`, `processorName`, `pipeline`, `dlq` (recommended), `autoStart` (optional). Leave empty: all workspace and connection fields. (See [references/pipeline-patterns.md](references/pipeline-patterns.md) for pipeline examples.)
+- **resource = "privatelink":** Fill: `projectId`, `privateLinkConfig`. Note: PrivateLink is **project-level**, not workspace-level. `workspaceName` is not required — omit it. Leave empty: all connection and processor fields.
+
+### atlas-streams-manage — ALL update/state operations
+| Action | Notes |
+|--------|-------|
+| `start-processor` | Begins billing. Optional `tier` override, `resumeFromCheckpoint` |
+| `stop-processor` | Stops billing. Retains state 45 days |
+| `modify-processor` | Processor must be stopped first. Change pipeline, DLQ, or name |
+| `update-workspace` | Change tier or region |
+| `update-connection` | Update config (networking is immutable — must delete and recreate) |
+| `accept-peering` / `reject-peering` | VPC peering management |
+
+**Field mapping** — always fill `projectId`, `workspaceName`, then by action:
+
+- `"start-processor"` → `resourceName`. Optional: `tier`, `resumeFromCheckpoint`, `startAtOperationTime` (ISO 8601 timestamp to resume from a specific point)
+- `"stop-processor"` → `resourceName`
+- `"modify-processor"` → `resourceName`. At least one of: `pipeline`, `dlq`, `newName`
+- `"update-workspace"` → `newRegion` or `newTier`
+- `"update-connection"` → `resourceName`, `connectionConfig`. **Exception: networking config (e.g., PrivateLink) cannot be modified after creation** — delete and recreate.
+- `"accept-peering"` → `peeringId`, `requesterAccountId`, `requesterVpcId`
+- `"reject-peering"` → `peeringId`
+
+**State pre-checks:**
+- `start-processor` → errors if processor is already STARTED
+- `stop-processor` → no-ops if already STOPPED or CREATED (not an error)
+- `modify-processor` → errors if processor is STARTED (must stop first)
+
+**Processor states:** `CREATED` → `STARTED` (via start) → `STOPPED` (via stop). Can also enter `FAILED` on runtime errors. Modify requires STOPPED or CREATED state.
+
+**Teardown safety checks:**
+- **Processor deletion** → auto-stops before deleting (no need to stop manually first)
+- **Connection deletion** → blocks if any running processor references it. Stop/delete referencing processors first.
+- **Workspace deletion** → See detailed workflow below (lines 108-111).
+
+### atlas-streams-teardown — ALL delete operations
+| Resource | Safety behavior |
+|----------|----------------|
+| `processor` | Auto-stops before deleting |
+| `connection` | Blocks if referenced by running processor |
+| `workspace` | Cascading delete of all connections and processors |
+| `privatelink` / `peering` | Remove networking resources |
+
+**Field mapping** — always fill `projectId`, `resource`, then:
+
+- `resource: "workspace"` → `workspaceName`
+- `resource: "connection"` or `"processor"` → `workspaceName`, `resourceName`
+- `resource: "privatelink"` or `"peering"` → `resourceName` (the ID). These are project-level resources, not tied to a specific workspace.
+
+**Before deleting a workspace**, inspect it first:
+1. `atlas-streams-discover` → `inspect-workspace` — get connection/processor counts
+2. Present to user: "Workspace X contains N connections and M processors. Deleting permanently removes all. Proceed?"
+3. Wait for confirmation before calling `atlas-streams-teardown`
+
+## CRITICAL: Validate Before Creating Processors
+
+**You MUST call `search-knowledge` before composing any processor pipeline.** This is not optional.
+- **Field validation:** Query with the sink/source type, e.g. "Atlas Stream Processing $emit S3 fields" or "Atlas Stream Processing Kafka $source configuration". This catches errors like `prefix` vs `path` for S3 `$emit`.
+- **Pattern examples:** Query with `dataSources: [{"name": "devcenter"}]` for working pipelines, e.g. "Atlas Stream Processing tumbling window example".
+
+Also fetch examples from the official ASP examples repo when building non-trivial processors: **https://github.com/mongodb/ASP_example** (quickstarts, example processors, Terraform examples). Start with `example_processors/README.md` for the full pattern catalog.
+
+Key quickstarts:
+| Quickstart | Pattern |
+|-----------|---------|
+| `00_hello_world.json` | Inline `$source.documents` with `$match` (zero infra, ephemeral) |
+| `01_changestream_basic.json` | Change stream → tumbling window → `$merge` to Atlas |
+| `03_kafka_to_mongo.json` | Kafka source → tumbling window rollup → `$merge` to Atlas |
+| `04_mongo_to_mongo.json` | Chained processors: rollup → archive to separate collection |
+| `05_kafka_tail.json` | Real-time Kafka topic monitoring (sinkless, like `tail -f`) |
+
+## Pipeline Rules & Warnings
+
+**Invalid constructs** — these are NOT valid in streaming pipelines:
+- **`$$NOW`**, **`$$ROOT`**, **`$$CURRENT`** — NOT available in stream processing. NEVER use these. Use the document's own timestamp field or `_stream_meta` metadata for event time instead of `$$NOW`.
+- **HTTPS connections as `$source`** — HTTPS is for `$https` enrichment or sink only, NOT as a data source
+- **Kafka `$source` without `topic`** — topic field is required
+- **Pipelines without a sink** — terminal stage (`$merge`, `$emit`, `$https`, or `$externalFunction` async) required for deployed processors (sinkless only works via `sp.process()`)
+- **Lambda as `$emit` target** — Lambda uses `$externalFunction` (mid-pipeline enrichment), not `$emit`
+- **`$validate` with `validationAction: "error"`** — crashes processor; use `"dlq"` instead
+
+**Required fields by stage:**
+- **`$source` (change stream)**: include `fullDocument: "updateLookup"` to get the full document content
+- **`$source` (Kinesis)**: use `stream` (NOT `streamName` or `topic`)
+- **`$emit` (Kinesis)**: MUST include `partitionKey`
+- **`$emit` (S3)**: use `path` (NOT `prefix`)
+- **`$https`**: must include `connectionName`, `path`, `method`, `as`, `onError: "dlq"`
+- **`$externalFunction`**: must include `connectionName`, `functionName`, `execution`, `as`, `onError: "dlq"`
+- **`$validate`**: must include `validator` with `$jsonSchema` and `validationAction: "dlq"`
+- **`$lookup`**: include `parallelism` setting (e.g., `parallelism: 2`) for concurrent I/O
+- **AWS connections** (S3, Kinesis, Lambda): IAM role ARN must be registered via Atlas Cloud Provider Access first. Always confirm this with user. See [references/connection-configs.md](references/connection-configs.md) for details.
+
+See [references/pipeline-patterns.md](references/pipeline-patterns.md) for stage field examples with JSON syntax.
+
+**SchemaRegistry connection:** `connectionType` must be `"SchemaRegistry"` (not `"Kafka"`). Schema type values are case-sensitive (use lowercase `avro`, not `AVRO`). See [references/connection-configs.md](references/connection-configs.md#schemaregistry) for required fields and auth types.
+
+## MCP Tool Behaviors
+
+**Elicitation:** When creating connections, the build tool auto-collects missing sensitive fields (passwords, bootstrap servers) via MCP elicitation. Do NOT ask the user for these — let the tool collect them.
+
+**Auto-normalization:**
+- `bootstrapServers` array → auto-converted to comma-separated string
+- `schemaRegistryUrls` string → auto-wrapped in array
+- `dbRoleToExecute` → defaults to `{role: "readWriteAnyDatabase", type: "BUILT_IN"}` for Cluster connections
+
+**Workspace creation:** `includeSampleData` defaults to `true`, which auto-creates the `sample_stream_solar` connection.
+
+**Region naming:** The `region` field uses Atlas-specific names that differ by cloud provider. Using the wrong format returns a cryptic `dataProcessRegion` error.
+
+| Provider | Cloud Region | Streams `region` Value |
+|----------|-------------|----------------------|
+| **AWS** | us-east-1 | `VIRGINIA_USA` |
+| **AWS** | us-east-2 | `OHIO_USA` |
+| **AWS** | eu-west-1 | `DUBLIN_IRL` |
+| **GCP** | us-central1 | `US_CENTRAL1` |
+| **GCP** | europe-west1 | `EUROPE_WEST1` |
+| **Azure** | eastus | `eastus` |
+| **Azure** | westeurope | `westeurope` |
+
+See [references/connection-configs.md](references/connection-configs.md) for the full region mapping table. If unsure, inspect an existing workspace with `atlas-streams-discover` → `inspect-workspace` and check `dataProcessRegion.region`.
+
+## Connection Capabilities — Source/Sink Reference
+
+Know what each connection type can do before creating pipelines:
+
+| Connection Type | As Source ($source) | As Sink ($merge / $emit) | Mid-Pipeline | Notes |
+|-----------------|---------------------|--------------------------|--------------|-------|
+| **Cluster** | ✅ Change streams | ✅ $merge to collections | ✅ $lookup | Change streams monitor insert/update/delete/replace operations |
+| **Kafka** | ✅ Topic consumer | ✅ $emit to topics | ❌ | Source MUST include `topic` field |
+| **Sample Stream** | ✅ Sample data | ❌ Not valid | ❌ | Testing/demo only |
+| **S3** | ❌ Not valid | ✅ $emit to buckets | ❌ | Sink only - use `path`, `format`, `compression`. Supports AWS PrivateLink. |
+| **Https** | ❌ Not valid | ✅ $https as sink | ✅ $https enrichment | Can be used mid-pipeline for enrichment OR as final sink stage |
+| **AWSLambda** | ❌ Not valid | ✅ $externalFunction (async only) | ✅ $externalFunction (sync or async) | **Sink:** `execution: "async"` required. **Mid-pipeline:** `execution: "sync"` or `"async"` |
+| **AWS Kinesis** | ✅ Stream consumer | ✅ $emit to streams | ❌ | Similar to Kafka pattern |
+| **SchemaRegistry** | ❌ Not valid | ❌ Not valid | ✅ Schema resolution | **Metadata only** - used by Kafka connections for Avro schemas |
+
+**Common connection usage mistakes to avoid:**
+- ❌ Using `$externalFunction` as sink with `execution: "sync"` → Must use `execution: "async"` for sink stage
+- ❌ Forgetting change streams exist → Atlas Cluster is a powerful source, not just a sink
+- ❌ Using `$merge` with Kafka → Use `$emit` for Kafka sinks
+
+See [references/connection-configs.md](references/connection-configs.md) for detailed connection configuration schemas by type.
+
+## Core Workflows
+
+### Setup from scratch
+1. `atlas-streams-discover` → `list-workspaces` (check existing)
+2. `atlas-streams-build` → `resource: "workspace"` (region near data, SP10 for dev)
+3. `atlas-streams-build` → `resource: "connection"` (for each source/sink/enrichment)
+4. **Validate connections:** `atlas-streams-discover` → `list-connections` + `inspect-connection` for each — verify names match targets, present summary to user
+5. Call `search-knowledge` to validate field names. Fetch relevant examples from https://github.com/mongodb/ASP_example
+6. `atlas-streams-build` → `resource: "processor"` (with DLQ configured)
+7. `atlas-streams-manage` → `start-processor` (warn about billing)
+
+### Workflow Patterns
+
+**Incremental pipeline development (recommended):**
+See [references/development-workflow.md](references/development-workflow.md) for the full 5-phase lifecycle.
+1. Start with basic `$source` → `$merge` pipeline (validate connectivity)
+2. Add `$match` stages (validate filtering)
+3. Add `$addFields` / `$project` transforms (validate reshaping)
+4. Add windowing or enrichment (validate aggregation logic)
+5. Add error handling / DLQ configuration
+
+**Modify a processor pipeline:**
+1. `atlas-streams-manage` → `action: "stop-processor"` — **processor MUST be stopped first**
+2. `atlas-streams-manage` → `action: "modify-processor"` — provide new pipeline
+3. `atlas-streams-manage` → `action: "start-processor"` — restart
+
+**Debug a failing processor:**
+1. `atlas-streams-discover` → `diagnose-processor` — one-shot health report. Always call this first.
+2. **Commit to a specific root cause.** Match symptoms to diagnostic patterns:
+   - **Error 419 + "no partitions found"** → Kafka topic doesn't exist or is misspelled
+   - **State: FAILED + multiple restarts** → connection-level error (bypasses DLQ), check connection config
+   - **State: STARTED + zero output + windowed pipeline** → likely idle Kafka partitions blocking window closure; add `partitionIdleTimeout` to Kafka `$source` (e.g., `{"size": 30, "unit": "second"}`)
+   - **State: STARTED + zero output + non-windowed** → check if source has data; inspect Kafka offset lag
+   - **High memoryUsageBytes approaching tier limit** → OOM risk; recommend higher tier
+   - **DLQ count increasing** → per-document errors; use MongoDB `find` on DLQ collection
+   See [references/output-diagnostics.md](references/output-diagnostics.md) for the full pattern table.
+3. Classify processor type before interpreting output volume (alert vs transformation vs filter).
+4. Provide concrete, ordered fix steps specific to the diagnosed root cause. Do NOT present a list of hypothetical scenarios.
+5. If detailed logs are needed, direct the user to the Atlas UI: **Atlas → Stream Processing → Workspace → Processor → Logs tab**.
+
+### Chained processors (multi-sink pattern)
+**CRITICAL: A single pipeline can only have ONE terminal sink** (`$merge` or `$emit`). When users request multiple output destinations (e.g., "write to Atlas AND emit to Kafka"), you MUST acknowledge the single-sink constraint and propose chained processors using an intermediate destination. See [references/pipeline-patterns.md](references/pipeline-patterns.md) for the full pattern with examples.
+
+## Pre-Deploy & Post-Deploy Checklists
+
+See [references/development-workflow.md](references/development-workflow.md) for the complete pre-deploy quality checklist (connection validation, pipeline validation) and post-deploy verification workflow.
+
+## Tier Sizing & Performance
+
+See [references/sizing-and-parallelism.md](references/sizing-and-parallelism.md) for tier specifications, parallelism formulas, complexity scoring, and performance optimization strategies.
+
+## Troubleshooting
+
+See [references/development-workflow.md](references/development-workflow.md) for the complete troubleshooting table covering processor failures, API errors, configuration issues, and performance problems.
+
+## Billing & Cost
+
+**Atlas Stream Processing has no free tier.** All deployed processors incur continuous charges while running.
+
+- Charges are per-hour, calculated per-second, only while the processor is running
+- `stop-processor` stops billing; stopped processors retain state for 45 days at no charge
+- **For prototyping without billing:** Use `sp.process()` in mongosh — runs pipelines ephemerally without deploying a processor
+- See `references/sizing-and-parallelism.md` for tier pricing and cost optimization strategies
+
+## Safety Rules
+
+- `atlas-streams-teardown` and `atlas-streams-manage` require user confirmation — do not bypass
+- **BEFORE calling `atlas-streams-teardown` for a workspace**, you MUST first inspect the workspace with `atlas-streams-discover` to count connections and processors, then present this information to the user before requesting confirmation
+- **BEFORE creating any processor**, you MUST validate all connections per the "Pre-Deployment Validation" section in [references/development-workflow.md](references/development-workflow.md)
+- Deleting a workspace removes ALL connections and processors permanently
+- After stopping a processor, state is preserved 45 days — then checkpoints are discarded
+- `resumeFromCheckpoint: false` drops all window state — warn user first
+- Moving processors between workspaces is not supported (must recreate)
+- Dry-run / simulation is not supported — explain what you would do and ask for confirmation
+- Always warn users about billing before starting processors
+- Store API authentication credentials in connection settings, never hardcode in processor pipelines
+
+## Reference Files
+
+| File | Read when... |
+|------|-------------|
+| [`references/pipeline-patterns.md`](references/pipeline-patterns.md) | Building or modifying processor pipelines |
+| [`references/connection-configs.md`](references/connection-configs.md) | Creating connections (type-specific schemas) |
+| [`references/development-workflow.md`](references/development-workflow.md) | Following lifecycle management or debugging decision trees |
+| [`references/output-diagnostics.md`](references/output-diagnostics.md) | Processor output is unexpected (zero, low, or wrong) |
+| [`references/sizing-and-parallelism.md`](references/sizing-and-parallelism.md) | Choosing tiers, tuning parallelism, or optimizing cost |

package/dist/global-skills/mongodb-atlas-stream-processing/references/connection-configs.md

@@ -0,0 +1,298 @@
+# Connection Configuration Reference
+
+**Official examples repo**: https://github.com/mongodb/ASP_example — check quickstarts, example processors, and Terraform examples. Start with quickstarts.
+
+## Connection Capabilities — Source/Sink Reference
+
+Know what each connection type can do before creating pipelines:
+
+| Connection Type | As Source ($source) | As Sink ($merge / $emit) | Mid-Pipeline | Notes |
+|-----------------|---------------------|--------------------------|--------------|-------|
+| **Cluster** | ✅ Change streams | ✅ $merge to collections | ✅ $lookup | Change streams monitor insert/update/delete/replace operations |
+| **Kafka** | ✅ Topic consumer | ✅ $emit to topics | ❌ | Source MUST include `topic` field |
+| **Sample Stream** | ✅ Sample data | ❌ Not valid | ❌ | Testing/demo only |
+| **S3** | ❌ Not valid | ✅ $emit to buckets | ❌ | Sink only - use `path`, `format`, `compression` |
+| **Https** | ❌ Not valid | ✅ $https as sink | ✅ $https enrichment | Can be used mid-pipeline for enrichment OR as final sink stage |
+| **AWSLambda** | ❌ Not valid | ✅ $externalFunction (async only) | ✅ $externalFunction (sync or async) | **Sink:** `execution: "async"` required. **Mid-pipeline:** `execution: "sync"` or `"async"` |
+| **AWS Kinesis** | ✅ Stream consumer | ✅ $emit to streams | ❌ | Similar to Kafka pattern |
+| **SchemaRegistry** | ❌ Not valid | ❌ Not valid | ✅ Schema resolution | **Metadata only** - used by Kafka connections for Avro schemas |
+
+**Common connection usage mistakes to avoid:**
+- ❌ Using HTTPS connections as `$source` → HTTPS is for enrichment or sink only
+- ❌ Using `$externalFunction` as sink with `execution: "sync"` → Must use `execution: "async"` for sink stage
+- ❌ Forgetting change streams exist → Atlas Cluster is a powerful source, not just a sink
+- ❌ Using `$merge` with Kafka → Use `$emit` for Kafka sinks
+
+**$externalFunction execution modes:**
+- **Mid-pipeline:** Can use `execution: "sync"` (blocks until Lambda returns) or `execution: "async"` (non-blocking)
+- **Final sink stage:** MUST use `execution: "async"` only
+
+## Connection Naming Best Practices
+
+**CRITICAL**: Connection names should clearly indicate their actual targets to avoid confusion and prevent writing data to wrong destinations.
+
+### Good Naming Patterns
+
+**Match the actual target name:**
+- Cluster connection to "ClusterRestoreTest" → name it `cluster-restore-test` or `ClusterRestoreTest`
+- Cluster connection to "AtlasCluster" → name it `atlas-cluster` or `AtlasCluster`
+
+**Use descriptive names with context:**
+- `prod-kafka-orders` (indicates environment + service + purpose)
+- `dev-atlas-main` (indicates environment + service + designation)
+- `staging-s3-exports` (indicates environment + service + purpose)
+
+### Bad Naming Patterns (AVOID)
+
+❌ **Generic names that don't match targets:**
+- Connection "atlascluster" pointing to "ClusterRestoreTest" ← CONFUSING!
+- Connection "kafka" pointing to multiple different topics ← NOT SPECIFIC!
+
+❌ **Reusing names across workspaces without context:**
+- "myconnection" in workspace A and workspace B with different targets
+
+❌ **Names that don't indicate connection type:**
+- "connection1", "test", "temp" ← NO CONTEXT!
+
+### Verification Workflow
+
+**Before creating processors**, always inspect your connections to verify they point where you expect:
+```
+1. atlas-streams-discover → action: "list-connections"
+2. atlas-streams-discover → action: "inspect-connection" for each
+3. Verify connection name matches actual target (clusterName, bootstrapServers, url, etc.)
+4. If mismatch exists, consider renaming or warn the user
+```
+
+See [development-workflow.md](development-workflow.md) "Pre-Deployment Connection Validation" section for the complete validation procedure.
+
+## Important Notes
+- HTTPS connections are for `$https` enrichment ONLY — they are NOT valid as `$source` data sources
+- Store API authentication in connection settings, never hardcode in processor pipelines
+- AWS connections (S3, Kinesis, Lambda) require IAM role ARN registered via Atlas Cloud Provider Access first
+- Supported `connectionType` values: `Kafka`, `Cluster`, `S3`, `Https`, `AWSKinesisDataStreams`, `AWSLambda`, `SchemaRegistry`, `Sample`
+
+## AWS Cloud Provider Access Prerequisites
+
+**For S3, Kinesis, and Lambda connections:**
+
+AWS connections (S3, Kinesis, Lambda) require that the IAM role ARN be **registered in the Atlas project via Cloud Provider Access** before creating the connection. This is a prerequisite — the connection creation will fail without it.
+
+**Always mention this prerequisite** in your response when the user wants to create AWS connections, even if the user says connections already exist. Confirm with language like:
+- "IAM role ARNs are registered via Atlas Cloud Provider Access"
+- "Ensure IAM role ARNs are registered via Atlas Cloud Provider Access before creating connections"
+
+**Security best practice:** Use a dedicated IAM role per processor (or group of related processors) with least-privilege permissions scoped only to the specific S3 buckets, Kinesis streams, or Lambda functions that processor needs. Avoid sharing broad-access roles across unrelated processors.
+
+## Region Mapping Reference
+
+The `region` field for workspace creation uses Atlas-specific names that differ by cloud provider. Using the wrong format returns a cryptic `dataProcessRegion` error.
+
+| Provider | Cloud Region | Streams `region` Value |
+|----------|-------------|----------------------|
+| **AWS** | us-east-1 | `VIRGINIA_USA` |
+| **AWS** | us-east-2 | `OHIO_USA` |
+| **AWS** | us-west-2 | `OREGON_USA` |
+| **AWS** | ca-central-1 | `MONTREAL_CAN` |
+| **AWS** | sa-east-1 | `SAOPAULO_BRA` |
+| **AWS** | eu-west-1 | `DUBLIN_IRL` |
+| **AWS** | ap-southeast-1 | `SINGAPORE_SGP` |
+| **AWS** | ap-south-1 | `MUMBAI_IND` |
+| **AWS** | ap-northeast-1 | `TOKYO_JPN` |
+| **GCP** | us-central1 | `US_CENTRAL1` |
+| **GCP** | europe-west1 | `EUROPE_WEST1` |
+| **GCP** | us-east4 | `US_EAST4` |
+| **Azure** | eastus | `eastus` |
+| **Azure** | eastus2 | `eastus2` |
+| **Azure** | westus | `westus` |
+| **Azure** | westeurope | `westeurope` |
+
+This is a partial list. If unsure, inspect an existing workspace with `atlas-streams-discover` → `inspect-workspace` and check `dataProcessRegion.region`.
+
+## MCP Tool Behaviors for Connections
+
+**Elicitation:** When required fields are missing, the build tool auto-prompts for them via an interactive form (MCP elicitation protocol). Do NOT manually ask the user for passwords or bootstrap servers — let the tool collect them.
+
+**Auto-normalization:**
+- `bootstrapServers` passed as array → auto-converted to comma-separated string
+- `schemaRegistryUrls` passed as string → auto-wrapped in array
+- Cluster `dbRoleToExecute` → auto-defaults to `{role: "readWriteAnyDatabase", type: "BUILT_IN"}` if omitted
+
+## connectionConfig by type
+
+### Kafka
+```json
+{
+  "bootstrapServers": "broker1:9092,broker2:9092",
+  "authentication": {
+    "mechanism": "SCRAM-256",
+    "username": "my-user",
+    "password": "my-password"
+  },
+  "security": {
+    "protocol": "SASL_SSL"
+  }
+}
+```
+**Important:** `bootstrapServers` is a **comma-separated string**, not an array.
+
+All fields above are required. The tool will prompt the user for username/password via elicitation if not provided.
+
+Authentication mechanisms: `PLAIN`, `SCRAM-256`, `SCRAM-512`, `OAUTHBEARER`
+Security protocols: `SASL_SSL`, `SASL_PLAINTEXT`, `SSL`
+
+For Confluent Cloud, use `mechanism: "PLAIN"` with your API key as `username` and API secret as `password`.
+
+Kafka supports both **PrivateLink** and **VPC Peering** for private networking. See the [PrivateLink Reference](#privatelink-reference-all-vendors) section below for all supported vendors and providers.
+
+**VPC Peering:**
+- Supported for outbound connections to Kafka brokers in your own VPC
+- Requires `SASL_SSL` security protocol
+- Use `atlas-streams-manage` with `accept-peering` action to complete the peering setup
+- Requires AWS account ID, VPC ID, and region information
+
+**Important: Networking cannot be modified after connection creation.** To add or change PrivateLink/VPC peering on an existing Kafka connection, you must delete it and recreate it with the networking config.
+
+Use `atlas-streams-discover` → `action: "get-networking"` to list available PrivateLink endpoints and VPC peering connections.
+
+### Cluster (Atlas)
+```json
+{
+  "clusterName": "my-atlas-cluster",
+  "dbRoleToExecute": {
+    "role": "readWriteAnyDatabase",
+    "type": "BUILT_IN"
+  }
+}
+```
+`clusterName` is **required** — must be a cluster in the same project (use `atlas-list-clusters` to verify).
+
+`dbRoleToExecute` defaults to `{role: "readWriteAnyDatabase", type: "BUILT_IN"}` if not provided.
+
+Optional: `clusterGroupId` (if cluster is in a different project — requires cross-project access to be enabled at the org level).
+
+### S3
+```json
+{
+  "aws": {
+    "roleArn": "arn:aws:iam::123456789:role/streams-s3-role",
+    "testBucket": "my-test-bucket"
+  }
+}
+```
+**Prerequisite:** The IAM role ARN must be registered in the Atlas project via Cloud Provider Access before creating the connection.
+
+Required IAM policy permissions: `s3:ListBucket`, `s3:GetObject`, `s3:PutObject`.
+
+### Https
+```json
+{
+  "url": "https://api.example.com/webhook",
+  "headers": {
+    "Authorization": "Bearer token123"
+  }
+}
+```
+**IMPORTANT:** HTTPS connections are for `$https` enrichment stages ONLY. They are NOT valid data sources — do not use them in `$source`.
+
+Store all API authentication in the connection config headers, not in the processor pipeline.
+
+#### HTTPS Auth Patterns
+
+**API Key:**
+```json
+{"url": "https://api.example.com", "headers": {"X-API-Key": "your-api-key"}}
+```
+
+**Bearer Token:**
+```json
+{"url": "https://api.example.com", "headers": {"Authorization": "Bearer your-token"}}
+```
+
+**Basic Auth:**
+```json
+{"url": "https://api.example.com", "headers": {"Authorization": "Basic base64-encoded-credentials"}}
+```
+
+**OAuth 2.0 (pre-obtained token):**
+```json
+{"url": "https://api.example.com", "headers": {"Authorization": "Bearer oauth-access-token"}}
+```
+
+### AWSKinesisDataStreams
+```json
+{
+  "aws": {
+    "roleArn": "arn:aws:iam::123456789:role/streams-kinesis-role"
+  }
+}
+```
+**Prerequisite:** The IAM role ARN must be registered in the Atlas project via Cloud Provider Access before creating the connection.
+
+Required IAM policy permissions: `kinesis:ListShards`, `kinesis:SubscribeToShard`, `kinesis:PutRecords`, `kinesis:DescribeStreamSummary`.
+
+### AWSLambda
+```json
+{
+  "aws": {
+    "roleArn": "arn:aws:iam::123456789:role/streams-lambda-role"
+  }
+}
+```
+**Prerequisite:** The IAM role ARN must be registered in the Atlas project via Cloud Provider Access before creating the connection.
+
+### SchemaRegistry
+```json
+{
+  "connectionType": "SchemaRegistry",
+  "connectionConfig": {
+    "schemaRegistryUrls": ["https://schema-registry.example.com"],
+    "schemaRegistryAuthentication": {
+      "type": "USER_INFO",
+      "username": "...",
+      "password": "..."
+    }
+  }
+}
+```
+- `connectionType` MUST be `"SchemaRegistry"` (not `"Kafka"` or `"Https"`)
+- `schemaRegistryUrls` is an **array** (not a string). The tool auto-wraps a string into an array if needed.
+- `schemaRegistryAuthentication.type`: `"USER_INFO"` (explicit credentials) or `"SASL_INHERIT"` (inherit from Kafka connection)
+- Tool elicitation will collect sensitive fields (password) — don't ask the user for these directly
+
+### Sample
+No connectionConfig required. Provides built-in test data. Useful for development and testing without external infrastructure.
+
+Available sample formats: `sample_stream_solar` (default, auto-created when `includeSampleData: true` on workspace), `samplestock`, `sampleweather`, `sampleiot`, `samplelog`, `samplecommerce`.
+
+### PrivateLink Reference (All Vendors)
+
+PrivateLink is supported for Kafka, S3, Kinesis, and Azure EventHub connections. Create a project-level PrivateLink first, then reference it in the connection's `networking.access` config.
+
+**Step 1: Create project-level PrivateLink** via `atlas-streams-build` resource='privatelink':
+
+| Provider | Vendor | Required privateLinkConfig fields |
+|----------|--------|----------------------------------|
+| AWS | CONFLUENT | provider, vendor, dnsDomain, dnsSubDomain (array, [] if none) |
+| AWS | MSK | provider, vendor, arn |
+| AWS | S3 | provider, vendor, region, serviceEndpointId (`com.amazonaws.<region>.s3`) |
+| AWS | KINESIS | provider, vendor, region, serviceEndpointId |
+| AZURE | EVENTHUB | provider, vendor, dnsDomain, serviceEndpointId |
+| AZURE | CONFLUENT | provider, vendor, dnsDomain |
+| GCP | CONFLUENT | provider, vendor, gcpServiceAttachmentUris |
+
+**Step 2: Reference in connection networking config:**
+```json
+{
+  "networking": {
+    "access": {
+      "type": "PRIVATE_LINK",
+      "connectionId": "<PrivateLink _id from Step 1>"
+    }
+  }
+}
+```
+
+Use `atlas-streams-discover` action='get-networking' to find the PrivateLink `_id`.
+
+**Note:** Networking config cannot be modified after connection creation — delete and recreate to change.