@ductape/mcp 0.1.38 → 0.1.39

package/dist/index.js

@@ -1727,6 +1727,31 @@ Important:
     apps: `
 DUCTAPE APPS
 
+WHAT AN APP IS:
+A Ductape App is a pre-configured, versioned API integration definition. It is NOT a generic HTTP
+client, NOT a job scheduler, and NOT anything you can call without registering first.
+
+An App must be fully set up in Ductape before any code can use it:
+  1. The App record must be created (name, tag, description)
+  2. Environments must be added (each environment slug → base URL for that stage)
+  3. Auth scheme must be configured (how outbound requests authenticate: apikey, bearer, OAuth2, etc.)
+  4. Action endpoints must be defined (each action = one HTTP endpoint spec: method, path, body/query/header shape, response shape)
+  5. The App must be connected to the product (product.apps.add) and its envs mapped
+
+ONLY after all five steps can any code call:
+  ctx.api.run({ app: '<app_tag>', event: '<action_tag>', input: { ... } })   ← in a feature handler
+  actions.run([{ product, env, app: '<app_tag>', action: '<action_tag>', input }])   ← at runtime
+
+ctx.api (alias for ctx.action) is NOT a generic HTTP call. It ONLY invokes a pre-registered
+Ductape App Action. If the App or action tag does not exist in the product, the call will fail.
+actions.dispatch is the same as actions.run but scheduled as a background job — it still requires
+a registered App. There is no way to dispatch a job to an arbitrary URL via ctx.api or actions.dispatch.
+
+If a feature step needs to call an external service and no App is registered for it yet:
+  → Flag it as "App to create" in your plan (STEP 4 of the feature design workflow)
+  → Create the App and all its actions first (see below)
+  → Only then write the ctx.api.run call
+
 An app is a versioned API integration definition. It contains environments (base URLs), actions
 (individual endpoint specs), auth schemes, webhooks, variables, and constants.
 
@@ -1742,6 +1767,21 @@ Manage environments (base URLs per stage):
   ductape_execute("app.environments.create", [app_tag, { slug, env_name, base_url }])
   ductape_execute("app.environments.list",   [app_tag])
 
+Discover apps in a product and their actions (ALWAYS do this before writing any ctx.api.run call):
+  Step 1 — list apps connected to the product:
+    ductape_cli("products get --tag <product_tag> --json")    → full product document; check apps[]
+    ductape_cli("products apps list --product <product_id> --json")  → apps[] with access_tag, envs
+  Step 2 — list actions in an app:
+    ductape_execute("actions.list", [app_tag])                → returns all action tags + names
+  Step 3 — fetch the input schema for an action:
+    ductape_execute("actions.fetch", [app_tag, action_tag])
+      → returns { body: {fieldName: {type, required}}, params: {}, query: {}, headers: {} }
+    OR: call ductape_generate_payload (operation_family="action", method="run",
+          targets={app: "app_tag", action: "action_tag"}) to get the exact resolved payload shape
+  Step 4 — call ductape_schema({ module: "app" }) if you need the JSON schema for creating/updating
+           app resources (not for runtime input — use actions.fetch or ductape_generate_payload for that)
+  NEVER assume action input field names. Always fetch the action definition first.
+
 Manage actions (individual API endpoints):
   ductape_execute("actions.create", [app_tag, {
     tag, name, resource: "/users/{id}", method: "GET"|"POST"|"PUT"|"PATCH"|"DELETE",
@@ -1755,6 +1795,16 @@ Manage actions (individual API endpoints):
   ductape_execute("actions.list",   [app_tag])
   ductape_execute("actions.fetch",  [app_tag, action_tag])
 
+Action input — flat input format:
+  Fields are resolved to the correct location (body/params/query/headers) by matching the action schema.
+  For ambiguous keys, use explicit prefixes:
+    input: { amount: 1000 }                       → auto-resolved (body.amount if body field exists)
+    input: { "body:amount": 1000 }                → explicit body
+    input: { "params:id": "user_123" }            → route parameter
+    input: { "query:limit": 10 }                  → query string
+    input: { "headers:X-Idempotency-Key": "..." } → request header
+  Always use ductape_generate_payload or actions.fetch to know the exact field names — never guess.
+
 Run an action at runtime:
   → CALL ductape_generate_payload FIRST (operation_family="action", method="run", targets={app, action})
   ductape_execute("actions.run", [{ product, env, app, action, input: { "body:field": value } }])
@@ -1774,10 +1824,21 @@ Variables (per-env mutable values) and Constants (fixed values):
   ductape_execute("app.constants.create", [app_tag, { key, value }])
 
 Connecting an app to a product (after creation):
-  NOTE: All product.* module methods require the access key and CANNOT use ductape_execute (publishable key only).
-  Use ductape_cli for all product-level operations:
-  ductape_cli("products apps list --product <product_id> --json")
-  ductape_cli("products get --tag <product_tag> --json")    ← includes apps[], databases[], features[] etc.
+  NOTE: All product.* module methods require the access key and CANNOT use ductape_execute.
+  Use ductape_cli for all product-level operations.
+
+  FULL FLOW to make an app callable from a product:
+  1. Create the app:     ductape_cli("apps create --name \\"Service Name\\" --description \\"...\\"")
+  2. Add environments:   ductape_execute("app.environments.create", [app_tag, { slug: "prd", env_name: "Production", base_url: "https://api.example.com" }])
+  3. Configure auth:     ductape_execute("auths.create", [app_tag, { tag, name, setup_type: "apikey"|"bearer"|"basic"|"oauth2", expiry, period }])
+  4. Define actions:     ductape_execute("actions.create", [app_tag, { tag, name, resource, method, body?, params?, query?, headers?, response? }])
+                         OR import: ductape_cli("apps import <file.json> -t postman|openapi")
+  5. Connect to product: Requires the product_id (from ductape_cli("products get --tag <tag> --json")).
+                         There is no CLI command for this step — the SDK product.apps.add method requires
+                         an access key which only the backend can provide. Use ductape_execute via an
+                         admin-authenticated context, or connect via the Workbench UI.
+  6. Verify:             ductape_cli("products get --tag <product_tag> --json") → check apps[] contains the app
+                         ductape_execute("actions.list", [app_tag]) → verify actions are registered
 `.trim(),
     products: `
 DUCTAPE PRODUCTS
@@ -2086,7 +2147,13 @@ STEP 2 — INVENTORY existing Ductape components
   (product.* requires the access key — never use ductape_execute for product reads, it will return 403)
   Note what already exists:
     - databases[]      → available for ctx.database.insert/query/update/delete steps
-    - apps[]           → available for ctx.api.run steps (check app.events[] for event tags)
+    - apps[]           → available for ctx.api.run steps ONLY if the App is fully registered:
+                         (a) App record exists, (b) environments defined with base URLs,
+                         (c) auth scheme configured, (d) action endpoints defined,
+                         (e) connected to the product via product.apps.add.
+                         Check app.events[] for action tags. If no app exists for a service the
+                         feature needs to call, flag it as "App to create" in the plan — do NOT
+                         assume ctx.api can call any URL or schedule any job without a registered App.
     - notifications[]  → available for ctx.notification.email/sms/push steps
     - storage[]        → available for ctx.storage.upload/download steps
     - messageBrokers[] → available for ctx.events.produce steps
@@ -2094,10 +2161,14 @@ STEP 2 — INVENTORY existing Ductape components
     - features[]       → can be called as child features via ctx.feature.execute()
     - caches[], sessions[]
   Do NOT assume a component or event tag exists — verify from the product before using it.
+  Do NOT treat ctx.api as a generic HTTP call or job scheduler. It requires a registered App.
 
 STEP 3 — PLAN each step
   For every logical step:
-    a. Identify which existing component handles it, or flag it as needing creation
+    a. Identify which existing component handles it, or flag it as needing creation.
+       If a step calls an external service, it MUST go through a registered Ductape App.
+       If no App for that service exists in the product → mark it "App to create: <service name>".
+       DO NOT plan a raw HTTP call, a direct dispatch to a URL, or any workaround in place of a missing App.
     b. Note how inputs flow: ctx.input fields, or return values from earlier steps (plain JS variables — no special notation needed)
     c. Decide if a rollback handler is needed (e.g. charge → refund on later failure)
     d. Decide allow_fail: true for non-critical steps (email, analytics, audit logs)
@@ -2141,7 +2212,8 @@ STEP 7 — HANDLE conditionals, loops, and branching (when applicable)
     → Only the scenario whose input matches runs at execution time
 
 STEP 8 — SET rollbacks for reversible steps
-  Any step that allocates a resource should undo it if a later step fails:
+  Any step that allocates a resource should undo it if a later step fails.
+  ctx.api.run requires a pre-registered Ductape App — 'stripe' below is the tag of a registered App:
     const charge = await ctx.step(
       'charge',
       async () => ctx.api.run({ app: 'stripe', event: 'create-charge', input: { amount: ctx.input.amount } }),
@@ -2201,6 +2273,52 @@ Signals and queries (for long-running features):
 
 Rollback strategies: reverse_all | reverse_critical | compensate | none
 Feature statuses: pending | running | completed | failed | rolled_back | rolling_back | paused
+
+━━━ FEATURE RECORDING SEMANTICS ━━━
+
+When you call features.define({ handler }), the handler runs TWICE:
+
+  1. RECORDING PHASE (at define time) — handler is called with a RecordingContext.
+     All ctx.step() calls return lightweight proxy objects, not real data.
+     This phase captures the step graph: which steps exist, their types, tags, and declared
+     inputs/outputs. No real API calls, DB queries, or side effects occur.
+     Arbitrary JS code OUTSIDE ctx.step() ALSO runs during recording — with proxy values.
+     For loops: use recordInput so the handler sees sample data and all iterations are recorded.
+     For branches: use branchOverrides so each path is captured.
+
+  2. EXECUTION PHASE (at runtime) — handler is called with a real ExecutionContext.
+     ctx.step() actually executes. All real Ductape component calls happen.
+     Arbitrary JS logic (math, string ops, conditionals on step results) runs for real.
+
+  Implication: put all meaningful business logic INSIDE ctx.step() handlers, not in the
+  outer handler body. Code in the outer body runs during recording with proxy values and
+  may behave unexpectedly (e.g. typeof proxy === 'object' is true but .someField is a proxy).
+
+  Features do NOT execute arbitrary NestJS or server code directly. A feature handler can only
+  call Ductape component primitives (ctx.api, ctx.database, ctx.notification, etc.) as steps.
+  To invoke internal application business logic, produce a broker event from a feature step
+  (ctx.messaging.produce) and consume it in your NestJS service — that is the correct pattern.
+
+━━━ ORCHESTRATION DECISION RULE ━━━
+
+  One component operation at a future time:
+    → use that component's own dispatch method
+      e.g. ductape.events.dispatch({ ..., schedule: { start_at: ... } })
+      e.g. ductape.api.dispatch({ ..., schedule: { start_at: ... } })
+      e.g. ductape.database.dispatch({ ..., schedule: { start_at: ... } })
+
+  Several durable Ductape component operations in sequence (with rollback / retry / state):
+    → define a Feature, then features.dispatch to schedule it
+
+  Invoke internal application business logic (your own NestJS/backend service code):
+    → produce a broker event (ctx.messaging.produce or ductape.events.produce)
+    → consume it in your NestJS service with events.consume in onModuleInit
+    → your service method runs with full access to DI, DB transactions, etc.
+    Do NOT create an App Action just to call your own service over HTTP.
+
+  Invoke an external/public HTTP service:
+    → create a Ductape App (register base URL, auth, action endpoints) then use ctx.api.run
+    → requires the App to be fully registered and connected to the product first
 `.trim(),
     events: `
 DUCTAPE EVENTS (MESSAGE BROKERS)
@@ -2298,81 +2416,173 @@ Import (register an EXISTING cloud resource):
     redis:    { host: "...", port: 6379, password?: "..." }
     nats:     { servers: ["nats://host:4222"], token?: "...", user?: "...", pass?: "...", tls?: true }
 
-━━━ STEP 2: ADD TOPIC DEFINITIONS ━━━
-
-  For most providers (GCP Pub/Sub, Kafka, RabbitMQ, Redis, NATS, Azure Service Bus):
-  SKIP this step. Topics are auto-created when you create a producer (step 3).
-  You do NOT need to call messageBrokers.topics.create before creating a producer.
+━━━ STEP 2: DEFINE TOPICS ━━━
 
-  Only call topics.create explicitly when:
-    - Using AWS SQS (must supply queueUrls per env — auto-creation cannot know the queue URL)
-    - You want to pre-set sample data or idempotency config on the topic
+  Topics MUST be defined before any consumer can subscribe to them.
+  Producing to a topic also calls ensureTopicRegistered in the background — but DO NOT rely on
+  auto-registration for consume paths. Always create topics explicitly.
 
-  If you do need it (SQS or explicit config):
   ductape_execute("messageBrokers.topics.create", [product_tag, {
-    tag: "player-joined",
-    name: "Player Joined",
-    broker: "notifications-broker",   // broker component tag
+    tag: "order-created",             // topic tag (just the topic part, NOT "broker:topic")
+    name: "Order Created",
+    broker: "order-events",           // broker component tag
     description?: string,
-    sample: { playerId: "string", username: "string" },  // example message shape
-    idempotent?: boolean,
-    // SQS only — map per-env queue URLs (each topic can be a different queue):
+    sample: { orderId: "string", total: 0 },   // documents expected message shape
+    idempotent?: boolean,             // if true, Ductape deduplicates by idempotency_key
+    // AWS SQS only — must supply per-env queue URL:
     queueUrls?: [
-      { env_slug: "snd", url: "https://sqs.us-east-1.amazonaws.com/123456/my-product-player-joined-snd" },
-      { env_slug: "prd", url: "https://sqs.us-east-1.amazonaws.com/123456/my-product-player-joined-prd" }
+      { env_slug: "snd", url: "https://sqs.us-east-1.amazonaws.com/123/queue-snd" },
+      { env_slug: "prd", url: "https://sqs.us-east-1.amazonaws.com/123/queue-prd" }
     ]
   }])
 
-  List topics on a broker:
+  List / fetch topics:
     ductape_execute("messageBrokers.topics.list", [product_tag, "broker-tag"])
+    ductape_execute("messageBrokers.fetch", [product_tag, "broker-tag"])  → includes topics[]
 
-━━━ STEP 3: PRODUCE AND CONSUME — WRITTEN IN APPLICATION CODE ━━━
+━━━ STEP 3: PRODUCE — WRITTEN IN APPLICATION CODE ━━━
 
-  There is NO admin command or file to declare producers/consumers.
-  There is NO "create producer" step before writing code.
-  Producers and consumers are registered automatically by the SDK the first time your code calls
-  produce/consume — you do not pre-declare them.
+  There is NO admin command to declare a producer. Producers are auto-registered by the SDK on
+  the first produce call — you do not pre-declare them.
+  Do NOT call ductape_generate_payload for messaging. The producer owns the schema.
+  Infer the message shape from context, present it to the user for approval, then implement.
 
-  The entire producer/consumer contract is the code you write in your controllers or services:
-
-  Produce (publish a message) — write in your service/controller:
-    Do NOT call ductape_generate_payload for messaging. Events have no pre-existing backend schema
-    to discover — the producer defines the schema. Instead, infer the message shape from context
-    (event name, existing data models, user input), present it to the user for approval, then implement.
+  GENERAL BACKEND (TypeScript/Node.js — not NestJS):
+    import Ductape from '@ductape/sdk';
+    const ductape = new Ductape({ accessKey: 'your-access-key' });
     await ductape.events.produce({
       product: "my-product",
       env: "prd",
-      event: "broker_tag:topic_tag",   // "broker_tag:topic_tag" — always colon-separated
-      message: { key: value },         // shape inferred from context, approved by user
+      event: "broker-tag:topic-tag",   // always colon-separated
+      message: { key: value },
+      session?: "session-tag:jwt",     // optional — traces message to a user session
+    });
+    // Idempotent publish (deduplicates — prevents double-processing on retries):
+    await ductape.events.publishIdempotent({
+      product, env, event, message,
+      idempotencyKey: "order-123-charge",  // stable key unique to this logical operation
+      idempotencyTtl?: 86400,             // seconds; default 86400 (24h)
+    });
+
+  NESTJS — method decorator:
+    import { Messaging } from '@ductape/nestjs';
+    @Injectable() export class OrdersService {
+      @Messaging.Produce({ event: 'order-events:order-created' })
+      emitOrderCreated(payload: { orderId: string; total: number }) { return payload; }
+
+      // Scheduled dispatch — fire-and-forget with optional schedule:
+      @Messaging.Dispatch({ broker: 'order-events', event: 'order-events:order-created',
+                            schedule?: { start_at?, cron?, every?, limit?, tz? } })
+      scheduleOrderNotification(payload: Record<string, unknown>) { return payload; }
+    }
+
+  CLIENT-SIDE (browser — publishable key):
+    Clients CAN produce messages using a publishable key + session token.
+    Only produce to topics whose schema is safe for client authorship.
+    NEVER allow clients to produce to topics that trigger privileged server-side operations
+    (payments, admin actions, state mutations) — those must go through a backend endpoint first.
+    import Ductape from '@ductape/sdk';
+    const ductape = new Ductape({ publishableKey: 'pk_...', env: 'prd', product: 'my-product' });
+    await ductape.events.produce({
+      event: "user-events:user-action",
+      message: { action: "button-click", screen: "dashboard" },
+      session: "user-session:eyJ...",  // REQUIRED for client-side produce
     });
-    Idempotent publish (deduplicates by key):
-      await ductape.events.publishIdempotent({ product, env, event, message, idempotencyKey, idempotencyTtl? })
 
-  Consume (subscribe) — write in your service/controller:
+  SCHEDULED DISPATCH (background job):
+    ductape_execute("messageBrokers.dispatch", [{
+      product, env,
+      broker: "order-events",                // broker tag
+      event: "order-events:reminder-due",    // "broker:topic"
+      input: { message: { orderId: "123" } },
+      retries?: 3,
+      session?: "session-tag:jwt",
+      schedule?: {
+        start_at?: 1735689600000,    // Unix ms or ISO string
+        cron?: "0 9 * * *",          // recurring cron
+        every?: 86400000,            // recurring interval ms
+        limit?: 10,                  // max repetitions
+        endDate?: "2026-12-31",
+        tz?: "America/New_York",
+      },
+    }])
+    Returns: { job_id, status: "scheduled"|"queued", scheduled_at, recurring, next_run_at? }
+
+━━━ STEP 4: CONSUME — WRITTEN IN APPLICATION CODE ━━━
+
+  Consumers are auto-registered by the SDK on first consume call.
+  Consumer registration options (all optional — used for tracking in Workbench):
+    consumer?: { tag?: string, name?: string, description?: string }
+  If tag is omitted, Ductape generates one: "consumer-<brokerTag>-<topicTag>".
+
+  ACK BEHAVIOR (automatic):
+    - Callback returns successfully → message is acknowledged (ack)
+    - Callback throws → message is tracked as failed; broker nacks/retries per provider behavior
+    - After max retries → message moves to dead-letter queue (DLQ)
+    There is no manual ack API. Acknowledgement is implicit from callback outcome.
+
+  CONSUMER GROUPS (Kafka-specific):
+    Consumer groups are set in the broker's envs[].config.groupId (at broker registration time).
+    All service instances sharing the same groupId form a consumer group and share partition load.
+    To configure: set groupId in the kafka config when creating/updating the broker.
+
+  CONCURRENCY:
+    Ductape has no per-consumer concurrency setting. Concurrency is determined by:
+      - Number of running service instances (horizontal scale)
+      - Broker-level partition count (Kafka) or visibility timeout (SQS)
+    Run multiple instances of your service to scale consumption.
+
+  GENERAL BACKEND (TypeScript/Node.js — not NestJS):
+    Start consuming in your module init or service startup:
     await ductape.events.consume({
       product: "my-product",
       env: "prd",
-      event: "broker_tag:topic_tag",
-      callback: async (message) => { /* handle message */ },
+      event: "order-events:order-created",
+      callback: async (message) => {
+        // All real processing logic goes here.
+        // Throw to nack. Return to ack.
+        await processOrder(message as { orderId: string; total: number });
+      },
+      consumer?: { tag: "order-processor", name: "Order Processor" },
     });
-    Callback errors are re-thrown so the broker can nack/retry.
 
-  Background dispatch with scheduling — write in your service/controller:
-    await ductape.events.dispatch({ product, env, broker, event, input: { message },
-      schedule?: { start_at?, cron?, every?, limit?, endDate?, tz? } })
+  NESTJS — use SDK in onModuleInit (no @Messaging.Consume decorator exists yet):
+    @Injectable()
+    export class OrderConsumerService implements OnModuleInit {
+      constructor(private readonly ductape: Ductape) {}
+      async onModuleInit() {
+        await this.ductape.events.consume({
+          product: "my-product",
+          env: process.env.DUCTAPE_ENV || 'prd',
+          event: "order-events:order-created",
+          callback: async (message) => { await this.handle(message); },
+          consumer: { tag: "order-consumer", name: "Order Consumer" },
+        });
+      }
+      private async handle(message: unknown) { /* business logic */ }
+    }
 
-  For the four standard producer declarations (match-state, match-report, projection-updated,
-  notification), write these produce calls in the relevant application service methods — there is
-  no separate configuration file or CLI step. The SDK creates the producer metadata on first call.
+  CLIENT-SIDE: Clients CANNOT consume. Event consumption is always server-side only.
+  This is the key distinction between server topics (produce + consume) and client-observable
+  topics (produce from client, consume on server). Never set up a consumer in browser code.
 
-Message payload is AES-encrypted before the tracking API call — tracking never sees plaintext.
+  DEAD-LETTER QUEUE (DLQ):
+    Messages whose callbacks consistently throw are automatically moved to the DLQ.
+    Query:     ductape_execute("messageBrokers.messages.getDeadLetters",
+                 [{ product, env, brokerTag, topicTag?, consumerTag?, limit? }])
+    Reprocess: ductape_execute("messageBrokers.reprocessDLQ",
+                 [{ product, env, brokerTag, topicTag?, messageIds?, limit? }])
+    Replay:    ductape_execute("messageBrokers.replayEvent",
+                 [{ product, env, eventId, force? }])
 
 ━━━ OBSERVABILITY ━━━
 
   messageBrokers.messages.query        [{ product, env, brokerTag, topicTag?, status?, page?, limit? }]
   messageBrokers.messages.getStats     [{ product, env, brokerTag }]
   messageBrokers.messages.getDashboard [{ product, env, brokerTag }]
-  messageBrokers.messages.getDeadLetters [{ product, env, brokerTag, topicTag?, limit? }]
+  messageBrokers.messages.getDeadLetters [{ product, env, brokerTag, topicTag?, consumerTag?, limit? }]
+  messageBrokers.messages.getProducers [{ product, env, brokerTag, topicTag?, page?, limit? }]
+  messageBrokers.messages.getConsumers [{ product, env, brokerTag, topicTag?, page?, limit? }]
   messageBrokers.replayEvent           [{ product, env, eventId, force? }]
   messageBrokers.reprocessDLQ         [{ product, env, brokerTag, topicTag?, messageIds?, limit? }]
   messageBrokers.checkIdempotency      [{ product, env, brokerTag, idempotency_key }]

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ductape/mcp",
-  "version": "0.1.38",
+  "version": "0.1.39",
   "description": "MCP server that exposes Ductape SDK operations via the backend proxy",
   "type": "module",
   "main": "dist/index.js",

package/src/index.ts

@@ -1793,6 +1793,31 @@ Important:
   apps: `
 DUCTAPE APPS
 
+WHAT AN APP IS:
+A Ductape App is a pre-configured, versioned API integration definition. It is NOT a generic HTTP
+client, NOT a job scheduler, and NOT anything you can call without registering first.
+
+An App must be fully set up in Ductape before any code can use it:
+  1. The App record must be created (name, tag, description)
+  2. Environments must be added (each environment slug → base URL for that stage)
+  3. Auth scheme must be configured (how outbound requests authenticate: apikey, bearer, OAuth2, etc.)
+  4. Action endpoints must be defined (each action = one HTTP endpoint spec: method, path, body/query/header shape, response shape)
+  5. The App must be connected to the product (product.apps.add) and its envs mapped
+
+ONLY after all five steps can any code call:
+  ctx.api.run({ app: '<app_tag>', event: '<action_tag>', input: { ... } })   ← in a feature handler
+  actions.run([{ product, env, app: '<app_tag>', action: '<action_tag>', input }])   ← at runtime
+
+ctx.api (alias for ctx.action) is NOT a generic HTTP call. It ONLY invokes a pre-registered
+Ductape App Action. If the App or action tag does not exist in the product, the call will fail.
+actions.dispatch is the same as actions.run but scheduled as a background job — it still requires
+a registered App. There is no way to dispatch a job to an arbitrary URL via ctx.api or actions.dispatch.
+
+If a feature step needs to call an external service and no App is registered for it yet:
+  → Flag it as "App to create" in your plan (STEP 4 of the feature design workflow)
+  → Create the App and all its actions first (see below)
+  → Only then write the ctx.api.run call
+
 An app is a versioned API integration definition. It contains environments (base URLs), actions
 (individual endpoint specs), auth schemes, webhooks, variables, and constants.
 
@@ -1808,6 +1833,21 @@ Manage environments (base URLs per stage):
   ductape_execute("app.environments.create", [app_tag, { slug, env_name, base_url }])
   ductape_execute("app.environments.list",   [app_tag])
 
+Discover apps in a product and their actions (ALWAYS do this before writing any ctx.api.run call):
+  Step 1 — list apps connected to the product:
+    ductape_cli("products get --tag <product_tag> --json")    → full product document; check apps[]
+    ductape_cli("products apps list --product <product_id> --json")  → apps[] with access_tag, envs
+  Step 2 — list actions in an app:
+    ductape_execute("actions.list", [app_tag])                → returns all action tags + names
+  Step 3 — fetch the input schema for an action:
+    ductape_execute("actions.fetch", [app_tag, action_tag])
+      → returns { body: {fieldName: {type, required}}, params: {}, query: {}, headers: {} }
+    OR: call ductape_generate_payload (operation_family="action", method="run",
+          targets={app: "app_tag", action: "action_tag"}) to get the exact resolved payload shape
+  Step 4 — call ductape_schema({ module: "app" }) if you need the JSON schema for creating/updating
+           app resources (not for runtime input — use actions.fetch or ductape_generate_payload for that)
+  NEVER assume action input field names. Always fetch the action definition first.
+
 Manage actions (individual API endpoints):
   ductape_execute("actions.create", [app_tag, {
     tag, name, resource: "/users/{id}", method: "GET"|"POST"|"PUT"|"PATCH"|"DELETE",
@@ -1821,6 +1861,16 @@ Manage actions (individual API endpoints):
   ductape_execute("actions.list",   [app_tag])
   ductape_execute("actions.fetch",  [app_tag, action_tag])
 
+Action input — flat input format:
+  Fields are resolved to the correct location (body/params/query/headers) by matching the action schema.
+  For ambiguous keys, use explicit prefixes:
+    input: { amount: 1000 }                       → auto-resolved (body.amount if body field exists)
+    input: { "body:amount": 1000 }                → explicit body
+    input: { "params:id": "user_123" }            → route parameter
+    input: { "query:limit": 10 }                  → query string
+    input: { "headers:X-Idempotency-Key": "..." } → request header
+  Always use ductape_generate_payload or actions.fetch to know the exact field names — never guess.
+
 Run an action at runtime:
   → CALL ductape_generate_payload FIRST (operation_family="action", method="run", targets={app, action})
   ductape_execute("actions.run", [{ product, env, app, action, input: { "body:field": value } }])
@@ -1840,10 +1890,21 @@ Variables (per-env mutable values) and Constants (fixed values):
   ductape_execute("app.constants.create", [app_tag, { key, value }])
 
 Connecting an app to a product (after creation):
-  NOTE: All product.* module methods require the access key and CANNOT use ductape_execute (publishable key only).
-  Use ductape_cli for all product-level operations:
-  ductape_cli("products apps list --product <product_id> --json")
-  ductape_cli("products get --tag <product_tag> --json")    ← includes apps[], databases[], features[] etc.
+  NOTE: All product.* module methods require the access key and CANNOT use ductape_execute.
+  Use ductape_cli for all product-level operations.
+
+  FULL FLOW to make an app callable from a product:
+  1. Create the app:     ductape_cli("apps create --name \\"Service Name\\" --description \\"...\\"")
+  2. Add environments:   ductape_execute("app.environments.create", [app_tag, { slug: "prd", env_name: "Production", base_url: "https://api.example.com" }])
+  3. Configure auth:     ductape_execute("auths.create", [app_tag, { tag, name, setup_type: "apikey"|"bearer"|"basic"|"oauth2", expiry, period }])
+  4. Define actions:     ductape_execute("actions.create", [app_tag, { tag, name, resource, method, body?, params?, query?, headers?, response? }])
+                         OR import: ductape_cli("apps import <file.json> -t postman|openapi")
+  5. Connect to product: Requires the product_id (from ductape_cli("products get --tag <tag> --json")).
+                         There is no CLI command for this step — the SDK product.apps.add method requires
+                         an access key which only the backend can provide. Use ductape_execute via an
+                         admin-authenticated context, or connect via the Workbench UI.
+  6. Verify:             ductape_cli("products get --tag <product_tag> --json") → check apps[] contains the app
+                         ductape_execute("actions.list", [app_tag]) → verify actions are registered
 `.trim(),
 
   products: `
@@ -2158,7 +2219,13 @@ STEP 2 — INVENTORY existing Ductape components
   (product.* requires the access key — never use ductape_execute for product reads, it will return 403)
   Note what already exists:
     - databases[]      → available for ctx.database.insert/query/update/delete steps
-    - apps[]           → available for ctx.api.run steps (check app.events[] for event tags)
+    - apps[]           → available for ctx.api.run steps ONLY if the App is fully registered:
+                         (a) App record exists, (b) environments defined with base URLs,
+                         (c) auth scheme configured, (d) action endpoints defined,
+                         (e) connected to the product via product.apps.add.
+                         Check app.events[] for action tags. If no app exists for a service the
+                         feature needs to call, flag it as "App to create" in the plan — do NOT
+                         assume ctx.api can call any URL or schedule any job without a registered App.
     - notifications[]  → available for ctx.notification.email/sms/push steps
     - storage[]        → available for ctx.storage.upload/download steps
     - messageBrokers[] → available for ctx.events.produce steps
@@ -2166,10 +2233,14 @@ STEP 2 — INVENTORY existing Ductape components
     - features[]       → can be called as child features via ctx.feature.execute()
     - caches[], sessions[]
   Do NOT assume a component or event tag exists — verify from the product before using it.
+  Do NOT treat ctx.api as a generic HTTP call or job scheduler. It requires a registered App.
 
 STEP 3 — PLAN each step
   For every logical step:
-    a. Identify which existing component handles it, or flag it as needing creation
+    a. Identify which existing component handles it, or flag it as needing creation.
+       If a step calls an external service, it MUST go through a registered Ductape App.
+       If no App for that service exists in the product → mark it "App to create: <service name>".
+       DO NOT plan a raw HTTP call, a direct dispatch to a URL, or any workaround in place of a missing App.
     b. Note how inputs flow: ctx.input fields, or return values from earlier steps (plain JS variables — no special notation needed)
     c. Decide if a rollback handler is needed (e.g. charge → refund on later failure)
     d. Decide allow_fail: true for non-critical steps (email, analytics, audit logs)
@@ -2213,7 +2284,8 @@ STEP 7 — HANDLE conditionals, loops, and branching (when applicable)
     → Only the scenario whose input matches runs at execution time
 
 STEP 8 — SET rollbacks for reversible steps
-  Any step that allocates a resource should undo it if a later step fails:
+  Any step that allocates a resource should undo it if a later step fails.
+  ctx.api.run requires a pre-registered Ductape App — 'stripe' below is the tag of a registered App:
     const charge = await ctx.step(
       'charge',
       async () => ctx.api.run({ app: 'stripe', event: 'create-charge', input: { amount: ctx.input.amount } }),
@@ -2273,6 +2345,52 @@ Signals and queries (for long-running features):
 
 Rollback strategies: reverse_all | reverse_critical | compensate | none
 Feature statuses: pending | running | completed | failed | rolled_back | rolling_back | paused
+
+━━━ FEATURE RECORDING SEMANTICS ━━━
+
+When you call features.define({ handler }), the handler runs TWICE:
+
+  1. RECORDING PHASE (at define time) — handler is called with a RecordingContext.
+     All ctx.step() calls return lightweight proxy objects, not real data.
+     This phase captures the step graph: which steps exist, their types, tags, and declared
+     inputs/outputs. No real API calls, DB queries, or side effects occur.
+     Arbitrary JS code OUTSIDE ctx.step() ALSO runs during recording — with proxy values.
+     For loops: use recordInput so the handler sees sample data and all iterations are recorded.
+     For branches: use branchOverrides so each path is captured.
+
+  2. EXECUTION PHASE (at runtime) — handler is called with a real ExecutionContext.
+     ctx.step() actually executes. All real Ductape component calls happen.
+     Arbitrary JS logic (math, string ops, conditionals on step results) runs for real.
+
+  Implication: put all meaningful business logic INSIDE ctx.step() handlers, not in the
+  outer handler body. Code in the outer body runs during recording with proxy values and
+  may behave unexpectedly (e.g. typeof proxy === 'object' is true but .someField is a proxy).
+
+  Features do NOT execute arbitrary NestJS or server code directly. A feature handler can only
+  call Ductape component primitives (ctx.api, ctx.database, ctx.notification, etc.) as steps.
+  To invoke internal application business logic, produce a broker event from a feature step
+  (ctx.messaging.produce) and consume it in your NestJS service — that is the correct pattern.
+
+━━━ ORCHESTRATION DECISION RULE ━━━
+
+  One component operation at a future time:
+    → use that component's own dispatch method
+      e.g. ductape.events.dispatch({ ..., schedule: { start_at: ... } })
+      e.g. ductape.api.dispatch({ ..., schedule: { start_at: ... } })
+      e.g. ductape.database.dispatch({ ..., schedule: { start_at: ... } })
+
+  Several durable Ductape component operations in sequence (with rollback / retry / state):
+    → define a Feature, then features.dispatch to schedule it
+
+  Invoke internal application business logic (your own NestJS/backend service code):
+    → produce a broker event (ctx.messaging.produce or ductape.events.produce)
+    → consume it in your NestJS service with events.consume in onModuleInit
+    → your service method runs with full access to DI, DB transactions, etc.
+    Do NOT create an App Action just to call your own service over HTTP.
+
+  Invoke an external/public HTTP service:
+    → create a Ductape App (register base URL, auth, action endpoints) then use ctx.api.run
+    → requires the App to be fully registered and connected to the product first
 `.trim(),
 
   events: `
@@ -2371,81 +2489,173 @@ Import (register an EXISTING cloud resource):
     redis:    { host: "...", port: 6379, password?: "..." }
     nats:     { servers: ["nats://host:4222"], token?: "...", user?: "...", pass?: "...", tls?: true }
 
-━━━ STEP 2: ADD TOPIC DEFINITIONS ━━━
-
-  For most providers (GCP Pub/Sub, Kafka, RabbitMQ, Redis, NATS, Azure Service Bus):
-  SKIP this step. Topics are auto-created when you create a producer (step 3).
-  You do NOT need to call messageBrokers.topics.create before creating a producer.
+━━━ STEP 2: DEFINE TOPICS ━━━
 
-  Only call topics.create explicitly when:
-    - Using AWS SQS (must supply queueUrls per env — auto-creation cannot know the queue URL)
-    - You want to pre-set sample data or idempotency config on the topic
+  Topics MUST be defined before any consumer can subscribe to them.
+  Producing to a topic also calls ensureTopicRegistered in the background — but DO NOT rely on
+  auto-registration for consume paths. Always create topics explicitly.
 
-  If you do need it (SQS or explicit config):
   ductape_execute("messageBrokers.topics.create", [product_tag, {
-    tag: "player-joined",
-    name: "Player Joined",
-    broker: "notifications-broker",   // broker component tag
+    tag: "order-created",             // topic tag (just the topic part, NOT "broker:topic")
+    name: "Order Created",
+    broker: "order-events",           // broker component tag
     description?: string,
-    sample: { playerId: "string", username: "string" },  // example message shape
-    idempotent?: boolean,
-    // SQS only — map per-env queue URLs (each topic can be a different queue):
+    sample: { orderId: "string", total: 0 },   // documents expected message shape
+    idempotent?: boolean,             // if true, Ductape deduplicates by idempotency_key
+    // AWS SQS only — must supply per-env queue URL:
     queueUrls?: [
-      { env_slug: "snd", url: "https://sqs.us-east-1.amazonaws.com/123456/my-product-player-joined-snd" },
-      { env_slug: "prd", url: "https://sqs.us-east-1.amazonaws.com/123456/my-product-player-joined-prd" }
+      { env_slug: "snd", url: "https://sqs.us-east-1.amazonaws.com/123/queue-snd" },
+      { env_slug: "prd", url: "https://sqs.us-east-1.amazonaws.com/123/queue-prd" }
     ]
   }])
 
-  List topics on a broker:
+  List / fetch topics:
     ductape_execute("messageBrokers.topics.list", [product_tag, "broker-tag"])
+    ductape_execute("messageBrokers.fetch", [product_tag, "broker-tag"])  → includes topics[]
 
-━━━ STEP 3: PRODUCE AND CONSUME — WRITTEN IN APPLICATION CODE ━━━
+━━━ STEP 3: PRODUCE — WRITTEN IN APPLICATION CODE ━━━
 
-  There is NO admin command or file to declare producers/consumers.
-  There is NO "create producer" step before writing code.
-  Producers and consumers are registered automatically by the SDK the first time your code calls
-  produce/consume — you do not pre-declare them.
+  There is NO admin command to declare a producer. Producers are auto-registered by the SDK on
+  the first produce call — you do not pre-declare them.
+  Do NOT call ductape_generate_payload for messaging. The producer owns the schema.
+  Infer the message shape from context, present it to the user for approval, then implement.
 
-  The entire producer/consumer contract is the code you write in your controllers or services:
-
-  Produce (publish a message) — write in your service/controller:
-    Do NOT call ductape_generate_payload for messaging. Events have no pre-existing backend schema
-    to discover — the producer defines the schema. Instead, infer the message shape from context
-    (event name, existing data models, user input), present it to the user for approval, then implement.
+  GENERAL BACKEND (TypeScript/Node.js — not NestJS):
+    import Ductape from '@ductape/sdk';
+    const ductape = new Ductape({ accessKey: 'your-access-key' });
     await ductape.events.produce({
       product: "my-product",
       env: "prd",
-      event: "broker_tag:topic_tag",   // "broker_tag:topic_tag" — always colon-separated
-      message: { key: value },         // shape inferred from context, approved by user
+      event: "broker-tag:topic-tag",   // always colon-separated
+      message: { key: value },
+      session?: "session-tag:jwt",     // optional — traces message to a user session
+    });
+    // Idempotent publish (deduplicates — prevents double-processing on retries):
+    await ductape.events.publishIdempotent({
+      product, env, event, message,
+      idempotencyKey: "order-123-charge",  // stable key unique to this logical operation
+      idempotencyTtl?: 86400,             // seconds; default 86400 (24h)
+    });
+
+  NESTJS — method decorator:
+    import { Messaging } from '@ductape/nestjs';
+    @Injectable() export class OrdersService {
+      @Messaging.Produce({ event: 'order-events:order-created' })
+      emitOrderCreated(payload: { orderId: string; total: number }) { return payload; }
+
+      // Scheduled dispatch — fire-and-forget with optional schedule:
+      @Messaging.Dispatch({ broker: 'order-events', event: 'order-events:order-created',
+                            schedule?: { start_at?, cron?, every?, limit?, tz? } })
+      scheduleOrderNotification(payload: Record<string, unknown>) { return payload; }
+    }
+
+  CLIENT-SIDE (browser — publishable key):
+    Clients CAN produce messages using a publishable key + session token.
+    Only produce to topics whose schema is safe for client authorship.
+    NEVER allow clients to produce to topics that trigger privileged server-side operations
+    (payments, admin actions, state mutations) — those must go through a backend endpoint first.
+    import Ductape from '@ductape/sdk';
+    const ductape = new Ductape({ publishableKey: 'pk_...', env: 'prd', product: 'my-product' });
+    await ductape.events.produce({
+      event: "user-events:user-action",
+      message: { action: "button-click", screen: "dashboard" },
+      session: "user-session:eyJ...",  // REQUIRED for client-side produce
     });
-    Idempotent publish (deduplicates by key):
-      await ductape.events.publishIdempotent({ product, env, event, message, idempotencyKey, idempotencyTtl? })
 
-  Consume (subscribe) — write in your service/controller:
+  SCHEDULED DISPATCH (background job):
+    ductape_execute("messageBrokers.dispatch", [{
+      product, env,
+      broker: "order-events",                // broker tag
+      event: "order-events:reminder-due",    // "broker:topic"
+      input: { message: { orderId: "123" } },
+      retries?: 3,
+      session?: "session-tag:jwt",
+      schedule?: {
+        start_at?: 1735689600000,    // Unix ms or ISO string
+        cron?: "0 9 * * *",          // recurring cron
+        every?: 86400000,            // recurring interval ms
+        limit?: 10,                  // max repetitions
+        endDate?: "2026-12-31",
+        tz?: "America/New_York",
+      },
+    }])
+    Returns: { job_id, status: "scheduled"|"queued", scheduled_at, recurring, next_run_at? }
+
+━━━ STEP 4: CONSUME — WRITTEN IN APPLICATION CODE ━━━
+
+  Consumers are auto-registered by the SDK on first consume call.
+  Consumer registration options (all optional — used for tracking in Workbench):
+    consumer?: { tag?: string, name?: string, description?: string }
+  If tag is omitted, Ductape generates one: "consumer-<brokerTag>-<topicTag>".
+
+  ACK BEHAVIOR (automatic):
+    - Callback returns successfully → message is acknowledged (ack)
+    - Callback throws → message is tracked as failed; broker nacks/retries per provider behavior
+    - After max retries → message moves to dead-letter queue (DLQ)
+    There is no manual ack API. Acknowledgement is implicit from callback outcome.
+
+  CONSUMER GROUPS (Kafka-specific):
+    Consumer groups are set in the broker's envs[].config.groupId (at broker registration time).
+    All service instances sharing the same groupId form a consumer group and share partition load.
+    To configure: set groupId in the kafka config when creating/updating the broker.
+
+  CONCURRENCY:
+    Ductape has no per-consumer concurrency setting. Concurrency is determined by:
+      - Number of running service instances (horizontal scale)
+      - Broker-level partition count (Kafka) or visibility timeout (SQS)
+    Run multiple instances of your service to scale consumption.
+
+  GENERAL BACKEND (TypeScript/Node.js — not NestJS):
+    Start consuming in your module init or service startup:
     await ductape.events.consume({
       product: "my-product",
       env: "prd",
-      event: "broker_tag:topic_tag",
-      callback: async (message) => { /* handle message */ },
+      event: "order-events:order-created",
+      callback: async (message) => {
+        // All real processing logic goes here.
+        // Throw to nack. Return to ack.
+        await processOrder(message as { orderId: string; total: number });
+      },
+      consumer?: { tag: "order-processor", name: "Order Processor" },
     });
-    Callback errors are re-thrown so the broker can nack/retry.
 
-  Background dispatch with scheduling — write in your service/controller:
-    await ductape.events.dispatch({ product, env, broker, event, input: { message },
-      schedule?: { start_at?, cron?, every?, limit?, endDate?, tz? } })
+  NESTJS — use SDK in onModuleInit (no @Messaging.Consume decorator exists yet):
+    @Injectable()
+    export class OrderConsumerService implements OnModuleInit {
+      constructor(private readonly ductape: Ductape) {}
+      async onModuleInit() {
+        await this.ductape.events.consume({
+          product: "my-product",
+          env: process.env.DUCTAPE_ENV || 'prd',
+          event: "order-events:order-created",
+          callback: async (message) => { await this.handle(message); },
+          consumer: { tag: "order-consumer", name: "Order Consumer" },
+        });
+      }
+      private async handle(message: unknown) { /* business logic */ }
+    }
 
-  For the four standard producer declarations (match-state, match-report, projection-updated,
-  notification), write these produce calls in the relevant application service methods — there is
-  no separate configuration file or CLI step. The SDK creates the producer metadata on first call.
+  CLIENT-SIDE: Clients CANNOT consume. Event consumption is always server-side only.
+  This is the key distinction between server topics (produce + consume) and client-observable
+  topics (produce from client, consume on server). Never set up a consumer in browser code.
 
-Message payload is AES-encrypted before the tracking API call — tracking never sees plaintext.
+  DEAD-LETTER QUEUE (DLQ):
+    Messages whose callbacks consistently throw are automatically moved to the DLQ.
+    Query:     ductape_execute("messageBrokers.messages.getDeadLetters",
+                 [{ product, env, brokerTag, topicTag?, consumerTag?, limit? }])
+    Reprocess: ductape_execute("messageBrokers.reprocessDLQ",
+                 [{ product, env, brokerTag, topicTag?, messageIds?, limit? }])
+    Replay:    ductape_execute("messageBrokers.replayEvent",
+                 [{ product, env, eventId, force? }])
 
 ━━━ OBSERVABILITY ━━━
 
   messageBrokers.messages.query        [{ product, env, brokerTag, topicTag?, status?, page?, limit? }]
   messageBrokers.messages.getStats     [{ product, env, brokerTag }]
   messageBrokers.messages.getDashboard [{ product, env, brokerTag }]
-  messageBrokers.messages.getDeadLetters [{ product, env, brokerTag, topicTag?, limit? }]
+  messageBrokers.messages.getDeadLetters [{ product, env, brokerTag, topicTag?, consumerTag?, limit? }]
+  messageBrokers.messages.getProducers [{ product, env, brokerTag, topicTag?, page?, limit? }]
+  messageBrokers.messages.getConsumers [{ product, env, brokerTag, topicTag?, page?, limit? }]
   messageBrokers.replayEvent           [{ product, env, eventId, force? }]
   messageBrokers.reprocessDLQ         [{ product, env, brokerTag, topicTag?, messageIds?, limit? }]
   messageBrokers.checkIdempotency      [{ product, env, brokerTag, idempotency_key }]