vybekiit 0.7.0 → 0.7.2

package/dist/global-skills/aws-serverless/references/orchestration.md

@@ -1,18 +1,20 @@
 # Orchestration Reference
 
-Step Functions and EventBridge decision matrices, error semantics, and limits. Assumes you can write Amazon States Language (Saga/Parallel/Map/Choice JSON) and EventBridge patterns — this file focuses on the choices and gotchas.
+AWS Step Functions and Amazon EventBridge patterns and configuration.
 
 ## Contents
 
-- [Standard vs Express](#standard-vs-express)
-- [State machine limits and patterns](#state-machine-limits-and-patterns)
+- [Step Functions Standard vs Express](#step-functions-standard-vs-express)
+- [State machine patterns](#state-machine-patterns)
 - [Error handling](#error-handling)
-- [EventBridge rules, pipes, and patterns](#eventbridge-rules-pipes-and-patterns)
-- [Step Functions vs Lambda durable functions](#step-functions-vs-lambda-durable-functions)
+- [EventBridge rules and patterns](#eventbridge-rules-and-patterns)
+- [EventBridge Pipes](#eventbridge-pipes)
 
 ---
 
-## Standard vs Express
+## Step Functions Standard vs Express
+
+### Decision Matrix
 
 | Dimension | Standard | Express |
 |---|---|---|
@@ -23,108 +25,425 @@ Step Functions and EventBridge decision matrices, error semantics, and limits. A
 | `.waitForTaskToken` | Supported | **Not supported** |
 | Distributed Map | Supported | **Not supported** |
 | Activities | Supported | **Not supported** |
-| Idempotency | Automatic (execution name unique 90 days) | Not managed |
+| Idempotency | Automatic (execution name unique for 90 days) | Not managed |
+
+Express sub-types:
 
-Express sub-types: **Async** (fire-and-forget, results via CloudWatch Logs); **Sync** (blocks until completion, invokable from API Gateway/Lambda/`StartSyncExecution`, 5-min max).
+- **Asynchronous**: Fire-and-forget. Results via CloudWatch Logs.
+- **Synchronous**: Blocks until completion. Invokable from API Gateway, Lambda, or `StartSyncExecution`. 5-min max.
 
-| Use case | Type |
+| Use Case | Type |
 |---|---|
-| Long-running, `.sync`/callback, non-idempotent (payments) | Standard |
+| Long-running orchestration, `.sync`/callback patterns | Standard |
+| Non-idempotent operations (payments, exactly-once) | Standard |
 | Distributed Map (large-scale parallel) | Standard |
 | High-volume event processing (IoT, streaming) | Express |
 | API-backed synchronous microservice orchestration | Synchronous Express |
 
 ---
 
-## State machine limits and patterns
+## State Machine Patterns
+
+### Saga Pattern (Compensating Transactions)
+
+Each step has a corresponding undo step invoked on failure via `Catch`. Compensations chain in reverse.
+
+```json
+{
+  "Comment": "Saga pattern — book travel",
+  "StartAt": "BookHotel",
+  "States": {
+    "BookHotel": {
+      "Type": "Task",
+      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:book-hotel",
+      "TimeoutSeconds": 30,
+      "Catch": [{
+        "ErrorEquals": ["States.ALL"],
+        "ResultPath": "$.BookHotelError",
+        "Next": "NotifyFailure"
+      }],
+      "Next": "BookFlight"
+    },
+    "BookFlight": {
+      "Type": "Task",
+      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:book-flight",
+      "TimeoutSeconds": 30,
+      "Catch": [{
+        "ErrorEquals": ["States.ALL"],
+        "ResultPath": "$.BookFlightError",
+        "Next": "CancelHotel"
+      }],
+      "Next": "BookCar"
+    },
+    "BookCar": {
+      "Type": "Task",
+      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:book-car",
+      "TimeoutSeconds": 30,
+      "Catch": [{
+        "ErrorEquals": ["States.ALL"],
+        "ResultPath": "$.BookCarError",
+        "Next": "CancelFlight"
+      }],
+      "Next": "ConfirmBooking"
+    },
+    "CancelFlight": {
+      "Type": "Task",
+      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:cancel-flight",
+      "Next": "CancelHotel"
+    },
+    "CancelHotel": {
+      "Type": "Task",
+      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:cancel-hotel",
+      "Next": "NotifyFailure"
+    },
+    "NotifyFailure": {
+      "Type": "Fail",
+      "Error": "SagaFailed",
+      "Cause": "One or more bookings failed; compensations executed"
+    },
+    "ConfirmBooking": { "Type": "Succeed" }
+  }
+}
+```
+
+### Parallel State
+
+Executes branches concurrently. **Output is an array** with one element per branch. All branches must succeed or the entire Parallel state fails. Supports `Retry` and `Catch`.
+
+```json
+{
+  "Type": "Parallel",
+  "Branches": [
+    {
+      "StartAt": "ProcessImages",
+      "States": {
+        "ProcessImages": {
+          "Type": "Task",
+          "Resource": "arn:aws:lambda:us-east-1:123456789012:function:process-images",
+          "End": true
+        }
+      }
+    },
+    {
+      "StartAt": "ProcessMetadata",
+      "States": {
+        "ProcessMetadata": {
+          "Type": "Task",
+          "Resource": "arn:aws:lambda:us-east-1:123456789012:function:process-metadata",
+          "End": true
+        }
+      }
+    }
+  ],
+  "Next": "AggregateResults"
+}
+```
+
+### Map State
+
+**Inline Map**: Iterates over an array in the same execution. Max **40 concurrent** iterations.
+
+```json
+{
+  "Type": "Map",
+  "ItemsPath": "$.orders",
+  "MaxConcurrency": 10,
+  "ItemProcessor": {
+    "ProcessorConfig": { "Mode": "INLINE" },
+    "StartAt": "ProcessOrder",
+    "States": {
+      "ProcessOrder": {
+        "Type": "Task",
+        "Resource": "arn:aws:lambda:us-east-1:123456789012:function:process-order",
+        "End": true
+      }
+    }
+  },
+  "Next": "Done"
+}
+```
+
+**Distributed Map**: Up to **10,000 parallel child executions**. Reads from S3 (JSON, CSV, S3 inventory). Supports `ItemBatcher`, `ItemReader`, `ResultWriter`. **Standard workflows only.**
 
-- **Payload limit: 256 KiB between states** — store large data in S3, pass S3 keys.
-- **Inline Map:** max **40 concurrent** iterations, same execution.
-- **Distributed Map:** up to **10,000 parallel child executions**, reads from S3 (JSON/CSV/inventory), supports `ItemBatcher`/`ItemReader`/`ResultWriter`. **Standard workflows only.**
-- **25,000 execution-history entries** (Standard) — split long workflows into child executions.
-- **Parallel state output is an array** (one element per branch); all branches must succeed or the state fails.
-- **Choice state:** always include a `Default` branch.
+### Choice State
 
-Common patterns (write the ASL directly): **Saga** (each step has a compensating undo via `Catch`, chained in reverse); **Parallel** (concurrent branches); **Map** (iterate an array); **Agentic AI loop** (`bedrock:invokeModel` → Choice on `stop_reason = 'tool_use'` → execute tool → loop). Prefer **direct SDK integrations** (200+ services) over Lambda intermediaries to cut latency, and prefer **JSONata** for inline transforms over a Lambda task.
+Routes execution based on input conditions. Always include a `Default` branch.
+
+Comparison operators: `StringEquals`, `StringMatches`, `NumericGreaterThan`, `NumericLessThanEquals`, `BooleanEquals`, `IsPresent`, `IsNull`, `TimestampEquals`, and `Path` variants.
+
+```json
+{
+  "Type": "Choice",
+  "Choices": [
+    { "Variable": "$.orderTotal", "NumericGreaterThan": 1000, "Next": "HighValueOrder" },
+    { "Variable": "$.isPrime", "BooleanEquals": true, "Next": "PrimeProcessing" }
+  ],
+  "Default": "StandardProcessing"
+}
+```
+
+### Agentic AI Loop Pattern (Tool Use)
+
+Model outputs a structured response indicating a tool call or final answer. Choice state routes accordingly. Tool results feed back in a loop.
+
+```json
+{
+  "Comment": "Agentic AI loop with tool use",
+  "QueryLanguage": "JSONata",
+  "StartAt": "InvokeModel",
+  "States": {
+    "InvokeModel": {
+      "Type": "Task",
+      "Resource": "arn:aws:states:::bedrock:invokeModel",
+      "Arguments": {
+        "ModelId": "global.anthropic.claude-sonnet-4-6",
+        "Body": {
+          "anthropic_version": "bedrock-2023-05-31",
+          "max_tokens": 4096,
+          "messages": "{% $states.input.messages %}"
+        },
+        "ContentType": "application/json",
+        "Accept": "application/json"
+      },
+      "Next": "CheckAction"
+    },
+    "CheckAction": {
+      "Type": "Choice",
+      "Choices": [
+        { "Condition": "{% $states.input.Body.stop_reason = 'tool_use' %}", "Next": "ExecuteTool" }
+      ],
+      "Default": "ReturnResult"
+    },
+    "ExecuteTool": {
+      "Type": "Task",
+      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:execute-tool",
+      "TimeoutSeconds": 60,
+      "Next": "InvokeModel"
+    },
+    "ReturnResult": { "Type": "Succeed" }
+  }
+}
+```
 
 ---
 
-## Error handling
+## Error Handling
 
-### Built-in error names
+### Built-in Error Names
 
-| Error Name | Retriable? | Notes |
-|---|:---:|---|
-| `States.ALL` | Yes | Wildcard — but does **NOT** match the two terminal errors below |
-| `States.TaskFailed` | Yes | Wildcard for task errors (except `States.Timeout`) |
-| `States.Timeout` / `States.HeartbeatTimeout` | Yes | Exceeded `TimeoutSeconds` / missed `HeartbeatSeconds` |
-| `States.Permissions` | Yes | Insufficient IAM privileges |
-| `States.DataLimitExceeded` | **No** | Payload > 256 KiB — **terminal** |
-| `States.Runtime` | **No** | Invalid JSONPath, null payload — **terminal** |
-| `States.ItemReaderFailed` / `States.ResultWriterFailed` | Yes | Map source/destination errors |
+| Error Name | Description | Retriable? |
+|---|---|---|
+| `States.ALL` | Wildcard — matches any error | Yes |
+| `States.TaskFailed` | Wildcard for task errors (except `States.Timeout`) | Yes |
+| `States.Timeout` | Task exceeded `TimeoutSeconds` or `HeartbeatSeconds` | Yes |
+| `States.HeartbeatTimeout` | No heartbeat within `HeartbeatSeconds` | Yes |
+| `States.Permissions` | Insufficient IAM privileges | Yes |
+| `States.DataLimitExceeded` | Payload exceeds 256 KiB — **terminal** | **No** |
+| `States.Runtime` | Invalid JSONPath, null payload — **terminal** | **No** |
+| `States.ItemReaderFailed` | Map couldn't read from ItemReader source | Yes |
+| `States.ResultWriterFailed` | Map couldn't write to ResultWriter destination | Yes |
+
+`States.ALL` does **not** match `States.DataLimitExceeded` or `States.Runtime`.
 
-> **`States.ALL` does NOT catch `States.DataLimitExceeded` or `States.Runtime`.** These are terminal and must be designed around, not retried.
+### Retry Configuration
 
-### Retry config
+Available on `Task`, `Parallel`, and `Map` states. Retries are attempted before catchers.
 
 ```json
 "Retry": [
-  { "ErrorEquals": ["States.Timeout"], "IntervalSeconds": 3, "MaxAttempts": 2,
-    "BackoffRate": 2.0, "MaxDelaySeconds": 30, "JitterStrategy": "FULL" },
-  { "ErrorEquals": ["Lambda.ServiceException", "Lambda.SdkClientException"],
-    "IntervalSeconds": 1, "MaxAttempts": 3, "BackoffRate": 2.0 },
-  { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 1, "MaxAttempts": 3, "BackoffRate": 2.0 }
+  {
+    "ErrorEquals": ["States.Timeout"],
+    "IntervalSeconds": 3,
+    "MaxAttempts": 2,
+    "BackoffRate": 2.0,
+    "MaxDelaySeconds": 30,
+    "JitterStrategy": "FULL"
+  },
+  {
+    "ErrorEquals": ["Lambda.ServiceException", "Lambda.SdkClientException"],
+    "IntervalSeconds": 1,
+    "MaxAttempts": 3,
+    "BackoffRate": 2.0
+  },
+  {
+    "ErrorEquals": ["States.ALL"],
+    "IntervalSeconds": 1,
+    "MaxAttempts": 3,
+    "BackoffRate": 2.0
+  }
+]
+```
+
+| Field | Default | Description |
+|---|---|---|
+| `ErrorEquals` | (required) | Array of error names to match |
+| `IntervalSeconds` | 1 | Initial wait before first retry |
+| `MaxAttempts` | 3 | Max retries; 0 = never retry |
+| `BackoffRate` | 2.0 | Multiplier for exponential backoff |
+| `MaxDelaySeconds` | — | Cap on computed backoff interval |
+| `JitterStrategy` | `"NONE"` | `"FULL"` randomizes wait between 0 and computed interval |
+
+Rules:
+
+- `States.ALL` must be **last** in the Retry array
+- Retries count as state transitions (billed in Standard workflows)
+- `States.Runtime` and `States.DataLimitExceeded` **cannot be retried**
+- Use `JitterStrategy: "FULL"` to prevent thundering herd
+
+### Catch (Fallback States)
+
+```json
+"Catch": [
+  {
+    "ErrorEquals": ["CustomBusinessError"],
+    "ResultPath": "$.error-info",
+    "Next": "HandleBusinessError"
+  },
+  {
+    "ErrorEquals": ["States.ALL"],
+    "ResultPath": "$.error-info",
+    "Next": "GenericErrorHandler"
+  }
 ]
 ```
 
-Defaults: `IntervalSeconds` 1, `MaxAttempts` 3 (0 = never), `BackoffRate` 2.0, `JitterStrategy` `"NONE"`.
+- `ResultPath` preserves original input alongside the error (e.g., `"$.error-info"`)
+- Without `ResultPath`, error output replaces entire input
+- Retries are attempted first; catchers apply only after retries are exhausted
+
+### Error handling best practices
+
+1. **Always set `TimeoutSeconds`** on every Task state
+2. **Always retry Lambda service exceptions**: `Lambda.ServiceException`, `Lambda.SdkClientException`
+3. **Use `HeartbeatSeconds`** for long-running tasks
+4. **Combine Retry + Catch**: Retry transient, Catch permanent
+5. **Use `JitterStrategy: "FULL"`** to prevent thundering herd
+6. **Listen for execution failures via EventBridge** for top-level failures
+
+---
+
+## EventBridge Rules and Patterns
+
+### Event Pattern Structure
+
+All specified fields must match (AND). Values within an array are OR'd.
+
+```json
+{
+  "source": ["aws.ec2"],
+  "detail-type": ["EC2 Instance State-change Notification"],
+  "detail": { "state": ["terminated", "stopped"] }
+}
+```
+
+### Advanced Pattern Operators
+
+| Operator | Syntax | Description |
+|---|---|---|
+| Exact match | `["value"]` | Field equals value |
+| Prefix | `[{"prefix": "prod-"}]` | Starts with string |
+| Suffix | `[{"suffix": ".json"}]` | Ends with string |
+| Anything-but | `[{"anything-but": ["val"]}]` | Not in list |
+| Numeric range | `[{"numeric": [">", 0, "<=", 100]}]` | Numeric comparison |
+| Exists | `[{"exists": true}]` | Field must be present |
+| Wildcard | `[{"wildcard": "prod-*-east"}]` | Glob-style matching |
+
+### EventBridge best practices
+
+1. **Dedicated event bus per application domain** — default bus for AWS service events only
+2. **Be precise with patterns** — broad patterns increase risk of infinite loops
+3. **One target per rule** — simplifies debugging and IAM permissions
+4. **Use DLQs on targets** — capture failed event deliveries
+5. **Use the EventBridge Sandbox** to test patterns before deploying
+
+### Step Functions Status Change Events
+
+Step Functions emits to the default bus automatically:
+
+```json
+{
+  "source": ["aws.states"],
+  "detail-type": ["Step Functions Execution Status Change"],
+  "detail": { "status": ["FAILED", "TIMED_OUT", "ABORTED"] }
+}
+```
+
+### Integration Patterns
+
+**SFN → EventBridge** (publish events from a workflow):
+
+```json
+{
+  "Type": "Task",
+  "QueryLanguage": "JSONata",
+  "Resource": "arn:aws:states:::events:putEvents",
+  "Arguments": {
+    "Entries": [{
+      "Detail": { "orderId": "{% $states.input.orderId %}", "status": "PROCESSED" },
+      "DetailType": "OrderProcessed",
+      "EventBusName": "my-app-bus",
+      "Source": "my-app.orders"
+    }]
+  },
+  "Next": "Done"
+}
+```
 
-Rules and best practices:
+**EventBridge → SFN**: Rule target is the state machine ARN. Event payload becomes execution input.
 
-- `States.ALL` must be **last** in the Retry array; retries are attempted **before** catchers.
-- Retries count as state transitions (billed in Standard).
-- Always set `TimeoutSeconds` on every Task; always retry `Lambda.ServiceException` / `Lambda.SdkClientException`.
-- Use `JitterStrategy: "FULL"` to prevent thundering herd; `HeartbeatSeconds` for long tasks.
-- `Catch` with `ResultPath: "$.error-info"` preserves the original input alongside the error (without it, error output replaces the input).
-- Listen for top-level execution failures via EventBridge (`source: aws.states`, `detail-type: Step Functions Execution Status Change`, `status: [FAILED, TIMED_OUT, ABORTED]`).
+**Fan-out**: Single event triggers multiple workflows via multiple rules on the same bus.
 
 ---
 
-## EventBridge rules, pipes, and patterns
+## EventBridge Pipes
 
-### Event patterns
+### Architecture
 
-All specified fields must match (AND); values within an array are OR'd. Operators: exact `["value"]`, `{"prefix"}`, `{"suffix"}`, `{"anything-but"}`, `{"numeric": [">", 0, "<=", 100]}`, `{"exists": true}`, `{"wildcard": "prod-*-east"}`.
+```
+Source → [Filter] → [Enrichment] → [Transform] → Target
+```
+
+Eliminates intermediary Lambda functions for point-to-point integrations.
 
-### Best practices
+### Supported Sources
+
+| Source | Notes |
+|---|---|
+| Amazon SQS | Standard and FIFO queues |
+| Amazon Kinesis Data Streams | Shard-level polling |
+| Amazon DynamoDB Streams | Change data capture |
+| Amazon MSK / Self-managed Kafka | Topic-level consumption |
+| Amazon MQ | ActiveMQ and RabbitMQ |
 
-1. **Dedicated event bus per application domain** — default bus for AWS service events only.
-2. **Be precise with patterns** — broad patterns risk infinite loops.
-3. **One target per rule** — simplifies debugging and IAM.
-4. **DLQs on all targets.**
-5. Use the EventBridge Sandbox to test patterns before deploying.
+### Enrichment Options
+
+Lambda, API Gateway, EventBridge API Destinations, Step Functions (Synchronous Express).
+
+### Key Features
+
+- **Filtering**: Event patterns filter at the source — pay only for matched events
+- **Ordering**: Maintains event ordering within batches
+- **Built-in retry + DLQ**: Source-level retry with dead-letter queue support
 
 ### Pipes vs Rules
 
 | Dimension | Pipes | Rules |
 |---|---|---|
 | Topology | Point-to-point (1→1) | Fan-out (1→N) |
-| Flow | Source → Filter → Enrichment → Transform → Target | Event routing on a bus |
-| Sources | SQS, Kinesis, DynamoDB Streams, MSK, MQ | Any event on a bus |
-| Enrichment | Built-in (Lambda, API GW, API Destinations, Sync Express SFN) | Not built-in |
-| Use case | **Replace Lambda glue** for source→target | Event routing and distribution |
-
-Pipes filtering happens **at the source** — you pay only for matched events — with built-in retry + DLQ.
+| Sources | SQS, Kinesis, DDB Streams, MSK, MQ | Any event on a bus |
+| Enrichment | Built-in | Not built-in |
+| Use case | Replace Lambda glue | Event routing and distribution |
 
 ---
 
-## Step Functions vs Lambda durable functions
+## Lambda durable functions vs Step Functions
 
-Lambda durable functions let you write reliable multi-step workflows as **plain code** (TS/Python/Java) with automatic checkpointing — the SDK persists each step and replays from the checkpoint on interruption, enabling executions up to 1 year with zero compute during waits. **For full guidance use the aws-lambda-durable-functions skill** (see SKILL.md routing).
+Lambda durable functions let you write reliable multi-step workflows as plain code (TypeScript, Python, Java) with automatic checkpointing — the SDK persists each step's result and replays from the checkpoint on interruption, enabling executions up to 1 year with zero compute during waits. Use the **aws-lambda-durable-functions** skill for full guidance.
 
 | Question | Lambda durable functions | Step Functions |
 |---|---|---|
-| Programming model | Standard code (TS/Python/Java) | Amazon States Language / visual designer |
-| AWS service integrations | Primarily Lambda | 200+ native integrations |
-| Who reads the workflow | Developers | Non-technical stakeholders too |
-| Best for | Distributed transactions, stateful logic, AI agent loops | Business process automation, multi-service orchestration |
+| Primary focus? | Application logic in Lambda | Orchestration across AWS services |
+| Programming model? | Standard code (TS/Python/Java) | Amazon States Language (ASL) or visual designer |
+| AWS service integrations? | Primarily Lambda | 200+ native integrations |
+| Who reads the workflow? | Developers | Non-technical stakeholders |
+| Best for? | Distributed transactions, stateful logic, AI agent loops | Business process automation, multi-service orchestration |