@aws/durable-execution-sdk-js-otel 0.1.1 → 1.1.0

package/README.md

@@ -1,276 +1,624 @@
 # AWS Durable Execution SDK - OpenTelemetry Plugin
 
-> **⚠️ Experimental Beta:** This plugin is currently in experimental beta. Functionality may change without notice between releases. It is not recommended for production workloads at this time.
+OpenTelemetry instrumentation for the AWS Durable Execution SDK. The package
+provides two plugins that emit Workflow, Invocation, durable operation, and
+operation-attempt spans.
 
-OpenTelemetry instrumentation plugin for AWS Durable Execution SDK. Emits distributed traces that correlate across multiple Lambda invocations of a single durable execution, producing deterministic span and trace IDs so that spans from different invocations are stitched into a single coherent trace.
+The whole durable execution shares **one trace**, anchored at the execution
+ancestor resolved at invocation start. The `Workflow` span and every per-invocation
+`Invocation` span join that trace, so a single trace ID spans all invocations of
+the execution.
 
-## Features
+They differ in where durable operation spans are parented:
 
-- **Deterministic Trace IDs**: All invocations of the same durable execution share a single trace, derived from the X-Ray trace header or execution ARN
-- **Span-per-Operation**: Each durable operation (step, wait, invoke) gets its own span with accurate timing
-- **Continuation Spans**: Operations completing in a different invocation are linked back to the original span
-- **Log Correlation**: Enrich application logs with trace ID and span ID for end-to-end observability
-- **Configurable Sampling**: Control trace volume via environment variable or plugin options
-- **Self-Contained Setup**: No manual TracerProvider configuration required
+| Plugin                 | Operation hierarchy                  | Cross-link                              |
+| ---------------------- | ------------------------------------ | --------------------------------------- |
+| `ExecutionOtelPlugin`  | `Workflow -> operation -> attempt`   | Operations and attempts link Invocation |
+| `InvocationOtelPlugin` | `Invocation -> operation -> attempt` | Operations and attempts link Workflow   |
+
+The plugins do not create or register an OpenTelemetry provider, exporter,
+sampler, resource, propagator, context manager, or library instrumentation.
+They use the global provider by default, or an application-owned provider
+created through `tracerProviderFactory`.
 
 ## Installation
 
 ```bash
-npm install @aws/durable-execution-sdk-js-otel
+npm install @aws/durable-execution-sdk-js-otel \
+            @aws/durable-execution-sdk-js \
+            @opentelemetry/api \
+            @opentelemetry/core \
+            @opentelemetry/sdk-trace-node
 ```
 
-## Quick Start using Xray/CloudWatch Tracing
+The package requires Node.js 22 or later. Exporters, span processors,
+propagators, resources, and library instrumentation are application or ADOT
+responsibilities.
 
-1. Add the ADOT Lambda Layer to your function and set `AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument`
-2. Enable X-Ray Active Tracing on the function
-3. Pass `OtelPlugin` to your handler's `plugins` array
-4. Add Xray Write Permissions
+## Quick Start
 
-### 1. ADOT Lambda Layer
+When an SDK provider is registered globally, no plugin configuration is
+required:
 
-This plugin requires the [AWS Distro for OpenTelemetry (ADOT) Lambda layer](https://aws-otel.github.io/docs/getting-started/lambda) to export traces from your Lambda function.
+```typescript
+import { withDurableExecution } from "@aws/durable-execution-sdk-js";
+import { ExecutionOtelPlugin } from "@aws/durable-execution-sdk-js-otel";
 
-The layer ARN follows the format:
+const plugin = new ExecutionOtelPlugin();
 
+export const handler = withDurableExecution(
+  async (event, context) => {
+    return context.step("process", async () => process(event));
+  },
+  { plugins: [plugin] },
+);
 ```
-arn:aws:lambda:<region>:<awsAccountId>:layer:AWSOpenTelemetryDistroJs:<version>
+
+Use `InvocationOtelPlugin` instead when operations should appear under each
+Lambda invocation rather than under the durable Workflow.
+
+## Provider Setup
+
+### Global Provider
+
+Omitting `tracerProviderFactory` uses `trace.getTracerProvider()`. This is the
+normal setup for the ADOT Lambda layer, OpenTelemetry zero-code
+instrumentation, or an application that registers its own provider globally.
+
+When using ADOT, activate its instrumentation wrapper:
+
+```text
+AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
 ```
 
-Refer to the [ADOT Lambda Layer ARNs](https://aws-otel.github.io/docs/getting-started/lambda#aws-lambda-layer-for-opentelemetry-arns) page for the latest version number and supported regions.
+The OpenTelemetry JavaScript API does not provide a public way to retrieve or
+replace a provider's configured ID generator. For a compatible SDK tracer, the
+plugin therefore:
 
-**AWS CLI:**
+1. reads the tracer's runtime `_idGenerator` field;
+2. replaces it with a guarded deterministic wrapper;
+3. delegates all ID generation outside plugin span creation to the original
+   generator.
 
-```bash
-aws lambda update-function-configuration \
-  --function-name your-function-name \
-  --layers "arn:aws:lambda:<region>:<<awsAccountId>>:layer:AWSOpenTelemetryDistroJs:<version>"
+This private-field integration is isolated behind runtime shape and assignment
+checks. It does not change IDs for unrelated spans, including spans created
+concurrently or with the same instrumentation scope.
+
+If the plugin is constructed before the SDK provider is globally registered,
+its initial tracer may be a proxy without `_idGenerator`. At each invocation
+start, the plugin re-resolves the global provider until a compatible SDK tracer
+is available. When installation still fails:
+
+- plugin telemetry and log enrichment are disabled for that invocation;
+- a warning is emitted;
+- provider resolution is retried on the next invocation.
+
+Providers or future SDK tracers that do not expose the compatible runtime field
+cannot use this global-provider path. Use `tracerProviderFactory` to install the
+generator through the supported provider constructor API.
+
+### Application-Owned Provider
+
+`tracerProviderFactory` receives a function that creates the plugin's
+deterministic ID wrapper. The factory is called during plugin construction.
+
+```typescript
+import { InvocationOtelPlugin } from "@aws/durable-execution-sdk-js-otel";
+import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
+import {
+  NodeTracerProvider,
+  SimpleSpanProcessor,
+} from "@opentelemetry/sdk-trace-node";
+
+const plugin = new InvocationOtelPlugin({
+  tracerProviderFactory: (createIdGenerator) => {
+    const provider = new NodeTracerProvider({
+      idGenerator: createIdGenerator(),
+      spanProcessors: [
+        new SimpleSpanProcessor(
+          new OTLPTraceExporter({
+            url: "http://localhost:4318/v1/traces",
+          }),
+        ),
+      ],
+    });
+
+    provider.register();
+    return provider;
+  },
+});
 ```
 
-You must also set the `AWS_LAMBDA_EXEC_WRAPPER` environment variable:
+`createIdGenerator()` delegates non-durable IDs to OpenTelemetry's standard
+random generator. To preserve another generator, pass it as the fallback:
 
-```bash
-aws lambda update-function-configuration \
-  --function-name your-function-name \
-  --environment "Variables={AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument}"
+```typescript
+idGenerator: createIdGenerator(applicationIdGenerator);
 ```
 
-> **Note:** Replace `<region>`, `<awsAccountId>` `<version>` with the latest layer information from the ADOT docs.
+The deterministic override is active only for the synchronous `startSpan()`
+call made by the plugin. All unrelated IDs use the fallback generator.
+
+The application owns the returned provider and all associated configuration:
+
+- exporters and span processors;
+- sampling and resources;
+- context management and propagation;
+- HTTP, AWS SDK, Lambda, and other library instrumentation;
+- global registration;
+- shutdown.
 
-**CloudFormation / SAM:**
+The provider does not have to be the global provider for the plugin to obtain a
+tracer from it. Register it globally when the application also needs its
+context manager, propagator, or library instrumentation.
 
-```yaml
-MyFunction:
-  Type: AWS::Serverless::Function
-  Properties:
-    Layers:
-      - !Sub arn:aws:lambda:${AWS::Region}:<awsAccountId>:layer:AWSOpenTelemetryDistroJs:<version>
-    Environment:
-      Variables:
-        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
+At the end of each enabled invocation, both plugins call `forceFlush()` when
+the provider exposes it. They never call `shutdown()`.
+
+## Dynamic Loading from a Lambda Layer
+
+The SDK can load either plugin without importing it in function code. Package
+this module and its OpenTelemetry peer dependencies in a Lambda layer under
+`nodejs/node_modules`, then configure one entry point:
+
+```text
+DURABLE_EXECUTION_PLUGINS=@aws/durable-execution-sdk-js-otel/otel-execution
 ```
 
-For raw CloudFormation (`AWS::Lambda::Function`):
-
-```yaml
-MyFunction:
-  Type: AWS::Lambda::Function
-  Properties:
-    Layers:
-      - !Sub arn:aws:lambda:${AWS::Region}:<awsAccountId>:layer:AWSOpenTelemetryDistroJs:<version>
-    Environment:
-      Variables:
-        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
+or:
+
+```text
+DURABLE_EXECUTION_PLUGINS=@aws/durable-execution-sdk-js-otel/otel-invocation
 ```
 
-**CDK:**
+Do not package `@aws/durable-execution-sdk-js` in the layer. The provider entry
+points use SDK types only, and the SDK peer dependency is optional so package
+installation does not add a second copy. At runtime, the plugin is loaded and
+driven by the SDK instance bundled with the function.
 
-```typescript
-import * as lambda from "aws-cdk-lib/aws-lambda";
+Dynamic providers construct plugins with default configuration, so they
+require a compatible globally registered SDK provider. Use code-based
+registration when `tracerProviderFactory` or other custom configuration is
+needed.
 
-const adotLayer = lambda.LayerVersion.fromLayerVersionArn(
-  this,
-  "AdotLayer",
-  `arn:aws:lambda:${this.region}:<awsAccountId>:layer:AWSOpenTelemetryDistroJs:<version>`,
-);
+## Choosing a Plugin
 
-const fn = new lambda.Function(this, "MyFunction", {
-  runtime: lambda.Runtime.NODEJS_22_X,
-  handler: "index.handler",
-  code: lambda.Code.fromAsset("lambda"),
-  layers: [adotLayer],
-  environment: {
-    AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-instrument",
-  },
-});
-```
+### `ExecutionOtelPlugin`
 
-> **Tip:** Pin the layer version to a specific number in production deployments to avoid unexpected behavior from automatic version changes.
+Use this plugin for a workflow-centered view. Operations and attempts are
+children of the `Workflow` span. Each also links to the `Invocation` span that
+observed it. The whole execution shares one trace.
 
-### 2. AWS X-Ray Active Tracing
+```text
+Execution trace
 
-Enable active tracing on your Lambda function so the `_X_AMZN_TRACE_ID` environment variable is populated at invocation time. The plugin uses this header to derive deterministic trace IDs that remain consistent across all invocations of the same durable execution.
+Execution ancestor
+├── Workflow
+│   └── Operation: fetch-data
+│       └── Attempt 1
+└── Invocation
 
-**AWS Console:** Lambda → Configuration → Monitoring and operations tools → Active tracing → Enable
+Links:
+Operation -> Invocation
+Attempt   -> Invocation
+```
 
-**AWS CLI:**
+The plugin makes Workflow the active span while durable handler code runs.
+Completed operation spans use durable operation start/end timestamps and
+deterministic span IDs. While an operation spans multiple invocations its
+identity is carried as a non-recording context; nothing is exported until it
+completes. The single recording operation span is then created and ended once,
+in the invocation where the operation terminates, under a **deterministic span
+ID on the execution trace**. Because there is one span per logical operation,
+no cross-invocation link is needed to stitch it together — unlike
+`InvocationOtelPlugin` below.
+
+### `InvocationOtelPlugin`
+
+Use this plugin for an invocation-centered view. Operations and attempts are
+children of the current `Invocation` span. Each links to the `Workflow` span.
+The whole execution shares one trace.
+
+```text
+Execution trace
+
+Execution ancestor
+├── Workflow
+└── Invocation
+    └── Operation: fetch-data
+        └── Attempt 1
+
+Links:
+Operation -> Workflow
+Attempt   -> Workflow
+```
 
-```bash
-aws lambda update-function-configuration \
-  --function-name your-function-name \
-  --tracing-config Mode=Active
+The plugin makes Invocation the active span while durable handler code runs.
+Open operation spans are ended at the invocation boundary and retain
+`durable.operation.status=STARTED`. When an operation completes in a later
+invocation, the plugin emits a continuation span in that invocation.
+
+Because the original span context is not checkpointed, replayed `STEP` and
+`CONTEXT` spans and cross-invocation continuation spans use new provider IDs.
+They correlate through two links: the real exported `Workflow` span, and the
+**initial logical operation span** — whose ID is deterministic on
+`(operationId, execution ARN)` and lives on the execution trace, so the segments
+of one logical operation stay stitched together across invocations. Replay-only
+`WAIT`, `INVOKE`, `CHAINED_INVOKE`, and `CALLBACK` operations are not emitted
+again.
+
+## Execution Trace and the Execution Ancestor
+
+Both plugins resolve one **execution ancestor** at invocation start — the common
+parent the `Workflow` and `Invocation` spans join — so the whole execution
+shares a single trace. The ancestor is chosen by precedence:
+
+1. a **complete propagated remote parent** (a valid `Root` trace ID and a valid
+   `Parent` span ID): used directly, whether or not `Sampled` is present;
+2. otherwise a **synthetic execution root** with a deterministic span ID derived
+   from the execution ARN.
+
+The canonical trace ID follows the same precedence: the propagated remote trace
+ID when valid, else one derived from the ARN and start time.
+
+A live ambient span (a `context.active()` span created by an auto-instrumentation
+layer) is deliberately **never** used as the execution ancestor. Its trace ID is
+not guaranteed stable across Lambda reinvocations, so anchoring the
+multi-invocation execution on it could change the execution trace ID on replay
+and break the cross-invocation continuation and replay links (which target a
+deterministic operation span ID on the canonical trace). Both anchors used above
+are stable: the durable backend keeps the propagated `Root` identical for every
+invocation, and the ARN-derived synthetic root is a pure hash of the ARN.
+
+### Valid Workflow parent shapes
+
+`ADOT` and community Node.js auto-instrumentation layers can create the ambient
+handler span before the durable plugin runs, on the trace the durable backend
+propagated. OpenTelemetry `SpanContext` has no ancestor pointer, so the plugin
+cannot walk from that local span to the durable backend span. The `Workflow`
+span therefore joins the canonical trace and parents onto whichever ancestor the
+precedence above resolves. When a complete remote parent is propagated, that is
+the ancestor:
+
+```text
+Propagated remote parent
+├── Workflow
+└── Invocation
 ```
 
-**CloudFormation / SAM:**
+When no complete remote parent can be constructed, the synthetic execution root
+anchors the trace and the `Workflow` span parents onto it:
 
-```yaml
-MyFunction:
-  Type: AWS::Lambda::Function
-  Properties:
-    TracingConfig:
-      Mode: Active
+```text
+Synthetic execution root
+└── Workflow
 ```
 
-**CDK:**
+The `Invocation` span may still nest under a same-trace ambient handler span —
+see [Invocation parent](#invocation-parent) — without changing the execution
+ancestor the `Workflow` span joins.
+
+### Workflow identity and lifecycle
+
+`Workflow` is an `INTERNAL` span that **joins the execution trace** by parenting
+onto the execution ancestor (its span ID is forced; the trace ID comes from the
+parent). Its span ID is reproducible across replays:
+
+- span ID: the first 16 hexadecimal characters of SHA-256 over
+  `workflow:<execution ARN>`.
+
+The synthetic execution root, when used, gets a distinct deterministic span ID
+from SHA-256 over `execution-root:<execution ARN>`, and the ARN-derived trace ID
+is the first 32 hexadecimal characters of SHA-256 over
+`<execution ARN>:<execution start timestamp in ISO 8601 format>` (falling back
+to the ARN alone when the timestamp is unavailable).
+
+The plugins carry the same `Workflow` span identity on each invocation as a
+non-recording span context, and create the single recording `Workflow` span only
+when the execution reaches `SUCCEEDED` or `FAILED`, backdated to the execution
+start. `PENDING` and `RETRYING` invocations create no recording `Workflow` span,
+so none is ever left unended, and exactly one `Workflow` root is exported for the
+durable execution while intermediate invocation spans export normally. The
+non-recording context carries the execution's sampling decision, so operations
+under it are sampled consistently with the eventual root. In-flight attempt spans
+are ended at invocation cleanup, so a non-terminal invocation leaves no recording
+span unended. Suspended operations differ by plugin: `InvocationOtelPlugin` ends
+the open operation span at the boundary and continues it later, while
+`ExecutionOtelPlugin` holds the operation identity as a non-recording context and
+exports one recording span only when the operation completes.
+
+When a chained parent and target execution share a propagated remote parent,
+both join that one trace, each keeping its own deterministic `Workflow` span ID.
+
+### Invocation parent
+
+The `Invocation` span parents onto the **same-trace ambient span** when one is
+active on the canonical trace (preserving the Lambda/X-Ray linkage), otherwise
+onto the execution ancestor so it stays on the execution trace. Provider
+ownership does not change this topology.
+
+### Sampling
+
+Sampling follows this precedence, highest first:
+
+1. **Explicit upstream decision** — an explicit `Sampled=1` / `Sampled=0` in a
+   valid propagated header is authoritative and preserved. It is applied only
+   when the extracted trace ID is itself valid, so an unusable trace never
+   carries its sampling bit into the derived execution trace.
+2. **Configured sampler** — when the header carries no usable decision, the
+   provider sampler decides. It is evaluated at its **root policy**
+   (`ROOT_CONTEXT`, no parent) with the real trace ID, span name, and attributes,
+   so a `ParentBasedSampler` applies its `root` sampler rather than inheriting an
+   unrelated ambient span's decision.
+3. **Default to sampled** — used only when no sampler decision can be obtained.
+
+The resolved execution decision is enforced for all SDK-created spans through a
+durable sampler wrapper around the provider sampler. This makes explicit backend
+sampling authoritative even for direct/custom samplers that ignore parent
+`traceFlags`. Unrelated application spans still use the original configured
+sampler. If the resolved tracer does not expose a compatible writable sampler,
+the plugin preserves normal tracing through the configured provider; in that
+case direct samplers may make their normal span-level decisions.
+
+### Context extractors
+
+The default `xRayContextExtractor` parses `Root`, `Parent`, and `Sampled`
+independently from `_X_AMZN_TRACE_ID`, rejecting an all-zero (invalid) `Root` or
+`Parent`. The durable backend keeps the X-Ray `Root` stable for every invocation
+of one execution, so its trace ID anchors the whole execution. The package also
+exports `w3cClientContextExtractor`, which reads W3C `traceparent` data from
+`context.clientContext.custom.traceparent`.
+
+A custom extractor can return:
 
 ```typescript
-new lambda.Function(this, "MyFunction", {
-  tracing: lambda.Tracing.ACTIVE,
-});
+type Sampling = "SAMPLED" | "NOT_SAMPLED" | "UNDECIDED";
+
+type ContextExtractor = (info: InvocationInfo) =>
+  | {
+      traceId: string;
+      parentSpanId?: string;
+      traceFlags?: number;
+      sampling?: Sampling;
+    }
+  | undefined;
 ```
 
-### 3. In your Lambda handler (index.js)
+A complete remote parent needs a valid `traceId` and a valid `parentSpanId`.
+`sampling` is the tri-state upstream decision; when omitted it is derived from
+`traceFlags` (sampled bit), and when neither is present the decision is
+`UNDECIDED` and the sampling rules above decide. A custom extractor must return
+the durable execution's own trace context: a valid trace ID anchors the
+execution across replays, so returning stale or per-invocation context (for
+example an ambient span that changes each invocation) would split one durable
+execution across traces.
+
+## Shared Configuration
+
+Both plugins accept `OtelPluginConfig`:
 
 ```typescript
-import { withDurableExecution } from "@aws/durable-execution-sdk-js";
-import { OtelPlugin } from "@aws/durable-execution-sdk-js-otel";
+interface OtelPluginConfig {
+  /** Creates an application-owned provider with a deterministic ID wrapper. */
+  tracerProviderFactory?: (
+    createIdGenerator: (fallbackIdGenerator?: IdGenerator) => IdGenerator,
+  ) => TracerProvider;
 
-export const handler = withDurableExecution(
-  async (event, context) => {
-    const result = await context.step("fetch-data", async () => {
-      return fetchData(event.id);
-    });
+  /** Extracts upstream trace context. Defaults to xRayContextExtractor. */
+  contextExtractor?: ContextExtractor;
 
-    await context.wait({ seconds: 5 });
+  /** Instrumentation scope name. */
+  instrumentationName?: string;
 
-    await context.step("process", async () => {
-      return process(result);
-    });
+  /** Custom Workflow span name. Defaults to "Workflow". */
+  workflowSpanName?: string;
 
-    return result;
-  },
-  { plugins: [new OtelPlugin()] },
-);
+  /** Add active OTel IDs to durable log records. Defaults to true. */
+  enrichLogger?: boolean;
+}
 ```
 
-That's it. The plugin handles TracerProvider setup, deterministic ID generation, and span lifecycle internally.
+Provider selection is implicit:
 
-### 4. Grant Permissions
+- omit `tracerProviderFactory` to use the global provider;
+- provide `tracerProviderFactory` to use the returned application-owned
+  provider.
 
-The function's execution role needs the `AWSXRayDaemonWriteAccess` managed policy (or equivalent permissions) if using X-Ray as the tracing backend.
+The default instrumentation scope name is
+`aws-durable-execution-sdk-js`.
 
-### AWS Console
+## Spans, Attributes, and Status
 
-See https://aws-otel.github.io/docs/getting-started/lambda#use-the-lambda-console.
+All plugin-created spans use `SpanKind.INTERNAL`.
 
-### Environment Variables for ADOT layer
+- **Workflow:** Named by `workflowSpanName`, which defaults to `Workflow`.
+  Carries `durable.execution.arn` and, when terminal,
+  `durable.execution.status`.
+- **Invocation:** Named `Invocation`. Carries `durable.execution.arn`,
+  `durable.invocation.first`, and `durable.invocation.status`.
+- **Operation:** Named from the configured operation name, or the operation
+  type when unnamed. Carries `durable.execution.arn`,
+  `durable.operation.id`, `durable.operation.type`, optional
+  `durable.operation.name`, and optional `durable.operation.subtype`.
+  `InvocationOtelPlugin` sets `durable.operation.status=STARTED` at span
+  creation; both plugins apply a supplied terminal operation status at
+  completion. `durable.attempt.number` is added to `STEP` and
+  `WAIT_FOR_CONDITION` operation spans when supplied.
+- **Attempt:** Named `<operation name or type> attempt <number>`. Carries the
+  operation identity attributes, `durable.attempt.number`, and terminal
+  `durable.attempt.outcome`. Attempt spans do not carry
+  `durable.operation.status`.
 
-| Variable                      | Description                                                                                   | Default           |
-| ----------------------------- | --------------------------------------------------------------------------------------------- | ----------------- |
-| `OTEL_EXPORTER_OTLP_ENDPOINT` | Endpoint for the OTLP exporter (e.g., `http://localhost:4318` for the ADOT collector sidecar) | Set by ADOT layer |
-| `AWS_LAMBDA_EXEC_WRAPPER`     | Set to `/opt/otel-instrument` for the ADOT layer to instrument your function                  | —                 |
-| `OTEL_TRACES_SAMPLER`         | Sampler to use (e.g., `traceidratio` for ratio-based sampling)                                | `always_on`       |
-| `OTEL_TRACES_SAMPLER_ARG`     | Argument for the sampler (e.g., `0.3` to sample 30% of traces)                                | —                 |
+For `ExecutionOtelPlugin` with an application-owned provider, Invocation also
+adds `cloud.resource_id` and `faas.max_memory` when the corresponding Lambda
+environment values are available.
 
-See the [ADOT sampling configuration](https://aws-otel.github.io/docs/getting-started/lambda#sampling-configuration) for more details.
+OpenTelemetry status mapping is:
 
-## Configuration
+| Span       | Durable result                                                     | OTel status              |
+| ---------- | ------------------------------------------------------------------ | ------------------------ |
+| Workflow   | `SUCCEEDED` / `FAILED`                                             | `OK` / `ERROR`           |
+| Invocation | `SUCCEEDED` or `PENDING` / `FAILED` / `RETRYING`                   | `OK` / `ERROR` / `UNSET` |
+| Operation  | `SUCCEEDED` / error object / other failure without an error object | `OK` / `ERROR` / `UNSET` |
+| Attempt    | `FAILED` / any other outcome                                       | `ERROR` / `OK`           |
 
-### Plugin Options
+Operation and attempt errors are recorded as exception events when an error
+object is available. Workflow and Invocation failures set an `ERROR` status and
+status message without recording an exception event.
 
-```typescript
-import { OtelPlugin } from "@aws/durable-execution-sdk-js-otel";
+## Log Correlation
 
-const plugin = new OtelPlugin({
-  // Use a custom context extractor (default: xRayContextExtractor)
-  contextExtractor?: xRayContextExtractor,
+When `enrichLogger` is enabled, durable log records receive the currently
+active OpenTelemetry `traceId`, `spanId`, and `otelTraceSampled` values.
+Outside an active span, no fields are added.
 
-  // Provide your own TracerProvider if you already have one configured
-  tracerProvider?: myTracerProvider,
+Disable enrichment when another logging integration already injects equivalent
+fields:
 
-  // Custom instrumentation scope name (default: "aws-durable-execution-sdk-js")
-  instrumentationName?: "my-service",
+```typescript
+const plugin = new ExecutionOtelPlugin({
+  enrichLogger: false,
 });
 ```
 
-### Context Extractors
-
-The plugin supports multiple strategies for extracting upstream trace context:
+For `ExecutionOtelPlugin`, handler-level logs correlate with Workflow. For
+`InvocationOtelPlugin`, handler-level logs correlate with Invocation. Logs
+inside child contexts and attempts correlate with their active operation or
+attempt span.
 
-```typescript
-import {
-  OtelPlugin,
-  xRayContextExtractor,
-  w3cClientContextExtractor,
-} from "@aws/durable-execution-sdk-js-otel";
+## Public API
 
-// Default: X-Ray trace header (recommended for most Lambda deployments)
-new OtelPlugin({ contextExtractor: xRayContextExtractor });
+### Plugins
 
-// W3C Trace Context via clientContext (requires backend propagation support (TODO))
-new OtelPlugin({ contextExtractor: w3cClientContextExtractor });
+```typescript
+new ExecutionOtelPlugin(config?: OtelPluginConfig);
+new InvocationOtelPlugin(config?: OtelPluginConfig);
 ```
 
-## Verification
+### Provider Types
 
-After deploying your function with the plugin configured:
-
-1. **Invoke your durable function** — trigger at least one execution that includes multiple steps or a wait/resume cycle.
+```typescript
+type IdGeneratorFactory = (fallbackIdGenerator?: IdGenerator) => IdGenerator;
 
-2. **Check Cloudwatch console** — Navigate to Cloudwatch → Traces in the AWS Console. You should see a trace with:
-   - An "invocation" span per invocation
-   - Child spans for each durable operation (named after your step names)
-   - All invocations of the same execution grouped under one trace ID
+type TracerProviderFactory = (
+  createIdGenerator: IdGeneratorFactory,
+) => TracerProvider;
+```
 
-3. **Check log correlation** — If you use `enrichLogContext()`, verify that your logs include `traceId` and `spanId` fields matching the spans in X-Ray.
+### `DeterministicIdGenerator`
 
-4. **Confirm sampling** — If you set `OTEL_TRACES_SAMPLER=traceidratio` and `OTEL_TRACES_SAMPLER_ARG` to a value less than 1.0, verify that only the expected proportion of traces appear.
+An OpenTelemetry `IdGenerator` with `AsyncLocalStorage`-scoped deterministic
+overrides. `withIds()` applies optional trace and span IDs only while its
+synchronous callback starts a span. The span ID override is consumed after its
+first use. All other ID generation delegates to the fallback generator.
 
-5. **span links** — For operations that span multiple invocations (e.g., after a wait resumes), though span links are set, they are not visualized within CloudWatch console.
+```typescript
+const generator = new DeterministicIdGenerator(fallbackIdGenerator);
 
-### Troubleshooting
+const span = generator.withIds(
+  { traceId: executionTraceId, spanId: deterministicSpanId },
+  () => tracer.startSpan("operation"),
+);
+```
 
-| Symptom                           | Likely Cause                                                    |
-| --------------------------------- | --------------------------------------------------------------- |
-| No traces appear                  | ADOT layer not configured, or `AWS_LAMBDA_EXEC_WRAPPER` not set |
-| Traces appear but are fragmented  | X-Ray active tracing not enabled on the Lambda function         |
-| Missing spans for some operations | `OTEL_TRACES_SAMPLER_ARG` set below 1.0                         |
-| `_X_AMZN_TRACE_ID` not populated  | X-Ray active tracing not enabled                                |
+### ID Helpers
 
-## API Reference
+```typescript
+deriveTraceIdFromArn(
+  executionArn: string,
+  executionStartTimestamp?: Date,
+): string;
 
-### `OtelPlugin`
+deriveTraceIdFromXRayRoot(xRayRoot: string): string | undefined;
 
-The main plugin class. Implements `DurableInstrumentationPlugin` from `@aws/durable-execution-sdk-js`.
+deriveExecutionTraceId(
+  environment: ExecutionTraceEnvironment,
+  executionArn: string,
+  executionStartTimestamp?: Date,
+): string;
 
-```typescript
-new OtelPlugin(config?: OtelPluginConfig)
-```
+deriveWorkflowSpanId(executionArn: string): string;
 
-### `DeterministicIdGenerator`
+deriveExecutionRootSpanId(executionArn: string): string;
 
-A custom OpenTelemetry `IdGenerator` that produces reproducible trace and span IDs from execution metadata. Exported for advanced use cases.
+deriveSpanIdFromOperationId(
+  operationId: string,
+  executionArn: string,
+): string;
+```
 
-### `xRayContextExtractor`
+`deriveTraceIdFromArn` returns a 32-character hexadecimal trace ID.
+`deriveTraceIdFromXRayRoot` converts a valid X-Ray `Root` value to an
+OpenTelemetry trace ID and returns `undefined` for invalid input.
+`deriveExecutionTraceId` applies the default plugin precedence to an explicit
+environment: a valid `_X_AMZN_TRACE_ID` `Root` wins, otherwise it uses the same
+ARN-and-start-time fallback as the plugins. Pass `process.env` in Lambda or a
+plain object in tests. When no valid X-Ray Root is available, pass the same
+execution start timestamp supplied to the plugin; omit it only when it is
+unavailable to both callers.
+`deriveWorkflowSpanId` hashes `workflow:<execution ARN>`,
+`deriveExecutionRootSpanId` hashes `execution-root:<execution ARN>` (a distinct
+namespace so the synthetic root never collides with the Workflow or operation
+spans on the shared trace), and `deriveSpanIdFromOperationId` hashes
+`<execution ARN>:<operation ID>`. All three span helpers return 16-character
+hexadecimal IDs.
+
+Wrapper-style instrumentation can create spans on the same execution trace and
+refer to the same Workflow span without copying the plugin's derivations:
 
-Default context extractor. Reads the `_X_AMZN_TRACE_ID` environment variable to derive trace context.
+```typescript
+import {
+  deriveExecutionTraceId,
+  deriveWorkflowSpanId,
+} from "@aws/durable-execution-sdk-js-otel";
 
-### `w3cClientContextExtractor`
+const executionTraceId = deriveExecutionTraceId(
+  process.env,
+  executionArn,
+  executionStartTimestamp,
+);
+const workflowSpanId = deriveWorkflowSpanId(executionArn);
+```
 
-Alternative context extractor. Reads `traceparent` from `context.clientContext.custom.traceparent` (W3C Trace Context format).
+### Context Extractors
 
-### `ContextExtractor`
+```typescript
+xRayContextExtractor(info: InvocationInfo): ContextExtractorResult;
+w3cClientContextExtractor(info: InvocationInfo): ContextExtractorResult;
+```
 
-Type definition for custom context extractor functions.
+The package also exports the `ContextExtractor`, `ContextExtractorResult`,
+`ExecutionTraceEnvironment`, `IdGeneratorFactory`, `TracerProviderFactory`, and
+`OtelPluginConfig` types.
+
+## Verification and Troubleshooting
+
+After deployment:
+
+1. Invoke a durable function with multiple steps or a wait/resume cycle.
+2. Verify Invocation spans and completed operation spans appear after enabled
+   invocations.
+3. Verify one Workflow span appears after the execution becomes terminal.
+4. Expect the Workflow span, every Invocation span, and the operation/attempt
+   spans to share one execution trace ID.
+5. Confirm the documented cross-links connect the workflow and invocation views.
+6. Confirm unrelated root spans retain provider-generated trace IDs.
+7. Verify durable log records contain correlation fields when
+   `enrichLogger` is enabled.
+
+If no plugin spans appear:
+
+- confirm a compatible SDK provider is registered before invocation start, or
+  return one through `tracerProviderFactory`;
+- confirm the provider has a span processor and exporter;
+- check for the plugin warning that says telemetry was disabled because the
+  global tracer was incompatible;
+- remember that dynamic loading supports only the global-provider path;
+- remember that Workflow is not exported until `SUCCEEDED` or `FAILED`.
+
+One execution trace is expected. The Workflow span and every per-invocation
+Invocation span share it, anchored at the execution ancestor (the propagated
+remote parent, or a synthetic execution root).
 
 ## License