@aws/durable-execution-sdk-js-otel 1.1.0 → 1.2.1

package/README.md

@@ -153,7 +153,25 @@ tracer from it. Register it globally when the application also needs its
 context manager, propagator, or library instrumentation.
 
 At the end of each enabled invocation, both plugins call `forceFlush()` when
-the provider exposes it. They never call `shutdown()`.
+the provider exposes it. For the default global API provider, they also check
+its public delegate, so an SDK `BatchSpanProcessor` is flushed when the API proxy
+does not expose `forceFlush()` itself. Explicit factory providers are used as
+returned and are not unwrapped. The plugins never call `shutdown()`.
+
+Both views use one `flushTimeoutMillis` budget across the invocation-end
+drain/flush rounds (default **1000 ms**, positive and at most **2147483647 ms**,
+Node.js's maximum timer delay). This bounds
+asynchronous provider waiting on both the global and explicit-provider paths.
+Choose a value that leaves room in the Lambda invocation timeout and configure
+the exporter's own transport timeouts too. The plugin has no remaining-runtime
+deadline in its invocation metadata and cannot guarantee that this budget fits
+the time left after the handler.
+
+On expiry, the plugin drains one bounded snapshot of terminal notifications
+already received, then returns without starting another provider flush. Export
+delivery is best effort: spans may remain queued. Timing out does not cancel an
+export, shut down the provider, or preempt synchronously blocking provider or
+processor code.
 
 ## Dynamic Loading from a Lambda Layer
 
@@ -183,6 +201,13 @@ needed.
 
 ## Choosing a Plugin
 
+Configure exactly one durable OTel view: `ExecutionOtelPlugin` for a workflow
+view, or `InvocationOtelPlugin` for an invocation view. Registering both (or
+registering one view twice) raises `PluginLoadError` before invocation hooks
+run. This applies to explicit `plugins`, `DURABLE_EXECUTION_PLUGINS`, and any
+combination of the two. Other instrumentation plugins can run alongside the
+selected view.
+
 ### `ExecutionOtelPlugin`
 
 Use this plugin for a workflow-centered view. Operations and attempts are
@@ -238,6 +263,33 @@ Open operation spans are ended at the invocation boundary and retain
 `durable.operation.status=STARTED`. When an operation completes in a later
 invocation, the plugin emits a continuation span in that invocation.
 
+Live invocation-view span boundaries and exception events use a wall-clock
+origin captured with `Date.now()` at invocation start, advanced by monotonic
+elapsed time. This matches the clock model of ordinary `tracer.startSpan()`
+spans instead of using the process-wide `performance.timeOrigin`. Absolute
+`HrTime` tuples keep epoch milliseconds distinct from relative performance time.
+Each resumed invocation captures its own anchor. Supplied execution-start
+Dates used to backdate Workflow and synthetic roots are preserved; execution-view
+provided timestamps are retained without adding cross-clock boundary compensation.
+
+The wall and monotonic reads are not atomic. Sampling delay, millisecond wall
+precision, later wall-clock adjustments, and clocks on different machines can
+still disagree. The plugin does not rewrite backend Dates or user spans, retain
+a clock high-water mark across invocations, or promise simultaneous strict
+ordering and complete containment across arbitrary clock skew.
+
+Independently of the invocation clock origin, execution-view operation and attempt
+exception events use the supplied backend end timestamp when one is available,
+just as their span end does. Recording the event at local receipt time can place
+it after the backend-ended span. Missing end timestamps retain the normal tracer
+default; this does not add rounding or cross-clock ordering policies.
+
+When an execution-view completion has no recorded start timestamp, an available
+backend end timestamp supplies a zero-duration fallback. This keeps its exception
+at that known completion time instead of before a later local default start.
+Recorded starts and missing-end defaults retain their existing behavior; this
+does not reconcile a captured local start with an earlier backend end.
+
 Because the original span context is not checkpointed, replayed `STEP` and
 `CONTEXT` spans and cross-invocation continuation spans use new provider IDs.
 They correlate through two links: the real exported `Workflow` span, and the
@@ -298,6 +350,66 @@ The `Invocation` span may still nest under a same-trace ambient handler span —
 see [Invocation parent](#invocation-parent) — without changing the execution
 ancestor the `Workflow` span joins.
 
+### Synthetic root export lifecycle
+
+When fallback tracing is sampled, the backend supplies `executionStartTimestamp`,
+and the configured SDK tracer allows the durable sampler wrapper to be installed,
+both views create and end `DurableExecutionRoot` during each invocation's start
+hook. Its start and end are both that backend timestamp. The normal
+invocation-boundary flush can export it before a `PENDING` or `RETRYING` invocation
+returns. A retry with checkpoint history or an intermediate resume can therefore
+recover an earlier missing export, including when tracing became available only
+after the first invocation. Recovery requires a later sampled invocation to
+actually deliver its copy; stopping or timing out before any successful delivery
+can still leave the execution without an exported root. This policy does not rely
+on `isFirstInvocation`: a retry after checkpointed work runs in replay mode, while
+an attempt without that history can still be identified as the first invocation.
+
+Each sampled invocation emits at most one anchor: a terminal invocation does not
+emit a second copy if its start hook already created one. Additional copies on
+resumes are intentional, allowing consumers to deduplicate by trace ID and span ID.
+The SDK-owned span fields are identical when the execution's trace identity,
+backend start timestamp and instrumentation configuration remain stable. The
+anchor carries `durable.execution.arn`, but no terminal status, request ID or
+execution duration; `Workflow` continues to carry the duration and outcome.
+`DurableExecutionRoot` is a zero-duration identity span: its descendants end after
+it, which some trace viewers or containment validators may flag.
+
+A forwarding tracer that hides the configured sampler retains the existing
+terminal-only root creation path. Creating an early root through that tracer would
+query the hidden sampler independently and could sample a root after dropping its
+Invocation child. The plugin does not add that early sampler call; the underlying
+provider still controls sampling, and these tracers do not receive early-anchor
+recovery. A terminal backup can recover a missing first export only if a sampled
+terminal invocation actually runs and delivers it.
+
+Resources belong to the configured provider. Resource attributes such as
+`faas.instance` can differ across execution environments even for identical
+anchor span content; backends may retain whichever resource accompanies their
+chosen copy. Span processors that enrich these spans must likewise avoid
+invocation-varying span fields when relying on identity-based deduplication.
+Export delivery and flushing remain the provider/layer's responsibility. The
+plugin awaits an available provider or global-delegate `forceFlush()`, but a killed invocation
+or a failed export cannot guarantee delivery; this is not exactly-once export.
+
+If a caller omits `executionStartTimestamp`, the plugin retains terminal-only
+synthetic-root materialization with its existing timestamp fallback. It does not
+invent a wall-clock-based early anchor. These callers do not get the early-anchor
+or identical-timestamp guarantee; the backend timestamp must be consistently
+supplied to enable that contract. Complete propagated remote parents are never
+materialized by the SDK. With the durable sampler wrapper installed, explicit
+`NOT_SAMPLED` executions emit no root.
+
+With the durable sampler wrapper installed, sampling is resolved once per
+invocation and the anchor reuses that decision without a second sampler query.
+Explicit upstream decisions or a deterministic trace-based policy (for example a
+ratio sampler) keep anchor and terminal `Workflow` decisions consistent across
+resumes. A non-deterministic custom sampler can select different invocations,
+leaving an anchor without a sampled terminal `Workflow`. Decisions are not
+persisted. These once-per-invocation guarantees do not apply to an opaque tracer:
+its provider can query its sampler independently for each span, including the
+existing terminal root; the early-anchor path is skipped to preserve that baseline.
+
 ### Workflow identity and lifecycle
 
 `Workflow` is an `INTERNAL` span that **joins the execution trace** by parenting
@@ -317,7 +429,7 @@ The plugins carry the same `Workflow` span identity on each invocation as a
 non-recording span context, and create the single recording `Workflow` span only
 when the execution reaches `SUCCEEDED` or `FAILED`, backdated to the execution
 start. `PENDING` and `RETRYING` invocations create no recording `Workflow` span,
-so none is ever left unended, and exactly one `Workflow` root is exported for the
+so none is ever left unended, and one terminal `Workflow` span is exported for the
 durable execution while intermediate invocation spans export normally. The
 non-recording context carries the execution's sampling decision, so operations
 under it are sampled consistently with the eventual root. In-flight attempt spans
@@ -415,6 +527,9 @@ interface OtelPluginConfig {
 
   /** Add active OTel IDs to durable log records. Defaults to true. */
   enrichLogger?: boolean;
+
+  /** Total invocation-end provider wait budget; defaults to 1000 ms. */
+  flushTimeoutMillis?: number;
 }
 ```
 
@@ -455,16 +570,28 @@ environment values are available.
 
 OpenTelemetry status mapping is:
 
-| Span       | Durable result                                                     | OTel status              |
-| ---------- | ------------------------------------------------------------------ | ------------------------ |
-| Workflow   | `SUCCEEDED` / `FAILED`                                             | `OK` / `ERROR`           |
-| Invocation | `SUCCEEDED` or `PENDING` / `FAILED` / `RETRYING`                   | `OK` / `ERROR` / `UNSET` |
-| Operation  | `SUCCEEDED` / error object / other failure without an error object | `OK` / `ERROR` / `UNSET` |
-| Attempt    | `FAILED` / any other outcome                                       | `ERROR` / `OK`           |
-
-Operation and attempt errors are recorded as exception events when an error
-object is available. Workflow and Invocation failures set an `ERROR` status and
-status message without recording an exception event.
+| Span       | Durable result | OTel status |
+| ---------- | -------------- | ----------- |
+| Workflow   | `SUCCEEDED` / `FAILED` | `OK` / `ERROR` |
+| Invocation | `SUCCEEDED` or `PENDING` / `FAILED` / `RETRYING` | `OK` / `ERROR` / `UNSET` |
+| Operation  | `SUCCEEDED` / `FAILED`, `TIMED_OUT`, `STOPPED` or an error object / otherwise | `OK` / `ERROR` / `UNSET` |
+| Attempt    | `FAILED` / any other outcome | `ERROR` / `OK` |
+
+Operation failure status and exception details are independent. `FAILED`,
+`TIMED_OUT`, and `STOPPED` remain `ERROR` when error details are absent, including when
+the handler catches its failure and returns successfully. The status message uses
+the supplied error message, or the failed status name when no error exists.
+
+`CANCELLED` without error details remains `UNSET`. A supplied error still makes
+a cancelled operation `ERROR`; this mapping does not introduce a general
+cancellation policy.
+
+Operation and attempt exceptions are recorded only when an error object is
+available. Wire errors with no defined values do not synthesize exception events,
+including when an unrecognized property is `undefined`. Explicit partial
+details, including empty strings or an empty stack trace, remain error information.
+Workflow and Invocation failures set `ERROR` without recording an exception event.
+Caller exceptions, checkpoint payloads and replay behavior are unchanged.
 
 ## Log Correlation
 
@@ -591,6 +718,102 @@ The package also exports the `ContextExtractor`, `ContextExtractorResult`,
 `ExecutionTraceEnvironment`, `IdGeneratorFactory`, `TracerProviderFactory`, and
 `OtelPluginConfig` types.
 
+### External completions and replay
+
+Both views retain terminal wait, invoke, and callback notifications received at
+invocation start or in checkpoint responses. If workflow traversal does not
+reach a completed operation before suspending or returning, its completion is
+exported at invocation end. Supplied operation timestamps, parent identity,
+status, and error details are preserved in the execution view; the invocation
+view completes the live segment or emits a linked continuation.
+
+A failed invoke still rejects with `InvokeError` when backend error details are
+absent. Its telemetry retains the absent error metadata in live, resumed, and
+deferred delivery, rather than inventing an exception. Provided error details
+continue through the existing metadata converter.
+
+For a named `waitForCallback`, a deferred inner callback recovers its derived
+`<name>-callback` display name from a known `WaitForCallback` parent when replay
+skips that branch. This does not change checkpointed names, operation identity,
+parent identity, or the source notification; explicit names remain unchanged.
+
+At the final cleanup of a terminal invocation, the execution view parents a
+deferred completion to `Workflow` if its direct parent is still an open local
+placeholder that will be discarded. During the open drain/flush phase, a live
+parent can still finish and its normal end hook exports the queued child with
+the real parent identity. An open parent alone does not trigger repeated flushes.
+The plugin does not invent a completion or span for an abandoned context. Pending
+or retrying invocations retain the parent identity
+because a later invocation can still complete and export that context. An
+exception is an unfinished parent below a known completed context in checkpoint
+history: replay cannot restart that unfinished branch, so its deferred completion
+also uses `Workflow`. `ReplayChildren` can traverse completed children to rebuild
+a result, but `ReplaySucceededContext` still blocks unfinished child operations.
+Missing or inconclusive ancestry is not treated as proof of abandonment.
+
+In both views, a completion notification already queued when its live context
+ends is exported before that context closes, preserving its OTel parent. Queued
+child draining and parent finalization run synchronously within the End hook;
+flush completion cannot reset invocation state between them. Invocation-end
+drains likewise finish their current snapshot before yielding. This
+includes an unawaited callback from a prior invocation: the invocation view emits
+its linked continuation while the resumed parent segment is still open.
+
+In the execution view, a completed context in checkpoint history remains a valid
+parent for a later completion. A context span actually recorded during the current
+invocation also remains a valid parent when an enclosing context suppressed its
+terminal checkpoint and history still shows it as unfinished. This recording takes
+precedence over abandonment inference within that invocation; knowledge of an
+uncheckpointed recording is not persisted into later invocations.
+
+A notification delivered after the parent's end hook may carry a
+completion that occurred earlier in the backend. Delivery order does not establish
+that chronology. Reusing the recorded parent identity does not recreate a parent
+span, compare timestamps, retime supplied dates, or promise temporal containment.
+The unfinished-parent safeguards still apply when no such recorded parent is known.
+
+This requires the SDK to receive the operation metadata. Lambda normally removes
+a completed context's children from subsequent execution state unless
+`ReplayChildren` preserves them. For completion visibility after a child context
+has finished, enable the appropriate `pluginsConfig.childOperationsDepth` (for
+example, `1` for direct children of a top-level context). Preserved contexts
+replay their orchestration; completed durable work still uses stored results.
+The plugin cannot reconstruct an operation that was pruned and whose lifecycle
+hooks do not run. The local runner now applies the service's pruning rule to
+invocation input and `GetDurableExecutionState`; its inspection results and
+checkpoint responses still retain their existing operation data.
+
+Within the configured flush budget, the invocation-end hook drains notifications
+received while it awaits a provider flush, then flushes the newly exported
+completions. On expiry it drains one snapshot of already-received notifications
+without another provider wait before clearing its state. In the execution view,
+completions held for a still-open parent are finalized after the open drain/flush
+phase. Both views close their invocation state before a final provider flush for
+operation or attempt spans ended during the previous wait or final cleanup,
+using only the original budget remaining. New operation-end hooks after
+finalization do not record additional spans during that final wait. At deadline
+expiry these final exports remain best effort. Change hooks enqueue notifications;
+the end hook owns this loop. In the execution view, these drains use the latest operation ancestry received during
+the end hook to detect an unfinished parent abandoned by a completed ancestor.
+Missing ancestry alone does not prove abandonment. In the
+invocation view, a completion after its individual segment has ended uses a linked
+continuation. A segment first created while a provider flush is waiting remains
+eligible for normal completion. Any such segment still open at final cleanup is
+ended before state is cleared and included in the final bounded flush. The
+execution view likewise closes recording attempt spans that started during the
+wait, without inventing a completed operation span for unfinished work.
+Enclosing spans may already have ended, so this does not guarantee
+late-child timestamp containment. Updates after the final drain/reset, a frozen
+runtime, and concurrent executions sharing a plugin instance remain outside this
+shutdown guarantee; the shared-instance limitation is tracked in [#950](https://github.com/aws/aws-durable-execution-sdk-js/issues/950).
+
+Notifications and operation-end hooks are deduplicated within each invocation.
+Plain replay without a fresh update does not re-export stored external
+completions. Deduplication is not persisted: a retry carrying the same updated
+operation IDs can export the completion again, preserving recovery eligibility.
+This is a conditional redelivery behavior, not a claim that Lambda always sends
+those IDs again or that the exporter has a durable delivery acknowledgement.
+
 ## Verification and Troubleshooting
 
 After deployment:
@@ -623,3 +846,106 @@ remote parent, or a synthetic execution root).
 ## License
 
 Apache-2.0
+
+## Chained-invoke propagation
+
+When preparing a new `context.invoke()` START, the SDK calls the optional
+`providePropagationMetadata(input)` hook after assigning the operation identity
+and before serializing the checkpoint. The readonly input contains
+`executionArn`, `operationId`, optional `parentOperationId`, and
+`targetFunctionName`. The SDK places the collected `xAmznTraceId` in the flat
+`ChainedInvokeOptions.XAmznTraceId` wire member, separately for each operation
+in a batched checkpoint. Function name, tenant, payload and operation identity
+are unchanged; the SDK does not inject tracing fields into user payloads.
+
+With the durable sampler bridge still installed on a compatible SDK tracer,
+both OTel views encode the resolved execution trace ID as `Root`, the calling
+CHAINED_INVOKE operation span ID as `Parent`, and the resolved sampling decision
+as `Sampled`, including `Sampled=0`. Producing metadata does not create an extra
+span or use an unrelated active span. For example, existing plugin registration
+also covers both calls here:
+
+```typescript
+const first = context.invoke("first", "first-function:1", { item: 1 });
+const second = context.invoke("second", "second-function:1", { item: 2 });
+return context.promise.all([first, second]);
+```
+
+An opaque tracer or a tracer whose sampler was replaced may make an independent
+decision for the invoke span. The plugins omit their optional propagation metadata
+when they cannot verify that their sampler bridge still controls that decision.
+They do not probe the sampler, create a span to obtain a decision, reinstall a
+customer's sampler, or change ordinary span behavior. The check occurs when the
+START is prepared; later provider reconfiguration cannot retroactively change a
+checkpointed header.
+
+The dispatcher calls providers synchronously with a frozen input snapshot.
+The first non-null supported value wins; identical values do not conflict.
+Later different values generate a warning with both plugin identities and a
+conflict count. Invalid results and ordinary plugin failures are isolated.
+Blank or whitespace-only headers are ignored so later providers can contribute;
+nonblank values remain opaque and are not trimmed or parsed by the core SDK.
+With no plugin, an absent hook or no returned value, the wire member is omitted
+and the backend's existing tracing fallback applies.
+
+Replaying a persisted START, including a pending or completed invoke, does not
+call the hook or send another START. A failed, uncommitted START can collect
+metadata again when the invocation is retried. Providers must therefore use the
+stable identity, avoid side effects, and not assume exactly-once callbacks.
+
+The runtime dependency requires `@aws-sdk/client-lambda` 3.1152.0 or newer.
+That published release declares the optional `XAmznTraceId` field and includes
+it in the generated serializer. The strict wire-model parity guard and actual
+serializer regression pass with that release.
+
+**Deployed backend trace-topology qualification remains separate.** Local
+serialization and package compatibility do not establish
+that a deployed chained invocation receives the intended Root/Parent/Sampled
+or exports the expected caller/child topology. Existing sampled cloud traces
+do not qualify unsampled delivery.
+
+The design also adds `DistributedMapOptions.XAmznTraceId` to a future service
+model. This SDK currently has no distributed-map options or START API; its
+`map` and `parallel` APIs use CONTEXT operations. They are not distributed-map
+transport paths, and this change does not introduce a new operation API or
+invent the future Source/Processor/Destination structures.
+## Core compatibility
+
+OTel 1.2 preserves valid core 2.4–2.x registrations, including separate
+OTel-only Lambda layers. Its core peer is optional (`>=2.4.0 <3.0.0`), and the
+dynamic provider contract remains version 1. Existing on-demand environment
+carrier behavior remains available on older cores; upgrading the plugin alone
+does not add fields to the invocation metadata those cores supply.
+
+The new invocation-local LMI carrier and core registration exclusivity require
+the coordinated core 2.7 / OTel 1.2 pair. Upgrade both for those capabilities.
+Older core/plugin combinations do not gain those guarantees. The two OTel views
+remain an unsupported combination, and the updated core rejects it with updated
+plugin metadata. The later core 3 factory migration is a separate major release.
+
+Instrumentation error details are optional even for failed operations. The core
+metadata correction for an empty callback error payload no longer invents an
+error object. API-v1 plugins that previously used only `OperationInfo.error`
+presence as a failure signal should use `OperationInfo.status` instead (and
+`AttemptEndInfo.outcome` for an attempt). Missing error details do not imply a
+successful operation; supplied partial or explicitly empty detail fields are
+retained. This does not change the provider/hook shape, callback rejection,
+persisted error payload, or replay identity.
+
+Registration metadata uses the namespaced symbol
+`Symbol.for("aws.lambda.durable.instrumentation.plugin-registration")`.
+The bundled views declare it on their constructors, so the core can validate
+all selected provider types before calling any environment-selected factory.
+Conflicting dynamic views therefore do not install sampler or ID-generator
+wrappers on the global tracer. Subclasses retain all inherited static exclusive
+groups; declaring their own static metadata adds a constraint and cannot replace
+an inherited one. Repeating a group within one constructor chain does not count
+as a second plugin registration. Subclasses are identified by their actual
+constructor names in diagnostics. Explicit instances
+have already been constructed by the application before registration; validation
+cannot undo that construction. Existing instance metadata remains supported and
+returned instances are checked too, including subclasses returned by providers
+that declare a broader base type.
+The base hook interface stays unchanged, and ordinary properties such as a
+custom plugin's `registration` field are not interpreted by the SDK. This keeps
+existing valid plugin classes and generic hook dispatchers source-compatible.