@blockcast/mmt-render 0.1.0-main.c0e9e40265a1

package/README.md

@@ -0,0 +1,207 @@
+# @blockcast/mmt-render
+
+Shared render pipeline for MMT players: FEC-aware wall-clock pacing, per-AU presentation-time reconstruction under FEC interleave, an `AudioWorkletProcessor` with an exact-timestamp PCM FIFO, and a keyframe-gated `VideoDecoder` driver. The package is the canonical substrate for moqtail, `hang-mmt-fec`, and Shaka integrations so they do not independently invent timing, buffering, or sample-correction policy. It is *not* a "reference renderer + diagnostics UI" — there is no DOM, chart, or catalog browser. Diagnostics are structured snapshots and worklet telemetry messages that callers can render however they like.
+
+## Install
+
+```bash
+pnpm add @blockcast/mmt-render
+```
+
+Peer-installs through normal deps: `@blockcast/mmt-container` (provides `MmtpDecodePipeline` and the FEC-aware MFU reassembler) and `@blockcast/mmt-fec` (provides `FecBlockMapper`, which carries `pacingThresholdMs`, `K`, `D`, `symbolsPerGroup`).
+
+## Quick start (audio)
+
+```ts
+import { AudioRenderPipeline } from "@blockcast/mmt-render/audio"
+import { FecBlockMapper } from "@blockcast/mmt-fec"
+
+const ctx = new AudioContext({ sampleRate: 48000 })
+const mapper = new FecBlockMapper({ interleaveDepthMs: 66, gopDurationMs: 21.33, k: 32 })
+
+const audio = new AudioRenderPipeline({
+  context: ctx,
+  codec: "mp4a.40.2",
+  packetId: 0x0011,
+  sampleRate: 48000,
+  channels: 2,
+  description: aacEsdsDescription,   // from catalog / init segment
+  mapper,
+  reassemblyPolicy: { transport: "reliable-moq" },
+  latencyMs: mapper.interleaveDepthMs, // one catalog FEC block
+  diag: true,
+  onDiag: (d) => console.log("ring", d),
+})
+
+await audio.ready()
+audio.rootNode.connect(ctx.destination)
+
+for await (const obj of moqSubscription) {
+  audio.feedMoqObject(obj.payload)
+  const sleep = audio.paceSleepMs()
+  if (sleep > 0) await new Promise((r) => setTimeout(r, sleep))
+}
+```
+
+## Quick start (video)
+
+```ts
+import { VideoRenderPipeline } from "@blockcast/mmt-render/video"
+
+const video = new VideoRenderPipeline({
+  codec: "hev1.1.6.L120.90",
+  packetId: 0x0010,
+  description: hvccBox,
+  width: 1920,
+  height: 1080,
+  framerate: 30,
+  mapper,
+  reassemblyPolicy: { transport: "reliable-moq" },
+  onFrame: (frame) => {
+    canvas.getContext("2d")?.drawImage(frame, 0, 0)
+    frame.close()
+  },
+  onError: (e) => console.warn(e),
+})
+
+for await (const obj of moqSubscription) {
+  video.feedMoqObject(obj.payload)
+  const sleep = video.paceSleepMs()
+  if (sleep > 0) await new Promise((r) => setTimeout(r, sleep))
+}
+```
+
+## Exports
+
+Per `package.json`:
+
+| Subpath | Symbols |
+|---------|---------|
+| `@blockcast/mmt-render` | re-exports of `./pacing`, `./audio`, `./video` |
+| `@blockcast/mmt-render/pacing` | `FrameCountPacer`, `InterleaveTiming`, `auDurationMsFromCodec`, `pictureDurationMs` |
+| `@blockcast/mmt-render/pacing/frame-pacer` | `FrameCountPacer` only |
+| `@blockcast/mmt-render/pacing/interleave-timing` | `InterleaveTiming` only |
+| `@blockcast/mmt-render/audio` | `AudioRenderPipeline`, `AudioRingBuffer`, message types, `WORKLET_PROCESSOR_NAME`, `workletModuleUrl` |
+| `@blockcast/mmt-render/audio/ring-buffer` | `AudioRingBuffer` only |
+| `@blockcast/mmt-render/audio/render-messages` | `FromMain` / `ToMain` message types |
+| `@blockcast/mmt-render/audio/render-pipeline` | `AudioRenderPipeline` only |
+| `@blockcast/mmt-render/audio/worklet-url` | `workletModuleUrl()`, `WORKLET_PROCESSOR_NAME` |
+| `@blockcast/mmt-render/audio/worklet.js` | Prebuilt self-contained worklet IIFE — pass to `audioWorklet.addModule()` |
+| `@blockcast/mmt-render/video` | `VideoRenderPipeline`, `KeyframeGate`, frame meta types |
+| `@blockcast/mmt-render/video/keyframe-gate` | `KeyframeGate` only |
+| `@blockcast/mmt-render/video/render-pipeline` | `VideoRenderPipeline` only |
+
+The fine-grained subpaths exist so a lean consumer (e.g. moqtail's `player.ts`) can import just the pacer + AU-duration helpers without pulling in `@blockcast/mmt-container` (see `src/audio/index.ts:5-7`).
+
+## Components
+
+### `FrameCountPacer` (`src/pacing/frame-pacer.ts`)
+
+Media-agnostic wall-clock pacer. Anchors on the first `notifyFrame()` call; computes media time as `frameCount × frameDurationMs`. Sleeps the read loop when the decoder runs ahead of wall-clock by more than `mapper.pacingThresholdMs`. Returns `0` for `NullBlockMapper` (pacing disabled). Why not derive media time from MPU timestamps? Under FEC interleaving, encoders pack multiple AUs per MMTP MPU with a single `mpu_timestamp`; the observed delta is a fraction of the real per-AU duration, so timestamp-driven pacing never sleeps and lets burst delivery saturate the decoder. See the file-level comment for the Sony libatsc3 reference.
+
+`PacerSnapshot` exposes `frameCount`, `mediaMs`, `wallMs`, `aheadMs`, `thresholdMs`, `anchored` for diagnostics overlays.
+
+### `InterleaveTiming` (`src/pacing/interleave-timing.ts`)
+
+Per-AU presentation-time reconstruction from FEC coordinates. Implements `draft-ramadan-moq-fec-00 §8.3`:
+
+```
+au_index_in_block = groupInBlock × symbolsPerGroup + objectId
+ssId              = sbn × K + au_index_in_block
+presentationUs    = anchorMpuUs + ssId × frameDurationUs
+```
+
+Same answer whether the AU arrived via MoQ unicast (group_id/object_id) or multicast UDP (SS_ID from the FEC Payload ID). Methods: `presentationUs(groupId, objectId)`, `presentationUsFromSsId(ssId)`, `frameAtOffset(offsetUs)` (inverse), `blockTimeRangeUs(sbn)`. Consumers: CMAF segment cut points, debug overlays, cross-path FEC combining.
+
+### `auDurationMsFromCodec` / `pictureDurationMs` (`src/pacing/au-duration.ts`)
+
+Nominal AU duration helpers. AAC-LC = 1024 samples; Opus default = 960; FLAC = 4096. Throws on unknown codec — no silent fallback. Lives in the pacing subpath specifically to avoid pulling in `mmt-container`.
+
+### `AudioRingBuffer` (`src/audio/audio-ring-buffer.ts`)
+
+Timestamp-anchored PCM FIFO. Decoder output stays in emission order, and every decoded chunk adds an exact MMTP timestamp anchor for playout reporting. Timestamps never reposition samples, guess discontinuities, or trigger sample correction. Startup waits for the full caller-supplied, catalog-derived capacity. Underflow re-stalls until that capacity is full again. Source changes use an explicit `reset`; there is no elapsed-time heuristic. The compatibility `skips` and `dups` telemetry fields remain present but are always zero.
+
+Optional telemetry via `getDiagAndReset()`: `reads`, `avgFill`, `capacity`, `skips`, `dups`, `overflowSamples`.
+
+### `AudioRenderPipeline` (`src/audio/audio-render-pipeline.ts`)
+
+End-to-end glue: `MoQ object → MmtpDecodePipeline → WebCodecs AudioDecoder → AudioWorkletNode (ring buffer)`. Owns the worklet, decoder, decode pipeline, and pacer. Audio uses `reorderWindow: 0` because AAC/Opus frames are self-contained.
+
+API:
+
+```ts
+await pipeline.ready()                  // load worklet + create decoder
+pipeline.feedMoqObject(packet)          // hot path
+pipeline.paceSleepMs()                  // throttle read loop
+pipeline.rootNode.connect(destination)  // connect to AudioContext
+pipeline.resetPacer()                   // on source switch (MoQ ↔ multicast)
+pipeline.resetPlayout()                 // drop queued PCM + old timestamp epoch
+pipeline.close()                        // teardown
+```
+
+`onState` and `onDiag` callbacks surface worklet messages (`StateMessage`, `DiagMessage`) for player UIs.
+
+### `KeyframeGate` (`src/video/keyframe-gate.ts`)
+
+Tiny state machine. Drops delta frames until the first keyframe arrives, then opens permanently. `reset()` re-arms after a decoder error so the next IDR reopens the gate. Required for mid-stream subscribe — relays may deliver P/B frames from the middle of a GOP before the first IDR, which yields "key frame required" decoder errors otherwise.
+
+### `VideoRenderPipeline` (`src/video/video-render-pipeline.ts`)
+
+End-to-end glue for video: `MoQ object → MmtpDecodePipeline → KeyframeGate → WebCodecs VideoDecoder → onFrame(VideoFrame)`. Datagram delivery uses a catalog-derived frame reorder window and explicit catalog/in-band deadline. Reliable MoQ uses protocol-verified sequential groups with no frame window or timeout. Decoder errors trigger an internal `reset()` that closes the gate, resets the pacer, and reconfigures the decoder so the next keyframe re-opens playback.
+
+This replaces the `decodeQueueSize > 10 drop-deltas` workaround in moqtail's `webcodecs-backend.ts` and the `reorderWindow = 0 // TODO` in hang-mmt-fec's `video/decoder.ts` (see file header at `src/video/video-render-pipeline.ts:1-15`).
+
+## AudioWorklet bundle
+
+`AudioWorkletGlobalScope` cannot resolve bare specifiers, so the worklet is shipped as a self-contained IIFE at `dist/worklet/render-worklet.js`. Two ways to load it:
+
+```ts
+// 1. Vite-style static URL (recommended for bundled apps)
+import workletUrl from "@blockcast/mmt-render/audio/worklet.js?url"
+await ctx.audioWorklet.addModule(workletUrl)
+
+// 2. import.meta.url resolution (no bundler)
+import { workletModuleUrl } from "@blockcast/mmt-render/audio/worklet-url"
+await ctx.audioWorklet.addModule(workletModuleUrl())
+```
+
+`AudioRenderPipeline.ready()` uses option (2) internally. Processor name is exported as `WORKLET_PROCESSOR_NAME = "mmt-render"`.
+
+## Diagnostics surface
+
+The package does not render its own UI. It exposes typed data the host can render:
+
+- `FrameCountPacer.snapshot` → `PacerSnapshot` (frameCount, mediaMs, wallMs, aheadMs, thresholdMs, anchored)
+- `AudioRingBuffer.getDiagAndReset()` → `AudioRingBufferDiag` (reads, avgFill, capacity, skips, dups, overflowSamples)
+- Worklet `StateMessage` (exact timestamp, stalled, bufferedFrames) every ~13 ms
+- Worklet `DiagMessage` (fill/overflow/underflow telemetry; skip/dup remain zero) when `diag: true` was passed at init
+- `audio underflow: <ms>ms` `console.warn` when the worklet runs dry (`src/audio/render-worklet.ts:58`)
+
+There is **no CMCD `x-bc-*` emission, no catalog (CMSF) inspector, and no MPU/MFU packet inspector** in this package. CMCD lives in `@blockcast/mmt-container/cmcd-producer`; catalog parsing lives in `@blockcast/mmt-container/carp`. This package is purely the render side.
+
+## Use cases
+
+- **Player implementations.** moqtail and hang-mmt-fec both consume this — adding a third player only requires wiring a transport into `feedMoqObject()`.
+- **Debugging multicast losses.** `PacerSnapshot.aheadMs` going negative + ring-buffer underflow warns indicate the decoder can't keep up; large positive `aheadMs` indicates burst delivery and pacing is throttling correctly.
+- **Customer demos / overlays.** `onState` + `onDiag` callbacks feed real-time HUDs without the host having to instrument the pipeline.
+- **Source-switch testing.** `resetPlayout()` / `VideoRenderPipeline.reset()` clear the old epoch when flipping between MoQ unicast and SSM multicast.
+
+## Related packages
+
+- `@blockcast/mmt-container` — `MmtpDecodePipeline` (consumed for AU framing), MFU reassembly, catalog parser
+- `@blockcast/mmt-fec` — `FecBlockMapper` (provides `pacingThresholdMs`, `K`, `D`, `symbolsPerGroup`); `NullBlockMapper` disables pacing
+- `@blockcast/mmt-transport` — produces the MoQ objects that feed `feedMoqObject()`
+
+## Testing
+
+```bash
+pnpm test            # vitest, runs src/__tests__/*.test.ts
+pnpm test:watch
+pnpm build           # tsc --emitDeclarationOnly + esbuild bundles (incl. worklet IIFE)
+```
+
+Existing suites: `frame-pacer.test.ts`, `interleave-timing.test.ts`, `audio-ring-buffer.test.ts`, `keyframe-gate.test.ts`. The pacer suite uses `vi.spyOn(performance, "now")` to drive deterministic wall time. There is no DOM / `AudioContext` integration test — the worklet is exercised in-browser by the consuming players' E2E suites.
+
+## License
+
+Apache-2.0 — see `LICENSE`. (libmmt npm packages are all Apache-2.0; the Rust crates are dual `MIT OR Apache-2.0`.)

package/dist/audio/audio-buffer-plan.d.ts

@@ -0,0 +1,27 @@
+export interface AudioBurstBufferPlanInput {
+    sampleRate: number;
+    chunkFrames: number;
+    /** Catalog media-group duration. Integer milliseconds may be rounded. */
+    groupDurationMs: number;
+    /** Optional catalog-derived datagram/FEC delivery span. */
+    deliveryWindowMs?: number;
+}
+export interface AudioBurstBufferPlan {
+    /** PCM required before playout starts or resumes. */
+    startupFrames: number;
+    /** Maximum retained PCM, including the next complete delivery burst. */
+    capacityFrames: number;
+    /** Maximum PCM frames delivered as one catalog-defined burst. */
+    deliveryFrames: number;
+}
+/**
+ * Convert catalog delivery timing into exact codec-frame FIFO geometry.
+ *
+ * `groupDurationMs` is commonly rounded to an integer, so media-group units
+ * use nearest-unit rounding. FEC delivery windows are upper bounds and use
+ * ceiling. Storage holds the startup burst plus one complete successor burst;
+ * this prevents an atomic MoQ group from overwriting PCM while the worklet is
+ * draining the preceding group without adding a guessed time margin.
+ */
+export declare function deriveAudioBurstBufferPlan(input: AudioBurstBufferPlanInput): AudioBurstBufferPlan;
+//# sourceMappingURL=audio-buffer-plan.d.ts.map

package/dist/audio/audio-buffer-plan.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"audio-buffer-plan.d.ts","sourceRoot":"","sources":["../../src/audio/audio-buffer-plan.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,yBAAyB;IACxC,UAAU,EAAE,MAAM,CAAA;IAClB,WAAW,EAAE,MAAM,CAAA;IACnB,yEAAyE;IACzE,eAAe,EAAE,MAAM,CAAA;IACvB,2DAA2D;IAC3D,gBAAgB,CAAC,EAAE,MAAM,CAAA;CAC1B;AAED,MAAM,WAAW,oBAAoB;IACnC,qDAAqD;IACrD,aAAa,EAAE,MAAM,CAAA;IACrB,wEAAwE;IACxE,cAAc,EAAE,MAAM,CAAA;IACtB,iEAAiE;IACjE,cAAc,EAAE,MAAM,CAAA;CACvB;AAOD;;;;;;;;GAQG;AACH,wBAAgB,0BAA0B,CACxC,KAAK,EAAE,yBAAyB,GAC/B,oBAAoB,CA2BtB"}

package/dist/audio/audio-render-pipeline.d.ts

@@ -0,0 +1,83 @@
+/**
+ * AudioRenderPipeline — end-to-end audio render for MoQ/MMT players.
+ *
+ * Pipeline: MoQ object → MmtpDecodePipeline → WebCodecs AudioDecoder
+ *           → AudioWorkletNode (exact-timestamp FIFO)
+ *
+ * The consumer:
+ *   1. constructs with codec/sampleRate/channels/mapper from catalog
+ *   2. awaits ready()
+ *   3. calls feedMoqObject(packet) for each arriving MoQ object
+ *   4. between objects, awaits paceSleepMs() to throttle burst delivery
+ *   5. connects rootNode to an AudioContext destination (or a GainNode)
+ *   6. close() on teardown
+ *
+ * Pacing uses FrameCountPacer (anchor on first decoded AU, frame count × AU
+ * duration) with threshold = mapper.pacingThresholdMs — NOT MPU timestamps
+ * (ambiguous under FEC interleave).
+ */
+import { type PipelineFecClient } from '@blockcast/mmt-container/mmtp-decode-pipeline';
+import type { MfuReassemblyPolicy } from '@blockcast/mmt-container/mfu-reassembly-policy';
+import type { FecBlockMapper } from '@blockcast/mmt-fec';
+export interface AudioRenderPipelineOpts {
+    context: AudioContext;
+    codec: string;
+    /** MMTP packet_id for this audio track — from catalog. */
+    packetId?: number;
+    sampleRate: number;
+    channels: number;
+    description?: Uint8Array;
+    mapper: FecBlockMapper;
+    fecClient?: PipelineFecClient;
+    /** Reliable MoQ is clock-free; datagram deadlines come from catalog/in-band timing. */
+    reassemblyPolicy: MfuReassemblyPolicy;
+    /** Catalog-derived PCM startup buffer in milliseconds. */
+    latencyMs: number;
+    /** Exact PCM startup threshold when catalog grouping is codec-aligned. */
+    startupFrames?: number;
+    /** Exact PCM storage ceiling, including one successor delivery burst. */
+    capacityFrames?: number;
+    /** Enable ring-buffer telemetry messages. Default false. */
+    diag?: boolean;
+    /** Called when worklet reports state (playback timestamp + exact buffer accounting). Optional. */
+    onState?: (state: {
+        timestampUs: number | undefined;
+        stalled: boolean;
+        bufferedFrames: number;
+        writtenFrames: number;
+    }) => void;
+    /** Called when worklet reports ring-buffer diag (telemetry). Optional. */
+    onDiag?: (diag: {
+        reads: number;
+        avgFill: number;
+        capacity: number;
+        skipsPerSec: number;
+        dupsPerSec: number;
+        overflowSamplesPerSec: number;
+        underflowSamplesPerSec: number;
+    }) => void;
+    /** Warmup frames before pacing engages. Default 0 (audio tolerates instant pacing). */
+    warmupFrames?: number;
+    /** Audio underflow / config event logger. Optional. */
+    onError?: (err: unknown) => void;
+}
+export declare class AudioRenderPipeline {
+    #private;
+    constructor(opts: AudioRenderPipelineOpts);
+    /** Load worklet module + create decoder + pipeline. Call once. */
+    ready(): Promise<void>;
+    /** Feed one MoQ object (raw MMTP packet). */
+    feedMoqObject(packet: Uint8Array): void;
+    /** ms to sleep before the next object read. 0 when not anchored / not ahead. */
+    paceSleepMs(): number;
+    /** Pacer snapshot (for diagnostics / UI). */
+    get pacerSnapshot(): import("../pacing/frame-pacer.js").PacerSnapshot;
+    /** Reset pacer on source switch (MoQ ↔ multicast). */
+    resetPacer(): void;
+    /** Drop queued PCM and its timestamp epoch at an explicit source boundary. */
+    resetPlayout(startUnstalled?: boolean): void;
+    /** Node to connect to destination / gain. */
+    get rootNode(): AudioNode;
+    close(): void;
+}
+//# sourceMappingURL=audio-render-pipeline.d.ts.map

package/dist/audio/audio-render-pipeline.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"audio-render-pipeline.d.ts","sourceRoot":"","sources":["../../src/audio/audio-render-pipeline.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;GAiBG;AAEH,OAAO,EAAsB,KAAK,iBAAiB,EAAE,MAAM,+CAA+C,CAAA;AAC1G,OAAO,KAAK,EAAE,mBAAmB,EAAE,MAAM,gDAAgD,CAAA;AACzF,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAA;AAYxD,MAAM,WAAW,uBAAuB;IACtC,OAAO,EAAE,YAAY,CAAA;IACrB,KAAK,EAAE,MAAM,CAAA;IACb,0DAA0D;IAC1D,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,UAAU,EAAE,MAAM,CAAA;IAClB,QAAQ,EAAE,MAAM,CAAA;IAChB,WAAW,CAAC,EAAE,UAAU,CAAA;IACxB,MAAM,EAAE,cAAc,CAAA;IACtB,SAAS,CAAC,EAAE,iBAAiB,CAAA;IAC7B,uFAAuF;IACvF,gBAAgB,EAAE,mBAAmB,CAAA;IACrC,0DAA0D;IAC1D,SAAS,EAAE,MAAM,CAAA;IACjB,0EAA0E;IAC1E,aAAa,CAAC,EAAE,MAAM,CAAA;IACtB,yEAAyE;IACzE,cAAc,CAAC,EAAE,MAAM,CAAA;IACvB,4DAA4D;IAC5D,IAAI,CAAC,EAAE,OAAO,CAAA;IACd,kGAAkG;IAClG,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE;QAChB,WAAW,EAAE,MAAM,GAAG,SAAS,CAAA;QAC/B,OAAO,EAAE,OAAO,CAAA;QAChB,cAAc,EAAE,MAAM,CAAA;QACtB,aAAa,EAAE,MAAM,CAAA;KACtB,KAAK,IAAI,CAAA;IACV,0EAA0E;IAC1E,MAAM,CAAC,EAAE,CAAC,IAAI,EAAE;QACd,KAAK,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,QAAQ,EAAE,MAAM,CAAA;QAChD,WAAW,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAC;QAAC,qBAAqB,EAAE,MAAM,CAAA;QACtE,sBAAsB,EAAE,MAAM,CAAA;KAC/B,KAAK,IAAI,CAAA;IACV,uFAAuF;IACvF,YAAY,CAAC,EAAE,MAAM,CAAA;IACrB,uDAAuD;IACvD,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,OAAO,KAAK,IAAI,CAAA;CACjC;AAED,qBAAa,mBAAmB;;gBA6BlB,IAAI,EAAE,uBAAuB;IA+BzC,kEAAkE;IAC5D,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAuG5B,6CAA6C;IAC7C,aAAa,CAAC,MAAM,EAAE,UAAU,GAAG,IAAI;IAIvC,gFAAgF;IAChF,WAAW,IAAI,MAAM;IAIrB,6CAA6C;IAC7C,IAAI,aAAa,qDAEhB;IAED,sDAAsD;IACtD,UAAU,IAAI,IAAI;IAIlB,8EAA8E;IAC9E,YAAY,CAAC,cAAc,UAAQ,GAAG,IAAI;IAa1C,6CAA6C;IAC7C,IAAI,QAAQ,IAAI,SAAS,CAGxB;IAED,KAAK,IAAI,IAAI;CAgCd"}