@displayxr/inline3d 1.23.0 → 1.25.0

package/CHANGELOG.md

@@ -5,6 +5,145 @@ entry points (`.`, `./three`) are frozen for 1.x, while the **scene subpaths** (
 `./splat`, `./model`) are a preview tier whose options may change in any release. Entries below say
 which tier they touch, because that is what tells you whether an upgrade can move your pixels.
 
+## 1.25.0 — 2026-09-25
+
+Touches the **preview tier** (a new subpath, `./player`) and the **core** (`.`, frozen tier): new
+tracking-state API and an opt-in `untrackedFallback`. No pixel change for a page that does not opt
+in, and none on a browser without the tracking-state surface.
+
+### Fixed — `./splat` (`engine: 'playcanvas'`), `setLayerRig`
+
+Reported on the panel: layers set to the display rig still looked flat, with no way to tell why. 1.23
+could fail silently; now it cannot:
+
+- **Works on both view paths.** The N-camera fallback (one camera per view) used to leave the layer
+  on the photo rig with a single warning at the call. It now gets one display camera per view, the
+  shear carried in its projection, and measures identical to the RenderView path.
+- **`layerRigState()` says what happened**: `path` ('renderviews' | 'ncamera' | 'mono'), `engaged`
+  (the display-rig views were applied on the last frame), `reason` when not (e.g. a layer drawn by
+  another camera, a layer not in the composition), `viewerDistance`, `gain`, `planeM`,
+  `photoConvergenceM`, `planeOffset`, `located`. The same line is WARNed on the first 3D frame and on
+  every change.
+- A display layer that no camera draws is now drawn by the display camera. A layer another camera
+  draws is named, not taken.
+- **`planeOffset` / `planeDistance`** choose which stage plane lands on the glass (a stage whose
+  "z = 0" is not the photo's convergence plane read as recessed). The change is exact, and the 2D
+  picture does not move. **`handle.setLayerRigOptions(opts)`** changes these options live, and
+  options now merge across calls (`null` clears a key).
+
+### Added — core
+
+- **Tracking state: `wall.trackingState` and `on('trackingstatechange', (state, ev) => …)`**, on
+  every tile handle too (`'tracking' | 'searching' | 'unknown'`), mirroring the browser's
+  `session.trackingState` (DisplayXR Browser patch 0195; browser-pvt#161). Read off the session at
+  construction and on every frame, emitted on change only, released to `'unknown'` when the
+  session ends. The unsupported shape carries `trackingState: 'unknown'`. A browser without the
+  surface (1.0.5 and earlier) reports `'unknown'`, silently. Pages use it for "step back into
+  view" hints. (API and tests from #35.)
+- **`createInline3D({ untrackedFallback: 'none' | 'mono' })`**, default `'none'`. `'mono'` eases
+  every image/video window to its left eye in both halves of its SBS buffer on `'searching'`, and
+  back on `'tracking'`, over the mode switch's duration; `'unknown'` leaves it where it is. No
+  buffer reallocation and no layer close, so there is no squeezed-pair flash. Scene windows are
+  never touched. It is for **MANUAL** eye-tracking displays, where the app handles tracking loss;
+  on **MANAGED** displays (the default, and Leia's) the vendor already goes 2D before it reports
+  `'searching'`, so leave it off. Docs: authoring guide § Knowing when nobody is tracked. RFC
+  0001 and the player header say where `untrackedFallback` lives again.
+
+### Added — player
+
+- **`addPlayer(wall, canvas, src, opts)` — a media player module (preview tier).** Built on
+  `addVideo`: creates and owns a hidden `<video>`, and for `format:'sbs'` (default) hands it
+  straight to `wall.addVideo()`, so the stereo paint, the SBS buffer contract and the
+  display-mode fallback are inherited rather than re-implemented. Adds real transport
+  (`play/pause/seek/currentTime/duration/volume/muted/setSource/remove`), an `on/off` event pair
+  (`play|pause|ended|timeupdate|ready|error`), an SDK-drawn partial-region transport bar
+  (`data-inline3d-overlay`, auto-hiding, keyboard-accessible), and a keyboard map
+  (Space/K/←/→/J/L/M) scoped to the canvas/transport. `format:'mono'` paints genuinely flat
+  content full-frame on a small paint loop of its own — see `js/inline3d-player.js`'s module doc
+  comment for why this is a different code path from the unsupported-`wall` fallback, and why
+  neither one is the RFC's `untrackedFallback`/`trackingState` mechanism, which does not exist in
+  this codebase. v1 is narrower than `docs/rfcs/0001-media-player.md`: no `'tb'`/`'auto'` format
+  detection, no `opts.group` one-active-player policy, no `opts.fit`/letterboxing. Sample:
+  `samples/player/`.
+- **`setSource()` transitions in `./splat`'s vocabulary.** `transition: 'cut' | 'crossfade'`,
+  `durationMs` (600), `easing` (the same named curves, `'easeInOutSine'`), `outgoing: 'frozen'`,
+  and `fadeMs` as the legacy alias. Per-call options override the player's own field by field.
+  The crossfade dissolves from the outgoing title's last frame through an opt-in mixer canvas (a
+  player built without one keeps the plain `addVideo` path). `./splat`'s other transitions,
+  unknown easings and `outgoing: 'live'` throw by name.
+- **Transport: title line, ±10 s skip, fullscreen.** `title` (a now-playing line in a bounded top
+  band, its own overlay, fading with the bar; `setSource(src, { title })` changes it),
+  `skipButtons` (default on, plain rewind / fast-forward glyphs, hidden under 420 px) and `fullscreen` (default on, plus `F`): the
+  canvas's container goes fullscreen so the transport comes along, and the canvas is sized to fit
+  at its own aspect — never `object-fit`, which would letterbox inside the woven rect. The sample
+  gains chapter chips (real `setSource()` crossfades) and an honest caption for the mono tile.
+- **`skin: 'dock'`** — a floating rounded dock, lit round buttons, an accent orb for play, a glowing
+  scrub fill and knob, a title pill. `'classic'` stays the default. Every glow is inside its
+  overlay's own box: a shadow outside the excluded rect would be woven with the canvas. The sample
+  uses `'dock'` (`?skin=classic` for the band) and stacks its tiles at 90% of the window.
+- **Developer customisation: `size: 's' | 'm' | 'l'` and `handle.setAppearance({ accent, size, skin })`**
+  (live re-skin). Size is CSS `zoom` on each overlay's content, so icons, fonts and hit targets
+  scale together and the centre badge stays centred (measured at all three). Round controls draw
+  their own feathered edge (a 1.6 px mask) instead of relying on the border-radius clip, which can
+  stair-step where an overlay is re-composited on its own. The sample gains a developer bar
+  (accent picker, S/M/L, skin) showing the exact `setAppearance` call.
+- **Docs: the overlay no-glow rule and three media traps** (codec: VP9/AV1 + Opus WebM only, no
+  H.264/AAC; non-square-pixel sources play squashed, rewrap with square pixels; black picture with
+  working sound = an opaque page background over the woven canvas), and a note that `addPlayer`
+  shares `./splat` `setVideo`'s option names (`format`, later `fit`).
+- **`group`: one decoding player at a time.** Players sharing a group take turns: a `play` pauses
+  every other member, so a shelf of muted, looping previews decodes one video at a time. Leaving
+  on `remove()`.
+- **Controls inside the black bars, and skins `'bars'` / `'call'`** (the Show Spatial team's
+  layouts). With a `band`, when the bottom bar is at least 44 CSS px (times `size`), every skin
+  puts its controls in it and the title in the top bar; the centre button and key pip stand down,
+  so nothing sits over the picture; a thinner bar falls back to the overlay. Re-measured on
+  resize. `'bars'`: back / play / elapsed / scrub / remaining in one row (a solid bottom strip
+  without a band). `'call'`: name + timer bar on top, controls at the bottom, band 2.39 by default.
+- **Playlist API** (RFC 0001 Addendum A4, for the Show Spatial team's voice commands): `titles`
+  (`[{ id, src, title?, poster? }]`, bare sources accepted), `loopList`, `autoAdvance`; on the
+  handle `titles`, `current`, `setTitles()`, `play(id)`, `toggle()`, `next()`, `back()` (restart
+  past 3 s, else the previous title), and a `'titlechange'` event. Switches go through
+  `setSource`, so transitions, the title line and posters follow. `normalizeTitles`,
+  `nextIndex` and `backTarget` are pure and tested.
+- **For the Show Spatial team** (quick wins from their list): `format: 'tb'` (top/bottom, repacked
+  into the woven pair; `setVideo`'s vocabulary); `band` — a letterbox slot of a target aspect
+  (`2.39`, `'2.39:1'`) the picture is fitted into; `posterFormat: 'sbs' | 'tb'` — stereo poster
+  stills painted eye by eye; `on('ended')` also fires once if the clip had already ended when the
+  listener was attached; `src` / `setSource` take candidates best first and `pickSource()` plays
+  the first the browser supports by full `canPlayType` string. Docs: woven-canvas rule 9 (nothing
+  with a background above the woven canvas) and a Hosting video section (codec, range requests,
+  cache headers, CORS, square pixels).
+- **`fit: 'contain' | 'cover'`** — `./splat` `setVideo`'s names and meaning: contain letterboxes each
+  eye (transparent bars), cover fills the tile and crops. Unset keeps the stretch-to-tile pixels.
+  On the woven path a small per-eye fit canvas sits after the crossfade mixer (one extra draw per
+  frame, only when set); the flat paths fit directly. `fitRect()` is exported and tested.
+- **Named accents** — `accent: 'azure' | 'violet' | 'magenta' | 'sunset' | 'amber' | 'lime' | 'mint' |
+  'ice'` (or any CSS colour), exported as `PLAYER_ACCENTS`. Each is light enough for the dock's dark
+  play glyph. The sample shows them as swatches.
+
+## 1.24.0 — 2026-09-25
+
+Touches the **preview tier** (`./splat`, `engine: 'playcanvas'` only). No API change. Pixels change
+during a live transition between two camera-rig photos (they were wrong), and for the frames in
+which a newly declared rig has not yet reached the runtime's views.
+
+- **No camera jump at the start of a live transition.** Crossfade, wavefront and the particle
+  transitions keep the outgoing photo alive next to the incoming one. Between two photos with
+  camera rigs (SOG camera blocks: each its own convergence and vertical FOV), the outgoing photo
+  was drawn from eyes scaled by the two rigs' window ratio from the swap on, a jump in its
+  disparity and head parallax. For the frame(s) before the declaration reached the views, both
+  photos were drawn through the wrong rig. Each photo is now drawn through its own rig for the whole
+  window. The SDK reads off the views which declared rig they were located for, and remaps them
+  exactly (window onto window, eye onto eye; the runtime's projection verbatim, no Kooima) where
+  they are not the photo's own. `reassemble` was never affected. Headless: 102–208 px of
+  reference-point error → < 0.005 px, and the live outgoing image equals the frame before the swap
+  (MAE 0.000). Docs: `docs/playcanvas-adapter.md` § Each photo through its own rig.
+- **Kill switch `?dxrdiag=oldrig`** (the 1.23 behaviour). The diag's per-frame record gains
+  `rigAt` / `rigIn` / `rigOut`: the rig the views were located for, the rig the current photo was
+  drawn through (`+` = remapped), and how the live outgoing photo was drawn.
+- `tools/rig-swap-capture`: the headless harness behind the numbers.
+
 ## 1.23.0 — 2026-09-25
 
 Touches the **preview tier** (`./splat`, `engine: 'playcanvas'` only). New API; no pixel change for a

package/README.md

@@ -27,6 +27,8 @@ import { addSplat } from '@displayxr/inline3d/splat';               // experimen
 import { addModel } from '@displayxr/inline3d/model';               // experimental: glTF/GLB in a tile
                                                                     //   (Draco / meshopt / KTX2 too —
                                                                     //    you serve the decoder files)
+import { addPlayer } from '@displayxr/inline3d/player';             // preview: a media player, built
+                                                                    //   on addVideo, with transport
 import { SceneViewer } from '@displayxr/inline3d/viewer';           // experimental: framing + orbit
 ```
 
@@ -147,6 +149,15 @@ No projection math lands in your page or in the SDK — the off-axis frustum sta
 > OS weave service has bound (typically at page load), a false-negative that silently drops you to 2D.
 > `createInline3D()` detects by actually acquiring a session, which is authoritative.
 
+> **Nobody in front of the display?** `wall.trackingState` is `'tracking'`, `'searching'` or
+> `'unknown'`, and `wall.on('trackingstatechange', state => …)` tells you when it moves (tile
+> handles carry both too). `'searching'` means nobody is being tracked: the viewer left the
+> display's 3D zone, or the panel is in a 2D mode. On a MANAGED display (the default, and Leia's)
+> the vendor has already gone 2D by then, so use it for hints only. On a MANUAL display,
+> `createInline3D({ untrackedFallback: 'mono' })` eases `addImage` / `addVideo` windows flat for you.
+> `'unknown'` is "no opinion" (an older browser, or a closed session): render normally. See
+> [knowing when nobody is tracked](docs/authoring-inline-3d.md#knowing-when-nobody-is-tracked).
+
 Full API + authoring guidance: [`docs/authoring-inline-3d.md`](docs/authoring-inline-3d.md).
 Before you ship a page that navigates or remounts, read
 [`docs/woven-canvas-rules.md`](docs/woven-canvas-rules.md): how to avoid a raw side-by-side
@@ -164,6 +175,8 @@ samples/
                       each woven with one SDK call, all on one session
   splat/              a 3D Gaussian splat in a tile (PlayCanvas; ?engine=spark), with a 2D plate over it
   model/              a glTF mesh, a mesh+splat scene, and a Draco-COMPRESSED glTF (PlayCanvas; ?engine=three)
+  player/             a media player — SDK transport (play/pause/seek/mute), one call, built on
+                      addVideo; default (SBS) and format:'mono' side by side
   composition/        the 14-case 2D/3D overlap matrix — demo AND standing hardware regression
                       surface; red cases ship red (see samples/README.md)
 vendor/draco/         three's Draco decoder, served for samples/model (compressed glTF needs it)
@@ -185,6 +198,8 @@ js/
   inline3d-splat-effects.js     splat shader effects: the runner, the effect registry (PlayCanvas)
   inline3d-splat-live.js        setSource's live outgoing photo: its own camera, layer and target (PlayCanvas)
                       from what the asset declares (you serve the decoder files — see the guide)
+  inline3d-player.js  preview: addPlayer() — a media player window built on addVideo (stereo
+                      paint, display modes and the fallback are inherited) + an SDK transport bar
 docs/
   authoring-inline-3d.md   the authoring guide
   authoring-motion-and-effects.md

package/index.d.ts

@@ -157,6 +157,24 @@ export interface HardwareDisplayStateChange {
 
 export type DisplayModeChange = RenderingModeChange | HardwareDisplayStateChange;
 
+/**
+ * Is anyone being tracked in front of the display? From the DisplayXR runtime (which may hand the
+ * decision to the vendor plug-in), via the browser's `session.trackingState` (patch 0195).
+ *
+ * - `'tracking'` — a viewer is in the display's 3D zone.
+ * - `'searching'` — the runtime's derived isTracking is FALSE: nobody is in the zone, **or** the
+ *   display is in an untracked / 2D mode. It does **not** necessarily mean the tracker lost lock
+ *   on a viewer who is still there, so do not word a page's UI as if it did.
+ * - `'unknown'` — no opinion: this browser has no tracking-state surface, or the session ended.
+ */
+export type XRTrackingState = 'unknown' | 'tracking' | 'searching';
+
+/** The tracking state changed. The callback is handed the STATE first, this object second. */
+export interface TrackingStateChange {
+  type: 'trackingstatechange';
+  state: XRTrackingState;
+}
+
 /** What a page may lift into the floating native viewer. `null` = no `XRDisplayLayer.undock`. */
 export interface UndockCapabilities {
   model: boolean;
@@ -314,9 +332,17 @@ export interface TileHandle {
   /** Listen for one display event, re-emitted on this handle. Returns an unsubscribe function. */
   on(type: 'renderingmodechange', cb: (e: RenderingModeChange) => void): () => void;
   on(type: 'hardwaredisplaystatechange', cb: (e: HardwareDisplayStateChange) => void): () => void;
+  /**
+   * Nobody-is-tracked changes, with the STATE as the first argument. The same subscription as the
+   * manager's; silent forever on a browser without the surface. See {@link Inline3D.trackingState}.
+   */
+  on(type: 'trackingstatechange', cb: (state: XRTrackingState, e: TrackingStateChange) => void): () => void;
   /** Drop a listener registered with {@link TileHandle.on}. */
   off(type: 'renderingmodechange', cb: (e: RenderingModeChange) => void): void;
   off(type: 'hardwaredisplaystatechange', cb: (e: HardwareDisplayStateChange) => void): void;
+  off(type: 'trackingstatechange', cb: (state: XRTrackingState, e: TrackingStateChange) => void): void;
+  /** The manager's {@link Inline3D.trackingState}, read live. */
+  readonly trackingState: XRTrackingState;
   /**
    * Both display events through one callback — the older shape, still supported. Returns an
    * unsubscribe function; inert (a no-op unsubscribe) on a browser without the API.
@@ -367,10 +393,19 @@ export interface Inline3D {
   setStereoEnabled(enabled: boolean): Promise<boolean>;
   on(type: 'renderingmodechange', cb: (e: RenderingModeChange) => void): () => void;
   on(type: 'hardwaredisplaystatechange', cb: (e: HardwareDisplayStateChange) => void): () => void;
+  on(type: 'trackingstatechange', cb: (state: XRTrackingState, e: TrackingStateChange) => void): () => void;
   off(type: 'renderingmodechange', cb: (e: RenderingModeChange) => void): void;
   off(type: 'hardwaredisplaystatechange', cb: (e: HardwareDisplayStateChange) => void): void;
+  off(type: 'trackingstatechange', cb: (state: XRTrackingState, e: TrackingStateChange) => void): void;
   /** As last REPORTED by `hardwaredisplaystatechange` — never what was last requested. */
   readonly hardwareDisplayState: XRHardwareDisplayMode | null;
+  /**
+   * Is anyone being tracked in front of the display right now? Mirrors `session.trackingState` as
+   * last read — `'unknown'` where the browser has no such surface (silently), and again once the
+   * session ends. Pages use it for "step back into view" hints; see
+   * {@link CreateInline3DOptions.untrackedFallback} for having the SDK flatten image/video tiles.
+   */
+  readonly trackingState: XRTrackingState;
   /** The active mode as last read/reported. `viewCount: 0` means "not read yet". */
   readonly activeMode: { modeIndex: number; viewCount: number };
   /** True while the SDK is holding every window's rig flat because a 1-view mode is active. */
@@ -441,6 +476,8 @@ export interface Inline3D {
 /** The shape {@link createInline3D} resolves to when inline-3D is unavailable. */
 export interface Inline3DUnsupported {
   supported: false;
+  /** Always `'unknown'`: no session, nobody to track. Present so a page need not branch first. */
+  trackingState: 'unknown';
   error?: unknown;
 }
 
@@ -470,6 +507,26 @@ export interface CreateInline3DOptions {
   autoChrome?: boolean;
   /** The eased 2D<->3D transition. On by default; see {@link ModeSwitchOptions}. */
   modeSwitch?: ModeSwitchOptions;
+  /**
+   * What the SDK does to the image and video windows it owns the pixels of while **nobody is
+   * tracked** (`trackingState === 'searching'`). Default `'none'`.
+   *
+   * Whose job this is depends on the display's eye-tracking mode:
+   * - **MANAGED** (the default, and Leia's): the vendor owns tracking loss. It eases the eyes
+   *   together and reports `'searching'` only once the display is already 2D. Leave this at
+   *   `'none'`: a flatten here would be a second transition.
+   * - **MANUAL**: the vendor does nothing and the app handles tracking loss. Use `'mono'`.
+   *
+   * - `'none'` — nothing changes.
+   * - `'mono'` — each `addImage` / `addVideo` window eases to its LEFT eye in both halves of its
+   *   side-by-side buffer on `'searching'`, and back on `'tracking'`, over the mode switch's
+   *   duration. The buffer is never reallocated and the layer never closed; `'unknown'` leaves it
+   *   where it is.
+   *
+   * **Scene windows are never touched** — the page owns those pixels. Listen for
+   * `trackingstatechange` and do the same for a scene.
+   */
+  untrackedFallback?: 'none' | 'mono';
 }
 
 /**