@spatius/avatarkit 1.3.8 → 1.3.10-beta.1

package/CHANGELOG.md

@@ -5,6 +5,59 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.10-beta.1] - 2026-09-21
+
+### Added
+- `AvatarViewOptions.audioFormat`: the audio format is now set per view, `new AvatarView(avatar, container, { audioFormat })`, instead of once for the whole page in `AvatarSDK.initialize()`. Two views on one page can run different sample rates. Defaults to `{ channelCount: 1, sampleRate: 16000 }` as before; `inputAudioFormat`, `opusBitrate` and `opusUplinkEnabled` move with it and keep their meaning.
+- `controller.setAudioFormat(patch)`: change the audio format of a live view without tearing the SDK down. It is a partial update — `{ sampleRate?, opusBitrate?, opusUplinkEnabled?, inputAudioFormat? }`, fields left out keep their current value. Until now the only way to change any of these was `AvatarSDK.cleanup()` followed by `initialize()`, which also released the avatar, cleared the asset cache and rebuilt the rendering runtime, so the avatar disappeared from the page. Whatever field changes, the call interrupts whatever is playing, discards the audio context and, in Direct Mode, disconnects (reported through `onConnectionState` as `disconnected`); the host must then call `controller.initializeAudioContext()` again from a user gesture before the next `send()` (browsers require a gesture to create an audio context, and its sample rate cannot be changed once created), and audio fed after the call is interpreted in the new format. Switching `inputAudioFormat` to `'opus'` pins the sample rate at 48000; switching back to `'pcm'` keeps the current rate unless a `sampleRate` is passed in the same call. Accepted sample rates are 8000, 16000, 22050, 24000, 32000, 44100 and 48000; an unsupported rate, or a `sampleRate` paired with Opus input, logs an error and leaves the format untouched. `controller.audioFormat` reads the format currently in effect. Same API on the Android and iOS SDKs.
+
+### Removed
+- **Breaking**: `Configuration.audioFormat` (the field passed to `AvatarSDK.initialize()`) and `AvatarSDK.getAudioFormat()`. Pass the format to `new AvatarView(...)` instead (see Added above). The field no longer exists on the `Configuration` type, so TypeScript hosts get a compile error pointing at the line to move. No deprecation shim is kept: this is a reorganisation of the initialization options rather than a renamed method, and `getAudioFormat()` was marked internal. See ADR-0008.
+- The SDK no longer reports telemetry to PostHog. The `posthog-js` dependency is gone, and the SDK no longer opens a connection to `i.spatialwalk.ai` from the host page. Internal events continue to be reported through the existing OpenTelemetry channel. No change to public API.
+- The SDK no longer uploads its locally stored diagnostic logs to `hogtool.spatialwalk.ai`. That upload was only ever triggered by a PostHog feature flag, which no longer exists. Logs are still kept locally in IndexedDB (`avatarkit_logs`) for offline diagnosis.
+
+### Fixed
+- With Opus uplink enabled (the default in Direct Mode), a round whose connection dropped *while* a chunk was still being encoded went silent: no sound, no motion, and no fallback. The disconnect check happened before encoding, so a chunk that entered the encoder while connected was simply never sent once the socket closed, and the round waited forever for animation that would not come. The SDK now notices the failed send and falls back the same way it does when the connection is already down at `send()` time: the round plays through as audio only with locally generated animation. PCM uplink was not affected.
+
+### Changed
+- `AvatarView.dispose()` now reports `disconnected` through `onConnectionState` when the driving connection was still open at that moment, as `controller.close()` already did and as the Android SDK does. A host that keys UI off the connection state no longer sees it stuck at `connected` after disposing the view.
+- Internal telemetry only, no change to public API or runtime behaviour: the per-round playback report now includes how many uplink writes were attempted and how many failed, so the Direct Mode uplink success rate can be computed. Rounds in Backend Mode, or with no uplink at all, do not report these.
+
+## [1.3.9] - 2026-09-16
+
+### Added
+- `AvatarSDK.isDeviceSupported()` now checks whether a rendering context can actually be created before running the benchmark, and returns `false` immediately when neither WebGL2 nor WebGPU is available. Previously it always returned `true`: a device whose browser had hardware acceleration switched off passed the check, loaded the avatar, and then silently failed to draw anything. The check adds no measurable cost.
+- Two new `ErrorCode` values: `renderNotSupported` (no WebGL2 or WebGPU — typically hardware acceleration disabled or the GPU driver blocklisted; deterministic, tell the user rather than retry) and `renderInitFailed` (any other failure while setting up the renderer). A host that listens for errors on the view can now tell "this machine can't draw" apart from "something else went wrong".
+
+### Changed
+- The error thrown when the WebAssembly module cannot be loaded now says which URL was tried and links to the toolchain setup page (`https://docs.spatius.ai/sdk-reference/web-sdk/toolchain`). Nearly every such failure is a build tool inlining or relocating `avatar_core_wasm.wasm`; the previous message only carried the runtime's own "both async and sync fetching of the wasm failed", which named neither the path nor the fix.
+- **Behaviour change**: the SDK no longer reconnects the driving-service WebSocket on its own. Until now the Web SDK silently retried up to 5 times with exponential backoff on almost every disconnect — including abnormal closures (1006), server-initiated normal closures and server rejections it did not recognise — while the Android and iOS SDKs never retried. The server drops connections without a close frame when it intentionally ends a session, so a client cannot tell "the server ended this" from "the network dropped", and the automatic retries turned every intentional disconnect into a burst of failed connections. All three SDKs now behave the same way: on disconnect `onConnectionState` reports `disconnected` (or `failed`), the conversation ends, and the host decides whether to reconnect — typically by calling `controller.start()` again after checking the error reported via `onError`. Hosts that relied on the SDK reconnecting by itself must add that call. See ADR-0006.
+- Internal telemetry only, no change to public API or runtime behaviour: metrics no longer emit a same-named log record alongside every data point. Until now each metric (`fetch_avatar_latency`, `ws_connect_latency`, `time_calibrated`, `template_resources_load_measure`, …) also wrote a log line that was ~95% repeated resource fields; the number itself lives in the metric. Where a metric needs high-cardinality detail (avatar / conversation / trace ids, failure reasons) the SDK now sends one explicit detail log record instead. **Consequence for anyone querying the log stream by these event names**: the success side of `time_calibrated`, `template_resources_load_measure` and `ws_connect_latency` no longer appears in logs at all — only failure-side records remain — so success rates for these must be computed from the metric, not from logs. `fetch_avatar_latency`, `fetch_avatar_metadata_latency` and `download_avatar_assets_latency` keep a detail record on both sides.
+- Internal telemetry only, no change to public API or runtime behaviour: server error messages received over the driving connection are rate-limited per error code (first occurrence always sent, then at most one per minute per code), so a server that answers every message with an error cannot flood the pipeline.
+- Internal telemetry only, no change to public API or runtime behaviour: audio send failures now record whether the audio was PCM or Opus, matching the Android and iOS SDKs.
+- Internal telemetry only, no change to public API or runtime behaviour: a failure anywhere in `AvatarView` initialization — loading the character, setting up the renderer, drawing the first frame — is now reported with which stage failed and why. Until now the only trace of such a failure was a browser-console line, so a device that could not render at all showed up in our data as a perfectly healthy session that never played anything. This is the gap that produced the 2026-09-03 "avatar never appears" report.
+- Internal telemetry only, no change to public API or runtime behaviour: every session now reports once whether WebGL2 and WebGPU are available on that device, so environments that cannot render can be counted by browser and OS rather than found one support ticket at a time.
+
+- Internal telemetry only, no change to public API or runtime behaviour: failures that until now were only written to the browser console are reported as structured diagnostics — playback failing to start, the render loop stopping on an error, Opus input decoding, audio-context state changes, renderer blit, WebSocket message handling and reconnects, server error messages, and WebAssembly runtime errors. Diagnostics raised on per-frame or per-packet paths are rate-limited to one per minute per kind, with the number of suppressed occurrences attached.
+- Internal telemetry only, no change to public API or runtime behaviour: server error messages received over the driving connection were never reported (the handler that would have done so was unreachable). They are now.
+- Internal telemetry only, no change to public API or runtime behaviour: the SDK now records when the host tears it down — `AvatarView.dispose()`, `controller.close()` and a host-initiated `controller.interrupt()` each report what was playing at that moment, how long since the last disconnect, and the calling code — and when the avatar vanishes without any of those being called: the canvas being removed from the DOM while the view is still alive, the container collapsing to zero size, and the page being unloaded mid-playback. Until now an avatar disappearing mid-sentence looked identical in our data whether the host destroyed it, the page reloaded, or the renderer failed; this is the gap that made the 2026-09-16 "avatar vanishes while speaking" report take a week of log correlation to attribute.
+- Internal telemetry only, no change to public API or runtime behaviour: an abnormal WebSocket closure (code 1006) is now reported once instead of twice, and audio-send failures after a disconnect carry the id of the connection that just dropped, so both can be matched to the server-side session log.
+
+### Deprecated
+- `AvatarView.getCameraConfig()` and `AvatarView.updateCameraConfig()`. The camera is defined by the avatar's own settings, which the rendering baselines are calibrated against; overriding it from the host produces a view that no longer matches those baselines. Both methods still work in this release and log a one-time console warning when called; they will be removed in the next release. There is no replacement: hosts that were adjusting `fov` or the camera position through these methods should remove that code.
+
+### Removed
+- `ErrorCode.appIDUnrecognized`, `ErrorCode.websocketClosedAbnormally` and `ErrorCode.audioOnlyInitFailed`. None of the three could ever be produced: the service has no separate signal for an unknown app ID (it surfaces as a session-token error), close code 1006 is reported as `websocketClosedUnexpected` like every other unexpected close, and the audio-only path never raised its own error. Code that switched on them can drop those branches.
+- `AvatarView.playTransitionFromProtobuf()` and `AvatarView.playTransitionToIdleFromProtobuf()`. Both were superseded by `generateTransitionToFrame()` / `generateTransitionToIdle()`, which hand the transition frames back to the host to play at its own cadence; the removed methods played them internally at a fixed 25 fps and had no callers.
+- `extractResourceUrls()` and the `ResourceType` enum from the character types. Neither was ever used, and the URL set they produced no longer matches the current asset schema.
+- `isClockCalibrated()` from the `@spatius/avatarkit/internal-telemetry` entry, which exists only as a bridge for the companion RTC SDK; no RTC release ever used it. (`clientContextFields()` on the same entry is kept: RTC 1.1.3, still deployed, imports it.)
+
+### Fixed
+- After a driving-service connection was closed before it finished opening (handshake failure, or the server dropping the socket before confirming the session), calling `controller.start()` again silently did nothing: the SDK still believed a connection was in progress and returned without opening a socket, raising an error or changing the connection state, and only `controller.close()` could unblock it. The pending connection attempt is now settled and the "connecting" flag cleared as soon as the socket closes, so `start()` opens a new connection as documented. This mattered little while the SDK retried on its own; now that reconnection is the host's responsibility it would have left the host with no way to recover.
+- `ErrorCode.failedToDownloadAvatarAssets` is now actually raised when a required avatar asset fails to download or the download completes incomplete. Previously the raw network error (or a plain `Error`) was thrown instead, so the code existed but never reached a host.
+- Internal telemetry only, no change to public API or runtime behaviour: the blank-canvas diagnostic introduced in 1.3.8 was being sent to the product-analytics channel only and never reached the log backend, so it could not be queried or alerted on. It now goes through the same path as every other diagnostic.
+- Internal telemetry only, no change to public API or runtime behaviour: the blank-canvas check now reports on every view once the first frame has been sampled, with a `blank` field saying whether anything was drawn, instead of reporting only the blank case. Until now the blank-screen rate had no denominator — only failures were counted, so it could not be expressed as a rate. Matches the Android and iOS SDKs.
+
 ## [1.3.8] - 2026-08-31
 
 ### Added