@furious.luke/argus-js 0.3.1 → 0.5.0

package/README.md

@@ -1,6 +1,6 @@
 # @furious.luke/argus-js
 
-Browser client for the [Argus](https://github.com/furious-luke/go-projects/tree/main/argus) video streaming platform. It publishes a `MediaStream` (camera, screen share, etc.) from a browser to an Argus media server over WebRTC, handling gateway selection, the signaling handshake, and ICE negotiation for you.
+Browser client for [Argus](https://github.com/furious-luke/go-projects/tree/main/argus) WebRTC sessions. It establishes the reliable agent text channel and optionally publishes camera, screen, or microphone media, handling gateway selection, signaling, and ICE negotiation for you.
 
 Zero runtime dependencies — it uses only the browser's built-in `WebSocket` and `RTCPeerConnection`.
 
@@ -9,8 +9,11 @@ Zero runtime dependencies — it uses only the browser's built-in `WebSocket` an
 Argus is a distributed video ingestion service. A stream flows through three parties:
 
 - **Your server** mints a short-lived **join token** using its secret API key (see the [Go client](https://github.com/furious-luke/go-projects/tree/main/argus/client), published as the `github.com/furious-luke/argus-go` module, or the control-plane HTTP API).
-- **The browser** (this library) uses that token to publish video to the nearest Argus media server. It never sees the API key.
-- **Your server** later fetches frames from the stream on demand.
+- **The browser** (this library) uses that token to establish a text-only or media-backed session with the nearest Argus media server. It never sees the API key.
+- **The browser** relays the winning gateway URL and its region-scoped read token
+  back to your server. The URL routes both frame reads and notifications to the
+  selected region; the read token authorizes frame reads only, while notifications
+  use the server-held control token.
 
 This library is only the browser publishing half. It deliberately does **not** talk to the control plane or mint tokens — that requires your secret API key and must stay server-side.
 
@@ -20,7 +23,8 @@ This library is only the browser publishing half. It deliberately does **not** t
 │          │ ◀─────────────────────── │            │ ◀── token + gateway_urls ──
 │ argus-js │  2. token + gateway_urls └────────────┘
 │          │
-│          │  3. publish WebRTC video ────────────────▶ Argus media server
+│          │  3. WebRTC text + optional media ────────▶ Argus media server
+│          │  4. frameReadToken + selectedGatewayURL ─▶ your server
 └──────────┘
 ```
 
@@ -42,24 +46,50 @@ const publisher = new Publisher({
   gatewayURLs: join.gateway_urls,
   token: join.token,
   callbacks: {
-    onConnected: () => console.log("streaming live"),
+    onConnected: async () => {
+      console.log("streaming live");
+      // Relay these as a pair; the read token is valid in the selected region.
+      await fetch("/api/stream-credentials", {
+        method: "POST",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify({
+          stream_id: join.stream_id,
+          read_token: publisher.frameReadToken,
+          gateway_url: publisher.selectedGatewayURL,
+        }),
+      });
+    },
     onConnectionStateChange: (state) => console.log("state:", state),
     onRecoveryStateChange: (event) => console.log("media recovery:", event),
     // Show a button or prompt. From its click handler, call captureScreen()
-    // and then publisher.replaceStream(newStream).
-    onRecoveryRequired: (event) => console.warn("screen share must be restarted", event),
+    // and then publisher.publish(newStream, "screen").
+    onRecoveryRequired: (event) => console.warn(`${event.track} must be restarted`, event),
     onError: (err) => console.error("publish error:", err),
   },
 });
 
 // captureCamera() applies sensible capture defaults; any MediaStream works too.
+// The second argument labels the track type (defaults to "camera").
 const stream = await captureCamera();
-await publisher.start(stream);
+await publisher.start(stream, "camera");
+```
+
+For a typed agent that needs no initial media or capture permission, start the
+same WebRTC session with only its ordered text data channel:
 
-// After start() resolves, this token lets your server fetch frames for the stream.
-console.log("read token:", publisher.frameReadToken);
+```ts
+await publisher.startTextOnly();
+publisher.sendUserText(crypto.randomUUID(), "Hello");
 ```
 
+Camera, screen, microphone, and the optional inbound speech track can still be
+added later through `publish`, `publishMicrophone`, and `enableSpeech`.
+
+Each published track is labelled with a **track type** (`"camera"` or
+`"screen"`). The label is declared to Argus during signaling, so frame reads and
+change-notification subscriptions can address a specific track by type. A stream
+may carry one track of each type at a time.
+
 ### Capture helpers
 
 `captureCamera()` and `captureScreen()` wrap `getUserMedia` / `getDisplayMedia` with defaults tuned for streaming — a capped resolution and modest frame rate, audio off. They matter most for screen sharing: on HiDPI/Retina displays a raw `getDisplayMedia` captures at native resolution (often 3456px+ / effectively 4k), wasting upload bandwidth and downstream decode for no benefit. `captureScreen()` caps the width instead.
@@ -68,18 +98,34 @@ console.log("read token:", publisher.frameReadToken);
 import { captureScreen } from "@furious.luke/argus-js";
 
 const stream = await captureScreen();
-await publisher.start(stream);
+await publisher.start(stream, "screen");
 ```
 
 Both accept overrides (shallow-merged over the defaults), and any `MediaStream` you build yourself still works if you'd rather manage constraints directly.
 
-### Switching sources without reconnecting
+### Adding and removing tracks live
 
-`replaceStream` renegotiates in place, e.g. to toggle between camera and screen:
+Start with one track, then add or remove others without reconnecting. `publish`
+adds a track of a given type (renegotiating in place); `unpublish` removes it.
+Each supplied stream must contain exactly one video track. Publishing a second
+track of a type replaces the first.
 
 ```ts
+// Live camera already publishing from start(stream, "camera")...
 const screen = await captureScreen();
-await publisher.replaceStream(screen);
+await publisher.publish(screen, "screen"); // now streaming camera + screen
+
+// Later, stop sharing the screen but keep the camera live.
+await publisher.unpublish("screen");
+```
+
+`replaceStream(stream, type?)` remains available as a convenience — it delegates
+to `publish` and defaults to the `"camera"` type. It is handy from
+`onRecoveryRequired` to swap in a freshly reacquired screen share:
+
+```ts
+const screen = await captureScreen();
+await publisher.replaceStream(screen, "screen");
 ```
 
 ### Stopping
@@ -90,13 +136,50 @@ publisher.stop(); // stops all tracks and tears down the peer connection
 
 ## API
 
+### Assistant speech and text
+
+Every publisher creates an ordered, reliable `argus.text` data channel. Enable
+the persistent outbound speech track only after the user opts in:
+
+```ts
+const publisher = new Publisher({
+  gatewayURLs: join.gateway_urls,
+  token: join.token,
+  callbacks: {
+    onSpeechTrack(track) {
+      const audio = new Audio();
+      audio.srcObject = new MediaStream([track]);
+      void audio.play();
+    },
+    onAssistantText({ utteranceId, text }) {
+      renderCaption(utteranceId, text);
+    },
+    onUserTextResult(result) {
+      console.log(result.messageId, result.accepted);
+    },
+  },
+});
+
+await publisher.startTextOnly();
+await publisher.enableSpeech(); // call from the application's user action
+publisher.sendUserText(crypto.randomUUID(), "Stop and explain that again");
+```
+
+`enableSpeech()` is idempotent. The remote track is labelled `speech` and stays
+attached but silent between utterances. `sendUserText` is admitted only while
+the customer server owns a live control-token notify subscription.
+
 ### `new Publisher(options)`
 
 | Option | Type | Description |
 | --- | --- | --- |
-| `gatewayURLs` | `string[]` | **Required.** The `gateway_urls` from the join response. All are raced simultaneously; the fastest to accept wins. |
+| `gatewayURLs` | `string[]` | **Required.** The `gateway_urls` from the join response. All are raced simultaneously; the first to return `ready` wins. |
 | `token` | `string` | **Required.** The short-lived join token from the join response. |
 | `iceServers` | `RTCIceServer[]` | Optional extra ICE servers (e.g. your own STUN). TURN is supplied automatically by the winning gateway. |
+| `iceTransportPolicy` | `RTCIceTransportPolicy` | Passed to the underlying `RTCPeerConnection`. Defaults to `"all"`. Set `"relay"` to force media through TURN only (verifies the relay path end to end). |
+| `turnTransportPolicy` | `"all" \| "udp" \| "tls"` | Restricts gateway TURN URLs. Defaults to `"all"`; use `"tls"` with relay-only ICE to verify TURN over TLS. Startup fails if the required transport was not advertised. |
+| `gatewayHandshakeTimeoutMs` | `number` | Overall deadline for the initial gateway race and `accepted` → `ready` handshake. Unaccepted sockets are replaced after 3 seconds so a blackholed TCP flow cannot consume the full deadline. Defaults to 20 seconds. |
+| `peerConnectionTimeoutMs` | `number` | Deadline after the initial offer for WebRTC to reach `connected`. Defaults to 30 seconds. |
 | `signalingReconnectTimeoutMs` | `number` | How long to retry a dropped signaling socket against the selected regional gateway. Defaults to 20 seconds. |
 | `callbacks` | `PublisherCallbacks` | Optional lifecycle callbacks (see below). |
 
@@ -104,10 +187,19 @@ publisher.stop(); // stops all tracks and tears down the peer connection
 
 | Member | Description |
 | --- | --- |
-| `start(stream)` | Races the gateways, completes the handshake, and sends the SDP offer. Resolves once the offer is sent — use `onConnected` to know when media is actually flowing. |
-| `replaceStream(stream)` | Replaces the published tracks and renegotiates in place. |
+| `start(stream, type?)` | Races the gateways, completes the handshake, and sends the SDP offer. The stream must contain exactly one video track; `type` labels it, defaulting to `"camera"`. Resolves once the offer is sent — use `onConnected` to know when media is actually flowing. |
+| `startTextOnly()` | Starts with only the reliable ordered `argus.text` data channel and requests no capture permission. Media or speech may be added later. |
+| `startAudioOnly(stream)` | Starts with the stream's single microphone track plus the text data channel. |
+| `publish(stream, type)` | Adds the stream's single video track under `type` and renegotiates in place. Replaces any existing track of the same type. |
+| `publishMicrophone(stream)` | Adds or replaces the session's microphone track and renegotiates in place (turns on transcription). |
+| `unpublish(type)` | Removes the live track(s) of `type`, stops their capture, and renegotiates so Argus ends that track. |
+| `unpublishMicrophone()` | Removes the microphone track and renegotiates (ends transcription). |
+| `enableSpeech()` | Opts into the inbound `speech` track for text-to-speech. Idempotent; call from a user action. |
+| `sendUserText(messageId, text)` | Sends typed input over the `argus.text` data channel (≤4 KiB). |
+| `replaceStream(stream, type?)` | Convenience wrapper over `publish`; defaults to `"camera"`. |
 | `stop()` | Stops all local tracks and closes the peer connection. |
 | `frameReadToken` | The one-hour read token from the gateway's `ready` message, or `null` before connecting. The publisher uses it internally to resume signaling; hand it to your server to fetch frames. |
+| `selectedGatewayURL` | The signaling URL that won the regional race, or `null` before connecting. Relay it so your server uses the same region for frame reads and notifications; `frameReadToken` authorizes only the frame reads. |
 | `peerConnection` | The underlying `RTCPeerConnection`, or `null` if not started. |
 | `isConnected` | `true` when the peer connection state is `"connected"`. |
 
@@ -119,13 +211,20 @@ publisher.stop(); // stops all tracks and tears down the peer connection
 | `onConnectionStateChange(state)` | The `RTCPeerConnectionState` changed. |
 | `onRecoveryStateChange(event)` | Argus detected stalled media and the publisher started, escalated, completed, or failed automatic recovery. |
 | `onRecoveryRequired(event)` | Automatic recovery could not restore media, or capture ended and the host must ask the user for a new screen share. |
-| `onError(error)` | A fatal error occurred (signaling error, initial gateway failure, or signaling resume timed out). |
+| `onSpeechTrack(track, streams)` | The inbound `speech` track arrived after `enableSpeech()` — attach it to an `<audio>` element to play text-to-speech. |
+| `onAssistantText({ utteranceId, text })` | Assistant text arrived; it is paced with synthesized speech when speech is enabled and delivered immediately in text-only mode. |
+| `onUserTextResult({ messageId, accepted, reason })` | The server accepted or rejected a `sendUserText` message. |
+| `onError(error)` | A fatal error occurred (signaling error, WebRTC connection failure/timeout, or signaling resume timed out). |
 
 ## How `start()` works
 
 1. **Gateway race.** Every URL in `gatewayURLs` is opened at once with the token in the query string. The first to complete the two-phase handshake (`accepted` → `proceed` → `ready`) wins; the rest are closed. This picks the lowest-latency region without a separate probe.
 2. **TURN + read token.** The winning gateway's `ready` message carries per-session TURN credentials (merged into the ICE configuration) and the read token exposed as `frameReadToken`.
-3. **WebRTC.** A peer connection is created, tracks are added, and an offer is sent. Remote ICE candidates that arrive before the SDP answer are buffered and flushed once the answer is applied.
+3. **WebRTC.** A peer connection and the text data channel are created. Any
+   initial media track is added and labelled, then an offer is sent. Remote ICE
+   candidates that arrive before the SDP answer are buffered and flushed once
+   the answer is applied. Locally gathered candidates remain queued until their
+   signaling write succeeds and are replayed after signaling resume.
 
 After this initial race, the publisher is pinned to the selected regional
 gateway. If signaling drops, it reconnects to that same gateway with the
@@ -133,6 +232,22 @@ one-hour read token and waits for `resumed`; it does not race regions, rebuild
 the peer connection, or repeat the `ready` handshake. If the retry deadline
 expires, the publisher closes the stream and calls `onError`.
 
+Leave both transport policies at their defaults for normal use. To prove the
+firewall-friendly TURN TLS path from a VPN or restrictive network, force both
+relay ICE and TLS TURN:
+
+```ts
+const publisher = new Publisher({
+  gatewayURLs: join.gateway_urls,
+  token: join.token,
+  iceTransportPolicy: "relay",
+  turnTransportPolicy: "tls",
+});
+```
+
+This diagnostic configuration fails startup when the winning gateway did not
+advertise a TLS TURN URL instead of silently testing another path.
+
 ## Automatic media recovery
 
 Argus measures freshness from complete encoded samples received by the media
@@ -143,11 +258,12 @@ first detaches and reattaches the live sender and renegotiates. If samples do no
 resume within four seconds, it performs an ICE restart and waits another eight
 seconds. These fixed timings deliberately are not application configuration.
 
-`onRecoveryStateChange` reports the recovery transitions. If both automatic
-steps fail, `onRecoveryRequired` is called and the host can acquire a replacement
-stream and pass it to `replaceStream`. A browser cannot silently reacquire a
-screen share after the user or operating system ends it, so `capture_ended`
-always requires host UI and a fresh `captureScreen()` call.
+`onRecoveryStateChange` reports the recovery transitions, and the event's `track`
+field names the affected track type. If both automatic steps fail,
+`onRecoveryRequired` is called and the host can acquire a replacement stream and
+pass it to `publish` (or `replaceStream`) for that track type. A browser cannot
+silently reacquire a screen share after the user or operating system ends it, so
+`capture_ended` always requires host UI and a fresh `captureScreen()` call.
 
 ## Browser support
 
@@ -164,7 +280,7 @@ npm run typecheck  # tsc --noEmit
 
 ## Publishing
 
-The package is published to npm under the `@furious-luke` scope. `prepublishOnly` runs typecheck, tests, and build first, so a release is:
+The package is published to npm under the `@furious.luke` scope. `prepublishOnly` runs typecheck, tests, and build first, so a release is:
 
 ```bash
 npm version <patch|minor|major>