replicas-computer 0.3.2 → 0.3.4

package/README.md

@@ -21,15 +21,17 @@ The viewer and agent command paths are independent. One on-demand encoder pair i
 
 ffmpeg captures the display and encodes 1920×1080 H.264 (and Opus from PulseAudio) as RTP onto loopback. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
 
-WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without a displayed frame switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser.
+WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. The watchdog tracks decoded frames so CSS-hidden viewers do not fall back merely because the browser stops presenting video. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without decoded video switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser.
 
 The server makes the offer, so payload types and SSRCs are fixed and ffmpeg emits them directly. Four H.264 payload types are offered (High 4:4:4 Predictive, High, Main, Constrained Baseline); the encoder runs the best profile every connected viewer accepts, so one viewer without 4:4:4 support moves everyone to 4:2:0, and socket viewers or caps below 8 Mbps do the same. Two profiles select the encoder tuning: `balanced` (CRF 14, default chroma quantisation) and `text` (CRF 10, chroma QP offset −6); `?profile=` picks one and the most recent viewer's choice applies to the shared encoder. Quality adapts on a six-rung bitrate ladder that keeps resolution and frame rate and adds two CRF per rung under the 20 Mbps cap; `?adapt=0` pins the top rung.
 
-Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP; a REMB estimate below the sent rate, RTT growth, loss, jitter-buffer growth and stale-queue cuts all drive the ladder down, and clean seconds drive it up.
+Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It only bounds how far a severe descent (heavy loss or a starved decoder) steps, and only while it has stopped climbing below the sent rate. `scripts/probe-viewer.ts` drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; `scripts/probe-rtt-jitter.ts` feeds the controller a synthetic WebRTC viewer with jittery RTT probes.
 
 Encoding uses at most four threads and one queued capture frame, skips rapid catch-up captures after CPU stalls, and uses periodic intra refresh for WebRTC viewers; socket viewers switch it to 250 ms IDR intervals because they resynchronise only on IDR frames. When encoder settings change, a replacement discards two initial frames, emits a forced keyframe and then takes over. The active encoder continues if replacement fails.
 
-The viewer exposes `?view=fit|native|pixel` (letterboxed, one source pixel per CSS pixel, one source pixel per device pixel) and `?diagnostics=1`, which shows transport and ICE route, negotiated codec profile, source/decoded/CSS/device dimensions, displayed and decoded frame rates, bitrate, RTT, loss, jitter buffer, and the server's encoder settings, CPU, pacer budget, queue, stale cuts, repairs and copies. `globalThis.__replicasDesktop` carries the same state for tooling.
+The viewer exposes `?view=fit|native|pixel` (letterboxed, one source pixel per CSS pixel, one source pixel per device pixel) and `?diagnostics=1`, which shows transport and ICE route, negotiated codec profile, source/decoded/CSS/device dimensions, displayed and decoded frame rates, bitrate, RTT, loss, jitter buffer, the server's encoder settings, keyframe interval, CPU, pacer budget, queue, stale cuts, repairs and copies, the socket fallback's unsent bytes, withheld delta frames and decoder resets, and the latest server and viewer events. `globalThis.__replicasDesktop` carries the same state for tooling.
+
+The server records every decision that changes the picture (viewer connects and transport fallbacks with their reason, WebRTC state changes, quality rung moves with the signals that voted for them, encoder swaps, preset changes and keyframe requests) and writes each to the stream log (`/tmp/replicas-computer/stream.log`). `GET /telemetry` returns the `/health` snapshot plus the last 200 events and five minutes of one-second samples (encoder rung, CRF, cap, keyframe interval, preset, fps and kbps; CPU; per-viewer transport, ICE path, pacer counters, socket backlog and the viewer's own reported stats). Fetch it inside the sandbox with `curl -s localhost:6080/telemetry` or through the authenticated preview URL; the dashboard's stream settings menu has **Open stream telemetry**. `REPLICAS_DESKTOP_STREAM_TELEMETRY=1` also writes every sample to the stream log for sessions longer than the in-memory history.
 
 | Variable | Default | Meaning |
 |---|---|---|
@@ -42,6 +44,7 @@ The viewer exposes `?view=fit|native|pixel` (letterboxed, one source pixel per C
 | `REPLICAS_DESKTOP_STREAM_COLOR` | `bt709` | `legacy` disables the explicit BT.709 conversion and signalling |
 | `REPLICAS_DESKTOP_STREAM_VIDEO_BITRATE` | `20M` | Encoder bitrate cap across all quality rungs |
 | `REPLICAS_DESKTOP_STREAM_FPS` | `60` | Capture and encode rate |
+| `REPLICAS_DESKTOP_STREAM_TELEMETRY` | unset | `1` writes one-second telemetry samples to the stream log while a viewer is connected |
 
 ```bash
 replicas-computer setup --check

package/assets/desktop/stream-viewer.css

@@ -38,6 +38,8 @@ html[data-view="native"] body, html[data-view="pixel"] body { width: max-content
   right: 12px;
   display: none;
   max-width: calc(100vw - 24px);
+  max-height: calc(100vh - 24px);
+  overflow: hidden;
   padding: 6px 10px;
   border: 1px solid var(--replicas-hairline);
   color: var(--replicas-ink);

package/dist/index.js

@@ -2937,7 +2937,7 @@ async function computerStatusCommand() {
 }
 
 // src/version.ts
-var COMPUTER_USE_VERSION = "0.3.2";
+var COMPUTER_USE_VERSION = "0.3.4";
 var COMPUTER_USE_PROTOCOL_VERSION = 1;
 
 // src/program.ts
@@ -3446,8 +3446,8 @@ function superviseEncoder(options) {
   let restart;
   let candidateDeadline;
   let chain = Promise.resolve();
-  const log = (message) => process.stderr.write(`[desktop-stream] ${options.label} ${message}
-`);
+  const log = options.log ?? ((message) => process.stderr.write(`[desktop-stream] ${options.label} ${message}
+`));
   const retire = (generation) => {
     if (!generation) return;
     generation.live = false;
@@ -4253,6 +4253,9 @@ var fallbackBufferedBytes = 512 * 1024;
 var maximumBufferedBytes = 4 * 1024 * 1024;
 var probeIntervalMs = 250;
 var telemetryIntervalMs = 500;
+var eventHistory = 200;
+var sampleHistory = 300;
+var telemetryLog = process.env.REPLICAS_DESKTOP_STREAM_TELEMETRY === "1";
 var budgetFactor = 1.25;
 var protectMs = 1e4;
 var packageRoot = resolve2(dirname6(fileURLToPath(import.meta.url)), "..");
@@ -4352,6 +4355,14 @@ async function runMediaServer() {
   let audioPackets = 0;
   let videoBytes = 0;
   let audioBytes = 0;
+  const events = [];
+  const samples = [];
+  const record = (event, detail) => {
+    if (events.push({ at: Date.now(), event, detail }) > eventHistory) events.shift();
+    process.stderr.write(`[desktop-stream] ${event}${detail ? `: ${detail}` : ""}
+`);
+  };
+  const transportOf = (viewer) => viewer.fallback ? "websocket" : viewer.rtc?.connected ? "webrtc" : "negotiating";
   const ladder = () => qualityLadder(activeProfile);
   const socketViewers = () => [...viewers.values()].filter((viewer) => viewer.fallback);
   const profile = () => {
@@ -4389,11 +4400,13 @@ async function runMediaServer() {
       const { socket } = viewer;
       if (!viewer.fallback || socket.readyState !== WebSocket2.OPEN) continue;
       if (socket.bufferedAmount > maximumBufferedBytes) {
+        record("viewer dropped", `socket holds ${Math.round(socket.bufferedAmount / 1024)} KiB unsent`);
         socket.close(1013, "stream client is too slow");
         continue;
       }
       if (!keyframe2 && (viewer.waitingForKeyframe || queueMs(viewer) > targetBacklogMs || socket.bufferedAmount > fallbackBufferedBytes)) {
         viewer.waitingForKeyframe = true;
+        viewer.withheldFrames += 1;
         continue;
       }
       if (keyframe2) viewer.waitingForKeyframe = false;
@@ -4454,6 +4467,7 @@ async function runMediaServer() {
   const env = { ...process.env, DISPLAY: display };
   const videoEncoder = superviseEncoder({
     label: "video",
+    log: (message) => record("video encoder", message),
     // Initial encoder work can leave its first image older than the outgoing stream.
     warmupFrames: 2,
     env,
@@ -4556,6 +4570,7 @@ async function runMediaServer() {
   });
   const audioEncoder = superviseEncoder({
     label: "audio",
+    log: (message) => record("audio encoder", message),
     env,
     onPacket: onAudioPacket,
     joinable: () => true,
@@ -4614,14 +4629,14 @@ async function runMediaServer() {
     if (!videoWanted) {
       videoWanted = true;
       encoderSignature = signature;
+      record("encoder", signature);
       void videoEncoder.start();
       return false;
     }
     if (signature === encoderSignature) return false;
     encoderSignature = signature;
     lastSwapAt = Date.now();
-    process.stderr.write(`[desktop-stream] encoder \u2192 ${signature}
-`);
+    record("encoder", signature);
     void videoEncoder.replace();
     return true;
   };
@@ -4635,9 +4650,11 @@ async function runMediaServer() {
       void audioEncoder.stop();
     }, 5e3);
   };
-  const setRung = (next) => {
+  const setRung = (next, reason) => {
     if (next === rung) return;
     const now = Date.now();
+    const quality = ladder()[next];
+    record(next > rung ? "quality down" : "quality up", `rung ${rung} \u2192 ${next} (${quality.kbps} kbps cap, crf ${quality.crf}): ${reason}`);
     if (next > rung) {
       if (now - lastDescentAt > 1e4) descents += 1;
       lastDescentAt = now;
@@ -4665,20 +4682,37 @@ async function runMediaServer() {
     blockedUntil = 0;
   };
   const stableRttWindows = /* @__PURE__ */ new WeakMap();
-  const freshRemb = (viewer, now) => viewer.remb && now - viewer.remb.at < 2e3 ? viewer.remb.kbps : void 0;
+  const latestRemb = (viewer) => viewer.rembs[viewer.rembs.length - 1];
+  const freshRemb = (viewer, now) => {
+    const remb = latestRemb(viewer);
+    return remb && now - remb.at < 2e3 ? remb.kbps : void 0;
+  };
   const rembBelowSent = (viewer, now) => {
     const remb = freshRemb(viewer, now);
+    const earlier = [...viewer.rembs].reverse().find((sample) => sample.at <= now - 2e3);
     const sent = Math.min(...recentVideoKbps);
-    return remb !== void 0 && sent > 500 && remb < sent * 0.9;
+    return remb !== void 0 && earlier !== void 0 && remb <= earlier.kbps && sent > 500 && remb < sent * 0.9;
   };
-  const struggling = (viewer, stats) => {
+  const signals = (viewer, stats) => {
+    const now = Date.now();
+    if (viewer.fallback) {
+      return [
+        ...queueMs(viewer) > congestedBacklogMs ? [`probe queue ${Math.round(queueMs(viewer))} ms`] : [],
+        ...stats.decodeQueue > congestedDecodeQueue ? [`decode queue ${stats.decodeQueue}`] : []
+      ];
+    }
     const stableSince = stableRttWindows.get(viewer)?.since;
-    return viewer.fallback ? queueMs(viewer) > congestedBacklogMs || stats.decodeQueue > congestedDecodeQueue : stats.rttMs > viewer.rtcBaselineMs + 40 && (stableSince === void 0 || Date.now() - stableSince < 2e3) || stats.videoFps < fps * 0.6 && viewer.lowFpsReports >= 2 || (stats.lossPercent ?? 0) > 3 || (stats.jitterBufferMs ?? 0) > 80 || rembBelowSent(viewer, Date.now());
+    return [
+      ...viewer.rttFloorMs > viewer.rtcBaselineMs + 40 && (stableSince === void 0 || now - stableSince < 2e3) ? [`rtt floor ${Math.round(viewer.rttFloorMs)} ms over ${Math.round(viewer.rtcBaselineMs)} ms baseline`] : [],
+      ...stats.videoFps < fps * 0.6 && viewer.lowFpsReports >= 2 ? [`decoded ${stats.videoFps.toFixed(0)} fps`] : [],
+      ...(stats.lossPercent ?? 0) > 3 ? [`loss ${(stats.lossPercent ?? 0).toFixed(1)}%`] : [],
+      ...(stats.jitterBufferMs ?? 0) > 80 ? [`jitter buffer ${(stats.jitterBufferMs ?? 0).toFixed(0)} ms`] : []
+    ];
   };
   const lowFpsObservationAt = /* @__PURE__ */ new WeakMap();
   const severe = (viewer, stats) => !viewer.fallback && ((stats.lossPercent ?? 0) > 10 || stats.videoFps < fps * 0.25 && viewer.lowFpsReports >= 2);
   let lastSevereDescentAt = 0;
-  const descendSevere = (now) => {
+  const descendSevere = (now, trigger) => {
     comfortableSeconds = 0;
     congestedSeconds = 0;
     if (!adaptive || now - lastSevereDescentAt < 1e3) return;
@@ -4690,7 +4724,10 @@ async function runMediaServer() {
     });
     const rungs = ladder();
     const estimatedRung = estimates.length ? rungs.findIndex((quality) => quality.kbps <= Math.min(...estimates) * 0.8) : rung + 2;
-    setRung(Math.min(rungs.length - 1, Math.max(rung + 2, estimatedRung < 0 ? rungs.length - 1 : estimatedRung)));
+    setRung(
+      Math.min(rungs.length - 1, Math.max(rung + 2, estimatedRung < 0 ? rungs.length - 1 : estimatedRung)),
+      `severe: ${trigger}; link estimates ${estimates.length ? `${estimates.map(Math.round).join("/")} kbps` : "none"}`
+    );
   };
   const control = setInterval(() => {
     encoderFps = videoFrames - framesAtTick;
@@ -4708,6 +4745,7 @@ async function runMediaServer() {
       starvedSeconds += 1;
       if (starvedSeconds >= 5 && preset !== "ultrafast") {
         preset = "ultrafast";
+        record("encoder starved", `${encoderFps} fps for ${starvedSeconds}s; preset \u2192 ultrafast`);
         syncEncoder();
       }
       return;
@@ -4717,26 +4755,27 @@ async function runMediaServer() {
     if (encoderHealthyTicks < 3) return;
     if (preset !== basePreset && ++presetHealthySeconds >= 120) {
       preset = basePreset;
+      record("encoder recovered", `preset \u2192 ${basePreset}`);
       syncEncoder();
     }
     if (!adaptive) {
-      if (rung !== 0) setRung(0);
+      if (rung !== 0) setRung(0, "quality pinned");
       return;
     }
     const now = Date.now();
     const reporting = [...viewers.values()].filter((viewer) => viewer.stats !== void 0 && viewer.stats.videoPacketsPerSecond !== 0 && now - viewer.stats.at < 3e3 && now - viewer.connectedAt > 2500);
-    cleanTransportSeconds = reporting.length === viewers.size && reporting.length > 0 && reporting.every((viewer) => !viewer.fallback && viewer.stats.rttMs <= viewer.rtcBaselineMs + 20 && (viewer.stats.lossPercent ?? 0) === 0 && (viewer.stats.nacks ?? 0) === 0 && (viewer.stats.jitterBufferMs ?? 0) <= 20 && viewer.stats.videoFps >= fps * 0.95) ? cleanTransportSeconds + 1 : 0;
+    cleanTransportSeconds = reporting.length === viewers.size && reporting.length > 0 && reporting.every((viewer) => !viewer.fallback && viewer.rttFloorMs <= viewer.rtcBaselineMs + 20 && (viewer.stats.lossPercent ?? 0) === 0 && (viewer.stats.nacks ?? 0) === 0 && (viewer.stats.jitterBufferMs ?? 0) <= 20 && viewer.stats.videoFps >= fps * 0.95) ? cleanTransportSeconds + 1 : 0;
     if (reporting.length > 0 && reporting.every((viewer) => severe(viewer, viewer.stats))) {
-      descendSevere(now);
+      descendSevere(now, reporting.map((viewer) => `${(viewer.stats.lossPercent ?? 0).toFixed(1)}% loss, ${viewer.stats.videoFps.toFixed(0)} fps decoded`).join(" | "));
       return;
     }
-    const votes = reporting.flatMap((viewer) => viewer.everHealthy || now - viewer.connectedAt > 15e3 ? [struggling(viewer, viewer.stats)] : []);
+    const votes = reporting.flatMap((viewer) => viewer.everHealthy || now - viewer.connectedAt > 15e3 ? [signals(viewer, viewer.stats)] : []);
     if (votes.length === 0) return;
-    const strugglers = votes.filter(Boolean).length;
+    const strugglers = votes.filter((vote) => vote.length > 0).length;
     if (strugglers === votes.length) {
       comfortableSeconds = 0;
       congestedSeconds += 1;
-      if (congestedSeconds >= 2) setRung(Math.min(ladder().length - 1, rung + 1));
+      if (congestedSeconds >= 2) setRung(Math.min(ladder().length - 1, rung + 1), votes.map((vote) => vote.join(", ")).join(" | "));
       return;
     }
     if (strugglers > 0) {
@@ -4745,7 +4784,7 @@ async function runMediaServer() {
       return;
     }
     congestedSeconds = Math.max(0, congestedSeconds - 0.5);
-    if (descents === 0 && reporting.some((viewer) => !viewer.fallback && (viewer.stats.rttMs > viewer.rtcBaselineMs + 20 || (viewer.stats.playbackDroppedPercent ?? 0) > 5 && cleanTransportSeconds < 15))) {
+    if (descents === 0 && reporting.some((viewer) => !viewer.fallback && (viewer.rttFloorMs > viewer.rtcBaselineMs + 20 || (viewer.stats.playbackDroppedPercent ?? 0) > 5 && cleanTransportSeconds < 15))) {
       comfortableSeconds = 0;
       return;
     }
@@ -4755,7 +4794,7 @@ async function runMediaServer() {
     if (comfortableSeconds < (descents === 0 ? startupSeconds : recoverySeconds)) return;
     const target = Math.max(0, rung - 1);
     if (target === blockedRung && now < blockedUntil) return;
-    setRung(target);
+    setRung(target, `${comfortableSeconds}s comfortable, ${cleanTransportSeconds}s clean transport`);
   }, 1e3);
   control.unref();
   const probe = setInterval(() => {
@@ -4779,46 +4818,61 @@ async function runMediaServer() {
       encoderPercent: Math.max(0, Math.round((encoder - cpuSample.encoder) * 100 / elapsed))
     };
     cpuSample = { at: now, server, encoder };
-    const { quality, profile: videoProfile, pixelFormat } = encoderParams();
+    const { quality, profile: videoProfile, pixelFormat, keyframeMs } = encoderParams();
+    const encoderState = {
+      profile: activeProfile,
+      codecProfile: videoProfile,
+      pixelFormat,
+      color: colorMode,
+      rung,
+      rungs: ladder().length,
+      kbpsCap: quality.kbps,
+      crf: quality.crf,
+      preset,
+      keyframeMs,
+      fps: encoderFps,
+      kbps: videoKbps,
+      adaptive
+    };
+    const cpuState = { ...cpu, load: Math.round(loadavg()[0] * 100) / 100, cores: availableParallelism() };
+    const sample = {
+      at: now,
+      encoder: { rung, crf: quality.crf, kbpsCap: quality.kbps, keyframeMs, preset, fps: encoderFps, kbps: videoKbps },
+      cpu: cpuState,
+      viewers: []
+    };
     for (const viewer of viewers.values()) {
       if (viewer.socket.readyState !== WebSocket2.OPEN) continue;
       const pacer = viewer.rtc?.snapshot();
-      const message = {
-        type: "telemetry",
-        encoder: {
-          profile: activeProfile,
-          codecProfile: videoProfile,
-          pixelFormat,
-          color: colorMode,
-          rung,
-          rungs: ladder().length,
-          kbpsCap: quality.kbps,
-          crf: quality.crf,
-          preset,
-          fps: encoderFps,
-          kbps: videoKbps,
-          adaptive
-        },
-        cpu: { ...cpu, load: Math.round(loadavg()[0] * 100) / 100, cores: availableParallelism() },
-        viewer: {
-          budgetKbps: pacer?.budgetKbps ?? 0,
-          queueMs: pacer?.queueMs ?? Math.round(queueMs(viewer)),
-          queuedPackets: pacer?.queuedPackets ?? 0,
-          staleEvents: pacer?.staleEvents ?? 0,
-          droppedFrames: pacer?.droppedFrames ?? 0,
-          repairsSent: pacer?.repairsSent ?? 0,
-          repairsSkipped: pacer?.repairsSkipped ?? 0,
-          duplicatesSent: pacer?.duplicatesSent ?? 0,
-          protecting: pacer?.protecting ?? false,
-          rembKbps: pacer?.rembKbps ?? null,
-          sentKbps: pacer?.sentKbps ?? 0
-        }
+      const viewerState = {
+        budgetKbps: pacer?.budgetKbps ?? 0,
+        queueMs: pacer?.queueMs ?? Math.round(queueMs(viewer)),
+        queuedPackets: pacer?.queuedPackets ?? 0,
+        staleEvents: pacer?.staleEvents ?? 0,
+        droppedFrames: pacer?.droppedFrames ?? 0,
+        repairsSent: pacer?.repairsSent ?? 0,
+        repairsSkipped: pacer?.repairsSkipped ?? 0,
+        duplicatesSent: pacer?.duplicatesSent ?? 0,
+        protecting: pacer?.protecting ?? false,
+        rembKbps: pacer?.rembKbps ?? null,
+        sentKbps: pacer?.sentKbps ?? 0,
+        transport: transportOf(viewer),
+        bufferedBytes: viewer.socket.bufferedAmount,
+        withheldFrames: viewer.withheldFrames,
+        waitingForKeyframe: viewer.fallback && viewer.waitingForKeyframe
       };
+      sample.viewers.push({ ...viewerState, sentKbps: Math.round((viewerState.sentKbps + (viewer.lastSentKbps ?? viewerState.sentKbps)) / 2), path: viewer.rtc?.path ?? null, stats: viewer.stats ?? null });
+      viewer.lastSentKbps = viewerState.sentKbps;
+      const message = { type: "telemetry", encoder: encoderState, cpu: cpuState, viewer: viewerState, events: events.slice(-5) };
       viewer.socket.send(JSON.stringify(message));
     }
+    if (now - (samples[samples.length - 1]?.at ?? 0) < 900) return;
+    if (samples.push(sample) > sampleHistory) samples.shift();
+    if (telemetryLog) process.stderr.write(`[desktop-stream] sample ${JSON.stringify(sample)}
+`);
   }, telemetryIntervalMs);
   telemetry.unref();
-  const requestKeyframe = (joinedAt, requestingViewer) => {
+  const requestKeyframe = (reason, joinedAt, requestingViewer) => {
     const now = Date.now();
     if (!videoWanted || viewers.size === 0 || videoEncoder.replacing() || lastKeyframeAt < now - videoEncoder.activeForMs() || joinedAt !== void 0 && lastKeyframeAt > joinedAt) return;
     const startup = requestingViewer && !requestingViewer.hasDecodedFrame && now - requestingViewer.connectedAt < 1e4;
@@ -4826,12 +4880,13 @@ async function runMediaServer() {
     if (wait > 0) {
       if (joinedAt !== void 0) {
         clearTimeout(joinKeyframe);
-        joinKeyframe = setTimeout(() => requestKeyframe(joinedAt), wait);
+        joinKeyframe = setTimeout(() => requestKeyframe(reason, joinedAt), wait);
       }
       return;
     }
     if (!startup && encoderParams().keyframeMs - (now - lastKeyframeAt) <= 350) return;
     lastSwapAt = now;
+    record("keyframe", reason);
     void videoEncoder.replace();
   };
   const enterFallback = (viewer, reason) => {
@@ -4841,9 +4896,8 @@ async function runMediaServer() {
     viewer.input.release();
     viewer.rtc?.close();
     viewer.rtc = void 0;
-    process.stderr.write(`[desktop-stream] viewer streaming over the socket: ${reason}
-`);
-    if (!syncEncoder()) requestKeyframe();
+    record("fallback to socket", reason);
+    if (!syncEncoder()) requestKeyframe("socket fallback");
   };
   const optional = (value, max) => value === void 0 ? void 0 : clamp(value, 0, max);
   const handleMessage = (viewer, message, reply) => {
@@ -4883,8 +4937,10 @@ async function runMediaServer() {
           viewer.lowFpsReports = Math.min(2, viewer.lowFpsReports + 1);
           lowFpsObservationAt.set(viewer, now);
         }
-        if (!viewer.fallback && viewer.stats.rttMs > 0) viewer.rtcBaselineMs = Math.min(viewer.rtcBaselineMs, viewer.stats.rttMs);
-        const rtt = viewer.stats.rttMs;
+        viewer.rttSamples = [...viewer.rttSamples.filter((sample) => now - sample.at < 1e3), { at: now, ms: viewer.stats.rttMs }];
+        viewer.rttFloorMs = Math.min(...viewer.rttSamples.map((sample) => sample.ms).filter((ms) => ms > 0), viewer.stats.rttMs);
+        if (!viewer.fallback && viewer.rttFloorMs > 0) viewer.rtcBaselineMs = Math.min(viewer.rtcBaselineMs, viewer.rttFloorMs);
+        const rtt = viewer.rttFloorMs;
         let window = stableRttWindows.get(viewer);
         const stableNetwork = !viewer.fallback && rtt > viewer.rtcBaselineMs + 20 && (viewer.stats.videoPacketsPerSecond ?? 0) > 0 && (viewer.stats.lossPercent ?? 0) <= 1 && (viewer.stats.jitterBufferMs ?? 0) <= 80;
         if (stableNetwork && viewer.stats.videoFps >= fps * 0.9) {
@@ -4919,7 +4975,7 @@ async function runMediaServer() {
         }
         if (viewer.stats.videoFps > 0) viewer.hasDecodedFrame = true;
         if (viewer.stats.videoFps >= fps * 0.8) viewer.everHealthy = true;
-        if (encoderHealthyTicks >= 3 && videoEncoder.running() && [...viewers.values()].every((v) => !v.fallback && v.stats && now - v.stats.at < 750 && now - v.connectedAt > 2500 && (v.stats.videoPacketsPerSecond ?? 0) > 0 && (v.stats.lossPercent ?? 0) > 10)) descendSevere(now);
+        if (encoderHealthyTicks >= 3 && videoEncoder.running() && [...viewers.values()].every((v) => !v.fallback && v.stats && now - v.stats.at < 750 && now - v.connectedAt > 2500 && (v.stats.videoPacketsPerSecond ?? 0) > 0 && (v.stats.lossPercent ?? 0) > 10)) descendSevere(now, "every viewer reports over 10% loss");
         return;
       }
       case "answer":
@@ -4933,7 +4989,7 @@ async function runMediaServer() {
         enterFallback(viewer, message.reason);
         return;
       case "keyframe":
-        if (!viewer.fallback) requestKeyframe(void 0, viewer);
+        if (!viewer.fallback) requestKeyframe("viewer watchdog", void 0, viewer);
         return;
       case "theme":
         applyDesktopTheme(message.theme);
@@ -4955,7 +5011,7 @@ async function runMediaServer() {
   };
   const httpServer = createServer2((request, response) => {
     const url = new URL(request.url || "/", "http://localhost");
-    if (url.pathname === "/health") {
+    if (url.pathname === "/health" || url.pathname === "/telemetry") {
       const all = [...viewers.values()];
       const memory = process.memoryUsage();
       const { keyframeMs, profile: videoProfile, pixelFormat, quality } = encoderParams();
@@ -4964,7 +5020,7 @@ async function runMediaServer() {
         version: COMPUTER_USE_VERSION,
         clients: viewers.size,
         webrtc,
-        transports: all.map((viewer) => viewer.fallback ? "websocket" : viewer.rtc?.connected ? "webrtc" : "negotiating"),
+        transports: all.map(transportOf),
         requests: all.map((viewer) => viewer.request),
         encoder: videoEncoder.running(),
         audioEncoder: audioEncoder.running(),
@@ -4998,7 +5054,8 @@ async function runMediaServer() {
         heapUsed: memory.heapUsed,
         width,
         height,
-        fps
+        fps,
+        ...url.pathname === "/telemetry" ? { events, samples } : {}
       }));
       return;
     }
@@ -5038,6 +5095,7 @@ async function runMediaServer() {
       request,
       fallback: false,
       waitingForKeyframe: true,
+      withheldFrames: 0,
       everHealthy: false,
       lowFpsReports: 0,
       connectedAt: Date.now(),
@@ -5046,17 +5104,20 @@ async function runMediaServer() {
       probeAnswered: false,
       baselineMs: Number.POSITIVE_INFINITY,
       rtcBaselineMs: Number.POSITIVE_INFINITY,
-      protectBlockedUntil: 0
+      protectBlockedUntil: 0,
+      rembs: [],
+      rttFloorMs: 0,
+      rttSamples: []
     };
     if (viewers.size === 0) resetLadder(request);
     if (request.profile !== activeProfile || request.adaptive !== adaptive) {
-      process.stderr.write(`[desktop-stream] viewer requests ${request.profile} profile, ${request.adaptive ? "adaptive" : "pinned"} quality
-`);
+      record("viewer request", `${request.profile} profile, ${request.adaptive ? "adaptive" : "pinned"} quality`);
       activeProfile = request.profile;
       adaptive = request.adaptive;
       if (!adaptive) rung = 0;
     }
     viewers.set(socket, viewer);
+    record("viewer connected", `${request.transport} requested, ${viewers.size} connected`);
     const reply = (message) => {
       if (socket.readyState === WebSocket2.OPEN) socket.send(JSON.stringify(message));
     };
@@ -5072,23 +5133,26 @@ async function runMediaServer() {
           candidate: (candidate, mid) => reply({ type: "candidate", candidate, mid }),
           message: (message, answer) => handleMessage(viewer, message, answer),
           keyframe: (reason) => {
-            if (reason === "stale") descendSevere(Date.now());
-            requestKeyframe(void 0, viewer);
+            if (reason === "stale") descendSevere(Date.now(), "send queue went stale and was cut back to a keyframe");
+            requestKeyframe(reason === "pli" ? "receiver pli" : "stale send queue", void 0, viewer);
           },
           remb: (bitsPerSecond) => {
-            viewer.remb = { kbps: bitsPerSecond / 1e3, at: Date.now() };
+            if (viewer.rembs.push({ kbps: bitsPerSecond / 1e3, at: Date.now() }) > 16) viewer.rembs.shift();
+          },
+          connected: () => {
+            record("webrtc connected", viewer.rtc?.path ?? void 0);
+            requestKeyframe("viewer joined", Date.now());
           },
-          connected: () => requestKeyframe(Date.now()),
           // A viewer that loses its peer cannot send the key-ups for whatever it
           // was holding, and its socket may stay open for a long time yet.
           disconnected: (state) => {
+            record(`webrtc ${state}`);
             viewer.input.release();
             if (state === "failed") reply({ type: "webrtc-failed" });
           }
         });
       } catch (error) {
-        process.stderr.write(`[desktop-stream] could not create a peer connection: ${error instanceof Error ? error.message : String(error)}
-`);
+        record("webrtc error", `could not create a peer connection: ${error instanceof Error ? error.message : String(error)}`);
       }
     }
     if (!viewer.rtc) enterFallback(viewer, offerWebRtc ? "webrtc unavailable" : "websocket requested");
@@ -5102,6 +5166,7 @@ async function runMediaServer() {
       viewer.rtc?.close();
       viewer.input.close();
       viewers.delete(socket);
+      record("viewer disconnected", `${viewers.size} connected`);
       if (viewers.size > 0) {
         syncEncoder();
         return;

package/dist/stream-viewer.js

@@ -24,7 +24,7 @@ function validServerMessage(value) {
     case "config":
       return finite(value.protocolVersion) && (value.inputRace === void 0 || typeof value.inputRace === "boolean") && finite(value.width) && finite(value.height) && finite(value.fps) && typeof value.videoCodec === "string" && typeof value.audioCodec === "string" && finite(value.audioSampleRate) && finite(value.audioChannels) && isRecord(value.request) && (value.iceServers === void 0 || Array.isArray(value.iceServers) && value.iceServers.every(iceServer));
     case "telemetry":
-      return isRecord(value.encoder) && isRecord(value.cpu) && isRecord(value.viewer);
+      return isRecord(value.encoder) && isRecord(value.cpu) && isRecord(value.viewer) && Array.isArray(value.events);
     default:
       return false;
   }
@@ -129,6 +129,13 @@ var telemetry;
 var route = { path: "", codec: "", decoded: "", kbps: 0, displayedFps: 0 };
 var state = {};
 globalThis.__replicasDesktop = state;
+var viewerEvents = [];
+state.events = viewerEvents;
+var decoderResets = 0;
+var note = (event, detail) => {
+  if (viewerEvents.push({ at: Date.now(), event, detail }) > 50) viewerEvents.shift();
+};
+var describeEvents = (list) => list.map((entry) => `${new Date(entry.at).toLocaleTimeString([], { hour12: false })} ${entry.event}${entry.detail ? ` (${entry.detail})` : ""}`).join(" \xB7 ");
 var showStatus = (message) => {
   statusText.textContent = message;
   status.classList.add("visible");
@@ -267,6 +274,7 @@ var configureVideo = async (next, preferHardware = true) => {
       if (videoDecoder !== decoder) return;
       videoDecoder = void 0;
       waitingForKeyframe = true;
+      note("decoder error", error.message);
       showStatus(`recovering video: ${error.message}`);
       if (config) void configureVideo(config, false).catch((reason) => showStatus(describe(reason)));
     }
@@ -335,6 +343,7 @@ var handleBinary = (data) => {
     videoDecoder.reset();
     videoDecoder.configure(activeVideoConfig);
     waitingForKeyframe = true;
+    state.decoderResets = ++decoderResets;
   }
   if (waitingForKeyframe && !key) return;
   waitingForKeyframe = false;
@@ -344,6 +353,7 @@ var handleBinary = (data) => {
     videoDecoder.reset();
     videoDecoder.configure(activeVideoConfig);
     waitingForKeyframe = true;
+    state.decoderResets = ++decoderResets;
   }
 };
 var unlockAudio = () => {
@@ -367,6 +377,7 @@ var useFallback = (reason, failed = false) => {
   if (transport === "websocket" || !config) return;
   closePeer();
   if (failed) webrtcSkips = webrtcSkipsAfterFailure;
+  note("fallback to socket", reason);
   transport = "websocket";
   transportReason = reason;
   route = { ...route, path: "proxy", codec: config.videoCodec };
@@ -412,8 +423,12 @@ var negotiate = async (sdp) => {
     if (peer !== pc) return;
     if (event.candidate?.candidate) sendSocket({ type: "candidate", candidate: event.candidate.candidate, mid: event.candidate.sdpMid ?? "0" });
   };
+  pc.oniceconnectionstatechange = () => {
+    if (peer === pc) note("ice", pc.iceConnectionState);
+  };
   pc.onconnectionstatechange = () => {
     if (peer !== pc) return;
+    note("webrtc", pc.connectionState);
     if (pc.connectionState === "connected") {
       transport = "webrtc";
       transportReason = "";
@@ -491,7 +506,7 @@ var handleServerMessage = (message, viaSocket) => {
     useFallback("recent WebRTC failure");
   } else if (!transport) {
     showStatus("Establishing connection\u2026");
-    webrtcDeadline ??= setTimeout(() => useFallback("WebRTC did not display video in time", true), webrtcFrameTimeoutMs);
+    webrtcDeadline ??= setTimeout(() => useFallback("WebRTC did not decode video in time", true), webrtcFrameTimeoutMs);
   }
 };
 var handleServerText = (data, viaSocket) => {
@@ -516,6 +531,7 @@ var connect = () => {
   socket = nextSocket;
   nextSocket.addEventListener("open", () => {
     if (socket !== nextSocket) return;
+    note("socket open");
     actionId = 0;
     reconnectAttempt = 0;
     sendSocket({ type: "theme", theme: document.documentElement.dataset.theme === "light" ? "light" : "dark" });
@@ -534,8 +550,9 @@ var connect = () => {
     transport = void 0;
     announceTransport();
     reconnectAttempt += 1;
-    const delay = Math.min(8e3, 250 * 2 ** Math.min(reconnectAttempt, 5));
-    reconnectTimer = setTimeout(connect, delay * (0.85 + Math.random() * 0.3));
+    const delay = Math.min(8e3, 250 * 2 ** Math.min(reconnectAttempt, 5)) * (0.85 + Math.random() * 0.3);
+    note("socket closed", `reconnect in ${Math.round(delay)} ms`);
+    reconnectTimer = setTimeout(connect, delay);
   });
   nextSocket.addEventListener("error", () => nextSocket.close());
 };
@@ -561,11 +578,6 @@ var flushPointer = () => {
 var frameCallback = 0;
 var observeFrame = () => {
   frameCallback = screen.requestVideoFrameCallback((_now, metadata) => {
-    const stream = screen.srcObject;
-    if (webrtcDeadline && transport === "webrtc" && stream instanceof MediaStream && peer?.getReceivers().some(({ track }) => stream.getVideoTracks().includes(track))) {
-      clearTimeout(webrtcDeadline);
-      webrtcDeadline = void 0;
-    }
     renderedFrames += 1;
     presentedFrames += lastPresentedCount ? Math.max(0, metadata.presentedFrames - lastPresentedCount) : 1;
     lastPresentedCount = metadata.presentedFrames;
@@ -654,8 +666,11 @@ var updateDiagnostics = (stats) => {
     `${transport ?? "connecting"}${transportReason ? ` (${transportReason})` : ""} \xB7 ${route.path || "\u2014"} \xB7 ${route.codec || config?.videoCodec || ""} \xB7 ${view} view`,
     `source ${dims.source} \xB7 decoded ${dims.decoded || "\u2014"} \xB7 css ${dims.css} \xB7 device ${dims.physical}`,
     `${route.displayedFps.toFixed(1)} displayed/s \xB7 ${stats.videoFps.toFixed(1)} decoded/s \xB7 ${route.kbps.toFixed(0)} kbps \xB7 ${stats.rttMs.toFixed(0)} ms rtt \xB7 ${socketRtt.toFixed(0)} ms via proxy \xB7 ${network}`,
-    encoder ? `encoder ${encoder.profile} crf ${encoder.crf} \xB7 ${encoder.codecProfile} ${encoder.pixelFormat} ${encoder.color} \xB7 rung ${encoder.rung}/${encoder.rungs - 1} cap ${encoder.kbpsCap} kbps${encoder.adaptive ? "" : " pinned"} \xB7 ${encoder.fps} fps ${encoder.kbps} kbps \xB7 ${encoder.preset}` : "",
-    telemetry && server ? `server cpu: encoder ${telemetry.cpu.encoderPercent}% \xB7 stream ${telemetry.cpu.serverPercent}% \xB7 load ${telemetry.cpu.load} on ${telemetry.cpu.cores} cores \xB7 budget ${server.budgetKbps} kbps sent ${server.sentKbps} \xB7 queue ${server.queueMs} ms ${server.queuedPackets} pkt \xB7 stale ${server.staleEvents} dropped ${server.droppedFrames} \xB7 repairs ${server.repairsSent}/${server.repairsSkipped} skipped \xB7 copies ${server.duplicatesSent}${server.protecting ? " on" : ""} \xB7 remb ${server.rembKbps ?? "\u2014"}` : ""
+    encoder ? `encoder ${encoder.profile} crf ${encoder.crf} \xB7 ${encoder.codecProfile} ${encoder.pixelFormat} ${encoder.color} \xB7 rung ${encoder.rung}/${encoder.rungs - 1} cap ${encoder.kbpsCap} kbps${encoder.adaptive ? "" : " pinned"} \xB7 keyframe every ${encoder.keyframeMs} ms \xB7 ${encoder.fps} fps ${encoder.kbps} kbps \xB7 ${encoder.preset}` : "",
+    telemetry && server ? `server cpu: encoder ${telemetry.cpu.encoderPercent}% \xB7 stream ${telemetry.cpu.serverPercent}% \xB7 load ${telemetry.cpu.load} on ${telemetry.cpu.cores} cores \xB7 budget ${server.budgetKbps} kbps sent ${server.sentKbps} \xB7 queue ${server.queueMs} ms ${server.queuedPackets} pkt \xB7 stale ${server.staleEvents} dropped ${server.droppedFrames} \xB7 repairs ${server.repairsSent}/${server.repairsSkipped} skipped \xB7 copies ${server.duplicatesSent}${server.protecting ? " on" : ""} \xB7 remb ${server.rembKbps ?? "\u2014"}` : "",
+    transport === "websocket" && server ? `socket: ${Math.round(server.bufferedBytes / 1024)} KiB unsent \xB7 ${server.withheldFrames} deltas withheld${server.waitingForKeyframe ? " \xB7 awaiting keyframe" : ""} \xB7 ${decoderResets} decoder resets` : "",
+    telemetry?.events.length ? `server events: ${describeEvents(telemetry.events)}` : "",
+    viewerEvents.length ? `viewer events: ${describeEvents(viewerEvents.slice(-5))}` : ""
   ].filter(Boolean).join("\n");
 };
 var inboundOf = (kind) => (stat) => stat.type === "inbound-rtp" && "kind" in stat && stat.kind === kind;
@@ -691,6 +706,11 @@ var reportWebRtcStats = async (pc, stats) => {
     jitterBufferEmittedCount: video?.jitterBufferEmittedCount ?? 0,
     audioPackets: audio?.packetsReceived ?? 0
   };
+  if (counters.framesDecoded > (inbound?.framesDecoded ?? 0)) {
+    lastFrameAt = now;
+    clearTimeout(webrtcDeadline);
+    webrtcDeadline = void 0;
+  }
   if (!inbound) inboundHistory = [];
   while (inboundHistory.length > 1 && inboundHistory[1].at <= now - 1e3) inboundHistory.shift();
   const oldest = inboundHistory[0];
@@ -723,8 +743,6 @@ var reportWebRtcStats = async (pc, stats) => {
   updateDiagnostics(stats);
   state.framesDropped = video?.framesDropped;
 };
-var watchdogPlaybackFrames = 0;
-var watchdogPlaybackAt = 0;
 setInterval(() => {
   if (transport !== "webrtc") return;
   if (screen.paused && screen.srcObject instanceof MediaStream && screen.srcObject.getVideoTracks().some((track) => track.readyState === "live")) return;
@@ -733,15 +751,10 @@ setInterval(() => {
     useFallback("WebRTC video stopped arriving", true);
     return;
   }
-  const quality = screen.getVideoPlaybackQuality();
-  const playbackFrames = quality.totalVideoFrames - quality.droppedVideoFrames;
-  if (playbackFrames !== watchdogPlaybackFrames) {
-    watchdogPlaybackFrames = playbackFrames;
-    watchdogPlaybackAt = now;
-  }
   if (inputChannel?.readyState !== "open") return;
-  if (now - watchdogPlaybackAt < 300 || now - lastFrameAt < 300 || now - lastKeyframeRequestAt < 1e3) return;
+  if (now - lastFrameAt < 2500 || now - lastKeyframeRequestAt < 1e3) return;
   lastKeyframeRequestAt = now;
+  note("keyframe requested", `no frame for ${Math.round(now - lastFrameAt)} ms`);
   sendInput({ type: "keyframe" });
 }, 100);
 setInterval(() => {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "replicas-computer",
-  "version": "0.3.2",
+  "version": "0.3.4",
   "description": "Replicas workspace computer-use runtime",
   "type": "module",
   "main": "dist/index.js",