replicas-computer 0.3.17 → 0.3.18

package/README.md

@@ -23,15 +23,15 @@ Chrome disables GPU compositing when no `/dev/dri/renderD*` device is available.
 
 ## How the stream works
 
-The Go capture process listens for XDamage updates, copies the current screen with XShm, converts RGB to YUV, and encodes H.264 through libx264 directly into RTP on loopback. The libyuv path copies and converts the changed bounding rectangle, aligned to preserve 4:2:0 chroma samples, while reusing unchanged pixels. It captures up to 60fps when the desktop changes and sends 5fps keepalives while idle. It reads control lines on stdin between frames: `keyframe` encodes the next frame as an IDR at once, `quality <crf> <kbps>` changes the rate control through `x264_encoder_reconfig`, and `fps <n>` changes the capture rate, so none of them restarts the process. FFmpeg supplies Opus audio and remains the video fallback when the native binary is missing or fails. The stream uses the desktop resolution, 1920×1080 by default. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
+The Go capture process listens for XDamage updates, copies the current screen with XShm, converts RGB to YUV, and encodes H.264 through libx264 directly into RTP on loopback. The libyuv path copies and converts the changed bounding rectangle, aligned to preserve 4:2:0 chroma samples, while reusing unchanged pixels. It captures up to 60fps when the desktop changes and sends 5fps keepalives while idle. It reads control lines on stdin between frames: `keyframe` encodes the next frame as an IDR at once, `quality <crf> <kbps>` changes the rate control through `x264_encoder_reconfig`, and `fps <n>` changes the capture rate, so none of them restarts the process. x264 keeps the frame rate it opened with and sizes its rate cap by it, so the encoder always opens at the stream's full rate and `--capture-fps` sets the rate capture starts at. FFmpeg supplies Opus audio and remains the video fallback when the native binary is missing or fails. The stream uses the desktop resolution, 1920×1080 by default. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
 
 WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. The watchdog tracks decoded frames so CSS-hidden viewers do not fall back merely because the browser stops presenting video. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without decoded video (at least three after the connection completes, so a slow network is not abandoned just as it connects) switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser. A viewer that fell back after a WebRTC failure keeps trying WebRTC in the background, 15 seconds after the fallback and then at doubling intervals up to five minutes: the server opens a new peer while the socket keeps carrying media, the new peer plays into an unseen element without offering 4:4:4, and once it decodes video the viewer moves its picture, audio and input to it and tells the server to stop the socket media (`retry-webrtc`, then `webrtc-ready`). The dashboard records each return as `desktop_stream_webrtc_recovered` with the time spent on the socket.
 
 The server makes the offer, so payload types and SSRCs are fixed and ffmpeg emits them directly. Four H.264 payload types are offered (High 4:4:4 Predictive, High, Main, Constrained Baseline); the encoder runs the best profile every connected viewer accepts, so one viewer without 4:4:4 support moves everyone to 4:2:0, and socket viewers or caps below 8 Mbps do the same. Two profiles select the encoder tuning: `balanced` (CRF 14, default chroma quantisation) and `text` (CRF 10, chroma QP offset −6); `?profile=` picks one and the most recent viewer's choice applies to the shared encoder. Quality adapts on a bitrate ladder down to 750 kbps that keeps resolution and frame rate and adds two CRF per rung under the default 20 Mbps cap; `?adapt=0` pins the top rung.
 
-Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. Loss, RTT growth and jitter count only while the encoder has sent over 500 kbps for three seconds: on a quiet screen they belong to the network, and no rung fixes them. Climbing back to the rung below the one a descent left takes a few clean seconds; retrying that rung waits 15, 30 and then 60 seconds as episodes repeat, and an episode is forgotten after three minutes. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It and the browser's available-bitrate estimate only bound how far a severe descent (heavy loss or a starved decoder) steps, and only while the stream has been sending more than they report. `scripts/probe-viewer.ts` drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; `scripts/probe-rtt-jitter.ts` feeds the controller a synthetic WebRTC viewer with jittery RTT probes.
+Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. Loss, RTT growth and jitter count only while the encoder has sent over 500 kbps for three seconds: on a quiet screen they belong to the network, and no rung fixes them. Climbing back to the rung below the one a descent left takes a few clean seconds; retrying that rung waits 15, 30 and then 60 seconds as episodes repeat, and an episode is forgotten after three minutes. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It only bounds how far a severe descent (heavy loss, a starved decoder or a cut send queue) steps, and only once it has stopped climbing below what the stream sends. The browser's available-bitrate stat is the same estimate and is reported but never used. `scripts/probe-viewer.ts` drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; `scripts/probe-rtt-jitter.ts` feeds the controller a synthetic WebRTC viewer with jittery RTT probes.
 
-The superfast preset uses integer-pixel motion search to reduce encoder CPU while retaining CABAC compression; ultrafast is unchanged. Encoding uses at most four threads and one queued capture frame and skips rapid catch-up captures after CPU stalls. Native capture sends no periodic keyframes or refresh wave: an IDR follows within a frame when a viewer joins, a receiver sends PLI, a send queue is cut, the viewer's watchdog sees no video, or a socket viewer whose frames were withheld has drained its backlog or reset its decoder. Requests within a round trip of the last keyframe share it. Browsers join only at an IDR, so the SPS that intra refresh puts before a recovery point is not treated as a keyframe. Rung and standby changes are applied in place; a change of H.264 profile, chroma or preset starts a replacement, which discards two initial frames, emits a forced keyframe and then takes over. The FFmpeg fallback cannot be told to make a keyframe, so it keeps periodic intra refresh for WebRTC viewers, switches to 250 ms IDR intervals for socket viewers, and makes requested keyframes by replacement. Every new encoder has four seconds to produce joinable media. Native startup or replacement failures switch subsequent launches to FFmpeg; the active encoder continues until its replacement is ready.
+The superfast preset uses integer-pixel motion search to reduce encoder CPU while retaining CABAC compression; ultrafast is unchanged. Encoding uses at most four threads and one queued capture frame and skips rapid catch-up captures after CPU stalls. Native capture sends no periodic keyframes or refresh wave: an IDR follows within a frame when a viewer joins, a receiver sends PLI, a send queue is cut, the viewer's watchdog sees no video, or a socket viewer whose frames were withheld has drained its backlog or reset its decoder. Requests within a round trip of the last keyframe share it. Browsers join only at an IDR, so the SPS that intra refresh puts before a recovery point is not treated as a keyframe. Rung and standby changes are applied in place; a change of H.264 profile, chroma or preset starts a replacement, which discards two initial frames, emits a forced keyframe and then takes over. Until then the outgoing encoder takes the new rate, and keyframe requests wait for the replacement's. The FFmpeg fallback cannot be told to make a keyframe, so it keeps periodic intra refresh for WebRTC viewers, switches to 250 ms IDR intervals for socket viewers, and makes requested keyframes by replacement. Every new encoder has four seconds to produce joinable media. Native startup or replacement failures switch subsequent launches to FFmpeg; the active encoder continues until its replacement is ready.
 
 The viewer exposes `?view=fit|native|pixel` (letterboxed, one source pixel per CSS pixel, one source pixel per device pixel) and `?diagnostics=1`, which shows transport and ICE route, negotiated codec profile, source/decoded/CSS/device dimensions, displayed and decoded frame rates, bitrate, RTT, loss, jitter buffer, the server's encoder settings, keyframe interval, CPU, pacer budget, queue, stale cuts, repairs and copies, the socket fallback's unsent bytes, withheld delta frames and decoder resets, and the latest server and viewer events. `globalThis.__replicasDesktop` carries the same state for tooling.
 

package/dist/index.js

@@ -2937,7 +2937,7 @@ async function computerStatusCommand() {
 }
 
 // src/version.ts
-var COMPUTER_USE_VERSION = "0.3.17";
+var COMPUTER_USE_VERSION = "0.3.18";
 var COMPUTER_USE_PROTOCOL_VERSION = 1;
 
 // src/program.ts
@@ -3478,13 +3478,21 @@ function superviseEncoder(options) {
     if (active || candidate) return;
     await spawnCandidate();
   });
-  const replace = () => enqueue(async () => {
-    if (!wanted) return;
-    clearTimeout(restart);
-    clearTimeout(candidateDeadline);
-    retire(candidate);
-    await spawnCandidate();
-  });
+  let queuedReplacements = 0;
+  const replace = () => {
+    queuedReplacements += 1;
+    return enqueue(async () => {
+      try {
+        if (!wanted) return;
+        clearTimeout(restart);
+        clearTimeout(candidateDeadline);
+        retire(candidate);
+        await spawnCandidate();
+      } finally {
+        queuedReplacements -= 1;
+      }
+    });
+  };
   const stop = () => enqueue(async () => {
     wanted = false;
     clearTimeout(restart);
@@ -3507,7 +3515,7 @@ function superviseEncoder(options) {
     stop,
     control,
     running: () => Boolean(active && alive(active.child)),
-    replacing: () => Boolean(candidate),
+    replacing: () => Boolean(candidate) || queuedReplacements > 0,
     generation: () => active?.id ?? 0,
     pid: () => active?.child.pid ?? 0,
     command: () => active?.child.spawnfile,
@@ -4200,6 +4208,8 @@ function superviseVideoEncoder(options) {
         "--height",
         String(height2),
         "--fps",
+        String(options.maxFps),
+        "--capture-fps",
         String(fps2),
         "--threads",
         String(Math.min(4, availableParallelism())),
@@ -4807,6 +4817,7 @@ async function runMediaServer() {
     width,
     height,
     colorMode,
+    maxFps: fps,
     params: encoderParams,
     onPacket: onVideoPacket,
     record
@@ -4882,17 +4893,16 @@ async function runMediaServer() {
     }
     if (description !== encoderDescription) record("encoder", description);
     encoderDescription = description;
-    if (signature !== encoderSignature) {
+    const replacing = signature !== encoderSignature;
+    if (replacing) {
       encoderSignature = signature;
-      encoderLive = live;
       lastSwapAt = Date.now();
       void videoEncoder.replace();
-      return true;
     }
     if (live.crf !== encoderLive.crf || live.kbps !== encoderLive.kbps) videoEncoder.control(`quality ${live.crf} ${live.kbps}`);
     if (live.fps !== encoderLive.fps) videoEncoder.control(`fps ${live.fps}`);
     encoderLive = live;
-    return false;
+    return replacing;
   };
   const syncEncoder = () => videoWanted && viewers.size > 0 && ensureEncoders();
   const stopEncodersSoon = () => {
@@ -4974,12 +4984,9 @@ async function runMediaServer() {
     congestedSeconds = 0;
     if (!adaptive || now - lastSevereDescentAt < 1e3) return;
     lastSevereDescentAt = now;
-    const sent = Math.min(...recentVideoKbps);
     const estimates = [...viewers.values()].flatMap((viewer) => {
-      const available = viewer.stats && now - viewer.stats.at < 750 ? viewer.stats.availableKbps ?? 0 : 0;
-      const reported = sent > 500 && available < sent * 0.9 ? available : 0;
-      const remb = rembBelowSent(viewer, now) ? freshRemb(viewer, now) ?? 0 : 0;
-      return [...reported > 0 ? [reported] : [], ...remb > 0 ? [remb] : []];
+      const remb = rembBelowSent(viewer, now) ? freshRemb(viewer, now) : void 0;
+      return remb ? [remb] : [];
     });
     const rungs = ladder();
     const estimatedRung = estimates.length ? rungs.findIndex((quality) => quality.kbps <= Math.min(...estimates) * 0.8) : rung + 2;

package/native/internal/capture/options.go

@@ -1,7 +1,7 @@
 package capture
 
 type Options struct {
-	Width, Height, FPS, Threads, CRF, Kbps                     int
+	Width, Height, FPS, CaptureFPS, Threads, CRF, Kbps         int
 	KeyframeMS, IntraRefresh, ChromaQP, Warmup, Payload, BT709 int
 	Preset, Profile, URL, SSRC                                 string
 }

package/native/main.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"bytes"
+	"cmp"
 	"fmt"
 	"os"
 	"os/signal"
@@ -59,7 +60,10 @@ func run(o capture.Options) error {
 	}
 	started := capture.NowMicros()
 	last, lastKeyframe, reportAt := started-1000000, started-1000000, started
-	interval := int64((1000000 + o.FPS - 1) / o.FPS)
+	// x264 keeps the frame rate it opened with and sizes its rate cap by it, so
+	// --fps is the stream's full rate and --capture-fps the rate capture starts at.
+	captureFPS := cmp.Or(o.CaptureFPS, o.FPS)
+	interval := int64((1000000 + captureFPS - 1) / captureFPS)
 	busyUS, captureUS, scaleUS, encodeUS, previousPTS := int64(0), int64(0), int64(0), int64(0), int64(-1)
 	dirty, forceKeyframe := true, false
 	number, changedFrames, sentFrames := 0, 0, 0

package/native/options.go

@@ -18,7 +18,7 @@ func parseOptions() (capture.Options, error) {
 		initial, low, high int
 	}{
 		{"width", &o.Width, 0, 2, 8192}, {"height", &o.Height, 0, 2, 8192},
-		{"fps", &o.FPS, 60, 1, 240}, {"threads", &o.Threads, 4, 1, 16},
+		{"fps", &o.FPS, 60, 1, 240}, {"capture-fps", &o.CaptureFPS, 0, 1, 240}, {"threads", &o.Threads, 4, 1, 16},
 		{"crf", &o.CRF, 14, 0, 51}, {"kbps", &o.Kbps, 20000, 1, 1000000},
 		{"keyframe-ms", &o.KeyframeMS, 1000, 0, 10000}, {"intra-refresh", &o.IntraRefresh, 1, 0, 1},
 		{"chroma-qp", &o.ChromaQP, 0, -12, 12}, {"warmup", &o.Warmup, 0, 0, 60},

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "replicas-computer",
-  "version": "0.3.17",
+  "version": "0.3.18",
   "description": "Replicas workspace computer-use runtime",
   "type": "module",
   "main": "dist/index.js",