replicas-computer 0.3.7 → 0.3.9

package/README.md

@@ -5,7 +5,8 @@ The versioned computer-use runtime for Replicas workspace images.
 - `src/daemon.ts` owns the serialized agent command boundary.
 - `src/commands/` owns desktop, browser, and recording operations.
 - `src/media-server.ts` owns the shared H.264 and Opus encoders, the viewer control plane, and the quality controller.
-- `src/encoder.ts` supervises each ffmpeg process and keeps the active encoder running during replacement.
+- `src/encoder.ts` supervises each encoder process and keeps the active encoder running during replacement.
+- `src/video-encoder.ts` owns video backend selection, arguments, fallback, and capture metrics.
 - `src/webrtc.ts` owns one WebRTC peer per viewer (node-datachannel / libdatachannel) and parses receiver feedback.
 - `src/pacer.ts` owns the per-viewer send budget: fresh media, NACK repairs and protective copies share one token bucket.
 - `src/rtp.ts` owns RTP continuity, keyframe detection, and frame reassembly for the socket fallback.
@@ -18,9 +19,11 @@ The versioned computer-use runtime for Replicas workspace images.
 
 The viewer and agent command paths are independent. One on-demand encoder pair is shared by all viewers, stops when the last viewer disconnects, and restarts without reloading the page.
 
+Chrome disables GPU compositing when no `/dev/dri/renderD*` device is available. This avoids Mesa's CPU-rendered compositor overhead before capture while retaining WebGL support. Desktops with a hardware render device keep Chrome's normal compositing path.
+
 ## How the stream works
 
-ffmpeg captures the display and encodes 1920×1080 H.264 (and Opus from PulseAudio) as RTP onto loopback. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
+The Go capture process listens for XDamage updates, copies the current screen with XShm, converts RGB to YUV, and encodes H.264 through libx264 directly into RTP on loopback. The libyuv path copies and converts the changed bounding rectangle, aligned to preserve 4:2:0 chroma samples, while reusing unchanged pixels. It captures up to 60fps when the desktop changes and sends 5fps keepalives while idle. FFmpeg supplies Opus audio and remains the video fallback when the native binary is missing or fails. The stream uses the desktop resolution, 1920×1080 by default. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
 
 WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. The watchdog tracks decoded frames so CSS-hidden viewers do not fall back merely because the browser stops presenting video. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without decoded video switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser.
 
@@ -28,10 +31,12 @@ The server makes the offer, so payload types and SSRCs are fixed and ffmpeg emit
 
 Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It only bounds how far a severe descent (heavy loss or a starved decoder) steps, and only while it has stopped climbing below the sent rate. `scripts/probe-viewer.ts` drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; `scripts/probe-rtt-jitter.ts` feeds the controller a synthetic WebRTC viewer with jittery RTT probes.
 
-Encoding uses at most four threads and one queued capture frame, skips rapid catch-up captures after CPU stalls, and uses periodic intra refresh for WebRTC viewers; socket viewers switch it to 250 ms IDR intervals because they resynchronise only on IDR frames. When encoder settings change, a replacement discards two initial frames, emits a forced keyframe and then takes over. The active encoder continues if replacement fails.
+The superfast preset uses integer-pixel motion search to reduce encoder CPU while retaining CABAC compression; ultrafast is unchanged. Encoding uses at most four threads and one queued capture frame, skips rapid catch-up captures after CPU stalls, and uses periodic intra refresh for WebRTC viewers; socket viewers switch it to 250 ms IDR intervals because they resynchronise only on IDR frames. When encoder settings change, a replacement discards two initial frames, emits a forced keyframe and then takes over. Every new encoder has four seconds to produce joinable media. Native startup or replacement failures switch subsequent launches to FFmpeg; the active encoder continues until its replacement is ready.
 
 The viewer exposes `?view=fit|native|pixel` (letterboxed, one source pixel per CSS pixel, one source pixel per device pixel) and `?diagnostics=1`, which shows transport and ICE route, negotiated codec profile, source/decoded/CSS/device dimensions, displayed and decoded frame rates, bitrate, RTT, loss, jitter buffer, the server's encoder settings, keyframe interval, CPU, pacer budget, queue, stale cuts, repairs and copies, the socket fallback's unsent bytes, withheld delta frames and decoder resets, and the latest server and viewer events. `globalThis.__replicasDesktop` carries the same state for tooling.
 
+WebRTC receiver timings use one-second counter differences: `decodeMs` is decode time per decoded frame; `receiverProcessingMs` runs from the first received packet to the decoded frame and already includes buffering and decoding. Neither measures physical display latency. Unsupported counters, counter resets and windows without decoded frames leave these fields absent. The timings appear in diagnostics, `__replicasDesktop` and per-viewer telemetry. See the [WebRTC stats definitions](https://www.w3.org/TR/webrtc-stats/#dom-rtcinboundrtpstreamstats-totalprocessingdelay).
+
 The server records every decision that changes the picture (viewer connects and transport fallbacks with their reason, WebRTC state changes, quality rung moves with the signals that voted for them, encoder swaps, preset changes and keyframe requests) and writes each to the stream log (`/tmp/replicas-computer/stream.log`). `GET /telemetry` returns the `/health` snapshot plus the last 200 events and five minutes of one-second samples (encoder rung, CRF, cap, keyframe interval, preset, fps and kbps; CPU; per-viewer transport, ICE path, pacer counters, socket backlog and the viewer's own reported stats). Fetch it inside the sandbox with `curl -s localhost:6080/telemetry` or through the authenticated preview URL; the dashboard's stream settings menu has **Open stream telemetry**. `REPLICAS_DESKTOP_STREAM_TELEMETRY=1` also writes every sample to the stream log for sessions longer than the in-memory history.
 
 | Variable | Default | Meaning |
@@ -44,7 +49,8 @@ The server records every decision that changes the picture (viewer connects and
 | `REPLICAS_DESKTOP_STREAM_WEBRTC` | `1` | `0` serves the WebSocket transport only |
 | `REPLICAS_DESKTOP_STREAM_COLOR` | `bt709` | `legacy` disables the explicit BT.709 conversion and signalling |
 | `REPLICAS_DESKTOP_STREAM_VIDEO_BITRATE` | `20M` | Encoder bitrate cap across all quality rungs |
-| `REPLICAS_DESKTOP_STREAM_FPS` | `60` | Capture and encode rate |
+| `REPLICAS_DESKTOP_STREAM_FPS` | `60` | Maximum capture and encode rate |
+| `REPLICAS_DESKTOP_STREAM_CAPTURE` | native when installed | `ffmpeg` bypasses native capture |
 | `REPLICAS_DESKTOP_STREAM_TELEMETRY` | unset | `1` writes one-second telemetry samples to the stream log while a viewer is connected |
 
 ```bash
@@ -56,3 +62,7 @@ replicas-computer daemon
 `setup --check` reports missing OS dependencies, including the native WebRTC module, without changing the machine. `setup` installs package-owned assets and the `replicas-computer.service` systemd unit. Operational commands use the daemon over a user-only Unix socket.
 
 The sandbox image builds node-datachannel from pinned libdatachannel source with `scripts/build-webrtc.sh`, which applies `patches/libdatachannel-dtls-bio.patch` (upstream DTLS BIO synchronisation fix, [2059d2a](https://github.com/paullouisageneau/libdatachannel/commit/2059d2a3bb81dc3dc24686fdc92c90b6167ec4fd)) and `patches/libdatachannel-nack-duplicates.patch`. It also applies `patches/libjuice-ice-role-presence.patch` so valid zero-valued ICE role attributes do not break Chrome TURN/TCP connectivity. It needs CMake, Ninja, a C++ compiler, Git and OpenSSL development libraries. Remove each patch when the pinned addon includes its fix.
+
+The sandbox image builds the Go module in `native/` with `scripts/build-capture.sh` against its installed FFmpeg and X11 libraries. On x86-64, frames use a pinned, statically linked libyuv build; its chroma-rounding patch keeps SIMD conversion consistent with the reference converter. Builds without libyuv use libswscale for full-frame color conversion at the same resolution. Local builds need Go (the script selects Go 1.27.1), C/C++ compilers, CMake, Git, pkg-config, and development packages for X11, Xext, Xdamage, Xfixes, libavcodec, libavformat, libavutil and libswscale. The module has no third-party Go packages. The capture loop and CLI are ordinary Go; `native/internal/capture` confines cgo calls, native buffers, the small C callback adapter, and explicit cleanup to one package. Its stream stays on one OS thread for Xlib. X11, libyuv, FFmpeg and x264 remain native dependencies. The build enables the `libyuv` tag on x86-64; a plain `go build` uses libswscale only. Run `bun run build:capture` after the TypeScript build; the binary's libyuv license is included in `dist`. `/health` exposes `captureBackend` and native `captureMetrics`: elapsed time, busy time, damage lookup and screen copy, color conversion, encode/RTP time, sent frames and changed frames for the latest interval. Timings are milliseconds of wall time; divide stage totals by changed frames (copy/conversion) or sent frames (encode) for per-frame cost. Receiver FPS is compared with the current one-second sending rate so quiet desktops do not trigger congestion downgrades.
+
+Run `bun run test:capture` on x86-64 with Xvfb and xauth installed. The isolated X11 test compares all YUV planes against a fresh full-frame conversion over 1,024 captures, covering I420/I444, both color matrices, odd damage coordinates, right/bottom clipping, and repeated region-size changes. The `Desktop capture` CI workflow runs it with strict cgo pointer checks.

package/assets/desktop/replicas-chrome

@@ -45,6 +45,12 @@ unset GTK_THEME
 # password-store skips a keyring D-Bus probe that stalls first launch for seconds
 # on a headless image; the network flags keep first-run downloads off the critical path.
 # ignore-gpu-blocklist keeps WebGL available on the GPU-less Xvfb display.
+# Mesa's GPU compositor adds latency on desktops without a hardware renderer.
+CHROME_COMPOSITING_FLAGS=()
+if ! compgen -G '/dev/dri/renderD*' > /dev/null; then
+  CHROME_COMPOSITING_FLAGS=(--disable-gpu-compositing)
+fi
+
 exec /usr/bin/google-chrome-stable \
   --user-data-dir="$PROFILE_DIR" \
   --profile-directory=Default \
@@ -59,5 +65,6 @@ exec /usr/bin/google-chrome-stable \
   --remote-debugging-address=127.0.0.1 \
   --remote-debugging-port="$DEBUG_PORT" \
   --ignore-gpu-blocklist \
+  "${CHROME_COMPOSITING_FLAGS[@]}" \
   --start-maximized \
   "$@"