replicas-computer 0.2.15 → 0.3.0

package/README.md

@@ -4,14 +4,44 @@ The versioned computer-use runtime for Replicas workspace images.
 
 - `src/daemon.ts` owns the serialized agent command boundary.
 - `src/commands/` owns desktop, browser, and recording operations.
-- `src/media-server.ts` owns the shared H.264 and Opus encoder, WebSocket transport, and input bridge.
-- `src/stream-viewer.ts` owns browser decoding, audio playback, reconnects, and interactive input.
+- `src/media-server.ts` owns the shared H.264 and Opus encoders, the viewer control plane, and the quality controller.
+- `src/encoder.ts` supervises each ffmpeg process and keeps the active encoder running during replacement.
+- `src/webrtc.ts` owns one WebRTC peer per viewer (node-datachannel / libdatachannel) and parses receiver feedback.
+- `src/pacer.ts` owns the per-viewer send budget: fresh media, NACK repairs and protective copies share one token bucket.
+- `src/rtp.ts` owns RTP continuity, keyframe detection, and frame reassembly for the socket fallback.
+- `src/stream-viewer.ts` owns the browser side: WebRTC playback, input routing, the WebSocket fallback, diagnostics and reconnects.
 - `src/viewer.ts` owns the optional noVNC fallback.
 - `assets/desktop/start-services.sh` owns the X11 and stream process lifecycle.
 - `xsettingsd` broadcasts theme changes to running GTK apps and Chrome. Chrome follows the dashboard theme without reopening tabs; launch it without a `GTK_THEME` override.
 - `assets/desktop/` owns the desktop theme: `brand-wallpaper.svg` (regenerate with `bun scripts/render-wallpaper.ts`), `tint2rc` dock, `openbox-theme/`, `gtk-settings.ini`, the `replicas-terminal` wrapper, `themes/` palettes that `start-services.sh` renders into those templates (`--theme light|dark` switches a running desktop), and the bundled fonts (static TTF instances of General Sans and Geist Mono; variable and WOFF2 builds render with wrong metrics in Pango and VTE).
 
-The viewer and agent command paths are independent. One on-demand encoder is shared by all viewers, stops when the last viewer disconnects, and restarts without reloading the page. The default 1920×1080 stream uses low-latency H.264 and Opus; noVNC remains available as an opt-in fallback.
+The viewer and agent command paths are independent. One on-demand encoder pair is shared by all viewers, stops when the last viewer disconnects, and restarts without reloading the page.
+
+## How the stream works
+
+ffmpeg captures the display and encodes 1920×1080 H.264 (and Opus from PulseAudio) as RTP onto loopback. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
+
+WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without a displayed frame switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser.
+
+The server makes the offer, so payload types and SSRCs are fixed and ffmpeg emits them directly. Four H.264 payload types are offered (High 4:4:4 Predictive, High, Main, Constrained Baseline); the encoder runs the best profile every connected viewer accepts, so one viewer without 4:4:4 support moves everyone to 4:2:0, and socket viewers or caps below 8 Mbps do the same. Two profiles select the encoder tuning: `balanced` (CRF 14, default chroma quantisation) and `text` (CRF 10, chroma QP offset −6); `?profile=` picks one and the most recent viewer's choice applies to the shared encoder. Quality adapts on a six-rung bitrate ladder that keeps resolution and frame rate and adds two CRF per rung under the 20 Mbps cap; `?adapt=0` pins the top rung.
+
+Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP; a REMB estimate below the sent rate, RTT growth, loss, jitter-buffer growth and stale-queue cuts all drive the ladder down, and clean seconds drive it up.
+
+Encoding uses at most four threads and one queued capture frame, skips rapid catch-up captures after CPU stalls, and uses periodic intra refresh for WebRTC viewers; socket viewers switch it to 250 ms IDR intervals because they resynchronise only on IDR frames. When encoder settings change, a replacement discards two initial frames, emits a forced keyframe and then takes over. The active encoder continues if replacement fails.
+
+The viewer exposes `?view=fit|native|pixel` (letterboxed, one source pixel per CSS pixel, one source pixel per device pixel) and `?diagnostics=1`, which shows transport and ICE route, negotiated codec profile, source/decoded/CSS/device dimensions, displayed and decoded frame rates, bitrate, RTT, loss, jitter buffer, and the server's encoder settings, CPU, pacer budget, queue, stale cuts, repairs and copies. `globalThis.__replicasDesktop` carries the same state for tooling.
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `REPLICAS_DESKTOP_STREAM_TRANSPORT` | `webrtc` | Transport for viewers whose URL does not choose one |
+| `REPLICAS_DESKTOP_STREAM_PROFILE` | `balanced` | Encoder profile for viewers whose URL does not choose one |
+| `REPLICAS_DESKTOP_STREAM_ADAPTIVE` | `1` | `0` pins the top quality rung by default |
+| `REPLICAS_DESKTOP_STREAM_ICE_SERVERS` | `stun:stun.l.google.com:19302` | Comma-separated STUN/TURN servers; `turn:user:secret@host:port?transport=tcp` adds a relay. Re-read from the workspace runtime env on every viewer, so the monolith's `DESKTOP_STREAM_ICE_SERVERS` reaches running workspaces |
+| `REPLICAS_DESKTOP_STREAM_ICE_PORTS` | ephemeral | `begin-end` UDP port range for the server's ICE candidates |
+| `REPLICAS_DESKTOP_STREAM_WEBRTC` | `1` | `0` serves the WebSocket transport only |
+| `REPLICAS_DESKTOP_STREAM_COLOR` | `bt709` | `legacy` disables the explicit BT.709 conversion and signalling |
+| `REPLICAS_DESKTOP_STREAM_VIDEO_BITRATE` | `20M` | Encoder bitrate cap across all quality rungs |
+| `REPLICAS_DESKTOP_STREAM_FPS` | `60` | Capture and encode rate |
 
 ```bash
 replicas-computer setup --check
@@ -19,4 +49,6 @@ sudo replicas-computer setup
 replicas-computer daemon
 ```
 
-`setup --check` reports missing OS dependencies without changing the machine. `setup` installs package-owned assets and the `replicas-computer.service` systemd unit. Operational commands use the daemon over a user-only Unix socket.
+`setup --check` reports missing OS dependencies, including the native WebRTC module, without changing the machine. `setup` installs package-owned assets and the `replicas-computer.service` systemd unit. Operational commands use the daemon over a user-only Unix socket.
+
+The sandbox image builds node-datachannel from pinned libdatachannel source with `scripts/build-webrtc.sh`, which applies `patches/libdatachannel-dtls-bio.patch` (upstream DTLS BIO synchronisation fix, [2059d2a](https://github.com/paullouisageneau/libdatachannel/commit/2059d2a3bb81dc3dc24686fdc92c90b6167ec4fd)) and `patches/libdatachannel-nack-duplicates.patch`. It needs CMake, Ninja, a C++ compiler, Git and OpenSSL development libraries. Remove each patch when the pinned addon includes its fix.

package/assets/desktop/start-services.sh

@@ -418,6 +418,10 @@ fi
 
 kill_pids "$(legacy_stream_websockify_pids)"
 clear_bridge_pid_if_unhealthy stream has_stream stream_listener_pids stream_pids
+# Encoded keyframes leave ffmpeg as a burst of hundreds of UDP packets and go
+# back out the same way over WebRTC; the default 208KB socket buffers drop the
+# tail of the burst and every viewer waits out a GOP.
+sudo sysctl -q -w net.core.rmem_max=8388608 net.core.wmem_max=8388608 >/dev/null 2>&1 || true
 # Forwarded only when set, so the stream server keeps its own default.
 start_bg stream \
   "has_stream" \

package/assets/desktop/stream-index.html

@@ -4,9 +4,9 @@
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
   <title>Replicas Desktop</title>
-  <script src="./viewer-theme.js"></script>
-  <link rel="stylesheet" href="./viewer-theme.css">
-  <link rel="stylesheet" href="./stream-viewer.css">
+  <script src="./viewer-theme.js?v=__VERSION__"></script>
+  <link rel="stylesheet" href="./viewer-theme.css?v=__VERSION__">
+  <link rel="stylesheet" href="./stream-viewer.css?v=__VERSION__">
   <link rel="icon" href="./viewer-icon.svg" type="image/svg+xml">
 </head>
 <body>
@@ -18,6 +18,6 @@
   </div>
   <button id="audio-prompt" type="button">Enable audio</button>
   <output id="diagnostics"></output>
-  <script type="module" src="./stream-viewer.js"></script>
+  <script type="module" src="./stream-viewer.js?v=__VERSION__"></script>
 </body>
 </html>

package/assets/desktop/stream-viewer.css

@@ -1,5 +1,7 @@
 html, body { background: var(--replicas-surface); }
 body { display: grid; place-items: center; }
+html[data-view="native"], html[data-view="pixel"] { overflow: auto; }
+html[data-view="native"] body, html[data-view="pixel"] body { width: max-content; min-width: 100%; height: max-content; min-height: 100%; }
 
 #screen {
   display: block;
@@ -35,11 +37,14 @@ body { display: grid; place-items: center; }
   top: 12px;
   right: 12px;
   display: none;
+  max-width: calc(100vw - 24px);
   padding: 6px 10px;
   border: 1px solid var(--replicas-hairline);
   color: var(--replicas-ink);
   background: var(--replicas-panel);
   font-size: 11px;
+  line-height: 1.5;
+  white-space: pre-wrap;
   pointer-events: none;
 }
 

package/assets/desktop/viewer-theme.css

@@ -3,7 +3,7 @@
   font-style: normal;
   font-weight: 400;
   font-display: swap;
-  src: url("./geist-mono-400.ttf") format("truetype");
+  src: url("./geist-mono-400.woff2") format("woff2");
 }
 
 @font-face {
@@ -11,7 +11,7 @@
   font-style: normal;
   font-weight: 600;
   font-display: swap;
-  src: url("./geist-mono-600.ttf") format("truetype");
+  src: url("./geist-mono-600.woff2") format("woff2");
 }
 
 @font-face {
@@ -19,7 +19,7 @@
   font-style: normal;
   font-weight: 700;
   font-display: swap;
-  src: url("./geist-mono-700.ttf") format("truetype");
+  src: url("./geist-mono-700.woff2") format("woff2");
 }
 
 @font-face {
@@ -27,7 +27,7 @@
   font-style: normal;
   font-weight: 300;
   font-display: swap;
-  src: url("./general-sans-300.ttf") format("truetype");
+  src: url("./general-sans-300.woff2") format("woff2");
 }
 
 :root {