replicas-computer 0.3.14 → 0.3.16

package/README.md

@@ -17,21 +17,21 @@ The versioned computer-use runtime for Replicas workspace images.
 - `xsettingsd` broadcasts theme changes to running GTK apps and Chrome. Chrome follows the dashboard theme without reopening tabs; launch it without a `GTK_THEME` override.
 - `assets/desktop/` owns the desktop theme: `brand-wallpaper.svg` (regenerate with `bun scripts/render-wallpaper.ts`), `tint2rc` dock, `openbox-theme/`, `gtk-settings.ini`, the `replicas-terminal` wrapper, `themes/` palettes that `start-services.sh` renders into those templates (`--theme light|dark` switches a running desktop), and the bundled fonts (static TTF instances of General Sans and Geist Mono; variable and WOFF2 builds render with wrong metrics in Pango and VTE).
 
-The viewer and agent command paths are independent. One on-demand encoder pair is shared by all viewers, stops when the last viewer disconnects, and restarts without reloading the page. A viewer out of sight (the dashboard keeps one connected while the Desktop tab is hidden) reports `standby`, detected with an `IntersectionObserver` because a hidden iframe keeps its last layout size; while every viewer is in standby the encoder runs at 5 fps on the 1.5 Mbps rung, and the first viewer to become visible swaps it back to full rate without a gap.
+The viewer and agent command paths are independent. `start-services.sh` runs Xvfb, the compositor, PulseAudio and the stream server (and so the encoders it spawns) at nice −10 when `sudo` is available, so CPU load from the apps on the desktop slows those apps instead of stalling the picture of them. One on-demand encoder pair is shared by all viewers, stops when the last viewer disconnects, and restarts without reloading the page. A viewer out of sight (the dashboard keeps one connected while the Desktop tab is hidden) reports `standby`, detected with an `IntersectionObserver` because a hidden iframe keeps its last layout size; while every viewer is in standby the encoder runs at 5 fps on the 1.5 Mbps rung, and the first viewer to become visible returns it to full rate in place.
 
 Chrome disables GPU compositing when no `/dev/dri/renderD*` device is available. This avoids Mesa's CPU-rendered compositor overhead before capture while retaining WebGL support. Desktops with a hardware render device keep Chrome's normal compositing path.
 
 ## How the stream works
 
-The Go capture process listens for XDamage updates, copies the current screen with XShm, converts RGB to YUV, and encodes H.264 through libx264 directly into RTP on loopback. The libyuv path copies and converts the changed bounding rectangle, aligned to preserve 4:2:0 chroma samples, while reusing unchanged pixels. It captures up to 60fps when the desktop changes and sends 5fps keepalives while idle. FFmpeg supplies Opus audio and remains the video fallback when the native binary is missing or fails. The stream uses the desktop resolution, 1920×1080 by default. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
+The Go capture process listens for XDamage updates, copies the current screen with XShm, converts RGB to YUV, and encodes H.264 through libx264 directly into RTP on loopback. The libyuv path copies and converts the changed bounding rectangle, aligned to preserve 4:2:0 chroma samples, while reusing unchanged pixels. It captures up to 60fps when the desktop changes and sends 5fps keepalives while idle. It reads control lines on stdin between frames: `keyframe` encodes the next frame as an IDR at once, `quality <crf> <kbps>` changes the rate control through `x264_encoder_reconfig`, and `fps <n>` changes the capture rate, so none of them restarts the process. FFmpeg supplies Opus audio and remains the video fallback when the native binary is missing or fails. The stream uses the desktop resolution, 1920×1080 by default. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
 
-WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. The watchdog tracks decoded frames so CSS-hidden viewers do not fall back merely because the browser stops presenting video. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without decoded video switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser.
+WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. The watchdog tracks decoded frames so CSS-hidden viewers do not fall back merely because the browser stops presenting video. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without decoded video (at least three after the connection completes, so a slow network is not abandoned just as it connects) switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser. A viewer that fell back after a WebRTC failure keeps trying WebRTC in the background, 15 seconds after the fallback and then at doubling intervals up to five minutes: the server opens a new peer while the socket keeps carrying media, the new peer plays into an unseen element without offering 4:4:4, and once it decodes video the viewer moves its picture, audio and input to it and tells the server to stop the socket media (`retry-webrtc`, then `webrtc-ready`). The dashboard records each return as `desktop_stream_webrtc_recovered` with the time spent on the socket.
 
 The server makes the offer, so payload types and SSRCs are fixed and ffmpeg emits them directly. Four H.264 payload types are offered (High 4:4:4 Predictive, High, Main, Constrained Baseline); the encoder runs the best profile every connected viewer accepts, so one viewer without 4:4:4 support moves everyone to 4:2:0, and socket viewers or caps below 8 Mbps do the same. Two profiles select the encoder tuning: `balanced` (CRF 14, default chroma quantisation) and `text` (CRF 10, chroma QP offset −6); `?profile=` picks one and the most recent viewer's choice applies to the shared encoder. Quality adapts on a bitrate ladder down to 750 kbps that keeps resolution and frame rate and adds two CRF per rung under the default 20 Mbps cap; `?adapt=0` pins the top rung.
 
-Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It only bounds how far a severe descent (heavy loss or a starved decoder) steps, and only while it has stopped climbing below the sent rate. `scripts/probe-viewer.ts` drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; `scripts/probe-rtt-jitter.ts` feeds the controller a synthetic WebRTC viewer with jittery RTT probes.
+Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. Loss, RTT growth and jitter count only while the encoder has sent over 500 kbps for three seconds: on a quiet screen they belong to the network, and no rung fixes them. Climbing back to the rung below the one a descent left takes a few clean seconds; retrying that rung waits 15, 30 and then 60 seconds as episodes repeat, and an episode is forgotten after three minutes. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It and the browser's available-bitrate estimate only bound how far a severe descent (heavy loss or a starved decoder) steps, and only while the stream has been sending more than they report. `scripts/probe-viewer.ts` drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; `scripts/probe-rtt-jitter.ts` feeds the controller a synthetic WebRTC viewer with jittery RTT probes.
 
-The superfast preset uses integer-pixel motion search to reduce encoder CPU while retaining CABAC compression; ultrafast is unchanged. Encoding uses at most four threads and one queued capture frame, skips rapid catch-up captures after CPU stalls, and uses periodic intra refresh for WebRTC viewers; socket viewers switch it to 250 ms IDR intervals because they resynchronise only on IDR frames. When encoder settings change, a replacement discards two initial frames, emits a forced keyframe and then takes over. Every new encoder has four seconds to produce joinable media. Native startup or replacement failures switch subsequent launches to FFmpeg; the active encoder continues until its replacement is ready.
+The superfast preset uses integer-pixel motion search to reduce encoder CPU while retaining CABAC compression; ultrafast is unchanged. Encoding uses at most four threads and one queued capture frame and skips rapid catch-up captures after CPU stalls. Native capture sends no periodic keyframes or refresh wave: an IDR follows within a frame when a viewer joins, a receiver sends PLI, a send queue is cut, the viewer's watchdog sees no video, or a socket viewer whose frames were withheld has drained its backlog or reset its decoder. Requests within a round trip of the last keyframe share it. Browsers join only at an IDR, so the SPS that intra refresh puts before a recovery point is not treated as a keyframe. Rung and standby changes are applied in place; a change of H.264 profile, chroma or preset starts a replacement, which discards two initial frames, emits a forced keyframe and then takes over. The FFmpeg fallback cannot be told to make a keyframe, so it keeps periodic intra refresh for WebRTC viewers, switches to 250 ms IDR intervals for socket viewers, and makes requested keyframes by replacement. Every new encoder has four seconds to produce joinable media. Native startup or replacement failures switch subsequent launches to FFmpeg; the active encoder continues until its replacement is ready.
 
 The viewer exposes `?view=fit|native|pixel` (letterboxed, one source pixel per CSS pixel, one source pixel per device pixel) and `?diagnostics=1`, which shows transport and ICE route, negotiated codec profile, source/decoded/CSS/device dimensions, displayed and decoded frame rates, bitrate, RTT, loss, jitter buffer, the server's encoder settings, keyframe interval, CPU, pacer budget, queue, stale cuts, repairs and copies, the socket fallback's unsent bytes, withheld delta frames and decoder resets, and the latest server and viewer events. `globalThis.__replicasDesktop` carries the same state for tooling.
 
@@ -44,7 +44,7 @@ The server records every decision that changes the picture (viewer connects and
 | `REPLICAS_DESKTOP_STREAM_TRANSPORT` | `webrtc` | Transport for viewers whose URL does not choose one |
 | `REPLICAS_DESKTOP_STREAM_PROFILE` | `balanced` | Encoder profile for viewers whose URL does not choose one |
 | `REPLICAS_DESKTOP_STREAM_ADAPTIVE` | `1` | `0` pins the top quality rung by default |
-| `REPLICAS_DESKTOP_STREAM_ICE_SERVERS` | `stun:stun.l.google.com:19302` | Comma-separated STUN/TURN servers; `turn:user:secret@host:port?transport=tcp` adds a relay. Percent-encode the username and password; the browser receives their decoded values. Re-read from the workspace runtime env on every viewer, so the monolith's `DESKTOP_STREAM_ICE_SERVERS` reaches running workspaces |
+| `REPLICAS_DESKTOP_STREAM_ICE_SERVERS` | `stun:stun.l.google.com:19302` | Comma-separated STUN/TURN servers; `turn:user:secret@host:port?transport=tcp` adds a relay. Percent-encode the username and password; the browser receives their decoded values. Re-read from the workspace runtime env on every viewer, so the monolith's `DESKTOP_STREAM_ICE_SERVERS` reaches running workspaces. Short-lived credentials for the [TURN relay](../services/turn/README.md) are fetched from the monolith and appended; the server itself only uses UDP relays |
 | `REPLICAS_DESKTOP_STREAM_ICE_PORTS` | ephemeral | `begin-end` UDP port range for the server's ICE candidates |
 | `REPLICAS_DESKTOP_STREAM_WEBRTC` | `1` | `0` serves the WebSocket transport only |
 | `REPLICAS_DESKTOP_STREAM_COLOR` | `bt709` | `legacy` disables the explicit BT.709 conversion and signalling |

package/assets/desktop/start-services.sh

@@ -82,6 +82,16 @@ start_bg() {
   echo $! > "$pid_file"
 }
 
+# The stream, the X server it reads, the compositor that draws into it and the
+# audio server outrank the apps they show, so CPU load from those apps slows
+# the apps rather than stalling the picture of them. The encoders the stream
+# server spawns inherit its priority. Without sudo they run at normal priority.
+prioritize() {
+  local pid
+  pid="$(cat "$STATE_DIR/$1.pid" 2>/dev/null)"
+  [ -n "$pid" ] && sudo -n renice -n -10 -p "$pid" >/dev/null 2>&1 || true
+}
+
 count_lines() {
   awk 'NF { count++ } END { print count + 0 }'
 }
@@ -346,6 +356,7 @@ EOF
 start_bg xvfb \
   "pgrep -af 'Xvfb :$DISPLAY_NUM '" \
   "exec Xvfb :$DISPLAY_NUM -screen 0 ${WIDTH}x${HEIGHT}x${DEPTH} -ac +extension RANDR +extension GLX"
+prioritize xvfb
 
 for _ in $(seq 1 30); do
   if DISPLAY=":$DISPLAY_NUM" xdpyinfo >/dev/null 2>&1; then break; fi
@@ -363,6 +374,7 @@ start_bg xsettingsd \
 start_bg pulseaudio \
   "pactl info" \
   "exec pulseaudio --daemonize=no --exit-idle-time=-1 --log-target=stderr"
+prioritize pulseaudio
 
 # Paint twice because some themes reset root pixmap during startup.
 paint_wallpaper
@@ -388,6 +400,7 @@ if [ "$COMPOSITOR" = "1" ]; then
   start_bg picom \
     "false" \
     "DISPLAY=:$DISPLAY_NUM exec picom --config $HOME/.config/picom/picom.conf --no-fading-openclose"
+  prioritize picom
 fi
 
 if [ "$ENABLE_NOVNC_FALLBACK" = "1" ]; then
@@ -426,6 +439,7 @@ sudo sysctl -q -w net.core.rmem_max=8388608 net.core.wmem_max=8388608 >/dev/null
 start_bg stream \
   "has_stream" \
   "exec env REPLICAS_DESKTOP_DISPLAY=:$DISPLAY_NUM REPLICAS_DESKTOP_WIDTH=$WIDTH REPLICAS_DESKTOP_HEIGHT=$HEIGHT ${REPLICAS_DESKTOP_STREAM_FPS:+REPLICAS_DESKTOP_STREAM_FPS=$REPLICAS_DESKTOP_STREAM_FPS }REPLICAS_DESKTOP_STREAM_PORT=$STREAM_PORT REPLICAS_DESKTOP_STREAM_WEBROOT=${REPLICAS_DESKTOP_STREAM_WEBROOT:-$REPLICAS_DESKTOP_TEMPLATES/stream} replicas-computer stream"
+prioritize stream
 
 # Takes Chrome's cold-start cost at boot instead of on the user's first click;
 # later launches join this instance through the profile singleton.