replicas-computer 0.3.14 → 0.3.15

package/README.md

@@ -17,21 +17,21 @@ The versioned computer-use runtime for Replicas workspace images.
 - `xsettingsd` broadcasts theme changes to running GTK apps and Chrome. Chrome follows the dashboard theme without reopening tabs; launch it without a `GTK_THEME` override.
 - `assets/desktop/` owns the desktop theme: `brand-wallpaper.svg` (regenerate with `bun scripts/render-wallpaper.ts`), `tint2rc` dock, `openbox-theme/`, `gtk-settings.ini`, the `replicas-terminal` wrapper, `themes/` palettes that `start-services.sh` renders into those templates (`--theme light|dark` switches a running desktop), and the bundled fonts (static TTF instances of General Sans and Geist Mono; variable and WOFF2 builds render with wrong metrics in Pango and VTE).
 
-The viewer and agent command paths are independent. One on-demand encoder pair is shared by all viewers, stops when the last viewer disconnects, and restarts without reloading the page. A viewer out of sight (the dashboard keeps one connected while the Desktop tab is hidden) reports `standby`, detected with an `IntersectionObserver` because a hidden iframe keeps its last layout size; while every viewer is in standby the encoder runs at 5 fps on the 1.5 Mbps rung, and the first viewer to become visible swaps it back to full rate without a gap.
+The viewer and agent command paths are independent. `start-services.sh` runs Xvfb, the compositor, PulseAudio and the stream server (and so the encoders it spawns) at nice −10 when `sudo` is available, so CPU load from the apps on the desktop slows those apps instead of stalling the picture of them. One on-demand encoder pair is shared by all viewers, stops when the last viewer disconnects, and restarts without reloading the page. A viewer out of sight (the dashboard keeps one connected while the Desktop tab is hidden) reports `standby`, detected with an `IntersectionObserver` because a hidden iframe keeps its last layout size; while every viewer is in standby the encoder runs at 5 fps on the 1.5 Mbps rung, and the first viewer to become visible returns it to full rate in place.
 
 Chrome disables GPU compositing when no `/dev/dri/renderD*` device is available. This avoids Mesa's CPU-rendered compositor overhead before capture while retaining WebGL support. Desktops with a hardware render device keep Chrome's normal compositing path.
 
 ## How the stream works
 
-The Go capture process listens for XDamage updates, copies the current screen with XShm, converts RGB to YUV, and encodes H.264 through libx264 directly into RTP on loopback. The libyuv path copies and converts the changed bounding rectangle, aligned to preserve 4:2:0 chroma samples, while reusing unchanged pixels. It captures up to 60fps when the desktop changes and sends 5fps keepalives while idle. FFmpeg supplies Opus audio and remains the video fallback when the native binary is missing or fails. The stream uses the desktop resolution, 1920×1080 by default. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
+The Go capture process listens for XDamage updates, copies the current screen with XShm, converts RGB to YUV, and encodes H.264 through libx264 directly into RTP on loopback. The libyuv path copies and converts the changed bounding rectangle, aligned to preserve 4:2:0 chroma samples, while reusing unchanged pixels. It captures up to 60fps when the desktop changes and sends 5fps keepalives while idle. It reads control lines on stdin between frames: `keyframe` encodes the next frame as an IDR at once, `quality <crf> <kbps>` changes the rate control through `x264_encoder_reconfig`, and `fps <n>` changes the capture rate, so none of them restarts the process. FFmpeg supplies Opus audio and remains the video fallback when the native binary is missing or fails. The stream uses the desktop resolution, 1920×1080 by default. RGB capture is converted with the BT.709 matrix in limited range and the same matrix is signalled in the bitstream, so browsers reconstruct the colours the desktop drew (`REPLICAS_DESKTOP_STREAM_COLOR=legacy` restores the untagged BT.601 conversion). Frames are forwarded unchanged apart from a sequence/timestamp splice and a playout-delay header extension.
 
-WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. The watchdog tracks decoded frames so CSS-hidden viewers do not fall back merely because the browser stops presenting video. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without decoded video switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser.
+WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. The watchdog tracks decoded frames so CSS-hidden viewers do not fall back merely because the browser stops presenting video. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without decoded video (at least three after the connection completes, so a slow network is not abandoned just as it connects) switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser. A viewer that fell back after a WebRTC failure keeps trying WebRTC in the background, 15 seconds after the fallback and then at doubling intervals up to five minutes: the server opens a new peer while the socket keeps carrying media, the new peer plays into an unseen element without offering 4:4:4, and once it decodes video the viewer moves its picture, audio and input to it and tells the server to stop the socket media (`retry-webrtc`, then `webrtc-ready`). The dashboard records each return as `desktop_stream_webrtc_recovered` with the time spent on the socket.
 
 The server makes the offer, so payload types and SSRCs are fixed and ffmpeg emits them directly. Four H.264 payload types are offered (High 4:4:4 Predictive, High, Main, Constrained Baseline); the encoder runs the best profile every connected viewer accepts, so one viewer without 4:4:4 support moves everyone to 4:2:0, and socket viewers or caps below 8 Mbps do the same. Two profiles select the encoder tuning: `balanced` (CRF 14, default chroma quantisation) and `text` (CRF 10, chroma QP offset −6); `?profile=` picks one and the most recent viewer's choice applies to the shared encoder. Quality adapts on a bitrate ladder down to 750 kbps that keeps resolution and frame rate and adds two CRF per rung under the default 20 Mbps cap; `?adapt=0` pins the top rung.
 
-Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It only bounds how far a severe descent (heavy loss or a starved decoder) steps, and only while it has stopped climbing below the sent rate. `scripts/probe-viewer.ts` drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; `scripts/probe-rtt-jitter.ts` feeds the controller a synthetic WebRTC viewer with jittery RTT probes.
+Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. Loss, RTT growth and jitter count only while the encoder has sent over 500 kbps for three seconds: on a quiet screen they belong to the network, and no rung fixes them. Climbing back to the rung below the one a descent left takes a few clean seconds; retrying that rung waits 15, 30 and then 60 seconds as episodes repeat, and an episode is forgotten after three minutes. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It and the browser's available-bitrate estimate only bound how far a severe descent (heavy loss or a starved decoder) steps, and only while the stream has been sending more than they report. `scripts/probe-viewer.ts` drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; `scripts/probe-rtt-jitter.ts` feeds the controller a synthetic WebRTC viewer with jittery RTT probes.
 
-The superfast preset uses integer-pixel motion search to reduce encoder CPU while retaining CABAC compression; ultrafast is unchanged. Encoding uses at most four threads and one queued capture frame, skips rapid catch-up captures after CPU stalls, and uses periodic intra refresh for WebRTC viewers; socket viewers switch it to 250 ms IDR intervals because they resynchronise only on IDR frames. When encoder settings change, a replacement discards two initial frames, emits a forced keyframe and then takes over. Every new encoder has four seconds to produce joinable media. Native startup or replacement failures switch subsequent launches to FFmpeg; the active encoder continues until its replacement is ready.
+The superfast preset uses integer-pixel motion search to reduce encoder CPU while retaining CABAC compression; ultrafast is unchanged. Encoding uses at most four threads and one queued capture frame and skips rapid catch-up captures after CPU stalls. Native capture sends no periodic keyframes or refresh wave: an IDR follows within a frame when a viewer joins, a receiver sends PLI, a send queue is cut, the viewer's watchdog sees no video, or a socket viewer whose frames were withheld has drained its backlog or reset its decoder. Requests within a round trip of the last keyframe share it. Browsers join only at an IDR, so the SPS that intra refresh puts before a recovery point is not treated as a keyframe. Rung and standby changes are applied in place; a change of H.264 profile, chroma or preset starts a replacement, which discards two initial frames, emits a forced keyframe and then takes over. The FFmpeg fallback cannot be told to make a keyframe, so it keeps periodic intra refresh for WebRTC viewers, switches to 250 ms IDR intervals for socket viewers, and makes requested keyframes by replacement. Every new encoder has four seconds to produce joinable media. Native startup or replacement failures switch subsequent launches to FFmpeg; the active encoder continues until its replacement is ready.
 
 The viewer exposes `?view=fit|native|pixel` (letterboxed, one source pixel per CSS pixel, one source pixel per device pixel) and `?diagnostics=1`, which shows transport and ICE route, negotiated codec profile, source/decoded/CSS/device dimensions, displayed and decoded frame rates, bitrate, RTT, loss, jitter buffer, the server's encoder settings, keyframe interval, CPU, pacer budget, queue, stale cuts, repairs and copies, the socket fallback's unsent bytes, withheld delta frames and decoder resets, and the latest server and viewer events. `globalThis.__replicasDesktop` carries the same state for tooling.
 

package/assets/desktop/start-services.sh

@@ -82,6 +82,16 @@ start_bg() {
   echo $! > "$pid_file"
 }
 
+# The stream, the X server it reads, the compositor that draws into it and the
+# audio server outrank the apps they show, so CPU load from those apps slows
+# the apps rather than stalling the picture of them. The encoders the stream
+# server spawns inherit its priority. Without sudo they run at normal priority.
+prioritize() {
+  local pid
+  pid="$(cat "$STATE_DIR/$1.pid" 2>/dev/null)"
+  [ -n "$pid" ] && sudo -n renice -n -10 -p "$pid" >/dev/null 2>&1 || true
+}
+
 count_lines() {
   awk 'NF { count++ } END { print count + 0 }'
 }
@@ -346,6 +356,7 @@ EOF
 start_bg xvfb \
   "pgrep -af 'Xvfb :$DISPLAY_NUM '" \
   "exec Xvfb :$DISPLAY_NUM -screen 0 ${WIDTH}x${HEIGHT}x${DEPTH} -ac +extension RANDR +extension GLX"
+prioritize xvfb
 
 for _ in $(seq 1 30); do
   if DISPLAY=":$DISPLAY_NUM" xdpyinfo >/dev/null 2>&1; then break; fi
@@ -363,6 +374,7 @@ start_bg xsettingsd \
 start_bg pulseaudio \
   "pactl info" \
   "exec pulseaudio --daemonize=no --exit-idle-time=-1 --log-target=stderr"
+prioritize pulseaudio
 
 # Paint twice because some themes reset root pixmap during startup.
 paint_wallpaper
@@ -388,6 +400,7 @@ if [ "$COMPOSITOR" = "1" ]; then
   start_bg picom \
     "false" \
     "DISPLAY=:$DISPLAY_NUM exec picom --config $HOME/.config/picom/picom.conf --no-fading-openclose"
+  prioritize picom
 fi
 
 if [ "$ENABLE_NOVNC_FALLBACK" = "1" ]; then
@@ -426,6 +439,7 @@ sudo sysctl -q -w net.core.rmem_max=8388608 net.core.wmem_max=8388608 >/dev/null
 start_bg stream \
   "has_stream" \
   "exec env REPLICAS_DESKTOP_DISPLAY=:$DISPLAY_NUM REPLICAS_DESKTOP_WIDTH=$WIDTH REPLICAS_DESKTOP_HEIGHT=$HEIGHT ${REPLICAS_DESKTOP_STREAM_FPS:+REPLICAS_DESKTOP_STREAM_FPS=$REPLICAS_DESKTOP_STREAM_FPS }REPLICAS_DESKTOP_STREAM_PORT=$STREAM_PORT REPLICAS_DESKTOP_STREAM_WEBROOT=${REPLICAS_DESKTOP_STREAM_WEBROOT:-$REPLICAS_DESKTOP_TEMPLATES/stream} replicas-computer stream"
+prioritize stream
 
 # Takes Chrome's cold-start cost at boot instead of on the user's first click;
 # later launches join this instance through the profile singleton.

package/dist/index.js

@@ -2937,7 +2937,7 @@ async function computerStatusCommand() {
 }
 
 // src/version.ts
-var COMPUTER_USE_VERSION = "0.3.14";
+var COMPUTER_USE_VERSION = "0.3.15";
 var COMPUTER_USE_PROTOCOL_VERSION = 1;
 
 // src/program.ts
@@ -3251,13 +3251,17 @@ import { createInterface } from "readline";
 
 // src/rtp.ts
 var RTP_HEADER_BYTES = 12;
+var NAL_SLICE = 1;
 var NAL_IDR = 5;
+var NAL_SEI = 6;
 var NAL_SPS = 7;
+var SEI_RECOVERY_POINT = 6;
 var NAL_STAP_A = 24;
 var NAL_FU_A = 28;
 var nalType = (byte) => byte & 31;
 var headerBytesOf = (packet) => RTP_HEADER_BYTES + (packet[0] & 15) * 4;
 var joinable = (nal) => nal === NAL_IDR || nal === NAL_SPS;
+var refreshFrame = (nal) => nalType(nal[0]) === NAL_SLICE || nalType(nal[0]) === NAL_SEI && nal[1] === SEI_RECOVERY_POINT;
 var isRtp = (data) => data.length >= RTP_HEADER_BYTES && data[0] >> 6 === 2 && (data[1] < 192 || data[1] > 223);
 var rtpPayload = (packet) => packet.subarray(headerBytesOf(packet));
 function aggregatedNals(payload) {
@@ -3286,7 +3290,10 @@ function isKeyframeStart(payload) {
     const fragment = fragmentHeader(payload);
     return fragment.start && joinable(fragment.type);
   }
-  if (type === NAL_STAP_A) return aggregatedNals(payload).nals.some((nal) => joinable(nalType(nal[0])));
+  if (type === NAL_STAP_A) {
+    const { nals } = aggregatedNals(payload);
+    return nals.some((nal) => joinable(nalType(nal[0]))) && !nals.some(refreshFrame);
+  }
   return joinable(type);
 }
 var RtpSplicer = class {
@@ -3363,6 +3370,8 @@ var RtpClock = class {
 };
 var startCode = new Uint8Array([0, 0, 0, 1]);
 var H264RtpAssembler = class {
+  /** Frames abandoned for a missing or out-of-order packet; every delta after one is undecodable until a keyframe. */
+  discarded = 0;
   clock = new RtpClock(9e4);
   chunks = [];
   key = false;
@@ -3380,6 +3389,7 @@ var H264RtpAssembler = class {
   }
   push(packet) {
     if (this.timestamp !== void 0 && packet.timestamp !== this.timestamp) {
+      if (this.chunks.length > 0 || this.damaged) this.discarded += 1;
       this.clearFrame();
     }
     this.timestamp ??= packet.timestamp;
@@ -3412,6 +3422,7 @@ var H264RtpAssembler = class {
     } else this.damaged = true;
     if (!packet.marker) return null;
     if (this.damaged || this.fragmentType !== void 0 || this.chunks.length === 0) {
+      this.discarded += 1;
       this.clearFrame();
       return null;
     }
@@ -3469,7 +3480,9 @@ function superviseEncoder(options) {
     const warmupFrames = active ? options.warmupFrames ?? 0 : 0;
     const command = options.command?.() ?? "ffmpeg";
     const args = options.args(socket.address().port, warmupFrames);
-    const child = spawn3(command, args, { env: options.env, stdio: ["ignore", options.onOutput ? "pipe" : "ignore", "pipe"] });
+    const child = spawn3(command, args, { env: options.env, stdio: [options.controllable?.(command) ? "pipe" : "ignore", options.onOutput ? "pipe" : "ignore", "pipe"] });
+    child.stdin?.on("error", () => {
+    });
     child.stderr?.on("data", (chunk) => process.stderr.write(`[desktop-stream] ${chunk}`));
     const generation = { id: ++nextId, child, socket, live: true, startedAt: Date.now(), warmupFrames };
     candidate = generation;
@@ -3539,10 +3552,18 @@ function superviseEncoder(options) {
     candidate = void 0;
     active = void 0;
   });
+  const control = (line) => {
+    for (const generation of [candidate, active]) {
+      if (generation?.live && generation.child.stdin?.writable) generation.child.stdin.write(`${line}
+`);
+    }
+    return Boolean(active?.live && active.child.stdin?.writable);
+  };
   return {
     start,
     replace,
     stop,
+    control,
     running: () => Boolean(active && alive(active.child)),
     replacing: () => Boolean(candidate),
     generation: () => active?.id ?? 0,
@@ -3625,6 +3646,8 @@ function validInput(value) {
   switch (value.type) {
     case "release":
     case "keyframe":
+    case "retry-webrtc":
+    case "webrtc-ready":
       return true;
     case "theme":
       return value.theme === "light" || value.theme === "dark";
@@ -4078,6 +4101,7 @@ function superviseVideoEncoder(options) {
   const encoder = superviseEncoder({
     label: "video",
     command: () => nativeCapture ? captureBinary : "ffmpeg",
+    controllable: (command) => command === captureBinary,
     onFailure: (command) => {
       if (command !== captureBinary) return;
       nativeCapture = false;
@@ -4234,6 +4258,8 @@ function superviseVideoEncoder(options) {
   });
   return {
     ...encoder,
+    /** Native capture takes keyframe, quality and frame-rate changes over stdin; ffmpeg needs a replacement for each. */
+    controllable: () => nativeCapture,
     capture: () => ({
       captureBackend: (encoder.command() ?? (nativeCapture ? captureBinary : "ffmpeg")) === captureBinary ? "native" : "ffmpeg",
       captureMetrics: captureMetrics?.generation === encoder.generation() && Date.now() - captureMetrics.at <= 2500 ? captureMetrics : void 0
@@ -4479,6 +4505,7 @@ var qualityLadder = (profile) => rungCaps.map((kbps, index) => ({ kbps, crf: pro
 var fullChromaMinKbps = 8e3;
 var fallbackKeyframeMs = 250;
 var webrtcStartupRung = 2;
+var descentMemoryMs = 18e4;
 var targetBacklogMs = 120;
 var congestedBacklogMs = 250;
 var congestedDecodeQueue = 3;
@@ -4558,7 +4585,8 @@ async function runMediaServer() {
   let activeProfile = "balanced";
   let adaptive = true;
   let rung = 0;
-  let descents = 0;
+  let descents = [];
+  let failedRung = -1;
   let congestedSeconds = 0;
   let comfortableSeconds = 0;
   let cleanTransportSeconds = 0;
@@ -4577,6 +4605,8 @@ async function runMediaServer() {
   let encoderFps = 0;
   let videoWanted = false;
   let encoderSignature = "";
+  let encoderDescription = "";
+  let encoderLive = { crf: 0, kbps: 0, fps: 0 };
   let encoderStop;
   let joinKeyframe;
   let lastVideoAt = 0;
@@ -4611,10 +4641,13 @@ async function runMediaServer() {
     const quality = ladder()[allStandby() ? Math.max(rung, standbyRung) : rung];
     const encodeFps = streamFps();
     const sockets = socketViewers().length > 0;
-    const keyframeMs = sockets ? Math.min(quality.keyframeMs, fallbackKeyframeMs) : quality.keyframeMs;
+    const onDemand = videoEncoder.controllable();
+    const keyframeMs = onDemand ? 0 : sockets ? Math.min(quality.keyframeMs, fallbackKeyframeMs) : quality.keyframeMs;
     const videoProfile = profile();
-    const intraRefresh = !sockets;
+    const intraRefresh = !onDemand && !sockets;
     const pixelFormat = videoProfile === "high444" ? "yuv444p" : "yuv420p";
+    const live = `rung ${rung} \xB7 crf ${quality.crf} \xB7 ${encodeFps} fps`;
+    const restart = `${activeProfile} \xB7 ${preset} \xB7 ${videoProfile} \xB7 keyframe ${keyframeMs ? `${keyframeMs}ms${intraRefresh ? " refresh" : ""}` : "on demand"}`;
     return {
       quality,
       preset,
@@ -4624,7 +4657,8 @@ async function runMediaServer() {
       pixelFormat,
       fps: encodeFps,
       chromaQpOffset: profiles[activeProfile].chromaQpOffset,
-      signature: `rung ${rung} \xB7 ${activeProfile} crf ${quality.crf} \xB7 ${preset} \xB7 ${videoProfile} \xB7 keyframe ${keyframeMs}ms${intraRefresh ? " refresh" : ""} \xB7 ${encodeFps} fps`
+      signature: onDemand ? restart : `${live} \xB7 ${restart}`,
+      description: `${live} \xB7 ${restart}`
     };
   };
   const budgetBps = () => ladder()[rung].kbps * 1e3 * budgetFactor;
@@ -4645,6 +4679,7 @@ async function runMediaServer() {
       if (!keyframe2 && (viewer.waitingForKeyframe || queueMs(viewer) > targetBacklogMs || socket.bufferedAmount > fallbackBufferedBytes)) {
         viewer.waitingForKeyframe = true;
         viewer.withheldFrames += 1;
+        if (videoEncoder.controllable() && queueMs(viewer) <= targetBacklogMs && socket.bufferedAmount <= fallbackBufferedBytes) requestKeyframe("socket viewer caught up", void 0, viewer);
         continue;
       }
       if (keyframe2) viewer.waitingForKeyframe = false;
@@ -4676,7 +4711,7 @@ async function runMediaServer() {
     let sockets = false;
     for (const viewer of viewers.values()) {
       if (viewer.fallback) sockets = true;
-      else viewer.rtc?.enqueueVideo(rtp ??= withPlayoutDelay(packet, PLAYOUT_DELAY_ID), frameId, keyframe, last);
+      viewer.rtc?.enqueueVideo(rtp ??= withPlayoutDelay(packet, PLAYOUT_DELAY_ID), frameId, keyframe, last);
     }
     frameStart = last;
     if (last) {
@@ -4685,7 +4720,12 @@ async function runMediaServer() {
     }
     if (!sockets) return;
     const parsed = parseRtpPacket(packet);
+    const discarded = videoAssembler.discarded;
     const frame = parsed && videoAssembler.push(parsed);
+    if (videoAssembler.discarded !== discarded) {
+      for (const viewer of socketViewers()) viewer.waitingForKeyframe = true;
+      if (videoEncoder.controllable()) requestKeyframe("socket frame lost");
+    }
     if (frame) broadcast(mediaPacket(frame.key ? 1 /* VideoKey */ : 2 /* VideoDelta */, frame.timestamp, frame.data), frame.key);
   };
   const onAudioPacket = (packet, generation) => {
@@ -4696,7 +4736,7 @@ async function runMediaServer() {
     let sockets = false;
     for (const viewer of viewers.values()) {
       if (viewer.fallback) sockets = true;
-      else viewer.rtc?.sendAudio(packet);
+      viewer.rtc?.sendAudio(packet);
     }
     if (!sockets) return;
     const parsed = parseRtpPacket(packet);
@@ -4770,20 +4810,30 @@ async function runMediaServer() {
   const ensureEncoders = () => {
     clearTimeout(encoderStop);
     void audioEncoder.start();
-    const { signature } = encoderParams();
+    const { signature, description, quality, fps: encodeFps } = encoderParams();
+    const live = { crf: quality.crf, kbps: quality.kbps, fps: encodeFps };
     if (!videoWanted) {
       videoWanted = true;
       encoderSignature = signature;
-      record("encoder", signature);
+      encoderDescription = description;
+      encoderLive = live;
+      record("encoder", description);
       void videoEncoder.start();
       return false;
     }
-    if (signature === encoderSignature) return false;
-    encoderSignature = signature;
-    lastSwapAt = Date.now();
-    record("encoder", signature);
-    void videoEncoder.replace();
-    return true;
+    if (description !== encoderDescription) record("encoder", description);
+    encoderDescription = description;
+    if (signature !== encoderSignature) {
+      encoderSignature = signature;
+      encoderLive = live;
+      lastSwapAt = Date.now();
+      void videoEncoder.replace();
+      return true;
+    }
+    if (live.crf !== encoderLive.crf || live.kbps !== encoderLive.kbps) videoEncoder.control(`quality ${live.crf} ${live.kbps}`);
+    if (live.fps !== encoderLive.fps) videoEncoder.control(`fps ${live.fps}`);
+    encoderLive = live;
+    return false;
   };
   const syncEncoder = () => videoWanted && viewers.size > 0 && ensureEncoders();
   const stopEncodersSoon = () => {
@@ -4801,8 +4851,9 @@ async function runMediaServer() {
     const quality = ladder()[next];
     record(next > rung ? "quality down" : "quality up", `rung ${rung} \u2192 ${next} (${quality.kbps} kbps cap, crf ${quality.crf}): ${reason}`);
     if (next > rung) {
-      if (now - lastDescentAt > 1e4) descents += 1;
+      if (now - lastDescentAt > 1e4) descents.push(now);
       lastDescentAt = now;
+      failedRung = rung;
       if (now - lastAscentAt < 15e3) {
         blockedRung = rung;
         blockedUntil = now + 12e4;
@@ -4818,7 +4869,8 @@ async function runMediaServer() {
   const resetLadder = (request) => {
     rung = request.transport === "webrtc" && request.adaptive ? webrtcStartupRung : 0;
     cleanTransportSeconds = 0;
-    descents = 0;
+    descents = [];
+    failedRung = -1;
     lastDescentAt = 0;
     lastAscentAt = 0;
     congestedSeconds = 0;
@@ -4838,6 +4890,7 @@ async function runMediaServer() {
     const sent = Math.min(...recentVideoKbps);
     return remb !== void 0 && earlier !== void 0 && remb <= earlier.kbps && sent > 500 && remb < sent * 0.9;
   };
+  const loaded = () => Math.min(...recentVideoKbps) > 500;
   const signals = (viewer, stats) => {
     const now = Date.now();
     if (viewer.fallback) {
@@ -4848,22 +4901,24 @@ async function runMediaServer() {
     }
     const stableSince = stableRttWindows.get(viewer)?.since;
     return [
-      ...viewer.rttFloorMs > viewer.rtcBaselineMs + 40 && (stableSince === void 0 || now - stableSince < 2e3) ? [`rtt floor ${Math.round(viewer.rttFloorMs)} ms over ${Math.round(viewer.rtcBaselineMs)} ms baseline`] : [],
+      ...viewer.rttFloorMs > viewer.rtcBaselineMs + 40 && (stableSince === void 0 || now - stableSince < 2e3) && loaded() ? [`rtt floor ${Math.round(viewer.rttFloorMs)} ms over ${Math.round(viewer.rtcBaselineMs)} ms baseline`] : [],
       ...stats.videoFps < expectedFps() * 0.6 && viewer.lowFpsReports >= 2 ? [`decoded ${stats.videoFps.toFixed(0)} fps`] : [],
-      ...(stats.lossPercent ?? 0) > 3 ? [`loss ${(stats.lossPercent ?? 0).toFixed(1)}%`] : [],
-      ...(stats.jitterBufferMs ?? 0) > 80 ? [`jitter buffer ${(stats.jitterBufferMs ?? 0).toFixed(0)} ms`] : []
+      ...(stats.lossPercent ?? 0) > 3 && loaded() ? [`loss ${(stats.lossPercent ?? 0).toFixed(1)}%`] : [],
+      ...(stats.jitterBufferMs ?? 0) > 80 && loaded() ? [`jitter buffer ${(stats.jitterBufferMs ?? 0).toFixed(0)} ms`] : []
     ];
   };
   const lowFpsObservationAt = /* @__PURE__ */ new WeakMap();
-  const severe = (viewer, stats) => !viewer.fallback && ((stats.lossPercent ?? 0) > 10 || stats.videoFps < expectedFps() * 0.25 && viewer.lowFpsReports >= 2);
+  const severe = (viewer, stats) => !viewer.fallback && ((stats.lossPercent ?? 0) > 10 && loaded() || stats.videoFps < expectedFps() * 0.25 && viewer.lowFpsReports >= 2);
   let lastSevereDescentAt = 0;
   const descendSevere = (now, trigger) => {
     comfortableSeconds = 0;
     congestedSeconds = 0;
     if (!adaptive || now - lastSevereDescentAt < 1e3) return;
     lastSevereDescentAt = now;
+    const sent = Math.min(...recentVideoKbps);
     const estimates = [...viewers.values()].flatMap((viewer) => {
-      const reported = viewer.stats && now - viewer.stats.at < 750 ? viewer.stats.availableKbps ?? 0 : 0;
+      const available = viewer.stats && now - viewer.stats.at < 750 ? viewer.stats.availableKbps ?? 0 : 0;
+      const reported = sent > 500 && available < sent * 0.9 ? available : 0;
       const remb = rembBelowSent(viewer, now) ? freshRemb(viewer, now) ?? 0 : 0;
       return [...reported > 0 ? [reported] : [], ...remb > 0 ? [remb] : []];
     });
@@ -4880,9 +4935,12 @@ async function runMediaServer() {
     videoKbps = Math.round((videoBytes - bytesAtTick) * 8 / 1e3);
     bytesAtTick = videoBytes;
     if (recentVideoKbps.push(videoKbps) > 3) recentVideoKbps.shift();
-    if (viewers.size === 0) return;
+    if (viewers.size === 0) {
+      starvedSeconds = 0;
+      return;
+    }
     const { captureBackend, captureMetrics } = videoEncoder.capture();
-    const captureBusy = captureBackend !== "native" || !captureMetrics || captureMetrics.busyMs / captureMetrics.elapsedMs > 0.8;
+    const captureBusy = captureBackend !== "native" || (captureMetrics ? captureMetrics.busyMs / captureMetrics.elapsedMs > 0.8 : videoEncoder.activeForMs() > 2500);
     if (videoEncoder.running() && encoderFps < streamFps() * 0.9 && captureBusy) {
       cleanTransportSeconds = 0;
       congestedSeconds = 0;
@@ -4931,15 +4989,16 @@ async function runMediaServer() {
       return;
     }
     congestedSeconds = Math.max(0, congestedSeconds - 0.5);
-    if (descents === 0 && reporting.some((viewer) => !viewer.fallback && (viewer.rttFloorMs > viewer.rtcBaselineMs + 20 || (viewer.stats.playbackDroppedPercent ?? 0) > 5 && cleanTransportSeconds < 15))) {
+    descents = descents.filter((at) => now - at < descentMemoryMs);
+    if (descents.length === 0 && reporting.some((viewer) => !viewer.fallback && (viewer.rttFloorMs > viewer.rtcBaselineMs + 20 && loaded() || (viewer.stats.playbackDroppedPercent ?? 0) > 5 && cleanTransportSeconds < 15))) {
       comfortableSeconds = 0;
       return;
     }
     comfortableSeconds += 1;
-    const startupSeconds = reporting.every((viewer) => !viewer.fallback) ? 2 : 3;
-    const recoverySeconds = lastAscentAt > lastDescentAt && cleanTransportSeconds >= 15 ? 15 : Math.min(60, 15 * 2 ** Math.min(descents, 2));
-    if (comfortableSeconds < (descents === 0 ? startupSeconds : recoverySeconds)) return;
     const target = Math.max(0, rung - 1);
+    const startupSeconds = reporting.every((viewer) => !viewer.fallback) ? 2 : 3;
+    const retrySeconds = lastAscentAt > lastDescentAt && cleanTransportSeconds >= 15 ? 15 : Math.min(60, 15 * 2 ** Math.min(descents.length - 1, 2));
+    if (comfortableSeconds < (descents.length > 0 && target <= failedRung ? retrySeconds : startupSeconds)) return;
     if (target === blockedRung && now < blockedUntil) return;
     setRung(target, `${comfortableSeconds}s comfortable, ${cleanTransportSeconds}s clean transport`);
   }, 1e3);
@@ -5019,9 +5078,18 @@ async function runMediaServer() {
 `);
   }, telemetryIntervalMs);
   telemetry.unref();
+  let lastKeyframeRequestAt = 0;
   const requestKeyframe = (reason, joinedAt, requestingViewer) => {
     const now = Date.now();
     if (!videoWanted || viewers.size === 0 || videoEncoder.replacing() || lastKeyframeAt < now - videoEncoder.activeForMs() || joinedAt !== void 0 && lastKeyframeAt > joinedAt) return;
+    if (videoEncoder.controllable()) {
+      const rttMs = requestingViewer?.rtc?.rttMs ?? requestingViewer?.stats?.rttMs ?? 0;
+      if (joinedAt === void 0 && now - Math.max(lastKeyframeAt, lastKeyframeRequestAt) < Math.max(300, rttMs + 150)) return;
+      if (!videoEncoder.control("keyframe")) return;
+      lastKeyframeRequestAt = now;
+      record("keyframe", reason);
+      return;
+    }
     const startup = requestingViewer && !requestingViewer.hasDecodedFrame && now - requestingViewer.connectedAt < 1e4;
     const wait = (joinedAt === void 0 ? startup ? 500 : 2e3 : 350) - (now - lastSwapAt);
     if (wait > 0) {
@@ -5031,7 +5099,9 @@ async function runMediaServer() {
       }
       return;
     }
-    if (!startup && encoderParams().keyframeMs - (now - lastKeyframeAt) <= 350) return;
+    const { keyframeMs, intraRefresh } = encoderParams();
+    const untilPeriodic = keyframeMs - (now - lastKeyframeAt);
+    if (!startup && !intraRefresh && untilPeriodic >= 0 && untilPeriodic <= 350) return;
     lastSwapAt = now;
     record("keyframe", reason);
     void videoEncoder.replace();
@@ -5044,16 +5114,51 @@ async function runMediaServer() {
       process.stderr.write(`[desktop-stream] webrtc fallback diagnostics: ${JSON.stringify(entry)}
 `);
     }
-    if (viewer.fallback) return;
+    const retried = viewer.rtc;
+    viewer.rtc?.close();
+    viewer.rtc = void 0;
+    if (viewer.fallback) {
+      if (retried) record("webrtc retry failed", reason);
+      return;
+    }
     viewer.fallback = true;
     viewer.waitingForKeyframe = true;
     viewer.input.release();
-    viewer.rtc?.close();
-    viewer.rtc = void 0;
     record("fallback to socket", reason);
     if (!syncEncoder()) requestKeyframe("socket fallback");
   };
   const optional = (value, max) => value === void 0 ? void 0 : clamp(value, 0, max);
+  const openWebRtc = (viewer, reply, iceServers) => {
+    try {
+      viewer.rtc = new WebRtcViewer({ iceServers, iceTransportPolicy: "all", portRange: icePorts, budgetBps: budgetBps() }, {
+        offer: (sdp) => reply({ type: "offer", sdp }),
+        candidate: (candidate, mid) => reply({ type: "candidate", candidate, mid }),
+        message: (message, answer) => handleMessage(viewer, message, answer),
+        keyframe: (reason) => {
+          if (reason === "stale" && Date.now() - lastDescentAt > 1e3) descendSevere(Date.now(), "send queue went stale and was cut back to a keyframe");
+          requestKeyframe(reason === "pli" ? "receiver pli" : "stale send queue", void 0, viewer);
+        },
+        remb: (bitsPerSecond) => {
+          if (viewer.rembs.push({ kbps: bitsPerSecond / 1e3, at: Date.now() }) > 16) viewer.rembs.shift();
+        },
+        connected: () => {
+          record("webrtc connected", viewer.rtc?.path ?? void 0);
+          requestKeyframe("viewer joined", Date.now());
+        },
+        // A viewer that loses its peer cannot send the key-ups for whatever it
+        // was holding, and its socket may stay open for a long time yet. A socket
+        // viewer trying WebRTC again still sends its input over the socket.
+        disconnected: (state) => {
+          record(`webrtc ${state}`);
+          if (!viewer.fallback) viewer.input.release();
+          if (state === "failed") reply({ type: "webrtc-failed" });
+        }
+      });
+    } catch (error) {
+      record("webrtc error", `could not create a peer connection: ${error instanceof Error ? error.message : String(error)}`);
+      reply({ type: "webrtc-failed" });
+    }
+  };
   const handleMessage = (viewer, message, reply) => {
     switch (message.type) {
       case "ping":
@@ -5131,7 +5236,7 @@ async function runMediaServer() {
         }
         if (viewer.stats.videoFps > 0) viewer.hasDecodedFrame = true;
         if (viewer.stats.videoFps >= expectedFps() * 0.8) viewer.everHealthy = true;
-        if (encoderHealthyTicks >= 3 && videoEncoder.running() && [...viewers.values()].every((v) => !v.fallback && v.stats && now - v.stats.at < 750 && now - v.connectedAt > 2500 && (v.stats.videoPacketsPerSecond ?? 0) > 0 && (v.stats.lossPercent ?? 0) > 10)) descendSevere(now, "every viewer reports over 10% loss");
+        if (encoderHealthyTicks >= 3 && videoEncoder.running() && loaded() && [...viewers.values()].every((v) => !v.fallback && v.stats && now - v.stats.at < 750 && now - v.connectedAt > 2500 && (v.stats.videoPacketsPerSecond ?? 0) > 0 && (v.stats.lossPercent ?? 0) > 10)) descendSevere(now, "every viewer reports over 10% loss");
         return;
       }
       case "answer":
@@ -5144,8 +5249,25 @@ async function runMediaServer() {
       case "fallback":
         enterFallback(viewer, message.reason, message.failure);
         return;
+      // A socket viewer whose WebRTC attempt failed tries again in the background;
+      // the socket keeps carrying its media until the new peer shows video.
+      case "retry-webrtc": {
+        const ice = webrtc && viewer.fallback && !viewer.rtc && viewer.request.transport === "webrtc" ? currentIceServers() : void 0;
+        if (!ice) return;
+        record("retrying webrtc", "socket media continues meanwhile");
+        openWebRtc(viewer, reply, ice.native);
+        return;
+      }
+      case "webrtc-ready":
+        if (!viewer.fallback || !viewer.rtc?.connected) return;
+        viewer.fallback = false;
+        viewer.withheldFrames = 0;
+        record("back on webrtc", viewer.rtc.path ?? void 0);
+        syncEncoder();
+        return;
       case "keyframe":
-        if (!viewer.fallback) requestKeyframe("viewer watchdog", void 0, viewer);
+        if (viewer.fallback) viewer.waitingForKeyframe = true;
+        if (!viewer.fallback || videoEncoder.controllable()) requestKeyframe(viewer.fallback ? "socket viewer decoder reset" : "viewer watchdog", void 0, viewer);
         return;
       case "theme":
         applyDesktopTheme(message.theme);
@@ -5294,36 +5416,7 @@ async function runMediaServer() {
     const fallbackProfile = profile() === "high444" ? "main" : profile();
     const configMessage = { ...config, videoCodec: H264_PROFILES[fallbackProfile].codec, iceServers: ice?.browser, request };
     reply(configMessage);
-    if (ice) {
-      try {
-        viewer.rtc = new WebRtcViewer({ iceServers: ice.native, iceTransportPolicy: "all", portRange: icePorts, budgetBps: budgetBps() }, {
-          offer: (sdp) => reply({ type: "offer", sdp }),
-          candidate: (candidate, mid) => reply({ type: "candidate", candidate, mid }),
-          message: (message, answer) => handleMessage(viewer, message, answer),
-          keyframe: (reason) => {
-            if (reason === "stale") descendSevere(Date.now(), "send queue went stale and was cut back to a keyframe");
-            requestKeyframe(reason === "pli" ? "receiver pli" : "stale send queue", void 0, viewer);
-          },
-          remb: (bitsPerSecond) => {
-            if (viewer.rembs.push({ kbps: bitsPerSecond / 1e3, at: Date.now() }) > 16) viewer.rembs.shift();
-          },
-          connected: () => {
-            record("webrtc connected", viewer.rtc?.path ?? void 0);
-            requestKeyframe("viewer joined", Date.now());
-          },
-          // A viewer that loses its peer cannot send the key-ups for whatever it
-          // was holding, and its socket may stay open for a long time yet.
-          disconnected: (state) => {
-            record(`webrtc ${state}`);
-            viewer.input.release();
-            if (state === "failed") reply({ type: "webrtc-failed" });
-          }
-        });
-      } catch (error) {
-        record("webrtc error", `could not create a peer connection: ${error instanceof Error ? error.message : String(error)}`);
-        reply({ type: "webrtc-failed" });
-      }
-    }
+    if (ice) openWebRtc(viewer, reply, ice.native);
     if (!viewer.rtc) enterFallback(viewer, request.transport === "webrtc" ? "webrtc unavailable" : "websocket requested");
     ensureEncoders();
     socket.on("message", (raw, binary) => {

package/dist/stream-viewer.js

@@ -99,6 +99,10 @@ var view = params.get("view") === "native" ? "native" : params.get("view") === "
 document.documentElement.dataset.view = view;
 diagnostics.classList.toggle("visible", showDiagnostics);
 var webrtcFrameTimeoutMs = 6e3;
+var webrtcFirstFrameMs = 3e3;
+var webrtcRetryFirstMs = 15e3;
+var webrtcRetryMaxMs = 3e5;
+var webrtcRetryTimeoutMs = 1e4;
 var webrtcSkipsAfterFailure = 3;
 var pointerIntervalMs = 4;
 var qualityWindowMs = 3e4;
@@ -119,6 +123,7 @@ var peer;
 var inputChannel;
 var pointerChannel;
 var webrtcDeadline;
+var webrtcDeadlineAt = 0;
 var webrtcSkips = 0;
 var videoTrack;
 var videoWriter;
@@ -174,6 +179,7 @@ state.events = viewerEvents;
 var decoderResets = 0;
 var connectionStartedAt = performance.now();
 var failure;
+var recovery;
 var localCandidateTypes = /* @__PURE__ */ new Set();
 var remoteCandidateTypes = /* @__PURE__ */ new Set();
 var iceErrorCodes = /* @__PURE__ */ new Set();
@@ -250,9 +256,10 @@ var sendInput = (message, unreliable = false) => {
 };
 var announceTransport = () => {
   if (window.parent === window) return;
-  window.parent.postMessage({ type: DESKTOP_TRANSPORT_MESSAGE, transport: transport ?? "connecting", reason: transportReason, path: route.path, codec: route.codec, failure }, "*");
+  window.parent.postMessage({ type: DESKTOP_TRANSPORT_MESSAGE, transport: transport ?? "connecting", reason: transportReason, path: route.path, codec: route.codec, failure, recovery }, "*");
 };
 var closePeer = (preserveDeadline = false) => {
+  cancelRetry();
   if (!preserveDeadline) {
     clearTimeout(webrtcDeadline);
     webrtcDeadline = void 0;
@@ -394,7 +401,14 @@ var handleBinary = (data) => {
     waitingForKeyframe = true;
     state.decoderResets = ++decoderResets;
   }
-  if (waitingForKeyframe && !key) return;
+  if (waitingForKeyframe && !key) {
+    const now = performance.now();
+    if (now - lastKeyframeRequestAt >= 1e3) {
+      lastKeyframeRequestAt = now;
+      sendSocket({ type: "keyframe" });
+    }
+    return;
+  }
   waitingForKeyframe = false;
   try {
     videoDecoder.decode(new EncodedVideoChunk({ type: key ? "key" : "delta", timestamp, data: payload }));
@@ -465,6 +479,9 @@ var useFallback = (reasonCode, failed = false) => {
   route = { ...route, path: "proxy", codec: config.videoCodec };
   announceTransport();
   sendSocket({ type: "fallback", reason, failure });
+  socketSince = performance.now();
+  retryAttempts = 0;
+  if (reasonCode !== "websocket_selected" && reasonCode !== "server_unavailable" && reasonCode !== "browser_unavailable" && reasonCode !== "codec_unsupported") scheduleRetry();
   if (!TrackGenerator) {
     showStatus("This browser cannot render the desktop stream");
     return;
@@ -477,35 +494,15 @@ var useFallback = (reasonCode, failed = false) => {
 var negotiate = async (sdp) => {
   if (!config?.iceServers || transport === "websocket") return;
   closePeer(true);
-  const pc = new RTCPeerConnection({ iceServers: config.iceServers, bundlePolicy: "max-bundle", iceTransportPolicy: config.request.iceTransportPolicy });
-  peer = pc;
-  pc.ontrack = (event) => {
-    if (peer !== pc) return;
-    try {
-      event.receiver.jitterBufferTarget = 0;
-    } catch {
-    }
-    try {
-      event.receiver.playoutDelayHint = 0;
-    } catch {
-    }
-    if (screen.srcObject !== event.streams[0]) screen.srcObject = event.streams[0] ?? new MediaStream([event.track]);
+  const pc = openPeer(config, () => peer === pc, (stream) => {
+    if (screen.srcObject !== stream) screen.srcObject = stream;
     void screen.play().catch(() => {
     });
-  };
-  pc.ondatachannel = (event) => {
-    if (peer !== pc) return;
-    if (event.channel.label === "pointer") pointerChannel = event.channel;
-    else inputChannel = event.channel;
-    event.channel.onmessage = (message) => {
-      if (typeof message.data === "string") handleServerText(message.data, false);
-    };
-  };
-  pc.onicecandidate = (event) => {
-    if (peer !== pc) return;
-    if (event.candidate?.type) localCandidateTypes.add(event.candidate.type);
-    if (event.candidate?.candidate) sendSocket({ type: "candidate", candidate: event.candidate.candidate, mid: event.candidate.sdpMid ?? "0" });
-  };
+  }, (channel) => {
+    if (channel.label === "pointer") pointerChannel = channel;
+    else inputChannel = channel;
+  }, (type) => localCandidateTypes.add(type));
+  peer = pc;
   pc.onicecandidateerror = (event) => {
     if (peer !== pc) return;
     iceErrorCodes.add(event.errorCode);
@@ -522,6 +519,7 @@ var negotiate = async (sdp) => {
       transportReason = "";
       webrtcSkips = 0;
       lastFrameAt = performance.now();
+      if (webrtcDeadline && webrtcDeadlineAt - lastFrameAt < webrtcFirstFrameMs) armWebrtcDeadline(webrtcFirstFrameMs);
       announceTransport();
       unlockAudio();
     } else if (pc.connectionState === "failed") {
@@ -530,26 +528,166 @@ var negotiate = async (sdp) => {
     }
   };
   try {
-    await pc.setRemoteDescription({ type: "offer", sdp });
-    if (peer !== pc) return;
-    if (skipFullChroma) {
-      const codecs = RTCRtpReceiver.getCapabilities("video")?.codecs.filter((codec) => h264ProfileLabel(codec.sdpFmtpLine ?? "") !== H264_PROFILES.high444.label);
-      if (codecs) pc.getTransceivers().find((transceiver) => transceiver.receiver.track.kind === "video")?.setCodecPreferences(codecs);
-    }
-    const answer = await pc.createAnswer();
-    if (peer !== pc) return;
-    await pc.setLocalDescription(answer);
-    if (peer !== pc || !answer.sdp) return;
-    const video = answer.sdp.split(/^m=/m).find((section) => section.startsWith("video")) ?? "";
-    if (video.startsWith("video 0 ") || !/^a=rtpmap:\d+ H264\/90000/m.test(video)) {
-      useFallback("codec_unsupported", true);
-      return;
-    }
-    sendSocket({ type: "answer", sdp: answer.sdp });
+    if (!await answerOffer(pc, sdp, skipFullChroma, () => peer === pc)) useFallback("codec_unsupported", true);
   } catch (error) {
     if (peer === pc) throw error;
   }
 };
+var renderOnArrival = (receiver) => {
+  try {
+    receiver.jitterBufferTarget = 0;
+  } catch {
+  }
+  try {
+    receiver.playoutDelayHint = 0;
+  } catch {
+  }
+};
+var openPeer = ({ iceServers, request: { iceTransportPolicy } }, current, play, attach, candidateType) => {
+  const pc = new RTCPeerConnection({ iceServers, bundlePolicy: "max-bundle", iceTransportPolicy });
+  pc.ontrack = (event) => {
+    if (!current()) return;
+    renderOnArrival(event.receiver);
+    play(event.streams[0] ?? new MediaStream([event.track]));
+  };
+  pc.ondatachannel = (event) => {
+    if (!current()) return;
+    attach(event.channel);
+    event.channel.onmessage = (message) => {
+      if (typeof message.data === "string") handleServerText(message.data, false);
+    };
+  };
+  pc.onicecandidate = (event) => {
+    if (!current()) return;
+    if (event.candidate?.type) candidateType?.(event.candidate.type);
+    if (event.candidate?.candidate) sendSocket({ type: "candidate", candidate: event.candidate.candidate, mid: event.candidate.sdpMid ?? "0" });
+  };
+  return pc;
+};
+var answerOffer = async (pc, sdp, withoutFullChroma, current) => {
+  await pc.setRemoteDescription({ type: "offer", sdp });
+  if (!current()) return true;
+  if (withoutFullChroma) {
+    const codecs = RTCRtpReceiver.getCapabilities("video")?.codecs.filter((codec) => h264ProfileLabel(codec.sdpFmtpLine ?? "") !== H264_PROFILES.high444.label);
+    if (codecs) pc.getTransceivers().find((transceiver) => transceiver.receiver.track.kind === "video")?.setCodecPreferences(codecs);
+  }
+  const answer = await pc.createAnswer();
+  if (!current()) return true;
+  await pc.setLocalDescription(answer);
+  if (!current() || !answer.sdp) return true;
+  const video = answer.sdp.split(/^m=/m).find((section) => section.startsWith("video")) ?? "";
+  if (video.startsWith("video 0 ") || !/^a=rtpmap:\d+ H264\/90000/m.test(video)) return false;
+  sendSocket({ type: "answer", sdp: answer.sdp });
+  return true;
+};
+var retry;
+var retryRequested = false;
+var retryTimer;
+var retryDeadline;
+var retryDelayMs = webrtcRetryFirstMs;
+var retryAttempts = 0;
+var socketSince = 0;
+var retryScreen = document.createElement("video");
+retryScreen.muted = true;
+retryScreen.playsInline = true;
+retryScreen.setAttribute("aria-hidden", "true");
+retryScreen.style.cssText = "position:fixed;left:0;top:0;width:2px;height:2px;opacity:0;pointer-events:none";
+document.body.append(retryScreen);
+var cancelRetry = () => {
+  clearTimeout(retryTimer);
+  clearTimeout(retryDeadline);
+  retryTimer = void 0;
+  retryRequested = false;
+  retry?.pc.close();
+  retry = void 0;
+  retryScreen.srcObject = null;
+};
+var scheduleRetry = () => {
+  if (retryTimer || retryRequested || request.transport !== "webrtc" || !config?.iceServers || typeof RTCPeerConnection !== "function") return;
+  retryTimer = setTimeout(() => {
+    retryTimer = void 0;
+    if (transport !== "websocket" || socket?.readyState !== WebSocket.OPEN) return;
+    retryRequested = true;
+    retryAttempts += 1;
+    note("retrying WebRTC", `attempt ${retryAttempts}`);
+    sendSocket({ type: "retry-webrtc" });
+    retryDeadline = setTimeout(() => failRetry("no video in time"), webrtcRetryTimeoutMs);
+  }, retryDelayMs);
+  retryDelayMs = Math.min(webrtcRetryMaxMs, retryDelayMs * 2);
+};
+var failRetry = (reason) => {
+  if (!retryRequested) return;
+  cancelRetry();
+  note("WebRTC retry failed", reason);
+  sendSocket({ type: "fallback", reason: `WebRTC retry: ${reason}` });
+  scheduleRetry();
+};
+var answerRetry = async (sdp) => {
+  if (!retryRequested || retry || !config?.iceServers) return;
+  const pc = openPeer(config, () => retry === attempt, (stream) => {
+    attempt.stream = stream;
+    retryScreen.srcObject = stream;
+    void retryScreen.play().catch(() => {
+    });
+  }, (channel) => {
+    if (channel.label === "pointer") attempt.pointer = channel;
+    else attempt.input = channel;
+  });
+  const attempt = { pc };
+  retry = attempt;
+  pc.onconnectionstatechange = () => {
+    if (peer === pc) {
+      note("webrtc", pc.connectionState);
+      if (pc.connectionState === "failed") socket?.close();
+    } else if (retry === attempt && pc.connectionState === "failed") failRetry("connection failed");
+  };
+  try {
+    if (!await answerOffer(pc, sdp, true, () => retry === attempt)) failRetry("codec unsupported");
+  } catch (error) {
+    if (retry === attempt) failRetry(describe(error));
+  }
+};
+var checkRetry = async (attempt) => {
+  const video = [...(await attempt.pc.getStats()).values()].find(inboundOf("video"));
+  if (retry === attempt && (video?.framesDecoded ?? 0) > 0) completeRetry(attempt);
+};
+var completeRetry = (attempt) => {
+  clearTimeout(retryDeadline);
+  retry = void 0;
+  retryRequested = false;
+  videoConfiguration += 1;
+  if (videoDecoder?.state !== "closed") videoDecoder?.close();
+  videoDecoder = void 0;
+  activeVideoConfig = void 0;
+  if (audioDecoder?.state !== "closed") audioDecoder?.close();
+  audioDecoder = void 0;
+  void videoWriter?.close().catch(() => {
+  });
+  videoTrack?.stop();
+  videoTrack = void 0;
+  videoWriter = void 0;
+  peer = attempt.pc;
+  inputChannel = attempt.input;
+  pointerChannel = attempt.pointer;
+  inbound = void 0;
+  inboundHistory = [];
+  screen.srcObject = attempt.stream ?? null;
+  retryScreen.srcObject = null;
+  recovery = { id: crypto.randomUUID?.() ?? `${Date.now()}`, afterMs: Math.round(performance.now() - socketSince), attempts: retryAttempts };
+  note("back on WebRTC", `${Math.round(recovery.afterMs / 1e3)} s and ${retryAttempts} attempts on the socket`);
+  transport = "webrtc";
+  transportReason = "";
+  failure = void 0;
+  webrtcSkips = 0;
+  retryDelayMs = webrtcRetryFirstMs;
+  retryAttempts = 0;
+  lastFrameAt = performance.now();
+  restartQuality();
+  route = { ...route, path: "", codec: "" };
+  sendSocket({ type: "webrtc-ready" });
+  announceTransport();
+  unlockAudio();
+};
 var handleServerMessage = (message, viaSocket) => {
   if (message.type === "pong") {
     const elapsed = performance.now() - message.sentAt;
@@ -570,10 +708,15 @@ var handleServerMessage = (message, viaSocket) => {
     return;
   }
   if (message.type === "webrtc-failed") {
-    useFallback("server_connection_failed", true);
+    if (retryRequested) failRetry("server connection failed");
+    else useFallback("server_connection_failed", true);
     return;
   }
   if (message.type === "offer") {
+    if (retryRequested) {
+      void answerRetry(message.sdp);
+      return;
+    }
     void negotiate(message.sdp).catch((error) => {
       negotiationErrorName = error instanceof Error ? error.name : "unknown";
       useFallback("negotiation_failed", true);
@@ -583,7 +726,7 @@ var handleServerMessage = (message, viaSocket) => {
   if (message.type === "candidate") {
     const candidateType = message.candidate.match(/\btyp (host|srflx|prflx|relay)\b/)?.[1];
     if (candidateType) remoteCandidateTypes.add(candidateType);
-    const pc = peer;
+    const pc = retry?.pc ?? peer;
     void pc?.addIceCandidate({ candidate: message.candidate, sdpMid: message.mid }).catch(() => {
       if (peer === pc) candidateErrors += 1;
     });
@@ -605,18 +748,24 @@ var handleServerMessage = (message, viaSocket) => {
     useFallback("recent_failure");
   } else if (!transport) {
     showStatus("Establishing connection\u2026", true);
-    webrtcDeadline ??= setTimeout(() => {
-      if (skipFullChroma || !inbound?.packetsReceived || !route.codec.endsWith(H264_PROFILES.high444.label)) {
-        useFallback("video_timeout", true);
-        return;
-      }
-      skipFullChroma = true;
-      if (config) fullChromaFailure = captureFailure("full_chroma_undecodable", config);
-      note("retrying WebRTC without 4:4:4");
-      socket?.close();
-    }, webrtcFrameTimeoutMs);
+    armWebrtcDeadline(webrtcFrameTimeoutMs);
   }
 };
+var armWebrtcDeadline = (ms) => {
+  clearTimeout(webrtcDeadline);
+  webrtcDeadlineAt = performance.now() + ms;
+  webrtcDeadline = setTimeout(() => {
+    webrtcDeadline = void 0;
+    if (skipFullChroma || !inbound?.packetsReceived || !route.codec.endsWith(H264_PROFILES.high444.label)) {
+      useFallback("video_timeout", true);
+      return;
+    }
+    skipFullChroma = true;
+    if (config) fullChromaFailure = captureFailure("full_chroma_undecodable", config);
+    note("retrying WebRTC without 4:4:4");
+    connect();
+  }, ms);
+};
 var handleServerText = (data, viaSocket) => {
   let parsed;
   try {
@@ -633,6 +782,7 @@ var connect = () => {
   transport = void 0;
   transportReason = "";
   failure = void 0;
+  recovery = void 0;
   config = void 0;
   telemetry = void 0;
   connectionStartedAt = performance.now();
@@ -807,7 +957,7 @@ var updateDiagnostics = (stats) => {
     `source ${dims.source} \xB7 decoded ${dims.decoded || "\u2014"} \xB7 css ${dims.css} \xB7 device ${dims.physical}`,
     `${route.displayedFps.toFixed(1)} displayed/s \xB7 ${stats.videoFps.toFixed(1)} decoded/s \xB7 ${route.kbps.toFixed(0)} kbps \xB7 ${stats.rttMs.toFixed(0)} ms rtt \xB7 ${socketRtt.toFixed(0)} ms via proxy \xB7 ${network}`,
     transport === "webrtc" ? `receiver: ${stats.decodeMs?.toFixed(2) ?? "\u2014"} ms decode \xB7 ${stats.receiverProcessingMs?.toFixed(2) ?? "\u2014"} ms receive \u2192 decoded (includes buffer + decode; excludes display)` : "",
-    encoder ? `encoder ${encoder.profile} crf ${encoder.crf} \xB7 ${encoder.codecProfile} ${encoder.pixelFormat} ${encoder.color} \xB7 rung ${encoder.rung}/${encoder.rungs - 1} cap ${encoder.kbpsCap} kbps${encoder.adaptive ? "" : " pinned"} \xB7 keyframe every ${encoder.keyframeMs} ms \xB7 ${encoder.fps} fps ${encoder.kbps} kbps \xB7 ${encoder.preset}` : "",
+    encoder ? `encoder ${encoder.profile} crf ${encoder.crf} \xB7 ${encoder.codecProfile} ${encoder.pixelFormat} ${encoder.color} \xB7 rung ${encoder.rung}/${encoder.rungs - 1} cap ${encoder.kbpsCap} kbps${encoder.adaptive ? "" : " pinned"} \xB7 keyframe ${encoder.keyframeMs ? `every ${encoder.keyframeMs} ms` : "on demand"} \xB7 ${encoder.fps} fps ${encoder.kbps} kbps \xB7 ${encoder.preset}` : "",
     telemetry && server ? `server cpu: encoder ${telemetry.cpu.encoderPercent}% \xB7 stream ${telemetry.cpu.serverPercent}% \xB7 load ${telemetry.cpu.load} on ${telemetry.cpu.cores} cores \xB7 budget ${server.budgetKbps} kbps sent ${server.sentKbps} \xB7 queue ${server.queueMs} ms ${server.queuedPackets} pkt \xB7 stale ${server.staleEvents} dropped ${server.droppedFrames} \xB7 repairs ${server.repairsSent}/${server.repairsSkipped} skipped \xB7 copies ${server.duplicatesSent}${server.protecting ? " on" : ""} \xB7 remb ${server.rembKbps ?? "\u2014"}` : "",
     transport === "websocket" && server ? `socket: ${Math.round(server.bufferedBytes / 1024)} KiB unsent \xB7 ${server.withheldFrames} deltas withheld${server.waitingForKeyframe ? " \xB7 awaiting keyframe" : ""} \xB7 ${decoderResets} decoder resets` : "",
     telemetry?.events.length ? `server events: ${describeEvents(telemetry.events)}` : "",
@@ -995,6 +1145,8 @@ setInterval(() => {
   renderedFrames = 0;
   decodedAudioPackets = 0;
   renderedWindowStartedAt = now;
+  if (retry) void checkRetry(retry).catch(() => {
+  });
   if (peer) void reportWebRtcStats(peer, stats).catch(() => {
   });
   else if (transport === "websocket") {

package/native/internal/capture/native.go

@@ -152,15 +152,19 @@ func (d *desktop) changed() (bool, error) {
 	return changed, nil
 }
 
-func (d *desktop) wait(micros int64) error {
+func (d *desktop) wait(micros int64, control int) error {
 	C.XFlush(d.display)
-	fd := C.struct_pollfd{fd: C.XConnectionNumber(d.display), events: C.POLLIN}
+	fds := [2]C.struct_pollfd{{fd: C.XConnectionNumber(d.display), events: C.POLLIN}, {fd: C.int(control), events: C.POLLIN}}
+	count := C.nfds_t(1)
+	if control >= 0 {
+		count = 2
+	}
 	timeout := C.struct_timespec{tv_sec: C.time_t(micros / 1000000), tv_nsec: C.long(micros % 1000000 * 1000)}
-	ready, err := C.ppoll(&fd, 1, &timeout, nil)
+	ready, err := C.ppoll(&fds[0], count, &timeout, nil)
 	if ready < 0 && err != syscall.EINTR {
 		return fmt.Errorf("display poll failed: %w", err)
 	}
-	if fd.revents&(C.POLLERR|C.POLLHUP|C.POLLNVAL) != 0 {
+	if fds[0].revents&(C.POLLERR|C.POLLHUP|C.POLLNVAL) != 0 {
 		return errors.New("display disconnected")
 	}
 	return nil
@@ -217,6 +221,7 @@ type encoder struct {
 	packet        *C.AVPacket
 	scale         *C.struct_SwsContext
 	matrix        colorMatrix
+	vbvMS         int
 	headerWritten bool
 }
 
@@ -251,10 +256,19 @@ func newEncoder(o Options) (e *encoder, err error) {
 	}
 	c.time_base = C.AVRational{num: 1, den: 90000}
 	c.framerate = C.AVRational{num: C.int(o.FPS), den: 1}
-	c.gop_size = C.int(max(1, (o.FPS*o.KeyframeMS+500)/1000))
+	keyint := "infinite"
+	c.gop_size = C.int(1 << 30)
+	if o.KeyframeMS > 0 {
+		c.gop_size = C.int(max(1, (o.FPS*o.KeyframeMS+500)/1000))
+		keyint = fmt.Sprintf("%d:min-keyint=%d", c.gop_size, c.gop_size)
+	}
 	c.max_b_frames, c.thread_count = 0, C.int(o.Threads)
+	e.vbvMS = 100
+	if o.KeyframeMS > 0 {
+		e.vbvMS = min(o.KeyframeMS, 100)
+	}
 	c.rc_max_rate = C.int64_t(o.Kbps) * 1000
-	c.rc_buffer_size = C.int(o.Kbps * min(o.KeyframeMS, 100))
+	c.rc_buffer_size = C.int(o.Kbps * e.vbvMS)
 	c.flags |= C.AV_CODEC_FLAG_LOW_DELAY
 	if o.BT709 != 0 {
 		c.color_primaries = C.AVCOL_PRI_BT709
@@ -264,7 +278,7 @@ func newEncoder(o Options) (e *encoder, err error) {
 	}
 	dict := dictionary{}
 	defer dict.close()
-	parameters := fmt.Sprintf("fps=%d/1:repeat-headers=1:sliced-threads=1:slices=4:keyint=%d:min-keyint=%d:scenecut=0:chroma-qp-offset=%d:intra-refresh=%d", o.FPS, c.gop_size, c.gop_size, o.ChromaQP, o.IntraRefresh)
+	parameters := fmt.Sprintf("fps=%d/1:repeat-headers=1:sliced-threads=1:slices=4:keyint=%s:scenecut=0:chroma-qp-offset=%d:intra-refresh=%d", o.FPS, keyint, o.ChromaQP, o.IntraRefresh)
 	if o.Preset == "superfast" {
 		parameters += ":subme=0"
 	}
@@ -366,6 +380,20 @@ func (e *encoder) close() {
 	e.matrix.close()
 }
 
+// reconfigure changes the rate control between frames; libavcodec's x264
+// wrapper applies a changed CRF or VBV with x264_encoder_reconfig before the
+// next frame, so the stream continues without a keyframe.
+func (e *encoder) reconfigure(crf, kbps int) error {
+	name := C.CString("crf")
+	defer C.free(unsafe.Pointer(name))
+	if err := avCheck(C.av_opt_set_double(e.codec.priv_data, name, C.double(crf), 0)); err != nil {
+		return err
+	}
+	e.codec.rc_max_rate = C.int64_t(kbps) * 1000
+	e.codec.rc_buffer_size = C.int(kbps * e.vbvMS)
+	return nil
+}
+
 func (e *encoder) convert(d *desktop, x, y int) error {
 	im := d.image
 	if e.scale != nil {
@@ -442,10 +470,11 @@ func (s *Stream) Close() {
 	runtime.UnlockOSThread()
 }
 
-func NowMicros() int64                                { return int64(C.av_gettime_relative()) }
-func (s *Stream) Changed() (bool, error)              { return s.desktop.changed() }
-func (s *Stream) Wait(micros int64) error             { return s.desktop.wait(micros) }
-func (s *Stream) Send(pts int64, keyframe bool) error { return s.encoder.send(pts, keyframe) }
+func NowMicros() int64                                 { return int64(C.av_gettime_relative()) }
+func (s *Stream) Changed() (bool, error)               { return s.desktop.changed() }
+func (s *Stream) Wait(micros int64, control int) error { return s.desktop.wait(micros, control) }
+func (s *Stream) Send(pts int64, keyframe bool) error  { return s.encoder.send(pts, keyframe) }
+func (s *Stream) Reconfigure(crf, kbps int) error      { return s.encoder.reconfigure(crf, kbps) }
 
 func (s *Stream) Capture(first bool) (copied, converted int64, err error) {
 	if err = avCheck(C.av_frame_make_writable(s.encoder.frame)); err != nil {

package/native/main.go

@@ -1,14 +1,49 @@
 package main
 
 import (
+	"bytes"
 	"fmt"
 	"os"
 	"os/signal"
+	"strconv"
+	"strings"
 	"syscall"
 
 	"github.com/replicas-group/replicas/computer-use/native/internal/capture"
 )
 
+// controls reads the media server's commands from stdin, one per line. They
+// change the running encoder between frames, where a restart would cost a
+// new process and a keyframe. A closed stdin ends control, not capture.
+type controls struct {
+	fd      int
+	pending []byte
+}
+
+func (c *controls) read() []string {
+	var buffer [256]byte
+	for c.fd >= 0 {
+		n, err := syscall.Read(c.fd, buffer[:])
+		if n > 0 {
+			c.pending = append(c.pending, buffer[:n]...)
+			continue
+		}
+		if err == syscall.EINTR {
+			continue
+		}
+		if err != syscall.EAGAIN {
+			c.fd = -1
+		}
+		break
+	}
+	var lines []string
+	for index := bytes.IndexByte(c.pending, '\n'); index >= 0; index = bytes.IndexByte(c.pending, '\n') {
+		lines = append(lines, string(c.pending[:index]))
+		c.pending = c.pending[index+1:]
+	}
+	return lines
+}
+
 func run(o capture.Options) error {
 	stop := make(chan os.Signal, 1)
 	signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
@@ -18,11 +53,15 @@ func run(o capture.Options) error {
 		return err
 	}
 	defer stream.Close()
+	control := &controls{fd: 0}
+	if syscall.SetNonblock(control.fd, true) != nil {
+		control.fd = -1
+	}
 	started := capture.NowMicros()
 	last, lastKeyframe, reportAt := started-1000000, started-1000000, started
 	interval := int64((1000000 + o.FPS - 1) / o.FPS)
 	busyUS, captureUS, scaleUS, encodeUS, previousPTS := int64(0), int64(0), int64(0), int64(0), int64(-1)
-	dirty := true
+	dirty, forceKeyframe := true, false
 	number, changedFrames, sentFrames := 0, 0, 0
 	for {
 		select {
@@ -30,6 +69,27 @@ func run(o capture.Options) error {
 			return nil
 		default:
 		}
+		for _, line := range control.read() {
+			fields := strings.Fields(line)
+			values := make([]int, len(fields))
+			for index := 1; index < len(fields); index++ {
+				if values[index], err = strconv.Atoi(fields[index]); err != nil {
+					values[index] = -1
+				}
+			}
+			switch {
+			case len(fields) == 1 && fields[0] == "keyframe":
+				forceKeyframe = true
+			case len(fields) == 3 && fields[0] == "quality" && values[1] >= 0 && values[1] <= 51 && values[2] > 0:
+				if err = stream.Reconfigure(values[1], values[2]); err != nil {
+					return err
+				}
+			case len(fields) == 2 && fields[0] == "fps" && values[1] > 0 && values[1] <= 240:
+				interval = int64((1000000 + values[1] - 1) / values[1])
+			default:
+				fmt.Fprintf(os.Stderr, "capture: ignoring control %q\n", line)
+			}
+		}
 		changed, err := stream.Changed()
 		if err != nil {
 			return err
@@ -37,15 +97,19 @@ func run(o capture.Options) error {
 		dirty = dirty || changed
 		now := capture.NowMicros()
 		idleDue := last + max(interval, 200000)
-		if o.IntraRefresh == 0 {
+		periodic := o.IntraRefresh == 0 && o.KeyframeMS > 0
+		if periodic {
 			idleDue = min(idleDue, lastKeyframe+int64(o.KeyframeMS)*1000)
 		}
 		due := max(last+interval, idleDue)
 		if dirty || number <= o.Warmup || number < 3 {
 			due = last + interval
 		}
+		if forceKeyframe {
+			due = now
+		}
 		if now < due {
-			if err = stream.Wait(due - now); err != nil {
+			if err = stream.Wait(due-now, control.fd); err != nil {
 				return err
 			}
 			continue
@@ -61,7 +125,8 @@ func run(o capture.Options) error {
 		}
 		pts := max(previousPTS+1, (now-started)*90/1000)
 		previousPTS = pts
-		keyframe := number == 0 || (o.Warmup > 0 && number == o.Warmup) || (o.IntraRefresh == 0 && now-lastKeyframe >= int64(o.KeyframeMS)*1000)
+		keyframe := forceKeyframe || number == 0 || (o.Warmup > 0 && number == o.Warmup) || (periodic && now-lastKeyframe >= int64(o.KeyframeMS)*1000)
+		forceKeyframe = false
 		if keyframe {
 			lastKeyframe = now
 		}

package/native/options.go

@@ -20,7 +20,7 @@ func parseOptions() (capture.Options, error) {
 		{"width", &o.Width, 0, 2, 8192}, {"height", &o.Height, 0, 2, 8192},
 		{"fps", &o.FPS, 60, 1, 240}, {"threads", &o.Threads, 4, 1, 16},
 		{"crf", &o.CRF, 14, 0, 51}, {"kbps", &o.Kbps, 20000, 1, 1000000},
-		{"keyframe-ms", &o.KeyframeMS, 1000, 1, 10000}, {"intra-refresh", &o.IntraRefresh, 1, 0, 1},
+		{"keyframe-ms", &o.KeyframeMS, 1000, 0, 10000}, {"intra-refresh", &o.IntraRefresh, 1, 0, 1},
 		{"chroma-qp", &o.ChromaQP, 0, -12, 12}, {"warmup", &o.Warmup, 0, 0, 60},
 		{"payload", &o.Payload, 96, 0, 127}, {"bt709", &o.BT709, 1, 0, 1},
 	}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "replicas-computer",
-  "version": "0.3.14",
+  "version": "0.3.15",
   "description": "Replicas workspace computer-use runtime",
   "type": "module",
   "main": "dist/index.js",