replicas-computer 0.3.6 → 0.3.7

package/README.md

@@ -24,7 +24,7 @@ ffmpeg captures the display and encodes 1920×1080 H.264 (and Opus from PulseAud
 
 WebRTC is the default transport, with automatic WebSocket fallback. A viewer can choose `?transport=websocket`; `REPLICAS_DESKTOP_STREAM_TRANSPORT=websocket` changes the server default for viewers that do not choose. Dashboard URLs carry the selected transport explicitly, and saved transport preferences are preserved. The viewer's WebSocket to `/stream` is the control plane either way: it carries the server's SDP offer, the viewer's answer, trickled ICE candidates, keepalives and telemetry. Pointer moves ride an unreliable data channel. When the server advertises `inputRace`, clicks, keys, wheel events and held-input release race a direct copy against ordered WebSocket delivery. The input bridge accepts only the next action ID, ignoring duplicates and out-of-order direct copies; the socket fills gaps without SCTP retransmission delays or a reorder queue. Older peers keep their existing input path. The watchdog tracks decoded frames so CSS-hidden viewers do not fall back merely because the browser stops presenting video. An explicit server-side WebRTC failure, a network that cannot pass UDP, or six seconds without decoded video switches the viewer to the socket, where the server reassembles frames and WebCodecs decodes them in the browser.
 
-The server makes the offer, so payload types and SSRCs are fixed and ffmpeg emits them directly. Four H.264 payload types are offered (High 4:4:4 Predictive, High, Main, Constrained Baseline); the encoder runs the best profile every connected viewer accepts, so one viewer without 4:4:4 support moves everyone to 4:2:0, and socket viewers or caps below 8 Mbps do the same. Two profiles select the encoder tuning: `balanced` (CRF 14, default chroma quantisation) and `text` (CRF 10, chroma QP offset −6); `?profile=` picks one and the most recent viewer's choice applies to the shared encoder. Quality adapts on a six-rung bitrate ladder that keeps resolution and frame rate and adds two CRF per rung under the 20 Mbps cap; `?adapt=0` pins the top rung.
+The server makes the offer, so payload types and SSRCs are fixed and ffmpeg emits them directly. Four H.264 payload types are offered (High 4:4:4 Predictive, High, Main, Constrained Baseline); the encoder runs the best profile every connected viewer accepts, so one viewer without 4:4:4 support moves everyone to 4:2:0, and socket viewers or caps below 8 Mbps do the same. Two profiles select the encoder tuning: `balanced` (CRF 14, default chroma quantisation) and `text` (CRF 10, chroma QP offset −6); `?profile=` picks one and the most recent viewer's choice applies to the shared encoder. Quality adapts on a bitrate ladder down to 750 kbps that keeps resolution and frame rate and adds two CRF per rung under the default 20 Mbps cap; `?adapt=0` pins the top rung.
 
 Every video byte for a WebRTC viewer passes through `PacedSender`. libdatachannel's own pacer and NACK responder are not used because repairs bypassed the pacer and neither accounted for duplicates, so recovery traffic could exceed the link while fresh frames queued for seconds. The budget is 1.25× the rung's encoder cap and covers media, repairs and protective copies together: repairs take at most 30% of it over a rolling second, copies are sent only from idle budget while loss is observed and never exceed the original media bytes, and a queue that grows past 150 ms is cut back to whole frames a decoder can still follow (unsent delta frames go, the in-flight frame completes, a keyframe is requested and further deltas are dropped until it arrives). Sequence numbers are assigned as packets leave, so dropped frames never read as loss. Receiver PLI, NACK and REMB are parsed from RTCP. RTT growth (the lowest probe of the last second against the session's baseline, so a single late probe is not a queue), loss, jitter-buffer growth and stale-queue cuts drive the ladder down, and clean seconds drive it up. REMB is not a congestion vote: the receiver's estimate is bounded by what recently arrived and climbs by tens of kbps per second, so it trails every burst of motion on a quiet screen for a minute or more, and its delay detector reads the sender's pacing bursts on a quiet screen as overuse. It only bounds how far a severe descent (heavy loss or a starved decoder) steps, and only while it has stopped climbing below the sent rate. `scripts/probe-viewer.ts` drives the viewer page in a Chrome with remote debugging and prints what it measures and what the server decides each second; `scripts/probe-rtt-jitter.ts` feeds the controller a synthetic WebRTC viewer with jittery RTT probes.
 
@@ -39,7 +39,7 @@ The server records every decision that changes the picture (viewer connects and
 | `REPLICAS_DESKTOP_STREAM_TRANSPORT` | `webrtc` | Transport for viewers whose URL does not choose one |
 | `REPLICAS_DESKTOP_STREAM_PROFILE` | `balanced` | Encoder profile for viewers whose URL does not choose one |
 | `REPLICAS_DESKTOP_STREAM_ADAPTIVE` | `1` | `0` pins the top quality rung by default |
-| `REPLICAS_DESKTOP_STREAM_ICE_SERVERS` | `stun:stun.l.google.com:19302` | Comma-separated STUN/TURN servers; `turn:user:secret@host:port?transport=tcp` adds a relay. Re-read from the workspace runtime env on every viewer, so the monolith's `DESKTOP_STREAM_ICE_SERVERS` reaches running workspaces |
+| `REPLICAS_DESKTOP_STREAM_ICE_SERVERS` | `stun:stun.l.google.com:19302` | Comma-separated STUN/TURN servers; `turn:user:secret@host:port?transport=tcp` adds a relay. Percent-encode the username and password; the browser receives their decoded values. Re-read from the workspace runtime env on every viewer, so the monolith's `DESKTOP_STREAM_ICE_SERVERS` reaches running workspaces |
 | `REPLICAS_DESKTOP_STREAM_ICE_PORTS` | ephemeral | `begin-end` UDP port range for the server's ICE candidates |
 | `REPLICAS_DESKTOP_STREAM_WEBRTC` | `1` | `0` serves the WebSocket transport only |
 | `REPLICAS_DESKTOP_STREAM_COLOR` | `bt709` | `legacy` disables the explicit BT.709 conversion and signalling |
@@ -55,4 +55,4 @@ replicas-computer daemon
 
 `setup --check` reports missing OS dependencies, including the native WebRTC module, without changing the machine. `setup` installs package-owned assets and the `replicas-computer.service` systemd unit. Operational commands use the daemon over a user-only Unix socket.
 
-The sandbox image builds node-datachannel from pinned libdatachannel source with `scripts/build-webrtc.sh`, which applies `patches/libdatachannel-dtls-bio.patch` (upstream DTLS BIO synchronisation fix, [2059d2a](https://github.com/paullouisageneau/libdatachannel/commit/2059d2a3bb81dc3dc24686fdc92c90b6167ec4fd)) and `patches/libdatachannel-nack-duplicates.patch`. It needs CMake, Ninja, a C++ compiler, Git and OpenSSL development libraries. Remove each patch when the pinned addon includes its fix.
+The sandbox image builds node-datachannel from pinned libdatachannel source with `scripts/build-webrtc.sh`, which applies `patches/libdatachannel-dtls-bio.patch` (upstream DTLS BIO synchronisation fix, [2059d2a](https://github.com/paullouisageneau/libdatachannel/commit/2059d2a3bb81dc3dc24686fdc92c90b6167ec4fd)) and `patches/libdatachannel-nack-duplicates.patch`. It also applies `patches/libjuice-ice-role-presence.patch` so valid zero-valued ICE role attributes do not break Chrome TURN/TCP connectivity. It needs CMake, Ninja, a C++ compiler, Git and OpenSSL development libraries. Remove each patch when the pinned addon includes its fix.

package/dist/index.js

@@ -2937,7 +2937,7 @@ async function computerStatusCommand() {
 }
 
 // src/version.ts
-var COMPUTER_USE_VERSION = "0.3.6";
+var COMPUTER_USE_VERSION = "0.3.7";
 var COMPUTER_USE_PROTOCOL_VERSION = 1;
 
 // src/program.ts
@@ -4126,7 +4126,15 @@ function parseIceServers(spec) {
   const native = (spec ?? "stun:stun.l.google.com:19302").split(",").map((server) => server.trim()).filter(Boolean);
   const browser = native.map((url) => {
     const match = url.match(/^(turns?):([^:@]+):(.+)@([^@]+)$/);
-    return match ? { urls: `${match[1]}:${match[4]}`, username: match[2], credential: match[3] } : { urls: url };
+    if (!match) return { urls: url };
+    const [username, credential] = match.slice(2, 4).map((value) => {
+      try {
+        return decodeURIComponent(value);
+      } catch {
+        return value;
+      }
+    });
+    return { urls: `${match[1]}:${match[4]}`, username, credential };
   });
   return { native, browser };
 }
@@ -4284,7 +4292,7 @@ var profiles = {
   balanced: { crf: 14, chromaQpOffset: 0 },
   text: { crf: 10, chromaQpOffset: -6 }
 };
-var rungCaps = [configuredTopKbps, 12e3, 8e3, 5e3, 3e3, 1500].map((kbps) => Math.min(kbps, configuredTopKbps));
+var rungCaps = [configuredTopKbps, 12e3, 8e3, 5e3, 3e3, 1500, 750].map((kbps) => Math.min(kbps, configuredTopKbps));
 var qualityLadder = (profile) => rungCaps.map((kbps, index) => ({ kbps, crf: profiles[profile].crf + 2 * index, keyframeMs: index >= 4 ? 2e3 : 1e3 }));
 var fullChromaMinKbps = 8e3;
 var fallbackKeyframeMs = 250;
@@ -4527,6 +4535,7 @@ async function runMediaServer() {
         colorMode === "bt709" ? `scale=out_color_matrix=bt709:out_range=tv,format=${pixelFormat}` : `format=${pixelFormat}`
       ];
       const x264 = [
+        `fps=${fps}/1`,
         "repeat-headers=1",
         "sliced-threads=1",
         "slices=4",
@@ -4564,6 +4573,8 @@ async function runMediaServer() {
         "1",
         "-i",
         `${display}.0`,
+        "-filter_threads",
+        "1",
         "-vf",
         filters.join(","),
         ...warmupFrames ? ["-force_key_frames", `expr:eq(n,${warmupFrames})`, "-forced-idr", "1"] : [],

package/dist/stream-viewer.js

@@ -362,7 +362,7 @@ var handleBinary = (data) => {
   }
   if (!videoDecoder || !activeVideoConfig || videoDecoder.state !== "configured") return;
   const key = kind === 1 /* VideoKey */;
-  if (videoDecoder.decodeQueueSize > 3) {
+  if (videoDecoder.decodeQueueSize > 12) {
     videoDecoder.reset();
     videoDecoder.configure(activeVideoConfig);
     waitingForKeyframe = true;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "replicas-computer",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "description": "Replicas workspace computer-use runtime",
   "type": "module",
   "main": "dist/index.js",
@@ -12,7 +12,8 @@
     "assets",
     "scripts/build-webrtc.sh",
     "patches/libdatachannel-dtls-bio.patch",
-    "patches/libdatachannel-nack-duplicates.patch"
+    "patches/libdatachannel-nack-duplicates.patch",
+    "patches/libjuice-ice-role-presence.patch"
   ],
   "scripts": {
     "build": "bun run typecheck && tsup",

package/patches/libjuice-ice-role-presence.patch

@@ -0,0 +1,96 @@
+diff --git a/src/agent.c b/src/agent.c
+--- a/src/agent.c
++++ b/src/agent.c
+@@ -1437,7 +1437,7 @@
+ 			return -1;
+ 
+ 		ice_candidate_pair_t *pair = entry->pair;
+-		if (msg->ice_controlling == msg->ice_controlled) {
++		if (msg->has_ice_controlling == msg->has_ice_controlled) {
+ 			JLOG_WARN("Controlling and controlled attributes mismatch in request");
+ 			agent_send_stun_binding(agent, entry, STUN_CLASS_RESP_ERROR, 400, msg->transaction_id,
+ 			                        NULL);
+@@ -1451,7 +1451,7 @@
+ 		//  ERROR-CODE attribute with a value of 487 (Role Conflict) but retains its role.
+ 		//  * If the agent's tiebreaker value is less than the contents of the ICE-CONTROLLING
+ 		//  attribute, the agent switches to the controlled role.
+-		if (agent->mode == AGENT_MODE_CONTROLLING && msg->ice_controlling) {
++		if (agent->mode == AGENT_MODE_CONTROLLING && msg->has_ice_controlling) {
+ 			JLOG_WARN("ICE role conflict (both controlling)");
+ 			if (agent->ice_tiebreaker >= msg->ice_controlling) {
+ 				JLOG_DEBUG("Asking remote peer to switch roles");
+@@ -1471,7 +1471,7 @@
+ 		//  * If the agent's tiebreaker value is less than the contents of the ICE-CONTROLLED
+ 		//  attribute, the agent generates a Binding error response and includes an ERROR-CODE
+ 		//  attribute with a value of 487 (Role Conflict) but retains its role.
+-		if (agent->mode == AGENT_MODE_CONTROLLED && msg->ice_controlled) {
++		if (agent->mode == AGENT_MODE_CONTROLLED && msg->has_ice_controlled) {
+ 			JLOG_WARN("ICE role conflict (both controlled)");
+ 			if (agent->ice_tiebreaker >= msg->ice_controlling) {
+ 				JLOG_DEBUG("Switching to controlling role");
+@@ -1485,7 +1485,7 @@
+ 			break;
+ 		}
+ 		if (msg->use_candidate) {
+-			if (!msg->ice_controlling) {
++			if (!msg->has_ice_controlling) {
+ 				JLOG_WARN("STUN message use_candidate missing ice_controlling attribute");
+ 				agent_send_stun_binding(agent, entry, STUN_CLASS_RESP_ERROR, 400,
+ 				                        msg->transaction_id, NULL);
+@@ -1715,6 +1715,8 @@
+ 			password = agent->remote.ice_pwd;
+ 			msg.ice_controlling = agent->mode == AGENT_MODE_CONTROLLING ? agent->ice_tiebreaker : 0;
+ 			msg.ice_controlled = agent->mode == AGENT_MODE_CONTROLLED ? agent->ice_tiebreaker : 0;
++			msg.has_ice_controlling = agent->mode == AGENT_MODE_CONTROLLING;
++			msg.has_ice_controlled = agent->mode == AGENT_MODE_CONTROLLED;
+ 
+ 			// RFC 8445 7.1.1. PRIORITY
+ 			// The PRIORITY attribute MUST be included in a Binding request and be set to the value
+diff --git a/src/stun.c b/src/stun.c
+--- a/src/stun.c
++++ b/src/stun.c
+@@ -157,14 +157,14 @@
+ 			goto overflow;
+ 		pos += len;
+ 	}
+-	if (msg->ice_controlling) {
++	if (msg->has_ice_controlling || msg->ice_controlling) {
+ 		uint64_t ice_controlling = htonll(msg->ice_controlling);
+ 		len = stun_write_attr(pos, end - pos, STUN_ATTR_ICE_CONTROLLING, &ice_controlling, 8);
+ 		if (len <= 0)
+ 			goto overflow;
+ 		pos += len;
+ 	}
+-	if (msg->ice_controlled) {
++	if (msg->has_ice_controlled || msg->ice_controlled) {
+ 		uint64_t ice_controlled = htonll(msg->ice_controlled);
+ 		len = stun_write_attr(pos, end - pos, STUN_ATTR_ICE_CONTROLLED, &ice_controlled, 8);
+ 		if (len <= 0)
+@@ -916,6 +916,7 @@
+ 		}
+ 		uint32_t *value32 = (uint32_t *)attr->value;
+ 		msg->ice_controlling = ((uint64_t)ntohl(value32[0]) << 32) | ntohl(value32[1]);
++		msg->has_ice_controlling = true;
+ 		break;
+ 	}
+ 	case STUN_ATTR_ICE_CONTROLLED: {
+@@ -926,6 +927,7 @@
+ 		}
+ 		uint32_t *value32 = (uint32_t *)attr->value;
+ 		msg->ice_controlled = ((uint64_t)ntohl(value32[0]) << 32) | ntohl(value32[1]);
++		msg->has_ice_controlled = true;
+ 		break;
+ 	}
+ 	case STUN_ATTR_CHANNEL_NUMBER: {
+diff --git a/src/stun.h b/src/stun.h
+--- a/src/stun.h
++++ b/src/stun.h
+@@ -319,6 +319,8 @@
+ 	uint32_t priority;
+ 	uint64_t ice_controlling;
+ 	uint64_t ice_controlled;
++	bool has_ice_controlling;
++	bool has_ice_controlled;
+ 	bool use_candidate;
+ 	addr_record_t mapped;
+ 

package/scripts/build-webrtc.sh

@@ -19,6 +19,7 @@ git -C "$source_dir" checkout -q --detach FETCH_HEAD
 git -C "$source_dir" submodule update --init --recursive --depth 1
 git -C "$source_dir" apply "$script_dir/../patches/libdatachannel-dtls-bio.patch"
 git -C "$source_dir" apply "$script_dir/../patches/libdatachannel-nack-duplicates.patch"
+git -C "$source_dir/deps/libjuice" apply "$script_dir/../patches/libjuice-ice-role-presence.patch"
 
 cmake -S "$build_root" -B "$build_root/build" -G Ninja \
   -DCMAKE_BUILD_TYPE=Release \