@desklink/host 0.1.1 → 0.2.0

package/docs/PROTOCOL.md

@@ -23,7 +23,10 @@ to `stderr`. One JSON object per line, no length prefix, UTF-8.
 Why stdio rather than a socket: the consumer already holds the process, so the
 channel inherits the consumer's own access control (same uid, same session, no
 filesystem permission to get wrong) and no second authorization mechanism has to
-exist. A consumer that wants a socket can wrap this process.
+exist. A consumer that wants a socket can wrap this process. (A screen that
+needs no protocol at all — a private `Xvfb` whose windows should simply be kept
+in order and reported — can run the engine as a display keeper instead; see
+"Display keeper" below.)
 
 Cancellation is by closing the consumer's end. The engine treats EOF on `stdin`
 exactly like `shutdown`: it releases held input, stops capture, closes peers, and
@@ -63,7 +66,7 @@ Request parameters use the engine's own snake_case names (`session_id`,
 Result:
 
 ```jsonc
-{"protocol":3,"engine":"desklink-host/0.1.0","platform":"linux", /* the rest of Capabilities */ }
+{"protocol":3,"engine":"desklink-host/0.2.0","platform":"linux", /* the rest of Capabilities */ }
 ```
 
 The consumer must send `hello` first, and a version the engine does not speak is
@@ -105,7 +108,7 @@ session may require fresh consent. The macOS engine requires a VP9-enabled build
 ```jsonc
 {
   "protocol": 3,
-  "engine": "desklink-host/0.1.0",
+  "engine": "desklink-host/0.2.0",
   "platform": "linux",
   "session": {"kind": "wayland"},
   "x11": {"available": true, "size": [2560, 1440]},
@@ -125,10 +128,15 @@ session may require fresh consent. The macOS engine requires a VP9-enabled build
     "unavailable_reason": null,
     "grant": "granted"
   },
-  "clipboard": {"read": true, "write": true, "mime": ["text/plain;charset=utf-8"], "maxBytes": 262144}
+  "clipboard": {"read": true, "write": true, "mime": ["text/plain;charset=utf-8"], "maxBytes": 262144},
+  "encoded": {"codecs": ["h264"]}
 }
 ```
 
+`encoded` lists the codecs this build will carry when the consumer supplies the
+video itself (see *Encoded sources*): a consumer reads it rather than inferring
+support from the engine's version string.
+
 `input.mechanism` names the build's portal input path, `inputtino/uinput`,
 where the engine creates its own virtual devices. A session opened against an X
 display applies input through XTest instead, where the X server applies the
@@ -156,6 +164,7 @@ trigger a capture request.
 {"id":4,"method":"session.open","params":{
   "source": {"kind":"portal"},          // or {"kind":"x11","display":":99"}
                                            // macOS: {"kind":"display","display_id":123}
+                                           // or {"kind":"encoded","codec":"h264","width":486,"height":1080}
   "permissions": ["view","control","clipboard"],
   "max_width": 3840, "max_height": 2160, // encode box (default); never upscales
   "bitrate_kbps": 0,                    // 0 (default): sized to the encoded surface
@@ -182,8 +191,9 @@ Result:
 screen cast, which is the only path that carries a Wayland user's consent;
 `x11` reads a named X display's root window and applies input through XTest;
 `display` selects a macOS display by the ID reported in capabilities (or the
-main display when omitted). The choice is normally the *consumer's* decision,
-because a client is not in a position to know which backend a machine can offer.
+main display when omitted); `encoded` carries video the consumer encoded itself.
+The choice is normally the *consumer's* decision, because a client is not in a
+position to know which backend a machine can offer.
 
 `permissions` are the engine's authority for this session. `view` is required
 to open a session. The engine enforces the remaining scopes on every input and
@@ -233,6 +243,22 @@ the SDP and candidates to the client and brings back the answer.
 
 A candidate that arrives before the remote description is buffered, not dropped.
 
+A client whose ICE connection goes `disconnected` may restart ICE on the same
+peer connection instead of opening a new session: it sends
+`session.restart_ice` with the session's id and generation, and the engine
+re-offers — a fresh offer with a new ICE generation — as a new
+`session.description` event, which the client answers as usual. The session
+and its generation survive; only the ICE generation turns over. Fresh
+candidates from the restart trickle through `session.candidate` on both sides
+exactly as the initial ones do.
+
+```jsonc
+{"id":7,"method":"session.restart_ice","params":{"session_id":"…","generation":1}}
+// → {"id":7,"result":{"accepted":true}}
+// …then {"event":"session.description","params":{"sessionId":"…","generation":1,
+//   "description":{"type":"offer","sdp":"v=0\r\n…"}}}
+```
+
 With `loopback_tcp`, the engine also listens for ICE over TCP on
 `127.0.0.1` and offers that passive candidate. It is for a client whose only
 way to this computer is a forward of its loopback, such as an SSH tunnel: the
@@ -246,10 +272,21 @@ always does.
 ```jsonc
 {"event":"session.state","params":{"sessionId":"…",
   "capture":"streaming",              // consented|streaming|ended
-  "transport":"connected",            // new|connecting|connected|failed|closed
+  "transport":"connected",            // new|connecting|connected|disconnected|failed|closed
   "firstFrame":true}}                 // false until a frame has been encoded
 ```
 
+`disconnected` is ICE losing the path: the picture is frozen but the session
+is alive, and a client should show reconnecting and may restart ICE (above)
+rather than open a new session. `failed` is terminal: the engine revokes and
+closes the session, and only a new `session.open` recovers it.
+
+While its control channel is open the engine also sends a heartbeat twice a
+second, `{"kind":"ping"}`. It lets a client notice a stalled path within
+about a second and a half, long before ICE consent timers declare it
+`disconnected` — and a resumed heartbeat is the path working again. Clients
+that predate it ignore the unknown kind.
+
 A stopped capture emits `{"event":"session.capture.stopped","params":{"sessionId":"…","reason":"SCStream stopped (SCStreamErrorDomain -3821): …"}}`. Frame requests then return `stream-stopped` rather than stale pixels. On macOS the engine retries opening the stream up to three times and emits `session.state` with `capture=streaming` on recovery; Linux capture errors also surface instead of serving an old frame.\n\nThe `geometry` a client maps touch against is the one `session.open` returned,
 and the one the control channel's `hello` repeats. A session keeps the one
 `generation` it was opened with: a different source or scale is a new
@@ -287,7 +324,8 @@ Metrics are a request, not a notification:
 ```jsonc
 {"id":7,"result":{"captured_frames":812,"dropped_frames":3,"encoded_frames":809,
   "encoded_bytes":12345678,"key_frames":1,"refined_frames":37,
-  "encode_micros":7390000,"target_kbps":16430,"input_applied":44,"input_rejected":0}}
+  "encode_micros":7390000,"target_kbps":16430,"input_applied":44,"input_rejected":0,
+  "fed_frames":0,"input_forwarded":0}}
 ```
 
 `dropped_frames` counts frames superseded by a newer one before they were coded;
@@ -327,10 +365,11 @@ are JSON, one per message:
 | `release_all` | — | explicit safety net; the engine also does this on close and on channel loss |
 | `clipboard_read` / `clipboard_write` | `request` (echoed back), `text` | explicit, on user action; never polled |
 
-The engine answers on the same channel:
+The engine answers on the same channel, and sends two things unprompted:
 
 ```jsonc
 {"kind":"hello","protocol":3,"geometry":{…}}     // once, when the channel opens
+{"kind":"ping"}                                  // heartbeat, twice a second; see "State" above
 {"kind":"ack","seq":44}
 {"kind":"rejected","seq":44,"code":"coordinates","message":"(9000,4) is outside the 1280x720 surface"}
 {"kind":"clipboard","request":"…","text":"…","truncated":false}   // an empty "text" with "error" when the read failed
@@ -354,6 +393,111 @@ layout can produce. macOS reports Unicode text support and currently maps
 layout is diagnostic, not proof that this map matches it. Anything outside the
 map is refused with `text-unsupported`.
 
+## Encoded sources
+
+An `encoded` source is video the **consumer** produced: a phone mirror from a
+device helper, a simulator, a camera. The engine captures nothing, encodes
+nothing and holds no input device for it; it packetizes what it is handed and
+returns the client's control messages to the consumer, which owns whatever
+device the picture came from. That is the whole point of the source: the helper
+that already speaks H.264 keeps its own encoder, and the engine keeps the
+WebRTC, congestion and teardown logic that is already proven on this path.
+
+```jsonc
+{"source": {"kind":"encoded","codec":"h264","width":486,"height":1080}}
+```
+
+`codec` is `h264`, the only one this build carries (`capabilities.encoded`).
+`width`/`height` are the stream's own pixel size, chosen by the consumer because
+it is the only party that knows them (`1..7680` by `1..4320`). Nothing is scaled:
+`geometry.source` and `geometry.encoded` are both that size, the client aims at
+those pixels, and the consumer maps them to its device. `max_width`/
+`max_height`/`bitrate_kbps` are ignored for this source, and `permissions` works
+as it does anywhere else — with one difference: `control` needs no local input
+backend, because nothing is applied locally.
+
+`session.open` returns before any offer exists. The offer must name the profile
+the stream actually is (`packetization-mode=1` and the SPS's own
+`profile-level-id`), and the only place that is written down is the stream's own
+SPS, so the engine sends `session.description` **after** the first fed access
+unit that carries one. A consumer must therefore not wait for the offer before
+it starts feeding, and a consumer that never feeds an SPS gets no offer at all.
+
+### Feeding access units
+
+```jsonc
+{"id":8,"method":"session.feed","params":{
+  "session_id":"…",
+  "keyframe":true,                       // this unit is an IDR
+  "data_b64":"AAAA…"                     // one Annex-B access unit, base64
+}}
+```
+
+```jsonc
+{"id":8,"result":{"accepted":true}}
+```
+
+One request is one access unit: start codes and all, exactly as the encoder
+emitted it, with SPS/PPS in the unit that carries the IDR. `keyframe` is the
+consumer's own statement about the unit; the engine uses it for `session.metrics`
+and for nothing else, because the RTP payloader reads the NAL types itself. The
+reply acknowledges the hand-off, not the picture: units are packetized behind
+the request, and `session.metrics` reports what was sent and what was dropped.
+
+Base64 in the same JSON line keeps one ordered channel and one inherited fd.
+Measured, that is not a close call: one second of a 2 Mbit/s stream is 250 KB,
+and base64 costs about 0.09 ms to encode on the consumer's side and 1.0 ms to
+decode on the engine's — roughly a tenth of one per cent of a core, against the
+5 % that would have justified a second fd. Decoded units are
+limited to 8 MiB (`malformed` beyond that); a unit the buffer cannot hold right
+away is **dropped, not queued** — a live picture is worth more than a backlog of
+stale ones — and is counted in `dropped_frames`.
+
+An access unit is dropped until the transport connects, because nothing sent
+before it arrives. The engine therefore asks for a key frame the moment the
+transport connects (below); a consumer that ignores that request shows a black
+picture.
+
+### Key frames the engine cannot make
+
+```jsonc
+{"event":"session.keyframeRequest","params":{"sessionId":"…","generation":1}}
+```
+
+Raised when the receiver reports a loss it cannot repair (RTCP PLI or FIR), and
+once when the transport connects. The consumer responds by feeding a keyframe
+access unit — for a device helper, "reset the video stream". This is the encoded
+source's version of the reference frame the engine's own encode loop sends when
+the same thing happens on a captured desktop.
+
+### Input belongs to the consumer
+
+For this source only, the control channel is **forwarded, not applied**:
+`pointer`, `wheel`, `key`, `text` and `release_all` arrive as events, and the
+consumer injects them wherever the picture came from (a device helper's own
+touch injection, for instance). Coordinates are the encoded surface's own
+pixels, exactly as a client sends them.
+
+```jsonc
+{"event":"session.input","params":{"sessionId":"…",
+  "input":{"kind":"pointer","phase":"down","x":100,"y":200,"button":1,"seq":4}}}
+```
+
+The event carries the control message as it arrived. The engine still validates
+permission, ordering and replay before forwarding, so the client keeps the same
+`ack`/`rejected` behaviour and a replayed gesture still cannot be delivered
+twice. `ack` means "accepted for this session", not "applied to a desktop" —
+`session.metrics.input_forwarded` counts what was handed on, and `input_applied`
+stays zero. Clipboard messages are answered with an empty `clipboard` reply and
+an error, because this source has no desktop clipboard: the device's clipboard
+is the consumer's to carry.
+
+`session.frame` and `session.wait_frame` are refused with `error.code =
+"operation"` — there is no lossless frame to read — and `captured_frames`,
+`refined_frames` and `target_kbps` stay at zero while `fed_frames` counts what
+the consumer handed over and `encoded_frames`, `encoded_bytes`, `key_frames`
+and `encode_micros` describe what was packetized and sent.
+
 ## Latest frame over the local protocol
 
 `session.frame` requires `view` permission. It writes the latest pre-encode frame as
@@ -416,6 +560,58 @@ session that has already ended is refused with `error.code = "session"`.
 
 Closes every session and exits.
 
+## Display keeper
+
+```
+desklink-host keep --display :42
+```
+
+A kiosk window manager for one X screen the consumer created itself — typically
+one `Xvfb` per screen, with its own `-auth` cookie file. It is not for the
+user's desktop: it takes SubstructureRedirect on the root window, so it is that
+screen's window manager, and a display that already has one refuses the keeper.
+Authentication is the ambient `XAUTHORITY`; a screen the consumer created should
+have its own cookie file, and the keeper inherits whatever the consumer's
+environment carries. There is no handshake and no stdin: the contract is a
+stdout JSON-lines stream, and closing the consumer's stdout end ends the keeper.
+
+The keeper fills the screen with every normal (non-override-redirect,
+input-output) top-level window at `0,0,W,H`, raises the newest one and keeps the
+input focus on the newest content window, so keyboard events an engine session
+injects on that display land where a person looking at the screen would expect. A top-level window
+smaller than 120 px on either side is not content — an emulator draws its side
+toolbar as a tiny second window — and is parked far off-screen instead, so a
+capture of the screen never shows it.
+
+On startup, and after every change on the screen, it writes one line:
+
+```jsonc
+{"windows":[{"id":4194305,"title":"Pricing — Acme Store - Chromium","class":"Chromium","pid":31671,"width":1280,"height":800}]}
+```
+
+`windows` lists every visible top-level window, bottom of the stacking order
+first, newest last. `id` is the X window id; `title` comes from `_NET_WM_NAME`
+with `WM_NAME` as the fallback; `class` is the second string of `WM_CLASS` (the
+one consumers filter on, e.g. `Chromium`, `Google-chrome`); `pid` is
+`_NET_WM_PID`. Fields the window does not carry are `null`, never absent. A
+change is reported after 100 ms of quiet, and never later than 300 ms after the
+first unreported change, so a busy screen cannot starve its consumer.
+
+A refused startup is one error line and a non-zero exit:
+
+```jsonc
+{"error":"another window manager"}
+```
+
+The same shape reports a display that could not be opened. When the display
+goes away — an `Xvfb` the consumer stopped, most commonly — the keeper exits 0
+on its own; a consumer treats the process ending as the screen ending.
+
+The keeper deliberately does not decorate, move, resize on request, or remember
+anything across runs. A screen with no windows produces one `"windows":[]` line
+at startup and then nothing: the absence of a report is the absence of
+anything to show.
+
 ## What the engine deliberately does not do
 
 - It does not decide *who* the user is. It trusts the consumer's local

package/engine/Cargo.lock

@@ -517,11 +517,12 @@ checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
 
 [[package]]
 name = "desklink-host"
-version = "0.1.1"
+version = "0.2.0"
 dependencies = [
  "anyhow",
  "ashpd",
  "async-trait",
+ "base64 0.22.1",
  "block2",
  "bytes",
  "cc",

package/engine/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "desklink-host"
-version = "0.1.1"
+version = "0.2.0"
 edition = "2021"
 rust-version = "1.97.1"
 license = "Apache-2.0"
@@ -14,6 +14,7 @@ path = "src/main.rs"
 [dependencies]
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
+base64 = "0.22"
 
 [target.'cfg(target_os = "linux")'.dependencies]
 anyhow = "1"