@linkshell/mac 0.1.3 → 0.1.5

package/README.md

@@ -7,7 +7,38 @@ unix socket; nobody starts it by hand.
 
 It has no place in the Dock (`LSUIElement`) and one window, shown only when asked for
 (`--setup`): the one in which the user gives it its two permissions. macOS 13 or later, on
-Apple silicon only: there is no build for Intel Macs.
+Apple silicon and Intel. The universal app and its WebRTC framework contain both arm64 and x86_64;
+macOS chooses the native architecture at launch. No separate installer or Rosetta is needed.
+Intel real-hardware capture, encoding and end-to-end performance validation is pending.
+
+## Current media path
+
+ScreenCaptureKit produces NV12 `CVPixelBuffer`s backed by IOSurface. The video track excludes
+the cursor. `ScreenEncoderFactory` defaults to the app's low-latency VideoToolbox H.264
+adapter, retaining libwebrtc's transport, bandwidth feedback, NACK/RTX and negotiated FlexFEC.
+Encoder initialization/operation failures fall back to stock H.264; `--encoder stock` is the
+explicit comparison path. `playout-delay` 0/0 asks for prompt display, not zero end-to-end delay.
+The attempted `MaxFrameDelayCount=0` property is not supported by the hardware in the recorded
+checks and does not establish zero encoder queueing.
+
+The media peer is an iOS native receiver or a web player, independently of the Node host's
+werift bulk-data connection. iOS normally decodes H.264 through VideoToolbox into a latest-frame
+mailbox, then maps the pixel buffer to Metal. Its transparent WebView still owns gestures,
+controls and the cursor. It requests a ceiling of 120 fps, bounded by both displays, power,
+thermal state and adaptation; standard receivers retain the 60/30 defaults. No video frames
+cross JS. The native module shares the app's JitsiWebRTC M124 framework; this sender uses M154,
+so codec/extension support must be established by the actual negotiation.
+
+Failed media falls back to the host's H.264 byte stream. iOS first tries the standard WebView
+receiver and then bypasses the bulk direct channel for that screen, using RPC. Android/Web
+retain their existing stream path. Both Mac capture paths use ScreenCaptureKit/VideoToolbox;
+only the byte stream includes the cursor in the picture. There is no TURN service.
+
+See the [current architecture and diagrams](../../docs/v2/screen-realtime.md),
+[iOS receiver](../client/modules/link-screen/README.md) and
+[historical measurements](../../docs/v2/screen-realtime-history.md). Stable 120 fps, stable
+4K/60 and phone-side FlexFEC recovery are not established; the recorded periodic stutter is
+still unresolved. These are validation limits, not reasons to change defaults without evidence.
 
 ## Why it is an app
 
@@ -114,7 +145,7 @@ sleeping when the lid is closed or the user chooses Sleep.
 A package manager can't carry the app as it is: npm leaves out the framework's symbolic links,
 pnpm the program's permission to run. The package carries one file,
 `build/LinkShell.app.tar.gz`; `@linkshell/host` has this package as an optional dependency
-(`os: darwin`, `cpu: arm64`) and unpacks the archive with `/usr/bin/tar -xzf` into `~/.linkshell/LinkShell.app`,
+(`os: darwin`, `cpu: arm64, x64`) and unpacks the archive with `/usr/bin/tar -xzf` into `~/.linkshell/LinkShell.app`,
 once per archive (by its SHA-256). In a checkout the host uses `build/LinkShell.app` directly.
 
 ## Build and check
@@ -128,10 +159,10 @@ pnpm test                         # swift test: what needs no screen and no netw
 pnpm check                        # that, then pass or fail for the built app, in about a minute
 ```
 
-`build-app.mjs [--out <directory>]` builds for arm64, embeds the framework (its arm64 half),
+`build-app.mjs [--out <directory>] [--scratch-path <directory>]` builds a universal arm64/x86_64 executable, embeds both framework architectures, checks that neither is missing,
 signs from the inside out (hardened runtime, timestamp), swaps the
 finished bundle into place, packs the archive and proves that the archive unpacks into an app
-whose signature verifies. It signs with the "Developer ID Application" identity in the keychain,
+whose signature verifies for both architectures. It signs with the "Developer ID Application" identity in the keychain,
 or the one `LINKSHELL_SIGN_IDENTITY` names. Without one it signs ad hoc: that app runs, but is
 another app to the system, with permissions of its own — not for release.
 `LINKSHELL_REQUIRE_SIGNED=1` (set by `prepack`) makes a missing identity fatal. The app is not
@@ -147,6 +178,7 @@ already, and stop if it is not rather than have the system ask.
 | `stream-check.mjs` | The stream for the host's socket: record format, rates, key frames, change of size, beside a video track; decoded by ffmpeg where installed, and by Chrome's WebCodecs unless `--no-chrome`. Part of `check`. |
 | `app-smoke.mjs` | Opens the app, asks for an offer, says what it holds. Part of `check`. |
 | `loopback.mjs` | The video track with no viewer: the app answers its own offer, and both ends' numbers are printed. Without `--fps`, at the app's own choice of frame rate; `--fps 30,60` holds each of those. `--narrow <bit/s>:<from second>:<for seconds>` holds the bandwidth estimate down for a while, prints every second, and says what each change of frame rate cost; `--timeline`, `--seconds 10`, `--motion`, `--still`, `--encoder own\|stock`, `--json <file>`; the rest in its header. |
+| `flexfec-check.mjs` | A short functional check over its own loopback UDP proxy: 60 ms each way, every twentieth original video packet dropped. Checks real FlexFEC packet receipt, then successful video without FlexFEC when the test receiver declines it. `--json <file>` saves the evidence. No system network changes or latency claims. |
 | `setup-shots.mjs` | The setup window drawn into PNG files in every state, light and dark, Chinese and English (24 of them, in `build/setup-shots` or the directory named), with nothing shown and nothing asked of the system. For looking at the window after changing it. |
 | `app.mjs`, `h264.mjs`, `pointer-shapes.swift` | Shared by the above: opening the app, reading H.264, changing the pointer's picture. |
 
@@ -168,9 +200,12 @@ The host's own tests run its real code against this app: `packages/host/test/app
 | `--clock` | Shows the clock strip on a display while it is sent (see Measuring). `status` gains `clock`. |
 | `--motion` | Shows a window of scrolling text and a sliding block on a display while it is sent: a busy screen's work for the encoder, the same every run. |
 | `--loopback` | Each `rtc.open` is answered inside the app by a second connection that decodes the track. Adds `rtc.loopback*` (below). |
-| `--encoder own\|stock` | The video track's H.264 encoder: the app's low-latency one (`LowLatencyEncoder.swift`), or libwebrtc's VideoToolbox encoder with the level fixed (`Encoders.swift`). Default `stock`. |
+| `--encoder own\|stock` | The video track's H.264 encoder: the app's low-latency one (`LowLatencyEncoder.swift`), or libwebrtc's VideoToolbox encoder with the level fixed (`Encoders.swift`). Default `own`; initialization or encoding failures switch to stock for the rest of that encoder's lifetime. |
 | `--no-low-latency` | Never asks VideoToolbox for the low-latency rate control: encodes as a Mac without it does. |
 | `--no-playout-delay` | Leaves out the field trial that sends playout-delay 0. |
+| `--no-flexfec` | Disables FlexFEC advertisement and sending for an explicit comparison. |
+| `--loopback-no-flexfec` | Makes the test receiver decline FlexFEC through codec preferences, to check an unsupported viewer's path. |
+| `--loopback-network` | Only with `--loopback`: passes its ICE candidates to `flexfec-check.mjs`, which supplies proxy candidates. The proxy exists only while that tool runs. |
 | `--trial <name>=<value>` | Another libwebrtc field trial (repeatable). |
 | `--rtc-log` | libwebrtc's log, as `log` messages. |
 
@@ -230,12 +265,12 @@ and takes its place; what is then said for the earlier `v` is dropped.
 
 | | Message | |
 |---|---|---|
-| → | `{t:"rtc.open", v, screen?, iceServers?, maxWidth?, fps?, codec?}` | `iceServers`: `[{urls, username?, credential?}]`, `urls` a string or a list. `maxWidth`: default 1920, at least 160; the picture is never larger than the display. `fps`: 1–60, the one rate to send at; left out, 60, and 30 while 60 is not being carried (see Tuning). `codec`: `"h264"` (default) or `"hevc"`, which this build of libwebrtc can't encode: H.264 is offered, and a `log` says so. |
+| → | `{t:"rtc.open", v, screen?, iceServers?, maxWidth?, fps?, maxFps?, codec?}` | `iceServers`: `[{urls, username?, credential?}]`, `urls` a string or list. `maxWidth`: default 1920, clamped to 160–3840 and bounded by the source display. `maxFps`: receiver ceiling, 30 / 60 / 120, default 60; automatic rate selection also considers the source display. `fps`: optional fixed 1–120 rate, overriding automatic selection (internal comparison). `codec`: `"h264"` (default) or `"hevc"`, which this build cannot encode; H.264 is offered and a log explains why. |
 | ← | `{t:"rtc.offer", v, sdp}` | The app offers: one send-only video track, and the four data channels. |
 | → | `{t:"rtc.answer", v, sdp}` | |
 | ↔ | `{t:"rtc.ice", v, candidate, sdpMid, sdpMLineIndex}` | Trickled, either way. Gathering goes on for the life of the session, so a viewer that changes network is found again. |
 | ← | `{t:"rtc.state", v, ice, connection}` | On every change of either, named as a browser names them. |
-| ← | `{t:"rtc.stats", v, …}` | Once a second: `bitrate` (bit/s), `fps` (frames encoded), `frameRate` (the rate in force: 60 or 30, or the one `rtc.open` named), `frameRateReason` (why it last changed, as the `log` said it; `null` while it hasn't), `width`, `height`, `encoder`, `hardware`, `codec`, `fmtp`, `keyFrames`, `qualityLimitation`, `targetBitrate`, `availableOutgoing`, `rttMs`, `packetsLost`, `nack`, `pli`, `encodeMs`, `sendDelayMs`, `hugeFrames`, `captureFps`, `repeated` (the last picture sent again), `gapMs` (the longest libwebrtc went without a picture, new or repeated), `heard` (messages received on each channel), `clock` (with `--clock`: the strip as captured against the time it was displayed, `{n, unreadable, min, median, max}` in ms). A number libwebrtc doesn't have is `null`. |
+| ← | `{t:"rtc.stats", v, …}` | Once a second: `bitrate` (bit/s), `fps` (frames encoded), `frameRate` (the rate in force: up to 120 / 60 / 30 subject to source and receiver limits, or the fixed `fps` requested), `frameRateReason` (why it last changed, as the `log` said it; `null` while it hasn't), `width`, `height`, `encoder`, `hardware`, `codec`, `fmtp`, `keyFrames`, `qualityLimitation`, `targetBitrate`, `availableOutgoing`, `rttMs`, `packetsLost`, `nack`, `pli`, `encodeMs`, `sendDelayMs`, `hugeFrames`, `captureFps`, `repeated` (the last picture sent again), `gapMs` (the longest libwebrtc went without a picture, new or repeated), `heard` (messages received on each channel), `clock` (with `--clock`: the strip as captured against the time it was displayed, `{n, unreadable, min, median, max}` in ms). A number libwebrtc doesn't have is `null`. |
 | ← | `{t:"rtc.error", v, message}` | The session has ended, and why. |
 | → | `{t:"rtc.close", v}` | |
 
@@ -247,15 +282,42 @@ be made`, `the video could not be added`, `no offer: …`, `the offer was not se
 The picture has no pointer in it; the viewer draws one from `cursor` and `shape`. A failed
 connection does not end the session: the host closes it.
 
+`rtc.stats.flexfec` reports SDP evidence: `{state, payloadType, mediaSSRC, repairSSRC}`. The
+state is `not-offered`, `awaiting-answer`, `declined`, `missing-ssrc` or `negotiated`. The last
+means the answer accepted the codec and the offer has an FEC-FR protection group; it does not
+claim that protection packets have been needed or that a loss was repaired.
+
 With `--loopback`: `← {t:"rtc.loopback", v, …}` once a second (the receiving end's numbers,
 among them `gapMs`, the longest it went without a decoded frame, and with `--clock` `latency`
-`{n, min, p50, p95, max}`: glass to decoded, ms),
+`{n, min, p50, p95, max}`: glass to decoded, ms). `fecPacketsReceived` and `fecBytesReceived`
+are the receiver's protection counters (`null` when unavailable), not recovery counts.
+The test control messages are
 `→ {t:"rtc.loopback.limit", v, bitrate?}` to hold the sender's bandwidth estimate to so many
 bits a second, as a narrow network does (without its loss and delay), or without `bitrate` to
 let it go,
 `← {t:"rtc.loopback.channel", v, channel}` when a channel opens there,
 `→ {t:"rtc.loopback.send", v, channel, events:[…]}` to send messages up one, and
 `← {t:"rtc.loopback.heard", v, channel, bytes, message}` for each that came down one.
+With `--loopback-network`, `↔ {t:"rtc.loopback.ice", v, candidate, sdpMid, sdpMLineIndex}`
+exchanges the test receiver's ICE candidates with the tool. Normal viewers never use it.
+
+**FlexFEC.** M154 needs both `WebRTC-FlexFEC-03-Advertised=Enabled` (send codec) and
+`WebRTC-FlexFEC-03=Enabled` (protection SSRC and sending). They are on by default. Codec
+preferences retain `flexfec-03`; libwebrtc negotiates it normally and produces the FEC-FR
+group for our single video stream. A viewer which declines it continues with H.264 and
+NACK/RTX. There is no SDP rewriting to pretend a viewer supports it. See M154's
+[video engine](https://webrtc.googlesource.com/src/+/refs/branch-heads/8037/media/engine/webrtc_video_engine.cc),
+[stream negotiation](https://webrtc.googlesource.com/src/+/refs/branch-heads/8037/pc/media_session.cc), and
+[FEC sender creation](https://webrtc.googlesource.com/src/+/refs/branch-heads/8037/call/rtp_video_sender.cc).
+
+Upstream [M124](https://webrtc.googlesource.com/src/+/refs/branch-heads/6367/media/engine/webrtc_video_engine.cc)
+already exposes FlexFEC receive support unless explicitly disabled; the installed Jitsi build
+and WKWebView still have to prove their capability in their actual answers. A clean network
+can negotiate FlexFEC and send no repair packets, because M154's protection controller uses
+loss and RTT to choose redundancy. The advertised `repair-window=10000000` is a required
+format parameter which this upstream implementation does not use as a ten-second playback
+wait. `flexfec-check.mjs` checks actual packet receipt under injected loss; it does not measure
+real-device recovery, visual stalls or latency. Those remain phone acceptance checks.
 
 ### The data channels
 
@@ -291,7 +353,7 @@ and a stream beside an `rtc` session, run independently.
 
 | | Message | |
 |---|---|---|
-| → | `{t:"stream.open", v, socket, screen?, width?, fps?, bitrate?, ceiling?, gop?, profile?}` | `socket`: the path the app connects to and writes records on. `width`: the widest the picture may be, default 1920, at least 160. `fps`: default 30, 1–60. `bitrate`: bit/s on average, default 2 000 000, at least 50 000. `ceiling`: the most in any one second, never below `bitrate`. `gop`: seconds between key frames at most, default 1, at least 0.2. `profile`: `"baseline"` or `"high"`; left out, High from the low-latency encoder and Baseline from the usual one. |
+| → | `{t:"stream.open", v, socket, screen?, width?, fps?, bitrate?, ceiling?, gop?, profile?}` | `socket`: the path the app connects to and writes records on. `width`: the widest the picture may be, default 1920, at least 160. `fps`: default 30, 1–120 at this helper interface; the host's compatibility ladder requests only 8–20 fps. `bitrate`: bit/s on average, default 2 000 000, at least 50 000. `ceiling`: the most in any one second, never below `bitrate`. `gop`: seconds between key frames at most, default 1, at least 0.2. `profile`: `"baseline"` or `"high"`; left out, High from the low-latency encoder and Baseline from the usual one. |
 | → | `{t:"stream.set", v, width?, fps?, bitrate?, ceiling?}` | Takes effect without the stream stopping. |
 | → | `{t:"stream.key", v}` | The next frame is a key frame, and goes now. |
 | ← | `{t:"stream.started", v, width, height, fps, generation, mode, profile, hardware}` | After the first frame of each generation is written. `mode`: `"low-latency"` or `"real-time"`, the rate control the encoder gave. |
@@ -328,44 +390,48 @@ again once.
 
 ## Tuning
 
-`Sources/LinkShell/Tuning.swift` has the numbers that shape the picture, each with its reason:
-the defaults (1920 wide; 60 frames for the track, 30 for the stream), the track's ceiling (8 Mbit/s for 1920×1080 at 30, in
-proportion to the pixels, half as much again above 30, between 2 and 16) and where its
-bandwidth estimate starts (2 Mbit/s), the degradation preference (the picture gets smaller, not
-jerkier), the field trials (playout-delay 0), how much more than the frame rate the capture is
-asked for (1.1×), how often a still screen's picture is sent again (every 0.1 s for a second,
-then every 0.5 s), the pointer's rates (60 and 5 a second), and everything about the stream's
-pacing and key frames.
-
-**The track's frame rate** (`FrameRate.swift`, a rule with no clock and no network in it,
-tested in `Tests/`; what a change takes is `ScreenSession.change`). A session that `rtc.open`
-gave no `fps` starts at 60 and is judged once a second, while its connection is up, on the
-numbers `rtc.stats` says:
-
-- **Down to 30** when, for 3 seconds running, the picture sent has under 0.9 of the captured
-  picture's pixels (libwebrtc made it smaller: too little bandwidth, or too little processor),
-  or the screen gives at least 0.8 of the rate and under 0.75 of what it gives is encoded.
-  Packets lost, a low estimate while the picture is whole, and the encoder's time over a frame
-  are not reasons.
-- **Back to 60** when, for 10 seconds running at 30, the picture is whole, libwebrtc says
-  nothing limits it, frames are not being lost and the bandwidth estimate is at least 0.4 of
-  the ceiling at 60 (4.8 Mbit/s for 1920×1080). The 10 seconds double, up to 160, each time 60
-  is lost again within 30 seconds of being tried.
-- Nothing is concluded in a connection's first 4 seconds, nor in the 4 after each change.
-
-A change is made with the capture running and nothing negotiated: ScreenCaptureKit's frame
-interval, the sender's `maxFramerate` and its ceiling. Going down, a picture libwebrtc had
-shrunk is whole again from the next frame (a key frame) and stays so for 4 seconds, after which
-libwebrtc may shrink it again; going up, the first second may send half of what 30 was sending.
-Each change is a `log`: `frame rate 60 → 30: the picture was being shrunk for 3 s (bandwidth)`,
-`frame rate 30 → 60: nothing limited the picture for 10 s, and the network is estimated at
-9.8 Mbit/s`.
-
-Elsewhere: the quantizers at which libwebrtc resizes the picture (the encoders say 28 and 39,
-`LowLatencyEncoder.swift`; libwebrtc goes by its own for H.264, 24 and 37, as its log says), the sizing of key frames (`VideoCompressor.swift`), the 0.3 s after an event in
-which the pointer is the viewer's own and the ±2000 on a scroll (`Control.swift`), the pause
-and the 2.5 s around the input source (`Keys.swift`), and the pointer picture's scale
-(`PointerShape.swift`).
+`Sources/LinkShell/Tuning.swift` defines the shared limits. The video defaults to a maximum
+width of 1920, with a 3840 cap; an ordinary receiver gets a 60 fps ceiling, while `maxFps`
+can request 30 / 60 / 120 without disabling adaptation. The source display bounds the starting
+rate. The helper's separate byte-stream interface defaults to 30 fps, but `screen-pacer.ts`
+selects its own 8–20 fps ladder (1280-wide / 12 fps / 900 kbit/s target when `q=low`). These
+are different paths; selecting a video width does not override the compatibility ladder.
+
+The video bitrate ceiling starts from 8 Mbit/s for 1920×1080 at 30 fps. Pixel counts below
+1080p scale proportionally; above it they scale by their square root. The frame factor is 1
+through 30 fps, 1.5 through 60, and `1.5 × fps / 60` above 60. The result is clamped to
+2–30 Mbit/s: 1080p30/60/120 ceilings are 8/12/24 Mbit/s. Actual bitrate follows WebRTC's
+estimate, initialized at 2 Mbit/s, and the content.
+
+`maintainFramerate` permits WebRTC to reduce resolution under resource pressure. Separately,
+`ScreenEncoderFactory.scaling` returns no QP thresholds, suppressing QP-only quality scaling
+that unnecessarily shrank text. `FrameRate.swift` judges the connection once a second and
+reduces rate when the full picture/rate cannot be carried:
+
+- Start at the receiver/source ceiling, normally 120 or 60, with lower stages at 60 and 30
+  where applicable. A source below these rates bounds the starting rate further. Explicit
+  `fps` is a fixed internal override.
+- Reduce after 3 consecutive troubled seconds: estimated bandwidth below the full-rate
+  ceiling × 0.2 × current/full, sent pixel share below 0.9, or encoded output below 0.75 of
+  the requested active input while capture supplies at least 0.8 of the current rate.
+  Packet loss alone and time per encoded frame are not independent triggers.
+- Raise to the next stage after 10 good seconds: full picture, no quality limitation,
+  sufficient encoded output and estimated bandwidth at least the full-rate ceiling × 0.4
+  × next/full. If a raised rate fails again within 30 seconds, the next wait doubles, up
+  to 160 seconds.
+- The first 4 seconds and the 4 after a change are settling periods.
+
+Rate changes update ScreenCaptureKit, sender parameters and the bitrate ceiling without
+renegotiating the media connection. A decrease requests restoration of the captured dimensions;
+the sender can adapt them again later. An increase eases the first second's bitrate using
+half the previous observed bitrate, with a 1 Mbit/s floor. Neither change guarantees physical
+presentation cadence.
+
+Other settings include capture rate slack (1.1×), capture surface queue depth (5), static-frame
+repeats (0.1 s for a second, then 0.5 s), cursor position/shape polling (60/5 Hz), playout-delay
+0/0 and FlexFEC field trials. Surface pool sizes are not measured queued-frame counts.
+`VideoCompressor.swift` controls keyframe sizing, `Control.swift` input state and scroll bounds,
+and `Keys.swift` the temporary input-source switch for text.
 
 ## Measuring
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@linkshell/mac",
-  "version": "0.1.3",
+  "version": "0.1.5",
   "description": "LinkShell.app: the Mac side of the screen — capture, encoding, WebRTC and input, under the app's own permissions.",
   "license": "MIT",
   "repository": {
@@ -13,7 +13,8 @@
     "darwin"
   ],
   "cpu": [
-    "arm64"
+    "arm64",
+    "x64"
   ],
   "files": [
     "build/LinkShell.app.tar.gz",