@linkshell/mac 0.1.4 → 0.1.6

package/README.md

@@ -7,7 +7,44 @@ unix socket; nobody starts it by hand.
 
 It has no place in the Dock (`LSUIElement`) and one window, shown only when asked for
 (`--setup`): the one in which the user gives it its two permissions. macOS 13 or later, on
-Apple silicon only: there is no build for Intel Macs.
+Apple silicon and Intel. The universal app and its WebRTC framework contain both arm64 and x86_64;
+macOS chooses the native architecture at launch. No separate installer or Rosetta is needed.
+Intel real-hardware capture, encoding and end-to-end performance validation is pending.
+
+## Current media path
+
+ScreenCaptureKit produces NV12 `CVPixelBuffer`s backed by IOSurface. The video track excludes
+the cursor. `ScreenEncoderFactory` defaults to the app's low-latency VideoToolbox H.264
+adapter, retaining libwebrtc's transport, bandwidth feedback, NACK/RTX and negotiated FlexFEC.
+Encoder initialization/operation failures fall back to stock H.264; `--encoder stock` is the
+explicit comparison path. `playout-delay` 0/0 asks for prompt display, not zero end-to-end delay.
+The attempted `MaxFrameDelayCount=0` property is not supported by the hardware in the recorded
+checks and does not establish zero encoder queueing.
+The adapter now admits at most two raw frames to VideoToolbox at once. If both are pending,
+it skips new raw input and retains any key-frame request for the next accepted frame. It never
+discards already-compressed reference frames to enforce this bound.
+
+The media peer is an iOS native receiver or a web player, independently of the Node host's
+werift bulk-data connection. iOS normally decodes H.264 through VideoToolbox into a latest-frame
+mailbox, then maps the pixel buffer to Metal. Its transparent WebView still owns gestures,
+controls and the cursor. It requests a ceiling of 120 fps, bounded by both displays, power,
+thermal state and adaptation; standard receivers retain the 60/30 defaults. No video frames
+cross JS. Display-link callbacks, texture mapping and GPU submission now run on a dedicated
+iOS render thread; UIKit publishes small layout snapshots. This removes dependence on UI
+event processing, without establishing a phone latency or sustained-frame-rate result.
+The native module shares the app's JitsiWebRTC M124 framework; this sender uses M154,
+so codec/extension support must be established by the actual negotiation.
+
+Failed media falls back to the host's H.264 byte stream. iOS first tries the standard WebView
+receiver and then bypasses the bulk direct channel for that screen, using RPC. Android/Web
+retain their existing stream path. Both Mac capture paths use ScreenCaptureKit/VideoToolbox;
+only the byte stream includes the cursor in the picture. There is no TURN service.
+
+See the [current architecture and diagrams](../../docs/v2/screen-realtime.md),
+[iOS receiver](../client/modules/link-screen/README.md) and
+[historical measurements](../../docs/v2/screen-realtime-history.md). Stable 120 fps, stable
+4K/60 and phone-side FlexFEC recovery are not established; the recorded periodic stutter is
+still unresolved. These are validation limits, not reasons to change defaults without evidence.
 
 ## Why it is an app
 
@@ -114,7 +151,7 @@ sleeping when the lid is closed or the user chooses Sleep.
 A package manager can't carry the app as it is: npm leaves out the framework's symbolic links,
 pnpm the program's permission to run. The package carries one file,
 `build/LinkShell.app.tar.gz`; `@linkshell/host` has this package as an optional dependency
-(`os: darwin`, `cpu: arm64`) and unpacks the archive with `/usr/bin/tar -xzf` into `~/.linkshell/LinkShell.app`,
+(`os: darwin`, `cpu: arm64, x64`) and unpacks the archive with `/usr/bin/tar -xzf` into `~/.linkshell/LinkShell.app`,
 once per archive (by its SHA-256). In a checkout the host uses `build/LinkShell.app` directly.
 
 ## Build and check
@@ -128,10 +165,10 @@ pnpm test                         # swift test: what needs no screen and no netw
 pnpm check                        # that, then pass or fail for the built app, in about a minute
 ```
 
-`build-app.mjs [--out <directory>]` builds for arm64, embeds the framework (its arm64 half),
+`build-app.mjs [--out <directory>] [--scratch-path <directory>]` builds a universal arm64/x86_64 executable, embeds both framework architectures, checks that neither is missing,
 signs from the inside out (hardened runtime, timestamp), swaps the
 finished bundle into place, packs the archive and proves that the archive unpacks into an app
-whose signature verifies. It signs with the "Developer ID Application" identity in the keychain,
+whose signature verifies for both architectures. It signs with the "Developer ID Application" identity in the keychain,
 or the one `LINKSHELL_SIGN_IDENTITY` names. Without one it signs ad hoc: that app runs, but is
 another app to the system, with permissions of its own — not for release.
 `LINKSHELL_REQUIRE_SIGNED=1` (set by `prepack`) makes a missing identity fatal. The app is not
@@ -148,12 +185,57 @@ already, and stop if it is not rather than have the system ask.
 | `app-smoke.mjs` | Opens the app, asks for an offer, says what it holds. Part of `check`. |
 | `loopback.mjs` | The video track with no viewer: the app answers its own offer, and both ends' numbers are printed. Without `--fps`, at the app's own choice of frame rate; `--fps 30,60` holds each of those. `--narrow <bit/s>:<from second>:<for seconds>` holds the bandwidth estimate down for a while, prints every second, and says what each change of frame rate cost; `--timeline`, `--seconds 10`, `--motion`, `--still`, `--encoder own\|stock`, `--json <file>`; the rest in its header. |
 | `flexfec-check.mjs` | A short functional check over its own loopback UDP proxy: 60 ms each way, every twentieth original video packet dropped. Checks real FlexFEC packet receipt, then successful video without FlexFEC when the test receiver declines it. `--json <file>` saves the evidence. No system network changes or latency claims. |
+| `network-check.mjs` | A 48-second congestion/recovery scenario: an owned UDP proxy changes media capacity from 20 to 0.7 to 20 Mbps, with 10 ms propagation each way and a 200 ms router queue. Includes RTX/FEC in the budget. `--screen`, `--maxFps`, `--maxWidth`, `--app`, repeatable `--trial`, `--json`; passing proves traversal and decoding, not acceptable latency or iPhone performance. |
+| `media-lab.mjs` | Loopback-only encrypted UDP experiment using the same capture, VT H.264 and native decoder. `--network` uses the congestion scenario above; `--no-ltr` checks recovery without acknowledged long-term references. Requires `--loopback --media-lab` in the helper. Not a remote transport. |
+| `codec-check.mjs` | Encode-only H.264/HEVC hardware comparison with fixed `--fps`, `--bitrate`, `--maxWidth` and raw input bound `--pending`. No transport, decoder, picture-quality or display result. HEVC is not enabled for shipping media. |
 | `setup-shots.mjs` | The setup window drawn into PNG files in every state, light and dark, Chinese and English (24 of them, in `build/setup-shots` or the directory named), with nothing shown and nothing asked of the system. For looking at the window after changing it. |
 | `app.mjs`, `h264.mjs`, `pointer-shapes.swift` | Shared by the above: opening the app, reading H.264, changing the pointer's picture. |
 
 The host's own tests run its real code against this app: `packages/host/test/app.test.ts`, and
 `packages/host/scripts/screen-check.ts` with a real viewer page.
 
+## Media experiments
+
+The default remains H.264 over WebRTC. The alternative transport is a laboratory, gated behind
+`--loopback --media-lab`: both sockets bind to `127.0.0.1`, share a fresh in-process key, and
+authenticate every packet with ChaCha20-Poly1305 using direction-specific HKDF keys. A
+256-packet replay window, 1200-byte datagrams, at most eight assembling frames / 4 MiB / 800 ms,
+and bounded raw inputs keep malformed or delayed input from making queues grow indefinitely.
+There is no remote handshake, pairing integration, NAT traversal or iOS receiver for it.
+
+Frame deadlines abandon an expired prediction chain; only an IDR or a refresh based on a
+successfully decoded, acknowledged LTR may resume it. Recovery waits for a decoded ACK before
+admitting speculative deltas. Quiet probes avoid an application-limited feedback deadlock.
+The experimental controller uses delivery feedback, never the proxy's configured capacity.
+These mechanisms are implemented and tested, but the full-screen weak-network comparison
+showed longer freezes than WebRTC. They have not been adopted in the product path.
+
+Run the built helper through these tools, selecting an actual display index:
+
+```sh
+TMPDIR=/tmp node tools/loopback.mjs --screen 1 --maxFps 120 --motion --media-probe --seconds 15
+TMPDIR=/tmp node tools/network-check.mjs --screen 1 --maxFps 120 --full-screen-motion --json /tmp/rtc.json
+TMPDIR=/tmp node tools/media-lab.mjs --screen 1 --maxFps 120 --full-screen-motion --network --deadline-ms 300 --json /tmp/udp.json
+TMPDIR=/tmp node tools/codec-check.mjs --screen 1 --fps 120 --codec hevc --bitrate 5000000 --seconds 8
+```
+
+`--media-probe` carries encoder-input monotonic time in an H.264 SEI and matches it at decode,
+instead of assuming sender/receiver RTP timestamps have the same origin. It is loopback-only
+and disabled in normal use; reported time excludes capture waiting and physical presentation.
+`network-check.mjs` enables it automatically. Use `--quality` on the two network tools in a
+separate run to compare sampled, frame-matched luma grids (about 1/31 frames plus recovery
+frames, video/full-range normalized); this adds CPU work and does not measure chroma or text
+readability. `--full-screen-motion` covers the chosen screen with a repeatable scrolling
+fixture and closes it with the helper. No synthetic input or system network settings change.
+
+`--budget-keyframes` on the WebRTC diagnostic tool limits requested recovery key-frame quality
+using the previous key frame's size and 0.2 seconds of target bitrate. It is off by default;
+smaller recovery frames can still mean worse pictures. `--trial WebRTC-FrameDropper=Disabled`
+is also experimental: more decoded frames in a narrow link did not ensure fresh content.
+Compare delivered updates, age, freezes, bitrate and quality together. Conditions, binary
+hashes and results are recorded in the [architecture experiment section](../../docs/v2/screen-realtime.md).
+Remove owned temporary JSON/log/build outputs after recording the useful results.
+
 ## Command line
 
 | Mode | |
@@ -174,7 +256,10 @@ The host's own tests run its real code against this app: `packages/host/test/app
 | `--no-playout-delay` | Leaves out the field trial that sends playout-delay 0. |
 | `--no-flexfec` | Disables FlexFEC advertisement and sending for an explicit comparison. |
 | `--loopback-no-flexfec` | Makes the test receiver decline FlexFEC through codec preferences, to check an unsupported viewer's path. |
-| `--loopback-network` | Only with `--loopback`: passes its ICE candidates to `flexfec-check.mjs`, which supplies proxy candidates. The proxy exists only while that tool runs. |
+| `--loopback-network` | Only with `--loopback`: passes its ICE candidates to `flexfec-check.mjs` or `network-check.mjs`, which supplies proxy candidates. The proxy exists only while that tool runs. |
+| `--media-probe`, `--media-quality` | Loopback-only frame timing, and optional sampled luma comparison. Normal sessions do neither. |
+| `--media-lab`, `--lab-codec-only` | Loopback-only alternative media lab, or encode-only codec probe. Use the tools above; these are not host/phone modes. |
+| `--full-screen-motion`, `--budget-keyframes` | Full-display diagnostic workload; optional experimental key-frame budget, never enabled by default. |
 | `--trial <name>=<value>` | Another libwebrtc field trial (repeatable). |
 | `--rtc-log` | libwebrtc's log, as `log` messages. |
 
@@ -234,12 +319,12 @@ and takes its place; what is then said for the earlier `v` is dropped.
 
 | | Message | |
 |---|---|---|
-| → | `{t:"rtc.open", v, screen?, iceServers?, maxWidth?, fps?, codec?}` | `iceServers`: `[{urls, username?, credential?}]`, `urls` a string or a list. `maxWidth`: default 1920, at least 160; the picture is never larger than the display. `fps`: 1–60, the one rate to send at; left out, 60, and 30 while 60 is not being carried (see Tuning). `codec`: `"h264"` (default) or `"hevc"`, which this build of libwebrtc can't encode: H.264 is offered, and a `log` says so. |
+| → | `{t:"rtc.open", v, screen?, iceServers?, maxWidth?, fps?, maxFps?, codec?}` | `iceServers`: `[{urls, username?, credential?}]`, `urls` a string or list. `maxWidth`: default 1920, clamped to 160–3840 and bounded by the source display. `maxFps`: receiver ceiling, 30 / 60 / 120, default 60; automatic rate selection also considers the source display. `fps`: optional fixed 1–120 rate, overriding automatic selection (internal comparison). `codec`: `"h264"` (default) or `"hevc"`, which this build cannot encode; H.264 is offered and a log explains why. |
 | ← | `{t:"rtc.offer", v, sdp}` | The app offers: one send-only video track, and the four data channels. |
 | → | `{t:"rtc.answer", v, sdp}` | |
 | ↔ | `{t:"rtc.ice", v, candidate, sdpMid, sdpMLineIndex}` | Trickled, either way. Gathering goes on for the life of the session, so a viewer that changes network is found again. |
 | ← | `{t:"rtc.state", v, ice, connection}` | On every change of either, named as a browser names them. |
-| ← | `{t:"rtc.stats", v, …}` | Once a second: `bitrate` (bit/s), `fps` (frames encoded), `frameRate` (the rate in force: 60 or 30, or the one `rtc.open` named), `frameRateReason` (why it last changed, as the `log` said it; `null` while it hasn't), `width`, `height`, `encoder`, `hardware`, `codec`, `fmtp`, `keyFrames`, `qualityLimitation`, `targetBitrate`, `availableOutgoing`, `rttMs`, `packetsLost`, `nack`, `pli`, `encodeMs`, `sendDelayMs`, `hugeFrames`, `captureFps`, `repeated` (the last picture sent again), `gapMs` (the longest libwebrtc went without a picture, new or repeated), `heard` (messages received on each channel), `clock` (with `--clock`: the strip as captured against the time it was displayed, `{n, unreadable, min, median, max}` in ms). A number libwebrtc doesn't have is `null`. |
+| ← | `{t:"rtc.stats", v, …}` | Once a second: `bitrate` (bit/s), `fps` (frames encoded), `frameRate` (the rate in force: up to 120 / 60 / 30 subject to source and receiver limits, or the fixed `fps` requested), `frameRateReason` (why it last changed, as the `log` said it; `null` while it hasn't), `width`, `height`, `encoder`, `hardware`, `codec`, `fmtp`, `keyFrames`, `qualityLimitation`, `targetBitrate`, `availableOutgoing`, `rttMs`, `packetsLost`, `nack`, `pli`, `encodeMs`, `sendDelayMs`, `hugeFrames`, `captureFps`, `repeated` (the last picture sent again), `gapMs` (the longest libwebrtc went without a picture, new or repeated), `heard` (messages received on each channel), `clock` (with `--clock`: the strip as captured against the time it was displayed, `{n, unreadable, min, median, max}` in ms). A number libwebrtc doesn't have is `null`. |
 | ← | `{t:"rtc.error", v, message}` | The session has ended, and why. |
 | → | `{t:"rtc.close", v}` | |
 
@@ -322,7 +407,7 @@ and a stream beside an `rtc` session, run independently.
 
 | | Message | |
 |---|---|---|
-| → | `{t:"stream.open", v, socket, screen?, width?, fps?, bitrate?, ceiling?, gop?, profile?}` | `socket`: the path the app connects to and writes records on. `width`: the widest the picture may be, default 1920, at least 160. `fps`: default 30, 1–60. `bitrate`: bit/s on average, default 2 000 000, at least 50 000. `ceiling`: the most in any one second, never below `bitrate`. `gop`: seconds between key frames at most, default 1, at least 0.2. `profile`: `"baseline"` or `"high"`; left out, High from the low-latency encoder and Baseline from the usual one. |
+| → | `{t:"stream.open", v, socket, screen?, width?, fps?, bitrate?, ceiling?, gop?, profile?}` | `socket`: the path the app connects to and writes records on. `width`: the widest the picture may be, default 1920, at least 160. `fps`: default 30, 1–120 at this helper interface; the host's compatibility ladder requests only 8–20 fps. `bitrate`: bit/s on average, default 2 000 000, at least 50 000. `ceiling`: the most in any one second, never below `bitrate`. `gop`: seconds between key frames at most, default 1, at least 0.2. `profile`: `"baseline"` or `"high"`; left out, High from the low-latency encoder and Baseline from the usual one. |
 | → | `{t:"stream.set", v, width?, fps?, bitrate?, ceiling?}` | Takes effect without the stream stopping. |
 | → | `{t:"stream.key", v}` | The next frame is a key frame, and goes now. |
 | ← | `{t:"stream.started", v, width, height, fps, generation, mode, profile, hardware}` | After the first frame of each generation is written. `mode`: `"low-latency"` or `"real-time"`, the rate control the encoder gave. |
@@ -359,44 +444,54 @@ again once.
 
 ## Tuning
 
-`Sources/LinkShell/Tuning.swift` has the numbers that shape the picture, each with its reason:
-the defaults (1920 wide; 60 frames for the track, 30 for the stream), the track's ceiling (8 Mbit/s for 1920×1080 at 30, in
-proportion to the pixels, half as much again above 30, between 2 and 16) and where its
-bandwidth estimate starts (2 Mbit/s), the degradation preference (the picture gets smaller, not
-jerkier), the field trials (playout-delay 0), how much more than the frame rate the capture is
-asked for (1.1×), how often a still screen's picture is sent again (every 0.1 s for a second,
-then every 0.5 s), the pointer's rates (60 and 5 a second), and everything about the stream's
-pacing and key frames.
-
-**The track's frame rate** (`FrameRate.swift`, a rule with no clock and no network in it,
-tested in `Tests/`; what a change takes is `ScreenSession.change`). A session that `rtc.open`
-gave no `fps` starts at 60 and is judged once a second, while its connection is up, on the
-numbers `rtc.stats` says:
-
-- **Down to 30** when, for 3 seconds running, the picture sent has under 0.9 of the captured
-  picture's pixels (libwebrtc made it smaller: too little bandwidth, or too little processor),
-  or the screen gives at least 0.8 of the rate and under 0.75 of what it gives is encoded.
-  Packets lost, a low estimate while the picture is whole, and the encoder's time over a frame
-  are not reasons.
-- **Back to 60** when, for 10 seconds running at 30, the picture is whole, libwebrtc says
-  nothing limits it, frames are not being lost and the bandwidth estimate is at least 0.4 of
-  the ceiling at 60 (4.8 Mbit/s for 1920×1080). The 10 seconds double, up to 160, each time 60
-  is lost again within 30 seconds of being tried.
-- Nothing is concluded in a connection's first 4 seconds, nor in the 4 after each change.
-
-A change is made with the capture running and nothing negotiated: ScreenCaptureKit's frame
-interval, the sender's `maxFramerate` and its ceiling. Going down, a picture libwebrtc had
-shrunk is whole again from the next frame (a key frame) and stays so for 4 seconds, after which
-libwebrtc may shrink it again; going up, the first second may send half of what 30 was sending.
-Each change is a `log`: `frame rate 60 → 30: the picture was being shrunk for 3 s (bandwidth)`,
-`frame rate 30 → 60: nothing limited the picture for 10 s, and the network is estimated at
-9.8 Mbit/s`.
-
-Elsewhere: the quantizers at which libwebrtc resizes the picture (the encoders say 28 and 39,
-`LowLatencyEncoder.swift`; libwebrtc goes by its own for H.264, 24 and 37, as its log says), the sizing of key frames (`VideoCompressor.swift`), the 0.3 s after an event in
-which the pointer is the viewer's own and the ±2000 on a scroll (`Control.swift`), the pause
-and the 2.5 s around the input source (`Keys.swift`), and the pointer picture's scale
-(`PointerShape.swift`).
+`Sources/LinkShell/Tuning.swift` defines the shared limits. The video defaults to a maximum
+width of 1920, with a 3840 cap; an ordinary receiver gets a 60 fps ceiling, while `maxFps`
+can request 30 / 60 / 120 without disabling adaptation. The source display bounds the starting
+rate. The helper's separate byte-stream interface defaults to 30 fps, but `screen-pacer.ts`
+selects its own 8–20 fps ladder (1280-wide / 12 fps / 900 kbit/s target when `q=low`). These
+are different paths; selecting a video width does not override the compatibility ladder.
+
+The video bitrate ceiling starts from 8 Mbit/s for 1920×1080 at 30 fps. Pixel counts below
+1080p scale proportionally; above it they scale by their square root. The frame factor is 1
+through 30 fps, 1.5 through 60, and `1.5 × fps / 60` above 60. The result is clamped to
+2–30 Mbit/s: 1080p30/60/120 ceilings are 8/12/24 Mbit/s. Actual bitrate follows WebRTC's
+estimate, initialized at 2 Mbit/s, and the content.
+
+`maintainFramerate` permits WebRTC to reduce resolution under resource pressure. Separately,
+`ScreenEncoderFactory.scaling` returns no QP thresholds, suppressing QP-only quality scaling
+that unnecessarily shrank text. `FrameRate.swift` judges the connection once a second and
+reduces rate when the full picture/rate cannot be carried:
+
+- Start at the receiver/source ceiling, normally 120 or 60, with lower stages at 60 and 30
+  where applicable. A source below these rates bounds the starting rate further. Explicit
+  `fps` is a fixed internal override.
+- Reduce after 3 consecutive troubled seconds: sent pixel share below 0.9, encoded output
+  below 0.75 of the requested active input, or average packet send wait above 50 ms.
+  Output/queue triggers require capture to supply at least 0.8 of the current rate. A low
+  bandwidth estimate, packet loss alone and time per encoded frame are not independent triggers.
+- Probe the next stage after 10 good seconds: full picture, no quality limitation,
+  sufficient encoded output and average packet send wait at most 10 ms. Missing delivery
+  statistics don't count as health. No fraction of the maximum bitrate is reserved for a
+  probe: low-complexity desktop content can sustain a higher rate on less bandwidth.
+  If a raised rate fails again within 30 seconds, the next wait doubles, up to 160 seconds.
+- The first 4 seconds and the 4 after a change are settling periods.
+
+Rate changes update ScreenCaptureKit, sender parameters and the bitrate ceiling without
+renegotiating the media connection. A decrease requests restoration of the captured dimensions;
+the sender can adapt them again later. An increase eases the first second's bitrate using
+half the previous observed bitrate, with a 1 Mbit/s floor. Neither change guarantees physical
+presentation cadence.
+
+The queue thresholds identify sustained pressure; they do not cap the age of individual
+packets already inside WebRTC. A quiet screen's occasional large frame does not lower the
+frame-rate ceiling. `--rtc-log` also logs the bitrate and input-rate estimates supplied to
+the custom encoder, to diagnose overshoot when motion resumes or the rate changes.
+
+Other settings include capture rate slack (1.1×), capture surface queue depth (5), static-frame
+repeats (0.1 s for a second, then 0.5 s), cursor position/shape polling (60/5 Hz), playout-delay
+0/0 and FlexFEC field trials. Surface pool sizes are not measured queued-frame counts.
+`VideoCompressor.swift` controls keyframe sizing, `Control.swift` input state and scroll bounds,
+and `Keys.swift` the temporary input-source switch for text.
 
 ## Measuring
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@linkshell/mac",
-  "version": "0.1.4",
+  "version": "0.1.6",
   "description": "LinkShell.app: the Mac side of the screen — capture, encoding, WebRTC and input, under the app's own permissions.",
   "license": "MIT",
   "repository": {
@@ -13,7 +13,8 @@
     "darwin"
   ],
   "cpu": [
-    "arm64"
+    "arm64",
+    "x64"
   ],
   "files": [
     "build/LinkShell.app.tar.gz",