@desklink/react-native 0.1.1 → 0.3.0

package/README.md

@@ -1,6 +1,16 @@
-# @desklink/react-native
+<h1 align="center">@desklink/react-native</h1>
 
-Show a live desktop in a React Native app, and drive it from the phone.
+<p align="center">
+  <a href="https://www.npmjs.com/package/@desklink/react-native"><img alt="npm" src="https://img.shields.io/npm/v/@desklink/react-native?style=flat&label=npm" /></a>
+  <a href="https://github.com/umeranjum17/desklink/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/umeranjum17/desklink/ci.yml?style=flat&branch=main" /></a>
+  <a href="LICENSE"><img alt="Apache 2.0" src="https://img.shields.io/badge/license-Apache--2.0-666?style=flat" /></a>
+  <img alt="Android, iOS and web" src="https://img.shields.io/badge/Android%20%7C%20iOS%20%7C%20web-111?style=flat" />
+</p>
+
+<p align="center">
+  <strong>Show a live desktop in a React Native app, and drive it from the phone.</strong><br/>
+  It renders a desktop that <code>@desklink/host</code> serves on Android, iOS and the web, with touch as a desktop pointer the phone can see, sticky modifier keys, and a session that recovers from a dropped network on its own.
+</p>
 
 The package is a **view and a session**, not a screen: it renders the picture and
 turns touch, the keyboard and the clipboard into that session's input. Where it
@@ -9,6 +19,10 @@ the application.
 
 ## Install
 
+The npm badge at the top of this page tracks the registry version, and npm
+verifies package integrity on install. See the [changelog](../../CHANGELOG.md)
+for release changes. Supported platforms: Android, iOS and web.
+
 The package contains native code and is not an Expo Go package. Inside this
 repository it resolves through the workspace; anywhere else, install it from npm:
 
@@ -17,10 +31,29 @@ npx expo install @desklink/react-native      # or yarn add, then prebuild/rebuil
 ```
 
 Android uses an Expo native module; iOS uses the app's existing
-`react-native-webrtc` peer connection and `RTCView` from JavaScript. Both need a
-dev client or native build containing that binding. Web uses the browser's WebRTC.
+`react-native-webrtc` peer connection and `RTCView` from JavaScript, plus a small
+Expo module of its own for the hardware keyboard and an iPad pointer. Both need a
+dev client or native build containing that binding ([development builds](https://docs.expo.dev/develop/development-builds/introduction/); [Expo Go](https://docs.expo.dev/workflow/expo-go/) cannot load them). Web uses the browser's WebRTC.
 No second WebRTC binary is linked. The Android module has its own hardware-first
-decoder factory; iOS uses the decoder shipped by `react-native-webrtc`.
+decoder factory; iOS uses the decoder shipped by `react-native-webrtc` and asks
+the host for H.264, which that decoder runs in hardware.
+
+On iOS the WebRTC binding also needs its config plugin
+(`@config-plugins/react-native-webrtc` — 14.x for Expo SDK 55, 15.x for
+Expo SDK 56 and later):
+
+```json
+{ "expo": { "plugins": ["@config-plugins/react-native-webrtc", "@desklink/react-native"] } }
+```
+
+On iPad, a trackpad or mouse hovers and scrolls the desktop as it is. For its
+presses to carry their buttons — a click-drag that selects, a secondary click
+that is a right click — the package's entry in the plugins list above sets
+[`UIApplicationSupportsIndirectInputEvents`](https://developer.apple.com/documentation/bundleresources/information-property-list/uiapplicationsupportsindirectinputevents).
+
+That key changes how every screen of the app receives trackpad input, so the
+package never sets it on its own; without the plugin nothing about the app
+changes, and trackpad presses reach the desktop as touches.
 
 ## Use
 
@@ -56,21 +89,52 @@ import { desktopAvailable } from '@desklink/react-native/availability';
 {desktopAvailable && <ComputerAction onPress={openDesktop} />}
 ```
 
+### Frame rate and encode size
+
+The engine streams at up to 60 fps unless `session.maxFps` names a lower cap —
+omit it for the smooth path. A still desktop costs nothing at 60, because the
+engine only produces frames on change.
+
+Keep the encoded picture inside a 1920×1080 box:
+
+```tsx
+session: { permissions: CONTROL_PERMISSIONS, maxWidth: 1920, maxHeight: 1080 },
+```
+
+The engine never upscales, so a smaller desktop is sent at its own size and a
+larger one is scaled to fit. Above that box the encoder cannot sustain 50+ fps
+under full-screen motion.
+
+On iOS, keep the box at or below the device's own pixels and `maxFps` at 30:
+the iOS receiver asks the engine for H.264, which iOS decodes in hardware
+(VideoToolbox), but a host without an H.264 encoder still sends VP9, which
+iOS decodes in software — so a 1440p or 4K picture on such a host costs CPU,
+heat and battery.
+
 ### What the package guarantees
 
-- **VP9 decoding.** Android builds a session-scoped hardware-first decoder
+- **Hardware decoding where the platform has it.** Android and web decode the
+  host's default VP9; Android builds a session-scoped hardware-first decoder
   factory with software fallback. iOS uses the app's `react-native-webrtc`
-  decoder; neither platform links a second WebRTC stack.
+  decoder, which decodes H.264 through VideoToolbox but VP9 only in software,
+  so the iOS receiver asks for H.264 by putting it first in the offer it
+  applies; a host with an H.264 encoder then sends H.264, and one without sends
+  VP9. Neither platform links a second WebRTC stack.
 - **A sharp, zoomable picture.** The desktop fits the view by default; a pinch
   zooms up to 2.5 view pixels per desktop pixel and one finger moves around the
   zoomed desktop. On Android the decoded frame is copied once into the view's
   own texture and drawn with a multi-tap filter when it is shown smaller than
   its size, so a fitted 4K desktop does not alias and a pinch redraws at once.
-  iOS uses `RTCView`'s native renderer. `fitToView()` shows the whole desktop again.
+  iOS uses `RTCView`'s native renderer. That renderer draws a frame only when
+  its timestamp is new, and a receiver honouring the engine's zero playout delay
+  stamps every frame alike, so the iOS peer declines that hint and keeps the
+  receiver's usual jitter buffer. `fitToView()` shows the whole desktop again.
 - **A pointer a phone can see.** Once a touch has sent the desktop's pointer
   somewhere, the view draws it there at a readable size, over a picture whose
   own cursor is a few pixels tall or not captured at all. Android hides the
-  extra mark when a mouse is attached.
+  extra mark when a mouse is attached, iOS when an iPad trackpad or mouse
+  drives it, and the `device` profile draws no mark on any platform — the
+  screen being touched shows the finger itself.
 - **The picture above the keyboard.** While the phone's keyboard is up, the
   picture sits above it and above the room the app keeps for its own controls
   (`keyboardClearance`). Android follows the keyboard animation and keeps the
@@ -88,14 +152,65 @@ import { desktopAvailable } from '@desklink/react-native/availability';
   under the finger while the picture catches up; two fingers scroll the desktop
   under them, or pinch; a quick two-finger tap is a right click too. Scrolling
   is fractional wheel steps, smooth where the desktop supports high-resolution
-  wheels. Android coalesces pending pointer moves under congestion.
+  wheels. Android coalesces pending pointer moves under congestion. The
+  `gestures` prop reshapes one finger for what the picture shows, and can be
+  changed mid-session: `browser` scrolls the page under one finger (a tap
+  still clicks and a hold still right-clicks), `device` presses the screen —
+  the finger lands with the button down, drags with it held and lifts to
+  release, so a tap clicks and a hold holds with no long-press right click,
+  and no drawn pointer — while the default `desktop` is the trackpad above.
+  Two fingers scroll and pinch the same under every profile.
+- **A hardware keyboard and a trackpad.** A hardware keyboard's keys reach the
+  desktop with their down and up and what is held: arrows, Esc, Tab, Home/End,
+  Page Up/Down, Delete and the modifiers, with Control and Command
+  chords as the key's own letter (iOS sends Command as the desktop's Meta);
+  other keys type their text. On iOS the view holds the keyboard while control
+  is enabled and gives it to the on-screen keyboard while the app shows it;
+  shortcuts the system keeps, such as Command-Tab and the Globe key, stay with
+  the system. An iPad pointer hovers the desktop's pointer, scrolls under it,
+  and — with the config plugin — presses, drags and right-clicks with its own
+  buttons.
 - **The keys a phone lacks.** `modifiers`, `tapModifier` and `pressKey` give
   sticky Ctrl and Shift: tap arms one for the next key, tap again locks it.
   While one is armed, the next key or character the phone's keyboard types is
   sent as that key's chord, so Ctrl then v is Ctrl+V. The app draws the keys.
-- **Readiness is a rendered frame.** Android marks the first draw; iOS uses
-  `RTCView`'s native video-dimensions callback, which fires after a decoded
-  picture arrives. Neither marks a track or ICE connection as live.
+- **Readiness is a rendered frame.** Android marks the first draw; iOS marks
+  the first `RTCView` video-dimensions callback and ignores later ones, so a
+  resize or remount during a reconnect does not read as live. Neither marks
+  a track or ICE connection as live.
+- **Connection states, as they happen.** The session status is `idle`,
+  `opening`, `connecting`, `live`, `reconnecting`, `ended` or `failed`. The
+  peer's ICE state and the engine's `session.state` both feed it: a
+  `disconnected` path reads as `reconnecting` within about a second of the
+  drop — never a frozen picture that still says live — and a recovered path
+  reads as `live` again once ICE is `connected`. A path that reports
+  `failed` reopens the session with fresh authority instead of staying stuck.
+  Every event carries its engine session id where the carrier preserves it,
+  and the session ignores anything naming a session it no longer holds, so a
+  previous generation's queued offer or revocation cannot corrupt or kill a
+  healthy recovery.
+  Detection is two-layered: ICE and engine `session.state` events, plus the
+  engine's twice-a-second control-channel heartbeat (`{"kind":"ping"}`) — a
+  path that stops heartbeating reads as stalled within ~1.5 s even while ICE
+  consent timers are still making up their minds, and resumed heartbeats read
+  as recovered. Clients on engines that predate the heartbeat simply never
+  arm that watchdog.
+- **ICE restarts before reopens.** When the path drops after frames were
+  shown, the session asks the engine (`session.restart_ice`) to re-offer on
+  the same peer connection: the fresh offer arrives as the usual
+  `session.description` event and is answered as usual, all without reopening
+  the session. The first restart waits ~2 s for blips that heal alone;
+  retries back off across ~60 s, so a 30–60 s outage still recovers when the
+  network returns. When the restart schedule is spent without the peer
+  reporting `failed`, the session reopens instead of sitting in
+  `reconnecting`. An engine revocation with code `transport` (the path was
+  lost) also reopens the session; any other revocation code, or none, ends
+  it, as does a deliberate close. A return to the foreground after ~30 s away
+  reopens at once rather than spending restarts on a lapsed path. Those
+  reopens back off across ~60 s too instead of spending one attempt while the
+  network is still down. Renegotiation transients from a restart the session
+  asked for itself are expected, not a new outage: they neither flip the
+  status nor trigger another restart for a short grace window.
 - **Control starts off.** `setInputEnabled(true)` enables pointer, keyboard and
   clipboard input; `setInputEnabled(false)` blocks new input, releases held keys
   and buttons, and resets pending gestures. The app decides when to enable it
@@ -109,7 +224,10 @@ import { desktopAvailable } from '@desklink/react-native/availability';
 ### `Signaling`
 
 The one thing the app must supply. It carries the engine's local protocol over
-whatever authenticated channel the app already has:
+whatever authenticated channel the app already has.
+
+See the [BYOKit integration guidance](../../README.md#why-desklink-exists)
+for the application's pairing, identity and signaling transport.
 
 ```ts
 interface Signaling {
@@ -119,9 +237,72 @@ interface Signaling {
 ```
 
 `method` is one of `session.open`, `session.description`, `session.candidate`,
-`session.close`; `SessionEvent` is the engine's offer, candidates, state and
-revocation. Pixels and input do **not** use this channel — they are the
-desktop's own WebRTC session.
+`session.restart_ice`, `session.close`; `SessionEvent` is the engine's offer,
+candidates, state and revocation. Pixels and input do **not** use this
+channel — they are the desktop's own WebRTC session.
+
+## Shipping on iOS
+
+What the consuming app — not this package — must get right before review.
+
+- **Local network.** A direct LAN session connects to host ICE candidates,
+  which prompts for local-network access, so the app needs
+  `NSLocalNetworkUsageDescription`. The first connection can be denied before
+  the user answers; the session's restart/reopen path recovers from that.
+  `NSBonjourServices` is only needed if the app adds its own Bonjour
+  discovery — desklink discovers no hosts. Relay-only sessions never prompt.
+  For app-supplied routes, see the
+  [BYOKit integration guidance](../../README.md#why-desklink-exists).
+  See [TN3179](https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy).
+- **Export compliance.** The viewer's DTLS-SRTP comes from libwebrtc bundled
+  with `react-native-webrtc`, not from the OS, so the app must check the
+  exemption criteria itself rather than assume
+  `ITSAppUsesNonExemptEncryption` is `NO` — standard-protocol encryption is
+  commonly exempt but can still need a yearly self-classification report.
+  See [complying with encryption export regulations](https://developer.apple.com/documentation/security/complying-with-encryption-export-regulations).
+- **Review.** A viewer that mirrors the whole host device generically falls
+  outside the remote-desktop conditions; one that mirrors only specific
+  software must connect LAN-only to a user-owned host. Which case applies is
+  the app's product shape to decide. Declare no `audio` or `voip` background
+  mode to keep the stream alive — review rejects that. See the
+  [App Store review guidelines](https://developer.apple.com/app-store/review/guidelines/)
+  (§4.2.7, §2.5.4).
+- **Backgrounding.** There is no compliant background mode for a desktop
+  stream: an app away longer than ~30 s comes back to a closed session, and
+  the session reopens on return to the foreground rather than resuming. Keep
+  disabling input on background; the engine releases held keys within
+  seconds of the path dying, so nothing stays stuck.
+
+## iOS simulator proof
+
+`example/` is the smallest app that shows a desktop: a native (non-Expo Go)
+build with `react-native-webrtc`, connecting to `desklink-host bridge` with the
+URL it is launched with. `test/ios-flow.mjs` builds it for the iOS simulator on
+a Mac over ssh, serves a private Xvfb desktop from this machine through the
+bridge, and checks that the first frame is presented, that the answer puts
+H.264 first with NACK when the host offers it (VP9 otherwise; both descriptions
+are written out), that a tap clicks the host, that
+the picture keeps following the desktop afterwards, and that hardware keys
+pressed on the simulator reach the desktop. With `DESKLINK_IOS_IPAD=1` it runs on
+an iPad simulator and also drives a trackpad through XCUITest
+(`test/ios-pointer/`): hover, a click-drag that selects text, a right click and
+a scroll:
+
+```sh
+npm run build && cargo build --manifest-path packages/desktop-host/engine/Cargo.toml
+DESKLINK_IOS_MAC=user@mac npm run test:ios --workspace @desklink/react-native
+```
+
+The Mac needs Xcode with an iOS simulator runtime, CocoaPods, node and
+[`axe`](https://github.com/cameroncooke/AXe) on its login `PATH`; everything
+the run builds or caches stays in `~/desklink-ios` there. When the selected
+Xcode has no simulator platform, point `DESKLINK_IOS_XCODE` at one that does;
+`DESKLINK_IOS_RUNTIME` picks the iOS version (for example `18.6`); the
+script's header lists the other settings. To try the example by hand against any bridge:
+
+```sh
+xcrun simctl launch <device> dev.desklink.example -desklinkUrl 'ws://HOST:PORT/desktop?token=TOKEN'
+```
 
 ## Licence
 

package/android/src/main/java/expo/modules/desklink/DesklinkModule.kt

@@ -126,6 +126,10 @@ class DesklinkModule : Module() {
         view.setKeyboardClearance(clearance ?: 0f)
       }
 
+      Prop("gestures") { view: DesktopView, gestures: String? ->
+        view.setGestures(gestures ?: "desktop")
+      }
+
       OnViewDestroys { view: DesktopView -> view.release() }
 
       OnViewDidUpdateProps { view: DesktopView ->

package/android/src/main/java/expo/modules/desklink/DesktopView.kt

@@ -66,7 +66,9 @@ private const val MIN_WHEEL_STEP = 0.05f
  *  - press and hold: a right click on release, or drag after it to hold the left
  *    button (select text, move a window);
  *  - one finger: move around a zoomed-in desktop; on the whole desktop,
- *    move its pointer, which follows the finger without pressing a button;
+ *    move its pointer, which follows the finger without pressing a button —
+ *    unless `gestures` is `browser`, where it scrolls the page, or `device`,
+ *    where it lands with the button down and drags with it held;
  *  - two fingers: scroll the desktop under them, or pinch to zoom (and move) the
  *    picture; a quick two-finger tap is a right click.
  */
@@ -81,6 +83,23 @@ class DesktopView(context: Context, appContext: AppContext) : ExpoView(context,
   private var surfaceWidth = 0
   private var surfaceHeight = 0
 
+  /** Which touch meaning this surface uses; a browser page and a device screen answer one finger differently. */
+  var gestures = "desktop"
+    private set
+
+  fun setGestures(value: String) {
+    if (value == gestures) return
+    // A held drag belongs to the old meaning: let go rather than release the
+    // button somewhere the finger never meant.
+    if (gesture == Gesture.DRAG) session?.sendCancel()
+    cancelGesture()
+    gestures = value
+    if (value == "device" && pointerAt != null) {
+      pointerAt = null
+      placePointer()
+    }
+  }
+
   // The picture's placement: surface pixels per desktop pixel, and where the
   // desktop's top-left corner sits in this view. `fitted` keeps a picture that
   // was showing the whole desktop showing it when the view changes size.
@@ -449,7 +468,8 @@ class DesktopView(context: Context, appContext: AppContext) : ExpoView(context,
   /** Send the desktop's pointer somewhere, and show it there. */
   private fun pointerTo(active: DesktopSession, phase: String, at: Pair<Int, Int>, withButton: Boolean = false) {
     active.sendPointer(phase, at.first, at.second, withButton)
-    if (!mouseInput) showPointer(at)
+    // A device screen shows the touch itself; a mark would be a second finger that lies.
+    if (!mouseInput && gestures != "device") showPointer(at)
   }
 
   private fun showPointer(at: Pair<Int, Int>) {
@@ -519,6 +539,25 @@ class DesktopView(context: Context, appContext: AppContext) : ExpoView(context,
     }
     when (event.actionMasked) {
       MotionEvent.ACTION_DOWN -> {
+        // A device screen is pressed, not pointed at: the finger lands with the
+        // button down, drags with it held, and lifts to release. A tap is the
+        // same down and up with no move, so it clicks for free, and a hold
+        // holds for free — there is no long press to arm.
+        if (gestures == "device") {
+          downX = event.x
+          downY = event.y
+          lastX = event.x
+          lastY = event.y
+          val at = point(event.x, event.y)
+          if (at == null) {
+            gesture = Gesture.LETTERBOX
+          } else {
+            gesture = Gesture.DRAG
+            lastTapPoint = null
+            pointerTo(active, "down", at)
+          }
+          return true
+        }
         downOnPicture = point(event.x, event.y) != null
         gesture = if (fitted && !downOnPicture) Gesture.LETTERBOX else Gesture.PENDING
         downX = event.x
@@ -573,6 +612,15 @@ class DesktopView(context: Context, appContext: AppContext) : ExpoView(context,
         Gesture.PENDING -> if (hypot(event.x - downX, event.y - downY) > touchSlop) {
           removeCallbacks(longPress)
           lastTapPoint = null
+          // A page scrolls under one finger; the desktop's pointer rides along
+          // so the scroll lands where the finger is.
+          if (gestures == "browser") {
+            gesture = Gesture.SCROLL
+            lastX = event.x
+            lastY = event.y
+            point(event.x, event.y)?.let { pointerTo(active, "move", it) }
+            return true
+          }
           // The whole desktop has nowhere to move to, so the finger moves the
           // desktop's pointer instead, without a button.
           gesture = if (!fitted) Gesture.PAN else if (downOnPicture) Gesture.HOVER else Gesture.LETTERBOX
@@ -599,6 +647,15 @@ class DesktopView(context: Context, appContext: AppContext) : ExpoView(context,
         Gesture.DRAG -> dragTo(active, event.x, event.y)
 
         Gesture.TWO, Gesture.PINCH, Gesture.SCROLL -> if (event.pointerCount >= 2) twoFingers(active, event)
+        else if (gesture == Gesture.SCROLL && gestures == "browser") {
+          // One finger dragging a page: content follows it, in pixel deltas
+          // turned to detents like the two-finger scroll.
+          wheelX -= (event.x - lastX) / scale / PIXELS_PER_DETENT
+          wheelY -= (event.y - lastY) / scale / PIXELS_PER_DETENT
+          lastX = event.x
+          lastY = event.y
+          flushWheel(active, force = false)
+        }
 
         else -> {}
       }
@@ -618,6 +675,7 @@ class DesktopView(context: Context, appContext: AppContext) : ExpoView(context,
         removeCallbacks(longPress)
         when (gesture) {
           Gesture.PENDING -> tap(active, event.x, event.y)
+          Gesture.SCROLL -> flushWheel(active, force = true)
           Gesture.HOVER -> hoverTo(active, event.x, event.y)
           Gesture.ARMED -> point(downX, downY)?.let { rightClick(active, it) }
           Gesture.DRAG -> endDrag(active, event.x, event.y)
@@ -669,7 +727,7 @@ class DesktopView(context: Context, appContext: AppContext) : ExpoView(context,
 
   private fun rightClick(active: DesktopSession, at: Pair<Int, Int>) {
     active.sendRightClick(at.first, at.second)
-    if (!mouseInput) showPointer(at)
+    if (!mouseInput && gestures != "device") showPointer(at)
   }
 
   private fun twoFingers(active: DesktopSession, event: MotionEvent) {

package/app.plugin.js

@@ -0,0 +1,16 @@
+/**
+ * Opt-in config plugin: an iPad trackpad's or mouse's presses reach the
+ * desktop with the button that made them — a drag selects, a secondary click
+ * is a right click. It sets `UIApplicationSupportsIndirectInputEvents`, which
+ * changes how every screen of the app receives that input, so the package
+ * never sets it on its own. Hover, scroll and the hardware keyboard work
+ * without it.
+ *
+ *   "plugins": ["@desklink/react-native"]
+ */
+module.exports = function withDesklinkPointer(config) {
+    return {
+        ...config,
+        ios: { ...config.ios, infoPlist: { ...config.ios?.infoPlist, UIApplicationSupportsIndirectInputEvents: true } },
+    };
+};

package/expo-module.config.json

@@ -1,6 +1,9 @@
 {
-  "platforms": ["android"],
+  "platforms": ["android", "ios"],
   "android": {
     "modules": ["expo.modules.desklink.DesklinkModule"]
+  },
+  "ios": {
+    "modules": ["DesklinkInputModule"]
   }
 }

package/ios/DesklinkInput.podspec

@@ -0,0 +1,22 @@
+require 'json'
+
+package = JSON.parse(File.read(File.join(__dir__, '..', 'package.json')))
+
+Pod::Spec.new do |s|
+  s.name           = 'DesklinkInput'
+  s.version        = package['version']
+  s.summary        = package['description']
+  s.description    = package['description']
+  s.license        = package['license']
+  s.author         = 'desklink'
+  s.homepage       = 'https://github.com/umeranjum17/desklink'
+  s.platforms      = { :ios => '15.1' }
+  s.swift_version  = '5.9'
+  s.source         = { git: 'https://github.com/umeranjum17/desklink.git' }
+  s.static_framework = true
+
+  s.dependency 'ExpoModulesCore'
+
+  s.pod_target_xcconfig = { 'DEFINES_MODULE' => 'YES' }
+  s.source_files = '*.swift'
+end

package/ios/DesklinkInputModule.swift

@@ -0,0 +1,20 @@
+import ExpoModulesCore
+
+/**
+ * The iOS view's hardware input: the keys and the pointer React Native's touch
+ * system cannot see. The session, the picture and touch gestures stay in
+ * JavaScript; this module only reports what UIKit delivers.
+ */
+public final class DesklinkInputModule: Module {
+  public func definition() -> ModuleDefinition {
+    Name("DesklinkInput")
+
+    View(DesklinkInputView.self) {
+      Events("onKey", "onPointer", "onWheel")
+
+      Prop("enabled") { (view: DesklinkInputView, enabled: Bool) in
+        view.enabled = enabled
+      }
+    }
+  }
+}