appium-ios-remotexpc 0.0.1

package/README.md

@@ -0,0 +1,178 @@
+# appium-ios-remotexpc
+
+A Node.js library for interacting with iOS devices
+through Appium using remote XPC services.
+This library enables communication with iOS devices
+through various services like system logs and network tunneling.
+
+## Overview
+
+This library provides functionality for:
+
+- Remote XPC (Cross Process Communication) with iOS devices
+- Lockdown communication
+- USB device multiplexing (usbmux)
+- Property list (plist) handling
+- IPv6 tunneling services to iOS devices using TUN/TAP interfaces
+- System log access
+
+## Installation
+
+```bash
+npm install appium-ios-remotexpc
+```
+
+## Requirements
+
+- Node.js 16 or later
+- iOS device for testing
+- Proper device pairing and trust setup
+- Root/sudo privileges for tunnel creation (TUN/TAP interface requires elevated permissions)
+
+## Features
+
+- **Plist Handling**: Encode, decode, parse, and create property lists for iOS device communication.
+- **USB Device Communication**: Connect to iOS devices over USB using the usbmux protocol.
+- **Remote XPC**: Establish Remote XPC connections with iOS devices.
+- **Service Architecture**: Connect to various iOS services:
+    - System Log Service: Access device logs
+    - Tunnel Service: Network tunneling to/from iOS devices
+    - Diagnostic Service: Device diagnostics
+- **Pair Record Management**: Read and write device pairing records.
+- **Packet Streaming**: Stream packets between host and device for service communication.
+
+## Architecture Flow
+
+The following diagram illustrates the high-level flow of how the tunnel is created:
+
+<div align="center">
+  <img src="assets/images/ios-arch.png" alt="iOS Architecture" width="70%">
+</div>
+
+### Role of TUN/TAP
+
+The `appium-ios-tuntap (previously tuntap-bridge)` module plays a crucial role in establishing network connectivity:
+
+1. **TLS Socket Input**: Receives the secure TLS socket connection from CoreDeviceProxy
+2. **Virtual Network Interface**: Creates a TUN/TAP virtual network interface on the host system
+3. **IPv6 Tunnel**: Establishes an IPv6 tunnel between the host and iOS device
+4. **Packet Routing**: Routes network packets between the virtual interface and the iOS device
+5. **Service Access**: Enables access to iOS shim services through the tunnel
+
+**Technical Details:**
+- **Platform Support**: Works on both macOS and Linux
+- **IPv6 Support**: Creates IPv6 tunnels for modern iOS communication
+- **Packet Handling**: Manages packet routing between virtual interface and device
+- **Automatic Cleanup**: Properly closes tunnels and cleans up interfaces
+
+**Security Considerations:**
+- Requires root/sudo access for TUN/TAP interface creation
+- Uses TLS for secure communication with iOS devices
+
+## Usage
+
+### Creating a Tunnel (Low-level approach)
+
+```typescript
+import { createLockdownServiceByUDID, startCoreDeviceProxy, TunnelManager } from 'appium-ios-remotexpc';
+
+// Create lockdown service
+const { lockdownService, device } = await createLockdownServiceByUDID(udid);
+
+// Start CoreDeviceProxy
+const { socket } = await startCoreDeviceProxy(
+  lockdownService,
+  device.DeviceID,
+  device.Properties.SerialNumber,
+  { rejectUnauthorized: false }
+);
+
+// Create tunnel using tuntap
+const tunnel = await TunnelManager.getTunnel(socket);
+console.log(`Tunnel created at ${tunnel.Address} with RSD port ${tunnel.RsdPort}`);
+
+// Create RemoteXPC connection
+const remoteXPC = await TunnelManager.createRemoteXPCConnection(
+  tunnel.Address,
+  tunnel.RsdPort
+);
+
+// Access services
+const services = remoteXPC.getServices();
+```
+
+## Development
+
+### Setup
+
+```bash
+# Clone the repository
+git clone https://github.com/yourusername/appium-ios-remotexpc.git
+cd appium-ios-remotexpc
+
+# Install dependencies
+npm install
+
+# Build the project
+npm run build
+```
+
+### Continuous Integration
+
+This project uses GitHub Actions for continuous integration and Dependabot for dependency management:
+
+- **Lint and Build**: Automatically runs linting and builds the project on Node.js LTS.
+- **Format Check**: Ensures code formatting adheres to project standards
+- **Test Validation**: Validates that test files compile correctly (actual tests require physical devices)
+- **Dependabot**: Automatically creates PRs for dependency updates weekly
+
+All pull requests must pass these checks before merging. The workflows are defined in the `.github/workflows` directory.
+
+### Scripts
+
+- `npm run build` - Clean and build the project
+- `npm run lint` - Run ESLint
+- `npm run format` - Run prettier
+- `npm run lint:fix` - Run ESLint with auto-fix
+- `npm test` - Run tests (requires sudo privileges for tunneling)
+- `npm run test:tunnel-creation` - Create tunnels for testing (requires sudo)
+
+## Project Structure
+
+- `/src` - Source code
+  - `/lib` - Core libraries
+    - `/lockdown` - Device lockdown protocol
+    - `/pair-record` - Pairing record handling
+    - `/plist` - Property list processing
+    - `/remote-xpc` - XPC connection handling
+    - `/tunnel` - Tunneling implementation with tuntap integration
+    - `/usbmux` - USB multiplexing protocol
+  - `/services` - Service implementations
+    - `/ios`
+      - `/diagnostic-service` - Device diagnostics
+      - `/syslog-service` - System log access
+      - `/tunnel-service` - Network tunneling
+
+## Testing
+
+```bash
+# Run all tests
+npm test
+```
+
+Note: Integration tests require:
+- Physical iOS devices connected
+- Sudo privileges for tunnel creation
+- Device trust established
+
+## License
+
+Apache-2.0
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## Notes
+
+This project is under active development. APIs may change without notice.

package/eslint.config.js

@@ -0,0 +1,45 @@
+import appiumConfig from '@appium/eslint-config-appium-ts';
+import unicorn from 'eslint-plugin-unicorn';
+
+export default [
+  ...appiumConfig,
+  {
+    files: ['**/*.ts'],
+    linterOptions: {
+      reportUnusedDisableDirectives: 'off',
+    },
+    plugins: {
+      unicorn: unicorn,
+    },
+    rules: {
+      quotes: ['error', 'single'],
+      semi: ['error', 'always'],
+      '@typescript-eslint/no-explicit-any': 'off',
+      '@typescript-eslint/member-ordering': [
+        'error',
+        {
+          default: [
+            // ─── PUBLIC METHODS ─────────────────────────────────────────────────────
+            'public-static-method',
+            'public-instance-method',
+
+            // ─── PROTECTED METHODS ──────────────────────────────────────────────────
+            'protected-static-method',
+            'protected-instance-method',
+
+            // ─── PRIVATE METHODS ────────────────────────────────────────────────────
+            'private-static-method',
+            'private-instance-method',
+          ],
+        },
+      ],
+      'unicorn/filename-case': [
+        'error',
+        {
+          case: 'kebabCase',
+        },
+      ],
+    },
+    ignores: ['**/build/**', '**/node_modules/**'],
+  },
+];

package/package.json

@@ -0,0 +1,78 @@
+{
+  "name": "appium-ios-remotexpc",
+  "version": "0.0.1",
+  "main": "build/src/index.js",
+  "types": "build/src/index.d.ts",
+  "type": "module",
+  "exports": {
+    ".": {
+      "import": "./build/src/index.js",
+      "types": "./build/src/index.d.ts"
+    }
+  },
+  "engines": {
+    "node": ">=20 <23"
+  },
+  "scripts": {
+    "clean:build": "rimraf ./build",
+    "build:es": "tsc",
+    "build": "run-s clean:* build:*",
+    "lint": "eslint src --ext .ts --quiet",
+    "format": "prettier --write \"{src,test}/**/*.{ts,tsx}\"",
+    "format:check": "prettier --check \"{src,test}/**/*.{ts,tsx}\"",
+    "lint:fix": "eslint src --ext .ts --fix",
+    "test": "mocha test/integration/**/*.ts",
+    "test:all": "mocha -r tsx/cjs test/run-integration-tests.ts",
+    "test:tunnel": "mocha test/integration/tunnel-test.ts --exit --timeout 1m",
+    "test:pair-record": "mocha test/integration/read-pair-record-test.ts --exit --timeout 1m",
+    "test:diagnostics": "mocha test/integration/diagnostics-test.ts --exit --timeout 1m",
+    "test:unit": "mocha 'test/unit/**/*.ts' --exit --timeout 2m",
+    "test:tunnel-creation": "sudo tsx scripts/test-tunnel-creation.ts",
+    "test:tunnel-creation:lsof": "sudo tsx scripts/test-tunnel-creation.ts --keep-open"
+  },
+  "keywords": [],
+  "author": "Appium Contributors",
+  "license": "Apache-2.0",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/appium/appium-ios-remotexpc.git"
+  },
+  "bugs": {
+    "url": "https://github.com/appium/appium-ios-remotexpc/issues"
+  },
+  "description": "",
+  "devDependencies": {
+    "@appium/eslint-config-appium-ts": "^1.0.4",
+    "@appium/tsconfig": "^0.3.5",
+    "@eslint/js": "^9.23.0",
+    "@semantic-release/changelog": "^6.0.3",
+    "@semantic-release/git": "^10.0.1",
+    "@trivago/prettier-plugin-sort-imports": "^5.2.2",
+    "@types/chai": "^5.2.1",
+    "@types/chai-as-promised": "^8.0.2",
+    "@types/mocha": "^10.0.10",
+    "appium": "^2.17.1",
+    "chai": "^5.2.0",
+    "chai-as-promised": "^8.0.1",
+    "conventional-changelog-conventionalcommits": "^8.0.0",
+    "eslint": "^9.23.0",
+    "eslint-config-prettier": "^10.1.2",
+    "eslint-plugin-unicorn": "^58.0.0",
+    "mocha": "^11.1.0",
+    "prettier": "^3.5.3",
+    "rimraf": "^6.0.1",
+    "semantic-release": "^24.0.0",
+    "ts-node": "^10.9.2",
+    "tsx": "^4.7.0",
+    "typescript": "^5.2.2",
+    "typescript-eslint": "^8.29.0"
+  },
+  "dependencies": {
+    "@appium/strongbox": "^0.x",
+    "@appium/support": "^6.1.0",
+    "@types/node": "^24.0.10",
+    "@xmldom/xmldom": "^0.9.8",
+    "npm-run-all2": "^7.0.2",
+    "tuntap-bridge": "^0.x"
+  }
+}

package/scripts/test-tunnel-creation.ts

@@ -0,0 +1,378 @@
+#!/usr/bin/env node
+/**
+ * Test script for creating lockdown service, starting CoreDeviceProxy, and creating tunnel
+ * This script demonstrates the tunnel creation workflow for all connected devices
+ */
+import { logger } from '@appium/support';
+import type { ConnectionOptions } from 'tls';
+
+import {
+  PacketStreamServer,
+  SocketInfo,
+  TunnelManager,
+  createLockdownServiceByUDID,
+  createUsbmux,
+  startCoreDeviceProxy,
+  TunnelRegistry,
+} from '../src/index.js';
+import { startTunnelRegistryServer } from '../src/lib/tunnel/tunnel-registry-server.js';
+import type { Device } from '../src/lib/usbmux/index.js';
+
+const log = logger.getLogger('TunnelCreation');
+/**
+ * Update tunnel registry with new tunnel information
+ */
+async function updateTunnelRegistry(
+  results: TunnelResult[],
+): Promise<TunnelRegistry> {
+  const now = Date.now();
+  const nowISOString = new Date().toISOString();
+  
+  // Initialize registry if it doesn't exist
+  const registry: TunnelRegistry = {
+    tunnels: {},
+    metadata: {
+      lastUpdated: nowISOString,
+      totalTunnels: 0,
+      activeTunnels: 0,
+    }
+  };
+
+  // Update tunnels
+  for (const result of results) {
+    if (result.success) {
+      const udid = result.device.Properties.SerialNumber;
+      registry.tunnels[udid] = {
+        udid,
+        deviceId: result.device.DeviceID,
+        address: result.tunnel.Address,
+        rsdPort: result.tunnel.RsdPort ?? 0,
+        packetStreamPort: result.packetStreamPort,
+        connectionType: result.device.Properties.ConnectionType,
+        productId: result.device.Properties.ProductID,
+        createdAt: registry.tunnels[udid]?.createdAt ?? now,
+        lastUpdated: now,
+      };
+    }
+  }
+
+  // Update metadata
+  registry.metadata = {
+    lastUpdated: nowISOString,
+    totalTunnels: Object.keys(registry.tunnels).length,
+    activeTunnels: Object.keys(registry.tunnels).length, // Assuming all are active for now
+  };
+
+  return registry;
+}
+
+const activeServers: Array<{ server: any; port: number }> = [];
+const packetStreamServers: Map<string, PacketStreamServer> = new Map();
+
+const deviceInfoMap: Map<
+  string,
+  {
+    udid: string;
+    address: string;
+    rsdPort?: number;
+    connectionType: string;
+    productId: number;
+  }
+> = new Map();
+
+let PACKET_STREAM_BASE_PORT = 50000;
+/**
+ * Setup cleanup handlers for graceful shutdown
+ */
+function setupCleanupHandlers(): void {
+  const cleanup = async (signal: string) => {
+    log.warn(`\nReceived ${signal}. Cleaning up...`);
+
+    // Close all packet stream servers
+    if (packetStreamServers.size > 0) {
+      log.info(
+        `Closing ${packetStreamServers.size} packet stream server(s)...`,
+      );
+      for (const [udid, server] of packetStreamServers) {
+        try {
+          await server.stop();
+          log.info(`Closed packet stream server for device ${udid}`);
+        } catch (err) {
+          log.warn(
+            `Failed to close packet stream server for device ${udid}: ${err}`,
+          );
+        }
+      }
+      packetStreamServers.clear();
+    }
+
+    log.info('Cleanup completed. Exiting...');
+    process.exit(0);
+  };
+
+  // Handle various termination signals
+  process.on('SIGINT', () => cleanup('SIGINT (Ctrl+C)'));
+  process.on('SIGTERM', () => cleanup('SIGTERM'));
+  process.on('SIGHUP', () => cleanup('SIGHUP'));
+
+  // Handle uncaught exceptions and unhandled rejections
+  process.on('uncaughtException', async (error) => {
+    log.error('Uncaught Exception:', error);
+    await cleanup('Uncaught Exception');
+    process.exit(1);
+  });
+
+  process.on('unhandledRejection', async (reason, promise) => {
+    log.error('Unhandled Rejection at:', promise, 'reason:', reason);
+    await cleanup('Unhandled Rejection');
+    process.exit(1);
+  });
+}
+
+/**
+ * Interface for tunnel result
+ */
+interface TunnelResult {
+  device: Device;
+  tunnel: {
+    Address: string;
+    RsdPort?: number;
+  };
+  packetStreamPort?: number;
+  success: boolean;
+  error?: string;
+}
+
+async function createTunnelForDevice(
+  device: Device,
+  tlsOptions: Partial<ConnectionOptions>,
+): Promise<TunnelResult & { socket?: any; socketInfo?: SocketInfo }> {
+  const udid = device.Properties.SerialNumber;
+
+  try {
+    log.info(`\n--- Processing device: ${udid} ---`);
+    log.info(`Device ID: ${device.DeviceID}`);
+    log.info(`Connection Type: ${device.Properties.ConnectionType}`);
+    log.info(`Product ID: ${device.Properties.ProductID}`);
+
+    log.info('Creating lockdown service...');
+    const { lockdownService, device: lockdownDevice } =
+      await createLockdownServiceByUDID(udid);
+    log.info(
+      `Lockdown service created for device: ${lockdownDevice.Properties.SerialNumber}`,
+    );
+
+    log.info('Starting CoreDeviceProxy...');
+    const { socket } = await startCoreDeviceProxy(
+      lockdownService,
+      lockdownDevice.DeviceID,
+      lockdownDevice.Properties.SerialNumber,
+      tlsOptions,
+    );
+    log.info('CoreDeviceProxy started successfully');
+
+    log.info('Creating tunnel...');
+    const tunnel = await TunnelManager.getTunnel(socket);
+    log.info(
+      `Tunnel created for address: ${tunnel.Address} with RsdPort: ${tunnel.RsdPort}`,
+    );
+
+    let packetStreamPort: number | undefined;
+    try {
+      packetStreamPort = PACKET_STREAM_BASE_PORT++;
+      const packetStreamServer = new PacketStreamServer(packetStreamPort);
+      await packetStreamServer.start();
+
+      const consumer = packetStreamServer.getPacketConsumer();
+      if (consumer) {
+        tunnel.addPacketConsumer(consumer);
+      }
+
+      packetStreamServers.set(udid, packetStreamServer);
+
+      log.info(`Packet stream server started on port ${packetStreamPort}`);
+    } catch (err) {
+      log.warn(`Failed to start packet stream server: ${err}`);
+    }
+
+    log.info(`✅ Tunnel creation completed successfully for device: ${udid}`);
+    log.info(`   Tunnel Address: ${tunnel.Address}`);
+    log.info(`   Tunnel RsdPort: ${tunnel.RsdPort}`);
+    if (packetStreamPort) {
+      log.info(`   Packet Stream Port: ${packetStreamPort}`);
+    }
+
+    try {
+      if (socket && typeof socket === 'object' && socket.setNoDelay) {
+        socket.setNoDelay(true);
+      }
+
+      const deviceInfo = {
+        udid: device.Properties.SerialNumber,
+        address: tunnel.Address,
+        rsdPort: tunnel.RsdPort,
+        connectionType: device.Properties.ConnectionType,
+        productId: device.Properties.ProductID,
+      };
+
+      deviceInfoMap.set(device.Properties.SerialNumber, deviceInfo);
+
+      return {
+        device,
+        tunnel: {
+          Address: tunnel.Address,
+          RsdPort: tunnel.RsdPort,
+        },
+        packetStreamPort,
+        success: true,
+        socket
+      };
+    } catch (err) {
+      log.warn(`Could not add device to info server: ${err}`);
+
+      return {
+        device,
+        tunnel: {
+          Address: tunnel.Address,
+          RsdPort: tunnel.RsdPort,
+        },
+        packetStreamPort,
+        success: true,
+        socket,
+      };
+    }
+  } catch (error) {
+    const errorMessage = `Failed to create tunnel for device ${udid}: ${error}`;
+    log.error(`❌ ${errorMessage}`);
+    return {
+      device,
+      tunnel: { Address: '', RsdPort: 0 },
+      success: false,
+      error: errorMessage,
+    };
+  }
+}
+
+/**
+ */
+async function main(): Promise<void> {
+  setupCleanupHandlers();
+
+  const args = process.argv.slice(2);
+  const keepOpenFlag = args.includes('--keep-open') || args.includes('-k');
+  const specificUdid = args.find((arg) => !arg.startsWith('-'));
+
+  if (specificUdid) {
+    log.info(
+      `Starting tunnel creation test for specific UDID: ${specificUdid}`,
+    );
+  } else {
+    log.info('Starting tunnel creation test for all connected devices');
+  }
+
+  if (keepOpenFlag) {
+    log.info('Running in "keep connections open" mode for lsof inspection');
+  }
+
+  try {
+    const tlsOptions: Partial<ConnectionOptions> = {
+      rejectUnauthorized: false,
+      minVersion: 'TLSv1.2',
+    };
+
+    log.info('Connecting to usbmuxd...');
+    const usbmux = await createUsbmux();
+
+    log.info('Listing all connected devices...');
+    const devices = await usbmux.listDevices();
+
+    await usbmux.close();
+
+    if (devices.length === 0) {
+      log.warn(
+        'No devices found. Make sure iOS devices are connected and trusted.',
+      );
+      process.exit(0);
+    }
+
+    log.info(`Found ${devices.length} connected device(s):`);
+    devices.forEach((device, index) => {
+      log.info(`  ${index + 1}. UDID: ${device.Properties.SerialNumber}`);
+      log.info(`     Device ID: ${device.DeviceID}`);
+      log.info(`     Connection: ${device.Properties.ConnectionType}`);
+      log.info(`     Product ID: ${device.Properties.ProductID}`);
+    });
+
+    let devicesToProcess = devices;
+    if (specificUdid) {
+      devicesToProcess = devices.filter(
+        (device) => device.Properties.SerialNumber === specificUdid,
+      );
+
+      if (devicesToProcess.length === 0) {
+        log.error(
+          `Device with UDID ${specificUdid} not found in connected devices.`,
+        );
+        log.error('Available devices:');
+        devices.forEach((device) => {
+          log.error(`  - ${device.Properties.SerialNumber}`);
+        });
+        process.exit(1);
+      }
+    }
+
+    log.info(`\nProcessing ${devicesToProcess.length} device(s)...`);
+
+    const results: TunnelResult[] = [];
+
+    for (const device of devicesToProcess) {
+      const result = await createTunnelForDevice(device, tlsOptions);
+      results.push(result);
+
+      if (devicesToProcess.length > 1) {
+        await new Promise((resolve) => setTimeout(resolve, 1000));
+      }
+    }
+
+    log.info('\n=== TUNNEL CREATION SUMMARY ===');
+    const successful = results.filter((r) => r.success);
+    const failed = results.filter((r) => !r.success);
+
+    log.info(`Total devices processed: ${results.length}`);
+    log.info(`Successful tunnels: ${successful.length}`);
+    log.info(`Failed tunnels: ${failed.length}`);
+
+    if (successful.length > 0) {
+      log.info('\n✅ Successful tunnels:');
+      const registry = await updateTunnelRegistry(results);
+      await startTunnelRegistryServer(registry);
+
+      log.info('\n📁 Tunnel registry API:');
+      log.info('   The tunnel registry is now available through the API at:');
+      log.info('   http://localhost:42314/remotexpc/tunnels');
+      log.info('\n   Available endpoints:');
+      log.info('   - GET /remotexpc/tunnels - List all tunnels');
+      log.info('   - GET /remotexpc/tunnels/:udid - Get tunnel by UDID');
+      log.info('   - GET /remotexpc/tunnels/metadata - Get registry metadata');
+
+      log.info('\n💡 Example usage:');
+      log.info('   curl http://localhost:4723/remotexpc/tunnels');
+      log.info('   curl http://localhost:4723/remotexpc/tunnels/metadata');
+      if (successful.length > 0) {
+        const firstUdid = successful[0].device.Properties.SerialNumber;
+        log.info(
+          `   curl http://localhost:4723/remotexpc/tunnels/${firstUdid}`,
+        );
+      }
+    }
+  } catch (error) {
+    log.error(`Error during tunnel creation test: ${error}`);
+    process.exit(1);
+  }
+}
+
+// Run the main function
+main().catch(async (error) => {
+  log.error(`Fatal error: ${error}`);
+  process.exit(1);
+});