@houwert/conductor 0.28.0 → 0.29.0

package/README.md

@@ -2,9 +2,9 @@
 
 <img src="assets/banner.png" alt="Conductor" width="800" />
 
-# 🎼 Conductor
+# Conductor
 
-**Give Claude hands. Let it drive your app.**
+**Mobile and web UI automation for AI agents.**
 
 [![CI](https://github.com/DouweBos/conductor/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/DouweBos/conductor/actions/workflows/ci.yml)
 [![npm version](https://img.shields.io/npm/v/@houwert/conductor)](https://www.npmjs.com/package/@houwert/conductor)
@@ -14,13 +14,9 @@
 
 ---
 
-Conductor is a token-efficient CLI for mobile UI interactions, built for AI agents. It's a TypeScript reimplementation and partial fork of [Maestro](https://maestro.mobile.dev) that bundles its own native drivers — no external CLI, no setup friction, no nonsense.
+Conductor is a token-efficient CLI for driving and inspecting running apps, built for AI agents. It is a TypeScript reimplementation and partial fork of [Maestro](https://maestro.mobile.dev) that ships its own native drivers, so there is no external CLI to install and no JVM to configure.
 
-It started as an experiment: what if Claude could tap through your app while it's writing the code? Turns out that's extremely useful. Now it's a proper tool.
-
-## ✨ What it does
-
-Conductor gives Claude Code the ability to interact with iOS simulators and Android emulators directly from a coding session. It can navigate UI, inspect the live hierarchy, take screenshots, run flows, and manage multiple devices in parallel across concurrent agents.
+It gives a coding agent the ability to operate an app while writing the code for it: navigate the UI, read the live view hierarchy, take screenshots, run flows, and drive several devices in parallel across concurrent agents.
 
 ```bash
 conductor launch-app com.example.myapp
@@ -30,47 +26,85 @@ conductor assert-visible "Dashboard"
 conductor take-screenshot --output /tmp/screen.png
 ```
 
-One agent writes the feature. Another taps through the app. They talk. It works. 🤝
+Targets iOS and tvOS simulators, physical iOS/tvOS devices, Android emulators and devices, Amazon Fire TV, and web via Playwright.
 
-## 🚀 Quick start
+## Quick start
 
 ```bash
 npm install -g @houwert/conductor
 ```
 
-That's it. Conductor is a pure CLI. To teach an AI agent how to use it, set up the bundled skills in your repo:
+Conductor is a pure CLI. To teach an AI agent how to use it, install the bundled skills into your repository:
 
 ```bash
-conductor init            # interactive: pick scope + skills, writes them into .claude/skills/
+conductor init            # interactive: pick scope + skills, writes to .claude/skills/
 conductor init --yes      # non-interactive: install all skills into ./.claude/skills/
 conductor init --global   # install into ~/.claude/skills/ for all repos
-conductor init --force    # re-sync skills you've already installed
+conductor init --force    # re-sync skills you have already installed
 ```
 
-`init` is the one manual setup step — run it once per repo. In a terminal it walks you through which skills and where; piped or headless (CI/agent) it installs everything non-interactively. It drops a set of capability-scoped Claude Code skills — `conductor-device-interact`, `conductor-inspect`, `conductor-create-flow`, `conductor-metro-debugger`, `conductor-profiler`, and `conductor-device-setup` — that document every command and the act → observe → act workflow.
+`init` is the one manual setup step — run it once per repository. In a terminal it prompts for which skills and where; piped or headless (CI, agent) it installs everything non-interactively. It writes a set of capability-scoped Claude Code skills — `conductor-device-interact`, `conductor-inspect`, `conductor-create-flow`, `conductor-native`, `conductor-metro-debugger`, `conductor-profiler` and `conductor-device-setup` — documenting every command and the act → observe → act workflow.
+
+After upgrading conductor, re-run `conductor init --force` to re-sync the installed skills. `init` stamps the version it installed, so it can tell you when they are stale and prune skills no longer shipped. Integrating another way (a custom `CLAUDE.md`, a slash command) works equally well.
 
-When you upgrade conductor, re-run `conductor init --force` to re-sync the installed skills (it stamps the installed version, so `init` tells you when they're out of date, and prunes any skills no longer shipped). Or wire it in however you like (a custom `CLAUDE.md`, a slash command — it's up to you). Run `conductor --help` for the full command reference, or `conductor <command> --help` for per-command flags.
+Run `conductor --help` for the full command reference, or `conductor <command> --help` for per-command flags.
 
-### 📱 What the CLI can do
+### What the CLI can do
 
 | Capability | Commands |
 |---|---|
-| App lifecycle | `launch-app`, `stop-app`, `clear-state`, `uninstall-app`, `install-app`, `foreground-app`, `copy-app` |
-| Interaction | `tap-on`, `input-text`, `scroll`, `scroll-until-visible`, `swipe`, `press-key`, `erase-text`, `hide-keyboard` |
-| Inspection | `inspect`, `focused`, `take-screenshot`, `capture-ui`, `list-apps` |
-| Assertions | `assert-visible`, `assert-not-visible` |
+| App lifecycle | `launch-app`, `stop-app`, `clear-state`, `install-app`, `uninstall-app`, `foreground-app`, `copy-app`, `download-app` |
+| Interaction | `tap-on`, `input-text`, `scroll`, `scroll-until-visible`, `swipe`, `gesture`, `pinch`, `press-key`, `erase-text`, `hide-keyboard` |
+| Inspection | `inspect`, `focused`, `capture-ui`, `take-screenshot`, `list-apps` |
+| Assertions | `assert-visible`, `assert-not-visible`, `assert-true`, `assert-screenshot` |
 | Navigation | `open-link`, `back` |
-| Flows | `run-flow`, `run-flow-inline`, `run-parallel` |
-| Devices | `start-device`, `list-devices`, `set-location`, `set-orientation` |
+| Flows | `run-flow`, `run-flow-inline`, `run-parallel`, `run-sequence`, `flow` |
+| Devices | `start-device`, `stop-device`, `list-devices`, `device-pool`, `set-location`, `set-orientation`, `set-permissions` |
+| Debugging | `logs`, `crashes`, `network`, `memory`, `profile`, `metro`, `record-video`, `stream-server` |
+| In-process (iOS/tvOS) | `native-inspect`, `native-find`, `native-set`, `native-eval`, `native-heap`, and more |
+| Test cases | `cases` |
 | Web setup | `install-web [browser]` (installs a Playwright browser; `--check` prints status) |
-| Discovery | `list-options [command]` / `<command> --options` (valid values for enumerated params) |
+| Discovery | `list-options [command]` / `<command> --options`, `workspace` |
+
+## Conductor Studio
+
+**[Conductor Studio](apps/studio)** is a desktop app (Electron + React) built on this CLI. It does three jobs: writing and managing Maestro tests, writing them with an agent, and tracking them as test cases.
+
+It provides a flow editor with autocomplete and linting, a live device stream with element picking and a record mode that turns your interactions into flow steps, an agentic test writer that verifies described behaviour on a device and files a visual report, and local test case management. Light and dark, signed and notarized, auto-updating.
+
+Studio bundles its own copy of the conductor CLI, so it needs nothing installed globally. The version it uses can be pinned from Settings.
+
+**Download:** [Releases](https://github.com/DouweBos/conductor/releases) — macOS (Apple silicon). Studio releases are tagged `studio-v*`; the CLI's are tagged `cli-v*`.
+
+See the [Studio README](apps/studio/README.md) for the full feature tour, architecture and release process.
+
+```bash
+pnpm dev:studio    # run it from source
+```
 
-## 🔨 Building locally
+## Repository structure
+
+```
+conductor/
+├── packages/
+│   ├── cli/              # TypeScript CLI (@houwert/conductor)
+│   ├── android-driver/   # Kotlin/Gradle instrumentation driver
+│   ├── ios-driver/       # Swift/Xcode XCTest driver
+│   ├── ios-inproc/       # Library injected into the app for a second inspection plane
+│   ├── ios-hid/          # Host binary injecting HID below the XCTest layer
+│   ├── ios-capture/      # Host binary capturing the Simulator framebuffer
+│   └── studio-ui/        # Design system for Conductor Studio
+├── apps/
+│   └── studio/           # Conductor Studio — the desktop app
+└── Makefile
+```
+
+## Building locally
 
 ### Prerequisites
 
-- Node.js + pnpm v9
-- **iOS:** Xcode with command-line tools
+- Node.js and pnpm 10
+- **iOS/tvOS:** Xcode with command-line tools
 - **Android:** Android SDK with `adb` on `PATH`
 
 ### Full build
@@ -79,7 +113,7 @@ When you upgrade conductor, re-run `conductor init --force` to re-sync the insta
 make build
 ```
 
-Builds the iOS driver (xcodebuild), the Android driver (Gradle), packages both into the CLI, and compiles TypeScript. Then link it globally:
+Builds the iOS and tvOS XCTest drivers, the in-process library and capture binary (xcodebuild), and the Android driver (Gradle); packages them all into the CLI and compiles TypeScript. Then link it globally:
 
 ```bash
 cd packages/cli && pnpm link --global
@@ -97,48 +131,23 @@ pnpm install && pnpm build
 ### Individual targets
 
 ```bash
-make build-cli          # CLI TypeScript only
-make build-ios-driver   # iOS XCTest driver
+make build-cli            # CLI TypeScript only
+make build-ios-driver     # iOS XCTest driver
 make build-android-driver # Android instrumentation driver
-make package-cli        # Bundle drivers into CLI package
-```
-
-## 🗂️ Repository structure
-
-```
-conductor/
-├── packages/
-│   ├── cli/              # TypeScript CLI (@houwert/conductor)
-│   ├── android-driver/   # Kotlin/Gradle instrumentation driver
-│   ├── ios-driver/       # Swift/Xcode XCTest driver
-│   └── studio-ui/        # Design system for Conductor Studio
-├── apps/
-│   └── studio/           # Conductor Studio — the desktop app
-└── Makefile
-```
-
-## 🖥️ Conductor Studio
-
-There's also a desktop app: **Conductor Studio**, a Maestro test workbench built
-on this CLI — flow editor with autocomplete, live device stream with element
-picking, an agentic test writer, and test case management. It lives in
-[`apps/studio`](apps/studio); see its
-[README](apps/studio/README.md) for what it does and how to run it.
-
-```bash
-pnpm dev:studio
+make package-cli          # Bundle drivers into CLI package
 ```
 
-## 🛠️ Development
+## Development
 
 ```bash
-pnpm dev     # TypeScript watch mode
-pnpm lint    # ESLint + Prettier check
+pnpm dev       # TypeScript watch mode
+pnpm lint      # ESLint + Prettier check
 pnpm lint:fix  # Auto-fix formatting
-pnpm test    # Run test suite
+pnpm test      # Run test suite
 ```
 
-## 📋 Requirements
+## Requirements
 
-- **iOS:** Xcode with a booted simulator
+- **iOS/tvOS:** Xcode with a booted simulator, or a paired physical device
 - **Android:** `adb` on `PATH` with a running emulator or connected device
+- **Web:** a Playwright browser (`conductor install-web`)

package/dist/android/device.js

@@ -0,0 +1,94 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.adb = adb;
+exports.adbShell = adbShell;
+exports.parseResumedPackage = parseResumedPackage;
+exports.resolveAndroidForegroundApp = resolveAndroidForegroundApp;
+exports.shellBracketed = shellBracketed;
+exports.buildClockAnchor = buildClockAnchor;
+exports.parseProcUptimeMs = parseProcUptimeMs;
+exports.toDeviceRealtimeMs = toDeviceRealtimeMs;
+/**
+ * Small adb helpers for read-only device queries.
+ *
+ * Deliberately spawns adb directly rather than going through `runner.ts` so the
+ * profiling commands can query a device without booting the gRPC driver daemon —
+ * `dumpsys` needs neither, and starting the daemon would perturb what we measure.
+ */
+const child_process_1 = require("child_process");
+const sdk_js_1 = require("./sdk.js");
+function adb(deviceId, args) {
+    return new Promise((resolve) => {
+        const proc = (0, child_process_1.spawn)((0, sdk_js_1.resolveAndroidTool)('adb'), ['-s', deviceId, ...args], {
+            stdio: ['ignore', 'pipe', 'pipe'],
+            env: (0, sdk_js_1.androidSpawnEnv)(),
+        });
+        let stdout = '';
+        let stderr = '';
+        proc.stdout.on('data', (c) => (stdout += c.toString()));
+        proc.stderr.on('data', (c) => (stderr += c.toString()));
+        proc.on('close', (code) => resolve({ success: code === 0, stdout, stderr }));
+        proc.on('error', (err) => resolve({ success: false, stdout: '', stderr: err.message }));
+    });
+}
+function adbShell(deviceId, args) {
+    return adb(deviceId, ['shell', ...args]);
+}
+/**
+ * Package name of the resumed activity, or undefined. Android spells this
+ * differently across API levels, hence the alternation.
+ */
+function parseResumedPackage(dumpsysActivities) {
+    const m = dumpsysActivities.match(/(?:m|top)?ResumedActivity[=:].*?([a-zA-Z][a-zA-Z0-9_]*(?:\.[a-zA-Z][a-zA-Z0-9_]*)+)\//);
+    return m ? m[1] : undefined;
+}
+async function resolveAndroidForegroundApp(deviceId) {
+    const dump = await adbShell(deviceId, ['dumpsys', 'activity', 'activities']);
+    return dump.success ? parseResumedPackage(dump.stdout) : undefined;
+}
+/**
+ * Run `command` on the device bracketed by two CLOCK_REALTIME reads, in one
+ * adb invocation. The caller supplies the monotonic reading by parsing it out
+ * of `stdout` (for gfxinfo, its own `Uptime:` header line).
+ */
+async function shellBracketed(deviceId, command) {
+    const hostStart = Date.now();
+    // One argument, not argv: adb joins everything after `shell` with spaces and
+    // the device's own shell parses it, so the semicolons survive.
+    const res = await adbShell(deviceId, [`date +%s%N; ${command}; date +%s%N`]);
+    const hostElapsedMs = Date.now() - hostStart;
+    if (!res.success)
+        return undefined;
+    const stamps = [...res.stdout.matchAll(/^\s*(\d{16,})\s*$/gm)].map((m) => Number(m[1]) / 1e6);
+    if (stamps.length < 2)
+        return undefined;
+    const realtimeStartMs = stamps[0];
+    const realtimeEndMs = stamps[stamps.length - 1];
+    // Strip the two stamp lines so callers parse only the command's own output.
+    const stdout = res.stdout
+        .split(/\r?\n/)
+        .filter((l) => !/^\s*\d{16,}\s*$/.test(l))
+        .join('\n');
+    return { stdout, realtimeStartMs, realtimeEndMs, hostElapsedMs };
+}
+function buildClockAnchor(bracket, deviceMonotonicMs) {
+    const spread = bracket.realtimeEndMs - bracket.realtimeStartMs;
+    return {
+        deviceMonotonicMs,
+        deviceRealtimeMs: (bracket.realtimeStartMs + bracket.realtimeEndMs) / 2,
+        anchorErrorMs: Math.round((spread / 2) * 100) / 100,
+        hostEpochMs: Date.now(),
+        // The device cannot have spent more wall time on this than the host did
+        // waiting for it; if it claims to, the realtime clock was stepped.
+        clockStepped: spread > bracket.hostElapsedMs + 50 ? true : undefined,
+    };
+}
+/** Device CLOCK_MONOTONIC ms from /proc/uptime. Cheap (~1ms) but 10ms-grained. */
+function parseProcUptimeMs(out) {
+    const m = out.match(/^\s*(\d+\.\d+)\s+\d+\.\d+\s*$/m);
+    return m ? Number(m[1]) * 1000 : undefined;
+}
+/** Convert a framestats vsync timestamp (ns, monotonic) to the app's Date.now() domain. */
+function toDeviceRealtimeMs(anchor, monotonicNs) {
+    return monotonicNs / 1e6 - anchor.deviceMonotonicMs + anchor.deviceRealtimeMs;
+}

package/dist/commands/debug.js

@@ -47,11 +47,11 @@ function resolveSession(sessionName) {
     };
 }
 async function debugStatus(opts, sessionName, debugOpts) {
-    const port = debugOpts.port ?? 8081;
     try {
-        const targets = await (0, metro_js_1.fetchTargets)(port, 'localhost');
         const { deviceId, platformPromise } = resolveSession(sessionName);
         const platform = await platformPromise;
+        const port = await (0, metro_cdp_js_1.resolveMetroPort)({ port: debugOpts.port, deviceId, platform });
+        const targets = await (0, metro_js_1.fetchTargets)(port, 'localhost');
         const client = new metro_cdp_js_1.MetroCdpClient();
         await client.connect({ port, deviceId, platform, targetIndex: debugOpts.targetIndex });
         await client.enableDomain('Runtime');
@@ -120,10 +120,10 @@ async function debugEvaluate(expr, opts, sessionName, debugOpts) {
             return 1;
         }
     }
-    const port = debugOpts.port ?? 8081;
     const { deviceId, platformPromise } = resolveSession(sessionName);
     try {
         const platform = await platformPromise;
+        const port = await (0, metro_cdp_js_1.resolveMetroPort)({ port: debugOpts.port, deviceId, platform });
         const client = new metro_cdp_js_1.MetroCdpClient();
         await client.connect({ port, deviceId, platform, targetIndex: debugOpts.targetIndex });
         const value = await client.evaluate(expr);
@@ -140,10 +140,10 @@ async function debugEvaluate(expr, opts, sessionName, debugOpts) {
     }
 }
 async function debugComponentTree(opts, sessionName, debugOpts) {
-    const port = debugOpts.port ?? 8081;
     const { deviceId, platformPromise } = resolveSession(sessionName);
     try {
         const platform = await platformPromise;
+        const port = await (0, metro_cdp_js_1.resolveMetroPort)({ port: debugOpts.port, deviceId, platform });
         const client = new metro_cdp_js_1.MetroCdpClient();
         await client.connect({ port, deviceId, platform, targetIndex: debugOpts.targetIndex });
         const awaitCallback = await client.installCallbackBinding();
@@ -214,10 +214,10 @@ async function debugInspectElement(at, opts, sessionName, debugOpts) {
     }
     const x = Number(m[1]);
     const y = Number(m[2]);
-    const port = debugOpts.port ?? 8081;
     const { deviceId, platformPromise } = resolveSession(sessionName);
     try {
         const platform = await platformPromise;
+        const port = await (0, metro_cdp_js_1.resolveMetroPort)({ port: debugOpts.port, deviceId, platform });
         const client = new metro_cdp_js_1.MetroCdpClient();
         await client.connect({ port, deviceId, platform, targetIndex: debugOpts.targetIndex });
         const awaitCallback = await client.installCallbackBinding();
@@ -253,10 +253,10 @@ async function debugLogRegistry(opts, sessionName) {
     return (0, logs_js_1.logs)(opts, sessionName, { source: 'metro', list: true });
 }
 async function debugReload(opts, sessionName, debugOpts) {
-    const port = debugOpts.port ?? 8081;
     const { deviceId, platformPromise } = resolveSession(sessionName);
     try {
         const platform = await platformPromise;
+        const port = await (0, metro_cdp_js_1.resolveMetroPort)({ port: debugOpts.port, deviceId, platform });
         await (0, metro_cdp_js_1.cdpCall)('Page.reload', undefined, {
             port,
             deviceId,

package/dist/commands/focused.js

@@ -1,6 +1,8 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.HELP = void 0;
+exports.queryFocused = queryFocused;
+exports.focusKey = focusKey;
 exports.focused = focused;
 exports.HELP = `  focused [--poll [interval_ms]]       Print metadata of the currently focused element
                                        --poll continuously watches for focus changes (default 500ms)`;
@@ -216,6 +218,31 @@ async function queryFocused(driver) {
     }
     throw new Error('Unknown driver type');
 }
+/**
+ * Stable identity for a focused element, for detecting *that focus moved*.
+ *
+ * Deliberately excludes the visible label: TV rows repeat titles, so two
+ * neighbouring cards can print identically. Bounds plus whichever id the
+ * platform exposes distinguishes them.
+ */
+function focusKey(data) {
+    if (!data)
+        return '(none)';
+    const bounds = data.bounds;
+    const boundsKey = bounds
+        ? Object.entries(bounds)
+            .sort(([a], [b]) => a.localeCompare(b))
+            .map(([k, v]) => `${k}=${v}`)
+            .join(',')
+        : '';
+    const id = data.identifier ||
+        data.resourceId ||
+        data.ref ||
+        data.contentDesc ||
+        '';
+    const type = data.elementTypeName || data.className || data.role || '';
+    return `${id}|${type}|${boundsKey}`;
+}
 async function focused(opts = {}, sessionName = 'default', { poll = false, interval = 500 } = {}) {
     try {
         const driver = await (0, runner_js_1.getDriver)(sessionName);