serve-sim-sjchmiela 0.1.34

package/README.md

@@ -0,0 +1,241 @@
+# serve-sim
+
+The `npx serve` of Apple Simulators. 
+
+Host your simulator for use with Agent tools like Codex, Cursor, or Claude Desktop — locally, over your LAN, or host on a remote mac and tunnel anywhere. 
+
+```sh
+npx serve-sim
+# → Preview at http://localhost:3200
+```
+
+https://github.com/user-attachments/assets/fbf890f4-c8c7-4684-82be-d677b8a188f8
+
+`serve-sim` spawns a small Swift helper that captures the simulator's framebuffer via `simctl io`, exposes it as an MJPEG stream + WebSocket control channel, and serves a React preview UI on top. It works with any booted iOS Simulator — no Xcode plugin, no instrumentation in your app.
+
+## Features 
+
+- Full 60 FPS video stream in the browser.
+- Swipe from the bottom to go home.
+- gestures like pinch to zoom by holding the option key.
+- Simulator logs are forwarded to the browser for browser-use MCP tools to read from.
+- Drag and drop videos and images to add them to the simulator device. 
+- Keyboard commands and hot keys are forwarded to the simulator, including CMD+SHIFT+H to go home.
+- Apple Watch, iPad, and iOS support.
+
+## Why?
+
+Hosted simulators can be hard to test, `serve-sim` enables you to test the hosted infra locally first for faster iteration. When you're ready to host a simulator remotely, simply tunnel the served URL and users can interact with the simulator as if it were running locally on their device.
+
+I develop the Expo framework, but this tool is completely agnostic to React Native and can be used for any iOS interaction you need.
+
+## Install
+
+Requires macOS with Xcode command line tools (`xcrun simctl`) and Node.js 18+. `bun` is **not** required to run the CLI. Camera injection uses a host-side helper built for macOS 14+.
+
+> **Note:** Apple Silicon (arm64) only. The bundled `serve-sim-bin` helper ships as an arm64 binary and does not run on Intel (x86_64) Macs.
+
+## CLI
+
+```
+serve-sim [device...]                 Start preview server (default: localhost:3200)
+serve-sim --no-preview [device...]    Stream in foreground without a preview server
+serve-sim gesture '<json>' [-d udid]  Send a touch gesture
+serve-sim button [name] [-d udid]     Send a button press (default: home)
+serve-sim type <text> [-d udid]       Type text via the simulator keyboard
+                                      (US keyboard only; also --stdin / --file <path>)
+serve-sim rotate <orientation> [-d udid]
+                                      portrait | portrait_upside_down |
+                                      landscape_left | landscape_right
+serve-sim ca-debug <option> <on|off> [-d udid]
+                                      Toggle a CoreAnimation debug flag
+                                      (blended|copies|misaligned|offscreen|slow-animations)
+serve-sim memory-warning [-d udid]    Simulate a memory warning
+
+serve-sim camera <bundle-id> [-d udid] [source-options]
+                                      Inject a synthetic camera feed and (re)launch the app
+serve-sim camera switch <placeholder|webcam|file> [arg] [-d udid]
+                                      Hot-swap the running helper's source (no relaunch)
+serve-sim camera mirror <auto|on|off> [-d udid]
+                                      Hot-swap preview-layer mirror mode
+serve-sim camera status [-d udid]     Print helper state as JSON ({alive, source, ...})
+serve-sim camera --list-webcams       List host camera devices
+serve-sim camera --stop-webcam [-d udid]
+                                      Stop the camera helper for a device
+
+Options:
+  -p, --port <port>   Starting port (preview default: 3200, stream default: 3100)
+  -d, --detach        Spawn helper and exit (daemon mode)
+  -q, --quiet         JSON-only output
+      --no-preview    Skip the web UI; stream in foreground only
+      --list [device] List running streams
+      --kill [device] Kill running stream(s)
+
+Camera options (used with `serve-sim camera <bundle-id>`):
+  -f, --file <path>          Image or video file (kind auto-detected from
+                             extension/magic bytes; videos loop at native FPS)
+      --webcam [name]        Live host webcam (defaults to the built-in
+                             front camera when [name] is omitted)
+      --mirror [on|off|auto] Override preview-layer mirroring (default: auto =
+                             front mirrored, back not). Data-output buffers
+                             are never auto-mirrored, matching AVF defaults.
+      --no-mirror            Shortcut for --mirror off
+      --build                Rebuild the dylib + helper from source
+```
+
+### Examples
+
+```sh
+serve-sim                              # auto-detect booted sim, open preview
+serve-sim "iPhone 16 Pro"              # target a specific device
+serve-sim --detach                     # start a background helper, return JSON
+serve-sim --list                       # show running streams
+serve-sim --kill                       # stop all helpers
+
+# Type text into the focused field
+serve-sim type "Hello, world!"
+echo "from stdin" | serve-sim type --stdin
+serve-sim type --file ./snippet.txt
+
+# Camera injection
+serve-sim camera com.acme.MyApp                            # animated placeholder
+serve-sim camera com.acme.MyApp --webcam                   # default webcam
+serve-sim camera com.acme.MyApp --webcam "MacBook Pro Camera"
+serve-sim camera com.acme.MyApp --file ~/Pictures/face.png # static image
+serve-sim camera com.acme.MyApp --file ~/Movies/loop.mp4   # looping video
+
+# Hot-swap source on a running helper (no app relaunch)
+serve-sim camera switch placeholder
+serve-sim camera switch webcam
+serve-sim camera switch ~/Movies/loop.mp4                  # auto-detects file kind
+
+# Other helpers
+serve-sim camera mirror on
+serve-sim camera status                                    # JSON: alive, source, mirror
+serve-sim camera --list-webcams
+serve-sim camera --stop-webcam
+```
+
+Multiple booted simulators are supported — pass several device names, or leave it empty to attach to all of them.
+
+### Camera
+
+`serve-sim camera <bundle-id>` replaces the simulator's camera feed for a single app. A small host-side helper writes BGRA frames into a POSIX shared-memory region; an injected dylib (`DYLD_INSERT_LIBRARIES`) swizzles AVFoundation inside the simulator process so the app reads from that region instead of the simulator's stub camera.
+
+The helper is one-per-device and outlives any single app launch, so multiple apps on the same simulator can share the feed — just run `serve-sim camera <other-bundle-id>` again to relaunch the next app with the dylib attached. Source changes (`camera switch`) and mirror changes (`camera mirror`) flow through the helper's control socket and don't relaunch the app.
+
+Sources:
+
+- **placeholder** — animated programmatic frames (default).
+- **file** — image (PNG/JPEG/HEIC/…) or video (mp4/mov/m4v/webm/…). The CLI sniffs the kind from the extension and falls back to magic bytes for files without an extension.
+- **webcam** — live `AVCaptureDevice` (built-in, Continuity, external).
+
+## Connectors
+
+`serve-sim` can be used with dev servers, browser, and AI editors for more seamless integration.
+
+### Agent Skill
+
+An [Agent Skill](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview) ships in [`skills/serve-sim`](skills/serve-sim) — it teaches AI coding agents (Claude Code, Cursor, Codex CLI, Gemini CLI, and any host implementing the open Agent Skills standard) how to drive a simulator through the CLI: taps, gestures, hardware buttons, rotation, camera injection, and handing the stream off to the host's preview pane.
+
+```sh
+bunx add-skill EvanBacon/serve-sim
+# in Claude Code:
+/plugin marketplace add EvanBacon/serve-sim
+```
+
+See [`skills/serve-sim/README.md`](skills/serve-sim/README.md) for the full capability list.
+
+### Claude Code Desktop
+
+Create a `.claude/launch.json` and define a server:
+
+```json
+{
+  "version": "0.0.1",
+  "configurations": [
+    {
+      "name": "Apple",
+      "runtimeExecutable": "npx",
+      "runtimeArgs": ["serve-sim"],
+      "port": 3200
+    }
+  ]
+}
+```
+
+### Expo
+
+Automatically start the serve-sim process with `npx expo start` and access the URL at `http://localhost:8081/.sim`.
+
+First, customize the `metro.config.js` file (`bunx expo customize`):
+
+```js
+// Learn more https://docs.expo.io/guides/customizing-metro
+const { getDefaultConfig } = require("expo/metro-config");
+const connect = require("connect");
+const { simMiddleware } = require("serve-sim/middleware");
+
+/** @type {import('expo/metro-config').MetroConfig} */
+const config = getDefaultConfig(__dirname);
+
+config.server = config.server || {};
+const originalEnhanceMiddleware = config.server.enhanceMiddleware;
+config.server.enhanceMiddleware = (metroMiddleware, server) => {
+  const middleware = originalEnhanceMiddleware
+    ? originalEnhanceMiddleware(metroMiddleware, server)
+    : metroMiddleware;
+  const app = connect();
+  app.use(simMiddleware({ basePath: "/.sim" }));
+  app.use(middleware);
+  return app;
+};
+
+module.exports = config;
+```
+
+## Embed in your dev server
+
+`serve-sim/middleware` is a Connect-style middleware that mounts the same preview UI inside your existing dev server (Metro, Vite, Next, plain Express, etc.). Run `serve-sim --detach` once to start the streaming helper, then add the middleware:
+
+```ts
+import { simMiddleware } from "serve-sim/middleware";
+
+app.use(simMiddleware({ basePath: "/.sim" }));
+// → preview HTML at /.sim
+// → state JSON  at /.sim/api
+```
+
+The middleware reads the helper's state from `$TMPDIR/serve-sim/` and forwards the user's browser to the live MJPEG + WebSocket endpoints. CORS is wide-open on the helper, so the page renders without a proxy.
+
+## How it works
+
+```
+┌──────────────┐   simctl io   ┌─────────────────┐  MJPEG / WS  ┌─────────┐
+│ iOS Simulator│ ────────────► │ serve-sim-bin   │ ───────────► │ Browser │
+└──────────────┘   (Swift)     │ (per-device)    │              └─────────┘
+                               └─────────────────┘
+                                       ▲
+                                  state file in
+                                $TMPDIR/serve-sim/
+                                       ▲
+                               ┌──────────────────┐
+                               │ serve-sim CLI /  │
+                               │ middleware       │
+                               └──────────────────┘
+```
+
+The Swift helper (`bin/serve-sim-bin`) is a tiny standalone binary — no Xcode dependency at runtime. The CLI embeds it via `bun build --compile`, so installing the npm package is enough.
+
+## Development
+
+```sh
+bun install
+bun run --filter serve-sim build         # build the JS bundles
+bun run --filter serve-sim build:swift   # rebuild the Swift helper
+bun run --filter serve-sim dev           # watch mode
+```
+
+## License
+
+Apache-2.0

package/Sources/SimAXSettings/build.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+OUT_DIR="${1:-$HERE/../../dist/simax}"
+mkdir -p "$OUT_DIR"
+
+SDK="$(xcrun --sdk iphonesimulator --show-sdk-path)"
+BIN="$OUT_DIR/serve-sim-ax-settings"
+
+# Build a fat simulator executable (arm64 + x86_64); it runs inside the sim
+# via `simctl spawn`.
+xcrun --sdk iphonesimulator clang \
+    -arch arm64 -arch x86_64 \
+    -mios-simulator-version-min=15.0 \
+    -isysroot "$SDK" \
+    -framework CoreFoundation \
+    -o "$BIN" \
+    "$HERE/sim-ax-settings.m"
+
+echo "Built: $BIN"
+file "$BIN"

package/Sources/SimAXSettings/sim-ax-settings.m

@@ -0,0 +1,273 @@
+// sim-ax-settings — tiny CLI that runs *inside* the iOS Simulator (via
+// `simctl spawn`) to read and write the simulator-wide settings that
+// `simctl ui` does not cover. It drives the same private libAccessibility
+// and MediaAccessibility setters the Xcode Devices app uses, which write the
+// backing preference *and* post the darwin notification that makes running
+// apps pick the change up live.
+//
+// Usage:
+//   sim-ax-settings get <key>
+//   sim-ax-settings set <key> <value>
+//   sim-ax-settings status            # JSON object of every key
+//
+// Keys / values:
+//   reduce-motion        on|off    -> _AXSSetReduceMotionEnabled
+//   show-borders         on|off    -> _AXSSetButtonShapesEnabled
+//   reduce-transparency  on|off    -> _AXSSetEnhanceBackgroundContrastEnabled
+//   voiceover            on|off    -> _AXSVoiceOverTouchSetEnabled
+//   color-filter         none|grayscale|red-green|green-red|blue-yellow
+//                                  -> MADisplayFilterPrefSetType/CategoryEnabled
+//   liquid-glass         clear|tinted
+//                                  -> com.apple.UIKit UIViewGlassLegibilitySetting
+
+#include <CoreFoundation/CoreFoundation.h>
+#include <dlfcn.h>
+#include <notify.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+typedef int (*GetBoolFn)(void);
+typedef void (*SetBoolFn)(int);
+typedef long (*MAGetTypeFn)(long);
+typedef void (*MASetTypeFn)(long, long);
+typedef int (*MAGetEnabledFn)(long);
+typedef void (*MASetEnabledFn)(long, int);
+
+// MADisplayFilterPref* "category" argument: 1 maps to the "__Color__." key
+// prefix in com.apple.mediaaccessibility (the Settings > Color Filters pane).
+static const long kMAColorCategory = 1;
+
+// MADisplayFilterType values, confirmed against libAccessibility's
+// _AXS{GreenRed,BlueYellow}FilterSetEnabled disassembly on iOS 26.5.
+enum {
+  kFilterNone = 0,
+  kFilterGrayscale = 1,
+  kFilterRedGreen = 2,   // protanopia
+  kFilterGreenRed = 4,   // deuteranopia
+  kFilterBlueYellow = 8, // tritanopia
+};
+
+static void *axHandle(void) {
+  static void *handle;
+  if (!handle) handle = dlopen("/usr/lib/libAccessibility.dylib", RTLD_NOW);
+  return handle;
+}
+
+static void *maHandle(void) {
+  static void *handle;
+  if (!handle) {
+    handle = dlopen(
+        "/System/Library/Frameworks/MediaAccessibility.framework/MediaAccessibility",
+        RTLD_NOW);
+  }
+  return handle;
+}
+
+static void *requireSym(void *handle, const char *name) {
+  void *sym = handle ? dlsym(handle, name) : NULL;
+  if (!sym) {
+    fprintf(stderr, "sim-ax-settings: missing symbol %s\n", name);
+    exit(2);
+  }
+  return sym;
+}
+
+// ─── Boolean AXS settings ───
+
+typedef struct {
+  const char *key;
+  const char *getter;
+  const char *setter;
+} BoolSetting;
+
+static const BoolSetting kBoolSettings[] = {
+    {"reduce-motion", "_AXSReduceMotionEnabled", "_AXSSetReduceMotionEnabled"},
+    {"show-borders", "_AXSButtonShapesEnabled", "_AXSSetButtonShapesEnabled"},
+    {"reduce-transparency", "_AXSEnhanceBackgroundContrastEnabled",
+     "_AXSSetEnhanceBackgroundContrastEnabled"},
+    {"voiceover", "_AXSVoiceOverTouchEnabled", "_AXSVoiceOverTouchSetEnabled"},
+};
+static const size_t kBoolSettingCount =
+    sizeof(kBoolSettings) / sizeof(kBoolSettings[0]);
+
+static const BoolSetting *findBoolSetting(const char *key) {
+  for (size_t i = 0; i < kBoolSettingCount; i++) {
+    if (strcmp(kBoolSettings[i].key, key) == 0) return &kBoolSettings[i];
+  }
+  return NULL;
+}
+
+static int getBoolSetting(const BoolSetting *s) {
+  GetBoolFn fn = (GetBoolFn)requireSym(axHandle(), s->getter);
+  return fn() ? 1 : 0;
+}
+
+static void setBoolSetting(const BoolSetting *s, int enabled) {
+  SetBoolFn fn = (SetBoolFn)requireSym(axHandle(), s->setter);
+  fn(enabled);
+}
+
+// ─── Color filter ───
+
+static const char *filterName(long type, int enabled) {
+  if (!enabled) return "none";
+  switch (type) {
+    case kFilterGrayscale: return "grayscale";
+    case kFilterRedGreen: return "red-green";
+    case kFilterGreenRed: return "green-red";
+    case kFilterBlueYellow: return "blue-yellow";
+    default: return "none";
+  }
+}
+
+static long filterTypeForName(const char *name) {
+  if (strcmp(name, "grayscale") == 0) return kFilterGrayscale;
+  if (strcmp(name, "red-green") == 0) return kFilterRedGreen;
+  if (strcmp(name, "green-red") == 0) return kFilterGreenRed;
+  if (strcmp(name, "blue-yellow") == 0) return kFilterBlueYellow;
+  return kFilterNone;
+}
+
+static const char *getColorFilter(void) {
+  MAGetTypeFn getType =
+      (MAGetTypeFn)requireSym(maHandle(), "MADisplayFilterPrefGetType");
+  MAGetEnabledFn getEnabled = (MAGetEnabledFn)requireSym(
+      maHandle(), "MADisplayFilterPrefGetCategoryEnabled");
+  return filterName(getType(kMAColorCategory), getEnabled(kMAColorCategory));
+}
+
+static void setColorFilter(const char *name) {
+  MASetTypeFn setType =
+      (MASetTypeFn)requireSym(maHandle(), "MADisplayFilterPrefSetType");
+  MASetEnabledFn setEnabled = (MASetEnabledFn)requireSym(
+      maHandle(), "MADisplayFilterPrefSetCategoryEnabled");
+  long type = filterTypeForName(name);
+  if (type == kFilterNone) {
+    setEnabled(kMAColorCategory, 0);
+  } else {
+    setType(kMAColorCategory, type);
+    setEnabled(kMAColorCategory, 1);
+  }
+  // Keep the com.apple.Accessibility grayscale flag in sync, matching what
+  // the Xcode Devices app writes for the Grayscale filter.
+  SetBoolFn setGrayscale =
+      (SetBoolFn)requireSym(axHandle(), "_AXSGrayscaleSetEnabled");
+  setGrayscale(type == kFilterGrayscale);
+}
+
+// ─── Liquid Glass (iOS 26+) ───
+
+static CFStringRef kGlassDomain = CFSTR("com.apple.UIKit");
+static CFStringRef kGlassKey = CFSTR("UIViewGlassLegibilitySetting");
+
+static const char *getLiquidGlass(void) {
+  CFPropertyListRef value = CFPreferencesCopyValue(
+      kGlassKey, kGlassDomain, kCFPreferencesCurrentUser, kCFPreferencesAnyHost);
+  int tinted = 0;
+  if (value) {
+    if (CFGetTypeID(value) == CFNumberGetTypeID()) {
+      int n = 0;
+      CFNumberGetValue((CFNumberRef)value, kCFNumberIntType, &n);
+      tinted = n == 1;
+    }
+    CFRelease(value);
+  }
+  return tinted ? "tinted" : "clear";
+}
+
+static void setLiquidGlass(const char *name) {
+  int tinted = strcmp(name, "tinted") == 0;
+  CFNumberRef value = CFNumberCreate(NULL, kCFNumberIntType, &tinted);
+  CFPreferencesSetValue(kGlassKey, value, kGlassDomain,
+                        kCFPreferencesCurrentUser, kCFPreferencesAnyHost);
+  CFRelease(value);
+  CFPreferencesSynchronize(kGlassDomain, kCFPreferencesCurrentUser,
+                           kCFPreferencesAnyHost);
+  notify_post("UIViewGlassLegibilityUpdateNotification");
+}
+
+// ─── Entry point ───
+
+static void printStatus(void) {
+  printf("{");
+  for (size_t i = 0; i < kBoolSettingCount; i++) {
+    printf("\"%s\":\"%s\",", kBoolSettings[i].key,
+           getBoolSetting(&kBoolSettings[i]) ? "on" : "off");
+  }
+  printf("\"color-filter\":\"%s\",", getColorFilter());
+  printf("\"liquid-glass\":\"%s\"}\n", getLiquidGlass());
+}
+
+static int parseOnOff(const char *value) {
+  if (strcmp(value, "on") == 0 || strcmp(value, "1") == 0 ||
+      strcmp(value, "true") == 0 || strcmp(value, "enabled") == 0)
+    return 1;
+  if (strcmp(value, "off") == 0 || strcmp(value, "0") == 0 ||
+      strcmp(value, "false") == 0 || strcmp(value, "disabled") == 0)
+    return 0;
+  return -1;
+}
+
+int main(int argc, char **argv) {
+  if (argc >= 2 && strcmp(argv[1], "status") == 0) {
+    printStatus();
+    return 0;
+  }
+
+  if (argc == 3 && strcmp(argv[1], "get") == 0) {
+    const char *key = argv[2];
+    const BoolSetting *bs = findBoolSetting(key);
+    if (bs) {
+      printf("%s\n", getBoolSetting(bs) ? "on" : "off");
+      return 0;
+    }
+    if (strcmp(key, "color-filter") == 0) {
+      printf("%s\n", getColorFilter());
+      return 0;
+    }
+    if (strcmp(key, "liquid-glass") == 0) {
+      printf("%s\n", getLiquidGlass());
+      return 0;
+    }
+    fprintf(stderr, "sim-ax-settings: unknown key %s\n", key);
+    return 1;
+  }
+
+  if (argc == 4 && strcmp(argv[1], "set") == 0) {
+    const char *key = argv[2];
+    const char *value = argv[3];
+    const BoolSetting *bs = findBoolSetting(key);
+    if (bs) {
+      int enabled = parseOnOff(value);
+      if (enabled < 0) {
+        fprintf(stderr, "sim-ax-settings: %s wants on|off, got %s\n", key, value);
+        return 1;
+      }
+      setBoolSetting(bs, enabled);
+      return 0;
+    }
+    if (strcmp(key, "color-filter") == 0) {
+      if (strcmp(value, "none") != 0 && filterTypeForName(value) == kFilterNone) {
+        fprintf(stderr, "sim-ax-settings: unknown color filter %s\n", value);
+        return 1;
+      }
+      setColorFilter(value);
+      return 0;
+    }
+    if (strcmp(key, "liquid-glass") == 0) {
+      if (strcmp(value, "clear") != 0 && strcmp(value, "tinted") != 0) {
+        fprintf(stderr, "sim-ax-settings: liquid-glass wants clear|tinted\n");
+        return 1;
+      }
+      setLiquidGlass(value);
+      return 0;
+    }
+    fprintf(stderr, "sim-ax-settings: unknown key %s\n", key);
+    return 1;
+  }
+
+  fprintf(stderr,
+          "Usage: sim-ax-settings status | get <key> | set <key> <value>\n");
+  return 64;
+}

package/Sources/SimCameraHelper/build.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+set -euo pipefail
+HERE="$(cd "$(dirname "$0")" && pwd)"
+OUT_DIR="${1:-$HERE/../../dist/simcam}"
+mkdir -p "$OUT_DIR"
+
+SDK="$(xcrun --sdk macosx --show-sdk-path)"
+BIN="$OUT_DIR/serve-sim-camera-helper"
+
+xcrun --sdk macosx clang \
+    -arch arm64 -arch x86_64 \
+    -mmacosx-version-min=14.0 \
+    -isysroot "$SDK" \
+    -fobjc-arc -fmodules \
+    -framework Foundation \
+    -framework AVFoundation \
+    -framework CoreMedia \
+    -framework CoreVideo \
+    -framework CoreGraphics \
+    -framework CoreImage \
+    -framework CoreText \
+    -framework ImageIO \
+    -framework IOSurface \
+    -framework Accelerate \
+    -O2 \
+    -o "$BIN" \
+    "$HERE/main.m"
+
+# Re-sign so the camera privacy prompt persists per-build instead of restarting.
+codesign -s - -f "$BIN" 2>/dev/null || true
+
+echo "Built: $BIN"
+file "$BIN"