simdrive 0.2.0a1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,155 @@
1
+ Metadata-Version: 2.4
2
+ Name: simdrive
3
+ Version: 0.2.0a1
4
+ Summary: Hand your iOS simulator to your agent. Claude-native MCP driver for iOS simulator testing — vision, taps, recordings.
5
+ Author-email: SyncTek LLC <info@synctek.io>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://synctek.io/simdrive
8
+ Project-URL: Repository, https://github.com/SyncTek-LLC/simdrive
9
+ Project-URL: Issues, https://github.com/SyncTek-LLC/simdrive/issues
10
+ Keywords: ios,simulator,mcp,claude,testing,qa,agent,anthropic
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Operating System :: MacOS
18
+ Classifier: Topic :: Software Development :: Testing
19
+ Classifier: Topic :: Software Development :: Quality Assurance
20
+ Requires-Python: >=3.10
21
+ Description-Content-Type: text/markdown
22
+ Requires-Dist: Pillow>=10.0
23
+ Requires-Dist: pyyaml>=6.0
24
+ Requires-Dist: mcp>=1.0
25
+ Requires-Dist: pyobjc-framework-Quartz>=10.0
26
+ Requires-Dist: pyobjc-framework-Vision>=10.0
27
+ Provides-Extra: ssim
28
+ Requires-Dist: scikit-image>=0.22; extra == "ssim"
29
+ Provides-Extra: dev
30
+ Requires-Dist: pytest>=7.0; extra == "dev"
31
+ Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
32
+ Requires-Dist: ruff>=0.1.0; extra == "dev"
33
+
34
+ # simdrive
35
+
36
+ > **Hand your iOS simulator to your agent.**
37
+
38
+ Claude-native MCP server for driving iOS simulators. Vision-first. No XCTest, no accessibility-tree query, no daemons. Your agent looks at a screenshot, picks a pixel, and `simdrive` taps it.
39
+
40
+ ## Why
41
+
42
+ You stay in your editor. Your agent drives the sim in the background. Taps don't steal focus, your keyboard doesn't get hijacked.
43
+
44
+ Automating an iOS simulator from inside an LLM session has historically required:
45
+ - A Swift XCTest runner that breaks every Xcode release
46
+ - An accessibility tree your agent has to mentally reconstruct from JSON dumps
47
+ - Bespoke selectors (`label:"Sign in"`) that drift with every UI change
48
+ - Watchdogs killing your runner mid-test
49
+
50
+ simdrive replaces all of that with: **screenshot in, click out**. Your agent already understands screenshots — the LLM is the selector engine.
51
+
52
+ ## Install
53
+
54
+ ```bash
55
+ pip install simdrive
56
+ ```
57
+
58
+ Requirements:
59
+ - macOS with Xcode + iOS Simulator (for native HID input)
60
+ - A booted simulator. simdrive will use a running one or boot one for you.
61
+
62
+ simdrive runs in the background by default — taps and keystrokes go straight to the simulator without raising its window or stealing your keyboard focus. Verify via `session_status` (`mode: "background"`).
63
+
64
+ ## Wire into Claude
65
+
66
+ Add to your `.mcp.json`:
67
+
68
+ ```json
69
+ {
70
+ "mcpServers": {
71
+ "simdrive": { "command": "simdrive" }
72
+ }
73
+ }
74
+ ```
75
+
76
+ Restart Claude Code. The 12 simdrive tools are now available.
77
+
78
+ ## Quickstart
79
+
80
+ ```
81
+ You: open Settings on iPhone 17 Pro and turn on Airplane Mode.
82
+
83
+ Claude (using simdrive):
84
+ → session_start({device: "iPhone 17 Pro", app_bundle_id: "com.apple.Preferences"})
85
+ → observe() # screenshot + annotated copy with numbered marks
86
+ → tap({text: "Airplane Mode"}) # by visible text
87
+ → observe() # sees the toggle
88
+ → tap({mark: 12}) # by mark number from the annotation
89
+ → observe() # confirms it's green
90
+ ```
91
+
92
+ You can also `tap({x, y})` if you have specific pixel coords (great for replay). Pick whichever is lowest-friction per call:
93
+
94
+ | Form | Use it for |
95
+ |------|------------|
96
+ | `{text: "..."}` | Buttons, labels, anything with visible text |
97
+ | `{mark: N}` | When the agent has just looked at the annotated screenshot |
98
+ | `{x, y}` | Replays, deterministic UI tests, icons without text |
99
+
100
+ That's the whole loop. No selectors. No waits. No XCTest.
101
+
102
+ ## Tool surface (12 tools)
103
+
104
+ | Tool | Purpose |
105
+ |------|---------|
106
+ | `session_start` | Boot/find a sim, optionally launch an app |
107
+ | `session_end` | End session (sim stays booted) |
108
+ | `session_status` | Inspect active session(s) |
109
+ | `observe` | Capture screenshot (returns file path), optional log tail |
110
+ | `tap` | Click at screenshot pixel coordinate |
111
+ | `swipe` | Drag from (x1,y1)→(x2,y2) |
112
+ | `type_text` | Send keyboard input |
113
+ | `press_key` | Hardware buttons (home, lock, siri, shake, return, etc.) |
114
+ | `record_start` | Begin recording every action |
115
+ | `record_stop` | Finalize recording.yaml |
116
+ | `replay` | Re-execute a recording with SSIM drift detection |
117
+ | `logs` | Tail simulator logs (NSPredicate filterable) |
118
+
119
+ Coordinates are always in **screenshot pixel space** — same pixels the agent sees in the most recent `observe`.
120
+
121
+ ## Recording + replay
122
+
123
+ ```
124
+ record_start({name: "checkout-flow"})
125
+ ... agent does the flow naturally, calling tap/swipe/type_text ...
126
+ record_stop() # writes ~/.simdrive/recordings/checkout-flow/recording.yaml
127
+ ```
128
+
129
+ Later:
130
+
131
+ ```
132
+ replay({name: "checkout-flow", on_drift: "halt"})
133
+ ```
134
+
135
+ Each step is gated on visual similarity: if the live screen has drifted from the recorded pre-screenshot, the replay halts (`halt`), warns and continues (`warn`), or proceeds blind (`force`). The recording is a self-contained YAML+PNG bundle you can commit to your repo.
136
+
137
+ ## Testing
138
+
139
+ ```bash
140
+ pip install simdrive[dev]
141
+ pytest # 22 unit tests, no sim required
142
+ pytest -m live # 26 live tests against TestKitApp
143
+ ```
144
+
145
+ Live tests boot a fresh TestKitApp session per test and exercise every tool: tap by text/mark/coords, type into focused fields, swipe-to-scroll, alert-while-focused dismissal (the iOS 26 case that defeated v15), record + replay with drift detection.
146
+
147
+ ## What this isn't
148
+
149
+ - **Not** a real-device tool. v0.1 is simulator-only. Real device support via `idb`/`devicectl` is on the roadmap.
150
+ - **Not** a CI replacement (yet). Designed for interactive Claude sessions; CI integration is a follow-up.
151
+ - **Not** a fork of XCTest. We deliberately avoid Apple's testing stack to stay durable across Xcode releases.
152
+
153
+ ## License
154
+
155
+ MIT. Built by [SyncTek](https://synctek.io).
@@ -0,0 +1,122 @@
1
+ # simdrive
2
+
3
+ > **Hand your iOS simulator to your agent.**
4
+
5
+ Claude-native MCP server for driving iOS simulators. Vision-first. No XCTest, no accessibility-tree query, no daemons. Your agent looks at a screenshot, picks a pixel, and `simdrive` taps it.
6
+
7
+ ## Why
8
+
9
+ You stay in your editor. Your agent drives the sim in the background. Taps don't steal focus, your keyboard doesn't get hijacked.
10
+
11
+ Automating an iOS simulator from inside an LLM session has historically required:
12
+ - A Swift XCTest runner that breaks every Xcode release
13
+ - An accessibility tree your agent has to mentally reconstruct from JSON dumps
14
+ - Bespoke selectors (`label:"Sign in"`) that drift with every UI change
15
+ - Watchdogs killing your runner mid-test
16
+
17
+ simdrive replaces all of that with: **screenshot in, click out**. Your agent already understands screenshots — the LLM is the selector engine.
18
+
19
+ ## Install
20
+
21
+ ```bash
22
+ pip install simdrive
23
+ ```
24
+
25
+ Requirements:
26
+ - macOS with Xcode + iOS Simulator (for native HID input)
27
+ - A booted simulator. simdrive will use a running one or boot one for you.
28
+
29
+ simdrive runs in the background by default — taps and keystrokes go straight to the simulator without raising its window or stealing your keyboard focus. Verify via `session_status` (`mode: "background"`).
30
+
31
+ ## Wire into Claude
32
+
33
+ Add to your `.mcp.json`:
34
+
35
+ ```json
36
+ {
37
+ "mcpServers": {
38
+ "simdrive": { "command": "simdrive" }
39
+ }
40
+ }
41
+ ```
42
+
43
+ Restart Claude Code. The 12 simdrive tools are now available.
44
+
45
+ ## Quickstart
46
+
47
+ ```
48
+ You: open Settings on iPhone 17 Pro and turn on Airplane Mode.
49
+
50
+ Claude (using simdrive):
51
+ → session_start({device: "iPhone 17 Pro", app_bundle_id: "com.apple.Preferences"})
52
+ → observe() # screenshot + annotated copy with numbered marks
53
+ → tap({text: "Airplane Mode"}) # by visible text
54
+ → observe() # sees the toggle
55
+ → tap({mark: 12}) # by mark number from the annotation
56
+ → observe() # confirms it's green
57
+ ```
58
+
59
+ You can also `tap({x, y})` if you have specific pixel coords (great for replay). Pick whichever is lowest-friction per call:
60
+
61
+ | Form | Use it for |
62
+ |------|------------|
63
+ | `{text: "..."}` | Buttons, labels, anything with visible text |
64
+ | `{mark: N}` | When the agent has just looked at the annotated screenshot |
65
+ | `{x, y}` | Replays, deterministic UI tests, icons without text |
66
+
67
+ That's the whole loop. No selectors. No waits. No XCTest.
68
+
69
+ ## Tool surface (12 tools)
70
+
71
+ | Tool | Purpose |
72
+ |------|---------|
73
+ | `session_start` | Boot/find a sim, optionally launch an app |
74
+ | `session_end` | End session (sim stays booted) |
75
+ | `session_status` | Inspect active session(s) |
76
+ | `observe` | Capture screenshot (returns file path), optional log tail |
77
+ | `tap` | Click at screenshot pixel coordinate |
78
+ | `swipe` | Drag from (x1,y1)→(x2,y2) |
79
+ | `type_text` | Send keyboard input |
80
+ | `press_key` | Hardware buttons (home, lock, siri, shake, return, etc.) |
81
+ | `record_start` | Begin recording every action |
82
+ | `record_stop` | Finalize recording.yaml |
83
+ | `replay` | Re-execute a recording with SSIM drift detection |
84
+ | `logs` | Tail simulator logs (NSPredicate filterable) |
85
+
86
+ Coordinates are always in **screenshot pixel space** — same pixels the agent sees in the most recent `observe`.
87
+
88
+ ## Recording + replay
89
+
90
+ ```
91
+ record_start({name: "checkout-flow"})
92
+ ... agent does the flow naturally, calling tap/swipe/type_text ...
93
+ record_stop() # writes ~/.simdrive/recordings/checkout-flow/recording.yaml
94
+ ```
95
+
96
+ Later:
97
+
98
+ ```
99
+ replay({name: "checkout-flow", on_drift: "halt"})
100
+ ```
101
+
102
+ Each step is gated on visual similarity: if the live screen has drifted from the recorded pre-screenshot, the replay halts (`halt`), warns and continues (`warn`), or proceeds blind (`force`). The recording is a self-contained YAML+PNG bundle you can commit to your repo.
103
+
104
+ ## Testing
105
+
106
+ ```bash
107
+ pip install simdrive[dev]
108
+ pytest # 22 unit tests, no sim required
109
+ pytest -m live # 26 live tests against TestKitApp
110
+ ```
111
+
112
+ Live tests boot a fresh TestKitApp session per test and exercise every tool: tap by text/mark/coords, type into focused fields, swipe-to-scroll, alert-while-focused dismissal (the iOS 26 case that defeated v15), record + replay with drift detection.
113
+
114
+ ## What this isn't
115
+
116
+ - **Not** a real-device tool. v0.1 is simulator-only. Real device support via `idb`/`devicectl` is on the roadmap.
117
+ - **Not** a CI replacement (yet). Designed for interactive Claude sessions; CI integration is a follow-up.
118
+ - **Not** a fork of XCTest. We deliberately avoid Apple's testing stack to stay durable across Xcode releases.
119
+
120
+ ## License
121
+
122
+ MIT. Built by [SyncTek](https://synctek.io).
@@ -0,0 +1,67 @@
1
+ [build-system]
2
+ requires = ["setuptools>=68.0", "wheel"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "simdrive"
7
+ version = "0.2.0a1"
8
+ description = "Hand your iOS simulator to your agent. Claude-native MCP driver for iOS simulator testing — vision, taps, recordings."
9
+ readme = "README.md"
10
+ license = "MIT"
11
+ requires-python = ">=3.10"
12
+ authors = [
13
+ {name = "SyncTek LLC", email = "info@synctek.io"},
14
+ ]
15
+ keywords = ["ios", "simulator", "mcp", "claude", "testing", "qa", "agent", "anthropic"]
16
+ classifiers = [
17
+ "Development Status :: 3 - Alpha",
18
+ "Intended Audience :: Developers",
19
+ "Programming Language :: Python :: 3",
20
+ "Programming Language :: Python :: 3.10",
21
+ "Programming Language :: Python :: 3.11",
22
+ "Programming Language :: Python :: 3.12",
23
+ "Operating System :: MacOS",
24
+ "Topic :: Software Development :: Testing",
25
+ "Topic :: Software Development :: Quality Assurance",
26
+ ]
27
+ dependencies = [
28
+ "Pillow>=10.0",
29
+ "pyyaml>=6.0",
30
+ "mcp>=1.0",
31
+ "pyobjc-framework-Quartz>=10.0",
32
+ "pyobjc-framework-Vision>=10.0",
33
+ ]
34
+
35
+ [project.urls]
36
+ Homepage = "https://synctek.io/simdrive"
37
+ Repository = "https://github.com/SyncTek-LLC/simdrive"
38
+ Issues = "https://github.com/SyncTek-LLC/simdrive/issues"
39
+
40
+ [project.optional-dependencies]
41
+ ssim = ["scikit-image>=0.22"]
42
+ dev = [
43
+ "pytest>=7.0",
44
+ "pytest-asyncio>=0.23",
45
+ "ruff>=0.1.0",
46
+ ]
47
+
48
+ [project.scripts]
49
+ simdrive = "simdrive.server:serve"
50
+ simdrive-mcp = "simdrive.server:serve"
51
+
52
+ [tool.setuptools.packages.find]
53
+ where = ["src"]
54
+
55
+ [tool.setuptools.package-data]
56
+ simdrive = ["_bin/simdrive-input"]
57
+
58
+ [tool.pytest.ini_options]
59
+ testpaths = ["tests"]
60
+ pythonpath = ["src"]
61
+ markers = [
62
+ "live: tests that require a booted iOS Simulator and Xcode",
63
+ ]
64
+
65
+ [tool.ruff]
66
+ line-length = 110
67
+ target-version = "py310"
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,64 @@
1
+ """Build hook: compile the native HID injection helper before packaging.
2
+
3
+ This runs `make` in `native/` whenever setup builds the wheel. The output
4
+ binary lands at `src/simdrive/_bin/simdrive-input` and is picked up by
5
+ [tool.setuptools.package-data] in pyproject.toml.
6
+
7
+ setup.py is intentionally minimal — pyproject.toml is the source of truth
8
+ for project metadata.
9
+ """
10
+ from __future__ import annotations
11
+
12
+ import os
13
+ import platform
14
+ import subprocess
15
+ import sys
16
+ from pathlib import Path
17
+
18
+ from setuptools import setup
19
+ from setuptools.command.build_py import build_py
20
+
21
+
22
+ HERE = Path(__file__).parent
23
+ NATIVE_DIR = HERE / "native"
24
+ BINARY_PATH = HERE / "src" / "simdrive" / "_bin" / "simdrive-input"
25
+
26
+
27
+ class BuildPyWithNative(build_py):
28
+ """build_py subclass that compiles the native HID helper as a pre-step."""
29
+
30
+ def run(self) -> None:
31
+ if platform.system() == "Darwin" and NATIVE_DIR.exists():
32
+ self._build_native()
33
+ else:
34
+ print(
35
+ f"simdrive: skipping native build (system={platform.system()}, "
36
+ f"native_dir_exists={NATIVE_DIR.exists()})",
37
+ file=sys.stderr,
38
+ )
39
+ super().run()
40
+
41
+ def _build_native(self) -> None:
42
+ if BINARY_PATH.exists() and os.environ.get("SIMDRIVE_SKIP_NATIVE_BUILD"):
43
+ print(f"simdrive: SIMDRIVE_SKIP_NATIVE_BUILD set; using existing {BINARY_PATH}")
44
+ return
45
+ print("simdrive: building native helper via make...")
46
+ try:
47
+ subprocess.run(
48
+ ["make", "clean", "all"],
49
+ cwd=str(NATIVE_DIR),
50
+ check=True,
51
+ )
52
+ except subprocess.CalledProcessError as exc:
53
+ raise RuntimeError(
54
+ f"Native build failed (rc={exc.returncode}). "
55
+ "simdrive requires Xcode + macOS to compile its HID helper."
56
+ ) from exc
57
+ if not BINARY_PATH.exists():
58
+ raise RuntimeError(
59
+ f"Native build reported success but binary not found at {BINARY_PATH}"
60
+ )
61
+ print(f"simdrive: built {BINARY_PATH}")
62
+
63
+
64
+ setup(cmdclass={"build_py": BuildPyWithNative})
@@ -0,0 +1,3 @@
1
+ """simdrive — hand your iOS simulator to your agent."""
2
+
3
+ __version__ = "0.2.0a1"
@@ -0,0 +1,245 @@
1
+ """Synthetic input dispatch.
2
+
3
+ Internal module. Coordinates passed in are screenshot pixels from the most
4
+ recent observe; translated to logical iOS device points using the cached
5
+ scale, then dispatched via the HID-injection backend.
6
+
7
+ Three backends, in preference order:
8
+ 1. hid — bundled native helper (real UITouch events; focuses TextFields)
9
+ 2. cliclick — synthetic mouse via the macOS window (fallback)
10
+ """
11
+ from __future__ import annotations
12
+
13
+ import os
14
+ import subprocess
15
+ import time
16
+ from typing import Iterable, Optional
17
+
18
+ from . import hid_inject, sim
19
+ from .sim import cliclick_path
20
+ from .window import WindowBounds, activate, get_bounds
21
+
22
+
23
+ class ActError(RuntimeError):
24
+ """Raised when an act dispatch fails."""
25
+
26
+
27
+ # Cache of UDID → (logical_w, logical_h, scale) from hid_inject.device_size_points()
28
+ _DEVICE_GEOM_CACHE: dict[str, tuple[float, float, float]] = {}
29
+
30
+
31
+ def _backend() -> str:
32
+ """Return which backend will be used. Override via SIMDRIVE_INPUT_BACKEND."""
33
+ requested = os.environ.get("SIMDRIVE_INPUT_BACKEND", "").lower()
34
+ if requested == "cliclick":
35
+ return "cliclick"
36
+ if hid_inject.available():
37
+ return "hid"
38
+ return "cliclick"
39
+
40
+
41
+ def _device_geom(udid: str) -> tuple[float, float, float]:
42
+ cached = _DEVICE_GEOM_CACHE.get(udid)
43
+ if cached is not None:
44
+ return cached
45
+ geom = hid_inject.device_size_points(udid)
46
+ _DEVICE_GEOM_CACHE[udid] = geom
47
+ return geom
48
+
49
+
50
+ def _pixels_to_points(udid: str, pixel_x: int, pixel_y: int, screenshot_w: int, screenshot_h: int) -> tuple[float, float]:
51
+ """Map a screenshot pixel coord to logical iOS device points for the HID path."""
52
+ if screenshot_w <= 0 or screenshot_h <= 0:
53
+ raise ActError(f"Invalid screenshot size: {screenshot_w}x{screenshot_h}")
54
+ logical_w, logical_h, _scale = _device_geom(udid)
55
+ px = (pixel_x / screenshot_w) * logical_w
56
+ py = (pixel_y / screenshot_h) * logical_h
57
+ return px, py
58
+
59
+
60
+ def _pixels_to_screen(
61
+ bounds: WindowBounds, pixel_x: int, pixel_y: int, screenshot_w: int, screenshot_h: int
62
+ ) -> tuple[int, int]:
63
+ if screenshot_w <= 0 or screenshot_h <= 0:
64
+ raise ActError(f"Invalid screenshot size: {screenshot_w}x{screenshot_h}")
65
+ sx = bounds.x + (pixel_x / screenshot_w) * bounds.width
66
+ sy = bounds.y + (pixel_y / screenshot_h) * bounds.height
67
+ return int(round(sx)), int(round(sy))
68
+
69
+
70
+ def _run_cliclick(args: Iterable[str], timeout: float = 5.0) -> None:
71
+ cli = cliclick_path()
72
+ cmd = [cli, *args]
73
+ res = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, check=False)
74
+ if res.returncode != 0:
75
+ raise ActError(f"cliclick failed (rc={res.returncode}): {res.stderr.strip() or res.stdout.strip()}")
76
+
77
+
78
+ # ----------------------------- Public API ------------------------------ #
79
+
80
+
81
+ def tap(pixel_x: int, pixel_y: int, screenshot_w: int, screenshot_h: int, udid: Optional[str] = None) -> tuple[int, int]:
82
+ """Click at screenshot-pixel coordinates. Returns the macOS screen coords used (or 0,0 for HID path)."""
83
+ if _backend() == "hid" and udid:
84
+ x_pt, y_pt = _pixels_to_points(udid, pixel_x, pixel_y, screenshot_w, screenshot_h)
85
+ hid_inject.tap(udid, x_pt, y_pt)
86
+ return (0, 0)
87
+
88
+ bounds = get_bounds()
89
+ sx, sy = _pixels_to_screen(bounds, pixel_x, pixel_y, screenshot_w, screenshot_h)
90
+ activate()
91
+ time.sleep(0.15)
92
+ _run_cliclick([f"c:{sx},{sy}"])
93
+ return sx, sy
94
+
95
+
96
+ def swipe(
97
+ x1: int, y1: int, x2: int, y2: int, screenshot_w: int, screenshot_h: int, duration_ms: int = 300,
98
+ udid: Optional[str] = None,
99
+ ) -> None:
100
+ if _backend() == "hid" and udid:
101
+ x1p, y1p = _pixels_to_points(udid, x1, y1, screenshot_w, screenshot_h)
102
+ x2p, y2p = _pixels_to_points(udid, x2, y2, screenshot_w, screenshot_h)
103
+ steps = max(4, duration_ms // 25)
104
+ hid_inject.swipe(udid, x1p, y1p, x2p, y2p, steps=steps, step_delay_ms=25)
105
+ return
106
+
107
+ bounds = get_bounds()
108
+ sx1, sy1 = _pixels_to_screen(bounds, x1, y1, screenshot_w, screenshot_h)
109
+ sx2, sy2 = _pixels_to_screen(bounds, x2, y2, screenshot_w, screenshot_h)
110
+ activate()
111
+ time.sleep(0.15)
112
+ duration_ms = max(50, min(duration_ms, 5000))
113
+ steps = max(2, duration_ms // 30)
114
+ moves: list[str] = []
115
+ for i in range(1, steps + 1):
116
+ t = i / steps
117
+ mx = int(round(sx1 + (sx2 - sx1) * t))
118
+ my = int(round(sy1 + (sy2 - sy1) * t))
119
+ moves.append(f"m:{mx},{my}")
120
+ args = ["-w", "30", f"dd:{sx1},{sy1}", *moves, f"du:{sx2},{sy2}"]
121
+ _run_cliclick(args, timeout=10.0)
122
+
123
+
124
+ def type_text(text: str, udid: Optional[str] = None) -> None:
125
+ """Send keystrokes. Caller is responsible for tapping a focused field first.
126
+
127
+ For non-ASCII characters (accented, emoji, non-Latin), falls back to the
128
+ pasteboard path: simctl pbcopy + Cmd-V — preserves the focused-field state
129
+ and works around the HID keyboard's US-ASCII-only key map.
130
+ """
131
+ if not text:
132
+ return
133
+
134
+ is_ascii = all(ord(c) < 128 for c in text)
135
+
136
+ if _backend() == "hid" and udid:
137
+ if is_ascii:
138
+ hid_inject.type_text(udid, text)
139
+ return
140
+ # Non-ASCII path: pbcopy + paste-shortcut
141
+ sim.set_pasteboard(udid, text)
142
+ time.sleep(0.05)
143
+ # Cmd-V via HID — issue Cmd modifier hold + V keypress
144
+ # HID usage 0xE3 = Left Cmd; 0x19 = V
145
+ _hid_paste(udid)
146
+ return
147
+
148
+ activate()
149
+ time.sleep(0.15)
150
+ _run_cliclick(["t:" + text])
151
+
152
+
153
+ def _hid_paste(udid: str) -> None:
154
+ """Cmd-V via the HID helper — works in background mode."""
155
+ hid_inject.chord(udid, "cmd", "v")
156
+
157
+
158
+ _CLICLICK_KEY_MAP = {
159
+ "return": "kp:return",
160
+ "enter": "kp:return",
161
+ "tab": "kp:tab",
162
+ "escape": "kp:esc",
163
+ "esc": "kp:esc",
164
+ "space": "kp:space",
165
+ "delete": "kp:delete",
166
+ "backspace": "kp:delete",
167
+ "arrow-up": "kp:arrow-up",
168
+ "arrow-down": "kp:arrow-down",
169
+ "arrow-left": "kp:arrow-left",
170
+ "arrow-right": "kp:arrow-right",
171
+ }
172
+
173
+ # HID usage codes for special keys (US layout, HID Keyboard/Keypad page)
174
+ _HID_KEY_MAP = {
175
+ "return": 40,
176
+ "enter": 40,
177
+ "tab": 43,
178
+ "escape": 41,
179
+ "esc": 41,
180
+ "space": 44,
181
+ "delete": 42,
182
+ "backspace": 42,
183
+ "arrow-up": 82,
184
+ "arrow-down": 81,
185
+ "arrow-left": 80,
186
+ "arrow-right": 79,
187
+ }
188
+
189
+ _DEVICE_BUTTONS = {"home", "lock", "siri"} # buttons routed to hid_inject.press_button
190
+
191
+ # Sim-only buttons that go through Simulator's "Device" menu (cliclick fallback path).
192
+ _DEVICE_MENU_KEYS = {
193
+ "home": "Home",
194
+ "lock": "Lock",
195
+ "shake": "Shake",
196
+ "siri": "Siri",
197
+ "app-switcher": "App Switcher",
198
+ "screenshot": "Trigger Screenshot",
199
+ "rotate-left": "Rotate Left",
200
+ "rotate-right": "Rotate Right",
201
+ "action-button": "Action Button",
202
+ }
203
+
204
+
205
+ def press_key(key: str, udid: Optional[str] = None) -> None:
206
+ key_lower = key.lower().strip()
207
+
208
+ if _backend() == "hid" and udid:
209
+ if key_lower in _DEVICE_BUTTONS:
210
+ hid_inject.press_button(udid, key_lower)
211
+ return
212
+ hid_code = _HID_KEY_MAP.get(key_lower)
213
+ if hid_code is not None:
214
+ hid_inject.press_key(udid, hid_code)
215
+ return
216
+ # fall through to cliclick path for unknown keys
217
+
218
+ if key_lower in _DEVICE_MENU_KEYS:
219
+ _menu_click("Device", _DEVICE_MENU_KEYS[key_lower])
220
+ return
221
+
222
+ cli_arg = _CLICLICK_KEY_MAP.get(key_lower)
223
+ if cli_arg is None:
224
+ raise ActError(
225
+ f"unsupported key: {key!r}. Supported: {sorted(_CLICLICK_KEY_MAP)} "
226
+ f"+ {sorted(_DEVICE_MENU_KEYS)}"
227
+ )
228
+ activate()
229
+ time.sleep(0.15)
230
+ _run_cliclick([cli_arg])
231
+
232
+
233
+ def _menu_click(menu_title: str, item_title: str) -> None:
234
+ script = f'''
235
+ tell application "System Events"
236
+ tell process "Simulator"
237
+ click menu item "{item_title}" of menu "{menu_title}" of menu bar 1
238
+ end tell
239
+ end tell
240
+ '''
241
+ res = subprocess.run(
242
+ ["osascript", "-e", script], capture_output=True, text=True, timeout=5.0, check=False
243
+ )
244
+ if res.returncode != 0:
245
+ raise ActError(f"menu click {menu_title} > {item_title} failed: {res.stderr.strip()}")