device-devtools-mcp 0.1.0

package/AGENTS.md

@@ -0,0 +1,327 @@
+# DeviceTools
+
+You are operating a real application on a real phone, plugged into this Mac.
+There is no person watching the screen for you.
+
+This file is the contract. It is the only document you need; the README is for
+whoever set the machine up.
+
+## The one rule
+
+**Read before you act, and act on what you read.**
+
+`snapshot` is cheap and exact. Guessing where a control is — from a screenshot,
+or from memory of the last screen — is the one failure that matters here. A
+wrong tap is not a failed assertion. It is a thing that happened, on somebody's
+phone, and every step you take afterwards reasons about a screen you are no
+longer on.
+
+When a verb refuses, it is refusing because it could not act unambiguously. The
+answer is another `snapshot`, not another attempt.
+
+## Start here, every session
+
+```sh
+./scripts/doctor.sh
+```
+
+A fixed list of checks, always the same ones in the same order,
+`OK|WARN|FAIL <name> — <detail>`. Exit 0 means you can work. Run it again the
+moment any verb exits **3**: that code means the connection to the device died,
+and nothing you do afterwards is real until it is back.
+
+If it fails with the driver missing, run `./scripts/setup.sh` once and doctor
+again. Do not try to install the driver by hand.
+
+Some failures are not yours to fix. Signing in to Xcode, registering the device
+with Apple, trusting a certificate on the phone, and turning on Settings →
+Developer → Enable UI Automation are all things a person has to do while holding
+the device. When doctor says one of those, stop and ask — retrying will not
+change the answer, and neither will installing anything.
+
+## The loop
+
+```
+you edit code
+you run xcodebuild / gradlew        ← your own shell. DeviceTools knows nothing
+                                       about Xcode, Gradle or signing
+app relaunch                        ← the app is back on its first screen
+waypoint action=goto name=…         ← walk back to where you were
+snapshot                            ← what is on it, with the geometry
+tap 7 snapshot=true                 ← act, and get the new screen in the same call
+screenshot                          ← only when the tree cannot answer
+```
+
+**Two things collapse that loop, and you should reach for both.**
+
+`snapshot=true` on any action prints the screen it produced and renumbers the
+uids, so the read you were about to do is already done. It costs nothing extra:
+the action reads the tree anyway to tell you whether the screen moved.
+
+`run` takes several steps in one call. What a person does as one continuous
+act — type the password, hide the keyboard, press submit — is one call, not
+four. The number of calls is what this loop costs, not the size of any one
+answer:
+
+```
+run steps='[{"tap":"label:Đăng nhập"},{"type":"{{ password }}"},
+            {"key":"hide"},{"tap":"label:Xác nhận"},{"wait":"screen:Home"}]'
+     vars='{"password":"…"}'
+```
+
+It stops at the first failing step and shows you the screen it stopped on. uids
+are refused inside a batch: a uid names a position on a screen that is already
+two steps old by step three, so use selectors. `snapshot: true` inside a step is
+a flag on that step, not a step of its own. Start a batch with
+`{"expect": "screen:X"}` when you know where it should be beginning — it is a
+wait with a three-second deadline, and it fails at the door instead of tapping
+into the wrong screen.
+
+`flow save <name>` keeps a batch under a name, in the project. If you have just
+worked out how to log in, save it — the next session says `flow run login`.
+
+**Name the project.** You have no working directory: whatever the host set when
+it spawned this server is fixed for the whole session, and it is the DeviceTools
+checkout, not the repository the person is working in. So `init` takes the
+repository path once, and every verb after that takes `project=<that path>`.
+Without it a saved flow lands in the checkout, where every app on the machine
+shares it and nobody reviews it.
+
+DeviceTools owns everything from **"the app is installed"** onward, and nothing
+before it. Building and installing is your shell's job, and you are better at it
+than a wrapper here would be.
+
+## Verbs
+
+`./scripts/<verb>.sh`, or `devicetools <verb>` where the command is on PATH — the
+same program, same arguments, same output, same exit codes. Use whichever
+spelling the pointer in this repository gave you and do not mix them.
+
+| Verb | Use |
+| --- | --- |
+| `doctor` | can this machine drive the device right now |
+| `devices` | what is attached, and which one the config selects |
+| `snapshot [--all] [--grep S]` | **read the screen.** The default way to see |
+| `screenshot [name] [--scale N] [--full]` | an image, when the tree cannot answer |
+| `measure <uid>` | one element in full: parent, neighbours, gaps, overlaps |
+| `tap <uid\|selector> [--index N] [--count N]` | tap exactly one thing, or refuse. `--count` repeats it |
+| `type <text>` | type into whatever has focus, and verify it landed |
+| `swipe <direction>` | a gesture between two points |
+| `scroll <direction> [--until S]` | scroll, optionally until something appears |
+| `key hide` | put the keyboard away |
+| `back` | one screen back; fails if the screen does not move |
+| `wait <selector> [--gone]` | until something appears or disappears |
+| `open <url> [--expect-app <id>]` | a deep link, and which app answered |
+| `app <launch\|relaunch\|kill\|reset\|state>` | the app's lifecycle, by bundle id — no icon to find |
+| `app install <path> [--launch]` | put a build on the device the config selects |
+| `app uninstall [id]` | remove it |
+| `logs [start\|stop\|status] [--network] [--grep S]` | what the app printed |
+| `waypoint <mark\|goto\|list\|forget> <name>` | get back after a reinstall |
+| `settings <key> <value>` | dark mode, text size, orientation |
+| `permission <allow\|deny\|reset> <name>` | a system permission |
+| `find <selector>` | resolve a selector without acting on it |
+| `run --steps '[…]'` | **several steps in one call.** The biggest thing you can do about cost |
+| `flow <save\|run\|list\|show\|delete> <name>` | a batch, kept under a name, in the project |
+
+Every acting verb takes `--snapshot` (`snapshot=true` over MCP): it prints the
+screen the action produced and renumbers the uids, in the same call.
+
+`tree` and `controls` still exist and are what `snapshot` was built from. Prefer
+`snapshot`: it carries the geometry and assigns the numbers everything else
+takes.
+
+## Reading a screen
+
+```
+$ snapshot
+SCREEN "Liên hệ" #9b41  428x926  portrait
+
+[1]  Application "DemoApp"  (0,0)  428x926
+[2]  NavigationBar #Liên hệ  (0,47)  428x101
+[3]  Button  (20,51)  36x36  enabled
+[4]  StaticText "Liên hệ"  (19,108)  116x31
+…
+[10]  TextField "Tìm kiếm"  (72,368)  316x25  enabled
+
+⚠ hit area below the 44pt minimum — [3] [10] [19] [23] [27] [31] [35] [39]
+⚠ no identifier: a selector has to use the label, which changes with the copy — [3] [10] [19] [23] [27] [31] [35] [39]
+⚠ partly off screen by 8pt — [19] [23] [27] [31] [35] [39]
+```
+
+**The first line identifies the screen.** `#9b41` is a short hash over the set
+of (type, label) pairs on it. It is not a URL and not a class name — XCUITest
+and UiAutomator publish neither — but it is stable while the screen is at rest,
+which is what change detection needs.
+
+**The number in brackets is a uid, and it dies at the next `snapshot`.** Acting
+on one re-reads the screen and requires that element to still be exactly what
+was numbered: same type, same identifier, same label, same value, same rectangle
+give or take two points. If it is not, the action is refused and says what
+changed. Nothing is scored, and there is no runner-up.
+
+**Warnings are grouped by fault, not printed under each element.** A list of six
+identical rows would otherwise report the same thing six times, and an agent
+that reads fifty warnings per screen stops reading warnings. To find out which
+sibling `[10]` overlaps, ask `measure 10` — that is what it is for.
+
+The whole list of warnings, and nothing beyond it:
+
+| Warning | Computed from |
+| --- | --- |
+| `overlaps a sibling by Xpt` | two sibling rectangles intersect |
+| `outside its parent by Xpt` | a child's rect is not contained by its parent's |
+| `off screen` / `partly off screen by Xpt` | the rect against the window size |
+| `zero size, but it carries text` | width or height is 0 on an element with text |
+| `hit area below the Npt minimum` | 44pt on iOS, 48dp on Android |
+| `the OS truncated this text with an ellipsis` | the string ends in `…` or `...` |
+
+Nothing at or below **two points** is reported at all. Both drivers hand back
+floating-point rectangles and this pipeline floors them, so any pair of edges
+carries up to two points of error; without that floor, every pair of stacked
+text lines "overlaps by 1pt". Measured on a real screen: sixty-three warnings
+became eight, and the eight were true.
+
+**There is no "this text is clipped, it needs 244pt".** That needs font metrics,
+which neither driver exposes. Guessing a string's width from its length would be
+wrong often enough that you would learn to skip the warnings, and a signal you
+skip is worse than one that was never printed. When the OS truncates a string it
+leaves an ellipsis, and that is what gets reported.
+
+## Looking at a screen
+
+```sh
+screenshot            # longest edge 768px, about 360 tokens
+screenshot --full     # device resolution, about 1500
+```
+
+Use it for what the tree cannot say: a wrong icon, a blurry asset, a colour, a
+contrast, a gradient. Neither driver reports any of those, so looking is the
+only way to know.
+
+For anything about position, size or text, `snapshot` is cheaper and exact.
+**The normal loop contains no images at all.**
+
+Two things to know before you read one. Apps that mark secure inputs render them
+as blank regions in any captured image — the pixels are genuinely absent, and a
+blank field is not evidence it failed to draw; the verb warns when a text input
+is on screen. And the image leaves this machine: it goes to your model provider,
+with no masking. That is a deliberate trade for an internal tool on a
+developer's own device, and it is written here rather than left to be
+discovered.
+
+## Selectors
+
+| Spelling | Means |
+| --- | --- |
+| `7` | uid — the seventh thing the last snapshot showed |
+| `id:<s>` | iOS accessibilityIdentifier, Android resource-id |
+| `label:<s>` | iOS accessibility label, Android content-desc |
+| `text:<s>` | iOS value, Android text |
+| `xy:<x>,<y>` | a raw point, checked against nothing |
+| anything else | a plain string, matched against id, label and value in turn |
+
+The first kind that matches wins — not the union of all three. An identifier
+that matches exactly is a better answer than two labels that also do, and
+merging them would manufacture an ambiguity the screen does not have.
+
+`id:`, `label:` and `text:` must resolve to **exactly one** visible element
+unless `--index` says which. Zero matches, or several without an index, and the
+verb exits non-zero naming what it saw, without touching anything.
+
+`kind:<Type>` matches the element's class — the word `snapshot` prints first on
+the line. It is what to use for a control that has no identifier and no label,
+usually with `--index`, instead of falling back to `xy:`. Coordinates break: on
+this app, hiding the keyboard moved a sheet from y=507 to y=774 and every
+recorded tap in it missed.
+
+**Every selector matches whole.** `text:OTP` does not find a label reading
+"Vui lòng nhập mã OTP…"; `snapshot --grep OTP` does. Reading may guess, acting
+may not.
+
+## What the app said
+
+```sh
+logs start        # once per session
+… act …
+logs --grep error
+logs --network
+```
+
+Until you start a collector, actions carry no `LOG` section and `logs` has
+nothing to read. That is deliberate: an action must not spawn a background
+process as a side effect of touching a button, and the device under test
+produced twenty-one thousand log lines in thirty seconds.
+
+`--network` is the Network panel, **filtered from the app's own log output and
+not from a proxy**. A proxy needs a CA certificate on the device, and an app
+that pins its certificates defeats one. So this shows what the app chose to
+print: no bodies, no headers it did not log, and nothing at all from an app that
+prints none of it. On the app this was built against it returns nothing — an
+empty result here means the app logs no traffic, not that the network was
+quiet.
+
+## After a rebuild
+
+Installing a new build puts the app back on its first screen. Every action that
+lands is appended to `.state/session.jsonl`, so:
+
+```sh
+waypoint mark otp     # name where you are
+… rebuild, reinstall …
+waypoint goto otp     # relaunch and walk back
+```
+
+`goto` replays the recorded actions, preferring the recorded identifier over the
+recorded label — a label is application copy and gets rewritten, an identifier
+is chosen by an engineer and rarely does. When a step no longer resolves it
+**stops** and says which step and what it was looking for. It does not search,
+score or improvise. Read the screen and walk the rest yourself.
+
+## What every action tells you
+
+```
+TAP    7 — resolved by uid 7, verified still at 214,69 → tapped 214,69
+SCREEN changed — "Trang chủ" #60a8 → "Xác thực" #abf8
+LOG    2 lines
+       [error] AuthService: token refresh failed
+```
+
+Three facts: what was done, whether the screen moved, and what the app said
+while it moved. Not the new tree by default — attaching one to every action
+would triple the cost of a loop that mostly does not need it, and "changed" or
+"unchanged" already answers the usual question.
+
+Ask for it with `--snapshot` when the next thing you would do is read the
+screen, which is most of the time while you are finding your way. It is free:
+the action already fetched that tree to decide whether the screen moved.
+
+`SCREEN same — #x, contents changed` is a fourth: the same controls holding
+different text. A code being entered, a counter moving, a field filling up. Not
+a new screen, and not nothing.
+
+`SCREEN unknown` means the tree could not be read on one side or the other. It
+is not the same claim as "unchanged".
+
+## Exit codes
+
+| Code | Meaning |
+| --- | --- |
+| 0 | done |
+| 1 | usage error — you called it wrong |
+| 2 | environment or configuration problem |
+| 3 | the device is unreachable. **Run `doctor`.** Nothing after this is real |
+| 4 | the command ran, and the screen did not satisfy it |
+
+There is no 5. It used to mean "tests failed" and retired with the test runner.
+
+Exit 4 is an ordinary outcome, not a fault: zero matches, several matches, a
+condition that never held, a uid that has gone stale. Read the message. It says
+what was on the screen.
+
+## Where things are
+
+`DT_HOME` is the checkout: scripts, adapters, this file. `DT_ROOT` is the
+project you are working in, found by walking up for a `.devicetools` marker;
+`config.json`, `.state/` and `.runs/` live there. Every relative path in a
+config resolves against `DT_ROOT`, and nothing you author ever resolves into
+`DT_HOME`.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hưng Phan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.