@flxapps/adbax 0.0.0-stage → 0.1.0

package/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/NOTICE

@@ -0,0 +1,2 @@
+adbax
+Copyright 2026 Felix Heller

package/README.md

@@ -1,3 +1,133 @@
-# Temporary Holding Version
+# adbax
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+adbax drives an Android phone or emulator by its accessibility tree. Run
+`adbax ui` and you get the screen as a numbered list of labels with their
+positions; `adbax tap "Sign in"` then taps whatever is labelled Sign in. You
+never squint at a screenshot or guess a coordinate. The name is adb plus AX, a
+common shorthand for accessibility.
+
+I built it for Claude Code, after watching it screenshot the emulator, estimate
+where a button was and tap beside it, again and again. Typed by hand or run from
+a shell script, the commands behave the same.
+
+```console
+$ adbax ui
+# com.android.settings/.Settings   1080x1920   11 nodes
+  [0]  TAP     "Search settings"  id=search_action_bar  @540,147  1038x126
+  [5]  TAP     "Network & Internet / Wi‑Fi, mobile, data usage"  id=dashboard_tile  @540,903  1080x189
+  [6]  TAP     "Connected devices / Bluetooth, Cast"  id=dashboard_tile  @540,1092  1080x189
+  ...
+$ adbax tap "Network & Internet" --ui
+tapped TAP "Network & Internet" id=dashboard_tile @540,903
+# com.android.settings/.Settings$NetworkDashboardActivity   1080x1920   12 nodes
+  [0]  TAP     "Navigate up"  @74,137  147x147
+  ...
+```
+
+## Install
+
+You need Node 18 or later and adb, either on the PATH or in the SDK that
+`ANDROID_HOME` points to. adbax itself has zero dependencies.
+
+```sh
+npm install -g @flxapps/adbax
+```
+
+npm wouldn't take the bare name, hence the scope. The command is `adbax`.
+
+## Usage
+
+```sh
+adbax ui                                # the screen; --all for every node, --json for scripts
+adbax tap "Sign in"                     # by text, content-desc or resource-id
+adbax tap 5                             # row 5 of the last `ui`, no fresh read
+adbax tap "Sign in" --ui                # act, then print the screen it led to
+adbax type you@example.com --into Email --enter
+adbax scroll-to "Delete account" --tap
+adbax wait "Welcome" --timeout 20
+adbax launch com.example.app            # returns once the activity is drawn
+adbax screenshot --small                # 540 px wide JPEG
+```
+
+Every invocation is a separate process, which means steps string together with
+`&&`.
+
+Matching ignores case and prefers an exact hit, then a prefix, then a
+substring. If what matched can't be tapped itself, like the title text inside a
+list row, adbax goes up to the closest clickable parent. When a selector fits
+more than one node, adbax won't choose for you. It lists them and exits with
+status 2:
+
+```console
+$ adbax tap "Play Store"
+adbax: "Play Store" matches 2 nodes:
+  --nth 0  TAP "Play Store" @158,1679
+  --nth 1  TAP "Play Store" @923,2003
+       Narrow the query, or add --nth N / --first.
+```
+
+A game, or anything else painted onto a canvas, exposes no tree at all. There,
+`adbax screenshot --grid` draws a labelled coordinate grid over the image and
+`adbax tap 270,480 --shot` takes coordinates in that image's own pixels. Whoever
+reads the picture, usually a model, taps what it sees without converting to
+screen size.
+
+Dark mode, font scale, logcat, the environment variables and every other
+command are in [docs/commands.md](docs/commands.md).
+
+## The on-device server
+
+Reading the screen with `uiautomator dump` costs roughly two seconds, even when
+there's hardly anything on it. Each run boots a fresh VM, and since a brand-new
+connection has no idea when the screen last changed, it then sits through a
+whole second of waiting for things to go idle.
+
+adbax gets around that. The first time you use it, it copies a 13 KB jar to
+`/data/local/tmp` and launches it through `app_process` as the shell user:
+there's no APK, and nothing asks for permission. That server holds one
+accessibility connection for as long as it runs, so it always knows when the
+screen last moved. Measured end to end on an arm64 Android 13 emulator,
+`adbax ui` goes from 2.2 s down to 0.21-0.27 s, and most of what's left is Node
+starting up. Only the very first call is slower, about a second, because it
+starts the server. Taps, keys and swipes travel over that connection too, and
+after each one adbax waits until the accessibility events die down rather than
+sleeping for some fixed delay.
+
+The server wants Android 7.0 (API 24) or later, and I've tried it on 8.1, 13
+and 17. Older devices, or `--no-server`, get the same commands on top of
+`uiautomator dump` and `input`, at the old speed.
+
+There is one catch. Android hands out a single UiAutomation connection at a
+time, so while the server holds it, UiAutomator and screenshot tests, Appium and
+Maestro are locked out. After ten minutes without a request it exits on its
+own, and `adbax server stop` releases it immediately.
+
+The internals are written up in [docs/architecture.md](docs/architecture.md).
+
+## Compared with an MCP server
+
+Shell commands can be chained. For an agent,
+`adbax launch com.example.app && adbax wait Start && adbax tap Settings --ui` is a
+single tool call. Through an MCP server the same steps take four calls, and
+every one of them goes back through the model. You can also type the identical
+commands yourself, put them in CI, or hand them to any agent with shell access,
+with no client to configure.
+
+The price is that you need the shell. Chat apps that don't have one can't run
+adbax at all. There's no schema either, so an agent picks the commands up from
+`adbax help` or from the skill below. And a screenshot arrives as a path the
+agent has to open, whereas an MCP tool returns the picture directly.
+
+## With Claude Code
+
+[`skill/SKILL.md`](skill/SKILL.md) shows an agent how to work with it: read the
+tree, act on labels, chain steps, and keep screenshots for screens that have
+nothing to read.
+
+```sh
+ln -s "$(npm root -g)/@flxapps/adbax/skill" ~/.claude/skills/android-device
+```
+
+## License
+
+Apache 2.0, see [LICENSE](LICENSE).

package/bin/adbax.js

@@ -0,0 +1,15 @@
+#!/usr/bin/env node
+// SPDX-License-Identifier: Apache-2.0
+import { main } from '../src/cli.js';
+
+main(process.argv.slice(2)).then(
+  (code) => process.exit(code),
+  (err) => {
+    if (typeof err.code === 'number') {
+      process.stderr.write(`adbax: ${err.message}\n`);
+      process.exit(err.code);
+    }
+    process.stderr.write(`adbax: ${err.stack || err}\n`);
+    process.exit(1);
+  },
+);

package/docs/architecture.md

@@ -0,0 +1,166 @@
+# How adbax works
+
+Two halves: a Node CLI on the host, and a small Java server that runs on the
+device. The CLI works without the server, just slower.
+
+```
+bin/adbax.js              entry point, turns errors into exit codes
+src/cli.js               command table and help
+src/args.js              argv into command, positionals and flags
+src/adb.js               finding adb, choosing the device, running adb
+src/device.js            one device: picks a backend, reads the tree, acts
+src/backends/server.js   host half of the on-device server
+src/backends/shell.js    the fallback: uiautomator dump and `input`
+src/tree.js              XML into nodes, nodes into rows, rows into text
+src/select.js            matching a selector against the tree
+src/commands/*.js        one file per group of commands
+server/src/              the on-device server
+server/build.sh          javac + d8 into server/adbax-server.jar
+```
+
+Commands only ever talk to a `Device`. Which backend sits behind it is decided
+once per call, the first time it is needed.
+
+## Why `uiautomator dump` is slow
+
+`adb shell uiautomator dump` runs `/system/framework/uiautomator.jar` through
+`app_process`, so every call:
+
+1. starts a VM (a few hundred ms, more on old releases),
+2. registers a new UiAutomation connection with the accessibility manager,
+3. calls `waitForIdle(1000, 10000)`. A fresh connection has never seen an
+   event, so it counts "idle" from the moment it connected and always waits the
+   full second,
+4. dumps the active window to a file and exits.
+
+Steps 1 and 3 are almost all of the two seconds, and both exist only because
+the process is new.
+
+## The server
+
+`server/src/es/flx/adbax` is about 870 lines of Java, compiled against
+`android.jar` and dexed with d8. adbax pushes the jar to
+`/data/local/tmp/adbax-server.jar` and starts it:
+
+```sh
+CLASSPATH=/data/local/tmp/adbax-server.jar nohup app_process / --nice-name=adbax-server \
+    es.flx.adbax.Main --socket adbax --token <jar md5> --serial <adb serial> --idle 600
+```
+
+It runs as the shell user, the same as `uiautomator`, so it may inject input and
+read every window, and it needs no APK and no permission prompt.
+
+- `Automation` creates and connects the UiAutomation through hidden API, by
+  reflection. The constructor takes a Looper up to Android 13 and may take a
+  Context later, so both are tried. It connects with
+  `FLAG_DONT_SUPPRESS_ACCESSIBILITY_SERVICES`, so TalkBack keeps running. The
+  same class finds the top activity: `getStackInfo(getFocusedStackId())` up to
+  8.1, `getFocusedStackInfo` from 9, on IActivityTaskManager from 10,
+  `getFocusedRootTaskInfo` from 12.
+- `Hierarchy` writes the tree in the format `uiautomator dump` uses: the same
+  attributes in the same order, the same rule of skipping children that are not
+  visible to the user, bounds clipped to the display. The host parses both with
+  one parser, and the fallback prints the same rows.
+- `Input` injects touches and keys through the connection, and types through
+  the virtual keyboard's key map, falling back to `ACTION_SET_TEXT` for
+  characters the map does not have.
+- `Grid` draws the screenshot grid. Its labels are stroked seven-segment digits:
+  an `app_process` has no font setup, and drawing text on Android 8 aborts it.
+- `Server` listens on the abstract unix socket `@adbax`.
+
+### Protocol
+
+One request per connection. The host writes one line of JSON and the server
+answers with one line of JSON, then the payload (XML or image bytes) until it
+closes the socket.
+
+```
+> {"cmd":"dump","settle":150,"idle":300,"timeout":3000}
+< {"ok":true,"busy":false,"activity":"com.android.settings/.SubSettings","width":1080,
+   "height":2400,"rotation":0,"waitMs":212,"dumpMs":18,"token":"7fa7…","serial":"emulator-5554"}
+< <?xml version='1.0' …><hierarchy …>…</hierarchy>
+```
+
+| cmd | |
+| --- | --- |
+| `ping` | sdk, pid, uptime |
+| `dump` | `settle`, `idle`, `timeout`, `windows` |
+| `screenshot` | `width`, `format` (`png`/`jpeg`), `quality`, `grid` |
+| `tap` | `x`, `y` |
+| `swipe` | `x1`, `y1`, `x2`, `y2`, `ms` |
+| `key` | `code` or `name`, `long` |
+| `text` | `text`: key events, or `ACTION_SET_TEXT` for what the key map lacks |
+| `settext` | `text`, `append`: always `ACTION_SET_TEXT` |
+| `focus` | the node with input focus: editable, text, password |
+| `activity` | the top activity |
+| `quit` | disconnect and exit |
+
+Requests are served one at a time. A client that connects and sends nothing is
+dropped after 10 s.
+
+### Reading after an action
+
+A plain `dump` waits up to 2 s for 300 ms without accessibility events. On a
+settled screen the last event is long past and that returns at once. That is
+the part a fresh connection cannot do.
+
+After an action the host sends `settle: 150`. The server sleeps that long so
+the app has started reacting, waits for the events to stop, and then compares
+the top activity's package with the package of the active window. When they
+differ, an activity switch has not reached the accessibility tree yet, and it
+keeps waiting until they agree or `timeout` runs out. It only checks that after
+an action: with the notification shade or a system dialog open the two disagree
+for as long as it stays open, and a plain read should not wait that out.
+
+One gap remains: a new activity of the same package can go quiet before its
+content is attached, which reads as an empty screen. If a read after an action
+has no rows, the host reads again every 250 ms until something shows or two
+reads come back identical. A game's canvas has no rows either, and two
+identical reads end the loop for it after a quarter of a second.
+
+### The node cache
+
+The framework caches accessibility nodes per connection and invalidates them
+from accessibility events. A long-lived connection depends on every toolkit
+sending the right events, and they do not: a Compose text field kept reporting
+the text of three edits ago. So the server clears the cache before every dump
+and before every look at the focused field. `UiAutomation.clearCache()` exists
+from Android 14; before that it is `AccessibilityInteractionClient.clearCache(id)`,
+which is what `setServiceInfo()` calls first. On a settled screen a full read
+still takes 10-15 ms.
+
+### Lifecycle and versioning
+
+- **Start.** On the first command that needs it. The host pushes the jar when
+  the MD5 on the device differs, kills any older server (only one UiAutomation
+  client may exist), starts the new one and polls `ping` until it answers.
+- **Version.** The jar's MD5 is passed as `--token` and echoed in every reply. A
+  server answering with another token is an older build, so the host sends it
+  `quit` and starts the current one.
+- **Port.** `adb forward tcp:0 localabstract:adbax` picks a free port, which is
+  cached in `~/.cache/adbax/port-<serial>` to save an `adb forward --list` per
+  call. The server echoes its `--serial`, so a cached port that adb has since
+  given to another device is noticed rather than used.
+- **Exit.** After `--idle` seconds without a request (600 by default), on
+  `quit`, or when killed. The UiAutomation registration dies with the process.
+- **Failure.** A server that cannot start on a device (below API 24, or a stack
+  trace in its log) is recorded in `~/.cache/adbax/server-<serial>.json` against
+  the jar's MD5, and adbax stays on the shell backend for that device until the
+  jar changes or `adbax server start` asks again. If another UiAutomation client
+  holds the connection, adbax says so and does not record anything.
+
+### Interactive windows
+
+`getWindows()` only works with `FLAG_RETRIEVE_INTERACTIVE_WINDOWS` set. With that
+flag on, the framework also reports views scrolled underneath other views as
+invisible, so a dump stops matching what `uiautomator dump` prints for the same
+screen (on the Android 17 Settings home, 81 nodes against 88). The server
+therefore turns the flag on only for the duration of a `--windows` dump.
+
+## Building
+
+`server/build.sh` compiles with `javac -source 8 -target 8` against the newest
+`android.jar` in the SDK and dexes with d8 at `--min-api 24`. The android.jar
+stubs have no `LambdaMetafactory`, so the server uses anonymous classes instead
+of lambdas. The jar is about 13 KB and checked in, so neither a clone nor the
+npm package needs an Android SDK.