tapyr-cli 0.1.0__tar.gz

tapyr_cli-0.1.0/CHANGELOG.md

@@ -0,0 +1,80 @@
+# Changelog
+
+## 0.1.0 (2026-04-10)
+
+Initial public release. **977 tests, 56 MCP tools, 35 CLI subcommands.**
+
+### Intent Resolver
+- `tapyr do "intent"`: natural-language intent → tapyr action, three-tier pipeline
+  (14 builtin regex patterns → workflow suggestion → element-text matching),
+  no LLM calls, dry-run support
+
+### Perception
+- Screenshot + UI dump + Set-of-Mark annotated PNG with numbered elements
+- Vision-first fallback v0: grid-based region synthesis when uiautomator returns nothing
+- Delta mode (`--since-last`): return only changed elements between calls
+- Compact mode (`--compact`): minimal id/text/clickable payload for token efficiency
+- Clickable-only filter (`--clickable-only`)
+- Continuous watch polling (`tapyr watch`) with JSON Lines output on change
+
+### Actions
+- Semantic tap (`--text`, `--id`, `--content-desc`, `--xy` with percent coords)
+- Swipe, type, key, scroll-until, wait-for
+- Unlock (idempotent, PIN support)
+- Device status (one-shot composed query)
+- Read notifications with OTP extraction (`--extract-otp`)
+- Named screenshot snapshots (`tapyr screenshot --save`)
+
+### Workflow Engine
+- YAML workflows with template variables, weighted `choose_one`, `repeat`, `sequence`
+- `if` / `skip_if` / `run_if` conditional gates
+- `expect_screen` / `verify` assertions with soft-mismatch `needs_llm_fallback` markers
+- Per-step `retries` with exponential backoff
+- Checkpoints with `requires_confirmation` halt + `--resume-from`
+- Patching protocol: detect drift → respond with patches → persist as new workflow
+- Dry-run with synthetic screen state (`--dry-run-state`)
+- Workflow pipeline: `tapyr kb run A B C` chains multiple workflows
+- Ad-hoc step execution (`tapyr exec --steps-json`)
+
+### Knowledge Base
+- CRUD: save, read, list, delete, rename, clone, update metadata
+- Tag management: add/remove/list with FTS reindex
+- Search with FTS + tag filter composition
+- Suggest workflows based on screen content
+- Diff, find-refs, check-calibration for workflow maintenance
+- History: JSONL audit log, stats aggregation, CSV export
+- Record: materialize workflows from action history
+- Export/import/install bundles (zip + manifest, 3 merge strategies, URL fetch with sha256)
+
+### Calibration
+- Rail scanner (vertical icon columns) and nav scanner (horizontal bars)
+- Unified v2 multi-region files with namespaced labels (`right_rail.comment`)
+- `calibrate-app` mega-command composing both scanners
+- `tap_rail` and `tap_saved` actions consuming calibrations with cross-device percent coords
+
+### Safety
+- Action allowlist/denylist via `~/.tapyr/safety.yaml`
+- Three-tier rules: deny / require_confirmation / allow
+- Predicate matching: equals, contains, regex, bare-field AND-semantics
+- CLI management: `tapyr safety list/set/delete`
+- `--confirm-safety` runtime bypass for require_confirmation rules
+- Per-device override support
+
+### Credential Vault
+- macOS (security CLI), Linux (secret-tool/libsecret), Windows (DPAPI/PowerShell)
+- `fill_credential` pastes without returning the secret to the agent
+- No `vault_get` MCP tool by design — secrets never cross the agent boundary
+- In-memory backend (`TAPYR_VAULT_TEST_BACKEND=memory`) for CI
+
+### Distribution
+- `tapyr install-adb`: Playwright-style platform-tools bootstrap
+- `tapyr doctor --fix`: auto-remediate adb-not-found
+- `tapyr demo`: 7-step zero-side-effect scripted showcase
+- `tapyr install claude-desktop/claude-code/cursor`: MCP client setup
+- PyPI-ready: 210KB wheel, PEP 561 py.typed marker
+- GitHub Actions CI: Python 3.10-3.13 × Ubuntu + macOS
+
+### Security
+- Subprocess injection audit: 4 vectors patched with 28 regression tests
+- Input validation on package names, keyevent names, deep-link URIs
+- Shell metacharacter escaping for `adb shell input text`

tapyr_cli-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing to Tapyr
+
+Thanks for your interest in contributing.
+
+## Development setup
+
+```bash
+git clone https://github.com/RafieAmandio/tapyr.git
+cd tapyr
+pip install -e ".[dev]"
+python -m pytest -q          # 977 tests, ~12s
+```
+
+## Running tests
+
+```bash
+python -m pytest -q                    # full suite
+python -m pytest tests/test_foo.py -x  # single file, stop on first failure
+python -m pytest -k "keyword"          # filter by name
+```
+
+All tests are hermetic — no real device, no network calls. Monkeypatching
+is used for adb, urllib, and OS-level subprocess calls.
+
+## Smoke-testing with a device
+
+```bash
+tapyr doctor --fix                     # ensure adb is available
+tapyr demo                            # 7-step safe walkthrough
+tapyr do "go home"                    # AI CLI mode — intent resolver
+tapyr perceive --pretty               # inspect the screen
+```
+
+## Adding a feature
+
+1. Read the existing code in the module you're extending.
+2. Add tests alongside the implementation. Every PR must keep the suite green.
+3. Follow existing patterns — match the idioms of adjacent functions.
+4. No new dependencies without discussion.
+
+## Workflow bundle contributions
+
+Share your workflows via `tapyr kb export`:
+
+```bash
+tapyr kb export ~/my-pack.tapyr.zip --workflow my_flow --calibration my_cal
+```
+
+See `examples/README.md` for the showcase workflow conventions.
+
+## Code of conduct
+
+Be kind, be constructive, ship working code.

tapyr_cli-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Rafie Amandio
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tapyr_cli-0.1.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+include CONTRIBUTING.md
+recursive-include examples *.yaml *.yml *.md

tapyr_cli-0.1.0/PKG-INFO

@@ -0,0 +1,390 @@
+Metadata-Version: 2.4
+Name: tapyr-cli
+Version: 0.1.0
+Summary: Playwright for Android, built for AI agents. CLI + MCP server for LLM-driven phone control.
+Author-email: Rafie Amandio <88525718+RafieAmandio@users.noreply.github.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/RafieAmandio/tapyr
+Project-URL: Documentation, https://github.com/RafieAmandio/tapyr
+Project-URL: Repository, https://github.com/RafieAmandio/tapyr
+Project-URL: Bug Tracker, https://github.com/RafieAmandio/tapyr/issues
+Keywords: android,adb,automation,mcp,ai-agents,playwright
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: Pillow>=10.0
+Requires-Dist: PyYAML>=6.0
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.0; extra == "mcp"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Dynamic: license-file
+
+![Tests](https://github.com/RafieAmandio/tapyr/actions/workflows/test.yml/badge.svg)
+
+# Tapyr
+
+**Playwright for Android, built for AI agents.**
+
+A CLI and stdio MCP server that gives an LLM (Claude, Cursor, any MCP-capable
+agent) structured perception and action primitives for a connected Android
+device, plus a file-based knowledge layer so learned workflows and app notes
+persist across sessions.
+
+Tapyr is **not** an agent. It is the hands, eyes, and memory that an agent
+uses. Bring your own brain.
+
+## Why
+
+LLMs can already read text and reason about intent. They cannot, by default,
+open your Gojek app, tap "GoFood", type "sate", and hand off to checkout.
+Tapyr closes that gap with a small, opinionated set of primitives designed
+for tool-use from the ground up:
+
+- **JSON-first output.** Every command returns `{"ok": bool, ...}` — no
+  log-scraping.
+- **Semantic targeting.** `tap --text "Order"` or `tap --id 7` (from the
+  last perception), not `tap 540 1820`.
+- **Set-of-Mark perception.** Screenshot + UI dump + an annotated PNG with
+  numbered boxes so vision models can pick an element by number.
+- **Cross-device coordinates.** `tap_xy` and `swipe` accept percent strings
+  (`"50%"`) that resolve against the live `wm size` at replay time. Workflows
+  written on 720x1604 replay on 1080x2400 unchanged.
+- **Workflow executor.** YAML workflows with template variables, weighted
+  `choose_one` branches, `repeat`, `sequence`, `expect_screen`, `verify`,
+  `if` / `skip_if` / `run_if` conditional gates, per-step `retries` with exponential
+  backoff, and a debug mode that snapshots fingerprints and annotated PNGs
+  per step. `kb run tiktok_engage --arg count=10` — done.
+- **Safety gates.** `checkpoint: name, requires_confirmation: true` halts a
+  replay cleanly at payment / destructive steps; resume past the gate with
+  `tapyr kb run NAME --resume-from checkpoint_name` without re-executing the
+  expensive setup.
+- **Observability.** `tapyr kb stats` surfaces per-workflow success rate,
+  halted runs, and drift-marker counts from the JSONL audit log — flaky
+  workflows bubble to the top. Soft `expect_screen` mismatches emit a
+  structured `needs_llm_fallback` marker so an orchestrating agent can
+  machine-detect drift without parsing prose warnings.
+- **Calibration for canvas apps.** Apps that render via custom canvas
+  (TikTok, Instagram Reels, YouTube Shorts) are invisible to uiautomator.
+  `calibrate-rail` scans for a vertical icon rail (right-side action column),
+  `calibrate-nav` scans for a horizontal one (bottom nav), and the `tap_rail`
+  workflow action consumes saved calibrations by semantic name (`icon:
+  comment`). See below.
+- **Lock screen + device status.** `tapyr unlock [--pin XXXX]` dismisses the
+  keyguard idempotently (no-ops if already unlocked). `tapyr status` returns
+  a one-shot snapshot — locked? foreground? screen size? wakefulness?
+  battery? — in a single call instead of five sequential adb round trips.
+- **File-backed memory.** Workflows are YAML. App notes are Markdown. The
+  agent can read, write, and share them.
+- **One binary, many front doors.** The same core runs as a CLI, an MCP
+  stdio server, or a self-installing plugin for Claude Desktop / Claude
+  Code / Cursor.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────┐
+│  executor     replay workflows, verify      │
+├─────────────────────────────────────────────┤
+│  knowledge    workflows + app notes + FTS   │
+├─────────────────────────────────────────────┤
+│  calibration  whiteness scan for canvas UIs │
+├─────────────────────────────────────────────┤
+│  actions      semantic tap, swipe, type     │
+├─────────────────────────────────────────────┤
+│  perception   screenshot + ui_dump + SoM    │
+├─────────────────────────────────────────────┤
+│  driver       adb subprocess wrapper        │
+└─────────────────────────────────────────────┘
+```
+
+Each layer is usable independently. Agents operate at whatever level the
+task needs: replay a saved workflow (fast path), drop to semantic taps
+(normal path), coordinate-based taps on a calibrated rail (canvas-app
+escape hatch), or raw pixels (last resort).
+
+## Install
+
+```bash
+pipx install tapyr-cli        # or: pip install tapyr-cli
+tapyr doctor                  # tells you exactly what's missing
+tapyr doctor --fix            # auto-install adb if missing, re-run checks
+tapyr install-adb             # one-shot Playwright-style fetch of platform-tools
+tapyr demo --dry-run          # preview the 7-step scripted walkthrough
+tapyr demo                    # with a phone plugged in: the hero shot
+```
+
+`tapyr install-adb` fetches the official Android platform-tools bundle from
+`dl.google.com` into `~/.tapyr/bin/`, falling back to a `find_adb()` chain
+extension so PATH-installed adb still wins if present. Cached after first
+run; `--force` re-downloads. Optional `TAPYR_ADB_SHA256` env var for
+enterprise users who need a pinned checksum.
+
+`tapyr demo` runs a zero-side-effect, 7-step walkthrough (doctor → wake →
+perceive → HOME → deep-link Settings → BACK → save a marker workflow) that
+gives the first-run experience a single command. No typing into fields, no
+taps inside third-party apps, no installs. `--dry-run` works without a
+phone for curious users and CI.
+
+You will also need Android `platform-tools` on your `PATH` (for `adb`) and a
+phone with USB debugging enabled. `doctor` walks you through both.
+
+The `TAPYR_HOME` environment variable overrides the default `~/.tapyr`
+state directory. Useful for multi-device test rigs or CI.
+
+### Wire it into your agent
+
+```bash
+tapyr install claude-desktop   # writes ~/Library/.../claude_desktop_config.json
+tapyr install claude-code      # writes ~/.claude/mcp.json
+tapyr install cursor           # writes ~/.cursor/mcp.json
+tapyr install print            # just prints the JSON snippet
+```
+
+Then restart the client. The agent now has a `tapyr` tool group with:
+
+- **Intent resolver:** `do` (natural-language → action, no API knowledge needed)
+- **Perception:** `perceive`, `screenshot`
+- **Actions:** `tap`, `swipe`, `type_text`, `key`, `scroll_until`, `wait_for`,
+  `unlock`, `exec_steps`
+- **Device:** `device_status`, `list_devices`
+- **Apps:** `open_app`, `deep_link`, `foreground`, `list_packages`,
+  `read_notifications`
+- **Calibration:** `calibrate_rail`, `calibrate_nav`, `calibrations_list`,
+  `calibration_show`, `calibration_delete`
+- **Credential vault:** `fill_credential`, `vault_set`, `vault_list`,
+  `vault_delete` (macOS / Linux / Windows; no `vault_get` by design)
+- **Safety policy:** `safety_list`, `safety_set_rule`, `safety_delete_rule`
+- **Knowledge base:** `kb_search_workflows`, `kb_suggest_workflows`,
+  `kb_read_workflow`, `kb_run_workflow`, `kb_save_workflow`,
+  `kb_delete_workflow`, `kb_rename_workflow`, `kb_clone_workflow`,
+  `kb_update_workflow_metadata`, `kb_lint_workflow`, `kb_diff_workflows`,
+  `kb_find_references`, `kb_check_calibration`, `kb_record_workflow`,
+  `kb_add_tags`, `kb_remove_tags`, `kb_read_history`, `kb_stats`,
+  `kb_read_app_note`, `kb_append_app_note`, `kb_list_app_notes`,
+  `kb_list_workflows`, `kb_export_bundle`, `kb_import_bundle`,
+  `kb_install_bundle`
+- **Environment:** `doctor`
+
+**56 MCP tools** across 11 families — the full surface an agent needs to
+perceive, act, learn, and manage workflows on any connected Android device.
+
+## Usage (CLI)
+
+```bash
+# Sanity check
+tapyr doctor --pretty
+tapyr devices
+tapyr status --pretty            # one-shot: locked, foreground, battery, ...
+
+# See the screen the way an agent sees it
+tapyr perceive --pretty
+#   -> ~/.tapyr/cache/last_screenshot.png
+#   -> ~/.tapyr/cache/last_annotated.png  (numbered overlays)
+#   -> JSON element list with ids
+tapyr perceive --clickable-only    # trim layout noise, keep only tap targets
+tapyr perceive --compact           # minimal payload: id + text + clickable only
+tapyr watch --interval 2 --max-polls 10  # poll for screen changes, JSON Lines output
+
+# AI CLI mode — just say what you want
+tapyr do "like this post"                  # resolves intent → action automatically
+tapyr do "go home"                         # builtin: HOME key (no perceive needed)
+tapyr do "type hello world"                # detects type intent
+tapyr do "open com.whatsapp"               # opens app by package
+tapyr do "scroll down"                     # swipe gesture
+tapyr do "like this" --dry-run             # preview the resolved action without executing
+tapyr perceive --since-last        # delta mode: only what changed since the last call
+tapyr --timeout 60 perceive        # global adb-shell timeout override (slow devices)
+
+# Drive the phone
+tapyr unlock --pin 1234          # dismisses keyguard, idempotent
+tapyr tap --text "GoFood"
+tapyr type "sate"
+tapyr key enter
+tapyr tap --id 3
+tapyr tap --xy 50% 90%           # percent coords work everywhere
+tapyr swipe 50% 75% 50% 25%
+tapyr scroll-until --text "Add to cart"
+tapyr wait-for --text "Order confirmed" --timeout 15
+
+# Skip the UI entirely when you know the intent
+tapyr deep-link "gojek://gofood/home"
+tapyr open com.whatsapp
+
+# Read push notifications / SMS OTPs (needs --noredact on device)
+tapyr read-notifications --package com.android.messaging --pretty
+
+# Knowledge base
+tapyr kb list
+tapyr kb list --query "food order"
+tapyr kb search "checkout" --tag food                  # FTS + tag filter
+tapyr kb read gojek_order_food
+tapyr kb run tiktok_natural_engagement --arg count=10
+tapyr kb run unlock_flow gofood_order verify_order    # pipeline: run 3 in sequence
+tapyr kb run my_workflow --debug                      # per-step PNGs
+tapyr kb run my_workflow --dry-run                    # preview only
+tapyr kb run pay_flow --confirm-checkpoints           # walk past gates
+tapyr kb run pay_flow --confirm-safety                # walk past require_confirmation safety rules
+tapyr kb run pay_flow --resume-from confirm_payment   # pick up from halt
+tapyr kb run my_workflow --dry-run --dry-run-state '{"screen_texts":["OK"]}' # evaluate if/skip_if gates against a fixture
+tapyr kb lint my_workflow                             # static validation
+tapyr kb stats --days 7                               # flaky workflows
+tapyr kb history --limit 20 --action run_workflow
+tapyr kb delete old_workflow
+tapyr kb rename old_name new_name                      # atomic rename + FTS reindex
+tapyr kb tag add gojek_order food delivery             # add tags to a saved workflow
+tapyr kb tag remove gojek_order wip                    # remove tags
+tapyr kb clone gojek_order --as gojek_order_v2         # duplicate for safe editing
+tapyr kb diff workflow_v1 workflow_v2 --pretty         # colored unified-diff
+tapyr kb history --export csv > history.csv             # spreadsheet-friendly audit export
+tapyr kb find-refs --calibration tiktok_rail           # who uses this?
+tapyr kb check-calibration tiktok_rail                 # did re-cal break anything?
+tapyr kb app com.gojek.app
+tapyr kb app com.gojek.app --append "Payment confirm button is bottom-right"
+tapyr kb apps                                          # list all documented packages
+tapyr kb export ~/gojek-pack.tapyr.zip --all            # pack workflows+cals+notes
+tapyr kb import ~/gojek-pack.tapyr.zip --merge rename   # skip/overwrite/rename
+tapyr kb install https://example.com/gojek-pack.tapyr.zip --sha256 ...  # fetch + verify + import
+
+# Credential vault (macOS / Linux / Windows)
+tapyr vault set com.whatsapp password --stdin           # reads from stdin, never argv
+tapyr vault list                                         # {package, field} only — no secrets
+tapyr fill-credential com.whatsapp password              # paste into focused field
+
+# Action safety policy
+tapyr safety list                                        # show current deny/require_confirmation/allow rules
+tapyr safety set deny key --equals POWER                 # never POWER key on this device
+tapyr safety set deny shell --contains "pm uninstall"    # forbid uninstalls via shell action
+tapyr safety set require_confirmation deep_link --package com.android.vending
+tapyr safety delete deny key --equals POWER              # exact-match delete
+
+# Parallel test fleets
+TAPYR_MULTI_DEVICE=1 tapyr --serial emulator-5554 perceive  # scoped cache + history
+```
+
+## Calibration for canvas-rendered apps
+
+Some apps render their entire UI through a custom canvas engine — TikTok,
+Instagram Reels, YouTube Shorts. Their windows have focus but `shown=false`
+in `dumpsys`, so `uiautomator dump` falls through to whatever's drawn
+underneath (usually the launcher). `perceive` returns zero meaningful
+elements. Semantic `tap --text` and `tap --content_desc` do not work.
+
+Tapyr's workaround: scan the screen for white icon clusters along a vertical
+rail and use the cluster centers as blind-tap targets.
+
+```bash
+# 1. One-time setup per app+device: scan the rail, label the clusters,
+#    save the result. The x-column auto-detects in the right 15% of the
+#    screen; pass --x-col to override.
+tapyr calibrate-rail \
+  --y-min 500 --y-max 1400 \
+  --min-cluster-pixels 30 \
+  --labels heart,comment,bookmark,share \
+  --save tiktok_rail.json
+
+# 2. In a workflow yaml, reference the saved calibration by name:
+#
+#    - action: tap_rail
+#      calibration: tiktok_rail
+#      icon: comment                       # v1: bare label
+#      icon: right_rail.comment            # v2: namespaced (region.label)
+#
+#    Unified v2 calibrations merge multiple regions (right_rail, bottom_nav)
+#    into one file per app. Bare labels auto-resolve when unambiguous;
+#    namespaced labels (right_rail.comment) disambiguate across regions.
+#    v1 files (single rail) keep working unchanged.
+
+# 3. Discover what's saved
+tapyr calibrations --pretty
+```
+
+The `examples/tiktok_natural_engagement.yaml` workflow is a fully worked
+reference: a weighted `choose_one` loop that randomly likes, favorites,
+shares, or comments (with Indonesian gen-Z phrases) 5-45% of the time,
+passively scrolls the rest, and posts real comments via TikTok's native
+input field.
+
+See `examples/README.md` for four more showcase workflows:
+**whatsapp_send_message** (the hello-world tutorial),
+**gofood_order** (Indonesian use case with a payment checkpoint),
+**spotify_play_playlist** (morning-routine weighted rotation), and
+**instagram_reels_engagement** (canvas-app mirror of the TikTok example).
+Every file in `examples/` is pinned lint-clean by a regression test.
+
+## Knowledge on disk
+
+```
+~/.tapyr/
+├── workflows/              # agent-written task recipes (yaml)
+├── apps/                   # per-app declarative notes (markdown)
+├── calibration/            # rail-icon calibrations for canvas apps (json)
+├── history/                # jsonl audit log of every action
+├── cache/                  # last screenshot, dump, annotated PNG
+├── debug/                  # per-step snapshots from `kb run --debug`
+└── index.sqlite            # FTS index — rebuildable from the above
+```
+
+Everything is git-friendly. You can version-control your `~/.tapyr/`
+directory and share workflows + calibrations across machines or with
+teammates.
+
+## Usage (from an agent, via MCP)
+
+Once installed into your client, the agent can call the tools directly.
+A typical loop for a well-known app:
+
+1. `perceive` → read the element list + annotated PNG
+2. `tap` with a selector that matches an element
+3. `perceive` → confirm the new screen
+4. `kb_save_workflow` once the task completes, so the next agent is faster
+
+For a canvas-rendered app (TikTok etc.) where `uiautomator dump` returns
+nothing: **vision-first fallback (v0)** automatically kicks in — the
+screenshot is analyzed via grid-based variance detection to synthesize
+clickable regions even without a UI tree. The agent sees numbered boxes
+on the annotated PNG and can `tap --id N`. For higher precision on
+known apps, calibrate once and replay:
+
+1. `calibrate_app` with `--rail` and `--nav` flags to scan both icon
+   regions in one shot and save a unified v2 calibration file
+2. `kb_save_workflow` with `tap_rail` or `tap_saved` steps referencing
+   namespaced labels (`right_rail.comment`, `bottom_nav.home`)
+3. `kb_run_workflow` to replay
+
+## Status
+
+**v0 / alpha.** Validated on a real Oppo 720x1604 device through 40+ loop
+iterations of incremental hardening. The TikTok engagement workflow posts
+real comments, likes, bookmarks, and shares through the coordinate-based
+`tap_rail` path. **977 tests passing**, covering the executor, actions,
+calibration scanners, knowledge base, MCP server, CLI dispatch, and the
+workflow observability surface.
+
+What's here (as of this README revision): the full primitive set listed
+above, 10 MCP tool families, `kb stats` observability over the JSONL audit
+log, soft-drift `needs_llm_fallback` markers on `expect_screen` mismatches,
+idempotent `unlock` and composed `device_status`, and a cross-device
+coordinate system that works on any screen that `wm size` can report.
+
+**Vision-first perception fallback (v0)** shipped — when `uiautomator dump`
+returns nothing, perception now synthesizes clickable regions from the
+screenshot via grid-based variance analysis so agents can at least `tap --id N`
+on numbered areas in the annotated PNG. Still directional (v1): replacing
+the heuristic grid with a segmentation model (SAM / YOLO) for element-level
+precision, plus OCR for text field extraction. See `BACKLOG.md` for the full
+list with priority ordering.
+
+## License
+
+MIT.