@reticlehq/server 2.3.0 → 2.5.0

package/docs/telemetry-contract.md

@@ -0,0 +1,214 @@
+# The telemetry contract
+
+> For anyone — human or agent — adding a tool, an event, a finding kind, or a failure path to Reticle.
+>
+> The rules here are enforced by `packages/server/src/telemetry/telemetry-contract.test.ts`. If you break one, that test tells you which and where. This page is why.
+
+## Why this has its own contract
+
+**Telemetry fails silently.** Nothing throws when an event is missed. No test goes red. No user complains. The data is simply, permanently absent — and you find out months later when someone asks a question the data cannot answer, about a period you can never re-collect.
+
+That has already happened here twice, and both times the code looked correct:
+
+- `daemon_stopped` was emitted fire-and-forget microseconds before `process.exit(0)`. The POST was killed every single time. The event **never once arrived**, and nothing anywhere indicated a problem.
+- `bug_found` hand-copied the twelve contradiction kinds into a local `Set`. Correct on the day it was written; the thirteenth kind would have been silently miscounted, quietly deflating the one number we intend to publish.
+
+So the rule is not "remember to add telemetry". The rule is that the guard lives in a test.
+
+## The five rules
+
+### 1. Everything routes through a chokepoint
+
+Tool usage, timing, errors, verifications and bugs are all recorded in **one place** — `runTool` in `tools/invoke-tool.ts`. Adding a tool to `TOOLS` is all it takes to be instrumented.
+
+Do **not** add telemetry inside a tool handler. If you find yourself wanting to, the metric probably belongs at the chokepoint, read off the result.
+
+> The one exception is a path that genuinely does not go through `runTool` — currently only the verification runner (`reticle verify`, the HTTP verify surface), which has its own reporter in `telemetry/run-telemetry.ts`. **If you add a second dispatch path, it needs the same treatment**, and until it has one it is invisible. That gap existed for real: CI-found bugs were uncounted.
+
+### 2. Names say what happened
+
+`<noun>_<verbed>`, lowercase, no abbreviations: `verification_completed`, `bug_found`, `runtime_crashed`.
+
+## Counting defects: instances vs distinct
+
+`bug_found` fires once per OCCURRENCE. A defect hit five times in a session is five events, which is the right raw signal — frequency is what says which classes of defect actually cost anybody anything. But it means a naive count answers "how often were defects hit", not "how many defects were found", while looking like it answers the second.
+
+So every `bug_found` carries **`repeat`**: false the first time a KIND is seen in a session, true after. Count `repeat: false` for **distinct defects**; count everything for **instances**. Measured on a real app: 7 events, 3 defects, 4 repeats. Publishing the 7 as defects would have inflated the claim by more than double.
+
+The denominator is **`verification_completed`**, which fires per verdict with `via`, `verified`, `passed` and `falseGreenCaught`. Defects per verification is the honest rate; raw defect counts grow with usage and say nothing on their own.
+
+**And `repeat` only means anything if the session remembers.** `SessionMetrics.reset()` runs at every periodic flush and used to clear the seen-kinds set with the window counters — so the same defect, re-found after a flush, reported `repeat: false` again. Sessions in the data run to 11.5 hours. Window counters zero on a flush; session-lifetime memory does not. (`session-window.test.ts`)
+
+Two rules follow, and both are gated:
+
+- **`repeat` is set at the EMISSION site, never by the classifier.** `bugsInResult` is a pure function over one tool result and cannot know what a session has already seen; if it ever grows a `repeat` field it will be guessing, and the guess becomes the published number. (`telemetry-contract.test.ts`)
+- **Session-scoped, and it cannot be otherwise.** The payload carries no selector, URL or app detail by design, so the same defect in two sessions is unrecognisable as one — and making it recognisable would require collecting exactly what this event refuses to collect.
+
+The old set failed this so badly it confused its own authors — `invoke` meant "the CLI ran" while `tool` meant "a tool was called", which is the opposite of how both read. A name that has to be looked up is a name that gets misread on a dashboard a year from now.
+
+### 3. Names, never values
+
+| Send                                 | Never send              |
+| ------------------------------------ | ----------------------- |
+| Parameter and flag **names**         | What they were set to   |
+| Error **shape** (variables stripped) | The message             |
+| **Our** stack frames                 | The user's stack frames |
+| The **kind** of a defect             | What it was found in    |
+| A hash of the git origin             | The origin              |
+
+`--http-token` holds a secret. `reticle_act`'s `args` holds the text being typed into the app, which on a login form is a password. Assume every value is the worst thing it could be.
+
+There is one narrow exception, and it is explicit rather than heuristic: parameters whose values are enums **we** defined are allow-listed in `telemetry/argument-shape.ts`, and anything unrecognised reports as `other` so a schema change cannot start forwarding free text.
+
+### 4. Never derive a vocabulary by copying it
+
+If a set of kinds already exists in `@reticlehq/core`, **import it**. Do not re-list it.
+
+```ts
+// ✗ correct today, wrong the moment core gains a member — and silent about it
+const KINDS = new Set(['ui-advanced-request-failed', 'signal-contradicted' /* …10 more */]);
+
+// ✓ cannot drift
+const KINDS = new Set(Object.values(ContradictionKind));
+```
+
+A copied enum is a drift hazard anywhere. It is a **correctness** hazard when the thing that drifts is a number you publish.
+
+### 5. A metric may never change behaviour
+
+Every send is wrapped and best-effort. A telemetry failure must not fail a tool call, a verification, a daemon start, or `reticle init`.
+
+The single exception is `daemon_stopped`, which is **awaited** — because the process exits immediately after and the send would otherwise be killed. Even then a failure resolves rather than throws.
+
+## The event kinds, all of them
+
+`TelemetryEventKind` in `@reticlehq/core` is the closed list. Seven of these went undocumented here for months — the doc described only the session-lifecycle half — so anyone building a dashboard from this page could not know that the transport and install events existed at all. A kind nobody documents is a kind nobody queries, and telemetry that nobody queries is telemetry nobody notices has stopped arriving. `telemetry-contract.test.ts` now fails when a kind is missing from this table.
+
+| Kind | When | What it answers |
+| --- | --- | --- |
+| `reticle_installed` | first-ever run on a machine | install count, and the new-user curve |
+| `cli_command_run` | a human ran a `reticle` subcommand | human intent — `verify`/`gate` mean something very different from `status`. Never emitted for the internal `_daemon` spawn |
+| `daemon_started` | the daemon came up | active sessions, DAU/WAU/MAU |
+| `daemon_stopped` | clean exit | the rich session roll-up. **Count sessions with this one** — see below |
+| `session_progress` | periodic flush from a LIVE daemon | same payload, `final: false`. Sum work across both |
+| `verification_completed` | a verdict was produced | the product's reason to exist: was an app actually verified |
+| `project_profiled` | once per daemon start | stack, size, and how deeply the feature surface is used — activation vs retention |
+| `version_changed` | the running version differs from last seen | upgrade adoption, and whether a nudge caused it |
+| `runtime_crashed` | an unhandled failure in the daemon | stability |
+| `feedback_submitted` | `reticle feedback`, or an agent's report | the qualitative channel |
+| `identified` | `reticle identify` | joins anonymous machine ids to a person who volunteered one |
+| `mcp_client_connected` | an MCP client attached | how many sessions are agent-driven at all |
+| `mcp_connection_lost` | the proxy lost its daemon | **the transport-stability metric.** The disconnect that makes a user reopen `/mcp` is invisible without it |
+| `init_completed` | `reticle init` finished | does install actually work, outside the fixtures gate |
+| `bug_found` | a defect was detected in the app under test | the value delivered, as opposed to the work done |
+
+## Sessions: `daemon_stopped` vs `session_progress`
+
+Count sessions with **`daemon_stopped`** (`final: true`). It fires once, at a clean exit.
+
+A running daemon rolls its window up every 5 minutes as **`session_progress`** (`final: false`), same payload shape. Sum work across both; count sessions with neither summed nor doubled.
+
+This split exists because the flush used to be emitted AS `daemon_stopped` — an event named for an exit, fired while the process was alive. One day's export: 98 `daemon_stopped` events were 73 exits + 25 flushes, so a session count was 34% high. Worse, the two populations are opposites — every one of the 25 flushes had tool calls and **not one of the 73 exits did**, because a daemon that has served a tool never idle-exits and so never reaches a clean shutdown. A funnel over the raw event describes active sessions at one end and abandoned ones at the other.
+
+The flush interval is also the **bound on what is lost**: nothing calls shutdown when a working daemon is finally killed, so its last partial window dies with it. At 30 minutes against a median 28-minute session that was most of the session. Only non-empty windows emit, so a short interval costs nothing on the daemons that never serve a tool.
+
+## The session summary's newer fields
+
+Four counters and one flag were added because the data could not answer questions we were already asking. All are properties on events that already exist — no new kinds — and all four counters are **omitted rather than sent as zero**, so a field's presence is itself the signal.
+
+| Field | On | Means |
+| --- | --- | --- |
+| `noSessionErrors` | session summary | tool calls that failed because there was no app to reach — no session, no session by that id, or several with none named. The largest drop-off in the funnel; it was previously reachable only by unpacking `errors[]`. |
+| `consecutiveRepeats` | session summary | longest back-to-back run per tool name. `toolCounts` reports five useful calls and five retries of one failing call identically, and those are opposite facts. |
+| `abandonedActions` | session summary | actions driven with no verdict AFTER them — the trailing unsettled run, not `actions - verifications`. That difference ignores order, so a verdict that drove nothing (a `flow_verify` over saved flows) silently paid for an abandoned action elsewhere. |
+| `verification.browser` | `verification_completed` | `headless` \| `headed` \| `attached` — who DROVE the browser. `attached` (Reticle launched nothing, the SDK connected from a browser somebody else opened) is the common case in production, so on its own this is mostly "somebody's own browser". |
+| `verification.brand` | `verification_completed` | WHICH browser it was: `chrome` \| `edge` \| `arc` \| `dia` \| `brave` \| `opera` \| `firefox` \| `safari` \| `other` — the closed `BrowserBrand` list in core. The axis `engine` cannot answer, since Chrome, Edge, Arc, Dia and Brave are all `blink`. The SDK reads `navigator.userAgentData.brands` (and the UA string on Firefox/Safari, which expose no `userAgentData`) and normalises IN THE PAGE: a raw brand or UA string is unbounded and fingerprintable and never leaves. Anything unrecognised is `other`. **Omitted rather than `"unknown"`** when the page did not say — a desktop webview has no brand and an older SDK does not report one, and a guess is indistinguishable from a measurement on a dashboard. |
+| `verification.reason` | `verification_completed` | WHICH clause of `decideVerified` produced the verdict, from core's closed `VerifiedReason`: `inconclusive` \| `assertion_failed` \| `contradicted` \| `already_true` \| `unclean_capture` \| `vacuous_grade` \| `outcome_pending` \| `outcome_unread` \| `unsettled` \| `proved`. See below. |
+| `bug.attribution` | `bug_found` | `app` \| `request` \| `reticle` — whose fault the defect was. **Absent means unclassified**, never `app`. See below. |
+| `outage.stage` / `outage.reason` / `outage.attempts` | `mcp_connection_lost` | which stage of the outage, why the stream went away (closed `OutageReason`, `other` for anything unnamed), and how many reconnects had been tried. See below. |
+| `tzOffsetMin` | every event | minutes offset from UTC. One integer, no location. |
+| `versionChange.nudged` | `version_changed` | an agent had been told about exactly this version recently, so the nudge plausibly caused the update. The daemon that nudges and the `reticle update` that acts are different processes, so a marker file joins them. |
+
+## Why a verdict came out that way — `verification.reason`
+
+`verified` has three values. The rule that produces it has **ten clauses**. Everything in between was thrown away at the moment it was known.
+
+Captured against the real classifier: `verified: 'unknown'` covered "the agent malformed the call", "the consequence was already true", "the app answered 202", "a 2xx body went unread", "the capture was not clean", "nothing was asserted at a real grade" and "the page never settled" — **seven causes, two wire payloads**. They belong to three different owners (the agent, the app, Reticle) and need opposite responses: teach the agent, wait and re-check, or ship a fix. On a dashboard they were one bar. `verified: 'no'` collapsed the same way — "channels disagree" (Reticle earning its keep) and "the agent's predicate failed" were the same string.
+
+`VerifiedReason` lives in `@reticlehq/core` and is the **single list**. `decideVerified` returns a member from every clause, so a new clause cannot compile without one; `verified.test.ts` drives all ten and fails if a member exists that no clause produces. `verification-of.ts` narrows the result field against `Object.values(VerifiedReason)` — anything else is dropped rather than forwarded, because a string nobody can group by is worse than a gap. **Nothing re-lists these**, including the battery spec, which imports the enum from core's build.
+
+Optional on purpose: a suite verdict (`flow_verify`) is a pass/fail with no clause behind it, and an older sender has none. Absent means unclassified.
+
+## Whose defect it was — removed, on purpose
+
+`bug.attribution` shipped twice and was wrong both times. A real drive found that across two full runs EVERY `attribution: 'app'` was a misattribution, while the one defect that genuinely was a bad agent predicate carried none — a single session would have published "2 defects in the app" against a true count of 0.
+
+A metric that is confidently wrong about whose fault a defect is, is worse than no metric: it is the number a founder steers on, and it points at the customer. Counting nothing is recoverable; publishing a false accusation about somebody's product is not.
+
+It comes back when a verdict carries the reason it came out that way. Today the payload cannot separate the cases — `element.present` covers "the button is missing", "the API is down" and "the agent mistyped a testid" identically.
+
+## What is NOT a crash — expected disconnects
+
+`runtime_crashed` answers exactly one question: is Reticle stable. One real session put **nine** events into it, all `write EPIPE` — the MCP client closed its half of the stdio pipe and the next `process.stdout.write` failed, which is how a client is supposed to leave.
+
+`daemon-resilience.ts` matches `err.code` against `EPIPE` / `ECONNRESET` / `ERR_STREAM_DESTROYED` and logs `reticle_daemon_client_disconnected` (or `reticle_mcp_proxy_client_disconnected`) instead of emitting. Two rules make this safe rather than a hole:
+
+- **Code, never message.** Prose gets wrapped, localised and rewritten; matching it would eventually swallow a real crash that merely mentioned a pipe. `daemon-resilience.test.ts` drives an error whose message says `write EPIPE` and carries no `code`, and asserts it is still a crash.
+- **Visible, never swallowed.** It still logs a line with the code. A daemon emitting a hundred of these is a finding, just not a crash.
+
+A disconnect is also **no longer fatal for the daemon**: Node's "process state is undefined" guidance is about a throw that escaped everything, not about writing to a socket somebody closed, and exiting there let one departing client take down the daemon serving every other agent.
+
+## Did MCP stay up — the `outage` block
+
+The transport-stability metric shipped with an **empty payload** for months, and it is the exact failure this page opens with. `reportMcpOutage` passed `{ outage: { stage, reason, attempts } }`, `TelemetryExtra` declared the field, and it typechecked — but `emit()` builds its event from an **explicit allow-list of keys** and `outage` was not on it, nor in the `blocks` flattening map, nor in core's `TelemetryEventSchema`. Two deliberately different outages produced byte-identical events. Nothing threw, no test went red, and the data for that whole period cannot be recovered.
+
+The lesson is not "wire the field". It is that **the battery asserted the event ARRIVED and never that it carried anything**, and a kind-only assertion cannot see an empty payload. When you add an event kind, the live check has to assert the FIELDS.
+
+- `stage` — `first` (this session lost MCP at all) or `budget_spent` (it stopped retrying). These are the two facts the event exists to separate: the share of sessions that lose MCP, and the share where it never came back on its own.
+- `reason` — closed `OutageReason`: `sse_ended` | `sse_error` | `sse_aborted` | `sse_closed` | `connect_error` | `other`. The proxy's own reason strings are free text that also feeds a log, so `mcp-outage.ts` narrows them and reports **`other`** for anything unnamed. A classifier that cannot say "I don't know" lies instead, and an unbounded string must never reach the wire.
+- `attempts` — consecutive reconnects tried when this was reported.
+
+Still true and worth knowing when you query it: `mcp_connection_lost` carries **no `sessionId`** (it fires from the proxy process, not the daemon), and is capped at two per proxy process by design.
+
+## Recording locally instead of sending — `RETICLE_TELEMETRY_FILE`
+
+Set it to a path and every event is appended there as one JSON object per line, and **nothing is sent**. The payload is the one the wire would have carried, built by the same code and redacted by the same rules, so what a run records is what a user would have sent.
+
+It exists for two reasons that pull the same way:
+
+- **A release sweep is not a user.** Driving dozens of sessions through a gate emits real `daemon_started` / `verification_completed` / `bug_found` events, indistinguishable in PostHog from people. Test runs polluting the numbers is the same class of error as counting `cli_command_run { mcp }` as human intent: the metric stops describing what it claims to.
+- **Verifying telemetry should not need a hand-rolled HTTP server.** Ad-hoc harnesses are how a check ends up measuring nothing.
+
+One deliberate exception to the rules above: `RETICLE_TELEMETRY_FILE` keeps telemetry ENABLED inside a Reticle source checkout. The checkout guard exists to stop us phoning home, and writing a local file is not phoning home — while a release sweep is driven from exactly there, so a sink that inherited the guard would record nothing and look like it had worked.
+
+`sent: true` from `reticle_feedback` means the record landed in the file, which is the honest reading of "captured" for a recorded run. An unwritable path degrades to a no-op and reports `false`; it never takes the daemon down.
+
+## Adding things: what to do
+
+| You are adding | Do this | Enforced by |
+| --- | --- | --- |
+| **A tool** | Add it to `TOOLS`. Nothing else. If its name implies a verdict (`assert`/`verify`), also add it to `VERIFICATION_TOOLS` | `telemetry-contract.test.ts` |
+| **A verdict-producing tool** | `VERIFICATION_TOOLS` — otherwise it emits no `verification_completed` and stops counting toward the product's headline metric | ✓ |
+| **A contradiction / anomaly kind** | Add it to core's enum only. `bug-found.ts` derives from it | ✓ |
+| **A new finding shape** in a tool result | Teach `bugsInResult` the field. Add a case to the contract test | ✓ |
+| **A failure path** (connect, install, crash) | Classify it into an enum with an explicit `OTHER` bucket — a classifier that cannot say "I don't know" lies instead | ✓ |
+| **An event kind** | Add to `TelemetryEventKind` + a payload schema + emit it + add a live check to `apps/e2e/specs/telemetry-events-test.mjs` that asserts the **fields**, not just that it arrived | partly — the live check is on you |
+| **A block on `TelemetryExtra`** | Also add it to the `emit()` event build, the `blocks` flattening map, AND `TelemetryEventSchema`. Missing any one of the three drops the payload in silence — see `outage` | ✗ **not enforced — be careful** |
+| **A dispatch path** that bypasses `runTool` | Give it a reporter like `run-telemetry.ts`, or it is invisible | ✗ **not enforced — be careful** |
+
+## Verifying it actually works
+
+Unit tests cannot see the failure mode that matters, because nothing throws. Two things do:
+
+```bash
+pnpm test:unit                                  # the contract test + the fingerprint/redaction guards
+node apps/e2e/specs/telemetry-events-test.mjs   # fires every event at a real endpoint, checks it lands
+```
+
+The second is the one that matters. It drives the real built modules against a real capture server — real network, real process semantics, real redaction — and asserts each event **arrives**. Half its checks are leak checks, asserting that secrets, passwords, customer emails and home directories are _absent_.
+
+**Both halves are mutation-tested.** Reintroducing the fire-and-forget bug fails 9 checks; disabling redaction fails 3. A guard that cannot fail is theatre, so these are periodically proven to bite.
+
+## The privacy line, in one sentence
+
+We measure **that** something happened and **what class** of thing it was — never **what** it was, in whose app, or containing what.

package/docs/telemetry.md

@@ -0,0 +1,184 @@
+# Telemetry
+
+Reticle collects a small amount of anonymous usage data to help us understand whether the tool is useful — which commands people run, which tools agents actually use, and whether people keep using Reticle after they try it. That's what this data is for, and all it is for: making the product better.
+
+This page is the complete description of what is collected. If something is not listed here, it is not sent.
+
+## The short version
+
+- **Anonymous.** We cannot tell who you are, and we do not try.
+- **No code, no app data.** Nothing from the app under test — no DOM, no network traffic, no console output, no source, no file paths — ever leaves your machine.
+- **The one exception is feedback you deliberately send us**, and it is never collected passively. See [Feedback](#feedback) below.
+- **Opt out any time**, permanently, with one command:
+
+  ```bash
+  reticle telemetry disable
+  ```
+
+## What is sent
+
+Thirteen kinds of events, each a single small JSON object:
+
+| Event | When | Extra data |
+| --- | --- | --- |
+| `reticle_installed` | The first time Reticle runs on a machine | — |
+| `cli_command_run` | You run a `reticle` command | Which subcommand (`verify`, `status`, …) and which flags were present, by name |
+| `daemon_started` | The local daemon starts | — |
+| `daemon_stopped` | The local daemon stops | A summary of the session — see below |
+| `verification_completed` | A verification produces a verdict | Whether it passed, and whether Reticle refused to call a passing check verified |
+| `project_profiled` | Once per daemon start | The shape of the project — see below |
+| `version_changed` | You update or roll back | The two version numbers, and which direction |
+| `runtime_crashed` | The daemon hits an uncaught error | The error's type, **Reticle's own** stack frames, and the message with variables stripped — see below |
+| `mcp_client_connected` | An agent attaches to the daemon | Whether it is a reconnect, and how long the daemon had been idle |
+| `mcp_connection_lost` | The agent's MCP tools go away | Which stage (`first`, or `budget_spent` when it stopped retrying), the cause, and the attempt count. **At most twice per session** — one measured afternoon produced 547 reconnects, and an event each would bill for the pathology instead of measuring it |
+| `init_completed` | `reticle init` finishes | Whether it worked, and a classified reason when it did not |
+| `bug_found` | Reticle finds a defect in the app under test | The **kind** of defect (`signal-contradicted`, `console-error`, …) and how it was found — never what it was found in |
+| `feedback_submitted` | **Only** when you or your agent explicitly send feedback | The report — see [Feedback](#feedback). The AGENT's call does not wait for the network: the receipt says `accepted` (validated, redacted, queued), never `sent`, and a delivery that then fails is reported back on the agent's next tool result. `reticle feedback` typed by a human still waits, because a person at a terminal is owed the real answer |
+| `identified` | **Only** when you run `reticle identify` | What you chose to tell us — see [Telling us who you are](#telling-us-who-you-are) |
+
+**There is no per-tool-call event.** Tool usage is counted in memory and leaves once, inside `daemon_stopped`, as a histogram like `{"reticle_act": 40, "reticle_assert": 12}`. That is counts of tool NAMES from a fixed list we define — never arguments, results, selectors, or URLs.
+
+**Parameter and flag NAMES are collected; their VALUES are not.** We record that `reticle_act` was called with `ref` and `action`, and that you ran `reticle serve --headed --port`. We do not record what you set them to. This distinction is the whole safety property, and it is absolute for CLI flags: `--http-token` holds a secret, `--drive` holds a URL, `--storage-state` holds a file path, so no flag value is ever sent. For tool parameters there is one narrow exception — a short, explicit allowlist of parameters whose values are enums _we_ defined (`action: "click"`), listed in [`argument-shape.ts`](../packages/server/src/telemetry/argument-shape.ts). A value outside that allowlist reports as `other`, so a future schema change cannot quietly start forwarding free text. `reticle_act`'s `args` — the text being typed into your app, which on a login form is a password — is **never** in that allowlist.
+
+`daemon_stopped` carries: how long the session ran, how many tool calls and of which tools, how long each tool took (total and worst case), how many failed, how many verifications ran, browser/lease connection attempts with their failure causes, and which MCP clients connected (`claude-code`, `cursor`).
+
+It also carries a snapshot of the **machine's** state — our own process's memory, free and total system RAM, load average and CPU count — taken at shutdown and again on any crash. This is what separates "your machine ran out of memory" from "Reticle has a bug", which otherwise produce identical-looking failures. No hostname, no username, no paths, no process list.
+
+### Bugs Reticle finds
+
+When Reticle catches a defect in the app it is verifying, it records **that a class of defect was found** — never what it was found in. The event carries the kind (`signal-contradicted`, `duplicate-request`, `console-error`, …) and how it surfaced, and nothing else: no selector, no URL, no element, no description of your app or its behaviour.
+
+This is the number we use to say whether Reticle works at all, and the one we would publish. It is counted conservatively on purpose — a defect explained by a contradiction is not also counted as a failed assertion, because an inflated number would be worse than none.
+
+### Errors and crashes
+
+The SDK that runs inside your page reports its own failures too — an observer that could not start, a patch that would not install. It does this over the **local bridge it is already connected to**; the SDK still makes no outbound request of its own, and the daemon decides what (if anything) is reported onward. What travels is our module name (`network_observer`), the error type, and the message with variables stripped — never your page's URL, and never anything from your app.
+
+Errors are grouped by a **hash of the error's shape**, with every variable part removed first — so `no baseline named 'checkout-v3'` and `no baseline named 'login'` become one anonymous group and neither flow name is sent. Alongside the hash we send that stripped shape (`no baseline named *`) and the tool that produced it, because a hash on its own can be counted but never understood.
+
+A crash additionally carries **Reticle's own stack frames** — `resolveAnchor@act-tools.js:142` — plus the tool that was running, the preceding tool names, and the Node version and CPU architecture. Those frames are our published npm code, readable by anyone who unpacks the tarball.
+
+**Stack frames belonging to your application are dropped entirely**, along with node internals. In the example below only the two Reticle frames survive; your file, your function, and your home directory do not:
+
+```
+at doCheckout (/Users/ada/secret-app/src/checkout.tsx:42:9)   ← dropped
+at resolveAnchor (…/@reticlehq/server/dist/tools/act-tools.js:142:19)   ← sent
+```
+
+`project_profiled` carries: the framework and its major version, a size **bucket** (`tiny` … `huge`, never a file count), whether it is a monorepo, its age in whole **weeks**, how many saved flows/baselines/runs exist, and which Reticle feature families have been used. It also carries whether the project has git at all, whether it has ever been pushed (`none` / `local_only` / `remote`), and which forge hosts it — `github`, `gitlab`, `bitbucket`, `azure`, `sourcehut`, `codeberg`, or just `self_hosted` for anything else. A private git host is usually `git.<company>.com`, so self-hosted repos report **only** that they are self-hosted, never the hostname.
+
+This is how we learn whether people are getting value from the whole product or only a corner of it. No file names, no paths, no flow names, no dependency list.
+
+Every event carries the same few fields:
+
+| Field | What it is | What it is not |
+| --- | --- | --- |
+| `anonymousId` | A random UUID minted locally on first run, stored at `~/.reticle/telemetry-id` | Not derived from your name, email, hardware, or anything else about you |
+| `projectId` | A one-way SHA-256 hash of the git origin URL, or of the directory path outside a repo | Not reversible — we can count _distinct_ projects, but not learn any project's name, URL, or location. Hashing the origin means one repo counts once instead of once per teammate and per clone |
+| `actor` | Whether a person or an agent caused this — a typed `reticle` command is a person, an MCP tool call is the agent | Not a claim about _why_. Whether you asked your agent to verify, or it decided to, lives in a prompt Reticle never sees, and we do not guess |
+| `sessionId` | A random id for this daemon run, held in memory and never written to disk | Not a device id and not persistent — a restarted daemon gets a new one. It only groups one run's own events together |
+| `projectIdSource` | Whether `projectId` came from a shared git origin or from the local directory path | Not the origin or the path. It exists so we can tell when "how many people share this project" is a real measurement — outside a pushed repo there is nothing shared to hash, so those rows always show one user |
+| `version` | The Reticle version running | — |
+| `os` | The platform (`darwin` / `linux` / `win32`) | Not the OS version, hostname, or hardware |
+| `ci` | Whether the run is inside CI | — |
+
+There are no IP-based profiles, no cookies, no fingerprinting, and no person profiles: events are processed in "personless" mode, so they are never joined into an identity.
+
+## What is never sent
+
+Your code. Your app's DOM, network requests or responses, console logs, application state, or screenshots. File paths, project names, or git remote URLs. Your name, email, employer, or any account identifier. Environment variables. Anything typed into the app under test. The names of your flows, baselines, or tests. Error messages, and any stack frame belonging to your application.
+
+A few of these are worth being explicit about, because they are the ones a product team is most tempted by:
+
+- **We do not send your project's name or its GitHub URL.** `projectId` is a one-way hash. It lets us count distinct projects and see that a project came back next week; it cannot be turned back into a repository, and we cannot look you up from it.
+- **We do not try to work out who you work for.** No domain sniffing, no email inference, no matching a repo against a company. If you _want_ us to know, [`reticle identify`](#telling-us-who-you-are) exists and you decide what it says.
+- **We do not record what you asked your agent to verify.** The `verification_completed` event knows that a verification happened and how it turned out. The prompt behind it is not something Reticle can see, and we do not reconstruct it.
+
+## Feedback
+
+Reticle verifies apps for AI agents, and for a long time it had no way to hear when it got something wrong. An agent would hit a tool that misbehaved, work around it, and that knowledge would vanish at the end of the turn. The feedback channel is the fix — and because it is the only part of Reticle that transmits words someone wrote, it gets stricter rules than everything above.
+
+**It is never passive.** There is no code path that sends feedback on its own. A `feedback` event exists only because one of these happened:
+
+| Who | How | What it carries |
+| --- | --- | --- |
+| Your agent | Calls the `reticle_feedback` MCP tool — after a failure, **or to ask for a feature** | What it wrote: an analysis, or a request with the goal behind it, what would improve, and how it works around the gap today. Plus the model it is running, which it tells us because MCP cannot |
+| You | Run `reticle feedback [--rating 1-5] [--bug] "your words"` | Your words and your rating |
+
+Agents are instructed, in the tool description itself, never to include app source, secrets, or user data — to describe the failure in their own words instead.
+
+**It is redacted before it leaves your machine.** Emails, credentials in URLs, `Authorization` headers and API-key assignments, recognizable vendor tokens (`sk-…`, `ghp_…`, `AKIA…`, `xoxb-…`, JWTs), and home-directory paths are stripped and replaced with `[redacted]`. This runs client-side, in [`feedback.ts`](../packages/server/src/telemetry/feedback.ts), so you can read exactly what it removes rather than take our word for it. It is a safety net, not a guarantee — it cannot catch a secret that looks like an ordinary sentence, which is why the instruction above comes first.
+
+**It shows you what it sends.** `reticle feedback` prints the exact payload before transmitting, and names any redaction rule that fired.
+
+Alongside the report, a feedback event carries context about the environment it came from — never about you:
+
+| Field | Example | Why |
+| --- | --- | --- |
+| `stack` / `stackMajor` | `next` / `15` | Which frameworks a bug actually affects. Major version only |
+| `runtime` | `web`, `electron`, `tauri` | Desktop bugs look nothing like browser bugs |
+| `engine` | `blink`, `gecko`, `webkit` | Coarse bucket, never the user-agent string |
+| `driver` | `cdp`, `sdk` | Whether Reticle drove the page or observed your own browser |
+| `client` | `claude-code`, `cursor` | The MCP client's own name from its handshake |
+| `mcpScope` | `user`, `project` | How Reticle is registered |
+| `kind` | `bug`, `gap`, `ambiguity`, `experience` | A defect, a blind spot, an undecidable verdict, or an overall take |
+
+Plus the `version`, `os`, and `ci` fields every event carries. No paths, no project name, no dependency list, no user-agent string.
+
+**It has its own off switch.** To keep the anonymous counters but never send free text:
+
+```bash
+RETICLE_FEEDBACK=0
+```
+
+Every switch in [Your choices](#your-choices) disables feedback too. If telemetry is off, feedback is off — and `reticle feedback` will tell you so rather than silently discard what you wrote.
+
+## Telling us who you are
+
+Everything above is anonymous, and stays that way unless you decide otherwise. If you want us to know who you are — to get support, to ask about an enterprise licence, or to be a design partner — there is one command, and running it is the only way it ever happens:
+
+```bash
+reticle identify --context company --company "Acme" --email you@acme.com
+```
+
+`--context` is the only required part, and `company | side_project | open_source | learning` is the whole vocabulary. You can say "this is a company" without naming it, or name it without leaving an email.
+
+Before it sends anything, it prints what it will send and one thing worth reading carefully: **the identity is linked to this machine's anonymous id, so identifying yourself also connects the anonymous usage already recorded from this machine to what you enter.** That is what makes it useful to us, and it is why you are told before choosing rather than after.
+
+To undo it:
+
+```bash
+reticle identify --forget
+```
+
+That deletes the local file and stops any further sends. To have what was already sent removed, email support@reticlehq.com.
+
+## Where it goes
+
+Events are sent over HTTPS to [PostHog](https://posthog.com) (US cloud), a product-analytics service acting as our data processor, and are used only in aggregate (counts, retention curves, tool popularity). We do not sell or share this data.
+
+## Your choices
+
+Telemetry is on by default and Reticle tells you so the first time it runs — once, in one line, with a pointer to this page. To see the current state at any time:
+
+```bash
+reticle telemetry status
+```
+
+Three ways to opt out, in whatever form fits your setup:
+
+| Method | Scope |
+| --- | --- |
+| `reticle telemetry disable` | This machine, permanently (until `reticle telemetry enable`) |
+| `RETICLE_TELEMETRY=0` | Wherever the variable is set — handy for CI or a fleet-wide profile |
+| `DO_NOT_TRACK=1` | The [cross-tool convention](https://consoledonottrack.com) — Reticle honors it |
+
+Opting out changes nothing about how Reticle works. A failed or blocked telemetry send never delays, alters, or fails a command either — sends are best-effort and asynchronous by design.
+
+To also remove the locally stored random id, delete `~/.reticle/telemetry-id`.
+
+## A note on data protection
+
+The data described above is designed not to identify you: the only identifier is a locally minted random UUID, the project reference is a one-way hash, and no personal data is collected. We collect it on the basis of our legitimate interest in understanding and improving Reticle, we minimize what is collected to the fields listed here, and we honor every opt-out signal above. If you believe something in this design falls short of that intent, please open an issue — that is a bug, and we will treat it as one.
+
+Any change to what is collected will be listed on this page and called out in the release notes of the version that introduces it.

package/docs/testing.md

@@ -0,0 +1,82 @@
+# Testing with `@reticlehq/test` — declarative, signal-bound specs
+
+Driving Reticle interactively is reconnaissance. To turn it into a **repeatable, CI-runnable** suite, write declarative specs with `@reticlehq/test`. Specs bind to **signals and testids — never DOM structure** — so they inherit Reticle's refactor-resistance.
+
+```ts
+import { reticleTest } from '@reticlehq/test';
+
+reticleTest('add a task', async (t) => {
+  await t.act('add-task', 'click');
+  await t.expectElement({ testid: 'task-list' }, 'visible');
+});
+
+reticleTest('ai chat edit', async (t) => {
+  await t.fill('chat-input', 'Make the hook punchier');
+  await t.act('chat-send', 'click');
+  await t.expectNet('POST', '/chat-script', 200);
+  await t.expectSignal('chat:edit-applied', { sections: ['hook'] });
+});
+```
+
+## The test context `t`
+
+A thin, typed façade over Reticle's tools — it resolves testids → refs for you, so specs never touch refs or DOM:
+
+| Method | What it does |
+| --- | --- |
+| `t.act(testid, action, args?)` | perform an action on a testid'd element |
+| `t.fill(testid, value)` | fill an input |
+| `t.actAndWait(testid, action, until)` | act, then block until a predicate holds |
+| `t.expectSignal(name, dataMatches?)` | assert an app signal fired (with optional data match) |
+| `t.expectNet(method, urlContains, status?)` | assert a network call happened |
+| `t.expectElement(query, state?)` / `t.expectText(contains)` / `t.expectAbsent(query)` | DOM assertions |
+| `t.expectNoConsoleErrors()` | assert the flow produced no console errors |
+| `t.state(storeOrRef)` | read a registered store / a component's state |
+| `t.clock.freeze() / advance(ms) / reset()` | deterministic time (toasts, debounces, auto-dismiss) |
+| `t.expectInputModeReal()` | guard: pass under real input, else **skip with a reason** |
+
+Any failed matcher throws with the structured evidence (near-miss, failure reason) so the runner reports _why_.
+
+## Deterministic + honest
+
+- **`t.clock`** bakes `reticle_clock` into the spec, so time-gated UI (a 5s auto-dismiss, a 500ms hover dwell) is tested deterministically instead of racing real timers.
+- **`t.expectInputModeReal()`** — a hover/drag spec asserts native input is active; if it's running synthetic (no CDP), the spec is **skipped with a reason**, never silently passing on a no-op. Enable real input headless with `reticle drive` (see [usage §18](usage.md#18-real-input-mode--native-hover--drag)).
+
+## Run a suite (headless, the same path CI uses)
+
+`bootSession` launches a headless real-input browser at your app and gives the runner a programmatic tool invoker (no MCP/stdio):
+
+```ts
+import { reticleTest, bootSession, runSpecs, createTestContext } from '@reticlehq/test';
+
+// … reticleTest(...) registrations above …
+
+const booted = await bootSession({ driveUrl: 'http://localhost:4310', headless: true });
+const { summary } = await runSpecs({
+  invoke: booted.invoke,
+  now: () => Date.now(),
+  buildContext: (invoke) => createTestContext(invoke, { sessionId: 'my-app' }),
+  print: (line) => process.stdout.write(line + '\n'),
+});
+await booted.close();
+process.exit(summary.failed === 0 ? 0 : 1);
+```
+
+Each spec reports `pass` | `fail` (with evidence) | `skip` (with reason). For CI, emit JUnit:
+
+```ts
+import { toJUnitXml, writeJUnit } from '@reticlehq/test';
+```
+
+## Flows become specs
+
+`.reticle/` flows (see [Flows](flows.md)) can be executed directly as specs — replayed with their `expect`/`success` predicates and skipping `dynamic` (LLM-output) regions — so the recorded map and the suite can't drift apart:
+
+```ts
+import { flowsAsSpecs } from '@reticlehq/test';
+// register one reticleTest per flow under .reticle/flows/
+```
+
+## Authoring tip: record → prune → commit
+
+You don't have to hand-write steps. Drive the flow once (or record it via the panel), let Reticle emit the program, trim it, and commit it as a spec — the regression test is a byproduct of testing, not separate work.

package/docs/token-efficiency.md

@@ -0,0 +1,80 @@
+# Token efficiency: Reticle vs. a full-tree snapshot (Playwright MCP)
+
+Agent browser tools that feed the **whole accessibility tree** to the model every step get expensive fast. Playwright MCP's own ecosystem notes its snapshots _"can exceed 50,000 tokens on complex pages,"_ with a _typical task ~114,000 tokens through MCP._ Reticle is built to ask **narrow questions** instead, so the per-interaction cost stays tiny.
+
+## Head-to-head (measured, same page, same moment)
+
+Measured against the bench dashboard (`apps/bench-app`) **with a 1,000-item list rendered**, after login. Token estimate = characters ÷ 4. Reproduce with the benchmark harness — see `bench/README.md`.
+
+| Payload                                                                        |     Tokens |
+| ------------------------------------------------------------------------------ | ---------: |
+| **Playwright MCP** — with-refs snapshot (the real payload it sends every step) | **~7,300** |
+| Playwright MCP — bare accessibility tree (what we measured directly)           |     ~6,856 |
+| Reticle — `snapshot` `full` (whole page, incl. all 1,000 items)                |     ~4,144 |
+| Reticle — `snapshot` `interactive` (actionable elements only)                  |       ~110 |
+| Reticle — `snapshot` `status` (route / dialogs / counters)                     |        ~31 |
+| Reticle — `query` one element                                                  |        ~28 |
+| Reticle — `observe` (reaction after an action)                                 |        ~39 |
+| Reticle — `assert` verdict                                                     |        ~33 |
+| **Reticle — a full verify loop** (`query` + `observe` + `assert`)              |   **~100** |
+
+**Result on this page:** the common Reticle loop is **~73× leaner** than Playwright MCP's per-step snapshot (100 vs ~7,300 tokens). The bare a11y tree we measured directly is 6,856; Playwright MCP's actual payload adds a `[ref=…]` to every node, pushing it to ~7,300. On the complex pages Playwright's ecosystem cites (50k+), the gap widens to **~100–500×**.
+
+## Diffed snapshots: pay once, then only for changes
+
+After the first snapshot, pass `reticle_snapshot({ diff: true })` to get back **only what changed** since your last look of the same scope/mode (`mode:delta` with added/removed lines, or `mode:unchanged`). A route change auto-resets to a full snapshot, so you never read a misleading cross-page diff.
+
+Measured on a representative 150-row dashboard (the shipped regression benchmark `packages/server/src/tools/snapshot-cost.test.ts`, char/4 proxy):
+
+| Payload                            |    Tokens |
+| ---------------------------------- | --------: |
+| Full re-snapshot (150-row table)   | **4,246** |
+| `diff:true` after a one-row change |    **60** |
+| `diff:true` when nothing changed   |    **17** |
+
+**~99% fewer tokens** to re-look after an action — and because a `delta` carries no stale full tree, it also removes the 60–80K-token stale-context buildup that makes long-running agents start hallucinating selectors that no longer exist.
+
+Every `reticle_snapshot`/`reticle_query` result also carries `cost:{ bytes, tokens }` (estimated) so you can **re-scope before reading** a large body (`mode:interactive`/`status`, a tighter `scope`, or a narrower `query`) instead of paying for it first.
+
+## The other tax: tool schemas, paid on every request
+
+Per-payload leanness is only half the token story. Before an agent reads a single result, it pays for the tool SCHEMAS injected into its context on every request — and this is the metric the field now organises around. A filed issue measures Playwright MCP's default tool list at 14.4k tokens = 7.2% of a Claude Code context window, and Microsoft's own README steers coding agents to the CLI over its MCP on exactly these grounds.
+
+Measured live, all servers in one run, same tokenizer (`bench/harness/schema-tax.mjs`):
+
+| MCP server                                | tools | schema tokens |
+| ----------------------------------------- | ----: | ------------: |
+| **Reticle — the tool surface**            |    17 |    **~4,500** |
+| Playwright MCP                            |    23 |         3,725 |
+| Chrome DevTools MCP                       |    29 |         5,116 |
+| Reticle — `RETICLE_ADVERTISE_ALL_TOOLS=1` |    48 |       ~32,000 |
+
+There is one tool surface: the verify loop advertised directly, plus two meta-tools (`reticle_tools`, `reticle_run`) that reach every other tool on demand. Nothing is unreachable; the cold tail simply is not re-sent every turn.
+
+`RETICLE_ADVERTISE_ALL_TOOLS=1` advertises everything WITH output schemas — a verification switch for suites that call by name, not a mode to run agents in. It is roughly 7x the per-turn cost, which is why it is opt-in: measured, carrying output schemas on the default surface takes it from 18,183 to 41,117 bytes.
+
+The typed result object still travels as `structuredContent` either way; the default surface simply does not advertise the output schema, which an agent reading the `text` block never consumed.
+
+## The honest version
+
+- **Full-tree vs full-tree, the gap is modest (~1.8×):** Reticle `full` (4,144) vs Playwright's with-refs snapshot (~7,300). Reticle collapses generic wrapper nodes, but both include every list item. If you force Reticle to dump the whole page each step, you don't save much.
+- **The savings come from _not needing_ the full tree.** Playwright MCP's primary perception primitive is "return the accessibility tree"; Reticle's is "answer a specific question" (`query`/`assert`/`observe`/scoped or interactive `snapshot`). The win is architectural, not a cleverer serializer.
+- **Cost scales with interactive elements + what you look at, not total DOM.** The 1,000 list items cost ~0 in `interactive` mode because they aren't interactive.
+- **This is tool-output tokens only.** The agent's own reasoning tokens dominate either way — which is the point: keep observation cheap so the budget goes to thinking.
+
+## Why it matters in practice
+
+A 20-step verification flow:
+
+- **Full-tree approach:** ~7,300 tokens × 20 ≈ **~146,000 tokens** (and more on complex pages), plus a vision model if it also screenshots.
+- **Reticle:** ~100 tokens × 20 ≈ **~2,000 tokens**, any model, deterministic.
+
+At scale (long flows, large dashboards, frequent re-runs for regression) that difference is the difference between "too expensive to run every change" and "run it on every edit."
+
+## Method & caveats
+
+- One page, one tool, char/4 token proxy — directional, not a benchmark suite. Absolute numbers vary by page; the _ratio_ is the point.
+- `_snapshotForAI()` (Playwright MCP's exact with-refs payload) was unavailable in the installed Playwright build, so we measured `body.ariaSnapshot()` — the same accessibility tree it serializes; the real MCP payload is equal or slightly larger (it adds `[ref=…]`).
+- Playwright MCP is excellent and Microsoft-backed; this is not a knock on it. It optimizes for cross-browser _driving_; Reticle optimizes for cheap, in-app _verification_. They can coexist (drive with one, assert with the other).
+
+Run it yourself: the benchmark harness in `bench/` (see `bench/README.md`), with the demo + api running.