@reticlehq/server 2.7.0 → 2.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -5
- package/SKILL.md +64 -29
- package/dist/capsule/causal-summary.d.ts +20 -0
- package/dist/capsule/causal-summary.js +31 -6
- package/dist/capsule/divergence.js +7 -1
- package/dist/cli/chromium-hint.d.ts +33 -5
- package/dist/cli/chromium-hint.js +82 -12
- package/dist/cli/cli-doctor.js +28 -25
- package/dist/cli/cli-kill.d.ts +58 -0
- package/dist/cli/cli-kill.js +162 -0
- package/dist/cli/cli-launch.d.ts +17 -0
- package/dist/cli/cli-launch.js +28 -0
- package/dist/cli/cli-parse.d.ts +16 -1
- package/dist/cli/cli-parse.js +32 -9
- package/dist/cli/cli-port.d.ts +8 -0
- package/dist/cli/cli-port.js +16 -0
- package/dist/cli/cli-verify.js +2 -2
- package/dist/cli/doctor-sessions-line.d.ts +12 -0
- package/dist/cli/doctor-sessions-line.js +66 -0
- package/dist/cli/port-holder.d.ts +10 -0
- package/dist/cli/port-holder.js +26 -2
- package/dist/cli.js +99 -14
- package/dist/crawl/crawl-tools.js +12 -2
- package/dist/crawl/crawl.d.ts +13 -0
- package/dist/crawl/crawl.js +50 -2
- package/dist/events/contradictions.d.ts +14 -0
- package/dist/events/contradictions.js +26 -17
- package/dist/events/predicate-eval.d.ts +20 -1
- package/dist/events/predicate-eval.js +125 -3
- package/dist/events/predicate.js +170 -18
- package/dist/flows/flow-classify.js +1 -1
- package/dist/flows/flow-tools.js +12 -4
- package/dist/flows/heal-run.js +1 -1
- package/dist/honesty/blind-spots.js +6 -2
- package/dist/honesty/unsettled.d.ts +40 -0
- package/dist/honesty/unsettled.js +78 -0
- package/dist/honesty/verified.d.ts +7 -0
- package/dist/honesty/verified.js +3 -3
- package/dist/index.js +3 -0
- package/dist/init/agent-rules.d.ts +16 -0
- package/dist/init/agent-rules.js +96 -22
- package/dist/init/astro-patch.d.ts +2 -1
- package/dist/init/astro-patch.js +11 -5
- package/dist/init/capabilities.d.ts +22 -0
- package/dist/init/capabilities.js +76 -0
- package/dist/init/confirm.d.ts +96 -0
- package/dist/init/confirm.js +138 -0
- package/dist/init/cra.js +3 -2
- package/dist/init/non-js-project.js +29 -5
- package/dist/init/plan-framework.js +53 -15
- package/dist/init/plan.d.ts +40 -4
- package/dist/init/plan.js +163 -40
- package/dist/init/run.d.ts +24 -2
- package/dist/init/run.js +148 -42
- package/dist/init/slash-command.d.ts +1 -1
- package/dist/init/slash-command.js +46 -6
- package/dist/init/snippets.d.ts +44 -4
- package/dist/init/snippets.js +161 -37
- package/dist/init/vite-config.js +29 -3
- package/dist/input/network-mock-tools.d.ts +19 -0
- package/dist/input/network-mock-tools.js +29 -4
- package/dist/license/license.js +1 -1
- package/dist/mcp/mcp-proxy.d.ts +69 -3
- package/dist/mcp/mcp-proxy.js +205 -20
- package/dist/mcp/mcp.js +23 -7
- package/dist/mcp/proxy-lifecycle.d.ts +1 -1
- package/dist/mcp/proxy-lifecycle.js +1 -1
- package/dist/pool/browser-pool.d.ts +22 -2
- package/dist/pool/browser-pool.js +44 -4
- package/dist/project/learned-routes.d.ts +24 -0
- package/dist/project/learned-routes.js +63 -0
- package/dist/project/project-store.d.ts +6 -0
- package/dist/project/project-store.js +25 -1
- package/dist/session/no-session-diagnosis.d.ts +19 -0
- package/dist/session/no-session-diagnosis.js +71 -18
- package/dist/session/no-session-watch.js +12 -0
- package/dist/session/ref-provenance.d.ts +54 -0
- package/dist/session/ref-provenance.js +61 -0
- package/dist/session/session-lease.js +1 -1
- package/dist/telemetry/automation-hint.d.ts +17 -0
- package/dist/telemetry/automation-hint.js +71 -0
- package/dist/telemetry/bug-found.d.ts +0 -20
- package/dist/telemetry/bug-found.js +38 -21
- package/dist/telemetry/cli-telemetry.js +7 -1
- package/dist/telemetry/feedback.js +1 -1
- package/dist/telemetry/init-telemetry.js +9 -1
- package/dist/telemetry/install-source.d.ts +36 -0
- package/dist/telemetry/install-source.js +40 -0
- package/dist/telemetry/run-telemetry.js +4 -1
- package/dist/telemetry/session-metrics.js +5 -0
- package/dist/telemetry/telemetry.d.ts +5 -1
- package/dist/telemetry/telemetry.js +15 -2
- package/dist/telemetry/tool-refused.d.ts +19 -0
- package/dist/telemetry/tool-refused.js +55 -0
- package/dist/telemetry/verification-of.js +46 -7
- package/dist/temp-dir.d.ts +27 -0
- package/dist/temp-dir.js +46 -0
- package/dist/tools/act-tools.js +45 -1
- package/dist/tools/assert-grade.d.ts +1 -1
- package/dist/tools/assert-grade.js +27 -5
- package/dist/tools/assert-verdict.js +18 -1
- package/dist/tools/contract-tools.js +4 -0
- package/dist/tools/dynamic-tools.js +1 -1
- package/dist/tools/error-recovery.d.ts +10 -0
- package/dist/tools/error-recovery.js +64 -4
- package/dist/tools/feedback-invite.js +1 -1
- package/dist/tools/invoke-tool.js +139 -23
- package/dist/tools/lease-tools.d.ts +29 -1
- package/dist/tools/lease-tools.js +58 -5
- package/dist/tools/navigate-result.js +1 -1
- package/dist/tools/observe-tools.js +1 -1
- package/dist/tools/observed-nothing.js +1 -1
- package/dist/tools/react-grace.d.ts +48 -0
- package/dist/tools/react-grace.js +87 -0
- package/dist/tools/read-tools.js +20 -4
- package/dist/tools/settle-in-flight.d.ts +79 -0
- package/dist/tools/settle-in-flight.js +106 -0
- package/dist/tools/snapshot-delta.d.ts +7 -0
- package/dist/tools/snapshot-delta.js +130 -6
- package/dist/tools/tool-surface.js +19 -0
- package/dist/tools/tools.js +16 -4
- package/docs/README.md +56 -11
- package/docs/actions.mdx +87 -0
- package/docs/agent-cheatsheet.md +93 -29
- package/docs/architecture.md +46 -35
- package/docs/benchmarks.md +76 -72
- package/docs/best-practices.mdx +211 -0
- package/docs/capabilities.mdx +387 -0
- package/docs/cli/affected.mdx +51 -0
- package/docs/cli/capsules.mdx +69 -0
- package/docs/cli/cloud.mdx +127 -0
- package/docs/cli/doctor.mdx +76 -0
- package/docs/cli/drive.mdx +55 -0
- package/docs/cli/feedback.mdx +67 -0
- package/docs/cli/gate.mdx +64 -0
- package/docs/cli/help.mdx +92 -0
- package/docs/cli/hunt.mdx +57 -0
- package/docs/cli/identify.mdx +62 -0
- package/docs/cli/init.mdx +94 -0
- package/docs/cli/kill.mdx +91 -0
- package/docs/cli/license.mdx +48 -0
- package/docs/cli/mcp.mdx +61 -0
- package/docs/cli/open.mdx +68 -0
- package/docs/cli/restart.mdx +60 -0
- package/docs/cli/rollback.mdx +35 -0
- package/docs/cli/serve.mdx +62 -0
- package/docs/cli/status.mdx +64 -0
- package/docs/cli/stop.mdx +56 -0
- package/docs/cli/telemetry.mdx +57 -0
- package/docs/cli/update.mdx +52 -0
- package/docs/cli/verify.mdx +57 -0
- package/docs/cli/version.mdx +39 -0
- package/docs/cli/watch.mdx +55 -0
- package/docs/cli.mdx +219 -0
- package/docs/debugging.md +13 -9
- package/docs/deploy-checks.md +15 -11
- package/docs/desktop-apps.md +49 -40
- package/docs/desktop.mdx +188 -0
- package/docs/docs.json +363 -0
- package/docs/enterprise.md +20 -16
- package/docs/faq.mdx +88 -0
- package/docs/fixtures.md +11 -7
- package/docs/flows.md +26 -22
- package/docs/for-agents.md +71 -0
- package/docs/frameworks.mdx +199 -0
- package/docs/gate-plan.md +45 -41
- package/docs/gates.md +22 -18
- package/docs/getting-started.md +76 -65
- package/docs/human-control.md +22 -18
- package/docs/index.mdx +101 -0
- package/docs/install-agentic.mdx +215 -0
- package/docs/install-manual.mdx +194 -0
- package/docs/instrumentation.mdx +232 -0
- package/docs/integration-patterns.md +38 -34
- package/docs/local-registry.md +16 -12
- package/docs/multi-agent-testing.md +16 -12
- package/docs/packages/babel-plugin.mdx +70 -0
- package/docs/packages/browser.mdx +111 -0
- package/docs/packages/core.mdx +106 -0
- package/docs/packages/electron.mdx +87 -0
- package/docs/packages/eslint-plugin.mdx +84 -0
- package/docs/packages/next.mdx +91 -0
- package/docs/packages/react.mdx +91 -0
- package/docs/packages/server.mdx +98 -0
- package/docs/packages/tauri.mdx +63 -0
- package/docs/packages/test.mdx +108 -0
- package/docs/packages/vite-plugin.mdx +96 -0
- package/docs/packages.mdx +217 -0
- package/docs/platform-integration.md +33 -26
- package/docs/predicates.mdx +179 -0
- package/docs/quickstart.mdx +211 -0
- package/docs/recipes.mdx +242 -0
- package/docs/skill-file.mdx +105 -0
- package/docs/state-management.mdx +197 -0
- package/docs/style.css +14 -0
- package/docs/system-map.md +47 -32
- package/docs/telemetry-contract.md +93 -50
- package/docs/telemetry-events.mdx +430 -0
- package/docs/telemetry.md +57 -52
- package/docs/testing.md +10 -6
- package/docs/token-efficiency.md +30 -26
- package/docs/tools-act-and-wait.mdx +123 -0
- package/docs/tools-act-sequence.mdx +120 -0
- package/docs/tools-act.mdx +82 -0
- package/docs/tools-assert.mdx +70 -0
- package/docs/tools-console.mdx +59 -0
- package/docs/tools-inspect.mdx +85 -0
- package/docs/tools-navigate.mdx +63 -0
- package/docs/tools-network.mdx +86 -0
- package/docs/tools-observe.mdx +89 -0
- package/docs/tools-overview.mdx +157 -0
- package/docs/tools-query.mdx +82 -0
- package/docs/tools-session-and-feedback.mdx +91 -0
- package/docs/tools-sessions.mdx +65 -0
- package/docs/tools-snapshot.mdx +75 -0
- package/docs/tools-state.mdx +81 -0
- package/docs/tools-tools-and-run.mdx +125 -0
- package/docs/tools-wait-for.mdx +66 -0
- package/docs/troubleshooting.mdx +193 -0
- package/docs/usage.md +175 -153
- package/docs/vs-chrome-devtools-mcp.mdx +82 -0
- package/docs/vs-playwright-mcp.mdx +85 -0
- package/docs/vs-screenshots.mdx +96 -0
- package/docs/what-is-reticle.mdx +63 -0
- package/docs/why-reticle.mdx +107 -0
- package/package.json +2 -2
- package/docs/matrix/2.5.0/claude-code-machine.json +0 -19
- package/docs/matrix/2.5.0/codex-machine.json +0 -20
- package/docs/matrix/2.5.0/cursor-machine.json +0 -22
- package/docs/matrix/2.5.0/gemini-machine.json +0 -22
- package/docs/matrix/2.5.0/opencode-machine.json +0 -22
- package/docs/matrix/2.5.0/vscode-machine.json +0 -22
- package/docs/matrix/2.5.0/windsurf-machine.json +0 -22
- package/docs/matrix/MATRIX.md +0 -26
- package/docs/matrix/README.md +0 -59
- package/docs/mint.json +0 -61
package/docs/telemetry.md
CHANGED
|
@@ -1,18 +1,22 @@
|
|
|
1
|
-
|
|
1
|
+
---
|
|
2
|
+
title: Telemetry
|
|
3
|
+
description: 'The complete description of the anonymous usage data Reticle collects, what it is for, and how to turn it off.'
|
|
4
|
+
icon: chart-line
|
|
5
|
+
---
|
|
2
6
|
|
|
3
|
-
Reticle collects a small amount of anonymous usage data to help us understand whether the tool is useful
|
|
7
|
+
Reticle collects a small amount of anonymous usage data to help us understand whether the tool is useful: which commands people run, which tools agents actually use, and whether people keep using Reticle after they try it. That's what this data is for, and all it is for: making the product better.
|
|
4
8
|
|
|
5
9
|
This page is the complete description of what is collected. If something is not listed here, it is not sent.
|
|
6
10
|
|
|
7
11
|
## The short version
|
|
8
12
|
|
|
9
13
|
- **Anonymous.** We cannot tell who you are, and we do not try.
|
|
10
|
-
- **No code, no app data.** Nothing from the app under test
|
|
14
|
+
- **No code, no app data.** Nothing from the app under test (no DOM, no network traffic, no console output, no source, no file paths) ever leaves your machine.
|
|
11
15
|
- **The one exception is feedback you deliberately send us**, and it is never collected passively. See [Feedback](#feedback) below.
|
|
12
16
|
- **Opt out any time**, permanently, with one command:
|
|
13
17
|
|
|
14
18
|
```bash
|
|
15
|
-
|
|
19
|
+
npx @reticlehq/server telemetry disable
|
|
16
20
|
```
|
|
17
21
|
|
|
18
22
|
## What is sent
|
|
@@ -21,50 +25,51 @@ Thirteen kinds of events, each a single small JSON object:
|
|
|
21
25
|
|
|
22
26
|
| Event | When | Extra data |
|
|
23
27
|
| --- | --- | --- |
|
|
24
|
-
| `reticle_installed` | The first time Reticle runs on a machine |
|
|
28
|
+
| `reticle_installed` | The first time Reticle runs on a machine | n/a |
|
|
25
29
|
| `cli_command_run` | You run a `reticle` command | Which subcommand (`verify`, `status`, …) and which flags were present, by name |
|
|
26
|
-
| `daemon_started` | The local daemon starts |
|
|
27
|
-
| `daemon_stopped` | The local daemon stops | A summary of the session
|
|
28
|
-
| `verification_completed` | A verification produces a verdict | Whether it passed, whether Reticle refused to call a passing check verified, and **why** the verdict came out that way
|
|
29
|
-
| `project_profiled` | Once per daemon start | The shape of the project
|
|
30
|
+
| `daemon_started` | The local daemon starts | n/a |
|
|
31
|
+
| `daemon_stopped` | The local daemon stops | A summary of the session; see below |
|
|
32
|
+
| `verification_completed` | A verification produces a verdict | Whether it passed, whether Reticle refused to call a passing check verified, and **why** the verdict came out that way; see below |
|
|
33
|
+
| `project_profiled` | Once per daemon start | The shape of the project; see below |
|
|
30
34
|
| `version_changed` | You update or roll back | The two version numbers, and which direction |
|
|
31
|
-
| `runtime_crashed` | The daemon hits an uncaught error | The error's type, **Reticle's own** stack frames, and the message with variables stripped
|
|
35
|
+
| `runtime_crashed` | The daemon hits an uncaught error | The error's type, **Reticle's own** stack frames, and the message with variables stripped; see below |
|
|
32
36
|
| `mcp_client_connected` | An agent attaches to the daemon | Whether it is a reconnect, and how long the daemon had been idle |
|
|
33
|
-
| `mcp_connection_lost` | The agent's MCP tools go away | Which stage (`first`, or `budget_spent` when it stopped retrying), the cause, and the attempt count. **At most twice per session
|
|
34
|
-
| `init_completed` | `reticle init` finishes | Whether it worked,
|
|
35
|
-
| `bug_found` | Reticle finds a defect in the app under test | The **kind** of defect (`signal-contradicted`, `console-error`, …) and how it was found
|
|
36
|
-
| `
|
|
37
|
-
| `
|
|
37
|
+
| `mcp_connection_lost` | The agent's MCP tools go away | Which stage (`first`, or `budget_spent` when it stopped retrying), the cause, and the attempt count. **At most twice per session**: one measured afternoon produced 547 reconnects, and an event each would bill for the pathology instead of measuring it |
|
|
38
|
+
| `init_completed` | `reticle init` finishes | Whether it worked, a classified reason when it did not, and which published install route you came through, DECLARED by that route via `RETICLE_INSTALL_SOURCE` and never inferred, and `unknown` when nothing declared one |
|
|
39
|
+
| `bug_found` | Reticle finds a defect in the app under test | The **kind** of defect (`signal-contradicted`, `console-error`, …) and how it was found, never what it was found in |
|
|
40
|
+
| `tool_refused` | A tool cannot do what an agent asked | Which tool, why (one of six buckets: no session, no match, unsupported, bad arguments, not ready, other), and whether the same call was retried. **Never the message**, because a refusal interpolates whatever you asked for, so only the tool name and the bucket are sent. Capped at 50 per daemon run |
|
|
41
|
+
| `feedback_submitted` | **Only** when you or your agent explicitly send feedback | The report; see [Feedback](#feedback). The AGENT's call does not wait for the network: the receipt says `accepted` (validated, redacted, queued), never `sent`, and a delivery that then fails is reported back on the agent's next tool result. `reticle feedback` typed by a human still waits, because a person at a terminal is owed the real answer |
|
|
42
|
+
| `identified` | **Only** when you run `reticle identify` | What you chose to tell us; see [Telling us who you are](#telling-us-who-you-are) |
|
|
38
43
|
|
|
39
|
-
**There is no per-tool-call event.** Tool usage is counted in memory and leaves once, inside `daemon_stopped`, as a histogram like `{"reticle_act": 40, "reticle_assert": 12}`. That is counts of tool NAMES from a fixed list we define
|
|
44
|
+
**There is no per-tool-call event.** Tool usage is counted in memory and leaves once, inside `daemon_stopped`, as a histogram like `{"reticle_act": 40, "reticle_assert": 12}`. That is counts of tool NAMES from a fixed list we define, never arguments, results, selectors, or URLs.
|
|
40
45
|
|
|
41
|
-
**Parameter and flag NAMES are collected; their VALUES are not.** We record that `reticle_act` was called with `ref` and `action`, and that you ran `reticle serve --headed --port`. We do not record what you set them to. This distinction is the whole safety property, and it is absolute for CLI flags: `--http-token` holds a secret, `--drive` holds a URL, `--storage-state` holds a file path, so no flag value is ever sent. For tool parameters there is one narrow exception
|
|
46
|
+
**Parameter and flag NAMES are collected; their VALUES are not.** We record that `reticle_act` was called with `ref` and `action`, and that you ran `reticle serve --headed --port`. We do not record what you set them to. This distinction is the whole safety property, and it is absolute for CLI flags: `--http-token` holds a secret, `--drive` holds a URL, `--storage-state` holds a file path, so no flag value is ever sent. For tool parameters there is one narrow exception: a short, explicit allowlist of parameters whose values are enums _we_ defined (`action: "click"`), listed in [`argument-shape.ts`](../packages/server/src/telemetry/argument-shape.ts). A value outside that allowlist reports as `other`, so a future schema change cannot quietly start forwarding free text. `reticle_act`'s `args` (the text being typed into your app, which on a login form is a password) is **never** in that allowlist.
|
|
42
47
|
|
|
43
48
|
`daemon_stopped` carries: how long the session ran, how many tool calls and of which tools, how long each tool took (total and worst case), how many failed, how many verifications ran, browser/lease connection attempts with their failure causes, and which MCP clients connected (`claude-code`, `cursor`).
|
|
44
49
|
|
|
45
|
-
It also carries a snapshot of the **machine's** state
|
|
50
|
+
It also carries a snapshot of the **machine's** state (our own process's memory, free and total system RAM, load average and CPU count), taken at shutdown and again on any crash. This is what separates "your machine ran out of memory" from "Reticle has a bug", which otherwise produce identical-looking failures. No hostname, no username, no paths, no process list.
|
|
46
51
|
|
|
47
52
|
### Why a verdict came out that way
|
|
48
53
|
|
|
49
54
|
`verification_completed` carries the **clause** that decided the verdict, from a fixed list we define: `proved`, `contradicted`, `assertion_failed`, `already_true`, `unclean_capture`, `vacuous_grade`, `outcome_pending`, `outcome_unread`, `unsettled`, `observation_lost`, `inconclusive`.
|
|
50
55
|
|
|
51
|
-
It exists because `verified: "unknown"` covered seven different situations belonging to three different owners
|
|
56
|
+
It exists because `verified: "unknown"` covered seven different situations belonging to three different owners (your app, your agent, and Reticle's own blind spots), and they arrived as one value. It is a **name from our own vocabulary**, never a description of your app: `contradicted` says two channels disagreed, not which ones, about what, or on which page.
|
|
52
57
|
|
|
53
|
-
When the clause is `unclean_capture`, one further name says which of our three losses caused it
|
|
58
|
+
When the clause is `unclean_capture`, one further name says which of our three losses caused it: `buffer_loss` (our server's event buffer), `transport_gap` (our browser-side queue), `blind_spot` (a boundary in the page, such as a cross-origin frame), or `other`. All four are facts about **Reticle's** ability to observe, not about what it observed. This field is how we found that Reticle was refusing to answer over windows that were completely intact.
|
|
54
59
|
|
|
55
60
|
### Bugs Reticle finds
|
|
56
61
|
|
|
57
|
-
When Reticle catches a defect in the app it is verifying, it records **that a class of defect was found
|
|
62
|
+
When Reticle catches a defect in the app it is verifying, it records **that a class of defect was found**, never what it was found in. The event carries the kind (`signal-contradicted`, `duplicate-request`, `console-error`, …) and how it surfaced, and nothing else: no selector, no URL, no element, no description of your app or its behaviour.
|
|
58
63
|
|
|
59
|
-
This is the number we use to say whether Reticle works at all, and the one we would publish. It is counted conservatively on purpose
|
|
64
|
+
This is the number we use to say whether Reticle works at all, and the one we would publish. It is counted conservatively on purpose: a defect explained by a contradiction is not also counted as a failed assertion, because an inflated number would be worse than none.
|
|
60
65
|
|
|
61
66
|
### Errors and crashes
|
|
62
67
|
|
|
63
|
-
The SDK that runs inside your page reports its own failures too
|
|
68
|
+
The SDK that runs inside your page reports its own failures too: an observer that could not start, a patch that would not install. It does this over the **local bridge it is already connected to**; the SDK still makes no outbound request of its own, and the daemon decides what (if anything) is reported onward. What travels is our module name (`network_observer`), the error type, and the message with variables stripped, never your page's URL, and never anything from your app.
|
|
64
69
|
|
|
65
|
-
Errors are grouped by a **hash of the error's shape**, with every variable part removed first
|
|
70
|
+
Errors are grouped by a **hash of the error's shape**, with every variable part removed first, so `no baseline named 'checkout-v3'` and `no baseline named 'login'` become one anonymous group and neither flow name is sent. Alongside the hash we send that stripped shape (`no baseline named *`) and the tool that produced it, because a hash on its own can be counted but never understood.
|
|
66
71
|
|
|
67
|
-
A crash additionally carries **Reticle's own stack frames**
|
|
72
|
+
A crash additionally carries **Reticle's own stack frames** (`resolveAnchor@act-tools.js:142`), plus the tool that was running, the preceding tool names, and the Node version and CPU architecture. Those frames are our published npm code, readable by anyone who unpacks the tarball.
|
|
68
73
|
|
|
69
74
|
**Stack frames belonging to your application are dropped entirely**, along with node internals. In the example below only the two Reticle frames survive; your file, your function, and your home directory do not:
|
|
70
75
|
|
|
@@ -73,11 +78,11 @@ at doCheckout (/Users/ada/secret-app/src/checkout.tsx:42:9) ← dropped
|
|
|
73
78
|
at resolveAnchor (…/@reticlehq/server/dist/tools/act-tools.js:142:19) ← sent
|
|
74
79
|
```
|
|
75
80
|
|
|
76
|
-
**One narrow exception, for the crash that otherwise says nothing.** A refused connection
|
|
81
|
+
**One narrow exception, for the crash that otherwise says nothing.** A refused connection (`connect ECONNREFUSED`) has a stack that is _entirely_ node internals, so the rule above correctly keeps nothing and the report arrives with no location at all. Those crashes now also carry the failing **syscall** (`connect`), the **errno** (`ECONNREFUSED`), whether the target was **loopback** (one boolean), whether the port was **one of Reticle's own** (the enum `reticle` / `other`), and the innermost frame naming **Node's own source** (`node:net:1637`).
|
|
77
82
|
|
|
78
|
-
The address and the port number are used to compute those two answers and are then discarded
|
|
83
|
+
The address and the port number are used to compute those two answers and are then discarded; neither is ever sent. The Node frame is a line in Node's published source, not yours: it says a connect failed rather than a DNS lookup, and carries nothing about your machine, your app, or your directory layout. The frame is included **only** when the crash is a system error and no Reticle frame survived: the report that would otherwise be blind.
|
|
79
84
|
|
|
80
|
-
`project_profiled` carries: the framework and its major version, a size **bucket** (`tiny` … `huge`, never a file count), whether it is a monorepo, its age in whole **weeks**, how many saved flows/baselines/runs exist, and which Reticle feature families have been used. It also carries whether the project has git at all, whether it has ever been pushed (`none` / `local_only` / `remote`), and which forge hosts it
|
|
85
|
+
`project_profiled` carries: the framework and its major version, a size **bucket** (`tiny` … `huge`, never a file count), whether it is a monorepo, its age in whole **weeks**, how many saved flows/baselines/runs exist, and which Reticle feature families have been used. It also carries whether the project has git at all, whether it has ever been pushed (`none` / `local_only` / `remote`), and which forge hosts it: `github`, `gitlab`, `bitbucket`, `azure`, `sourcehut`, `codeberg`, or just `self_hosted` for anything else. A private git host is usually `git.<company>.com`, so self-hosted repos report **only** that they are self-hosted, never the hostname.
|
|
81
86
|
|
|
82
87
|
This is how we learn whether people are getting value from the whole product or only a corner of it. No file names, no paths, no flow names, no dependency list.
|
|
83
88
|
|
|
@@ -86,13 +91,13 @@ Every event carries the same few fields:
|
|
|
86
91
|
| Field | What it is | What it is not |
|
|
87
92
|
| --- | --- | --- |
|
|
88
93
|
| `anonymousId` | A random UUID minted locally on first run, stored at `~/.reticle/telemetry-id` | Not derived from your name, email, hardware, or anything else about you |
|
|
89
|
-
| `projectId` | A one-way SHA-256 hash of the git origin URL, or of the directory path outside a repo | Not reversible
|
|
90
|
-
| `actor` | Whether a person or an agent caused this
|
|
91
|
-
| `sessionId` | A random id for this daemon run, held in memory and never written to disk | Not a device id and not persistent
|
|
92
|
-
| `projectIdSource` | Whether `projectId` came from a shared git origin or from the local directory path | Not the origin or the path. It exists so we can tell when "how many people share this project" is a real measurement
|
|
93
|
-
| `version` | The Reticle version running |
|
|
94
|
+
| `projectId` | A one-way SHA-256 hash of the git origin URL, or of the directory path outside a repo | Not reversible: we can count _distinct_ projects, but not learn any project's name, URL, or location. Hashing the origin means one repo counts once instead of once per teammate and per clone |
|
|
95
|
+
| `actor` | Whether a person or an agent caused this: a typed `reticle` command is a person, an MCP tool call is the agent | Not a claim about _why_. Whether you asked your agent to verify, or it decided to, lives in a prompt Reticle never sees, and we do not guess |
|
|
96
|
+
| `sessionId` | A random id for this daemon run, held in memory and never written to disk | Not a device id and not persistent: a restarted daemon gets a new one. It only groups one run's own events together |
|
|
97
|
+
| `projectIdSource` | Whether `projectId` came from a shared git origin or from the local directory path | Not the origin or the path. It exists so we can tell when "how many people share this project" is a real measurement. Outside a pushed repo there is nothing shared to hash, so those rows always show one user |
|
|
98
|
+
| `version` | The Reticle version running | n/a |
|
|
94
99
|
| `os` | The platform (`darwin` / `linux` / `win32`) | Not the OS version, hostname, or hardware |
|
|
95
|
-
| `ci` | Whether the run is inside CI |
|
|
100
|
+
| `ci` | Whether the run is inside CI | n/a |
|
|
96
101
|
|
|
97
102
|
There are no IP-based profiles, no cookies, no fingerprinting, and no person profiles: events are processed in "personless" mode, so they are never joined into an identity.
|
|
98
103
|
|
|
@@ -108,23 +113,23 @@ A few of these are worth being explicit about, because they are the ones a produ
|
|
|
108
113
|
|
|
109
114
|
## Feedback
|
|
110
115
|
|
|
111
|
-
Reticle verifies apps for AI agents, and for a long time it had no way to hear when it got something wrong. An agent would hit a tool that misbehaved, work around it, and that knowledge would vanish at the end of the turn. The feedback channel is the fix
|
|
116
|
+
Reticle verifies apps for AI agents, and for a long time it had no way to hear when it got something wrong. An agent would hit a tool that misbehaved, work around it, and that knowledge would vanish at the end of the turn. The feedback channel is the fix, and because it is the only part of Reticle that transmits words someone wrote, it gets stricter rules than everything above.
|
|
112
117
|
|
|
113
118
|
**It is never passive.** There is no code path that sends feedback on its own. A `feedback` event exists only because one of these happened:
|
|
114
119
|
|
|
115
120
|
| Who | How | What it carries |
|
|
116
121
|
| --- | --- | --- |
|
|
117
|
-
| Your agent | Calls the `reticle_feedback` MCP tool
|
|
118
|
-
| You | Run `
|
|
119
|
-
| Your agent, before Reticle works | Runs `reticle feedback --agent --kind <bug\|gap\|ambiguity\|feature_request\|improvement> "what happened"` | The same words, filed from the shell. This exists for the phase where there is no daemon and no MCP tools
|
|
122
|
+
| Your agent | Calls the `reticle_feedback` MCP tool (after a failure, **or to ask for a feature**) | What it wrote: an analysis, or a request with the goal behind it, what would improve, and how it works around the gap today. Plus the model it is running, which it tells us because MCP cannot |
|
|
123
|
+
| You | Run `npx @reticlehq/server feedback [--rating 1-5] [--bug] "your words"` | Your words and your rating |
|
|
124
|
+
| Your agent, before Reticle works | Runs `reticle feedback --agent --kind <bug\|gap\|ambiguity\|feature_request\|improvement> "what happened"` | The same words, filed from the shell. This exists for the phase where there is no daemon and no MCP tools (a failed `init`, half-finished wiring), which is the failure we are otherwise never told about |
|
|
120
125
|
|
|
121
|
-
Agents are instructed, in the tool description itself, never to include app source, secrets, or user data
|
|
126
|
+
Agents are instructed, in the tool description itself, never to include app source, secrets, or user data, and to describe the failure in their own words instead.
|
|
122
127
|
|
|
123
|
-
**It is redacted before it leaves your machine.** Emails, credentials in URLs, `Authorization` headers and API-key assignments, recognizable vendor tokens (`sk-…`, `ghp_…`, `AKIA…`, `xoxb-…`, JWTs), and home-directory paths are stripped and replaced with `[redacted]`. This runs client-side, in [`feedback.ts`](../packages/server/src/telemetry/feedback.ts), so you can read exactly what it removes rather than take our word for it. It is a safety net, not a guarantee
|
|
128
|
+
**It is redacted before it leaves your machine.** Emails, credentials in URLs, `Authorization` headers and API-key assignments, recognizable vendor tokens (`sk-…`, `ghp_…`, `AKIA…`, `xoxb-…`, JWTs), and home-directory paths are stripped and replaced with `[redacted]`. This runs client-side, in [`feedback.ts`](../packages/server/src/telemetry/feedback.ts), so you can read exactly what it removes rather than take our word for it. It is a safety net, not a guarantee: it cannot catch a secret that looks like an ordinary sentence, which is why the instruction above comes first.
|
|
124
129
|
|
|
125
130
|
**It shows you what it sends.** `reticle feedback` prints the exact payload before transmitting, and names any redaction rule that fired.
|
|
126
131
|
|
|
127
|
-
Alongside the report, a feedback event carries context about the environment it came from
|
|
132
|
+
Alongside the report, a feedback event carries context about the environment it came from, never about you:
|
|
128
133
|
|
|
129
134
|
| Field | Example | Why |
|
|
130
135
|
| --- | --- | --- |
|
|
@@ -144,26 +149,26 @@ Plus the `version`, `os`, and `ci` fields every event carries. No paths, no proj
|
|
|
144
149
|
RETICLE_FEEDBACK=0
|
|
145
150
|
```
|
|
146
151
|
|
|
147
|
-
Every switch in [Your choices](#your-choices) disables feedback too. If telemetry is off, feedback is off
|
|
152
|
+
Every switch in [Your choices](#your-choices) disables feedback too. If telemetry is off, feedback is off, and `reticle feedback` will tell you so rather than silently discard what you wrote.
|
|
148
153
|
|
|
149
154
|
## Telling us who you are
|
|
150
155
|
|
|
151
|
-
Everything above is anonymous, and stays that way unless you decide otherwise. If you want us to know who you are
|
|
156
|
+
Everything above is anonymous, and stays that way unless you decide otherwise. If you want us to know who you are (to get support, to ask about an enterprise licence, or to be a design partner), there is one command, and running it is the only way it ever happens:
|
|
152
157
|
|
|
153
158
|
```bash
|
|
154
|
-
|
|
159
|
+
npx @reticlehq/server identify --context company --company "Acme" --email you@acme.com
|
|
155
160
|
```
|
|
156
161
|
|
|
157
162
|
`--context` is the only required part, and `company | side_project | open_source | learning` is the whole vocabulary. You can say "this is a company" without naming it, or name it without leaving an email.
|
|
158
163
|
|
|
159
164
|
Before it sends anything, it prints what it will send and one thing worth reading carefully: **the identity is linked to this machine's anonymous id, so identifying yourself also connects the anonymous usage already recorded from this machine to what you enter.** That is what makes it useful to us, and it is why you are told before choosing rather than after.
|
|
160
165
|
|
|
161
|
-
**One place mentions it to you.** After you send feedback as a human
|
|
166
|
+
**One place mentions it to you.** After you send feedback as a human (never as an agent, and never if you have already identified or already declined), the receipt prints one line offering this command, because a bug report we cannot reply to is a conversation that ends after one sentence. Your feedback has already been sent by then: the line is an offer, not a question, and nothing is gated on it. **No address is ever attached to the feedback itself**: it would put personal data on the anonymous stream, which is exactly what the rest of this page promises not to do.
|
|
162
167
|
|
|
163
168
|
To undo it:
|
|
164
169
|
|
|
165
170
|
```bash
|
|
166
|
-
|
|
171
|
+
npx @reticlehq/server identify --forget
|
|
167
172
|
```
|
|
168
173
|
|
|
169
174
|
That deletes the local file and stops any further sends. To have what was already sent removed, email support@reticlehq.com.
|
|
@@ -174,10 +179,10 @@ Events are sent over HTTPS to [PostHog](https://posthog.com) (US cloud), a produ
|
|
|
174
179
|
|
|
175
180
|
## Your choices
|
|
176
181
|
|
|
177
|
-
Telemetry is on by default and Reticle tells you so the first time it runs
|
|
182
|
+
Telemetry is on by default and Reticle tells you so the first time it runs: once, in one line, with a pointer to this page. To see the current state at any time:
|
|
178
183
|
|
|
179
184
|
```bash
|
|
180
|
-
|
|
185
|
+
npx @reticlehq/server telemetry status
|
|
181
186
|
```
|
|
182
187
|
|
|
183
188
|
Three ways to opt out, in whatever form fits your setup:
|
|
@@ -185,15 +190,15 @@ Three ways to opt out, in whatever form fits your setup:
|
|
|
185
190
|
| Method | Scope |
|
|
186
191
|
| --- | --- |
|
|
187
192
|
| `reticle telemetry disable` | This machine, permanently (until `reticle telemetry enable`) |
|
|
188
|
-
| `RETICLE_TELEMETRY=0` | Wherever the variable is set
|
|
189
|
-
| `DO_NOT_TRACK=1` | The [cross-tool convention](https://consoledonottrack.com)
|
|
193
|
+
| `RETICLE_TELEMETRY=0` | Wherever the variable is set; handy for CI or a fleet-wide profile |
|
|
194
|
+
| `DO_NOT_TRACK=1` | The [cross-tool convention](https://consoledonottrack.com), which Reticle honors |
|
|
190
195
|
|
|
191
|
-
Opting out changes nothing about how Reticle works. A failed or blocked telemetry send never delays, alters, or fails a command either
|
|
196
|
+
Opting out changes nothing about how Reticle works. A failed or blocked telemetry send never delays, alters, or fails a command either; sends are best-effort and asynchronous by design.
|
|
192
197
|
|
|
193
198
|
To also remove the locally stored random id, delete `~/.reticle/telemetry-id`.
|
|
194
199
|
|
|
195
200
|
## A note on data protection
|
|
196
201
|
|
|
197
|
-
The data described above is designed not to identify you: the only identifier is a locally minted random UUID, the project reference is a one-way hash, and no personal data is collected. We collect it on the basis of our legitimate interest in understanding and improving Reticle, we minimize what is collected to the fields listed here, and we honor every opt-out signal above. If you believe something in this design falls short of that intent, please open an issue
|
|
202
|
+
The data described above is designed not to identify you: the only identifier is a locally minted random UUID, the project reference is a one-way hash, and no personal data is collected. We collect it on the basis of our legitimate interest in understanding and improving Reticle, we minimize what is collected to the fields listed here, and we honor every opt-out signal above. If you believe something in this design falls short of that intent, please open an issue. That is a bug, and we will treat it as one.
|
|
198
203
|
|
|
199
204
|
Any change to what is collected will be listed on this page and called out in the release notes of the version that introduces it.
|
package/docs/testing.md
CHANGED
|
@@ -1,6 +1,10 @@
|
|
|
1
|
-
|
|
1
|
+
---
|
|
2
|
+
title: Specs for CI
|
|
3
|
+
description: 'Turn an interactive drive into a repeatable suite with @reticlehq/test: declarative specs bound to signals, never DOM structure.'
|
|
4
|
+
icon: vial
|
|
5
|
+
---
|
|
2
6
|
|
|
3
|
-
Driving Reticle interactively is reconnaissance. To turn it into a **repeatable, CI-runnable** suite, write declarative specs with `@reticlehq/test`. Specs bind to **signals and testids
|
|
7
|
+
Driving Reticle interactively is reconnaissance. To turn it into a **repeatable, CI-runnable** suite, write declarative specs with `@reticlehq/test`. Specs bind to **signals and testids, never DOM structure**, so they inherit Reticle's refactor-resistance.
|
|
4
8
|
|
|
5
9
|
```ts
|
|
6
10
|
import { reticleTest } from '@reticlehq/test';
|
|
@@ -20,7 +24,7 @@ reticleTest('ai chat edit', async (t) => {
|
|
|
20
24
|
|
|
21
25
|
## The test context `t`
|
|
22
26
|
|
|
23
|
-
A thin, typed façade over Reticle's tools
|
|
27
|
+
A thin, typed façade over Reticle's tools. It resolves testids → refs for you, so specs never touch refs or DOM:
|
|
24
28
|
|
|
25
29
|
| Method | What it does |
|
|
26
30
|
| --- | --- |
|
|
@@ -40,7 +44,7 @@ Any failed matcher throws with the structured evidence (near-miss, failure reaso
|
|
|
40
44
|
## Deterministic + honest
|
|
41
45
|
|
|
42
46
|
- **`t.clock`** bakes `reticle_clock` into the spec, so time-gated UI (a 5s auto-dismiss, a 500ms hover dwell) is tested deterministically instead of racing real timers.
|
|
43
|
-
- **`t.expectInputModeReal()
|
|
47
|
+
- **`t.expectInputModeReal()`**: a hover/drag spec asserts native input is active; if it's running synthetic (no CDP), the spec is **skipped with a reason**, never silently passing on a no-op. Enable real input headless with `reticle drive` (see [usage §18](usage.md#18-real-input-mode-native-hover--drag)).
|
|
44
48
|
|
|
45
49
|
## Run a suite (headless, the same path CI uses)
|
|
46
50
|
|
|
@@ -70,7 +74,7 @@ import { toJUnitXml, writeJUnit } from '@reticlehq/test';
|
|
|
70
74
|
|
|
71
75
|
## Flows become specs
|
|
72
76
|
|
|
73
|
-
`.reticle/` flows (see [Flows](flows.md)) can be executed directly as specs
|
|
77
|
+
`.reticle/` flows (see [Flows](flows.md)) can be executed directly as specs, replayed with their `expect`/`success` predicates and skipping `dynamic` (LLM-output) regions, so the recorded map and the suite can't drift apart:
|
|
74
78
|
|
|
75
79
|
```ts
|
|
76
80
|
import { flowsAsSpecs } from '@reticlehq/test';
|
|
@@ -79,4 +83,4 @@ import { flowsAsSpecs } from '@reticlehq/test';
|
|
|
79
83
|
|
|
80
84
|
## Authoring tip: record → prune → commit
|
|
81
85
|
|
|
82
|
-
You don't have to hand-write steps. Drive the flow once (or record it via the panel), let Reticle emit the program, trim it, and commit it as a spec
|
|
86
|
+
You don't have to hand-write steps. Drive the flow once (or record it via the panel), let Reticle emit the program, trim it, and commit it as a spec. The regression test is a byproduct of testing, not separate work.
|
package/docs/token-efficiency.md
CHANGED
|
@@ -1,24 +1,28 @@
|
|
|
1
|
-
|
|
1
|
+
---
|
|
2
|
+
title: Token efficiency
|
|
3
|
+
description: Why asking narrow questions costs a fraction of feeding the whole accessibility tree to the model every step.
|
|
4
|
+
icon: coins
|
|
5
|
+
---
|
|
2
6
|
|
|
3
7
|
Agent browser tools that feed the **whole accessibility tree** to the model every step get expensive fast. Playwright MCP's own ecosystem notes its snapshots _"can exceed 50,000 tokens on complex pages,"_ with a _typical task ~114,000 tokens through MCP._ Reticle is built to ask **narrow questions** instead, so the per-interaction cost stays tiny.
|
|
4
8
|
|
|
5
9
|
## Head-to-head (measured, same page, same moment)
|
|
6
10
|
|
|
7
|
-
Measured against the bench dashboard (`apps/bench-app`) **with a 1,000-item list rendered**, after login. Token estimate = characters ÷ 4. Reproduce with the benchmark harness
|
|
11
|
+
Measured against the bench dashboard (`apps/bench-app`) **with a 1,000-item list rendered**, after login. Token estimate = characters ÷ 4. Reproduce with the benchmark harness; see `bench/README.md`.
|
|
8
12
|
|
|
9
|
-
| Payload
|
|
10
|
-
|
|
|
11
|
-
| **Playwright MCP
|
|
12
|
-
| Playwright MCP
|
|
13
|
-
| Reticle
|
|
14
|
-
| Reticle
|
|
15
|
-
| Reticle
|
|
16
|
-
| Reticle
|
|
17
|
-
| Reticle
|
|
18
|
-
| Reticle
|
|
19
|
-
| **Reticle
|
|
13
|
+
| Payload | Tokens |
|
|
14
|
+
| ----------------------------------------------------------------------------- | ---------: |
|
|
15
|
+
| **Playwright MCP**, with-refs snapshot (the real payload it sends every step) | **~7,300** |
|
|
16
|
+
| Playwright MCP, bare accessibility tree (what we measured directly) | ~6,856 |
|
|
17
|
+
| Reticle `snapshot` `full` (whole page, incl. all 1,000 items) | ~4,144 |
|
|
18
|
+
| Reticle `snapshot` `interactive` (actionable elements only) | ~110 |
|
|
19
|
+
| Reticle `snapshot` `status` (route / dialogs / counters) | ~31 |
|
|
20
|
+
| Reticle `query`, one element | ~28 |
|
|
21
|
+
| Reticle `observe` (reaction after an action) | ~39 |
|
|
22
|
+
| Reticle `assert` verdict | ~33 |
|
|
23
|
+
| **Reticle, a full verify loop** (`query` + `observe` + `assert`) | **~100** |
|
|
20
24
|
|
|
21
|
-
**Result on this page:** the common Reticle loop is **~73× leaner** than Playwright MCP's per-step snapshot (100 vs ~7,300 tokens). The bare a11y tree we measured directly is 6,856; Playwright MCP's actual payload adds a `[ref=…]` to every node, pushing it to ~7,300. On the complex pages Playwright's ecosystem cites (50k+), the gap widens to **~100
|
|
25
|
+
**Result on this page:** the common Reticle loop is **~73× leaner** than Playwright MCP's per-step snapshot (100 vs ~7,300 tokens). The bare a11y tree we measured directly is 6,856; Playwright MCP's actual payload adds a `[ref=…]` to every node, pushing it to ~7,300. On the complex pages Playwright's ecosystem cites (50k+), the gap widens to **~100× to 500×**.
|
|
22
26
|
|
|
23
27
|
## Diffed snapshots: pay once, then only for changes
|
|
24
28
|
|
|
@@ -32,26 +36,26 @@ Measured on a representative 150-row dashboard (the shipped regression benchmark
|
|
|
32
36
|
| `diff:true` after a one-row change | **60** |
|
|
33
37
|
| `diff:true` when nothing changed | **17** |
|
|
34
38
|
|
|
35
|
-
**~99% fewer tokens** to re-look after an action
|
|
39
|
+
**~99% fewer tokens** to re-look after an action, and because a `delta` carries no stale full tree, it also removes the 60K to 80K-token stale-context buildup that makes long-running agents start hallucinating selectors that no longer exist.
|
|
36
40
|
|
|
37
41
|
Every `reticle_snapshot`/`reticle_query` result also carries `cost:{ bytes, tokens }` (estimated) so you can **re-scope before reading** a large body (`mode:interactive`/`status`, a tighter `scope`, or a narrower `query`) instead of paying for it first.
|
|
38
42
|
|
|
39
43
|
## The other tax: tool schemas, paid on every request
|
|
40
44
|
|
|
41
|
-
Per-payload leanness is only half the token story. Before an agent reads a single result, it pays for the tool SCHEMAS injected into its context on every request
|
|
45
|
+
Per-payload leanness is only half the token story. Before an agent reads a single result, it pays for the tool SCHEMAS injected into its context on every request, and this is the metric the field now organises around. A filed issue measures Playwright MCP's default tool list at 14.4k tokens = 7.2% of a Claude Code context window, and Microsoft's own README steers coding agents to the CLI over its MCP on exactly these grounds.
|
|
42
46
|
|
|
43
47
|
Measured live, all servers in one run, same tokenizer (`bench/harness/schema-tax.mjs`):
|
|
44
48
|
|
|
45
|
-
| MCP server
|
|
46
|
-
|
|
|
47
|
-
| **Reticle
|
|
48
|
-
| Playwright MCP
|
|
49
|
-
| Chrome DevTools MCP
|
|
50
|
-
| Reticle
|
|
49
|
+
| MCP server | tools | schema tokens |
|
|
50
|
+
| ---------------------------------------- | ----: | ------------: |
|
|
51
|
+
| **Reticle, the tool surface** | 18 | **~4,930** |
|
|
52
|
+
| Playwright MCP | 23 | 3,725 |
|
|
53
|
+
| Chrome DevTools MCP | 29 | 5,116 |
|
|
54
|
+
| Reticle, `RETICLE_ADVERTISE_ALL_TOOLS=1` | 48 | ~30,200 |
|
|
51
55
|
|
|
52
56
|
There is one tool surface: the verify loop advertised directly, plus two meta-tools (`reticle_tools`, `reticle_run`) that reach every other tool on demand. Nothing is unreachable; the cold tail simply is not re-sent every turn.
|
|
53
57
|
|
|
54
|
-
`RETICLE_ADVERTISE_ALL_TOOLS=1` advertises everything WITH output schemas
|
|
58
|
+
`RETICLE_ADVERTISE_ALL_TOOLS=1` advertises everything WITH output schemas. It is a verification switch for suites that call by name, not a mode to run agents in. It is roughly 7x the per-turn cost, which is why it is opt-in: measured, carrying output schemas on the default surface takes it from 18,183 to 41,117 bytes. (That 18,183 was taken when the surface was 16 tools; a fresh `tools/list` read on 2026-08-14 puts the current 18-tool surface at **21,468 bytes**, and all 48 with output schemas at **134,368**. The shape of the gap holds; the absolute numbers move every time a tool is added, which is why they are asserted in `surface-sizes.test.ts` rather than trusted from prose.)
|
|
55
59
|
|
|
56
60
|
The typed result object still travels as `structuredContent` either way; the default surface simply does not advertise the output schema, which an agent reading the `text` block never consumed.
|
|
57
61
|
|
|
@@ -60,7 +64,7 @@ The typed result object still travels as `structuredContent` either way; the def
|
|
|
60
64
|
- **Full-tree vs full-tree, the gap is modest (~1.8×):** Reticle `full` (4,144) vs Playwright's with-refs snapshot (~7,300). Reticle collapses generic wrapper nodes, but both include every list item. If you force Reticle to dump the whole page each step, you don't save much.
|
|
61
65
|
- **The savings come from _not needing_ the full tree.** Playwright MCP's primary perception primitive is "return the accessibility tree"; Reticle's is "answer a specific question" (`query`/`assert`/`observe`/scoped or interactive `snapshot`). The win is architectural, not a cleverer serializer.
|
|
62
66
|
- **Cost scales with interactive elements + what you look at, not total DOM.** The 1,000 list items cost ~0 in `interactive` mode because they aren't interactive.
|
|
63
|
-
- **This is tool-output tokens only.** The agent's own reasoning tokens dominate either way
|
|
67
|
+
- **This is tool-output tokens only.** The agent's own reasoning tokens dominate either way, which is the point: keep observation cheap so the budget goes to thinking.
|
|
64
68
|
|
|
65
69
|
## Why it matters in practice
|
|
66
70
|
|
|
@@ -73,8 +77,8 @@ At scale (long flows, large dashboards, frequent re-runs for regression) that di
|
|
|
73
77
|
|
|
74
78
|
## Method & caveats
|
|
75
79
|
|
|
76
|
-
- One page, one tool, char/4 token proxy
|
|
77
|
-
- `_snapshotForAI()` (Playwright MCP's exact with-refs payload) was unavailable in the installed Playwright build, so we measured `body.ariaSnapshot()
|
|
80
|
+
- One page, one tool, char/4 token proxy: directional, not a benchmark suite. Absolute numbers vary by page; the _ratio_ is the point.
|
|
81
|
+
- `_snapshotForAI()` (Playwright MCP's exact with-refs payload) was unavailable in the installed Playwright build, so we measured `body.ariaSnapshot()`, the same accessibility tree it serializes; the real MCP payload is equal or slightly larger (it adds `[ref=…]`).
|
|
78
82
|
- Playwright MCP is excellent and Microsoft-backed; this is not a knock on it. It optimizes for cross-browser _driving_; Reticle optimizes for cheap, in-app _verification_. They can coexist (drive with one, assert with the other).
|
|
79
83
|
|
|
80
84
|
Run it yourself: the benchmark harness in `bench/` (see `bench/README.md`), with the demo + api running.
|
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: reticle_act_and_wait
|
|
3
|
+
description: Act on an element and wait for a named consequence in one call. This is the tool that produces a verdict, and the one you should reach for by default.
|
|
4
|
+
icon: circle-check
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
This is the tool Reticle exists for. It does the act, observe and assert steps in one round trip, and it makes you name the expected consequence **before** the action runs.
|
|
8
|
+
|
|
9
|
+
That ordering is not a convenience. An agent that acts first and then decides what counts as success will always find something that counts. Naming the consequence up front is the difference between a check and a rationalisation.
|
|
10
|
+
|
|
11
|
+
## Example
|
|
12
|
+
|
|
13
|
+
```json
|
|
14
|
+
{
|
|
15
|
+
"ref": "e5",
|
|
16
|
+
"action": "click",
|
|
17
|
+
"until": { "kind": "net", "method": "POST" }
|
|
18
|
+
}
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Real response, trimmed:
|
|
22
|
+
|
|
23
|
+
```json
|
|
24
|
+
{
|
|
25
|
+
"verified": "yes",
|
|
26
|
+
"verifiedReason": "proved",
|
|
27
|
+
"because": "assertion held at presence grade over a clean capture with no channel disagreeing",
|
|
28
|
+
"effect": {
|
|
29
|
+
"ok": true,
|
|
30
|
+
"testid": "login-submit",
|
|
31
|
+
"component": "Login",
|
|
32
|
+
"effect": { "domMutatedWithin": 7, "appeared": "Signing in…" },
|
|
33
|
+
"source": { "file": "src/components/Login.tsx", "line": 81, "column": 8 }
|
|
34
|
+
},
|
|
35
|
+
"verdict": {
|
|
36
|
+
"pass": true,
|
|
37
|
+
"evidence": {
|
|
38
|
+
"method": "POST",
|
|
39
|
+
"url": "http://localhost:8787/api/login",
|
|
40
|
+
"status": 200,
|
|
41
|
+
"durationMs": 67,
|
|
42
|
+
"requestBody": "{\"email\":\"admin@reticle.dev\",\"password\":\"[REDACTED]\"}"
|
|
43
|
+
}
|
|
44
|
+
},
|
|
45
|
+
"summary": {
|
|
46
|
+
"net": { "total": 1, "errors": 0 },
|
|
47
|
+
"consoleErrors": 0,
|
|
48
|
+
"stateDiffs": [{ "path": "auth", "from": null, "to": "{\"email\":\"admin@reticle.dev\"}" }],
|
|
49
|
+
"signals": ["auth:granted"]
|
|
50
|
+
},
|
|
51
|
+
"honesty": {
|
|
52
|
+
"grade": "presence",
|
|
53
|
+
"coverage": { "pct": 100, "partial": false },
|
|
54
|
+
"integrity": { "clean": true, "issues": [] }
|
|
55
|
+
}
|
|
56
|
+
}
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
## The predicate grammar
|
|
60
|
+
|
|
61
|
+
`until` takes `{ kind, ...fields }`. The kinds:
|
|
62
|
+
|
|
63
|
+
| Kind | Asserts |
|
|
64
|
+
| ------------------------- | -------------------------------------------------------------- |
|
|
65
|
+
| `signal` | The app fired a named signal, the strongest evidence available |
|
|
66
|
+
| `state` | A store path reached a value |
|
|
67
|
+
| `net` | A request matched; `count` asserts an **exact** number |
|
|
68
|
+
| `route` | The route **changed** |
|
|
69
|
+
| `element` | An element appeared; `absent: true` asserts it went away |
|
|
70
|
+
| `text` | Text appeared or, with `absent`, disappeared |
|
|
71
|
+
| `console` | Something logged; `absent: true` asserts a clean console |
|
|
72
|
+
| `animation` | An animation ran or finished |
|
|
73
|
+
| `settled` | The page went quiet |
|
|
74
|
+
| `allOf` / `anyOf` / `not` | Combine the above |
|
|
75
|
+
|
|
76
|
+
### Predicates that catch real bugs
|
|
77
|
+
|
|
78
|
+
```json
|
|
79
|
+
{ "until": { "kind": "net", "method": "POST", "count": 1 } }
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Exactly one request. Double-submit fails this at two, and no screenshot will ever tell you.
|
|
83
|
+
|
|
84
|
+
```json
|
|
85
|
+
{ "until": { "kind": "console", "absent": true } }
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
The action completed **and** logged nothing. "It worked but threw a warning" stops passing.
|
|
89
|
+
|
|
90
|
+
```json
|
|
91
|
+
{ "until": { "kind": "signal", "name": "deploy:created" } }
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
The app itself declared success. This is the highest grade of evidence, and it requires [instrumenting your app](/instrumentation). Which is the main reason to bother.
|
|
95
|
+
|
|
96
|
+
## Reading `verified`
|
|
97
|
+
|
|
98
|
+
<CardGroup cols={3}>
|
|
99
|
+
<Card title="yes" icon="circle-check">
|
|
100
|
+
The named consequence happened, with evidence attached.
|
|
101
|
+
</Card>
|
|
102
|
+
<Card title="no" icon="circle-xmark">
|
|
103
|
+
It did not. A finding, with a source pointer.
|
|
104
|
+
</Card>
|
|
105
|
+
<Card title="unknown" icon="circle-question">
|
|
106
|
+
Reticle could not tell. **Not a pass.** Report it as unknown.
|
|
107
|
+
</Card>
|
|
108
|
+
</CardGroup>
|
|
109
|
+
|
|
110
|
+
## The `honesty` block
|
|
111
|
+
|
|
112
|
+
`grade` is how strong the evidence was. A fired signal outranks an element appearing. `coverage.partial: true` means Reticle did not observe the whole window. `integrity.clean: false` means something interfered with the capture and the verdict deserves less trust.
|
|
113
|
+
|
|
114
|
+
A verification tool that cannot describe the quality of its own evidence is asking you to take its word for it, which is precisely the problem we started with.
|
|
115
|
+
|
|
116
|
+
## Arguments
|
|
117
|
+
|
|
118
|
+
| Argument | What it does |
|
|
119
|
+
| ----------------------- | ---------------------------------------- |
|
|
120
|
+
| `ref`, `action`, `args` | As [`reticle_act`](/tools-act) |
|
|
121
|
+
| `until` | The predicate to wait for |
|
|
122
|
+
| `timeout_ms` | Maximum wait |
|
|
123
|
+
| `refuseWhenThrottled` | Throw rather than act on a throttled tab |
|