@reticlehq/server 2.7.0 → 2.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -5
- package/SKILL.md +130 -778
- package/dist/bridge/bridge.d.ts +23 -0
- package/dist/bridge/bridge.js +80 -2
- package/dist/bridge/bridge.test-harness.d.ts +5 -0
- package/dist/bridge/bridge.test-harness.js +6 -4
- package/dist/capsule/causal-summary.d.ts +36 -1
- package/dist/capsule/causal-summary.js +33 -7
- package/dist/capsule/divergence.js +7 -1
- package/dist/cli/chromium-hint.d.ts +33 -5
- package/dist/cli/chromium-hint.js +82 -12
- package/dist/cli/cli-doctor.js +65 -34
- package/dist/cli/cli-flow-commands.js +8 -8
- package/dist/cli/cli-kill.d.ts +58 -0
- package/dist/cli/cli-kill.js +162 -0
- package/dist/cli/cli-parse.d.ts +16 -1
- package/dist/cli/cli-parse.js +32 -9
- package/dist/cli/cli-port.d.ts +32 -0
- package/dist/cli/cli-port.js +82 -35
- package/dist/cli/cli-verify.d.ts +21 -0
- package/dist/cli/cli-verify.js +47 -3
- package/dist/cli/config-discovery.d.ts +35 -0
- package/dist/cli/config-discovery.js +163 -0
- package/dist/cli/doctor-daemon-line.js +4 -1
- package/dist/cli/doctor-project-line.d.ts +37 -0
- package/dist/cli/doctor-project-line.js +35 -0
- package/dist/cli/doctor-rows.d.ts +51 -0
- package/dist/cli/doctor-rows.js +54 -0
- package/dist/cli/doctor-sessions-line.d.ts +12 -0
- package/dist/cli/doctor-sessions-line.js +69 -0
- package/dist/cli/drive-attach.d.ts +99 -0
- package/dist/cli/drive-attach.js +203 -0
- package/dist/cli/drive-command.d.ts +13 -0
- package/dist/cli/drive-command.js +103 -0
- package/dist/cli/open-note.d.ts +20 -0
- package/dist/cli/open-note.js +37 -0
- package/dist/cli/port-holder.d.ts +10 -0
- package/dist/cli/port-holder.js +26 -2
- package/dist/cli/status-next-action.d.ts +37 -0
- package/dist/cli/status-next-action.js +61 -0
- package/dist/cli.js +119 -39
- package/dist/crawl/crawl-tools.js +12 -2
- package/dist/crawl/crawl.d.ts +13 -0
- package/dist/crawl/crawl.js +50 -2
- package/dist/events/contradictions.d.ts +46 -0
- package/dist/events/contradictions.js +35 -22
- package/dist/events/declared.d.ts +45 -0
- package/dist/events/declared.js +73 -0
- package/dist/events/echo-mismatch.d.ts +8 -2
- package/dist/events/echo-mismatch.js +33 -5
- package/dist/events/event-filters.d.ts +1 -1
- package/dist/events/event-filters.js +7 -4
- package/dist/events/predicate-eval.d.ts +17 -72
- package/dist/events/predicate-eval.js +117 -236
- package/dist/events/predicate-parse.js +31 -3
- package/dist/events/predicate-route.d.ts +32 -0
- package/dist/events/predicate-route.js +102 -0
- package/dist/events/predicate-schema.d.ts +99 -0
- package/dist/events/predicate-schema.js +309 -0
- package/dist/events/predicate.d.ts +8 -0
- package/dist/events/predicate.js +296 -22
- package/dist/flows/annotate-tools.js +3 -0
- package/dist/flows/annotate.js +18 -4
- package/dist/flows/coverage.d.ts +38 -5
- package/dist/flows/coverage.js +30 -4
- package/dist/flows/decision.d.ts +15 -0
- package/dist/flows/decision.js +1 -1
- package/dist/flows/flow-classify.js +1 -1
- package/dist/flows/flow-replay-run.js +8 -1
- package/dist/flows/flow-tools.js +21 -5
- package/dist/flows/heal-run.js +1 -1
- package/dist/flows/replay.js +3 -0
- package/dist/honesty/accepted-write.d.ts +3 -0
- package/dist/honesty/accepted-write.js +8 -3
- package/dist/honesty/already-true.js +15 -0
- package/dist/honesty/blind-spots.js +10 -2
- package/dist/honesty/honesty.d.ts +11 -0
- package/dist/honesty/honesty.js +1 -0
- package/dist/honesty/uncaptured-bodies.d.ts +4 -2
- package/dist/honesty/uncaptured-bodies.js +9 -2
- package/dist/honesty/unread-outcome.d.ts +8 -0
- package/dist/honesty/unread-outcome.js +36 -9
- package/dist/honesty/unsettled.d.ts +49 -0
- package/dist/honesty/unsettled.js +82 -0
- package/dist/honesty/verified.d.ts +36 -3
- package/dist/honesty/verified.js +54 -11
- package/dist/http-server.d.ts +19 -0
- package/dist/http-server.js +202 -4
- package/dist/impact/impact-recorder.d.ts +23 -0
- package/dist/impact/impact-recorder.js +76 -0
- package/dist/impact/impact-store.d.ts +47 -0
- package/dist/impact/impact-store.js +171 -0
- package/dist/index.d.ts +7 -0
- package/dist/index.js +47 -10
- package/dist/init/agent-rules.d.ts +36 -0
- package/dist/init/agent-rules.js +132 -22
- package/dist/init/astro-patch.d.ts +2 -1
- package/dist/init/astro-patch.js +38 -6
- package/dist/init/capabilities.d.ts +22 -0
- package/dist/init/capabilities.js +76 -0
- package/dist/init/confirm.d.ts +114 -0
- package/dist/init/confirm.js +165 -0
- package/dist/init/connect-steps.d.ts +2 -0
- package/dist/init/connect-steps.js +50 -0
- package/dist/init/cra.js +3 -2
- package/dist/init/csp-check.d.ts +38 -0
- package/dist/init/csp-check.js +92 -0
- package/dist/init/csp-doctor.d.ts +25 -0
- package/dist/init/csp-doctor.js +49 -0
- package/dist/init/cursor.js +6 -9
- package/dist/init/detect.d.ts +8 -0
- package/dist/init/detect.js +33 -3
- package/dist/init/mcp-clients.d.ts +1 -0
- package/dist/init/mcp-clients.js +51 -7
- package/dist/init/mcp.d.ts +15 -0
- package/dist/init/mcp.js +19 -0
- package/dist/init/non-js-project.js +29 -5
- package/dist/init/plan-framework.d.ts +10 -0
- package/dist/init/plan-framework.js +78 -16
- package/dist/init/plan.d.ts +47 -6
- package/dist/init/plan.js +228 -69
- package/dist/init/run.d.ts +24 -2
- package/dist/init/run.js +157 -45
- package/dist/init/slash-command.d.ts +1 -1
- package/dist/init/slash-command.js +54 -6
- package/dist/init/snippets.d.ts +45 -5
- package/dist/init/snippets.js +194 -42
- package/dist/init/vite-config.js +53 -5
- package/dist/input/network-mock-tools.d.ts +19 -0
- package/dist/input/network-mock-tools.js +37 -5
- package/dist/input/scroll-find.d.ts +6 -0
- package/dist/input/scroll-find.js +17 -2
- package/dist/input/scroll-tools.js +4 -0
- package/dist/journal/attach-journal.d.ts +9 -1
- package/dist/journal/attach-journal.js +13 -2
- package/dist/journal/journal-query.d.ts +13 -0
- package/dist/journal/journal-query.js +28 -3
- package/dist/journal/workspace-gitignore.d.ts +10 -0
- package/dist/journal/workspace-gitignore.js +59 -0
- package/dist/license/license.js +1 -1
- package/dist/mcp/mcp-proxy.d.ts +35 -30
- package/dist/mcp/mcp-proxy.js +306 -104
- package/dist/mcp/mcp.d.ts +10 -1
- package/dist/mcp/mcp.js +70 -34
- package/dist/mcp/proxy-handshake.d.ts +82 -1
- package/dist/mcp/proxy-handshake.js +109 -2
- package/dist/mcp/proxy-lifecycle.d.ts +11 -2
- package/dist/mcp/proxy-lifecycle.js +12 -3
- package/dist/mcp/proxy-log.d.ts +63 -0
- package/dist/mcp/proxy-log.js +132 -0
- package/dist/mcp/server-instructions.d.ts +17 -0
- package/dist/mcp/server-instructions.js +56 -0
- package/dist/mcp/shared-params.d.ts +27 -0
- package/dist/mcp/shared-params.js +36 -0
- package/dist/pool/browser-pool.d.ts +22 -2
- package/dist/pool/browser-pool.js +44 -4
- package/dist/project/learned-routes.d.ts +24 -0
- package/dist/project/learned-routes.js +63 -0
- package/dist/project/project-store.d.ts +6 -0
- package/dist/project/project-store.js +25 -1
- package/dist/project/reticle-dir.d.ts +2 -0
- package/dist/project/reticle-dir.js +1 -0
- package/dist/session/connection-memory.d.ts +59 -0
- package/dist/session/connection-memory.js +108 -0
- package/dist/session/dev-command.d.ts +26 -0
- package/dist/session/dev-command.js +89 -0
- package/dist/session/dev-server-probe.d.ts +68 -0
- package/dist/session/dev-server-probe.js +102 -7
- package/dist/session/last-act.d.ts +26 -0
- package/dist/session/last-act.js +35 -1
- package/dist/session/live-control-tools.js +42 -5
- package/dist/session/no-session-diagnosis.d.ts +67 -2
- package/dist/session/no-session-diagnosis.js +236 -27
- package/dist/session/no-session-next-action.d.ts +46 -0
- package/dist/session/no-session-next-action.js +85 -0
- package/dist/session/no-session-watch.d.ts +40 -1
- package/dist/session/no-session-watch.js +174 -8
- package/dist/session/pending-commands.d.ts +12 -0
- package/dist/session/pending-commands.js +13 -1
- package/dist/session/ref-provenance.d.ts +54 -0
- package/dist/session/ref-provenance.js +61 -0
- package/dist/session/session-lease.js +1 -1
- package/dist/session/session-manager.d.ts +4 -0
- package/dist/session/session-manager.js +35 -5
- package/dist/session/session.d.ts +10 -1
- package/dist/session/session.js +146 -5
- package/dist/telemetry/app-instrumented.d.ts +8 -0
- package/dist/telemetry/app-instrumented.js +10 -0
- package/dist/telemetry/automation-hint.d.ts +17 -0
- package/dist/telemetry/automation-hint.js +71 -0
- package/dist/telemetry/bug-found.d.ts +15 -17
- package/dist/telemetry/bug-found.js +52 -21
- package/dist/telemetry/cli-telemetry.js +7 -1
- package/dist/telemetry/daemon-telemetry.js +24 -0
- package/dist/telemetry/feedback.js +1 -1
- package/dist/telemetry/init-telemetry.js +9 -1
- package/dist/telemetry/install-source.d.ts +56 -0
- package/dist/telemetry/install-source.js +74 -0
- package/dist/telemetry/instrumentation-stall.d.ts +50 -0
- package/dist/telemetry/instrumentation-stall.js +90 -0
- package/dist/telemetry/run-telemetry.js +4 -1
- package/dist/telemetry/session-metrics.d.ts +8 -0
- package/dist/telemetry/session-metrics.js +15 -0
- package/dist/telemetry/telemetry.d.ts +7 -1
- package/dist/telemetry/telemetry.js +26 -2
- package/dist/telemetry/tool-refused.d.ts +19 -0
- package/dist/telemetry/tool-refused.js +55 -0
- package/dist/telemetry/verification-of.js +46 -7
- package/dist/temp-dir.d.ts +27 -0
- package/dist/temp-dir.js +46 -0
- package/dist/tools/act-tools.js +209 -28
- package/dist/tools/assert-grade.d.ts +1 -1
- package/dist/tools/assert-grade.js +27 -5
- package/dist/tools/assert-verdict.js +49 -7
- package/dist/tools/browser-tools.js +10 -0
- package/dist/tools/contract-tools.js +4 -0
- package/dist/tools/dynamic-tools.js +16 -7
- package/dist/tools/error-recovery.d.ts +10 -0
- package/dist/tools/error-recovery.js +73 -9
- package/dist/tools/feedback-invite.js +1 -1
- package/dist/tools/invoke-tool.js +159 -26
- package/dist/tools/lease-hint.d.ts +30 -8
- package/dist/tools/lease-hint.js +65 -14
- package/dist/tools/lease-tools.d.ts +40 -1
- package/dist/tools/lease-tools.js +181 -83
- package/dist/tools/navigate-result.js +1 -1
- package/dist/tools/observe-tools.js +10 -4
- package/dist/tools/observed-nothing.js +1 -1
- package/dist/tools/react-grace.d.ts +52 -0
- package/dist/tools/react-grace.js +90 -0
- package/dist/tools/read-tools.js +67 -5
- package/dist/tools/resolve-target.d.ts +33 -0
- package/dist/tools/resolve-target.js +53 -0
- package/dist/tools/settle-in-flight.d.ts +94 -0
- package/dist/tools/settle-in-flight.js +146 -0
- package/dist/tools/snapshot-delta.d.ts +10 -0
- package/dist/tools/snapshot-delta.js +194 -9
- package/dist/tools/tool-kit.d.ts +8 -0
- package/dist/tools/tool-surface.d.ts +60 -0
- package/dist/tools/tool-surface.js +86 -1
- package/dist/tools/tools.js +61 -8
- package/dist/tools/unadvertised-help.js +9 -6
- package/docs/README.md +55 -11
- package/docs/actions.mdx +120 -0
- package/docs/agent-cheatsheet.md +95 -29
- package/docs/architecture.md +52 -35
- package/docs/benchmarks.md +127 -75
- package/docs/best-practices.mdx +216 -0
- package/docs/capabilities.mdx +534 -0
- package/docs/cli/affected.mdx +53 -0
- package/docs/cli/capsules.mdx +70 -0
- package/docs/cli/cloud.mdx +153 -0
- package/docs/cli/doctor.mdx +109 -0
- package/docs/cli/drive.mdx +74 -0
- package/docs/cli/feedback.mdx +67 -0
- package/docs/cli/gate.mdx +64 -0
- package/docs/cli/help.mdx +94 -0
- package/docs/cli/hunt.mdx +73 -0
- package/docs/cli/identify.mdx +76 -0
- package/docs/cli/init.mdx +121 -0
- package/docs/cli/kill.mdx +91 -0
- package/docs/cli/license.mdx +48 -0
- package/docs/cli/mcp.mdx +61 -0
- package/docs/cli/open.mdx +74 -0
- package/docs/cli/restart.mdx +60 -0
- package/docs/cli/rollback.mdx +35 -0
- package/docs/cli/serve.mdx +69 -0
- package/docs/cli/status.mdx +88 -0
- package/docs/cli/stop.mdx +64 -0
- package/docs/cli/telemetry.mdx +57 -0
- package/docs/cli/update.mdx +52 -0
- package/docs/cli/verify.mdx +76 -0
- package/docs/cli/version.mdx +39 -0
- package/docs/cli/watch.mdx +62 -0
- package/docs/cli.mdx +241 -0
- package/docs/debugging.md +16 -10
- package/docs/deploy-checks.md +45 -12
- package/docs/desktop.mdx +384 -0
- package/docs/docs.json +364 -0
- package/docs/enterprise.md +22 -16
- package/docs/faq.mdx +102 -0
- package/docs/fixtures.md +13 -7
- package/docs/flows.md +29 -23
- package/docs/for-agents.md +71 -0
- package/docs/frameworks.mdx +238 -0
- package/docs/gate-plan.md +48 -42
- package/docs/gates.md +28 -22
- package/docs/getting-started.md +129 -77
- package/docs/human-control.md +24 -18
- package/docs/index.mdx +143 -0
- package/docs/install-agentic.mdx +224 -0
- package/docs/install-manual.mdx +196 -0
- package/docs/instrumentation.mdx +238 -0
- package/docs/integration-patterns.md +41 -35
- package/docs/local-registry.md +19 -13
- package/docs/multi-agent-testing.md +19 -13
- package/docs/packages/babel-plugin.mdx +79 -0
- package/docs/packages/browser.mdx +111 -0
- package/docs/packages/core.mdx +116 -0
- package/docs/packages/electron.mdx +85 -0
- package/docs/packages/eslint-plugin.mdx +86 -0
- package/docs/packages/next.mdx +89 -0
- package/docs/packages/react.mdx +91 -0
- package/docs/packages/server.mdx +98 -0
- package/docs/packages/tauri.mdx +63 -0
- package/docs/packages/test.mdx +119 -0
- package/docs/packages/vite-plugin.mdx +94 -0
- package/docs/packages.mdx +218 -0
- package/docs/platform-integration.md +35 -26
- package/docs/predicates.mdx +248 -0
- package/docs/quickstart.mdx +257 -0
- package/docs/recipes.mdx +282 -0
- package/docs/skill-file.mdx +107 -0
- package/docs/state-management.mdx +199 -0
- package/docs/style.css +14 -0
- package/docs/system-map.md +50 -33
- package/docs/telemetry-contract.md +106 -52
- package/docs/telemetry-events.mdx +475 -0
- package/docs/telemetry.md +63 -55
- package/docs/testing.md +44 -6
- package/docs/token-efficiency.md +37 -27
- package/docs/tools-act-and-wait.mdx +173 -0
- package/docs/tools-act-sequence.mdx +137 -0
- package/docs/tools-act.mdx +95 -0
- package/docs/tools-assert.mdx +129 -0
- package/docs/tools-console.mdx +63 -0
- package/docs/tools-inspect.mdx +88 -0
- package/docs/tools-navigate.mdx +99 -0
- package/docs/tools-network.mdx +119 -0
- package/docs/tools-observe.mdx +145 -0
- package/docs/tools-overview.mdx +160 -0
- package/docs/tools-query.mdx +126 -0
- package/docs/tools-session-and-feedback.mdx +112 -0
- package/docs/tools-sessions.mdx +94 -0
- package/docs/tools-snapshot.mdx +129 -0
- package/docs/tools-state.mdx +104 -0
- package/docs/tools-tools-and-run.mdx +156 -0
- package/docs/tools-wait-for.mdx +85 -0
- package/docs/troubleshooting.mdx +263 -0
- package/docs/usage.md +209 -165
- package/docs/vs-chrome-devtools-mcp.mdx +123 -0
- package/docs/vs-playwright-mcp.mdx +125 -0
- package/docs/vs-screenshots.mdx +137 -0
- package/docs/what-is-reticle.mdx +63 -0
- package/docs/why-reticle.mdx +109 -0
- package/package.json +5 -5
- package/docs/desktop-apps.md +0 -256
- package/docs/matrix/2.5.0/claude-code-machine.json +0 -19
- package/docs/matrix/2.5.0/codex-machine.json +0 -20
- package/docs/matrix/2.5.0/cursor-machine.json +0 -22
- package/docs/matrix/2.5.0/gemini-machine.json +0 -22
- package/docs/matrix/2.5.0/opencode-machine.json +0 -22
- package/docs/matrix/2.5.0/vscode-machine.json +0 -22
- package/docs/matrix/2.5.0/windsurf-machine.json +0 -22
- package/docs/matrix/MATRIX.md +0 -26
- package/docs/matrix/README.md +0 -59
- package/docs/mint.json +0 -61
package/docs/architecture.md
CHANGED
|
@@ -1,4 +1,10 @@
|
|
|
1
|
-
|
|
1
|
+
---
|
|
2
|
+
title: Architecture
|
|
3
|
+
description: 'The moving parts, the data flow between them, and the design decisions behind each one.'
|
|
4
|
+
icon: sitemap
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
Reticle has three moving parts: a dev-only **SDK** inside your page that instruments the DOM, network, console, routing and framework state; a local **server** that runs a WebSocket bridge to that SDK and an MCP server for your agent; and a git-checked **`.reticle/` workspace** holding flows, baselines and run artifacts. Your agent calls an MCP tool, the server turns it into a command over the WebSocket, the SDK runs it in the real page and streams structured events back.
|
|
2
8
|
|
|
3
9
|
> For engineers evaluating Reticle, integrating it at scale, or contributing. It explains the moving parts, the data flow, and the design decisions behind them. If you just want to get running, start with [getting-started.md](getting-started.md); come back here when you want to know _why_.
|
|
4
10
|
|
|
@@ -6,23 +12,28 @@
|
|
|
6
12
|
|
|
7
13
|
## The one-paragraph model
|
|
8
14
|
|
|
9
|
-
Your app, in dev, embeds a tiny **SDK** that instruments the page (DOM, network, console, routing, framework state) and opens a WebSocket to a local **bridge**. The bridge runs inside the Reticle **server**, which also exposes an **MCP server
|
|
15
|
+
Your app, in dev, embeds a tiny **SDK** that instruments the page (DOM, network, console, routing, framework state) and opens a WebSocket to a local **bridge**. The bridge runs inside the Reticle **server**, which also exposes an **MCP server**, the standard protocol AI agents speak.
|
|
10
16
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
17
|
+
Your coding agent calls MCP tools (`reticle_query`, `reticle_act`, `reticle_assert`, …); the server turns them into commands over the WebSocket; the SDK executes them in the page and streams back structured events. The agent thus **looks, acts, observes, and asserts** on the real running app, never on a screenshot.
|
|
18
|
+
|
|
19
|
+
```mermaid
|
|
20
|
+
flowchart LR
|
|
21
|
+
A["AI agent<br/>Claude Code, Cursor, ..."]
|
|
22
|
+
S["@reticlehq/server<br/>bridge + MCP + CLI<br/>(Node)"]
|
|
23
|
+
B["@reticlehq/browser<br/>SDK in your app<br/>(the DOM)"]
|
|
24
|
+
D[(".reticle/<br/>flows · baselines<br/>runs · contract")]
|
|
25
|
+
|
|
26
|
+
A -- "MCP (stdio/SSE)<br/>reticle_query / act / assert" --> S
|
|
27
|
+
S -- "tool results" --> A
|
|
28
|
+
S <-- "WebSocket<br/>commands / events" --> B
|
|
29
|
+
S <--> D
|
|
19
30
|
```
|
|
20
31
|
|
|
21
32
|
---
|
|
22
33
|
|
|
23
34
|
## The packages (and the boundaries between them)
|
|
24
35
|
|
|
25
|
-
Reticle is a pnpm + Turborepo monorepo. The split is not cosmetic
|
|
36
|
+
Reticle is a pnpm + Turborepo monorepo. The split is not cosmetic: each boundary enforces a rule.
|
|
26
37
|
|
|
27
38
|
| Package | Runs in | Responsibility | Hard rule |
|
|
28
39
|
| --- | --- | --- | --- |
|
|
@@ -31,20 +42,26 @@ Reticle is a pnpm + Turborepo monorepo. The split is not cosmetic — each bound
|
|
|
31
42
|
| `@reticlehq/server` | Node | The bridge, the MCP server, the `reticle` CLI, flow/run storage | Never imports DOM APIs |
|
|
32
43
|
| `@reticlehq/react` | the browser | The SDK **kit** you install in a browser app: re-exports the browser sensor (so one install gives both `reticle` and `install`) and maps a DOM node → React component → source `file:line` | Core works without the source-mapping half |
|
|
33
44
|
| `@reticlehq/babel-plugin`, `@reticlehq/next`, `@reticlehq/vite-plugin` | build time | Stamp `data-reticle-source` for source mapping (and, for Vite, inject `connect()`) | Plain tooling |
|
|
45
|
+
| `@reticlehq/electron` | Electron main process | Adapter for the desktop runtime: IPC observer, main-process capture | Desktop-only, outside the web gates |
|
|
46
|
+
| `packages/tauri` (`reticle-tauri`) | Tauri (Rust) | Capture backend for the Tauri desktop runtime | Rust, outside every JS gate; compiled by CI's `rust`/`rust-macos` jobs |
|
|
47
|
+
| `@reticlehq/test` | Node (dev dependency) | Spec runner + matchers used by CI's own test suites | Peer dependency on `vitest` |
|
|
48
|
+
| `@reticlehq/eslint-plugin` | dev tooling | Repo-internal lint rule: a state change must fire a signal | Dev-only, never shipped in a user's bundle |
|
|
34
49
|
|
|
35
50
|
> Note: pre-2.0, `@reticlehq/core` was a single umbrella package that re-exported all of the above under subpaths. In 2.0 the umbrella was retired and `@reticlehq/core` became the bottom-of-graph wire contract.
|
|
36
51
|
|
|
37
|
-
**Why core-as-contract matters:** because the browser and the server are two different runtimes (a DOM and a Node process) that must agree exactly on every message, the temptation is to inline a string like `"net.request"` in both. That's how drift and silent breakage start. Instead, every such string and shape lives once in `@reticlehq/core` as a named constant + a zod schema.
|
|
52
|
+
**Why core-as-contract matters:** because the browser and the server are two different runtimes (a DOM and a Node process) that must agree exactly on every message, the temptation is to inline a string like `"net.request"` in both. That's how drift and silent breakage start. Instead, every such string and shape lives once in `@reticlehq/core` as a named constant + a zod schema.
|
|
53
|
+
|
|
54
|
+
The browser and server both import it; neither can invent a message the other doesn't understand. The server zod-parses **every** inbound WebSocket message; malformed input closes the socket rather than flowing into logic.
|
|
38
55
|
|
|
39
56
|
---
|
|
40
57
|
|
|
41
58
|
## The data flow, end to end
|
|
42
59
|
|
|
43
|
-
1. **Connect.** In dev, your app calls `reticle.connect({ session })`. The SDK opens a WebSocket to the bridge (`ws://localhost:4400/reticle` by default) and sends a `HELLO` carrying the session id, protocol version, and (if configured) a pairing token. Each browser tab uses `SESSION_AUTO`
|
|
44
|
-
2. **Capture.** The SDK installs observers: a `MutationObserver` for DOM changes, wrappers around `fetch`/`XHR` for network, a console hook, a history hook for routing, and registries the app opts into (`registerStore`, `registerCapabilities`, `reticle.signal`). Events flow into a bounded **ring buffer
|
|
60
|
+
1. **Connect.** In dev, your app calls `reticle.connect({ session })`. The SDK opens a WebSocket to the bridge (`ws://localhost:4400/reticle` by default) and sends a `HELLO` carrying the session id, protocol version, and (if configured) a pairing token. Each browser tab uses `SESSION_AUTO` (a unique id), so multiple apps/tabs never collide.
|
|
61
|
+
2. **Capture.** The SDK installs observers: a `MutationObserver` for DOM changes, wrappers around `fetch`/`XHR` for network, a console hook, a history hook for routing, and registries the app opts into (`registerStore`, `registerCapabilities`, `reticle.signal`). Events flow into a bounded **ring buffer**, so recent history is always available and memory is capped.
|
|
45
62
|
3. **Look / act.** The agent calls an MCP tool. `reticle_query` finds an element by role/text/testid/ component and returns a stable **ref**. `reticle_act` dispatches an action against that ref and returns an **effect** report (did it land, did the DOM mutate, did focus move…). The server sends the command over the WebSocket; the SDK runs it and replies.
|
|
46
|
-
4. **Observe.** After an action, the agent reads what happened
|
|
47
|
-
5. **Assert.** `reticle_assert` (and a flow's declared `success`) evaluates a **predicate** over program truth
|
|
63
|
+
4. **Observe.** After an action, the agent reads what happened with `reticle_network`, `reticle_console`, `reticle_state`, or the reaction digest from `reticle_act_and_wait`. The server pulls the relevant slice of the ring buffer (scoped to a cursor so stale events can't leak in) and returns a compact summary.
|
|
64
|
+
5. **Assert.** `reticle_assert` (and a flow's declared `success`) evaluates a **predicate** over program truth (a network call that returned 200, a store value, a `signal` the app emitted), not just "an element exists." This is the difference between "looks done" and "is done."
|
|
48
65
|
|
|
49
66
|
---
|
|
50
67
|
|
|
@@ -54,23 +71,23 @@ Reticle is a pnpm + Turborepo monorepo. The split is not cosmetic — each bound
|
|
|
54
71
|
|
|
55
72
|
Most agent-browser tools can confirm "an element matching X is present." That's the weakest possible oracle: a wrong or healed-to-wrong element satisfies it, and the regression ships anyway. Reticle grades evidence in tiers:
|
|
56
73
|
|
|
57
|
-
- **Tier 1
|
|
58
|
-
- **Tier 2
|
|
59
|
-
- **Tier 3
|
|
74
|
+
- **Tier 1, an app signal** (`reticle.signal('order:placed')`): the strongest, because a wrong element can't fake it. Available when the app emits it (a ~30-second opt-in).
|
|
75
|
+
- **Tier 2, network + route + state**: a POST returned 200, the URL changed, a store value updated. Strong, and works on most SPAs with no setup.
|
|
76
|
+
- **Tier 3, DOM/text presence**: the weak fallback Reticle nudges _away_ from.
|
|
60
77
|
|
|
61
78
|
Reticle is honest about which tier a given assertion used, so "green" carries its own confidence.
|
|
62
79
|
|
|
63
80
|
### 2. Structured reads, not screenshots
|
|
64
81
|
|
|
65
|
-
A screenshot is ~1,365 image tokens per look, slow, non-deterministic, and **blind to everything non-visual
|
|
82
|
+
A screenshot is ~1,365 image tokens per look, slow, non-deterministic, and **blind to everything non-visual**: the failed request, the console error, the route that didn't change. Reticle reads the accessibility tree, the network log, the console, and framework state as compact structured data. That's an order of magnitude cheaper _and_ it sees the bugs pixels can't. See [benchmarks.md](benchmarks.md) for the measured comparison.
|
|
66
83
|
|
|
67
84
|
### 3. Record once, replay deterministically
|
|
68
85
|
|
|
69
|
-
The same verification runs over and over
|
|
86
|
+
The same verification runs over and over, every commit, every CI run. Reticle records a **flow** once, then replays it with **no AI model**: it re-resolves each element's durable anchor against the live DOM and re-asserts the declared consequence. A CI gate diffs the verdict exactly, at ~0% flake, for a couple hundred tokens, versus an agent re-driving the whole flow with the model every time. Self- healing rebinds a drifted anchor **only if the consequence still fires**, so it never "heals to the wrong element and ships the regression."
|
|
70
87
|
|
|
71
88
|
### 4. Dev-only, localhost-only, your app data stays local
|
|
72
89
|
|
|
73
|
-
The SDK is tree-shaken out of production builds and connects only to a local bridge. The bridge binds to loopback by default; exposing it beyond localhost _requires_ a pairing token (the server refuses to bind a non-loopback host without one). Every environment variable that gates a security control is a single named constant, so a typo can't silently disable auth. Nothing from the app under test
|
|
90
|
+
The SDK is tree-shaken out of production builds and connects only to a local bridge. The bridge binds to loopback by default; exposing it beyond localhost _requires_ a pairing token (the server refuses to bind a non-loopback host without one). Every environment variable that gates a security control is a single named constant, so a typo can't silently disable auth. Nothing from the app under test ever leaves your machine: no DOM, network, console, state, or source. The CLI reports anonymous, opt-out usage metrics only (a random id + event names like `invoke`/`session_start`; no code, no PII; see [telemetry](telemetry.md)); opt out with `reticle telemetry disable`, `RETICLE_TELEMETRY=0`, or `DO_NOT_TRACK=1`. The one thing that carries free text is feedback you or your agent deliberately send (`reticle feedback` / `reticle_feedback`). It is never collected passively, it is redacted before sending, and it is disabled separately with `RETICLE_FEEDBACK=0`.
|
|
74
91
|
|
|
75
92
|
---
|
|
76
93
|
|
|
@@ -78,27 +95,27 @@ The SDK is tree-shaken out of production builds and connects only to a local bri
|
|
|
78
95
|
|
|
79
96
|
The server persists project state under `.reticle/` in your repo:
|
|
80
97
|
|
|
81
|
-
- **`flows/`**
|
|
82
|
-
- **`baselines/`**
|
|
83
|
-
- **`runs/`**
|
|
84
|
-
- **the capabilities contract**
|
|
98
|
+
- **`flows/`** holds recorded, replayable flows (the golden journeys).
|
|
99
|
+
- **`baselines/`** holds saved snapshots for diffing.
|
|
100
|
+
- **`runs/`** holds verification run artifacts (the evidence trail; writes are atomic and bounded so a crash never leaves a half-written artifact, and the directory is pruned).
|
|
101
|
+
- **the capabilities contract** is the testids/signals/stores/flows the app advertises, frozen under a version so the public artifact can't break silently.
|
|
85
102
|
|
|
86
|
-
This is plain, reviewable, version-controllable data
|
|
103
|
+
This is plain, reviewable, version-controllable data, not a black box.
|
|
87
104
|
|
|
88
105
|
---
|
|
89
106
|
|
|
90
107
|
## Running at scale (multiple apps, multiple projects)
|
|
91
108
|
|
|
92
|
-
- **Multiple apps / tabs on one bridge:** fine
|
|
109
|
+
- **Multiple apps / tabs on one bridge:** fine. Each connection has a unique `SESSION_AUTO` id; a tool call targets the focused/most-recent session, or you pass an explicit `sessionId`.
|
|
93
110
|
- **Multiple isolated projects:** give each project its own bridge port via `RETICLE_PORT` (set it in the MCP config and dial the same port from the app). A port already in use fails fast with a clear error rather than hanging. See [getting-started.md → Running multiple apps](getting-started.md#running-multiple-apps-at-once).
|
|
94
|
-
- **CI / no MCP:** `reticle verify <url>` replays the saved flows headlessly and exits non-zero on failure
|
|
111
|
+
- **CI / no MCP:** `reticle verify <url>` replays the saved flows headlessly and exits non-zero on failure. It writes the same verdict artifact the MCP path produces, with no agent in the loop.
|
|
95
112
|
|
|
96
113
|
---
|
|
97
114
|
|
|
98
115
|
## Open-core licensing (what's free, what's protected)
|
|
99
116
|
|
|
100
|
-
- The embeddable **SDK** (`-core`, `-browser`, `-react`) is **Apache-2.0
|
|
101
|
-
- The **server / CLI** is under the **Functional Source License (FSL-1.1, Apache-2.0 future)
|
|
117
|
+
- The embeddable **SDK** (`-core`, `-browser`, `-react`) is **Apache-2.0**, safe to ship inside your own app.
|
|
118
|
+
- The **server / CLI** is under the **Functional Source License (FSL-1.1, Apache-2.0 future)**: source-available, converts to Apache-2.0 over time.
|
|
102
119
|
- Enterprise-only features live behind a license gate and are clearly separated.
|
|
103
120
|
|
|
104
121
|
See [LICENSE](../LICENSE) and each package's own `LICENSE` file. The licensing _mechanism_ is open and inspectable; activation is offline (no phone-home).
|
|
@@ -107,7 +124,7 @@ See [LICENSE](../LICENSE) and each package's own `LICENSE` file. The licensing _
|
|
|
107
124
|
|
|
108
125
|
## Where to go next
|
|
109
126
|
|
|
110
|
-
- [getting-started.md](getting-started.md)
|
|
111
|
-
- [benchmarks.md](benchmarks.md)
|
|
112
|
-
- [usage.md](usage.md)
|
|
113
|
-
- [CONTRIBUTING.md](../CONTRIBUTING.md)
|
|
127
|
+
- [getting-started.md](getting-started.md) wires Reticle into your app in a couple of minutes.
|
|
128
|
+
- [benchmarks.md](benchmarks.md) covers how we measure, and the honest results vs the alternatives.
|
|
129
|
+
- [usage.md](usage.md) is the full tool reference and advanced modes.
|
|
130
|
+
- [CONTRIBUTING.md](../CONTRIBUTING.md) has the development loop and the rules.
|
package/docs/benchmarks.md
CHANGED
|
@@ -1,12 +1,24 @@
|
|
|
1
|
-
|
|
1
|
+
---
|
|
2
|
+
title: Benchmarks
|
|
3
|
+
description: 'How we measure Reticle, what each number actually means, and the places Reticle loses.'
|
|
4
|
+
icon: chart-column
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
Two different measurements, because there are two different questions.
|
|
8
|
+
|
|
9
|
+
Against a **Playwright script** over an 88-bug registry, each tool running its own native check deterministically with no model in the loop, Reticle caught **85 of 88** and Playwright **59 of 88**, both with zero false alarms on the clean build. The gap is not spread evenly: it is 26 bugs Playwright **structurally cannot** see, and on everything observable from outside the app the two tie exactly. See Part 5a.
|
|
10
|
+
|
|
11
|
+
Against **Playwright MCP and Chrome DevTools MCP**, the agent-driven tools where cost per look matters, Reticle caught 10 of 10 injected regressions to Playwright MCP's 9 and DevTools' 8, averaging 815 tokens per look against 1,292 and 758. See Part 5b.
|
|
2
12
|
|
|
3
|
-
|
|
13
|
+
Where Reticle loses is stated plainly in Part 5, and it loses in more places than these two tables show.
|
|
14
|
+
|
|
15
|
+
> This page assumes **zero** testing background. By the end you'll understand what software testing is, why AI coding agents made it urgent, how to tell a good verification tool from a bad one, and exactly how Reticle measures up against the main alternatives, including the places Reticle loses. If a term looks like jargon, it's defined the first time it appears.
|
|
4
16
|
>
|
|
5
|
-
> _Where the pictures come from, stated accurately: the **SVG charts** (`bench/artifacts/`) are generated from measured raws by `bench/harness/charts.mjs`. The **headline PNG cards** (`assets/readme/`) are screenshots of hand-authored HTML in `assets/benchmarks/src/`, rendered by `assets/benchmarks/render.mjs
|
|
17
|
+
> _Where the pictures come from, stated accurately: the **SVG charts** (`bench/artifacts/`) are generated from measured raws by `bench/harness/charts.mjs`. The **headline PNG cards** (`assets/readme/`) are screenshots of hand-authored HTML in `assets/benchmarks/src/`, rendered by `assets/benchmarks/render.mjs`; their figures are typed in by a human and are **not** read from `history.jsonl`, which this line previously claimed. Treat a card as a design asset that has to be updated by hand when a number moves, and `bench/SCORECARD.md` (plus its freshness banner) as the source of truth._
|
|
6
18
|
|
|
7
19
|
---
|
|
8
20
|
|
|
9
|
-
## Part 1
|
|
21
|
+
## Part 1: The problem, from first principles
|
|
10
22
|
|
|
11
23
|
### What is "testing"?
|
|
12
24
|
|
|
@@ -14,69 +26,71 @@ When you build software, **testing** is the act of checking that it actually doe
|
|
|
14
26
|
|
|
15
27
|
There are roughly three ways teams do this:
|
|
16
28
|
|
|
17
|
-
1. **Manual testing
|
|
18
|
-
2. **Automated tests
|
|
19
|
-
3. **Nothing
|
|
29
|
+
1. **Manual testing.** A human clicks through the app. Accurate, but slow and easy to skip.
|
|
30
|
+
2. **Automated tests**: code that clicks through the app for you. Fast to re-run, but expensive to write and notoriously **flaky** (they break for reasons unrelated to real bugs).
|
|
31
|
+
3. **Nothing.** Ship and hope. More common than anyone admits: surveys put ~44% of teams with no automated testing at all.
|
|
20
32
|
|
|
21
|
-
A **regression** is the specific bug this page cares about: something that _used to work_ and quietly _stopped working_ after a change. Regressions are insidious because the feature looked done
|
|
33
|
+
A **regression** is the specific bug this page cares about: something that _used to work_ and quietly _stopped working_ after a change. Regressions are insidious because the feature looked done, right up until a later edit broke it and nobody noticed.
|
|
22
34
|
|
|
23
35
|
### Why AI agents made this urgent
|
|
24
36
|
|
|
25
|
-
AI coding agents now write a large share of new code. They're good at _producing_ code. They are bad at one specific thing: **knowing whether the code they wrote actually works.** The agent edits files, says _"done ✅"_, and moves on. It never opened the browser. So **you** become its QA department
|
|
37
|
+
AI coding agents now write a large share of new code. They're good at _producing_ code. They are bad at one specific thing: **knowing whether the code they wrote actually works.** The agent edits files, says _"done ✅"_, and moves on. It never opened the browser. So **you** become its QA department, clicking around to discover that "done" wasn't done. Developers describe being _"gaslit by coding assistants."_ This is the **"done lie."**
|
|
26
38
|
|
|
27
|
-
The honest fix is to give the agent a way to _check its own work_
|
|
39
|
+
The honest fix is to give the agent a way to _check its own work_: to drive the real running app and confirm the feature actually happened. That's what Reticle does. This page is about proving it does it **well**, not just that it does it.
|
|
28
40
|
|
|
29
41
|
---
|
|
30
42
|
|
|
31
|
-
## Part 2
|
|
43
|
+
## Part 2: How would you even measure "good"?
|
|
32
44
|
|
|
33
45
|
Imagine three verification tools. How do you decide which is best? Two things matter, and they pull against each other:
|
|
34
46
|
|
|
35
|
-
- **Coverage
|
|
36
|
-
- **Cost
|
|
47
|
+
- **Coverage.** Does it actually _catch the bug_? A tool that misses regressions is useless no matter how cheap it is.
|
|
48
|
+
- **Cost.** How much does it make the AI _read_ to do its job? Every observation is fed into the model's limited context as **tokens** (the unit models read/write in). A tool that floods the context with thousands of tokens per look is slow, expensive, and crowds out the actual work.
|
|
37
49
|
|
|
38
50
|
A tool can cheat either axis: be cheap by looking at almost nothing (and miss bugs), or be thorough by dumping everything (and blow the budget). So the real metric has to reward **both at once**, with coverage as a hard floor.
|
|
39
51
|
|
|
40
52
|
That gives us our headline number:
|
|
41
53
|
|
|
42
|
-
> **Verification Efficiency = real regressions caught per 1,000 tokens spent looking
|
|
54
|
+
> **Verification Efficiency = real regressions caught per 1,000 tokens spent looking**, and it only counts once the tool catches **100% of the bugs** with **zero false alarms** on a known-good control. (Historically abbreviated "VE"; the catch-rate floor was "RCR".)
|
|
43
55
|
|
|
44
|
-
A **false positive** (or "false alarm") is crying wolf
|
|
56
|
+
A **false positive** (or "false alarm") is crying wolf: flagging a bug when nothing is wrong. We require zero, because a tool you can't trust when it's quiet is a tool you'll learn to ignore.
|
|
45
57
|
|
|
46
|
-

|
|
47
59
|
|
|
48
60
|
---
|
|
49
61
|
|
|
50
|
-
## Part 3
|
|
62
|
+
## Part 3: The competitors, and why these three
|
|
51
63
|
|
|
52
64
|
We compare Reticle against the two most credible agent-native browser tools, because they're what an AI agent would otherwise reach for:
|
|
53
65
|
|
|
54
|
-
- **Playwright MCP
|
|
55
|
-
- **Chrome DevTools MCP**
|
|
56
|
-
- **(Baseline) screenshot agents
|
|
66
|
+
- **Playwright MCP**: Microsoft's tool that lets an agent drive a browser via the accessibility tree (not screenshots). The closest "serious" alternative.
|
|
67
|
+
- **Chrome DevTools MCP** is Google's tool exposing Chrome's DevTools to an agent.
|
|
68
|
+
- **(Baseline) screenshot agents**, the common "let the model look at a picture" approach.
|
|
69
|
+
|
|
70
|
+
Why not compare to traditional test frameworks (Playwright the library, Cypress, etc.) on the token passes? Because those are written and maintained by humans; they don't answer "can an _agent_ verify its own work in the loop," and a hand-written script has no token cost to compare. The three above do, so the cost comparison stays apples-to-apples.
|
|
57
71
|
|
|
58
|
-
|
|
72
|
+
There **is** a comparison against a Playwright script, and it answers a different question: not "what does it cost an agent to look" but "what can each tool structurally SEE, at all." That one is in Part 5a. It runs no model, so nothing about it depends on how well an agent drives, which is what makes it the more durable of the two results.
|
|
59
73
|
|
|
60
74
|
Tool versions are pinned in `bench/raw/run-meta.json` so any run is reproducible.
|
|
61
75
|
|
|
62
76
|
---
|
|
63
77
|
|
|
64
|
-
## Part 4
|
|
78
|
+
## Part 4: The three measurement passes
|
|
65
79
|
|
|
66
80
|
Different questions need different rigs. We run three (kept in the raw files as "Layer A/B/C"):
|
|
67
81
|
|
|
68
|
-
1. **Observation-cost pass** (no AI model involved). For each bug scenario we run each tool's natural recipe and measure the exact size of what it returns
|
|
82
|
+
1. **Observation-cost pass** (no AI model involved). For each bug scenario we run each tool's natural recipe and measure the exact size of what it returns: characters, bytes, and a tokenizer count. No model means no randomness, so the token cost of _looking_ is measured precisely and repeatably.
|
|
69
83
|
2. **Full-agent-loop pass** (a real model drives). Here an actual AI agent uses each tool end-to-end, and we record the _authoritative_ token usage the model reports. This is the real-world number; it needs an API key, so it's run periodically rather than on every change.
|
|
70
|
-
3. **Replay pass** (no model). Reticle can record a flow once and _replay_ it deterministically with no AI at all. This measures the cost of re-checking a known flow
|
|
84
|
+
3. **Replay pass** (no model). Reticle can record a flow once and _replay_ it deterministically with no AI at all. This measures the cost of re-checking a known flow, the thing a test suite does over and over, every commit.
|
|
71
85
|
|
|
72
86
|
### The scenarios
|
|
73
87
|
|
|
74
|
-
We inject **10 realistic regressions** into a demo app plus **one no-bug control** (to catch false alarms). The bugs span the failure modes that matter
|
|
88
|
+
We inject **10 realistic regressions** into a demo app plus **one no-bug control** (to catch false alarms). The bugs span the failure modes that matter, and deliberately include ones that are _invisible to a screenshot_:
|
|
75
89
|
|
|
76
90
|
| Scenario | The bug | Why it's here |
|
|
77
91
|
| --- | --- | --- |
|
|
78
92
|
| Hidden API 500 | A request silently returns a server error | A screenshot can't see a failed network call |
|
|
79
|
-
| Wrong status 404 | A request 404s | Same
|
|
93
|
+
| Wrong status 404 | A request 404s | Same, non-visual |
|
|
80
94
|
| CORS blocked | A request is blocked by the browser | Non-visual |
|
|
81
95
|
| Silent DOM removal | A KPI card vanishes with no error | Tests "did content disappear?" |
|
|
82
96
|
| Route break | Navigation doesn't change the page | Common SPA bug |
|
|
@@ -87,17 +101,55 @@ We inject **10 realistic regressions** into a demo app plus **one no-bug control
|
|
|
87
101
|
| Network timeout | A request hangs forever | A request that never resolves never "logs" |
|
|
88
102
|
| **Control (no bug)** | Nothing is wrong | Anything flagged here is a false positive |
|
|
89
103
|
|
|
90
|
-
Three of these (failed request, console error, hung request) are **categorically invisible to a screenshot
|
|
104
|
+
Three of these (failed request, console error, hung request) are **categorically invisible to a screenshot**. No number of pictures will ever catch them. That's a core part of the story.
|
|
91
105
|
|
|
92
106
|
---
|
|
93
107
|
|
|
94
|
-
## Part 5
|
|
108
|
+
## Part 5: The honest results, on two very different apps
|
|
109
|
+
|
|
110
|
+
We ran this two ways on purpose: a **controlled toy app** (where we can inject exact bugs and measure detection) and a **real production app** (where the numbers are messy and honest). The story holds in both, and the real app is where it gets interesting.
|
|
111
|
+
|
|
112
|
+

|
|
113
|
+
|
|
114
|
+
### 5a. The 88-bug head-to-head against a Playwright script
|
|
115
|
+
|
|
116
|
+
The largest and least model-dependent measurement here. An **88-bug registry** (`bench/pw-vs-reticle/bugs.mjs`), each bug injected into the same app, each tool running its NATIVE check: a Playwright script doing what a Playwright script does, Reticle doing what Reticle does. **No LLM in the loop**, so nothing depends on how well an agent happened to drive.
|
|
117
|
+
|
|
118
|
+
A tool "catches" a bug when its check correctly FAILS on the broken build **and** does not fail on the clean one. Failing on the clean build is a false alarm and is counted against the tool, not ignored.
|
|
119
|
+
|
|
120
|
+
| | Reticle | Playwright script |
|
|
121
|
+
| -------------------------------------- | ----------: | ----------------: |
|
|
122
|
+
| Bugs caught | **85 / 88** | 59 / 88 |
|
|
123
|
+
| Of what it can structurally catch | **85 / 86** | 57 / 60 |
|
|
124
|
+
| **False greens** (broken, reported OK) | **1** | **29** |
|
|
125
|
+
| False alarms on the clean build | 0 | 0 |
|
|
126
|
+
| Output bytes per bug | 9,261 | **5,849** |
|
|
127
|
+
|
|
128
|
+
Three things about that table are more important than the headline number.
|
|
129
|
+
|
|
130
|
+
**Two of Reticle's three non-catches are not misses.** They are the registry's `false-positive-trap` cases, builds that look broken and are not: flagging them would itself be a false alarm, so 0 of 2 is the correct score and a tool that "caught" them would be worse. The third, `iframe-stale-data` (deep-dom), is a **genuine miss** and is counted as a false green above rather than explained away.
|
|
131
|
+
|
|
132
|
+
**These figures were re-derived on the 2.9.0 branch and they moved.** The numbers first published here were recorded on 2026-07-24; 615 commits touched the harness, the fixture app, the SDK or the server before anybody re-ran it. Detection moved 86 → 85 and 60 → 59, and the per-bug output figure **inverted**: Reticle was published as the leaner of the two at 4,134 B against 7,899 B and measures 9,261 B against 5,849 B here. Reticle's own check being the more expensive one does not contradict the token-cost sections below. Those measure an AGENT-driven look against MCP tools, which is a different question. It does mean the earlier claim should not be repeated. Re-run `bench/pw-vs-reticle/run.mjs` before quoting any of this.
|
|
133
|
+
|
|
134
|
+
**Reticle does not win everything, and the ties are the honest part.** On everything observable from outside the app the two are exactly level: console 6/6 against 6/6, network 8/8 against 8/8, storage 5/5 against 5/5, visual UI 16/16 against 16/16, plus routing, timing, paint and chart bugs. Any evaluate-capable tool matches Reticle on what the DOM, the network buffer or a screenshot can show.
|
|
135
|
+
|
|
136
|
+
**The gap is one specific shape:** bugs where the screen is right and the app is wrong.
|
|
137
|
+
|
|
138
|
+
| Category | Reticle | Playwright | The bug |
|
|
139
|
+
| --- | --: | --: | --- |
|
|
140
|
+
| state | 8/8 | **0/8** | the UI renders a plausible value that contradicts the app's own store |
|
|
141
|
+
| business-logic | 6/6 | **0/6** | an action corrupts a field nothing renders. The record is wrong, the screen is fine |
|
|
142
|
+
| signal | 4/4 | **0/4** | an event the app emits about itself (hydration done, error boundary caught) |
|
|
143
|
+
| net-status | 4/4 | 1/4 | a swallowed 4xx/5xx. The request failed, a catch block ate it, the UI rendered |
|
|
144
|
+
| streams | 3/3 | 0/3 | an SSE or WebSocket frame anomaly a request-level view cannot see |
|
|
145
|
+
| perf | 3/3 | 1/3 | layout shift, or a render storm where the DOM is identical |
|
|
146
|
+
| deep-dom | 2/3 | 2/3 | a break deep in a subtree a snapshot elides. The one both now miss is `iframe-stale-data` |
|
|
95
147
|
|
|
96
|
-
|
|
148
|
+
That is the whole claim, stated as narrowly as it deserves: Reticle sees inside the running app, so it catches the class of bug where the outside looks correct. Outside that class it ties, and it says so.
|
|
97
149
|
|
|
98
|
-
|
|
150
|
+
Bug registry: `bench/pw-vs-reticle/bugs.mjs`. Runner: `bench/pw-vs-reticle/run.mjs`, which regenerates `results.json` locally (run artifacts are not committed). Scorecard: `bench/FALSE-GREEN-SCORECARD.md`.
|
|
99
151
|
|
|
100
|
-
###
|
|
152
|
+
### 5b. The controlled toy app (the demo)
|
|
101
153
|
|
|
102
154
|
Measured on the observation-cost pass (numbers regenerate from `bench/raw/`):
|
|
103
155
|
|
|
@@ -108,117 +160,117 @@ Measured on the observation-cost pass (numbers regenerate from `bench/raw/`):
|
|
|
108
160
|
| Playwright MCP | 9 / 10 | 0.91 | 1,292 | 6.97 |
|
|
109
161
|
|
|
110
162
|
- **Reticle is the only tool that caught every regression**, with zero false alarms on the control.
|
|
111
|
-
- DevTools is a hair cheaper per look (758 vs 815 tokens)
|
|
163
|
+
- DevTools is a hair cheaper per look (758 vs 815 tokens), **but it's cheaper because it catches less.** On the metric that combines both (Verification Efficiency), Reticle leads at 12.27 vs 10.55.
|
|
112
164
|
- Playwright catches a lot but is ~1.6× more expensive per look, so its efficiency is lowest here.
|
|
113
165
|
|
|
114
|
-
###
|
|
166
|
+
### 5c. A real production app (the Reticle dashboard)
|
|
115
167
|
|
|
116
|
-
A toy app is a fair lab, but it's small. The harder, more honest test is a **real, complex app
|
|
168
|
+
A toy app is a fair lab, but it's small. The harder, more honest test is a **real, complex app**. Take the [Reticle](https://reticle.sh) dashboard itself: React 19, authentication, live data, ~15 routes, a node-graph view, virtualized lists. We embedded the SDK (the Vite plugin) and drove the authenticated app with all three tools. Observing it **once** (the primary snapshot + the network log):
|
|
117
169
|
|
|
118
|
-

|
|
119
171
|
|
|
120
172
|
| Tool | Snapshot | Network | **Observe total** | Can it assert success? |
|
|
121
173
|
| --- | --- | --- | --- | --- |
|
|
122
|
-
| **Reticle** | 678 | 345 | **1,023** | ✅ via the app's own `auth:logged-in` signal
|
|
123
|
-
| Chrome DevTools MCP | 1,105 | 252 | 1,357 | ❌ DOM/network only
|
|
174
|
+
| **Reticle** | 678 | 345 | **1,023** | ✅ via the app's own `auth:logged-in` signal (**46 tok, un-fakeable**) |
|
|
175
|
+
| Chrome DevTools MCP | 1,105 | 252 | 1,357 | ❌ DOM/network only; can describe, can't verify intent |
|
|
124
176
|
| Playwright MCP | 1,522 | 671 | 2,193 | ❌ DOM/network only |
|
|
125
177
|
|
|
126
|
-
On a small page everything is cheap and the gap is modest. On a **big** page the structured read pulls ahead: Reticle is **2.1× leaner than Playwright MCP** and the cheapest overall. And only Reticle can read the app's own program state (`authenticated`, `userId`, `activeProjectId`) and assert the login _actually worked_ from a signal the app emits
|
|
178
|
+
On a small page everything is cheap and the gap is modest. On a **big** page the structured read pulls ahead: Reticle is **2.1× leaner than Playwright MCP** and the cheapest overall. And only Reticle can read the app's own program state (`authenticated`, `userId`, `activeProjectId`) and assert the login _actually worked_ from a signal the app emits. The others can only look at the DOM and the network and **guess**.
|
|
127
179
|
|
|
128
|
-
###
|
|
180
|
+
### 5d. The kicker: a real bug, caught live
|
|
129
181
|
|
|
130
|
-
Here's what makes the real-app test matter. On the **very first run, before we instrumented anything**, Reticle's network observation flagged two endpoints returning **`500
|
|
182
|
+
Here's what makes the real-app test matter. On the **very first run, before we instrumented anything**, Reticle's network observation flagged two endpoints returning **`500`**: `GET /api/v1/projects` and `/recovery/incidents`, both failing with `column "deleted_at" does not exist` (a database migration that hadn't been applied). **The page rendered fine.** The sidebar loaded, nothing looked broken; a screenshot agent would have called it _"done."_ Reticle saw the broken backend underneath, in one look.
|
|
131
183
|
|
|
132
184
|
That is the whole thesis, demonstrated on a real app we didn't cherry-pick: **"looks done" ≠ "is done," and the difference is usually non-visual.** (We then fixed the migration; all endpoints 200.)
|
|
133
185
|
|
|
134
|
-
###
|
|
186
|
+
### 5e. What each tool can actually do
|
|
135
187
|
|
|
136
|
-
Cost is half the story; capability is the other half. The marks below are what's **first-class and built-in** to each tool (all three are good tools
|
|
188
|
+
Cost is half the story; capability is the other half. The marks below are what's **first-class and built-in** to each tool (all three are good tools, and they compose: drive with theirs, assert with Reticle):
|
|
137
189
|
|
|
138
|
-

|
|
139
191
|
|
|
140
192
|
### Where Reticle loses (stated plainly)
|
|
141
193
|
|
|
142
194
|
Honesty is the point of this page, so here are the places Reticle does **not** win:
|
|
143
195
|
|
|
144
196
|
- **Raw cheapness on a single trivial look:** DevTools can be a few percent cheaper per observation when it isn't catching the harder bugs. We trade those tokens for catching more.
|
|
145
|
-
- **True pixels:** a screenshot is the _actual rendered frame_. A bug that only shows up in real paint (a font that failed to load, a GPU/compositing glitch) can be caught by a screenshot diff and missed by Reticle's structural reads
|
|
197
|
+
- **True pixels:** a screenshot is the _actual rendered frame_. A bug that only shows up in real paint (a font that failed to load, a GPU/compositing glitch) can be caught by a screenshot diff and missed by Reticle's structural reads, _unless_ Reticle is explicitly asked to do a visual diff. We're honest that structure isn't pixels.
|
|
146
198
|
- **The full-agent-loop pass** is measured for the three tools but a screenshot-agent variant is still future work; we don't claim a head-to-head there we haven't run.
|
|
147
199
|
|
|
148
200
|
### The part that compounds: re-running
|
|
149
201
|
|
|
150
|
-
The numbers above are for **one** verification. But a test suite's real job is the **same** check, over and over
|
|
202
|
+
The numbers above are for **one** verification. But a test suite's real job is the **same** check, over and over, every commit, every CI run. Here the picture changes shape:
|
|
151
203
|
|
|
152
|
-
- Reticle records a flow once, then **replays it with no AI model at all**
|
|
204
|
+
- Reticle records a flow once, then **replays it with no AI model at all** (re-resolving each element and re-asserting the outcome) for **~239 tokens per run** (re-measured 2026-08-11; it was ~175 before each flow started carrying a success oracle).
|
|
153
205
|
- Playwright MCP and DevTools MCP have no replay: re-checking means an agent **re-drives the whole flow with the model every time**, costing tens of thousands of tokens per run.
|
|
154
206
|
|
|
155
|
-
That's a **~
|
|
207
|
+
That's a **~127× cost difference per re-run**, and it grows with how often you run. This is where the "much cheaper than screenshots" claim becomes dramatic rather than incremental.
|
|
156
208
|
|
|
157
|
-

|
|
158
210
|
|
|
159
211
|
### The large-page test (where the wedge is biggest)
|
|
160
212
|
|
|
161
213
|
On a small page, structured reads and screenshots are both cheap, so the gap is modest. The advantage shows on a **big** page. On a deliberately large grid (thousands of DOM nodes):
|
|
162
214
|
|
|
163
215
|
- A **full page snapshot** costs ~**3,636 tokens**.
|
|
164
|
-
- A **targeted verify loop** (find one button → act → assert the success signal) costs ~**279 tokens
|
|
216
|
+
- A **targeted verify loop** (find one button → act → assert the success signal) costs ~**279 tokens**, and stays flat as the page grows to 5,000 rows.
|
|
165
217
|
|
|
166
218
|
That's a **13× difference**, and against a screenshot agent it's larger still: one screenshot at a normal window size is ~**1,365 image tokens** _per look_, a real loop takes several looks, and the screenshot still can't see the non-visual bugs.
|
|
167
219
|
|
|
168
220
|
### Many agents at once (the parallel wedge)
|
|
169
221
|
|
|
170
|
-
A fleet of agents (or a parallel suite) verifying the same app doesn't need a browser each. Reticle keeps **one** headless Chromium and leases each agent an **isolated context** (separate cookies/storage/DOM). Measured on 16 flows: **35.4s** one-at-a-time vs **5.2s** across 8 leased contexts
|
|
222
|
+
A fleet of agents (or a parallel suite) verifying the same app doesn't need a browser each. Reticle keeps **one** headless Chromium and leases each agent an **isolated context** (separate cookies/storage/DOM). Measured on 16 flows: **35.4s** one-at-a-time vs **5.2s** across 8 leased contexts, so **6.78× faster**, ~30s saved per batch, 8-way peak concurrency. The alternative, a browser per agent, costs hundreds of MB and seconds of startup each; the pool's edge grows with agent count up to its cap (`multi-agent-throughput`).
|
|
171
223
|
|
|
172
224
|
---
|
|
173
225
|
|
|
174
|
-
## Part 6
|
|
226
|
+
## Part 6: Why we believe these numbers (fairness + reproducibility)
|
|
175
227
|
|
|
176
228
|
- **Every number comes from a committed harness**, not a slide. Re-run it: `pnpm bench`.
|
|
177
229
|
- **A no-bug control** runs in every pass, so a tool can't look good by flagging everything.
|
|
178
230
|
- **Tool versions are pinned**; the raw payloads are saved to `bench/raw/` so anyone can audit them.
|
|
179
|
-
- **The token counter is a tokenizer**, not a guess
|
|
231
|
+
- **The token counter is a tokenizer**, not a guess. Where we use a proxy tokenizer instead of a specific model's, we say so and lean on _relative_ differences, which are robust to the choice.
|
|
180
232
|
- **We publish where we lose** (above). A benchmark that only flatters its author isn't a benchmark.
|
|
181
233
|
|
|
182
234
|
---
|
|
183
235
|
|
|
184
|
-
## Part 7
|
|
236
|
+
## Part 7: What this means for you
|
|
185
237
|
|
|
186
|
-
- **If you're a developer using an AI agent:** Reticle lets the agent confirm its own change actually worked
|
|
187
|
-
- **If you run CI / a platform team:** the replay pass is the headline
|
|
238
|
+
- **If you're a developer using an AI agent:** Reticle lets the agent confirm its own change actually worked (across the network call, the console, the route, the state) for a few hundred tokens, not a screenshot's thousands. "Done" starts meaning done.
|
|
239
|
+
- **If you run CI / a platform team:** the replay pass is the headline: deterministic re-verification with no model, so a known flow re-checks for a couple of hundred tokens at ~0% flake, every commit.
|
|
188
240
|
- **If you're evaluating tools:** the axis that matters is _bugs caught per token, gated on catching them all_. Cheapness that misses regressions is a false economy.
|
|
189
241
|
|
|
190
|
-
You are now equipped to read any verification benchmark critically: ask "what's the catch rate, what's the false-positive rate, and what's the cost _per re-run_?"
|
|
242
|
+
You are now equipped to read any verification benchmark critically: ask "what's the catch rate, what's the false-positive rate, and what's the cost _per re-run_?" Then be suspicious of any vendor that won't show you where they lose.
|
|
191
243
|
|
|
192
244
|
---
|
|
193
245
|
|
|
194
|
-
## Part 8
|
|
246
|
+
## Part 8: At enterprise scale (the honest "if a big company used this")
|
|
195
247
|
|
|
196
|
-
A fair question from a large engineering org (think a Datadog- or Salesforce-sized team): _does this actually pay off at our scale, or only in a demo?_ Here's the honest accounting, with every assumption stated so you can re-run it with your own numbers. **These are projections from the measured per-run costs above, not a measured enterprise deployment
|
|
248
|
+
A fair question from a large engineering org (think a Datadog- or Salesforce-sized team): _does this actually pay off at our scale, or only in a demo?_ Here's the honest accounting, with every assumption stated so you can re-run it with your own numbers. **These are projections from the measured per-run costs above, not a measured enterprise deployment; they are labeled as such.**
|
|
197
249
|
|
|
198
250
|
### Where it pays off (and compounds)
|
|
199
251
|
|
|
200
252
|
The win is **regression re-runs**, because that cost is paid over and over. Take a mid-size surface:
|
|
201
253
|
|
|
202
254
|
- **50 golden flows**, re-verified on **every PR**, at **200 PRs/day**.
|
|
203
|
-
- Reticle replays each flow **deterministically, with no model**: ~**
|
|
255
|
+
- Reticle replays each flow **deterministically, with no model**: ~**240 tokens/flow/run** (measured).
|
|
204
256
|
- The agent-driven alternative re-drives each flow with an LLM: ~**30,000 tokens/flow/run** (measured Layer-C comparison).
|
|
205
257
|
|
|
206
258
|
| | Per flow / run | 50 flows × 200 PRs/day | Per year (~250 working days) |
|
|
207
259
|
| --- | --- | --- | --- |
|
|
208
|
-
| Reticle replay (no model) | ~
|
|
260
|
+
| Reticle replay (no model) | ~240 tok | ~2.4 M tok/day | ~0.6 B tok/yr |
|
|
209
261
|
| Agent re-drive (LLM) | ~30,000 tok | ~300 M tok/day | ~75 B tok/yr |
|
|
210
262
|
|
|
211
|
-
That's a **~
|
|
263
|
+
That's a **~125× difference on the recurring axis**, and it scales linearly with flows × runs, the two numbers an enterprise has _a lot_ of. The deterministic replay also means **~0% verdict flake** (measured over repeated identical runs), which at this scale is the difference between a trusted gate and one engineers learn to ignore (recall: flaky suites get abandoned; that's the documented failure mode Reticle is built to avoid). And because replay needs no model, it has **no per-run API spend and no rate-limit ceiling**. It runs in CI like any other deterministic check.
|
|
212
264
|
|
|
213
265
|
### Where it does NOT help (stated plainly)
|
|
214
266
|
|
|
215
267
|
An honest enterprise evaluation has to include the limits:
|
|
216
268
|
|
|
217
|
-
- **Authoring still costs model tokens.** Recording/annotating a flow the first time uses an agent. The economics only turn positive once a flow is re-run enough times to amortize that
|
|
218
|
-
- **Tier-1 oracles need instrumentation.** The strongest, un-fakeable assertions come from app signals the team adds (a ~30-second opt-in per success event). Without them you get Tier-2 (network/route/ state)
|
|
269
|
+
- **Authoring still costs model tokens.** Recording/annotating a flow the first time uses an agent. The economics only turn positive once a flow is re-run enough times to amortize that, which is exactly the enterprise case (many runs), but a flow you run twice won't pay off.
|
|
270
|
+
- **Tier-1 oracles need instrumentation.** The strongest, un-fakeable assertions come from app signals the team adds (a ~30-second opt-in per success event). Without them you get Tier-2 (network/route/ state): still strong, still better than "element exists," but not the magic tier. Be realistic about the instrumentation rollout across a large codebase.
|
|
219
271
|
- **It is not a full replacement for an existing E2E platform on day one.** Reticle is verification in the dev/agent loop and in CI; a large org with a mature Playwright/Cypress estate adopts it alongside, starting with the agent loop and the highest-value journeys, not as a big-bang migration.
|
|
220
272
|
- **Structure is not pixels.** As in Part 5: a purely visual regression (a paint/font/compositing bug) needs the opt-in visual diff; the always-on structured reads won't catch it.
|
|
221
|
-
- **Per-project isolation is a setup step.** Multiple apps/teams on one machine each want their own bridge port (`RETICLE_PORT`)
|
|
273
|
+
- **Per-project isolation is a setup step.** Multiple apps/teams on one machine each want their own bridge port (`RETICLE_PORT`). Trivial, but it's configuration a platform team should standardize.
|
|
222
274
|
|
|
223
275
|
### The honest bottom line for a large org
|
|
224
276
|
|
|
@@ -226,21 +278,21 @@ Reticle is **most efficient exactly where big companies hurt most**: a large num
|
|
|
226
278
|
|
|
227
279
|
---
|
|
228
280
|
|
|
229
|
-
### Appendix
|
|
281
|
+
### Appendix: the raw artifacts
|
|
230
282
|
|
|
231
|
-
- `bench/SCORECARD.md
|
|
232
|
-
- `bench/METRIC.md`
|
|
233
|
-
- `bench/METHODOLOGY.md
|
|
234
|
-
- `bench/history.jsonl
|
|
235
|
-
- `bench/raw/`
|
|
283
|
+
- `bench/SCORECARD.md`: the one-page standing with the plain-language legend.
|
|
284
|
+
- `bench/METRIC.md` holds the exact metric definitions.
|
|
285
|
+
- `bench/METHODOLOGY.md`: full design (controls, scenarios, fairness).
|
|
286
|
+
- `bench/history.jsonl`: every measured run over time.
|
|
287
|
+
- `bench/raw/` holds raw payloads and per-scenario results (gitignored locally; regenerated by `pnpm bench`).
|
|
236
288
|
|
|
237
289
|
---
|
|
238
290
|
|
|
239
|
-
## SDK overhead
|
|
291
|
+
## SDK overhead: what instrumentation costs the app
|
|
240
292
|
|
|
241
|
-
An observability layer that slows the app corrupts its own performance verdicts, so Reticle holds itself to a budget: **total instrumentation overhead below 3% of main-thread time**, measured on the _hostile_ fixture
|
|
293
|
+
An observability layer that slows the app corrupts its own performance verdicts, so Reticle holds itself to a budget: **total instrumentation overhead below 3% of main-thread time**, measured on the _hostile_ fixture, a page that never goes quiet (a ~10 messages/sec feed, a 60fps ticker, a large list), not a polite demo.
|
|
242
294
|
|
|
243
|
-
The same page is loaded in one browser under three conditions
|
|
295
|
+
The same page is loaded in one browser under three conditions (full SDK, observers with the HUD suppressed, and Reticle absent), and Chrome's own cumulative `TaskDuration` is compared over an identical window (condition order rotated so warm-up can't favour any one of them). Three conditions, not two, so that what instrumenting costs is separable from what the HUD costs.
|
|
244
296
|
|
|
245
297
|
| | main-thread task time | busy |
|
|
246
298
|
| ----------------------------- | --------------------- | ------------------------------ |
|
|
@@ -250,6 +302,6 @@ The same page is loaded in one browser under three conditions — full SDK, obse
|
|
|
250
302
|
| **instrumentation** | **+0.21 pp** | below the ±1.23 pp noise floor |
|
|
251
303
|
| **presenter HUD** | **+0.53 pp** | opt out with `present: false` |
|
|
252
304
|
|
|
253
|
-
**Result: instrumentation overhead is smaller than the method can resolve
|
|
305
|
+
**Result: instrumentation overhead is smaller than the method can resolve. Report it as `< 1.2 percentage points`, inside the 3% budget.** The HUD is measured and reported separately, because it is a developer-visible affordance you can turn off, not a cost of observing.
|
|
254
306
|
|
|
255
|
-
Getting here required correcting the method and then fixing a real bug. The two-condition version of this bench compared the full SDK against Reticle being absent, so it charged instrumentation for the HUD too, and reported **+5.85 pp
|
|
307
|
+
Getting here required correcting the method and then fixing a real bug. The two-condition version of this bench compared the full SDK against Reticle being absent, so it charged instrumentation for the HUD too, and reported **+5.85 pp (FAIL)**. Decomposing it showed almost none of that was instrumentation: a single `backdrop-filter: blur(24px)` on the HUD panel was costing **+4.03 pp** on its own, more than every observer combined, because a backdrop blur re-samples everything behind it whenever that content changes, and this fixture repaints at 60fps. Removing it took total main-thread time down 22%. Full history, including a fix that measured nothing and was reverted rather than kept, is in `bench/overhead/README.md`. Reproduce it yourself with `node bench/overhead/measure.mjs` (see `bench/overhead/README.md`, which records the ruled-out causes).
|